arXiv CSopen access

Reinforcement Learning with Robust Rubric Rewards

Technical Report on Rubric-Based Post-Training II: Reinforcement Learning

Reinforcement Learning with Robust Rubric Rewards

Ya-Qi Yu∗,†B , Hao Wang∗ , Fangyu Hong∗ , Xiangyang Qu∗ , Gaojie Wu, Qiaoyu Luo, Nuo Xu, Huixin Wang, Wuheng Xu, Yongxin Liao, Zihao Chen, Haonan Li, Ziming Li, Dezhi Peng, Minghui Liao, Jihao Wu, Haoyu Ren, Dandan Tu

arXiv:2605.30244v1 [cs.CV] 28 May 2026

∗

Core Contributors

†

Project Leader

Huawei Technologies Co., Ltd.

Abstract While Reinforcement Learning with Verifiable Rewards (RLVR) is effective for deterministically checkable tasks, many vision-language tasks are partially verifiable, demanding multi-criteria supervision (e.g., perceptual details, reasoning steps, and constraints). Rubrics provide a natural interface for this fine-grained supervision, but their effectiveness depends on the execution accuracy during online RL. We propose Reinforcement Learning with Robust Rubric Rewards (RLR3 ), extending RLVR from task-level verification to criterion-level verification. RLR3 routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria. To ensure faithful scoring, RLR3 introduce a minimal exposure strategy that masks ground truths from extractors and images from judges. Furthermore, RLR3 employs hierarchical aggregation to prioritize essential criteria over additional criteria, and mitigates score saturation within rollout groups. Evaluated on Qwen3-VL-30B-A3B across 15 benchmarks, RLR3 consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap. Controlled audits confirm our deterministic verification and minimal exposure significantly reduce exploitable false positives.

1

Introduction

Reinforcement Learning with Verifiable Rewards (RLVR) has become a practical post-training recipe because deterministic outcome checks provide unambiguous reward signals [1]. It has been effective in math and code, where correctness can be decided by exact verifiers [2, 3, 4], and has recently been extended to vision-language tasks such as OCR, counting, and grounding [5, 6, 7]. However, this paradigm is bounded by a task-level assumption: the behavior being optimized must be verifiable [8]. Many vision-language tasks fall into a partially verifiable regime. A final answer may be exactly checkable, while intermediate perceptual facts, reasoning steps, and instruction-following details also benefit from extra supervision. Scoring the whole task from a single verifiability perspective collapses these distinctions into a coarse reward. This motivates instance-specific rubrics, which turn ambiguous response quality assessment into concrete criteria [9, 10, 11, 12, 13, 14]. Rubrics are useful only if their criteria can be accurately scored in online RL. In offline evaluation, imperfect rubric execution merely adds noise to a fixed response set. In online RL, any systematic rubric mis-execution becomes an incentive for the policy. An intuitive optimization is to match each criterion with an appropriate execution path: verifiable criteria can be handled by prediction extraction followed by deterministic checking, while the others can be handled by semantic judgment. B

E-mail: [email protected]

Preprint.

Checklist-Style Rubric Essential

VLM Input

Verifiable

Additional

Reward

Teacher

Extractor

Rollouts Policy

1

Limited Exposure

Verifier

Fuzzy

2

…

N

Judge

Text-Only

Final Rewards

Aggregation 1

2

…

N

Credit

GRPO

Figure 1: Overview of RLR3 . Instance-specific rubrics turn response quality assessment into concrete criteria and criterion-level rewards. Verifiable criteria are routed to a text-only LLM-as-an-extractor followed by deterministic verification, while fuzzy criteria are routed to a text-only LLM-as-a-Judge. We propose Reinforcement Learning with Robust Rubric Rewards (RLR3 ), a framework that extends RLVR from task-level verification to criterion-level verification, as illustrated in Figure 1. Each criterion is routed either to a text-only LLM-as-an-extractor followed by a deterministic verifier or to a text-only LLM-as-a-Judge when deterministic checking is unavailable. Verifier targets are hidden from the extractor, and source images are hidden from both execution paths to prevent shortcuts. The execution routing is the core of RLR3 , which uses deterministic verification wherever possible while retaining a judge path for the remaining criteria. Furthermore, RLR3 remaps saturated criterion scores within rollout groups, applies hierarchical aggregation so that supplementary response cannot compensate for critical failures, and trains the Generative Reward Model (GenRM) with RLVR. By making verifiability a criterion-level property, RLR3 supports fully and partially verifiable tasks in a single GRPO loop. When every criterion is verifiable, RLR3 reduces to standard RLVR. On Qwen3-VL-30B-A3B, RLR3 improves the macro average over RLVR across 3 open-source training mixtures: 76.4 to 77.7 on ViRL, 76.4 to 78.1 on OpenMMR, and 77.4 to 78.2 on DeepVision. Reward model audits further show that deterministic verification and minimal exposure reduce false positives on failure responses without harming scoring accuracy, while the RLVR-trained GenRM reaches 95.0% criterion-level accuracy on the held-out reward model test set. Our contributions are summarized as follows: • We identify partially verifiable vision-language tasks as a natural setting for rubric-based RL. • We propose RLR3 , a robust rubric reward framework that routes verifiable criteria to extraction plus deterministic verification and fuzzy criteria to text-only judgment under minimal exposure. • We improve reward informativeness through score remapping and hierarchical aggregation for multi-criteria. We improve reward reliability by minimal exposure strategy and GenRM RLVR. • We validate the effectiveness of RLR3 and its components through comparisons with RLVR, GenRM reliability evaluation, and failure-mode audits.

2

Related Works

Reinforcement learning with verifiable rewards. Recent studies underscore the effectiveness of RLVR, which leverages deterministic verifiers to provide precise rewards [1]. For Large Language Models (LLMs), this paradigm has been applied to tasks such as mathematics and programming [2, 3, 4], which allow automated verification through exact matches or unit tests. Recent multimodal extensions apply this approach to OCR, counting, grounding, and other tasks with well-defined targets [5, 6, 7]. Although these developments highlight the transparency of RLVR, they also clarify that its utility depends fundamentally on the existence of verifiable ground truth [8]. Rubric-based evaluation and alignment. Rubric-based evaluation makes supervision more interpretable by decomposing quality into explicit criteria. In language tasks, expert rubrics have been used to evaluate complex capabilities such as open-ended generation, research replication, and high-stakes professional reasoning [15, 16, 17, 18, 19]. More recently, rubrics have also been used in post-training and alignment for non-verifiable LLM domains [9, 10, 11, 20, 21]. Recent studies also 2

investigate automatic rubric construction, including synthetic rubric generation and elicitation from pairwise comparisons [22, 23, 24, 25]. In multimodal settings, evaluation with fixed or input-specific rubrics has also been explored [13, 14]. For multimodal alignment, rubrics have been used both for offline visual preference construction [12] and for multimodal reward modeling [26].

3

Preliminaries

This section introduces the policy optimization setup used in our framework. We optimize the policy with Group Relative Policy Optimization (GRPO) [2, 27] under a strict on-policy setting without a KL penalty. For each input x, the current policy πθ samples a group of G responses {y1 , . . . , yG }. Each sampled response yi is assigned a final scalar reward r̃i . We also enforce a simple length rule throughout policy training: if a response exceeds the task-specific maximum response length, its final reward is set to 0. We maximize the following objective: " # |yi | G X X 1 πθ (yi,t | x, yi,<t ) (1) JGRPO (θ) = E PG Âi , sg(π (y | x, y )) θ i,t i,<t |y | i i=1 i=1 t=1 G where sg(·) denotes the stop-gradient operator and Âi = (r̃i − mean({r̃j }G j=1 ))/ std({r̃j }j=1 ) is the group-relative advantage. Section 4 defines how the final scalar reward r̃i is constructed, including the rubric, its execution paths, and the aggregation procedure used by GRPO.

4

Methodology

A fundamental challenge in online RL is that the policy tends to hack the reward during continuous optimization. To keep rubric rewards accurate and robust, RLR3 follows three principles. First, we prioritize verifiability to limit the space for exploitation. Second, we restrict evidence exposure to prevent unintended shortcuts. Third, we preserve the multi-reward distinctions during aggregation. Our method implements these principles across three components. Section 4.1 introduces a rubric schema that integrates deterministic verifiers with probabilistic models, making verifiability explicit and prior for each criterion. Section 4.2 defines specific criterion execution paths, exposing only the strictly necessary context to the model. Section 4.3 turns criterion-level scores into the final reward through decoupled normalization and hierarchical aggregation, while reliably handling critical rule violations such as repetitive generation and language inconsistency. 4.1

Rubric Design

For an input x, we define the rubric as C x = {cx1 , . . . , cxK }, where each criterion is represented as cxk = ⟨dxk , txk , wkx , Vk , zkx ⟩. Here, dxk is the criterion description, txk ∈ {E SSENTIAL, A DDITIONAL} is the criterion type, wkx is a non-negative weight, Vk is the verifier tag, and zkx is the associated reference object. The verifier tag determines the execution path of each criterion: • Verifiable criteria (Vk ̸= ∅) are used when the relevant content can be extracted from the response and checked deterministically against known targets. In this case, zkx stores the target arguments passed to the corresponding verifier function, for example when the response must provide an option letter or a specific time string. • Fuzzy criteria (Vk = ∅) are used when evaluation still requires language understanding and cannot be reduced to deterministic matching. In this case, zkx stores the textual reference used by the LLM-as-a-Judge, for example whether the response conveys the same meaning as the reference using different wording or follows an instance-specific instruction. We use a compact verifier library that covers common value types, including text, expression, time, list, bounding box, and point. During rubric drafting, the rubric generator is given only the verifier tags allowed for the current task, together with the intended use, input schema, and examples for each verifier. For example, OCR tasks expose the text verifier, while grounding tasks expose the bounding box or point verifier. This restriction prevents the rubric generator from abusing verifiers. It also rules out special cases such as rewriting a complex criterion as a Boolean judgment and then checking it with the expression verifier, for example, “the response correctly states that the man is wearing a red 3

hat and standing to the left of the bicycle.” This design enforces a simple boundary. A verifier should check a value extracted from the response, not a judgment produced by the extractor. Rubrics are generated from the input context, available references, and task metadata. We use a multi-teacher aggregation pipeline in which several frontier models independently propose candidate criteria and a second-stage aggregation module merges them into the final checklist. Appendix A.1 summarizes the rubric schema and Appendix B.1 summarizes the generation pipeline. 4.2

Criterion Execution

We introduce distinct execution paths for verifiable and fuzzy criteria. Both paths follow a minimal exposure principle. For verifiable criteria, if the target arguments zkx were visible, the extractor might copy the required values from them instead of extracting them from the response. We therefore withhold zkx from the extractor on the verifiable path. Another failure mode could arise if the source image were visible. The judge might infer the task answer even when the response does not state it. To avoid this shortcut, both execution paths adopt text-only LLM and share the same parameters ϕ. 4.2.1

Verifiable Criteria with LLM-as-an-Extractor

Let x = (xt , xi ), where the input x is decomposed into a text prompt xt and a source image xi . For verifiable criteria, the extractor generates criterion-level reasoning and an extracted value (ηk , âk ) = E(xt , y, dxk , Vk ; ϕ), where E(·; ϕ) uses the shared LLM parameters ϕ. The deterministic verifier then computes sk = Vk (âk , zkx ), where zkx stores the target arguments required by the verifier function. For a time verifier, for example, these arguments can include the target time string and its string format. Appendix C summarizes the verifier specifications used in our implementation. The extractor only sees the text prompt xt , the response y, the criterion description dxk , and the verifier tag Vk . Hence, it must identify the value from the response y rather than copying it from zkx . 4.2.2

Fuzzy Criteria with LLM-as-a-Judge

For fuzzy criteria, the judge predicts criterion-level reasoning together with a discrete credit value. In the reference-grounded setting, the judge receives the text prompt xt , response y, criterion description dxk , and textual reference zkx , excluding the source image, and predicts (ηk , sk ) = J(xt , y, dxk , zkx ; ϕ). Here, sk ∈ {0, 0.5, 1} corresponds to no credit, partial credit, and full credit, following the discrete rubric scoring scheme used in rDPO [12]. 4.2.3

Structured Outputs

These paths together produce a single JSON object containing a global reasoning field and a list of criterion records. Appendix A.2 gives the full schema. This schema makes reward execution easy to review and simplifies the interface between rubric construction, reward execution, and RL training. 4.3 4.3.1

Reward Aggregation Decoupled Normalization

x For an input x and a response group {yi }G i=1 , let sk,i be the raw score assigned to criterion ck on response yi . For many verifiable criteria, these scores concentrate in a narrow score range and therefore provide weak resolution for ranking responses within a group. For example, under editdistance similarity, responses with one mistake and ten mistakes can score both above 0.9. During aggregation, their contributions to group-wise ranking are nearly indistinguishable.

We therefore remap raw scores within each group before aggregation. The remapping should improve within-group resolution without changing whether the whole group lies below or above the threshold τ . Otherwise, a group in which every response fails an essential criterion could be artificially stretched to full credit by within-group normalization alone. Given a threshold τ , let sk,min = mini sk,i and sk,max = maxi sk,i . We define the group-wise lower bound ℓk and upper bound uk as:   0, sk,min < τ, 1, sk,max > τ, ℓk = uk = (2) 0.5, sk,min ≥ τ, 0.5, sk,max ≤ τ. 4

We then define   sk,min = sk,max > τ, uk , sk,min = sk,max ≤ τ, s̃k,i = ℓk ,   sk,i −sk,min (uk − ℓk ) + ℓk , otherwise, sk,max −sk,min

(3)

which increases within-group separability before criterion scores are aggregated into the final reward. 4.3.2

Hierarchical Aggregation

Let s̃k denote the normalized score of criterion cxk for a given response, and compute the base content PK reward as r = k=1 wkx s̃k . This weighted sum captures fine-grained differences across responses, but by itself it treats all gains as mutually compensatory. As a result, strong performance on additional criteria could offset failures on essential ones, even though the rubric is meant to prioritize the latter. We therefore gate the base reward by criterion type, so that additional criteria refine the score only after the essential criteria are satisfied. We adopt a consistent scoring convention, treating scores below 0.5 as failures and scores in [0.5, 1) as partial satisfaction. The corresponding content mask is defined by:  x 0, |{k : tk = E SSENTIAL, s̃k < 0.5}| ≥ 1, mcontent = 0, |{k : txk = E SSENTIAL, 0.5 ≤ s̃k < 1}| ≥ 2, (4)  1, otherwise. Some response-level violations, such as repetition loops or language mixing, are also handled as hard format constraints. We define a binary format mask mformat ∈ {0, 1}, which is set to 0 when any such violation is triggered and to 1 otherwise. The final reward is r̃ = mcontent · mformat · r. 4.4

Reinforcement Learning for the Generative Reward Model

Reward robustness depends not only on rubric design, but also on reliable rubric execution. Since moderately-sized LLMs often struggle with complex instructions and multi-field structured outputs, we train the GenRM with RLVR before using it in policy optimization. We build the training and validation data by sampling candidate responses and scoring each response with multiple frontier models, each of which independently executes the target rubric. For each criterion, we retain the median credit across teachers. For verifiable criteria, we additionally keep a single extracted value from a teacher output whose credit matches the retained median credit. This multi-teacher setup helps reduce noise from any single model’s execution. The GenRM is trained with two families of verifiable rewards: • Format reward, which returns 1 only when the output is valid JSON and all required fields have the expected types. Otherwise, it returns 0. • Content reward, which is averaged over all criteria in the response. For verifiable criteria, we check the extracted values. For fuzzy criteria, we check the predicted credit. We supervise only the deterministically checkable fields in the final structured output and leave the free-form reasoning fields unsupervised, so this stage can be trained with an RLVR objective. 4.5

Reinforcement Learning for the Policy Model

The preceding components together define a fine-grained reward for any sampled response. Section 4.1 specifies the rubric, Section 4.2 defines criterion execution, and Section 4.3 maps criterion-level scores to a final scalar reward. We use this reward to train the target policy with GRPO. For each input x, we sample a group of responses {yi }G i=1 , execute the rubric on each response, obtain the final reward r̃i , and normalize these rewards within the group as in Section 3. In this framework, both fully and partially verifiable tasks are optimized within the same GRPO loop, so alignment in RLR3 does not necessarily split into a dedicated RLVR stage and a separate general RL stage. RLVR as a special case. When all criteria in the rubric are verifiable, RLR3 reduces to RLVR. The final reward is then obtained by aggregating verifier scores alone. For a single criterion, r̃ = V (E(y), z x ), where E is a non-parametric extractor and V verifies the extracted value against z x . 5

Table 1: Statistics of the training corpora after preprocessing. “Raw” denotes the original training split, “De-dup.” denotes the de-duplicated split, and “Filtered” denotes the subsets retained by the “Any” and “Essential” filtering rules. The “Essential” filtering rule is adopted for policy training. “MCQ Ratio” reports the fraction of multiple-choice questions. “Initial Reward” is the average rubric score of base-model rollouts on all of the converted instances and on the filtered subset. Dataset

# Raw

# De-dup.

ViRL OpenMMR DeepVision

38,870 74,971 103,484

38,870 74,145 92,491

5

Experiments

5.1

Experimental Setup

# Filtered

MCQ Ratio

Initial Reward

Any

Essential

De-dup.

Converted

Filtered

Converted

Filtered

16,444 36,627 63,549

9,551 26,070 55,550

31.8% 22.1% 48.4%

8.8% 6.5% 13.2%

13.7% 8.4% 15.2%

0.8156 0.7397 0.4713

0.4249 0.3556 0.3147

Data. For a controlled comparison with RLVR, we use the training splits of three open-source corpora as shared training sources for both methods: ViRL [6], OpenMMR [28], and DeepVision [29]. We further convert a subset of multiple-choice questions into open-ended questions when a VLM judges that the question remains well-posed without the answer options and still admits a unique correct answer. We remove duplicates at the image-question level and pair each remaining instance with an instance-specific rubric containing both verifiable and fuzzy criteria, following Section 4.1. We then perform offline filtering by sampling 8 rollouts from the base model for each instance and retaining only examples on which at least one rollout receives no credit on any criterion or on an essential criterion, depending on the target subset. Table 1 summarizes the resulting training data. Training. Models are trained using the Adam optimizer with a constant learning rate of 1 × 10−6 , a weight decay of 0.01, β1 = 0.9, and β2 = 0.999. By default, the maximum prompt and response lengths are set to 2K and 6K tokens, respectively. For DeepVision, which incorporates visual puzzles and longer inputs, we expand these limits to 4K and 12K tokens. Training proceeds with a global batch size of 128 and 8 rollouts for 1,000 steps, with checkpoints saved every 100 steps. For each method and training mix, we report the saved checkpoint with the best macro-average performance across the 15 benchmarks. We use the same early-stopping protocol for both RLVR and RLR3 to limit late-stage overfitting, which is more pronounced for RLVR. Benchmarks. We evaluate on 15 public benchmarks spanning math, general VQA, counting, and document VQA: We-Math [30], DynaMath [31], MathVision [32], MathVerse [33], MathVista [34], MMMU-Pro [35], RealWorldQA [36], MMStar [37], SimpleVQA [38], CountBenchQA [39], InfoVQA [40], DocVQA [41], ChartQA [42], and CharXiv (DQ/RQ) [43]. We use the official test or testmini split for each benchmark. We adopt two prompt templates for multiple-choice and open-ended QA, both of which require the model to think step by step and place the final answer in the last boxed span of the response. We apply rule-based matching to fixed-form answers such as option letter, numeric, and formula, while using GPT-4o-mini as a judge for open-ended QA. 5.2

Main Results

We compare RLVR and RLR3 across the three training mixes, with the results reported in Table 2. RLR3 improves average performance over RLVR. Specifically, on the macro average, RLR3 improves over RLVR from 76.4 to 77.7 on ViRL, from 76.4 to 78.1 on OpenMMR, and from 77.4 to 78.2 on DeepVision. At the benchmark level, RLR3 attains higher scores than RLVR on most benchmarks across the three training mixes. Representative gains appear on We-Math, MathVision, MathVerse, MMMU-Pro, and CountBenchQA, with especially large margins on OpenMMR We-Math (67.3 → 74.3), DeepVision MathVerse (73.9 → 79.6), and ViRL CountBenchQA (87.3 → 93.2). On the benchmarks where RLR3 does not improve over RLVR, the gaps are usually small. RLR3 achieves larger post-training gains using only open-source data. Our evaluated base model reaches a macro average of 73.5, compared with 72.5 for the officially reported instruct results, a 6

Table 2: Performance comparison on visual benchmarks. In the RLR3 columns, green highlights improvement over RLVR and red highlights degradation. “Official” and “GPT-5 mini” denotes results from the Qwen3-VL technical report. “Base” denotes our own evaluation of the instruct checkpoint. Base

ViRL

OpenMMR

DeepVision

Official

GPT-5 mini

Benchmark

instruct

RLVR

RLR3

RLVR

RLR3

RLVR

RLR3

thinking

instruct

high

minimal

We-Math DynaMath MathVision MathVerse (mini) MathVista (mini)

56.8 74.8 63.1 71.7 79.8

68.3 74.1 61.4 72.1 82.3

70.4 78.4 63.4 76.6 83.7

67.3 78.7 63.1 75.5 83.5

74.3 78.0 66.1 76.9 83.9

72.9 80.0 65.5 73.9 83.0

73.6 76.4 68.8 79.6 82.8

70.0 80.1 65.7 79.6 81.9

56.9 73.4 60.2 70.2 80.1

70.2 81.4 71.9 78.8 79.1

51.4 71.3 46.6 36.5 59.6

MMMU-Pro RealWorldQA MMStar SimpleVQA

64.2 74.4 74.1 49.9

64.5 77.9 75.7 54.0

66.8 76.7 76.9 53.4

64.1 76.7 77.8 52.9

66.6 77.6 76.9 54.6

66.8 76.6 77.1 52.1

67.4 75.6 76.6 52.4

63.0 77.4 75.5 54.3

60.4 73.7 72.1 52.7

67.3 79.0 74.1 56.8

53.7 73.3 61.3 50.3

CountBenchQA

88.7

87.3

93.2

89.7

90.3

88.3

92.6

90.0

89.8

91.0

84.1

InfoVQA DocVQA ChartQA CharXiv (DQ) CharXiv (RQ)

86.7 94.5 88.6 81.7 53.4

89.1 95.1 90.6 90.5 62.4

89.3 95.1 91.1 89.8 61.4

87.3 94.3 90.6 85.3 58.4

90.2 95.2 90.6 87.5 62.7

90.2 94.9 91.0 88.2 61.2

89.5 95.0 90.3 91.3 61.8

85.6 95.5 89.4 86.9 56.6

81.8 95.0 86.8 85.5 48.9

77.6 90.5 57.5 89.4 68.6

72.8 90.6 57.8 78.6 48.9

Macro Average

73.5

76.4

77.7

76.4

78.1

77.4

78.2

76.8

72.5

75.5

62.5

Table 3: Statistics of the RM data. OpenMMR denotes the portion that does not overlap with ViRL. Split Train Test

Source Mix

# Inst. 18,251 1,000

# Criteria

Verifier Usage

ViRL

OpenMMR

DeepVision

Total

Per inst.

Inst.

Criteria

4,631 (25.4%) 274 (27.4%)

4,701 (25.8%) 248 (24.8%)

8,919 (48.9%) 478 (47.8%)

58,224 3,148

3.19 3.15

15,575 (85.3%) 855 (85.5%)

35,572 (61.1%) 1,910 (60.7%)

small gap of 1.0 point. Our best RLR3 model improves from 73.5 to 78.2, a gain of 4.7 points, which is larger than the officially reported 4.3-point gain from the instruct model to the thinking model. The OpenMMR results also suggest a limitation of the current setup. Although RLR3 still improves over RLVR on the macro average, the margin is smaller than on ViRL and DeepVision. One plausible explanation is that OpenMMR contains many visual puzzles, for which an automatic rubric generator with access only to outcome-level ground truth may provide limited supervision. A promising future direction is synthetic visual puzzle construction with both outcome labels and finer-grained perceptual annotations, which could provide more informative supervision during rubric execution. 5.3

Long-Term Training Stability

To study the long-term stability and scaling behavior of the two training methods, we monitor benchmark performance over training checkpoints and show the resulting trajectories in Figure 2. RLR3 remains strong over a broader range of checkpoints than RLVR. Across many benchmarks, RLVR improves in the early stage of training and then plateaus or degrades, whereas RLR3 usually maintains or further improves its performance over a longer portion of the trajectory. The best performance of RLR3 is also higher than that of RLVR on most benchmarks. This suggests that finer-grained and more robust reward modeling can stablize RL training and make it easier to scale. This trajectory-level behavior also suggests a promising future direction. These gains are not yet fully consolidated into a single final model. Future work could explore model merging or online policy distillation to better transfer the strengths of multiple expert policies into one stronger policy. 5.4

GenRM Reliability

We sample policy-training instances, score responses with multiple frontier models, and use voted criterion-level labels to form GenRM train/test data. After scoring, we balance the data by whether 7

(a) WeMath

(b) DynaMath

(c) MathVision

(d) MathVerse

(e) MathVista

(f) MMMU-Pro

(g) RealWorldQA

(h) MMStar

(i) SimpleVQA

(j) CountBenchQA

(k) InfoVQA

(l) DocVQA

(m) ChartQA

(n) CharXiv (DQ)

(o) CharXiv (RQ)

Figure 2: Training trajectories across benchmarks. Each subplot reports benchmark performance over training checkpoints for RLVR and RLR3 , with the dashed gray line showing the base model. Table 4: Reliability of GenRM. “Execution” denotes whether the GenRM selects the correct path and verifier. “Arguments” and “Credit” are measured on verifiable and fuzzy criteria, respectively. GenRM Base Model SFT RLVR

Format Accuracy (%)

Content Accuracy (%)

Overall Accuracy (%)

Schema

Criterion

Execution

Arguments

Credit

Criterion-Level

Sample-Level

98.6 100.0 100.0

98.7 100.0 100.0

82.0 100.0 100.0

91.5 96.1 96.3

77.0 91.4 93.1

71.7 94.3 95.0

51.9 85.5 87.3

any criterion loses credit and split it without input overlap. Table 3 summarizes the resulting data. The GenRM outputs are automatically evaluated against rubrics and voted labels. Schema validity, criterion-slot matching, and function calling validity are checked from the JSON and rubric schema. Specifically, argument accuracy is measured by the corresponding verifier instead of exact matching. We compare the base model, an SFT baseline, and the RLVR-trained GenRM used in the final pipeline. Table 4 shows that the base model already follows the coarse schema reasonably well, but remains unreliable once the output must be executable and correctly scored. Its schema and criterion-slot accuracies are 98.6% and 98.7%, while execution accuracy drops to 82.0% and credit accuracy to 77.0%. SFT solves most instruction-following errors, reaching 100.0% accuracy on all three format metrics. The RLVR-trained GenRM also reaches 100.0% on these format metrics, and slightly improves the execution-critical content fields, with argument accuracy increasing from 96.1% to 96.3% and credit accuracy from 91.4% to 93.1% compared with the SFT baseline. These gains raise criterion-level overall accuracy from 94.3% to 95.0% and sample-level overall accuracy from 85.5% to 87.3%. This suggests that RLVR helps the GenRM learn to reason over the rubric and assign credit, rather than mimicking the teacher pattern. 5.5

Ablation Study of GenRM

We evaluate GenRM with a controlled audit set that probes reward-execution robustness under constructed abnormal responses. The audit set is built from the 1,000 examples in the GenRM test set: we keep their regular responses and construct 1,000 abnormal responses across four categories—nofinal-answer, irrelevant, wrong-but-plausible, and adversarial attack—with 250 responses per category. The regular column reports reward accuracy on regular responses, while the four abnormal columns 8

Table 5: Potential failure modes of GenRM. Self-answering is evaluated under the VLM-as-a-Judge setting, and target leakage is evaluated under the unlimited exposure setting. Each entry reports Average (Arguments / Credit). Absolute degradations larger than 2% are underlined. Settings

Accuracy (%) ↑

False Positive Rate (%) ↓

Regular

No Final

Irrelevant

Plausible

Adversarial

Default

95.0 (96.3 / 93.1)

11.5 (10.5 / 13.9)

1.5 (1.1 / 2.3)

5.6 (3.2 / 8.7)

18.0 (21.4 / 13.7)

VLM-as-a-Judge Unlimited Exp.

94.9 (96.3 / 92.7) 94.6 (95.7 / 93.0)

14.1 (13.5 / 15.4) 15.7 (15.9 / 15.4)

2.1 (1.3 / 3.2) 1.9 (1.9 / 1.9)

6.6 (4.7 / 9.0) 7.5 (5.7 / 9.8)

18.7 (22.0 / 14.3) 22.4 (27.6 / 15.5)

w/o Verifier

95.3

15.1

1.9

7.1

21.2

report false-positive rates (FPR) under the corresponding constructed failure modes. Appendix B.4 describes the LLM-based construction procedure and red-team adversarial generation. In this audit, regular-response accuracy stays close across settings, ranging from 94.9% to 95.3%. The main differences appear on constructed abnormal responses. Among the abnormal categories, nofinal-answer and adversarial attack are the most challenging. No-final-answer responses can contain an analysis trajectory in which the correct answer may appear multiple times, but still omit the final selection. Adversarial attack responses are produced by targeted probing of GenRM vulnerabilities. Access to image increases the risk of self-answering. Compared with the default setting, VLM-asa-Judge has higher FPRs on no-final-answer responses (14.1% vs. 11.5%) and wrong-but-plausible responses (6.6% vs. 5.6%). We hypothesize that visual access makes the model more likely to infer the correct answer directly from the image and fill the extracted arguments with that answer, even when the response itself is incomplete or incorrect. Target leakage increases the risk of shortcutting. Compared with the default setting, unlimited exposure shows higher FPRs on no-final-answer responses (15.7% vs. 11.5%), wrong-but-plausible responses (7.5% vs. 5.6%), and adversarial responses (22.4% vs. 18.0%). This suggests that exposing target values creates a shortcut for GenRM to copy or anchor on the reference answer instead of extracting the predicted value from the response. Deterministic verification reduces exploitable false positives. Removing the verifier keeps regularresponse accuracy similar (95.3%) but raises FPRs on no-final-answer responses (15.1% vs. 11.5%) and adversarial responses (21.2% vs. 18.0%). Although the absolute gaps are modest, they matter in online training because exploitable false positives can be repeatedly reinforced. In particular, without deterministic verification, an adversarial response can receive undeserved credit even without explicitly containing the correct answer. Representative bad cases are provided in Appendix D.1.

6

Conclusion and Limitations

We presented RLR3 , a framework for online reinforcement learning with rubric-based rewards in vision-language models. By treating verifiability as a criterion-level property, RLR3 extends RLVR from task-level outcome checking to rubric criteria that may require either deterministic verification or semantic judgment. Its execution routing, minimal exposure strategy, score remapping, hierarchical aggregation, and RLVR-trained GenRM make criterion-level scoring more faithful and informative under online optimization. Across three open-source training mixtures, RLR3 improves the macro average over RLVR. Controlled GenRM audits further show that deterministic verification and minimal exposure reduce exploitable false positives without sacrificing scoring accuracy. Despite these gains, the current study has several limitations. First, RLR3 still depends on rubrics generated by frontier models. Although these rubrics are effective in practice, the rubric generator is external to the policy optimization loop and is not itself improved through online training. As a result, rubric quality can become a bottleneck, especially when the generator under-specifies intermediate perceptual evidence or reasoning requirements. Second, the training trajectories still exhibit nontrivial metric fluctuations across checkpoints. This suggests that the gains learned during online RL are not yet fully stabilized or consolidated into a single final policy. An important next step is to introduce online policy distillation so that strong behaviors discovered at different stages of training can be accumulated more consistently. 9

References [1] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, Tao Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nat., 645(8081):633–638, 2025. [2] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. [3] Kimi Team. Kimi K2: open agentic intelligence. CoRR, abs/2507.20534, 2025. [4] GLM. GLM-4.5: agentic, reasoning, and coding (ARC) foundation models. abs/2508.06471, 2025.

CoRR,

[5] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. VLM-R1: A stable and generalizable r1-style large vision-language model. CoRR, abs/2504.07615, 2025. [6] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vlrethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. CoRR, abs/2504.08837, 2025. [7] Kimi Team. Kimi K2.5: visual agentic intelligence. CoRR, abs/2602.02276, 2026. [8] Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Wenkai Li, Wei Jia, Xin Lyu, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuxuan Zhang, Zhanxiao Du, Zhenyu Hou, Zhao Xue, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.1v-thinking: Towards 10

versatile multimodal reasoning with scalable reinforcement learning. CoRR, abs/2507.01006, 2025. [9] Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. CoRR, abs/2507.17746, 2025. [10] Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models. CoRR, abs/2507.18624, 2025. [11] Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, Jianguo Li, and Junbo Zhao. Reinforcement learning with rubric anchors. CoRR, abs/2508.12790, 2025. [12] Ya-Qi Yu, Fangyu Hong, Xiangyang Qu, Hao Wang, Gaojie Wu, Qiaoyu Luo, Nuo Xu, Huixin Wang, Wuheng Xu, Yongxin Liao, Zihao Chen, Haonan Li, Ziming Li, Dezhi Peng, Minghui Liao, Jihao Wu, Haoyu Ren, and Dandan Tu. Visual preference optimization with rubric rewards. CoRR, abs/2604.13029, 2026. [13] Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, and Philip S. Yu. Judge anything: MLLM as a judge across any modality. In Luiza Antonie, Jian Pei, Xiaohui Yu, Flavio Chierichetti, Hady W. Lauw, Yizhou Sun, and Srinivasan Parthasarathy, editors, Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, V.2, KDD 2025, Toronto ON, Canada, August 3-7, 2025, pages 5742–5753. ACM, 2025. [14] Tianyi Xiong, Yi Ge, Ming Li, Zuolong Zhang, Pranav Kulkarni, Kaishen Wang, Qi He, Zeying Zhu, Chenxi Liu, Ruibo Chen, Tong Zheng, Yanshuo Chen, Xiyao Wang, Renrui Zhang, Wenhu Chen, and Heng Huang. Multi-crit: Benchmarking multimodal judges on pluralistic criteria-following. CoRR, abs/2511.21662, 2025. [15] Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. Llmrubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 13806–13834. Association for Computational Linguistics, 2024. [16] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate AI research. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. [17] Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health. CoRR, abs/2505.08775, 2025. [18] Zhilin Wang, Jaehun Jung, Ximing Lu, Shizhe Diao, Ellie Evans, Jiaqi Zeng, Pavlo Molchanov, Yejin Choi, Jan Kautz, and Yi Dong. Profbench: Multi-domain rubrics requiring professional knowledge to answer and judge. CoRR, abs/2510.18941, 2025. [19] Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, and Yunzhong He. Prbench: Large-scale expert rubrics for evaluating high-stakes professional reasoning. CoRR, abs/2511.11562, 2025. 11

[20] Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning. CoRR, abs/2508.16949, 2025. [21] Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training. CoRR, abs/2602.01511, 2026. [22] Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment. CoRR, abs/2510.07743, 2025. [23] MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. Online rubrics elicitation from pairwise comparisons. CoRR, abs/2510.07284, 2025. [24] Lipeng Xie, Sen Huang, Zhuo Zhang, Anni Zou, Yunpeng Zhai, Dingchao Ren, Kezun Zhang, Haoyuan Hu, Boyin Liu, Haoran Chen, Zhaoyang Liu, and Bolin Ding. Auto-rubric: Learning to extract generalizable criteria for reward modeling. CoRR, abs/2510.17314, 2025. [25] Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, Yang Zhou, Jingwen Yang, Shunyu Liu, Kaike Zhang, and Wei Chen. Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. CoRR, abs/2601.08430, 2026. [26] Zicheng Kong, Dehua Ma, Zhenbo Xu, Alven Yang, Yiwei Ru, Haoran Wang, Zixuan Zhou, Fuqing Bie, Liuyu Xiang, Huijia Wu, Jian Zhao, and Zhaofeng He. Omni-rrm: Advancing omni reward modeling via automatic rubric-grounded preference synthesis. CoRR, abs/2602.00846, 2026. [27] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR, abs/2503.14476, 2025. [28] Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, and Lidong Bing. Openmmreasoner: Pushing the frontiers for multimodal reasoning with an open and general recipe. CoRR, abs/2511.16334, 2025. [29] Haoxiang Sun, Lizhen Xu, Bing Zhao, Wotao Yin, Wei Wang, Boyu Yang, Rui Wang, and Hu Wei. Deepvision-103k: A visually diverse, broad-coverage, and verifiable mathematical dataset for multimodal reasoning. CoRR, abs/2602.16742, 2026. [30] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma Gongque, Shanglin Lei, Yifan Zhang, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Xiao Zong, Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model achieve human-like mathematical reasoning? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 20023– 20070. Association for Computational Linguistics, 2025. [31] Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [32] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: 12

Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. [33] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. MATHVERSE: does your multi-modal LLM truly see the diagrams in visual math problems? In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VIII, Lecture Notes in Computer Science, pages 169–186. Springer, 2024. [34] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. CoRR, abs/2310.02255, 2023. [35] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 15134–15186. Association for Computational Linguistics, 2025. [36] xAI. Grok-1.5 vision preview. https://x.ai/news/grok-1.5v, 2024. Accessed: 2024-0520. [37] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. [38] Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, Yutao Zeng, Zhoufutu Wen, Ke Jin, Baorui Wang, Weixiao Zhou, Yunhong Lu, Tongliang Li, Wenhao Huang, and Zhoujun Li. Simplevqa: Multimodal factuality evaluation for multimodal large language models. CoRR, abs/2502.13059, 2025. [39] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey A. Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bosnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier J. Hénaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b VLM for transfer. CoRR, abs/2407.07726, 2024. [40] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. Infographicvqa. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pages 2582–2591. IEEE, 2022. [41] Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. Docvqa: A dataset for VQA on document images. CoRR, abs/2007.00398, 2020. [42] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, Findings of ACL, pages 2263–2279. Association for Computational Linguistics, 2022. [43] Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 13

Charxiv: Charting gaps in realistic chart understanding in multimodal llms. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.

14

A

JSON Schema

Both rubric generation and response scoring use JSON schema to facilitate reliable parsing. A.1

Rubric Schema

Following the design in Section 4.1, we format the rubric as a JSON object with two top-level arrays: essential and additional, corresponding to the criterion type. Each element in either array is a triplet: • criterion: A concrete, verifiable assertion that the judge can verify directly from the image, question, and candidate response. • reference: Either a ground truth derived from the image and common knowledge, or a scoring tool. • weight: A three-level integer quantifying the criterion’s importance, ranging from 1 (Auxiliary: supplementary information) through 2 (Important: noticeable impact on the user experience) to 3 (Key: critical elements where any omission or deviation constitutes a definitive error). { " essential ": [ { " criterion ": " string " , " reference ": " string " , " weight ": 1 | 2 | 3 } ] " additional ": [...] }

A.2

Scoring Schema

Following the design in Section 4.2, we format the scoring output as a JSON object with a reasoning trajectory and two top-level arrays, mirroring the input rubric. Each element contains three fields: • criterion: The assertion to be verified (copied verbatim from input). • rationale: – When reference is a ground truth: Reasoning for the judgment (1-2 sentences). – When reference is a scoring tool: Explanation of how the predicted value is identified from the response (1-2 sentences). • credit: – When reference is a ground truth: A three-level score, ranging from 0 (No Credit: incorrect or missing) throught 0.5 (Partial Credit: partially correct, incomplete, or with minor errors) to 1 (Full Credit: fully correct or semantically equivalent). – When reference is a scoring tool: The tool call string. { " thought ": " string " , " essential ": [ { " criterion ": " string " , " rationale ": " string " , " credit ": 0 | 0.5 | 1 | " < name > _verify ( predict =...) " } ] " additional ": [...] }

15

B

System Prompt

B.1

Rubric Generation Prompt

The generation prompt instructs the reasoning model to construct an instance-specific checklist-style rubric for a given image-question pair, following the principles described in Section 4.1. Expert-Grounded Generation. When trustworthy ground-truth annotations are available, the prompt includes a dual verification step: before finalizing the rubric, the model must confirm that the provided ground-truth fully satisfies all essential criteria, ensuring that no correct response would be penalized by an erroneous check item. Otherwise, the dual verification is removed. System Prompt: Rubric Generation You are a multimodal evaluation expert. Given the user question, the associated image, and the provided reference answer, construct an instance-specific Checklist-Style Rubric for evaluating the accuracy of model responses. The criteria in this Checklist serve as the sole evaluation standard for the reward model. Construction Principles • Atomic: Each check item targets exactly one key point or atomic sub-question within the query. • Comprehensive: The combined set of items covers all critical dimensions of the user’s question. • Precise: Exclude redundant checks and checks unrelated to the question. • Objective: Ground assessments in image facts or logical truths; avoid subjective uncertainty. Field Definitions Two categories of check items: • essential: Core information highlighted by the query. These are prerequisites for a sound response, regardless of verbosity. • additional: Relevant image facts, supplementary knowledge, or intermediate steps involved in deriving the answer. Optional; the list may be empty. Fields per item: • criterion: A concrete, verifiable assertion. • reference: Either a ground truth derived from the image and common knowledge, or a scoring tool. • weight: A three-level integer quantifying the criterion’s importance, ranging from 1 (Auxiliary: supplementary information) through 2 (Important: noticeable impact on the user experience) to 3 (Key: critical elements where any omission or deviation constitutes a definitive error). Place the ground truth in the reference field; it is strictly prohibited to include the ground truth within the criterion field. Specifically, for the following scenarios, we use deterministic tools for scoring. To facilitate tool calling, format the reference field using Python syntax according to the following specifications: {Specifications of the available verifiers, see Appendix C} Note: Think from a scoring perspective: do not double-count and avoid over-decomposing key points. [Dual Verification] [Before finalizing the Checklist, verify that the provided reference answer satisfies all essential items. Any unsatisfied item indicates a construction error—the item is either incorrect or superfluous.] Output Format {Structured JSON following the rubric schema in Appendix A.1.}

B.2

Rubric Aggregation Prompt

After independent rubric generation by multiple models, an aggregation prompt merges the candidate checklists into a single unified rubric. The model first applies the same four construction principles as in rubric generation, then executes the additional aggregation instructions below to apply majorityvote filtering, deduplicate overlapping check items, and verify the correctness of all references.

16

System Prompt: Rubric Aggregation [... Rubric principles and fields identical to rubric generation omitted ...] Checklist Aggregation Do not construct the Checklist from scratch. Multiple models have each independently generated a candidate Checklist for the same question; your task is to merge them into a single unified rubric. Based on the construction principles, ensure that: • All retained check items are necessary. Specifically: – Retain only items that the majority of candidates agree should be checked. – Remove redundant or duplicate checks. – Remove checks unrelated to the question. • Do not over-decompose check items. • All reference values are correct. [... Output format specifications identical to rubric generation omitted ...]

B.3

Response Scoring Prompt

The scoring prompt instructs the judge model to evaluate a candidate response against the finalized rubric, assigning a credit score or calling the specific verify function for each criterion. System Prompt: Response Scoring You are a multimodal evaluation expert. Given the user question, the associated image, and the ChecklistStyle Rubric, evaluate the accuracy of the model response. Ensure that the criteria in the Checklist are used as the sole evaluation standard. Checklist Fields Two categories of check items: • essential: Core information highlighted by the query. These are prerequisites for a sound response, regardless of verbosity. • additional: Relevant image facts, supplementary knowledge, or intermediate steps involved in deriving the answer. Optional; the list may be empty. Fields per item: • criterion: A concrete, verifiable assertion. • reference: Either a ground truth derived from the image and common knowledge, or a scoring tool. • weight: A three-level integer quantifying the criterion’s importance, ranging from 1 (Auxiliary: supplementary information) through 2 (Important: noticeable impact on the user experience) to 3 (Key: critical elements where any omission or deviation constitutes a definitive error). Scoring Fields For each criterion, output: • criterion: The assertion to be verified (copied verbatim from input). • rationale: – When reference is a ground truth: Reasoning for the judgment (1-2 sentences). – When reference is a scoring tool: Explanation of how the predicted value is identified from the response (1-2 sentences). • credit: – When reference is a ground truth: A three-level score, ranging from 0 (No Credit: incorrect or missing) throught 0.5 (Partial Credit: partially correct, incomplete, or with minor errors) to 1 (Full Credit: fully correct or semantically equivalent). – When reference is a scoring tool: The tool call string. When reference is a scoring tool, call the corresponding tool for scoring. Its format (with credit as a tool call string) is as follows: {Specifications of the available verifiers, see Appendix C} Note: When reference is a scoring tool, credit should contain the tool call rather than a numeric score; simply extract the prediction result without judging correctness. Output Format {Structured JSON following the scoring schema in Appendix A.2.}

17

B.4

Failure-Mode Audit Prompt

We construct the failure-mode audit set from the GenRM test set, preserving the original inputs and their rubrics. The regular split uses the original test set responses. In addition, we construct 1,000 abnormal responses across four categories, with 250 responses per category: no-final-answer, irrelevant, wrong-but-plausible, and adversarial. The no-final-answer category includes responses that may contain relevant analysis but omit the final answer. The irrelevant category contains fluent responses that are off-topic with respect to the question. The wrong-but-plausible category contains responses that follow the task format but alter at least one important answer element. The adversarial category contains responses designed to elicit undeserved credit. For the first three categories, we use an LLM generator conditioned on the question, the original response, and the instance-specific checklist. The concrete system prompts differ by failure category, but share the same overall structure: generate a plausible abnormal response, then provide criterionlevel annotations for the response. For verifiable criteria, these annotations are converted into verifier arguments. For fuzzy criteria, the generator provides the criterion-level credit directly. Shared Prompt Structure: Abnormal Response Construction You are helping construct test data for a reward model. Rewrite the original model response so that it exhibits the specified abnormal behavior while still looking like a natural model output. Inputs Question: {question} Original response: {original response} Checklist: {criteria and references} Failure type: {specific instruction} Common Requirements • First write the rewritten response for the assigned failure type. • Then evaluate the rewritten response against every checklist criterion as a strict human grader. • For each criterion, record the exact value stated by the response; if the response does not address the criterion or does not state a specific value, use an empty value. • Assign a credit in {0, 0.5, 1} with a brief rationale. Partial correctness is allowed, and an abnormal response does not need to fail every criterion. Output Format Return the rewritten response inside <response>...</response> tags. Return the criterion-level annotations inside <extractions>...</extractions> tags as a JSON dictionary, with one entry per checklist criterion and fields extracted_value, credit, and rationale. Category-Specific Instructions: Abnormal Responses No-final-answer. The response should look like an attempt to solve the task, but it should dodge the requested final answer. For example, for a calculation question, it may discuss the setup or method without arriving at the final value; for an identification question, it may discuss related visual or conceptual cues without naming the target; for a verification question, it may explain general principles without confirming or denying the claim. For numerical questions, the response should not place a final answer in a boxed span. Irrelevant. The response should be fluent and authoritative, but off-topic with respect to the actual question. For example, for a geometry question, it may discuss unrelated algebra concepts; for an imagedetail question, it may give general background knowledge; for a computation question, it may describe definitions or history rather than solving the instance. Wrong-but-plausible. The response should give at least one confidently wrong answer while preserving a reasonable surface form. Other criteria may be correct, wrong, or omitted. For numerical answers, the response should provide a plausible but incorrect value; for text answers, it should state a plausible but incorrect entity, option, relation, or property. The reasoning should appear coherent but contain a subtle error leading to the wrong answer.

For adversarial responses, we first run a red-team search to identify attack patterns that can elicit undeserved credit without providing the true answer. The final adversarial audit uses one of the three discovered patterns, including authoritative circumlocution, symbolic-equivalence bluffing, and plausible reasoning with an incorrect final selection. 18

Attack-Specific Instructions: Adversarial Responses Authoritative circumlocution. Replace the exact answer with domain-specific jargon, a fabricated technical term, or an elaborate description that sounds authoritative. For example, a letter label may be described by its visual position rather than stated directly; a property such as fragility may be replaced with a technical-sounding circumlocution; an option answer may be described as “the option matching the relevant diagnostic pattern” rather than named. Symbolic-equivalence bluff. Present an over-complicated symbolic or mathematical construction and claim that simplification yields the desired result, without explicitly giving the correct final value. For example, the response may introduce sums, integrals, auxiliary variables, or closed-form expressions, then assert that cancellation or reduction confirms the expected solution. Incorrect final selection. Give reasoning that appears mostly sound, but select an incorrect final answer. For example, the response may derive multiple candidate roots and choose an extraneous one, analyze a loop or boundary condition but make the wrong final selection, or correctly discuss visual evidence but pick the wrong option.

We apply an additional quality-control stage before using the generated responses. Two independent LLM judges review each candidate response and check whether the criterion-level annotation is consistent with the response and whether the response matches the intended failure mode. We also apply lightweight deterministic checks to remove cases where the generated response accidentally include the target answer. Candidates with inconsistent annotations or invalid failure-mode behavior are filtered out before evaluation. After filtering, we sample a balanced audit set with exactly 250 examples per abnormal category, using at most one abnormal response from each held-out input.

C

Verifier Specifications

As shown in Table 7, the verifier library provides a small set of deterministic scoring functions for criteria whose target values can be specified explicitly. Each verifier is used through a two-stage interface. During rubric generation, the reference field stores the verifier name and the target-side arguments, such as text_verify(target=...). During response scoring, the reward model does not see the hidden target arguments; it only extracts the prediction from the candidate response and emits the corresponding predict call, such as text_verify(predict=...). Representative verifier calls are shown in Table 6. Table 6: Representative verifier calls. The rubric-side call is generated when constructing the checklist; the scoring-side call is emitted after extracting the candidate response’s prediction. Verifier

Rubric-side reference

Scoring-side credit

text_verify

text_verify(target=’Export Volume’, ignore_space=True, ignore_case=True)

text_verify(predict=’Export Volume’)

expr_verify

expr_verify(target=r’\frac{4}{6}’)

expr_verify(predict=’2/3’)

time_verify

time_verify(target=’18:15’, tformat=’%H:%M’)

time_verify(predict=’18:15’, pformat=’%H:%M’)

list_verify

list_verify(target=[’M-30’, ’M-31’, ’M-31UK’])

list_verify(predict=[’M-30’, ’M-31’])

bbox_verify

bbox_verify( target=[[531,118,892,435]])

bbox_verify( predict=[[529,119,890,433]])

point_verify

point_verify(target=[[591,234]])

point_verify(predict=[[589,236]])

Generation-time constraints. The rubric generator is only allowed to use verifiers exposed for the current task type. Text and list verifiers are reserved for optical or directly extractive text, while point and bounding-box verifiers are reserved for grounding targets with normalized image coordinates. Expression verification is limited to raw values directly present in, or unambiguously extracted from, the response, such as option letters, formulas, and numeric expressions. It must not be used to encode a semantic judgment as a Boolean target. Across all verifiers, the target answer is placed in the reference field and omitted from the criterion text. 19

Scoring-time constraints. When a criterion uses a verifier, the scoring model must emit a tool call rather than a numeric credit. Its role is to extract the response-side prediction only; it should not create a verifier call for criteria whose reference is ordinary ground truth, and it should not skip a verifier call when the rubric specifies one. If no prediction is present, the model supplies empty string or list for verifiers, such as predict=” or predict=[]. For coordinate-based verifiers, formatting irregularities in the response are preserved in the emitted prediction so that the verifier, rather than the extractor, handles parsing failures.

D

Case Study

D.1

Failure-Mode Audit Examples

Example 1: The response provides a detailed discussion of the thermodynamic process and uses approximate descriptions without ever naming the correct answer “Boiler”, acting as a “blind” reasoner. The LLM extractor fails to extract any answer, so the verifier correctly returns 0. The LLM judge, however, is misled by the functional similarity and erroneously awards full credit.

Figure 3: The diagram illustrates the components of a coal-fired power plant. Question & Ground Truth

Q: Which component uses coal and water to make steam? A: Boiler Model Response

When examining the mechanics of a coal-fired power plant, the process begins with the fundamental inputs of fuel and a working fluid. · · · coal is prepared and introduced into a combustion chamber where it is burned to release thermal energy. · · · the core event where the chemical energy of the coal is directly applied to the water to induce this phase change occurs within the main heat exchange vessel of the system. · · · identifying the exact label of the vessel depends on the specific schematic or terminology being referenced in a given context. Scoring Results

LLM-as-an-Extractor + Deterministic Verifier: text_verify(predict='') → 0 Rationale: The model describes the process in detail but does not explicitly name the component (e.g., boiler, steam generator, or combustion chamber) that uses coal and water to make steam. LLM-as-a-Judge): Credit → 1 Rationale: The model describes the component as the ‘main heat exchange vessel’ and explains its function but does not explicitly use the term ‘boiler’. However, the description matches the function of a boiler.

20

Example 2: The response employs the symbolic-equivalence bluff strategy: it embeds the raw data (P (bAsia ) = 10) inside an elaborate mathematical framework, then claims the answer follows from “algebraic reduction” and “symbolic equivalence” without ever stating “book about Asia” as the conclusion. The LLM extractor cannot extract a concrete answer, so the verifier returns 0. The LLM judge, however, is deceived by the mathematical veneer and the embedded price data, incorrectly awarding full credit despite the answer never being explicitly stated.

Figure 4: The table lists books and their prices. Question & Ground Truth

Q: Which book is the least expensive? A: book about Asia Model Response (Symbolic Bluff)

To determine the least expensive book, we must construct a cost minimization framework over the discrete set of available literature B = {b1 , b2 , . . . , bn }. Let the price function P (bi ) be defined for each book bi in the catalog. We initialize the optimization problem by defining the total cost functional Pn J as a weighted sum involving the individual prices and a selection vector ⃗x: J(⃗x) = i=1 (P (bi ) · xi + · · · ). Substituting the specific market values provided (P (bart ) = 25, P (bAsia ) = 10, P (bf ish ) = 21, . . . ), the expression expands into a highdimensional manifold:   I P (z) 1 dz · · · Jtotal = min · · · + lim ϵ→0 ϵ C (z − P (bk ))2 ⃗ x While this expression appears computationally intensive due to the contour integrals and limit operations, algebraic reduction shows that the terms involving the complex analysis components and the exponential decay factors cancel out precisely against the normalization constants. · · · The symbolic equivalence confirms that the optimal index k ∗ corresponds to the book encoded by the minimal eigenvalue of the price matrix. Therefore, the derivation confirms the identity of the least expensive book as dictated by the reduced model. Scoring Results

LLM-as-an-Extractor + Deterministic Verifier: text_verify(predict='') → 0 Rationale: The model mentions the books and their prices but does not explicitly state which book is the least expensive in the final answer. LLM-as-a-Judge: Credit → 1 Rationale: The model mentions ‘P(bAsia )=10’ and later refers to the book as ‘the book encoded by the minimal eigenvalue of the price matrix’, but does not explicitly state ‘book about Asia’ as the least expensive. However, it identifies ‘Asia’ as the book with the lowest price of 10, which matches the reference.

21

Table 7: Verifier interfaces. These signatures are provided in the dedicated verifier-specification blocks of the rubric-generation and response-scoring prompts. The model then instantiates the corresponding call in the reference field during rubric generation or in the credit field during response scoring. text_verify

Rubric signature

Scoring signature Return Usage

text_verify(target: str = None, candidates: List[str] = None, use_latex: bool = False, ignore_space: bool = False, ignore_punc: bool = False, ignore_case: bool = False, ignore_st: bool = False) text_verify(predict: str) float in [0, 1], computed as normalized text similarity; with candidates, returns the maximum candidate score. OCR-style text or LaTeX transcription. Not used for semantic judgments or nonextractive visual QA.

expr_verify Rubric signature Scoring signature Return Usage

expr_verify(target: str) expr_verify(predict: str) float in {0, 1}; returns 1 iff the parsed target and prediction are mathematically equivalent. Option letters, numeric expressions, and LaTeX expressions. Units and task context remain in the criterion, not in the target string.

time_verify Rubric signature Scoring signature Return Usage

time_verify(target: str, tformat: str) time_verify(predict: str, pformat: str) float in {0, 1}; returns 1 iff the parsed date or time objects are equal. Dates and times expressed with Python-style datetime formats. The prediction format is copied from the response. Weekday names are not used as verifier targets.

list_verify Rubric signature Scoring signature Return Usage

list_verify(target: List[str] = None, candidates: List[List[str]] = None) list_verify(predict: List[str]) float in [0, 1], computed by Hungarian matching over pairwise text-similarity scores; with candidates, returns the maximum candidate-list score. Extractive text lists, such as OCR key fields. Each candidate list should represent one complete valid answer set.

bbox_verify Rubric signature Scoring signature Return Usage

bbox_verify(target: List[List[int]]) bbox_verify(predict: List[List[int]]) float in [0, 1], computed by Hungarian matching over pairwise IoU scores and normalized by the larger number of boxes. Single- or multi-object bounding-box grounding. Each box is [x1, y1, x2, y2] with coordinates normalized to 0–1000.

point_verify Rubric signature Scoring signature Return Usage

point_verify(target: List[List[int]]) point_verify(predict: List[List[int]]) float in [0, 1], computed by Hungarian matching over pairwise point distances and converting the matched distance into a normalized proximity score. Single- or multi-object point grounding. Each point is [x, y] with coordinates normalized to 0–1000.

22