Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR Esther Xin Independent Researcher estherxin0011@gmail.com Code and data: https://github.com/ethxin0011/verifier-error-budget

Abstract

arXiv:2609.01354v1 [cs.CL] 1 Sep 2026

Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free-text answer into a binary reward. Prior work reports that one evaluation harness accepts only about 94% of its own ground-truth answers, blaming LaTeX parsing. That is an aggregate: it does not say which answer forms consume the error budget. We supply the decomposition. We apply metamorphic testing to the verifier rather than the model, generating certified-equivalent answer variants—rewrites that preserve mathematical meaning by construction, so any rejection is a provable false negative needing no human adjudication—and measure rejection per answer category across four widely used verifiers over 307,420 verdicts. We find three things. (1) Self-validation ranges from 53.8% to 95.2% on identical inputs, a 41.3-point spread. The published figure describes one implementation, not the task; two configurations of the same library disagree on 49.9% of pairs. (2) The residual is not spread across parsing categories but concentrated in whitespace and punctuation, which account for 93.0% of in-contract failures for the default LaTeX configuration. A trailing period or newline dominates the budget. (3) Separating rejection from execution failure shows that verifiers with similar aggregate error fail for opposite reasons, and that a reference numeric cascade accepts off-by-one wrong answers as a step function of magnitude—0% below 104 , 100% at or above— because its relative tolerance is scale-invariant.

1

Introduction

Reinforcement learning with verifiable rewards (RLVR) has become the dominant post-training recipe for mathematical reasoning. Its central appeal is that the reward requires no human annotation (Guo et al., 2025): a model emits an answer,

an automatic verifier checks it against a reference, and the resulting binary signal drives policy optimisation. The same verifiers serve a second role, converting free-text model outputs into the accuracy figures reported on benchmarks such as MATH and GSM8K (Hendrycks et al., 2021; Cobbe et al., 2021). This shifts a substantial burden onto a component that is rarely examined. The verifier is not a mathematical oracle; it is a program that extracts a substring, normalises it, and compares it to a reference under some notion of equivalence. Every step is a design decision, and every decision has a failure mode. If the verifier rejects a correct answer, the policy is penalised for behaviour it should be rewarded for. If it accepts an incorrect one, the policy is rewarded for being wrong. Recent work has begun to treat this seriously. Cai et al. (2025) formalise verifier unreliability as a stochastic reward channel with asymmetric noise rates ρ0 (false positive) and ρ1 (false negative), and derive backward and forward corrections to the policy gradient; notably, the forward correction requires only the false-negative rate as input. Egashira et al. (2026) show that this framing is incomplete in an important way: verifier errors in practice are systematic rather than random, and systematic false positives can drive outcomes ranging from suboptimal plateaus to collapse, with the result determined by the pattern of errors rather than the aggregate rate. Huang et al. (2026) document that rule-based checkers and model-based judges fail in opposite directions, the former through brittle parsing and the latter through reward hacking. Both threads point at the same missing quantity. Corrections need error rates as parameters; the systematic-error analysis needs the shape of the error, not merely its magnitude. Yet the reliability literature reports aggregates. The most concrete published figure is that a standard evaluation harness validates ground truth against itself at roughly

94% accuracy, with the residual attributed to LaTeX parsing failures (Hugging Face, 2025b). That number tells a practitioner that six percent of correct answers are lost somewhere. It does not say where. This paper supplies the decomposition. We adapt metamorphic testing (Asgari et al., 2025; Hyun et al., 2024), a technique normally applied to models, and point it at the verifier instead. Given a gold answer g and a transformation T that preserves mathematical meaning by construction, the pair (g, T (g)) has a known correct verdict. Any verifier rejecting it commits a false negative that is certified: no human adjudication, no trusted arbiter, no appeal to a stronger model. The dual construction, applying a meaning-changing transformation and observing acceptance, certifies false positives. Because ground truth comes from the construction rather than from labelling, the procedure scales combinatorially in golds × transforms × verifiers and runs entirely on CPU. We apply this to four widely used implementations across 43 transformations in 14 strata, yielding 307,420 verdicts. A contract matrix records which strata each verifier claims to handle, so that a normalizer which never advertised answer extraction is not scored as defective for rejecting a boxed answer; out-of-contract behaviour is reported but never counted as a bug. We further separate rejection from execution failure, so that a verifier which crashes on an input cannot be credited with a correct verdict. Findings. (1) Self-validation is implementation-specific, not a property of the task. Acceptance of certified-equivalent variants ranges from 53.8% to 95.2% across the four implementations on identical inputs, a spread of 41.3 points. Two configurations of the same library differ by 26 points and disagree on 49.9% of individual pairs. A benchmark accuracy figure is therefore a function of the verifier configuration, which is almost never reported. (2) The residual is string hygiene, not grammar. Whitespace and punctuation handling accounts for 93.0% of in-contract failures for the default LaTeX configuration and 74.1% for the plain-expression configuration. A trailing period or a newline, which no reasonable specification treats as semantically meaningful, dominates the error budget. This reframes the problem: the prior attribution to LaTeX parsing suggests difficult grammar

work, whereas the measured distribution points at normalisation that is largely trivial to fix. (3) Similar aggregates hide opposite failure modes. Separating rejection from execution failure shows that sympy-cascade and strip-string have comparable aggregate residuals (12.7% and 4.8%) arising from disjoint causes: sympy-cascade returns a verdict on only 87.3% of inputs but is correct on every input it judges, whereas strip-string judges everything and errs by rejecting. Additionally, sympy-cascade accepts off-by-one adversarial answers as a deterministic step function of magnitude—0% below 104 , 100% at or above—because its relative tolerance is scale-invariant. This is maximally systematic in the sense of Egashira et al. (2026) and invisible in any aggregate rate. Contributions. (i) A certified metamorphic protocol for verifier auditing in which ground truth is constructed rather than adjudicated; (ii) a percategory decomposition across four implementations at 307,420 verdicts with Wilson intervals on every cell; (iii) a contract matrix separating implementation defects from specification ambiguity, and a coverage measure separating rejection from execution failure; (iv) identification of a scaledependent false-positive mechanism with an exact threshold; and (v) released transform suite, contract matrix, and per-sample verdict records.

2

Method

2.1

Certified equivalence

Let g be a gold answer and T a transformation. If T is semantics-preserving by construction, then T (g) is correct, and a verifier V returning V (g, T (g)) = FALSE commits a certified false negative. Dually, for a meaning-changing transformation T ′ , acceptance is a certified false positive. This design has three consequences: ground truth is free; the sample size is combinatorial in golds × transforms × verifiers; and the entire procedure is CPU-only. 2.2 Transform classes and the contract matrix We use 43 transformations across 14 strata, partitioned into three classes. CERTIFIED - EQUIV transformations are mathematically identical and within the verifier’s declared contract; rejection is scored as a false negative. CONTRACT- DEP transformations depend on the declared contract (boxing,

units, text wrappers) and are reported as specification ambiguity, never as defects. ADVERSAR IAL transformations change meaning; acceptance is scored as a false positive. Two design decisions matter. First, transformations producing empty output are discarded automatically. Second, a verifier is scored only on strata it claims to handle: a normalizer that never advertised answer extraction is not “wrong” to reject a boxed answer. Out-of-contract results appear in Table 7. The contract assignments are a judgement call and are published in the released code so that they can be contested. 2.3

Coverage: rejection versus execution failure

A verifier can fail in two distinct ways: it can return FALSE on a correct answer, or it can fail to return a verdict at all (parse exception, timeout, crash). Conflating these is a reporting error with real consequences—a verifier that crashes on every input in a stratum would otherwise be credited with a false-positive rate of 0.0%, which reads as “correctly rejected”. The same distinction between judgement accuracy and judgement availability has been argued for in security-scanner evaluation (Lan et al., 2026). We therefore define the evaluated denominator neval = nT RUE + nFALSE ,

coverage = neval /n,

(Albalak et al., 2025), plus 2,000 synthetic answers in standard MATH answer forms. Synthesis was necessary because set and interval notation appears in only 0.1% of corpus answers, leaving those strata underpowered. Synthetic golds are tagged and reported separately; §3.5 confirms all findings hold on corpus-derived answers alone. 2.6

The audit runs as an Azure ML parallel pipeline on five CPU nodes. Each verification executes in an isolated subprocess with a five-second peritem budget matching the documented symboliccomparison timeout; crashes and timeouts are recorded as distinct verdict classes.

3

Results Verifier strip-string sympy-cascade mv-latex mv-expr

2.5

Verifier

Cov.

Judged

95.2% 100.0% 95.2% 87.3% 87.3% 100.0% 79.9% 100.0% 79.9% 53.8% 100.0% 53.8%

Stratum

Fail

Share

mv-expr S6 whitespace 7,852 74.1% mv-expr S5 math-mode 2,417 22.8% mv-latex S6 whitespace 5,846 93.0% mv-latex S10 sets 238 3.8% strip-string S6 whitespace 694 50.3% strip-string S2 frac/dec 438 31.7% sympy-cascade S14 unreduced 2,062 52.0% sympy-cascade S6 whitespace 694 17.5%

Verifiers under test

We audit four implementations: math-verify (Hugging Face, 2025a) with LaTeX extraction (library default); math-verify restricted to plainexpression extraction; the DeepSeek-Math lineage string normalizer (Shao et al., 2024); and a reference three-level cascade (exact string, then numeric with relative tolerance 10−4 , then symbolic via SymPy). The ANTLR4 runtime is pinned to 4.13.2; behaviour differs across runtimes.

n Self-val. 28,570 31,266 31,266 22,953

Table 1: Self-validation: acceptance of certifiedequivalent variants, in-contract strata only. Self-val. is over all inputs; Cov. is the fraction on which the verifier returned a verdict at all; Judged restricts to those. The 41.3-point spread on identical inputs shows the figure is implementation-specific. sympy-cascade is correct on every input it judges: its residual is entirely execution failure.

and report both the all-inputs rate (denominator n) and the evaluated rate (denominator neval ). Cells with zero coverage report N / A, never 0. Overall in-contract coverage is 96.5%; the deficit is concentrated in a single implementation (§3). 2.4

Execution

Table 2: Error mass share: fraction of each verifier’s in-contract failures attributable to each stratum, top two strata per verifier. Whitespace and punctuation dominate both math-verify configurations. Every failure in the sympy-cascade rows is a parse exception rather than a rejected answer; the full breakdown is in the released T2_error_mass_share.csv.

Data

We use 4,990 unique gold answers drawn from GSM8K (Cobbe et al., 2021), MATH across all seven subjects (Hendrycks et al., 2021), Big-Math

3.1

Self-validation varies by 41 points

Table 1 reports self-validation: the fraction of certified-equivalent variants a verifier accepts. The

Stratum

n

Verif.

S6 white mv-expr S5 math mv-expr S10 sets mv-latex S6 white mv-latex S7 sqrt strip-string S2 frac strip-string S2 frac mv-expr S4 delim mv-latex S6 white strip-string S5 math strip-string S7 sqrt mv-latex S14 unred mv-latex S1 frac d. mv-latex S9 group all

FN 95% CI

15,704 50.0% [49.2, 50.8] 4,951 48.8% [47.4, 50.2] 634 37.5% [33.9, 41.4] 15,704 37.2% [36.5, 38.0] 404 35.1% [30.7, 39.9] 1,725 25.4% [23.4, 27.5] 1,725 19.1% [17.3, 21.0] 1,067 18.7% [16.5, 21.2] 15,704 4.4% [4.1, 4.8] 4,951 2.1% [1.8, 2.6] 404 0.2% [0.0, 1.4] 2,062 0.0% [0.0, 0.2] 4,146 0.0% [0.0, 0.1] 573 0.0% [0.0, 0.7]

zero coverage (no verdict returned) S10 sets sympy-cascade 634 S14 unred sympy-cascade 2,062

range is 53.8% to 95.2% on identical inputs. The published ∼94% figure (Hugging Face, 2025b) falls near the top of this range rather than at its centre, indicating that it characterises one particular harness on one particular input distribution rather than the task. Figure 1 plots the spread. The coverage column separates two failure modes that the aggregate hides. sympy-cascade returns a verdict on 87.3% of inputs and is correct on 100% of those it judges: its entire residual is parse failure, not equivalence error. The two math-verify configurations return verdicts on 100% of inputs and err by rejecting. These are different defects requiring different fixes, and an aggregate rate conflates them. 95.2%

strip_string

n/a n/a n/a n/a

87.3%

sympy cascade

79.9%

math-verify (latex)

Table 3: Certified false-negative rates by stratum and verifier, in-contract only, with Wilson score intervals. Rates are computed over verdicts actually returned; cells where the verifier never returned a verdict report N / A rather than 0. Strata with zero failures are retained to show which cases are handled correctly.

prior work (~94%)

0%

< 10 101 102 103 104 > 105

n V1 V2 V3 20 134 560 367 131 75

0.0 0.0 0.0 0.0 0.0 0.0

0.0 0.0 0.0 0.0 0.0 0.0

V4

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 100.0

Table 4: Off-by-one acceptance rate (%) by gold answer magnitude, over verdicts actually returned. V1 = mv-expr, V2 = mv-latex, V3 = strip-string, V4 = sympy-cascade. V4 steps deterministically at 104 : a relative tolerance of 10−4 is scale-invariant, so 10,001 differs from 10,000 by 0.01% and is accepted. Aggregate rate 16.0% (95% CI [14.1, 18.1], n = 1,287).

20%

40%

60%

80%

self-validation rate (in-contract)

100%

Figure 1: Self-validation rate by implementation, incontract strata. The dashed line marks the ∼94% figure reported in prior work (Hugging Face, 2025b).

3.2

Mag.

53.8%

math-verify (expr)

Whitespace dominates the error budget

Table 2 and Figure 2 decompose in-contract failures by stratum. For mv-latex, whitespace and punctuation account for 93.0% of all failures; for mv-expr, 74.1%. The remaining mass is spread thinly across sets, delimiters, and fraction handling. Table 3 gives per-stratum rates with Wilson intervals. The whitespace stratum is measured at n = 15,704 with intervals of ±0.8 points, so the effect is not a small-sample artefact. Table 8 shows representative pairs: an expression is accepted, and the same expression followed by a period is rejected. 3.3

A scale-dependent false-positive mechanism

Table 4 and Figure 3 report acceptance of off-byone adversarial answers by gold magnitude. Three verifiers reject all of them at every magnitude. sympy-cascade rejects all of them below 104 and accepts all of them at or above 104 . The mechanism is that a relative tolerance of 10−4 is scaleinvariant: for a gold answer of 10,000, the value

74%

math-verify (expr)

(strip-string), and 88.3% (sympy-cascade). Ordering and magnitude are preserved; synthesis is not driving the results.

23% 93%

math-verify (latex)

17%

sympy cascade

52% 50%

strip_string 0%

16%

4

delimiters

The residual is string hygiene, not grammar. Prior attribution—“LaTeX parsing failures”— implies difficult grammar work. The decomposition shows otherwise: 93.0% of one verifier’s in-contract failures are whitespace and punctuation. This is more actionable and more surprising than the aggregate suggests.

32%

20%

40%

60%

80%

100%

share of in-contract certified failures

whitespace math-mode

unreduced frac frac/decimal

sets sqrt/exponent

hatched = failure caused by execution error

Figure 2: Error budget: share of in-contract certified failures by stratum. Hatched segments indicate failures caused by execution error rather than rejection.

10,001 differs by 0.01% and is accepted. The aggregate off-by-one false-positive rate for this verifier is 16.0% (95% CI [14.1, 18.1], n = 1,287), but the aggregate conceals a threshold.

off-by-one acceptance

100% 75%

math-verify (expr) math-verify (latex) strip_string sympy cascade

50%

tolerance threshold |g| 1/rel_tol = 104

25% 0%

< 10

101

102

103

gold answer magnitude

104

105+

Figure 3: Off-by-one acceptance by gold answer magnitude. The reference cascade steps deterministically at 104 ; the other three verifiers reject all off-by-one variants at every magnitude.

3.4

Reported accuracy is not comparable across papers

Table 6 shows acceptance on contract-dependent input classes. Acceptance on boxed answers ranges from 0.0% to 75.1%; on scientific notation, from 0.0% to 100.0%. These are not defects—each implementation behaves as designed—but the differences are undocumented. Table 5 quantifies the consequence. Two configurations of the same library disagree on 49.9% of certified-equivalent pairs; panel (b) of the same table gives the pair counts supporting each cell. 3.5

Robustness: corpus-only

Excluding all synthetic golds, self-validation is 57.5% (mv-expr), 84.1% (mv-latex), 87.8%

Discussion

Verifier configuration must be reported. Acceptance on identical inputs ranges 0%–100% by configuration, and two configurations of one library disagree on 49.9% of cases. A reported benchmark accuracy is therefore a function of the verifier configuration, which is almost never stated. We recommend reporting verifier identity, version, extraction configuration, and runtime. Coverage is a first-class metric. §2.3 is not merely a reporting nicety. Two verifiers with similar aggregate residuals—sympy-cascade at 12.7% and strip-string at 4.8%—fail for opposite reasons, and only the coverage split reveals it. We recommend that verifier evaluations report coverage alongside accuracy. Scale-dependent false positives. Table 4 shows a threshold rather than a rate. Egashira et al. (2026) distinguish systematic from random verifier error and show that systematic false positives can produce plateaus or collapse, with outcomes determined by the error pattern rather than the overall rate. A magnitude threshold is maximally systematic and invisible in any aggregate figure. Relation to reported false-negative rates. Reported false-negative rates near 38% (Cai et al., 2025) are measured on real model outputs, which include mathematical non-equivalences such as unreduced fractions. Our measurement isolates the formatting and equivalence component under certified transformations and is therefore a lower bound on that quantity. The two are complementary.

5

Limitations

Verifier-level, not model-level. We do not estimate the effect on any model’s benchmark score; real outputs do not produce transform variants at equal rates. No training runs. We do not measure

downstream RLVR effect; Zhang (2026) examine hardened versus leaky rewards in code RLVR and find bounded effects. Open implementations only. No claims about closed frontier systems. Synthetic component. 2,000 of 4,990 golds are synthetic; §3.5 reports corpus-only results separately. Contract assignment is a judgement. We publish the matrix so it can be contested.

6

Evaluation brittleness. Su et al. (2025) show that changing the single character separating incontext examples moves MMLU accuracy by up to ±23%. Hua et al. (2025) argue much reported prompt sensitivity is an artefact of heuristic scoring. Our results are the verifier-side counterpart to both. Miller (2024) argues for statistical rigour in evaluation reporting; we attach Wilson intervals to every cell.

Related Work 7

Verifier noise in RLVR. Cai et al. (2025) model the verifier as a stochastic reward channel with asymmetric noise rates and derive corrections requiring the false-negative rate as input. Their motivating example, a checker marking 12 36 wrong 1 against a canonical 3 , is one of our strata. Their method assumes the rates; we measure and decompose them. Egashira et al. (2026) distinguish systematic from random verifier error and show outcomes are determined by the error pattern; our per-stratum decomposition characterises that pattern, and the magnitude threshold in §3 is a maximally systematic instance. Huang et al. (2026) compare rule-based and model-based verifiers and find opposing failure directions; we extend the falsenegative side with a category-level decomposition and add certified adversarial probes. Verifier and benchmark auditing. Ammanamanchi et al. (2026) audit five Lean theoremproving benchmarks with corpus-scale static checkers, surfacing 4,833 findings and proposing a fault taxonomy with released tooling. We adopt the same posture for natural-language answer verification. In code RLVR, Zhang (2026) run a preregistered causal contrast between leaky and hardened reward suites, and Rajan (2026) measure that 28.5% of SWE-bench Verified tasks accept Docker-verified incorrect patches. Outside RLVR, Lan et al. (2026) make a structurally similar argument for security scanners: conventional metrics characterise only cases where a tool yields a usable judgement, so coverage must be reported separately from accuracy. Metamorphic testing. Semantics-preserving transformation as a robustness probe has an established lineage (Asgari et al., 2025; Hyun et al., 2024; Zhou et al., 2026). All of this work perturbs the input to test the model. We invert the target: the transformation is applied to the answer string, and the system under test is the grader.

Conclusion

Verifier reliability is a measurable property that varies by 41 points across widely used implementations, is dominated by whitespace handling rather than mathematical parsing, and includes at least one deterministic false-positive mechanism triggered by answer magnitude. Separating rejection from execution failure shows that implementations with similar aggregate residuals fail for opposite reasons. The transform suite, contract matrix, and per-sample verdict records are released with this paper.

References Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. 2025. Big-math: A large-scale, highquality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502.17387. Pawan Sasanka Ammanamanchi, Siddharth Bhat, and Stella Biderman. 2026. Faults in our formal benchmarking: Dataset defects and evaluation failures in Lean theorem proving. arXiv preprint arXiv:2606.29493. Ali Asgari, Milan de Koning, Pouria Derakhshanfar, and Annibale Panichella. 2025. Metamorphic testing of deep code models: A systematic literature review. Journal of the ACM. ArXiv:2507.22610. Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, and Masashi Sugiyama. 2025. Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers. arXiv preprint arXiv:2510.00915. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, and Martin Vechev. 2026. Delay, plateau, or collapse: Evaluating the

impact of systematic verification error on RLVR. In Conference on Language Modeling (COLM). ArXiv:2605.02909. Daya Guo, Dejian Yang, Haowei Zhang, et al. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.

Jingtong Su, Jianyu Zhang, Karen Ullrich, Léon Bottou, and Mark Ibrahim. 2025. A single character can make or break your LLM evals. arXiv preprint arXiv:2510.05152. Chuyifei Zhang. 2026. When the reward suite is leaky: A preregistered causal contrast of natural verifier false positives in RLVR. arXiv preprint arXiv:2607.11022.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks Track.

Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou, Weibin Lin, and Zheng Zheng. 2026. LGMT: Logic-grounded metamorphic testing for evaluating the reasoning reliability of LLMs. arXiv preprint arXiv:2605.23965.

Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. 2025. Flaw or artifact? rethinking prompt sensitivity in evaluating LLMs. In Proceedings of EMNLP. ArXiv:2509.01790.

A

Yuzhen Huang, Weihao Zeng, Xingshan Zeng, Qi Zhu, and Junxian He. 2026. From accuracy to robustness: A study of rule- and model-based verifiers in mathematical reasoning. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP). ArXiv:2505.22203. Hugging Face. 2025a. Math-Verify: A robust mathematical expression evaluation system. https:// github.com/huggingface/Math-Verify. Version 0.9.0, ANTLR4 runtime 4.13.2. Hugging Face. 2025b. Troubleshooting math parsing. https : / / github . com / huggingface / evaluation-guidebook. Reports approximately 94% self-validation of ground truth under SymPy parsing. Sangwon Hyun, Mingyu Guo, and M. Ali Babar. 2024. METAL: Metamorphic testing framework for analyzing large-language model qualities. In IEEE International Conference on Software Testing, Verification and Validation (ICST). ArXiv:2312.06056. Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, and Indranil Sanyal. 2026. Beyond F1: Evaluating coverage and failure recovery in AI model security scanners. arXiv preprint arXiv:2608.27424. Evan Miller. 2024. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640. Shreshth Rajan. 2026. Auditing reward hackability in code RL training environments. arXiv preprint arXiv:2606.16062. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

Supplementary Tables

Tables 5–8 give the cross-verifier disagreement matrix with its pair support, acceptance on contractdependent input classes, out-of-contract behaviour, and representative certified-equivalent pairs that production verifiers reject. Throughout, V1 = mv-expr, V2 = mv-latex, V3 = strip-string, and V4 = sympy-cascade. (a) Disagreement rate (%) V1 V2 V3 V4

V1 — 49.9 48.0 53.4

V2 49.9 — 31.4 22.0

V3 48.0 31.4 — 0.4

V4 53.4 22.0 0.4 —

V3 31,266 31,266 31,266 27,299

V4 27,299 27,299 27,299 27,299

(b) Pairs judged by both V1 V2 V3 V4

V1 31,266 31,266 31,266 27,299

V2 31,266 31,266 31,266 27,299

Table 5: Pairwise disagreement on certified-equivalent pairs, computed only over pairs where both verifiers returned a verdict. V1 = mv-expr, V2 = mv-latex, V3 = strip-string, V4 = sympy-cascade. Two configurations of the same library (V1, V2) disagree on 49.9% of cases. Support is lower for pairs involving V4 because it does not return a verdict on every input.

n

Input class boxed scientific with unit text prefix percent

V1

V2

V3

19,952 8.0 75.1 0.0 326 0.0 100.0 0.0 2,782 50.0 100.0 100.0 9,980 28.2 39.6 48.9 20 0.0 100.0 0.0

Table 6: Acceptance rate (%) on contract-dependent input classes, over verdicts actually returned. V1 = mv-expr, V2 = mv-latex, V3 = strip-string. Classes are a boxed answer, scientific notation, a trailing unit, the prefix “The answer is”, and a percent sign. Acceptance spans 0–100% on identical inputs: these are undocumented contract differences, not defects. sympy-cascade is omitted because it returns no verdict on boxed or scientific inputs.

Stratum

Verifier

S10 sets S14 unred S14 unred S10 sets S1 frac d. S4 delim S7 sqrt

strip-string mv-expr strip-string mv-expr mv-expr mv-expr mv-expr

n Rejection 634 2,062 2,062 634 4,146 1,067 404

100.0% 100.0% 100.0% 97.5% 93.8% 87.1% 31.4%

Table 7: Out-of-contract behaviour: strata a verifier does not claim to handle. Reported for completeness; not counted as defects.

Gold

Variant

Rejected by

\frac{1}{2} \frac{1}{2} \frac{1}{2} (a+5)(b+2) 3\pm\sqrt{2}

\frac{1}{2}. \frac{1}{2}\n \dfrac{1}{2} \left(a+5\right) 3\,\pm\,\sqrt{2}

V2, V1 V2, V1 V1 V1 V3

Table 8: Certified-equivalent pairs rejected by production verifiers. V1 = mv-expr, V2 = mv-latex, V3 = strip-string. A trailing period changes the verdict. The fourth variant is truncated for width; the full form is \left(a+5\right)\left(b+2\right).

Related documents

• Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts#1129470
• Rater State Bias in RLHF Preference Data: An Audit Framework#1130589
• Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR#1131563
• VStress: Correlation-Aware Auditing and Adaptive Budget Allocation for Repeated Verifiers#1132159
• VStress: Correlation-Aware Auditing and Adaptive Budget Allocation for Repeated Verifiers#1122207