Conceptio › Archive › arXiv CS

arXiv CSopen access

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

software-architecture software-engineering testing

software engineering, software architecture, testing

Preprint. Under review.

B ENCH G UARD: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks Xinming Tu1,∗ Kexin Huang2

Tianze Wang1 Yuanhao Qu2,∗

Yingzhou (Minta) Lu2 Sara Mostafavi1,3,∗

1 Allen School, University of Washington, Seattle, WA, USA 2 Phylo, Inc., South San Francisco, CA, USA

arXiv:2604.24955v1 [cs.CL] 27 Apr 2026

3 Genentech, Inc., South San Francisco, CA, USA ∗ Corresponding authors

Abstract As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all—they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through B ENCH G UARD, the first automated auditing framework for taskoriented, execution-based agent benchmarks. B ENCH G UARD cross-verifies all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific benchmarks, B ENCH G UARD identified 12 author-confirmed issues in ScienceAgentBench—including fatal errors rendering tasks unsolvable—and exactly matched 83.3% of expertidentified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under $15, making automated benchmark auditing a practical and valuable complement to human review. These findings point toward AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.

Introduction

As LLM agents tackle increasingly complex tasks—resolving software issues (Jimenez et al., 2024), navigating web environments (Zhou et al., 2024), and automating end-to-end AI research (Lu et al., 2026)—a growing ecosystem of execution-based benchmarks has emerged to measure these capabilities (Jimenez et al., 2024; Zhou et al., 2024; Xie et al., 2024; Merrill et al., 2026). Unlike static, multiple-choice datasets (Hendrycks et al., 2021), these benchmarks are tightly coupled evaluation pipelines that comprise natural language instructions, executable ground-truth programs, evaluation scripts, and containerized environments, making the measurement infrastructure itself susceptible to systemic logical flaws. This fragility is particularly acute in specialized scientific domains (Chen et al., 2025; Mitchener et al., 2025), where evaluation demands deep domain expertise. Relying on domain experts to construct these benchmarks is necessary, and the limitations of current benchmarking methodology are well recognized (Bowman & Dahl, 2021), yet recent benchmark-repair efforts show that even widely used, human-validated evaluations contain enough noise to distort capability measurements—pervasive label errors have been documented across flagship static benchmarks (Gema et al., 2025; Northcutt et al., 2021). In execution-based settings, the problem is compounded: SWE-bench Verified was constructed specifically to filter unreliable tasks, yet subsequent analysis found residual issues even in this curated subset (OpenAI, 2024; 2026). This noise is uniquely difficult to detect because correctness emerges from interactions among instructions, ground-truth programs, evaluation 1

Preprint. Under review.

scripts, and environments—not from any single label. We interpret many of these persistent errors through the lens of solution fixation (Jansson & Smith, 1991) and the curse of knowledge (Camerer et al., 1989): benchmark creators anchor to their own implementations and unconsciously assume that implicit choices are self-evident, leaving instructions underspecified (Gema et al., 2025) and evaluation suites insufficient or misaligned (Liu et al., 2023; OpenAI, 2026)—a pattern consistent with pervasive annotation errors in static benchmarks (Northcutt et al., 2021), but harder to detect when spread across interacting artifacts. To break this bottleneck, we propose employing frontier LLMs not merely as subjects of evaluation or output judges (Zheng et al., 2023; Li et al., 2025), but as systematic auditors of the evaluation infrastructure itself—a direction supported by evidence that LLM critics can identify defects that human oversight missed (McAleese et al., 2024). We introduce B ENCH G UARD, to our knowledge the first automated auditing framework for task-oriented, execution-based agent benchmarks. While prior work has produced verified or revised benchmark variants through re-annotation and model-assisted auditing for static or question-answering settings (Gema et al., 2025; Zhai et al., 2026), and through expert re-annotation for execution-based settings such as SWE-bench Verified (OpenAI, 2024), no existing framework systematically audits the coupled artifact stack of executionbased benchmarks. The framework cross-references all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence to improve recall. We also introduce an empirically grounded taxonomy of 4 categories and 14 subcategories for classifying benchmark defects. We scope the framework to task-oriented, execution-based benchmarks (Section 3.1). Deployed on two prominent agent benchmarks, B ENCH G UARD exposed the fragility of current humanreviewed evaluations. On ScienceAgentBench, despite the benchmark undergoing multiple rounds of manual validation by annotators and subject matter experts, we identified 12 defects—all subsequently confirmed by the original authors. On BIXBench, which underwent independent review prior to publication, a five-model B ENCH G UARD ensemble exactly matched 83.3% of expert-identified issues on the Verified-50 subset, while also surfacing high-confidence defects that prior manual review had entirely missed. These results are consistent with growing evidence that follow-up audits continue to uncover benchmark flaws even after careful curation (Gema et al., 2025; Siddiq et al., 2024; OpenAI, 2026), but reveal a new dimension of the problem specific to execution-based benchmarks: tightly coupled software artifacts create failure modes invisible to dataset-only auditing, suggesting that automated, AI-assisted auditing is a valuable complement to human review in the benchmark lifecycle. Cases 1–2 illustrate two such failures discovered in ScienceAgentBench, where pink highlights mark the cross-artifact discrepancy: Case 1: Wrong Input File

Case 2: SMILES vs. Drug Names

INST

EVAL

Instruction: “...perform the analysis using ecg_1000hz.csv ...”

Instruction: “...save ordered list of drugs..., with one SMILES per line .”

Reference Solution: pd.read_csv( ’ bio_eventrelated_100hz.csv ’) Defect: Instruction points to the wrong file; the reference solution uses a different one.

Evaluation Script: if "Foscarnet" in pred[i]: Defect: Evaluator checks for drug names, contradicting the SMILES format the instruction requests.

Both defects passed multiple rounds of human review and were subsequently confirmed by the original benchmark authors. Such cross-artifact mismatches are invisible to singlecomponent review but precisely what automated auditing catches systematically. Our main contributions are as follows: 1. AI-Assisted Evaluation Paradigm and Error Taxonomy. We characterize cross-artifact consistency failures in agent benchmarks (motivated by solution fixation as an explanatory lens) and introduce a 4-category, 14-subcategory error taxonomy grounded in the constituent artifacts of execution-based benchmarks and validated through confirmed defects across production benchmarks. 2

Preprint. Under review.

2. The B ENCH G UARD Framework. We present the first automated audit framework for execution-based agent benchmarks, capable of auditing 50 complex tasks with five frontier models for under $15. B ENCH G UARD will be open-sourced upon publication to support community adoption. 3. Empirical Validation. We uncover author-confirmed, solve-rate-altering defects in ScienceAgentBench and demonstrate expert-level alignment on BIXBench, showing that even careful human review can be insufficient. We survey related work in Section 2, present the framework in Section 3, and report empirical findings in Sections 4–5.

Figure 1: Overview of the coupled artifact stack audited by B ENCH G UARD. Each task is defined jointly by four interlocking components: the natural-language instruction, the environment, the reference gold solution, and the evaluation logic. B ENCH G UARD treats benchmark auditing as a cross-artifact consistency problem: misalignment among any of these pieces can produce benchmark defects, which are surfaced as structured findings for expert review.

Related Work

B ENCH G UARD is most closely connected to two threads: (i) post-hoc auditing and “verified” revisions of benchmarks, and (ii) evaluation methodology for execution-based agent benchmarks, where correctness is implemented through coupled executable artifacts rather than static labels. 2.1

Benchmark Auditing and Verified Revisions

Benchmark integrity has become a first-order concern as leaderboards saturate and evaluation artifacts are reused at scale. In static QA benchmarks, careful audits have shown that seemingly small annotation or specification errors can materially alter measured performance, motivating verified subsets and re-annotation efforts such as MMLU-Redux (Gema et al., 2025). Similar dynamics are emerging in newly proposed expert-level evaluations: Humanity’s Last Exam (HLE) was introduced to raise the ceiling, yet it quickly attracted follow-up auditing and structured verification efforts, culminating in revision pipelines such as HLE-Verified (Phan et al., 2026; Zhai et al., 2026). In mathematical reasoning, systematic auditing of the Omni-MATH benchmark revealed dataset-induced noise that necessitated manual revision (Ballon et al., 2026). In the code domain, quality assessments have likewise highlighted that benchmark artifacts beyond labels—including prompts and 3

Preprint. Under review.

dataset construction choices—can systematically skew conclusions (Siddiq et al., 2024). In execution-based settings, SWE-bench Verified (OpenAI, 2024) adopted human re-annotation to filter unreliable tasks, yet subsequent evidence suggests that even this verified subset contains residual issues (OpenAI, 2026). These efforts establish an important lesson: benchmark quality is not guaranteed, and verification is often a necessary second step. However, most prior auditing pipelines still treat the evaluation oracle as relatively self-contained (a label, rubric, or a single checker). Execution-based agent benchmarks break this assumption: correctness is realized through interactions among instructions, ground-truth programs, evaluation logic, and runtime environments, creating failure modes that cannot be captured by dataset-only auditing. 2.2

Execution-Based Agent Benchmarks and Evaluation Reliability

Agent benchmarks increasingly assess long-horizon problem solving in executable environments, including repository-level software engineering (Jimenez et al., 2024), scientific analysis workflows (Chen et al., 2025), and research-style domains such as bioinformatics (Mitchener et al., 2025). A defining property of these settings is that each task is a coupled system: natural-language specifications are operationalized through evaluation harnesses (scripts, tests, and sometimes model judges) under specific environment assumptions. As a result, mis-measurement can arise from subtle specification–implementation mismatches even when each component appears reasonable in isolation. Human verification is a pragmatic remedy and has been adopted in prominent benchmarks (e.g., SWE-bench Verified) (OpenAI, 2024). Yet recent evidence suggests that one-off review is not sufficient as a lifecycle strategy: test suites can still reject functionally correct solutions, and contamination and evaluator limitations can increasingly dominate reported scores, leading to shifts away from previously “verified” benchmarks for frontier evaluation (OpenAI, 2026). More broadly, the growing use of model judges in evaluation introduces another source of measurement error: judge reliability can itself become a bottleneck as models advance (Zheng et al., 2023; Kim et al., 2024; Ballon et al., 2026).

Method

3.1

Overview

B ENCH G UARD takes as input the four constituent artifacts of an execution-based benchmark task: the natural-language instruction, the ground-truth program (reference solution), the evaluation script, and the environment configuration. It produces a structured set of findings, each annotated with an error category from our taxonomy (Section 3.2), a severity level, a confidence score, and supporting evidence with source-level citations. The core definition-level audit (Section 3.3) performs LLM-driven cross-verification of the four artifacts, supplemented by deterministic static checks, to detect logical inconsistencies visible from the benchmark specification alone. When agent solutions or execution traces are available, they can be appended to the audit context as additional diagnostic evidence (Section 3.3). Concretely, the pipeline ingests benchmark data in a standardized format compatible with Harbor (Harbor Framework Team, 2026) (Appendix C), constructs a verification context for each task bundling all available artifacts, and dispatches it to LLM verification protocols and deterministic checks running in parallel. Outputs are aggregated into a structured report with per-task cost tracking. Rather than acting as a fully autonomous judge, B ENCH G UARD surfaces high-confidence findings with supporting evidence for expert adjudication, establishing a human–AI collaborative workflow. We scope B ENCH G UARD to task-oriented, execution-based agent benchmarks: settings where each task has a well-defined input, a target output, and verification logic (deterministic scripts or structured judges, including model-based evaluators with reproducible protocols). It excludes open-ended conversational agents, interactive tasks without deterministic verifi4

Preprint. Under review.

Table 1: The B ENCH G UARD error taxonomy: four top-level categories of agent benchmark defects. Category

Sub.

GT (Ground Truth)

EVAL (Evaluation)

INST (Instruction)

ENV (Environment)

Description Errors in the reference solution: wrong logic, wrong data handling, or format mismatches Flaws in evaluation scripts: judge bias, spec–eval mismatches, incomplete coverage, wrong tolerances, unhandled non-determinism Defects in task specifications: underspecified requirements, internal contradictions, or infeasible tasks Runtime configuration issues: missing dependencies, path errors, or resource constraints

cation, and agents that produce irreversible real-world side effects (e.g., purchasing, email sending). 3.2

Error Taxonomy

We organize benchmark defects into a structurally benchmark-agnostic taxonomy of four top-level categories (Table 1), iteratively refined through audits of ScienceAgentBench and BIXBench; the full 14 subcategories are in Appendix A. While the top-level categories generalize to any execution-based benchmark, some subcategories may require extension for non-scientific domains (e.g., web navigation, OS-level tasks). Each finding produced by B ENCH G UARD is annotated with a severity level (Critical/High/Medium/Low) capturing impact on benchmark correctness, and a confidence score—the LLM’s self-assessed probability that the defect is real—bucketed into three tiers: Confirmed (0.8–1.0), Likely (0.55–0.79), and Possible (0.3–0.54); findings below 0.3 are suppressed. Detailed definitions are in Appendix A. 3.3

Definition-Level Audit

The definition-level audit detects defects that are visible from the benchmark specification alone, without requiring any agent execution. It combines LLM-driven verification protocols with deterministic static checks. LLM Verification Protocols. The core of the definition-level audit is a single consolidated LLM call that cross-verifies all four artifact types, with an explicit deduplication phase to prevent duplicate findings when a single root cause manifests across multiple categories. The protocol employs six-phase structured reasoning: (1) Task Understanding: parse the instruction to identify the target output specification; (2) Ground Truth Correctness: verify the gold program implements the correct algorithm, metrics, and data handling; (3) Evaluation Logic: verify the evaluation script measures what the specification asks, checking for judge bias, spec–eval mismatches, and incomplete coverage; (4) Task Specification: check for underspecified requirements, internal contradictions, and infeasible tasks; (5) Environment: flag runtime issues such as path mismatches and missing dependencies; and (6) Consolidation: deduplicate via the one-fix test (if one fix would resolve multiple findings, report the root cause once), verify each finding is atomic (the split test), filter runtime-specific artifacts, and emit the final structured report. The system prompt includes the full taxonomy as a reference, and the user prompt presents all available task artifacts. A key design choice is the categorization priority: when an issue could be classified as either an instruction or groundtruth defect, the protocol defaults to the instruction-level category. Ground-truth programs are typically derived from existing codebases and are inherently more complete; the instruction is where implicit assumptions most often go undocumented—a direct consequence of solution fixation (Section 1). 5

Preprint. Under review.

Execution-Level Audit. When agent traces are available, B ENCH G UARD runs an executionlevel audit that appends the agent’s generated program, evaluation result, and evaluation log to the audit context. Agent programs exercise code paths and edge cases invisible to static review—for instance, revealing that a tolerance is too strict only when a correct-butdifferently-ordered output is scored. On SAB, this improves recall (e.g., Opus 4.6 from 83.3% to 91.7%; parenthetical values in Table 2); on BIXBench, execution-level results are mixed across models and reported in Appendix I.

Experimental Setup

Benchmarks. We evaluate B ENCH G UARD on two agent benchmarks selected to cover complementary evaluation paradigms and quality-assurance histories: • ScienceAgentBench (SAB) (Chen et al., 2025): 102 scientific data-analysis tasks spanning computational biology, geoscience, psychology, and materials science. Each task is evaluated by a deterministic Python script that compares agent output to the gold program’s output using domain-specific metrics. The benchmark was validated through multiple rounds of review by its creators and nine subject-matter experts. We use the 12 authorconfirmed defects discovered through our auditing process as ground truth for measuring recall. • BIXBench (Mitchener et al., 2025): 205 bioinformatics tasks evaluated via a combination of LLM judges, exact string matching, and numerical range verification. The benchmark underwent independent review prior to publication, providing external expert revisions as ground truth for computing recall. We audit the Verified-50 subset (Phylo, 2026)—50 tasks that were independently re-reviewed by domain experts, of which 17 received revised question text, corrected ground-truth answers, or both. We use these 17 revisions as ground truth: a B ENCH G UARD finding is aligned if it identifies the same root cause as the human revision, partial if it flags the correct functional area without pinpointing the exact issue, and unrelated otherwise. We atomically decompose the 17 expert revisions into 24 independent issues (Appendix H) for finer-grained alignment matching. LLM Backends. To assess robustness across model families, we run the definition-level audit with five frontier LLM backends, all accessed through the litellm library: (1) Gemini 3.0 Flash (Google, via Vertex AI), (2) Gemini 3.1 Pro (Google, via Vertex AI), (3) GPT-5.4 (OpenAI), (4) Claude Opus 4.6 (Anthropic), (5) Claude Sonnet 4.6 (Anthropic). Using multiple models allows us to assess cross-model agreement and ensemble strategies. Configuration. All audits use definition mode with deterministic checks enabled, temperature = 0.0 for reproducibility (temperature = 1.0 for Gemini models, which do not support 0.0; these models were run once, so results may vary across runs), and a maximum of 4,096 output tokens per LLM call. Each task is audited independently in a single LLM call via the DefinitionProtocol. Evaluation Metrics.

We evaluate B ENCH G UARD along two axes:

• Recall: the fraction of known confirmed defects that B ENCH G UARD successfully identifies. For both benchmarks, we use an LLM alignment judge (Appendix J) to classify each (finding, gold issue) pair as aligned, partial, or unrelated. For SAB, gold issues are the 12 author-confirmed defects; for BIXBench, we decompose the 17 expert-revised questions in the Verified-50 subset into 24 atomic issues. We report RecallA (exact alignment only) and RecallA+P (including partial matches). • Precision: the fraction of B ENCH G UARD’s reported findings that correspond to genuine benchmark defects, as determined by the alignment judge. We report precision on the subset of tasks containing confirmed defects (flagged-task precision), which reflects the triage-relevant signal. Validation Protocol. All findings are reviewed by human experts with domain knowledge. For SAB, findings were independently submitted to and confirmed by the original 6

Preprint. Under review.

Table 2: Recall and precision on ScienceAgentBench (12 confirmed defects, 102 tasks). Precision is flagged-task precision (computed over findings within tasks containing confirmed defects, counting aligned and partial matches). Find. = number of findings on defective tasks. Cost is per-model total in USD (definition-only / + agent). Definition-Only

+ Agent Programs

Cost ($)

Model

RecA RecA+P Prec Find. RecA RecA+P Prec Find.

Def

+Ag

Gemini 3.0 Flash Gemini 3.1 Pro GPT-5.4 Opus 4.6 Sonnet 4.6

58.3 58.3 58.3 83.3 58.3

75.0 75.0 75.0 91.7 91.7

41.2 57.9 47.1 58.1 51.9

34 19 34 31 27

66.7 58.3 83.3 91.7 83.3

91.7 83.3 91.7 100 91.7

50.0 70.4 60.0 67.6 59.0

46 27 45 37 39

0.84 0.95 3.67 5.82 2.71 3.10 9.50 10.94 6.00 6.76

Ensemble

91.7

100

—

100

—

22.72 27.57

benchmark authors. For BIXBench, findings are validated against the Verified-50 ground truth and through independent expert review.

Results and Analysis

5.1

Exposing Measurement Noise in ScienceAgentBench

We evaluate B ENCH G UARD’s ability to recover the 12 author-confirmed defects in ScienceAgentBench. These are not marginal quality issues or stylistic preferences: they include fatal specification errors that render tasks completely unsolvable, metric mismatches that silently corrupt scores, and evaluation logic that rejects correct solutions. Recall is measured against these 12 defects, which were discovered during auditing and subsequently confirmed by the benchmark authors; additional latent defects may exist beyond our validated ground truth. Table 2 summarizes per-model recall and precision; the detailed per-defect breakdown is in Appendix E, which also shows that models exhibit complementary strengths across error categories. Precision in Context. The flagged-task precision in Table 2 conditions on tasks that contain confirmed defects; it measures how well practitioners can act on findings once a defective task is identified. We report flagged-task precision because it best reflects the intended triage workflow: B ENCH G UARD flags candidate tasks, and experts review the associated findings. For a human-in-the-loop auditing tool, recall is the primary objective: missing a genuine defect risks corrupting leaderboard conclusions, whereas a false positive costs only a few seconds of expert review. 5.2

Validating the Boundary of AI Auditing on BIXBench

BIXBench presents a distinct evaluation paradigm: tasks are evaluated by a mix of LLM judges and deterministic verifiers (string matching, range checks), introducing new classes of potential defects—particularly in LLM-judge rubrics. We validate B ENCH G UARD against the BIXBench Verified-50 subset, where 17 of 50 tasks were revised by independent domain experts. All BIXBench results use definition-level auditing; execution-level results are reported in Appendix I. We decompose these revisions into 24 atomic issues to enable fine-grained recall and precision measurement. Overall Findings. Across the 50-task subset, each of the five auditor models surfaces between 43 and 114 findings (Table 6), with INST-INCOMPLETE consistently dominating across all models and GT-LOGIC and EVAL-COVERAGE as frequent secondary categories. The most frequent issue is underspecified instructions and evaluation rubrics anchored to a single reference implementation, penalizing valid alternative approaches. Cases 3–4 illustrate 7

Preprint. Under review.

Table 3: Alignment with human expert revisions and audit cost on BIXBench Verified-50. Metrics are measured against 24 atomic issues from 17 expert-revised tasks; precision reflects alignment rate with expert revisions on flagged tasks (not all 50). Ensemble findings are pooled without cross-model deduplication. Cost is for the full 50-task audit. Model

RecallA

RecallA+P

PrecisionA

PrecisionA+P

Cost ($)

Find.

Gemini 3.0 Flash Gemini 3.1 Pro GPT-5.4 Opus 4.6 Sonnet 4.6

45.8 37.5 50.0 54.2 33.3

95.8 58.3 87.5 79.2 58.3

33.3 47.1 23.3 38.7 23.3

83.3 76.5 55.8 67.7 60.0

0.53 2.31 1.92 5.98 3.64

114 43 102 66 58

Ensemble (any)

83.3

95.8

—

14.38

383

representative defects where gold programs embed undocumented methodological choices absent from the instruction. Case 3: Silent Data Dropping

Case 4: Undocumented Covariates

INST

Instruction: “...differential expression analysis... comparing disease (ASXL1) vs control ...” Gold Program: design=~ sex + condition Defect: Gold program includes sex covariate not mentioned in the instruction.

Instruction: “...Pearson correlation between gene length and mean expression for protein-coding genes...” Gold Program: df_filtered = df[df.sum(axis=1) >= 10 ] Defect: Undocumented expression filter absent from the instruction.

Alignment with Human Expert Revisions. Using the 17 expert-revised tasks as ground truth, an LLM judge classifies each B ENCH G UARD finding against each atomic gold issue as aligned (identifies the same root cause), partial (flags the correct area without pinpointing the exact issue), or unrelated. Table 3 reports per-model alignment and audit cost. The five-model union achieves 83.3% exact alignment (20/24 issues detected by at least one model) and 95.8% recall when including partial matches (23/24), while the best single model (Opus 4.6) exactly aligns on 13 of 24 issues (54.2%). Only one issue (an ambiguous “frequency ratio” definition in bix-54-q7) eludes all models entirely; the remaining partially matched issues involve domain-specific conventions rather than structural cross-referencing of artifacts. We explore the practical collaboration boundary between human and automated auditing in Cases 5–6 below. Case 5: Format Over-Prescription

Case 6: Undocumented Sample Removal INST

INST

Original Question: “What is the difference between median treeness values for fungi versus animals?” Expert Added: “Report the difference as a decimal proportion (not percentage) .” Partial match. The LLM judge accepts both formats; the added constraint is arguably unnecessary.

Instruction: “... all significantly differentially expressed genes...” Gold Program: drop samples KL3, WL3 Caught by all 5 models, missed by experts. Gold silently removes two samples not mentioned in the instruction.

The Collaboration Boundary. Cases 5–6 illustrate where human and automated auditing have complementary blind spots. In Case 5, the expert revision added a format constraint to a question whose LLM judge already accepts equivalent representations—an arguably overprescriptive revision where B ENCH G UARD’s partial match reflects appropriate restraint. In Case 6, the gold program silently drops two samples not mentioned in the instruction; all five models flag this with high severity, yet the expert review did not annotate it—likely because the gold-program author knew the removal was methodologically motivated. The 8

Preprint. Under review.

collaboration boundary runs both ways: experts can over-correct where automated auditing shows restraint, and automated auditing catches structural inconsistencies that solution fixation renders invisible to experts. Because precision is measured against expert revisions, such novel detections are counted as unaligned; the reported precision is therefore better understood as a human alignment rate rather than a true error rate. 5.3

Complementary Diagnostic Profiles

Models exhibit complementary diagnostic strengths with substantial but imperfect overlap in their detection sets (Appendix G). The five-model union achieves 83.3% exact alignment (20/24 issues), compared to 54.2% for the best single model, with each model contributing unique detections (detailed breakdowns in Table 6). The entire five-model audit of 50 tasks costs under $15 and completes in under 12 minutes, making multi-model ensembles practical.

Discussion and Conclusion

Our audits of two prominent, human-reviewed benchmarks show that benchmark defects are far more prevalent than commonly assumed. B ENCH G UARD uncovered 12 defects in ScienceAgentBench that were later confirmed by the benchmark authors, including fatal errors that rendered tasks unsolvable, and exactly matched 83.3% of independent expert revisions on BIXBench while surfacing high-confidence defects that prior human review missed entirely. These results suggest that even multiple rounds of careful expert validation may be insufficient to ensure benchmark integrity, and that automated auditing is a valuable complement to human review. LLM-based auditing is not without limitations— findings can be hallucinated, requiring expert review in the loop, and our taxonomy may need extension beyond scientific computing—but at under $15 for a five-model audit of 50 complex tasks, the cost is negligible compared to that of publishing benchmarks with undetected measurement errors. Beyond defect detection, automated audit findings can directly inform benchmark revision, providing actionable evidence for repairing instructions, gold programs, and evaluation logic. More importantly, auditing need not remain a post-hoc remediation step: integrating it into the benchmark construction process itself would surface implicit assumptions before they propagate into published artifacts. This points toward a broader opportunity: if frontier models can reliably audit evaluation infrastructure, they can also serve as quality gates within automated benchmark construction pipelines—enabling the community to scale benchmark development from domain literature with built-in integrity guarantees. Reliable evaluation at scale is ultimately a prerequisite for reliable capability improvement; ensuring that benchmarks measure what they claim to measure is foundational to the progress they are designed to track.

LLM Disclosure This work uses frontier LLMs as core research instruments (benchmark auditors and alignment judges), as described in Sections 3–5. Additionally, LLMs were used to review and refine drafts of this manuscript; all scientific claims, experimental design, and analysis were conducted by the authors.

References Marthe Ballon, Andres Algaba, Brecht Verbeken, and Vincent Ginis. Benchmarks saturate when the model gets smarter than the judge. arXiv preprint arXiv:2601.19532, 2026. doi: 10.48550/arXiv.2601.19532. Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language understanding? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and 9

Preprint. Under review.

Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843–4855, Online, jun 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.385. URL https://aclanthology.org/2021.naacl-main.385/. Colin Camerer, George Loewenstein, and Martin Weber. The curse of knowledge in economic settings: An experimental analysis. Journal of Political Economy, 97(5):1232–1254, October 1989. doi: 10.1086/261651. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6z4YKr0GK6. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini. Are we done with MMLU? In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5069–5096, Albuquerque, New Mexico, apr 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.262. URL https://aclanthology.org/2025.naacl-long.262/. Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments. https://harborframework.com/, 2026. Repository: https://github.com/harbor-framework/harbor. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In The Ninth International Conference on Learning Representations, 2021. URL https://openreview.net/ forum?id=d7KBjmI3GmQ. David G. Jansson and Steven M. Smith. Design fixation. Design Studies, 12(1):3–11, January 1991. doi: 10.1016/0142-694X(91)90003-F. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4334–4353, Miami, Florida, USA, nov 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.248. URL https://aclanthology.org/2024.emnlp-main.248/. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Opportunities and challenges of LLM-as-ajudge. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 2757–2791, Suzhou, China, nov 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.138. URL https://aclanthology.org/2025.emnlp-main.138/. 10

Preprint. Under review.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 21558–21572. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/ hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html. Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of AI research. Nature, 651 (8107):914–919, 2026. doi: 10.1038/s41586-026-10265-5. Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. LLM critics help catch LLM bugs. arXiv preprint arXiv:2407.00215, 2024. doi: 10.48550/arXiv.2407.00215. Mike A. Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, et al. TerminalBench: Benchmarking agents on hard, realistic tasks in command line interfaces. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=a7Qa4CcHak. Ludovico Mitchener, Jon M Laurent, Alex Andonian, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. BixBench: a comprehensive benchmark for LLM-based agents in computational biology. arXiv preprint arXiv:2503.00096, 2025. doi: 10.48550/arXiv.2503.00096. Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2021. URL https://openreview.net/forum?id= XccDXrDNLek. OpenAI. Introducing SWE-bench Verified. https://openai.com/index/ introducing-swe-bench-verified/, August 2024. Published August 13, 2024; updated February 24, 2025. OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/, February 2026. Published February 23, 2026. Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, Ziwen Han, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649(8099):1139–1146, jan 2026. doi: 10.1038/s41586-025-09962-4. Phylo. BixBench-Verified-50. https://huggingface.co/datasets/phylobio/ BixBench-Verified-50, 2026. Hugging Face dataset with 50 verified questions across 33 unique data capsules. Mohammed Latif Siddiq, Simantika Dristi, Joy Saha, and Joanna C. S. Santos. The fault in our stars: Quality assessment of code generation benchmarks. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM), pp. 201–212. IEEE, oct 2024. doi: 10.1109/SCAM63643.2024.00028. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 52040–52094. Curran Associates, Inc., 2024. doi: 10.52202/079017-1650. 11

Preprint. Under review.

Table 4: Complete B ENCH G UARD error taxonomy with all subcategories. ID

Name

GT (Ground Truth) GT-LOGIC Wrong logic / methodology GT-DATA

Wrong data handling

GT-FMT

Output format mismatch

Description Gold uses incorrect algorithm, computes wrong metric, or logical opposite Gold uses wrong files/columns, drops data, or covers only partial scope Gold output format does not match the specification

EVAL (Evaluation) EVAL-JUDGE-BIAS Judge bias / anchoring

Evaluator penalizes valid functionally-equivalent alternatives EVAL-MISMATCH Spec–eval mismatch Eval checks something different from the specification EVAL-COVERAGE Incomplete output coverage Eval doesn’t handle all valid output formats EVAL-TOLERANCE Wrong tolerance Numeric tolerance or thresholds are incorrect EVAL-STOCHASTIC Unhandled non-determinism Eval assumes deterministic outputs when it should not INST (Instruction) INST-INCOMPLETE Underspecified requirements Essential information missing, preventing a unique solution INST-CONTRADICT Cross-artifact misalignment Instructions conflict internally or with gold/eval INST-INFEASIBLE Task infeasible as written Task cannot be solved with provided information ENV (Environment) ENV-DEP Missing/conflicting depen- Required packages unavailable or conflict dencies ENV-PATH Path configuration errors Hardcoded paths don’t match runtime environment ENV-RESOURCE Resource constraints Task requires network, external services, or exceeds time limits

Weiqi Zhai, Zhihai Wang, Jinghang Wang, Boyu Yang, Xiaogang Li, Xander Xu, Bohan Wang, Peng Wang, Xingzhe Wu, Anfeng Li, Qiyuan Feng, Yuhao Zhou, Shoulin Han, Wenjie Luo, Yiyuan Li, Yaxuan Wang, Ruixian Luo, Guojie Lin, Peiyao Xiao, Chengliang Xu, Ben Wang, Zeyu Wang, Zichao Chen, Jianan Ye, Yijie Hu, Jialong Chen, Zongwen Shen, Yuliang Xu, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Hu Wei, Que Shen, and Bing Zhao. HLE-Verified: A systematic verification and structured revision of humanity’s last exam. arXiv preprint arXiv:2602.13964, 2026. doi: 10.48550/arXiv.2602.13964. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 46595–46623. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/ hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= oKn9c6ytLx.

Full Error Taxonomy

Table 4 presents the complete B ENCH G UARD error taxonomy with all 14 subcategories. 12

Preprint. Under review.

Severity and Confidence Definitions. Severity captures the impact on benchmark correctness: Critical—task is impossible or fundamentally broken; High—changes evaluation correctness for valid solutions; Medium—affects some valid solution strategies; Low— minor issue or edge case. Confidence is the LLM’s self-assessed probability that the defect is real, bucketed into three tiers: Confirmed (0.8–1.0)—verified with conclusive evidence; Likely (0.55–0.79)—strong evidence, high probability; Possible (0.3–0.54)—suspicious pattern that warrants human review. Findings with confidence below 0.3 are suppressed.

Preprint. Under review.

Key Prompt Templates

This appendix presents the key prompt templates used in B ENCH G UARD’s verification protocols. The actual prompts additionally include the full error taxonomy (Table 4) as an inline reference and benchmark-specific calibration rules (e.g., runtime-mounted data heuristics for containerized tasks); these are omitted here for brevity. Editorial omissions within the prompt boxes are marked with [. . . ]. Complete prompt templates will be released with the open-source framework upon publication. DefinitionProtocol System Prompt. The system prompt establishes the auditor role, specifies four audit areas with their subcategory checks, and defines quality-control rules governing finding granularity and calibration. DefinitionProtocol — System Prompt (condensed) Role. You are an expert benchmark auditor performing a comprehensive definition audit. Your job is to find bugs in the BENCHMARK (not in agents). You will check four areas in a single pass: ground truth correctness, evaluation logic, task specification quality, and environment issues. Area 1: Ground Truth Correctness. Verify the gold program correctly implements the task instruction. • GT-LOGIC: Gold uses incorrect algorithm, computes wrong metric, or applies logical opposite. • GT-DATA: Gold uses wrong input files or columns, drops data, or covers only partial scope. • GT-FMT: Gold output format does not match the specification. Area 2: Evaluation Logic. Verify the evaluation script correctly measures what the specification asks. • EVAL-JUDGE-BIAS: LLM judge rigidly anchored to one implementation, rejecting valid alternatives. • EVAL-MISMATCH: Eval checks something different from what the specification requests. • EVAL-COVERAGE: Eval does not handle all valid output formats, types, or equivalent names. • EVAL-TOLERANCE: Numerical tolerances too strict or too lenient. • EVAL-STOCHASTIC: Eval assumes deterministic output for inherently non-deterministic computation. Area 3: Task Specification. Check whether the instruction provides sufficient and consistent information. • INST-INCOMPLETE: Essential information missing, preventing a unique correct solution. • INST-CONTRADICT: Instruction conflicts with the gold program or evaluation script. • INST-INFEASIBLE: Task cannot be solved with the provided information. Area 4: Environment & Infrastructure. Flag runtime issues in a sandboxed evaluation environment. • ENV-DEP: Required packages unavailable or version conflicts. • ENV-PATH: Hardcoded absolute paths that do not match the evaluation environment. • ENV-RESOURCE: Requires network access, external APIs, or exceeds time/compute limits. [Per-area guidelines and benchmark-specific calibration rules omitted.] Categorization Priority. When an issue could be classified as either an instruction problem (INST-*) or a ground truth problem (GT-*), always prefer the instruction-level category. The root cause of most benchmark bugs is an underspecified or ambiguous instruction—the gold program simply implements one interpretation of the vague spec. Use GT-* only when the gold program is objectively wrong independent of the instruction (e.g., a coding bug, off-by-one error, wrong formula). Deduplication Rule. Each bug must appear exactly once under its most specific subcategory. Apply the one-fix test: if fixing one issue would make another finding disappear, they are the same bug—report it only once. Atomicity Rule. Every finding MUST describe exactly one independently fixable root cause. Apply the split test to every finding: “Could concern A be fixed while concern B still remains?” If yes, they must be separate findings. Output Format. Respond with a JSON array. Each finding includes: category (GT/EVAL/INST/ENV), subcategory (e.g., GT-LOGIC), severity, finding_type, title, description, evidence (with source file, line numbers, and code snippet), recommendation, and confidence (0–1).

Preprint. Under review.

Severity scale: Critical = task is impossible or fundamentally broken; High = changes evaluation correctness; Medium = affects some valid solutions; Low = minor issue or edge case. Finding type: B UG = a concrete, verifiable error—a specific line of code contradicts the specification, an incorrect algorithm is used, or logic is demonstrably broken. WARNING = a concern or improvement opportunity—something is underspecified or not provably wrong. Default to WARNING when concrete scoring harm cannot be demonstrated. Confidence calibration: 0.8–1.0 (C ONFIRMED) = a specific line contradicts a specific statement in the spec; 0.55–0.79 (L IKELY) = clear discrepancy requiring cross-section reasoning or domain inference; 0.3–0.54 (P OSSIBLE) = suspicious pattern that requires domain expert verification. Below 0.3: do not report. [Full JSON schema, inline taxonomy reference, and runtime-data calibration rules omitted for brevity.]

DefinitionProtocol User Prompt. The user prompt presents all task artifacts in a structured context block, followed by a six-phase chain-of-thought analysis procedure. Phases 2–5 produce analysis only (no JSON output); Phase 6 consolidates, deduplicates, and emits the final JSON array. DefinitionProtocol — User Prompt Template Context Block [populated per task]: Task ID: {task_id} | Domain: {domain} | Expected output: {expected_output} Task Instruction: {task_instruction} Gold Program [with line numbers]: {gold_program_source} Evaluation Script [with line numbers]: {eval_script_source} Input Data Description: {input_data_description} Domain Knowledge [optional]: {domain_knowledge} Environment Information: {environment_info} Phase 1: Understand the Task. Parse the instruction to identify target output, metric, and computation. Classify the runtime environment (container/capsule vs. standalone) to calibrate discoverability of data details. Phase 2: Ground Truth Correctness [analysis only]. Trace the gold program line by line. Compare against instruction: correct metric? Correct files and columns? Right algorithm? Correct output format? List each concern as a separate bullet. Phase 3: Evaluation Logic [analysis only]. Trace the evaluation script. Does it check what the spec asks? Handle all valid output formats? Use appropriate tolerances? List concerns separately. Phase 4: Task Specification [analysis only]. For each candidate omission, apply a discoverability gate: is this a runtime-discoverable detail (filenames, column names, metadata) or a non-discoverable semantic choice (methodology, thresholds, covariates)? Report only the latter as genuine concerns. Phase 5: Environment & Infrastructure [analysis only]. Check for path mismatches, resource constraints, and sandbox compatibility. Phase 6: Consolidate & Report. (1) Deduplicate via the one-fix test. (2) Atomicity check via the split test. (3) Runtime-data filter: remove findings about purely discoverable details. (4) Output the final JSON array.

Agent Program as Supporting Evidence. When agent trajectories are available, the audit extends to execution-level mode by incorporating the agent’s generated program as supporting evidence. This mode corresponds to the execution-level results in Table 2: agent programs exercise code paths and edge cases invisible to static review, improving recall (e.g., Opus 4.6 from 83.3% to 91.7% on SAB). The agent program is appended to the user prompt as an additional section, and Phase 1 is augmented accordingly. Agent Program Evidence — User Prompt Extension Agent Program (Supporting Evidence Only)

Preprint. Under review.

Use this as supporting evidence about how a concrete agent interpreted the task. Do NOT assume the agent is correct. Use the benchmark artifacts as the primary source of truth, and only use the agent program to expose hidden assumptions, ambiguity, evaluator anchoring, or other benchmark issues that become clearer when compared against a real candidate solution. [Agent’s generated program source code] Phase 1 augmentation: “If an agent program is provided below, treat it as supporting evidence about one plausible interpretation of the task, not as ground truth.”

Benchmark Ingestion Details

To support diverse benchmarks, B ENCH G UARD adopts a standardized directory format compatible with Harbor (Harbor Framework Team, 2026). Each subdirectory under the benchmark root is treated as a task if and only if it contains a task.toml file. Listing 1: Benchmark directory layout. benchmark_root / | - benchguard_hints . yaml # optional : benchmark - level review hints | - task_id / | |- task . toml # required : metadata + config | |- instruction . md # required : natural - language task spec | |- tests / # required : evaluation logic | | |- test . sh | | `- *. py , *. sh , *. json , ... | |- solution / # recommended : reference solution | | |- solve . sh | | `- *. py , *. sh , ... | |- environment / # optional : Dockerfile , requirements | | |- Dockerfile | | `- requirements . txt | |- domain_knowledge . md # optional : background context | `- data_description . md # optional : dataset inventory `- another_task_id / `- ...

The task.toml file stores benchmark metadata and runtime configuration. All fields are optional; the only structural requirement is that the file exists. Listing 2: Representative task.toml configuration. [ metadata ] id = " task_001 " category = " bioinformatics " expected_output = " output . csv " benchmark_source = " ScienceAgentBench " [ verifier ] method = " script " [ environment ] runtime = " python3 .10" cpus = 2 memory = "4 G" [ agent ] timeout_sec = 1800.0

Tiered Input.

The framework supports three input tiers of increasing diagnostic power:

• Minimal (instruction + tests): enables EVAL, INST, and ENV checks. • Definition-level (+ gold program): enables full cross-artifact auditing. 16

Preprint. Under review.

Description

Pro

GPT -5.4

Opu

Son

4.6