arXiv CSopen access

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results Jan Batzner*,1-3 Sree Harsha Nelaturu*,4 Damian Stachura*,5 Anastassia Kornilova*,6

arXiv:2606.14516v1 [cs.AI] 12 Jun 2026

Jon Crall⋄, 7 Tommaso Cerruti⋄, 8 Yanan Long⋄, 9 Yifan Mai⋄, 10 Sanchit Ahuja⋄, 11 Asaf Yehudai⋄, 12 Marek Šuppa⋄, 13,14 John P. Lalor⋄, 15 Oluwagbemike Olowe⋄, 16 Jatin Ganhotra12 Brian H. Hu7 Eliya Habba17 Andrew M. Bean18 Chang Liu19 Sander Land20 Steven Dillmann10 Aniketh Garikaparthi21 Elron Bandel12 Saki Imai11 James Edgell22 Wm. Matthew Kennedy18 Jenny Chim23 Patrick Meusling24 Asteria Kaeberlein11 Venkata Ramachandra Karthik Chundi16 Manasi Patwardhan21 Martin Ku22 Austin Meek25 Leon Knauer26 Brian Wingenroth27 Srishti Yadav28,29 Usman Gohar30 Felix Friedrich31 Michelle Lin32,33 Jennifer Mickel34 Arman Cohan35 Stella Biderman†, 34 Irene Solaiman†, 36 Zeerak Talat†, 37 Anka Reuel†, 10,38 Mubashara Akhtar†, 39,8 Gjergji Kasneci†, 1,2 Avijit Ghosh†, 36 Leshem Choshen†, 40,41,12 * Lead Author ⋄ Top Contributor † Advisor This project was a part of the Evaluating Evaluations (EvalEval) Coalition: https://evalevalai.com/ 1 Technical University Munich 2 Munich Center for Machine Learning 3 Weizenbaum Institute Zuse Institute Berlin 5 Evidence Prime 6 Trustible 7 Kitware 8 ETH Zurich 9 StickFlux Labs 10 Stanford University 11 Northeastern University 12 IBM Research 13 Comenius University Bratislava 14 Cisco 15 University of Notre Dame 16 Independent 17 Hebrew University of Jerusalem 18 University of Oxford 19 Ohio University 20 Writer 21 TCS Research 22 Oxford University Press 23 Queen Mary University of London 24 Technical University Berlin 25 University of Delaware 26 Cinemo 27 Johns Hopkins University 28 University of Copenhagen 29 ELLIS 30 Iowa State University 31 Meta FAIR 32 University of Montreal 33 Mila Quebec AI Institute 34 EleutherAI 35 Yale University 36 Hugging Face 37 University of Edinburgh 38 Harvard University 39 ETH AI Center 40 MIT 41 MIT-IBM Watson Lab 4

Abstract AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First, results are saved in incompatible formats, scattered across leaderboards, papers, blog posts, evaluation harness logs, and custom repositories. Second, results are created by different evaluation frameworks, which produce divergent scores for nominally identical evaluations and record metadata inconsistently, hindering comparison, cross-community evaluation science, cost reduction, and reuse. We introduce Every Eval Ever, the first shared schema and community-crowdsourced repository for AI evaluation results. The schema standardizes how evaluations are represented, in a unified, single JSON document. It is source-agnostic by design, ingesting results from evaluation harnesses and papers alike, and optionally stores per-instance outputs for fine-grained analysis. We contribute: (i) a community-governed metadata schema with a companion instancelevel schema § evaleval/every_eval_ever, the first standardization effort of its kind; (ii) automatic converters from popular formats, evaluation harnesses, and leaderboards to the unified schema § ; and (iii) a crowdsourced community database hosted on Hugging Face, currently spanning to date 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats evaleval/EEE_datastore.

Preprint.

1

Introduction

Evaluations are critical for measuring AI progress, yet how they are reported is inconsistent, incomplete, and difficult to interpret. Evaluation results are often reduced to aggregated scores in a table, with important evaluation metadata, such as generation parameters, evaluation settings, and data provenance, omitted or scattered across papers, ad hoc log files, and code repositories. This fragmentation undermines reproducibility, complicates cross-benchmark comparisons, and limits the potential for systematic meta-analysis. In practice, this creates fundamental challenges for both researchers and practitioners. Comparative evaluation studies are typically constrained by the subset of results that can be reliably reproduced (e.g., architecture scaling [22, 80] or quantization comparisons [48]), often requiring substantial computational and financial resources [34, 70]. A lack of comparability is especially misleading when different parties evaluate the same model or benchmark, yet produce different scores [see §7.3 and 97, 89]. For example, the LLaMA 65B model has been reported to achieve both 63.7 and 48.8 on MMLU [39]. On a closer look, the difference in scores was found to arise from the use of different evaluation harnesses. Without this context, the scores are not directly comparable [29]. Similarly, our analysis of evaluations across over 22,235 models and 2,273 benchmarks reveals 31 distinct reporting formats, highlighting the lack of standardization and motivating the need for more structured reporting practices (See statistics in Fig. 2). Other parts of the AI pipeline have benefited from standardization: shared metadata schemas such as DCAT, Schema.org/Dataset, and Croissant [90, 81, 3]; documentation practices such as Datasheets for Datasets and Model Cards [33, 67]; and common evaluation and benchmarking protocols such as GLUE, SuperGLUE, HELM, BIG-bench, and MLPerf [92, 91, 53, 85, 76] have improved reproducibility, comparability, and transparency. In contrast, evaluation reporting remains fragmented [54, 18, 75, 27, 19], with reported implications for downstream analysis such as benchmark saturation studies [15, 4]. Similarly, psychometric analyses in the field depend on standardized example-level data, which is rare in current evaluation reporting [51, 74]. Finally, governance frameworks such as the EU AI Act [28] mandate reproducible risk assessments, yet current evaluation tooling and reporting lack even the basic standardization that reproducibility requires.

Leaderboards

Inspect AI

Custom Code

Stanford HELM

Papers Results

EleutherAI lm-eval

Benchmark Level Data

Other Sources

Every Evaluation Data CLI + EvalEval Bot

Eval Logs into EEE

Converters

Validator

Instance Level Data

EEE Database Unifying Eval Schema Figure 1: Every Eval Ever has four components: (1) heterogeneous evaluation data (leaderboards, papers, harness logs, custom scripts); (2) converters for known log formats (HELM, Inspect AI, lm-eval) and metadata parsers for community formats (Hugging Face, leaderboards); (3) a unified metadata schema supporting aggregate and instance-level results; and (4) a crowdsourced community database making public evaluation results accessible and processable. Every Eval Ever (EEE) addresses these gaps through a shared reporting schema and a crowdsourced repository for AI evaluation results. Just as data [3] and models [67] have documentation standards, EEE standardizes the core aspects of evaluation: who ran it, under what settings, and what the 2

resulting scores mean. It ingests results from any source, like harness logs, leaderboard scrapes, and paper results, and represents them in a single, interoperable format. In summary, EEE makes the following contributions: 1. A shared, versioned JSON schema for AI evaluation results that captures source provenance, model access mode, generation configuration, and metric semantics in a single record, with an optional instance-level companion schema supporting single-and multi-turn interaction types. 2. Automatic converters from major harnesses (HELM, lm-eval-harness, Inspect AI) and common formats producing schema-compliant records, including per-instance outputs where source logs provide them, paired with a validation pipeline that ensures schema compliance at contribution time. 3. A crowdsourced, community repository hosted on Hugging Face, already spanning 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats, that for the first time enables cross-framework comparison of evaluation results at scale. 4. Exemplary empirical analyses enabled by unified repository, where EEE can identify costaccuracy tradeoffs in agentic evaluations (7.1), reveal implementation-dependent perplexity scores (7.2), captures evaluation harness reproducibility gaps (7.3), and enable meta-analysis using Item Response Theory (7.4), none of which were previously feasible without a unified result format.

2

Related Work

Evaluation harnesses: Evaluation harnesses describe software to standardize model evaluation, from input prompts to output metrics. While evaluation harnesses like lm-eval-harness [32], HELM [53], and InspectAI [2] have proliferated, their format for results remain mutually incompatible [7, 14]. Every Eval Ever is not a new evaluation harness, but a translation layer that sits above those and enables better aggregation of evaluation results. Evaluation sharing: There are a few large sources that share evaluations. The main sources for those are leaderboards [53, 43], or websites [6, 65, 26] efforts that release what they run, and two concurrent works to ours that collect instance-level [41] or Inspect framework outputs specifically [1] and share them publicly. We collaborate with them to aggregate their results to Every Eval Ever. Public Leaderboards like Open LLM Leaderboard [12], Chatbot Arena [99], AlpacaEval [52], MT-Bench [98], aggregated results at scale but export limited structured metadata [93]. We created Every Eval Ever to combine all of those scores in a unified format and database, alongside local harness runs within the same format. Reproducibility: Comparison is unreliable when different evaluation settings are underspecified and carry the same benchmark name. Lacking standards prevents the community from reliably comparing, replicating, and reusing cost-intensive evaluations [14]. The same model, accessed through different providers or run with different engine configurations, can produce different outputs [69]. Moreover, prompt ordering [58] and data contamination [60] can introduce score variance. Large-scale analysis of evaluation results can require weeks of data wrangling before any research can begin [e.g. 80, 22, 70, 4], if such analysis is possible at all without rerunning full leaderboards at extreme cost [e.g. 35, 70, 34], we estimate the inference cost to reproduce our data in §6. Dataset and model documentation: Although larger efforts in the ML community centered around datasets and model documentation, evaluation, and result documentation itself remain a gap in the community [54, 18]; where multiple suggestions for metadata to report exist [86, 16, 84], but not the low-level evaluation ones. For datasets, Datasheets for Datasets [33] and Croissant [3] standardize metadata for ML datasets. For benchmarks, those efforts have been tailored to benchmark needs [78, 40, 84]. For models, Model Cards [67, 56] document artefacts and their intended uses. For evaluation results, Every Eval Ever addresses the most pressing remaining gap: a shared schema for the run-time context that determines whether two scores can be aggregated and compared. 3

Table 1: Design decisions behind Every Eval Ever and the capabilities they enable. Design Decision « Accept partial records

õ

Implementation in Every Eval Ever

Enabled capability

Required fields are minimal; unavailable Includes existing results that metadata can be omitted or recorded in would be excluded by a fully specadditional_details. ified reporting standard.

Use wide, typed metadata Common fields capture model identity, Makes heterogeneous results comcoverage provenance, generation settings, bench- parable and auditable across pamark metadata, and metric semantics. pers, leaderboards, and logs.

[ Preserve run identity with- Each run receives a UUID and stores out a canonical fingerprint available context, rather than assuming a fixed parameter tuple uniquely identifies an evaluation.

Keeps repeated, conflicting, or underspecified runs visible for later deduplication and longitudinal analysis.

à Version and extend the New versions add fields for new modal- Keeps the schema ities, reasoning models, and agentic compatible. traces without rewriting old records.

schema

forward-

C Separate aggregate and Aggregate JSON files link to optional Enables instance-level reanalysis, instance-level records sidecar files with prompts, outputs, auditing, and meta-evaluation bescores, and metadata. yond headline scores. ó Organize reusable infor- Source, model, configuration, result, and Improves interoperability across mation blocks instance-level metadata are grouped into converters and downstream analyrecurring schema components. sis tools.

Agentic evaluation standardization: Recent work begun to understand the importance of standardizing agentic evaluation [10, 43], and made first steps towards achieving it [8, 49, 64, 38, 96]. These efforts focus on the runtime and execution layers, standardizing agent evaluation across task representation, environment type, interface protocol, and tool specification format to enable easy and scalable agent and benchmark integration. Every Eval Ever provides a complementary focus by standardizing how agentic evaluation results are represented and stored. Hence, allowing for easy results analysis across different sources (Section 7.1).

3

The Every Eval Ever Schema

Every Eval Ever is a standardized representation of AI evaluation results across benchmarks, models, and reporting data sources (e.g., public model leaderboards, research papers, evaluation harness logs, among others; Figure 1). Instead of storing only final performance scores, each Every Eval Ever record captures the metadata required to interpret, compare, and reuse results: Who ran the evaluation, which model was evaluated, under what generation settings, how metrics were computed, and (if available) instance-level outputs. The schema is modular and organized into reusable information blocks. In this section, we describe how the schema was developed and the design principles that guided its construction (Section 3.1), and present the schema structure and core components (Section 3.2); full field references appear in App. B and Table 8. 3.1

Schema design principles and development methodology

Every Eval Ever balances broad adoption with enough structure to support downstream comparison, auditing, and reanalysis. Table 1 summarizes the main design decisions and the capabilities they enable. The schema was developed through an open, iterative community design process inspired by the Croissant metadata format [3]. We gathered structured feedback from about 40 researchers and unstructured feedback from about 110 researchers, including benchmark creators, evaluation framework developers, governance experts, leaderboard operators, and industry practitioners. The schema is open and had since discussions on improvements through GitHub and Slack. The schema versions were openly proposed, discussed, and revised by the community, with disagreements resolved by consensus meeting among core contributors (see Governance in App. §E). The fields were included if they were (i) reported in at least one existing evaluation framework or published result and (ii) considered necessary for the interpretability or reuse of the score by the majority of 4

contributors, (iii) anticipated to be available for others to report in the future; fields that did not meet all criteria were moved to additional_details or excluded.

3.2

Schema overview

Each EEE record stores data on a single evaluation run. The EEE schema is organized into five metadata blocks: First, Source Metadata: Who produced the results and from where did it originate? Second, Model Information: Which model was evaluated and how was it accessed? Third, General Configuration: Which configuration settings were used during evaluation? Fourth, Evaluation Results: How were the results reported (e.g., metrics, uncertainty estimates)? Finally, Instance-level Data: Optionally, what instance-level information is available? 1. Source metadata: The source_metadata field records who produced the result and how it was collected. source_type distinguishes results scraped from a leaderboard or paper (documentation) from those produced by a local evaluation run (evaluation_run). evaluator_relationship records whether the evaluation was run by the model developer (first_party), an independent party (third_party), or the metadata contributors themselves (self). Capturing the reporting source is important since the incentives, reproducibility, and trustworthiness of the reported results can differ between them. 2. Model information and access mode: model_info records the model identifier in developer/name using a standardized developer/model naming convention and the access mode. We store whether results were obtained through hosted APIs such as openai and anthropic (inference_platform) or local inference engines like vLLM (inference_engine). The same weights served through different providers or engine versions can produce different results (see §7.3); recording access mode makes these hidden confounds visible. 3. Generation configuration: Parameters such as temperature, number of samples, and stop sequences can effect benchmark outcomes substantially, yet they are frequently missing from leaderboards. generation_config makes them first-class fields. When a parameter is unknown, the field is omitted and its absence is recorded explicitly rather than silently defaulted. 4. Evaluation results and metric semantics: evaluation_results stores one entry per scored metric. Each entry includes a metric_config object capturing score direction (lower_is_better), type (continuous, binary, ordinal), and range. This prevents silent ambiguity. For example, a score of 0.31 is favorable on toxicity metrics where lower is better, but poor on pass@1 coding metrics where higher scores are desirable. Ordinal metrics (e.g. Low/Medium/High mapped to integers via level_names), uncertainty fields (standard errors, confidence intervals), and per-result timestamps are also supported (Case Studies §7.2). 5. Instance-level data: While aggregate scores support comparisons, understanding why scores differ often requires per-sample data [19]. EEE therefore stores instance outputs in a optional companion file _samples.jsonl (one JSON object per line). This file can store prompts, model outputs, references, scores, and metadata needed for detailed analysis. Three interaction types are supported: First, single_turn — QA, MCQ, classification; uses an output object. Second, multi_turn — multi-exchange conversations; uses a messages array. Third, agentic — toolusing agents with full tool-call traces and sandbox logs; uses messages with nested tool_calls. For agentic evaluations (e.g. SWE-Bench, GAIA), the aggregate record captures tool and sandbox configuration (Case Study §7.1).

4

Converters and validation

Every Eval Ever schema is supported by two components: (1) converters, which automatically parse existing evaluation outputs into the EEE schema, and (2) automated validation, which assesses whether the submitted datasets conform to the EEE schema specification: 5

(1) Converters: Manual re-formatting of existing logs is a large barrier to adoption. Hence, we provide converters for three widely-used LLM evaluation frameworks: HELM [53], lm-evalharness [32], and Inspect AI [2]. App. C describes these converters input formats and field mappings. More converters from leaderboards or other sources were contributed by the community (App. C.4). Each converter produces a schema-compliant .json file and, where source logs include per-sample data, a _samples.jsonl. (2) Validation: Every time a record is submitted to EEE, it is validated against the schema before being entered into the repository. The validation step checks for the required fields, data types, enum constraints, and object consistency. Schema compliance is enforced by Pydantic v2 models generated from eval.schema.json. Validation runs in two settings: (2.1) Locally via the command-line: Users can validate records locally with the Every Eval Ever CLI before submission. However, there are additional checks available which are provided by the validator. The CLI supports rich terminal, JSON, and GitHub annotation output formats. (2.2) EvalEvalBot: When contributors submit data via pull request to the datastore, either the author or a maintainer can request validation using the validator. The validator checks schema syntax compliance, presence of all files mentioned and provides warning for situations presence of duplicate records.

5

Community approach

Governance: The community-driven nature of EEE necessitates that governance is integrated into EEE. The project recognizes three roles: core maintainers, contributors, and community reviewers, where maintainers hold final authority on contested decisions (see App. E.1 for more). Decisions range from routine record additions to substantive schema changes, the latter following a structured community proposal and review process (App. E.2). Records are immutable once accepted. Errors are handled via explicit correction and retraction mechanisms that preserve immutability and reproducibility (App. E.4). For example, discrepancy between evaluation results for LLaMA [29] exists and EEE stores both evaluation results as valid records, with the discrepancy visible in metadata rather than discovered through a blog post (App. E.3, E.6.1). Full details, including worked examples, are in App. E. Contribution model: Evaluation infrastructure is a public good: every stakeholder needs it, yet no single entity has sufficient incentive to build it alone. EEE addresses this collectively: contributions range from evaluation records (aggregate JSON files from converters, instance-level companion files, or leaderboard scrapes; (Section 4) to converter extensions, schema proposals, and tooling (Section 3). Benchmark creators can register artifacts in EEE format, gaining visibility in downstream use. Evaluation framework developers (e.g., HELM [53], Inspect AI [2]) get a standardized export path for their users. Leaderboard operators can offload comparison and cross-source aggregation to shared infrastructure. Evaluation researchers (see Section 7 for examples) gain access to otherwise unattainable meta-evaluation data. Significant contributions are recognized through co-authorship and formal industry partnerships (See App. E.5).

6

Data analytics of the Every Eval Ever datastore

229K 22,235 2,273 Evaluations Unique models Benchmarks

135

Eval data PRs

127

Code PRs

31

Eval formats

28

Source orgs

Figure 2: Overview of the scale and diversity for Every Eval Ever data. As of May 4th 2026, current contributions total more than 200K aggregated results across over a hundred data contributions (see summary statistics in App. A), providing a foundation for evaluation research and a lens on community-wide reporting trends. In this section, we highlight how the cost of running evaluations underscores the value of a shared resource like EEE, and what the collected data reveals about community-wide evaluation trends. 6

Cost of AI evals: Several works have discussed the cost of evaluating models [70, 34]; while future work should make cost more explicitly derivable from the schema, it remains underreported and difficult to infer. Towards thism, we provide a conservative estimate of the savings such a shared resource may offer. We conservatively estimate that reproducing just the evaluation runs currently collected in EEE, would cost hundreds of thousands of dollars (App. D). This figure considers only running costs and excludes factors that would raise it by further orders of magnitude e.g., agentic evaluations (Case Study §F.1), thinking models, repeated runs, failed attempts, long benchmarks, code execution, and human labeling [9, 43]. Community trends: Beyond its value as a shared resource, the corpus offers a data-rich view of community-wide evaluation practices. While acknowledging that coverage is biased by data availability, we analyze what the collected results reveal about how the field approaches AI evaluations (App. A). We find that evaluations follow a long tail: popular benchmarks and models are reported at a scale far above the rest, yet thousands of less common ones appear, and the top 25 in each category account for barely 25% of all results. Geographically, we observe a strong concentration on U.S.based models, with GPT models dominating. Excluding human baselines, five companies account for 23 of the 24 most frequently evaluated systems, revealing a focus not only on specific models but on specific sources of models. This concentration carries implications beyond socio-political findings and suggests that much of the field is evaluating commercial products rather than underlying technologies, confirming recent claims in the literature [66]. Table 2: Macro-average fill rate of Every Eval Ever metadata fields across the 31 evaluation harnesses and formats in the EEE datastore. We provide three metadata field examples each for model metadata, benchmark metadata, and evaluation metadata. Model model name model parameters model license

Benchmark 100% source URL / HF repo 3% metric ID 3% uncertainty n_samples

Evaluation 96% inference platform 77% temperature 30% max tokens

27% 23% 23%

Format inconsistencies: Analyzing the data also corroborates our claims about format inconsistencies. For example, the common source of evaluations is academic papers, which are not machine-readable, and each uses a different reporting format. Moreover, many fields crucial for comparisons are unreported; for instance, the inference platform is either explicitly marked as unknown or omitted entirely in 98% of all evaluation rows (micro-average); even when each of the 31 formats is weighted equally, the field is reported in only 27% of rows on average (macro-average; Table 2).

7

Case Studies

EEE data allows AI evaluations to be compared, reproduced, and reused, enabling broader impact through additional research and meta-analyses. For example, EEE data can help identify where the evaluation ecosystem is thin, which capabilities are over-measured, and which risks are neglected. With instance-level data, researchers can move beyond leaderboard averages to study item difficulty, robustness, and temporal drift. Every Eval Ever also enables meta-evaluation: testing evaluation methods themselves to distinguish real progress from artifacts of setup and reporting. While different works already showcased uses for EEE-like data [e.g., for efficient benchmarking; 70], or even already used EEE data [e.g., to characterize benchmark saturation; 4], we perform several initial studies to showcase research that EEE enables. 7.1

Case Study 1: Every Eval Ever identifies cost–accuracy tradeoffs in agentic evaluation

We show EEE is useful for analyzing agentic evaluations beyond accuracy scores. EEE already contains results from several agentic benchmarks, including SWE-bench [42], HAL [43], Exgentic [8], and CocoaBench [37]. EEE also reports diverse metadata following previous work arguing that agent evaluations should track time, cost [44, 95], and other agent metadata [9], rather than reporting accuracy alone. We rely on the additional metadata to reveal cost-performance tradeoffs in scaffold and backbone choice. 7

50 80 GPT­5.4

70

40

Codex OpenClaw Cocoa Agent Claude Code

35 Gemini 3.1 pro

30

Accuracy (%)

Accuracy (%)

45

Opus 4.5

Claude Code CORE­Agent

Sonnet 4.5

50

Opus 4.1

Sonnet 4.6

40

25 20

60

100

150

200 250 Total cost ($)

300

30

350

(a) CocoaBench.

Sonnet 3.7

100

200 300 Total cost ($)

400

(b) CORE-Bench Hard from HAL.

Figure 3: Every Eval Ever enables cost–accuracy analysis across agent scaffolds and model backbones. Marker shape denotes the scaffold, and color denotes the backbone, with each point corresponding to one scaffold–backbone pair. Segments connect results sharing a backbone, isolating scaffold effects.

Fig. 3 illustrates two concrete findings about cost–accuracy tradeoffs. First, CocoaBench [37] shows that scaffold choice has substantial implication to costs, without necessarily showing performance gains in return. For example, Codex and OpenClaw with GPT-5.4 reach the same reported accuracy, but Codex costs less and is also faster on average (Appendix F.1). Second, CORE-Bench Hard [83] from HAL [43] shows that scaffold effects can depend on the model backbone. Claude code is cheaper than CORE-Agent for both Opus 4.5 and 4.1, but it substantially increases accuracy for one and decreases for the other. Taken together, we find agentic evaluations cannot be interpreted from scalar accuracy alone: agent scaffold and model backbone choice, runtime, and cost all significantly affect the conclusions one draws from a result. This kind of cross-source decomposition is precisely what EEE enables: without a common schema, these factors are scattered across incompatible logs and leaderboards, making systematic reanalysis challenging, particularly when metadata is reported inconsistently across sources. 7.2

Case Study 2: Every Eval Ever reveals version-dependent perplexity

Model compression techniques [31, 30, 87] aim to reduce the size and computational cost of models, while minimizing degradation in performance. WikiText perplexity [63] is a widely used metric to assess the impact of model compression, where lower perplexity indicates better predictive performance. However, reported values across papers for the same model and dataset can differ substantially based on implementation choices that often go unreported. GPTQ [31] and SpinQuant [57] shipped model-specific evaluation scripts that report perplexity normalized by the number of tokens. In contrast, the LM Evaluation Harness [32] reports byte_perplexity, word_perplexity, and bits_per_byte rather than token-level perplexity. “Perplexity” alone is ambiguous as the same loss normalized by tokens, words, or bytes yields different numbers that are not directly comparable, as shown in Table 3. EEE makes these distinctions explicit. Recording the evaluation backend, dataset version, and normalization convention prevents results from being compared merely because they share the “perplexity” label, and helps avoid drawing incorrect conclusions when, for example, a vLLMbased evaluator reports a different variant from the GPTQ-style script needed for direct comparison (see App. F.2 for implementation details).

Model OPT-6.7B Llama-2-7B

GPTQ script Token PPL e(Lsum /Ntokens ) 10.8605 5.4687

vLLM + lm-eval Word PPL e(Lsum /Nwords ) 12.2907 8.7939

Mismatch 1.4301 3.3252

Table 3: Perplexity on WikiText under two evaluation implementations. The summed cross-entropy Lsum is identical across columns; only the normalization denominator differs. Yet, the resulting values are not directly comparable.

8

7.3

Case Study 3: Every Eval Ever captures reproducibility gaps

E Im ntity pu tat IM ion DB [61 [5 ] 9] Sy Re nth as . on i Tr ng [ ut hf 94] ul QA [5 GS 5] M [2 5] LS AT QA [1 Ci 00 v ] Co il m m en M ts [ M LU 17] [3 9] Bo ol Q [2 3] Qu AC [2 1] Na rra tiv eQ A Na [4 6] Re tur as al on Sy . [ n. W 2 ik 4] iF ac t[ 71 E ] M ntity atc hi ng [4 7]

We use EEE to audit instance-level reproducibility. Although evaluation frameworks do not usually promise exact reproducibility, researchers often rerun public evaluations locally and compare them to shared results. We reproduced three models on fourteen single-turn HELM benchmarks and compared aligned per-instance scores between official HELM-released records [53] and local reproductions after converting both sides to EEE. Model and benchmark references appear in Fig. 4; implementation details are in App. F.3.

100

Pythia-6.9B [13] 100 Vicuna-7B v1.3 [20] 100 Falcon-7B [5] 100

100 100 99.1

99.9 99.7 98.2

99.8 99.8 97.7

99.6 99.0 98.3

99.8 99.8 96.0

99.9 99.7 94.0

100 100 93.5

99.6 100 93.4

99.7 98.7 94.2

98.4 98.5 91.0

99.7 99.8 78.8

92.8 92.0 92.2

N/A N/A N/A 75

Figure 4: Instance-level score agreement between model–benchmark pairs for official HELM records and local reproductions after conversion to Every Eval Ever. Values report the percentage of aligned (instance, core metric) score pairs with identical official and local scores up to numerical tolerance. N/A denotes no content-hash overlap, making Entity-Matching [47] incomparable.

Figure 4 shows that EEE exposes score and example mismatches. Entity-Matching (right column) is incomparable because official and reproduced records select different Abt–Buy [47] examples despite using the same HELM recipe; App. F.3 traces this to row-order changes in the data-processing stack. SyntheticReasoning-Natural reveals a serving artifact: official Pythia completions are empty and receive zero scores, whereas local completions are non-empty and receive low but non-zero scores. WikiFact [71] shows roughly 92% agreement across models, consistent with stochastic sampling. Smaller residual disagreements remain after these cases are explained. Overall, EEE enables reproducibility forensics by surfacing mismatched example sets, empty or truncated completions, stochastic disagreement, and residual score differences. Notably, EEE does not replace frameworklevel provenance: when serving details are missing, the schema can surface the discrepancies but cannot always determine their exact cause.

7.4

Case Study 4: Every Eval Ever enables meta-analysis using Item Response Theory

density

density

We showcase how the 0.30 0.30 instance-level data in EEE can be used for analy0.25 0.25 sis across datasets with 0.20 0.20 otherwise incomparable 0.15 0.15 data: GPQA Diamond 0.10 0.10 [77], Wordle Arena [68], 0.05 0.05 and JudgeBench [88]. 0.00 0.00 −4 −2 0 2 4 −10.0 −7.5 −5.0 −2.5 0.0 2.5 5.0 7.5 10.0 We fit a unidimensional Ability Estimate Difficulty Estimate Item Response Theory GPQA Diamond JudgeBench Wordle Arena (IRT) model to analyze model ability and example Figure 5: Estimated model abilities (left) and item difficulties (right) difficulty distributions for three datasets included in Every Eval Ever. (App. F.4). At the dataset level, we see that the distribution of item difficulties and model abilities varies (Fig. 5). In particular, the Wordle Arena examples are generally more difficult and their difficulty varies more. This suggests GPQA may quickly shift from hard to saturated, while Wordle Arena will likely continue to surface challenging cases. Overall, this showcases how the cross benchmark consistency simplifies comparing datasets and gaining new insights on existing evaluations. 9

8

Limitations

We note several limitations to the current EEE schema. First, coverage is strongest for text-based, single-model evaluations, while multi-modal evaluations, human preference judgments (e.g. Chatbot Arena Elo), and multi-agent settings are only partially supported; these areas are intended to be extended by the community. Second, the value of the schema depends on broad community adoption: although converters reduce the cost of contribution, labs and leaderboard operators may still omit important metadata, such as generation parameters for proprietary systems, and this missing metadata is explicitly recorded. Third, the UUID-per-run design preserves information losslessly but shifts deduplication to the analysis layer, with reference implementations of common equivalence criteria planned as package utilities. Fourth, while the schema aids in finding reproducibility issues, as it does not run evaluations, information unique to this running settings is likely not reported in it. Finally, despite automatic verification mechanisms and community governance for data additions, the resource remains participation-based, so errors, inconsistencies, and uneven reporting across evaluation areas are likely to occur.

9

Conclusion

We present Every Eval Ever: a schema, validation pipeline, and converter suite that establishes a common language for AI evaluation reporting, supported by a growing community dataset of evaluation results. By recording the context needed to interpret a score, not just the score itself, EEE makes existing evaluation results reusable and enables analyses that per-paper reporting cannot support. We invite the community to contribute results, extend the schema, and build upon the dataset in future research.

References [1] Alexandra Abbas, Celia Waggoner, and Justin Olive. 2025. Developing and maintaining an open-source repository of AI evaluations: Challenges and insights. In Championing Open-source DEvelopment in ML Workshop @ ICML25. [2] UK AI Security Institute. 2024. Inspect AI: Framework for Large Language Model Evaluations. [3] Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner-Miguelez, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Satyapriya Krishna, Michael Kuchnik, Sylvain Lesage, Quentin Lhoest, Pierre Marcenac, Manil Maskey, Peter Mattson, Luis Oala, Hamidah Oderinwale, Pierre Ruyssen, and 12 others. 2024. Croissant: A metadata format for ML-ready datasets. In Advances in Neural Information Processing Systems, volume 37, pages 82133–82148. [4] Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek Šuppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, and 18 others. 2026. When ai benchmarks plateau: A systematic study of benchmark saturation. Preprint, arXiv:2602.16763. [5] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language models. Preprint, arXiv:2311.16867. [6] Artificial Analysis. 2026. Independent analysis of ai models and hosting providers. https: //artificialanalysis.ai/. Accessed: 2026-05-01. [7] Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman, Ofir Arviv, Matan Orbach, Shachar Don-Yehiya, Dafna Sheinwald, Ariel Gera, Leshem Choshen, and 1 others. 2024. Unitxt: Flexible, shareable and reusable data preparation and evaluation for generative ai. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 207–215. 10

[8] Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, and Michal Shmueli-Scheuer. 2026. General agent evaluation. Preprint, arXiv:2602.22953. [9] Elron Bandel, Asaf Yehudai, Alexandre Lacoste, Avijit Ghosh, Graham Neubig, Margaret Mitchell, Michal Shmueli-Scheuer, and Leshem Choshen. 2026. Agentic systems should be general. SSRN Electronic Journal. [10] Elron Bandel, Asaf Yehudai, and Michal Shmueli-Scheuer. 2026. Ready for general agents? let’s test it. In ICLR Blogposts 2026. Https://iclr-blogposts.github.io/2026/blog/2026/generalagent-evaluation/. [11] Jan Batzner, Leshem Choshen, Sree Harsha Nelaturu, Damian Stachura, Anastassia Kornilova, Yanan Long, Usman Gohar, Andrew Tran, and Avijit Ghosh. 2026. Shared task of every eval ever: Building a unifying, standardized database of llm evaluations. Preprint. [12] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Younes Belkada, and Thomas Wolf. 2023. Open LLM leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard. [13] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR. [14] Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, and 11 others. 2024. Lessons from the trenches on reproducible evaluation of language models. Preprint, arXiv:2405.14782. [15] Kathrin Blagec, Georg Dorffner, Milad Moradi, Mehrdad Alam, and Matthias Samwald. 2021. Are NLP benchmarks saturating? Preprint, arXiv:2105.13977. [16] Florian Bordes, Candace Ross, Justine T Kao, Evangelia Spiliopoulou, and Adina Williams. 2025. Eval factsheets: A structured framework for documenting ai evaluations. Preprint, arXiv:2512.04062. [17] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. arXiv preprint arXiv:1903.04561. [18] Samuel R. Bowman and George E. Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855, Online. Association for Computational Linguistics. [19] Ryan Burnell, Wout Schellaert, John Burden, Tomer D. Ullman, Fernando Martinez-Plumed, Joshua B. Tenenbaum, Danaja Rutar, Lucy G. Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, Douwe Kiela, Murray Shanahan, Ellen M. Voorhees, Anthony G. Cohn, Joel Z. Leibo, and Jose Hernandez-Orallo. 2023. Rethink reporting of evaluation results in AI. Science, 380(6641):136– 138. [20] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [21] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184. Association for Computational Linguistics. 11

[22] Leshem Choshen, Yang Zhang, and Jacob Andreas. 2025. A hitchhiker’s guide to scaling law estimation. In International Conference on Machine Learning, pages 10683–10699. PMLR. [23] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. Association for Computational Linguistics. [24] Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3882–3890. [25] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. [26] Epoch AI. 2026. About us: Making sense of ai. https://epoch.ai/about. Accessed: 2026-05-01. [27] Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846–4853, Online. Association for Computational Linguistics. [28] European Parliament and Council of the European Union. 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council: Artificial intelligence act. https://eur-lex. europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689. [29] Clémentine Fourrier, Nathan Habib, Julien Launay, and Thomas Wolf. 2023. What’s going on with the open LLM leaderboard? Hugging Face Blog. [30] Elias Frantar and Dan Alistarh. 2023. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. [31] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations. [32] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2023. A framework for few-shot language model evaluation. [33] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86–92. [34] Avijit Ghosh, Yifan Mai, Georgia Channing, and Leshem Choshen. 2026. AI evals are becoming the new compute bottleneck. EvalEval Coalition Blog. [35] Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. 2025. Dove: A large-scale multi-dimensional predictions dataset towards meaningful llm evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11744–11763. [36] Eliya Habba, Itay Itzhak, Asaf Yehudai, Yotam Perlitz, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen, and Gabriel Stanovsky. 2026. Growing pains: Extensible and efficient llm benchmarking via fixed parameter calibration. arXiv preprint arXiv:2604.12843. [37] Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, Haoxiang Zhang, Junli Wang, Hexi Jin, Boyuan Zheng, Kun Zhou, Yu Wang, Feng Yao, Licheng Liu, Yijiang Li, Zhifei Li, and 12 others. 2026. Cocoabench: Evaluating unified digital agents in the wild. Preprint, arXiv:2604.11201. 12

[38] Harbor Framework Team. 2026. Harbor: A framework for evaluating and optimizing agents and models in container environments. [39] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. [40] Aris Hofmann, Inge Vejsbjerg, Dhaval Salwala, and Elizabeth M. Daly. 2025. Auto-benchmarkcard: Automated synthesis of benchmark documentation. Preprint, arXiv:2512.09577. [41] Han Jiang, Susu Zhang, Xiaoyuan Yi, Xing Xie, and Ziang Xiao. 2026. Position: Science of ai evaluation requires item-level benchmark data. Preprint, arXiv:2604.03244. [42] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? Preprint, arXiv:2310.06770. [43] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, and 12 others. 2025. Holistic agent leaderboard: The missing infrastructure for ai agent evaluation. Preprint, arXiv:2510.11977. [44] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2024. Ai agents that matter. Preprint, arXiv:2407.01502. [45] Alex Kipnis, Konstantinos Voudouris, Luca M Schulze Buschoff, and Eric Schulz. 2024. metabench–a sparse benchmark of reasoning and knowledge in large language models. arXiv preprint arXiv:2407.12844. [46] Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. [47] Hanna Köpcke, Andreas Thor, and Erhard Rahm. 2010. Evaluation of entity resolution approaches on real-world match problems. Proceedings of the VLDB Endowment, 3(1–2):484– 493. [48] Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. 2025. “give me bf16 or give me death”? accuracy-performance trade-offs in llm quantization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26872–26886. [49] Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko, Aman Jaiswal, Kusha Sareen, Shailesh Nanisetty, Joan Cabezas, Manuel Del Verme, Omar G. Younis, Simone Baratta, Matteo Avalle, Imene Kerboua, Xing Han Lù, Elron Bandel, Michal Shmueli-Scheuer, Asaf Yehudai, Leshem Choshen, Jonathan Lebensold, Sean Hughes, and 7 others. 2026. Cube: A standard for unifying agent benchmarks. Preprint, arXiv:2603.15798. [50] John Patrick Lalor and Pedro Rodriguez. 2023. py-irt: A scalable item response theory library for python. INFORMS Journal on Computing, 35(1):5–13. [51] Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, and Nitesh V. Chawla. 2026. Adaptive testing for llm evaluation: A psychometric alternative to static benchmarks. Preprint, arXiv:2511.04689. [52] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An automatic evaluator of instruction-following models. 13

[53] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew Hudson, and 31 others. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research. [54] Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. 2021. Are we learning yet? A meta review of evaluation failures across machine learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. [55] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252. Association for Computational Linguistics. [56] Jiarui Liu, Wenkai Li, Zhijing Jin, and Mona Diab. 2024. Automatic generation of model and data cards: A step towards responsible AI. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1975–1997, Mexico City, Mexico. Association for Computational Linguistics. [57] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2025. Spinquant: Llm quantization with learned rotations. Preprint, arXiv:2405.16406. [58] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098. Association for Computational Linguistics. [59] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150. Association for Computational Linguistics. [60] Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165. Association for Computational Linguistics. [61] Yinan Mei, Shaoxu Song, Chenguang Fang, Haifeng Yang, Jingyun Fang, and Jiang Long. 2021. Capturing semantics for imputation with pre-trained language models. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 61–72. IEEE. [62] Guangyu Meng, Qinkai Zeng, John P Lalor, and Hong Yu. 2025. A psychology-based unified dynamic framework for curriculum learning. Computational Linguistics, pages 1–49. [63] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843. [64] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, and 66 others. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. Preprint, arXiv:2601.11868. [65] METR. 2026. Time horizon 2026-1-29-time-horizon-1-1/.

1.1.

https://metr.org/blog/

[66] James A Michaelov, Catherine Arnett, Tyler A Chang, Pamela D Rivière, Samuel M Taylor, Cameron R Jones, Sean Trott, Roger P Levy, Benjamin K Bergen, and Micah Altman. 2026. How open must language models be to enable reliable scientific inference? arXiv preprint arXiv:2603.26539. 14

[67] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220–229. Association for Computing Machinery. [68] Colin Murphy and Chang Liu. 2025. Ai-assisted wordle demo: Combining llms and rule-based solvers for enhanced gameplay. In 2025 IEEE Conference on Games (CoG), pages 1–2. IEEE. [69] Sree Harsha Nelaturu, Nishaanth Kanna Ravichandran, Cuong Tran, Sara Hooker, and Ferdinando Fioretto. 2024. On the fairness impacts of hardware selection in machine learning. Proceedings of the 41st International Conference on Machine Learning (ICML). [70] Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. 2024. Efficient benchmarking (of language models). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2519–2536, Mexico City, Mexico. Association for Computational Linguistics. [71] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2463–2473. Association for Computational Linguistics. [72] Felipe M Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson F de Oliveira, Yuekai Sun, and Mikhail Yurochkin. 2024. Efficient multi-prompt evaluation of llms. Advances in Neural Information Processing Systems, 37:22483–22512. [73] Felipe Maia Polo, Leshem Choshen, Yuekai Sun, and Kristjan Greenewald. 2025. A statistical framework for game-based ai evaluation. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. [74] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. Preprint, arXiv:2402.14992. [75] Inioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. [76] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, and 28 others. 2020. MLPerf inference benchmark. In Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture. [77] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. [78] Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. 2024. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. Preprint, arXiv:2411.12990. [79] Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P Lalor, Robin Jia, and Jordan Lee Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change nlp leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4486–4503. [80] Yangjun Ruan, Chris J Maddison, and Tatsunori B Hashimoto. 2024. Observational scaling laws and the predictability of language model performance. Advances in Neural Information Processing Systems, 37:15841–15892. 15

[81] Schema.org. 2026. Dataset - Schema.org Type. Schema.org vocabulary documentation. Accessed 2026-05-01. [82] Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, Muhammad Jehanzeb Mirza, Leshem Choshen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, and 1 others. 2024. Livexiv-a multi-modal live benchmark based on arxiv papers content. In The Thirteenth International Conference on Learning Representations. [83] Zachary S Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. 2024. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. Transactions on Machine Learning Research. [84] Anna Sokol, Elizabeth Daly, Michael Hind, David Piorkowski, Xiangliang Zhang, Nuno Moniz, and Nitesh Chawla. 2025. Benchmarkcards: Standardized documentation for large language model benchmarks. Preprint, arXiv:2410.12974. [85] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, and 432 others. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Preprint, arXiv:2206.04615. [86] Leon Staufer, Mick Yang, Anka Reuel, and Stephen Casper. 2025. Audit cards: Contextualizing ai evaluations. arXiv preprint arXiv:2504.13839. [87] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. In Workshop on Efficient Systems for Foundation Models @ ICML2023. [88] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca A. Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. ArXiv, abs/2410.12784. [89] Fireworks AI Engineering Team. 2025. Quality-first with kimi k2.5: The importance of post-training and serving infrastructure. [90] W3C Dataset Exchange Working Group. 2024. Data Catalog Vocabulary (DCAT) – Version 3. W3C Recommendation. Accessed 2026-05-01. [91] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems. [92] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. [93] Angelina Wang, Aaron Hertzmann, and Olga Russakovsky. 2024. Benchmark suites instead of leaderboards for evaluating AI fairness. Patterns, 5(11):101080. [94] Yuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, and Christian Szegedy. 2021. LIME: Learning inductive bias for primitives of mathematical reasoning. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11251–11262. PMLR. [95] Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of llm-based agents. Preprint, arXiv:2503.16416. [96] Asaf Yehudai, Lilach Eden, and Michal Shmueli-Scheuer. 2026. Agentic clear: Automating multi-level evaluation of llm agents. Preprint, arXiv:2605.22608. 16

[97] Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. Understanding and mitigating numerical sources of nondeterminism in llm inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [98] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36. [99] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning. [100] Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2021. AR-LSAT: Investigating analytical reasoning of text. Preprint, arXiv:2104.06598.

Appendix Table of Contents A Summary Statistics Every Eval Ever Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.1 Overview of inference platform distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2 Overview of the top 25 models in Every Eval Ever . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.3 Overview of evaluation runs by source organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.4 Overview of evaluation activity across the top 25 benchmarks . . . . . . . . . . . . . . . . . . . . . . 21 B

Full Every Eval Ever Schema Field Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

C Converter Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.1 HELM Converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.2 lm-eval-harness Converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.3 Inspect AI Converter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 C.4 Community Converters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 D

Conservative Estimation of Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

E

Governance Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

F Case Studies Reproducibility and Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 F.1 Case Study 1: Cost-Accuracy Tradeoffs in Agentic Evals . . . . . . . . . . . . . . . . . . . . . . . . . . 35 F.2 Case Study 2: Version-Dependent Perplexity Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 F.3 Case Study 3: Evaluation Reproducibility Gaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 F.4 Case Study 4: Evaluation Meta-Analysis using Item Response Theory . . . . . . . . . . . . . . 37

17

A

Summary Statistics Every Eval Ever Datastore

We present a high level summary and breakdown of key fields in Every Eval Ever in Tables 4, 5, 6, 7. A.1

Overview of inference platform distribution

Table 4: Inference platform distribution by evaluation runs and model diversity. Over 98% of runs fall under Unreported or Unknown categories. Among identified providers, Ollama and OpenAI lead in both volume and unique model representation.

Inference Platform

Eval. Runs

Models

Unreported Unknown

184,929 42,260

(80.56%) (18.41%)

17,101 5,282

(75.71%) (23.38%)

Ollama OpenAI Google Together Anthropic Mistral DeepSeek Cohere xAI OpenRouter AWS Gemini Aliyun Perplexity Local Ark Moonshot MiniMax StepFun Qwen Tencent Zhipu Kuaishou

849 411 312 191 152 100 56 48 31 30 30 27 22 21 20 15 13 12 11 10 10 9 3

(0.37%) (0.18%) (0.14%) (0.08%) (0.07%) (0.04%) (0.02%) (0.02%) (0.01%) (0.01%) (0.01%) (0.01%) (0.01%) (0.01%) (0.01%) (0.01%) (0.01%) (0.01%) (0.00%) (0.00%) (0.00%) (0.00%) (0.00%)

55 32 28 13 16 10 4 5 3 10 3 3 5 2 1 5 2 1 1 2 1 1 1

(0.24%) (0.14%) (0.12%) (0.06%) (0.07%) (0.04%) (0.02%) (0.02%) (0.01%) (0.04%) (0.01%) (0.01%) (0.02%) (0.01%) (0.00%) (0.02%) (0.01%) (0.00%) (0.00%) (0.01%) (0.00%) (0.00%) (0.00%)

18

A.2

Overview of the top 25 models in Every Eval Ever

Table 5: This shows the breakdown of the top 25 models in Every Eval Ever and the total number of evaluations across all runs. The data highlights a strong concentration of evaluation runs for the GPT-4 family, while also showing the emergence of frontier models like DeepSeek-R1 and Gemini-3 previews in current evaluation cycles.

Models

Eval. Runs

GPT-4o GPT-4 Gemini-1.5-Pro GPT-4o-mini DeepSeek-R1 GPT-4.1 Claude-3.5-Sonnet Claude 3.5 Sonnet DeepSeek-V3 GPT-3.5 Gemini-2.5-Pro GPT-4V Human GPT-3.5-Turbo o4-mini GPT-5 Gemini 2.5 Pro Gemini-2.0-Flash Qwen2.5-VL-7B o3 Qwen2.5-VL-72B o1-mini Qwen2.5-72B-Instruct Claude-3.7-Sonnet InternVL2.5-8B

4,443 1,211 1,007 914 788 709 610 604 588 575 568 559 545 475 473 452 431 422 417 398 366 354 349 342 339

19

(1.94%) (0.53%) (0.44%) (0.40%) (0.34%) (0.31%) (0.27%) (0.26%) (0.26%) (0.25%) (0.25%) (0.24%) (0.24%) (0.21%) (0.21%) (0.20%) (0.19%) (0.18%) (0.18%) (0.17%) (0.16%) (0.15%) (0.15%) (0.15%) (0.15%)

A.3

Overview of evaluation runs by source organization

Table 6: The table shows individual evaluation runs by source organization. The dataset is characterized by a significant volume of records from alphaXiv, exceeding 160,000 entries. Notably, this schema includes a university consortium comprising Princeton University, New York University, University of Washington, University of California San Diego, and Canyon Crest Academy, which together contribute to the diverse academic representation.*

Organizations

Eval. Runs

alphaXiv Hugging Face Valsai Artificial Analysis crfm UC Berkeley Gorilla LLM Stats Allen Institute for AI TIGERLab LiveBench HumanCentered Eval ARC Prize Wordle Arena Project kaggle Princeton SAgE Team ByteDanceSeed Chang Liu’s Lab SWEbench Mercor Ai2 CapArena Google DeepMind Exgentic Princeton, NYU, UW, UCSD, CCA* CocoaBench AmazonScience La Leaderboard

162,616 27,444 8,144 8,104 4,712 3,350 3,002 2,404 2,220 1,761 1,218 1,020 1,002 912 342 261 238 127 125 114 110 108 90 87 40 16 5

20

(71.23%) (12.02%) (3.57%) (3.55%) (2.06%) (1.47%) (1.32%) (1.05%) (0.97%) (0.77%) (0.53%) (0.45%) (0.44%) (0.40%) (0.15%) (0.11%) (0.10%) (0.06%) (0.05%) (0.05%) (0.05%) (0.05%) (0.04%) (0.04%) (0.02%) (0.01%) (0.00%)

A.4

Overview of evaluation activity across the top 25 benchmarks

Table 7: Distribution of evaluation activity across the top 25 most popular benchmarks. The data shows a high density of testing within the Artificial Analysis LLM API framework, followed by foundational reasoning and knowledge benchmarks such as GPQA, IFEval, and MMLU-PRO, reflecting their role as industry standards for model performance assessment.

Benchmark

Eval. Runs

Artificial Analysis LLM API GPQA IFEval BBH MATH Level 5 MMLU-PRO MUSR BFCL leaderboard CSV helm_mmlu LiveBench MMLU-Pro leaderboard RewardBench 2 RewardBench ARC Prize evaluations leaderboard Japanese Financial Benchmark global-mmlu-lite OCRBench v2 FinanceMATH MM-InstructEval CharXiv EffiBench-X SuperGPQA SEED-Bench-2 fibble_arena_daily REST

8,104 4,662 4,662 4,585 4,574 4,574 4,574 3,350 2,844 2,286 2,220 1,379 1,025 1,020 943 912 874 817 783 780 716 614 607 559 528

21

(3.53%) (2.03%) (2.03%) (2.00%) (1.99%) (1.99%) (1.99%) (1.46%) (1.24%) (1.00%) (0.97%) (0.60%) (0.45%) (0.44%) (0.41%) (0.40%) (0.38%) (0.36%) (0.34%) (0.34%) (0.31%) (0.27%) (0.26%) (0.24%) (0.23%)

B

Full Schema Field Reference

This appendix describes the top-level interface of the unified evaluation schema used in Every Eval Ever. The schema is splitted into two linked records: an aggregate evaluation record for run-level metadata and summary metrics, and a companion instance-level record for per-sample outcomes. In the current release, the canonical interfaces are eval.schema.json (version 0.2.2) and instance_level_eval.schema.json (version instance_level_eval_0.2.2). Both schemas define closed top-level interfaces: unspecified top-level fields are not permitted. B.1

Aggregate Evaluation Records

The aggregate record represents a single evaluation run for one model and stores the provenance, model context, evaluation framework, and one or more reported metric results. It is defined by eval.schema.json version 0.2.2. Its top-level fields are summarized in Table 8. Each element of evaluation_results is an evaluation_result object. Its top-level structure is summarized in Table 9. Taken together, the fields in Tables 8 and 9 define the aggregate representation of an evaluation run. The separation between evaluation_timestamp and retrieved_timestamp distinguishes when the evaluation was executed from when the standardized record was created, while the evaluation_results array allows multiple benchmark outcomes or metrics to be attached to the same run-level record. B.2

Instance-Level Evaluation Records

The instance-level record represents a single benchmark sample associated with an aggregate evaluation run. It is defined by instance_level_eval.schema.json version instance_level_eval_0.2.2 and is typically stored as a companion JSONL file. Its top-level fields are summarized in Table 10. The instance-level schema is intentionally aligned with the aggregate schema but preserves samplelevel detail needed for auditing and re-analysis. The evaluation_id field links each row back to the aggregate JSON, while evaluation_result_id provides the preferred deterministic link to one specific aggregate metric result. The conditional use of output versus messages makes the schema applicable to standard single-turn tasks as well as conversational and tool-using evaluations.

C

Converter Implementation Details

This section describes how evaluation logs from HELM, lm-eval-harness, and Inspect AI are mapped into the unified EEE schema. All three converters produce an aggregate EvaluationLog. When instance-level data is available, they also emit a JSONL file referenced by detailed_evaluation_results. In addition to these core framework converters, the repository includes communitycontributed converters for public leaderboards and benchmark-specific result sources, summarized in a final subsection. C.1

Inspect AI Converter

Input format. The Inspect converter accepts evaluation logs with extension .eval or .json. Both formats represent the same logical object and are read through the Inspect log API1 . The top-level object is structurally rich: eval stores task, dataset, model path, package versions, generation settings, and task arguments; plan stores solver steps and plan configuration; results stores scorer-level aggregate metrics; stats stores run timestamps and summary counters; samples stores per-sample traces; and reductions stores scorer-level reduced sample values used for score resolution. This structure motivates field-wise extraction rather than direct key renaming. Aggregate mapping to EEE Core EEE fields are assembled from several Inspect structures. The converter derives evaluation_timestamp from stats.started_at with fallback 1 https://inspect.aisi.org.uk/reference/inspectAI.log.html

22

Table 8: Top-level fields of the aggregate evaluation record. Field

Type

Required

Description

Notes

schema_version

string

Yes

evaluation_id

string

Yes

evaluation_timestamp retrieved_timestamp source_metadata

string

No

string

Yes

object

Yes

eval_library

object

Yes

model_info

object

Yes

evaluation_results

array

Yes

detailed_evaluation_results

object

No

Version of the schema used for the For this appendix, the current released aggregate evaluation record. value is 0.2.2. Unique identifier for a specific eval- This field identifies the run as uation run. a whole rather than an individual metric result. The schema documentation recommends the form eval_name/model_id/ retrieved_timestamp. Timestamp indicating when the This is run time, not record-creation evaluation was run. time. Timestamp indicating when the ag- This is record-creation time in Unix gregate record was created. epoch format and is required. Metadata describing where the eval- This nested object captures proveuation evidence comes from. nance such as whether the result comes from documentation or a direct evaluation run, the providing organization, and the evaluator-model relationship. Metadata describing the evaluation This nested object records the frameframework used to obtain the result. work name and version, with optional additional details. Canonical description of the evalu- This nested object captures model ated model. identity and inference context, including model identifier, developer, and optional inference platform or local inference engine. Collection of metric results re- This is the main result-bearing field. ported for the evaluation run. Each element stores one metric outcome together with its dataset source, metric configuration, score details, and optional generation configuration. A single run may report multiple metric results. Reference to a companion file con- When present, this object points to the taining per-sample outcomes. instance-level JSONL file and records linkage metadata such as format, path, checksum, hash algorithm, and row count.

to eval.created, derives eval_library from eval.packages, and derives evaluation_results from results.scores. It initializes model_info from eval.model and, when available, refines that identifier with sample-level output model metadata. Inspect model paths vary across providers, so the converter applies provider-specific normalization to produce a canonical model_info.id and to infer inference_platform and, when possible, inference_engine. For example, openai/azure/gpt-4o-mini may be refined to openai/gpt-4o-mini-2024-07-18, while ollama/qwen2.5:0.5b is normalized to ollama/qwen2.5-0.5b. The converter derives per-result source data mainly from eval.dataset. In particular, eval.dataset.location is treated as a Hugging Face repository only when it matches the canonical namespace/name form; otherwise the converter preserves the raw Inspect dataset fields in additional_details. Each scorer metric is converted into an EvaluationResult, except standalone stderr, which is treated as uncertainty metadata rather than a separate metric. Metric values populate score_details.score, and uncertainty is populated from scorer-reported stderr together with optional std or stddev and sample counts. When scorer parameters expose grader_model and grader_template, the converter also populates metric_config.llm_scoring to preserve judge context. Finally, generation_config combines standard generation parameters with Inspect-specific execution context, including prompt template, available tools, serialized plan, limits, sandbox configuration, retry settings, and the reasoning flag. Instance-level mapping When sample logs are present, each sample is converted into one InstanceLevelEvaluationLog. Interaction type is inferred from the message structure: tool-role messages yield agentic, multiple assistant turns without tools yield multi_turn, and the remaining cases are treated as single_turn. Input text is serialized from user messages, with references and 23

choices preserved from sample targets and choices. For single-turn cases, output text and reasoning traces are stored in output; for multi-turn and agentic cases, the normalized message sequence is stored in messages, including tool calls. Score resolution prioritizes reductions matched by sample and scorer, then sample-level reduced values, and finally direct sample scores. If no score is available, the converter falls back to reference matching, and correctness follows the resolved score semantics. The instance-level record also preserves token usage, latency and generation time, sample hash, stop reason, epoch, answerattribution metadata, and full error traces when available. C.2

HELM Converter

Input format. The HELM converter operates on a HELM run directory. It requires run_spec.json, scenario_state.json, scenario.json, and per_instance_stats.json, while the optional stats.json file supplies aggregate metric values when present. Each file contributes a distinct part of the run state. run_spec.json provides adapter, metric, and scenario specification metadata; scenario_state.json provides request-level records, including prompts, references, outputs, and request timestamps; scenario.json provides scenario naming metadata used during dataset identification; and per_instance_stats.json provides per-sample metric and token statistics for instance-level conversion. When available, stats.json adds aggregate statistics such as mean, sum, count, and standard deviation. Because HELM distributes these signals across separate artifacts, the converter must coordinate extraction across multiple files rather than perform direct key renaming. Aggregate mapping to EEE Core EEE fields are assembled from multiple HELM artifacts. The converter derives model_info from run_spec.json, preferably through deployment registry entries referenced by adapter_spec.model_deployment; if deployment lookup is unavailable, it falls back to adapter-level model fields and best-effort platform inference. It derives evaluation_results primarily from stats.json, with candidate metric names anchored in run_spec.metric_specs. The converter derives per-result source data and timestamp fields from scenario_state.json and scenario.json. Dataset name comes from scenario.name when available, with fallback parsing from run_spec.name; sample counts and sample identifiers come from request-state instance ids; and scenario class names and arguments are preserved in additional_details. For each matched aggregate statistic, the score is taken from mean with fallback to sum/count, while uncertainty records stddev and sample-count metadata. If stats.json is absent, aggregate metric output may be empty. generation_config is derived from request-level and adapter-level settings, including temperature, top_p, top_k, max_tokens, stop sequences, penalties, completion count, and a reasoning flag inferred from HELM thinking traces. The converter computes evaluation_timestamp from the earliest available request datetime, with fallback to retrieval time, and forms evaluation_id from dataset, model, and timestamp after path-safe normalization of model identifier separators. Instance-level mapping When request-state records are available, the converter emits one consolidated JSONL file and references it through detailed_evaluation_results. Each row combines request_states with per_instance_stats: prompt text, references, and choices come from request-state records and output mapping metadata, while completions and optional reasoning traces come from model outputs. Score resolution prefers per-instance exact_match statistics and otherwise falls back to reference matching between generated completions and tagged correct references. The converter also records token usage, generation latency, stable sample identifiers, sample hash, answer-attribution metadata, and single-turn interaction typing. C.3

lm-eval-harness Converter

Input format. The lm-eval converter consumes result files named results_*.json. Optional instance-level conversion uses files named samples_<task>_*.jsonl and is enabled only when sample logging is available and conversion is executed with –include_samples. The aggregate record combines several top-level maps: config provides global run and model metadata, configs 24

provides task-level dataset and generation settings, results provides task-level metric values, higher_is_better provides metric directionality, n-samples provides sample-count metadata for uncertainty reporting, date maps to evaluation_timestamp, and lm_eval_version maps to eval_library.version. This layout motivates task-wise field extraction rather than direct key renaming. Table 9: Top-level fields of a single evaluation_result entry within evaluation_results. Field

Type

Required

Description

evaluation_result_id

string

No

evaluation_name

string

Yes

source_data

object

Yes

evaluation_timestamp

string

No

metric_config

object

Yes

score_details

object

Yes

generation_config

object

No

Stable identifier for a specific met- This is the preferred deterministic join ric result within the evaluation run. key for linking instance-level rows to one aggregate metric result. Name of the evaluation associated This identifies the benchmark or task with the metric result. reported by the metric entry. Description of the dataset source This object is a tagged union with used by the metric result. three variants: URL-based source data, Hugging Face dataset source data, and other for private or custom datasets. Timestamp indicating when this This field is useful when different respecific metric result was produced. sults within the same run were generated at different times. Metadata defining how the metric This object captures metric semantics should be interpreted. such as whether lower values are better, the score type, and optional normalized metric identifiers or parameters. Reported score and associated quan- This object stores the metric value ittitative details. self and may additionally include uncertainty estimates or supplementary score details. Configuration describing how This object can record generation armodel outputs were generated for guments and additional run details, this metric result. including settings relevant to agentic evaluations.

Notes

Aggregate mapping to EEE The results object may contain both metric-bearing tasks and structural placeholders, so the converter first excludes placeholders and tasks without numeric metrics and then emits one EvaluationLog per retained task. For each retained task, it derives evaluation_timestamp from date, derives eval_library.version from lm_eval_version, derives model_info from config, and derives both per-result source data and generation_config from task-specific entries in configs. model_info is constructed from config.model and parsed config.model_args. Because model_args is often a comma-delimited string, the converter parses it heuristically and prioritizes pretrained when present; inference platform and inference engine are then inferred from model-type mappings, with an optional command-line override for engine name and version. Perresult source data is derived from task configuration fields such as dataset_path and split metadata. Paths that match Hugging Face repository form are mapped to source_type=hf_dataset, while other paths are mapped to source_type=other. Metric keys typically follow the metric,filter convention, such as exact_match,none, and uncertainty keys follow the metric_stderr,filter convention. The converter decomposes these keys, creates one EvaluationResult per numeric metric, and maps standard error to score_details.uncertainty.standard_error. Metric directionality is derived from higher_is_better and inverted into lower_is_better; score bounds are inferred from a known-metrics table when available and left unset otherwise. Finally, generation_config is derived from generation_kwargs, including temperature, top_p, top_k, and max_gen_toks, while the remaining generation attributes and num_fewshot are preserved in additional_details. Instance-level mapping When sample JSONL files are available, each sample row is converted into one InstanceLevelEvaluationLog. Prompt and references are extracted from arguments and target, and for multiple-choice tasks the answer options are reconstructed from gen_args_*. For generation tasks, the converter uses the first response text; for multiple-choice tasks, it selects the option with the highest log probability from filtered_resps or resps. Scores and correctness 25

are derived from per-sample metric fields, with fallback to score=0.0 and is_correct=false when no numeric metric value is available. The converter also records a sample hash, lm-eval hashes doc_hash, prompt_hash, and target_hash, the applied filter name, and the serialized per-sample metric payload. C.4

Community-Contributed Converters

AlpacaEval. The AlpacaEval converter fetches the public AlpacaEval 1.0 and 2.0 leaderboard CSVs. It preserves pairwise preference metrics against the published baselines, including win rate, length-controlled win rate, discrete win rate, and average response length for each model. ARC-AGI. The arc_agi adapter reads the ARC Prize evaluations leaderboard JSON from arcprize.org. It records the published ARC score together with cost-per-task and total-cost fields while normalizing the often informal model aliases used on the leaderboard. Artificial Analysis. The artificial_analysis adapter ingests the Artificial Analysis LLM API, which combines benchmark scores with pricing and latency measurements for frontier models. It carries over composite indices such as the Artificial Analysis intelligence, coding, and math indices, benchmark scores such as MMLU-Pro, GPQA, HLE, LiveCodeBench, SciCode, AIME, and tau2, and token-pricing and latency summaries. BFCL. The bfcl adapter reads the BFCL leaderboard CSV published by Berkeley Gorilla. It preserves the leaderboard’s overall rank, overall accuracy, latency and cost fields, and the benchmark’s finer-grained tool-calling slices, including non-live, live, multi-turn, and web-search accuracies. CocoaBench. The cocoabench adapter reads CocoaBench’s published per-system CSV of agent performance, time, and cost. It preserves overall benchmark accuracy together with average runtime per task, average cost per task, and total evaluation cost for each released agent-model system. Exgentic. The exgentic adapter consumes Exgentic open-agent leaderboard aggregates, either from local results.json files or the Hugging Face dataset. These runs span agent benchmarks such as AppWorld, SWE-bench, BrowseComp+, and Tau2, and the adapter preserves benchmark score, session counts, and run-cost summaries for each agent-model submission. Global MMLU Lite. The global-mmlu-lite adapter fetches the Global MMLU Lite leaderboard from the Kaggle Benchmarks API. It preserves the reported Global MMLU Lite score for each model together with any confidence-interval or standard-deviation information exposed by the leaderboard payload. Open LLM Leaderboard v2. The hfopenllm_v2 adapter ingests the Hugging Face Open LLM Leaderboard v2 API. It preserves the benchmark panel used by that leaderboard, including IFEval, BBH, MATH Level 5, GPQA, MUSR, and MMLU-Pro, together with basic model metadata such as architecture, precision, and parameter count when available. LLM Stats. The llm_stats adapter consumes the LLM Stats API’s combined model, benchmark, and score payloads. It is designed for a broad benchmark catalog rather than a single leaderboard, so it preserves benchmark-specific provenance URLs, relationship metadata, pricing and context-window model details, and the score entries attached to each model. Multi-SWE-Bench. The multi_swe_bench adapter clones the Multi-SWE-Bench experiments repository and reads verified submissions under each language-specific leaderboard. It preserves resolved-instance rates and submission metadata for C, C++, Go, Java, JavaScript, Rust, and TypeScript tracks. RewardBench. The rewardbench adapter fetches RewardBench v1 leaderboard CSV data and RewardBench v2 JSON results from Hugging Face. It preserves the v1 overall, chat, chat-hard, safety, reasoning, and prior-set scores, as well as the v2 factuality, precise instruction following, math, safety, focus, and tie-handling metrics. 26

SciArena. The sciarena adapter reads the SciArena leaderboard API maintained by Allen AI. It preserves the published rank, arena rating, and cost-per-100-calls metadata for each model, while keeping the source model aliases close to the leaderboard’s own naming. SWE-bench Verified. The swe_bench_verified adapter reads verified submission directories from the public SWE-bench experiments repository. It preserves the standard verified leaderboard signal, namely the fraction of the 500 benchmark instances resolved by each submission, along with submission metadata and agent tooling context. SWE-PolyBench. The swe_polybench adapter reads submission artifacts for SWE-PolyBench and SWE-PolyBench Verified from the public experiments repository. It preserves resolved-instance rates separately for each dataset variant and programming language, so one submission may yield distinct records for different language tracks. Terminal-Bench 2.0. The terminal_bench_2 adapter captures the published Terminal-Bench 2.0 leaderboard for agentic coding systems. It preserves the leaderboard’s accuracy and standard-error values for each agent-model pair on the 87-task benchmark, together with the agent and model organization metadata shown on the leaderboard.

27

Table 10: Top-level fields of the instance-level evaluation record. Field

Type

Required

Description

schema_version

string

Yes

evaluation_id

string

Yes

model_id

string

Yes

evaluation_name

string

Yes

evaluation_result_id

string

No

sample_id

string

Yes

sample_hash

string | null

No

interaction_type

string

Yes

input

object

Yes

output

object | null array | null

No

answer_attribution

array

Yes

evaluation

object

Yes

token_usage

object | null

No

performance

object | null

No

error

string | null object | null

No

Version of the schema used for the For this appendix, the current reinstance-level record. leased value is instance_level_eval_0.2.2. Foreign key linking the instance- This value must match the aggregate level record to the aggregate evalu- record and anchors the per-sample row ation JSON. to a specific evaluation run. Identifier of the evaluated model. This is the model identifier for the sample-level record and supports joins or filtering even when the aggregate file is not loaded. It should use Hugging Face-style formatting, i.e., model_developer/model_name. Name of the evaluation associated This is primarily a display and filterwith the sample. ing field when no deterministic metriclevel join key is available. Preferred foreign key to a specific This is the preferred one-to-one link aggregate metric result. from an instance-level row to a particular element of evaluation_results. If one underlying sample contributes to multiple aggregate metrics, separate instance-level records should be emitted. Identifier of the source benchmark This is typically inherited from the sample. original dataset. Hash-based identifier for the sam- This supports cross-model matchple content. ing when sample_id is unstable or inconsistent across sources. Operationally, it is computed from the concatenation of input.raw and input.reference, yielding a content-based identifier for the sample. Interaction regime for the sample. Allowed values are single_turn, multi_turn, and agentic. This field controls which output container is valid. Input content presented to the This nested object stores the raw model for the sample. prompt, reference answers, and optional formatted input or answer choices. Model output for single-turn evalu- single_turn requires output and ations. forbids messages. Message transcript for multi-turn or multi_turn and agentic require agentic evaluations. messages and forbid output. The message list can include tool calls and tool outputs. Record of how the scored answer This array identifies the source lowas extracted from the model be- cation, extraction method, extracted havior. value, and whether the extracted answer is terminal. Instance-level scoring outcome for This nested object stores the sample the sample. score and correctness, and may additionally report turn counts or tool-call counts. Token accounting for the sample. This nested object records input, output, and total token counts, with optional cache and reasoning-token fields. Latency and runtime measurements This nested object supports perfor the sample. sample performance analysis, including latency and time-to-first-token metrics. Error information associated with This field can capture failures such as the sample. refusals, timeouts, or API errors. Optional sample-level metadata. This object can store benchmarkspecific annotations such as subject, difficulty, or tags without changing the top-level interface.

messages

metadata

No

No

28

Notes

D

Conservative Estimation of Costs

We explain here our assumptions on how we estimate the cost for running evaluations to reproduce all of our data. While we note that this is a vast underapproximation of the actual cost of reproduction all this work, we still see it as a sign for the importance of collecting such data. D.1

Dataset and Evaluation Scale

The dataset comprises approximately 230,000 model–benchmark evaluation pairs, where each evaluation represents running a model on a single benchmark. Each benchmark is assumed to contain 1,000 examples, with roughly 100 input tokens and 20 output tokens per example. Under these assumptions, each evaluation uses about 100,000 input tokens and 20,000 output tokens, for a total of 120,000 tokens before additional overhead. D.2

LLM-as-Judge Overhead

Modern evaluation pipelines frequently incorporate an additional language model to automatically grade or compare outputs, commonly referred to as an “LLM-as-judge.” Based on production observations, this introduces an additional 60% token overhead relative to the base evaluation. This overhead is modeled as a multiplicative factor applied uniformly to both input and output tokens, such that the adjusted token count is given by 1.6 times the base tokens. Consequently, each evaluation involves approximately 160,000 input tokens and 32,000 output tokens after accounting for this overhead. D.3

Total Token Volume

Aggregating across all 230,000 evaluations, the total token volume is obtained by multiplying the per-evaluation total of 192,000 tokens by the number of evaluations. This results in approximately 4.416 × 1010 tokens, corresponding to roughly 44 billion tokens processed in total. D.4

Cost Model

The total inference cost is computed as the sum of input and output token costs. Specifically, the cost is given by the product of input tokens and their per-million-token price, plus the product of output tokens and their corresponding price. The pricing parameters are denoted by Cin for input tokens and Cout for output tokens. We consider three levels of approximation corresponding to different pricing regimes. D.5

Low-Cost Estimate (No Judge)

As a lower bound, we consider a highly cost-efficient model with pricing of $0.10 per million input tokens and $0.40 per million output tokens. This estimate excludes any judge overhead and therefore uses the base token counts. Under these assumptions, the input cost per evaluation is computed as 100,000 × 0.10 106 , which equals $0.01. The output cost per evaluation is 20,000 × 0.40 , which equals $0.008. The total cost per 106 evaluation is therefore $0.018. Across all 230,000 evaluations, the total cost is approximately 230,000 × 0.018, which yields about $4,140. This corresponds to a total low-cost estimate of approximately $4.1K. D.6

Mid-Cost Estimate (Sonnet with Judge)

For a more realistic estimate, we consider a mid-tier model with pricing of $3 per million input tokens and $15 per million output tokens. This estimate incorporates the 60% judge overhead. 29

With adjusted token counts, the input cost per evaluation is 160,000 × 1036 , which equals $0.48, and 15 the output cost is 32,000 × 10 6 , which also equals $0.48. The total cost per evaluation is therefore $0.96. Across all evaluations, the total cost is approximately 230,000 × 0.96, which yields about $220,800. This corresponds to a total mid-cost estimate of approximately $221K. D.7

High-Cost Estimate (Opus with Judge)

Finally, we consider a higher-end model with pricing of $5 per million input tokens and $25 per million output tokens, again including the 60% judge overhead. Under these conditions, the input cost per evaluation is 160,000 × 1056 , which equals $0.80, and the 25 output cost is 32,000 × 10 6 , which also equals $0.80. The total cost per evaluation is therefore $1.60. Across all evaluations, the total cost is approximately 230,000 × 1.60, resulting in about $368,000. This corresponds to a total high-cost estimate of approximately $368K. D.8

Summary

Under the stated assumptions, the total cost of evaluating 230,000 model–benchmark pairs ranges from a lower bound of approximately $4K, assuming no judge and highly optimized pricing, to approximately $370K when using a high-end model with judge overhead. A mid-tier estimate of roughly $220K is also given. While the lower bound is likely unrealistic, the others might be closer to actual pricing as most models are not of the smaller kinds and usually top or middle models are evaluated, with or without a judge.

E

Governance Card

Every Eval Ever is a community project. This appendix documents the governance mechanisms currently in place. We follow the spirit of the Croissant governance process [3] and adapt it to the specifics of Every Eval Ever. The governance model remains dynamic, and we expect it to evolve as the project progresses. E.1

Decision-Making and Roles

The project recognizes three key roles. Core maintainers are responsible for repository upkeep, schema releases, converter maintenance, reviewing contributions, and final decisions on contested proposals. Contributors submit data, converters, schema proposals, tooling, or documentation through pull requests and issues or discussions through GitHub or, on occasion, Slack. Community reviewers are volunteer experts who participate in schema discussions and review proposals in their area of expertise. Roles are not exclusive: maintainers also contribute and become so through community acceptance and after several contributions. Routine decisions (record additions that pass validation, bug fixes, documentation updates, and additive non-breaking schema fields) are made by maintainers on a rolling basis. Substantive decisions (breaking schema changes, new interaction types, deprecations, deduplication policy changes) follow the proposal process below. E.2

Schema Change Proposal Process

Substantive schema changes follow a lightweight three-stage process modeled on the iterative methodology used to produce the vx.x.x schema (Section 3). 1. Proposal. A contributor opens an issue in the repository describing the proposed change, or is raised during discussion between maintainers. The problem it solves and the implications are discussed, and alternatives are weighed. 2. Community review. The proposal is open for discussion until disagreements are resolved. If necessary, maintainers solicit feedback from relevant community experts based on the area of the proposal. 30

3. Resolution. Maintainers summarize the discussion and propose a resolution: accept, accept with modification, defer, or decline. Decisions are made by consensus among maintainers; when consensus cannot be reached, a documented majority decision is recorded, with dissenting positions preserved in the schema change-log (Section 3). E.3

Conflicting Submissions and Duplicate Records

Because the schema assigns a unique UUID to each evaluation run and defers deduplication to the analysis layer (Section 3.1), conflicting or near-duplicate records are expected and, by themselves, are not a governance problem. The validator flags likely duplicates (same model, same benchmark, same metric, same evaluator) at submission time but does not reject them. When users encounter conflicting records that cannot be reconciled from metadata alone, they are encouraged to open an issue; maintainers may then request additional metadata from the contributors, annotate records with a disputed flag in additional_details, or, in cases of clear error, mark records as superseded (see Section E.4 below). We do not arbitrate which of two methodologically valid evaluation runs is "correct." E.4

Corrections, Retractions, and Supersession

While there have not yet been any disputes over data contributions, we present here a proposal for how to address them when they do arise. We will continue to adapt this process in response to emerging real-world needs. Records are immutable once accepted: modifying a record in place would invalidate downstream analyses that reference it. Three mechanisms handle errors and updates: 1. Correction. For minor fixes (e.g., typos in identifiers), a new record is added that supersedes the original. The original is retained and annotated with a superseded_by field pointing to the corrected UUID. Similarly, a preceded_by field points to the original UIUD. 2. Retraction. For records that were submitted in error or are based on faulty source data, the record is annotated with a retracted flag and a brief reason. The record itself is not deleted, so prior analyses remain reproducible. 3. Schema migration. When schema versions advance, records remain valid under the version they were submitted with. Migration utilities are provided where possible, but historical records are not overwritten. E.5

Code of Conduct and Acknowledgment

The project follows a standard contributor code of conduct. Contributors are acknowledged in three ways: through git commit history, through the contributor list maintained in the repository, and, for substantive contributions to a release, through co-authorship on the associated release paper. The first such instance is the present submission, organized as a shared task [11]; subsequent releases will follow the same pattern with criteria documented in the contributor guide. E.5.1

Copy of the Contributor Guide

New data can be contributed to the Hugging Face Datastore using the following process: Leaderboard/evaluation data is split-up into files by individual model, and data for each model is stored using eval.schema.json. The repository is structured into folders as data/benchmark_name/developer_name/model_name/. TL;DR How to successfully submit 1. Data must conform to eval.schema.json (current version: 0.2.2) 2. The validation pipeline will automatically verify the data submitted in the pull request, but can also be manually triggered by typing /eee validate changed in a comment on the HF PR. 3. A core maintainer will review and merge your submission PR Naming Convention 31

Use these prefixes in your pull request titles: • [Submission] - New evaluation data • [Issue #N] - Fix for a specific GitHub issue • [Feature] - New functionality not tied to an issue • [Docs] - Documentation changes UUID Naming Convention Each JSON file is named with a UUID (Universally Unique Identifier) in the format uuid.json. The UUID is automatically generated (using standard UUID v4) when creating a new evaluation result file. This ensures that: • Multiple evaluations of the same model can exist without conflicts (each gets a unique UUID) • Different timestamps are stored as separate files with different UUIDs (not as separate folders) • A model may have multiple result files, with each file representing different iterations or runs of the leaderboard/evaluation • UUID’s can be generated using Python’s uuid.uuid4() function. Example: The model openai/gpt-4o-2024-11-20 might have multiple files like: • e70acf51-30ef-4c20-b7cc-51704d114d70.json (evaluation run #1) • a1b2c3d4-5678-90ab-cdef-1234567890ab.json (evaluation run #2) Note: Each file can contain multiple individual results related to one model. How to add new eval: 1. Add a new folder under data/ on the Hugging Face datastore with a codename for your eval. 2. For each model, use the Hugging Face (developer_name/model_name) naming convention to create a 2-tier folder structure. 3. Add a JSON file with results for each model and name it uuid.json. Optional Include a utils/ folder in your benchmark name folder with any scripts used to generate the data (e.g., utils/global-mmlu-lite/adapter.py). Submit Two ways to submit your evaluation data: • Option A: Drag & drop via Hugging Face — Go to the datastore → click “Files and versions” → “Contribute” → “Upload files” → drag and drop your data → select “Open as a pull request to the main branch”. • Option B: Clone & PR — Clone the repo, add your data under data, and open a pull request Schema Instructions 1. model_info: Use Hugging Face formatting (developer_name/model_name). If a model does not come from Hugging Face, use the exact API reference. Check examples in data/livecodebenchpro. Notably, some do have a date included in the model name, but others do not. For example: • OpenAI: gpt-4o-2024-11-20, gpt-5-2025-08-07, o3-2025-04-16 • Anthropic: claude-3-7-sonnet-20250219, claude-3-sonnet-20240229 • Google: gemini-2.5-pro, gemini-2.5-flash • xAI (Grok): grok-2-2024-08-13, grok-3-2025-01-15 2. evaluation_id: Use benchmark_name/model_id/retrieved_timestamp format (e.g. livecodebenchpro/qwen3-235b-a22b-thinking-2507/1760492095.8105888). 32

3. inference_platform vs inference_engine: Where possible specify where the evaluation was run using one of these two fields. • inference_platform: Use this field when the evaluation was run through a remote API (e.g., openai, huggingface, openrouter, anthropic, xai). • inference_engine: Use this field when the evaluation was run locally. This is now an object with name and version (e.g. "name": "vllm", "version": "0.6.0"). 4. The source_type on source_metadata has two options: documentation and evaluation_run. Use documentation when results are scraped from a leaderboard or paper. Use evaluation_run when the evaluation was run locally (e.g. via an eval converter). 5. source_data is specified per evaluation result (inside evaluation_results), with three variants: • source_type: "url" - link to a web source (e.g. leaderboard API) • source_type: "hf_dataset" — reference to a Hugging Face dataset (e.g. "hf_repo": "google/IFEval") • source_type: "other" — for private or proprietary datasets 6. The schema is designed to accommodate both numeric and level-based (e.g. Low, Medium, High) metrics. For level-based metrics, the actual ’value’ should be converted to an integer (e.g. Low = 1, Medium = 2, High = 3), and the level_names property should be used to specify the mapping of levels to integers. 7. Timestamps: The schema has three timestamp fields — use them as follows: • retrieved_timestamp (required) — when this record was created, in Unix epoch format (e.g. 1760492095.8105888) • evaluation_timestamp (top-level, optional) — when the evaluation was run • evaluation_results[].evaluation_timestamp (per-result, optional) — when a specific evaluation result was produced, if different results were run at different times 8. Additional details can be provided in several places in the schema. They are not required, but can be useful for detailed analysis. • model_info.additional_details: Use this field to provide any additional information about the model itself (e.g. number of parameters) • evaluation_results.generation_config.generation_args: Specify additional arguments used to generate outputs from the model • evaluation_results.generation_config.additional_details: Use this field to provide any additional information about the evaluation process that is not captured elsewhere Instance-Level Data For evaluations that include per-sample results, the individual results should be stored in a companion uuid_samples.jsonl file in the same folder (one JSONL per JSON, sharing the same UUID). The aggregate JSON file refers to its JSONL via the detailed_evaluation_results field. The instance-level schema (instance_level_eval.schema.json) supports three interaction types: • single_turn: Standard QA, MCQ, classification — uses output object • multi_turn: Conversational evaluations with multiple exchanges — uses messages array • agentic: Tool-using evaluations with function calls and sandbox execution — uses messages array with tool_calls Each instance captures: input (raw question + reference answer), answer_attribution (how the answer was extracted), evaluation (score, is_correct), and optional token_usage and performance metrics. Instance-level JSONL files are produced automatically by the eval converters. 33

E.6 E.6.1

Worked Examples Example 1: Conflicting MMLU Records

To make the governance mechanisms concrete, consider the LLaMA 65B/MMLU example from Section 1, where the model scores 63.7 under HELM and 48.8 under lm-eval-harness [29]. Under EEE, both results are valid records: each receives its own UUID, each carries eval_library metadata identifying the harness, and each preserves the generation configuration and prompt template available at submission time. The validator does not flag these as duplicates because the eval_library field differs. A downstream user comparing the two records sees the discrepancy in the metadata directly and can decide whether to treat them as comparable, rather than discovering the difference through a blog post months later. If a third contributor later submits a third MMLU record for LLaMA 65B without specifying the harness, the validator emits a warning, the record is accepted with the missing field recorded as absent (Section 3.2), and any downstream analysis that requires harness-level disambiguation can filter it out. No human governance intervention is needed for this case; the schema and validator handle it. Governance intervention is reserved for cases where metadata is contested rather than merely missing. E.6.2

Example 2: Disputed Agentic Record

A contributor submits records scraped from a public agentic-evaluation leaderboard. The records pass validation, but the agent’s developers later contest them, claiming the leaderboard ran a deprecated harness version and that the current version yields lower scores; another contributor argues the original entry should remain as the authoritative public record at the time of reporting. The schema cannot resolve this because both parties agree on the metadata. Maintainers handle it by retaining the original record (records are immutable; Section E.4), annotating it with a disputed flag under additional_details, pointing to the issue thread, inviting the developers to submit a new record under the current harness, and documenting the resolution in the changelog. EEE does not arbitrate which run is “correct”; it ensures both runs and their dispute are visible. Listing 1: Installing and running a converter. pip install ‘ every - eval - ever [ all ] ’ every_eval_ever convert helm -- log_path path / to / helm / logs every_eval_ever convert inspect -- log_path path / to / run . eval every_eval_ever convert lm_eval -- log_path path / to / results . json \ -- include_samples

34

Listing 2: CLI validation examples. # Validate a single aggregate file uv run python -m every_eval_ever validate path / to / uuid . json # Validate instance - level data uv run python -m every_eval_ever validate path / to / uuid_samples . jsonl # Validate an entire benchmark directory uv run python -m every_eval_ever validate data / mmlu /

F

Case Studies: Reproducibility and Implementation Details

F.1

Case 1

We report the aggregate records used for the agentic cost–accuracy analysis in Section 7.1. CocoaBench [37] is used to illustrate how runtime and cost can change the interpretation of accuracy across scaffold–backbone combinations. CORE-Bench Hard results from HAL [43] are used as a representative within-benchmark slice showing how both scaffold and backbone choices affect the cost–accuracy tradeoff. The corresponding records are available in the Every Eval Ever datastore under the CocoaBench and HAL benchmark directories, illustrated respectively in Tables 11 and 12. Table 11: Aggregate CocoaBench results represented in Every Eval Ever. Agent Scaffold Model Backbone Codex OpenClaw Cocoa Agent OpenClaw Claude Code Cocoa Agent

GPT-5.4 GPT-5.4 GPT-5.4 Claude Sonnet 4.6 Claude Sonnet 4.6 Gemini 3.1 pro

Accuracy (%)

Avg. time (s)

Avg. cost ($)

Total cost ($)

45.1 45.1 36.6 34.0 25.5 30.7

377.8 502.1 596.8 693.7 673.8 715.4

0.7 1.0 2.3 2.0 1.5 1.2

111.4 166.3 342.4 300.6 194.5 186.0

Table 12: Representative HAL results on CORE-Bench Hard represented in Every Eval Ever.

F.2

Agent Scaffold

Model Backbone

Accuracy (%)

Total cost ($)

Claude Code Claude Code Claude Code CORE-Agent CORE-Agent CORE-Agent

Claude Opus 4.1 Claude Sonnet 4.5 Claude Opus 4.5 Claude Sonnet 3.7 Claude Opus 4.1 Claude Opus 4.5

42.2 62.2 77.8 35.6 51.1 42.2

331.8 68.3 87.2 73.0 412.4 169.0

Case 2

This appendix gives the implementation details for Case Study 2 (Section 7.2). Records for the perplexity comparison in Table 3 were obtained from two sources: lm-eval-harness logs ingested via the automated lm_eval converter (Section 4, App. C.3), and GPTQ-style evaluation scripts contributed as manual records. The lm_eval converter preserves metric keys verbatim from harness output — word_perplexity and byte_perplexity are stored as distinct evaluation_name values in the MetricConfig block (Section 3.2), ensuring records with different normalization conventions are never silently aggregated (Section 3.1). GPTQ-style records were contributed with metric_name set to reflect token-level normalization. The corresponding records are available in the Every Eval Ever datastore under the WikiText benchmark directory. 35

F.3

Case 3

This appendix gives the implementation details for Case Study 3 (Section 7.3). It expands the four reproducibility patterns summarized in the main text—non-comparable examples, serving artifacts, stochastic disagreement, and residual answer differences—by describing how official and local records are aligned, how the heatmap values are computed, and the evidence supporting each diagnosis. Code used to produce these experiments is available on Anonymous Github. We compare HELM-released records, which we call the official side, to local reproductions for Pythia-6.9B, Vicuna-7B v1.3, and Falcon-7B on fourteen single-turn benchmarks, i.e., evaluations where each instance consists of a single prompt-response pair rather than multi-turn evaluations containing multiple prompts and responses within an instance or agentic tool-use tasks. Both sides are converted to Every Eval Ever and aligned by sample_hash, which the HELM converter computes as a SHA-256 hash of the rendered prompt concatenated with the first correct reference. Figure 4 summarizes the 42 (model, benchmark) comparisons. For each cell with a successful join, the displayed value is a micro-average over aligned (instance, core metric) score pairs: the number whose official. A micro average is equivalent to the local score difference divided by the total number of score pairs compared. The local score difference is at most 1e-9. Thirty-nine cells align successfully under this metric; the three failed joins are all Entity-Matching. The heatmap is a metric-level audit calculated after content-based alignment. Its unit of comparison is a scored (instance, metric) pair, so agreement implies that the two recorded scores are numerically equal up to the tolerance rather than that the text outputs are identical. Every Eval Ever provides the data needed for this comparison—sample hashes, per-instance outputs, per-metric scores, and generation parameters. F.3.1

Entity-Matching

Entity-Matching is a non-comparability failure. Official and local records each contain 1,000 examples from the Abt–Buy dataset [47], and HELM positional identifiers overlap 1,000/1,000, but the sample_hash sets have zero overlap. EntityMatchingScenario constructs a candidate table with pd.merge, then applies fixed-seed sampling over that ordered table and assigns ids positionally. A pandas row-order change makes the same input files, seed, and HELM recipe select different product pairs. End-to-end HELM reproduction confirms the split: the audit environment (pandas 2.3.3, numpy 2.2.6) matches the local scenario_state.json, while the HELM v0.3.0-era environment (pandas 2.0.3, numpy 1.23.5) matches the official one; cross-environment prompt-reference overlap, measured by sample_hash, is 0/1000. Thus, an id-based join would align different prompts, while the Every Eval Ever content join correctly reports join_failed. F.3.2

SyntheticReasoning-Natural

SyntheticReasoning-Natural is a synthetic-reasoning benchmark in natural language based on RuleTaker-style facts and rules [53, 24]. In our results, the SyntheticReasoning-Natural × Pythia6.9B is the largest discrepancy among cells that joined successfully. The sample_hash join succeeds, so the official and local records compare the same prompt-reference pairs. The divergence is in the scored outputs: official records contain empty output.raw completions and score 0.0 under the set-match metrics, while local records contain non-empty completions and recover non-zero scores. Lower-level official artifacts show that the backend returned generated text, but each completion began with the newline stop sequence and was trimmed to the empty string before scoring. The HELM recipe itself requests deterministic decoding: temperature=0.0, num_outputs=1, max_tokens=20, and stop_sequences=[“\n”]. Lower-level official artifacts show that the model produced outputs, but each began with the newline stop sequence and was trimmed before scoring. The audit trail indicates that HELM v0.3.0 routed the official Pythia call through Together.ai when model_deployment was unset, whereas the local run used Hugging Face transformers. Thus, Every Eval Ever exposes the symptom, while the historical backend must be recovered from HELM runtime artifacts or source code. 36

F.3.3

WikiFact

WikiFact is a factual-knowledge probing scenario, related to LAMA-style cloze probes for relational knowledge [53, 71]. WikiFact has similar cell-level score agreement for all models: 0.922 for Pythia, 0.920 for Vicuna, and 0.927 for Falcon. Its recipe uses temperature=1.0, num_outputs=5, and max_tokens=8, so independent executions can differ. A simple independent Bernoulli model for binary score-pair outcomes explains why agreement can stabilize below one. Because the heatmap micro-averages over aligned (instance, core metric) score pairs, let i index a binary WikiFact score pair. Let Xi and Yi be the binary scores from two independent executions of the same stochastic recipe on score pair i, and let pi be the probability that one full num_outputs=5 execution scores 1 on that prompt-metric pair. For a fixed score pair, agreement has two disjoint cases: Pr[Xi = Yi ] = | {z } agreement on prompt i

p2i |{z}

Xi =1, Yi =1 both score 1

+ (1 − pi )2 . | {z } Xi =0, Yi =0 both score 0

Averaging uniformly over score-pairs, with p̄ = Ei [pi ], gives Ei [p2i + (1 − pi )2 ] = | {z } heterogeneous expected agreement

p̄2 + (1 − p̄)2 + {z } |

homogeneous Bernoulli agreement at average rate

2 Vari (pi ) | {z }

.

extra agreement from prompt-dependent difficulty

The homogeneous Bernoulli agreement is therefore a lower bound in the symmetric @5-versus-@5 idealization, since the variance term is nonnegative. Intuitively, easy score pairs tend to agree by both succeeding, while hard score pairs tend to agree by both failing. The actual WikiFact artifacts are slightly asymmetric: the official HELM scenario_state.json stores one completion per request state even though the adapter records num_outputs=5, and the official exact_match and exact_match@5 scores are pointwise identical. The local audit artifacts store five completions, so the observed comparison is better modeled as a official-@1 versus local-@5 stochastic score-pair comparison. The observed WikiFact values are consistent with this stochastic recipe, with any residual plausibly due to estimator uncertainty or backend drift. Every Eval Ever makes this diagnosis checkable by preserving decoding parameters, per-instance scores, and raw outputs. F.3.4

Residual disagreements and scope

Several other cells fall below perfect agreement without exhibiting non-comparability, empty completions, or stochastic decoding. Spot checks show aligned instances and mostly matching answers; the residual differences are genuine answer changes concentrated in Pythia-6.9B. A plausible lead is serving-stack sensitivity, but the evidence cannot separate model strength from checkpoint, quantization, precision, tokenizer, or generation-kernel differences. F.4

Case 4

In Case Study 4, we conduct an Item Response Theory (IRT) meta-analysis of instance-level evaluation data collected and stored in Every Eval Ever. IRT models estimate latent parameters for dataset items (i.e., instances/examples) and subjects (i.e., AI models), and have been used in prior evaluation practices [73, 72] leaderboards [45, 82], meta-evaluations [79] and applications such as curriculum learning [79, 62, 74], often those uses were limited by the amount of available data [35, 36]. The one-parameter logistic (1PL) IRT model estimates item difficulty and subject ability, while more complex IRT models include more item-level parameters such as discriminability and feasibility [79]. IRT models the probability of subject j labeling item i correctly (zij = 1) as a function of subject j’s latent ability and item i’s latent difficulty. Parameters are learned via optimization from a dataset of graded (i.e., correct or incorrect) responses from subjects for a set of items: 1 1 + e−(θj −bi )

(1)

log p(Zij = zij |θj , bi )

(2)

p(zij = 1|θj , bi ) = log L =

J X I X j=1 i=1

37

Instance-level data collection is an expensive prerequisite and thus often a bottleneck for IRT research in NLP. For the case study, we selected three datasets currently available in Every Eval Ever with instance-level evaluations. GPQA Diamond [77] includes responses for 198 items from 69 subjects; Wordle Arena [68] includes responses for 63 items from 46 subjects; JudgeBench [88] includes responses for 350 items from 55 subjects. We extract the is_correct value for each instance-level record in a dataset to construct the response matrix Z. For example, Z JudgeBench has 55 rows (subjects) and 350 columns (items). We fit a 1PL model for each dataset using the py-irt package version 0.7.1 [50]. py-irt implements IRT model fitting via variational inference and can scale to large evaluation datasets via GPU-scaled training. Specifically, the joint posterior distribution p(Θ, B|Z) is approximated by a variational distribution q(Θ, B), and latent variables are learned by minimizing the KL-Divergence between q(Θ, B) and p(Θ, B|Z) [50].

38