One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation

One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation

arXiv:2608.29420v1 [cs.LG] 29 Aug 2026

Louis Yiven Zhu∗ University of Oxford yiven.zhu@oii.ox.ac.uk

Abstract Frontier-model leaderboards now rank systems based on economic benchmarks, tests of how well models carry out professional tasks from software engineering to banking workflows, and those rankings inform what organisations buy, what regulators scrutinise, and expectations of how work will change. Whether such benchmarks measure a capability distinct from general test-taking or re-express the one axis along which every benchmark rises as models improve is a question of construct validity that has not yet been studied. We test it on a hash-pinned leaderboard snapshot of 421 model configurations across twelve benchmarks, four of them economic, treating benchmarks as items and models as respondents in a latent-variable model with four hypotheses and their thresholds fixed before analysis. A single factor explains 74.5% of common variance and tracks model release date (R2 = 0.505), so the leading axis of capability is substantially a time trend; where prior work controls for scale, compute adds little once date is removed. Removing the date trend lowers that share by 14.9 percentage points and by 24.1 when the analysis is repeated with one row per base model. Under the dimensionality rule fixed in advance the economic benchmarks form no distinct factor, yet a leave-one-benchmark-out test with factors re-estimated inside every fold shows that a multi-factor representation predicts held-out economic scores better than a single general-capability index (pooled ∆MSE 0.037, 95% bootstrap interval [0.019, 0.055]). Economic benchmarks therefore add incremental predictive information to a largely date-driven general factor, and the evidence does not support treating them as a distinct latent capability. Leaderboards remain a sound guide to overall progress, but most of the gap between models released months apart is calendar, so a small gap between contemporaneous models should be date-adjusted before it is read as a capability difference; for benchmark builders, we give a two-test protocol for showing that a new suite measures more.

1

Introduction

Public leaderboards have become the dominant instrument for comparing and procuring frontier language models, and they increasingly inform policy, so a recent shift in what they are based on raises a measurement question with direct economic stakes. Alongside academic knowledge tests such as GPQA (Graduate-Level Google-Proof Q&A) [Rein et al., 2024] and Humanity’s Last Exam (HLE) [Phan et al., 2025], leaderboards now carry economic benchmarks such as GDPval [Patwardhan et al., 2026], the τ -Bench family [Yao et al., 2025, Barres et al., 2026], and terminal or agentic coding suites [Merrill et al., 2026], all built to proxy paid professional work and multi-step tool use, one of them (τ 3 -Banking) in banking customer support, a policy-compliance setting that evaluation suites ∗ ORCID 0009-0001-5579-0340.

Preprint.

(a) Benchmarks as items, models as respondents economic (4)

release date

others (8)

(b) One capability, or many?

(c) Two pre-specified tests

release date

release date

g

g

− date

e

project out the date trend, re-fit: does e survive? (H1–H3)

?

not run on earlier models 548 → coverage rule → 421 × 12

a single date-driven general factor

. . . plus a distinct economic factor

hold out each economic column; predict it from the rest (H4)

Figure 1: Study design. (a) Benchmarks as items (columns, coloured by capability block, economic in red) and model configurations as respondents (rows, ordered by release date); a coverage rule fixed in advance reduces 548 configurations to a 421 × 12 matrix. (b) The two candidate structures, a single date-driven general factor g, or g together with a distinct economic factor e. (c) The two pre-specified tests, date adjustment followed by re-fitting (H1–H3) and leave-one-benchmark-out prediction (H4); grids are stylised. cover thinly. Whether these benchmarks deserve a separate column depends on what they measure. If they capture a capability distinct from general test-taking, they carry information that a single “intelligence” score cannot; if they merely re-express one dominant axis of model progress, a separate column overstates what the leaderboard knows. This is a question of construct validity, the degree to which a measurement captures the construct it claims to [Cronbach and Meehl, 1955, Messick, 1998]. Although machine learning has been urged to hold its benchmarks to that psychometric standard [Raji et al., 2021, Jacobs and Wallach, 2021, Bowman and Dahl, 2021], recent audits find construct validity systematically under-examined in AI evaluation [Bean et al., 2025, Kearns, 2026]. We frame a snapshot of frontier models scored across twelve benchmarks as a psychometric problem, with benchmarks as items, models as respondents, and scores as the responses from which latent structure is inferred, and answer it under a pre-specified design (Figure 1). Four research questions organise the analysis, each mapping to one pre-specified hypothesis (Section 4.3); the first three concern structure (Task 1, unsupervised) and the fourth prediction (Task 2, supervised). RQ1 (dominance) How many latent capabilities does the battery measure, and how dominant is the leading one? RQ2 (temporal confound) Is the leading axis a release-date artefact, and how much survives date adjustment? RQ3 (structural distinctiveness) After date adjustment, do the economic benchmarks load on a factor of their own? RQ4 (predictive validity) Does a multi-factor representation predict held-out economic scores better than a single general index? Contributions. The paper makes four contributions. First, it subjects the economic column of a frontier leaderboard to a construct-validity test for the first time. Second, it isolates model release date as a first-order confound and removes it before reading any factor structure, whereupon the dominant capability axis proves substantially a temporal artefact, and where prior work controls for scale, the calendar carries what scale carries across generations. Third, it introduces a leave-onebenchmark-out predictive-validity test, since a benchmark that adds nothing to held-out prediction adds nothing a practitioner can use. Fourth, it delivers the analysis under a protocol fixed in advance with hash-pinned inputs, documenting every deviation and reporting the two failed tests as failed.

2

Related work

Two strands of the construct-validity programme bear on this paper. The first treats LLM scores as psychometric data, following the case for measuring artificial systems with the instruments built 2

for measuring minds [Hernández-Orallo, 2017] and the century-old observation that cognitive tests yield a positive manifold and a general factor [Spearman, 1904]. Ilić and Gignac [2024] recover that manifold across 591 models on twelve tests, with a mean inter-test correlation of 0.73 and a general factor correlating with parameter count at about 0.6; Burnell et al. [2023], applying factor analysis to 29 LLMs across 27 cognitive tasks, resolve capability instead into three factors. Ruan et al. [2024] then put the low-dimensional structure to work, using principal components of a score matrix as a capability space in which performance becomes predictable, including agentic performance forecast from non-agentic benchmarks [see also Zhou et al., 2025]. Closest to our design, Kearns [2026] makes the validity question quantitative on the Open LLM Leaderboard, where a plain factor analysis fits a single factor explaining 72% of communal variance that tracks parameter count, and residualising every score on a fitted scaling law lowers the mean inter-item correlation from 0.64 to 0.48; that residualisation is the template for our date adjustment. Across all of these studies scale is the variable to control; on a frontier snapshot we find that release date takes that role and compute adds little once date is removed, which suggests the confound has moved with the field. The second strand audits benchmarks as artefacts. Calls to repair benchmarking predate the frontier era [Bowman and Dahl, 2021, Kiela et al., 2021], and holistic evaluation frameworks [Liang et al., 2023], reproducibility guidance [Biderman et al., 2024], and proposals for a full evaluation science [Weidinger et al., 2025] have since sharpened them. Reviewing 445 benchmarks, Bean et al. [2025] find that 53.4% present any evidence for the construct validity of their benchmark and 16.0% report uncertainty estimates or statistical tests when comparing models, a gap Miller [2024] addresses with error bars and Reuel et al. [2024] with a quality checklist. On the temporal side, Akhtar et al. [2026] find that nearly half of sixty benchmarks are saturated and that saturation rises with benchmark age, which makes release date a first-order confound whenever capabilities are compared across a moving frontier, and Reuel et al. [2025] map an uneven division of evaluation labour between developers and third parties. None of this work isolates the economic column, and none pairs a structural test with an out-of-sample predictive one.

3

Data and design

Source and provenance. The primary data set is a snapshot of the Artificial Analysis model leaderboard [Artificial Analysis, 2026], captured on 6 July 2026 directly from the public leaderboard page and pinned by its SHA-256 hash (Appendix N). It contains 548 model configurations with per-benchmark scores, release dates, and provider metadata. A secondary source, the Epoch AI Notable AI Models data set [Epoch AI, 2026], supplies training-compute figures for a subset of models and serves only a scale robustness check. Both snapshots are pinned, so every number here is reproducible from fixed inputs. Inclusion rule and analysis grids. To avoid selecting benchmarks or models on the very outcomes under study, we fixed the coverage rule in advance. A benchmark is retained when at least sixty models carry a score, and a model when it carries at least eight of the thirteen sufficiently covered benchmarks. Both floors were set as indicative values in the analysis plan and confirmed unchanged at the coverage audit; that confirmation used coverage counts alone, never a score, a correlation, or a factor solution. Appendix G shows what each protects. Under this rule thirteen of fourteen candidates pass; we drop APEX-Agents [Vidgen et al., 2026] for sparsity and demote MMMU-Pro [Yue et al., 2025] to a sensitivity-only role, leaving twelve primary benchmarks and 421 model configurations, four of them economic benchmarks proxying paid professional or agentic work and eight spanning academic knowledge, scientific coding [Tian et al., 2024, Zhu et al., 2025], long-context retrieval, and instruction following [Pyatkin et al., 2025] (Appendix F). The 421 rows are configurations, since one base model can appear under several reasoning or effort settings; the plan deduplicates to one row per base model as robustness check R1, and we keep the configuration level as primary because it is the more conservative choice. Because the economic benchmarks are scored on far fewer models than the academic core, we use two grids. The complete-case grid of 96 models scored on all twelve benchmarks is the only one on which the economic block is complete, and it carries every hypothesis. The dense grid of 409 models scored on the nine near-universal benchmarks serves the corroborating clustering (Appendix K), where sample size matters more than economic completeness. This trade-off is the central datadesign decision, and the 96-model grid is the set of frontier configurations on which the economic benchmarks have actually been run, so it is representative of that population by construction. 3

Preprocessing and scope. Since the benchmarks are reported on incommensurable scales, mostly accuracy fractions alongside one hallucination-penalised score and GDPval’s Elo rating (Appendix F), we standardise each benchmark to zero mean and unit variance before any analysis, using training-fold statistics only in the supervised task. Missingness is structural, so we use complete-case analysis and impute nothing. On the complete-case grid the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy is 0.933, well above the 0.8 threshold [Kaiser, 1974], and Bartlett’s test of sphericity is decisively rejected [Bartlett, 1950], so factor analysis is appropriate. The battery is also highly collinear before any modelling, with a mean off-diagonal Spearman correlation of ρ = 0.79, every benchmark pair positive, and the economic benchmarks correlating with the academic core almost as strongly as with one another (Appendix H); the manifold is thus at least as strong here as the 0.73 Ilić and Gignac [2024] report on a broader population. We fixed the scope of the work to internal validity, meaning the correlational and predictive structure of aggregate scores on one cross-section; item-level responses, external economic impact such as adoption or revenue, and saturation dynamics lie outside it.

4

Methods

4.1

Task 1: latent structure

Each structural question has its own quantity, and Appendix D states each method formally with the reasoning behind it. The first factor’s share of common variance measures dominance (RQ1), the change in that share after date adjustment measures the temporal confound (RQ2), and the dateadjusted loading pattern tests distinctiveness (RQ3). To fix dimensionality, we use Horn’s parallel analysis [Horn, 1965], retaining factors whose eigenvalue exceeds the 95th percentile of eigenvalues from random data of the same shape, with the scree criterion [Cattell, 1966] as a cross-check, since on collinear data it is more conservative than the Kaiser rule [Kaiser, 1960] and than information criteria (Appendix D). We prefer factor analysis to the principal-component description of Ruan et al. [2024] because components mix common and unique variance, whereas the common-factor model isolates shared capability, the distinction a construct-validity question turns on. The model is accordingly a maximum-likelihood exploratory factor analysis (EFA) with an oblique oblimin rotation [Jennrich and Sampson, 1966, Thurstone, 1947], chosen because the date-adjusted factors are strongly correlated (Section 5.3) and an orthogonal rotation would spread shared variance across artificially independent axes. Every factor is oriented so that higher means more capable. To separate a general capability from a shared time trend, we regress each standardised benchmark on release date, and on a subsample also on training compute, and re-fit the factor analysis on the residuals; we call this date adjustment. It removes from the benchmark covariance the rank-one component aligned with date (Appendix D), so the resulting change in the first factor’s variance share measures how much of the apparent general factor is a date artefact, the analogue of the scaling-law residualisation Kearns [2026] reports. Clustering corroborates the structure without assuming a factor model (Appendix K). 4.2

Task 2: predictive validity

Task 2 recasts the distinctiveness question as a forecast, asking whether a model’s score on a held-out economic benchmark follows from its behaviour on the others, a sharper test than the in-sample capability space of Ruan et al. [2024] because the target column is withheld from estimation entirely. To that end, the leave-one-benchmark-out (LOBO) protocol takes each economic benchmark as the target in turn and predicts it from representations of the remaining eleven on the n = 96 completecase grid, so that success demands transfer across benchmarks. A five-rung predictor ladder makes the comparison concrete. Rung (i) uses release timing and scale only, rung (ii) a single mean-score general index, rung (iii) the first factor only, rung (iv) the k factors (the three-factor solution of Section 4.3), and rung (v) the k factors plus model covariates. The contrast of interest sets rung (iv) against the single-index rung (ii). So that the ladder comparison is tied to no single model class, we fit each rung with four learners spanning the linear-regularised and tree families [Hastie et al., 2009], ridge [Hoerl and Kennard, 1970], elastic net [Zou and Hastie, 2005], random forest [Breiman, 2001], and gradient boosting [Friedman, 2001]. Hyperparameters are tuned by grid search in an inner five-fold cross-validation nested inside an outer five-fold split that supplies the reported out-of-fold error, the nesting that 4

removes selection bias from the estimate [Varma and Simon, 2006] (grid in Appendix I). Because the factors are estimated quantities, we re-estimate them inside the loop, fitting the factor analysis on each training fold alone, projecting the held-out fold through the training-fold Thurstone factor-score weights, and re-orienting by the training-fold mean index; a naive fit-on-all-data pipeline would silently open that leakage path (full procedure in Appendix B). Two error metrics then serve two purposes. Root mean squared error (RMSE) on the standardised target is the descriptive metric since it stays on the target’s own scale. The test statistic is ∆MSE = MSE(single index) − MSE(k factors) per economic target, positive when the multi-factor representation wins, with a 95% interval from a paired bootstrap of the pooled out-of-fold errors [Efron and Tibshirani, 1993] with B = 2000. 4.3

Pre-specified hypotheses

Before analysis, the design fixed four hypotheses with quantitative thresholds set with reference to the figures Kearns [2026] reports for the Open LLM Leaderboard. H1 (dominance, RQ1) holds that the first factor carries more than 50% of common variance. H2 (date artefact, RQ2) has two parts, (i) that the dominant factor’s release-date R2 is at least 0.30 and the strongest of the factors, and (ii) that date adjustment lowers the first factor’s share by at least fifteen percentage points. H3 (structural distinctiveness, RQ3) holds that after date adjustment parallel analysis retains a factor on which the four economic benchmarks load at least 0.40 and exceed their cross-loadings, which is the discriminant criterion of Campbell and Fiske [1959] applied to a benchmark battery. H4 (predictive validity, RQ4) holds that the multi-factor representation’s ∆MSE interval excludes zero and a majority of the four economic targets individually favour it. We report every outcome, including the two that fail. Two factor counts are used in what follows. The primary count is k = 1, returned by parallel analysis, the Kaiser rule, and the scree elbow, and it governs H1 to H3. The second is k = 3, an exploratory over-extraction that no criterion selects, which supplies the loading pattern of Section 5.3 and the k-factor predictor of Task 2 and is a documented deviation from the plan (Appendix J); the predictive conclusion does not hinge on it, since sweeping rung (iv) over k ∈ {2, 3, 4, 5} beats the single index at every count (Appendix L).

5

Results

5.1

Dominance of a single factor (H1)

Every dimensionality diagnostic points to a dominant capability axis. The first principal component alone accounts for 79.4% of total score variance, and parallel analysis retains a single factor, since only the first observed eigenvalue of 9.53 exceeds its random-data 95th percentile of 1.79 (Appendix Figure 5). In the maximum-likelihood factor analysis the first factor holds 74.5% of common variance, above the 50% dominance threshold and close to the 72% Kearns [2026] reports on a different leaderboard, and more concentrated than the three-factor structure Burnell et al. [2023] recover. H1 is therefore supported, which answers RQ1 in full, and because rank-based and logittransformed variants only raise the share (Section 5.5), the Pearson-based headline is the conservative one. 5.2

A date-driven artefact (H2)

Having established a dominant factor, we ask whether it reflects stable capability or the passage of time. The factor’s scores track model release date with a logistic-fit R2 of 0.505 (0.477 under an ordinary-least-squares fit, Appendix L), the strongest of the three factors, so H2(i) is supported and the leading axis of “capability” is substantially a time trend. Adjusting every benchmark for release date and re-fitting then lowers the first factor’s common-variance share from 74.5% to 59.6% (Figure 2c), a drop of 14.9 percentage points, which means date alone accounts for about a fifth of the general factor. That point estimate falls just short of the pre-specified fifteen-point threshold, with a bootstrap 95% interval of [−5.3, +32.7], so on the primary grid the drop is indistinguishable both from zero and from the threshold. We report H2(ii) as not met on the point estimate without moving the line after seeing the result. 5

b Date-adjusted

GDPval (Elo) 0.87 0.12 -0.03

0.84 0.13 -0.04

Terminal-Bench v2.1 1.05 -0.12 0.04

1.01 -0.12 0.03

τ³-Banking 0.57 0.06 0.36

0.54 0.04 0.38

τ²-Bench 0.71 0.25 -0.13

0.50 0.31 -0.13

GPQA Diamond -0.02 0.97 0.03

0.01 0.95 0.04

HLE 0.35 0.30 0.43

0.32 0.28 0.49

AA-Omniscience 0.56 0.18 0.20

0.69 0.12 0.14

SciCode 0.11 0.69 0.23

0.26 0.61 0.20

CritPt 0.04 0.03 0.95

0.02 -0.00 0.99

Terminal-Bench Hard 0.81 0.11 0.10

0.82 0.10 0.10

AA-LCR 0.27 0.73 -0.09

0.34 0.67 -0.12

IFBench -0.05 0.90 -0.01

-0.13 0.86 0.02

F1

Economic

F2

Academic

F3

F1

F2

F3

Scientific-coding

Factor-1 share: 74.5% → 59.6%

1.00 0.75 0.50 0.25 0.00

−0.25 −0.50 −0.75 −1.00

Long-context

Common-variance share (%)

Raw scores

c

loading

a

75

raw date-adjusted

74 60

50 35 22

25

3 5 0

F1

F2

F3

Instruction-following

Figure 2: Factor loadings from the three-factor oblimin EFA. (a) Raw scores; the economic benchmarks (red) already concentrate on F1. (b) After date adjustment they still concentrate on F1, with a largest cross-loading of 0.38. (c) The first factor’s common-variance share falls from 74.5% to 59.6%, a 14.9-point drop.

Whether H2(ii) passes depends on the unit of analysis. Under the plan’s base-model unit the drop is 24.1 points and clears the threshold; under the configuration-level primary it is 14.9 points and fails; on the 58-model compute-known subsample date alone gives 16.5 points and clears it. Two of the three specifications therefore pass. The alternative, promoting the deduplicated grid to primary, would pass the threshold but would turn the plan’s robustness unit into the headline on fewer models (89 against 96), so we keep the configuration-level specification as primary and the confirmation of a temporal confound rests on the subsample and the deduplicated grid. RQ2 is therefore answered in part, since the leading axis is substantially a release-date artefact while the size of the correction is established on two of the three specifications. Adding training compute does not deepen the correction, since on the compute-known subsample the joint date-plus-compute drop is smaller, at 9.3 points, plausibly because later models are also larger and the two predictors split their shared variance. This inverts the usual control, since prior work treats scale as the variable to model or remove [Kearns, 2026, Ilić and Gignac, 2024, Ruan et al., 2024], whereas on a frontier snapshot the calendar carries what scale carries across generations. 5.3

Economic distinctiveness (H3)

With the temporal confound removed, we test whether the economic benchmarks form a distinct factor, a pre-specified test with two conditions, that parallel analysis retain a factor and that its loadings concentrate discriminantly on the economic block. On the date-adjusted data parallel analysis retains a single factor, which leaves no block to separate; the separation appears only once three factors are extracted, which the dimensionality rule does not endorse. Holding H3 to the plan as H2(ii) was held to its 15-point line, we report H3 as not supported as specified, which answers RQ3 negatively under the plan. Under the three-factor over-extraction the economic block does separate, and we report the pattern as exploratory. All four economic benchmarks load at least 0.40 on a single factor, F1, with GDPval at 0.84, Terminal-Bench v2.1 at 1.01, τ 3 -Banking at 0.54, and τ 2 -Bench at 0.50, against a largest economic cross-loading of 0.38 (Figure 2b, tabulated in Appendix E); the mean absolute F1 loading is 0.72 for the economic benchmarks and 0.32 for the rest. Because the three factors are strongly correlated (0.67 between F1 and F3 and 0.72 between F2 and F3) the separation reads as a secondary refinement of one dominant axis. Bootstrap intervals on the economic loadings are wide (0.10 to 0.78 for τ 2 -Bench and 0.44 to 0.99 for GDPval), excluding zero but overlapping one another, so the pattern is suggestive and loosely estimated. As discussed in Section 5.4, out-of-sample transfer is 6

Table 1: Predictor-ladder cross-validated errors (economic block) and the H4 bootstrap test. Left, train and test RMSE on standardised targets, out-of-fold R2 , and the best learner per rung (RF random forest, EN elastic net). Right, ∆MSE = MSE(mean index) − MSE(k factors), B = 2000; every interval except τ 2 -Bench’s excludes zero. Rung

Learner Train

(i) timing / scale RF (ii) mean index ridge (iii) first factor EN (iv) k factors ridge (v) k factors + cov. ridge

b

0.95

∆MSE 95% CI

GDPval (Elo) +0.026 [+0.010, +0.044] Terminal-Bench v2.1 +0.028 [+0.001, +0.056] τ 3 -Banking +0.056 [+0.008, +0.103] τ 2 -Bench +0.038 [−0.011, +0.086] Pooled economic +0.037 [+0.019, +0.055]

H4: k-factor gain over single index → favours k-factor

0.44

reasoning log-params

τ³-Banking

F2 (academic)

τ²-Bench

params-missing open-weights

POOLED economic 0 0.05 0.10 dMSE (mean-index − k-factor)

0.0 0.2 0.4 mean |SHAP| (GDPval, GBM)

(i)

tim (ii i ) ng m /s ea ca ( (v iii l ) ) fi n in e k rs de fa t ct (iv f x or ) ac k t s + fa or co ct o va rs ria te s

What predicts economic score

F3 (reasoning)

Terminal-Bench v2.1

0.0

c

F1 (economic/agentic)

GDPval (Elo) 0.43

0.5

0.74

all 12 benchmarks economic block 1.0

Target

R2

0.521 0.741 0.459 0.463 0.474 0.771 0.933 0.950 0.110 0.410 0.433 0.808 0.391 0.438 0.800

k factors beat single index

0.47

LOBO test RMSE (z-units)

a

Test

Figure 3: Predictive validity under the LOBO protocol. (a) Test RMSE by rung, economic block (red) and all twelve targets (grey). (b) Bootstrap ∆MSE (single index minus k factors) with 95% intervals; positive favours the multi-factor representation. (c) SHAP (SHapley Additive exPlanations) importances for the rung-(iv) gradient-boosting model of GDPval. a stronger dimensionality argument than an in-sample retention rule, and we let the predictive test carry the claim the structural test cannot. 5.4

Predictive validity (H4)

The final test asks whether the economic factor improves prediction. Across the predictor ladder a single mean-score index is already a strong baseline, at economic-block test RMSE 0.474 and R2 0.771; the k-factor representation improves on it to RMSE 0.433 and R2 0.808, and adding covariates gives no further gain (Table 1, Figure 3a). The first factor alone predicts poorly at RMSE 0.950, which locates the economic signal in the secondary factors. Bootstrapping ∆MSE over the pooled out-of-fold errors gives a pooled improvement of +0.037 with a 95% interval of [+0.019, +0.055], and because that resample includes near-duplicate settings of the same base model, we re-run it on the deduplicated grid of 89 distinct models, where the improvement is +0.038 with [+0.020, +0.056] (Appendix L). Three of the four economic benchmarks improve individually, GDPval by +0.026, Terminal-Bench v2.1 by +0.028, and τ 3 -Banking by +0.056, while τ 2 -Bench’s interval includes zero (Table 1, Figure 3b). H4 is therefore supported on the rule fixed in advance (Section 4.3), which answers RQ4 affirmatively. In direction this agrees with Ruan et al. [2024], and in size it qualifies them. An improvement of 0.037 sits on top of a single index that already explains 77% of economic-block variance, so the gain is incremental, yet it survives a validation that rewards transfer and penalises memorisation. The one exception, τ 2 -Bench, carries the widest interval, and its null reflects limited coverage as much as absent signal. The 96 configurations mix reasoning and effort variants of base models across providers; two design features address their comparability, since one operator scores every model under a single harness [Artificial Analysis, 2026] and the deduplicated re-run reproduces 7

the gain. On GDPval, SHAP attribution [Lundberg and Lee, 2017] on the rung-(iv) boosting learner ranks the agentic factor F1 first (Figure 3c, Appendix M). 5.5

Robustness

The headline structure is stable under perturbations to the unit of analysis, the correlation measure, the adjustment covariates, and the factor count (Appendices J and L). Collapsing the 421 configurations to one row per base model raises the first-factor share to 90.2% and the date-adjustment drop to 24.1 points, rank-based and logit-transformed correlations raise the share to 94.5% and 91.8%, so H1 is no metric artefact [Schaeffer et al., 2023], the predictor ladder’s ordering holds on all twelve targets, and the H4 verdict depends on neither the regularisation strength nor the factor count.

6

Discussion

Taken together, the two tasks answer the four research questions. On RQ1 the battery is governed by one dominant axis holding 74.5% of common variance; on RQ2 that axis is substantially a release-date artefact, as removing the date trend takes away about a fifth of it (74.5% to 59.6%). On RQ3 they form no separate capability in the pre-specified structure and separate only under an over-extraction, yet on RQ4 the secondary economic factor still predicts held-out economic scores better than a single general index on three of four benchmarks. Economic validity is therefore limited, in the precise sense that a date-driven progress factor accounts for most of the economic signal. For leaderboard use, a single intelligence index captures most of what distinguishes models, yet the residual economic signal is the part most relevant to claims about professional value, so reporting economic benchmarks separately is justified. Small gaps between contemporaneous models still call for caution. A leaderboard ranking models released months apart is ranking them largely on release date, an external factor set by the pace at which new models and capabilities arrive. A date-adjusted score or a comparison restricted to one release window would show how much of a gap is capability and how much is calendar. In both respects the findings reinforce the call for psychometric scrutiny of AI benchmarks [Bean et al., 2025, Kearns, 2026] and the evidence that benchmarks lose discriminating power as they age [Akhtar et al., 2026]. Contributing a two-test protocol for benchmark builders. Beyond how leaderboards are read, the result points to how new economic benchmarks should be validated, since a benchmark’s value for distinguishing contemporaneous models depends on the variance it adds once the shared date trend is removed. A builder who wants to show that a new suite measures something distinct can apply the two tests directly with what a leaderboard already publishes, per-model scores on the new benchmark and the existing battery and their release dates. Step one, the structural test, regresses every benchmark on release date, fits an oblique factor model to the residuals, and checks whether the new benchmark’s loading separates from the general factor; step two, the predictive test, holds the new benchmark out, predicts it from a k-factor representation of the others with factors re-estimated in every fold, and checks whether that beats a single mean-score index with a bootstrap interval excluding zero. A benchmark that passes both adds a construct, and one that passes neither is re-measuring the progress trend. An operator can run both tests on every benchmark it adds and publish the two statistics beside the score, as the code accompanying this paper does from a single score matrix (https://github.com/louisyzhu/frontier-ai-economic-validity). Limitations. Four limitations bound the claims. First, the economic factor comes from an exploratory over-extraction, so it stands as a hypothesis for a confirmatory factor model on an independent snapshot. Second, the analysis is observational, so it establishes how strongly release date correlates with the general factor and not what drives that association. Third, coverage is uneven, so the economic results carry wider uncertainty; fourth, the results describe one leaderboard on one date and will shift as new models arrive, a risk the hash-pinned snapshot mitigates. Conclusion. Economic benchmarks measure neither one capability nor many. Most of what they record is a single general factor that is largely a release-date trend, and what remains is a small but real economic component that predicts held-out economic scores. Read with the date in view they add information to a general index without yet constituting a distinct capability. Every test and threshold 8

here was fixed before analysis, every verdict is reported as it fell, and every number is reproducible from pinned inputs, which is the standard we propose for evaluation claims of this kind.

Acknowledgements I thank Marcos Barreto for comments and guidance on this work.

References Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, et al. When AI benchmarks plateau: A systematic study of benchmark saturation. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. arXiv:2602.16763. Artificial Analysis. Artificial Analysis LLM Leaderboard and Intelligence Index. https:// artificialanalysis.ai, 2026. Snapshot accessed 6 July 2026. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -Bench: Evaluating conversational agents in a dual-control environment. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. arXiv:2506.07982. Maurice S. Bartlett. Tests of significance in factor analysis. British Journal of Psychology (Statistical Section), 3:77–85, 1950. Andrew M. Bean, Ryan O. Kearns, Angelika Romanou, Franziska S. Hafner, Harry Mayne, et al. Measuring what matters: Construct validity in large language model benchmarks. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track, 2025. arXiv:2511.04703. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, et al. Lessons from the trenches on reproducible evaluation of language models, 2024. arXiv:2405.14782. Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4843–4855, 2021. Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. Ryan Burnell, Han Hao, Andrew R. A. Conway, and José Hernández-Orallo. Revealing the structure of language model capabilities, 2023. arXiv:2306.10062. Donald T. Campbell and Donald W. Fiske. Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin, 56(2):81–105, 1959. Raymond B. Cattell. The scree test for the number of factors. Multivariate Behavioral Research, 1 (2):245–276, 1966. Lee J. Cronbach and Paul E. Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302, 1955. Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993. Epoch AI. Notable AI Models (data set, CC-BY 4.0). notable-ai-models, 2026.

https://epoch.ai/data/

Jerome H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29(5):1189–1232, 2001. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009. José Hernández-Orallo. The Measure of All Minds: Evaluating Natural and Artificial Intelligence. Cambridge University Press, 2017. 9

Arthur E. Hoerl and Robert W. Kennard. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. John L. Horn. A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2): 179–185, 1965. David Ilić and Gilles E. Gignac. Evidence of interrelated cognitive-like capabilities in large language models: indications of artificial general intelligence or achievement? Intelligence, 106:101858, 2024. Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 375–385, 2021. Robert I. Jennrich and Paul F. Sampson. Rotation for simple loadings. Psychometrika, 31(3):313–323, 1966. Henry F. Kaiser. The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1):141–151, 1960. Henry F. Kaiser. An index of factorial simplicity. Psychometrika, 39(1):31–36, 1974. Ryan O. Kearns. Quantifying construct validity in large language model evaluations, 2026. MSc thesis, Oxford Internet Institute, University of Oxford, Trinity Term 2025. arXiv:2602.15532. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, et al. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4110–4124, 2021. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. arXiv:2211.09110. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), pages 4765–4774, 2017. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. arXiv:2601.11868. Samuel Messick. Test validity: a matter of consequence. Social Indicators Research, 45:35–44, 1998. Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations, 2024. arXiv:2411.00640. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.04374. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, et al. Humanity’s Last Exam, 2025. arXiv:2501.14249. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025. arXiv:2507.02833. Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Datasets and Benchmarks Track, 2021. 10

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling (COLM), 2024. arXiv:2311.12022. Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. BetterBench: Assessing AI benchmarks, uncovering issues, and establishing best practices. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024. Anka Reuel, Avijit Ghosh, Jenny Chim, Andrew Tran, et al. Who evaluates AI’s social impacts? Mapping coverage and gaps in first and third party evaluations, 2025. Preprint, Evaluating Evaluations (EvalEval) Coalition. arXiv:2511.05613. Peter J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65, 1987. Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. Gideon Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6(2):461–464, 1978. Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor Barres. τ -Knowledge: Evaluating conversational agents over unstructured knowledge, 2026. arXiv:2603.04370. Charles Spearman. “General Intelligence,” Objectively Determined and Measured. The American Journal of Psychology, 15(2):201–292, 1904. Louis L. Thurstone. Multiple-Factor Analysis. University of Chicago Press, 1947. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, et al. SciCode: A research coding benchmark curated by scientists. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024. arXiv:2407.13168. Robert Tibshirani, Guenther Walther, and Trevor Hastie. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B, 63(2):411–423, 2001. Sudhir Varma and Richard Simon. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7:91, 2006. Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, et al. APEX-Agents, 2026. arXiv:2601.14242. Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, et al. Toward an evaluation science for generative AI systems, 2025. arXiv:2503.05336. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations (ICLR), 2025. arXiv:2406.12045. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, et al. MMMUPro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15134–15186, 2025. arXiv:2409.02813. Lexin Zhou, Lorenzo Pacchiardi, Fernando Martínez-Plumed, et al. General scales unlock AI evaluation with explanatory and predictive power, 2025. arXiv:2503.06378. Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, et al. Probing the critical point (CritPt) of AI reasoning: A frontier physics research benchmark, 2025. arXiv:2509.26574. 11

Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B, 67(2):301–320, 2005.

12

Appendix The appendix collects the material that supports the main text without interrupting its argument. Each section opens with the main-text section it supports, and every figure and table is placed with the section that discusses it.

A

Abbreviations and notation Table 2: Abbreviations and symbols, in order of first appearance in the main text. GPQA HLE LLM RQ H1–H4 KMO EFA PCA ML R2 ρ k Λ, Φ, Ψ

B

Graduate-Level Google-Proof Q&A Humanity’s Last Exam large language model research question pre-specified hypotheses Kaiser–Meyer–Olkin sampling adequacy exploratory factor analysis principal-component analysis maximum likelihood coefficient of determination Spearman rank correlation number of retained factors loadings, factor covariance, uniquenesses

LOBO RMSE MSE ∆MSE CI OLS BIC SHAP RF, EN GBM R1, C1–C7 AA W

leave-one-benchmark-out root mean squared error mean squared error MSE(single index) − MSE(k factors) confidence interval ordinary least squares Bayesian information criterion SHapley Additive exPlanations random forest, elastic net gradient-boosting machine planned and delivered robustness checks Artificial Analysis (leaderboard operator) Thurstone factor-score weights

Leave-one-benchmark-out procedure

Supports Section 4.2, which describes the protocol in words. Figure 4 states the complete procedure, including the two safeguards on which the predictive test depends. First, the factor model is re-fitted inside every outer training fold, so the held-out models never influence the loadings used to score them. Second, hyperparameters are tuned in an inner loop nested within the outer split, so the reported out-of-fold error is never used for selection [Varma and Simon, 2006]. Procedure Leave-one-benchmark-out predictive validity with nested cross-validation Input: standardised score matrix Z (models × benchmarks), economic targets T , learners L, rungs R. for each target t ∈ T do partition models into 5 outer folds for each outer fold (train, test) do fit standardisation and EFA on train only; form Thurstone weights W = R−1 Λ project test models through W ; orient factors by train mean index for each rung r ∈ R and learner ℓ ∈ L do tune ℓ by inner 5-fold grid search on train; predict test, store out-of-fold error select best learner per rung by pooled out-of-fold RMSE compute ∆MSE = MSE(rung ii) − MSE(rung iv); bootstrap B = 2000 for 95% CI Output: per-target and pooled ∆MSE with confidence intervals (H4).

Figure 4: The leave-one-benchmark-out procedure with leakage-safe factor projection and nested tuning.

C

Positionality, directionality, and threats to validity

Supports Section 4.3. A factor solution is identified only up to sign, so we state the direction of every factor explicitly. We orient each factor to correlate positively with the model’s mean standardised score. “Higher” then always means “more capable”, and every loading in Table 3 reads in the same direction. The choice is not cosmetic, since an un-oriented solution could report the same economic factor with flipped signs and invite the opposite interpretation. The direction of the date adjustment is equally deliberate. We regress scores on release date and analyse the residuals, which measures how much benchmark co-movement survives once the release date is known. We do not run the reverse regression, as we make no claim that the general factor causes the calendar. Finally, we fix the sign of 13

the predictive contrast in advance, so that ∆MSE is the single-index error minus the k-factor error, a positive value favours the richer representation, and H4 requires this positive interval to exclude zero. Two threats to validity follow from the study’s vantage point. First, we observe only the models that vendors chose to submit to a public leaderboard. The sample is therefore a convenience sample of frontier systems, and the claims generalise to models on this leaderboard alone. Second, the economic benchmarks are newer and sparser, so any economic-specific structure is estimated from fewer observations and is more sensitive to the particular models that carry those scores. The wide bootstrap intervals in Table 1 quantify that limitation.

D

Mathematical basis of the two tasks

Supports Sections 4.1 and 4.2. Each subsection states a component of the method, derives the quantity the main text reports, and gives the reason the component was chosen over its nearest alternative. The common-factor model and the dominance statistic. Let z ∈ Rp be the vector of p perbenchmark standardised scores for a model. The common-factor model writes z = Λf + ε,

Σ = ΛΦΛ⊤ + Ψ,

with Λ ∈ Rp×k the loading matrix, f ∈ Rk the latent factors with Cov(f ) = Φ, and ε benchmarkspecific uniquenesses with diagonal Cov(ε) = Ψ and Cov(f , ε) = 0. The decomposition of Σ into a common part ΛΦΛ⊤ and a unique part Ψ is the reason we use factor analysis and not principal components for a validity question. A principal component maximises total variance and so absorbs benchmark-specific noise along with shared capability, whereas the common-factor model attributes to the factors only the variance that benchmarks share [Thurstone, 1947]. Parameters are estimated by maximum likelihood under multivariate normality. The dominance statistic for H1 is computed on the unrotated solution, where Φ = I and each factor’s contribution to common variance is its sum of squared loadings, the m-th diagonal entry of Λ⊤ Λ, Pp 2 (Λ⊤ Λ)11 j=1 λj1 s1 = Pk . = Pp 2 tr(Λ⊤ Λ) m=1 j=1 λjm On the complete-case grid the three factors carry sums of squared loadings 7.79, 2.34, and 0.32, so s1 = 7.79/10.45 = 74.5%. This is a factor-level share, distinct from a single benchmark’s P communality m λ2jm , a diagonal entry of ΛΛ⊤ . Dimensionality by parallel analysis. Let ℓ1 ≥ · · · ≥ ℓp be the eigenvalues of the sample correla(0.95) tion matrix R, and let ℓm be the 95th percentile of the m-th eigenvalue over correlation matrices of independent standard-normal data of the same dimensions n × p. Parallel analysis retains factor (0.95) m when ℓm > ℓm [Horn, 1965]. The rule asks whether a factor explains more than chance sampling structure would, which is why it is more conservative than the Kaiser rule ℓm > 1 [Kaiser, 1960] on strongly collinear data, where sampling eigenvalues of later factors fall below one and the Kaiser rule over-retains. Information criteria such as the BIC [Schwarz, 1978] compare likelihoods of nested models and reward any factor that improves fit, which on a near-rank-one correlation matrix means retaining factors that carry very little common variance; the BIC’s preference for k = 4 here (0.95) (Section 4.3) is an instance. On our grid ℓ1 = 9.53 against ℓ1 = 1.79, and ℓ2 = 0.91 against (0.95) ℓ2 = 1.57, so one factor is retained (Figure 5). Oblique rotation. The maximum-likelihood solution is identified only up to rotation. Direct oblimin [Jennrich and Sampson, 1966] chooses the rotation that minimises p p p X X γX 2 X 2 Q(Λ) = λ2jm λ2jl − λjm λjl , p j=1 j=1 j=1 m<l

with γ = 0 giving the quartimin criterion we use, while allowing Φ ̸= I. An orthogonal rotation such as varimax forces Φ = I; on data with a strong general factor this distributes shared variance across artificially independent axes, so that no single factor can carry the general variance and the dominance we set out to measure is obscured by construction. The date-adjusted factors correlate at 0.67 and 0.72 (Section 5.3), which confirms that the oblique choice was the right one. 14

Scree plot (PCA)

a

observed eigenvalues 95th percentile (random)

79.4 8

60

Eigenvalue

Variance explained (%)

80

Horn parallel analysis retains k = 1

b

40

20 7.6

6

4

2 3.5

2.8

1.8

3

4

5

0

0 1

2

1

Principal component

2

3

4

5

Factor

Figure 5: Dimensionality of the battery. (a) The scree plot of principal-component variance shares drops sharply after the first component (79.4%). (b) Horn parallel analysis retains a single factor, as only the first observed eigenvalue (9.53) exceeds its random-data 95th percentile (1.79), while the second (0.91) already falls below (1.57). Factor scores and the leakage-safe projection. Regression (Thurstone) factor scores are the linear predictor f̂ = W ⊤ z that minimises E∥f − W ⊤ z∥2 . Setting the derivative to zero gives W = Σ−1 Cov(z, f ) = R−1 ΛΦ, where R is the sample correlation matrix, Λ the pattern loadings, and Φ the inter-factor correlation matrix of the sorted solution, so that W = R−1 Λ when the factors are orthogonal [Thurstone, 1947]. The structural factor scores behind H2 and the model clustering use the oblique form. The prediction task uses the orthogonal-form weights W = R−1 Λ built from the pattern loadings; there W is estimated on the training fold only and applied to the held-out fold, the leakage-safe projection of Section 4.2. The two score sets are an invertible reparameterisation of one another, f̂oblique = f̂pattern Φ, so an unpenalised regression on either gives identical fitted values; the ridge learners of Task 2 are not invariant to the reparameterisation, and the reported ladder uses the pattern form throughout. The point is that W is a function of the data. If it were estimated on all n models, every held-out model’s score would have contributed to the weights used to predict it, and the predictive comparison in Task 2 would be contaminated by exactly the information it is meant to withhold. Date adjustment as removal of a rank-one component. Each standardised benchmark zj is regressed on release date d (and, in a subsample, additionally on log training compute), zj = β0j + β1j d + rj , and the factor model is re-fitted on the residuals rj . Writing b = (β11 , . . . , β1p )⊤ , the covariance of the residual vector is Σr = Σ − σd2 bb⊤ , so the adjustment removes from the benchmark covariance exactly the rank-one component that lies along the date vector. If the general factor were nothing but the shared time trend, Λ1 would be proportional to b and the first factor’s share would collapse after adjustment; if it were unrelated to date, the share would be unchanged. The observed change, 74.5% → 59.6%, is the H2(ii) statistic and measures where the battery sits between those two cases. The design mirrors the scaling-law residualisation of Kearns [2026], with release date in place of parameter count as the nuisance variable. Nested leave-one-benchmark-out estimator. For an economic target t, the out-of-fold error of rung r is n 1X (−κ(i))
2 MSEr = zit − ẑit , n i=1 (−κ(i))

where ẑit is the prediction for model i from a learner trained on the outer folds excluding the fold κ(i) that contains i, with hyperparameters chosen by an inner five-fold grid search on those 15

training folds alone. Selecting hyperparameters on the same folds that supply the reported error biases the estimate optimistically, and the bias grows with the size of the search space [Varma and Simon, 2006]; nesting removes it at the cost of a five-fold increase in computation. The predictive-validity statistic contrasts the single-index rung against the k-factor rung, ∆MSE = MSEii − MSEiv . Paired bootstrap for ∆MSE. Let eiii and eiv i be model i’s out-of-fold squared errors under the two rungs. A 95% interval for ∆MSE is obtained by resampling models with replacement, recomputing P ii 1 iv i (ei − ei ) on each of B = 2000 resamples, and taking the 2.5th and 97.5th percentiles n [Efron and Tibshirani, 1993]. Resampling the paired differences, and not the two error vectors separately, matters because the two rungs are evaluated on the same models and their errors are strongly correlated; the paired design removes the between-model variance that the two rungs share, so the interval reflects only the variance of the contrast. H4 is supported when this interval excludes zero. Because the resampling unit is a model, near-duplicate configurations of one base model would be treated as independent draws; the deduplicated re-run in Appendix L, whose resampling unit is a distinct base model, is the check on that assumption.

E

Date-adjusted three-factor loadings

Supports Section 5.3. Table 3 tabulates the loadings shown as a heatmap in Figure 2b. F1 is an agentic and work-realistic factor, F2 an academic-knowledge factor, and F3 a physics and hard-reasoning factor; all four economic benchmarks load highest on F1. Table 3: Date-adjusted three-factor oblimin loadings, economic benchmarks in bold. Benchmark GDPval (Elo) Terminal-Bench v2.1 τ 3 -Banking τ 2 -Bench GPQA Diamond HLE AA-Omniscience SciCode CritPt Terminal-Bench Hard AA-LCR IFBench

F

F1

F2

F3

0.84 1.01 0.54 0.50 0.01 0.32 0.69 0.26 0.02 0.82 0.34 −0.13

0.13 −0.12 0.04 0.31 0.95 0.28 0.12 0.61 −0.00 0.10 0.67 0.86

−0.04 0.03 0.38 −0.13 0.04 0.49 0.14 0.20 0.99 0.10 −0.12 0.02

Benchmark battery

Supports Section 3. Table 4 describes every candidate benchmark, the construct it targets, the type of score on which the leaderboard reports it, its coverage before and after the inclusion rule, and the role the rule assigned, with the source paper for each listed beneath. The economic benchmarks differ from the academic ones in kind, not only in coverage. They score multi-step, tool-using, or humanpreference-judged task completion on work-like tasks, whereas the academic benchmarks score single-response accuracy on knowledge or reasoning items. Three of the twelve (AA-Omniscience, AA-LCR, and Terminal-Bench Hard) are constructed or subset by the leaderboard operator itself; all twelve are run by that operator under a single harness [Artificial Analysis, 2026]. Sources. GDPval [Patwardhan et al., 2026]; Terminal-Bench v2.1 [Merrill et al., 2026]; τ 3 -Banking [Shi et al., 2026]; τ 2 -Bench [Barres et al., 2026]; GPQA Diamond [Rein et al., 2024]; HLE [Phan et al., 2025]; SciCode [Tian et al., 2024]; CritPt [Zhu et al., 2025]; IFBench [Pyatkin et al., 2025]; AA-Omniscience, AA-LCR, and Terminal-Bench Hard are constructed or subset by the leaderboard operator [Artificial Analysis, 2026]. GDPval’s Elo rating comes from pairwise, head-to-head judgements of model outputs on economically valuable tasks, so a higher rating means more often preferred on a scale with no fixed maximum. The τ -Bench family [Yao et al., 2025, Barres et al., 2026] scores an agent on whether the final database state and required confirmations match a policy-compliant goal after a multi-turn conversation with a 16

Table 4: Candidate benchmarks by capability block, with the construct each targets, the type of score reported, the number of models scored on the raw 548-configuration snapshot and among the 421 retained, and the role assigned by the inclusion rule. Twelve benchmarks are primary; MMMU-Pro is sensitivity-only for sparse and multimodal coverage; APEX-Agents falls below the sixty-model floor. Higher is better throughout; sources are listed below the table. Benchmark

Construct measured

Economic GDPval (Elo) Paid professional task quality Terminal-Bench v2.1 Terminal and agentic task completion τ 3 -Banking Multi-step banking workflows τ 2 -Bench Tool-use agentic tasks APEX-Agents Agentic task completion Academic GPQA Diamond Graduate-level science QA HLE Frontier academic knowledge AA-Omniscience Broad knowledge, hallucination-penalised MMMU-Pro Multimodal understanding Scientific coding SciCode Research-level scientific coding CritPt Frontier physics problem solving Terminal-Bench Hard Hard terminal coding tasks Long-context / instruction AA-LCR Long-context reasoning IFBench Instruction following

Score type

Raw Retained Role

pairwise-judged Elo pass rate task success task success task success

117 121 112 428 26

112 121 112 413 –

primary primary primary primary dropped (sparse)

accuracy accuracy penalised score accuracy

513 509 418 204

421 421 418 –

primary primary primary sensitivity only

pass rate accuracy pass rate

507 422 421

421 primary 419 primary 413 primary

accuracy accuracy

441 437

420 primary 417 primary

simulated user; τ 3 -Banking is the banking-knowledge domain of its third generation, in which agents must complete policy-compliant state changes over a large document base. The leaderboard labels its Terminal-Bench column v2.1; the cited paper presents the suite’s 2.0 release. MMMU-Pro [Yue et al., 2025] is demoted to a sensitivity-only role and APEX-Agents [Vidgen et al., 2026] is dropped for sparsity (Appendix G).

G

Full coverage audit

Supports the inclusion rule of Section 3. Table 4 records every candidate benchmark’s coverage on the raw snapshot and the decision applied to it, and Figure 6 shows that coverage together with the pairwise overlaps that set the effective sample size for every correlation in the study. The sixtymodel floor is the smallest count at which the sparsest retained benchmark still overlaps every other benchmark on more than one hundred models (Figure 6b), which keeps each pairwise correlation estimable with a standard error below about 0.1.

H

Exploratory data analysis

Supports Section 3. We examined coverage and missingness, correlation structure, and temporal patterns before fitting any model (Figure 7). All standardisation and model fitting uses the training portion of each split only, so these views characterise the data without leaking test information into the models. The battery is highly collinear before any modelling. The mean off-diagonal Spearman correlation is ρ = 0.79, every benchmark pair is positive, and the economic benchmarks correlate with the academic core almost as strongly as with one another (Figure 7a). A dominant general factor would produce this near-block structure, and the structure analysis must explain it, since a strong positive manifold is equally consistent with one underlying capability and with several capabilities that improve together over time. Scores rise steeply and jointly with release date (Figure 7b), so release date confounds any claim that two benchmarks measure the same thing; the saturation literature finds saturation rising with benchmark age [Akhtar et al., 2026], and we therefore treat date as a first-class nuisance variable. Most benchmarks are bounded accuracies in [0, 1], and several are strongly right-skewed, with CritPt at Fisher skewness 3.8, HLE at 1.5, and τ 3 -Banking at 0.9, their scores piling up near the floor 17

(a) Benchmark coverage by capability block inclusion floor (60) 117

GDPval (Elo) Terminal-Bench v2.1

121

τ³-Banking

112 428

τ²-Bench 26

APEX-Agents GPQA Diamond

513

HLE

509 418

AA-Omniscience 204

MMMU-Pro

507

SciCode CritPt

422

Terminal-Bench Hard

421

AA-LCR

441

IFBench

437 0

100

200

300

400

500

600

Models with a score (of 548 configurations in the raw snapshot)

Economic Academic

Scientific-coding Long-context

Instruction-following

112

108

104

106

112

112

111

66

112

111

105

111

108

Terminal-Bench v2.1

108

121

109

115

121

121

120

73

121

120

114

120

117

τ³-Banking

104

109

112

106

112

112

111

71

112

111

105

111

108

τ²-Bench

106

115

106

413

413

413

411

192

413

412

411

413

413

GPQA Diamond

112

121

112

413

421

421

418

197

421

419

413

420

417

HLE

112

121

112

413

421

421

418

197

421

419

413

420

417

AA-Omniscience

111

120

111

411

418

418

418

194

418

418

411

418

414

MMMU-Pro

66

73

71

192

197

197

194

197

197

195

192

196

194

SciCode

112

121

112

413

421

421

418

197

421

419

413

420

417

CritPt

111

120

111

412

419

419

418

195

419

419

412

419

415

Terminal-Bench Hard

105

114

105

411

413

413

411

192

413

412

413

413

413

AA-LCR

111

120

111

413

420

420

418

196

420

419

413

420

416

IFBench

108

117

108

413

417

417

414

194

417

415

413

416

417

Terminal-Bench v2.1

τ³-Banking

τ²-Bench

GPQA Diamond

HLE

AA-Omniscience

MMMU-Pro

SciCode

CritPt

Terminal-Bench Hard

AA-LCR

IFBench

400

350

300

250

# models

GDPval (Elo)

GDPval (Elo)

(b) Pairwise model overlap (both benchmarks scored)

200

150

100

Figure 6: Coverage audit behind the inclusion rule. (a) Number of models carrying a score on each candidate benchmark on the raw 548-configuration snapshot, coloured by capability block, with the sixty-model inclusion floor marked. Three economic benchmarks (Terminal-Bench v2.1, GDPval, and τ 3 -Banking) are scored on only 112 to 121 models, while τ 2 -Bench and the academic core are each scored on four hundred or more; APEX-Agents, with 26 models, falls below the floor. (b) Pairwise counts of models scored on both benchmarks in a pair, which set the effective sample size for every correlation in the study and are the evidence for the two-grid design.

18

a

Pairwise rank correlation 1.00 GDPval (Elo)

1

.94

.90

.81

.88

.86

.81

.87

.82

.94

.85

.68

Terminal-Bench v2.1

.94

1

.86

.77

.86

.84

.83

.87

.82

.94

.83

.64

τ³-Banking

.90

.86

1

.73

.86

.84

.77

.86

.81

.91

.84

.66

τ²-Bench

.81

.77

.73

1

.81

.80

.63

.72

.71

.79

.71

.73

GPQA Diamond

.88

.86

.86

.81

1

.96

.82

.92

.90

.92

.90

.80

HLE

.86

.84

.84

.80

.96

1

.80

.89

.91

.89

.88

.77

AA-Omniscience

.81

.83

.77

.63

.82

.80

1

.86

.76

.86

.84

.62

SciCode

.87

.87

.86

.72

.92

.89

.86

1

.84

.91

.89

.71

CritPt

.82

.82

.81

.71

.90

.91

.76

.84

1

.85

.79

.66

Terminal-Bench Hard

.94

.94

.91

.79

.92

.89

.86

.91

.85

1

.89

.71

AA-LCR

.85

.83

.84

.71

.90

.88

.84

.89

.79

.89

1

.73

IFBench

.68

.64

.66

.73

.80

.77

.62

.71

.66

.71

.73

1

0.75

0.50

0.00

−0.25

−0.50

−0.75

Capability rises with release date

b Score (min-max scaled)

1.0 0.8

GPQA Diamond GDPval (Elo) SciCode

0.6 0.4 0.2 0.0

ch

CR

IFB en

AA -L

Cr itP t ch Ha rd l-B en

Te rm in a

HL E m ni sc ie nc e Sc iC od e AA -O

GD Pv Te rm al (E in al lo -B ) en ch v2 .1 τ³ -B an kin g τ² -B en GP ch QA Di am on d

−1.00

c Coverage by benchmark 112 121 112

GDPval (Elo) Terminal-Bench v2.1 τ³-Banking τ²-Bench GPQA Diamond HLE AA-Omniscience SciCode CritPt Terminal-Bench Hard AA-LCR IFBench

413 421 421 418 421 419 413 420 417 0

-11 -05 -11 -05 24 25 25 26 0 0 0 0 2 2 2 2

200

400

Models scored (of 421)

Release date Economic Academic

Scientific-coding Long-context

Instruction-following

Figure 7: Exploratory data analysis of the twelve-benchmark battery on the complete-case grid (n = 96). (a) Pairwise Spearman rank correlations, with benchmark labels coloured by capability block; almost every pair is strongly positive. (b) Per-benchmark scores (min–max scaled) against model release date, showing capability rising steeply with time. (c) Coverage per benchmark across the 421 retained models, with the four economic benchmarks (red) scored on far fewer models than the academic core. 19

Spearman ρ

0.25

a

Benchmark score distributions (n = 421 analysis set; per-benchmark complete cases) GDPval (Elo)

Terminal-Bench v2.1

γ=-0.15

0

1000

GPQA Diamond

0.0

HLE

AA-Omniscience

0.2

0.1

0.2

0.3

0.4

−50

0.2

0.4

0.6

0.3

0

AA-LCR

γ=0.67

0.0

0.2

γ=0.08

0.0

0.5

1.0

SciCode

γ=0.55

Terminal-Bench Hard γ=3.80

0.0

0.1

γ=1.46

0.0

CritPt

τ²-Bench γ=0.92

0.00 0.25 0.50 0.75

γ=-0.75

0.25 0.50 0.75

τ³-Banking

γ=0.36

γ=-0.57

0.0

0.2

0.4

0.6

IFBench γ=-0.11

γ=0.29

0.00 0.25 0.50 0.75

0.2

0.4

0.6

0.8

Skewness by benchmark

b GPQA Diamond

γ=1

SciCode GDPval (Elo) AA-LCR τ²-Bench IFBench Terminal-Bench v2.1 AA-Omniscience Terminal-Bench Hard τ³-Banking HLE CritPt −1

0

1

2

3

4

Skewness (γ)

(a) Raw score histograms by taxonomy block; γ = Fisher skewness. (b) CritPt, HLE, and τ³-Banking are strongly right-skewed (γ > 0.9), motivating the logit robustness check on bounded accuracies.

Figure 8: Univariate benchmark score distributions. (a) Raw score histograms for the twelve benchmarks, coloured by capability block; γ is the Fisher skewness. (b) Skewness ordered across benchmarks. Most benchmarks are bounded accuracies in [0, 1], and several are strongly right-skewed (CritPt γ = 3.8, HLE γ = 1.5, τ 3 -Banking γ = 0.9), with scores concentrated near the floor. GDPval (Elo) and AA-Omniscience are on unbounded scales and are closer to symmetric.

20

(Figure 8). Two benchmarks are unbounded, GDPval on an Elo scale and AA-Omniscience on a penalised score, and both sit closer to symmetric. Bounded and skewed scales can manufacture or hide structure through the metric alone [Schaeffer et al., 2023], which motivates the logit-transform check in Appendix L; that check confirms that the first-factor share is no artefact of the raw scale. The low-scoring tails hold weak models with genuinely low scores, so we retain every row and handle the boundedness through the transform, trimming no outliers. The dimensionality evidence behind H1, the scree plot and the parallel analysis, is Figure 5 in Appendix D; the first principal component accounts for 79.4% of total score variance, and only the first observed eigenvalue exceeds its random-data 95th percentile.

I

Hyperparameter grid and selections

Supports Section 4.2 and the robustness claim of Section 5.5 that the H4 verdict does not hinge on a single tuning choice. The inner grid searched ridge α ∈ {0.03, 0.1, 0.3, 1, 3, 10, 30}, elastic-net α ∈ {0.03, 0.1, 0.3, 1} with ℓ1 ratio in {0.2, 0.5, 0.8}, random-forest depth in {4, none} with 300 trees, and gradient-boosting depth in {2, 3} with learning rate in {0.05, 0.1}. The ridge and elastic-net grids span three orders of magnitude of regularisation strength on a log scale, the standard practice for a regularisation path [Hastie et al., 2009], and the tree grids bracket the shallow trees appropriate for n = 96 and k + 4 predictors. Table 5 reports the modal selection per rung; the modal ridge α = 0.03 sits at the weakly regularised end of the grid, and the H4 verdict is unchanged across the sweep of k in Appendix L. The regression learners and cross-validation use scikit-learn [Pedregosa et al., 2011]; the factor analysis uses the factor_analyzer package, and SHAP its reference implementation [Lundberg and Lee, 2017]. Table 5: Modal hyperparameters selected by the inner five-fold grid search, reported as the plurality choice across all twelve leave-one-benchmark-out targets with the agreement count. The H4 verdict rests on fold-averaged out-of-fold errors, not on any single setting.

J

Rung

Learner

Modal hyperparameters

(i) timing / scale (ii) mean index (iii) first factor (iv) k factors (v) k factors + covariates

random forest ridge elastic net ridge ridge

max_depth=4, min_samples_leaf=3, n_estimators=300 alpha=0.03 alpha=0.03, l1_ratio=0.2 alpha=0.03 alpha=1

Agreement 8/12 6/12 5/12 6/12 6/12

Deviations from the pre-specified design

Supports the reproducibility claims throughout. A study that fixes its analysis plan in advance owes the reader an explicit account of where the delivery departed from it. Five departures matter. First, the plan’s primary unit was one row per base model at its best default configuration. The analysis instead uses all 421 configurations; the deduplicated base-model dataset appears in Appendix L as planned check R1, and the configuration level serves as the primary. Second, the plan’s factor-count rule is parallel analysis, and it retains one factor. The three-factor solution that exhibits economic separation is a pre-declared exploratory over-extraction, and we report H3 as not supported under the plan in consequence. Third, the taxonomy places Terminal-Bench Hard in scientific coding, while the plan grouped it with Terminal-Bench v2.1 as economic. The date-adjusted loadings show Hard at 0.82 on the economic factor, so the assignment is genuinely contestable, and we disclose the split openly. Fourth, the delivered robustness checks depart from the planned suite. We label them distinctly below, and we do not renumber a different battery onto the planned names. Fifth, the plan set the two inclusion floors as indicative values to be confirmed at the coverage audit rather than as fixed constants; the audit confirmed both unchanged, and no score was used to choose them. The plan also targeted 180 to 220 base models, where coverage on the economic block delivered 96 complete cases and 89 deduplicated base models. Two further differences are presentational and are recorded for readers comparing the paper with the deposited plan. The plan organised the work under two questions, structure and prediction; the paper splits the structural question so that each hypothesis has its own research question, with no change to 21

any hypothesis or test. And the H2(ii) fifteen-point threshold was calibrated to a reading of Kearns [2026] in which a scale correction lowered a dominant factor’s share from 72% to 41%. That reading was mistaken, and Section 2 gives the corrected account; the threshold itself was fixed in advance and has not been moved.

K

Clustering of models and benchmarks

Supports Section 4.1. A factor model can impose structure the raw data do not support, so we sought an independent check, and clustering corroborates the low dimensionality without assuming a factor model at all. We cluster models in factor-score space with k-means, Gaussian mixtures, and agglomerative Ward linkage, selecting the cluster count by silhouette [Rousseeuw, 1987] and the gap statistic [Tibshirani et al., 2001]; three algorithms guard against a clustering that reflects one algorithm’s inductive bias. All three prefer k = 2 by silhouette, at 0.480 for k-means, 0.499 for agglomerative, and 0.467 for the Gaussian mixture (Figure 9b). The gap statistic increases monotonically in k and finds no interior optimum, so it is inconclusive on data with a strong general factor and we set it aside. The two clusters mark capability tiers along one axis of overall strength. A larger, earlier, lower-scoring group of 302 models, with median release 2025-09 and mean Intelligence Index 13.7, separates from a smaller, newer, higher-scoring group of 107 models, with median release 2026-03 and mean index 37.1. The split sorts models by how much capability they hold, along the same date-driven axis (Figure 9a). Two capability tiers

a

Validation selects k = 2

0.50

tier low (n=302) tier high (n=107)

1

Silhouette score

Factor 2 (academic)

2

b

0

−1

−2

k-means GMM agglomerative

0.45

0.40

0.35

0.30 −1

0

1

2

2

Factor 1 (agentic / economic)

3

4

5

6

k (clusters)

Figure 9: Model clustering on the dense grid (n = 409). (a) Models in the space of the first two factors, coloured by the two-cluster agglomerative solution. The factor axes are the dense-grid solution fitted largely without the economic columns, so they track overall capability, and the two clusters mark tiers of it. (b) Silhouette scores by k; all three algorithms agree on k = 2. Clustering the benchmarks separately by correlation distance 1 − ρ under average linkage places ten of twelve, and three of four economic benchmarks, in one block, with τ 2 -Bench and IFBench branching separately (Figure 10).

L

Robustness checks

Supports Section 5.5. We first report the one planned check that bears directly on the deduplication concern, then the additional checks actually delivered. Planned check R1 (configuration versus deduplicated). We collapse the 421 configurations to one row per base model, keeping the best default configuration as the plan specified. We operationalise it as the highest-intelligence-index row for each base model, after stripping reasoning, effort, and preview suffixes from the model name. Of the resulting base models, 89 carry the complete battery. 22

Benchmark clustering (hierarchical, average linkage) AA-LCR SciCode HLE GPQA Diamond CritPt Terminal-Bench Hard Terminal-Bench v2.1 GDPval (Elo) τ³-Banking AA-Omniscience τ²-Bench IFBench 0.00

0.05

0.10

0.15

0.20

0.25

0.30

Correlation distance (1 - rho)

Figure 10: Benchmark clustering by correlation distance (1 − ρ, average linkage). Ten of the twelve benchmarks, including three of four economic benchmarks, fall in one block, with τ 2 -Bench and IFBench branching separately. On this deduplicated grid the first-factor share rises to 90.2% and the date-adjustment drop rises to 24.1 points, and the four economic loadings remain above 0.40. Deduplication therefore strengthens H1 and H2(ii) and leaves the exploratory economic pattern intact, and the configuration-level analysis is the conservative choice for the structural headline. We also re-run the H4 predictive test on this deduplicated grid, where the resampling unit is a distinct base model. The pooled economic-block improvement of the k-factor representation over the single index is +0.038 with a 95% interval of [+0.020, +0.056]. It matches the configuration-level +0.037 and again excludes zero, so the predictive gain does not depend on treating near-duplicate configurations as exchangeable. Delivered checks C1–C7. (C1) A rank-based factor analysis raises the first-factor share to 94.5%, and (C2) a logit transform of the bounded accuracy benchmarks raises it to 91.8%, so H1 is not a Pearson artefact. (C3) On the 58-model compute-known subsample, the date-only adjustment drops the first-factor share by 16.5 points, and adding log-compute jointly gives a smaller 9.3-point drop, so compute does not deepen the temporal correction beyond date on the same models. (C4) The three clustering algorithms agree on k = 2. (C5) The predictor-ladder ordering holds on both the economic block and all twelve targets. (C6) The dominant-factor date fit is robust to the link function, since a logistic fit gives R2 = 0.505 against an ordinary-least-squares R2 = 0.477, and both clear 0.30 with the first factor ranked strongest, so H2(i) does not depend on that modelling choice. (C7) The k-factor predictive advantage does not depend on the exploratory factor count. Sweeping the rung-(iv) predictor over k ∈ {2, 3, 4, 5}, the improvement over the single index is positive at every count with a bootstrap interval excluding zero, reading +0.021 at k = 2, +0.037 at the k = 3 we use, +0.058 at the criterion’s k = 4, and +0.059 at k = 5, with pooled test R2 rising from 0.79 to 0.83 across the range. The gain increases with k, so the k = 3 we adopt is conservative for this test relative to the criterion’s k = 4. The planned checks not run are R3 (FIML and MICE missingness sensitivity), R4 (replication of the single-factor foil then correction), R5 (reasoning-effort variants as a second scale axis), and R6 (the taxonomy swap test). Each was deferred for space; R5 and R6 bear on the effort and taxonomy departures above and are the natural next step.

M

Explainability and error analysis

Supports Section 5.4. To interpret the predictive gain, we open the models fitted to the largest economic target. GDPval is predicted best by a k-factor ridge with out-of-fold R2 0.88 (Figure 11). 23

SHAP attribution on the gradient-boosting learner fitted at the same rung ranks the agentic factor F1 first, followed by the reasoning flag and the hard-reasoning factor F3, with scale and open-weights status contributing little (Figure 3c). The ordering is informative in the light of prior work, since Ruan et al. [2024] find agentic performance forecastable from a low-dimensional capability space, and Ilić and Gignac [2024] find parameter count correlated with the general factor at about 0.6; here, once the factors are in the model, parameter count and open-weights status add almost nothing, so the scale signal that prior work reports reaches GDPval through the factors and not alongside them. The predicted-versus-observed plot lies on the identity line, and the residuals show no systematic trend (Figure 11a,b). The largest residuals are interpretable outliers. They include a non-reasoning variant of a frontier model that outperforms its economic prediction, Grok 4.3 Non-reasoning at +1.14, and two models that under-perform theirs (Figure 11c). The pattern fits GDPval rewarding agentic behaviour that the other benchmarks capture only indirectly. Predicted vs observed 2

Residuals - no systematic trend

b

R2 = 0.88

1.0

Residual (obs - pred)

Predicted (out-of-fold, z)

a

1

0

−1

−2

0.5

0.0

−0.5

−1.0 −2

−1

0

1

2

−1

Observed GDPval (z)

c

0

1

2

Predicted (z)

Largest prediction errors (named) Grok 4.3 (Non-reasoning) Gemini 3.1 Pro Preview Llama 4 Maverick Gemma 3 27B Instruct Qwen3.6 27B (Non-reasoning Qwen3.7 Plus Gemma 3 12B Instruct Mistral Small 3.2 −1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

Residual (obs - pred, z)

Figure 11: Regression diagnostics for the GDPval k-factor ridge model, out-of-fold. (a) Predicted versus observed, on the identity line with R2 = 0.88. (b) Residuals versus predicted show no systematic trend. (c) The eight largest residuals, named. These are interpretable outliers, and none signals misspecification.

N

Data provenance and reproducibility

Supports Section 3 and the reproducibility claims throughout. The primary data set is the Artificial Analysis model leaderboard [Artificial Analysis, 2026], captured 2026-07-06 as a raw HTML snapshot and pinned by the SHA-256 hash recorded in the shipped manifest (aa_snapshot_manifest.json), 6f19f8f08befaaf14cb2d9952e379404c39f246b52d7ae4d1c2aa44795404de6. The Data API refused an unauthenticated request, so we captured the public page and stored the exact bytes. Every downstream number is therefore reproducible regardless of live-page drift [Biderman et al., 2024]. The secondary data set is Epoch AI Notable AI Models [Epoch AI, 2026], released under CC-BY 4.0 and used only for the scale robustness check. The analysis plan was fixed before the data were captured 24

and before any analysis was run, and is deposited at https://doi.org/10.17605/OSF.IO/VD34J; the deposit is retrospective, and the note there records the timeline, the plan’s checksum, and the limits of that evidence. Both snapshots, every result table, and the analysis notebook are released at https://github.com/louisyzhu/frontier-ai-economic-validity. The pipeline is a single notebook that runs top to bottom from the pinned snapshots with no manual step, no live network call and no path outside the repository, and all four learners run by default. It regenerates the figures that carry the results, Figures 2, 3, 7, 9 and 11, while the remaining figures and twelve result tables listed in the repository are shipped rather than recomputed within it; the notebook prints every value it reads, so each remains checkable against its own output. The SHAP attribution behind Figure 3c and the R1 deduplication with its H4 re-run verify themselves against the tables cited here and reproduce exactly. The prediction ladder is refitted on every run and written alongside the canonical table rather than over it, so no value reported here can be altered by re-execution. The notebook then recomputes the quantities this paper prints, the per-target and pooled economic ∆MSE and the five economicblock rows of Table 1, and compares them with the canonical values at a tolerance of 5 × 10−4 , half a unit at the third decimal and so at the precision reported here; it separately verifies that the best-learner ranking behind the table’s learner column is unchanged. Those quantities reproduce in every printed digit on both machines we have tested. Individual ladder cells are reported rather than enforced, because one rung is not platform-invariant. The maximum-likelihood solution has a weakly determined leading direction for this correlation structure, so the single-factor rung can settle on a slightly different point under a different linear-algebra backend, and gradient-boosted trees carry that difference most visibly because a split threshold either flips or does not. The largest deviation we observed was 7.2 × 10−4 , reaching the k-factor rung only through tree learners on a non-economic target, while the mean-index and k-factor ridge fits from which every reported effect and interval is built agreed to about 10−7 . The divergence is a property of the platform and of that rung, not of the results.

25

Related documents

• Zero knowledge verification for frontier AI training is possible#665796
• Zero knowledge verification for frontier AI training is possible#664296
• Understanding as an Explicit and Assessable Component of Frontier AI Safety Decisions#658913
• Understanding as an Explicit and Assessable Component of Frontier AI Safety Decisions#656777
• Contextual Multi-Objective Optimization: Rethinking Objectives in Frontier AI Systems#155333