Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence

[2608.14903] Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Artificial Intelligence arXiv:2608.14903 (cs) [Submitted on 14 Aug 2026] Title: Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence Authors: Fabricio F Costa View a PDF of the paper titled Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence, by Fabricio F Costa View PDF HTML (experimental) Abstract: Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records, and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50 percent task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95 percent CI 1.021 to 1.390), whereas a six-system MMLU to MMLU-Pro comparison appears shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80 percent power only for slope departures near 25 percent. Provenance is concentrated: 52 of 71 substantive quantitative events, or 73.2 percent, come from one measurement programme, and 76.1 percent are laboratory releases. A review of 56 methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes, and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measurement system with explicit joins, protocols, links, and source dependence, not merely a fitted curve or calendar date. Comments: 16 pages, 6 figures, 1 table. Evidence cutoff: 12 August 2026. Ancillary code and data included. Preprint; comments welcome Subjects: Artificial Intelligence (cs.AI) MSC classes: 68T01, 68T05 ACM classes: I.2.0; I.2.6 Cite as: arXiv:2608.14903 [cs.AI] (or arXiv:2608.14903v1 [cs.AI] for this version) https://doi.org/10.48550/arXiv.2608.14903 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Fabricio Costa [ view email ] [v1] Fri, 14 Aug 2026 21:28:14 UTC (319 KB) Full-text links: Access Paper: View a PDF of the paper titled Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence, by Fabricio F Costa View PDF HTML (experimental) TeX Source view license Ancillary-file links: Ancillary files ( details ) : DATA_AND_SOURCE_AUDIT_v4.md FIGURE_AND_PDF_QA_REPORT.txt MEASUREMENT_DIRECTIONS_RESEARCH_v4.md README.md README_ANCILLARY.txt model/analyse.py model/analysis_output.txt model/build_graph.py model/build_model.py model/data/benchmarks.csv model/data/bridges.csv model/data/compute_horizon.csv model/data/criteria.csv model/data/cutoff_release_check.csv model/data/data_dictionary.csv model/data/data_manifest.csv model/data/data_quality_report.csv model/data/dataset_inventory.csv model/data/direction_source_links.csv model/data/equating.csv model/data/events.csv model/data/graph.ttl model/data/join_intersections.csv model/data/labs.csv model/data/literature_sources.csv model/data/measurement_designs.csv model/data/measurement_directions.csv model/data/metr_th11.csv model/data/model_observability.csv model/data/models.csv model/data/pairwise_overlap.csv model/data/power.csv model/data/relations.csv model/data/source_audit.csv model/data/source_verification_log.csv model/data/sources.csv model/directions_fig.py model/equating.py model/equating_fig.py model/equating_output.txt model/equating_results.json model/figure_utils.py model/finalize_data_inventory.py model/graph_stats.txt model/inventory_output.txt model/lifecycle_fig.py model/measurement_directions.py model/measurement_directions_output.txt model/method_fig.py model/observability_fig.py model/power.py model/power_output.txt model/research_audit.py model/research_audit_output.txt model/synthesis_fig.py requirements.txt (51 additional files not shown) Current browse context: cs.AI < prev | next > new | recent | 2026-08 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• When a High Score Is an Illusion: Certifying Genuine versus Repackaged Forecasting Skill#1013822
• A Continuous-Time Ensemble Kalman-Bucy Smoother for Causal Inference and Model Discovery#1011086
• Can Integrated Machine Learning Models Improve Rainfall Forecasting? A Case Study from the Australian Tropics#443670
• wastewater-informed-covid-forecasting#367705
• Robustness of Deep Learning Models for PV Power Forecasting under NWP Forecast Errors: A Spatiotemporal and Physically Interpretable Analysis#366243