arXiv CSopen access

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

arXiv:2607.24573v1 [cs.AI] 27 Jul 2026

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports Jonas Schröder†

Jonas Schweisthal†

Oliver Müller

MCML & LMU Munich Munich, Germany [email protected]

MCML & LMU Munich Munich, Germany [email protected]

Paderborn University Paderborn, Germany [email protected]

Markus Weinmann

Stefan Feuerriegel

University of Cologne Cologne, Germany [email protected]

MCML & LMU Munich Munich, Germany [email protected]

Abstract

Keywords

Large language models (LLMs) increasingly support decisions about uncertain future events, yet evaluating their ability to forecast real-world outcomes remains difficult. In particular, existing benchmarks are typically static and retrospective, and therefore cannot test how information is synthesized by LLMs to predict future events under uncertainty. We introduce LLM-SoccerArena (https://llm-soccerarena.com), a prospective live benchmark that evaluates how well LLMs forecast real-world sports events before the outcomes are known. LLM-SoccerArena provides (1) a prospective live benchmark protocol, (2) a public open-source platform, and (3) a factorial benchmark design together with tournamentrelated questions (e.g., which team will win). LLM-SoccerArena automatically records timestamped, schema-validated forecasts of unresolved events, together with prompts, model versions, tool traces, and costs. The factorial design varies along four dimensions: (1) model version (e.g., GPT-5.5, Claude Opus 4.8); (2) information access (i.e., with or without web search); (3) prompting strategy (i.e., scoreline or outcome probabilities), and (4) forecast horizon (i.e., whether predictions are made at stage opening, 24 hours before kickoff, or 2 hours before kickoff). We demonstrate LLM-SoccerArena through a large-scale evaluation of the 2026 FIFA World Cup, in which seven LLMs generated forecasts for all 104 matches and 15 tournament-related questions. We provide a detailed analysis of model performance across information access, prompting strategy, and forecast horizon. As a result, LLM-SoccerArena provides new evidence about the forecasting performance of state-of-theart LLMs. For example, LLMs with web access outperform those without, but only by a small margin (i.e., a 0.023 improvement in Brier score). Overall, LLM-SoccerArena provides a flexible, opensource platform for prospective benchmarking of unresolved events. LLM-SoccerArena will be continuously updated, and can be directly applied to future national and international tournaments and league competitions.

large language models, forecasting, live benchmark, soccer, sports analytics

CCS Concepts • Computing methodologies → Machine learning; Natural language processing.

† Joint first authors.

Resources: The

LLM-SoccerArena benchmark infrastructure

Platform & live leaderboard (continuously updated) https://www.llm-soccerarena.com/ Dataset for the World Cup 2026 (CSV) https://github.com/jonas-srd/world_cup_LLM_rank/blob/main/data/ worldcup2026-full-prediction-dataset-2026-07-21.csv Code & reproducibility materials https://github.com/jonas-srd/world_cup_LLM_rank/tree/main

1

Introduction

Forecasting future events is important for decision-making in business, public policy, or personal life [16, 19, 32]. Recently, there has been a growing interest in whether large language models (LLMs) can provide such forecasts at scale [15, 20, 43]. However, forecasting real-world events is challenging for various reasons: relevant information changes over time, outcomes remain unresolved, and models must reason under uncertainty based on incomplete evidence [12, 18, 40]. As such, forecasting with LLMs implicitly tests whether they can synthesize evolving information into wellfounded probabilistic judgments about future events before the outcome is known. Existing benchmarks for LLM-based forecasting have important limitations. Most are static and inherently retrospective. This means they either evaluate questions for which outcomes are already known [40, 43] or which can be solved through reasoning from available information [2, 21, 38]. In contrast, forecasting is fundamentally prospective. It requires a model to predict an outcome that is still unresolved using only the information available at the time of prediction. Hence, static benchmarks cannot reliably assess real-world forecasting ability because their questions and outcomes may already appear in model training data, causing performance to reflect memorization [5] or benchmark familiarity rather than real-world forecasting ability [21, 28, 38]. Further, asking a model to forecast from a historical date does not ensure that it ignores ex ante knowledge acquired after that date [27]. Dynamic benchmarks partly address this problem by continually adding, replacing, or updating tasks as models and training data

Schröder and Schweisthal, et al.

change [21, 28, 38]. In particular, live forecasting benchmarks go furdozens of new forecasting events each week and thereby enable ther by collecting predictions before the corresponding outcomes frequent benchmark updates. 5 The benchmark can evaluate a are known [20, 30, 41]. For example, ForecastBench [20] covers broad range of currently relevant system capabilities, including broad questions drawn from prediction markets and continuously probabilistic predictions (e.g., how well models can predict exact updated datasets; Prophet Arena [41] evaluates prediction-market scores vs. outcome probabilities), calibration, and web search. Here, events across politics, economics, entertainment, and science; and established models from sports analytics as well as bookmaker TS-Arena [30] focuses on live numerical time series forecasts in the probabilities also provide external reference points for evaluating energy sector. However, continuous prospective evaluation remains these forecasts [24, 25, 42]. difficult because existing benchmarks are constrained by the limLLM-SoccerArena is a flexible and fully open-source forecastited availability of standardized unresolved events. Moreover, the ing framework. LLM-SoccerArena supports the complete real-time available event streams are often small, irregular, or heterogeneous, benchmark environment in an automated, end-to-end manner, making repeated and controlled comparisons challenging. from event registration and forecast scheduling to model execuIn this paper, we introduce LLM-SoccerArena (https://llm-soccerarena.com tion, schema ), validation, archival, outcome retrieval, evaluation, and a general-purpose system for prospectively evaluating LLM foremaintaining a live leaderboard. We make available not only the casts of unresolved real-world sports events. A key strength of resulting predictions and evaluation data, but also the underlying LLM-SoccerArena is that it provides an automated, prospective platform (under a permissive MIT License; see our GitHub). Below, evaluation protocol to assess LLM forecasts before outcomes we demonstrate the system using the 2026 FIFA World Cup; however, are known. LLM-SoccerArena provides 1 a prospective live the framework is designed to operate continuously across compebenchmark protocol that registers unresolved events and records titions, and we are currently extending it to cover major soccer timestamped, schema-validated forecasts before their outcomes leagues (e.g., the English Premier League, the German Bundesliga, are known; 2 a public open-source platform that stores forecasts where the tournaments launch in fall 2026) to offer a continuous, together with the corresponding prompts, model versions, reaprospective benchmarking platform. soning traces, observed tool use, generated evidence (i.e., the Our contributions LLM-generated rationale for the forecast), token use, and cost; and 3 a factorial benchmark design to evaluate predictions of match 1 Prospective live benchmark protocol. We provide a results and of tournament-related questions (e.g., which team will prospective, real-time protocol that records and evaluates win). The factorial benchmark design allows us to compare various LLM forecasts for unresolved events before their outcomes settings that differ in 1 model versions (e.g., GPT-5.5, Claude are known. Opus 4.8), 2 information access (i.e., with or without web search); 2 Public open-source platform. We provide a public plat3 prompting strategy (i.e., scoreline or outcome probabilities), and form under an open-source license, together with a live leader4 forecast horizon (i.e., whether predictions are made at start of a board and an auditable archive of past LLM forecasts and tournament, 24 hours before a match, or 2 hours before a match). results. Further, LLM-SoccerArena also supports a broad set of evaluation 3 Factorial benchmark design. We compare LLM foremeasures, including accuracy and Brier score. casts across different model versions, information access conLLM-SoccerArena has several strengths as a general benchmark ditions, prompting strategies, and forecast horizons on the for LLM forecasting. 1 It is fully prospective: forecasts are recorded same events. Thereby, we derive recommendations for effecwhile outcomes remain unresolved, so performance reflects actual tive LLM use in real-world forecasts. prediction (and prevents artifacts due to reconstruction, memorization, or access to known outcomes). 2 It supports complex 4 Case study using the 2026 FIFA World Cup. We demonreasoning and information synthesis in a highly standardized realstrate LLM-SoccerArena using on all 104 matches and 15 tourworld setting. Sports matches take place at scheduled times and nament questions of the 2026 FIFA World Cup and analyze produce objective, verifiable outcomes. The setting is further chalforecast quality, reliability, and tool use. We also provide a lenging because the relevant information changes continuously qualitative analysis of the mentioned evidence that is provided before kickoff and may include recent results, team rankings, hisby the LLM to justify a prediction. torical matchups, injuries, expected lineups, venue information, weather, betting odds, and news. Models must therefore identify, 2 Related Work retrieve, weigh, and combine heterogeneous evidence under uncertainty. More broadly, sports are often used in research as a proxy Below, we position LLM-SoccerArena relative to three relevant for studying complex management and decision-making problems. research streams (i.e., )LLM evaluation, prospective LLM forecasting, 3 Soccer provides a challenging forecasting problem: low score and soccer forecasting) and explain how LLM-SoccerArena is novel counts, draws, and unexpected results preserve substantial uncer(see Table 1). tainty even when teams differ in strength, which has motivated LLM evaluation. Standardized benchmarks and public evaluaextensive statistical and machine learning research on match and tion platforms make comparisons across LLMs more transparent tournament prediction [8, 13, 14, 29]. 4 Sports provide a continuous, and reproducible by providing shared tasks, metrics, and evaluation scalable stream of comparable events. Our platform is interoperable procedures [6, 26, 28]. Existing benchmarks are often static and with different league competitions (e.g., English Premier League, the therefore retrospective. [11, 40, 43] Dynamic benchmarks address German Bundesliga, and Spain’s La Liga), which together provide benchmark aging and training data contamination by collecting

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

new examples or regularly updating tasks [2, 5, 21, 38]. These approaches establish the importance of LLM evaluations that are public, repeatable, and continuously updated. However, dynamic benchmarks are often still retrospective: their outputs can generally be evaluated against an existing answer or human judgment and therefore cannot test whether LLMs can forecast future outcomes that are unknown. Prospective LLM forecasting. Prior work evaluates LLMs in prospective forecasting tasks. One approach is represented by Autocast [43] and Autocast++ [40], which reconstruct the information available at earlier dates using time-indexed news and retrieval to preserve the temporal order of information and reduce leakage from events occurring after the forecast date. However, such retrospective reconstruction cannot fully prevent information leakage from post-forecast knowledge already encoded in the model and does not support benchmarking capabilities such as live agentic search and tool use under real-world conditions. As second research stream provides live forecasting benchmarks, often in comparison with human forecasters [15, 18, 35]. Existing examples are scarce. ForecastBench [20] and Prophet Arena [41] collect probabilistic forecasts for heterogeneous questions for which the outcomes are unresolved at forecast time, and TS-Arena [30] records live numerical forecasts for recurring energy data. As such, these benchmarks can establish the value of LLMs in prospective forecasting. However, they are often difficult to scale because unresolved events may be limited or irregular, and the tasks are heterogeneous and not standardized (e.g., questions or events vary across time). In addition, benchmarks from this stream often do not systematically evaluate prompting strategies or information access (e.g., web search) and provide limited qualitative insight into reasoning traces or evidence from LLM-generated justifications. LLM-SoccerArena addresses this gap through a recurring stream of comparable soccer events evaluated under a standardized protocol. Soccer forecasting. Soccer forecasting traditionally uses curated domain information, such as historical results, team strength, rankings, player availability, and betting odds, to predict match scores, outcomes, or tournament progress [8, 14, 25, 29, 42]. Several works develop statistical and machine learning models specifically for this setting [4, 36]. At the same time, bookmaker probabilities provide a strong external baseline that is often hard to beat [13, 24]. However, LLM-SoccerArena has a different goal: we aim to evaluate the general-purpose forecasting abilities of LLMs rather than proposing a new specialized soccer model. We therefore deliberately do not provide curated statistics or domain features beyond basic event information. This design keeps the benchmark simple and inexpensive to extend to new events and domains. It also allows us to directly test whether LLMs can synthesize relevant information and reason under uncertainty, and examine what additional information is retrieved through web search and how it is integrated into the forecasts. Research gap. To the best of our knowledge, LLM-SoccerArena is the first prospective benchmark for systematically evaluating LLMs on recurring real-world sports forecasts.

Table 1: Comparison of related research streams. ✓ = supported, ~ = partially supported, ✗ = not supported. Research stream

Opensource

New unresolved events

Factorial benchmark

Auditable forecast records

Public LLM evaluation [6, 26, 28] Dynamic LLM evaluation [2, 21, 38] Live forecasting benchmarks [20, 30, 41]

✓ ~ ~

✗ ✗ ✓

✗ ✗ ~

~ ~ ~

LLM-SoccerArena

✓

✓

✓

✓

1. INPUTS & CONTEXT

3. EVALUATION & INSIGHTS

2. LLM-SOCCERARENA PLATFORM

Tournament & Match Data FIFA FIFA World World Cup Cup2026 2026 · Final schedule, teams, venues · 19 Jul 2026, 16:00 ET · New Jersey Stadium, USA

Live Leaderboard

Interactive Visualizations

Forecast Explorer

Real-time Updates

Model rankings & performance

Match & tournament insights

Search, filters & detailed views

New forecasts & outcomes

Teams vs Spain

Match-Level Forecast: Final Predicted Scoreline (Top-1) Spain

Forecast Recorded

Outcome Barrier

Before outcome occurs

No information leakage

Outcome Revealed After kickoff/ end of match

24h before kickoff (18 Jul 2026, 15:00 ET)

Away (ARG) 27.2%

Evaluation Unlocked Metrics computed & stored

Real-time Information

Forecast Timing

Actual Outcome

Home (ESP) 42.5% Draw 30.3%

Argentina

• Recent form: Spain beat France 2–0; Argentina beat England 2–1. • Market signal: Spain were favorites; a tight, low-scoring final was expected. • Venue: neutral site.

Argentina

1-0

Predicted Probabilities

Unified Data Store (All Timestamped)

Historical Data & Stats • ELO: ARG 2110 | ESP 2043 • FIFA Ranking: ESP 2 | ARG 1 • H2H (all comps): ESP 3W, ARG 3W • Recent Form: ESP WDW|ARG WLW

Ex.

1-0 Spain Win (AET)

Metric: Brier Score (Lower is Better) Outcome (one-hot): [1, 0, 0] (ESP wins) Prediction: [0.425, 0.303, 0.272] Brier Score = mean((p − y)²) = 0.240 0 (best)

Open Resources & Reproducibility Dataset (CSV)

Code & Scripts

API / Export

Documentation

Timestamped forecasts & outcomes

Evaluation, analysis & reproduction

Programmatic access to all records

Guides, methodology & data dictionary

1 (worst) Evaluation Dimensions

Proper Scoring Rules (Brier, Log Loss) Accuracy Metrics (Exact Score, Directional, Top-1) Calibration & Reliability (ECE, Reliability Curves) Robustness & Validity (Consistency, Diversity) Efficiency (Latency, Tokens, Cost)

All inputs are timestamped and versioned

LLM FORECASTERS

GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro Preview, Grok 4.3, DeepSeek V4 Pro, Qwen 3.7 Max, and Mistral Large 2512

FORECAST CONDITIONS

FORECAST OUTPUTS

Information Access

Prompting Style

Forecast Timing

Web search / Internal Only

Scoreline First / Probabilities First

Stage Start / 24h / 2h before KO

Scoreline Probabilities Confidence (e.g. 2-1) (Home / Draw / Away) (0-100%)

Rationales & Citations

Figure 1: LLM-SoccerArena architecture and public interface.

3

LLM-SoccerArena

LLM-SoccerArena consists of three components (see Figure 1): • 1 Prospective live benchmark protocol: to register unresolved events and record LLM forecasts before their outcomes are known (§3.1). • 2 Public open source platform: to provide forecasts, outcomes, data, code, and evaluation results using an auditable archive [33]. • 3 Factorial benchmark design: to compare selected model versions and other conditions (e.g., information access, prompting strategies, and forecast horizons) on the same event [31].

3.1

Live Benchmark Protocol

The live benchmark protocol has two parts: (1) The forecasting tasks define what each LLM must predict. (2) The prospective benchmark environment defines how unresolved events and their forecasts are registered, collected, validated, stored, resolved, and evaluated. 3.1.1 Forecasting Tasks. We define two complementary forecasting tasks. (a) Match forecasts evaluate repeated short horizon predictions for individual matches. (b) Tournament forecasts evaluate structured predictions over longer horizons, such as group winners, semifinalists, or the champion. • Match forecasts. For each match 𝑖, let 𝑌𝑖 ∈ C = {H, D, A} denote the realized outcome after 90 min plus stoppage time (home win, draw, away win); extra time and penalty shootouts are excluded.
The LLM outputs a probability vector p𝑖 = 𝑝𝑖,H, 𝑝𝑖,D, 𝑝𝑖,A ∈ Δ2, where 𝑝𝑖,𝑐 is the probability the model assigns to the outcome 𝑌𝑖 = 𝑐 and where Δ2 denotes the probability Í simplex, i.e., 𝑝𝑖,𝑐 ≥ 0 with 𝑐 ∈ C 𝑝𝑖,𝑐 = 1. For a knockout match, the LLM additionally reports advancement probabilities, denoted by 𝑎𝑖,H and 𝑎𝑖,A with 𝑎𝑖,H + 𝑎𝑖,A = 1, where advancement accounts for extra time and penalty shootouts. We separate the 90 minute outcome from advancement because a knockout match can be drawn after 90 minutes even though exactly one team advances.

Schröder and Schweisthal, et al.

Each response also contains the expected number of goals for both teams, the most likely scoreline (e.g., 2–1), the model’s selfreported overall confidence on a continuous scale from 0 to 1, and a short generated rationale (which we refer to as evidence). Predicted mean goals are not the football expected goals (“xG”) statistic. Confidence is the model’s overall self assessment and is not treated as a probability that the selected outcome is correct. The generated evidence provides a rationale supporting the forecast and is therefore distinct from the model’s private reasoning traces. These additional information later allow us to evaluate how confident the model is and how the model reasoned over a forecast. • Tournament forecasts. Tournament forecasts are collected through natural-language questions over a predefined candidate set T𝑞 . Depending on the question, exactly one or exactly 𝑘𝑞 candidate teams are correct. For every candidate 𝑗 ∈ T𝑞 , the LLM reports an associated probability together with its final prediction(s). For one-team questions (e.g., “Which team will win the FIFA World Cup?”) there is exactly one correct answer 𝑌𝑞 ∈ T𝑞 . The LLM reports a probability distribution 𝝅 𝑞 over T𝑞 , where 𝜋𝑞,𝑗 denotes the probability assigned to team 𝑗, together with one final selection. Some questions instead require selecting exactly 𝑘𝑞 teams from the candidate set, such as predicting the four semifinalists. In this case, the LLM reports the marginal probability 𝜌𝑞,𝑗 that team 𝑗 belongs to the correct set 𝑆𝑞 ⊆ T𝑞 , subject to Í 𝜌𝑞,𝑗 ∈ [0, 1], 𝑗 ∈ T𝑞 𝜌𝑞,𝑗 = 𝑘𝑞 , and outputs exactly 𝑘𝑞 final selections. The marginal probabilities sum to 𝑘𝑞 because exactly 𝑘𝑞 candidate teams are correct.

3.1.2 LLM prompt. For both tasks, we provide only basic information in prompt blocks shared across all benchmark conditions (details in Appendix F.4). In particular, the prompt has two blocks: (1) The match block contains the competition, edition, stage, UTC kickoff time, teams, known venue, and knockout status. (2) The tournament block contains the tournament structure, official fixtures, valid candidate teams, and the exact question definition. We deliberately do not provide curated historical results (e.g., form statistics, rankings, injuries, lineups, betting odds, or news). This design has two advantages. First, LLM forecasts can be collected without building and maintaining a separate structured data pipeline. Second, this allows us to evaluate the reasoning ability of web search, to assess which current information the LLM retrieves, synthesized, and included in the generated evidence. Table 4 defines all recorded forecast fields and their interpretation. Appendix F provides the exact prompts, response schemas, candidate lists, and repair instructions.

3.1.3 Prospective Benchmark Environment. The prospective benchmark environment separates forecast collection from resolving the outcome. This separation ensures that every LLM forecast is recorded under known conditions before the corresponding outcome becomes available. It proceeds in four steps: (1) Event registration. We register each match or tournament question while the corresponding outcome is unresolved. The registration contains a unique event identifier, the participating teams, the stage, the scheduled time, the available venue information, and the rule used to resolve the outcome. The same schema can register

future tournaments and league fixtures, which creates a continually updated stream of unresolved events without changing the forecasting tasks. (2) Forecast collection. For each registered event, a cron-based scheduler automatically triggers the required forecast requests at prespecified forecast horizons. For match-level events, requests are targeted at 𝑇 − 24 h and 𝑇 − 2 h relative to the scheduled kickoff. Stage-opening forecasts are generated before the group stage begins or, for knockout stages, once all relevant pairings are known. The scheduler checks for due requests every 15 minutes and submits them through OpenRouter, which serves as a common API gateway to the registered model versions. Each request specifies the model version, information access condition, prompting strategy, and forecast horizon. Because execution may deviate from the target time, we store both the scheduled and actual request timestamps and calculate the realized lead time as the interval between the actual request and kickoff. We additionally retain the exact prompt, raw response, parsed forecast, generated evidence, observed tool use, latency, token use, and cost. (3) Validation. A deterministic parser checks the response schema, required fields, the values of numbers, the ranges of probabilities, probability sums, score formats, and event-specific fields. Small deviations in probability vectors that must sum to one are normalized using a predefined rule and recorded. Other invalid responses receive at most one repair call. We retain the original response, every permitted transformation, API errors, timeouts, and unrepaired outputs. This procedure makes response reliability transparent by allowing us to evaluate not only successful forecasts but also operational failures. (4) Outcome resolution. Official outcomes enter the benchmark only after the corresponding forecasts have been stored. This step acts as a barrier that prevents known outcomes from affecting the recorded forecasts. After resolution, we link each valid forecast to its outcome, calculate the evaluation measures (see Section 4.1), and create a frozen analysis snapshot with integrity checks and hashes. The snapshot makes each benchmark release reproducible, while later events and model versions can be evaluated through the same prospective live benchmark protocol. Appendix B provides the complete validation rules, archived fields, and snapshot checks. Appendix F provides the exact response schemas and repair prompts.

3.2

Public Open-Source Platform

The open-source platform (https://llm-soccerarena.com) implements the live benchmark protocol and makes all records public. It provides timestamped match and tournament forecasts, resolved outcomes, a filterable leaderboard, released data, and evaluation code. The public website, released data, and analyses all use the same auditable archive. Every reported result can therefore be traced to the exact forecast, prompt, model version, benchmark conditions, validation record, observed tool use, evidence, latency, token use, and cost stored before the corresponding outcome was known. The website is accessed in a read-only mode, so that website interaction cannot modify stored forecasts or affect their evaluation. The released data retain both valid forecasts and operational failures, while the released code provides the corresponding evaluation procedures. Figure 1 summarizes the platform architecture.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Leaderboard. The website presents the LLM forecasts through four complementary views. Match forecasts display the predictions for individual matches, including probabilities, predicted scorelines, generated evidence, and benchmark conditions. Tournament questions summarize longer-horizon forecasts, such as group winners, semifinalists, or the tournament champion. Bracket views visualize the predicted tournament progression and allow comparisons with the realized tournament path. Performance summaries aggregate evaluation metrics across models and benchmark conditions, providing rankings and interactive comparisons. Figure 3 shows screenshots of the leaderboard. Future updates. The platform is flexible and can be extended with new unresolved soccer events and new model versions, including future tournaments and league fixtures, under the same live benchmark protocol. Together, these resources make LLM-SoccerArena a public, continuously updated, auditable, and reproducible benchmark.

3.3

Factorial Benchmark Design

The factorial benchmark design compares LLM forecasts under controlled combinations along four dimensions: 1 model version, 2 information access, 3 prompting strategy, and 4 forecast horizon. This design allows us to isolate how each factor contributes to forecast quality and to derive practical recommendations for using LLMs in real-world forecasting. Let Z = M × A × P × H,

𝑧 = (𝑚, 𝑎, 𝑝, ℎ) ∈ Z,

(1)

where M, A, P, and H denote the sets of model versions, information access conditions, prompting strategies, and forecast horizons, respectively. For each registered match 𝑖, LLM-SoccerArena creates forecasts for every applicable configuration 𝑧 ∈ Z. All configurations receive the same event information as input and produce the same structured forecast outputs. Consequently, the benchmark dimensions modify only the conditions under which a forecast is generated (i.e., the model version, information access, prompting strategy, and forecast horizon), while the forecasting task itself remains identical across all configurations. This controlled design enables direct comparisons of individual benchmark factors without confounding them with differences in the underlying prediction task. (1) Model version. The model version identifies the exact model endpoint used to produce a forecast. For every forecast, we record both the configured OpenRouter API identifier and the canonical frozen model version. Different model versions may vary in their numerical reasoning capabilities, reasoning strength, tool use. Here, we instantiate LLM-SoccerArena with the following LLMs: GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro Preview, Grok 4.3, DeepSeek V4 Pro, Qwen 3.7 Max, and Mistral Large 2512. Comparing them on the same events and under identical conditions tests which model versions produce better forecasts, rather than confounding model differences with event difficulty or deployment settings. Recording the exact endpoint also makes comparisons reproducible and allows future model versions to be evaluated under the same protocol. (2) Information access. The information access condition tests whether knowledge encoded during LLM training is already sufficient for forecasting current events and whether access to recent external information improves the forecasts. (i) In the closed-book condition, web search is disabled and the LLM relies solely on

its internal knowledge and the provided event information. (ii) In the open-book condition, web search is enabled and the LLM is instructed to retrieve current public information before forecasting. This comparison tests whether current information changes predictions and improves accuracy, as well as agentic web search can identify relevant information. For the subsequent analysis, we additionally record observed search use and the mentioned evidence to qualitatively study whether an open-book model actually retrieves relevant information and whether this information is reflected in its forecast. (3) Prompting strategy. The reasoning abilities of LLMs are known to deteriorate depending on whether qualitative or quantitative outputs are prompted [1]. Hence, by varying the prompting strategy, we can later test how LLMs can deal with events under uncertainty. (i) In the scoreline condition, the LLM is asked to report the most likely scoreline (and should then append the predicted mean goals and the outcome probabilities). (ii) In the probabilistic forecast condition, the LLM is asked to report the outcome probabilities and predicted mean goals (and should then combine these into the most likely scoreline). Both conditions use the same information, definitions, and response fields. Comparing the two conditions tests whether eliciting a concrete scoreline improves score prediction or whether eliciting probabilistic forecast improves probabilistic forecast quality. (4) Forecast horizon. The forecast horizon tests whether forecasts change as more recent event information becomes available and whether these changes improve prediction quality. Each match is forecast at three prespecified horizons. (i) Stage opening provides an early forecast before the corresponding tournament stage begins. (ii) T–24h provides the primary forecast approximately 24 hours before kickoff. (iii) T–2h provides a late forecast that may incorporate recent injuries, suspensions, expected lineups, and other team news. Comparing these horizons tests whether LLMs can effectively retrieve, synthesize, and reason over newly available information and thus become more accurate. Statistical analysis. For each comparison between two model versions or benchmark conditions (e.g., open versus closed book), we calculate one withinmatch metric difference and test whether its mean differs from zero using 10,000 two-sided sign-flip permutations [11, 22]. We apply Holm correction within prespecified families of related comparisons to correct for multiple comparisons [17]. Appendix B provides further details about the statistical analysis.

4 Case Study using the 2026 FIFA World Cup 4.1 Evaluation Setup We demonstrate the abilities of LLM-SoccerArena using 2026 FIFA World Cup. The tournament contains 104 matches among 48 teams and provides a sequence of match events with objective outcomes and rich information that can be used for web search. This setting allows us to evaluate all benchmark conditions repeatedly on the same forecast targets. Note that LLM-SoccerArena is flexible and can be seamlessly adapted to other tournaments or sports leagues. Model versions. Our main evaluation includes seven state-ofthe-art LLMs from different provider families: GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro Preview, Grok 4.3, DeepSeek V4 Pro, Qwen 3.7

Schröder and Schweisthal, et al.

Max, and Mistral Large 2512. The selection is intended to cover a diverse set of models from different providers. Table 3 reports a model registry with the exact API identifiers, provider names, and details on web access. Match forecasts. For every match, we apply the full factorial design with seven model versions, three forecast horizons, two information access conditions, and two prompting strategies. The main evaluation therefore contains 104 × 7 × 3 × 2 × 2 = 8,736 match forecasts. Each LLM thus comes with 1,248 forecasts. This balanced and fully matched design allows every model version and benchmark condition to be compared on the same matches without imputation. Tournament forecasts. Before the tournament began, the seven model versions answered 15 longer horizon questions: the 12 group winners, the four semifinalists, the world champion, and the team of the top scorer. Each question was evaluated under both information access conditions and both prompting strategies. The main evaluation therefore contains 15 × 7 × 2 × 2 = 420 tournament forecasts. We recorded these forecasts once at stage opening and froze them before the corresponding outcomes were known. We do not update them during the tournament because their purpose is to test whether LLMs can forecast the final tournament structure before it unfolds. Performance Metrics. We use different performance metrics to assess scoreline predictions, probability forecasts, and the accuracy of one-team questions. • Our primary metric is the unscaled multiclass Brier score [↓]. For match 𝑖, let p𝑖 denote the reported H/D/A probability vector and let y𝑖 denote the one-hot encoding of the realized 90-minute outcome. We additionally report the log loss [↓].
2 Í The two metrics are defined as BS𝑖 = 𝑐 ∈ C 𝑝𝑖,𝑐 − 𝑦𝑖,𝑐 , LL𝑖 = − log 𝑝𝑖,𝑌𝑖 . Lower values indicate better forecasts. A perfect forecast has a Brier score of zero, the worst possible forecast has a score of two, and a uniform forecast has a score of 2/3. We use the Brier score because it evaluates the complete probability distribution, rather than only the outcome with the highest probability [3, 12]. Log loss penalizes confident incorrect forecasts more strongly. • The modal H/D/A accuracy [↑] measures whether the realized outcome has the highest reported probability; if 𝑘 outcomes share the maximum probability, each receives credit 1/𝑘. • The exact-score accuracy [↑] measures whether the reported scoreline equals the realized 90-minute score. • The Scoring System [↑] provides a graded scoreline measure: five points are awarded for the exact score, two for the correct goal difference, one for the correct H/D/A tendency, and zero otherwise.1 For one-team questions, such as the world champion, we report Brier score, log loss, and accuracy. For the semifinalist question, we also report whether all four teams are correct, how many semifinalists are correct, the mean squared error between each team’s reported probability of reaching the semifinals and the observed outcome, and whether these probabilities sum to four. Table 5 defines all measures. We organize the following results around five evaluation dimensions: (1) model performance, (2) information access, (3)

1 The scoring system is inspired by https://www.kicktipp.de/.

prompting strategy, (4) calibration, and (5) tournament forecasts. Unless stated otherwise, match results use the balanced sevenmodel panel and compare forecasts on the same matches under matched benchmark conditions.

4.2

Model Performance

Forecast quality. Overall, the seven LLMs exhibit comparable forecasting performance, with small differences depending on the evaluation metric. Gemini achieves the best probabilistic forecasts, obtaining the lowest Brier score and log loss, whereas other models perform best on complementary metrics such as modal H/D/A accuracy, exact score accuracy, or Scoring System points. Across models, Brier scores range only from 0.506 to 0.546. Consistent with these small differences, none of the 21 paired model comparisons remains statistically significant after Holm correction (smallest adjusted 𝑝 = 0.055 for GPT versus Mistral). The absence of statistically significant differences is unsurprising given the sample of 104 matches and is consistent with findings from other prospective LLM forecasting benchmarks. Table 2 reports the complete results, while Figure 4 in Appendix C.2 shows cumulative and stage-specific performance over the tournament. The comparison of open-book, probabilistic T–2h forecasts with de-vigged closing bookmaker odds shows competitive absolute performance: Gemini 3.1 Pro Preview achieves a mean Brier score of 0.497, compared with 0.498 for the market consensus. Figures 15 and 16 in the appendix report the absolute and paired comparisons. Forecast diversity. We now study whether forecasts are similar or correlated. Within the same match and benchmark conditions, the seven LLMs assign highly correlated H/D/A probabilities, with a mean pairwise correlation of 0.943. The corresponding mean Jensen–Shannon divergence is only 0.0044. The models therefore provide largely overlapping forecast signals, rather than independent forecasts of the same event. We thus test whether the across-model diversity is useful through an equal-weight ensemble. The ensemble improves on the average member by only 0.0047 Brier, which directly reflects the low disagreement among the forecasts, and none of the seven ensemble–model comparisons remains significant after Holm correction. Aggregation therefore provides only a small gain because the underlying forecasts are already highly similar. Figure 5 in Appendix C.2 reports the pairwise similarity matrices and ensemble contrasts. The same appendix provides the complete pairwise tests and results by tournament stage. Key finding. The leaderboard shows no clear winner, and the seven models produce highly similar forecasts.

4.3

Information Access

Forecast quality. Open-book access provides the clearest controlled improvement in forecast quality. At T–24h, the mean Brier score decreases from 0.535 in the closed-book condition to 0.512 in the open-book condition. The paired closed-minus-open difference is 0.0228, with a 95% confidence interval of [0.0044, 0.0403] and a Holm-adjusted 𝑝-value of 0.045. This corresponds to a 4.3% reduction from the closed-book mean. The information access effect is significant after Holm correction, whereas the 21 pairwise model comparisons indicate similar performance across the seven models.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Table 2: Complete-panel T–24h probabilistic forecast performance. Values are means with 95% match-bootstrap intervals.

Gemini 3.1 Pro Preview GPT-5.5 DeepSeek V4 Pro Grok 4.3 Qwen 3.7 Max Claude Opus 4.8 Mistral Large 2512

Brier [95% CI]↓ Log loss [95% CI]↓ Modal H/D/A accuracy, % [95% CI]↑ Exact score, % [95% CI]↑ Scoring System [95% CI]↑ 0.506 [0.433, 0.587] 0.517 [0.454, 0.584] 0.520 [0.452, 0.596] 0.524 [0.455, 0.600] 0.527 [0.463, 0.597] 0.528 [0.465, 0.597] 0.546 [0.493, 0.604]

0.853 [0.757, 0.963] 0.873 [0.789, 0.963] 0.877 [0.781, 0.980] 0.876 [0.781, 0.979] 0.887 [0.803, 0.978] 0.888 [0.803, 0.982] 0.913 [0.839, 0.989]

Information access therefore represents a more important source of variation in forecast quality than the observed differences among model versions. Figure 2(a) shows the model-specific effects, while panel (b) shows how the effect develops over the tournament (i.e., hinting toward a slight upward trend). Forecast horizon. More recent open-book forecasts (i.e., T– 2h instead of T–24h) improve only slightly. The mean Brier score decreases by 0.0054 from stage opening to T–24h and by 0.0021 from T–24h to T–2h. Closed-book forecasts change little across the same horizons. Thus, web access improves forecast quality, but forecasting closer to kickoff does not by itself improve performance. Search use. Interestingly, not all LLMs with enabling web search eventually use it. We observe search functionality only in 84.5% of open-book forecasts, while model-specific rates range from 48.4% to 100%. Open-book access adds on average 22,306 input tokens, 885 output tokens, 3.92 seconds of latency, and USD 0.110 per forecast. Figure 6 in Appendix C.3 reports the pooled access effect, forecast horizon comparisons, forecast changes across snapshots, and operational costs. Mentioned evidence. We qualitatively analyze the LLMgenerated evidence from 1,456 forecasts using the T–24h forecast horizon and the probabilistic forecast prompting strategy. We analyze the evidence using the following prespecified categories: markets or odds, recent form, injuries or lineups, rankings or team strength, tactics, venue or travel, tournament context, explicit sources, and generic unsupported claims. For this, we use a released keyword lexicon together with a frozen GLM 5.2 annotator to label the rationales. Following recommendations in [10], we cross-checked the labels using a blinded human audit of 196 rationales. Figure 12 in Appendix D reports detailed results across different evidence categories and model-specific patterns. Compared with closed-book rationales, open-book rationales mention recent form 68.0 percentage points more often, markets or odds 60.2 percentage points more often, and injuries or lineups 53.0 percentage points more often. Conversely, generic unsupported claims—predictive assertions or football clichés without a concrete factual or mechanistic basis in the rationale—occur 18.1 percentage points less often. All seven LLMs show the same directional patterns. Details are in Figure 2(d). Appendix C.3 reports robustness and observed-search analyses and decomposes Brier-score variation across models and conditions. Appendix D details the annotation protocol and human audit. Key finding. Web access improves forecasts, but forecasting closer to kickoff adds little and models use search very differently.

63.7 [54.0, 72.6] 62.0 [52.4, 70.9] 63.7 [54.2, 72.4] 63.5 [53.6, 72.4] 61.1 [51.1, 70.1] 63.0 [53.2, 71.8] 59.6 [50.2, 68.1]

15.4 [9.8, 22.8] 15.4 [10.0, 22.5] 13.9 [9.0, 20.6] 11.5 [7.0, 18.1] 13.5 [8.6, 20.3] 14.9 [9.5, 22.3] 12.5 [7.6, 19.2]

Mistral Claude Opus

(a)

Grok

(c)

DeepSeek Gemini Qwen −0.4

−0.2

0.0

0.2

0.4

Δ Brier = closed − open (positive favors open book) Group stage SF

(b)

R32 Third place

R16 Final

QF

n

1.36 [1.10, 1.68] 104 1.39 [1.12, 1.70] 104 1.27 [1.02, 1.56] 104 1.19 [0.96, 1.48] 104 1.24 [0.99, 1.54] 104 1.25 [0.98, 1.57] 104 1.17 [0.93, 1.47] 104

Correct picks

Random guess (25%)

GPT

Δ Brier = closed − open (positive favors open book)

Model

Group A Group B Group C Group D Group E Group F Group G Group H Group I Group J Group K Group L Semifinalists Top-scorer team Winner

26/28 28/28 28/28 28/28 28/28 28/28 28/28 28/28 28/28 28/28 0/28 28/28 10/28 exact 25/28 5/28

8.3% 2.1%

0%

20%

40%

60%

80%

100%

Probability assigned to realized outcome

0.2 0.1 0.0

(d)

−0.1 −0.2

Recent form

+68.0 pp

Markets / odds

+60.2 pp

Injuries / lineups

+53.0 pp

Explicit sources

+18.0 pp -14.4 pp

Team strength

20

40

60

80

100

Completed matches in chronological order

-18.1 pp

Unsupported claims 0%

50%

100%

Share of rationales mentioning category

Figure 2: (a) Model-specific and (b) match-level effects of open-book access. (c) Tournament-question forecasts. (d) Evidence-category shares open-book vs. closed-book.

4.4

Prompting Strategy

We find no evidence that prompt order changes average forecast quality. For each match, we compare Brier scores from score-first and probabilistic forecast under otherwise identical conditions. The mean difference, calculated as score first minus probabilistic , is −0.0008, with a 95% confidence interval of [−0.0044, 0.0029] and a Holm-adjusted 𝑝-value of 0.693. Positive values would favor probabilistic forecast. Forecast structure. Compared with score-first prompting, probabilistic forecast produces 3.98 percentage points more predicted draw scorelines and 4.05 percentage points more 1–1 scorelines, while the mean probability assigned to a draw is 0.21 percentage points lower. Six of the seven model versions show an increase in predicted draw scorelines. Details are in Appendix C.4 and Table 6 in Appendix B. Key finding. Prompt order does not change accuracy, but probabilistic forecast produces more draw scorelines and more disagreement with outcome probabilities.

4.5

Analysis for Calibration and Confidence

We evaluate two reported outputs. First, we test the calibration, that is, whether the H/D/A probabilities match observed frequencies (i.e., whether outcomes assigned 60% occur about 60% of the time). Second, we test whether higher self-reported confidence (i.e., the model’s overall self-assessment) is linked to more accurate forecasts. (1) Calibration. For each T–24h forecast horizon, information access condition, and prompting strategy, we average the probability vectors of the seven model versions. The calibration curves broadly follow the diagonal, but several probability ranges deviate.

Schröder and Schweisthal, et al.

(2) Confidence. A pooled correlation would conflate modelspecific confidence scales and treat repeated forecasts for the same match as independent. We therefore rank confidence within each model-version, information-access, and prompting-strategy cell and compare relative confidence groups. From the lowest to the highest group, the mean Brier score decreases from 0.620 to 0.423, while modal H/D/A accuracy increases from 49.8% to 73.1%. Higher relative confidence is thus associated with better forecasts, but is not a calibrated probability of correctness. Figure 10 and Appendix C.5 provide the detailed results. Key finding. Probability calibration shows no consistent bias, while higher self reported confidence identifies more accurate forecasts.

5

Discussion

Strengths. LLM-SoccerArena offers several advantages as a benchmark setting for real-world LLM forecasting. (1) It is fully prospective yet scalable: forecasts are recorded before outcomes are known, while recurring tournaments and league competitions provide a continuous stream of events with standardized outcomes for evaluating new LLMs over time. (2) Sports provide a well-established empirical setting for studying broader management and organizational phenomena, including decision-making, competition, collaboration, leadership, and performance under uncertainty [7, 39]. Hence, the findings can inform recommendations for effective LLM prompting in practice. The standardized format of sports outcomes further enables direct comparisons between LLMs, but also against specialized statistical and machine learning models, human forecasts, prediction markets, and bookmaker-implied probabilities on the same outcomes. (3) Forecasting soccer results presents a challenging test of information retrieval and synthesis; web-enabled agents must distinguish relevant evidence from noise and conflicting reports, account for uneven coverage across countries and languages, and integrate signals such as form, injuries, lineups, weather, and market information. The collected rationales and observable tool traces provide qualitative insights into the evidence that LLMs use to support their forecasts. (4) The complete benchmark platform is open source. Practical implications for LLM prompting and deployment. Our results suggest three concrete recommendations for managers deploying LLM forecasts. (1) Provide web access when decisions depend on current information rather than relying solely on a larger or newer model. Our results show that open-book forecasts outperform closed-book forecasts, whereas model differences are small. However, practitioners should verify actual search use and record the retrieved evidence, because access to search does not ensure that the model uses it. (2) Prompt design can benefit from probabilistic approaches. Score-first forecasts may be easier to communicate, but in our results, score-first prompting did not improve predictive accuracy (and only increased consistency across the reported outputs). For decision support [34, 37], probability distributions may be more useful because they make uncertainty and alternative outcomes explicit. In practice, this could imply that managers prompt LLMs to construct several plausible scenarios, assess their relative likelihoods, and combine them into a coherent probabilistic forecast rather than requesting only a single prediction. (3) Self-reported confidence should be used carefully. Although practitioners may be

tempted to ask an LLM how confident it is, similar to how managers consult human experts, the reported confidence is not necessarily well calibrated. In some cases, it may help prioritize relatively easier or harder tasks, but it should not be interpreted as a signal whether the forecast is correct. Limitations. As with any benchmark, LLM-SoccerArena has limitations. First, our case study covers only one tournament, even though the FIFA World Cup is a high-profile, globally followed competition with a large and diverse set of matches and large attention with an audience of more than one billion viewers. Our platform itself is flexible, and we are currently extending it to regularly cover additional competitions, including the English Premier League and the German Bundesliga. Second, analyzing the generated justifications and mentioned evidence by LLMs provides useful qualitative insights, but these outputs do not reveal private model reasoning or establish which information causally influenced a forecast. Third, we deliberately use simple and standardized prompts to ensure comparability and to test how LLMs benefit from web searches. This provides many opportunities for future research to examine alternative prompting strategies, more advanced agent architectures, retrieval procedures, and additional forms of tool use. Fourth, similar to humans, LLM forecasts are not perfect; however, this also reflects the inherent difficulty of soccer forecasting, where substantial uncertainty, draws, and unexpected outcomes persist even for specialized statistical models and betting markets. As such, the irreducible uncertainty is precisely what makes soccer a challenging setting for evaluating real-world forecasting capabilities. Research opportunities. LLM-SoccerArena creates several concrete opportunities for future research. (1) The design as a continuous benchmark enables longitudinal studies of whether new model generations improve in forecast accuracy, calibration, search behavior, and cost over time. (2) The standardized setting allows researchers to test how prompting strategies and agent designs affect numerical reasoning. For example, researchers could test whether probabilistic forecasts could be improved through more structured prompting strategies that require intermediate calculations or consistency checks. (3) The recorded evidence and tool traces enable systematic analysis of which search and reasoning behaviors are associated with better forecasts. For example, models sometimes retrieve archival or otherwise weakly relevant sources (e.g., arxiv.org), which makes it possible to study how source selection, evidence quality, and prompting interventions affect forecasting performance and how these behaviors change across model generations. Conclusion. We propose LLM-SoccerArena, an open-source, prospective benchmark and live leaderboard for evaluating LLM forecasts of real-world events where the outcomes are unknown. We demonstrate the benchmark through a case study of the 2026 FIFA World Cup, while the design of LLM-SoccerArena is flexible and can easily be extended to other tournaments, leagues, and model versions. To the best of our knowledge, LLM-SoccerArena is the first benchmark to provide such a continuously operating and standardized evaluation of LLM forecasting in sports.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

References [1] Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2025. What’s in a Prompt?: A Large-Scale Experiment to Assess the Impact of Prompt Design on the Compliance and Accuracy of LLM-Generated Text Annotations. Proceedings of the International AAAI Conference on Web and Social Media 19 (June 2025), 122–145. doi:10.1609/icwsm.v19i1.35807 [2] Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. MathArena: Evaluating LLMs on Uncontaminated Math Competitions. In Advances in Neural Information Processing Systems, Vol. 38. https://proceedings.neurips.cc/paper_ files/paper/2025/hash/1d27c01ebd3e3aebe226b44fc970d803-AbstractDatasets_and_Benchmarks_Track.html [3] Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78, 1 (1950), 1–3. [4] Rory Bunker, Calvin Yeung, and Keisuke Fujii. 2024. Machine Learning for Soccer Match Result Prediction. https://arxiv.org/abs/2403.07669v1. [5] Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. 2025. Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Suzhou, China, 10080–10098. doi:10.18653/v1/2025.emnlp-main.511 [6] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI] https://arxiv.org/abs/2403. 04132 [7] David V. Day, Sandy Gordon, and Corinna Fink. 2012. The Sporting Life: Exploring Organizations through the Lens of Sport. Academy of Management Annals 6, 1 (2012), 397–433. doi:10.1080/19416520.2012.678697 [8] Mark J. Dixon and Stuart G. Coles. 1997. Modelling Association Football Scores and Inefficiencies in the Football Betting Market. Journal of the Royal Statistical Society: Series C 46, 2 (1997), 265–280. doi:10.1111/1467-9876.00065 [9] Stefan Feuerriegel, Christopher Barrie, M. J. Crockett, Laura K. Globig, Killian L. McLoughlin, Dan-Mircea Mirea, Arthur Spirling, Diyi Yang, Tim Althoff, Maria Antoniak, Lisa P. Argyle, Ashwini Ashokkumar, Mohammad Atari, Hannah Bailey, Kevin Bauer, Umang Bhatt, Yidong Chai, Tanmoy Chakraborty, Yanto Chandra, Huimin Chen, Hal Daumé III, Gianmarco De Francisci Morales, Morteza Dehghani, Danica Dillion, Johannes C. Eichstaedt, Kerstin Forster, Dominique Geissler, Kurt Gray, Thomas L. Griffiths, Hartmann Jochen, Oliver P. Hauser, James K. He, Rahul Hemrajani, Felix Holzmeister, Angel Hsing-Chi Hwang, Tiancheng Hu, Anna A. Ivanova, Nils Köbis, Yara Kyrychenko, Himabindu Lakkaraju, Jia Liu, Abdurahman Maarouf, Sebastian Maier, Lennart Meincke, Rada Mihalcea, Brent Mittelstadt, Saif M. Mohammad, Mor Naaman, Oded Netzer, Alice Oh, Desmond C. Ong, Francesco Pierri, Barbara Plank, Iyad Rahwan, Talal Rahwan, Pooja S. B. Rao, Claire E. Robertson, David M. Rothschild, Matthew J. Salganik, Eric Schulz, Chirag Shah, Yash Raj Shrestha, Ekaterina Shutova, Alexandra A. Siegel, Almog Simchon, Huan Sun, Malte Toetzke, Jay J. Van Bavel, Michelle Vaccaro, Jennifer Wortman Vaughan, Effy Vayena, Pedro O. S. Vaz de Melo, Briana Vecchione, Angelina Wang, Robert West, Robb Willer, Dirk U. Wulff, Renwen Zhang, Simone Zhang, Steve Rathje, and Manoel Horta Ribeiro. 2026. A Reporting Checklist for Large Language Models in Behavioural Science. Nature Human Behaviour 10, 7 (July 2026), 1182–1186. doi:10.1038/s41562-026-02492-7 [10] Stefan Feuerriegel, Abdurahman Maarouf, Dominik Bär, Dominique Geissler, Jonas Schweisthal, Nicolas Pröllochs, Claire E. Robertson, Steve Rathje, Jochen Hartmann, Saif M. Mohammad, Oded Netzer, Alexandra A. Siegel, Barbara Plank, and Jay J. Van Bavel. 2025. Using Natural Language Processing to Analyse Text Data in Behavioural Science. Nature Reviews Psychology 4, 2 (2025), 96–111. doi:10.1038/s44159-024-00392-z [11] Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, and Stefan Feuerriegel. 2026. Nonparametric LLM Evaluation from Preference Data. arXiv preprint arXiv:2601.21816 (2026). arXiv:2601.21816 [12] Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. J. Amer. Statist. Assoc. 102, 477 (2007), 359–378. doi:10.1198/016214506000001437 [13] Andreas Groll, Christophe Ley, Gunther Schauberger, and Hans Van Eetvelde. 2019. A Hybrid Random Forest to Predict Soccer Matches in International Tournaments. Journal of Quantitative Analysis in Sports 15, 4 (2019), 271–287. doi:10.1515/jqas-2018-0060 [14] Andreas Groll, Gunther Schauberger, and Gerhard Tutz. 2015. Prediction of Major International Soccer Tournaments Based on Team-Specific Regularized Poisson Regression: An Application to the FIFA World Cup 2014. Journal of Quantitative Analysis in Sports 11, 2 (2015), 97–115. doi:10.1515/jqas-2014-0051 [15] Danny Halawi, Fred Zhang, Yueh-Han Chen, and Jacob Steinhardt. 2024. Approaching Human-Level Forecasting with Language Models. Advances in Neural Information Processing Systems 37. doi:10.52202/079017-1598

[16] Robin M. Hogarth and Spyros Makridakis. 1981. Forecasting and Planning: An Evaluation. Management Science 27, 2 (Feb. 1981), 115–138. doi:10.1287/mnsc.27. 2.115 [17] Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70. [18] Elvis Hsieh, Preston Fu, and Jonathan Chen. 2024. Reasoning and Tools for Human-Level Forecasting. In Proceedings of the Workshop on the Future of Event Detection (FuturED). Association for Computational Linguistics, Miami, Florida, USA, 48–57. doi:10.18653/v1/2024.futured-1.5 [19] Rob J. Hyndman and George Athanasopoulos. 2018. Forecasting: Principles and Practice. OTexts. [20] Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. 2025. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. International Conference on Learning Representations. doi:10.48550/arXiv.2409.19839 [21] Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 4110–4124. doi:10.18653/v1/2021.naacl-main.324 [22] Frank Konietschke and Markus Pauly. 2014. Bootstrapping and Permuting Paired T-Test Type Statistics. Statistics and Computing 24, 3 (May 2014), 283–296. doi:10.1007/s11222-012-9370-4 [23] Anders Krogh and Jesper Vedelsby. 1994. Neural Network Ensembles, Cross Validation, and Active Learning. In Advances in Neural Information Processing Systems 7. MIT Press. https://proceedings.neurips.cc/paper/1994/hash/ b8c37e33defde51cf91e1e03e51657da-Abstract.html [24] Christoph Leitner, Achim Zeileis, and Kurt Hornik. 2010. Forecasting Sports Tournaments by Ratings of (Prob)abilities: A Comparison for the EURO 2008. International Journal of Forecasting 26, 3 (2010), 471–481. doi:10.1016/j.ijforecast. 2009.10.001 [25] Christophe Ley, Tom Van de Wiele, and Hans Van Eetvelde. 2019. Ranking Soccer Teams on the Basis of Their Current Strength: A Comparison of Maximum Likelihood Approaches. Statistical Modelling 19, 1 (2019), 55–73. doi:10.1177/ 1471082X18817650 [26] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic Evaluation of Language Models. arXiv:2211.09110 [cs.CL] https://arxiv.org/abs/2211.09110 [27] Yachuan Liu, Xiaochun Wei, Lin Shi, Xinnuo Li, Bohan Zhang, Paramveer Dhillon, and Qiaozhu Mei. 2026. ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Rabat, Morocco, 1551–1571. doi:10.18653/v1/2026. eacl-long.72 [28] Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021. Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking. Advances in Neural Information Processing Systems 34. https://proceedings. neurips.cc/paper/2021/hash/55b1927fdafef39c48e5b73b5d61ea60-Abstract.html [29] M. J. Maher. 1982. Modelling Association Football Scores. Statistica Neerlandica 36, 3 (1982), 109–118. doi:10.1111/j.1467-9574.1982.tb00782.x [30] Marcel Meyer, Sascha Kaltenpoth, Henrik Albers, Kevin Zalipski, and Oliver Müller. 2025. TS-Arena – A Live Forecast Pre-Registration Platform. arXiv preprint arXiv:2512.20761. doi:10.48550/arXiv.2512.20761 [31] Douglas C. Montgomery. [n. d.]. Design and Analysis of Experiments, 10th Edition | Wiley. https://www.wiley.com/en-us/design-and-analysis-of-experiments10th-edition-p-9781119492443. [32] Fotios Petropoulos, Daniele Apiletti, Vassilios Assimakopoulos, Mohamed Zied Babai, Devon K. Barrow, Souhaib Ben Taieb, Christoph Bergmeir, Ricardo J. Bessa, Jakub Bijak, John E. Boylan, Jethro Browell, Claudio Carnevale, Jennifer L. Castle, Pasquale Cirillo, Michael P. Clements, Clara Cordeiro, Fernando Luiz Cyrino Oliveira, Shari De Baets, Alexander Dokumentov, Joanne Ellison, Piotr Fiszeder, Philip Hans Franses, David T. Frazier, Michael Gilliland, M. Sinan Gönül, Paul Goodwin, Luigi Grossi, Yael Grushka-Cockayne, Mariangela Guidolin, Massimo Guidolin, Ulrich Gunter, Xiaojia Guo, Renato Guseo, Nigel Harvey, David F. Hendry, Ross Hollyman, Tim Januschowski, Jooyoung Jeon, Victor Richmond R.

Schröder and Schweisthal, et al.

Jose, Yanfei Kang, Anne B. Koehler, Stephan Kolassa, Nikolaos Kourentzes, Sonia Leva, Feng Li, Konstantia Litsiou, Spyros Makridakis, Gael M. Martin, Andrew B. Martinez, Sheik Meeran, Theodore Modis, Konstantinos Nikolopoulos, Dilek Önkal, Alessia Paccagnini, Anastasios Panagiotelis, Ioannis Panapakidis, Jose M. Pavía, Manuela Pedio, Diego J. Pedregal, Pierre Pinson, Patrícia Ramos, David E. Rapach, J. James Reade, Bahman Rostami-Tabar, Michał Rubaszek, Georgios Sermpinis, Han Lin Shang, Evangelos Spiliotis, Aris A. Syntetos, Priyanga Dilini Talagala, Thiyanga S. Talagala, Len Tashman, Dimitrios Thomakos, Thordis Thorarinsdottir, Ezio Todini, Juan Ramón Trapero Arenas, Xiaoqian Wang, Robert L. Winkler, Alisa Yusupova, and Florian Ziel. 2022. Forecasting: Theory and Practice. International Journal of Forecasting 38, 3 (2022), 705–871. doi:10.1016/j.ijforecast.2021.11.001 [33] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché-Buc, Emily Fox, and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arXiv:2003.12206 [cs.LG] doi:10.48550/arXiv.2003.12206 [34] Daniel J. Power. 2002. Decision Support Systems: Concepts and Resources for Managers (1. publ ed.). Quorum Books, Westport, Conn. [35] Sarah Pratt, Seth Blumberg, Pietro Kreitlon Carolino, and Meredith Ringel Morris. 2024. Can Language Models Use Forecasting Strategies? arXiv preprint arXiv:2406.04446. doi:10.48550/arXiv.2406.04446 [36] Markel Rico-González, José Pino-Ortega, Amaia Méndez, Filipe Clemente, and Arnold Baca. 2022. Machine Learning Application in Soccer: A Systematic Review. Biology of Sport 40, 1 (2022), 249–263. doi:10.5114/biolsport.2023.112970 [37] J. Shim, Merrill Warkentin, James Courtney, Daniel Power, Ramesh Sharda, and Christer Carlsson. 2002. Past, Present, And Future Of Decision Support Technology. Decision Support Systems 33, 2 (Jan. 2002), 111–126. doi:10.1016/ S0167-9236(01)00139-7 [38] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/ 2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html [39] Richard A. Wolfe, Karl E. Weick, John M. Usher, James R. Terborg, Laura Poppo, A. J. Murrell, John M. Dukerich, Deborah C. Core, Kevin E. Dickson, and James S. Jourdan. 2005. Sport and Organizational Studies: Exploring Synergy. Journal of Management Inquiry 14, 2 (2005), 182–210. doi:10.1177/1056492605275245 [40] Qi Yan, Raihan Seraj, Jiawei He, Lili Meng, and Tristan Sylvain. 2024. AutoCast++: Enhancing World Event Prediction with Zero-Shot RankingBased Context Retrieval. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 93f01c8d9b355d7bbe3f353b44ccde66-Abstract-Conference.html [41] Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu. 2026. LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena. International Conference on Learning Representations. https: //openreview.net/forum?id=VpiHkMSPqI [42] Achim Zeileis, Christoph Leitner, and Kurt Hornik. 2018. Probabilistic Forecasts for the 2018 FIFA World Cup Based on the Bookmaker Consensus Model. Working Papers in Economics and Statistics 2018-09. University of Innsbruck, Innsbruck, Austria. https://EconPapers.repec.org/RePEc:inn:wpaper:2018-09 [43] Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecasting Future World Events with Neural Networks. Advances in Neural Information Processing Systems 35. https://proceedings.neurips.cc/paper/2022/hash/ aec870a6772336c15dac992c16f2e7c9-Abstract-Datasets_and_Benchmarks.html

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Disclosure of LLM use

B.3

OpenAI Codex and Anthropic Claude were used as assistive tools for software development, analysis implementation, visualization, and manuscript drafting and editing; the authors conceived the study, directed and verified the work, and take responsibility for the final content.

Table 4 defines the content and provenance stored for every match forecast. All benchmark conditions use the same response schema. Information access and prompting strategy change how the forecast is produced, not what the model must report.

A

Table 5 defines the complete evaluation set. We keep the measures separate because probability quality, categorical accuracy, score prediction, confidence, output validity, and operational behavior capture different properties of an LLM forecast.

B.4 Future Deployment

Long-term operation, maintenance, and extensions. LLM-SoccerArena is designed as a continuously operating benchmark. Its automated end-to-end pipeline supports event registration, forecast collection, validation, outcome resolution, evaluation, and leaderboard updates with limited manual intervention. We plan to extend the platform to the 2026/27 seasons of La Liga, beginning on 15–16 August 2026; the Premier League, beginning on 21 August 2026; and the Bundesliga, beginning on 28 August 2026. Long-term maintenance includes regularly registering new model versions and forecasting configurations. Continuous operation enables evaluation at scale and create a longitudinal record of changes in LLM forecasting, tool use, and probabilistic reasoning.

B

Evaluation and Reproducibility Details

This appendix provides the details required to reproduce the World Cup 2026 evaluation. It defines the frozen analysis snapshot, model registry, recorded fields, evaluation measures, statistical procedures, and supplementary analyses. The main results use the balanced seven-model panel without imputation. Claude Fable is retained only as additional archive coverage and is excluded from all main analyses. Reporting follows [9].

B.1

Frozen Analysis Snapshot

We create the analysis snapshot with SQLite’s backup mechanism inside a read transaction. The pipeline runs PRAGMA integrity_check and records the UTC freeze time, database SHA-256, schema version, table counts, source path, and repository commit. Final-paper mode requires 104 reconciled fixtures and one official 90-minute outcome for every completed match. We derive typed Parquet tables with explicit schemas and UTC timestamps. Each table records the source-database hash, configuration hash, code commit, row count, key-uniqueness result, and file hash. We read the public website CSV once for reconciliation. Any discrepancy is logged explicitly; the pipeline never resolves conflicts by silently preferring one source.

B.2

Frozen Model Registry

Table 3 resolves every short model name used in the paper. All calls were routed through OpenRouter to the listed provider endpoint. The frozen registry did not store a verified public-weight release for any exact deployed endpoint, so the table reports “unverified” rather than inferring availability from a related model family. Web search was assigned in the open-book condition and disabled in the closed-book condition. Observed search use is analyzed separately.

B.5

Recorded Forecast Fields

Evaluation Measures

Statistical Analysis

The primary match analysis evaluates T–24h H/D/A forecasts from the balanced seven-model panel without imputation. Information access is analyzed by assigned closed-book or open-book condition regardless of observed search. A secondary sensitivity analysis retains only open-book forecasts with observed search. Modelversion and prompting-strategy comparisons hold the remaining benchmark conditions fixed. Each paired contrast is reduced to one difference per match after the stated aggregation. We estimate uncertainty with 10,000 studentized bootstrap replicates that resample complete matches within group-stage and knockout strata. We test paired mean differences with 10,000 sign-flip permutations. We report estimates, 95% intervals, raw 𝑝-values, Holm-adjusted 𝑝-values where applicable, and the relevant match and record counts. The declared Holm families cover information-access and prompting-strategy contrasts, complete-panel model pairs, operational snapshots, eligible stages, ensemble comparisons, and the soccer-specific external baseline. All random procedures use master seed 20260715. Robustness checks preserve the paired design. They restrict realized lead times, repeat the information-access estimate after deleting each match in turn, summarize tournament stages, and resample complete matches for calibration intervals. Tournament forecasts are analyzed at the question level because all configurations for one question share the same realized outcome. Confidence and rationale results are associative; they do not establish causal information use or private model reasoning.

B.6

Output Validation

The archive contains 9,984 scheduled match attempts. The seven main model versions each provide a complete panel of 1,248 valid forecasts, giving the 8,736 forecasts used in the main evaluation. Claude Fable contributes 580 additional valid forecasts and 668 API errors. Across the full archive, 9,106 responses are directly valid and 210 become valid after one repair. No match response requires deterministic probability normalization. Table 6 reports the model-specific counts.

B.7

Artifact Provenance

Every figure and table is accompanied by a machine-readable JSON artifact containing its estimand, estimate, interval, 𝑝-values, counts, aggregation rule, configuration hash, source-table hashes, output hash, and manifest key. The manifest also records the Python, R, package, operating-system, and external-tool versions.

Schröder and Schweisthal, et al.

Table 3: Model registry. The seven complete model versions form the balanced main analysis panel. Note: the online archive also includes Claude Fable 5, but it has partial coverage because data collection was temporarily paused due to U.S. sanctions. Model

Provider

Configured API identifier

Canonical frozen version

Public weights

Web access

Analysis role

GPT-5.5 Claude Opus 4.8 Gemini 3.1 Pro Preview

OpenAI Anthropic Google

openai/gpt-5.5 anthropic/claude-opus-4.8 google/gemini-3.1-pro-preview

Unverified Unverified Unverified

Assigned Assigned Assigned

Main panel Main panel Main panel

Grok 4.3 DeepSeek V4 Pro Qwen 3.7 Max Mistral Large 2512

xAI DeepSeek Qwen Mistral AI

x-ai/grok-4.3 deepseek/deepseek-v4-pro qwen/qwen3.7-max mistralai/mistral-large-2512

openai/gpt-5.5-20260423 anthropic/claude-4.8-opus-20260528 google/gemini-3.1-pro-preview20260219 x-ai/grok-4.3-20260430 deepseek/deepseek-v4-pro-20260423 qwen/qwen3.7-max-20260520 mistralai/mistral-large-2512

Unverified Unverified Unverified Unverified

Assigned Assigned Assigned Assigned

Main panel Main panel Main panel Main panel

Table 4: Forecast content and provenance recorded for each match-level attempt. Object

Stored fields

Interpretation and use

90-minute probabilities

𝑝𝑖,H , 𝑝𝑖,D , and 𝑝𝑖,A

Goal and score forecast

Predicted mean home and away goals; most likely 90-minute score

Advancement forecast

Home and away advancement probabilities

Self assessment and rationale

Confidence in [0, 1]; short generated rationale

Benchmark identity

Match, model version, provider, forecast horizon, information access, prompting strategy, sample ID Scheduled and actual timestamps, minutes to kickoff, prompt and template hashes, raw response, response ID, tool trace, latency, tokens, and cost Parsed fields, validation status and errors, original and final probability sums, normalization and repair flags, scoring eligibility

Probabilities of a home win, draw, and away win after regulation time plus stoppage time. Values must lie in [0, 1] and sum to one. This is the primary forecast target. Predicted mean goal counts and one most likely scoreline. These fields support exact-score, goal-difference, tendency, error, and scoreline-probability agreement analyses. For knockout matches, the probability that each team advances after extra time and penalties if required. For group-stage matches, these fields are null. Confidence is the model’s overall self assessment, not a probability that its selected outcome is correct. The rationale supports analysis of mentioned evidence and is not treated as private model reasoning. These fields form the stable forecast key and prevent accidental duplication or mixing of benchmark conditions.

Execution provenance

Validation provenance

The final manifest links the complete database SHA-256 to every derived table and paper artifact. Acceptance checks cover database integrity, the 104-match outcome universe, stable keys, probability bounds and sums, metric recomputation, and the complete preresults checklist. The public export contains the same 9,984 matchattempt rows and reproduces the stored probabilities, Brier scores, and log losses.

B.8

Public Platform Views

These fields make timing, observed search use, failures, and resource use auditable without reconstructing them from the public website.

Raw responses are never overwritten. Validated values and every permitted transformation remain linked to the original response.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Table 5: Evaluation measures. Lower is better for losses and errors; higher is better for accuracy, points, validity, and observedsearch rates. Calibration, confidence, agreement, diversity, and rationale analyses are diagnostic. Dimension

Measure

Definition

Probability quality

Brier score; log loss

Categorical quality

Modal H/D/A accuracy; advancement accuracy

Scoreline quality

Exact score; goal difference; tendency; absolute errors Scoring System points

Unscaled three-class Brier score and negative log probability assigned to the realized 90-minute outcome. Analogous two-class scores are computed for advancement forecasts. Whether the realized class belongs to the set of maximum-probability classes. If 𝑘 classes share the maximum, each receives credit 1/𝑘 . Advancement uses the larger of the two advancement probabilities. Indicators for the exact 90-minute score, correct signed goal difference, and correct H/D/A tendency implied by the scoreline. Absolute errors cover home goals, away goals, total goals, and goal difference. Five points for the exact score, two for the correct goal difference, one for the correct tendency, and zero otherwise. The categories are mutually exclusive. For home, draw, and away separately, mean predicted probabilities are compared with observed frequencies in five approximately equal-frequency bins. Uncertainty resamples complete matches. Confidence is ranked within each model-version, information-access, and prompting-strategy condition. Brier score and modal accuracy are compared across relative confidence groups. Agreement between the H/D/A outcome implied by the most likely scoreline and the outcome with the highest reported probability; distance between predicted mean goals and scoreline goals; deviations from required probability sums. Pairwise similarity of model probability vectors within identical match and benchmark conditions, plus the performance of an equal-weight probability average formed without mixing conditions. Shares of scheduled calls that are directly valid, valid after deterministic normalization, valid after one repair, invalid after repair, API errors, or timeouts. Among assigned open-book calls, the shares with observed search, no observed search, and unknown status. Closedbook calls form a separate tools-disabled audit. Per-attempt distributions and aggregate consumption by model version and benchmark condition. Failed calls remain in operational denominators when metadata are available. Word count and adjudicated mentions of prespecified evidence types in non-empty rationales. Categories describe generated text, not factual correctness, actual source use, or private model reasoning. Brier score, log loss, and final-selection accuracy for one-team questions; exact four-team set recovery, mean team recovery, marginal probability error, and probability-sum audit for semifinalists.

Game-style score Calibration

Outcome-specific reliability

Self reported confidence

Within-condition confidence association

Scoreline probability agreement

Modal agreement; mean-goal distance; sum audits

Forecast diversity

Observed search use

Probability correlation; Jensen–Shannon divergence; same-condition ensemble Validity, repair, normalization, and missingness rates Search-observed and trace-availability rates

Resource use

Latency, input and output tokens, and cost

Output validity

Generated rationale

Length and evidence-category prevalence

Tournament forecasts

Choice scores, set recovery, and probability sums

Table 6: Validation outcomes for all scheduled match attempts. Repaired responses are valid for scoring. Claude Fable is excluded from the balanced main analysis. Model version

Scheduled

Directly valid

Valid after repair

API error

Scorable

GPT 5.5 Claude Opus 4.8 Gemini 3.1 Pro Preview Grok 4.3 DeepSeek V4 Pro Qwen 3.7 Max Mistral Large 2512

1,248 1,248 1,248 1,248 1,248 1,248 1,248

1,248 1,246 1,194 1,246 1,230 1,232 1,134

0 2 54 2 18 16 114

0 0 0 0 0 0 0

1,248 1,248 1,248 1,248 1,248 1,248 1,248

Main panel Claude Fable 5

8,736 1,248

8,530 576

206 4

0 668

8,736 580

Full archive

9,984

9,106

210

668

9,316

Schröder and Schweisthal, et al.

(a) Match forecasts with scorelines, probabilities, benchmark conditions, and generated rationales.

(b) Filterable performance summaries and trajectories.

Figure 3: Public interfaces to the auditable benchmark archive. Figure 1 summarizes the platform architecture; this figure shows the records that readers can inspect on the public website.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

C Supplementary Results C.1 Results We organize the results around five evaluation dimensions: (1) model performance, whether model versions differ in forecast quality and provide distinct forecast signals; (2) information access, whether current web information is retrieved, changes forecasts, and improves accuracy; (3) prompting strategy, whether score-first or probabilistic forecast prompting changes forecast quality and cross-field consistency; (4) calibration, whether reported probabilities and self-reported confidence correspond to observed accuracy; and (5) tournament forecasts, whether LLMs can produce coherent longer-horizon predictions about the tournament structure. Unless stated otherwise, match results use the balanced seven-model panel and compare forecasts on the same matches under matched benchmark conditions. The supplementary results follow the five evaluation dimensions in the main paper. They provide the robustness checks, diagnostic analyses, and complete figures that support the concise main-text findings. The final subsection reports a soccer-specific closing-odds

comparison. This external baseline is deliberately confined to the appendix because it is not part of the general LLM-SoccerArena protocol.

C.2

Model Performance Details

Model comparisons. The complete family contains 21 paired Brierscore comparisons among the seven model versions. None remains significant after Holm correction. The smallest adjusted 𝑝-value is 0.055 for GPT compared with Mistral. The descriptive leaderboard therefore does not establish statistically distinct model tiers. Tournament progression. Cumulative Brier score and modal accuracy are volatile early and stabilize as matches accumulate. Match-level median Brier increases from 0.403 in the group stage to 0.453 in the round of 32, 0.504 in the round of 16, 0.591 in the quarterfinals, and 0.663 in the semifinals. The late stages contain few matches, so this pattern describes the realized tournament rather than a general increase in difficulty.

Mean Brier score

A

Cumulative Brier score

0.6

0.4

0.2 0

20

40

60

80

100

Accuracy (fractional tie credit)

Schröder and Schweisthal, et al.

B

Cumulative modal H/D/A accuracy

100% 80% 60% 40% 0

Completed matches

Mean Brier score across models (lower is better)

C

20

40

60

80

100

Completed matches

Match-level forecast error by tournament stage 1.5 1.2 0.9 0.6 0.3 0.0 Group stage (n=72)

Round of 32 (n=16)

Round of 16 (n=8)

Quarterfinal (n=4)

Semifinal Third-place match (n=2) (n=1)

Final (n=1)

Figure 4: Performance across the tournament. Cumulative trajectories use the balanced T–24h aggregation across both information-access conditions and both prompting strategies. Stage distributions show one complete-panel mean per match with raw observations and stage sample sizes. Forecast similarity. We compare model versions only within the same T–24h match, information-access condition, and prompting strategy. Mean pairwise Jensen–Shannon divergence is 0.00442 nats, with a range from 0.00162 to 0.00834. Mean pairwise probability correlation is 0.943, with a range from 0.903 to 0.984. These values show that the seven model versions produce highly similar probability forecasts. Open-book access does not detectably increase this convergence. Mean pairwise Jensen–Shannon divergence is 0.00452 in the closedbook condition and 0.00432 in the open-book condition. The paired

open-minus-closed difference is −0.2 × 10−3 nats, with a 95% confidence interval of [−0.9, 0.6] × 10−3 and 𝑝 = 0.593. Same-condition ensemble. We average the seven probability vectors only within the same match and benchmark conditions. The ensemble Brier score is 0.0047 lower than the average member score. Under Brier loss, this difference is the nonnegative disagreement term created by averaging nonidentical probability forecasts [23]. The ensemble directionally outperforms four model versions and trails three; none of the seven ensemble comparisons remains significant after Holm correction.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

A

B

Pairwise JS divergence Access contrast (open − closed) Δ = -0.2 [-0.9, 0.6], p = 0.593

Pairwise probability correlation Fisher-z mean across H/D/A

10−3 nats

10

3.7

Mean r

1.00

0.95

8 3.3 4.2 1.6 4.2 3.5

C

2.5 5.6 3.2 7.6 4.5

0.96 0.97

6

4.4 2.4 8.3 4.4

6.0 6.0

0.95 0.94 0.95

4

4.1

3.2

0.93 0.91 0.92 0.92 0.92

0

5.6

0.90

0.97 0.96 0.98 0.95

2

4.4

0.95

0.85

0.94 0.93 0.95 0.93 0.94 0.90

Same-cell ensemble comparisons

Average member (identity)

−0.04

−0.03

−0.02

−0.01

0.00

0.01

0.02

Δ Brier (ensemble − comparator) Negative values favor the ensemble Figure 5: Forecast similarity and same-condition ensemble performance. The matrices report pairwise Jensen–Shannon divergence and combined probability correlation. The forest plot compares the equal-weight ensemble with the average member and each model version without mixing forecast horizons, information-access conditions, or prompting strategies.

C.3

Information Access Details

Robustness. The T–24h closed-minus-open Brier improvement remains stable when any one match is removed. The leave-onematch-out estimates range from 0.0208 to 0.0253. Restricting the analysis to calls made within 1,440 ± 90 minutes of kickoff gives an effect of 0.0217, with a 95% confidence interval of [0.0021, 0.0401], across 101 matches. The three excluded matches fall outside this prespecified timing window. The primary analysis retains all 104 matches. The effect is largest in the round of 32, where the paired difference is 0.0611, with a 95% confidence interval of [0.0216, 0.0935] and a Holm-adjusted 𝑝-value of 0.012. Stage-specific estimates remain secondary because the knockout stages contain few matches. Observed search sensitivity. The primary analysis compares assigned open-book and closed-book conditions. This intent-totreat comparison preserves the factorial design even when an openbook model does not search. A secondary sensitivity analysis restricts open-book forecasts to calls with observed search. It measures the association with actual search use but no longer compares

the original randomized conditions and can reflect model-specific search behavior. Model-specific effects. Access gains vary descriptively across model versions. They range from approximately zero for Qwen to 0.0349 for GPT. These estimates are not treated as separate confirmatory effects. The model-specific distributions appear in Figure 7. Variation across factors. A two-way decomposition of the 7×4 matrix of T–24h cell means attributes 44.8% of the variation to the four information-access and prompting-strategy conditions, 40.5% to model version, and 14.7% to their interaction. Bootstrap distributions are broad. The decomposition supports treating information access as a first-class benchmark factor without claiming that it dominates every model difference.

Schröder and Schweisthal, et al.

Access effect by tournament stage

A

Access effect over the match calendar

B

Overall

Closed − open Brier

Group stage (n=72) Round of 32 (n=16) Round of 16 (n=8) Quarterfinal (n=4) Semifinal (n=2; desc.) Third-place match (n=1; desc.)

Knockout rounds

Group stage

0.2 0.1 0.0 −0.1 −0.2

Final (n=1; desc.) −0.16 −0.08

0.00

0.08

0.16

0.24

20

C

40

60

80

100

Chronological match number

Δ Brier (closed − open) Positive values favor open-book forecasts

Forecast change across snapshots

D

Open: stage → T−24h

Operational burden

Mistral DeepSeek

Open: T−24h → T−2h

Grok Qwen

Closed: stage → T−24h

Claude Opus

Closed: T−24h → T−2h

Gemini

GPT

−0.004 0.000 0.004 0.008 0.012 0.016

0%

50%

100%

Observed search use

Δ Brier (earlier − later) Positive values favor the later snapshot

$0.00

$0.15

$0.30

Added cost per forecast

Figure 6: Information-access results. Panels show the pooled T–24h access effect, horizon comparisons, forecast changes across snapshots, and operational burden. Positive Brier differences favor open-book forecasts.

GPT Mistral Claude Opus Grok DeepSeek Gemini Qwen −0.4

−0.2

0.0

0.2

0.4

Δ Brier = closed − open (positive favors open book) Figure 7: Model-specific distributions of T–24h closed-minus-open Brier-score differences. The pooled access contrast is prespecified; model-specific results are descriptive.

C.4

Prompting Strategy Details

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Complete metric comparison. We orient effects so that positive values favor probabilistic forecast prompting. The Brier-score effect is −0.0008, with a 95% confidence interval of [−0.0044, 0.0029] and a Holm-adjusted 𝑝-value of 0.693. No significant effect appears for log loss, modal H/D/A accuracy, exact-score accuracy, or Scoring System points. Modal accuracy directionally favors score-first prompting by 0.89 percentage points, but the difference does not remain significant after correction. Information-access interaction. Score-first prompting is directionally better in the closed-book condition, while probabilistic forecast prompting is directionally better in the open-book condition. The additional probabilistic forecast advantage under open-book access is 0.0067, with a 95% confidence interval of [0.0001, 0.0138], raw 𝑝 = 0.055, and Holm-adjusted 𝑝 = 0.110. Reported forecasts. Probabilistic forecast prompting produces 3.98 percentage points more score-implied draws, 4.05 points more exact 1–1 predictions, and 5.36 points more cases in which the

A

scoreline implies a draw while another H/D/A outcome has the highest probability. At the same time, the mean reported draw probability decreases by 0.21 percentage points. Six of the seven model versions show the increase in draw scorelines. Scoreline probability agreement. Agreement means that the most likely scoreline implies the outcome with the highest H/D/A probability, such as 2:1 together with home win. In the closedbook condition, agreement decreases from 69.0% under score-first prompting to 62.4% under probabilistic forecast prompting. In the open-book condition, it decreases from 83.8% to 79.7%. Disagreement does not necessarily make a forecast invalid because one exact scoreline can be most likely even when another aggregate outcome has the highest probability. Output validation. No prompting-strategy condition requires deterministic probability normalization. Repairs are retained as valid forecasts when the repaired response satisfies the original schema. Table 6 reports the complete match-level validation counts.

B

Score-implied draw

Overall prompt effect

Exact 1-1

Draw probability

Modal divergence

Open-book interaction 0.000

0.006

0.012

Brier advantage (positive favors probabilities first)

0.0

2.5

5.0

7.5

Probabilities first - score first (percentage points)

Figure 8: Prompting-strategy effects. The left panel reports paired Brier contrasts. The right panel reports changes in scoreimplied draws, exact 1–1 predictions, mean draw probability, and scoreline-probability disagreement. Positive values favor probabilistic forecast prompting.

Schröder and Schweisthal, et al.

Closed book

62.4%

69.0%

Score first

Probabilities first 79.7%

83.8%

Open book 55.0%

65.0%

75.0%

85.0%

Share with score tendency matching a modal H/D/A outcome Figure 9: Agreement between the outcome implied by the most likely scoreline and the outcome with the highest reported probability. Exact-score and aggregate-outcome modes need not coincide, so disagreement is not labeled a logical inconsistency.

C.5

Calibration Details

Probability calibration. For each T–24h match, informationaccess condition, and prompting strategy, we average the seven model probability vectors. This produces one probability vector for each of the 104 × 2 × 2 = 416 matched conditions. We divide the predicted probabilities into five approximately equal-frequency bins separately for home, draw, and away outcomes. Pointwise intervals resample complete matches. The calibration curves broadly follow the ideal diagonal, but several bins deviate and the uncertainty intervals are wide. Home probabilities vary around the diagonal without a simple directional pattern. Mean draw probability is 24.7%, compared with an observed draw rate of 27.9%. The predicted-minus-observed difference is −3.2

percentage points, with a 95% confidence interval of [−12.5, 4.9] and 𝑝 = 0.461. The outcome-specific deviations do not establish one consistent bias across all three outcomes. Self reported confidence. We rank confidence within each model version, information-access condition, and prompting strategy because model versions use different confidence scales. We preserve ties and group forecasts by relative confidence rank. From the lowest to the highest group, mean Brier score decreases from 0.620 to 0.423, while modal H/D/A accuracy increases from 49.8% to 73.1%. Confidence therefore identifies easier and harder forecasts within a model configuration, but it is not interpreted as a calibrated probability of correctness.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

A

Home outcome

Draw outcome

Away outcome

100%

Observed frequency

80% 60% 40% 20% 0% 0%

20%

40%

60%

80% 100%

0%

20%

40%

60%

80% 100%

0%

20%

40%

60%

80% 100%

Mean predicted probability

C

Brier score vs confidence

Mean Brier score (lower is better)

1

2

0.6

3 4

0.5

5

0.4 0.3

Modal accuracy (higher is better)

B

Accuracy vs confidence 80% 3

5

4

2

60%

1

40% 56%

60%

64%

68%

72%

76%

Mean elicited confidence

56%

60%

64%

68%

72%

76%

Mean elicited confidence

Figure 10: Probability calibration and self reported confidence. Home, draw, and away outcomes use separate reliability panels with pointwise match-bootstrap intervals. Confidence groups are defined within model version, information access, and prompting strategy; lower Brier score and higher modal accuracy indicate better forecasts.

C.6

Tournament Forecast

Before the tournament, the seven model versions answered 15 longer-horizon questions: the 12 group winners, the four semifinalists, the world champion, and the team of the top scorer. We evaluate both the final selections and the reported probabilities. Appendix F, including Table 8, provides the exact question definitions. Group winner. Across the 28 configurations (7 model versions × 2 information access conditions × 2 prompting strategies), 91.1% of the 12 group winner selections are correct. However, all 28 select Portugal for Group K, which Colombia wins. This shared error shows that agreement does not guarantee correctness. The top scorer team is also predicted well: 25 of 28 forecasts select France. The world champion is apparently much harder. Only 5 of 28 forecasts select Spain, while 18 select France. Figure 2(c) summarizes the final selections across all questions. Semifinalists. Ten of 28 forecasts recover the exact four-team semifinal set. On average, each forecast selects 3.29 of the four correct teams. The final selections are therefore often close even when the complete set is not correct. The reported marginal probabilities show a separate problem. They should sum to four because exactly four teams reach the semifinals, but several forecasts violate this requirement. We therefore

evaluate final set recovery separately from probability quality. Figure 11 in Appendix C.6 shows the complete probability sum audit across all 28 configurations. Question set. The seven model versions answer 15 questions once at stage opening under both information-access conditions and both prompting strategies. Each question therefore has 28 configurations. The questions cover the 12 group winners, the four semifinalists, the world champion, and the team of the tournament’s top scorer. Appendix F provides the exact question definitions and candidate lists. One-team questions. Across the 12 group-winner questions, 91.1% of final selections are correct. Group A has 26 correct selections, ten groups have 28, and Group K has none. All 28 configurations select Portugal for Group K, which Colombia wins. For the two tournament-wide one-team questions, 25 configurations select France as the team of the top scorer and five select Spain as champion. Figure 2(c) reports the probability assigned to each realized outcome and the correct-selection counts. Semifinalists. Ten of 28 configurations recover the exact fourteam semifinal set. The mean forecast contains 3.29 of the four correct teams. We evaluate final-set recovery separately from the

Schröder and Schweisthal, et al.

marginal inclusion probabilities because the latter must satisfy an additional sum constraint. Probability-sum audit. The 48 marginal semifinal probabilities should sum to four because exactly four teams reach the semifinals. Several configurations violate this requirement, with strongly

model-specific deviations. We audit the reported probabilities without retrospective normalization. Figure 11 shows the complete distribution.

Coherent target = 4 0

2

4

6

8

10

12

Sum across 48 semifinalist inclusion probabilities Figure 11: Sum of the 48 reported semifinal inclusion probabilities for each model version and benchmark condition. The required sum is four. Raw marginals are audited without retrospective normalization.

D D.1

Rationale Analysis Scope

The rationale analysis covers 1,456 complete-panel T–24h probabilistic forecast rationales. It studies generated text and observable provider traces, not private model reasoning. In the balanced sevenmodel panel, 84.5% of open-book forecasts show observed search. Model-specific rates range from 48.4% to 100%. Closed-book search is not applicable because tools are disabled by design.

D.2

Annotation Procedure

A released keyword lexicon and the frozen annotator z-ai/glm-5.2 label every rationale independently. The annotator receives a fixed prompt, a temperature of zero, and no access to outcomes, forecast performance, model identity, or keyword labels. Raw responses from 73 batches covering all 1,456 rationales are cached and frozen because temperature zero does not guarantee identical regeneration. The prespecified categories are markets or odds, recent form, injuries or lineups, rankings or strength, tactics, venue or travel, tournament context, explicit sources, and generic unsupported claims. The labels indicate that a rationale mentions a category. They do

not establish factual correctness, actual source use, causal influence on the forecast, or private reasoning.

D.3

Human Audit

One blinded human auditor labels a fixed-seed sample of 14 rationales per model-version and information-access cell, giving 196 texts. Final prevalence estimates use human labels for these 196 texts and frozen GLM labels for the remaining 1,260. The keyword coder remains an independent comparison. Across the full scoped corpus, keyword-GLM agreement ranges from 48.0% to 92.7%, with 𝜅 from 0.04 to 0.75. Against the human audit, GLM reaches 89.8% to 100% agreement and 𝜅 = 0.59 to 1.00 for seven categories. Agreement is weaker for tournament context at 57.1% and 𝜅 = 0.25. Generic unsupported claims reach 83.2% agreement but 𝜅 = 0.10 because positive cases are rare. Humanonly information-access differences have the same direction as the resolved estimates for every category. Tournament context is therefore retained with this validation caveat.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

D.4

Complete Evidence Categories

The main-text evidence directions hold for every model version, although their size differs. GPT shows the richest current-evidence profile. Qwen searches frequently but shows smaller open-book

increases in odds, injuries, and explicit sources. Grok produces the shortest rationales and the largest reduction in generic unsupported claims. These differences describe generated text and do not rank reasoning quality.

Markets / odds

+60.2 pp

Recent form

+68.0 pp

Injuries / lineups

+53.0 pp

Rankings / strength

-14.4 pp

Tactics

+13.0 pp

Venue / travel

+4.9 pp

Tournament context

+16.2 pp

Explicit sourcing

+18.0 pp

Generic unsupported claims

-18.1 pp

0%

20%

40%

60%

80%

100%

Share of rationales mentioning category Figure 12: Complete evidence-category results for the 1,456 T–24h probabilistic forecast rationales. Points mark closed-book prevalence and arrows terminate at open-book prevalence. Tournament context has weaker GLM-human validation and should be read with the stated annotation sensitivity.

Schröder and Schweisthal, et al.

Closed book

Open book

Claude Opus DeepSeek Gemini Grok GPT Mistral Qwen Claude Fable 0

20

40

60

80

100

Rationale length (words) Figure 13: Model-specific median and interquartile range of rationale length by information-access condition. Claude Fable appears only as additional archive coverage: 580 of its 1,248 planned match cells contain a valid rationale, while the seven main model versions cover every planned cell.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Closed book

Open book

t-SNE dimension 2 (arbitrary units)

40

20

0

−20

−40 −40

−20

0

20

40

t-SNE dimension 1 (arbitrary units) Figure 14: Exploratory fixed-seed t-SNE projection of frozen, L2-normalized MPNet embeddings for the 1,456 rationale texts. Color denotes model version and marker denotes information access. Axes and distances are arbitrary; the projection supports no confirmatory or mechanistic interpretation.

E

Soccer-specific Closing Odds Comparison

This analysis compares LLM-SoccerArena with a soccer-specific external reference. It is included only in the appendix because closing odds are not part of the general benchmark protocol and do not transfer directly to other domains.

update time, oriented fixture key, and three decimal prices. The 92 distinct raw snapshots are cached and hash verified. Because the feed is subscription data, raw odds remain local; derived consensus probabilities and results are released.

E.1

E.2

Source and Timing

De-vigged bookmaker consensus is an established forecasting reference because it aggregates information and judgments across market participants [24, 42]. We retrieve odds from the historical endpoint of The Odds API for the soccer_fifa_world_cup sport key, European bookmaker region, and h2h market. Prices are decimal odds for the 90-minute home, draw, and away outcomes. For each distinct kickoff time, the collector requests the closest historical snapshot at or before one second before kickoff. Events are matched to the local fixture by kickoff time within one minute and by oriented team names. The extract covers all 104 matches and contains 2,487 bookmaker-match rows, with 15 to 25 bookmakers per match and a median of 24. Each row stores the provider event ID, bookmaker, requested and returned snapshot times, market

De-vigging and Consensus

For match 𝑚, bookmaker 𝑏, and outcome 𝑘 ∈ {𝐻, 𝐷, 𝐴}, let 𝑜𝑚𝑏𝑘 > 1 denote the decimal odds. Raw inverse odds sum to more than one when the bookmaker margin is positive. We remove the margin within each bookmaker: 1/𝑜𝑚𝑏𝑘 fair 𝑝𝑚𝑏𝑘 =Í . (2) 𝑗 ∈ {𝐻,𝐷,𝐴} 1/𝑜𝑚𝑏 𝑗 We then take the cross-bookmaker median for each outcome and renormalize the three values:   fair 𝑝e𝑚𝑘 = median𝑏 𝑝𝑚𝑏𝑘 , (3) market 𝑝𝑚𝑘 =Í

𝑝e𝑚𝑘 e𝑚 𝑗 𝑗 ∈ {𝐻,𝐷,𝐴} 𝑝

.

(4)

Schröder and Schweisthal, et al.

This yields one valid closing-odds probability vector per match without selecting a preferred bookmaker or imputing prices.

E.3

information arriving during the final two hours. Table 7 reports the seven model versions, their equal-weight probability ensemble, and the market reference.

Matched Comparison

We compare closing odds with the balanced seven-model openbook, probabilistic forecast T–2h forecasts. This is the closest registered LLM condition, although the market can still incorporate

Table 7: Closing odds versus balanced open-book, probabilistic forecast LLM forecasts at T–2h over all 104 matches. Values are means with 95% stratified match-bootstrap intervals. Brier skill is 100 × (1 − BrierLLM /Briermarket ), so positive values favor the LLM. Lower Brier score, log loss, and ranked probability score are better; higher modal H/D/A accuracy is better. Forecaster

Brier [95% CI]

Gemini Closing odds GPT DeepSeek LLM ensemble Grok Claude Opus Mistral Qwen

0.497 [0.423, 0.582] 0.498 [0.424, 0.584] 0.500 [0.429, 0.581] 0.506 [0.434, 0.586] 0.507 [0.439, 0.585] 0.511 [0.437, 0.598] 0.517 [0.451, 0.592] 0.518 [0.457, 0.586] 0.529 [0.462, 0.601]

Brier skill vs. market, %

Log loss [95% CI]

RPS [95% CI]

Modal H/D/A accuracy, % [95% CI]

0.3 0.0 −0.3 −1.5 −1.7 −2.6 −3.8 −4.0 −6.2

0.842 [0.738, 0.958] 0.844 [0.741, 0.958] 0.846 [0.749, 0.953] 0.859 [0.761, 0.972] 0.857 [0.765, 0.962] 0.865 [0.759, 0.989] 0.871 [0.779, 0.973] 0.875 [0.794, 0.969] 0.888 [0.797, 0.989]

0.150 [0.128, 0.177] 0.150 [0.128, 0.176] 0.152 [0.131, 0.176] 0.153 [0.133, 0.178] 0.154 [0.134, 0.176] 0.153 [0.132, 0.179] 0.158 [0.139, 0.180] 0.159 [0.142, 0.179] 0.163 [0.144, 0.185]

63.5 [54.0, 72.7] 63.5 [54.1, 72.7] 64.4 [54.6, 73.1] 63.5 [54.0, 72.7] 64.4 [54.9, 73.7] 62.5 [52.2, 71.8] 61.5 [51.2, 70.9] 65.9 [56.3, 74.6] 60.1 [50.2, 69.6]

Gemini is descriptively closest to the market, with Brier score 0.497 compared with 0.498 for closing odds. Its paired marketminus-LLM advantage is 0.001, with a 95% confidence interval of [−0.006, 0.008] and a Holm-adjusted 𝑝-value of 1.000. One of the

𝑛 104 104 104 104 104 104 104 104 104

seven individual model versions has a positive mean Brier advantage over closing odds, and none remains positive after the prespecified correction. These intervals describe uncertainty within this tournament, not performance across future tournaments.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

A

Multiclass Brier score

B

Odds

Odds

Ensemble

Ensemble

C

Log loss (nats)

0.44 0.48 0.52 0.56 0.60

0.78 0.84 0.90 0.96

Mean across 104 matches (lower is better)

Mean across 104 matches (lower is better)

Ranked probability score

D

Odds

Odds

Ensemble

Ensemble 0.135 0.150 0.165 0.180

Mean across 104 matches (lower is better)

Modal H/D/A accuracy

50%

60%

70%

Mean across 104 matches (higher is better)

Figure 15: Absolute performance of closing odds, individual model versions, and the equal-weight LLM ensemble in the matched open-book, probabilistic forecast T–2h condition. Points are means across 104 matches; bars are 95% stratified match-bootstrap intervals.

Schröder and Schweisthal, et al.

A

Multiclass Brier score

−0.06

−0.04

−0.02

B

0.00

−0.05

Paired advantage vs closing odds (score units) C

Ranked probability score

−0.02

−0.01

Log loss (nats)

0.00

Paired advantage vs closing odds (score units)

0.00

Paired advantage vs closing odds (score units) D

Modal H/D/A accuracy

−10

−5

0

5

Paired advantage vs closing odds (percentage points)

Positive values favor the LLM. Bars show stratified bootstrap-t 95% CIs. * Holm-adjusted p < 0.05. Figure 16: Paired LLM advantage over closing odds for Brier score, log loss, ranked probability score, and modal H/D/A accuracy. Positive values favor the LLM; bars are stratified bootstrap-𝑡 95% intervals with inference adjusted across model versions within each metric.

Market Brier - LLM Brier (positive favors LLM)

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

0.15 0.10 0.05 0.00 −0.05 −0.10 0

20

40

60

80

100

Completed matches

Figure 17: Cumulative Brier advantage relative to closing odds. Positive values indicate a lower cumulative mean Brier score for the LLM in the matched open-book, probabilistic forecast T–2h condition. Early trajectories are volatile and are interpreted descriptively.

E.4

External Baseline Eligibility

External baselines enter the analysis only after a common provenance, timing, target-compatibility, and reproduction review. Timestamped bookmaker-level odds pass this review for all 104 matches. Opta and five coauthor-supplied classical football models were also considered: time-weighted ability Poisson, ordered-logit Elo, Elo– Poisson, Dixon–Coles, and ability Poisson with market value. They

remain excluded because row-level provenance, exact timing, or faithful reproduction is unavailable. We remove bookmaker overround before consensus aggregation. Closing odds appear only in this dedicated appendix comparison and are matched to the registered open-book, probabilistic forecast T–2h condition. Coauthor-supplied predictions are never used directly. A classical baseline can enter only after its supplied code is executed in an isolated pinned environment and all successfully reproduced candidates are retained without post hoc selection.

Schröder and Schweisthal, et al.

F

Exact LLM Prompt Templates

This section reproduces the prompt builders used by the released project. Match prompts use template identifier wc2026_v1; tournament-question prompts use wc2026_special_v1. The blocks follow the exact concatenation order in packages/llm/ src/prompt.ts and packages/llm/src/special-questions. ts. Text in angle brackets denotes a runtime value; optional lines are emitted only for the relevant question type. The four benchmark variants combine one information-access header with one prompting-strategy instruction while holding the remaining blocks fixed.

F.1

Tournament Questions and Candidate Lists

The 15 questions are: group_winner_A through group_winner_L, semifinalists, world_champion, and top_scorer_team. Groupwinner questions are one-team questions over the four teams in the relevant group. The champion and top-scorer-team questions are one-team questions over all 48 teams. The semifinalist question is a fixed-set question with exactly four final selections and marginal inclusion probabilities for all 48 teams.

F.2

Prompting Consistency

Probabilistic forecast prompting also reduces agreement between the predicted scoreline and the most probable H/D/A outcome (e.g., 2:1 with home win). Agreement decreases from 69.0% to 62.4% in the closed-book condition and from 83.8% to 79.7% in the openbook condition. Disagreement does not necessarily make a forecast invalid because one exact scoreline can be most likely even when another outcome has the highest total probability. Figures 8 and 9 in Appendix C.4 summarize the accuracy, reported forecast, and agreement effects.

F.3

Match-Level Forecast Prompt (wc2026_v1)

The prompt concatenates the common preamble, one informationaccess header, one prompting-strategy instruction, the common definitions, the runtime fixture block, and the response schema. Common preamble.

You do not have access to this project's stored predictions, analytics, scores, or tournament-tree outputs.

Open-book access header. Open-book prompt component Access condition: OPEN_BOOK. You may use the available web-search tool for current public factual information about teams, squads, injuries, form, fixtures, and tournament context. Do not read, request, infer from, or use this project's stored predictions, analytics, scores, or tournament-tree outputs. Base the final forecast on public information, the match information below, and calibrated football reasoning.

Score-first instruction. Score-first prompt component Prompt strategy: DIRECT_SCORE. Choose the single most likely 90-minute scoreline first. Then assign calibrated 90-minute home/draw/away probabilities, expected goals, full-match probabilities, and advancement probabilities consistent with that forecast. Do not overstate certainty.

Probabilistic forecast instruction. Probabilistic forecast prompt component Prompt strategy: PROBABILISTIC_FORECAST. Estimate calibrated 90-minute home/draw/away probabilities and expected goals first. Then derive the single most likely 90-minute scoreline, full-match probabilities, and advancement probabilities from that forecast. Do not overstate certainty.

Common definitions block.

Common match prompt preamble

Common match definitions

You are forecasting a football match before it is played. Return only valid JSON. Do not include markdown or any text outside JSON. Return a concise reason only; do not reveal hidden reasoning or chain-ofthought. Use calibrated probabilities and do not overstate certainty.

Definitions: - 90-minute result means regulation time plus stoppage time, excluding extra time and penalties. - home_win_90_prob is the probability that the listed home team leads after 90 minutes plus stoppage time. - draw_90_prob is the probability that the match is tied after 90 minutes plus stoppage time. - away_win_90_prob is the probability that the listed away team leads after 90 minutes plus stoppage time. - expected_home_goals_90 and expected_away_goals_90 are expected goals scored in regulation time plus stoppage time. - most_likely_score_90 is the single most likely score after regulation time plus stoppage time. - Full-match result means final match outcome after all applicable extra time and penalty shootout procedures. - For group-stage matches, full-match result is the same as the 90-minute result.

Closed-book access header. Closed-book prompt component Access condition: CLOSED_BOOK. Do not use internet search, browsing, tools, APIs, external data sources, or project databases. Use only the match information below plus your internal football knowledge.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

Table 8: Exact candidate teams supplied to the tournament-question prompts. The union of the 12 groups forms the 48-team candidate list for the semifinalist, champion, and top-scorer-team questions. Group A B C D E F G H I J K L

Candidate teams Mexico; South Africa; South Korea; Czechia Canada; Switzerland; Qatar; Bosnia-Herzegovina Brazil; Morocco; Haiti; Scotland United States; Paraguay; Australia; Turkey Germany; Ecuador; Ivory Coast; Curaçao Netherlands; Japan; Tunisia; Sweden Belgium; Egypt; Iran; New Zealand Spain; Uruguay; Cape Verde Islands; Saudi Arabia France; Senegal; Iraq; Norway Argentina; Algeria; Austria; Jordan Portugal; Colombia; Uzbekistan; Congo DR England; Croatia; Ghana; Panama

- For knockout matches, home_advances_prob and away_advances_prob are probabilities that each team advances/wins the tie after extra time and penalties if needed. - Probabilities must be numbers between 0 and 1. - home_win_90_prob + draw_90_prob + away_win_90_prob must sum to 1. - home_win_full_prob + draw_full_prob + away_win_full_prob must sum to 1. - For knockout matches, home_advances_prob + away_advances_prob must sum to 1. - Confidence is the model's confidence in the overall forecast, between 0 and 1.

The code-level field name expected_goals denotes a predicted mean goal count, not the football expected-goals event statistic xG.

"home": number, "away": number }, "home_win_full_prob": number, "draw_full_prob": number, "away_win_full_prob": number, "most_likely_score_full": { "home": number, "away": number }, "home_advances_prob": number or null, "away_advances_prob": number or null, "confidence": number, "reason": "short reason" }

Runtime fixture block.

F.4 Runtime fixture template Match: Sport: football / soccer Competition: <COMPETITION> Tournament edition: <TOURNAMENT_EDITION> Stage: <STAGE> Date UTC: <UTC_DATE> Home/listed first team: <HOME_TEAM> Away/listed second team: <AWAY_TEAM> Venue: <VENUE_OR_UNKNOWN> Is knockout match: <YES_NO_OR_UNKNOWN>

Match-response schema. Required JSON schema Return only valid JSON. Do not include markdown or explanation outside JSON. JSON schema: { "home_win_90_prob": number, "draw_90_prob": number, "away_win_90_prob": number, "expected_home_goals_90": number, "expected_away_goals_90": number, "most_likely_score_90": {

Tournament-Question Prompt (wc2026_special_v1)

All 15 tournament questions use the common preamble below. It is followed by one information-access header, one prompting-strategy instruction, a question-specific block, the complete candidate list, the static group and fixture context, and the relevant JSON schema. Common preamble. Common tournament prompt preamble You are forecasting \emph{2026 FIFA World Cup} tournament outcomes for Scoring System-style special questions. Return only valid JSON. Do not include markdown or any text outside JSON. Return concise reasoning_summary only; do not reveal hidden reasoning or chain-of-thought. Use calibrated probabilities and valid candidate teams only. These special predictions are one-time pre-tournament/STAGE_OPENING forecasts.

Closed-book access header. Closed-book prompt component Access condition: CLOSED_BOOK.

Schröder and Schweisthal, et al.

Do not use internet search, browsing, tools, APIs, external data sources, or project databases. Use only the static tournament context below plus your internal football knowledge. You do not have access to this project's stored match predictions, special predictions, analytics, scores, or tournament-tree outputs.

Open-book access header. Open-book prompt component Access condition: OPEN_BOOK. You may use the available web-search tool for current public factual information about teams, squads, injuries, form, fixtures, and tournament context. Do not read, request, infer from, or use this project's stored match predictions, special predictions, analytics, scores, or tournamenttree outputs. Base the final forecast on public information, the static tournament context below, and calibrated football reasoning.

Score-first instruction. Score-first prompt component Prompt strategy: DIRECT_SCORE. Make the final pick(s) first, then assign calibrated probabilities that support those pick(s). Do not overstate certainty.

Probabilistic forecast instruction. Probabilistic forecast prompt component Prompt strategy: PROBABILISTIC_FORECAST. Estimate calibrated probabilities for every valid candidate first, then derive the final pick(s). Do not overstate certainty.

Allowed static context: Tournament edition: <TOURNAMENT_EDITION> Groups: Group <GROUP_NAME>: <TEAM_1>, <TEAM_2>, <TEAM_3>, <TEAM_4> <...> Official fixture data: <UTC_DATE> | <STAGE> | <OPTIONAL_GROUP> | <HOME_TEAM> vs <AWAY_TEAM> | < OPTIONAL_VENUE> <...>

The instantiated question identifiers are top_scorer_team, semifinalists, group_winner_A through group_winner_L, and world_champion. one-team response schema. Required one-team JSON schema Required JSON schema: { "question_id": "<QUESTION_ID>", "prediction_type": "single_choice", "stage": "STAGE_OPENING", "choices": [ { "team": "exact candidate team name", "probability": number, "rank": number, "is_final_pick": boolean } ], "final_pick": "exact candidate team name", "confidence": number, "reasoning_summary": "brief explanation" } Include exactly one choices entry for every valid candidate. For single_choice, probabilities must sum to 1. Rank may be omitted or approximate; stored ranks are recalculated from probabilities during validation.

Four-semifinalist response schema. Required four-semifinalist JSON schema

Runtime question, candidate, and context blocks. Runtime question and context template Special question: question_id: <QUESTION_ID> Question label: <QUESTION_LABEL> Meaning: <QUESTION_MEANING> prediction_type: <SINGLE_CHOICE_OR_MULTI_CHOICE_FIXED_K> Use team names exactly as listed in the valid candidates section. Do not translate, abbreviate, or rename teams. <OPTIONAL: Group: GROUP_NAME> <OPTIONAL: Required number of final picks: K> <TOP_SCORER_ONLY: Important: answer with the TEAM of the player who becomes top goalscorer, not the player name.> Valid candidates: - <EXACT_CANDIDATE_TEAM_NAME> - <...>

Required JSON schema: { "question_id": "semifinalists", "prediction_type": "multi_choice_fixed_k", "k": 4, "stage": "STAGE_OPENING", "choices": [ { "team": "exact candidate team name", "probability": number, "rank": number, "is_final_pick": boolean } ], "final_picks": ["exact candidate team name", "..."], "reasoning_summary": "brief explanation" } Include exactly one choices entry for every valid candidate. final_picks must contain exactly 4 unique teams. Rank may be omitted or approximate; stored ranks are recalculated from probabilities during validation.

LLM-SoccerArena : Benchmarking LLMs on Real-World Predictions in Sports

F.5

Single Schema-Repair Prompts

A response that fails parsing or validation receives at most one repair call. Repair calls preserve the substantive forecast whenever possible and are stored separately from the original response. Match-level repair prompt. Match-level repair prompt Your previous response could not be parsed or validated. Convert it into valid JSON matching the required schema. Do not change the substantive forecast unless required to satisfy probability constraints. Return only valid JSON. Do not include markdown or explanation outside JSON. Validation errors: - <VALIDATION_ERROR> Required schema: <MATCH_RESPONSE_SCHEMA_ABOVE> Previous response: <ORIGINAL_RESPONSE>

Tournament-question repair prompt. Tournament-question repair prompt Your previous response could not be parsed or validated. Convert it into valid JSON matching the required special-question schema. Do not change the substantive forecast unless required to satisfy candidate, probability, or pick constraints. Return only valid JSON. Do not include markdown or explanation outside JSON. Validation errors: - <VALIDATION_ERROR> <QUESTION_BLOCK> <CANDIDATE_BLOCK> <QUESTION-SPECIFIC_RESPONSE_SCHEMA_ABOVE> Previous response: <ORIGINAL_RESPONSE>