arXiv:2604.18576v1 [cs.AI] 20 Apr 2026
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
Kevin Murphy Department of Computer Science University of British Columbia Vancouver, BC, Canada [email protected]
Abstract We present BLF (Bayesian Linguistic Forecaster), an agentic system for binary forecasting that achieves state-of-the-art performance on the ForecastBench benchmark. The system is built on three ideas. (1) A Bayesian linguistic belief state: a semi-structured representation combining numerical probability estimates with natural-language evidence summaries, updated by the LLM at each step of an iterative tool-use loop. This contrasts with the common approach of appending all retrieved evidence to an ever-growing context. (2) Hierarchical multi-trial aggregation: running K independent trials and combining them using logit-space shrinkage with a data-dependent prior. (3) Hierarchical calibration: Platt scaling with a hierarchical prior, which avoids over-shrinking extreme predictions for sources with skewed base rates. On 400 backtesting questions from the ForecastBench leaderboard, BLF outperforms all the top public methods, including Cassi, GPT-5, Grok 4.20, and Foresight-32B. Ablation studies show that the structured belief state is as impactful as web search access, and that shrinkage aggregation and hierarchical calibration each provide significant additional gains. In addition, we develop a robust back-testing framework with a leakage rate below 1.5%, and use rigorous statistical methodology to compare different methods while controlling for various sources of noise.
1
Introduction
Forecasting the probability of future events is a fundamental challenge with applications in geopolitics, finance, and public health [Tetlock and Gardner, 2015, Spiegelhalter, 2025]. Recent work has shown that LLMs can approach human-level forecasting when given web search access [Halawi et al., 2024], and benchmarks such as ForecastBench [Karger et al., 2025] provide standardized evaluation with online leaderboards. We present BLF (Bayesian Linguistic Forecaster), an agentic system that achieves new state-of-the-art on ForecastBench. Our approach is organized around three key ideas: 1. Bayesian linguistic belief states. Most forecasting agents either search in parallel then reason once, or sequentially accumulate raw search results in context. BLF instead maintains a semistructured belief state — a probability estimate paired with natural-language evidence summaries — updated by the LLM at each step. This can be viewed as approximate sequential Bayesian inference. Ablations show that removing it degrades the Brier Index by 5.1, an effect larger than removing web search (3.4); see section 4. 2. Multi-trial aggregation. LLM forecasting exhibits high variance across runs. We run K=5 independent trials and aggregate by arithmetic mean, which improves convex scoring rules (such as Brier score, and the Metaculus log score) via Jensen’s inequality (section B.8) but not the linear Brier Preprint.
Index. We also explore hierarchical shrinkage (inspired by James-Stein / empirical Bayes), which nudges forecasts toward 0.5 when cross-trial variance is high; this helps on some datasets but not others (section D.3, section C.4). 3. Hierarchical calibration. To ensure the forecasts are calibrated, we use Platt scaling [Platt, 1999]. However, global Platt scaling can over-shrink well-calibrated extreme predictions. We use hierarchical Platt scaling with per-source intercept offsets, which outperforms global calibration in all settings and is critical when empirical priors produce source-specific biases, especially in the zero-shot setting (section 4). On 400 backtesting questions from ForecastBench, BLF significantly outperforms the top 5 methods on the FB leaderboard (p < 0.001 for all; Table 1), including agentic methods that use tools and web search (Cassi [Cassi AI, 2025], Grok 4.20, Foresight-32B [Lightning Rod Labs, 2025]), and strong zero-shot baselines (Gemini-3.1-Pro-zs and GPT-5-zs). Notably, BLF is the only method that significantly beats the crowd baseline on market questions; all other methods — including those with search and tool access — are statistically indistinguishable from simply returning the market price. Our difficulty-adjusted Brier Index (ABI = 71.0) is comparable to the human superforecaster median (ABI = 70.9) reported on the ForecastBench leaderboard.1 Beyond developing the core system itself, we make four additional contributions: (1) A four-layer date-leakage defense for backtesting, with a post-hoc audit showing only 1.5% residual leakage (section 2). (2) An analysis of the interaction between scoring rules (e.g., Brier score vs. Brier Index) and aggregation methods (section D.3), showing that the choice of metric fundamentally affects whether multi-trial averaging and shrinkage are beneficial. (3) Source-specific empirical priors for dataset questions, analogous to market prices for prediction markets, which provide strong baselines and enable fair comparison across question types (section B.7). (4) A rigorous statistical approach to quantifying the contribution of each component (belief state, search, tools, calibration, crowd signal, LLM choice) using paired analysis with bootstrap confidence intervals (section F.2), controlling for the confounding factor of radically varying question difficulty.
2
Experimental Setup
Problem definition. We focus on binary prediction problems of the following type. Let Y (t) denote the random variable of interest. Let f be the forecast date and r be the resolution date. Our task is to estimate P (Y (r) = 1 | data(≤ f )). We consider two main kinds of questions: open-ended text questions (e.g., “Will person X be the president of country C at time r?”), which require “judgemental forecasting” [Lawrence et al., 2006]; and more traditional time-series forecasting problems (e.g., “Will the stock price of company C at time r be greater than its current value of v at time f ?”), which require numerical reasoning, but may also leverage text data where relevant. Datasets. We did most of our initial development on the test set of 113 binary questions from the Q2 2025 Metaculus AI Benchmark Tournament (AIBQ2). (See section G.1 for details.) However, because AIBQ2 is so small, we also conducted a much larger scale study using data derived from ForecastBench [Karger et al., 2025]. This consists of judgemental forecasting questions from market sources (Polymarket, Manifold, Metaculus, Rand Forecasting Initiative (RFI)), and (binarized) time series forecasting problems from dataset sources (yfinance, FRED, DBnomics, Wikipedia, ACLED). The dataset questions require estimating P (Y (ri ) > v | data(≤ f )) for a set of up to 8 resolution dates ri = f + hi at multiple forecast horizons hi , where v = y(f ) is the current reference value. (See section G.2 for details). We construct two evaluation tranches from FB (A: 2025-10-26, B: 2025-11-09), each with 100 market + 100 dataset questions (400 questions, 791 resolution dates combined). These dates were chosen to meet 3 criteria: (1) be after the knowledge cutoff of current frontier LLMs; (2) maximize number of resolved questions; (3) maximize overlap with results from publicly submitted forecasts, to enable paired comparisons. See section A.1 for details. 1 Human superforecaster estimates from https://www.forecastbench.org/leaderboards/.
requires caution since the question sets and time periods differ.
2
Direct comparison
Metrics. We report three metrics: Metaculus Baseline Score [Aguirre, 2021] (MBS = 100(1 + log2 q), where q=p if o=1, else q=1−p; higher is better); Brier Score (BS = (p − o)2 ; lower is better); and Brier Index (BI = 100(1 − |p − o|); higher is better) [Kucinskas et al., 2026]. See section A.2 for more details on metrics, where we also discuss the Adjusted Brier Index, which is the official metric used by FB leaderboard. When evaluating on FB, overall scores are the unweighted average of market and dataset means, following the FB methodology. Backtesting validity. Our datasets all have dates that are after the knowledge cutoffs for all the models we use, which avoids parametric knowledge leakage. To minimize leakage from search and tool use, we implement a four-layer defense: (1) search engine date filtering, (2) LLM-based leak classifier on results, (3) data tool date clamping, and (4) URL blocking for resolution sources. A post-hoc audit shows only 1.5% undetected leakage rate. See section I for details. Baselines. For our FB experiments, we compare to the top 5 methods (as of 2026-04-15) on the FB leaderboard: “ensemble-2-crowdadj” (ensemble + crowd adjustment) from Cassi [Cassi AI, 2025], Gemini-3-Pro-Preview (zero shot with crowd forecast), Grok 4.20, GPT-5 (zero shot with crowd forecast), and Foresight-32B from Lightning Rod Labs [Lightning Rod Labs, 2025]. See fig. 4 for details of these models. All methods use the crowd estimate as a strong prior (this is only available for market questions). We also include two baselines: Crowd+emp (market price for market questions, empirical prior for dataset questions, no LLM) and ZS+crowd+emp (zero-shot LLM with crowd and empirical prior, but no tools or search). See section A.4 for details.
3
Method
We present the Bayesian Linguistic Forecaster (BLF), an agentic system for binary question forecasting. Given a question q with cutoff date d (set equal to the forecast date f to avoid leakage), BLF runs an iterative tool-use loop (Algorithm 1) that maintains a structured belief state, aggregates multiple independent trials, and applies hierarchical calibration. Full details of our system are in section B; below we give a brief summary. (See also Figure 1 for a system diagram.) Note: We mostly use Gemini-3.1-Pro as our base LLM, but we evaluate other base models in section 4. We use Brave as the agent’s search engine, although our method is engine-agnostic; we discuss other engines in section B.3. Belief state. The core innovation is the Bayesian linguistic belief state: at each step t, the LLM produces both an action at and an updated belief bt in a single generation: (at , bt ) = LLM(mt−1 ), where mt−1 is the full message history. The belief bt is a semi-structured JSON object containing: a probability estimate p ∈ [0, 1], a confidence level, key evidence for/against, and open questions. (See section B.1 for details.) This approach contrasts with (1) text accumulation, where retrieved evidence is simply appended to the context, and grows until it potentially exceeds the model’s attention span, and (2) batch search, where queries are issued in parallel followed by reasoning, rather than sequentially (iteratively) searching and reasoning. Our ablations show that removing the belief state degrades Brier Index by 5.1, an effect larger than removing web search, which degrades BI by 3.4. These results are statistically significant at the p < 0.001 level (see table 3). Agent loop and tools. At each step, the agent selects one of several actions: web_search (with automatic leak filtering), summarize_results (filter and summarize retrieved pages), lookup_url (fetch a specific URL), source-specific data tools (e.g., fetch_ts_yfinance, fetch_wikipedia_section), or submit. The loop runs for up to Tmax = 10 steps. A metacontroller selects the set of tools available to the agent on a per-question-type basis. See section B.2– section B.6 for the details. Our ablations show that replacing this sequential approach with a simpler batch-search-then-reason approach degrades Brier Index by 7.7, which is the single most harmful ablation we considered. These results are statistically significant at the p < 0.001 level (see table 3). Crowd and empirical prior. For market questions, the crowd signal (market price) is injected into the prompt as an anchor; adding it improves BI by 1.7 (Table 3). For dataset questions, we compute an empirical prior πq — the base rate for each source and question subtype — and provide it similarly. Our default system uses both forms of prior (denoted crowd=1, emp=1). See section B.7 for more details. 3
Question q
Cutoff date d
History mt−1
LLM (main)
query
submit(p̂)
Belief bt
Stop
Browse
url
file ids
URL lookup
call(args)
Read
Data tools
page snippets + pages
LLM (filter)
file ids
filtered pages
pages
File store
LLM (summ.)
filtered results
Concat → mt
summary time series
Figure 1: BLF agent loop. At each step, the LLM reads the message history mt−1 and produces an action at and updated belief state bt . The action is executed in the environment (with cutoff-date restrictions), producing an observation ot . The loop terminates at submit or max_steps.
Algorithm 1 BLF agent loop Require: Question q, cutoff date d, max steps T 1: b0 ← initial belief (p = 0.5); o0 ← ∅; m0 ← (q) 2: for t = 1, . . . , T do 3: (at , bt ) ← LLM(mt−1 ) // update belief and choose action 4: if at = submit(p̂) then 5: return p̂ 6: end if 7: ot ← Env(at ; q, d) // execute action with date clamping 8: if at is web search then 9: ot ← LeakFilter(ot , d) // drop post-cutoff results in case clamping fails 10: end if 11: mt ← mt−1 ⊕ (at , ot , bt ) // concatenation (deterministic update) 12: end for 13: Force submit: 14: return bT .p
Multi-trial aggregation. We run K=5 independent trials per question and aggregate using various methods. The simplest is arithmetic mean in probability space, discussed in section B.8, but we also consider an approach based on hierarchical shrinkage in logit space, which encourages the forecast to be closer to p = 0.5 when there is significant cross-trial variance, as discussed in section B.9. On AIBQ2, shrinkage improves MBS and BS metrics but hurts BI, because MBS and BS are convex penalties but BI is linear (see section B.8 for details). However, on ForecastBench, shrinkage hurts all three metrics (Table 4), suggesting the AIBQ2 result may be specific to that dataset’s difficulty distribution. We therefore use the arithmetic mean as the default. Hierarchical calibration. We apply LOO (leave-one-out) cross-validated Platt scaling to map raw forecasts to calibrated probabilities. On ForecastBench, we use hierarchical Platt scaling with 4
Table 1: Comparison to SOTA and LLM-free baseline on FB A∪B dataset (n=791). All methods use the crowd estimate for market questions as a strong prior. The baseline and BLF also use an empirical prior for dataset questions. Top block: absolute BI and paired ∆ BI vs BLF (∆ < 0 means BLF is better). Bottom block: paired ∆ BI of external methods vs crowd+emp baseline (∆ > 0 means method beats baseline). BLF is the only method that significantly beats crowd on market questions. ‡ FB leaderboard; † partial overlap (A-only or B-only). ∗∗∗ p<0.001; ∗∗ p<0.01; ∗ p<0.05; ns not significant (bootstrap, 5,000 resamples). Method
n
Mkt
Dat
All
BI ∆
BI ∆
BI ∆
BLF +crowd+emp+cal 790 94.8 ‡
Cassi GPT-5‡ Grok‡† Foresight‡†
72.2 ∗∗∗
83.5 ∗∗∗
791 90.4 −4.4 68.6 −3.5 79.5 −4∗∗∗ 791 90.9 −3.8∗∗∗ 69 −3.2∗∗∗ 79.9 −3.5∗∗∗ 423 89.3 −5.9∗∗∗ 68.3 −4.3∗∗∗ 78.8 −5.1∗∗∗ 418 91.4 −4∗∗∗ 67.1 −4.6∗∗∗ 79.2 −4.3∗∗∗
Crowd+emp (no LLM) 791 90.6 −4.2∗∗∗ 65.4 −6.7∗∗∗ External vs Crowd+emp Cassi‡ 791 GPT-5‡ 791 423 Grok‡† Foresight‡† 418
−0.3ns +0.3ns −2∗ +0.6ns
+3.2∗∗∗ +3.5∗∗∗ +4.4∗∗∗ +0.1ns
78 −5.4∗∗∗ +1.5∗∗ +1.9∗∗∗ +1.2ns +0.4ns
per-source intercept offsets δs (L2-regularized). See section B.11 for details. This hierarchical approach outperforms global Platt in all settings (Table 5), but the advantage is largest when the empirical prior is enabled — in such cases, global Platt over-shrinks extreme predictions from sources with skewed base rates, while hierarchical calibration preserves them.
4
Results
In this section, we present our results on ForecastBench; see section C for more details, and section D for results on AIBQ2. Comparison to SOTA. Table 1 compares BLF to leading methods on ForecastBench (A∪B). Our system achieves the best BI across all question types. The gains are largest on market questions (BI 94.8 vs. 91.4 for Foresight-32B), where most methods barely beat the crowd baseline (90.6). The paired ∆ BI values (top block) confirm these differences via bootstrap tests: BLF significantly outperforms all external methods on both market and dataset questions (p < 0.001). The bottom block shows that no external method significantly beats the crowd+emp baseline on market questions—only BLF does (+4.2∗∗∗ ). See Table 16 for additional pairwise comparisons. See section C for per-source breakdown, additional metrics (MBS, ABI), and detailed tables with confidence intervals. Core ablations. To identify which components matter most, we run 11 configurations on 791 resolution dates (5 trials each, crowd=0, emp=0, no calibration). Table 2 shows BI and MBS for each. Table 3 shows pairwise effect sizes (controlling for question difficulty via paired comparisons; see section F.2). The most important component is the belief state: removing it degrades BI by 5.1∗∗∗ , more than removing search (3.4∗∗∗ ) or tools (2.6∗∗∗ ). Weaker models also hurt (Flash −2.3∗∗∗ , Kimi −5.9∗∗∗ ). Thinking budget (med vs. high) is not significant. The crowd signal adds +1.7∗∗∗ . Aggregation ablations. Table 4 compares aggregation methods (all using the same 5 trials, no calibration). More trials consistently improve MBS and BS (by Jensen’s inequality on convex losses); however, BI is flat for plain mean, as predicted (section B.8). Median slightly improves BI (+0.2∗∗∗ ) but not MBS. Shrinkage hurts both BI and MBS on FB, even though it helps MBS on AIBQ2 (see section D.3). See section C.4 for more details and Figure 12 for a visualization. For simplicity, we use mean aggregation in probability space as our default method. Calibration ablations. Table 5 compares global vs. hierarchical Platt scaling. Hier-cal outperforms global in both settings, with the gap largest for the ZS baseline: global Platt provides no improvement 5
Table 2: Core ablations on FB (A∪B, n=791, crowd=0, emp=0, no calibration). Each row below the line changes one or more components relative to BLF. 95% bootstrap CIs. See Table 3 for pairwise effect sizes. BI ↑ Method
Mkt
MBS ↑
Dat
All
Mkt
Dat
All
†
clairvoyant 94.6 [93,96] 76.2 [74,78] 85.4 [84,86] 90.6 [87,93] 50.2 [45,55] 70.4 [67,73] BLF +crowd 90 [88,92] 70 [68,72] 80 [79,81] 82.2 [78,86] 37.6 [32,43] 59.9 [56,63] 87 [84,89] 69.4 [68,71] 78.2 [76,80] 70.9 [60,80] 37.4 [33,42] 54.1 [49,59]
BLF medthink flash notools nosearch nobelief kimi zs batch‡
87.3 [85,90] 69.5 [68,71] 78.4 [77,80] 74.3 [66,81] 37.0 [32,42] 55.7 [51,60] 83.4 [80,86] 68.4 [67,70] 75.9 [74,78] 63.3 [52,73] 35.1 [30,40] 49.2 [44,54] 87.8 [85,90] 66.4 [65,68] 77.1 [75,79] 75.3 [67,82] 30.3 [26,35] 52.8 [48,57] 81.3 [78,85] 68.2 [66,70] 74.7 [73,77] 55.9 [44,67] 35.2 [31,40] 45.6 [39,51] 78.8 [76,81] 63.8 [62,65] 71.3 [70,73] 60.2 [53,66] 26.4 [22,31] 43.3 [39,47] 78.6 [76,81] 65.4 [64,67] 72 [70,73] 60.3 [54,66] 27.8 [23,33] 44.0 [40,48] 80.9 [77,84] 66.9 [65,69] 73.9 [72,76] 54.8 [44,66] 32.9 [29,37] 43.8 [38,49] 86.3 [84,89] 53.7 [53,55] 70 [69,71] 72.5 [64,79] 5.9 [3,9] 39.2 [35,43]
† Clairvoyant sees future data (uses crowd=1). ‡ Batch = 5 parallel queries, non-agentic.
Table 3: Paired ablation comparisons vs BLF (no calibration, crowd=0, emp=0) on A∪B (n=791). ∆ = mean paired BI difference (positive = better than BLF). p = bootstrap probability that the method is better (∆ > 0) or worse (∆ < 0) than BLF (5,000 resamples). See Table 2 for method definitions. ∗∗∗ p < 0.001. LLM Srch Tool Crwd Bel ∆ BI
Method †
σ
clairvoyant +crowd
Pro Brave Pro Brave
1 1
1 1
1 +7.9 16.1 ∗∗∗ 1 +1.7 11.6 ∗∗∗
BLF
Pro Brave
1
0
1
(reference)
1 1 0 1 1 1 0 1
0 0 0 0 0 0 0 0
1 1 1 1 0 1 – 1‡
+0.2 5.6 −2.3 8.6 ∗∗∗ −2.6 12.6 ∗∗∗ −3.4 16.4 ∗∗∗ −5.1 10 ∗∗∗ −5.9 10.5 ∗∗∗ −6.2 19 ∗∗∗ −7.7 18.7 ∗∗∗
medthink flash notools nosearch nobelief kimi zs batch
∗
Pro Brave Flash Brave Pro Brave Pro None Pro Brave Kimi Brave Pro None Pro Brave
∗ med thinking; ‡ batch = 5 parallel queries; † clairvoyant (future data).
(−0.2, n.s.) because it over-shrinks the extreme predictions from the empirical prior, while hier-cal yields +3.5∗∗∗ . Our best system (BLF+crowd+emp+cal) achieves 83.5 overall BI. See section C.5 for more results, and Figure 13 for a reliability diagram and histogram of ECE (expected calibration error) before and after calibration. Behavioral analysis. Figures 2 and 3 illustrate the agent’s behavior on a single question, namely “Will WorldAtlas.com display the Gulf of America before July 1, 2025?” (The context is that President Trump had ordered that the “Gulf of Mexico” be renamed to “Gulf of America” on January 20, 2025.2 The belief trace (Figure 2) shows high inter-trial variance (σ=0.20), motivating multi-trial aggregation. The trace from one of the rollouts (Figure 3) shows how the belief state captures key evidence: Trial 2 correctly identifies that WorldAtlas uses static maps (not dynamic Google Maps), leading this agent to become skeptical that the change will happen before the deadline. (And indeed the true outcome is “No”.) See section C.6 for statistics on tool usage and number of steps taken by the agents.
5
Related Work
We organize the growing literature on LLM-based forecasting into six themes. 2 https://en.wikipedia.org/wiki/Executive_Order_14172.
6
Table 4: Aggregation ablations on FB (A∪B, n=791, no calibration). ∆ BI = paired difference vs mean:5. ∗∗∗
p < 0.001 (bootstrap, 5,000 resamples). BI ↑ Variant mean:5
Mkt
MBS ↑ All Mkt Dat All ∆ BI
Dat
87 [84,89] 69.5 [68,71] 78.2 [76,80] 70.9 37.5 54.2
σ
(reference)
median 87.9 [85,90] 69.5 [68,71] 78.7 [77,80] 70.9 37.5 54.2 +0.2 1.9 ∗∗∗ mean:3 85.9 [83,89] 69.5 [68,71] 77.7 [76,79] 68.8 37.5 53.1 −0.3 1.9 ∗∗∗ mean:1 83.5 [80,87] 69.5 [68,71] 76.5 [75,78] 63.6 37.5 50.5 −0.9 5.3 ∗∗∗ shrink 81.1 [78,84] 69.5 [68,71] 75.3 [73,77] 63.3 37.5 50.4 −1.5 4.1 ∗∗∗
Table 5: Calibration comparison on FB (A∪B, n=791). ∆ = paired BI difference vs uncalibrated. ∗∗∗ p < 0.001; ∗∗ p < 0.01 (bootstrap). See section C.5 for all four settings. BI ↑ Setting Calibration Mkt
Dat
All ∆ BI
BLF+crowd+emp (crowd=1, emp=1) uncal 91.9 70 81 (ref) global-cal 94.4 71.9 83.1 +2 ∗∗∗ hier-cal 94.8 72.2 83.5 +2.3 ∗∗∗ ZS+crowd+emp (search=none, tools=0) uncal 87.7 61.8 74.7 (ref) global-cal 87.1 62 74.5 −0.2 hier-cal 90.5 65.9 78.2 +3.5 ∗∗∗
Benchmarks and evaluation. Zou et al. [2022] introduced Autocast, the first large-scale dataset for neural forecasting of real-world events, and found that LM performance was far below human experts but improved with model scale and retrieval. Halawi et al. [2024] scaled this up with 5,000+ binary questions from five platforms, showing that retrieval-augmented GPT-4 approaches the human crowd’s Brier score. ForecastBench [Karger et al., 2025, Kucinskas et al., 2025] provides a rolling benchmark with market and dataset questions, using difficulty-adjusted Brier scores to compare methods on a common leaderboard. TFRBench [Ahamed et al., 2026] evaluates forecasting reasoning (not just accuracy). FutureX [Zeng et al., 2025] introduces a live benchmark with daily-updated questions and an automated pipeline to eliminate data contamination. Metaculus FutureEval [Metaculus, 2026] provides a continuously updated live benchmark that resists contamination (since answers are not yet known), with $175K in annual prizes for bot tournaments. OpenEP [Guan et al., 2024] moves beyond binary questions to open-ended outcome prediction. Paleka et al. [2025] provide a critical analysis of evaluation pitfalls, including temporal leakage and retrieval leakage. Li et al. [2026] show that LLMs fail to “simulate ignorance” of events before their knowledge cutoff, further complicating backtesting validity. Our four-layer leakage defense (section I) addresses several of these concerns. Financial forecasting and trading. Several benchmarks evaluate LLMs as financial predictors and traders. Prophet Arena [Yang et al., 2025] argues that economic utility (profit from bets) may be more relevant than Brier scores; they find that LLMs show impressive calibration but suffer from inaccurate event recall. Prediction Arena [Zhang et al., 2026] deploys six frontier models with real capital on Kalshi and Polymarket, finding that most models lose money (−16% to −31% on Kalshi). PolyBench [Cheng et al., 2026] evaluates seven LLMs on ∼38K Polymarket questions with order-book data, finding that only two models achieve positive returns. FinTradeBench [Agrawal et al., 2026] evaluates financial reasoning combining fundamentals and trading signals, finding that retrieval helps fundamental analysis but not time-series reasoning — consistent with our finding that LLMs struggle with raw numerical data (section B.4). Zero-shot and prompt-based forecasting. Karkar and Chopra [2025] show that LLM forecasting ability is “unevenly distributed” across domains and question types, with performance varying significantly depending on framing. Pratt et al. [2024] found that superforecasting prompting strategies (decomposition, base rates, retrieval) failed to improve PaLM 2 over a basic prompt, 7
Figure 2: Belief evolution across 5 trials on “Will WorldAtlas.com display the Gulf of America before July 1, 2025?” (outcome: No). All trials start at p0 = 0.5 and dip low at steps 1–2 (initial search finds skepticism), then diverge as different search paths lead to different conclusions. High inter-trial variance (σ=0.20) motivates aggregation. Question: Will WorldAtlas.com display the Gulf of America before July 1, 2025?
Outcome: No
Trial 2 (forecast: 0.28) — the “skeptic” among 5 trials Step pt Action & key evidence 0 0.50 Prior (no information) 1 0.05 search(“Gulf of America Gulf of Mexico”) ↓ “Renaming attempts usually fail; base rate <1%” 2 0.65 lookup_url(worldatlas.com/. . . /na.htm) ↑ Executive Order signed; Google Maps changed label 3–8 0.65 Multiple searches — inconclusive 9 0.65 search(“worldatlas.com maps API”) ↓ WorldAtlas uses static map images 10 0.28 submit — “Static sites take 1–3 years to update”
Figure 3: Agent trace for Trial 2 (Figure 2). The agent initially drops to p = 0.05 (skeptical prior), jumps to 0.65 after finding the executive order (from President Trump), then stays at 0.65 for 7 steps while searching inconclusively. At step 9, it discovers WorldAtlas uses static images (not dynamic Google Maps), causing a sharp drop to p = 0.28. This trial correctly identifies the key uncertainty — whether the site has updated its static maps — that the other four trials miss.
attributing the model’s apparent accuracy to a negativity bias that happens to align with low base rates. Schoenegger et al. [2025] tested 38 prompt variants across GPT-4o, Claude 3.5, and Llama 3.1, finding that most modifications yield negligible gains and some (e.g., explicit Bayesian reasoning prompts) actively hurt. These negative results motivate our agentic approach: rather than engineering better prompts, we provide tools (search, data access) and structured belief tracking. Agentic and tool-augmented systems. Hsieh et al. [2024] proposed RTF (Reasoning and Tools for Forecasting), a zero-shot framework using hierarchical agents with Python REPL and Google Search, achieving Brier scores competitive with human crowds on Manifold Markets without any weight updates. The AIA Forecaster [Alur et al., 2025] employs adaptive iterative search and statistical calibration (Platt scaling), similar to us, but uses simple text aggregation rather than a belief state. Also, they use an LLM-based calibration system, which we found to be inferior to simple mean aggregation (see section B.10). CogForecast [Wang et al., 2025] uses multi-agent debate with diverse cognitive profiles to mitigate inherited biases. Reinforcement learning for forecasting. Several recent works apply RL to improve forecasting. Time-R1 [Liu et al., 2025] uses a two-stage curriculum (temporal comprehension then prediction) to build “temporal logic” into the model’s representations. Turtel et al. [2025] apply RLVR (RL with Verifiable Rewards) to train a 14B model on historical Polymarket questions, achieving frontier8
level Brier scores (0.190) and demonstrating economic utility (10%+ ROI in simulated trading). OpenForecaster [Chandak et al., 2026] synthesizes 50K+ training questions from historical news and uses GRPO with a composite accuracy+Brier reward to combat hedging bias, showing that specialized 8B models can match 120B+ generalist models. Jeen et al. [2026] fine-tune models specifically for the Metaculus AI Benchmark Tournament, achieving the current AIBQ2 SOTA (MBS=45.8). Turtel et al. [2026] introduce “Foresight Learning,” using resolved outcomes as free supervision via proper scoring rule rewards, showing that a trained Qwen3-32B outperforms the untrained Qwen3-235B (27% better Brier score). These RL approaches are complementary to our work: they improve the base model, while we improve the agent harness (tools, belief tracking, calibration). Ensemble methods. Schoenegger et al. [2024b] showed that aggregating 12 diverse LLMs (a “silicon crowd”) matches human crowd accuracy, and that frontier models can update beliefs when shown human median forecasts. Our negative result — that model ensembling does not help when components share the same architecture (section E) — is consistent with their finding that ensemble gains require genuine diversity. Schoenegger et al. [2024a] showed that access to GPT-4 assistants improves human forecasting accuracy by 24–28%, regardless of the user’s skill level, suggesting that human–AI ensembles may be more promising than pure AI ensembles.
6
Conclusion
Summary. We developed a system that achieves SOTA performance on the ForecastBench challenge, by leveraging multiple ideas: agentic tool use, structured belief updates, and hierarchical Bayesian aggregation and calibration. Limitations. Our evaluation uses backtesting, so conclusions may not perfectly predict live forecasting performance (as discussed in Section 2). The ultimate validation would be to show that performance rankings from backtesting correlate with live ForecastBench performance; however, the competition’s submission limits (2 variants per fortnight) and 50 day reporting delay make comprehensive live ablation studies infeasible. Also our system was mostly evaluated on a specific base model (Gemini-3.1-Pro) and may not transfer to other models. Future work. We focused on predicting binary outcomes, but it would also be useful to predict categorical outcomes (since many questions are about alternatives from a fixed slate [Yang et al., 2025]) and numerical outcomes3 Other potential future directions include: better tools for dataset questions (especially temperature forecasting in dbnomics); online continual learning of the metacontroller (agentic harness); RL fine-tuning of the base model; and live testing, on benchmarks such as https://futurex-ai.github.io/, https://prophetarena.co/), etc.
3 For continuous values, could ask the agent to return a discretized cdf, as used in the Metaculus competition:
https://www.metaculus.com/faq/#question-types.
9
see
A
Experimental Setup Details
A.1
Datasets
We did most of our initial development on the test set of 113 binary questions from the Q2 2025 Metaculus AI Benchmark Tournament (AIBQ2).4 These questions cover a variety of topics (see section G for details), but do not have a crowd estimate. This allows us to directly measure the value of web search and reasoning without confounding factors, such as strong baselines and tool use. All questions were asked between 2025-04-22 and 2025-06-15 and only ones that resolved by 2025-07-01 are kept. However, because AIBQ2 is so small, and because there are very few reported results on it, we also conducted a much larger scale study using data derived from ForecastBench [Karger et al., 2025]. This consists of binary prediction questions from market sources (Polymarket, Manifold, Metaculus, Rand Forecasting Initiative (RFI)), which assess the ability to do “judgemental forecasting” [Lawrence et al., 2006] on various topics for various forecast horizons. We can formalize this as follows. Let Y (t) denote the random variable of interest (e.g., “Will A be president of country B at time t?”). Let f be the forecast date and r the resolution date. Our task is to estimate P (Y (r) = 1 | data(≤ f )). FB also contains questions derived from dataset sources (yfinance, FRED, DBnomics, Wikipedia, ACLED), which assess the ability to do (univariate) time series forecasting. These are converted to binary prediction problems by asking the forecaster what the probability will be that the value of interest at multiple future resolution dates, out to a horizon of h ∈ {7, 30, 90, 180, 365, 1095, 1825, 3650} days, will be larger or smaller (possibly by some factor) compared to its value at the forecast date. More formally we are required to estimate P (Y (ri ) > v | data(≤ f )), where ri = f + hi is the resolution date for the i’th forecast, for i = 1 : 8, Y (ri ) is the unknown future value, and v = y(f ) is the known threshold or reference value. For example, Y (ri ) > v might represent the event “the temperature at location X on date ri will be higher than its current value v”. (Note that the value v is not always given in the question, a subtlety we discuss in section B.4.) 500 new FB questions are released every two weeks (250 market and 250 dataset), and these resolve over time at different rates. We construct two evaluation datasets (which we call “tranches”) from ForecastBench, selecting dates that satisfied several criteria: after the knowledge cutoff of all models of interest (2025-08-31), has many resolved values (especially important for market questions, which resolve less often), and maximizes overlap with the submissions from existing leading methods. (See section A.4 for details on these existing methods which we compare to.) Each tranche uses questions from a single forecast_due_date and selects those which are resolved by 2026-04-10 (ensuring ground truth is available). Of the ∼500 questions per forecast date, approximately 100 market and 200 dataset questions had resolved by 2026-04-10; we select all 100 market questions and 100 dataset questions (20 per source, via stratified sampling with a fixed seed) for each tranche, to ensure neither dominates the performance metrics. (Note that each dataset question has multiple resolution dates, so the total number of binary questions with answers is n=791.) More precisely, we use these tranches: • Tranche A (forecast_due_date = 2025-10-26): 100 market + 100 dataset questions (200 total). Comparisons available: Cassi, xAI Grok 4.20, GPT-5. • Tranche B (forecast_due_date = 2025-11-09): 100 market + 100 dataset questions (200 total). Comparisons available: Cassi, Foresight-32B, GPT-5. The combined dataset (Tranche A∪B, n = 400) is used for the main comparison. Paired statistical tests are conducted within each tranche (where all methods forecast the same questions). Note that for market questions (but not dataset/timeseries questions), ForecastBench includes a crowd estimate, based on the market price for Polymarket, Manifold, and RFI, and the Community Prediction score for Metaculus. This is a very strong baseline, and most methods on the leaderboard rely on it quite heavily. A.2
Metrics
We use multiple metrics to evaluate performance, as defined below. 4 See https://www.metaculus.com/tournament/aibq2/.
The full list of questions and outcomes is available at https://gist.github.com/enjeeneer/86e24a52e6041a3d78e333bcab16984d, taken from [Jeen et al., 2026].
10
• Metaculus Baseline Score (MBS): MBS = 100 log2 (q/p0 ) = 100(log2 q + 1) where q = p if o = 1, q = 1 − p if o = 0, and p0 = 0.5 is the uniform baseline.5 A perfect prediction (q = 1) gives MBS = 100, while the baseline (q = 0.5) gives MBS = 0. MBS punishes incorrect predictions in the tails more severely than Brier score. Higher is better. • Brier Score: BS = (p − o)2 , where p is the forecast and o ∈ {0, 1} the outcome. Lower is better (despite the use of the word “score”), and predicting the uniform distribution of p = 0.5 scores 0.25. • Adjusted Brier Score: ABSi = BSi − γi is the difficulty adjusted Brier score for question i, where γi is an estimate of question i’s difficulty [Bastani et al., 2025, Kucinskas et al., 2025]. For market questions, γi = (mi − oi )2 where mi is the market price (crowd estimate). For dataset questions, γi is a fixed effect estimated across all methods on the ForecastBench.6 ABS controls for question difficulty, so a method that does well on hard questions scores better than one that only does well on easy questions. ABS was used as the official metric on the FB leaderboard until March 2026, when they switched to adjusted Brier index (see below). √ • Brier Index: BI = 100 × (1 − BS) = 100 × (1 − |p − o|). Higher is better; always0.5 scores 50%. This is designed to be an easier-to-interpret version of Brier score. See [Kucinskas et al., 2026] for details. However, because it is a linear function of the error, |p − o|, it behaves very differently under averaging compared to the non-linear BS and MBS scores, as we discuss below. √ • Adjusted Brier Index: ABI = 100 × (1 − ABS). As of March 2026, this is now the official metric used by FB leaderboard [Kucinskas et al., 2026]. MBS and (A)BS metrics are proper scoring rules, which reward well-calibrated forecasts (see e.g., [Gneiting and Raftery, 2007, Waghmare and Ziegel, 2025]). Unfortunately, (A)BI is not technically proper, although it converges to a proper scoring rule as the sample size increases [Kucinskas et al., 2026]. More importantly, BI uses a linear penalty, |p − o|, while BS uses a quadratic penalty, (p − o)2 . This distinction matters for aggregation, as we explain in section B.8. A.3
Backtesting Validity
A key concern with backtesting is information leakage: the LLM’s parametric knowledge or web search results may contain post-cutoff information that would not be available in a live setting. (In our backtesting context, the cutoff date is set equal to the forecast due date.) Mitigating search leakage. We employ four layers of defense: (1) the search engine itself filters results by date range; (2) all search results pass through an LLM-based date-leakage classifier (Gemini-3-Flash) that drops results containing post-cutoff content; (3) source-specific data tools clamp their date range so they never exceed the cutoff date; (4) direct lookup of certain URLs, such as from prediction market sites, are blocked, since they may contain the true outcome but cannot be date filtered. Note that algorithmic date filtering at the search engine level (layer 1) is critical: without it, we would need to retrieve many more results knowing that the LLM-based classifier (layer 2) will discard most of them. There is a potential risk that the core LLM could choose to “spontaneously” do unfiltered search on its own without us asking it to; however, both Gemini and Anthropic models require explicit opt-in to use search, which we do not enable. Instead web search is controlled entirely through our own tool schemas passed via the function-calling API, so we can enforce date filtering. 5 See https://metaculus.medium.com/a-primer-on-the-metaculus-scoring-rule-eb9a974cd204 for an introduction and https://www.metaculus.com/help/scores-faq/#baseline-score for the full derivation. Metaculus also defines a Metaculus Peer Score (MPS) that measures performance relative to other forecasters rather than a uniform baseline: MPS = 100 × (ln(q) − ln(GM(qi ))), where GM(qi ) is the geometric mean of all other predictions for the same question. (See https://www.metaculus.com/help/scores-faq/#peer-score.) We do not use MPS in this paper, but it could be approximated by using the crowd estimate as the reference prediction (cf. the adjusted Brier score below). 6 The γ values are available at https://www.forecastbench.org/datasets/. However, it is harder to apply this i metric to novel datasets. In section F.2, we use a paired analysis to achieve a similar effect in a more general way.
11
Mitigating parametric leakage. Besides leakage from web search, the LLM will contain certain forms of knowledge (including potentially the true outcome of a given forecasting question) up until its knowledge cutoff date. To avoid this risk, we only use models whose cutoff date is before the date range of the questions we test on. Post-hoc leakage audit. To assess the effectiveness of our leakage mitigation techniques, we run a separate “leak detective” that re-classifies all saved search results and URL lookups conducted by the agent, producing a confusion matrix. For these experiments (and all the other experiments in this paper), we use the Brave search engine, for reasons discussed in section B.3. As shown in Table 6, the runtime filter has 93.8% recall (catches 320 of 341 leaks) but also drops 577 clean results (29.9% false positive rate). Crucially, of the 1,375 results the agent actually sees (the “kept” column), only 21 are leaks—an undetected leakage rate of just 21/1,375 = 1.5%, suggesting that residual leakage has minimal impact on forecasts. Table 6: Leakage audit confusion matrix (n = 2,272 search results across 9 sources from Tranche A). Rows: post-hoc verdict from a second LLM classifier (Grok-4.1). Columns: runtime filter decision (Gemini-3-Flash). The runtime filter catches 93.8% of leaks but has a 29.9% false positive rate, meaning clean results are sometimes unnecessarily dropped. Using a stronger classifier could reduce FPs while maintaining the high leak detection rate.
Post-hoc: Clean Post-hoc: Leak
Filter: Kept
Filter: Dropped
1,354 21
577 320
To assess parametric leakage, we scan the LLM’s reasoning traces (including chain-of-thought) for mentions of post-cutoff events, dates, or outcomes that could not have been known at the forecast date (see section I for details). For ForecastBench, all the LLMs we use have a knowledge cutoff date before the tranches of interest, so there is no parametric leakage on FB. However, for AIBQ2, Kimi-K2-Thinking (knowledge cutoff 2025-06-30) exhibits parametric leakage on 2 of 113 questions, including specific knowledge of the South Korean martial law extension to August 2025 (see section I for details). A.4
Baselines we compare to
Since we cannot expect a forecaster to perfectly predict the future, it is important to establish credible reference values to compare to. We selected the top 5 scoring methods from the ForecastBench tournament leaderboard, shown in fig. 4. Predictions from all methods are publicly available.7 The phrase “zero shot” means the LLM is given the question text and asked to produce a probability in a single step (no search or tool use). The phrase “crowd” means that the market estimate is included in the question as a hint (not available for dataset questions). The top 5 methods and their availability in our tranches are: • Cassi (ensemble+crowd-adj): ABI=67.9. Available in both Tranches A and B. • Gem.-3.1-Pro (zs+crowd): ABI=67.8. Not available for our tranche dates,8 so we recreated them using Gemini-3.1-Pro with the zero-shot prompt from Halawi et al. [2024] and freeze values (crowd=1). • Grok 4.20: ABI=67.8. Available in Tranche A only. • GPT-5 (zs+crowd): ABI=67.2. Available in both Tranches A and B. • Foresight-32B: ABI=67.2. Available in Tranche B only. 7 Leaderboard: https://www.forecastbench.org/leaderboards/#tournament. Predictions: https://www. forecastbench.org/datasets/. 8 The ForecastBench leaderboard includes Gemini-3-Pro-Preview with the zero-shot prompt from Halawi et al. [2024], but only starting from 2025-11-23 — after both our tranche dates.
12
Figure 4: Screenshot of the ForecastBench tournament leaderboard (https://www.forecastbench.org/ leaderboards/#tournament) as of 2026-04-20. We compare to the top 5 methods (highlighted). ABI = Adjusted Brier Index (difficulty-adjusted). Note that each method’s ABI is computed on a different set of resolved questions, so direct comparison of leaderboard scores is approximate; our paired analysis (Table 1) provides rigorous comparisons on shared questions.
A.5
Compute requirements
We access all LLMs through OpenRouter (https://openrouter.ai/), using the LiteLLM interface (https://github.com/BerriAI/litellm), which provides a uniform way to use multiple LLM providers (e.g., Google, OpenAI, Anthropic, Moonshot). All experiments are launched from a laptop. A single question takes 1–8 minutes depending on API latency and the number of steps taken by the agent. (We limit the maximum number of steps to 10, and the maximum time to 10 minutes; if the agent has not submitted by then, we take its last estimate of p̂.) Using 50 parallel workers, a full evaluation (5 trials on 400 questions) completes in 1–3 hours wall clock. The agentic methods consume 50–100M tokens across 5 trials. At Gemini-3.1-Pro pricing ($2/M input, $12/M output), a single method evaluation costs approximately $230; output tokens dominate cost despite being fewer in number. (Smaller models like Gemini-3-Flash are much cheaper.)
B
Method Details
B.1
Bayesian Linguistic Belief State
The core innovation of our system is the Bayesian linguistic belief state: a semi-structured representation that the LLM maintains and updates at each step of the agent loop. The belief state consists of: • A probability estimate p ∈ [0, 1] for the binary outcome; • A confidence level (low/medium/high); • Key evidence for and against the outcome (natural-language summaries); • Open questions the agent plans to investigate next. At each step, the LLM receives the full message history (including the current belief state) alongside new evidence (search results, tool outputs), and produces an updated belief state, which is appended to the history for the next step. See section B.2 for details. This approach contrasts with two common alternatives: (1) text accumulation, where all retrieved evidence is appended to an ever-growing context, eventually exceeding the model’s effective attention span; and (2) batch-search then reason, where multiple queries are issued in parallel, and then the model reasons about all of them all at once, without iterative (sequential) refinement of the search queries. Our ablation nobelief in Table 3 shows that removing the structured belief state and falling back to naive text accumulation degrades BI by −5.1∗∗∗ , an effect comparable to removing web search entirely. In addition, our ablation batch in Table 3 shows that one-shot search-then-reason degrades BI by an even larger −7.7∗∗∗ . B.2
Agent Loop
The agent operates as an iterative tool-use loop, as shown in Figure 1 and Algorithm 1. To explain in more detail, let us define some notation: q is the question; t is the time step; at is the action chosen 13
by the agent; ot is the resulting observation from the environment; bt is the belief state; and mt is the (non-Markovian) message history, mt−1 = (q, a1:t−1 , o0:t−1 , b0:t−1 ). The belief state bt is a semi-structured object containing the agent’s probability estimate bt .p ≈ P (outcome = 1 | mt−1 ), as well as natural-language evidence summaries, as we discussed in section B.1. Consequently, we call our agent the “Bayesian Linguistic Forecaster” or BLF. At each step, the LLM produces both an action at and an updated belief state bt in a single generation: (at , bt ) = LLM(mt−1 ). The belief update is embedded as a structured JSON field (updated_belief) within the tool call arguments, so both outputs come from one LLM call. This field includes the updated probability bt .p, as well as an update_reasoning string in which the LLM explains why its belief changed (e.g., “page contradicts initial assessment” — see Algorithm 2 for an example trace). We have found that requiring the LLM to articulate its reasoning for each belief update encourages coherent probabilistic updating. The action at is executed in the environment, returning an observation, ot = Env(at ; q, d), where d is the cutoff date restricting data access. Finally the message history is updated deterministically by concatenation: mt = mt−1 ⊕ (at , ot , bt ). The loop terminates when the agent calls submit (returning p̂ = bT .p) or reaches max_steps, at which point submission is forced. This process can be viewed as a form of sequential Bayesian updating and decision making for a POMDP. Traditionally we first perform a belief update, bt = f (mt−1 ), integrating all available information, and then an action selection, at = π(bt ), choosing which tool to call next or whether to submit. However, this requires bt to be a sufficient statistic of all the past data, mt−1 , in order to render the system Markovian. By contrast, in our approach, actions are chosen with access to the entire history mt−1 ; the beliefs bt are an additional form of structured information (derived from mt−1 ), which helps the model maintain coherent probabilistic reasoning across steps. As we show in our ablations (Table 2), the benefits of this are even larger than allowing access to web search. Algorithm 2 Simplified agentic trace for T = 4 steps. (See fig. 3 for a real trace.) The web-search action returns a set of 10 snippets, as well as the corresponding page content; the full content is stored in a filing system (not shown), and the snippets are concatenated and returned to the agent. The summarize-results tool call will read the relevant files generated by a previous web search, and then summarize their contents and return the result to the agent. 1: b0 ← initial belief (p = 0.5); o0 ← ∅; m0 = (q) 2: (a1 , b1 ) ← LLM(m0 ) // a1 =(lookup-url, url); b1 .p=0.65 “Seems likely based on question” 3: o1 ← Env(a1 ) // page text 4: m1 ← m0 ⊕ (a1 , o1 , b1 ) 5: (a2 , b2 ) ← LLM(m1 ) // a2 =(search, query); b2 .p=0.40 “Page contradicts initial assessment” 6: o2 ← Env(a2 ) // (snippet1 , . . . , snippet10 ) 7: m2 ← m1 ⊕ (a2 , o2 , b2 ) 8: (a3 , b3 ) ← LLM(m2 ) // a3 =(summarize, [3, 6, 9]); b3 .p=0.35 “Sources confirm opposition” 9: o3 ← Env(a3 ) // (text3 , text6 , text9 ) 10: m3 ← m2 ⊕ (a3 , o3 , b3 ) 11: (a4 , b4 ) ← LLM(m3 ) // a4 =(submit, 0.30); b4 .p=0.30 “Confident: evidence is clear”
B.3
Web search tools
The agent can issue web search queries via an API. We call this tool browse-web (query). For backtesting, it is crucial that search support date filtering, to avoid leakage. We have found that the Brave search engine, which supports a freshness parameter to restrict results to before the cutoff date, to be quite reliable, although we still need the second-stage content-based date filter discussed in section 2. We briefly evaluated other search engines (Perplexity, AskNews, Exa, Serper, Tavily) but they all had one or more of the following problems: unreliable (or non-existent) date filtering; only returned snippets, not full page text; expensive. It might be interesting to explore these other search engines more carefully in future work. However, it is important to emphasize that BLF can work with any search engine. For each search, Brave returns up to 10 short snippets by default, but we also request extra snippets to get more informative results (which we find generally helps performance). The snippets are immediately added to the context, but the full results are not, to avoid overwhelming the attention 14
Table 7: List of all the tools and their arguments. Here q denotes the question, which contains information about its source sq , forecast date fq and resolution date(s) rq , which are needed by certain tools. The first letter is bolded, to represent the short-hand we use to describe each tool. Type Tools Web search b: browse-web (query), r: read-files (ids) Time series h: history-fetch (sq , fq ), m: model-fetch (sq , fq , rq ), c: combo-fetch (sq , fq , rq ) Other u: url-lookup (url), w: wikipedia-fetch (url, [section]), x: submit mechanism. Instead we save search results to a set of local files, search_{i}_result_{j}.md. We give the agent an additional tool, called read-files (id1 , . . . , idk ), which can load selected files into the context of a sub-LLM (we use Gemini Flash), which returns a summary to the main LLM. (The summarizer is given a prompt generated by the main LLM, so it extracts question-specific facts.) The main agent can choose which files it wants to read based on their snippets. This progressive disclosure approach — similar to Anthropic’s tool-use patterns [Anthropic, 2025] — avoids flooding the context with noise from irrelevant results while letting the agent drill into the most promising leads. In our experiments (section 4), we find that web search is the most important single tool: removing it degrades BI by −3.4 on FB (see Table 3). The one exception is DBnomics (temperature time series), where web search is disabled because it introduces noise that overrides the statistical signal (see section B.4). B.4
Time-series tools
As we briefly discussed in section 2, all the dataset questions in FB require the ability to do time series forecasting, i.e., predicting a value of interest at multiple fixed time points into the future and then comparing to a reference value, to create a binary prediction problem of the form p̂rq = P (Yq (r) > vq | data(≤ f ))
(1)
where q is the question, r is the resolution date, and f is the forecast date. Most of these dataset sources have publicly available APIs that allow you to retrieve historical values of the quantity of interest, which is very useful for tackling this problem. (The exception is ACLED, whose API requires special permission to use.) More precisely, these database retrieval tools return the history of values from a given source s up to a given date t in the form of a CSV list: H(s, t; W ) = {(tj , ys (tj )) : t − W ≤ tj ≤ t}
(2)
where W is the window size (which we set by hand, depending on the source).9 For brevity, we define Hq = H(sq , fq ; Wsq ), where sq is the source of question q (e.g, a FRED question), and fq is the forecast date of question q. We denote the tool for fetching this history by history-fetch(q). Note that Hq set contains vq = yq (fq ), allowing us to use the correct reference value in our comparisons, as opposed to the potentially stale “freeze value” given in the original question. This is needed because the freeze value is of the form vq′ = yq (t′ ), from the time t′ ≤ fq when the question was created. In some cases, vq′ may be quite different to vq (see e.g., fig. 30), so forecasters will be comparing to the wrong reference. Although providing Hq to the LLM is often helpful, sometimes we find that the LLM cannot make sense of this raw numerical data. In such cases, we can optionally compute an estimate of p̂q = {p̂rq : r ∈ res-dates(q)} directly from Hq by using simple statistical models (see Appendix H for details). We denote the tool for computing this model-based estimate by model-fetch(q). We can either use this estimate directly (bypassing the LLM), or pass it back to the LLM, just like any other tool call. Currently we use p̂q directly for DBnomics, since we found the LLM struggled with directly interpreting the raw data (see table 24). For yfinance and FRED, we use a combined tool, combo-fetch(q), that returns both Hq and p̂q in a single call (this is for efficiency reasons). 9We also consider more sophisticated versions of the history, that sample past data at different strides, allowing us to look back further without creating very long traces, but we omit this for simplicity. Furthermore, we always enforce t ≤ f , so the agent cannot cheat by looking into the future.
15
Table 8: Elementary actions (tools) available to each question, depending on its source type. Abbreviations are defined in table 7. We define Tsearch = {b, r, u, x}; this is available to all sources except DBnomics (which uses Tbasic = {u, x}, without search tools). Currently, time-series sources use the combo tool c (returning both history and model estimate); the separate h and m tools are a planned decomposition for future work.
B.5
Source s
Tools Ts
FRED, yfinance DBnomics Polymarket, Manifold Wikipedia RFI, Metaculus, ACLED
{c} ∪ Tsearch {m} ∪ Tbasic {h} ∪ Tsearch {w} ∪ Tsearch Tsearch
Other tools
We now describe two other tools we make available to the agent. A simple one is url-lookup(url), which retrieves the contents of a specific URL. This is useful because the question text, or resolution criteria, often contain specific URLs with relevant information. However, we have to be careful that the agent does not just lookup the answer (since date filtering cannot be applied to url lookup). To combat this, we create a blacklist of sites that might contain the true outcome for the question, such as polymarket.com. If the agent tries to perform a url-lookup call with such an argument, we return an empty string and an error message (in text form). This way the agent can learn not to try this invalid action again (c.f., [Lou et al., 2026]). (Such invalid calls should not happen, since the tool descriptions specifies the constraints, but smaller models sometimes struggle to obey these instructions). Finally, we discuss a special tool for Wikipedia questions. FB considers these dataset questions because they require the user to predict the value at multiple future resolution dates. For example, consider the question “According to Wikipedia, will Alexandra Kosteniuk have an Elo rating on [resolution dates] that’s at least 1% higher than on [forecast date]?". It is potentially possible to predict this answer just using information from web search, but it is easier to do if you can retrieve the values from the data table specified in the resolution criteria, namely https://en.wikipedia.org/ wiki/FIDE_rankings. See fig. 32. Of course, we must avoid leakage. Fortunately the wikipedia API allows retrieval of a page at or before a given timestamp, which we set to the forecast date f . We denote this tool by wikipedia-fetch(url). B.6
Policies
Table 9: Typical action sequences (traces) for different question types. Superscript ∗ denotes repetition (0 to T −1 times). Note that Bypass and Zero-shot have fixed traces. Policy
Sources
Typical trace
TS + search Market + search Wiki + search Search-only Bypass Zero-shot
yfinance, FRED Polymarket, Manifold Wikipedia Metaculus, RFI, ACLED DBnomics (ablation)
c → b∗ → x h → b∗ → x w → b∗ → x b∗ → x m→x x
The set of tools (actions) available to the agent depends on the type of question it is tackling, as shown in table 8. Denote this set by Aq . At each step t, the agent’s policy picks the action at = π(mt−1 ; Aq , L), where mt−1 is the incoming message history. In our case, the policy is implemented by the LLM L, as discussed in section B.2. In addition, the LLM generates the next belief state, (at , bt ) = π(mt−1 ; Aq , L); we can think of bt as an internal “mental” action, as opposed to at which is sent to the external environment. We call the sequence of actions generated by this process a policy trace, τ = (a1 , . . . , aT ), where at ∈ Aq , |τ | ≤ T , and aT = x (the submit/stop action). See table 9 for some typical traces. (Note that the agent can choose to stop at any time, but we also impose an upper bound of T = Tmax = 10 steps.) 16
Some traces do not make much sense. For example, calling read-files before browse-web has no effect, since no files have been retrieved by the search tool. In addition, one or more URLs are often provided in the resolution criteria, so it makes sense to initially perform url-lookup, before following up with optional browse-web calls, since we know that these URLs are relevant to the question, but web search may just add noise. Similarly, when a source-specific tool, such as wikipedia-fetch or combo-fetch, is available, it is usually best to call this first. We currently encourage this behavior by using source-specific prompt instructions (e.g., “call the FRED tool as your first action”), but we could enforce it more rigorously by using a meta-controller, which determines which set of actions are available at each step. We denote this by Aq,t = π2 (sq , t), where π2 is the second-level policy, indexed by source type sq and step t. This can be viewed as a form of agentic harness (c.f., [Lou et al., 2026, Lee et al., 2026]). Currently we use a very primitive form of meta-controller. For DBnomics questions, we set Aq,1 = {m} and Aq,2 = {x}; this forces the agent to first call the domain-specific modeling tool and then submit its (model-based) estimate, skipping all further steps. For all other question types, we use Aq,t = Aq = Tsq , so the base LLM can choose any action it wants in any order, limited only by the set of tools applicable to that source. However, since π2 depends only on (sq , t), it can be represented as a simple lookup table, which could be easy to learn, e.g., via offline cross-validation on backtesting data, or online via a bandit algorithm. (For questions that do not come from a well-defined source (i.e., for anything beyond the FB dataset), we can replace sq with cq , which is class label that we estimate for the question, as shown in table 22. We leave learning the meta-controller to future work. B.7
Crowd estimate and empirical priors
We optionally provide the LLM with an informative starting estimate πq , controlled by two independent flags: Crowd signal (crowd=0/1). For market questions, ForecastBench provides a crowd estimate πq = mq : the market price (Polymarket, Manifold, RFI) or Community Prediction (Metaculus). This is the most powerful single input for market questions: simply predicting p̂q = mq achieves BI ≈ 90.6 (Table 23). The crowd flag has no effect on dataset questions (no market price exists). Empirical prior (emp=0/1). For dataset questions, no crowd estimate exists. Instead, we use the empirical base rate for each source and question subtype as πq . This is computed from all available ForecastBench questions (not just our evaluation tranches), approximating what an online learner would quickly converge to. Table 10 shows the priors. The emp flag has no effect on market questions (which use the crowd signal instead). Our default configuration uses crowd=1, emp=1, providing informative anchors for all question types. Our core ablations (section 4) set both to zero (crowd=0, emp=0), using an implicit prior of 0.5 for all questions. The mixed setting crowd=1, emp=0 matches the ForecastBench zero-shot baseline setup, where only the market price (“freeze value”) is provided. Table 10: Empirical priors πq for dataset questions, computed from all ForecastBench questions. The different question subtypes are explained in section G. Source
Question subtype
ACLED ACLED Wikipedia Wikipedia Wikipedia Wikipedia FRED yfinance DBnomics
10× spike Any increase Vaccine FIDE Elo ≥1% FIDE rank Swimming WR (all) (all) (all)
Prior πq 0.00 0.23 0.00 0.01 0.68 0.99 0.42 0.58 0.56
These priors are strong baselines for sources with extreme base rates: for example, predicting πq = 0 for ACLED 10× spike questions (section G.8) or Wikipedia vaccine questions (section G.9) is nearly optimal. On dataset questions, simply submitting these priors achieves BI of 65.2; adding zero-shot LLM reasoning barely helps (65.6). Our full system (BLF) improves this to 69.8, a gain of +4.6 17
over the prior-only baseline (see Figure 6). Note that the crowd and emp signals are not always available (e.g., AIBQ2 has neither market prices nor empirical priors), so we also evaluate without them (crowd=0, emp=0; section 4). B.8
Multi-Trial Aggregation
We run K independent trials per question (typically K = 5), each producing a forecast pk ∈ (0, 1). Sometimes these can differ quite a lot, based on the particular search queries performed during each trial (see fig. 2).
Figure 5: (a) BI loss is linear in |p − o|; BS is quadratic. (b) All three scoring rules as functions of prediction error. MBS diverges for confident wrong predictions. (c) Jensen’s inequality: averaging two predictions with errors e1 , e2 reduces BS (convex gap) but not BI (linear, no gap).
The default aggregation is the arithmetic mean: K
p̂ =
1 X pk . K
(3)
k=1
Whether averaging helps or not depends on the metric we use for evaluation. In particular, we will see that while it probably helps BS and MBS metrics, it does not make any difference to BI! This can be explained by the fact that BI is linear in the error, |p − o|, whereas MBS and BS are convex (see Figure 5(a-b)). To see whyP this matters, consider K forecasts p1 , . . . , pK for a question with outcome 1 2 o, averaging gives p̄ = K k pk . By Jensen’s inequality applied to the convex function f (e) = e : BS(p̄) = (p̄ − o)2 ≤
1 X (pk − o)2 = BS(pk ) K
(4)
k
so averaging improves expected BS (and similarly for the log-based MBS). However, for the linear BI loss |p − o|, it is usually the case that the following holds:10 |p̄ − o| =
1 X |pk − o| K
(5)
k
So averaging does not improve expected BI, as indicated by the lack of “Jensen gap” in Figure 5c. B.9
Shrinkage aggregation
We also explored hierarchical shrinkage as a way to aggregate results. This is inspired by James–Stein estimation [Stein, 1956, Efron and Morris, 1973], which is a form of empirical Bayes estimation for a certain hierarchical model, as we explain below. 10 The equality holds whenever all p are on the same side of o (i.e., all p > o or all p < o), which is the typical case for k k k binary outcomes, where most questions have either very low or very high probability. Only when some pk straddle o would averaging help BI — but that’s rare in practice.
18
Model. We model the K trial forecasts in logit space. Let yk = logit(pk ) and let θ denote the “true” logit-forecast. The likelihood is yk | θ ∼ N (θ, σ 2 ),
k = 1, . . . , K,
(6)
2
where σ is estimated as the sample variance of the logit-transformed forecasts. The prior is θ ∼ N (0, τ 2 ), centered at logit(0.5) = 0. By standard calculations for linear Gaussian models, the posterior is θ | y1:K ∼ N (m, v) with m = αȳ, 1 where ȳ = K
α=
Kτ 2 , Kτ 2 + σ 2
(7)
P
k yk .
Prediction. Given the posterior mean E[θ|y1:K ] = αȳ, we compute the following plugin approximation to the posterior predictive distribution: p̂ = sigmoid (αȳ)
(8)
where sigmoid(x) = 1/(1 + e−x ) is the sigmoid function. When trials agree (σ 2 small), α ≈ 1, so we are averaging in logit space; when trials disagree, p̂ is pulled toward 0.5. Practical implementation. The downside of the above approach is that it relies on choosing a good prior variance τ 2 . Rather than using an exact empirical Bayes approach, that computes the ML-II estimate of τ given the data, we use a simple approximation to directly estimate the shrinkage coefficient: α ≈ max(f, 1 − c · s), where s = std(y1:K ), f = 0.3 (floor), and c = 0.7 (scale). This adapts the effective τ 2 per question based on local disagreement, and the floor prevents complete collapse to 0.5. The constants were tuned on AIBQ2. Connection to James–Stein. This method is analogous to James–Stein shrinkage [Stein, 1956, Efron and Morris, 1973], which shrinks individual estimates toward a common mean to reduce total MSE. Unlike classical James–Stein, our setting optimizes for log-scoring rules where overconfident wrong predictions are catastrophically penalized. Empirical results. Shrinkage improves Metaculus Score on AIBQ2 (+1.6 points) by reducing catastrophic overconfident predictions, but hurts Brier Index on both AIBQ2 (−0.032) and ForecastBench (−0.043). To see why, note that shrinkage pulls all predictions toward 0.5, which increases |p − o| for predictions that are already on the correct side of 0.5. Since most forecasts are approximately correct (i.e., p is on the same side as o), shrinkage increases the average |p − o| and thus hurts BI, even though it helps BS.11 B.10
LLM aggregation
We briefly experimented with the LLM-based aggregation method from the AIA Forecaster [Alur et al., 2025], which combines the reasoning traces (but not the probability estimates) from all K = 5 trials to a final aggregation LLM (that can also perform its own search), but it worked worse than simple mean aggregation on our FB benchmark. B.11
Calibration
We apply Platt scaling [Platt, 1999] as a post-processing calibration step: p̂cal = σ a · logit(p̂) + b ,
(9)
11 To see why shinkage helps BS but not BI, consider a question with outcome o = 0 and a high-variance set of trial forecasts pk ∈ {0.1, 0.1, 0.1, 0.1, 0.8}. We consider two aggregation schemes: Arithmetic mean: p̂ = 0.24. BS = 0.242 = 0.058; BI = 1 − 0.24 = 0.76; and Shrinkage (high std ⇒ low α): p̂ ≈ 0.44. BS = 0.442 = 0.194; BI = 1 − 0.44 = 0.56. Shrinkage moves p̂ from 0.24 to 0.44, hurting both BS and BI in this example. However, consider the expected BS across random trial subsets: without shrinkage, the outlier trial (p = 0.8) contributes 0.82 = 0.64 to BS when selected — a catastrophic penalty under the quadratic loss. Shrinkage dampens this outlier’s impact, reducing the variance of BS across subsets, which improves expected BS (by Jensen’s inequality applied to the convex function f (e) = e2 ). For BI, the linear loss |p − o| has no such convexity benefit: the outlier contributes |0.8 − 0| = 0.8, and shrinking it to 0.44 saves only 0.36, while the correct trials are each pushed from |0.1| to |0.44|, costing 4 × 0.34 = 1.36. The net effect on BI is negative.
19
where a and b are fitted by minimizing log loss on held-out data. We use leave-one-out crossvalidation (LOO CV) to avoid overfitting on small datasets: for each question, the model is fit on all other questions and applied to the held-out one. In our backtesting experiments, LOO-CV ensures that the calibration model never sees the question it is calibrating. For deployment on new unlabeled questions, one would use a calibration model trained on the full backtesting dataset (i.e., all of A∪B). Our LOO-CV estimates approximate this regime, since each fold trains on n−1 of n = 400 questions. For ForecastBench, which spans 9 heterogeneous sources (Table 23), we also evaluate hierarchical Platt scaling: a shared slope a and intercept b, plus per-source offsets δs : p̂cal = σ a · logit(p̂) + b + δs . (10) P 2 When fitting, we add L2 regularization, λ s δs , to the NLL. The per-source offsets account for systematic miscalibration that varies across question types (e.g., the model tends to be overconfident on polymarket questions but underconfident on dbnomics). On ForecastBench (Table 5), hierarchical calibration consistently outperforms global Platt scaling across all settings, achieving +2.4 vs +2 BI without crowd/emp, and +3.1 vs +2.6 with crowd. The advantage is most pronounced when the empirical prior (section B.7) is enabled: global Platt barely helps (−0.2 BI), because it over-shrinks the well-calibrated extreme predictions produced by the prior for sources with skewed base rates (e.g., πq ≈ 0 for Wikipedia vaccine questions). Hierarchical calibration avoids this by learning per-source offsets, yielding +3.5 BI.
C
ForecastBench Detailed Results
In this section, we present our results on ForecastBench (tranches A and B). See table 11 for details of the default system which we use (except for ablations). Table 11: Default BLF configuration on each benchmark. AIBQ2 has no crowd signal, empirical prior, or source-specific tools (all questions are Metaculus, search-only). DBnomics bypasses the LLM entirely (and uses a custom tool that implements a simple KNN estimator for the probability the temperature will be higher than the reference day).
C.1
Component
AIBQ2
ForecastBench
LLM Thinking level Search engine Source tools Belief state Agent mode Crowd signal Empirical prior Trials Aggregation Calibration DBnomics
Gemini-3.1-Pro high Brave (none available) structured sequential (Tmax = 10) off (not available) off (not available) K=5 plain mean Platt (LOO-CV) n/a
Gemini-3.1-Pro high Brave yfinance, FRED, Wikipedia; polymarket, manifold structured sequential (Tmax = 10) market price for market Qs base rate for dataset Qs K=5 plain mean hier. Platt (per-source, LOO-CV) KNN bypass (no LLM)
FB: Comparison to SOTA
Tables 12–15 and Figures 6–9 compare our system to leading methods (and various strong baselines) on ForecastBench (tranches A∪B), showing BI, ABI, and MBS metrics for market, dataset, and overall questions. Our method achieves the best results on all metrics across all question types. Our gains on market questions are particularly noticeable: we get BI of 94.8, whereas the best rival method (Foresight-32B) gets 91.4, which is barely above the baseline approach of simply returning the crowd estimate, which gives 90.6. Note that the ZS+crowd+emp baseline is approximately equivalent to the “Gemini-3-Pro-Preview (zero shot with crowd forecast)” entry on the FB leaderboard, since both use the same base model and zero-shot prompt with crowd signal. Table 16 extends the paired comparisons from Table 1 with separate ∆ BI for market and dataset questions. A striking finding is that no external method significantly beats the crowd+emp baseline on market questions: Cassi (−0.3, n.s.), GPT-5 (+0.3, n.s.), and Foresight (+0.6, n.s.) are all 20
Figure 6: Comparison to external methods and baselines on FB A∪B in terms of Brier Index, split by Overall, Market, and Dataset. Error bars: 95% bootstrap CI. See Table 12 for numeric values. Table 12: Brier Index comparison on FB A∪B (n=791 resolution dates from 400 questions). ∗ FB leaderboard; †
partial overlap. 95% bootstrap CIs in brackets. See fig. 6 for visualization. BI ↑ Method BLF +crowd+emp+cal Cassi∗ GPT-5 ZS+freeze∗ Grok 4.20∗† Foresight-32B∗†
Mkt
Dat
All
94.8 [93,97] 72.2 [70,74] 83.5 [82,85] 90.4 [88,92] 68.6 [67,71] 79.5 [78,81] 90.9 [88,93] 69 [67,71] 79.9 [78,82] 89.3 [86,93] 68.3 [66,71] 78.8 [77,81] 91.4 [89,94] 67.1 [64,70] 79.2 [77,81]
Crowd+emp (no LLM) 90.6 [88,93] 65.4 [64,67] 78 [77,80] ZS+crowd+emp 87.7 [85,90] 61.8 [58,66] 74.7 [73,77]
indistinguishable from simply returning the market price, while Grok is significantly worse (−2∗ ). Only BLF significantly outperforms the crowd on market questions (+4.2∗∗∗ ), demonstrating the value of agentic search and hierarchical calibration on top of the crowd signal. On dataset questions, all LLM methods significantly beat crowd+emp, with the exception of Foresight (+0.1, n.s.). C.2
FB: Per-source analysis
Figure 10 breaks down BI by source, and compares to the top two methods on FB leaderboard (at the time of writing). We see that all methods are at chance on yfinance, due to the random walk nature of stock prices (see section G.5). All methods struggle with FRED (Federal Reserve Economic Data) for similar reasons (see section G.6), although our method has an edge, likely due to the simple statistical model we use (eq. (20)), whose estimate is made available to the agent to supplement the raw data when the fetch-time-series tool is called. Our method has an edge on DBnomics for similar reasons: the agent uses a simple KNN-based probability estimator, bypassing the LLM entirely, since we found the LLM struggles with seasonal data such as temperature (see the ablation study in table 24). All methods do quite well on ACLED and Wikipedia, because the questions are quite easy, since the base rates are so extreme (see section G.8 and section G.9). On market questions, we outperform other methods by a large margin, which struggle to beat the prior. The sources of these gains are studied below, in section 4, and section 4 and section C.5. C.3
FB: Core Ablations
To try to identify the key components of our system that really matter, we developed a fractional factorial design of 11 configurations on n=400 questions, each run with 5 independent trials. The MBS and BI metrics for each of these configurations is shown in table 2. The reference system is 21
Figure 7: Same as fig. 6, but showing Adjusted Brier Index. See Table 13 for numeric values. Table 13: Adjusted Brier Index (ABI) comparison on FB A∪B. ABI uses question fixed effects to adjust for difficulty (see section A.2). ∗ FB leaderboard; † partial overlap. 95% approximate CIs in brackets. See fig. 7 for visualization. ABI ↑ Method BLF +crowd+emp+cal Cassi∗ GPT-5 ZS+freeze∗ Grok 4.20∗† Foresight-32B∗†
Mkt
Dat
All
84.7 [82,88] 62.0 [59,65] 71.0 [68,74] 82.0 [79,85] 59.6 [57,63] 68.7 [66,72] 80.4 [77,83] 60.1 [57,63] 68.5 [66,71] 77.6 [75,81] 61.4 [58,65] 68.4 [65,72] 82.3 [79,85] 57.6 [54,61] 67.5 [64,71]
Crowd+emp (no LLM) 81.5 [79,84] 58.3 [55,61] 67.7 [65,71] ZS+crowd+emp 78.6 [76,82] 51.9 [49,55] 62.8 [60,66]
BLF (without calibration); all ablations below the line remove or change one component. Calibration is evaluated separately in section C.5. We use crowd=0, emp=0 for all models, to isolate the effect of each component without the crowd/emp signal masking differences. The point estimates in table 2. do not account for the variance in these estimates, which is dominated by question difficulty (which accounts for 62.2% of total variance; see the ANOVA in section F.1), so small differences may not be significant. To estimate the marginal effect of each component while controlling for question difficulty, we fit a linear additive model that covers both core system and post-processing ablations. BIij = β0 + βsearch xsearch,j + βtools xtools,j + βcrowd xcrowd,j + βthink xthink,j + βmodel xmodel,j + βagentic xagentic,j + βbelief xbelief,j + βcal xcal,j + βntrials xntrials,j + ui + εij (11) where i indexes questions, j indexes configurations, ui ∼ N (0, σu2 ) is a random intercept capturing question difficulty, xagentic distinguishes sequential multi-step reasoning from batch parallel search (batch), and xbelief distinguishes structured belief tracking from raw text accumulation (nobelief). (See section F.2 for more details on paired analysis.) Table 3 show component effect sizes after controlling for question difficulty. (See also fig. 11.) The most notable result is the importance of the Bayesian linguistic belief state (Section B.1): removing it degrades BI by −5.1 ∗∗∗ , an effect larger than removing web search (−3.4) or source-specific tools (−2.6). This confirms that the semi-structured belief representation — mixing probability estimates with natural-language evidence summaries — provides a substantial advantage over simply accumulating retrieved text in the context window. With n=400, all components except thinking budget are individually significant. The crowd signal significantly improves BLF (+1.7). Among degradations, switching to batch mode (−7.7) and using 22
Figure 8: Same as fig. 6, but showing Metaculus Baseline Score (MBS). See table 14 for numerical values. Table 14: Metaculus Baseline Score (MBS) comparison on FB A∪B (n=791 resolution dates from 400 questions). ∗ FB leaderboard; † partial overlap. 95% bootstrap CIs in brackets. See fig. 8 for visualization. MBS ↑ Method BLF +crowd+emp+cal Cassi∗ GPT-5 ZS+freeze∗ Grok 4.20∗† Foresight-32B∗†
Mkt
Dat
All
88.3 [81,94] 39.9 [35,45] 64.1 [59,69] 81.5 [75,87] 33.1 [28,38] 57.3 [53,61] 78.5 [69,86] 35.4 [31,40] 56.9 [52,62] 72.2 [57,84] 37.2 [32,43] 54.7 [47,62] 81.4 [70,90] 27.6 [19,36] 54.5 [47,61]
Crowd+emp (no LLM) 80.8 [74,87] 29.4 [25,34] 55.1 [51,59] ZS+crowd+emp 75.5 [69,83] 9.0 [−4,21] 42.2 [36,49]
weaker models (Kimi −5.9, Flash −2.3) significantly hurt. Thinking budget (med vs high) remains the only non-significant factor (∆ = +0.2, p = 0.20). C.4
FB: Aggregation Ablations
In this section, we evaluate various different multi-trial aggregation ablations on FB. The results are shown in Table 4. Several patterns are visible. • More trials help MBS and BS consistently (fig. 12a–b), with no sign of diminishing returns up to k = 5. BI is flat for plain mean (fig. 12c), as predicted by the theory in section 2. • Median aggregation yields higher BI than mean (+0.2, p < 0.001), because it is more robust to outlier trials. However, mean and median achieve identical MBS (54.2), and on the full system with calibration (Table 5), calibration dominates the aggregation choice. We use mean as our default because it is the Bayes-optimal aggregator under the log score (Metaculus score) used by AIBQ2. • Shrinkage hurts both metrics on FB (MBS: 50.4 vs 54.2; BI: 75.3 vs 78.2), consistent with the AIBQ2 BI result but unlike the AIBQ2 MBS result. The AIBQ2 MBS improvement from shrinkage appears to be specific to that dataset’s difficulty distribution. C.5
FB: Calibration Ablations
In fig. 13, we show that (hierarchical) Platt scaling can help reduce the Expected Calibration Error on one particular model (Gemini-3-Flash). But how does ECE translate to prediction metrics? Table 17 shows the full comparison across all four base model settings (the main text Table 5 shows only the top and bottom blocks). Hierarchical calibration consistently outperforms global Platt across all settings. 23
Figure 9: Same as fig. 6, but showing Brier Score (BS = (p − o)2 , lower is better). Multi-trial averaging reduces BS via Jensen’s inequality, giving BLF a larger advantage on this metric. See table 15 for numerical values. Table 15: Brier Score (BS = (p − o)2 , ×100, lower is better) comparison on FB A∪B (n=791 resolution dates from 400 questions). Because BS is a quadratic (convex) loss, multi-trial averaging reduces BS via Jensen’s inequality, amplifying the advantage of BLF’s K=5 trials. ∗ FB leaderboard; † partial overlap. 95% bootstrap CIs in brackets. See fig. 9 for visualization. BS×100 ↓ Method BLF +crowd+emp+cal Cassi∗ GPT-5 ZS+freeze∗ Grok 4.20∗† Foresight-32B∗†
Mkt
Dat
All
2.2 [1,4] 14.1 [13,16] 8.2 3.2 [2,5] 16.3 [15,18] 9.8 3.9 [2,6] 15.9 [15,17] 9.9 5.0 [2,8] 14.9 [13,16] 10.0 3.1 [1,5] 17.9 [16,20] 10.5
Crowd+emp (no LLM) 3.4 [2,5] 17.4 [16,19] 10.4 ZS+crowd+emp 4.7 [3,7] 23.1 [20,26] 13.9
The advantage is most pronounced for the zero-shot baseline with emp=1 (bottom block): global Platt provides no improvement (−0.2, n.s.), because it over-shrinks the extreme predictions produced by the prior for sources with skewed base rates. Hierarchical calibration avoids this by learning per-source offsets, yielding a large improvement (+3.5∗∗∗ ), with dataset BI jumping from 61.8 to 65.9. Comparing the two emp=1 blocks also shows the value of search and tools: BLF+crowd+emp achieves 81 uncalibrated vs 74.7 for ZS (+6.3). Our best system (BLF+crowd+emp+cal) achieves 83.5 overall BI. C.6
FB: Behavioral Analysis
In this section, we examine the agent’s behavior: how it uses tools, how many steps it takes, and how its beliefs evolve across trials. Tool usage. Figure 14 shows the mean number of tool calls per question, broken down by source. Web search and the filter/lookup pipeline (section B.3) dominate across all sources. Source-specific data tools (market info, Wikipedia sections, time-series fetchers) are used selectively — only for their respective sources. DBnomics questions are handled entirely by the KNN model (section H.1) with no LLM tool calls. Polymarket questions use the most tools (∼5.5 per question), reflecting the additional market-info fetch on top of web search. Agent steps. Figure 15 shows the distribution of agent loop iterations per question. Most sources use 4–7 steps, with Polymarket requiring the most research (mean 6.7 steps) and FRED the least 24
Table 16: Extended paired SOTA comparisons on A∪B, with separate ∆ BI for market, dataset, and overall. ∆ > 0 means the first method is better. n = common resolution dates. ∗∗∗ p < 0.001; ∗∗ p < 0.01; ∗ p < 0.05 (bootstrap, 5,000 resamples). Mkt
Dat
All
∆
∆
∆
790 +4.4 ∗∗∗ 790 +3.8 ∗∗∗ 423 +5.9 ∗∗∗ 417 +4 ∗∗∗ 790 +4.2 ∗∗∗
+3.5 ∗∗∗ +3.2 ∗∗∗ +4.3 ∗∗∗ +4.6 ∗∗∗ +6.7 ∗∗∗
+4 ∗∗∗ +3.5 ∗∗∗ +5.1 ∗∗∗ +4.3 ∗∗∗ +5.4 ∗∗∗
External methods vs Crowd+emp baseline Cassi vs Crowd+emp 791 −0.3 GPT-5 vs Crowd+emp 791 +0.3 ∗ Grok† vs Crowd+emp 423 −2 Foresight† vs Crowd+emp 418 +0.6
+3.2 ∗∗∗ +3.5 ∗∗∗ +4.4 ∗∗∗ +0.1
+1.5 ∗∗ +1.9 ∗∗∗ +1.2 +0.4
Pairwise between external methods Cassi vs GPT-5 791 −0.6 Cassi vs Grok† 423 +0.4 Cassi vs Foresight† 418 +0 GPT-5 vs Grok† 423 +2.1 GPT-5 vs Foresight† 418 −0
−0.4 +1.3 +0.4 +0.9 +1.6
−0.5 +0.9 +0.2 +1.5 +0.8
Comparison BLF vs external methods BLF vs Cassi BLF vs GPT-5 BLF vs Grok† BLF vs Foresight† BLF vs Crowd+emp
n
∗
∗ ∗∗
∗
† A-only (Grok) or B-only (Foresight); n reflects partial overlap. Grok vs Foresight omitted (n=50). All ∆ values are BI
×100.
among LLM-based sources (mean 3.8), since the time-series tool provides most of the signal directly. DBnomics always uses exactly 1 step (KNN bypass). The agent rarely exhausts all 10 steps, suggesting the stop action (section B.3) is learned effectively. Belief evolution. Figure 2 shows how the agent’s probability estimate evolves across steps for 5 independent trials on a single AIBQ2 question. All trials start at p0 = 0.5 (no information) but quickly diverge as different search queries lead to different evidence. The high inter-trial variance (σ = 0.20) motivates multi-trial aggregation: the mean (0.57) is closer to the true outcome than most individual trials. Figure 3 shows a detailed trace for Trial 2 of the same question, illustrating how the belief state captures key evidence and reasoning at each step.
D
AIBQ2 Results
In this section, we present our results on AIBQ2. See table 11 for details of the default system which we use (except for ablations). D.1
AIBQ2: Comparison with SOTA
Table 18 compares our system to the current SOTA on AIBQ2. Our best system (BLF, which uses a frozen base model) achieves 39.5 MS, compared to 45.8 by RL-fine-tuned models [Jeen et al., 2026]. Although our mean score is worse, our 95% bootstrap CI is (23,54), so there is a lot of uncertainty about our estimate (and presumably theirs, too), since the sample size is small. We also evaluate on FB (see section C), whose dataset is 4x larger, and there we find that we are SOTA by a statistically significant margin. Table 18 also compares our method to a “clairvoyant” upper bound, which does not use a date cutoff. The fact that this is so much better is yet another indication that our date filtering is working. It may seem surprising that the clairvoyant predictor does not score perfectly (100 MBS and BI); the reason is that the true outcome may be hard to find on the web, even without date filtering. Furthermore, sometimes even if you add the true outcome to the context of the LLM, it does not necessarily know 25
Figure 10: Brier Index by source on FB A∪B.
Table 17: Calibration method comparison across four base model settings on FB (A∪B, n=791). ∆ BI = paired difference vs uncalibrated baseline in each block. Top three blocks use search=brave, tools=1; bottom block uses search=none, tools=0 (zero-shot). ∗∗∗ p < 0.001 (bootstrap). BI ↑ Setting Calibration Mkt
Dat
All ∆ BI
BLF (crowd=0, emp=0) uncal 87 69.4 78.2 (ref) global-cal 89.3 71.1 80.2 +2 ∗∗∗ hier-cal 89.8 71.3 80.6 +2.4 ∗∗∗ BLF+crowd (crowd=1, emp=0) uncal 90 70 80 (ref) global-cal 93.1 72.1 82.6 +2.6 ∗∗∗ hier-cal 93.8 72.3 83.0 +3.1 ∗∗∗ BLF+crowd+emp (crowd=1, emp=1) uncal 91.9 70 81 (ref) global-cal 94.4 71.9 83.1 +2 ∗∗∗ hier-cal 94.8 72.2 83.5 +2.3 ∗∗∗ ZS+crowd+emp (crowd=1, emp=1, search=none, tools=0) uncal 87.7 61.8 74.7 (ref) global-cal 87.1 62 74.5 −0.2 hier-cal 90.5 65.9 78.2 +3.5 ∗∗∗
how to interpret it, and may still make an incorrect prediction. However, we leave further investigation into this anomaly to future work. D.2
AIBQ2: Core Ablations
Table 19 shows the effects of ablating pieces of the core system. We use this to rank-order the importance of the system components, listed below. Note that, while the CIs are large, due to the small sample size, we replicate many of these findings in our much larger (and hence more reliable) FB experiments below. 26
Figure 11: Effect sizes vs BLF without calibration (crowd=0, emp=0) on A∪B (n=400). Green = significant improvement, red = significant degradation, grey = not significant (p > 0.05, bootstrap). Error bars show 95% CI. Numeric values in Table 3.
Table 18: AIBQ2 comparison to external methods. BLF+cal = mean of 5 trials + Platt calibration (LOO CV). 95% bootstrap CIs in brackets. Method
MBS ↑
BI ↑
BS ↓
BLF +cal 39.5 [23,54] 73.8 [69,78] .136 [.10,.18] SOTA (RL) [Jeen et al., 2026] 45.8 — — clairvoyant† 71.7 [56,85] 89.3 [85,93] .058 [.03,.09] † Clairvoyant sees future data, by setting search cutoff date to the present.
• Model capacity is key: Flash scores only 32.3 MS (−7.2); Kimi scores 32.1 MS (−7.4). • Belief state is the most important algorithmic component: nobelief drops to 35.1 MS (−4.4). • Search is also critical: zs (no search) scores 36.7 MS (−2.8). • Thinking budget: medium thinking level (instead of the default high) scores 37.5 MS (−2.0). However, this difference vanishes on FB (section F.2). D.3
AIBQ2: Aggregation Ablations
In this section, we consider different ways to aggregate multiple trials. In particular we compare the simple mean in eq. (3) to the hierarchical shrinkage method using the standard-deviation estimator in eq. (8). The results are shown in fig. 16(a-c) and Table 20. We see that MBS and BS both improve significantly with more trials, and shrinkage aggregation slightly improves MBS over mean aggregation (though it hurts BI). For BI metric, things are more interesting. First we see that mean aggregation has no effect at all. Second we see that shrinkage aggregation hurts performance. See section B.8 for an explanation. D.4
AIBQ2: Calibration Ablations
From Table 20, we see that calibration significantly improves MBS by 4.7, rising from 34.8 to 39.5, and improves BS by 0.008, dropping from 0.146 to 0.136. However, we see that calibration slightly 27
Figure 12: (a–c) Effect of increasing the number of trials on MBS, BS, and BI (FB, A∪B, n=400, crowd=0, emp=0). Error bars = 95% CI over random trial subsets. MBS and BS improve with more trials; BI is flat for plain mean. Shrinkage hurts all three metrics on FB. (d) Aggregation method effects vs mean:5. Solid = ∆ BI; hatched = ∆ MBS.
Table 19: AIBQ2 core ablations. BLF includes Platt calibration. Each row below the line changes one component (shown in bold). 95% bootstrap CIs in brackets. Configuration
MBS ↑
BI ↑
BS ↓
BLF +cal (Pro/high/Brave) 39.5 [23,54] 73.8 [69,78] .136 [.10,.18] zs (no search) medthink (default think) kimi (Kimi-K2) batch (parallel) nobelief (−belief) flash (Flash)
36.7 [21,52] 72.4 [68,77] .142 [.10,.18] 37.5 [21,52] 72.7 [68,77] .142 [.11,.19] 32.1 [17,47] 70.2 [66,75] .154 [.11,.20] 36.1 [21,51] 72.3 [67,77] .144 [.11,.19] 35.1 [20,51] 71.9 [67,77] .146 [.10,.19] 32.3 [17,47] 70.3 [66,75] .153 [.11,.19]
hurts BI by 0.4, dropping it from 74.2 to 73.8. Note, however, that the CIs for all these estimates are wide, and on the much larger FB dataset in section C.5, we see that (hierarchical) calibration helps on all metrics.
E
Ensemble Analysis
A common strategy for improving forecasts is to ensemble diverse models. We tested greedy ensembles of {Pro, Kimi-K2, Flash} (all without crowd, using mean aggregation of all trials from all members, uncalibrated). 28
Figure 13: Reliability diagram for the Flash model on FB A∪B (n=791 resolution dates). Hierarchical Platt scaling (green, ECE=0.043) improves on raw predictions (red, ECE=0.055). Right: ECE comparison.
Figure 14: Mean tool calls per question by source for BLF (trial 1, A∪B). Source-specific tools are used selectively. DBnomics bypasses the LLM (KNN model).
Figure 17 shows MBS and BI as ensemble members are added. On ForecastBench, both metrics decline monotonically — adding weaker models hurts more than diversity helps. On AIBQ2, the picture is more nuanced: Pro+Kimi improves MS from 34.6 to 37.1 (Kimi brings genuinely different predictions, JSD=0.052), but BI still declines (74.2 to 71.4), consistent with our finding that BI does not benefit from averaging (section 2). Adding Flash as a third member hurts MBS back down to 35.0. The reason ensembling fails on FB is twofold: (1) the component models are highly correlated — they receive identical prompts, tools, and search results, differing only in the base LLM; (2) Kimi and Flash are individually weaker than Pro. Figure 18 shows the pairwise JSD: FB diversity is very low (0.006–0.014 bits) while AIBQ2 is moderately higher (0.028–0.052 bits), explaining the different ensemble behavior across datasets. 29
Figure 15: Agent steps per question by source (BLF, trial 1, A∪B). Diamond = mean; box = IQR. Polymarket: mean 6.7 steps; DBnomics: 1 step (KNN bypass).
Table 20: AIBQ2 post-processing ablations. 95% bootstrap CIs in brackets. MBS ↑
Variant
BI ↑
BS ↓
BLF (= mean:5, no cal)
34.8 [15,53] 74.2 [69,80] .146 [.10,.19]
+cal (Platt, LOO-CV) shrink:5 (floor=0.3) median:5 mean:3 mean:1 (single trial)
39.5 [23,54] 73.8 [69,78] .136 [.10,.18] 36.0 [22,49] 70.8 [66,75] .143 [.11,.18] 33.5 [12,53] 75.6 [70,81] .144 [.10,.19] 29.2 [8,49] 73.7 [68,79] .148 [.11,.19] 13.8 [−12,38] 74.2 [69,80] .160 [.12,.21]
F
ANOVA and Paired Analysis
F.1
Variance Decomposition
We perform a two-way ANOVA (method × question) on trial-level BI scores pooled across all 12 configurations in Table 2. Each observation is BIijt for method i, question j, trial t. The total sum of squares is decomposed as: SStotal = SSmethod + SSquestion + SSresidual where SSmethod = i ni (ȳi· − ȳ)2 sums over the 12 methods, SSquestion = over the 400 questions, and the residual captures inter-trial variability. P
P
j nj (ȳ·j − ȳ)
2
sums
The F -statistic tests whether the group means differ significantly: F = MSeffect /MSresidual , where MS = SS/df is the mean square. A large F (relative to the F -distribution with appropriate degrees of freedom) indicates the effect is significant. Table 21: Two-way ANOVA on trial-level BI scores (FB A∪B, 12 methods × 400 questions × 5 trials). Source Method (M =12) Question (Q=400) Residual Total
SS
% total
df
F
38.1 877.6 494.3
2.7% 62.2% 35.1%
11 399 23,442
164.5 104.3 —
1,410.1
100%
23,852
From table 21, we see that question difficulty dominates (62.2% of total variance), consistent with the observation that some questions are inherently easy (e.g., “Will a vaccine for Sepsis exist by next 30
Figure 16: (a-c). Effect of increasing the number of trials on AIBQ2, as measured by MBS, BS and BI metrics. Error bars represent 95% CI over random trial subsets. We also compare plain mean (gray) vs std-shrinkage (blue). MBS and BS improve with more trials, but BI does not, for reasons explained in section B.8. (d) Effects of core ablations. Error bars represent the 95% bootstrap CIs shown in table 20.
month?”) while others are genuinely hard (e.g., stock price movements). See also fig. 19, which shows that on AIBQ2, most questions are very easy, so performance differences are dominated by a subset of hard questions. Method effects account for only 2.7% of total variance, but are highly significant (F = 164.5, p ≪ 0.001), confirming that the differences observed in Table 2 are real, not noise. The residual (35.1%) captures inter-trial variability within each (method, question) pair — i.e., the stochasticity of the LLM’s search and reasoning. F.2
Paired Analysis
For the pairwise comparisons in Table 3, we use an additive two-way fixed-effects model:12 BIij = µ + αi + γj + ϵij where µP is the grand mean, αi is the effect of method i (with j (with j γj = 0), and ϵij is the residual.
(12)
P
i αi = 0), γj is the effect of question
12 In a true mixed-effects model, the question effects γ would be treated as random (γ ∼ N (0, σ 2 )) and marginalized out j j q when estimating the method effects αi . This is statistically preferable (it accounts for uncertainty in γj and enables inference on new questions), but requires iterative algorithms such as REML. Since our design is balanced (every method answers every question), the fixed-effects estimates coincide with the mixed-effects estimates for αi , so the simpler approach suffices.
31
Figure 17: MBS and BI vs ensemble composition on FB (left) and AIBQ2 (right). On FB, both metrics decline with more members. On AIBQ2, MBS improves with Pro+Kimi (diversity helps) but BI always declines (linear penalty is invariant to averaging). All ensembles use uncalibrated forecasts.
Figure 18: Pairwise Jensen-Shannon divergence between model forecasts on FB (left) and AIBQ2 (right). FB has very low diversity (JSD 0.006–0.014), explaining why ensembling does not help. AIBQ2 has higher diversity (0.028–0.052), especially Pro vs Kimi, enabling modest MBS improvement.
Estimation via alternating projections. squares (ALS) algorithm, which iterates: γj ←
We estimate (µ, {αi }, {γj }) using an alternating least-
1 X (BIij − µ − αi ) |Ij |
(13)
1 X (BIij − µ − γj ) |Ji |
(14)
i∈Ij
αi ←
j∈Ji
where Ij is the set of methods that answered question j, and Ji is the set of questions P answered by method i. This is equivalent to coordinate descent on the least-squares objective ij (BIij − µ − αi − γj )2 , and converges to the unique global minimum since the objective is convex and separable in the two sets of parameters. In our setting, every method answers every question (balanced design), so ALS converges in a single iteration and produces the same estimates as direct computation: αi = ȳi· − ȳ and γj = ȳ·j − ȳ. However, the ALS formulation generalizes naturally to unbalanced designs (e.g., if some methods fail on some questions). Pairwise comparisons. The pairwise P ∆ BI values in Table 3 are the estimated α̂i − α̂ref , which equal the paired mean differences n1 j (BIij − BIref,j ) since the question effects cancel. Bootstrap confidence intervals are computed by resampling questions (5,000 resamples). The p-value is 32
computed as follows: for methods worse than the reference (∆ < 0), p is the fraction of resamples where ∆ ≥ 0 (i.e., the probability that the observed degradation is due to chance); for methods better than the reference (∆ > 0), p is the fraction of resamples where ∆ ≤ 0. We report significance using the convention ∗∗∗ p < 0.001, ∗∗ p < 0.01, ∗ p < 0.05. This approach is preferable to comparing raw means (Table 2) because it accounts for the fact that 62% of variance is due to question difficulty. Without pairing, a method that happens to be tested on easier questions might appear artificially good. By pairing on questions, the paired analysis isolates the true method effect and dramatically reduces the variance of the comparison.
Figure 19: Top: MBS for all n = 113 AIBQ2 questions (most are easy). Bottom: Zoom into questions at or below chance. Error bars: bootstrap 95% CIs across 5 trials.
G
Analysis of the Datasets
G.1
AIBQ2
The AIBQ2 benchmark consists of 113 binary questions from the Metaculus AI Benchmark Tournament (Q2 2025). Questions were asked between 2025-04-22 and 2025-06-15, with resolution dates up to 2025-07-01. The base rate is 22% (25 True, 88 False). Figure 20 shows the forecast horizon distribution: most questions have short horizons (1–8 weeks), with many resolving on 2025-07-01 (the tournament deadline). See also fig. 21 for another visualization of this data. Table 22 shows some example questions, which we have clustered into 10 manually chosen topics. (The clustering is done by applying an LLM classifier to each question, and then grouping questions with the same class label or tag into the same cluster). We see that the most common topics are Domestic Politics (35/113) and Geopolitics & Conflict (29/113). By contrast, the FB topic distribution is quite different (see fig. 22). 33
Figure 20: AIBQ2 date distribution of questions and answers. Green = resolved True (25), Red = resolved False (88).
Figure 21: AIBQ2 forecast horizon distribution (n = 113). Right panel shows outcome balance.
G.2
ForecastBench (overall)
In this section, we discuss some properties of the FB dataset. Table 23 shows some key statistics. Figure 22 shows the topic distribution: Financial Markets (76) leads, followed by Sports & Entertainment (69) and Geopolitics & Conflict (66), with the remaining 7 categories more evenly distributed. Figure 10 shows per-source BI for four methods. Performance of our best system (BLF+crowd+emp+cal) varies dramatically across sources: Wikipedia (96.8) and Manifold (93.5) are near-perfect, while yfinance (49) is essentially chance. FRED (56.4) is the second-weakest; improving it to match DBnomics (69.2) would raise overall BI by +1.3. We analyze each source in more detail below. G.3
FB: Market sources
Market questions (Polymarket, Manifold, Metaculus, RFI) are judgemental forecasting questions similar in nature to AIBQ2 (see Table 22 for example topics), but spanning a wider range of sources and with a different topic distribution (Figure 22). Forecast horizons range from 1 week to 5 months (Figure 23). 34
Table 22: AIBQ2 topic distribution with example questions across 10 categories (n = 113). Domestic Politics dominates (35 questions). Category
n
Example question
Domestic Politics Geopolitics & Conflict Business & Industry Society & Law Macroeconomics Science & Technology Health & Biology Financial Markets Other Sports & Entertainment
35 29 16 9 7 6 5 2 2 2
Will a state of emergency be in effect in Samoa on April 30, 2025? Will there be a successful coup in Africa or Latin America before Jul. . . Will Uber be available in the Turks and Caicos Islands on July 1, 2025? Will Harvard University lose its tax exempt status before July 1, 2025? Will South Sudan ratify AfCFTA (the pan-African free trade agreement). . . Will Starlink’s website show Lesotho as a country with Starlink avail. . . Will the CDC raise the Travel Health Notice status of Colombia to abo. . . Will any of these companies get dropped from the Dow Jones Industrial. . . Will WorldAtlas.com display the body of water northwest of the Caribb. . . Before July 1, 2025, will the International Traditional Karate Federa. . .
Total
113
Table 23: Question composition by source for FB Tranches A and B. “BR” = base rate (fraction resolving True). Market sources provide judgemental forecasting questions; dataset sources provide univariate time-series questions with multiple resolution dates per question. ACLED = Armed Conflict Location and Event Data. FRED = Federal Reserve Economic Data. DBnomics = database of international economic statistics. Source
URL
nA
nB
RFI Manifold Metaculus Polymarket
randforecastinginitiative.org manifold.markets metaculus.com polymarket.com
5 18 10 67
5 10 26 44 10 20 59 126
0.00 0.22 0.10 0.13
0.00 0.15 0.20 0.08
100 100 200
0.14
0.11
20 20 20 20 20
40 40 40 40 40
0.25 0.95 0.60 0.20 0.20
0.00 0.80 0.55 0.25 0.45
Dataset total
100 100 200
0.44
0.41
Total
200 200 400
0.29
0.26
Market total ACLED DBnomics FRED Wikipedia Yahoo Finance
acleddata.com db.nomics.world fred.stlouisfed.org wikipedia.org finance.yahoo.com
20 20 20 20 20
n BR(A) BR(B)
The base rates are very low (Table 23): Polymarket 11%, Manifold 18%, Metaculus 15%, RFI 0%. This means most questions resolve “No,”. Furthermore, these questions come with a market estimate; using this as the predicted probability achieves BI ≈ 90.6 — a strong baseline that is very hard to beat. Our system’s main advantage on market questions comes from calibration, which sharpens the already-good raw forecasts (Table 2). G.4
FB: Dataset sources
Dataset questions are univariate time series forecasting problems, which are converted to binary form by asking whether (some function of) the value of a quantity of interest at a future resolution date will be higher or lower than the value at the forecast date. Topics are very narrow, and depend on the source, as shown in Figure 22. Forecast horizons are fixed at 7, 30, and 90 days, as shown in Figure 24. We give more details on individual dataset sources below. G.5
FB: Yahoo Finance (stock prices)
Yahoo Finance questions ask whether a stock’s closing price on a future resolution date will be higher than its closing price on the forecast due date. Each question has 8 resolution dates ranging from 1 week to 10 years out. Figure 25 shows an example. The question text uses placeholders {resolution_date} and {forecast_due_date} that are substituted at preprocessing time. Crucially, the question does not specify the reference price on the 35
Figure 22: FB A∪B topic distribution across 9 sources.
Figure 23: FB market questions: forecast horizon distribution (n = 200). Most horizons are 1–5 months. Base rate is 12% (heavily skewed toward False). forecast due date — it only provides a freeze_datetime_value from an earlier date (when the question was created, typically 10 days before the forecast due date). For example, the CPB question below has freeze_datetime_value = 30.15 from 2025-10-16, but the actual closing price on the forecast due date (2025-10-24, since 10-26 is a Sunday) was 30.79. Our system retrieves the correct reference value via the fetch_ts_yfinance tool, which downloads the price history and returns the closing price on or before the forecast due date. Figure 25 illustrates why yfinance questions are inherently difficult: stock prices follow an approximate random walk, making it nearly impossible to predict whether the price will be higher or lower than the reference value on any given future date. The optimal strategy is to predict p ≈ 0.5, which our system correctly learns (yfinance BI ≈ 0.5 across both tranches; see Table 23). Example question (verbatim, abbreviated). { "forecast_due_date": "2025-10-26", // global "id": "CPB", "source": "yfinance", "question": "Will CPB’s market close price on {resolution_date} be higher than its market close price on {forecast_due_date}?", "resolution_criteria": "Resolves to the market close price at https://finance.yahoo.com/quote/CPB. If the resolution date coincides with a day the market is closed, the previous close is used.", "freeze_datetime": "2025-10-16T00:00:00+00:00", "freeze_datetime_value": "30.15", "resolution_dates": ["2025-11-02", "2025-11-25",
36
Figure 24: FB dataset questions: forecast horizon distribution (n = 200 questions, 591 resolution dates). Horizons are fixed at 7, 30, and 90 days (only the first 3 of 8 standard resolution dates have resolved by our evaluation cutoff). Base rate is 35%.
Figure 25: CPB (Campbell’s Company) closing price. Red dashed line: forecast date (2025-10-26). Star: actual price on 2025-10-24 ($29.94, since 10-26 was Sunday). Orange: freeze value ($30.15 from 10-16). Red diamonds: resolution date prices (all below reference, all resolved False). The decline to ∼$20 was unpredictable at forecast time.
"2026-01-24", ...], "resolved_to": [0, 0, 0, ...]
// all False
}
G.6
FB: FRED (economic indicators)
FRED (Federal Reserve Economic Data) questions ask whether an economic indicator will have increased by each resolution date compared to the forecast due date. An example question is shown below: Question (DTB6): “Will the Federal Reserve’s 6-month secondary market treasury bill rate have increased by {resolution_date} compared to its value on {forecast_due_date}?” Freeze value: 3.69% (as of 2025-10-16) Resolution dates: 2025-11-02, 2025-11-25, 2026-01-24, 2026-04-24, 2026-10-26, 202810-25, 2030-10-25, 2035-10-24
37
The 36 series in our tranches span interest rates (DTB6, DGS30), corporate bond spreads (BAMLC0A0CM), exchange rates (DEXUSUK), stock indices (DJIA), and banking data (DPSACBW027SBOG). Unlike yfinance (approximate random walk), many FRED series exhibit persistent trends. For example, Figure 26 shows the 6-month T-bill rate declining steadily from ∼5.25% to ∼3.5% over 2024–2026, reflecting Fed rate cuts. The first resolution date (2025-11-02) caught a brief uptick (resolved True), but subsequent dates resolved False as the rate continued falling. This trend is partially predictable from recent history, giving our fetch_ts_fred tool useful context. Figure 27 shows the USD/GBP exchange rate, which is harder to predict — more similar to a random walk, with a mix of True and False resolutions.
Figure 26: 6-Month Treasury Bill Rate (DTB6). Forecast date is 2025-10-26 (Sunday), so the reference value is from 2025-10-24 (red star). Clear downward trend from Fed rate cuts. Green diamond = resolved True (rate increased); red = resolved False.
Figure 27: USD/GBP exchange rate (DEXUSUK). Reference value from 2025-10-24 (red star, since forecast date 2025-10-26 is a Sunday). More erratic than interest rates — harder to predict. Overall, FRED has mixed predictability: of our 36 series, roughly 8 exhibit strong autocorrelation (trending), 18 are effectively random walks, and the rest are intermediate. Trend-following series (interest rates, banking aggregates) reward tools that detect recent direction, while noisy series (exchange rates, bond yields) are closer to chance. The combined base rate for FRED is 57% (Table 23), between DBnomics (88%) and yfinance (33%). 38
A natural improvement would be to classify each FRED series into a small number of patterns (random walk, trending, mean-reverting) and apply the appropriate statistical model — e.g., p = 0.5 for random walks, a trend-extrapolation model for persistent series. ForecastBench’s full question bank contains 165 unique FRED series, so this per-series approach would require a lightweight classifier, which we leave to future work. G.7
FB: DBnomics (temperature)
All DBnomics questions in ForecastBench ask whether the daily average temperature at a French weather station will be higher on a future resolution date than on the forecast due date. Our tranches contain 29 stations (24 metropolitan France, 5 overseas), as shown in Figure 28; each generates one question per forecast date with 8 resolution dates. Figure 29 shows ∼14 years of daily temperature data for one station (Mont-de-Marsan), exhibiting strong annual seasonality.
Figure 28: Météo-France weather stations used in ForecastBench. Stations are highly correlated (mean pairwise r = 0.93). Although the stations are spatially distributed, we found that pooling bias-corrected observations from nearby stations does not improve forecasts (+0.2 BI, negligible). This is because each station already has ∼200 historical observations per resolution date, so the per-station empirical exceedance estimate is already well-determined. The main bottleneck for dbnomics is near-horizon weather forecast data (e.g., 7-day forecasts), not statistical estimation from historical data. Figure 30 zooms into the period around the forecast date. The key challenge is the same as yfinance: the question provides a freeze_datetime_value of 14.4◦ C from 2025-10-16, but the actual value on the forecast date (2025-10-26) was 11.7◦ C — the threshold the agent must predict against. Our fetch_ts_dbnomics tool retrieves the correct value and computes an empirical exceedance probability (see Appendix H). Unlike yfinance (random walk), temperature has strong seasonal structure: the October-to-January decline is predictable from ∼14 years of historical data. Our empirical exceedance model achieves BI = 65.9 by computing P (temp > threshold) from same-calendar-date observations in prior years (see section H for model comparison). We bypass the LLM entirely for DBnomics, since LLM reasoning does not improve on the statistical model. Our system outperforms Cassi on this source (69.2 vs 59.3). Further improvement likely requires short-term weather forecast data for near-horizon resolution dates. Example question (verbatim, abbreviated). 39
Figure 29: Daily average temperature at Mont-de-Marsan, 2012–2025. Strong annual seasonality makes statistical forecasting effective.
Figure 30: Temperature around the forecast date for Mont-de-Marsan. Star: actual value on forecast date (11.7◦ C). Orange: stale freeze value (14.4◦ C). Green diamond: resolution date warmer than reference; red: colder. The seasonal decline from October to January is predictable from historical data.
{ "forecast_due_date": "2025-10-26", // global "id": "meteofrance_TEMPERATURE_celsius.07607.D", "source": "dbnomics", "question": "What is the probability that the daily average temperature at Mont-de-Marsan will be higher on {resolution_date} than on {forecast_due_date}?", "url": "https://db.nomics.world/meteofrance/ TEMPERATURE/celsius.07607.D", "freeze_datetime": "2025-10-16T00:00:00+00:00", "freeze_datetime_value": "14.438", "resolution_dates": ["2025-11-02", "2025-11-25", "2026-01-24", ...], "resolved_to": [1, 0, 0, ...] // True, False, False }
40
G.8
FB: ACLED (armed conflict)
ACLED (Armed Conflict Location & Event Data) questions ask whether the number of conflict events of a given type in a given country will exceed a threshold over a future 30-day window. There are two question templates (Figure 31): • Any increase (20 questions): will events exceed the baseline level? These have a 22% base rate. Example: Any increase: “Will there be more ‘Protests’ in Sri Lanka for the 30 days before {resolution_date} compared to the 30-day average of ‘Protests’ over the 360 days preceding 2025-11-09?” Resolution dates: 2025-11-16, 2025-12-09, 2026-02-07. Resolved: False, False, True (protest activity surged later).
• 10x spike (20 questions): will events in the next 30 days exceed ten times the 360-day rolling average? These have a 0% base rate — no country in our tranches experienced such an extreme spike. Example: 10x spike: “Will there be more than ten times as many fatalities in Finland for the 30 days before {resolution_date} compared to one plus the 30-day average of fatalities over the 360 days preceding 2025-11-09?” The “one plus” ensures the threshold is non-zero even when the baseline average is 0. Resolved False on all 3 dates.
Questions span 30 countries (from Antarctica to Venezuela) and 6 event types (Battles, Strategic developments, Riots, Violence against civilians, Explosions/Remote violence, Protests). No sourcespecific data tool is available for ACLED (the API requires special access), so the agent relies entirely on web search. The very low overall base rate (11%) means that predicting “No” for all questions is a strong baseline. The 10x-spike questions are particularly easy: since none resolved True, any well-calibrated system should predict near 0 for these. The agent’s main challenge is the “any increase” questions, which require understanding recent conflict dynamics via web search.
Figure 31: ACLED questions in A∪B. Left: event type distribution. Right: base rate by threshold type. The “10x spike” questions never resolved True.
G.9
FB: Wikipedia
Wikipedia questions ask whether specific facts on a Wikipedia page will change by the resolution date. Our 40 questions come from three articles: • List of infectious diseases (20 questions, BR=0%): “Will a vaccine have been developed for [disease] by [resolution date]?” Since vaccine development takes years, none of these resolved True over our 1–3 month horizons. 41
• FIDE rankings (15 questions, BR=22%): either “Will [player]’s FIDE ranking be as high or higher?” (5 rank questions) or “Will [player]’s Elo rating be at least 1% higher?” (10 Elo questions). The rank questions are easier (3/5 always True), while gaining 1% Elo in 1–3 months is very rare (0/10 True). See Figure 32. • List of world records in swimming (5 questions, BR=100%): “Will [swimmer] still hold the world record for [event]?” All resolved True (world records are rarely broken in a few months). The overall base rate is 21%, but it varies dramatically by article (0%, 22%, 100%). Our system achieves very high BI on Wikipedia (96.8; see Figure 10) because the questions are often answerable from the structure of the question itself: vaccine questions are almost certainly No, world record questions are almost certainly Yes. The fetch_wikipedia_toc/section tools allow the agent to verify these priors by checking the actual Wikipedia page content.
Figure 32: Left: FIDE question outcomes. Rank questions (top 5) mostly resolve True; Elo ≥1% questions (bottom 10) all resolve False. Wesley So (#8) is the only mixed case (True, False, False — his ranking briefly improved then dropped). Right: base rates for the three Wikipedia question types.
H
Time-Series Forecasting Models
In this section we discuss specialized statistical modeling tools which we optionally make available to the agent for certain data sources. These all return an estimate of p̂rq defined in eq. (1). These are all simple heuristic estimators that we developed to see if they helped the LLM with its task. Clearly many more sophisticated methods could be used, but that is not the focus of this paper. H.1
KNN model for DBnomics
Table 24: Time-series model comparison on Tranche A DBnomics questions (n=20 questions, 59 resolution dates). All models above the line are evaluated standalone (no LLM). “LLM + tool” shows the LLM agent’s performance when given the tool output and allowed to reason and search. Model
Mean BI
KNN (±10 days, 10+ years) Harmonic (linear + annual + semi-annual sinusoidal) Same-period prior years (Laplace-smoothed) Always 0.5 (baseline) Linear trend (no seasonality)
76.3 73.8 72.2 50 33.1
LLM + tool (agent reasons on data) Cassi (reference)
65.6 88.6
42
We tried several simple models for estimating p̂rq for DBnomics data, which we evaluated on the Tranche A DBnomics questions (n = 20 questions, 59 resolution dates). The best performing method, which we call KNN, is a non-parametric estimator which is analogous to a binary k-nearest-neighbor classifier, where “neighbors” are selected by day-of-year proximity. More precisely, define the neighbor set as follows: Na,w (q, r) = {yq (ti ) : |doy(ti ) − doy(r)| ≤ w, ti ≥ fq − a}, (15) where doy(·) extracts the day-of-year, a is the maximum age (to encourage focusing on data close to q’s forecast date), w is the window size around doy(r), and i indexes across all available data. The forecast is the Laplace-smoothed empirical exceedance frequency: |{yq (ti ) ∈ Na,w (q, r) : yq (ti ) > vq }| + 1 p̂knn . (16) q (a, w) = |Na,w (q, r)| + 2 This directly estimates eq. (1) without distributional assumptions. For DBnomics, we use a window of w = 10 days, and set a = ∞ to remove age restrictions, reflecting an assumption the periodic distribution is stationary. With ∼14 years of daily data, each resolution date typically has |Na,w (q, r)| ≈ 200 neighbors. The results of various models are shown in Table 24. We see that directly feeding the timeseries Hq and the estimate p̂rq into the LLM does not work well (BI is 65.6), and our KNN method is much better (BI is 76.3). We also considered a linear model, defined in eq. (18), as well as a linear model using basis function expansion to capture seasonality, but these were also inferior to KNN. However, on Tranche A, there is a gap between our best standalone model and the best external method. from Cassi (88.6). Closing this gap likely requires short-term weather forecast data for near-horizon resolution dates (e.g. 7-day forecasts for the first resolution date). We leave this to future work. H.2
Linear trend models for yfinance and FRED
For non-seasonal time series, such as yfinance and FRED, we use a parametric approach, which combines a local linear model with the baseline estimate of 0.5: psafe-linear (α) = α · plinear + (1 − α) · 0.5 (17) q q where α is a source-specific shrinkage factor controlling how much to trust the linear extrapolation. We set α = 0.1 for yfinance, which reflects that stock prices follow an approximate random walk, so the linear extrapolation is unreliable. We set α = 0.5 for FRED. (All α values were hand-tuned based on informal experimentation; systematic cross-validation is left to future work.) The linear probability estimate is given by vq − ŷ(r) (18) σ̂ where Φc is the standard normal survival function, and ŷ(t) = at + b is the linear model fit by OLS on the most recent W observations, H(sq , fq ; W ). (We use W = 30 for FRED and W = 60 for yfinance.) σ̂ is the standard deviation of the residuals for this data source sq . plinear = Φc q
H.3
Hybrid KNN and linear trend model for yfinance and FRED
For FRED and yfinance, we also consider a hybrid parametric and non-parametric estimate of the form p̂hybrid (α, β) = β · psafe-linear (α) + (1 − β) · pprior-year (19) q q q where pprior-year = pknn q q (a, w) uses the same nonparametric exceedance estimate as eq. (16), but with a narrower window (w = 7 days) and limited to the 5 most recent years, by setting a = 5 × 365 days. This captures annual periodicity: for a given resolution date, we check whether the value exceeded the threshold at the same time of year in prior years. Expanding out, this estimate combines 3 terms: p̂hybrid (α, β) q
+ (1 − β) pprior-year + (1 − α)β · 0.5 = αβ plinear q q
(20)
We set β = 0.5 for both FRED and yfinance when prior-year data is available (β = 1 otherwise, falling back to psafe-linear alone). For DBnomics, the hybrid model is not used — we apply the KNN model (eq. (16)) directly. 43
I
Leakage Analysis
Web search leakage. As mentioned in the main text, we use multiple layers of defense to minimize leakage of future information. The first layer is to add date range filtering to web search, so that we do not retrieve pages that are date stamped after the cutoff date, set equal to the forecast due date. Specifically, we use Brave Search, which supports a freshness parameter to restrict results to before the cutoff date. We evaluated other engines (Perplexity, AskNews, Exa, Serper) but they either did not support reliable date filtering, or returned only snippets (not full page text), which harmed downstream performance. Algorithmic date filtering at the search engine level is critical: without it, we would need to retrieve many more results knowing that the LLM-based classifier (layer 2) will discard most of them. Note that web search is controlled entirely through our own tool schemas passed via the function-calling API, so we can enforce date filtering. The core LLM could choose to “spontaneously” do search on its own without us asking it to; however, both Gemini and Anthropic models require explicit opt-in for built-in search [Google, 2025, Anthropic, 2025], and we do not enable it. Parametric knowledge leakage. Even without web search, the model’s training data may include events after the nominal knowledge cutoff. This is a fundamental challenge for backtesting LLM forecasters: the model may “know” outcomes it should not have access to. Our reasoning trace audit (section 2) scans chain-of-thought for mentions of post-cutoff events. On AIBQ2, Kimi-K2-Thinking (knowledge cutoff 2025-06-30) shows parametric leakage on two questions: • aibq2_0047 (cutoff 2025-05-08): “Will the USDA-posted recall [. . . ] be closed before June 1, 2025?” Kimi references the resolution date and specific timeframes that imply knowledge of the outcome (resolved True). • aibq2_0070 (cutoff 2025-05-16): “Will Ukraine announce a presidential election before July 1, 2025?” Kimi references the martial law extension to August 6, 2025 and the 357deputy approval — specific facts from after the cutoff date (resolved False; Kimi correctly predicts 0.037). Gemini-3.1-Pro (knowledge cutoff 2025-01-31) shows no parametric leakage on AIBQ2. On ForecastBench Tranche A, 3 of 99 aggregated reasoning traces contain post-cutoff information, but these originate from search results that passed through the runtime filter, not from parametric knowledge. Classifier error analysis. The runtime filter (Gemini-3-Flash) and post-hoc detective (Grok-4.1) are intentionally different models to reduce correlated errors. Both make mistakes: False positive (Flash too aggressive). A search result about historical weather patterns in Dijon mentioned “June 2025” in a generic seasonal context (e.g. “What’s the weather like in Dijon in June 2025?”). Flash classified this as a leak (“describes the weather in June 2025 which is after the March 2025 cutoff”), but Grok correctly recognized it as generic climatological information, not a post-cutoff observation. False negative (Flash too lenient). A search result about the Ethiopia conflict contained a link titled “Ethiopia situation update (30 April 2025)” while the cutoff was April 27, 2025. Flash kept the result because the content of the page did not include post-cutoff information — only the link title referenced a future date. Grok flagged this as a leak. These examples illustrate the fundamental difficulty of date-leakage classification: it requires understanding whether a date reference describes a specific observed event (leak) or a generic/scheduled reference (not a leak).
J
Prompts
This appendix lists the prompts used in our system and the zero-shot baseline. 44
J.1
System Prompt
The following is the system prompt used for all BLF configurations (abbreviated for space; sourcespecific tool descriptions omitted). You are an expert superforecaster. Your task is to predict the probability that a binary question will resolve to YES given information up to a certain date. You work in a tool-use loop: 1. Read the question, its resolution criteria, and background. 2. Form a base rate estimate (outside view / reference class reasoning). 3. Perform a loop: 3a. Choose ONE tool to call. 3b. After each tool call, your belief state is updated. 4. When you have gathered enough evidence, call submit(probability, reasoning). Belief state rules: - Evidence lists should ACCUMULATE across steps. - Each evidence item MUST cite its source (search_X_result_Y). - Include update_reasoning explaining WHY evidence changed your probability. - Consider RECENCY and AUTHORITATIVENESS of sources. Rules: - You MUST call submit before step {max_steps}. - Call submit once probability has stabilized. - Probabilities must be between 0.05 and 0.95.
J.2
Question Prompt
Each question is formatted as follows (with crowd/prior sections included only when crowd=1 or emp=1): # Question {question text} ## Background and resolution criteria {background} {resolution_criteria} ## Resolution dates {dates} You must submit {n} probabilities (one per resolution date). uncertainty should INCREASE with forecast horizon. ## Knowledge cutoff {cutoff_date} You must not use any information from after this date. ## Market estimate (if crowd=1, market questions only) The market estimate on {date} was {value}. ## Prior estimate (if emp=1, dataset questions only) {explanation}: {prior}. Use this as your starting point, but adjust based on question-specific evidence from search and tools.
J.3
Your
Zero-Shot Prompt
When search=none and tools=0 (zero-shot setting), we use a simplified prompt based on Halawi et al. [2024], which is also used by ForecastBench for its zero-shot leaderboard entries: You are an expert superforecaster, familiar with the work of Tetlock and others. Make a prediction of the probability that the question will be resolved as true. You MUST give a probability estimate between 0 and 1 UNDER ALL CIRCUMSTANCES. If for some reason you
45
can’t answer, pick the base rate, but return a number between 0 and 1. Question: {question} Question Background: {background} Resolution Criteria: {resolution_criteria} Question close date: {close_date} [If freeze value available:] The freeze value is {value}. {explanation} Output your answer (a number between 0 and 1) with an asterisk at the beginning and end of the decimal. Do not output anything else. Answer: {{ Insert answer here }}
References Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan, Santu Karmaker, and Aritra Dutta. FinTradeBench: A financial reasoning benchmark for LLMs. arxiv, 2026. URL https://arxiv. org/abs/2603.19225. Anthony Aguirre. A primer on the metaculus scoring rule, 2021. URL https://www.metaculus. com/notebooks/22486/a-primer-on-the-metaculus-scoring-rule/. Md Atik Ahamed, Mihir Parmar, Palash Goyal, Yiwen Song, Long T. Le, Qiang Cheng, Chun-Liang Li, Hamid Palangi, Jinsung Yoon, and Tomas Pfister. TFRBench: A reasoning benchmark for evaluating forecasting systems. arxiv, 2026. URL https://arxiv.org/abs/2604.05364. Rohan Alur, Bradly C. Stadie, Daniel Kang, Ryan Chen, Matt McManus, Michael Rickert, Tyler Lee, Michael Federici, Richard Zhu, Dennis Fogerty, Hayley Williamson, Nina Lozinski, Aaron Linsky, and Jasjeet S. Sekhon. Aia forecaster: Technical report. arxiv, 2025. URL https: //arxiv.org/abs/2511.07678. Anthropic. Web search tool. https://docs.anthropic.com/en/docs/agents-and-tools/ tool-use/web-search-tool, 2025. Houtan Bastani, Simas Kucinskas, and Ezra Karger. How well can large language models predict the future?, 2025. URL https://forecastingresearch.substack.com/p/ ai-llm-forecasting-model-forecastbench-benchmark. Cassi AI. Cassi: AI-powered forecasting. https://cassi-ai.com/, 2025. Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. OpenForecaster: Scaling open-ended reasoning to predict the future. arxiv, 2026. URL https://arxiv.org/ abs/2512.25070. Pu Cheng, Juncheng Liu, and Yunshen Long. PolyBench: Benchmarking LLM forecasting and trading capabilities on live prediction market data. arxiv, 2026. URL https://arxiv.org/abs/ 2604.14199. Bradley Efron and Carl Morris. Stein’s estimation rule and its competitors—an empirical Bayes approach. Journal of the American Statistical Association, 68(341):117–130, 1973. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc, 102:359–378, 2007. Google. Grounding with Google search. grounding, 2025.
https://ai.google.dev/gemini-api/docs/
Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou, and Juanzi Li. OpenEP: Open-ended future event prediction. arxiv, 2024. URL https://arxiv.org/abs/2408.06578. Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. arxiv, 2024. URL https://arxiv.org/abs/2402.18563. Elvis Hsieh, Preston Fu, and Jonathan Chen. Reasoning and tools for human-level forecasting. arxiv, 2024. URL https://arxiv.org/abs/2408.12036. 46
Scott Jeen, Matthew Aitchison, and Mantic. Training LLMs to predict world events. Thinking Machines Lab: News, 2026. URL https://thinkingmachines.ai/news/ training-llms-to-predict-world-events/. Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. ForecastBench: A dynamic benchmark of AI forecasting capabilities. In ICLR, 2025. Chinmay Karkar and Paras Chopra. Future is unevenly distributed: Forecasting ability of LLMs depends on what we’re asking. arxiv, 2025. URL https://arxiv.org/abs/2511.18394. Simas Kucinskas, Houtan Bastani, and Ezra Karger. ForecastBench: An updated ranking methodology, 2025. URL https://forecastbench.org/assets/pdfs/forecastbench_updated_ methodology.pdf. Simas Kucinskas, Houtan Bastani, and Matt Reynolds. Making forecasting scores easier to interpret: Introducing the brier index, 2026. URL https://forecastingresearch.substack.com/p/ introducing-the-brier-index. Michael Lawrence, Paul Goodwin, Marcus O’Connor, and Dilek Önkal. Judgmental forecasting: A review of progress over the last 25 years. International Journal of Forecasting, 22(3):493–518, 2006. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. MetaHarness: End-to-End Optimization of Model Harnesses. arxiv, 2026. URL https://arxiv. org/abs/2603.28052. Zehan Li, Yuxuan Wang, Ali El Lahib, Ying-Jieh Xia, and Xinyu Pi. Simulated ignorance fails: A systematic study of LLM behaviors on forecasting problems before model knowledge cutoff. arxiv, 2026. URL https://arxiv.org/abs/2601.13717. Lightning Rod Labs. Foresight-32b: An LLM-based forecasting system, 2025. URL https: //blog.lightningrod.ai/p/using-the-future-to-train-prediction-models. Zijia Liu, Peixuan Han, Haofei Yu, Haoru Li, and Jiaxuan You. Time-R1: Towards comprehensive temporal reasoning in LLMs. arxiv, 2025. URL https://arxiv.org/abs/2505.13508. Xinghua Lou, Miguel Lázaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P. Murphy. AutoHarness: improving LLM agents by automatically synthesizing a code harness. arxiv, 2026. URL https://arxiv.org/abs/2603.03329. Metaculus. FutureEval: Continuously updated AI forecasting benchmark, 2026. URL https: //www.metaculus.com/futureeval/. Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramer. Pitfalls in evaluating language model forecasters. arxiv, 2025. URL https://arxiv.org/abs/2506.00723. John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61–74. MIT Press, 1999. Sarah Pratt, Seth Blumberg, Pietro Kreitlon Carolino, and Meredith Ringel Morris. Can language models use forecasting strategies? arxiv, 2024. URL https://arxiv.org/abs/2406.04446. Philipp Schoenegger, Peter S. Park, Ezra Karger, Sean Trott, and Philip E. Tetlock. AI-augmented predictions: LLM assistants improve human forecasting accuracy. arxiv, 2024a. URL https: //arxiv.org/abs/2402.07862. Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, and Philip E. Tetlock. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. arxiv, 2024b. URL https://arxiv.org/abs/2402.19379. Philipp Schoenegger, Cameron R. Jones, Philip E. Tetlock, and Barbara Mellers. Prompt engineering large language models’ forecasting capabilities. arxiv, 2025. URL https://arxiv.org/abs/ 2506.01578. 47
David Spiegelhalter. The Art of Uncertainty: How to Navigate Chance, Ignorance, Risk and Luck. W.W. Norton, 2025. Charles Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, pages 197–206, 1956. Philip E. Tetlock and Dan Gardner. Superforecasting: The Art and Science of Prediction. Crown, 2015. Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt, and Philipp Schoenegger. Outcomebased reinforcement learning to predict the future. arxiv, 2025. URL https://arxiv.org/abs/ 2505.17989. Benjamin Turtel, Paul Wilczewski, Danny Franklin, and Kris Skothiem. Future-as-label: Scalable supervision from real-world outcomes. arxiv, 2026. URL https://arxiv.org/abs/2601. 06336. Kartik Waghmare and Johanna Ziegel. Proper scoring rules for estimation and forecast evaluation. arxiv, 2025. URL https://arxiv.org/abs/2504.01781. Zhen Wang, Xi Zhou, Yating Yang, Bo Ma, Lei Wang, Rui Dong, and Azmat Anwar. CogForecast: Mitigating cognitive bias in LLM forecasting. In Findings of EMNLP, 2025. Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu. LLM-as-a-prophet: Understanding predictive intelligence with prophet arena. arxiv, 2025. URL https://arxiv. org/abs/2510.17638. Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Yixiao Tian, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, Tianle Cai, Zehui Chen, Jiecao Chen, Yantao Du, Xiang Gao, Jiacheng Guo, Liang Hu, Jianpeng Jiao, Xiangsheng Li, Jingkai Liu, Shuang Ni, Zhoufutu Wen, Ge Zhang, Kaiyuan Zhang, Xin Zhou, Jose Blanchet, Xipeng Qiu, Mengdi Wang, and Wenhao Huang. FutureX: An advanced live benchmark for LLM agents in future prediction. arxiv, 2025. URL https://arxiv.org/abs/2508.11987. Jaden Zhang, Gardenia Liu, Oliver Johansson, Hileamlak Yitayew, Kamryn Ohly, and Grace Li. Prediction arena: Benchmarking AI models on real-world prediction markets. arxiv, 2026. URL https://arxiv.org/abs/2604.07355. Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. In NeurIPS (Datasets and Benchmarks), 2022. URL https://arxiv.org/abs/2206. 15474.
48