arXiv:2604.14961v1 [cs.LG] 16 Apr 2026
Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits Maksim Pershin
Ivan Golovanov
Pavel Baltabaev
Natalia Trankova
Abstract Contextual bandit algorithms suffer from high regret during cold-start, when the learner has insufficient data to distinguish good arms from bad. We propose augmenting Disjoint LinUCB with LLM pseudo-observations: after each round, a large language model predicts counterfactual rewards for the unplayed arms, and these predictions are injected into the learner as weighted pseudo-observations. The injection weight is controlled by a calibration-gated decay schedule that tracks the LLM’s prediction accuracy on played arms via an exponential moving average; high calibration error suppresses the LLM’s influence, while accurate predictions receive higher weight during the critical early rounds. We evaluate on two contextual bandit environments—UCI Mushroom (2-arm, asymmetric rewards) and MIND-small (5-arm news recommendation)—and find that when equipped with a task-specific prompt, LLM pseudo-observations reduce cumulative regret by 19% on MIND relative to pure LinUCB. However, generic counterfactual prompt framing increases regret on both environments, demonstrating that prompt design is the dominant factor — more important than the choice of decay schedule or calibration gating parameters. We analyze the failure modes of calibration gating on domains with small prediction errors and provide a theoretical motivation for the bias–variance trade-off governing pseudo-observation weight.
1
Introduction
Contextual bandit algorithms are a workhorse of interactive decision-making, powering applications from news recommendation [Li et al., 2010] and clinical trials to online advertising and content optimization [Ye et al., 2024]. In each round, the learner observes a context, selects an action (arm), and receives a reward—but only for the chosen arm. This partial-feedback structure creates a fundamental tension between exploration and exploitation, and the resulting cold-start regret—the cost paid while the learner has too few observations to make good decisions—can be substantial in practice. Large language models (LLMs) present a tantalizing opportunity to reduce this cold-start cost. An LLM that has been pre-trained on broad text corpora encodes significant world knowledge: it can read “user frequently reads technology news” alongside “candidate article: local sports recap” and infer that the click probability is likely low. This is genuine domain signal that a tabula-rasa bandit learner must discover through costly exploration. However, naively injecting LLM predictions into a bandit algorithm is fraught with risk. LLM reward predictions are noisy, inconsistent across rounds, and poorly calibrated. As we demonstrate in our experiments, a poorly framed prompt can cause the LLM to increase cumulative regret relative to the unaugmented baseline—even when the underlying model has relevant domain knowledge. A principled mechanism for weighting LLM contributions is therefore essential. We propose a framework in which, after each pull, the LLM predicts counterfactual rewards for the unplayed arms. These predictions are injected into Disjoint LinUCB [Li et al., 2010] as weighted 1
pseudo-observations, where the weight wt ∈ [0, 1) is determined by a pluggable decay schedule. We study several schedules, including time-based decay (inverse, power, exponential), calibration-gated decay that scales weight by exp(−η · Et ) where Et is an exponential moving average of the LLM’s squared prediction error on played arms, and hybrid combinations. As the bandit learner accumulates real data and the LLM’s marginal value decreases, the weight gracefully diminishes. Our contributions are: 1. A general framework for injecting LLM pseudo-observations into contextual bandits with calibration-gated weighting (Section 4). 2. An empirical evaluation on UCI Mushroom (2-arm) and MIND-small (5-arm news recommendation) showing that task-specific prompting reduces regret by 19% on MIND, while generic counterfactual framing hurts on both domains (Section 7). 3. The finding that prompt framing is the dominant factor —more impactful than the choice of decay schedule or calibration gating parameters (Section 7.3). 4. An analysis of when LLM augmentation helps (domains where the LLM has genuine knowledge to contribute) versus when it hurts (opaque feature spaces), and of the failure modes of calibration gating on domains with small prediction errors (Section 8). Related concurrent work. Several recent papers explore LLM-augmented bandits. Alamdari et al. [2024] use LLMs to generate synthetic prior data for pre-training; Ye et al. [2024] treat LLM predictions as auxiliary pseudo-samples via a two-UCB selection rule; Sun et al. [2025] propose Thompson Sampling variants with LLM reward oracles. Our approach differs in its emphasis on online calibration tracking as the gating mechanism and on the empirical finding that prompt design dominates algorithmic tuning. We discuss related work in detail in Section 2.
2
Related Work
2.1
LLMs for Sequential Decision Problems
The use of LLMs to accelerate or guide online learning has received growing attention. Alamdari et al. [2024] propose Contextual Bandits with LLM Initialization (CBLI), where an LLM generates synthetic user preference distributions from textual descriptions to pre-train the bandit before deployment. This is complementary to our approach: CBLI uses LLMs for initialization, while we use them for online augmentation with calibration tracking. Ye et al. [2024] introduce LOLA, an LLM-assisted online learning algorithm that converts LLM click-through rate predictions into auxiliary pseudo-samples of fixed size naux (tuned offline via hyperparameter search), then selects arms by taking the minimum of two UCB indices—one standard and one incorporating the pseudosamples. LOLA is methodologically closest to our work; the key differences are that (i) we use explicit online calibration tracking rather than a fixed offline-tuned pseudo-count, and (ii) our decay schedules adaptively modulate weight during learning rather than keeping it constant. Sun et al. [2025] propose three LLM-enhanced multi-armed bandit algorithms: TS-LLM uses temperature decay for exploration, RO-LLM treats the LLM as a deterministic regression oracle (temperature 0) with SquareCB inverse-gap-weighting exploration, and TS-LLM-DB extends to dueling bandits. None employ calibration gating. Chen et al. [2024] take a meta-level approach, using online model selection to dynamically balance between LLM-powered policies and classical contextual bandit algorithms. Nie et al. [2025] provide EVOLvE, a comprehensive benchmark for 2
evaluating LLM exploration in bandits, and show that algorithm-guided support improves LLM decision-making—a finding that motivates our calibration-tracking approach. Hazime and Farooq [2025] evaluate LLMs as zero-shot bandit solvers and find inconsistent performance, underscoring the need for principled weighting mechanisms. Baheri and Alm [2023] take an orthogonal approach, using LLMs as context encoders rather than reward predictors, enriching the feature representation fed to classical bandit algorithms. This is complementary: one could combine LLM-enhanced contexts with our pseudo-observation framework.
2.2
Contextual Bandits
Our base learner is Disjoint LinUCB [Li et al., 2010], which maintains independent ridge regressions per arm with upper-confidence-bound exploration. The theoretical foundations of linear contextual bandits are developed in Abbasi-Yadkori et al. [2011] (OFUL with self-normalized confidence sets) and the textbook treatment of Lattimore and Szepesvári [2020]. On the adversarial side, Liu et al. [2023] develop near-optimal algorithms for adversarial linear contextual bandits without simulator access, establishing the theoretical frontier against which LLM-augmented methods can be compared.
2.3
Calibration and Uncertainty in LLMs
Our calibration tracker draws on the broader literature on neural network calibration. Guo et al. [2017] demonstrate that modern deep networks are poorly calibrated and propose temperature scaling; Kadavath et al. [2022] study LLM self-evaluation and find that language models can partially assess their own uncertainty. Huch et al. [2024] address model misspecification in bandit settings through robust mixed-effects estimation, a related concern when LLM predictions are systematically biased. Our calibration-gated decay can be viewed as an online mechanism for adjusting the influence of a potentially misspecified auxiliary model.
3
Preliminaries
3.1
Contextual Bandits
We consider the stochastic contextual bandit setting. At each round t = 1, . . . , T : 1. Nature reveals a context xt ∈ Rd (or per-arm contexts {xt,a }K a=1 ). 2. The learner selects an arm at ∈ [K] := {1, . . . , K}. 3. The learner observes reward rt = r(xt , at ), drawn from an unknown distribution conditional on the context and arm. The learner’s goal is to minimize cumulative regret: T X ∗ RT = rt − rt ,
where rt∗ = max E[r(xt , a)]. a∈[K]
t=1
3.2
(1)
Disjoint LinUCB
We assume a disjoint linear reward model: E[r | x, a] = x⊤ θa∗ for unknown parameters θa∗ ∈ Rd , one per arm. Disjoint LinUCB [Li et al., 2010] maintains per-arm sufficient statistics: X X Aa = λI + xs x⊤ ba = rs · xs , (2) s, s:as =a
s:as =a
3
where λ > 0 is the ridge regularization parameter. The arm is selected by: q i h −1 ⊤ A−1 x , x at = arg max x⊤ A b + α a t t a a t a∈[K]
(3)
where α > 0 controls the exploration bonus. We augment each context vector with a bias term: x 7→ [x; 1]. Notation. K denotes the number of arms, T the horizon, d the context dimension (after augmentation), α the exploration parameter, and λ the ridge regularization.
4
Method
4.1
Weighted Observations in LinUCB
We extend the standard LinUCB update to accept weighted observations. Given a context–arm– reward triple (x, a, r) with weight w ∈ (0, 1], the update becomes: Aa ← Aa + w · xx⊤ ,
ba ← ba + w · r · x.
(4)
Real observations use w = 1; LLM pseudo-observations use w = wt < 1. This is equivalent to weighted ridge regression, where each pseudo-observation contributes a fraction of the information of a real sample to both the design matrix and the reward vector.
4.2
LLM Pseudo-Observations
After each round, we query an LLM to predict rewards for the arms that were not played. The per-round protocol is: 1. Calibration probe (before observing reward). Query the LLM for its prediction on the played arm at , given only the context and arm descriptions—crucially, without revealing the outcome LLM . rt . This yields r̂t,a t 2. Pull and observe. Execute arm at , observe reward rt , perform the standard LinUCB update with w = 1. LLM on all unplayed arms 3. Score unplayed arms. Query the LLM for reward predictions r̂t,a a ̸= at .
4. Compute weight. Determine the pseudo-observation weight wt from the decay schedule (Section 4.3). 5. Inject pseudo-observations. For each a = ̸ at , update LinUCB via Equation (4) with LLM , w ). (xt , a, r̂t,a t LLM − r )2 and update the 6. Update calibration tracker. Compute the squared error et = (r̂t,a t t EMA.
4
Prompt design. We find that prompt framing is the single most important design choice (Section 7.3). We test two framings: • Generic counterfactual: “Given this context, predict what reward arm a would have produced.” This asks the LLM to reason about the bandit abstraction. • Task-specific: “Given this user’s reading history, predict the probability that the user clicks each candidate article.” This asks the LLM to reason about the domain question directly. The task-specific framing (mind_click) dramatically outperforms the generic framing on MIND, achieving lower regret with fewer tokens. We hypothesize that the LLM’s pre-trained knowledge is better activated when the question matches its training distribution (natural language prediction tasks) rather than an abstract bandit formulation. Structured output. The LLM returns a structured JSON response containing, for each arm, a predicted_reward and a confidence score. In this work we use only the predicted reward; leveraging confidence as an additional weight modulator is left for future work.
4.3
Decay Schedules
The pseudo-observation weight wt is determined by a decay schedule D that maps the round index t and auxiliary state (e.g., calibration error) to a weight in [0, 1). We study the following families: Time-based schedules. These decay the weight as a function of round t alone: Constant: wt = λ, τ Inverse: wt = λ · , t+τ α τ Power: wt = λ · , t+τ Exponential: wt = λ · exp(−t/τ ),
(5) (6) (7) (8)
where λ ∈ (0, 1) is the base weight and τ > 0 controls the decay rate. Calibration-gated schedule. The weight is modulated by the LLM’s online prediction accuracy: wt = λ · exp −η · Et , (9) where η > 0 is the gating sensitivity and Et is the exponential moving average of squared calibration error: 2 LLM Et = β · Et−1 + (1 − β) · et , et = r̂t,a − rt , (10) t with smoothing parameter β ∈ [0, 1). Hybrid schedules. Time-based and calibration-gated components can be composed multiplicatively, e.g., wt = λ · g(t) · exp(−η · Et ), providing both a guaranteed fade-out and adaptive quality gating.
4.4
Algorithm
We summarize the full procedure in Algorithm 1. 5
Algorithm 1 LinUCB with Calibration-Gated LLM Pseudo-Observations Require: Arms [K], exploration α, regularization λ, base weight λw , decay schedule D, EMA parameter β, LLM scorer L 1: Initialize Aa ← λI, ba ← 0 for all a ∈ [K]; E0 ← 0 2: for t = 1, . . . , T do 3: Observe context xt (and per-arm contexts {xt,a } if applicable) 4: Select arm at via UCB (Equation (3)) LLM ← L(x , a ) 5: Calibration probe: r̂t,a {before observing reward} t t t 6: Pull arm at , observe reward rt 7: Real update: Aat ← Aat + xt x⊤ t , bat ← bat + rt · xt LLM 8: Score unplayed: {r̂t,a }a̸=at ← L(xt , [K] \ {at }) LLM − r )2 9: Compute calibration error: et ← (r̂t,a t t 10: Update EMA: Et ← β · Et−1 + (1 − β) · et 11: Compute weight: wt ← D(t, Et ) 12: for each a ̸= at do LLM 13: Pseudo update: Aa ← Aa + wt · xt x⊤ t , ba ← ba + wt · r̂t,a · xt 14: end for 15: end for
5
Theoretical Motivation
We provide an informal analysis of how pseudo-observation weight affects the bias–variance trade-off in the learner’s estimates, motivating the calibration-gated approach. Remark 1 (Effective sample size). In standard LinUCB, after T rounds the learner has accumulated at most T observations total, at most T /K per arm in expectation under uniform exploration. With LLM pseudo-observations injected at weight w for all K − 1 unplayed arms each round, the effective information per round increases. The design matrix for arm a accumulates information at rate 1 + (K − 1) · w per round in which a appears, accelerating convergence of θ̂a to θa∗ . Proposition 1 (Bias–variance trade-off). Suppose the LLM’s prediction for arm a at round t LLM = x⊤ θ ∗ + b + ϵ , where b is a systematic bias and ϵ satisfies r̂t,a a t,a a t,a is zero-mean noise with t a 2 variance σLLM . Then the weighted ridge estimator θ̂a based on n real observations (weight 1) and m pseudo-observations (weight w) satisfies: 2 2 σ + m · w2 · σLLM m · w · ba ∗ , Var θ̂a = O . (11) E θ̂a − θa = O n+m·w (n + m · w)2 The bias term grows with w · ba : if the LLM is systematically biased, higher weight injects more bias into the learner. The variance term decreases with m · w in the denominator, reflecting the information gain from pseudo-observations. The optimal weight balances these: w∗ ∝
1 , 2 σLLM + b2a
(12)
which is small when the LLM is inaccurate (high bias or variance) and large when it is well-calibrated. This directly motivates the calibration-gated schedule (Equation (9)): the EMA error Et tracks 2 b2a + σLLM online, and the exponential gating exp(−η · Et ) monotonically decreases weight as the estimated error grows. 6
Table 1: Environment comparison.
Mushroom MIND-small
Arms
Context dim
Reward range
LLM signal
2 5
117 per-arm
[−35, +5] {0, 1}
Low (raw codes) High (article titles)
Table 2: Experimental configurations. All use GPT-4o-mini (OpenAI) where an LLM is involved. Name
Prompt
Decay
λw
η
β
NoLLM LLM-Default LLM-Click LLM-CalGated LLM-Context
— counterfactual mind_click mind_click counterfactual+ctx
zero constant constant cal-gated constant
— 0.1 0.1 0.3 0.1
— — — 10.0 —
— 0.95 0.95 0.95 0.95
Remark 2 (Decay as a safety mechanism). Time-based decay provides a worst-case guarantee: regardless of LLM quality, the pseudo-observation weight vanishes as t → ∞, ensuring that LinUCB converges to its standard regret bound asymptotically. Calibration-gated decay provides an adaptive guarantee: the weight responds to the LLM’s actual performance. The hybrid approach offers both.
6
Experimental Setup
6.1
Environments
We evaluate on two contextual bandit environments that test different aspects of LLM augmentation. UCI Mushroom. Following Li et al. [2010], we construct a 2-arm bandit from the UCI Mushroom dataset (8124 instances, 22 categorical features one-hot encoded to d = 117 dimensions). Arm 1 is “eat” and arm 2 is “don’t eat,” with asymmetric rewards: eating an edible mushroom yields +5, eating a poisonous one yields −35, not eating an edible one yields 0, and not eating a poisonous one yields +5. This creates a high-penalty exploration problem where incorrect predictions on poisonous mushrooms are very costly. MIND-small. We use the Microsoft News Dataset (MIND-small) [Wu et al., 2020], a news recommendation benchmark with 50,000+ impressions. Each impression provides a user with a click history and K = 5 candidate articles. We hash user history and article title/abstract into fixed-dimensional vectors using feature hashing, yielding per-arm context vectors. The reward is Bernoulli: r = 1 if the chosen article matches the clicked article, 0 otherwise. Approximately 38% of rounds have a clickable article among the candidates, making this a sparse-reward environment.
6.2
Baselines and Configurations
All experiments use Disjoint LinUCB with α = 1.0 and λreg = 1.0. We compare the following configurations: LLM-Default uses the generic counterfactual prompt (“predict what the unplayed arm would have given”). LLM-Click uses the task-specific mind_click prompt (“predict click probability 7
Table 3: Cumulative regret on UCI Mushroom (T = 100, single seed).
NoLLM LLM-Default LLM-Context
t = 30
t = 50
t = 75
t = 100
125 140 180
170 205 205
190 220 305
265 295 365
Table 4: Cumulative regret and token usage on MIND-small (T = 100, single seed). t = 30
t = 50
t = 75
t = 100
Tokens (input)
10 11 7 9
17 18 11 15
22 26 15 22
32 35 26 33
— 163k 74k 74k
NoLLM LLM-Default LLM-Click LLM-CalGated
for each candidate article”). LLM-CalGated adds calibration-gated decay with η = 10, λw = 0.3 (higher ceiling since calibration gates it down). LLM-Context includes mushroom feature descriptions in the prompt history.
6.3
Implementation Details
All experiments run for T = 100 rounds with seed = 42 and nsims = 1 (single simulation). The LLM is GPT-4o-mini via the OpenAI API, called through LangChain’s with_structured_output for reliable JSON parsing. The mind_click prompt style makes a single LLM call per round (predicting all arms simultaneously), while the default prompt requires two calls (calibration probe + scoring). Calibration tracking uses an EMA with β = 0.95. We acknowledge that single-seed results without error bars are a limitation; these Phase 1 results establish directional findings that motivate targeted follow-up with multiple seeds and longer horizons.
7
Results
7.1
Mushroom: LLM Augmentation Hurts
On mushroom, all LLM variants increase regret relative to the baseline (Table 3, Figure 1). The default prompt adds +30 cumulative regret at t = 100 (+11%), while adding mushroom features to the prompt history (LLM-Context) adds +100 (+38%). The LLM’s predictions on mushroom are inconsistent: it flips between predicting −35 and +5 for the same arm across rounds, because the context consists of raw categorical feature codes (e.g., odor=foul) that do not activate the LLM’s pre-trained knowledge effectively. The constant decay at λw = 0.1 means this noise never stops influencing the learner.
7.2
MIND: Task-Specific Prompting Reduces Regret
On MIND, LLM-Click is the only variant that consistently beats the baseline (Table 4, Figure 2). At t = 100, it achieves cumulative regret of 26 versus 32 for NoLLM—a 19% reduction. The gap
8
Figure 1: Cumulative regret on UCI Mushroom. LLM augmentation increases regret in all configurations. opens early (t = 30: 7 vs. 10) and widens throughout, suggesting that the LLM provides genuine value during the cold-start phase that compounds over time. LLM-Click also uses 55% fewer input tokens than LLM-Default (74k vs. 163k), because the task-specific prompt makes a single call per round predicting all arms simultaneously, rather than separate calibration and scoring calls. LLM-Default slightly hurts on MIND (35 vs. 32), demonstrating that the same underlying LLM model can help or hurt depending entirely on how the prediction task is framed.
7.3
Ablation: Prompt Framing
The most striking finding across both environments is the dominance of prompt design. On MIND, holding the algorithm and decay schedule constant, switching from the generic counterfactual prompt to the task-specific mind_click prompt changes the outcome from +3 regret (worse than baseline) to −6 regret (better than baseline)—a swing of 9 points entirely attributable to prompt framing. The mechanism is straightforward: the generic prompt asks the LLM to reason about “what reward would arm a have produced?”—an abstract counterfactual question that it answers with hedged, mid-range predictions (0.2–0.5 for binary click probabilities). The task-specific prompt asks “will this user click this article?”—a concrete prediction task that aligns with the LLM’s pre-training on natural language understanding and enables it to leverage its knowledge of user preferences and article content.
7.4
Ablation: Calibration-Gated Decay
Calibration-gated decay with η = 10 (LLM-CalGated) performs worse than the simple constant schedule (LLM-Click), achieving regret of 33 versus 26 (Table 4). The failure mode is instructive. On MIND, the LLM’s calibration errors are consistently small— the EMA Et stabilizes around 0.05–0.08. With η = 10, this yields exp(−10 × 0.06) ≈ 0.55, causing the effective weight to decay from 0.3 × 0.55 ≈ 0.17 to below 0.02 by t = 100. The gating is too aggressive: it silences the LLM even though its predictions are directionally useful. 9
Figure 2: Cumulative regret on MIND-small. Only the task-specific prompt (LLM-Click) beats the baseline. This reveals a fundamental tension in calibration gating: when the LLM is reasonably good (small errors), aggressive gating removes useful signal. The η parameter must be tuned per domain, and the regime where calibration gating adds value over a simple constant schedule may be narrow—it requires that the LLM’s accuracy varies meaningfully across rounds, so that gating can selectively amplify good rounds.
8
Discussion
8.1
When Do LLMs Help?
Our results delineate a clear condition for LLM augmentation to reduce regret: the LLM must have genuine domain knowledge relevant to the prediction task, and the prompt must be designed to activate that knowledge. On MIND, the LLM can read article titles (“Tech Giants Report Record Earnings”), user reading history (previously clicked technology and business articles), and make informed click predictions. This is a task that aligns naturally with the LLM’s pre-training on text understanding. On mushroom, the LLM receives raw categorical codes (cap-shape=convex, odor=foul) that do not activate useful pre-trained knowledge—the LLM “knows” that foul odor suggests danger, but cannot reliably map this to the specific reward structure (−35 vs. +5) without in-context learning that the prompt history cannot efficiently support.
8.2
Prompt Framing as the Bottleneck
The dominance of prompt framing over algorithmic tuning is our most practically relevant finding. Researchers and practitioners designing LLM-augmented bandit systems should invest primarily in prompt engineering—crafting prompts that frame the prediction task in terms the LLM can reason about naturally—rather than in sophisticated decay schedule design. The decay schedule provides a safety mechanism (eventually silencing the LLM), but the prompt determines whether the LLM provides useful signal in the first place. 10
8.3
Limitations
Our results are subject to several important limitations: 1. Single seed. All results use a single random seed (= 42) with one simulation. Error bars and statistical significance tests require nsims ≥ 5. 2. Short horizon. At T = 100, we cannot determine whether the LLM’s advantage persists, widens, or reverses as LinUCB accumulates more data. Runs of 300–500 rounds would clarify the crossover dynamics. 3. Single LLM. We test only GPT-4o-mini. Different LLMs may exhibit different calibration profiles and domain knowledge. 4. Two environments. Broader evaluation across additional domains (e.g., advertising, clinical trials) is needed. 5. Cost. Even with the efficient single-call mind_click prompt (74k tokens over 100 rounds), LLM calls add latency and monetary cost per round. The cost–regret trade-off has not been formalized. 6. Confidence unused. The LLM returns per-prediction confidence scores that we do not currently use for weighting.
8.4
Future Work
Several extensions follow naturally: • Longer horizons and multiple seeds to establish statistical significance and crossover dynamics. • Gentler calibration gating (η ∈ [1, 3]) to preserve LLM signal while penalizing genuinely poor rounds. • Confidence-weighted injection, scaling pseudo-observation weight by the LLM’s selfreported confidence. • Sweep over λw to find the optimal base weight for the cold-start phase. • Task-specific prompts for mushroom (“is this mushroom safe to eat?”) to test whether the prompt framing insight transfers. • Formal regret analysis deriving bounds on the additional regret (or regret reduction) from pseudo-observations as a function of LLM bias and weight schedule.
9
Conclusion
We have presented a framework for augmenting contextual bandits with LLM pseudo-observations, where the injection weight is controlled by calibration-gated decay schedules. On MIND-small news recommendation, task-specific LLM prompting reduces cumulative regret by 19% relative to pure LinUCB. On UCI Mushroom, where the LLM lacks actionable domain knowledge, augmentation increases regret regardless of prompt design or decay schedule. 11
Our central finding is that prompt framing is the dominant factor in LLM-augmented bandit performance—more important than the choice of decay schedule, calibration gating parameters, or even whether calibration gating is used at all. This suggests that the most productive direction for practitioners is to invest in task-specific prompt design that activates the LLM’s pre-trained domain knowledge, while using simple constant decay as a sufficient algorithmic mechanism.
References Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems, 24, 2011. Parand A. Alamdari, Yanshuai Cao, and Kevin H. Wilson. Jump starting bandits with LLMgenerated prior knowledge. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 19821–19833. Association for Computational Linguistics, 2024. Ali Baheri and Cecilia O. Alm. LLMs-augmented contextual bandit. arXiv preprint arXiv:2311.02268, 2023. Dingyang Chen, Qi Zhang, and Yinglun Zhu. Efficient sequential decision making with large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9157–9170. Association for Computational Linguistics, 2024. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330, 2017. Jawad Hazime and Junaid Farooq. Evaluation of LLM powered agentic AI for solving multi-arm bandit problems. In 2025 IEEE International Conference on Omni-layer Intelligent Systems (COINS). IEEE, 2025. Easton Huch, Jieru Shi, Madeline R. Abbott, Jessica R. Golbus, Alexander Moreno, and Walter H. Dempsey. RoME: A robust mixed-effects bandit algorithm for optimizing mobile health interventions. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW), pages 661–670, 2010. Haolin Liu, Chen-Yu Wei, and Julian Zimmert. Bypassing the simulator: Near-optimal adversarial linear contextual bandits. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Allen Nie, Yi Su, Bo Chang, Jonathan N. Lee, Ed H. Chi, Quoc V. Le, and Minmin Chen. EVOLvE: Evaluating and optimizing LLMs for in-context exploration. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of PMLR, 2025. 12
Jiahang Sun, Zhiyong Wang, Runhan Yang, Chenjun Xiao, John C. S. Lui, and Zhongxiang Dai. Large language model-enhanced multi-armed bandits. arXiv preprint arXiv:2502.01118, 2025. Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Dongsheng Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. MIND: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3597–3606, 2020. Zikun Ye, Hema Yoganarasimhan, and Yufeng Zheng. LOLA: LLM-assisted online learning algorithm for content experiments. arXiv preprint arXiv:2406.02611, 2024.
A
Experimental Details
A.1
Hyperparameters
All experiments use the following shared hyperparameters: exploration parameter α = 1.0, ridge regularization λreg = 1.0, calibration EMA parameter β = 0.95, random seed = 42, single simulation. MIND experiments use K = 5 arms with hashed context vectors. Mushroom experiments use K = 2 arms with one-hot encoded categorical features (d = 117 after bias augmentation).
A.2
Prompt Templates
Generic counterfactual prompt (default). The prompt provides the current context description, lists available arms with descriptions, includes a history of the last 5 rounds (arm chosen, reward observed), and asks: “For each unplayed arm, predict the expected reward the learner would have received. Return predictions as JSON.” The calibration probe variant asks for the played arm’s expected reward before the outcome is revealed. Task-specific prompt (mind_click). The prompt provides the user’s recent reading history (last 10 article titles), lists 5 candidate articles with titles, and asks: “For each candidate article, predict the probability that this user will click on it. Return predictions as JSON with predicted_reward in [0, 1].” A single call covers all arms, serving as both calibration probe and scoring.
A.3
Compute Budget
All LLM experiments use GPT-4o-mini via the OpenAI API. Token usage per 100-round experiment: LLM-Default consumes ∼163k input tokens; LLM-Click consumes ∼74k input tokens. Wall-clock time is dominated by API latency (∼0.5–1s per call).
B
Additional Results
B.1
Calibration Error Trajectories
The calibration EMA Et on MIND stabilizes between 0.05 and 0.08 for both LLM-Click and LLMCalGated, indicating that the LLM’s click predictions are consistently close to actual outcomes. This small and stable error is precisely why aggressive calibration gating (η = 10) fails—the gating function cannot distinguish between “good enough” and “perfect,” and treats both as grounds for weight reduction. 13
On mushroom, calibration errors are larger and more variable, reflecting the LLM’s inconsistent predictions on raw feature codes. In this regime, calibration gating would correctly reduce weight— but since the underlying predictions are already harmful, gating alone is insufficient; the prompt framing must be fixed first.
B.2
MIND Tiny Results
Sanity-check experiments with T = 10 rounds show regret of 1–2 for all configurations, consistent with the sparse-reward structure (only ∼38% of rounds have a clickable article). These runs verified the pipeline but are too short to differentiate configurations.
14