Online Pandora’s Box for Contextual LLM Cascading Alexandre Belloni The Fuqua School of Business, Duke University, [email protected]
Yan Chen The Fuqua School of Business, Duke University, [email protected]
Yehua Wei The Fuqua School of Business, Duke University, [email protected]
arXiv:2606.07392v1 [cs.AI] 5 Jun 2026
Abstract. Motivated by Large Language Model (LLM) cascading, we propose an online contextual Pandora’s Box model for adaptively querying and selecting LLM APIs. In each period, a decision-maker observes a request context and faces a two-phase decision problem. In the query phase, the decision-maker sequentially queries APIs, where each query reveals a generated output and the decision-maker incurs an (output-dependent) cost. In the selection phase, the decision-maker selects one of the generated outputs to deploy and observes only the downstream reward of the deployed output. This output-mediated feedback structure differs from classical online contextual Pandora’s Box models, in which opening a box directly reveals its reward. Rather than estimating the full conditional output and cost distributions of each API, we directly model the reservation index and develop a learning approach for the query phase. Specifically, we impose a parametric structure on the contextual reservation index functions induced by the classical Weitzman’s policy. Our policy combines generalized method of moments (GMM) type estimation of these reservation indices with UCB-style confidence bounds for both these indices and the shared output-level reward √ e( 𝑇) cumulative regret evaluator. Under regularity conditions, we prove that the resulting policy achieves dimension-dependent 𝑂
over a horizon of 𝑇 periods.
1. Introduction The proliferation of Large Language Models (LLMs) has transformed the economics of content generation and decision support. Firms increasingly use generative AI systems to generate advertisements, produce code, serve customer requests, and conduct complex analytical tasks. In these applications, the operational challenge is often not whether content can be generated, but how to generate high-quality content reliably and cost-effectively at scale. A firm that produces a high volume of advertisements per day, for example, may have access to a portfolio of LLM APIs that differ in cost and output quality across task types. More capable proprietary models may, in general, deliver higher-quality outputs but incur substantially higher costs, while smaller or specialized models may be cheaper but less reliable. Because model performance varies across request contexts and it is difficult to predict the quality of the outputs a priori, committing to a single API is often suboptimal. Similarly, querying all of the available APIs is typically prohibitively costly. We study this problem from the perspective of an organization that uses external or internal LLM APIs to serve a stream of business requests. In such applications, requests arrive as distinct 1
2
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
decision units without a strict latency requirement, allowing the decision-maker to query more than one API before selecting a final output. It is worth noting that this request-level decision problem fundamentally differs from the model hosting problem faced by large LLM platforms, where a key operational challenge is to reduce latency through batching, scheduling, and load management (e.g. Yu et al. 2022, Kwon et al. 2023, Agrawal et al. 2024, Ao et al. 2025, Jaillet et al. 2025). Our focus is instead on the API consumer’s decision problem of dynamically querying a portfolio of APIs to secure high-quality outputs in a cost-effective manner. A prominent approach for managing this cost-quality trade-off is LLM cascading (see e.g., Chen et al. 2025a, Yue et al. 2024, Gupta et al. 2024). Rather than sending every request directly to a single expensive model, a cascade queries LLM APIs sequentially, typically beginning with lower-cost models and escalating only when earlier outputs appear insufficiently reliable. The pioneering work of Chen et al. (2025a), for example, illustrates this design by routing a query through a sequence of LLM APIs and stopping once a generated response receives a sufficiently high reliability score from an evaluator. Cascading allows the easy requests to be handled by cheaper APIs, with only the difficult requests escalated to the expensive ones. Empirical evidence from Chen et al. (2025a) suggests that such cascades can substantially reduce inference costs while preserving, and in some cases improving, task performance. The LLM cascading system raises a fundamental sequential decision problem. For each incoming request, the decision-maker must manage both a query phase and a selection phase. In the query phase, the decision-maker must decide which API to query next and whether the generated outputs justify continuing the search. In the selection phase, the decision-maker must decide which of the generated outputs to deploy. These decisions are tightly coupled. Querying another API is costly, but it may reveal a better output; stopping the query phase early saves cost, but significantly increases the risks of not generating a better outcome; and selecting among queried outputs requires evaluating their downstream value. Thus, LLM cascading is not simply a model-routing problem, but a sequential search problem with costly information acquisition. A natural starting point for principled analysis of this search problem is the classical Pandora’s Box problem. Consider an idealized benchmark in which, for each request, the decision-maker knows the context-dependent distribution of the output-cost pair generated by each API and can evaluate the downstream value of any generated output once it is observed. In this benchmark, querying an API corresponds to opening a box, the inference cost acts as the inspection cost, and the downstream value of the generated output serves as the realized reward. For this formulation,
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
3
the seminal work of Weitzman (1979) characterizes the structure of the optimal policy. Specifically, given the context of a request, the policy assigns each API a reservation index, then queries APIs in decreasing order of these indices, and terminates the search as soon as the reward of an output exceeds the highest remaining index. Real applications of LLM cascading, however, depart from this oracle benchmark in ways that make learning central. The value of querying an API is inherently contextual, and the primitives required by the oracle Weitzman’s policy are rarely known in advance. In some settings, firms may have substantial historical data on prior human- or AI-generated outputs and their downstream rewards, allowing them to construct an accurate offline model of the reward function. In many others, however, such output-reward pairs are limited or unavailable. Moreover, even when rewards can be modeled from offline data, a firm deploying LLM APIs typically does not know the conditional distribution of outputs and costs generated by each API under a given request context. Since requests arrive sequentially over time, the decision-maker must learn context-dependent reservation indices while continuing to make cost-sensitive query and selection decisions. This naturally leads to an online contextual Pandora’s Box formulation of LLM cascading. Recent work on LLM cascading has developed increasingly sophisticated methods for balancing cost and performance. Those include budget-constrained cascade policies, uncertainty or thresholdbased deferral rules, and hybrid methods that combine cascading with model routing (Chen et al. 2025a, Yue et al. 2024, Gupta et al. 2024, Zhang et al. 2024). While these approaches provide important insights into efficient multi-LLM deployment, they rely primarily on heuristic designs. To the best of our knowledge, existing work has not yet formulated LLM cascading as an online contextual Pandora’s Box problem with joint query and selection, nor developed formal regret guarantees. Conversely, existing Pandora’s Box literature are not directly tailored to this LLM setting. A prominent line of work takes distribution-learning approaches (Liu et al. 2025), in which opening a box reveals a scalar reward and the learner estimates how the reward distribution varies with context, under parametric structure on the full distribution. Closer to our work, Atsidakou et al. (2024) parameterizes contextual reservation indices directly, but their model remains a classical Pandora’s box setting in which opened boxes reveal rewards directly. In LLM cascading, by contrast, an API call reveals a stochastic output-cost pair. In our model, the downstream value depends on the request and the generated output, while API-specific heterogeneity enters through the distribution of outputs and costs.
4
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
1.1. Our Contributions We make three main contributions. First, we introduce an online contextual Pandora’s Box model motivated by LLM cascading. Unlike existing online Pandora’s Box models, our model features a natural two-phase decision structure: querying an API reveals a stochastic output and incurs a cost, while the downstream reward is observed only after one generated output is selected and deployed. This formulation captures a key practical feature of LLM cascading: API-specific heterogeneity enters through the distributions of generated outputs and costs, while the downstream value of a generated output is evaluated through a shared reward model on the request-output pair. That is, an API can influence the downstream value only through the generated output. Second, we propose a direct index modeling and learning approach for the query phase. In the full-information benchmark, Weitzman’s policy is governed by contextual reservation indices, which are induced by each API’s conditional output and cost distributions together with the reward evaluator. Rather than estimating these distributions directly, we impose a parametric structure on the reservation index functions themselves as a generalized linear function of observable context features. This modeling assumption leads to a practical optimism-based learning approach in the LLM cascading setting, as the decision-maker can focus on constructing optimistic box-specific reservation indices from queried output-cost samples through the moment equations instead of the full conditional output distributions. The policy also learns the shared output-level reward evaluator from deployed-output rewards, and the two learned objects jointly determine the cascade’s querying and selection decisions. Third, we provide regret guarantees for the resulting online policy. The policy combines generalized method of moments (GMM) estimation for reservation indices with upper confidence bound (UCB)-style approaches for both the indices and rewards. Our policy is motivated by a simple but crucial regret decomposition under optimism: cumulative regret separates into errors from estimating reservation indices for only the queried APIs and errors from evaluating generated outputs for only the selected APIs. This decomposition allows us to apply the UCB-style analysis to our √ moment estimation problem, yielding a dimension-dependent 𝑂˜ ( 𝑇) cumulative regret bound. The analysis covers the practically important known-evaluator regime, where offline data provide an accurate output-level reward model, as well as the full regime where the evaluator and reservation indices must be learned jointly online.
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
5
1.2. Related Literature Our work is related to the literature on cost-aware LLM systems, sequential test-time inference, and the theoretical frameworks of sequential search and online learning. First, our paper is closely related to the emerging literature on LLM cascading and cost-aware multi-LLM deployment. This literature studies how to allocate inference requests across multiple LLM APIs when models differ in cost, latency, and output quality. Representative approaches include FrugalGPT (Chen et al. 2025a), budget-constrained cascade policies, uncertainty- or reliability-based deferral rules, and hybrid methods that combine cascading with model routing (Nie et al. 2024, Chen et al. 2024, Zhang et al. 2024, Yue et al. 2024, Gupta et al. 2024). Our work contributes to this literature by formulating LLM cascading as an online contextual Pandora’s Box problem, where the decision-maker sequentially decides which APIs to query and when to stop under unknown, context-dependent output and cost distributions. Our work is also related to LLM routing and ensembling methods although they address different decision problems than our LLM cascading motivation. Routing methods usually choose a single LLM for each request before observing any generated output, either through non-predictive rules or through predicted quality, reward, or utility scores (Hu et al. 2024, Shnitzer et al. 2023, Hari and Thomson 2023, Lu et al. 2024, Šakota et al. 2024, Mei et al. 2025). Such methods capture one-shot model choice but not adaptive within-request search. Ensemble methods aggregate or synthesize outputs from multiple LLMs, such as by ranking and fusing candidate responses (Jiang et al. 2023, Fang et al. 2024, Hu et al. 2025); they focus on output aggregation rather than sequential stopping under query costs. Second, our paper is related to recent work on sequential testing and sequential stopping for LLM inference. Huang et al. (2026) study adaptive self-consistency for a single LLM, where the system sequentially samples reasoning paths and stops once the most-likely answer can be identified with high posterior confidence. Li et al. (2026) study Bayesian sequential testing with heterogeneous LLMs, where the decision-maker adaptively chooses which LLM to query and stops once the posterior confidence for a hypothesis crosses a target threshold. These papers formulate LLM inference as posterior-driven sequential testing, whereas our LLM cascading model focuses on contextual sequential search over realized output-cost pairs, with stopping governed by reservation indices and learning focused on both the index functions and the output-level reward model. Third, the query phase component of our model extends the contextual Pandora’s Box problem. The classic Pandora’s Box problem without contexts originates from the seminal work of Weitzman (1979) and has since been studied under a variety of extensions (e.g. Doval 2018,
6
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Chawla et al. 2020, Fu and Lin 2020, Boodaghians et al. 2020, Agarwal et al. 2024, Ezra et al. 2026), including online variants (Gergatsouli and Tzamos 2022, Atsidakou et al. 2024). In particular, Gergatsouli and Tzamos (2022) studies an adversarial online Pandora’s Box setting in which box rewards are chosen by an adversary. Gatmiry et al. (2024) subsequently shows that, in this setting, no algorithm can achieve sublinear regret against the optimal policy, even with full-information feedback. For the online contextual Pandora’s Box problem, Liu et al. (2025) obtains a regret bound √ e( 𝑇) for under the assumption that the reward distribution of each box is a location shift of a of 𝑂
context-invariant noise distribution, but their results does not apply heterogeneous output distributions across contexts within each box, a key feature for LLM cascading applications. Atsidakou et al.
(2024) studies a contextual Pandora’s Box model under adversarially arriving contexts, where the optimal reservation index is parametrized as a linear function of the observed context. In this gene(𝑇 5/6) regret bound by reducing the learning problem to online linear eral setting, they obtain an 𝑂
regression. Our approach builds on a similar model of Atsidakou et al. (2024) by imposing a para-
metric reservation index structure, but adopts a different learning approach through a combination
of GMM and UCB analysis. This allows us to propose an optimism-based learning algorithm with √ e( 𝑇) regret bound under suitable regularity assumptions. 𝑂 Further, the selection phase of our model draws on the literature on generalized linear contextual
bandits. This line of work originates from the seminal paper of Filippi et al. (2010), which introduces an optimism-based algorithm for generalized linear bandits, and has since been extended in various directions (e.g. Jun et al. 2017, Kveton et al. 2020, Ding et al. 2021, Kim et al. 2023). Finally, our learning algorithm combines the GMM with an UCB–type algorithm. GMM originates from the seminal work of Hansen (1982) and has been widely developed and applied in econometrics (e.g. Newey and McFadden 1994, Arellano and Bond 1991, Chamberlain 1987, Lin and Lee 2010, Cheng et al. 2024, Andrews and Mikusheva 2022, Hansen and Lee 2021). UCBtype algorithms trace back to the seminal paper of Lai and Robbins (1985) and have since been extensively studied in the bandit literature (e.g. Auer et al. 2002, Rusmevichientong and Tsitsiklis 2010, Filippi et al. 2010, Fan and Glynn 2025, Garivier and Cappé 2011, Audibert et al. 2009), as well as in broader online learning settings and applications in operations management (e.g. Rakhlin and Sridharan 2013, Gao et al. 2022, Cheung et al. 2022, Mao et al. 2025). In addition, our paper belongs to the growing literature on LLM-assisted decision making across a range of fields, including operations management (e.g. Chen and Chan 2024, Chen et al. 2025b, Simchi-Levi et al. 2026), advertising (e.g. Yang et al. 2023, Reisenbichler et al. 2025), healthcare (e.g. Thirunavukarasu et al. 2023, Hager et al. 2024, Hao et al. 2025), among others.
7
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
1.3. Notations Given any integer 𝑘 ≥ 1 and vector 𝑣 ∈ R𝑘 , we use k𝑣k 2 to denote the Euclidean norm of 𝑣, and
k𝑣k ∞ to denote max𝑖=1,...,𝑘 |𝑣𝑖 |. For any positive semidefinite matrix Φ ∈ R𝑘×𝑘 , we use the notation √ k𝑣k Φ = 𝑣 ⊤ Φ𝑣 to denote the Φ-weighted Euclidean norm of 𝑣. For matrices 𝐴, 𝐵 ∈ R𝑑×𝑑 , 𝐴 𝐵
(resp., 𝐴 𝐵) means that 𝐴 − 𝐵 (resp., 𝐵 − 𝐴) is positive semidefinite. For 𝑥, 𝑦 ∈ R, we write
𝑥 ∨ 𝑦 = max{𝑥, 𝑦}. For any positive semidefinite matrix 𝑀 ∈ R𝑑×𝑑 , 𝜆 min (𝑀) denotes its minimum
eigenvalue. For any integer 𝑘 ≥ 1, let [𝑘] = {1, 2, . . . , 𝑘}. Given any set 𝑆, we use 𝑆 𝑐 to denote the
complement of 𝑆. We write 𝜔 ∼ 𝑝(·) to indicate that the random variable 𝜔 follows distribution 𝑝(·). We use 𝑁 (𝜃, 𝜎 2 ) to denote the Gaussian distribution with mean 𝜃 and variance 𝜎 2 . Given
any 𝜇 ∈ R𝑑 and positive definite matrix Σ ∈ R𝑑×𝑑 , we use N (𝜇, Σ) to denote the 𝑑-dimensional Gaussian distribution with mean 𝜇 and covariance Σ. For any two random variables 𝑋, 𝑌 , 𝑋 ⊥ ⊥𝑌
means 𝑋 and 𝑌 are independent. For any random event E, we use I{E} to denote the indicator of E. The notation “a.s.” stands for “almost surely.” For any vector 𝑥 or matrix 𝐴, 𝑥 ⊤ and 𝐴⊤
denote their transposes. Finally, for any twice-differentiable function 𝑔, 𝑔′ and 𝑔′′ denote its firste(·) to hide logarithmic factors. Specifically, for functions and second-order derivatives. We use 𝑂
e(𝑔(𝑇)) if there exists a constant 𝐶 > 0 and a poly-logarithmic 𝑓 , 𝑔 : N → R+ , we write 𝑓 (𝑇) = 𝑂
function polylog(𝑇) such that 𝑓 (𝑇) ≤ 𝐶𝑔(𝑇)polylog(𝑇). For random variables 𝑌1 and 𝑌2 , 𝑌1 ® 𝑌2
denotes that 𝑌1 ≤ 𝑐𝑌2 with high probability for some constant 𝑐 > 0. For two positive sequences
𝑓 (𝑇) and 𝑔(𝑇), we write 𝑓 (𝑇) ≍ 𝑔(𝑇) if both 𝑓 (𝑇) ® 𝑔(𝑇) and 𝑔(𝑇) ® 𝑓 (𝑇) hold up to universal constant factors.
2. Model In this section, we formally introduce our online contextual Pandora’s box model motivated by LLM cascading. In our model, there are 𝐴 different boxes, each of which can be thought of as an API in LLM cascading. There are 𝑇 decision periods. Let F𝑡−1 denote the history available before period 𝑡, including all past observations and actions. At the beginning of period 𝑡, the decision-maker (DM) observes a request context vector 𝑥𝑡 ∈ X ⊂ R𝑑 𝑥 , where 𝑥𝑡 are independent across 𝑡 ∈ [𝑇]. For each box 𝑎 ∈ [𝐴], 𝜔 𝑎𝑡 , defined as the potential output vector of 𝑎, is random and its distribution is assumed to be independent across boxes and independent of the past history. Formally, for each 𝑎 ∈ [𝐴], 𝜔 𝑎𝑡 is drawn from a context-dependent distribution 𝑝 𝑎 (·|𝑥𝑡 ), and for any measurable sets 𝐵1 , . . . , 𝐵 𝐴 ⊆ Ω, we have P(𝜔1𝑡 ∈ 𝐵1 , . . . , 𝜔 𝐴𝑡 ∈ 𝐵 𝐴 | F𝑡−1 , 𝑥𝑡 ) =
𝐴 Ö 𝑎=1
𝑝 𝑎 (𝐵𝑎 | 𝑥𝑡 )
a.s.
(1)
8
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
The output 𝜔 𝑎𝑡 is revealed only if box 𝑎 is queried. In addition, if 𝑎 is queried, a cost 𝑐 𝑎 (𝑥𝑡 , 𝜔 𝑎𝑡 ) is incurred and observed, where 𝑐 𝑎 : X × Ω → (0, 1) is the cost function associated with box 𝑎. Finally, at the end of each period, the DM selects one queried box 𝑎 𝑡 whose generated output 𝜔 𝑎 𝑡 𝑡 will be deployed. The DM then receives and observes the realized reward associated with the deployed output, 𝑟 𝑡 = 𝜇∗ (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) + 𝜁𝑡 , where 𝜇∗ (·, ·) : X × Ω → [0, 1] is the (unknown) reward function and 𝜁𝑡 is the post-deployment noise satisfying E[𝜁𝑡 | F𝑡−1 , 𝑥𝑡 , 𝜔 𝑎 𝑡 ,𝑡 ] = 0. A key feature of our model is that the reward function is shared across boxes. Conditional on the same request-output pair (𝑥, 𝜔), the expected downstream reward is the same regardless of which box generated the output. Thus, box-specific heterogeneity only enters through the distributions of outputs and costs, while realized rewards depend solely on the common reward function 𝜇∗ (·, ·) and noise 𝜁𝑡 . Consequently, outputs 𝜔 𝑎𝑡 act as intermediaries linking boxes to rewards, which permits cross-learning of 𝜇∗ using reward observations from all selected boxes. This reflects our motivating LLM example, where the reward received by the DM depends on the generated output and the request it serves, not directly on the identity of the API that generated it. We preview the parametric structures used for learning, with the formal assumptions and estimators introduced in Sections 4 and 5. Suppose, as an oracle benchmark, the DM knows the conditional output distributions {𝑝 𝑎 (· | 𝑥)} 𝑎∈[𝐴],𝑥∈X , the cost functions {𝑐 𝑎 } 𝑎∈[𝐴] , and the reward
function 𝜇∗ . Then, after the context 𝑥𝑡 is observed, the decision problem in period 𝑡 reduces to a
contextual Pandora’s box problem of the type studied by Weitzman (1979). As we formally illustrate in Section 3, the optimal full-information policy is determined by two objects: the reward function 𝜇∗ , and the reservation indices, denoted as {𝜎𝑎∗ (𝑥𝑡 )} 𝑎∈[𝐴] . These indices are, in turn, determined by
the oracle objects 𝑝 𝑎 (·|𝑥𝑡 ), 𝜇∗ , and 𝑐 𝑎 through the reservation index equation (2) in Proposition 1. In the online learning problem, these oracle objects are not known. Rather than estimating the full conditional output distributions 𝑝 𝑎 (· | 𝑥), we impose generalized linear parametric structure directly
on the reward function 𝜇∗ and on the contextual reservation index functions {𝜎𝑎∗ (·)} 𝑎∈[𝐴] , namely
𝜇∗ (𝑥, 𝜔) = 𝐺 (𝜃 ∗⊤ 𝜙(𝑥, 𝜔)) and 𝜎𝑎∗ (𝑥) = Λ(𝜌 𝑎⊤ 𝜓(𝑥)) where the coefficients 𝜃 ∗ and 𝜌 𝑎 are unknown;
𝐺, Λ are known (monotonic) link functions and 𝜙, 𝜓 are known feature maps, respectively. This allows the DM to learn the shared reward and box specific contextual reservation indices through parametric estimation. R EMARK 1. Our contextual Pandora’s box model maps to LLM cascading as follows. A period corresponds to a service request, and a box corresponds to an LLM API or an API-prompt configuration. Querying box 𝑎 once produces one candidate output 𝜔 𝑎𝑡 , the DM then either stops or
9
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
queries another box in the cascade. We therefore restrict each box to be queried at most once within a period. This is consistent with the LLM cascading literature (Chen et al. 2025a), where a request is routed through a sequence of APIs and each API is invoked only if previous outputs are deemed to be not sufficiently reliable by the evaluator. The output-dependent cost 𝑐 𝑎 (𝑥𝑡 , 𝜔 𝑎𝑡 ) also arises naturally in LLM applications. For example, under token-based pricing, 𝑐 𝑎𝑡 = 𝛼𝑎 𝐼𝑎𝑡 + 𝛽𝑎 𝑂 𝑎𝑡 , where 𝐼𝑎𝑡 is the input token count of the query and 𝑂 𝑎𝑡 is the (random) number of output tokens. Here 𝐼𝑎𝑡 = 𝜅 𝑎,𝑖 (𝑥𝑡 ) and 𝑂 𝑎𝑡 = 𝜅 𝑎,𝑜 (𝜔 𝑎𝑡 ), where for each 𝑎 ∈ [𝐴], 𝜅 𝑎,𝑖 : R𝑑 𝑥 → R+ , 𝜅 𝑎,𝑜 : R𝑑 𝑤 → R+ are fixed functions.
2.1. DM’s Problem and Regret Next, we describe the decision-making pipeline in each period 𝑡 ∈ [𝑇] in more detail. Each period 𝑡 ∈ [𝑇] consists of two phases, which we call query phase and selection phase. During the query phase, the DM observes 𝑥𝑡 and sequentially queries boxes. A policy is nonanticipating: before each query, the DM may use F𝑡−1 , the current context 𝑥𝑡 , and the output-cost pairs of the boxes already observed in period 𝑡, but not the output or cost of any unqueried box. Let A𝑡 ⊆ [𝐴] denote the random set of boxes queried in period 𝑡. We assume each box can be queried at most once within a period. After each query, the DM observes the generated output and cost, and then decides whether to continue querying or stop. This non-anticipativity condition implies a useful sampling property. For any 𝑎 ∈ [𝐴], conditional on (F𝑡−1 , 𝑥𝑡 ), the event {𝑎 ∈ A𝑡 } is determined before observing box 𝑎’s own potential output-cost pair. Hence, whenever P(𝑎 ∈ A𝑡 | F𝑡−1 , 𝑥𝑡 ) > 0, 𝑑
(𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ) | F𝑡−1 , 𝑥𝑡 , {𝑎 ∈ A𝑡 } = (𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ) | F𝑡−1 , 𝑥𝑡 . This property will be used later to justify learning reservation indices from adaptively queried samples. After the query phase concludes, the DM enters the selection phase, selecting one queried box 𝑎 𝑡 ∈ A𝑡 whose generated output 𝜔 𝑎 𝑡 𝑡 will be deployed. Let Π𝑡 denote the class of admissible non-anticipating policies in period 𝑡. For a policy 𝜋𝑡 ∈ Π𝑡 , let A𝑡 (𝜋𝑡 ) be the set of queried boxes and 𝑎 𝑡 (𝜋𝑡 ) ∈ A𝑡 (𝜋𝑡 ) be the selected box. The conditional expected utility of policy 𝜋𝑡 in period 𝑡 is
Õ ∗ 𝑐 𝑎𝑡 F𝑡−1 , 𝑥𝑡 . 𝑈𝑡 (𝜋𝑡 ; F𝑡−1 , 𝑥𝑡 ) := E 𝜇 (𝑥𝑡 , 𝜔 𝑎 𝑡 (𝜋𝑡 )𝑡 ) − 𝑎∈A 𝑡 (𝜋𝑡 )
We note that the post-deployment noise 𝜁𝑡 does not enter the utility because it is conditionally mean-zero. It matters only as feedback for learning the reward function 𝜇∗ (·, ·).
10
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
We compare the DM against the full-information benchmark that knows the distribution 𝑝 𝑎 (·|𝑥𝑡 ),
and functions 𝜇∗ (𝑥𝑡 , ·), 𝑐 𝑎 (𝑥𝑡 , ·) for every box 𝑎 and every context 𝑥𝑡 . At each period 𝑡, after observing 𝑥𝑡 , the optimal policy 𝜋𝑡∗ ∈ arg max𝜋𝑡 ∈Π𝑡 𝑈𝑡 (𝜋𝑡 ; F𝑡−1 , 𝑥𝑡 ) reduces to the solution of the Pandora’s box
problem given context 𝑥𝑡 , which we formally characterize in Section 3. Define the condition expected utility of 𝜋𝑡∗ as Õ ∗ ∗ 𝑐 𝑎𝑡 F𝑡−1 , 𝑥𝑡 . 𝑈𝑡 (𝜋𝑡 ; F𝑡−1 , 𝑥𝑡 ) = E max∗ 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝑎∈A 𝑡
𝑎∈A 𝑡∗
Then, for any admissible policy 𝝅 = {𝜋𝑡 }𝑡∈[𝑇 ] , the cumulative regret is defined as 𝑇 Õ 𝑅𝑇 (𝝅) := E 𝑈𝑡 (𝜋𝑡∗ ; F𝑡−1 , 𝑥𝑡 ) − 𝑈𝑡 (𝜋𝑡 ; F𝑡−1 , 𝑥𝑡 ) . 𝑡=1
Readers familiar with the bandit literature might ask whether the problem can be modeled as a contextual bandit by treating each API as an arm. Such a formulation would lead to a one-shot APIas-arm policy, in which the decision-maker selects a single API for each request before observing any generated output. It does not, however, capture the adaptive information-acquisition structure of LLM cascading. The following example illustrates that such a restriction can create a constant per-period gap relative to the sequential-search benchmark, even when all primitives are known. Consider an example with two boxes and the context 𝑥𝑡 is fixed throughout. Querying box 1 costs 𝑐 1 = 0.5 and always generates output 𝜔1𝑡 = 1. Querying box 2 costs 𝑐 2 = 0.01 and generates output 𝜔2𝑡 = 1 with probability 0.1, and 𝜔2𝑡 = 0 otherwise. The downstream value is 𝜇∗ (𝑥𝑡 , 𝜔) = 𝜔. A one-shot API-as-arm policy obtains expected utility 0.5 from box 1 and 0.09 from box 2, so the best one-shot policy selects box 1. Now consider a cascade that first queries box 2. If 𝜔2𝑡 = 1, it stops and selects box 2; if 𝜔2𝑡 = 0, it queries box 1 and selects box 1. Its expected utility is −0.01 + 0.1 + 0.9(0.5) = 0.54 > 0.5. Thus, relative to the cascade benchmark, any one-shot APIas-arm formulation suffers a constant per-period optimality gap in this instance, and hence a loss that grows linearly with 𝑇.
3. Policy and Algorithm This section develops the online policy and the algorithm. We begin with the full-information benchmark. Fix a period 𝑡 and condition on the arriving context 𝑥𝑡 . Suppose the DM knows the reward function 𝜇∗ (·, ·) and the output distributions {𝑝 𝑎 (·|𝑥)} 𝑎∈[𝐴],𝑥∈X . The following proposition characterizes the optimal oracle policy during period 𝑡.
11
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
P ROPOSITION 1 (Oracle reservation index). For each box 𝑎 ∈ [𝐴] and any fixed 𝑡 ∈ [𝑇], let
𝜎𝑎∗ (·) be the index function such that h i + E 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎∗ (𝑥𝑡 ) 𝑥𝑡 = E[𝑐 𝑎𝑡 | 𝑥𝑡 ], ∀𝑥𝑡 .
(2)
Then there exists an optimal policy 𝜋𝑡∗ with the following structure: At any period 𝑡 with context 𝑥𝑡 , ∗ (𝑥 ) ≥ · · · ≥ 𝜎 ∗ (𝑥 ), sets 𝜎 ∗ it orders the boxes {(1), (2), . . . , ( 𝐴)} such that 𝜎(1) (𝑥 ) := −∞, 𝑡 ( 𝐴) 𝑡 ( 𝐴+1) 𝑡
∗ queries boxes in this order, stops at the 𝑘-th query if max1≤ 𝑗≤𝑘 𝜇∗ (𝑥𝑡 , 𝜔 ( 𝑗)𝑡 ) ≥ 𝜎(𝑘+1) (𝑥𝑡 ), and selects
𝑎 ∗𝑡 ∈ arg max𝑎∈A𝑡∗ 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), where A𝑡∗ is the set of boxes queried by the oracle. Moreover, the
oracle value satisfies 𝑈𝑡 (𝜋𝑡∗ ; F𝑡−1 , 𝑥𝑡 ) = E
max min 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝜎𝑎∗ (𝑥𝑡 )
𝑎∈[𝐴]
𝑥𝑡 , F𝑡−1 .
(3)
∗ := 𝜎 ∗ (𝑥 ). When the costs 𝑐 are known constants (i.e., 𝑐 ≡ 𝑐 for each 𝑎 ∈ [𝐴]), the Define 𝜎𝑎𝑡 𝑎𝑡 𝑎𝑡 𝑎 𝑎 𝑡
∗} indices {𝜎𝑎𝑡 𝑎∈[𝐴] are optimal by the seminal work of Weitzman (1979). When costs are stochastic
(as in our motivating setting), the result above follows from a minor adaptation of the proof in (Kleinberg et al. 2016) (see Appendix A). Proposition 1 formally identifies the reward evaluator 𝜇∗ (𝑥, 𝜔) and the reservation index function
𝜎𝑎∗ (𝑥) as the two objects required to make the optimal sequential decisions. We next describe the online policy as Algorithm 1. At a high level, the policy follows the oracle structure in Proposition 1,
but replaces the unknown reward evaluator and reservation indices with optimistic estimates. We call the resulting policy COst-aware Sequential MOdel query and Selection, or COSMOS . R EMARK 2. Our policy estimates the reservation index function 𝜎𝑎∗ (𝑥𝑡 ) through the conditional moment equation (2) in Proposition 1. Note that equation (2) characterizes the index at the population level. In our setting, however, the output-cost pair (𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ) is observed only when box 𝑎 is queried, that is, when 𝑎 ∈ A𝑡 . Hence, a direct empirical analogue of (2) cannot use unqueried boxes, whose outputs and costs are not observed. Nevertheless, conditioning on box 𝑎 being queried, the same moment equation continues to identify the reservation index. That is, we have E {𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎∗ (𝑥𝑡 )}+ − 𝑐 𝑎𝑡 |𝑥𝑡 , F𝑡−1 , 𝑎 ∈ A𝑡 = 0.
(4)
This is because conditional on 𝑥𝑡 , F𝑡−1 , the query event 𝑎 ∈ A𝑡 is determined before the realized output-cost pair (𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ) is observed, and hence is independent of this pair. Thus although outputs
and costs are only observed for queried boxes, these observations continue to identify 𝜎𝑎∗ (𝑥𝑡 ).
12
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
3.1. Optimism and Regret Decomposition Our COSMOS policy constructs optimistic estimates 𝜇 e𝑡 (𝑥, 𝜔) and e 𝜎𝑎𝑡 , ∀𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇]. The
estimate e 𝜇𝑡 (𝑥, 𝜔) is an optimistic estimate of 𝜇∗ (𝑥, 𝜔) for a realized output. The estimate e 𝜎𝑎𝑡 is
an optimistic estimate of the oracle reservation index 𝜎𝑎∗ (𝑥𝑡 ) for box 𝑎 at the current context. The
formal construction of these estimates is postponed to Sections 4 and 5, and here we focus only on their roles in the policy.
To understand why optimism is crucial in our COSMOS policy, we next present a theorem showing that, under optimism, the one-period regret decomposes into the reward estimation error for the selected output and the index estimation errors for the queried boxes. T HEOREM 1 (Regret decomposition under optimism). Fix any period 𝑡, and let Δ𝑡 (e 𝜋 ) := 𝑈𝑡 (𝜋𝑡∗ ; 𝑥𝑡 ) − 𝑈𝑡 (e 𝜋𝑡 ; 𝑥 𝑡 ) denote the period-𝑡 conditional regret of the policy e 𝜋 induced by Algorithm 1. Suppose that, for all
∗ . Let 𝑎 ∈ arg max 𝑎 ∈ [𝐴], 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) ≥ 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), e 𝜎𝑎𝑡 ≥ 𝜎𝑎𝑡 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) be the output selected 𝑡 𝑎∈A 𝑡 e
by Algorithm 1. Then,
Δ𝑡 (e 𝜋) ≤ E 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇∗ (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) | F𝑡−1 , 𝑥𝑡 + E
"
Õ
𝑎∈A 𝑡
∗
#
e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 | F𝑡−1 , 𝑥𝑡 .
We provide intuition on why Theorem 1 holds. Consider an auxiliary “optimistic” problem in which both its reward estimate e 𝜇𝑡 and its index e 𝜎𝑎𝑡 upper-bound the truth. Then COSMOS is the
optimal policy for the “optimistic” problem, and its corresponding optimistic expected value is thus at least as large as the optimal expected value of the true problem. As a result, the loss can
be bounded by the discrepancy between the optimistic quantities used by COSMOS and the true quantities realized along the same path. This yields exactly two errors: a reward estimation error for the selected output, and an index-estimation error for every queried box. The decomposition in Theorem 1 allows us to leverage the principle of optimism to control our regret. More specifically, if a box is queried frequently, its query dataset grows and its index uncertainty decreases. If a box is queried rarely, then it contributes only rarely to the second term in Theorem 1. This mirrors the classic self-correcting principle behind UCB analysis, adapted here to govern reservation indices and reward rather than the direct feedback in bandits. Consequently, Theorem 1 motivates us to construct high-probability confidence bounds for both reward and reservation indices and using them to define the optimistic estimators 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) and
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
13
ALGORITHM 1: COSMOS Policy Framework Initialization. Observe an initial context 𝑥0 . Query each box 𝑎 ∈ [𝐴] once and record the observed output-cost pairs {(𝜔 𝑎0 , 𝑐 𝑎0)} 𝑎∈[𝐴] . for 𝑡 = 1, 2, . . . , 𝑇 do Observe context 𝑥𝑡 . Initialize A𝑡 = ∅ and 𝑀𝑡 = −∞; Construct optimistic reservation index estimates {e 𝜎𝑎𝑡 } 𝑎∈[𝐴] ;
Order the boxes so that e 𝜎(1)𝑡 ≥ e 𝜎(2)𝑡 ≥ · · · ≥ e 𝜎( 𝐴)𝑡 . Set e 𝜎( 𝐴+1)𝑡 := −∞; for 𝑘 = 1, 2, . . . , 𝐴 do
Query box (𝑘) and observe its output and cost (𝜔 (𝑘)𝑡 , 𝑐 (𝑘)𝑡 );
Add (𝑘) to the queried set: A𝑡 ← A𝑡 ∪ {(𝑘)};
Compute the optimistic reward estimate e 𝜇𝑡 (𝑥𝑡 , 𝜔 (𝑘)𝑡 ); Update 𝑀𝑡 ← max𝑎∈A𝑡 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 );
if 𝑀𝑡 ≥ e 𝜎(𝑘+1)𝑡 , stop querying and break;
end
Select and deploy 𝑎 𝑡 ∈ arg max𝑎∈A𝑡 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ), and observe reward 𝑟 𝑡 ;
end
e 𝜎𝑎𝑡 . The goal is to ensure that, with high probability, these estimators upper bound their population counterparts while remaining sufficiently accurate. Together, optimism and error control provide
the key ingredients for deriving the final regret guarantee of the COSMOS policy, whose high-level structure is summarized as Algorithm 1. The full implementation, including the computation of optimistic reservation indices and reward estimates, is deferred to Algorithm 2 in the Appendix.
4. Results under Known Reward Function To build intuition, we first study the case in which the reward function 𝜇∗ (𝑥, 𝜔) is known. This setting is practically relevant when the reward model can be estimated offline using a substantially larger historical dataset. If the offline data are sufficiently rich, the resulting estimate of reward √ function 𝜇∗ (·, ·) may be accurate enough relative to the 𝑇 scale of the online regret analysis, to be treated as fixed in the subsequent online decision problem. Importantly, even with a known reward function, the conditional distribution of the output 𝜔 𝑎𝑡 given 𝑥𝑡 typically remains unknown. This case therefore captures regimes in which the main online learning challenge is not reward estimation, but learning the reservation indices and managing exploration across boxes. Indeed,
14
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
once box 𝑎 is opened in period 𝑡, the decision-maker observes its output 𝜔 𝑎𝑡 and cost 𝑐 𝑎𝑡 , and can therefore compute the conditional expected reward 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ). Thus, in Algorithm 1, we would set 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) = 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ). The selection rule then chooses the opened box with the largest true
conditional expected reward, and the regret component arising from reward estimation disappears. The remaining learning problem is then focused solely on estimating 𝜎𝑎∗ (𝑥𝑡 ) for each 𝑎.
This is analogous to the bandit-feedback setting in Atsidakou et al. (2024), in that feedback is obtained only from opened boxes.1 Atsidakou et al. (2024) develop a black-box reduction from contextual Pandora’s Box to online regression, using a specially designed linear-quadratic loss to learn all the reservation indices. In contrast, our approach follows the principle of optimism from the UCB literature. As we illustrate, under suitable regularity conditions, it is not necessary to learn all boxes’ index functions uniformly well. Instead, it is enough to maintain optimistic index estimates and refine the estimates of boxes that remain relevant to the query process. We next formalize this intuition. Recall from Section 2 that we impose the parametric structure on the indices 𝜎𝑎∗ (𝑥) = Λ(𝜌 𝑎⊤ 𝜓(𝑥)),
∀𝑥 ∈ X,
(5)
where Λ is a known monotonic link function, and 𝜓(·) is a known feature map. We impose the following regularity condition on the parametric reservation index function (5): A SSUMPTION 1 (Regularity conditions for reservation indices). The link function Λ : R → [−1, 1] is continuously differentiable, strictly increasing, and 𝐿-Lipschitz, lim𝑠→−∞ Λ(𝑠) = −1,
lim𝑠→+∞ Λ(𝑠) = 1. Moreover, inf 𝜌∈B, 𝑥∈X Λ′ (𝜌 ⊤ 𝜓(𝑥)) ≥ 𝜇1 > 0, k𝜓(𝑥)k 2 ≤ 𝐶¯𝜓 for all 𝑥 ∈ X. The parameter space B ⊂ R𝑚 is convex and compact, with diameter 𝑑 B : sup 𝜌,𝜌′ ∈B k 𝜌 − 𝜌 ′ k 2 ≤ 𝑑 B .
The range restriction Λ : R → [−1, 1] is consistent with the fact that 𝑐 𝑎𝑡 ∈ (0, 1) and 𝜇∗ ∈ [0, 1],
∗ ∈ [−1, 1]. In addition, Assumption 1 imposes only mild regularity conditions. which imply 𝜎𝑎𝑡
It accommodates the linear reservation index specification considered by Atsidakou et al. (2024), which corresponds to taking Λ as the identity map on the relevant, strictly bounded domain of 𝜌 ⊤ 𝜓(𝑥), while allowing it to smoothly flatten to ±1 as 𝑠 → ±∞. Assumption 1 yields the point identification of 𝜌 𝑎⊤𝜓(𝑥𝑡 ):
P ROPOSITION 2. Under Assumption 1, 𝜌 𝑎⊤𝜓(𝑥𝑡 ) is point identified through (4) for each 𝑎 ∈ [𝐴]. 1 The authors also study a full information setting where the DM observes the rewards of all boxes at the end of each round.
15
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
At the beginning of period 𝑡, Algorithm 1 estimates 𝜌 𝑎 by 𝜌ˆ 𝑎𝑡 and constructs an upper confidence bound for the linear index 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ). Let R 𝑎𝑡 (𝑥𝑡 ) denote the corresponding confidence radius, and ⊤ 𝜓(𝑥 ) + R (𝑥 ) . Let E denote the high-probability event on which define e 𝜎𝑎𝑡 = Λ 𝜌ˆ 𝑎𝑡 𝑡 𝑎𝑡 𝑡 𝜌 ⊤ 𝜌ˆ 𝑎𝑡 𝜓(𝑥𝑡 ) − 𝜌 𝑎⊤𝜓(𝑥𝑡 ) ≤ R 𝑎𝑡 (𝑥𝑡 ),
∀𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇].
Then on E 𝜌 , by the monotonicity of Λ, ∗ e 𝜎𝑎𝑡 ≥ Λ(𝜌 𝑎⊤𝜓(𝑥𝑡 )) = 𝜎𝑎𝑡 ,
∀𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇],
allowing us to invoke the regret decomposition of Theorem 1. Recall that e 𝜇𝑡 = 𝜇∗ in the known-𝜇∗
case, implying the reward estimation term in Theorem 1 is zero. Therefore, under the event E 𝜌 , Theorem 1 implies that the period-𝑡 regret satisfies # " Õ ∗ e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 𝑥𝑡 , F𝑡−1 , Δ𝑡 ( 𝜋) ˜ ≤E
(6)
𝑎∈A 𝑡
where A𝑡 is the random set of boxes opened by Algorithm 1 in period 𝑡. Inequality (6) illustrates that the regret depends only on the index estimation errors of boxes that are actually opened. Thus, frequently opened boxes yield larger sample sizes that shrink their confidence radii, while rarely opened boxes contribute negligibly to the overall regret. Once we obtain a valid high-probability confidence radius for the linear index 𝜌 𝑎⊤𝜓(𝑥𝑡 ), the cumulative regret can be controlled by the standard elliptical-potential argument used for contextual bandits (e.g., Abbasi-Yadkori et al. 2011). However, unlike in standard contextual bandits, an opened box does not produce a noisy observation ∗ of its reservation index 𝜎𝑎𝑡 ; but instead, a tuple (𝑥𝑡 , 𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ). Therefore, the index parameter must
be learned indirectly from the defining equation (4) of the reservation index. 4.1. Index Estimation and Loss Function We next describe how to estimate the box-specific reservation index function 𝜎𝑎∗ (·). Given (4) and (5), our estimation problem is moment-based and is thus inspired by the generalized method of moments (GMM). For this purpose, we define the known-𝜇∗ moment function, for any 𝑥 ∈ X, 𝜔 ∈ Ω, 𝑐 ∈ [0, 1], and 𝜌 ∈ B, as
+ 𝑚 ∗ (𝑥, 𝜔, 𝑐; 𝜌) := 𝑐 − 𝜇∗ (𝑥, 𝜔) − Λ(𝜌 ⊤ 𝜓(𝑥)) . ∗ By Equation (4), we have E 𝑚 (𝑥𝑡 , 𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ; 𝜌 𝑎 ) F𝑡−1 , 𝑥𝑡 , 𝑎 ∈ A𝑡 = 0. Next, let S𝑎𝑡 := {𝑠 < 𝑡 : 𝑎 ∈ A 𝑠 },
𝑛𝑎𝑡 := |S𝑎𝑡 |
(7)
16
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
denote the set and number of past periods in which box 𝑎 was queried before period 𝑡, and the elements in S𝑎𝑡 be ordered as 𝜏𝑎,1 < 𝜏𝑎,2 < · · · < 𝜏𝑎,𝑛 𝑎𝑡 . As we discuss below, our estimation problem is a special case of the GMM framework with conditional moment conditions. By setting the moment conditions based on the features 𝜓(𝑥 𝑠 ), we can formulate it as a 𝑀-estimation problem that is more suitable for our online learning setting. A standard GMM approach would instead form the empirical vector moment 𝑔¯ 𝑎𝑡 (𝜌) :=
1 Õ 𝜓(𝑥 𝑠 )𝑚 ∗ (𝑥 𝑠 , 𝜔 𝑎𝑠 , 𝑐 𝑎𝑠 ; 𝜌) 𝑛𝑎𝑡 𝑠∈S𝑎𝑡
and minimize a quadratic criterion such as 𝑔¯ 𝑎𝑡 (𝜌) ⊤𝑊𝑎𝑡 𝑔¯ 𝑎𝑡 (𝜌) for some positive definite weighting matrix 𝑊𝑎𝑡 at each period 𝑡. However, this approach focuses on estimating the coefficients 𝜌 𝑎 , which requires a full-rank condition on the local GMM curvature. Such a requirement can be overly stringent in our setting, as it requires that the observations in which box 𝑎 is queried contain sufficiently many “active” samples, namely those satisfying 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝜌 𝑎⊤ 𝜓(𝑥𝑡 )), and that the corresponding feature vectors 𝜓(𝑥𝑡 ) span the parameter space in a well-conditioned way conditional on 𝑎 ∈ A𝑡 . Our objective, however, is not to recover the entire vector 𝜌 𝑎 . For the UCB analysis, it is enough to construct a confidence interval for the scalar quantity 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ) at the current context. We therefore adopt an 𝑀-estimation formulation, whose first-order condition recovers the sample moment. For a queried sample of box 𝑎 in period 𝑠, define ∫ 𝜌⊤ 𝜓(𝑥𝑠 ) ℓ𝑎𝑠 (𝜌) := 𝑐 𝑎𝑠 − (𝜇∗ (𝑥 𝑠 , 𝜔 𝑎𝑠 ) − Λ(𝑢)) + 𝑑𝑢.
(8)
0
Note that ℓ𝑎𝑠 (𝜌) is convex in 𝜌, as Λ(·) is strictly increasing. Also, the derivative of ℓ𝑎𝑠 satisfies + ∇ 𝜌 ℓ𝑎𝑠 (𝜌) = 𝜓(𝑥 𝑠 ) 𝑐 𝑎𝑠 − 𝜇∗ (𝑥 𝑠 , 𝜔 𝑎𝑠 ) − Λ(𝜌 ⊤𝜓(𝑥 𝑠 )) = 𝜓(𝑥 𝑠 )𝑚 ∗ (𝑥 𝑠 , 𝜔 𝑎𝑠 , 𝑐 𝑎𝑠 ; 𝜌).
(9)
Therefore, the first-order condition of the empirical loss minimization is equivalent to the sample analogue of the vector moment equation. The known-𝜇∗ estimator of 𝜌 𝑎 for period 𝑡 is 𝜌ˆ 𝑎𝑡 ∈ arg min 𝜌∈B
Õ
𝑠∈S𝑎𝑡
ℓ𝑎𝑠 (𝜌).
(10)
We next state the regularity condition that yields curvature of the population loss. The condition requires that 𝜌 ⊤𝜓(𝑥) remains in a bounded region, and for every point in this region, there is some probability mass that the reward of the output is above its corresponding reservation index.
17
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
A SSUMPTION 2 (Local mass around reservation thresholds). There exist constants 𝜄¯ > 0 and 𝜅 > 0 such that, for all 𝑥 ∈ X and 𝜌 ∈ B, |𝜓(𝑥) ⊤ 𝜌| ≤ 𝜄¯, and for all 𝑎 ∈ [𝐴], 𝑥 ∈ X, |𝑢| ≤ 𝜄¯, we have
P𝜔∼𝑝 𝑎 (·|𝑥) (𝜇∗ (𝑥, 𝜔) > Λ(𝑢)) ≥ 𝜅.
Assumption 2 rules out the cases in which the reservation index lies in a region with no probability mass above it. Together with the lower derivative bound Λ′ (·) ≥ 𝜇1 from Assumption 1, it ensures that the reservation index moment crosses zero with slope bounded away from zero. This is the one-dimensional source of curvature for the loss function. L EMMA 1 (Population curvature of the loss function). Under Assumptions 1, 2, for any 𝑡 ∈ [𝑇], 𝑎 ∈ [𝐴] and 𝜌 ∈ B, E[ℓ𝑎𝑡 (𝜌) − ℓ𝑎𝑡 (𝜌 𝑎 ) | 𝑥𝑡 , F𝑡−1 ] ≥
2 𝜅𝜇1 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) . 2
(11)
The curvature bound in Lemma 1 illustrates the advantage for using the 𝑀-estimation formulation. It shows that the population loss increases quadratically in the prediction error 𝜓(𝑥 𝑠 ) ⊤ (𝜌 − 𝜌 𝑎 ). As a preview of the subsequent analysis, we will compare this quadratic curvature against the empirical Í fluctuation of the additive loss 𝑠∈S𝑎𝑡 ℓ𝑎𝑠 (𝜌). By controlling this fluctuation uniformly over 𝜌 ∈ B
via martingale concentration, we can absorb it into the curvature term. This procedure yields a
high-probability bound on |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )|, providing the confidence radius needed to construct the optimistic index e 𝜎𝑎𝑡 in (6).
4.2. Confidence Bound for a Fixed Box Next, we construct confidence radii for the reservation indices. For each fixed box 𝑎 ∈ [𝐴], because
∗ Λ(·) is monotone and 𝜎𝑎𝑡 = Λ(𝜌 𝑎⊤ 𝜓(𝑥𝑡 )), we focus on constructing a confidence interval for the
scalar index 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ) at the realized context 𝑥𝑡 . We define 𝐻𝑎𝑡 (𝜌), the empirical fluctuation of the
loss at 𝜌, as 𝐻𝑎𝑡 (𝜌) :=
Õ
𝑠∈S𝑎𝑡
{ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ]} .
By optimality of 𝜌ˆ 𝑎𝑡 , the empirical excess loss
Í
𝑠∈S𝑎𝑡 {ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 )} is non-positive. This
sum can be decomposed by its conditional expectation and empirical fluctuation as Õ
𝑠∈S𝑎𝑡
{ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 )} =
Õ
𝑠∈S𝑎𝑡
(12)
E [ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] + 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ).
18
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Lemma 1 implies that the conditional expectation is bounded below by the quadratic form Í 𝜅𝜇1 /2 𝑠∈S𝑎𝑡 {𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )} 2 . Consequently, to establish a confidence interval for 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ),
it suffices to bound the empirical fluctuation 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) using a comparable quadratic term and a
logarithmic complexity term, as established in the following lemma. L EMMA 2. Suppose Assumptions 1, 2 hold. Fix 𝑎 ∈ [𝐴], and given any constant 𝑐 0 > 0, with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇], for any 𝜌𝑡 adapted to F𝑡−1 we have 𝑐0 Õ 144 ⊤ 2 −𝐻𝑎𝑡 (𝜌𝑡 ) ≤ {(𝜌𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + + 𝐶0 (𝑚 log 𝑇 + log(𝑇/𝛿)), 8 𝑐0 𝑠∈S𝑎𝑡
where 𝐶0 > 0 is a constant depending on 𝑑 B , 𝐶¯𝜓 , 𝜄. Lemma 2 is derived through standard concentration arguments. For a fixed 𝜌, 𝐻𝑎𝑡 (𝜌) form a martingale difference sequence whose conditional variance is controlled by the quadratic term that appears in the population curvature lemma. A peeling argument over the size of this quadratic term and an 𝜖-net argument over B yield the uniform bound and hence for adaptive 𝜌𝑡 .
We now build the confidence interval for the scalar index 𝜌 𝑎⊤𝜓(𝑥𝑡 ) using Lemmas 1 and 2. Define
the regularized empirical design matrix 𝑉𝑎𝑡 (𝜂1 ) := 𝜂1 I𝑚 +
Õ
𝑠∈S𝑎𝑡
𝜓(𝑥 𝑠 )𝜓(𝑥 𝑠 ) ⊤ ,
(13)
where 𝜂1 > 0 is a regularization parameter and I𝑚 is the 𝑚 × 𝑚 identity matrix. Intuitively, 𝑉𝑎𝑡 (𝜂1 ) summarizes the contexts in which box 𝑎 has been queried before period 𝑡. The next proposition establishes a high-probability bound on |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )|. P ROPOSITION 3. Suppose Assumptions 1, 2 hold. When 𝜇∗ is known, fix any 𝑎 ∈ [𝐴], given any 𝛿 > 0, with probability at least 1 − 𝛿, uniformly over all 𝑡 ∈ [𝑇], we have |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )| ≤ 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 ,
(14)
q 2 , and 𝐶 is a constant depending on the constants where 𝐵∗𝑎𝑡 = 𝐶1 [𝑚 log(𝑇) + log(𝑇 𝐴/𝛿)] + 𝜂1 𝑑 B 1 𝜅, 𝜇1 , 𝜄¯, 𝐶¯𝜓 , 𝑑 B defined in Assumptions 1, 2. Proposition 3 has the usual online-learning interpretation (see, e.g., Abbasi-Yadkori et al. 2011, Lattimore and Szepesvári 2020) as the confidence bounds used in bandit UCB algorithms. It is large when the current context points in a direction that has not been well explored for box 𝑎, and small when the historical opened-box contexts provide enough information in that direction.
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Motivated by Proposition 3, we define the optimistic index in the known-𝜇∗ case as ⊤ e 𝜎𝑎𝑡 = Λ 𝜌ˆ 𝑎𝑡 𝜓(𝑥𝑡 ) + 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 ,
19
(15)
∗ Since Λ is increasing, Proposition 3 implies e 𝜎𝑎𝑡 ≥ 𝜎𝑎𝑡 uniformly over 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇] with proba-
bility at least 1 − 𝛿. This, in turn, is used to establish the uniform optimism event required for the regret guarantee. In implementation, one may calibrate this radius using a bootstrap approximation to the distribution of ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 ), which may yield less conservative confidence bounds. Such a bootstrap procedure is not covered by the present theory and proving its validity would require a uniform bootstrap approximation under adaptive sampling. 4.3. Regret under Known Reward Function Now we bound the cumulative regret when 𝜇∗ is known so that 𝜇 e𝑡 = 𝜇∗ . When the indices are optiÍ ∗ ∗ mistic, i.e., e 𝜎𝑎𝑡 ≥ 𝜎𝑎𝑡 , ∀𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], Theorem 1 gives Δ𝑡 ( 𝜋) 𝜎𝑎𝑡 − 𝜎𝑎𝑡 𝑥𝑡 , F𝑡−1 . ˜ ≤ E 𝑎∈A𝑡 e
∗ 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ≤ Using the Lipschitz continuity of Λ and the confidence bound in Proposition 3, we obtain e p 2𝐿𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 on the high-probability event, where 𝐵∗𝑎𝑡 ≤ O( 𝑚 log(𝑇) + log(𝑇 𝐴)). Thus Í Í the cumulative regret is controlled by 𝑇𝑡=1 𝑎∈A𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 . Every time box 𝑎 is queried, the
corresponding feature vector 𝜓(𝑥𝑡 ) is added to its design matrix. Therefore, the uncertainty in the same direction decreases over time. The standard elliptical-potential argument implies that for each box 𝑎 ∈ [𝐴],
𝑇 Õ 𝑡=1
√ e(𝑚 𝑇). 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 I{𝑎 ∈ A𝑡 } ≤ 𝑂
(16)
Summing this bound over the 𝐴 boxes yields the following regret guarantee: P ROPOSITION 4. Suppose Assumptions 1, 2 hold. When the reward function 𝜇∗ is known, " 𝑇 # √ Õ e E Δ𝑡 ( 𝜋) ˜ ≤ 𝑂 𝐴𝑚 𝑇 . 𝑡=1
e(𝑇 5/6) bound of Atsidakou et al. (2024) for a more general contextual PanCompared with the 𝑂 √ e 𝑇) rate relies on the additional local-curvature structure in Assumption 2, dora’s Box setting, our 𝑂( which enables UCB-style confidence control for the reservation indices.
R EMARK 3. The bound in (16) is pathwise and therefore Proposition 4 does not require 𝑥𝑡 to be independent. It holds for any sequence of contexts, including 𝑥𝑡 that are predictable with respect to F𝑡−1 . The independence assumption is, however, needed for the analysis when the reward function is unknown.
20
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
5. Results under Unknown Reward Function We now drop the assumption that 𝜇∗ is known and study the problem where 𝜇∗ must be learned online. The known-𝜇∗ result before guides the analysis when 𝜇∗ is unknown. Relative to the previous section, two additional challenges arise: (i) The reward estimation term in Theorem 1 is no longer zero. (ii) The reservation index estimation can no longer use the true conditional expected reward 𝜇∗ (𝑥 𝑠 , 𝜔 𝑎𝑠 ) when defining the loss. Recall from Section 2 that we impose the generalized linear parametric structure on the expected reward function: 𝜇∗ (𝑥, 𝜔) = 𝐺 (𝜃 ∗⊤ 𝜙(𝑥, 𝜔)),
∀(𝑥, 𝜔) ∈ X × Ω,
(17)
where 𝐺 : R → [0, 1] is strictly increasing. In Section 5.1 we address (i) using a standard penalized maximum likelihood estimator (MLE) construction based on (17). In Section 5.2 we address (ii) by showing that the main intuitions from Proposition 3 continue to work. Particularly, Proposition 8 extends Proposition 3 to this setting by accounting for the plug-in error for the loss. Similarly, the index estimator is controlled by population curvature, empirical concentration, and now an additional perturbation term from reward estimation. Once the reward parameter is learned at the √ e( 𝑇) regret rate. standard online rate, this perturbation is small enough to preserve the overall 𝑂 However, challenge (ii) with unknown reward is more delicate because reward learning uses
selected outputs while index estimation is based on queried outputs. Hence controlling the plugin loss requires an additional assumption ensuring that the reward-design matrix is sufficiently informative for the queried features that enter the index estimator. We begin by imposing the following regularity condition on the expected reward function: A SSUMPTION 3 (Regularity conditions for expected reward function). (i) 𝐺 (·) is continuously differentiable and strictly increasing, with Lipschitz constant 𝐿. Furthermore, 𝜇 :=
inf 𝜃∈Θ,𝑥∈X,𝜔∈Ω 𝐺 ′ (𝜃 ⊤ 𝜙(𝑥, 𝜔)) > 0, and for some absolute constant 𝐿 1 > 0, |𝐺 ′′ (𝜃 ⊤ 𝜙(𝑥, 𝜔))| ≤ 𝐿 1 , ∀𝑥 ∈ X, 𝜔 ∈ Ω, 𝜃 ∈ Θ. (ii) For an absolute constant 𝐶¯𝜙 , 𝛼¯ > 0, k𝜙(𝑥, 𝜔)k 2 ≤ 𝐶¯𝜙 , ∀𝜔 ∈ Ω, 𝑥 ∈ X and k𝜃 ∗ k 2 ≤ 𝛼. ¯ Assumption 3 is standard in the generalized linear contextual bandit literature (e.g., Filippi et al. 2010, Li et al. 2017, Lee et al. 2024). The Lipschitz constant 𝐿 controls how errors in 𝜃ˆ𝑡−1 propagate into the reward prediction, the lower bound 𝜇 on 𝐺 ′ ensures uniform strong convexity of the objective, and the boundedness of 𝜙 and 𝜃 ∗ keeps the analysis in the standard regime.
21
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
We next assume that the random reward observed by the DM is drawn from a canonical exponential family: A SSUMPTION 4. Given any 𝑥 ∈ X and 𝜔 ∈ Ω, conditioning on the pair of input context and output of the selected API (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) = (𝑥, 𝜔) during period 𝑡, the density of the random reward
𝑟 𝑡 = 𝐺 (𝜃 ∗⊤ 𝜙(𝑥, 𝜔)) + 𝜁𝑡 with respect to a reference measure is given by
𝑓 (𝑟 |𝑥, 𝜔) = exp 𝑟𝜃 ∗⊤ 𝜙(𝑥, 𝜔) − 𝑏(𝜃 ∗⊤ 𝜙(𝑥, 𝜔)) + 𝜉 (𝑟) ,
(18)
where 0 ≤ 𝑟 𝑡 ≤ 𝛾0 a.s. for some constant 𝛾0 > 1, 𝜉 (·) is a real function, 𝑏(·) is twice continuously differentiable and E[𝜁𝑡 |𝜔 𝑎 𝑡 𝑡 , 𝑥𝑡 , F𝑡−1 ] = 0 for any 𝑡 ∈ [𝑇] almost surely. Assumption 4 is standard in generalized linear contextual bandit (Filippi et al. 2010, Li et al. 2017, Zhou et al. 2019, Lee et al. 2024, etc.), which encompasses Bernoulli, logistic, and binomial reward distributions, and more generally the exponential family (e.g. Gaussian, Poisson, etc.) with support truncated to [0, 𝛾0 ]. By definition, taking derivative of 𝑏 we get 𝑏 ′ = 𝐺 as the inverse link function. Common examples of 𝐺 include any known cumulative distribution function (CDF) such as the logistic and probit CDFs. 5.1. Estimating the Reward Parameter The reward estimation follows essentially a generalized linear contextual bandit framework. At each period 𝑡 the deployment (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) yields a reward observation 𝑟 𝑡 , and the dataset {(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 , 𝑟 𝑘 )}𝑡−1 𝑘=1 accumulates over time. We estimate 𝜃 ∗ by a penalized (projected) maximum likelihood estimator 𝜃ˆ𝑡−1 := arg min 𝜃∈Θ
𝑡−1 Õ 𝑟 𝑘 − 𝐺 𝜃 ⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) − 𝜂0 𝜃 𝑘=1
where Φ𝑡−1 = 𝜂0 I𝑑 +
𝑡−1 Õ 𝑘=1
,
(19)
Φ𝑡−1 −1
𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 )𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) ⊤
(20)
is the regularized design matrix of reward features collected through period 𝑡 − 1, and 𝜂0 > 0 is a regularization parameter, I𝑑 is the 𝑑 × 𝑑 identity matrix. The term within the weighted norm k · k Φ−1 in (19) is the gradient of the objective function for the unconstrained penalized log𝑡 −1 Í likelihood max𝜃 𝑡−1 log 𝑓 (𝑟 𝑘 | 𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) − 𝜂0 k𝜃 k 2 , and 𝜃ˆ𝑡−1 can be viewed a “projection” onto Θ 𝑘=1
2
2
if the unconstrained solution falls outside Θ. This construction follows Filippi et al. (2010).
The following standard result provides a high-probability bound on the prediction error induced by using 𝜃ˆ𝑡−1 to evaluate the reward of a candidate context-output pair (𝑥𝑡 , 𝜔 𝑎𝑡 ).
22
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
L EMMA 3. Suppose Assumptions 3 and 4 hold and fix any 𝛿 ∈ (0, min{1, 2𝑑/𝑒}). With probability
at least 1 − 𝛿/2, uniformly over all 𝑎 ∈ [𝐴] and 𝑡 ≥ 2,
( 𝜃ˆ𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) ≤ 𝛽𝑡 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) Φ−1 , 𝑡 −1 p where 𝛽𝑡 = O 𝑑 log(𝑡) + log(1/𝛿) defined as in (61).
This is the ellipsoidal confidence bound for generalized linear bandits, which has similar structure as Proposition 3 for the index parameter estimation error bound. The matrix Φ𝑡−1 plays the role
that 𝑉𝑎𝑡 (𝜂1 ) played for the indices, and 𝛽𝑡 grows logarithmically in 𝑡. The proof of Lemma 3 adapts Filippi et al. (2010) via a self-normalized martingale inequality. Equipped with this bound, we construct the optimistic reward as
⊤ 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) := 𝐺 𝜃ˆ𝑡−1 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) + 𝛽𝑡 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) Φ−1 .
(21)
𝑡 −1
Because 𝐺 is monotonically increasing by Assumption 3, Lemma 3 implies that with probability at least 1 − 𝛿/2, uniformly over all 𝑎 ∈ [𝐴] and 𝑡 ≥ 2, 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) ≥ 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ). 5.2. Estimating Indices
Next, we estimate the reservation indices and construct their associated confidence intervals for the regime where the reward function is unknown. As an unknown-reward analogue to Section 4.2, we replace the true reward 𝜇∗ (𝑥 𝑠 , 𝜔 𝑎𝑠 ) by its plug-in estimate 𝐺 ( 𝜃ˆ⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) for each 𝑠 ≤ 𝑡. 𝑡−1
The main goal is to show that this substitution preserves the same confidence-bound structure as Proposition 3, up to an additional perturbation term controlled by the reward estimation error bound. Fix a period 𝑡 and a box 𝑎 ∈ [𝐴]. For any 𝜌 ∈ B and any past queried sample 𝑠 ∈ S𝑎𝑡 , define the plug-in loss ℓˆ𝑎𝑠,𝑡 (𝜌) :=
∫ 𝜌⊤ 𝜓(𝑥𝑠 ) h 0
i + ⊤ 𝑐 𝑎𝑠 − 𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢) 𝑑𝑢.
(22)
This is the same loss as (8), with 𝜇∗ replaced by the current reward estimate. The plug-in index estimator is 𝜌ˆ 𝑎𝑡 ∈ arg min 𝜌∈B
Õ
𝑠∈S𝑎𝑡
ℓˆ𝑎𝑠,𝑡 (𝜌).
(23)
The first-order condition of (23) is the empirical analogue of the Weitzman moment equation with the reward function evaluated at 𝜃ˆ𝑡−1 . The new challenge compared to Section 4.2 is that the empirical loss in (23) is not the oracle loss. To isolate this difference, define the plug-in perturbation Õ 𝐽𝑎𝑡 (𝜌) := ℓˆ𝑎𝑠,𝑡 (𝜌) − ℓ𝑎𝑠 (𝜌) − ℓˆ𝑎𝑠,𝑡 (𝜌 𝑎 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) , (24) 𝑠∈S𝑎𝑡
23
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
where ℓ𝑎𝑠 (𝜌) is the known-𝜇∗ loss in (8). The term 𝐽𝑎𝑡 (𝜌) measures how the estimation error of 𝜃ˆ𝑡−1 propagates into the index-estimation loss.
We impose the following additional assumption for the unknown-reward case. It ensures that the reward-design matrix contains enough information in all directions needed to control the plugin perturbation. We use polylog(𝑇) to denote a quantity bounded by a polynomial in log 𝑇 (i.e. polylog(𝑇) = O((log 𝑇) 𝑐 ) for some constant 𝑐 > 0). A SSUMPTION 5 (Anti-concentration of reward features). There exists a sequence 𝑀𝑇 ≥ 1 with p 𝑀𝑇 = polylog(𝑇)/𝐴 such that, for any 𝑡 ∈ [𝑇], 𝑎 ∈ [𝐴], 𝑣 ∈ S𝑑−1 , and 0 < 𝜖 ≤ (2𝐴𝑀𝑇 ) −1 , P |𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 ≤ 𝑀𝑇 𝜖
a.s.
Recall that the contexts 𝑥𝑡 are independent across time, implying that the distribution of the reward feature 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) is independent of F𝑡−1 . Assumption 5 leverages this independence to impose an anti-concentration condition that rules out degenerate reward features. Informally, every one-dimensional projection of 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) must have nontrivial variation near zero. This condition is imposed because 𝜃 ∗ is learned only from selected outputs, whereas the index loss for box 𝑎 uses queried outputs. Thus, the reward features must be sufficiently informative for evaluating the queried features that enter the index estimator. For example, it is satisfied when the embedding distribution is a truncated Gaussian, a truncated multivariate Student’s 𝑡, or a uniform distribution on a hyperrectangle; see Lemmas 12–14 in Appendix D. R EMARK 4. Our regret analysis can be extended to the case where 𝑥𝑡 is predictable with respect to F𝑡−1 with a more restrictive version of Assumption 5. Specifically, it would require that under the same conditions as stated in Assumption 5, we have P |𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 | F𝑡−1 , 𝑥𝑡 ≤ 𝑀𝑇 𝜖
a.s.
(25)
However, such an assumption implies structural restrictions on the reward features. For example, suppose 𝐺 is the identity map, 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) = (𝑥𝑡 , 𝜔 𝑎𝑡 ) and for simplicity, suppose both 𝑥𝑡 and 𝜔 𝑎𝑡 are one-dimensional. If 𝑥𝑡 is predictable from F𝑡−1 , then conditional on F𝑡−1 the context coordinate is fixed. In particular, if 𝑥1 follows a Bernoulli distribution with P(𝑥1 = 1) = 0.5 and 𝑥𝑡 = 𝑥𝑡−1 for 𝑡 ≥ 2, then the projection along the context direction 𝑣 = (1, 0) is identically zero when 𝑥1 = 0, so (25) fails. Thus, for general predictive 𝑥𝑡 , even a simple linear embedding can fail to satisfy the anti-concentration property.
24
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
The following lemma bounds the additional perturbation caused by using 𝜃ˆ𝑡−1 in the index loss. L EMMA 4 (Plug-in loss perturbation). Suppose Assumptions 1 and 3 hold. Then, for any 𝑐 0 > 0, −𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) ≤
2𝐿 2 Õ ˆ 𝑐0 Õ 2 2 𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) + ( 𝜃 𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 ) . 8 𝑐0 𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
Lemma 4 shows why Assumption 5 is needed. The first term on the right-hand side has the same quadratic form as the population curvature in Lemma 1 and can therefore be absorbed into the curvature argument. The second term is the reward estimation error evaluated on queried outputs. By Lemma 3, this term is controlled by the inverse-design norms k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k Φ−1 , where Φ𝑡−1 is 𝑡 −1
constructed from previously selected outputs, while 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 ) corresponds to queried outputs used for estimating the index of box 𝑎. Thus, there is a potential mismatch between the selected outputs that inform reward learning and the queried outputs that enter index estimation. Assumption 5 rules out severe mismatches of this form by ensuring that Φ𝑡−1 is sufficiently well conditioned, so these inverse-design norms do not remain large. We now state the confidence bound for the index estimator: P ROPOSITION 5 (Index confidence bound with unknown reward function). Suppose Assumptions 1–5 hold. Fix any 𝛿 > 0. Then, with probability at least 1 − 2𝛿/3, uniformly over all 𝑎 ∈ [𝐴] and all 𝑡 ∈ [𝑇], ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 ) ≤ 𝐵𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 ,
(26)
where 𝑉𝑎𝑡 (𝜂1 ) is defined as (13), s Õ 𝐵𝑎𝑡 = 𝐶2 𝑚 log(𝑇) + log( 𝐴𝑇/𝛿) + [𝑑 log(𝑡) + log(1/𝛿)] k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k 2 −1 , 𝑠∈S𝑎𝑡
Φ𝑡 −1
¯ and 𝐶2 is a constant depending on 𝜂0 , 𝜂1 , 𝜅, 𝜇1 , 𝜄¯, 𝑑 B , 𝐶¯𝜓 , 𝐿, 𝐶¯𝜙 , 𝛾0 , 𝜇, and 𝛼. The proof follows the same localized empirical-process argument as Proposition 3. By the Í optimality of 𝜌ˆ 𝑎𝑡 in (23), 𝑠∈S ℓˆ𝑎𝑠,𝑡 ( 𝜌ˆ 𝑎𝑡 ) − ℓˆ𝑎𝑠,𝑡 (𝜌 𝑎 ) ≤ 0. Decomposing this into the oracle 𝑎𝑡
population curvature, the empirical fluctuation, and the plug-in perturbation gives Õ
𝑠∈S𝑎𝑡
E[ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] + 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) ≤ 0.
The first term is lower bounded by the curvature result in Lemma 1, the second is controlled by the empirical-process bound in Lemma 2, and the third is controlled by Lemma 4 together with the
25
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
reward confidence bound in Lemma 3. Combining these bounds yields (26). The detailed proof is provided in Appendix E. Motivated by Proposition 5, define the optimistic reservation index as ⊤ e 𝜎𝑎𝑡 := Λ 𝜌ˆ 𝑎𝑡 𝜓(𝑥𝑡 ) + 𝐵𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 .
(27)
Since Λ is increasing, Proposition 5 implies that, with probability at least 1 − 𝛿, e 𝜎𝑎𝑡 ≥ Λ(𝜌 𝑎⊤ 𝜓(𝑥𝑡 )) = ∗ uniformly over all 𝑎 ∈ [𝐴] and 𝑡 ∈ [𝑇]. 𝜎𝑎𝑡
5.3. Regret Analysis under Unknown Reward Function We now combine the reward and index confidence bounds to control the cumulative regret of COSMOS when the reward function is unknown. The preceding subsections established two highprobability optimism events: Lemma 3 gives optimism for the reward estimate e 𝜇𝑡 , and Proposition 5
gives optimism for the reservation index estimate e 𝜎𝑎𝑡 . Therefore, when both events hold, the regret decomposition in Theorem 1 applies and separates the period-𝑡 regret into a reward estimation term
and an index-estimation term for all 𝑎 ∈ [𝐴] and 𝑡 ∈ [𝑇]. This structure parallels the known-reward analysis in Section 4. When the reward function is unknown, an additional generalized linear bandit
term appears from estimating the reward parameter. As shown in Appendix D, Assumption 5 implies that, with high probability, 𝜆 min (Φ𝑡−1 ) ≥
𝑡−1 16 polylog(𝑇 )
uniformly for all 𝑡 ≥ 𝜏 := 𝐶polylog(𝑇) log( 𝐴𝑑) for some absolute constant 𝐶. This
eigenvalue growth allows the reward estimation error in 𝐽𝑎𝑡 (𝜌) to be controlled uniformly over 𝑡 ≥ 𝜏, so that the cumulative regret is controlled after 𝑡 ≥ 𝜏. For these initial periods before ⌈𝜏⌉, the per-period regret is bounded by (2𝐴 + 1). Since 𝜏 = O(log( 𝐴𝑑)polylog(𝑇)), so the cumulative regret up to period ⌈𝜏⌉ is at most O( 𝐴 log( 𝐴𝑑)polylog(𝑇)), and is therefore dominated by the regret accumulated over periods 𝑡 ≥ 𝜏. T HEOREM 2. Suppose Assumptions 1–5 hold. Then, ignoring logarithmic factors, E
" 𝑇 Õ 𝑡=1
#
√ √ e Δ𝑡 ( 𝜋) ˜ ≤ 𝑂 [𝑑 + 𝐴(𝑚 + 𝑚𝑑)] 𝑇 .
In particular, the regret bound is asymptotically minimized by choosing
𝜂1 ≍ max 𝑚
1/2
, (𝑑𝑚)
1/4
,𝑑
1/3
,
2/3 ) 𝑑 𝑑 . 𝜂0 ≍ min √ , √ 𝐴 𝑚 𝐴 𝑚 (
26
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
The terms in Theorem 2 correspond directly to the two components in Theorem 1. The term √ e(𝑑 𝑇) is the reward estimation regret, matching the usual dimension dependence for generalized 𝑂 √ √ e( 𝐴(𝑚 + 𝑑𝑚) 𝑇) is the linear contextual bandits with reward feature dimension 𝑑. The term 𝑂
index-estimation regret, where 𝐴 is the number of boxes and 𝑚 is the dimension of the index feature √ e( 𝐴 𝑑𝑚𝑇 ) reflects the 𝜓. Relative to the known-reward case in Proposition 4, the additional term 𝑂 effect of reward estimation error on index estimation. On the high-probability optimism event, TheÍ Í Í Í ∗ orem 1 gives E 𝑇𝑡=1 Δ𝑡 ( 𝜋) 𝜎𝑎𝑡 − 𝜎𝑎𝑡 . ˜ ≤ E 𝑇𝑡=1 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇∗ (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) + E 𝑇𝑡=1 𝑎∈A𝑡 e
The first sum is controlled by the reward confidence radius from Lemma 3 and an elliptical-
potential argument (Abbasi-Yadkori et al. 2011) for the reward-design matrix Φ𝑡−1 . The second sum is controlled by the index confidence radius from Proposition 5 and a separate elliptical-potential argument for each box-specific design matrix 𝑉𝑎𝑡 (𝜂1 ). Thus the unknown-reward analysis adds a standard reward-learning term while preserving the same index-learning rate as in the known-reward benchmark. The detailed proof is provided in Appendix F.
6. Conclusion Motivated by LLM cascading, we introduce an online contextual Pandora’s Box model in which a decision-maker sequentially queries APIs to generate outputs at a cost and deploys a single output, observing only its downstream reward. This output-mediated feedback departs from the classical setting, where opening a box reveals its reward directly, and captures a key feature of LLM cascading systems: API-specific heterogeneity enters only through the distributions of generated outputs and costs, while downstream value is governed by a shared reward evaluator on the contextoutput pair. Rather than estimating the full conditional output and cost distributions, we impose a generalized linear structure directly on the reservation indices and the shared reward function. Our COSMOS policy is built by combining GMM estimation of the reservation indices with UCB-style confidence bounds for both the indices and the reward evaluator. Through a regret decomposition under optimism, which separates cumulative regret into reward-estimation errors for deployed √ √ e [𝑑 + 𝐴(𝑚 + 𝑑𝑚)] 𝑇 regret outputs and index-estimation errors for queried APIs, we derive a O
bound.
Our work opens several avenues for future research. First, because our theoretical guarantees rely on generalized linear specifications, extending the moment-based UCB framework to richer nonparametric or neural representations is a natural next step. Achieving this while preserving √ e( 𝑇) regret would further broaden the model’s applicability. Second, our current formulation 𝑂
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
27
assumes a standard LLM cascading structure where APIs are queried sequentially and a single output is selected and then deployed. Expanding this framework to allow for batched querying under latency budgets, or to support output ensembling and synthesis, would significantly change the query-selection dynamics. Addressing these richer action spaces raises compelling new questions about the structure of the learning policy.
AI Disclosure The authors used generative artificial intelligence tools, including GPT, Gemini, and Claude, to edit and polish earlier versions of the drafts. All AI-generated text and suggestions were checked, revised, and approved by the authors. The authors take full responsibility for the accuracy, integrity, and originality of the submitted work.
Acknowledgment We thank Ali Makhdoumi for helpful discussions on the general theory and intuition of Pandora’s Box, and Shreyas Sekar for an early conversation on AI-assisted coding and model selection for reducing LLM API costs that eventually led us to the FrugalGPT paper by Chen et al. (2025a). We also thank the participants from 2026 Marketplace Innovation Workshop for comments and discussion. Appendix A: Proofs for Section 3 In this section we prove three results in Section 3: the optimality of the oracle reservation index policy under full information (Proposition 1), point identification of 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ) from the queried samples for box 𝑎 before period 𝑡 (Proposition 2), and a regret decomposition under optimistic reward and index estimators (Theorem 1). The following lemma establishes that the query decision is independent of the stochastic output conditioning on the historical data and the current context at each period 𝑡. Although the proof is immediate, the result is used repeatedly in subsequent arguments, so we state it explicitly here. L EMMA 5. Given any 𝑡 ∈ [𝑇], 𝑥𝑡 ∈ X and 𝑎 ∈ [𝐴], we have I{𝑎 ∈ A𝑡 } ⊥ ⊥ (𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 )|𝑥𝑡 , F𝑡−1 . Proof of Lemma 5 For any 𝑥𝑡 ∈ X and 𝑎 ∈ [𝐴], under both the oracle reservation index policy and COSMOS algorithm, the event {𝑎 ∈ A𝑡 } is determined by the sequential construction, which depends on the history F𝑡−1 , the current context 𝑥𝑡 , and the outputs of boxes queried before 𝑎.
Under the conditional independence of {𝜔 𝑎 ′ 𝑡 : 𝑎 ′ ∈ [𝐴]} given 𝑥𝑡 according to (1), these preceding
28
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
outputs are independent of 𝜔 𝑎𝑡 . Further, 𝑐 𝑎𝑡 = 𝑐 𝑎 (𝑥𝑡 , 𝜔 𝑎𝑡 ). Hence, I{𝑎 ∈ A𝑡 } ⊥ ⊥ (𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 )|𝑥𝑡 , F𝑡−1 as claimed.
Proof of Proposition 1 For 𝑡 ∈ [𝑇], let 𝑄 𝑡𝑎 = I{𝑎 ∈ A𝑡 } and 𝑆 𝑡𝑎 = I{𝑎 𝑡 = 𝑎}. Since 𝜔 𝑎𝑡 ∼ 𝑝 𝑎 (·|𝑥𝑡 ), (2) implies E Thus
h
𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎∗ (𝑥𝑡 )
+
i 𝑥𝑡 , F𝑡−1 = E[𝑐 𝑎𝑡 | 𝑥𝑡 , F𝑡−1 ], ∀𝑥𝑡 .
Õ Õ 𝑡 ∗ 𝑡 𝑆 𝑎 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝑄 𝑎 𝑐 𝑎𝑡 𝑥𝑡 , F𝑡−1 E 𝑎∈[𝐴] 𝑎∈[𝐴] Õ Õ 𝑡 ∗ 𝑡 ∗ ∗ + 𝑆 𝑎 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝑄 𝑎 {𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 } 𝑥𝑡 , F𝑡−1 = (𝑖) E 𝑎∈[𝐴] 𝑎∈[𝐴] Õ Õ 𝑡 ∗ ∗ 𝑡 𝑡 ∗ ∗ + 𝑆 𝑎 min{𝜎𝑎𝑡 , 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 )} + (𝑆 𝑎 − 𝑄 𝑎 ){𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 } 𝑥𝑡 , F𝑡−1 = (𝑖𝑖) E 𝑎∈[𝐴] 𝑎∈[𝐴] Õ ∗ 𝑆 𝑡𝑎 min{𝜎𝑎𝑡 , 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 )} 𝑥𝑡 , F𝑡−1 ≤ (𝑖𝑖𝑖) E 𝑎∈[𝐴] ∗ ≤ (𝑖𝑣) E max min{𝜎𝑎𝑡 , 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 )} 𝑥𝑡 , F𝑡−1 , 𝑎∈[𝐴]
∗ where (i) holds because 𝑄 𝑡𝑎 ⊥ ⊥ (𝜔 𝑎𝑡 , 𝑐 𝑎𝑡 ) | 𝑥𝑡 , F𝑡−1 according to Lemma 5 and the definition of 𝜎𝑎𝑡
in (2) imply E[𝑄 𝑡𝑎 𝑐 𝑎𝑡 |𝑥𝑡 , F𝑡−1 ] = E[𝑄 𝑡𝑎 |𝑥𝑡 , F𝑡−1 ]E[𝑐 𝑎𝑡 |𝑥𝑡 , F𝑡−1 ]
∗ + = E[𝑄 𝑡𝑎 |𝑥𝑡 , F𝑡−1 ]E[{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 } |𝑥𝑡 , F𝑡−1 ]
∗ + = E[𝑄 𝑡𝑎 {𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 } |𝑥𝑡 , F𝑡−1 ],
(ii) holds by the identity 𝑏 = min{𝑎, 𝑏} + {𝑏 − 𝑎}+ , (iii) since 𝑆 𝑡𝑎 ≤ 𝑄 𝑡𝑎 , and (iv) since
and 𝑆 𝑡𝑎 ≥ 0.
Í
𝑡 𝑎∈[𝐴] 𝑆 𝑎 = 1
The inequalities (iii) and (iv) hold as equalities when using Weitzman’s principle via indices ∗ and selects the maximum of defined by (2). Indeed the policy selects 𝑎 whenever 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > 𝜎𝑎𝑡
∗ }. So the utility of the DM 𝑈 satisfies max𝑎∈[𝐴] min{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝜎𝑎𝑡 𝑡
Õ Õ ∗ ∗ 𝑡 ∗ 𝑡 𝑆 𝑎 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝑄 𝑎 𝑐 𝑎𝑡 𝑥𝑡 , F𝑡−1 = E max min{𝜎𝑎𝑡 , 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 )} 𝑥𝑡 , F𝑡−1 , 𝑈𝑡 = E 𝑎∈[𝐴] 𝑎∈[𝐴] 𝑎∈[𝐴] (28)
thus concludes the proof.
29
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Proof of Proposition 2 For any 𝑥𝑡 ∈ X and 𝑎 ∈ [𝐴], we have I{𝑎 ∈ A𝑡 } ⊥ ⊥ 𝜔 𝑎𝑡 |𝑥𝑡 , F𝑡−1 by Lemma 5. Since 𝑐 𝑎𝑡 = 𝑐 𝑎 (𝑥𝑡 , 𝜔 𝑎𝑡 ) we have
E {𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝜌 𝑎⊤ 𝜓(𝑥𝑡 ))}+ − 𝑐 𝑎𝑡 |𝑥𝑡 , F𝑡−1 , 𝑎 ∈ A𝑡 = E {𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝜌 𝑎⊤ 𝜓(𝑥𝑡 ))}+ − 𝑐 𝑎𝑡 |𝑥𝑡 , F𝑡−1 .
For any given 𝑎 ∈ [𝐴] and 𝑥, define
𝐹𝑎 (𝑥, 𝑠) := E {𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠)}+ − 𝑐 𝑎𝑡 |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 .
So we have 𝐹𝑎 𝑥, 𝜌 𝑎⊤ 𝜓(𝑥) = 0, ∀𝑥 ∈ X. Note that Λ(·) is strictly increasing and takes values in [−1, 1], and 𝑐 𝑎𝑡 ∈ (0, 1), so for every 𝑥 ∈ X, 𝐹𝑎 (𝑥, 𝑠) is decreasing and continuous in 𝑠. Note that lim𝑠→−∞ 𝐹𝑎 (𝑥, 𝑠) = lim𝑠→−∞ E [{𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠)}+ − 𝑐 𝑎𝑡 |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] = (𝑖) E {𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) + 1}+ − 𝑐 𝑎𝑡 |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡
≥ E [1 − 𝑐 𝑎𝑡 |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] = (𝑖𝑖) E [1 − 𝑐 𝑎𝑡 |𝑥𝑡 = 𝑥, F𝑡−1 ] > (𝑖𝑖𝑖) 0
where (i) follows from Assumption 1, (ii) follows because 𝑐 𝑎𝑡 = 𝑐 𝑎 (𝑥𝑡 , 𝜔 𝑎𝑡 ) and 𝜔 𝑎𝑡 ⊥ ⊥ I{𝑎 ∈ A𝑡 }|𝑥𝑡 , F𝑡−1 by Lemma 5, (iii) follows because 𝑐 𝑎𝑡 ∈ (0, 1). Thus lim𝑠→−∞ 𝐹𝑎 (𝑥, 𝑠) > 0. Similarly, Assumption 1 implies lim𝑠→+∞ 𝐹𝑎 (𝑥, 𝑠) < 0. Thus by intermediate value theorem, there exists a 𝜉 𝑎 (𝑥) ∈ R such that 𝐹𝑎 (𝑥, 𝜉 𝑎 (𝑥)) = 0. Now suppose that there exists 𝑠1 < 𝑠2 such that 𝐹𝑎 (𝑥, 𝑠1 ) = 𝐹𝑎 (𝑥, 𝑠2 ) = 0. Then we have 0 = 𝐹𝑎 (𝑥, 𝑠1 ) − 𝐹𝑎 (𝑥, 𝑠2 ) = E[(𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠1 )) + − (𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠2 )) + | 𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ]. (29) This implies that E[(𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠1 )) + |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] = E[(𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠2 )) + |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] = E[𝑐 𝑎𝑡 |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] > 0. Since 𝑠1 < 𝑠2 and Λ is strictly increasing, there must exists an event E where 𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) > Λ(𝑠2 ) > Λ(𝑠1 ) and P(E |𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ) > 0, and 𝐹𝑎 (𝑥, 𝑠1 ) − 𝐹𝑎 (𝑥, 𝑠2 ) = E[(𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠1 )) + − (𝜇∗ (𝑥, 𝜔 𝑎𝑡 ) − Λ(𝑠2 )) + | 𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] ≥ E[(Λ(𝑠2 ) − Λ(𝑠1 ))I{E}|𝑥𝑡 = 𝑥, F𝑡−1 , 𝑎 ∈ A𝑡 ] > 0, which contradicts (29). So the conditional moment restriction pins down the optimal index 𝜎𝑎∗ (𝑥) =
Λ(𝜉 𝑎 (𝑥)) uniquely, and 𝜉 𝑎 (𝑥) = 𝜌 𝑎⊤ 𝜓(𝑥) for any 𝑥 ∈ X.
30
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Proof of Theorem 1 Define 𝑈𝑡 ( 𝜋) ˜ := 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) −
Í
𝑎∈A 𝑡 𝑐 𝑎𝑡 , where 𝜋˜ is the COSMOS pol-
icy. Let 𝑐˜𝑎𝑡 := (e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + be defined as an auxiliary pseudo-cost for any 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇]. Let 𝑈˜ 𝑡 ( 𝜋) ˜ denote the realized utility of 𝜋˜ under cost 𝑐˜𝑎𝑡 , i.e. 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝑈˜ 𝑡 ( 𝜋) ˜ := max e 𝑎∈A 𝑡
Õ
𝑐˜𝑎𝑡 .
𝑎∈A 𝑡
By (3) in Proposition 1, we have
∗ E[𝑈𝑡 (𝜋 ∗ ) | 𝑥𝑡 , F𝑡−1 ] = E max min{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝜎𝑎𝑡 } | 𝑥𝑡 , F𝑡−1 , 𝑎∈[𝐴] ˜ E 𝑈𝑡 ( 𝜋) ˜ | 𝑥𝑡 , F𝑡−1 = E max min{e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ), e 𝜎𝑎𝑡 } | 𝑥𝑡 , F𝑡−1 . 𝑎∈[𝐴]
Note that for the same execution of 𝜋˜ pathwise,
𝑈𝑡 ( 𝜋) ˜ = 𝑈˜ 𝑡 ( 𝜋) ˜ − (e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )) −
Õ
𝑎∈A 𝑡
(𝑐 𝑎𝑡 − 𝑐˜𝑎𝑡 ).
Therefore, Δ𝑡 ( 𝜋) ˜ = E[𝑈𝑡 (𝜋 ∗ ) | 𝑥𝑡 , F𝑡−1 ] − E[𝑈𝑡 ( 𝜋) ˜ | 𝑥𝑡 , F𝑡−1 ] ∗ = E max min{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝜎𝑎𝑡 } | 𝑥𝑡 , F𝑡−1 − E max min{e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ), e 𝜎𝑎𝑡 } | 𝑥𝑡 , F𝑡−1 (30) 𝑎∈[𝐴] 𝑎∈[𝐴] Õ (𝑐 𝑎𝑡 − 𝑐˜𝑎𝑡 ) | 𝑥𝑡 , F𝑡−1 . +E[e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) | 𝑥𝑡 , F𝑡−1 ] + E 𝑎∈A 𝑡
∗ ∗ Since 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) ≥ 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) and e 𝜎𝑎𝑡 ≥ 𝜎𝑎𝑡 a.s., min{e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ), e 𝜎𝑎𝑡 } ≥ min{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝜎𝑎𝑡 } a.s.
for every 𝑎 ∈ [𝐴]. So
∗ E max min{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝜎𝑎𝑡 } | 𝑥𝑡 , F𝑡−1 − E max min{e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ), e 𝜎𝑎𝑡 } | 𝑥𝑡 , F𝑡−1 ≤ 0. 𝑎∈[𝐴]
𝑎∈[𝐴]
(31)
Further, for each 𝑎 ∈ [𝐴],
∗ ∗ = (e 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 )) + (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ) ∗ ≤ (e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ).
Taking positive parts and using subadditivity of 𝑥 ↦→ 𝑥 + gives
∗ + ∗ + ∗ (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 ) ≤ (e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + + (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ) = (e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + + (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ).
Hence
∗ + E[𝑐 𝑎𝑡 − 𝑐˜𝑎𝑡 | 𝑥𝑡 , F𝑡−1 ] = E[(𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − 𝜎𝑎𝑡 ) | 𝑥𝑡 , F𝑡−1 ] − E[(e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + | 𝑥𝑡 , F𝑡−1 ] ∗ ≤e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 .
(32)
31
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Thus E
Í
𝑎∈A 𝑡 (𝑐 𝑎𝑡 − 𝑐˜𝑎𝑡 ) 𝑥 𝑡 , F𝑡−1 =
Õ
Õ
𝑎∈[𝐴]
= (𝑖)
E[(𝑐 𝑎𝑡 − 𝑐˜𝑎𝑡 )I{𝑎 ∈ A𝑡 } | 𝑥𝑡 , F𝑡−1 ]
Õ
𝑎∈[𝐴]
E[𝑐 𝑎𝑡 − 𝑐˜𝑎𝑡 | 𝑥𝑡 , F𝑡−1 ]E[I{𝑎 ∈ A𝑡 } | 𝑥𝑡 , F𝑡−1 ]
∗ (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 )E[I{𝑎 ∈ A𝑡 } | 𝑥𝑡 , F𝑡−1 ] "𝑎∈[𝐴] # Õ ∗ (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ) 𝑥𝑡 , F𝑡−1 . ≤E
≤ (𝑖𝑖)
(33)
𝑎∈A 𝑡
𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − e 𝜎𝑎𝑡 ) + and I{𝑎 ∈ A𝑡 } ⊥ ⊥ where (i) of (33) follows since 𝑐 𝑎𝑡 = 𝑐 𝑎 (𝑥𝑡 , 𝜔 𝑎𝑡 ), 𝑐˜𝑎𝑡 = (e 𝜔 𝑎𝑡 |𝑥𝑡 , F𝑡−1 by Lemma 5, (ii) follows from (32). The result then follows from (30), (31), (33).
Appendix B: Regret under Known Reward Function In this section, we provide regret analysis when the reward parameter 𝜃 ∗ is known. We begin with the technical lemmas used in the analysis. B.1. Technical Lemmas For any 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], let
then 𝑚 𝑎𝑡 (𝜌 𝑎⊤𝜓(𝑥𝑡 )) = 0.
𝑚 𝑎𝑡 (𝑢) := E (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝑢)) + − 𝑐 𝑎𝑡 | F𝑡−1 , 𝑥𝑡 ,
(34)
L EMMA 6. Assume that, for every 𝑡, for some constant 𝜂 > 0, P 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑢) | F𝑡−1 , 𝑥𝑡 Λ′ (𝑢) ≥ 𝜂, ∀|𝑢| ≤ 𝜄,
(35)
−(𝑢 − 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ))𝑚 𝑎𝑡 (𝑢) ≥ 𝜂(𝑢 − 𝜌 𝑎⊤𝜓(𝑥𝑡 )) 2
(36)
Then, for every 𝑢 ∈ [−𝜄, 𝜄],
a.s.
Lemma 6 says that the reservation-value moment crosses zero with a slope bounded away from zero. If the candidate threshold 𝑢 is above the true threshold 𝑣 𝑎𝑠 , then the expected excess value is too small relative to cost, so the moment 𝑐 − (𝜇∗ − Λ(𝑢)) + is positive. If 𝑢 is below the true threshold, then the expected excess value is too large relative to cost, so the moment is negative. This monotonicity is what ultimately makes the loss locally curved around 𝜌 𝑎 .
32
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Proof of Lemma 6 Fix 𝑡 and 𝑢 ∈ 𝐼. For any real number 𝑎 and any continuously differentiable
𝑑 function 𝑔, the map 𝑣 ↦→ (𝑎 − 𝑔(𝑣)) + is absolutely continuous and has a.e. derivative 𝑑𝑣 (𝑎 − 𝑔(𝑣)) + =
−I{𝑎 > 𝑔(𝑣)}𝑔′ (𝑣). Applying the fundamental theorem of calculus with 𝑔 = Λ, we obtain for ∫𝑢 any 𝑢, 𝑣 ∈ R, (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝑢)) + − (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝑣)) + = − 𝑣 I{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑟)}Λ′ (𝑟) 𝑑𝑟.
Taking conditional expectations given F𝑡−1 yields ∫ 𝑢 P 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑟) | F𝑡−1 , 𝑥𝑡 Λ′ (𝑟) 𝑑𝑟. 𝑚 𝑎𝑡 (𝑢) − 𝑚 𝑎𝑡 (𝑣) = − 𝑣
Since 𝑚 𝑎𝑡 (𝜌 𝑎⊤ 𝜓(𝑥𝑡 )) = 0, setting 𝑣 = 𝜌 𝑎⊤𝜓(𝑥𝑡 ) gives 𝑚 𝑎𝑡 (𝑢) = −
∫ 𝑢
𝜌⊤ 𝑎 𝜓(𝑥𝑡 )
P 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑟) | F𝑡−1 , 𝑥𝑡 Λ′ (𝑟) 𝑑𝑟.
∫𝑢 If 𝑢 ≥ 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ), then (35) implies 𝑚 𝑎𝑡 (𝑢) ≤ − 𝜌⊤ 𝜓(𝑥 ) 𝜂 𝑑𝑟 = −𝜂 𝑢 − 𝜌 𝑎⊤𝜓(𝑥𝑡 ) , which proves (36). 𝑎
𝑡
If 𝑢 ≤ 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ), then ∫ 𝜌⊤𝑎 𝜓(𝑥𝑡 ) ∫ 𝜌⊤𝑎 𝜓(𝑥𝑡 ) ′ ∗ P 𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑟) | F𝑡−1 , 𝑥𝑡 Λ (𝑟) 𝑑𝑟 ≥ (𝑖) 𝜂 𝑑𝑟 = 𝜂 𝜌 𝑎⊤𝜓(𝑥𝑡 ) − 𝑢 , 𝑚 𝑎𝑡 (𝑢) = 𝑢
𝑢
where (i) follows from (35) and (36) follows again.
We next prove Lemma 1 using Lemma 6: Lemma (Restatement of Lemma 1). Under Assumptions 1, 2, for any 𝑡 ∈ [𝑇], 𝑎 ∈ [𝐴] and 𝜌 ∈ B, 2 E[ℓ𝑎𝑡 (𝜌) − ℓ𝑎𝑡 (𝜌 𝑎 ) | 𝑥𝑡 , F𝑡−1 ] ≥ 12 𝜅𝜇1 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) . ∫ 𝑧 ∗ + Proof of Lemma 1 The scalar map 𝑧 ↦→ 𝑐 𝑎𝑡 − (𝜇 (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝑢)) 𝑑𝑢 is differentiable, 0
with derivative 𝑐 𝑎𝑡 − (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝑧)) + . Therefore ∇ℓ𝑎𝑡 (𝜌) = 𝜓(𝑥𝑡 ) 𝑐 𝑎𝑡 − (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝜌 ⊤ 𝜓(𝑥𝑡 ))) + .
(37)
The derivative of 𝑧 ↦→ 𝑐 𝑎𝑡 − (𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) − Λ(𝑢)) + is I{𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑧)}Λ′ (𝑧) ≥ 0 a.e., so the
scalar map is nondecreasing and thus ℓ𝑎𝑡 (𝜌) is convex in 𝜌 ⊤ 𝜓(𝑥𝑡 ), hence convex in 𝜌. Taking
conditional expectations in (37) gives ∇E[ℓ𝑎𝑡 (𝜌) | 𝑥𝑡 , F𝑡−1 ] = −𝜓(𝑥𝑡 ) 𝑚 𝑎𝑡 (𝜓(𝑥𝑡 ) ⊤ 𝜌). By convexity
of B, the entire segment 𝜓(𝑥𝑡 ) ⊤ 𝜌 𝑎 + 𝑠𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) = 𝜓(𝑥𝑡 ) ⊤ (𝜌 𝑎 + 𝑠(𝜌 − 𝜌 𝑎 )), ∀𝑠 ∈ [0, 1] lies in
𝐼 = [−¯𝜄, 𝜄¯] by Assumption 2. Using the fundamental theorem of calculus along the line segment from 𝜌 𝑎 to 𝜌, E[ℓ𝑎𝑡 (𝜌) − ℓ𝑎𝑡 (𝜌 𝑎 ) | F𝑡−1 , 𝑥𝑡 ] =
∫ 1 0∫
=−
(𝜌 − 𝜌 𝑎 ) ⊤ ∇E[ℓ𝑎𝑡 (𝜌 𝑎 + 𝑠(𝜌 − 𝜌 𝑎 )) | F𝑡−1 , 𝑥𝑡 ] 𝑑𝑠
0
1
𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) 𝑚 𝑎𝑡 𝜓(𝑥𝑡 ) ⊤ 𝜌 𝑎 + 𝑠𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) 𝑑𝑠.
33
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Applying Lemma 6 in the Appendix with 𝑢 = 𝜓(𝑥𝑡 ) ⊤ 𝜌 𝑎 + 𝑠𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) yields
2 − 𝑠𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) 𝑚 𝑎𝑡 𝜓(𝑥𝑡 ) ⊤ 𝜌 𝑎 + 𝑠𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) ≥ 𝜅𝜇1 𝑠2 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) ,
where the last inequality follows because P 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) > Λ(𝑢) | F𝑡−1 , 𝑥𝑡 Λ′ (𝑢) ≥ 𝜅𝜇1 according to Assumptions 1, 2. For 𝑠 > 0, divide by 𝑠 to obtain
2 − 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) 𝑚 𝑎𝑡 𝜓(𝑥𝑡 ) ⊤ 𝜌 𝑎 + 𝑠𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) ≥ 𝜅𝜇1 𝑠 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) .
Integrating over 𝑠 ∈ [0, 1] yields
E[ℓ𝑎𝑡 (𝜌) − ℓ𝑎𝑡 (𝜌 𝑎 ) | F𝑡−1 , 𝑥𝑡 ] ≥
∫ 1 0
𝜂𝑠 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 )
as claimed.
2
𝑑𝑠 =
2 𝜅𝜇1 𝜓(𝑥𝑡 ) ⊤ (𝜌 − 𝜌 𝑎 ) 2
L EMMA 7. Fix any 𝑎 ∈ [𝐴] and 𝑡 ∈ [𝑇], for any 𝜌𝑡 adapted to F𝑡−1 , we have
𝜅𝜇1 1 Õ 1 1 Õ E [ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] ≥ [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )] 2 − . 𝑛𝑎𝑡 2 𝑛𝑎𝑡 𝑛𝑎𝑡 𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
Proof of Lemma 7 Set 𝜖=
1 ¯ 2𝑛𝑎𝑡 𝐶𝜓 (𝜅𝜇1 𝜄¯ + 1)
(38)
and take an Euclidean 𝜖-net N𝜖 of B. According to Assumptions 1 and 2, for any 𝑠 ∈ [𝑇] we have P 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) > Λ(𝑢) | 𝑥 𝑠 , F𝑠−1 Λ′ (𝑢) ≥ 𝜅𝜇1 > 0, ∀|𝑢| ≤ 𝜄. Lemma 1 implies that given any 𝑠 ∈ 𝜅𝜇1 [𝑇] and 𝜌¯ ∈ N𝜖 , it always holds that E [ℓ𝑎𝑠 ( 𝜌) ¯ − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] ≥ [𝜓(𝑥 𝑠 ) ⊤ ( 𝜌¯ − 𝜌 𝑎 )] 2 . Since 2 B has diameter 𝑑 B by Assumption 1, N𝜖 is finite. So after intersecting finitely many probability-one events, we have with probability one, simultaneously for all 𝜌 ∈ N𝜖 and 𝑠 ∈ [𝑇], E [ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] ≥
𝜅𝜇1 [𝜓(𝑥 𝑠 ) ⊤ (𝜌 − 𝜌 𝑎 )] 2 . 2
(39)
Choose 𝜌𝜖 ∈ N𝜖 such that k 𝜌𝑡 − 𝜌𝜖 k 2 ≤ 𝜖. Note that | [ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 )] − [ℓ𝑎𝑠 (𝜌𝜖 ) − ℓ𝑎𝑠 (𝜌 𝑎 )]| = |ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌𝜖 )| ∫ 𝜌𝑡⊤ 𝜓(𝑥𝑠 ) [𝑐 𝑎𝑠 − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + ]𝑑𝑢 = (𝑖) 𝜌⊤𝜖 𝜓(𝑥 𝑠 )
≤ (𝑖𝑖) 2|(𝜌𝑡 − 𝜌𝜖 ) ⊤ 𝜓(𝑥 𝑠 )| ≤ (𝑖𝑖𝑖) 2𝐶¯𝜓 𝜖,
(40)
34
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
where (i) follows by definition of ℓ𝑎𝑠 (𝜌), (ii) follows since by definition, 𝑐 𝑎𝑠 ∈ [0, 1],
𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) ∈ [0, 1], Λ(𝑢) ∈ [−1, 1], so |𝑐 𝑎𝑠 − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + | ≤ 2. (iii) follows by Cauchy-Schwarz inequality, the fact that k 𝜌 − 𝜌𝜖 k 2 ≤ 𝜖 and k𝜓(𝑥𝑡 )k 2 ≤ 𝐶¯𝜓 by Assumption 1. Hence
1 Õ E [ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 1 Õ E [ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌𝜖 ) + ℓ𝑎𝑠 (𝜌𝜖 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ] = 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜅𝜇1 1 Õ ≥ (1) [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝜖 − 𝜌 𝑎 )] 2 − 2𝐶¯𝜓 𝜖 2 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜅𝜇1 1 Õ [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝜖 − 𝜌𝑡 ) + 𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )] 2 − 2𝐶¯𝜓 𝜖 = 2 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜅𝜇1 1 Õ ≥ (2) [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )] 2 − 2𝜖 𝐶¯𝜓 |𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )| − 2𝐶¯𝜓 𝜖 2 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜅𝜇1 1 Õ ≥ (3) [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )] 2 − 4𝜖 𝐶¯𝜓 𝜄¯ − 2𝐶¯𝜓 𝜖 2 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜅𝜇1 1 Õ = [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )] 2 − 2(𝜅𝜇1 𝜄¯ + 1)𝐶¯𝜓 𝜖 2 𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜅𝜇1 1 Õ 1 [𝜓(𝑥 𝑠 ) ⊤ (𝜌𝑡 − 𝜌 𝑎 )] 2 − = (4) , 2 𝑛𝑎𝑡 𝑛𝑎𝑡 𝑠∈S𝑎𝑡
where (1) follows from (39) and (40), (2) follows from applying Cauchy-Schwarz inequality to 𝜓(𝑥 𝑠 ) ⊤ (𝜌𝜖 − 𝜌𝑡 ) and the fact that k 𝜌𝜖 − 𝜌𝑡 k 2 ≤ 𝜖, k𝜓(𝑥 𝑠 )k 2 ≤ 𝐶¯𝜓 , (3) follows since 𝜓(𝑥 𝑠 ) ⊤ 𝜌𝑡 ∈ [−¯𝜄, 𝜄¯], 𝜓(𝑥 𝑠 ) ⊤ 𝜌 𝑎 ∈ [−¯𝜄, 𝜄¯], (4) follows from (38).
L EMMA 8 (Freedman’s Inequality (Freedman 1975)). Consider a real-valued martingale 𝑌𝑘 = Í𝑘 𝑗=1 𝑋 𝑗 with 𝑌0 = 0 and difference sequence {𝑋 𝑘 : 𝑘 = 1, 2, 3, . . .}. Assume that 𝑋 𝑘 ≤ 𝑅 almost Í surely for 𝑘 ≥ 1, where 𝑅 is a constant. Let 𝑊 𝑘 := 𝑘𝑗=1 E[𝑋 2𝑗 | F 𝑗−1 ] for 𝑘 ≥ 1. Then for all 𝑡 ≥ 1 𝑡 2 /2 2 2 and 𝜎 > 0, P ∃𝑘 ≥ 0 : 𝑌𝑘 ≥ 𝑡, and 𝑊 𝑘 ≤ 𝜎 ≤ exp − 2 . 𝜎 + 𝑅𝑡/3 Lemma (Restatement of Lemma 2). Suppose Assumptions 1, 2 hold. Then given any constant 𝑐 0 > 0 and any 𝑎 ∈ [𝐴], with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇], for any 𝜌𝑡 adapted to F𝑡−1 , we have Õ − {ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | F𝑠−1 , 𝑥 𝑠 ]} 𝑠∈S𝑎𝑡 𝑐0 144 𝑐0 Õ ⊤ 2 ¯ {(𝜌𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + 4 + , ≤ 8 𝑐0 8 𝑠∈S𝑎𝑡
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
m l 2 𝐶¯ 2 } + 1 + log (6𝑇/𝛿). where Γ𝑎𝑡 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶¯𝜓 𝑇) + log log2 1 + 𝑛𝑎𝑡 min{9𝜄2 , 𝑑 B 𝜓
35
Proof of Lemma 2 For any 𝑠 ∈ S𝑎𝑡 , define G𝑠−1 := 𝜎(F𝑠−1 , 𝑥 𝑠 ). Let 𝐻𝑎𝑡 (𝜌) :=
Õ
𝑠∈S𝑎𝑡
{ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ]} .
Note that −2 ≤ 𝑐 𝑎𝑡 − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )) − Λ(𝑢)) + ≤ 1, so for any 𝜌 ∈ B, |ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ]| ∫ 𝜌⊤ 𝜓(𝑥𝑠 ) [𝑐 𝑎𝑠 − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 ) − Λ(𝑢)) + ]𝑑𝑢 = ⊤ 𝜌 𝑎 𝜓(𝑥 𝑠 ) ∫ 𝜌⊤ 𝜓(𝑥𝑠 ) ⊤ + [𝑐 𝑎𝑠 − (𝐺 (𝜃 ∗ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 ) − Λ(𝑢)) ]𝑑𝑢 | G𝑠−1 −E
𝜌⊤ 𝑎 𝜓(𝑥 𝑠 ) ≤ 3|(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )| ≤ min{6𝜄, 3𝐶¯𝜓 𝑑 B }.
Let 𝑊𝑡 (𝜌) :=
Õ
𝑠∈S𝑎𝑡
E {ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ]}2 | G𝑠−1 . So for any 𝜌 ∈ B, 𝑊𝑡 (𝜌) ≤ 9
Õ
𝑠∈S𝑎𝑡
{(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 ,
(41)
where (𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 ) is G𝑠−1 -measurable.
Note that for any 𝜌 ∈ B, 𝜌 ⊤ 𝜓(𝑥 𝑠 ) ∈ [−2𝜄, 𝜄] and |(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥)| ≤ k 𝜌 − 𝜌 𝑎 k 2 k𝜓(𝑥)k 2 ≤ 𝑑 B 𝐶¯𝜓 , Õ 2 ¯2 𝐶𝜓 }. Let thus {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 ≤ 𝑛𝑎𝑡 min{9𝜄2 , 𝑑 B 𝑠∈S𝑎𝑡
𝑄 𝑎𝑡 :=
l
2
2 ¯2 log2 1 + 𝑛𝑎𝑡 min{9𝜄 , 𝑑 B 𝐶𝜓 }
m
.
Consider the events {E 𝑞 } for 𝑞 = {0} ∪ [𝑄 𝑎𝑡 ], where
(42)
𝑞−1 Õ 2 < {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 ≤ 2𝑞 if 𝑞 ≥ 1 𝑠∈S𝑎𝑡 E 𝑞 := (43) Õ ⊤ 2 0≤ {(𝜌 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} ≤ 1 if 𝑞 = 0 𝑠∈S𝑎𝑡 Õ Fix any 𝑥 > 0. On the event E 𝑞 , 𝑊𝑡 (𝜌) ≤ 9 {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 ≤ 9 × 2𝑞 . So applying Freedman’s 𝑠∈S𝑎𝑡
inequality (Lemma 8) with 𝜎 2 = 9 × 2𝑞 , we have
Õ √ √ © ª P −𝐻𝑎𝑡 (𝜌) ≥ 3 2 2𝑞 𝑥 + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B }𝑥, {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 ≤ 2𝑞 ® ≤ 𝑒−𝑥 . 𝑠∈S𝑎𝑡 « ¬
36
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
v t p Õ √ √ ⊤ 2 Additionally, on event E 𝑞 , 6 𝑥 1 + {(𝜌 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} ≥ 6 2𝑞−1 𝑥 = 3 2 2𝑞 𝑥. Thus on 𝑠∈S𝑎𝑡
event E 𝑞 ,
v t Õ −𝐻𝑎𝑡 (𝜌) ≥ 6 𝑥 1 + {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B }𝑥 𝑠∈S
𝑎𝑡 √ √ ⇒ −𝐻𝑎𝑡 (𝜌) ≥ 3 2 2𝑞 𝑥 + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B }𝑥.
Summing the probability bound above over 𝑞 ∈ {0} ∪ [𝑄 𝑎𝑡 ], we have
s Õ © ª P −𝐻𝑎𝑡 (𝜌) ≥ 6 𝑥 + 𝑥 {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B }𝑥 ® ≤ 2(𝑄 𝑎𝑡 + 1)𝑒−𝑥 . 𝑠∈S𝑎𝑡 « ¬
Setting 𝑥 = log (2(𝑄 𝑎𝑡 + 1)/𝛿) above, then for any fixed 𝜌 ∈ B, with probability at least 1 − 𝛿 we have
v t
2(𝑄 𝑎𝑡 + 1) −𝐻𝑎𝑡 (𝜌) < 6 log 𝛿
Õ
2(𝑄 + 1) 𝑎𝑡 . {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + (2 min{2𝜄, 𝐶¯𝜓 𝑑 B } +6) log 𝛿 𝑠∈S𝑎𝑡 (44)
Set 𝜖=
1 𝑇 𝐶¯𝜓
(45)
and take an Euclidean 𝜖-net N𝜖 of B. Since B has diameter 𝑑 B by Assumption 1,
2𝑑 B |N𝜖 | ≤ 1 + 𝜖
𝑚
= (1 + 2𝑑 B 𝐶¯𝜓 𝑇) 𝑚 .
Note that (44) further implies that with probability at least 1 − 𝛿/3, uniformly over all 𝜌 ∈ N𝜖 and 𝑡 ∈ [𝑇],
v t
6𝑇 |N𝜖 |(𝑄 𝑎𝑡 + 1) −𝐻𝑎𝑡 (𝜌) < 6 log 𝛿
Õ
{(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2
6𝑇 |N𝜖 |(𝑄 𝑎𝑡 + 1) ¯ +(2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6) log 𝛿 s Õ ≤ 6 Γ𝑎𝑡 (𝛿) {(𝜌 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + (2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6)Γ𝑎𝑡 (𝛿), 𝑠∈S 𝑎𝑡
(46)
𝑠∈S𝑎𝑡
where Γ𝑎𝑡 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶¯𝜓 𝑇) + log(𝑄 𝑎𝑡 + 1) + log (6𝑇/𝛿) .
(47)
37
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Let 𝜌𝜖 ∈ N𝜖 satisfy k 𝜌𝑡 − 𝜌𝜖 k 2 ≤ 𝜖. Note that | [ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 )] − [ℓ𝑎𝑠 (𝜌𝜖 ) − ℓ𝑎𝑠 (𝜌 𝑎 )]| ∫ 𝜌𝑡⊤ 𝜓(𝑥𝑠 ) [𝑐 𝑎𝑠 − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + ]𝑑𝑢 = |ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌𝜖 )| = (𝑖) 𝜌⊤𝜖 𝜓(𝑥 𝑠 )
2 ≤ (𝑖𝑖) 2|(𝜌𝑡 − 𝜌𝜖 ) ⊤ 𝜓(𝑥 𝑠 )| ≤ (𝑖𝑖𝑖) 2𝐶¯𝜓 𝜖 ≤ (𝑖𝑣) . 𝑛𝑎𝑡
where (i) follows by definition of ℓ𝑎𝑠 (𝜌) in (8), (ii) follows since by definition, |𝑐 𝑎𝑠 − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + | ≤ 2, (iii) follows by Cauchy-Schwarz inequality, the fact that k 𝜌𝑡 − 𝜌𝜖 k 2 ≤ 𝜖 and k𝜓(𝑥𝑡 )k 2 ≤ 𝐶¯𝜓 by Assumption 1, (iv) follows by (45). The above inequality implies that |𝐻𝑎𝑡 (𝜌𝑡 ) − 𝐻𝑎𝑡 (𝜌𝜖 )| =
Õ
𝑠∈S𝑎𝑡
[ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌𝜖 )] − E[ℓ𝑎𝑠 (𝜌𝑡 ) − ℓ𝑎𝑠 (𝜌𝜖 ) | G𝑠−1 ] ≤
Õ 4 ≤ 4. 𝑛𝑎𝑡
𝑠∈S𝑎𝑡
Therefore,
Note that qÍ
sÕ
𝑠∈S𝑎𝑡
|𝐻𝑎𝑡 (𝜌𝑡 ) − 𝐻𝑎𝑡 (𝜌𝜖 )| ≤ 4.
(48)
{(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 = k ((𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )) 𝑗 ∈[𝑛 𝑎𝑡 ] k 2 . Thus by triangular inequality,
⊤ 2 𝑠∈S𝑎𝑡 {(𝜌 𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} −
qÍ
𝑠∈S𝑎𝑡 {(𝜌 𝜖 − 𝜌 𝑎 )
((𝜌𝑡 − 𝜌𝜖 ) ⊤ 𝜓(𝑥 𝑠 )) 𝑗 ∈[𝑛 𝑎𝑡 ] 2 q (49) 1 ≤ 𝑛𝑎𝑡 (𝐶¯𝜓 𝜖) 2 = √ . 𝑛𝑎𝑡
⊤ 𝜓(𝑥 )}2 ≤ 𝑠
On the event that (46) holds uniformly over all 𝜌 ∈ N𝜖 and 𝑡 ∈ [𝑇], we have −𝐻𝑎𝑡 (𝜌𝑡 ) ≤ (𝑖) −𝐻𝑎𝑡 (𝜌𝜖 ) + 4 s Õ ≤ (𝑖𝑖) 6 Γ𝑎𝑡 (𝛿) {(𝜌𝜖 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + (2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6)Γ𝑎𝑡 (𝛿) + 4 𝑠∈S𝑎𝑡
s 1 ª © Õ ≤ (𝑖𝑖𝑖) 6 Γ𝑎𝑡 (𝛿) {(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + √ ® + (2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6)Γ𝑎𝑡 (𝛿) + 4 𝑛𝑎𝑡 ¬ «s 𝑠∈S𝑎𝑡 Õ p {(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + 1 + (2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6)Γ𝑎𝑡 (𝛿) + 4 ≤ (𝑖𝑣) 6 2Γ𝑎𝑡 (𝛿) p
𝑠∈S𝑎𝑡 𝑐0 144 𝑐0 Õ ⊤ 2 ¯ {(𝜌𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + 4 + ≤ (𝑣) 8 𝑐0 8 𝑠∈S𝑎𝑡
38
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
where (i) follows (46), (iii) holds from (49), (iv) holds from the fact that q from (48), (ii) follows from Õ √ √ 1 √ {(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 and 1/ 𝑛𝑎𝑡 ≤ 1, (v) follows by 𝑠+ √ ≤ 2(𝑠 + 1/ 𝑛𝑎𝑡 ), where 𝑠 = 𝑛𝑎𝑡 s Õ 𝑠∈S𝑎𝑡 p 𝑐0 applying 2𝑎𝑏 ≤ 𝜖 𝑎 2 + 𝜖 −1 𝑏 2 to 𝑎 = {(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + 1, 𝑏 = 6 2Γ𝑎𝑡 (𝛿), 𝜖 = , so that 4 𝑠∈S𝑎𝑡
sÕ p 𝑐0 Õ 𝑐0 144 6 2Γ𝑎𝑡 (𝛿) {(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + 1 ≤ Γ𝑎𝑡 (𝛿) + . {(𝜌𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥 𝑠 )}2 + 8 𝑐0 8 𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
Hence with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] we have 𝑐0 𝑐0 Õ 144 ⊤ 2 ¯ −𝐻𝑎𝑡 (𝜌𝑡 ) ≤ {(𝜌𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + 4 + , 8 𝑐0 8 𝑠∈S𝑎𝑡
where Γ𝑎𝑡 (𝛿) is defined as in (47), thus the result follows.
B.2. Index Estimation under Known Reward Function Before introducing the formal proof, we first provide the proof’s intuition. The argument follows a standard localized empirical-process approach for M-estimation, adapted to our martingale setting. The estimator’s empirical optimality is combined with a population curvature lower bound, while the stochastic deviation is controlled uniformly over the parameter space using Freedman’s martingale inequality, a peeling argument, and an 𝜖-net. Readers familiar with localized empirical-process and martingale concentration arguments may skip the following intuition and proceed directly to the formal proof. (i) The optimality inequality. Because 𝜌ˆ 𝑎𝑡 minimizes the empirical primitive loss, the empirical excess loss at 𝜌ˆ 𝑎𝑡 cannot be positive. Plug 𝜌 = 𝜌ˆ 𝑎𝑡 into Õ
𝑠∈S𝑎𝑡
ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) =
Õ E ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) F𝑠−1 , 𝑥 𝑠 + 𝐻𝑎𝑡 (𝜌).
𝑠∈S𝑎𝑡
and use the population curvature lower bound from Lemma 1 (with a small Lipschitz adjustment so that the curvature holds uniformly over 𝜌, which costs only an additive 1). After rearranging, one obtains
𝜅𝜇1 Õ 2 𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ≤ −𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 1. 2 𝑠∈S𝑎𝑡
The left side is the local quadratic “signal” we want to bound, and the right side is essentially the “noise”. (ii) Bounding the noise at one fixed 𝜌. For each fixed 𝜌, every term inside 𝐻𝑎𝑡 (𝜌) is uniformly bounded and has a conditional variance controlled by the same quadratic quantity
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Í
𝑠∈S𝑎𝑡 {𝜓(𝑥 𝑠 )
39
⊤ (𝜌 − 𝜌 )}2 . Freedman’s (Freedman 1975) martingale inequality (Lemma 8) therefore 𝑎
gives, for each fixed 𝜌, |𝐻𝑎𝑡 (𝜌)| ®
r
Í
⊤ (𝜌 − 𝜌 )}2 · log(𝑇/𝛿) + log(𝑇/𝛿). {𝜓(𝑥 ) 𝑠 𝑎 𝑠
The key point is that the stochastic error enters through the same quadratic form that governs the population curvature, allowing the error term to be absorbed into the curvature in the final bound. (iii) Making the bound uniform in 𝜌. Since 𝜌ˆ 𝑎𝑡 is itself random, we need step (ii) to hold for all 𝜌 simultaneously. We do this in two passes. First, a peeling argument splits the parameter space Í into dyadic shells based on the size of 𝑠 {𝜓(𝑥 𝑠 ) ⊤ (𝜌 − 𝜌 𝑎 )}2 ; on each shell the variance proxy is replaced by a deterministic ceiling and Freedman’s bound applies. Summing over the shells costs only a log log factor. Second, an 𝜖-net argument extends the bound from a finite grid of 𝜌’s to all of B via the Lipschitz continuity of the loss primitive. The end product is a uniform bound on |𝐻𝑎𝑡 (𝜌)| that still scales with the same quadratic quantity, (iv) Closing the loop. Plug 𝜌 = 𝜌ˆ 𝑎𝑡 into the uniform bound from step (iii) and combine with p p step (i). The resulting inequality has the schematic form quadratic ® quadratic · log 𝑇 + log 𝑇.
Applying 2𝑎𝑏 ≤ 𝜖 𝑎 2 + 𝜖 −1 𝑏 2 with a suitable 𝜖 absorbs the square-root term into the quadratic, Í 2 leaving 1/𝑛𝑎𝑡 𝑠∈S𝑎𝑡 𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ® log 𝑇/𝑛𝑎𝑡 . Folding in the regularization 𝜂1 I𝑚 to ensure invertibility and applying Cauchy–Schwarz then yields the stated confidence radius.
P ROPOSITION 6 (Restatement of Proposition 3). Suppose Assumptions 1, 2 hold. When 𝜇∗ is known, given any 𝛿 > 0, with probability at least 1 − 𝛿, uniformly over all 𝑡 ∈ [𝑇] and 𝑎 ∈ [𝐴], |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )| ≤ 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 , where 𝐵∗𝑎𝑡 :=
s
8 144 ∗ (𝛿) + 40/(3𝜅𝜇 ) + 1/3 + 𝜂 𝑑 2 , + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6 Γ𝑎𝑡 1 1 B 3𝜅𝜇1 𝜅𝜇1
m l 2 ¯2 ∗ 𝐶𝜓 } + 1 + log(6𝐴𝑇/𝛿), Γ𝑎𝑡 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶¯𝜓 𝑇) + log log2 1 + 𝑛𝑎𝑡 min{9𝜄2 , 𝑑 B
and 𝑉𝑎𝑡 (𝜂1 ) = 𝜂1 I𝑚 +
Õ
𝑠∈S𝑎𝑡
𝜓(𝑥 𝑠 )𝜓(𝑥 𝑠 ) ⊤ .
(50)
(51)
40
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Proof of Proposition 6 For any 𝑠 ∈ [𝑇], define G𝑠−1 := 𝜎(F𝑠−1 , 𝑥 𝑠 ). Note that 0 ≥ (1) = (2) ≥ (3)
Õ
ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) −
Õ
𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
where (1) holds because 𝜌ˆ 𝑎𝑡 = arg min 𝜌∈B
𝐻𝑎𝑡 (𝜌) =
Õ
𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
ℓ𝑎𝑠 (𝜌 𝑎 )
E [ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ] + 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 )
𝑠∈S𝑎𝑡 𝜅𝜇1 Õ
2
Õ
(52)
[𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )] 2 − 1 + 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ), Õ
𝑠∈S𝑎𝑡
ℓ𝑎𝑠 (𝜌), (2) holds with
{ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ]}
defined as (12), and (3) follows from Lemma 7. So (52) implies that 𝜅𝜇1 Õ [𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )] 2 ≤ −𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 1. 2
(53)
𝑠∈S𝑎𝑡
Lemma 2 implies that with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] and 𝑎 ∈ [𝐴],
𝜅𝜇1 Õ 144 𝜅𝜇1 ⊤ 2 ∗ −𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) ≤ {( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6 Γ𝑎𝑡 , (54) (𝛿) +4 + 8 𝜅𝜇1 8 𝑠∈S𝑎𝑡
∗ (𝛿) = 𝑚 log(1 + 2𝑑 𝐶¯ 𝑇) + log(𝑄 + 1) + log (6𝐴𝑇/𝛿). So (53) further implies that where Γ𝑎𝑡 B 𝜓 𝑎𝑡
144 𝜅𝜇1 3𝜅𝜇1 Õ ⊤ 2 ∗ + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6 Γ𝑎𝑡 {( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} ≤ . (𝛿) + 5 + 8 𝜅𝜇1 8
(55)
𝑠∈S𝑎𝑡
2 , and recall that Combining with the fact that 𝜂1 k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k 2 ≤ 𝜂1 𝑑 B
𝑉𝑎𝑡 (𝜂1 ) = 𝜂1 I𝑚 +
Õ
𝑠∈S𝑎𝑡
𝜓(𝑥 𝑠 )𝜓(𝑥 𝑠 ) ⊤ ,
(55) then implies that with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] and 𝑎 ∈ [𝐴], 8 k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k𝑉2 𝑎𝑡 (𝜂1 ) ≤ 3𝜅𝜇
1
144 ∗ 2 ¯ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + 40/(3𝜅𝜇1 ) + 1/3 + 𝜂1 𝑑 B . (56) 𝜅𝜇1
By Cauchy-Schwarz inequality, |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )| ≤ k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k𝑉𝑎𝑡 (𝜂1 ) , so the result follows.
41
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
B.3. Regret Analysis under Known Reward Function Proof of Proposition 4 Let E denote the event that uniformly over all 𝑎 ∈ [𝐴] and 𝑡 ∈ [𝑇], |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )| ≤ 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 ,
(57)
∗ for 𝜎𝑎𝑡 ≥ 𝜎𝑎𝑡 where 𝑉𝑎𝑡 (𝜂1 ) is defined as in Proposition 3 and 𝐵∗𝑎𝑡 is defined as (51). On event E, e
all 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], where e 𝜎𝑎𝑡 is defined as (15). Then Theorem 1 implies that on E, " 𝑇 # " 𝑇 𝐴 # Õ ÕÕ ∗ E Δ𝑡 ( 𝜋) ˜ E ≤E (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 )I{𝑎 ∈ A𝑡 } E 𝑡=1 𝑡=1 𝑎=1 " 𝐴 𝑇 # ÕÕ ⊤ ≤E Λ 𝜌ˆ 𝑎𝑡 𝜓(𝑥𝑡 ) + 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 − Λ(𝜌 𝑎⊤ 𝜓(𝑥𝑡 )) I{𝑎 ∈ A𝑡 } E 𝑎=1 " 𝐴𝑡=1 𝑇 # ÕÕ ⊤ ≤ (𝑖) E 𝐿 𝜌ˆ 𝑎𝑡 𝜓(𝑥𝑡 ) + 𝐵∗𝑎𝑡 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 − 𝜌 𝑎⊤𝜓(𝑥𝑡 ) I{𝑎 ∈ A𝑡 } E 𝑎=1 𝑡=1 " 𝐴 𝑇 # ÕÕ ≤ (𝑖𝑖) 2𝐿𝐵𝑇∗ E k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 I{𝑎 ∈ A𝑡 } E , 𝑎=1 𝑡=1
where (i) follows from the Lipschitz of Λ according to Assumption 1, (ii) follows from r property 8 144 ¯𝜓 𝑑 B } + 6 Γ𝑇 (𝛿) + 40 + 1 + 𝜂1 𝑑 2 , (57), and 𝐵𝑇∗ = sup𝑎∈[𝐴],𝑡≤𝑇 𝐵∗𝑎𝑡 = 3𝜅𝜇 𝐶 + 2 min{2𝜄, 𝜅𝜇1 3𝜅𝜇1 3 B 1 m l 2 2 ¯2 ¯ Γ𝑇 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶𝜓 𝑇) + log log2 1 + 𝑇 min{9𝜄 , 𝑑 B 𝐶𝜓 } + 1 + log(6𝐴𝑇/𝛿).
Then following similar proof steps as in the proof for Theorem 2, we have that on E, 𝐴 Õ 𝑇 Õ 𝑎=1 𝑡=1
Hence on E, E
k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 I{𝑎 ∈ A𝑡 } =
𝐴 Õ Õ
r
k𝜓(𝑥 𝑠 )k𝑉𝑎𝑠 (𝜂1 ) −1
≤ 𝐴 𝑇 (1 + 𝐶¯𝜓2 /𝜂1 )𝑚 log 1 + 𝑇 𝐶¯𝜓2 /(𝜂1 𝑚) .
" 𝑇 Õ 𝑡=1
Δ𝑡 ( 𝜋) ˜ E
#
≤ 2𝐿𝐵𝑇∗ 𝐴
𝑎=1 𝑠∈S𝑎,𝑇+1
r
2 2 ¯ ¯ 𝑇 (1 + 𝐶𝜓 /𝜂1 )𝑚 log 1 + 𝑇 𝐶𝜓 /(𝜂1 𝑚) .
(58)
∗ Note that it always holds that e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ≤ 2, and recall from Proposition 3 that P(E) ≥ 1 − 𝛿. On E 𝑐 ,
the per-period regret is at most 1 + 2𝐴. Taking 𝛿 = 1/𝑇 " 𝑇 gives a# failure event contribution of order √ Õ e O( 𝐴). Thus ignoring logarithmic factors we have E Δ𝑡 ( 𝜋) ˜ ≤ 𝑂 𝐴𝑚 𝑇 . 𝑡=1
42
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Appendix C: Reward Estimation Lemma 9 below is Theorem 1 from Abbasi-Yadkori et al. (2011): L EMMA 9. Let {𝑣 𝑠 : 𝑠 ≥ 0} be an R𝑑 -valued stochastic process adapted to filtration {H𝑠 : 𝑠 ≥ 0}, {𝜖 𝑠 : 𝑠 ≥ 1} be a real-valued stochastic process adapted to {H𝑠 }. Assume that 𝜖 𝑠 is conditionally sub-Gaussian such that there exists some 𝛾 > 0 such that for any 𝑢 ∈ R, 𝑠 ≥ 1, 2 2 𝑢 𝛾 a.s. E [exp (𝑢𝜖 𝑠 ) |H𝑠−1 ] ≤ exp 2 Assume that 𝑉 is a 𝑑 × 𝑑 postive definite matrix. For any 𝑡 ≥ 0, define 𝑉¯𝑡 = 𝑉 + any 𝛿 > 0, with probability at least 1 − 𝛿, for all 𝑡 ≥ 0,
det(𝑉¯𝑡 ) 1/2 det(𝑉) −1/2 𝜖𝑠 𝑣𝑠 ≤ 2𝛾 log . 𝛿 −1 ¯ 𝑠=1
𝑡 Õ
2
2
𝑉𝑡
Í𝑡
⊤ 𝑠=1 𝑣 𝑠 𝑣 𝑠 , then for
Lemma 10 in the below provides a stronger result which implies Lemma 3 directly: L EMMA 10. Suppose Assumptions 3, 4 hold. Fix any 𝛿 > 0. Then with probability at least 1 − 𝛿, uniformly over all 𝑥 ∈ X, 𝜔 ∈ Ω, and 𝑡 ≥ 2, the following holds: ⊤ 𝜃 ∗⊤ 𝜙(𝑥, 𝜔) − 𝜃ˆ𝑡−1 𝜙(𝑥, 𝜔) q 2 √ 2 k𝜙(𝑥, 𝜔)k Φ−1 𝛾0 𝑑 log(1 + 𝑡 𝐶¯𝜙 /𝜂0 ) + 2 log(1/𝛿) + 𝜂0 𝛼¯ . ≤ 𝑡 −1 min{1, 𝜇}
(59)
Particularly, with probability at least 1 − 𝛿/2, uniformly over all 𝑎 ∈ [𝐴] and 𝑡 ≥ 2, ( 𝜃ˆ𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) ≤ 𝛽𝑡 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) Φ−1 ,
(60)
q 2 √ 2 𝛽𝑡 := 𝛾0 𝑑 log(1 + 𝑡 𝐶¯𝜙 /𝜂0 ) + 2 log(2/𝛿) + 𝜂0 𝛼¯ . min{1, 𝜇}
(61)
𝑡 −1
where
The proof of Lemma 10 follows closely from Proposition 1 of Filippi et al. (2010). 𝑡−1 Õ Proof of Lemma 10 Let 𝑔𝑡 (𝜃) = 𝐺 (𝜃 ⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) + 𝜂0 𝜃 be the invertible function such that 𝑔𝑡 ( 𝜃˜𝑡−1 ) =
𝑡−1 Õ 𝑘=1
𝑘=1
𝑟 𝑘 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ), where 𝜃˜𝑡−1 is the unique solution to (62):
𝑡−1 i Õ 𝜂0 2 𝑟 𝑘 − 𝐺 𝜃 ⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) = 0, ∇𝜃 L𝑡 (𝜃) − k𝜃 k 2 = −𝜂0 𝜃 + 2 𝑘=1
h
(62)
43
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
By Assumption 3, ∇𝑔𝑡 is continuous, so by the Fundamental Theorem of Calculus,
where Γ𝑡 =
∫ 1 0
𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃˜𝑡−1 ) = Γ𝑡 (𝜃 ∗ − 𝜃˜𝑡−1 ),
(63)
∇𝑔𝑡 (𝑠𝜃 ∗ + (1 − 𝑠) 𝜃˜𝑡−1 )𝑑𝑠, and ∇𝑔𝑡 (𝜃) = 𝜂0 I𝑑 +
𝑡−1 Õ 𝑘=1
(𝑖) 𝜂0 I𝑑 + 𝜇
𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 )𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) ⊤ 𝐺 ′ 𝜃 ⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) 𝑡−1 Õ 𝑘=1
𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 )𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) ⊤ ,
where (i) follows because 𝐺 ′ 𝜃 ⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ) ≥ 𝜇 according to Assumption 3. Hence for any 𝑡 ∈ [𝑇], Γ𝑡 min{1, 𝜇}Φ𝑡−1 min{1, 𝜇}𝜂0 I𝑑 ≻ 0,
(64)
So Γ𝑡 is positive definite and is non-singular for any 𝑡 ∈ [𝑇]. Therefore, ⊤ −1 ⊤ 𝜙(𝑥, 𝜔) = ˜ 𝜃 ∗⊤ 𝜙(𝑥, 𝜔) − 𝜃˜𝑡−1 (1) |𝜙(𝑥, 𝜔) Γ𝑡 {𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃 𝑡−1 )}|
(65)
≤ (2) k𝜙(𝑥, 𝜔)k Γ𝑡−1 k𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃˜𝑡−1 )k Γ𝑡−1 ,
where (1) of (65) follows from (63), and (2) follows from Cauchy-Schwarz inequality and the fact that Γ𝑡−1 is positive definite. (64) implies that Γ𝑡 min{1, 𝜇}Φ𝑡−1 , which further implies n p o −1 −1 max{1, 1/𝜇}Φ𝑡−1 Γ𝑡 , so k𝑣k Γ𝑡−1 ≤ max 1/ 𝜇, 1 k𝑣k Φ−1 , ∀𝑣 ∈ R𝑑 . Hence, (65) further implies ⊤ 𝜃 ∗⊤ 𝜙(𝑥, 𝜔) − 𝜃˜𝑡−1 𝜙(𝑥, 𝜔) ≤ max
𝑡 −1
1 , 1 k𝜙(𝑥, 𝜔)k Φ−1 k𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃˜𝑡−1 )k Φ−1 . 𝑡 −1 𝑡 −1 𝜇
(66)
Further, k𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃ˆ𝑡−1 )k Φ−1 ≤ k𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃˜𝑡−1 )k Φ−1 + k𝑔𝑡 ( 𝜃˜𝑡−1 ) − 𝑔𝑡 ( 𝜃ˆ𝑡−1 )k Φ−1 𝑡 −1
𝑡 −1
≤ 2k𝑔𝑡 (𝜃 ∗ ) − 𝑔𝑡 ( 𝜃˜𝑡−1 )k Φ−1 ,
𝑡 −1
(67)
𝑡 −1
where the first inequality of (67) follows from triangular inequality, and the second inequality of (67) follows from the fact that 𝜃 ∗ ∈ Θ and the optimality of 𝜃ˆ𝑡−1 in Θ by definition. Recall that 𝑔𝑡 ( 𝜃˜𝑡−1 ) − 𝑔𝑡 (𝜃 ∗ ) =
𝑡−1 Õ 𝑘=1
𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ){𝑟 𝑘 − 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))} − 𝜂0 𝜃 ∗ ,
44
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
so (66) and (67) imply that ⊤ 𝜃 ∗⊤ 𝜙(𝑥, 𝜔) − 𝜃ˆ𝑡−1 𝜙(𝑥, 𝜔) 𝑡−1 Õ 2 ≤ max , 2 k𝜙(𝑥, 𝜔)k Φ−1 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ){𝑟 𝑘 − 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))} − 𝜂0 𝜃 ∗ 𝑡 −1 𝜇 𝑘=1 Φ𝑡−1 −1 Õ 𝑡−1 2 ⊤ ≤ (𝑖) max , 2 k𝜙(𝑥, 𝜔)k Φ−1 + 𝜂0 k𝜃 ∗ k Φ−1 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ){𝑟 𝑘 − 𝐺 (𝜃 ∗ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))} 𝑡 −1 𝑡 −1 𝜇 −1 𝑘=1 Φ𝑡 −1 Õ 𝑡−1 2 √ ⊤ ≤ (𝑖𝑖) max , 2 k𝜙(𝑥, 𝜔)k Φ−1 + 𝜂0 k𝜃 ∗ k 2 , 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ){𝑟 𝑘 − 𝐺 (𝜃 ∗ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))} 𝑡 −1 𝜇 −1 𝑘=1 Φ𝑡 −1
holds for all 𝑥, 𝜔, where inequality (i) above holds from triangular inequality, and inequality (ii) √ above holds since Φ𝑡−1 𝜂0 I𝑑 so that k𝜃 ∗ k Φ−1 ≤ k𝜃 ∗ k 2 / 𝜂0 . We now apply Lemma 9 to bound 𝑡 −1
𝑡−1 Õ 𝑘=1
𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ){𝑟 𝑘 − 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))}
Set 𝑣 𝑘 = 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ), 𝜖 𝑘 = 𝜁 𝑘 , H𝑘 = 𝜎(𝑣 𝑠 , 𝜖 𝑠 ; 𝑠 ≤ 𝑘), 𝜉𝑡 =
Í𝑡−1
. Φ𝑡−1 −1
𝑘=1 𝑣 𝑘 𝜖 𝑘 and 𝑉𝑡 =
that |𝜖 𝑘 | ≤ 𝛾0 , so 𝜖 𝑘 is 𝛾0 -sub-Gaussian, meaning that for any 𝑢 ∈ R, 𝑡 ≥ 1, E [exp (𝑢𝜁𝑡 ) |H𝑡−1 ] ≤ exp
𝑢 2 𝛾02 2
!
Í𝑡
⊤ 𝑘=1 𝑣 𝑘−1 𝑣 𝑘−1 . Note
a.s.
𝑑 Further, by Assumption 3, k𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 )k 2 ≤ 𝐶¯𝜙 , implying that det(𝑉¯𝑡 ) ≤ 𝜂0 + (𝑡 − 1)𝐶¯𝜙2 . So by
Lemma 9, given any 𝛿 > 0, with probability at least 1 − 𝛿, for all 𝑡 ≥ 2 we have 𝑡−1 Õ 𝑘=1
𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ){𝑟 𝑘 − 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑘 , 𝜔 𝑎 𝑘 𝑘 ))}
Φ𝑡−1 −1
q ≤ 𝛾0 𝑑 log(1 + 𝑡 𝐶¯𝜙2 /𝜂0 ) + 2 log(1/𝛿).
(68)
Note that 𝛾0 > 1 according to Assumption 4, hence with probability at least 1 − 𝛿, uniformly over all 𝑡 ≥ 2 and all 𝑥 ∈ X and 𝜔 ∈ Ω, ⊤ 𝜃 ∗⊤ 𝜙(𝑥, 𝜔) − 𝜃ˆ𝑡−1 𝜙(𝑥, 𝜔) q 2 √ 2 ¯ k𝜙(𝑥, 𝜔)k Φ−1 𝛾0 𝑑 log(1 + 𝑡 𝐶𝜙 /𝜂0 ) + 2 log(1/𝛿) + 𝜂0 k𝜃 ∗ k 2 . ≤ 𝑡 −1 min{1, 𝜇}
¯ This immediately implies the highHence (59) follows from (iv) of Assumption 3 that k𝜃 ∗ k 2 ≤ 𝛼. probability bound (60) taken uniformly over all 𝑎 ∈ [𝐴], 𝑡 ≥ 2.
45
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Appendix D: Minimum Eigenvalue of Φ𝑡−1 In this section, we first verify the examples satisfying Assumption 5 (Lemma 12, Lemma 13, Lemma 14). Then we show that Assumption 5 implies a lower bound on the minimum eigenvalue for Φ𝑡−1 (Proposition 7). We first present Lemma 11, which is useful for proving Lemmas 12 – 14.
0 ∈ R𝑑 be a possibly unbounded random vector. Suppose P |𝑣 ⊤ 𝑍 0 | ≤ 𝜖 F L EMMA 11. Let 𝑍𝑎𝑡 𝑡−1 ≤ 𝑎𝑡 0 𝐶𝑇 𝜖, ∀𝑣 ∈ S𝑑−1 , 𝜖 > 0, and suppose that P k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 F𝑡−1 ≥ 𝑞𝑇 a.s. for some deterministic 0 k𝑍 0 k ≤ 𝐶¯ , F 𝑞𝑇 ∈ (0, 1]. Define the bounded vector 𝑍𝑎𝑡 by the conditional law 𝑍𝑎𝑡 ∼ L 𝑍𝑎𝑡 𝜙 𝑡−1 . 𝑎𝑡 2 If 𝐶𝑇 ≤ 𝑀𝑇 , then 𝑍𝑎𝑡 satisfies Assumption 5. Moreover, k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 almost surely. 𝑞𝑇
Proof of Lemma 11. By construction, k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 a.s.. Fix 𝑣 ∈ S𝑑−1 and 𝜖 > 0. Then P (|𝑣 ⊤ 𝑍
𝑎𝑡 | ≤ 𝜖 | F𝑡−1 ) = P
0 |𝑣 ⊤ 𝑍𝑎𝑡 |≤𝜖
0 k𝑍𝑎𝑡 k ≤ 𝐶¯𝜙 , F𝑡−1
0 | ≤ 𝜖, k𝑍 0 k ≤ 𝐶¯ F P |𝑣 ⊤ 𝑍𝑎𝑡 𝜙 𝑡−1 𝑎𝑡 = 0 ¯ P k𝑍𝑎𝑡 k ≤ 𝐶𝜙 F𝑡−1
0 |≤𝜖 F P |𝑣 ⊤ 𝑍𝑎𝑡 𝐶𝑇 𝑡−1 ≤ ≤ 𝜖 ≤ 𝑀𝑇 𝜖 . 𝑞𝑇 𝑞𝑇
In particular, the above inequality holds for all 0 < 𝜖 ≤ (2𝐴𝑀𝑇 ) −1 , so Assumption 5 holds.
L EMMA 12 (Truncated Gaussian). Assumption 5 holds under the following conditions: (i) conditional on F𝑡−1 , 𝑍𝑎𝑡 ∼ N (𝜇𝑎𝑡 , Σ𝑎𝑡 ) and 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) ∼ L (𝑍𝑎𝑡 | k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 , F𝑡−1 ).
(ii) There exist deterministic sequences 𝜎𝑇 > 0, 𝜎 ¯ 𝑇 > 0 and 𝜇¯𝑇 ∈ [0, 𝐶¯𝜙 ) such that uniformly
¯ 𝑇2 , k 𝜇𝑎𝑡 k 2 ≤ 𝜇¯𝑇 , a.s. over 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], 𝜆 min (Σ𝑎𝑡 ) ≥ 𝜎12 , 𝜆 max (Σ𝑎𝑡 ) ≤ 𝜎 𝑇
(iii) Let 𝐹𝜒 2 (·) denote the cumulative distribution function of a chi-square random variable with 𝑑 ¯ p p (𝐶 𝜙 − 𝜇¯𝑇 ) 2 𝑑 degrees of freedom. Suppose 2/𝜋𝜎𝑇 ≤ 𝑀𝑇 𝐹𝜒 2 polylog(𝑇)/𝐴. where 𝑀 = 𝑇 2 𝜎 ¯ 𝑑
𝑇
Proof of Lemma 12 Fix 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇] and 𝑣 ∈ S𝑑−1 . Conditional on F𝑡−1 , 𝑣 ⊤ 𝑍𝑎𝑡 ∼ N (𝑣 ⊤ 𝜇𝑎𝑡 , 𝑣 ⊤ Σ𝑎𝑡 𝑣).
𝜎𝑇 1 ≤ √ . Let 𝑓𝑎𝑡𝑣 Since 𝑣 ∈ S𝑑−1 , 𝑣 ⊤ Σ𝑎𝑡 𝑣 ≥ 1/𝜎𝑇2 . The density of 𝑣 ⊤ 𝑍𝑎𝑡 is bounded by √ √ 2𝜋 𝑣 ⊤ Σ𝑎𝑡 𝑣 2𝜋 ⊤ be the density of 𝑣 𝑍𝑎𝑡 . So for every 𝜖 > 0, ∫ 𝜖 2𝜖 𝜎𝑇 p ⊤ 𝑓𝑎𝑡𝑣 (𝑢)𝑑𝑢 ≤ √ P(|𝑣 𝑍𝑎𝑡 | ≤ 𝜖 | F𝑡−1 ) = = 2/𝜋𝜎𝑇 𝜖 . −𝜖 2𝜋 1/2 Next, note that 𝑍𝑎𝑡 = 𝜇𝑎𝑡 + Σ𝑎𝑡 𝑍 where 𝑍 ∼ N (0, I𝑑 ). Note that
p 1/2 ¯ 𝑇 k𝑍 k 2 , k𝑍𝑎𝑡 k 2 ≤ (1) k 𝜇𝑎𝑡 k 2 + kΣ𝑎𝑡 𝑍 k 2 ≤ (2) k 𝜇𝑎𝑡 k 2 + 𝜆 max (Σ𝑎𝑡 )k𝑍 k 2 ≤ (3) 𝜇¯𝑇 + 𝜎
46
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
p 1/2 where (1) follows from triangular inequality, (2) follows because kΣ𝑎𝑡 𝑍 k 2 ≤ 𝜆 max (Σ𝑎𝑡 )k𝑍 k 2 , (3)
follows from condition (ii) in the lemma. Therefore, the event {k𝑍 k 2 ≤ k𝑍𝑎𝑡 k 2 ≤ 𝜇¯𝑇 + 𝜎 ¯𝑇 Hence
𝐶¯ 𝜙 − 𝜇¯𝑇 𝜎 ¯ 𝑇 } implies
𝐶¯𝜙 − 𝜇¯𝑇 = 𝐶¯𝜙 . 𝜎 ¯𝑇
! 𝐶¯𝜙 − 𝜇¯𝑇 (𝐶¯𝜙 − 𝜇¯𝑇 ) 2 P(k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 | F𝑡−1 ) ≥ P k𝑍 k 2 ≤ | F𝑡−1 = 𝐹𝜒 2 = 𝑞𝑇 . 𝑑 𝜎 ¯𝑇 𝜎 ¯ 𝑇2
¯ p (𝐶 𝜙 − 𝜇¯𝑇 ) 2 gives Applying Lemma 11 with 𝐶𝑇 = 2/𝜋𝜎𝑇 and 𝑞𝑇 = 𝐹𝜒 2 𝜎 ¯2 𝑑
P(|𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 | F𝑡−1 ) ≤
p
𝑇
2/𝜋𝜎𝑇 𝜖 ≤ 𝑀𝑇 𝜖 . 𝑞𝑇
So Assumption 5 is satisfied.
L EMMA 13 (Truncated Student’s t distribution). Assumption 5 holds under conditions below: (i) 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) ∼ L (𝑍𝑎𝑡 | k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 , F𝑡−1 ), where 𝑍𝑎𝑡 ∼ 𝑡 𝜉 (𝜇𝑎𝑡 , Σ𝑎𝑡 ) conditioning on F𝑡−1 , and 𝑡 𝜉 (𝜇𝑎𝑡 , Σ𝑎𝑡 ) is a 𝑑-dimensional Student’s 𝑡 distribution with 𝜉 > 0 degrees of freedom, location 𝜇𝑎𝑡 , scale matrix Σ𝑎𝑡 . (ii) There exists deterministic sequences 𝜎𝑇 > 0, 𝜎 ¯ 𝑇 > 0 and 𝜇¯𝑇 ∈ [0, 𝐶¯𝜙 ) such that uniformly
over all 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], 𝜆 min (Σ𝑎𝑡 ) ≥ 1/𝜎𝑇2 , 𝜆 max (Σ𝑎𝑡 ) ≤ 𝜎 ¯ 𝑇2 , k 𝜇𝑎𝑡 k 2 ≤ 𝜇¯𝑇 a.s.. p √ (iii) 2𝑐 𝜉 𝜎𝑇 /𝑞𝑇 ≤ 𝑀𝑇 = polylog(𝑇)/𝐴, where 𝑐 𝜉 := Γ((𝜉+1)/2) , 𝑞𝑇 = 𝐹𝑑,𝜉 ((𝐶¯𝜙 − 𝜇¯𝑇 ) 2 /(𝑑 𝜎 ¯ 𝑇2 )), 𝜉𝜋Γ(𝜉/2) and 𝐹𝑑,𝜉 (·) denotes the cumulative distribution of the 𝐹-distribution with 𝑑 and 𝜉 degrees of freedom.
Proof of Lemma 13 Fix 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], 𝑣 ∈ S𝑑−1 . The multivariate Student’s 𝑡 random vector Σ
1/2
𝑍
𝑍𝑎𝑡 can be written as 𝑍𝑎𝑡 = 𝜇𝑎𝑡 + √𝑎𝑡 , where 𝑍 ∼ N (0, I𝑑 ), 𝑆 ∼ 𝜒𝜉2 , where 𝜒𝜉2 is the chi-square 𝑆/𝜉
distribution with 𝜉 degrees of freedom, 𝑍 and 𝑆 are independent. So 1/2
𝑣⊤ Σ 𝑍 𝑣 𝑍𝑎𝑡 = 𝑣 𝜇𝑎𝑡 + p 𝑎𝑡 . 𝑆/𝜉 ⊤
⊤
1/2 Since 𝑣 ⊤ Σ𝑎𝑡 𝑍 ∼ N (0, 𝑣 ⊤ Σ𝑎𝑡 𝑣), it holds that 𝑣 ⊤ 𝑍𝑎𝑡 ∼ 𝑡 𝜉 (𝑣 ⊤ 𝜇𝑎𝑡 , 𝑣 ⊤ Σ𝑎𝑡 𝑣), where by the eigenvalue
lower bound, we have 𝑣 ⊤ Σ𝑎𝑡 𝑣 ≥ 1/𝜎𝑇2 . The density 𝑓𝑎𝑡𝑣 of the univariate Student’s 𝑡 random variable 𝑣 ⊤ 𝑍𝑎𝑡 is
(𝑦 − 𝑣 ⊤ 𝜇𝑎𝑡 ) 2 1+ 𝑓𝑎𝑡𝑣 (𝑦) = √ 𝜉 [𝑣 ⊤ Σ𝑎𝑡 𝑣] 𝑣 ⊤ Σ𝑎𝑡 𝑣 𝑐𝜉
−(𝜉+1)/2
.
47
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
√ Hence sup 𝑦 𝑓𝑎𝑡𝑣 (𝑦) ≤ 𝑐 𝜉 / 𝑣 ⊤ Σ𝑎𝑡 𝑣 ≤ 𝑐 𝜉 𝜎𝑇 . So for every 𝜖 > 0, ∫ 𝜖 ⊤ P |𝑣 𝑍𝑎𝑡 | ≤ 𝜖 | F𝑡−1 = 𝑓𝑎𝑡𝑣 (𝑢)𝑑𝑢 ≤ 2𝑐 𝜉 𝜎𝑇 𝜖 . −𝜖
Next, we lower bound the probability of the truncation event {k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 }. Note that p p 1/2 ¯ 𝑇 k𝑍 k 2 / 𝑆/𝜉. k𝑍𝑎𝑡 k 2 ≤ k 𝜇𝑎𝑡 k 2 + kΣ𝑎𝑡 𝑍 k 2 / 𝑆/𝜉 ≤ 𝜇¯𝑇 + 𝜎
p 𝐶¯ − 𝜇¯ Therefore, k𝑍 k 2 / 𝑆/𝜉 ≤ (𝐶¯𝜙 − 𝜇¯𝑇 )/𝜎 ¯ 𝑇 implies k𝑍𝑎𝑡 k 2 ≤ 𝜇¯𝑇 + 𝜎 ¯ 𝑇 𝜙𝜎¯ 𝑇 𝑇 = 𝐶¯𝜙 . So ! ¯ 𝐶 − 𝜇 ¯ k𝑍 k 𝜙 𝑇 2 | F𝑡−1 P(k𝑍𝑎𝑡 k 2 ≤ 𝐶¯𝜙 | F𝑡−1 ) ≥ P p ≤ 𝜎 ¯𝑇 𝑆/𝜉 ! ! k𝑍 k 22 /𝑑 (𝐶¯𝜙 − 𝜇¯𝑇 ) 2 (𝐶¯𝜙 − 𝜇¯𝑇 ) 2 | F𝑡−1 = 𝐹𝑑,𝜉 = 𝑞𝑇 . ≤ =P 𝑆/𝜉 𝑑𝜎 ¯ 𝑇2 𝑑𝜎 ¯ 𝑇2 Hence applying Lemma 11, we have P(|𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 | F𝑡−1 ) ≤ 𝑀𝑇 𝜖, so Assumption 5 holds. L EMMA 14 (Uniform distribution on a hyperrectangle). Assumption 5 holds under (i)–(ii): (i) Conditional on F𝑡−1 , 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) = 𝜙¯𝑎𝑡 + 𝑈𝑎𝑡 , where 𝜙¯𝑎𝑡 is a fixed constant vector, 𝑈𝑎𝑡 ∼
Unif [−𝑅𝑎𝑡, 𝑗 , 𝑅𝑎𝑡, 𝑗 ] for 𝑗 ∈ [𝑑] and 𝑈𝑎𝑡, 𝑗 are independent across 𝑗 ∈ [𝑑]. Í 1/2 p √ 𝑑 2 ¯ (ii) 𝑅𝑎𝑡, 𝑗 ≥ 𝑑/𝑀𝑇 where 𝑀𝑇 = polylog(𝑇)/𝐴 and k 𝜙 𝑎𝑡 k 2 + ≤ 𝐶¯𝜙 , a.s. 𝑗=1 𝑅𝑎𝑡, 𝑗
Proof of Lemma 14 Fix 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], 𝑣 ∈ S𝑑−1 . Since k𝑣k 2 = 1, there exists a coordinate 𝑗 ∗ ∈ √ Í [𝑑], s.t. |𝑣 𝑗 ∗ | ≥ 1/ 𝑑. Note that 𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) = 𝑣 ⊤ 𝜙¯𝑎𝑡 + 𝑣 ⊤𝑈𝑎𝑡 = 𝑣 ⊤ 𝜙¯𝑎𝑡 + 𝑑𝑗=1 𝑣 𝑗 𝑈𝑎𝑡, 𝑗 . Conditional
on all {𝑈𝑎𝑡, 𝑗 : 𝑗 ≠ 𝑗 ∗ }, 𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 ) = 𝐶 + 𝑣 𝑗 ∗ 𝑈𝑎𝑡, 𝑗 ∗ where 𝐶 is a fixed constant. Conditional on
F𝑡−1 , 𝑈𝑎𝑡, 𝑗 are independent uniform random on [−𝑅𝑎𝑡, 𝑗 , 𝑅𝑎𝑡, 𝑗 ]. Since 𝑈𝑎𝑡, 𝑗 ∗ ∼ Unif [−𝑅𝑎𝑡, 𝑗 ∗ , 𝑅𝑎𝑡, 𝑗 ∗ ], 𝐶 + 𝑣 𝑗 ∗ 𝑈𝑎𝑡, 𝑗 ∗ is uniform on an interval of length 2𝑅𝑎𝑡, 𝑗 ∗ |𝑣 𝑗 ∗ |. Therefore, for any interval 𝐼 ⊂ R of length |𝐼 |, we have P(𝐶 + 𝑣 𝑗 ∗ 𝑈𝑎𝑡, 𝑗 ∗ ∈ 𝐼 | {𝑈𝑎𝑡, 𝑗 : 𝑗 ≠ 𝑗 ∗ }, F𝑡−1 ) ≤ Take 𝐼 = [−𝜖, 𝜖], then ⊤
∗
P |𝑣 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 | F𝑡−1 , {𝑈𝑎𝑡, 𝑗 : 𝑗 ≠ 𝑗 } ≤
|𝐼 |
2𝑅𝑎𝑡, 𝑗 ∗ |𝑣 𝑗 ∗ | 2𝜖
2𝑅𝑎𝑡, 𝑗 ∗ |𝑣 𝑗 ∗ |
≤
.
√
𝑑
𝑅𝑎𝑡, 𝑗 ∗
𝜖.
Taking expectation over the conditioned coordinates of 𝑈𝑎𝑡 in the above inequality gives
p with 𝑀𝑇 = polylog(𝑇)/𝐴.
P(|𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 | F𝑡−1 ) ≤ 𝑀𝑇 𝜖
48
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
L EMMA 15 (Theorem 3.1 of Tropp (2011)). Let F1 ⊂ F2 · · · be a filtration and consider a finite sequence {𝑋 𝑘 } of positive semi-definite matrices with dimension 𝑑 adapted to this filtration. Suppose Í Í that 𝜆 max (𝑋 𝑘 ) ≤ 𝑅 almost surely. Define the series 𝑌 ≡ 𝑘 𝑋 𝑘 and 𝑊 ≡ 𝑘 E[𝑋 𝑘 |H𝑘−1 ]. Then for −𝛾 𝜇/𝑅 𝑒 all 𝜇 ≥ 0, 𝛾 ∈ [0, 1), P (𝜆 min (𝑌 ) ≤ (1 − 𝛾)𝜇, 𝜆 min (𝑊) ≥ 𝜇) ≤ 𝑑 (1−𝛾) . 1−𝛾 P ROPOSITION 7 (Minimum eigenvalue). Suppose Assumption 5 holds. Then 𝜆 min E 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) ⊤ F𝑡−1 ≥
1/8 . polylog(𝑇)
(69)
Further, for any 𝛿 ∈ (0, 1), with probability at least 1 − 𝛿,
𝜆 min (Φ𝑡−1 ) ≥ (𝑡 − 1)/[16polylog(𝑇)]
(70)
holds uniformly for all 𝑡 ≥ 100𝐶¯𝜙2 polylog(𝑇) log(𝑇 𝑑/𝛿). Proof of Proposition 7 Fix 𝑣 ∈ S𝑑−1 . Since 𝑎 𝑡 ∈ [𝐴], for any 𝜖 > 0, {|𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )| ≤ 𝜖 } ⊆
𝐴 Ø 𝑎=1
{|𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 }.
Therefore, by applying union bound and Assumption 5,
⊤
P |𝑣 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )| ≤ 𝜖 F𝑡−1 ≤
𝐴 Õ 𝑎=1
P |𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| ≤ 𝜖 F𝑡−1 ≤ 𝐴𝑀𝑇 𝜖 .
Take 𝜖 = (2𝐴𝑀𝑇 ) −1 . Then the above inequality implies that
1 P |𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )| > (2𝐴𝑀𝑇 ) −1 F𝑡−1 ≥ . 2
Hence 𝑣 ⊤ E[𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) ⊤ | F𝑡−1 ]𝑣 = E[(𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )) 2 | F𝑡−1 ] ≥ (2𝐴𝑀𝑇 ) −2 P |𝑣 ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )| > (2𝐴𝑀𝑇 ) −1 F𝑡−1 1/8 1/8 = . ≥ ( 𝐴𝑀𝑇 ) 2 polylog(𝑇) Since this holds for any 𝑣 ∈ S𝑑−1 , (69) follows. Setting 𝛾 = 1/2, 𝑅 = 𝐶¯𝜙2 in Lemma 15, with probability at least 1 − exp{log(𝑑) − 0.01(𝑡 − 1)/[𝐶¯ 2 polylog(𝑇)]}, 𝜆 min (Φ𝑡−1 ) ≥ (𝑡 − 1)/[16polylog(𝑇)]. 𝜙
Particularly, for any 𝛿 ∈ (0, 1), with probability at least 1 − 𝛿, 𝜆 min (Φ𝑡−1 ) ≥ (𝑡 − 1)/[16polylog(𝑇)] holds uniformly for all 𝑡 ≥ 100𝐶¯ 2 polylog(𝑇) log(𝑇 𝑑/𝛿). 𝜙
49
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Appendix E: Index Estimation under Unknown Reward Function The same argument for Proposition 3 extends to Proposition 8 for the high probability bound of the index estimation error when the reward parameter 𝜃 ∗ is unknown and learned online. The main
difference is that the primitive loss is evaluated at the plug-in estimate 𝜃ˆ𝑡−1 rather than at the true parameter 𝜃 ∗ . This introduces an additional error term in the optimality inequality: population curvature ≤ empirical fluctuation + plug-in error from 𝜃ˆ𝑡−1 . Consequently, Proposition 8 yields the same type of confidence radius for 𝜌 𝑎⊤ 𝜓(𝑥𝑡 ), with an additional additive term that accounts for uncertainty in the reward parameter. Í Recall from Section 5.2 that 𝜌ˆ 𝑎𝑡 ∈ arg min 𝜌∈B 𝑠∈S ℓˆ𝑎𝑠,𝑡 (𝜌), where ℓˆ𝑎𝑠,𝑡 is the loss defined as 𝑎𝑡
(22) when plugging in 𝜃ˆ𝑡−1 for the unknown 𝜃 ∗ . Lemma 4 in the below characterizes how the estimation error of 𝜃 ∗ propagates into the estimation error of 𝜌 𝑎⊤𝜓(𝑥𝑡 ). Proof of Lemma 4 Recall from (24) that 𝐽𝑎𝑡 (𝜌) :=
Õ
𝑠∈S𝑎𝑡
[ℓˆ𝑎𝑠,𝑡 (𝜌) − ℓ𝑎𝑠 (𝜌)] − [ℓˆ𝑎𝑠,𝑡 (𝜌 𝑎 ) − ℓ𝑎𝑠 (𝜌 𝑎 )] .
Note that (8) and (22) imply that [ℓˆ𝑎𝑠,𝑡 (𝜌 𝑎 ) − ℓ𝑎𝑠 (𝜌 𝑎 )] − [ℓˆ𝑎𝑠,𝑡 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 )] ∫ 𝜌⊤𝑎 𝜓(𝑥𝑠 ) ⊤ = [(𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + − (𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + ]𝑑𝑢 0∫ ⊤ 𝜌ˆ 𝑎𝑡 𝜓(𝑥 𝑠 ) ⊤ [(𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + − (𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + ]𝑑𝑢 − 0
=
∫ 𝜌ˆ ⊤𝑎𝑡 𝜓(𝑥𝑠 ) 𝜌⊤ 𝑎 𝜓(𝑥 𝑠 )
⊤ [(𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + − (𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − Λ(𝑢)) + ]𝑑𝑢
⊤ ≤ (𝑖) 𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) · |𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )|,
where (i) uses the 1-Lipchitz property of 𝑧 ↦→ 𝑧 + . Thus sÕ sÕ ⊤ 𝜙(𝑥 , 𝜔 )) − 𝐺 (𝜃 ⊤ 𝜙(𝑥 , 𝜔 )) 2 ⊤ 2 −𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) ≤ (𝑖) {𝜓(𝑥 𝑠 ) ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )} 𝐺 ( 𝜃ˆ𝑡−1 𝑠 𝑎𝑠 𝑠 𝑎𝑠 ∗
2 2 Õ ˆ⊤ 𝑐0 Õ {𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )}2 + 𝐺 ( 𝜃 𝑡−1 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) − 𝐺 (𝜃 ∗⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )) ≤ (𝑖𝑖) 8 𝑐0 𝑠∈S𝑎𝑡 𝑠∈S𝑎𝑡 Õ Õ 𝑐0 2 ≤ (𝑖𝑖𝑖) {𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )}2 + 𝐿 2 {( 𝜃ˆ𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )}2 8 𝑐0 𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
50
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
where (i) above holds from Cauchy-Schwarz inequality, (ii) holds from applying 2𝑎𝑏 ≤ 𝜖 𝑎 2 + 𝜖 −1 𝑏 2 sÕ sÕ 𝑐0 ⊤ 𝜙(𝑥 , 𝜔 )) − 𝐺 (𝜃 ⊤ 𝜙(𝑥 , 𝜔 )) 2 , ⊤ 2 to 𝜖 = , 𝑎 = {𝜓(𝑥 𝑠 ) ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )} , 𝑏 = 𝐺 ( 𝜃ˆ𝑡−1 𝑠 𝑎𝑠 𝑠 𝑎𝑠 ∗ 4 𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
(iii) holds from the Lipschitz property of 𝐺 according to Assumption 3. Hence the result follows.
P ROPOSITION 8 (Restatement of Proposition 5). Suppose Assumptions 1, 2, 3, 4, 5 hold. Let 𝑉𝑎𝑡 (𝜂1 ) be defined as (13). Given any 𝛿 > 0, with probability 1 − 2𝛿/3, uniformly over all 𝑎 ∈ [𝐴] and 𝑡 ∈ [𝑇], |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )| ≤ k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1
"s
4Γ𝑎𝑡 (𝛿) 144 20 1 2 ¯ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 + + + 𝜂1 𝑑 B 𝜅𝜇1 𝜅𝜇1 𝜅𝜇1 2 # √ s Õ 2 2𝐿 ¯ 𝐶𝑎𝑡 (𝛿) k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k 2 −1 , + Φ𝑡 −1 𝜅𝜇1 𝑠∈S𝑎𝑡
where
l m 2 2 ¯2 ¯ Γ𝑎𝑡 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶𝜓 𝑇) + log log2 1 + 𝑛𝑎𝑡 min{9𝜄 , 𝑑 B 𝐶𝜓 } + 1 + log(6𝐴𝑇/𝛿), 𝐶¯𝑎𝑡 (𝛿) :=
q 2 4 √ 2 ¯ 𝛾0 𝑑 log(1 + 𝑡 𝐶𝜙 /𝜂0 ) + 2 log(3/𝛿) + 𝜂0 𝛼¯ . min{1, 𝜇}2
Proof of Proposition 8 For any 𝑠 ∈ [𝑇], define G𝑠−1 := 𝜎(F𝑠−1 , 𝑥 𝑠 ). Note that Õ Õ ℓˆ𝑎𝑠,𝑡 ( 𝜌ˆ 𝑎𝑡 ) − 0 ≥ (1) ℓˆ𝑎𝑠,𝑡 (𝜌 𝑎 ) Õ
𝑠∈S𝑎𝑡
= (2) ≥ (3)
𝑠∈S𝑎𝑡
E [ℓ𝑎𝑠 ( 𝜌ˆ 𝑎𝑡 ) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ] + 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 )
𝑠∈S𝑎𝑡 𝜅𝜇1 Õ
2
𝑠∈S𝑎𝑡
[𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )] 2 − 1 + 𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ),
where (1) holds because 𝜌ˆ 𝑎𝑡 = arg min 𝐻𝑎𝑡 (𝜌) :=
Õ
𝑠∈S𝑎𝑡
𝐽𝑎𝑡 (𝜌) :=
𝜌∈B
1 Õ ˆ ℓ𝑎𝑠,𝑡 (𝜌), (2) holds with 𝑛𝑎𝑡 𝑠∈S𝑎𝑡
{ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) − E[ℓ𝑎𝑠 (𝜌) − ℓ𝑎𝑠 (𝜌 𝑎 ) | G𝑠−1 ]} ,
Õ
𝑠∈S𝑎𝑡
[ℓˆ𝑎𝑠,𝑡 (𝜌) − ℓ𝑎𝑠 (𝜌)] − [ℓˆ𝑎𝑠,𝑡 (𝜌 𝑎 ) − ℓ𝑎𝑠 (𝜌 𝑎 )] ,
and (3) follows from Lemma 7. So (71) implies that 𝜅𝜇1 Õ [𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )] 2 ≤ −(𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 )) + 1. 2 𝑠∈S𝑎𝑡
(71)
(72)
51
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Lemma 2 implies that with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] and 𝑎 ∈ [𝐴] we have
144 𝜅𝜇1 Õ 𝜅𝜇1 ⊤ 2 ¯ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) +4 + {( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + , (73) −𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) ≤ 8 𝜅𝜇1 8 𝑠∈S𝑎𝑡
6𝐴𝑇 ¯ Γ𝑎𝑡 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶𝜓 𝑇) + log(𝑄 𝑎𝑡 + 1) + log . 𝛿
where
Lemma 4 implies −𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) ≤
𝜅𝜇1 Õ 2 Õ 2 ˆ {𝜓(𝑥 𝑠 ) ⊤ ( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 )}2 + 𝐿 {( 𝜃 𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )}2 . 8 𝜅𝜇1 𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
Hence combining this inequality and (73), with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇], 𝑎 ∈ [𝐴] we have −(𝐻𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 ) + 𝐽𝑎𝑡 ( 𝜌ˆ 𝑎𝑡 )) 144 𝜅𝜇1 Õ 𝜅𝜇1 ⊤ 2 {( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} + + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + 4 + ≤ 4 𝜅𝜇1 8 𝑠∈S𝑎𝑡 2𝐿 2 Õ ˆ {( 𝜃 𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )}2 + 𝜅𝜇1
(74)
𝑠∈S𝑎𝑡
Thus (72) further implies that with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] we have 𝜅𝜇1 𝜅𝜇1 Õ 144 ⊤ 2 ¯ {( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) 𝜓(𝑥 𝑠 )} ≤ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + 5 + 4 𝜅𝜇1 8 𝑠∈S𝑎𝑡 (75) 2𝐿 2 Õ ˆ ⊤ 2 {( 𝜃 𝑡−1 − 𝜃 ∗ ) 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )} , + 𝜅𝜇1 𝑠∈S𝑎𝑡
2 , and recall that combining with the fact that 𝜂1 k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k 2 ≤ 𝜂1 𝑑 B
𝑉𝑎𝑡 (𝜂1 ) = 𝜂1 I𝑚 +
Õ
𝑠∈S𝑎𝑡
𝜓(𝑥 𝑠 )𝜓(𝑥 𝑠 ) ⊤ ,
the above inequality then implies that with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] and 𝑎 ∈ [𝐴], we have
20 1 144 2 + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + + + 𝜂1 𝑑 B 𝜅𝜇1 2 1 𝜅𝜇1 8𝐿 2 Õ ˆ {( 𝜃 𝑡−1 − 𝜃 ∗ ) ⊤ 𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )}2 . + (𝜅𝜇1 ) 2 𝑠∈S
4 k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k𝑉2 𝑎𝑡 (𝜂1 ) ≤ 𝜅𝜇
𝑎𝑡
(76)
52
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Lemma 10 implies that with probability at least 1 − 𝛿/3, uniformly over all 𝑡 ∈ [𝑇] and 𝑎 ∈ [𝐴], Õ 8𝐿 2 Õ ˆ 8𝐿 2 ¯ ⊤ 2 {( 𝜃 − 𝜃 ) 𝜙(𝑥 , 𝜔 )} ≤ 𝐶 (𝛿) k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k 2Φ−1 , 𝑡−1 ∗ 𝑠 𝑎𝑠 𝑎𝑡 (𝜅𝜇1 ) 2 𝑠∈S (𝜅𝜇1 ) 2 𝑡 −1 𝑠∈S 𝑎𝑡
where
(77)
𝑎𝑡
q 2 4 √ 2 𝛾0 𝑑 log(1 + 𝑡 𝐶¯𝜙 /𝜂0 ) + 2 log(3/𝛿) + 𝜂0 𝛼¯ . 𝐶¯𝑎𝑡 (𝛿) := min{1, 𝜇}2
Thus with probability 1 − 2𝛿/3 we have s 4 144 20 1 2 ¯ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + + + 𝜂1 𝑑 B k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k𝑉𝑎𝑡 (𝜂1 ) ≤ 𝜅𝜇1 𝜅𝜇1 𝜅𝜇1 2 √ s Õ 2 2𝐿 ¯ + 𝐶𝑎𝑡 (𝛿) k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k 2 −1 . Φ𝑡 −1 𝜅𝜇1
(78)
𝑠∈S𝑎𝑡
By Cauchy-Schwarz inequality, |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) ⊤ 𝜓(𝑥𝑡 )| ≤ k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k𝑉𝑎𝑡 (𝜂1 ) , so combining (78), the result follows.
Appendix F: Regret under Unknown Reward Function Proof of Theorem 2 Let E denote the event that conditions (i)-(ii) hold in the following: (i) Uniformly over all 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], |(𝜃 ∗ − 𝜃ˆ𝑡−1 ) ⊤ 𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )| q 2 √ 2 ≤ k𝜙(𝑥𝑡 , 𝜔 𝑎𝑡 )k Φ−1 𝛾0 𝑑 log(1 + 𝑡 𝐶¯𝜙 /𝜂0 ) + 2 log(2/𝛿) + 𝜂0 𝛼¯ ; 𝑡 −1 min{𝜇, 1} (ii) Uniformly over all 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇], s 4 144 20 1 ⊤ 2 ¯ |( 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 ) 𝜓(𝑥𝑡 )| ≤ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + + + 𝜂1 𝑑 B 𝜅𝜇1 𝜅𝜇1 𝜅𝜇1 2 √ s Õ 2 2𝐿 2 𝐶𝑎𝑡 (𝛿) k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k −1 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 , + Φ𝑡 −1 𝜅𝜇1 where 𝑉𝑎𝑡 (𝜂1 ) = 𝜂1 I𝑚 +
Õ
𝑠∈S𝑎𝑡
𝑠∈S𝑎𝑡
l m 2 𝐶¯ 2 } , 𝜓(𝑥 𝑠 )𝜓(𝑥 𝑠 ) ⊤ , 𝑄 𝑎𝑡 := log2 1 + 𝑛𝑎𝑡 min{9𝜄2 , 𝑑 B 𝜓
Γ𝑎𝑡 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶¯𝜓 𝑇) + log(2𝑄 𝑎𝑡 + 2) + log (6𝐴𝑇/𝛿) ,
2 q 4 √ 2 ¯ 𝛾0 𝑑 log(1 + 𝑡 𝐶𝜙 /𝜂0 ) + 2 log(6/𝛿) + 𝜂0 𝛼¯ . 𝐶𝑎𝑡 (𝛿) := min{𝜇, 1}2
53
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Thus on E, e 𝜇𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ) ≥ 𝜇∗ (𝑥𝑡 , 𝜔 𝑎𝑡 ) holds uniformly over 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇] and e 𝜎𝑎𝑡 ≥ 𝜎𝑎𝑡 holds
uniformly over all 𝑎 ∈ [𝐴], 𝑡 ∈ [𝑇]. (21), (27), Lemma 10 and Proposition 8 imply that E holds with probability at least 1 − 𝛿. Theorem 1 implies " 𝑇 # " 𝑇 # # " 𝑇 Õ Õ ÕÕ E Δ𝑡 ( 𝜋) ˜ E ≤E (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ) E . (79) 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) E + E 𝑡=1
𝑡=1
𝑡=1 𝑎∈A 𝑡
On E, using the fact that
𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )
⊤ ⊤ ≤ 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )) + 𝐺 ( 𝜃ˆ𝑡−1 𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) ,
and the Lipschitz property of 𝐺 by Assumption 3, we have 𝑇 Õ
𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )
q Õ 𝑇 4𝐿 √ 2 𝛾0 𝑑 log(1 + 𝑡 𝐶¯𝜙 /𝜂0 ) + 2 log(2/𝛿) + 𝜂0 𝛼¯ ≤ k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k Φ−1 . 𝑡 −1 min{𝜇, 1} 𝑡=1 𝑡=1
Since det(Φ𝑡 ) = det(Φ𝑡−1 )(1 + k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k 2Φ−1 ) and Φ0 = 𝜂0 I𝑑 , we have 𝑡 −1
log
𝑇 det(Φ𝑇 ) Õ = log(1 + k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k 2Φ−1 ) det(Φ0 ) 𝑡=1 𝑡 −1 2 2 𝑇 k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k Φ−1 𝑇 k𝜙(𝑥 𝑡 , 𝜔 𝑎 𝑡 𝑡 )k −1 Õ Õ Φ𝑡 −1 𝑡 −1 , ≥ (𝑖) ≥ (𝑖𝑖) 2 ¯ 2 /𝜂0 1 + k𝜙(𝑥 , 𝜔 )k 1 + 𝐶 𝑡 𝑎 𝑡 −1 𝑡 𝜙 𝑡=1 𝑡=1 Φ 𝑡 −1
𝑎 where (i) uses the fact that log(1 + 𝑎) ≥ 1+𝑎 for 𝑎 ≥ 0 and (ii) uses k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k 2Φ−1 ≤ 𝐶¯𝜙2 /𝜂0 . Hence 𝑡 −1
the above inequality implies that 𝑇 Õ
det(Φ𝑇 ) k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k 2Φ−1 ≤ (1 + 𝐶¯𝜙2 /𝜂0 ) log ≤ (1) (1 + 𝐶¯𝜙2 /𝜂0 )𝑑 log det(Φ0 ) 𝑡 −1 𝑡=1
2 ¯ 1 + 𝑇 𝐶𝜙 /(𝜂0 𝑑) ,
𝑑 𝑑 tr(Φ𝑇 ) 2 ¯ ≤ 𝜂0 + 𝑇 𝐶𝜙 /(𝜂0 𝑑) and det(Φ0 ) = 𝜂0𝑑 so that where (1) follows because det(Φ𝑇 ) ≤ 𝑑 det(Φ𝑇 ) 2 ¯ log ≤ 𝑑 log 1 + 𝑇 𝐶𝜙 /(𝜂0 𝑑) . Thus by Cauchy-Schwarz inequality, det(Φ0 ) v u t 𝑇 r 𝑇 Õ Õ 2 k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k −1 ≤ 𝑇 (1 + 𝐶¯𝜙2 /𝜂0 )𝑑 log 1 + 𝑇 𝐶¯𝜙2 /(𝜂0 𝑑) , k𝜙(𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 )k Φ−1 ≤ 𝑇 𝑡=1
𝑡 −1
𝑡=1
Φ𝑡 −1
54
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
implying that on E, we have 𝑇 Õ 𝑡=1
where 𝐶𝜇 =
√ √ √ 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) − 𝜇(𝜃 ∗ ; 𝑥𝑡 , 𝜔 𝑎 𝑡 𝑡 ) ≤ (𝐶 𝜇 𝑑 + 4𝐿𝛾0 𝜂0 𝛼) ¯ 𝑇,
h q i √ 4𝐿 𝛾0 𝑑 log(1 + 𝑇 𝐶¯𝜙2 /𝜂0 ) + 2 log(2/𝛿) + 𝜂0 𝛼¯ r min{𝜇, 1}
(80)
1 + 𝐶¯𝜙2 /𝜂0 log 1 + 𝑇 𝐶¯𝜙2 /(𝜂0 𝑑) .
(81)
Additionally, Assumption 5 and Proposition 7 imply that with probability at least 1 − 𝛿/3, 𝜆 min (Φ𝑡−1 ) ≥ (𝑡 − 1)/[16polylog(𝑇)] holds uniformly for all 𝑡 ≥ 𝜏˜0 := 100𝐶¯𝜙2 polylog(𝑇) log(6𝑇 𝐴𝑑/𝛿) and 𝑎 ∈ [𝐴], under which Õ
𝑠∈S𝑎𝑡
k𝜙(𝑥 𝑠 , 𝜔 𝑎𝑠 )k 2Φ−1 ≤ 𝑡 −1
16polylog(𝑇)𝐶¯𝜙2 𝑛𝑎𝑡 𝑡 −1
≤ 16polylog(𝑇)𝐶¯𝜙2 ,
where the last inequality uses the fact that 𝑛𝑎𝑡 ≤ 𝑡. Henceforth, combining (76) and (77), with probability at least 1 − 𝛿, uniformly over all 𝑡 ≥ 100𝐶¯𝜙2 polylog(𝑇) log(6𝑇 𝐴𝑑/𝛿) we have s
20 1 4 144 2 ¯ k 𝜌ˆ 𝑎𝑡 − 𝜌 𝑎 k𝑉𝑎𝑡 (𝜂1 ) ≤ + 2 min{2𝜄, 𝐶𝜓 𝑑 B } + 6 Γ𝑎𝑡 (𝛿) + + + 𝜂1 𝑑 B 𝜅𝜇1 𝜅𝜇1 𝜅𝜇1 2 √ q 8 2𝐿 ¯ + 𝐶𝜙 𝐶¯𝑎𝑡 (𝛿)polylog(𝑇). 𝜅𝜇1
Furthermore, by (27) and the Lipschitz property of Λ by Assumption 1, conditional on event E, with probability at least 1 − 𝛿, uniformly over all 𝑎 ∈ [𝐴], 𝑡 ≥ 𝜏˜0 = 100𝐶¯ 2 polylog(𝑇) log(6𝑇 𝐴𝑑/𝛿), 𝜙
where 𝐶𝜎 = 2𝐿
s
e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ≤ 𝐶𝜎 k𝜓(𝑥𝑡 )k𝑉𝑎𝑡 (𝜂1 ) −1 ,
(82)
4 144 20 1 2 + 2 min{2𝜄, 𝐶¯𝜓 𝑑 B } + 6 Γ 𝑎 (𝛿) + + + 𝜂1 𝑑 B 𝜅𝜇1 𝜅𝜇1 𝜅𝜇1 2 √ q 8 2𝐿 ¯ 𝐶𝜙 𝐶¯𝑎 (𝛿)polylog(𝑇) , + 𝜅𝜇1
and Γ𝑎 (𝛿), 𝑄 𝑎 , 𝐶¯𝑎 (𝛿) are defined as
6𝐴𝑇 , Γ𝑎 (𝛿) := 𝑚 log(1 + 2𝑑 B 𝐶¯𝜓 𝑇) + log(2𝑄 𝑎 + 2) + log 𝛿
𝑄 𝑎 :=
l
(83)
2
2 ¯2 log2 1 + 𝑇 min{9𝜄 , 𝑑 B 𝐶𝜓 }
m
,
55
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
q 2 4 √ 2 ¯ 𝛾0 𝑑 log(1 + 𝑇 𝐶𝜙 /𝜂0 ) + 2 log(6/𝛿) + 𝜂0 𝛼¯ . min{𝜇, 1}2 Õ Define 𝑉˜𝑎,𝑛 𝑎𝑡 := 𝜂1 I𝑚 + 𝜓(𝑥 𝑠 )𝜓(𝑥 𝑠 ) ⊤ = 𝑉𝑎𝑡 (𝜂1 ) and let 𝑉˜𝑎,0 := 𝜂1 I𝑚 . Note that 𝐶¯𝑎 (𝛿) :=
𝑠∈S𝑎𝑡
det(𝑉˜𝑎,𝑛 𝑎𝑡 ) = det 𝑉˜𝑎,𝑛 𝑎𝑡 −1
1 + k𝜓(𝑥𝑡 𝑎 (𝑛 𝑎𝑡 ) )k𝑉2˜ −1 𝑎,𝑛
𝑎𝑡 −1
,
where 𝑡 𝑎 (𝑛𝑎𝑡 ) is the period where 𝑎 is queried the 𝑛𝑎𝑡 -th time. Thus Õ det(𝑉˜𝑎,𝑛 𝑎𝑇 ) 2 log = log 1 + k𝜓(𝑥 𝑠 )k𝑉˜ −1 𝑎,𝑛𝑎𝑠 det(𝑉˜𝑎,0 ) 𝑠∈S𝑎𝑇 2 2 Õ k𝜓(𝑥 𝑠 )k𝑉˜𝑎,𝑛 Õ k𝜓(𝑥 𝑠 )k𝑉˜𝑎,𝑛 −1 −1 𝑎𝑠 𝑎𝑠 , ≥ ≥ (𝑖) (𝑖𝑖) 2 2 /𝜂 ¯ 1 + k𝜓(𝑥 )k 1 + 𝐶 𝑠 1 −1 𝜓 ˜ 𝑠∈S 𝑠∈S 𝑉𝑎,𝑛𝑎𝑠
𝑎𝑇
𝑎𝑇
𝑎 for 𝑎 ≥ 0 and (ii) uses the fact that where inequality (i) uses the fact that log(1 + 𝑎) ≥ 1+𝑎 k𝜓(𝑥 𝑠 )k𝑉2˜ −1 ≤ 𝐶¯𝜓2 /𝜂1 . Hence the above inequality implies that 𝑎,𝑛 𝑎𝑠
det(𝑉˜𝑎,𝑛 𝑎𝑇 ) ≤ (1) (1 + 𝐶¯𝜓2 /𝜂1 )𝑚 log 1 + 𝑛𝑎𝑇 𝐶¯𝜓2 /(𝜂1 𝑚) , 𝑎,𝑛𝑎𝑡 det(𝑉˜𝑎,0 ) 𝑠∈S𝑎𝑇 ! ˜ 𝑚 ¯2 𝑚 𝑛 𝐶 𝑎𝑇 tr( 𝑉 ) 𝜓 𝑎,𝑛 𝑎𝑇 where (1) follows since det(𝑉˜𝑎,0 ) = 𝜂1𝑚 and det(𝑉˜𝑎,𝑛 𝑎𝑇 ) ≤ so that ≤ 𝜂1 + 𝑚 𝜂1 𝑚 ! 𝑛𝑎𝑇 𝐶¯𝜓2 det(𝑉˜𝑎,𝑛 𝑎𝑇 ) ≤ 𝑚 log 1 + . Further, for any 𝑎 ∈ [𝐴], log 𝜂1 𝑚 det(𝑉˜𝑎,0 ) Õ
k𝜓(𝑥 𝑠 )k𝑉2˜ −1 ≤ (1 + 𝐶¯𝜓2 /𝜂1 ) log
𝑇 Õ
𝑡=𝜏˜0
k𝜓(𝑥𝑡 )k𝑉˜𝑎,𝑛 −1 I{𝑎 ∈ A 𝑡 } ≤ 𝑎𝑡
𝑇 Õ
k𝜓(𝑥𝑡 )k𝑉˜𝑎,𝑛 −1 I{𝑎 ∈ A 𝑡 } =
𝑡=1s
≤ (𝑖)
𝑛𝑎𝑇
Õ
𝑠∈S𝑎𝑇
𝑎𝑡
Õ
𝑠∈S𝑎𝑇
k𝜓(𝑥 𝑠 )k𝑉˜𝑎,𝑛 −1
𝑎𝑠
k𝜓(𝑥 𝑠 )k 2˜ −1 , 𝑉𝑎,𝑛𝑎𝑠
where inequality (i) follows from Cauchy-Schwarz inequality. Note that 𝑛𝑎𝑇 ≤ 𝑇, thus for any 𝑎 ∈ [𝐴],
𝑇 Õ
𝑡=𝜏˜0
k𝜓(𝑥𝑡 )k𝑉˜𝑎,𝑛 −1 I{𝑎 ∈ A 𝑡 } ≤ 𝑎𝑡
r
2 2 ¯ ¯ 𝑇 (1 + 𝐶𝜓 /𝜂1 )𝑚 log 1 + 𝑇 𝐶𝜓 /(𝜂1 𝑚) .
Combining with (82), conditional on event E, with probability at least 1 − 𝛿, uniformly over all 𝑎 ∈ [𝐴], we have 𝑇 Õ 𝑡=1
(e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 )I{𝑎 ∈ A𝑡 } ≤
𝜏Õ ˜0 −1
𝑇 Õ
(e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 )I{𝑎 ∈ A𝑡 } + (e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 )I{𝑎 ∈ A𝑡 } 𝑡= 𝜏 ˜ 0 r ≤ 2( 𝜏˜0 − 1) + 𝐶𝜎 𝑇 (1 + 𝐶¯ 2 /𝜂1 )𝑚 log 1 + 𝑇 𝐶¯ 2 /(𝜂1 𝑚) , 𝑡=1
𝜓
𝜓
56
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
where the second inequality follows because for 𝑡 ∈ [1, 𝜏˜0 ], e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ≤ 2 by definition of the indices
according to Assumption 1. Henceforth, conditional on event E, with probability at least 1 − 𝛿, 𝑇 Õ Õ
𝑡=1 𝑎∈A 𝑡
(e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 ) =
𝑇 Õ Õ
(e 𝜎𝑎𝑡 − 𝜎𝑎𝑡 )I{𝑎 ∈ A𝑡 } r
≤ 2𝐴𝜏˜0 + 𝐴𝐶𝜎 𝑇 (1 + 𝐶¯𝜓2 /𝜂1 )𝑚 log 1 + 𝑇 𝐶¯𝜓2 /(𝜂1 𝑚) . 𝑎∈[𝐴] 𝑡=1
Note that by definition, Δ𝑡 ( 𝜋) ˜ ≤ 1 + 2𝐴 for any 𝑡 ∈ [𝑇]. So (79) and (80) together further imply that " 𝑇 # Õ √ √ √ E Δ𝑡 ( 𝜋) ˜ E ≤ (𝐶 𝜇 𝑑 + 4𝐿𝛾0 𝜂0 𝛼) ¯ 𝑇 + 2𝐴𝜏˜0 + 𝑇 (1 + 2𝐴)𝛿 𝑡=1 (84) r +𝐴𝐶𝜎 𝑇 (1 + 𝐶¯ 2 /𝜂1 )𝑚 log 1 + 𝑇 𝐶¯ 2 /(𝜂1 𝑚) . 𝜓
𝜓
Recall that E holds with probability at least 1 − 𝛿. Thus " 𝑇 # Õ 𝑐 E Δ𝑡 ( 𝜋)I{E ˜ } ≤ 𝑇 (2𝐴 + 1)𝛿.
(85)
𝑡=1
√ Thus setting 𝛿 = 1/ 𝑇, (84) and (85) imply that ignoring logarithmic factors, we have E
Õ 𝑇
√ √ √ Δ𝑡 ( 𝜋) ˜ ≤ (𝐶 𝜇 𝑑 + 4𝐿𝛾0 𝜂0 𝛼¯ + 4𝐴 + 2) 𝑇 + 2𝐴𝜏˜0 𝑡=1 r +𝐴𝐶𝜎 𝑇 (1 + 𝐶¯ 2 /𝜂1 )𝑚 log 1 + 𝑇 𝐶¯ 2 /(𝜂1 𝑚) . 𝜓
(86)
𝜓
Recall that 𝐶 𝜇 and 𝐶𝛿 are defined as (81) and (83). So ignoring logarithmic factors we have " 𝑇 # Õ √ √ e (𝑑 + 𝐴𝑚 + 𝐴 𝑑𝑚) 𝑇 . E Δ𝑡 ( 𝜋) ˜ ≤𝑂 𝑡=1
Next, we compute 𝜂0 , 𝜂1 that minimize the order of the right hand side of (86) (ignoring logarithmic e factor dependence). In the following, for functions 𝑓 , 𝑔 : N → R+ , we write 𝑓 (𝑇) = Θ(𝑔(𝑇)) if both e(𝑔(𝑇)) and 𝑔(𝑇) = 𝑂 e( 𝑓 (𝑇)) hold. 𝑓 (𝑇) = 𝑂
q e 1 + 1 . Moreover, Γ𝑎 (𝛿) = Θ(𝑚) e Suppressing logarithmic factors, we have 𝐶 𝜇 = Θ and 𝜂0 q √ √ e 𝑑 + √𝜂0 . Therefore, 𝐶𝜎 = Θ e √𝑚 + 𝜂1 + 𝑑 + √𝜂0 . Substituting these estimates 𝐶 𝑎 (𝛿) = Θ
into the regret bound and dropping additive terms independent of 𝜂0 , 𝜂1 , the relevant log-free objective is
s " s # √ √ √ 1 √ 1 √ √ 𝑚 + 𝜂1 + 𝑑 + 𝜂0 . 𝑅(𝜂0 , 𝜂1 ) ≍ 𝑇 𝑑 1 + + 𝜂0 + 𝐴 𝑚 1 + 𝜂0 𝜂1
57
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
√ 𝑇 is common to all 𝜂0 , 𝜂1 -dependent terms, it does not affect the minimizer. q √ √ Define 𝑔(𝜂1 ) := 1 + 𝜂11 , 𝑏(𝜂1 ) := 𝑚 + 𝜂1 + 𝑑. Then the relevant objective can be written as q √ √ √ 𝑟 (𝜂0 , 𝜂1 ) = 𝑑 1 + 𝜂10 + 1 + 𝐴 𝑚 𝐺 (𝜂1 ) 𝜂0 + 𝐴 𝑚 𝑔(𝜂1 )𝑏(𝜂1 ). We first optimize 𝜂0 for fixed 𝜂1 . p √ √ Let 𝐾 (𝜂1 ) := 1 + 𝐴 𝑚 𝑔(𝜂1 ). The 𝜂0 -dependent part is 𝑓 (𝜂0 ; 𝜂1 ) = 𝑑 1 + 1/𝜂0 + 𝐾 (𝜂1 ) 𝜂0 . The Since the factor
first-order condition is −
2𝜂02
√𝑑
1+1/𝜂0
+ 𝐾2√(𝜂𝜂10) = 0, equivalently, 𝐾 (𝜂1 ) =
Then the positive solution satisfies
𝜌(𝜂1 ), ★ 𝜂0 (𝜂1 ) ≍ 𝜌(𝜂1 ) 2/3 ,
𝜂0
√𝑑
1+𝜂0
𝑑 . . Let 𝜌(𝜂1 ) := 𝐾 (𝜂 1)
𝜌(𝜂1 ) ® 1, 𝜌(𝜂1 ) ¦ 1.
It remains to optimize 𝜂1 . By the envelope theorem, q √ 𝑑 ★ ′ ′ ★ 𝑟 (𝜂0 (𝜂1 ), 𝜂1 ) = 𝐴 𝑚 𝑔 (𝜂1 ) 𝑏(𝜂1 ) + 𝜂0 (𝜂1 ) + 𝑔(𝜂1 )𝑏 (𝜂1 ) . 𝑑𝜂1 𝑔 ′ (𝜂 )
1 , the first-order condition is equivalent to Since 𝑔(𝜂11) = − 2𝜂1 (𝜂11 +1) , 𝑏 ′ (𝜂1 ) = 2√𝑚+𝜂 1 q √ √ √ ★ 𝜂1 (𝜂1 + 1) = 𝑚 + 𝜂1 𝑚 + 𝜂1 + 𝑑 + 𝜂0 (𝜂1 ) .
Thus 𝜂12 − 𝑚 =
h√
i√ q q ★ 𝑑 + 𝜂0 (𝜂1 ) 𝑚 + 𝜂1 . We next show that the term 𝜂★0 (𝜂1 ) never changes the
polynomial order of the 𝜂1 minimizer. Since 𝐾 (𝜂1 ) ≥ 1, we have 𝜌(𝜂1 ) ≤ 𝑑. If 𝜌(𝜂1 ) ® 1, then q q √ √ ★ 1/2 𝜂0 (𝜂1 ) ≍ 𝜌(𝜂1 ) ® 1 ® 𝑑. If 𝜌(𝜂1 ) ¦ 1, then 𝜂★0 (𝜂1 ) ≍ 𝜌(𝜂1 ) 1/3 ® 𝑑 1/3 ® 𝑑. Therefore, √ q ★ √ √ √ 𝑑 + 𝜂0 (𝜂1 ) ≍ 𝑑, and the 𝜂1 balancing equation reduces to 𝜂12 ≍ 𝑚 + 𝑑 𝑚 + 𝜂1 . We now solve √ √ √ this equation by considering different regimes. If 𝑑 ® 𝑚, then at 𝜂1 ≍ 𝑚, 𝑑 𝑚 + 𝜂1 ® 𝑚. Hence √ 𝜂★1 ≍ 𝑚 1/2 . If 𝑚 ® 𝑑 ® 𝑚 3 , then the solution satisfies 𝜂1 ® 𝑚, so 𝑚 + 𝜂1 ≍ 𝑚. Hence 𝜂12 ≍ 𝑑𝑚, and therefore 𝜂★1 ≍ (𝑑𝑚) 1/4 . If 𝑑 ¦ 𝑚 3 , then the solution satisfies 𝜂1 ¦ 𝑚, so 𝑚 + 𝜂1 ≍ 𝜂1 . Hence 𝜂12 ≍ p 𝑑𝜂1 , and therefore 𝜂★1 ≍ 𝑑 1/3 . Combining the three regimes gives 𝜂★1 ≍ max 𝑚 1/2, (𝑑𝑚) 1/4 , 𝑑 1/3 . √ √ At this optimizer, 𝐺 (𝜂★1 ) = Θ(1). Therefore, 𝐾 (𝜂★1 ) = 1 + 𝐴 𝑚 𝐺 (𝜂★1 ) ≍ 𝐴 𝑚. Substituting this into the expression for 𝜂★0 gives 𝜌(𝜂★1 ) = 𝐾 (𝜂𝑑 ★) ≍ 𝐴√𝑑 𝑚 . Thus 1
√ 𝑑 𝑑 ® 𝐴 𝑚, √ , 𝐴 𝑚 𝜂★0 ≍ 𝑑 2/3 √ , 𝑑 ¦ 𝐴 𝑚. 𝐴√ 𝑚 2/3 Equivalently, 𝜂★0 ≍ min 𝐴√𝑑 𝑚 , 𝐴√𝑑 𝑚 . Consequently, suppressing logarithmic factors, the opti 2/3 1/2 𝑑 𝑑 ★ 1/4 1/3 ★ . mal polynomial choices are 𝜂1 ≍ max 𝑚 , (𝑑𝑚) , 𝑑 , 𝜂0 ≍ min 𝐴√𝑚 , 𝐴√𝑚
58
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
ALGORITHM 2: COSMOS Policy Algorithm Initialization. Observe an initial context 𝑥0 . Query each box 𝑎 ∈ [𝐴] once and record the observed output-cost pairs {(𝜔 𝑎0 , 𝑐 𝑎0)} 𝑎∈[𝐴] . for 𝑡 = 1, 2, . . . , 𝑇 do Observe context 𝑥𝑡 . Initialize A𝑡 = ∅ and 𝑀𝑡 = −∞;
⊤ 𝜓(𝑥 ) + 𝐵 k𝜓(𝑥 )k ˜ 𝑎𝑡 = Λ( 𝜌ˆ 𝑎𝑡 For any 𝑎 ∈ [𝐴], compute 𝜌ˆ 𝑎𝑡 by (23) and set 𝜎 𝑡 𝑎𝑡 𝑡 𝑉𝑎𝑡 (𝜂1 ) −1 ),
where 𝐵𝑎𝑡 is defined as in Proposition 5, 𝑉𝑎𝑡 (𝜂1 ) is defined as (13);
Order the boxes so that e 𝜎(1)𝑡 ≥ e 𝜎(2)𝑡 ≥ · · · ≥ e 𝜎( 𝐴)𝑡 . Set e 𝜎( 𝐴+1)𝑡 := −∞; for 𝑘 = 1, 2, . . . , 𝐴 do
Query box (𝑘) and observe its output and cost (𝜔 (𝑘)𝑡 , 𝑐 (𝑘)𝑡 );
Add (𝑘) to the queried set: A𝑡 ← A𝑡 ∪ {(𝑘)};
Compute the optimistic reward estimate ⊤ ˆ 𝜇 e𝑡 (𝑥𝑡 , 𝜔 (𝑘)𝑡 ) = 𝐺 𝜃 𝑡−1 𝜙(𝑥𝑡 , 𝜔 (𝑘)𝑡 ) + 𝛽𝑡 𝜙(𝑥𝑡 , 𝜔 (𝑘)𝑡 ) Φ−1 , where Φ𝑡−1 is defined as 𝑡 −1
in (20) and 𝛽𝑡 is as given in Lemma 3;
Update 𝑀𝑡 ← max𝑎∈A𝑡 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 );
if 𝑀𝑡 ≥ e 𝜎(𝑘+1)𝑡 , stop querying and break;
end
Select and deploy 𝑎 𝑡 ∈ arg max𝑎∈A𝑡 𝜇 e𝑡 (𝑥𝑡 , 𝜔 𝑎𝑡 ), and observe reward 𝑟 𝑡 ;
end
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
59
References Abbasi-Yadkori Y, Pál D, Szepesvári C (2011) Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24. Agarwal A, Ghuge R, Nagarajan V (2024) Semi-bandit learning for monotone stochastic optimization. 2024 IEEE 65th Annual Symposium on Foundations of Computer Science (FOCS), 1260–1274 (IEEE). Agrawal A, Kedia N, Panwar A, Mohan J, Kwatra N, Gulavani BS, Tumanov A, Ramjee R (2024) Taming throughputlatency tradeoff in llm inference with sarathi-serve. Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation. Andrews I, Mikusheva A (2022) Optimal decision rules for weak gmm. Econometrica 90(2):715–748. Ao R, Luo G, Simchi-Levi D, Wang X (2025) Optimizing llm inference: Fluid-guided online scheduling with memory constraints. arXiv preprint arXiv:2504.11320 . Arellano M, Bond S (1991) Some tests of specification for panel data: Monte carlo evidence and an application to employment equations. The review of economic studies 58(2):277–297. Atsidakou A, Caramanis C, Gergatsouli E, Papadigenopoulos O, Tzamos C (2024) Contextual pandora’s box. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 10944–10952. Audibert JY, Munos R, Szepesvári C (2009) Exploration–exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science 410(19):1876–1902. Auer P, Cesa-Bianchi N, Fischer P (2002) Finite-time analysis of the multiarmed bandit problem. Machine learning 47(2):235–256. Boodaghians S, Fusco F, Lazos P, Leonardi S (2020) Pandora’s box problem with order constraints. Proceedings of the 21st ACM Conference on Economics and Computation, 439–458. Chamberlain G (1987) Asymptotic efficiency in estimation with conditional moment restrictions. Journal of econometrics 34(3):305–334. Chawla S, Gergatsouli E, Teng Y, Tzamos C, Zhang R (2020) Pandora’s box with correlations: Learning and approximation. 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), 1214–1225 (IEEE). Chen L, Zaharia M, Zou J (2025a) Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research . Chen Y, Kirshner SN, Ovchinnikov A, Andiappan M, Jenkin T (2025b) A manager and an ai walk into a bar: does chatgpt make biased decisions like we do? Manufacturing & Service Operations Management 27(2):354–368. Chen Z, Chan J (2024) Large language model in creative work: The role of collaboration modality and user expertise. Management Science 70(12):9101–9117. Chen Z, Yang X, Lin J, Sun C, Chang KC, Huang J (2024) Cascade speculative drafting for even faster llm inference. Advances in Neural Information Processing Systems 37:86226–86242.
60
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Cheng T, Dong C, Gao J, Linton O (2024) Gmm estimation for high-dimensional panel data models. Journal of Econometrics 244(1):105853. Cheung WC, Ma W, Simchi-Levi D, Wang X (2022) Inventory balancing with online learning. Management Science 68(3):1776–1807. Ding Q, Hsieh CJ, Sharpnack J (2021) An efficient algorithm for generalized linear bandit: Online stochastic gradient descent and thompson sampling. International Conference on Artificial Intelligence and Statistics, 1585–1593 (PMLR). Doval L (2018) Whether or not to open pandora’s box. Journal of Economic Theory 175:127–158. Ezra T, Feldman M, Schlesinger M (2026) Contract design for sequential actions. Proceedings of the 2026 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 6537–6570 (SIAM). Fan L, Glynn PW (2025) The fragility of optimized bandit algorithms. Operations Research 73(6):3173–3198. Fang C, Li X, Fan Z, Xu J, Nag K, Korpeoglu E, Kumar S, Achan K (2024) Llm-ensemble: Optimal large language model ensemble method for e-commerce product attribute value extraction. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2910–2914. Filippi S, Cappe O, Garivier A, Szepesvári C (2010) Parametric bandits: The generalized linear case. Advances in neural information processing systems 23. Freedman DA (1975) On tail probabilities for martingales. the Annals of Probability 100–118. Fu H, Lin T (2020) Learning utilities and equilibria in non-truthful auctions. Advances in Neural Information Processing Systems 33:14231–14242. Gao X, Jasin S, Najafi S, Zhang H (2022) Joint learning and optimization for multi-product pricing (and ranking) under a general cascade click model. Management Science 68(10):7362–7382. Garivier A, Cappé O (2011) The kl-ucb algorithm for bounded stochastic bandits and beyond. Proceedings of the 24th annual conference on learning theory, 359–376 (JMLR Workshop and Conference Proceedings). Gatmiry K, Kesselheim T, Singla S, Wang Y (2024) Bandit algorithms for prophet inequality and pandora’s box. Proceedings of the 2024 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 462–500 (SIAM). Gergatsouli E, Tzamos C (2022) Online learning for min sum set cover and pandora’s box. International Conference on Machine Learning, 7382–7403 (PMLR). Gupta N, Narasimhan H, Jitkrittum W, Rawat AS, Menon AK, Kumar S (2024) Language model cascades: Token-level uncertainty and beyond. arXiv preprint arXiv:2404.10136 . Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, et al. (2024) Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine 30(9):2613–2622. Hansen BE, Lee S (2021) Inference for iterated gmm under misspecification. Econometrica 89(3):1419–1447.
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
61
Hansen LP (1982) Large sample properties of generalized method of moments estimators. Econometrica: Journal of the econometric society 1029–1054. Hao Y, Qiu Z, Holmes J, Löckenhoff CE, Liu W, Ghassemi M, Kalantari S (2025) Large language model integrations in cancer decision-making: a systematic review and meta-analysis. NPJ Digital Medicine 8(1):450. Hari SN, Thomson M (2023) Tryage: Real-time, intelligent routing of user prompts to large language models. arXiv preprint arXiv:2308.11601 . Hu J, Wang Y, Zhang S, Zhou K, Chen G, Hu Y, Xiao B, Tan M (2025) Efficient dynamic ensembling for multiple llm experts. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI, 16–22. Hu QJ, Bieker J, Li X, Jiang N, Keigwin B, Ranganath G, Keutzer K, Upadhyay SK (2024) Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031 . Huang J, Ma W, Zhou Z (2026) Optimal bayesian stopping for efficient inference of consistent llm answers. arXiv preprint arXiv:2602.05395 . Jaillet P, Jiang J, Mellou K, Molinaro M, Podimata C, Zhou Z (2025) Online scheduling for llm inference with kv cache constraints. arXiv preprint arXiv:2502.07115 . Jiang D, Ren X, Lin BY (2023) Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561 . Jun KS, Bhargava A, Nowak R, Willett R (2017) Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems 30. Kim W, Lee K, Paik MC (2023) Double doubly robust thompson sampling for generalized linear contextual bandits. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 8300–8307. Kleinberg R, Waggoner B, Weyl EG (2016) Descending price coordinates approximately efficient search. Extended abstract in the Proceedings of the 17th ACM Conference on Electronic Commerce (EC’16). Kveton B, Zaheer M, Szepesvari C, Li L, Ghavamzadeh M, Boutilier C (2020) Randomized exploration in generalized linear bandits. International Conference on Artificial Intelligence and Statistics, 2066–2076 (PMLR). Kwon W, Li Z, Zhuang S, Sheng Y, Zheng L, Yu CH, Gonzalez JE, Zhang H, Stoica I (2023) Efficient memory management for large language model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. Lai TL, Robbins H (1985) Asymptotically efficient adaptive allocation rules. Advances in applied mathematics 6(1):4– 22. Lattimore T, Szepesvári C (2020) Bandit algorithms (Cambridge University Press). Lee J, Yun SY, Jun KS (2024) A unified confidence sequence for generalized linear models, with applications to bandits. Advances in Neural Information Processing Systems 37:124640–124685.
62
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
Li G, Liang J, Liu M, Lei Y, Jasin S, Yang F, Baxi P (2026) Asymptotically optimal sequential testing with heterogeneous llms. arXiv preprint arXiv:2604.01086 . Li L, Lu Y, Zhou D (2017) Provably optimal algorithms for generalized linear contextual bandits. International Conference on Machine Learning, 2071–2080 (PMLR). Lin X, Lee Lf (2010) Gmm estimation of spatial autoregressive models with unknown heteroskedasticity. Journal of Econometrics 157(1):34–52. Liu J, Chen Z, Wang K, Luo H, Ratliff LJ (2025) Improved regret and contextual linear extension for pandora’s box and prophet inequality. arXiv preprint arXiv:2505.18828 . Lu K, Yuan H, Lin R, Lin J, Yuan Z, Zhou C, Zhou J (2024) Routing to the expert: Efficient reward-guided ensemble of large language models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 1964–1974. Mao W, Zhang K, Zhu R, Simchi-Levi D, Başar T (2025) Model-free nonstationary reinforcement learning: Nearoptimal regret and applications in multiagent reinforcement learning and inventory control. Management Science 71(2):1564–1580. Mei K, Xu W, Guo M, Lin S, Zhang Y (2025) Omnirouter: Budget and performance controllable multi-llm routing. ACM SIGKDD Explorations Newsletter 27(2):107–116. Newey WK, McFadden D (1994) Large sample estimation and hypothesis testing. Handbook of econometrics 4:2111– 2245. Nie L, Ding Z, Hu E, Jermaine C, Chaudhuri S (2024) Online cascade learning for efficient inference over streams. arXiv preprint arXiv:2402.04513 . Rakhlin S, Sridharan K (2013) Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems 26. Reisenbichler M, Reutterer T, Schweidel DA (2025) Applying large language models to sponsored search advertising. Marketing Science . Rusmevichientong P, Tsitsiklis JN (2010) Linearly parameterized bandits. Mathematics of Operations Research 35(2):395–411. Šakota M, Peyrard M, West R (2024) Fly-swat or cannon? cost-effective language model choice via meta-modeling. Proceedings of the 17th ACM International Conference on Web Search and Data Mining, 606–615. Shnitzer T, Ou A, Silva M, Soule K, Sun Y, Solomon J, Thompson N, Yurochkin M (2023) Large language model routing with benchmark datasets. First Conference on Language Modeling. Simchi-Levi D, Mellou K, Menache I, Pathuri J (2026) Large language models for supply chain decisions. AI in Supply Chains: Perspectives from Global Thought Leaders, 93–104 (Springer). Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW (2023) Large language models in medicine. Nature medicine 29(8):1930–1940.
Belloni, Chen, and Wei: Online Pandora’s Box for Contextual LLM Cascading
63
Tropp JA (2011) User-friendly tail bounds for matrix martingales. Technical report. Weitzman ML (1979) Optimal search for the best alternative. Econometrica 47(3). Yang Q, Ongpin M, Nikolenko S, Huang A, Farseev A (2023) Against opacity: Explainable ai and large language models for effective digital advertising. Proceedings of the 31st ACM International Conference on Multimedia, 9299–9305. Yu GI, Jeong JS, Kim GW, Kim S, Chun BG (2022) Orca: A distributed serving system for transformer-based generative models. Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation, 521– 538. Yue M, Zhao J, Zhang M, Du L, Yao Z (2024) Large language model cascades with mixture of thought representations for cost-efficient reasoning. The Twelfth International Conference on Learning Representations. Zhang X, Huang Z, Taga EO, Joe-Wong C, Oymak S, Chen J (2024) Efficient contextual llm cascades through budget-constrained policy learning. Advances in Neural Information Processing Systems 37:91691–91722. Zhou Z, Xu R, Blanchet J (2019) Learning in generalized linear contextual bandits with stochastic delays. Advances in Neural Information Processing Systems 32.