Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method Kyoungmin Kim
Martin Catheland
Anastasia Ailamaki
EPFL Lausanne, Switzerland [email protected]
EPFL Lausanne, Switzerland [email protected]
EPFL Lausanne, Switzerland [email protected]
arXiv:2606.08090v1 [cs.DB] 6 Jun 2026
Abstract Evaluating a natural-language yes/no predicate over a document corpus under a corpus-level accuracy target — the semantic filter — is a cornerstone of LLM-based data processing. The naive approach is to call the Large Language Model (LLM) on every document, which is considered as the oracle but prohibitively expensive as an online processing. Fast-proxy-then-LLM cascades are the standard remedy but, as deployed today, leave four structural limitations on the table. (1) Each existing family — model-free clustering, prebuilt small-LLM proxies, and online-trained proxies — commits to a single representation and a fixed pipeline, and so wins on only a narrow query regime. (2) The strongest online proxy invests heavily in a custom-designed training scheme while leaving established representation-learning tools on the table; its bi-encoder over dense embeddings captures only topical similarity, missing the token-level evidence and richer semantics many predicates require. (3) The same proxy is trained against binary yes/no labels, weighting every document identically, and wasting the LLM’s perdocument confidence at exactly the boundary documents the proxy most needs to learn. (4) Existing calibrations are too conservative. Proxy scores are aligned to the oracle LLM and a threshold is chosen to cascade lower-score documents to the oracle to meet accuracy target, which is often too larger than necessary, inflating the cascade cost. We observe a common root cause: they do not separate when the proxy is genuinely uncertain on those documents from when the labeled sample is simply too small to be sure, and treat both with the same safety margin. We address these limitations as follows. (1) We adapt to each query’s difficulty by composing families: we first try lightweight model-free clustering, and escalate to an online-trained proxy only when clustering becomes less efficient to classify all documents confidently. We share oracle LLM calls across phases to reduce the transition cost. (2) For that online proxy, we replace the cosine bi-encoder with a hybrid of off-the-shelf representation models that capture token-level evidence and fine-grained semantics. (3) We train the proxy with the oracle’s per-document confidence as a soft label, so the proxy itself learns to be unsure on documents the oracle was unsure about, instead of being forced toward yes/no only and being confident on documents it should not be confident on. (4) Our calibration separates when the proxy is uncertain from when the sample is too small to be sure, and adds a safety margin only on the latter. This leads to fewer LLM cascades at the same accuracy target. Also, we are the first to use the oracle LLM’s own per-document confidence for three new purposes in this setting: as a query-level difficulty compass that explains which family wins on which query, to derive a lower bound on the minimum oracle calls any proxy-based cascade can make at the target accuracy, and
as the training signal (soft label) for our proxy. At a 90% accuracy target on three 10K-document corpora, our methods are 1.6–2.0× faster than the best prior method on each corpus, and meet the target accuracy on 95% of queries. The latency lower bound derived from the oracle confidence indicates a further ∼4–20× of headroom for future work.
Keywords Semantic filtering, LLM cascades, proxy models, Bayes error rate, cross-encoder, ColBERT, two-phase execution, unified framework
1
Introduction
Database engines are increasingly asked to evaluate predicates on unstructured data that are not expressible in SQL on structured data [27, 34]. A clinician may want every medical publication abstract that “describes a randomized trial of an oncology drug”; an analyst may want every customer review that “expresses dissatisfaction with shipping.” Such predicates have natural-language form, no closed schema, and no cheap implementation: today the only general way to evaluate them is to call a Large Language Model (LLM, e.g., Llama-3.1-70B-Instruct) on every document [45, 49]. This is used as the oracle to avoid more costly and manual human labeling. Still, on a corpus of 10K documents a single such predicate already takes thousands of seconds to evaluate exhaustively [49]; the cost is linear in the corpus size and document length, i.e., the amount of tokens. Semantic filtering as a query plan. Recent work treats this as a classical database problem: the LLM is an expensive user-defined function, and the optimizer’s job is to apply it to as few documents as possible while still meeting a user-specified accuracy target 𝛼 (e.g. 0.90) [27, 34, 49]. The dominant plan is a two-tier cascade [5, 8, 17, 28]: a lightweight surrogate or proxy is evaluated on every document and the LLM is called only on the documents the proxy is uncertain about. The plan is parameterized by a score threshold that controls how many documents go to the LLM, and the optimization problem is to pick that threshold (and the proxy) so the SLA holds at the fewest oracle calls. Three families of cascades, none dominant. Three lines of work have emerged for the proxy. (i) Model-free. CSV [20] clusters the documents, samples a few per cluster, labels them with the oracle LLM, and propagates the majority label. There is no trained proxy. (ii) Prebuilt LLM proxy. BARGAIN [48] scores every document with a pre-trained small LLM (e.g. Llama-3.1-8B-Instruct) and falls back (or cascades) to the large oracle on the docs the small one is least confident about; the score threshold is calibrated on a small
Conference’17, July 2017, Washington, DC, USA
Figure 1: End-to-end latency vs. per-query difficulty (mean Bayes Error Rate (BER) of the oracle LLM over documents), on PubMed corpus at the 90% accuracy target (§8). Each point is a query; each line is a non-parametric smoothing curve. BERLB is our derivation of optimal performance. All methods cascade to more oracle LLM calls for harder queries to meet accuracy target.
labeled set to meet the target accuracy. There is no per-query training. (iii) Online-trained proxy. ScaleDoc [49] trains a per-query bi-encoder proxy on a small labeled training set with contrastive learning online. Each family has a niche — CSV on topical queries where embedding clusters align with the predicate; BARGAIN on reasoningheavy queries where a small LLM already separates yes from no, but classification with this small LLM adds an overhead; ScaleDoc on the middle ground, with an overhead of online labeling of training set and proxy training — but no family wins across the query mix we measure. Fig. 1 shows this. Also, no prior paper explains which family wins when, nor places the families on a single algorithmic skeleton, and nor expresses query difficulty as a quantifiable metric. A compass: the oracle’s own Bayes Error Rate (BER). Two facts make a difficulty-aware analysis possible. First, the oracle LLM returns not only a hard label 𝑦𝑖 but also a class probability 𝑝𝑖★ derived from output token logprobs — a soft label that is free in the sense that it requires no extra oracle call. Second, the per-document Bayes Error 𝜂𝑖 = min(𝑝𝑖★, 1 − 𝑝𝑖★) is a lower bound on the expected error of any classifier on that document [15], because the oracle is the most accurate label we have. The corpus average BER𝑞 for a query 𝑞 is therefore a query-level difficulty score that is methodindependent. We use it as a compass to explain which family wins, and to derive a lower bound on the minimum cascade calls any proxy can make. Four observations behind the contributions. This paper builds on four observations the prior literature has not isolated. The first three identify limitations in how existing cascades are designed; the fourth names a calibration that is often either too conservative in meeting the target accuracy. (O1) The three families share a skeleton. CSV, BARGAIN, and ScaleDoc all instantiate the same six-step template (partition, sample, label, build a proxy, calibrate, deploy). What differs is the choice along four orthogonal design knobs: proxy representation, proxy training, calibration, and partitioning. The conceptual benefit is large: prior “methods” become cells in a knobs × choices matrix.
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
New methods live in the empty cells or add new choices, and also can be designed through the seamless integration of multiple methods; we propose examples in this paper. (O2) ScaleDoc’s bottleneck is the architecture, not the training. ScaleDoc invests in a custom-designed multi-stage contrastive training scheme on a simple proxy (bi-encoder over dense embeddings). Before designing yet another training scheme, we revisit traditional representation-learning ML and find the trade-off is the wrong way around: the bi-encoder’s output cosine similarity is nonseparable on reasoning predicates, because (a) the dense embedding has already discarded the token-level evidence the predicate depends on, and (b) cosine similarity captures only topical relatedness of query and documents, not more complicated semantics. We replace the architecture with a hybrid of off-the-shelf token-aware representation models so the proxy can distinguish documents the bi-encoder fuses together; train it with the oracle’s per-document confidence as a soft label, so the proxy itself stays unsure on documents the oracle was unsure about instead of being forced toward yes/no; and impose a simple training-time constraint that ties training to the user’s accuracy target, shipping some of the calibration complexity to training. (O3) Existing calibrations are too conservative, cascading to the oracle LLM more often than necessary. The simplest calibration just uses the proxy’s measured error rate on the labeled sample to set the threshold; we observe this is optimistic on documents the sample does not cover densely — a handful of labels can leave the measured error rate arbitrarily off the truth. Existing state-of-the-art calibrations [48, 49] fix the optimism by adding a worst-case safety margin to the measured error rate. This is safe but conservative: the same safety margin is added on documents the labeled sample has covered densely and on documents it has not. We propose a calibration that adds a safety margin only on documents where the labeled sample is small, so the threshold tightens on the densely-covered documents and stays safe on the rest. (O4) No single method wins on all queries. Each family is the fastest plan in a different region of query difficulty: CSV on easy queries, our proxy on medium- to hard queries. The cheapest deployable plan would route each query to its best family. However, the routing decision itself costs oracle labels — a naive router spends a sample to identify the winner and then throws those labels away. We propose an adaptive Two-Phase method that tackles this: it runs CSV as Phase 1, terminates early when Phase 1 is efficient enough to classify all documents confidently, and otherwise reuses Phase-1’s oracle labels as Phase-2 training data for our proxy.
Contributions. We make the following contributions (C1–C5). (1) A unified framework (§3–§3.3) places CSV, BARGAIN, ScaleDoc, and our methods on one six-step skeleton with four explicit design knobs. The framework converts “different methods” into “different cells of one design matrix”. (2) A better online proxy by revisiting traditional ML (§4). We identify the bottleneck in ScaleDoc as the proxy’s architecture, not its training. We replace it with a hybrid of off-the-shelf token-aware representation models that capture richer query– document semantics than cosine over dense embeddings; train it with the oracle’s per-document confidence as the soft label;
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
and impose a small training-time constraint that aligns the proxy’s confidence with the deployment-time accuracy target. (3) A tighter calibration (§5). We increase the safety margin only on documents the labeled sample is too small to cover, and leave the safety margin at zero on documents the labeled sample already covers; our threshold calls the oracle LLM less often than prior calibrations at the same accuracy target. (4) Adaptive two-phase method (§6). A method that combines CSV (Phase 1) with our proxy and calibration (Phase 2), reusing Phase-1 oracle labels as Phase-2 training data to minimize the transition cost. Phase 2 is bypassed when Phase 1 is enough to classify all documents on easy queries. (5) BER as compass and headroom (§7). Cross-cutting across (C1)–(C4): we use the oracle’s own BER both as a query-level difficulty score (explaining which family wins on which query) and to derive a lower bound on the minimum LLM cascade calls any proxy-based method can make at the target accuracy. Our soft-label training (C2) reuses the same per-document confidence as BER. Headline empirical result. On three datasets of 10K documents and 20 queries each at a 90% accuracy target, our two-phase method runs 1.6–2.0× faster end-to-end than the best prior method per corpus, is at or near the fastest plan on every query, and meets the target accuracy on 95% of queries. The BER-derived latency lower bound sits a further ∼4–20× below it, indicating that semantic-filter optimization still has substantial headroom for future work. Plan of the paper. §2 surveys related work. §3 defines the problem and introduces the unified framework (C1). §4–6 present the three technical contributions: the revisited proxy (C2), calibration (C3), and the two-phase method (C4). §7 develops BER as a difficulty compass and a lower bound on cost (C5). §8 reports empirical results, and §9 concludes.
2
Related Work
This section explains the closest research areas with representative and state-of-the-art studies. LLM-powered data systems. Treating an LLM call as a user-defined function inside a relational plan goes back to probabilistic predicates over rich media [29] and continues with the recent wave of LLM-powered data systems such as Palimpzest [27], LOTUS [34], and DocETL [38]. These supply a query language (e.g., semantic Where/Group By/Join) and an optimizer that performs both logical optimization (operator reordering, plan rewrites) and physical optimization (selecting the best one among existing physical operators, or proxies in the fast-proxy-then-oracle-LLM cascade to reduce expensive oracle-LLM calls). Concurrent papers each refine a particular layer of this stack: PLOP [30] extends the logical layer with a dynamic-programming placement of semantic operators relative to relational ones; Chung et al. [10] characterize lightweight proxies as a class of physical operators for AI-extended SQL warehouses; and KEN [24] sits below the optimizer at the execution layer, scheduling cascade models on the GPU. Our work occupies a different point in this stack, proposing a new physical implementation of the semantic filter operator that the
Conference’17, July 2017, Washington, DC, USA
LLM-powered data systems above can employ. SUPG [21] used the cascading idea in the pre-LLM era, alike the speculation in database query processing [39]. LOTUS and Palimpzest above adopted it as one of their LLM-based physical implementations, and the focused state-of-the-art methods—our direct baselines—are ScaleDoc [49], CSV [20], and BARGAIN [48], discussed next. To our knowledge, no prior work has compared these state-of-the-art operators headto-head in a single experimental setting. New physical operator for semantic filter. The closest line is cascades that pair a cheap proxy score with the oracle. They are the prior art our contributions sit alongside. ScaleDoc [49] embeds documents into vectors offline, trains a per-query bi-encoder proxy (taking document and query embeddings as inputs and outputting a cosine similarity of transformed embeddings) online via a multi-stage contrastive scheme on a small oracle-labeled training sample. At deployment, it draws a stratified calibration sample, builds a 64-bin histogram of yes/no counts on the cosine score, smooths the per-bin counts, and searches for a two-sided band [ℓ, 𝑢] on the cosine score whose expected accuracy under the smoothed per-bin estimates meets the user target. Documents whose score falls outside the band are auto-labeled; documents inside the band are cascaded to the oracle. While its proxy adapts to each query online, (i) the bi-encoder over embeddings captures only topical similarity, leaving token-level evidence out of reach; and (ii) the calibration smooths the per-bin counts on the labeled sample before setting the threshold, which is a deliberate safety choice that prevents an optimistic threshold on score ranges the labeled sample covers only sparsely; but the same smoothing widens the auto-cascade band on every score range, including the well-covered ones where the measured error rate is already reliable on its own, so the threshold ends up calling the LLM on more documents than necessary to meet the target accuracy. CSV [20] is a model-free cascade: it clusters documents in dense embedding space, asks the oracle to label a small per-cluster sample, and propagates the majority label to the entire cluster if the percluster sample agrees on at least a vote-threshold fraction of its documents (CSV leaves this threshold as a free hyper-parameter; in our experiments we set it equal to the user’s target accuracy 𝛼 – see §6.3). Clusters whose sample does not agree are re-clustered into smaller ones; persistent disagreement falls back to per-document oracle calls. CSV is cheap when the embedding clustering aligns with the predicate (so most clusters are nearly all positive or all negative) but expensive when it does not: the same token-level evidence the ScaleDoc bi-encoder discards is also missing from the embeddings CSV clusters on, so predicates that depend on that evidence end up with many mixed clusters and a large oracle call cost. BARGAIN [48] differs from ScaleDoc in two ways. The proxy is a prebuilt small LLM rather than an online-trained bi-encoder, so the calibration sample is the only labeling cost beyond the cascade itself, without a separate training sample. The calibration replaces ScaleDoc’s smoothed empirical rate with a distribution-free highconfidence upper bound per score interval; this is finite-sample valid but more conservative than ScaleDoc’s smoothing on every well-covered interval, because the upper bound inflates the perinterval error rate above what the smoothed estimate would — so
Conference’17, July 2017, Washington, DC, USA
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
Table 1: Desired properties in semantic filter. ✓: delivered; ✗: not delivered; ∼: partially. Explanation of ∼ marks: (CSV) not declarative in the original paper, but setting internal parameters trivially meets target accuracy; (BARGAIN) prebuilt small LLM is partially trustworthy to oracle LLM; it processes tokens, but its scan over all documents adds a substantial cost. User-facing goal
ScaleDoc
CSV
BARGAIN
Ours
Declarative target accuracy. No need to manually tune internal parameters to meet target accuracy.
✓
∼ (no guide)
✓
✓
Headroom diagnosis. Measure how close the latency is to the optimal.
✗
✗
✗
✓
Trustworthy confidence. The proxy’s score reflects how likely the answer is to be correct.
✗
✗
∼ (small LLM)
✓
Expressive predicates. Efficiently support queries with token-level evidence beyond topical similarity.
✗
✗
∼ (small LLM)
✓
Tight calibration. The proxy score threshold is not too optimistic or too conservative.
✗
✗
✗
✓
Per-query competitiveness. A single method is competitive with the per-query best on every query.
✗
✗
✗
✓
the threshold calls the LLM more often than ScaleDoc does, and substantially more often than the target accuracy requires. The perdocument small-LLM scan over the whole corpus adds a moderate cost. What a semantic filter should offer its users. Table 1 states six desired properties and marks which methods deliver each. (1) Prior methods deliver declarative accuracy targets, but none of them lets the user (2) see how close cost is to the optimal performance bound, (3) trust the proxy’s score as reflecting how likely the answer is to be correct, (4) answer queries that go beyond topical relatedness without calling the oracle frequently, (5) get a tight proxy-score threshold that meets target accuracy without being too optimistic (SLA violated) or too conservative (oracle over-called), or (6) deploy a single method that is competitive with the per-query best on every query, without having to know or hand-pick which method is best in advance. To the best of our knowledge, our work is the first to deliver all six.
3
Algorithm 1 A unified cascade framework for semantic filtering. Input: Corpus D, predicate 𝑞, accuracy target 𝛼 Output: Predicted yes/no labels 1: Partition. Group the documents (e.g. by embedding clustering or trivially into one group). 2: Sample. Draw a small labeled set per group. 3: Label. Call the oracle on the sample; collect hard labels 𝑦 and optionally soft labels 𝑝 ★. 4: Build a proxy. One of (a) no model, (b) train an online proxy on the labeled training set, (c) reuse a prebuilt small-LLM proxy. 5: Calibrate. Choose a score threshold 𝜏 (or a decision rule) on a held-out portion of the labeled calibration set so that the SLA with 𝛼 is expected to hold. 6: Deploy. First check if the partition’s deploy condition fails (e.g. CSV’s vote does not agree), recurs to step 1 to re-partition (the dashed back-arrow in Figure 2). Then, for each remaining document, auto-label or cascade it to the oracle based on its proxy score and the calibration. 7: return per-document predictions.
Semantic Filtering and a Unified Framework
This section states the problem (§3.1), cost model (§3.2), and then unifies prior work on a single algorithmic framework (§3.3).
cost has two terms: 𝐶 =
3.1
Problem
A corpus D = {𝑑 1, . . . , 𝑑 𝑁 } and a natural-language predicate or query 𝑞 are given. A designated LLM (the oracle) returns, for any (𝑞, 𝑑𝑖 ), a hard label 𝑦𝑖 ∈ {0, 1} optionally with a class probability 𝑝𝑖★ = 𝑃 (𝑦𝑖 = 𝑦𝑒𝑠) ∈ [0, 1] derived from token logprobs. A semantic filtering method outputs predicted labels 𝑦ˆ1, . . . , 𝑦ˆ𝑁 such that the Í corpus accuracy Acc = 𝑁1 𝑖 1[𝑦ˆ𝑖 = 𝑦𝑖 ] is at least a user-specified target 𝛼 (e.g., 0.9). Throughout the paper we treat the oracle as the ground truth. This is the standard convention in semantic-filter work [20, 48, 49], and any disagreement between the oracle and an underlying notion of “truth” is folded into the irreducible noise we measure in §7.
3.2
Cost Model
We adopt the ScaleDoc-style end-to-end accounting [49] and refine it, assuming the proxy-then-oracle-LLM pipeline. The deployable
𝑇proxy (𝑛 tr, 𝑁 ) | {z }
+ 𝑛 tr + 𝑛 ca + 𝑛 cas 𝑡 LLM . | {z }
proxy train/score time
oracle calls
(1)
Here 𝑡 LLM is the average oracle latency per call (measured on the deployment GPU); 𝑛 tr and 𝑛 ca are the training and calibration set sizes; 𝑛 cas is the cascade count after proxy deployment, calling the oracle for the docs the proxy is uncertain about; and 𝑇proxy (𝑛 tr, 𝑁 ) is the time for proxy training and scoring documents with proxy. The oracle’s soft label [19, 35] is free. Every modern LLM API returns the logprobs of its output tokens, so 𝑝𝑖★ is exposed at no extra cost. Throughout the paper we exploit it as a soft label to train our proxy and compute the BER.
3.3
A Unified Algorithmic Framework
The three families of cascades in §2 share the same six-step skeleton, stated abstractly in Algorithm 1 and drawn as a DAG in Figure 2. Methods differ only in the choices they make for four orthogonal design knobs, summarized in Figure 3.
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
The framework as a DAG. Figure 2 draws Algorithm 1 as a sixnode DAG. The framework makes one structural observation natural: the oracle labels collected in step 3 are the same kind of object regardless of which method collected them, so they can also be used as the training input of a different method’s proxy. The dashed green arrow marks this cross-method join: two families can share a single labeling pass and a simple decision condition (e.g. “once ∼7% of the corpus is labeled and the first method is not confident enough, hand off to the second method”) can switch between them. Our Two-Phase composes CSV and our proxy this way (§6). Four design knobs. The differences between CSV, BARGAIN, ScaleDoc, and our Phase-2 and Two-Phase reduce to four orthogonal choices, displayed in Figure 3. We list its high-level options here; the per-method choices and the details behind them are in §2 and §4–6. • Representation. How does the proxy compute a document’s score? Options include using documents’ dense embeddings without a special ML model, an ML model over query and document, or a small LLM. • Training. Is the score function learned per-query, used as-is from a pre-training, or not learned at all? • Calibration. How is the threshold that meets the accuracy target chosen on the labeled sample? Choices range from a simple vote-agreement threshold set to 𝛼 to high-confidence upper bounds on the proxy’s error rate. • Partition granularity. Are documents grouped by embedding clustering or left as a single group? Our contributions on the matrix. The blue-cell rows in Figure 3 mark the cells unoccupied by prior work. Phase-2 introduces a new proxy in step 4 (cross-encoder (CE) + ColBERT (CB) + hybrid head over them, §4) and a new calibration in step 5 (per-score-bin upper-bound blend, §5); it keeps the trivial single-group partition and the random sample of ScaleDoc. Two-Phase (row 5) uses CSV as the first phase – clustering and per-cluster sample in steps 1–2, with the labels accumulated in step 3 over re-clustering (step 6 to 1). When ∼7% of corpus is labeled in step 3, it reuses the labeled docs to train our proxy (§6).
4
A Better Online Proxy by Revisiting Traditional ML
This section shows that the bottleneck in online-proxy approach is the proxy’s architecture, more than its training (§4.1), and presents our replacement: a token-aware hybrid (§4.2) trained with binary cross-entropy on the oracle’s soft labels and a small SLA-aware primal–dual constraint (§4.3).
4.1
Diagnosis: ScaleDoc’s suboptimal proxy
ScaleDoc’s proxy is a bi-encoder [14, 22, 36]. Query and document each pass through a frozen embedding model, then a 55M-parameter MLP projects each to another dense representation, and the score is the cosine similarity. ScaleDoc invests in a multi-stage contrastive training [9, 16] mechanism on this bi-encoder. However, dense embeddings discard
Conference’17, July 2017, Washington, DC, USA
token-level evidence, and cosine similarity over dense representations approximates topical similarity only, which makes it challenging to support reasoning-intensive predicates. Two documents that are nearly identical topically can have opposite labels two documents far apart in cosine space can both be yes. Contrastive training cannot fix what the representation has discarded.
4.2
Architecture: CE + CB + hybrid head
Before describing the components, we name the ideas we are borrowing and why. Two scoring architectures from neural information retrieval (IR) — the cross-encoder (CE), popularized for passage re-ranking by Nogueira and Cho [26, 32], and ColBERT (CB), introduced by Khattab and Zaharia [23, 37] (Figure 4) — have been the textbook examples of outperforming a bi-encoder on accuracy at IR benchmarks (e.g., MS MARCO [2], TREC passage ranking [13, 41]). A CE captures cross-query-doc interactions via a shared MLP, while CB keeps a separate vector per token on each side and aggregates similarity at the token level (called late interaction). The trade-off is well-known: a bi-encoder reduces each side to one fixed-size vector that is cheap to compare, but the compression is lossy on any predicate whose answer hinges on a specific token (a negation, a named entity, a number); CE captures more complex interactions, and CB recovers that token-level evidence at the cost of higher per-document scoring. Neither CE nor CB have, to our knowledge, been used as the proxy in the LLM-based semantic filtering. Rather than redesigning yet another proxy, we revisit these mature IR ideas in this setting: our proxy runs both CE and CB, and fuses the two scores with a small head. Also in this setting, the bottleneck is mostly on the model training, not scoring, and we can keep this training overhead small by limiting the model size, in total ∼6× smaller than ScaleDoc’s encoder. (1) Cross-encoder (CE) (∼9.5 M). An MLP takes query and document embeddings together and outputs a single score 𝑠 ce . CE captures cross-interaction between query and document unlike the separate MLPs in the bi-encoder. (2) ColBERT-style projection (CB) (∼0.10 M). Late-interaction scoring: project document and query tokens into a shared space and aggregate per-document by max-similarity over tokens (MaxSim). The CB score 𝑠 cb is complementary to CE because it weights individual matching tokens (negation cues, named entities, numbers) rather than the overall fusion. (3) Hybrid head (∼1.3 K). A small MLP on a six-dimensional feature vector 𝑋𝑖 = [𝑠𝑖ce, 𝑠𝑖cb, 𝑠𝑖ce𝑠𝑖cb, |𝑠𝑖ce − 𝑠𝑖cb |, (𝑠𝑖ce ) 2, (𝑠𝑖cb ) 2 ] produces the proxy’s predicted probability 𝑝𝑖 = 𝜎 (MLP(𝑋𝑖 )) for document 𝑑𝑖 . The cascade in §5–§6 thresholds the derived score 𝑠𝑖 = 2 |𝑝𝑖 − 12 | ∈ [0, 1] – large when the proxy is certain (either yes or no), small when the proxy is uncertain.
4.3
Training: soft-label BCE with SLA-aware primal–dual constraint
Our training loss has three terms to answer three questions. Throughout, T is the labeled training sample (∼7% of corpus), and the CE, CB, and hybrid head are trained separately for 60, 15, and 120 epochs each.
Conference’17, July 2017, Washington, DC, USA
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
label reuse across two methods (ours)
autolabel 4. Proxy Corpus D
1. Partition
2. Sample
3. Label
(group docs)
(pick docs to label)
call oracle
trained, prebuilt, or no model
5. Calibrate
6. Deploy
choose threshold 𝜏
score ⋛ 𝜏
cascade to oracle
recursion (CSV)
Figure 2: The unified framework as a DAG. The six steps of Algorithm 1 arranged left to right. Blue boxes are document sets; amber boxes are oracle-labeled subsets; gray boxes are operations; the pink diamond is the deploy-time decision (auto-label vs. cascade) gated by whether the proxy score passes the calibration threshold 𝜏. The dashed black arrow marks a recursion when the deploy condition fails on a partition (e.g. CSV re-clusters a cluster whose vote does not agree strongly enough, cascading individual documents to the oracle only as a last resort). The dashed green arrow is our cross-method join: the same labeled sample drawn for one method can be reused as the labeled sample of a second method (§6). 1. Partition (group docs)
(pick docs to label)
(call oracle)
(trained / prebuilt / none)
4. Proxy
5. Calibrate (choose τ)
(auto vs. cascade)
CSV
k-means on doc embeddings
0.5% per cluster
oracle
no model (majority vote)
vote-agreement threshold τ=α
propagate agreed label or ↺ re-cluster (back to step 1), cascade to oracle as a last resort
BARGAIN
single
calibration set only
oracle
prebuilt small LLM
high-confidence upper bound
auto-label if score ≥ τ cascade otherwise
ScaleDoc
single
7% for training set 5% for calibration set
oracle
bi-encoder
histogram band
auto-label if score ≥ τ cascade otherwise
Phase-2
single
7% for training set 5% for calibration set
oracle
soft-BCE + PD + cov
CE+CB+ hybrid head
per-score-bin CP blend
auto-label if score ≥ τ cascade otherwise
k-means on doc embeddings
0.5% per cluster
oracle
no model (majority vote)
vote-agreement threshold τ=α
propagate agreed label or ↺ re-cluster (back to step 1)
CE+CB+ hybrid head
per-score-bin CP blend
auto-label if score ≥ τ cascade otherwise
(ours)
2. Sample
Two-Phase
3. Label
multi-stage contrastive
6. Deploy
7% accumulated
(ours)
reused
soft-BCE + PD + cov
Figure 3: Method-by-step matrix view. Rows are methods; columns are the six steps. Cell color identifies the family (red = CSV, amber = BARGAIN, teal = ScaleDoc); blue cells are our contributions. CSV’s Deploy cell contains the recursion (⟲) that matches the dashed back-arrow in Figure 2. Two-Phase is shown as two sub-rows: the upper (CSV-style) first phase, the lower (Phase-2-style) second phase; the green arrow is the label reuse in Figure 2 and fires when ∼7% of the corpus has been labeled. (a) Soft-label cross-entropy: how should the proxy’s probability relate to the oracle’s probability? The auto-label vs. cascade calibration in §5 relies on the predicted probability 𝑝𝑖 to decide how likely document 𝑑𝑖 is to satisfy the predicate. Training with hard labels 𝑦𝑖 ∈ {0, 1} forces the proxy toward 𝑝𝑖 ∈ {0, 1} on every training document, including the ones where the oracle itself was unsure: the proxy might become confident on documents the oracle was not, and the calibration cannot tell “proxy is right” apart from “proxy is overconfident.” We instead use the oracle’s per-document probability 𝑝𝑖★ ∈ [0, 1] as the training target: Lsoft =
1 ∑︁ BCE 𝑝𝑖 , 𝑝𝑖★ , |T | 𝑑𝑖 ∈ T
(2)
ˆ 𝑝) = −𝑝 log 𝑝ˆ − (1 − 𝑝) log(1 − 𝑝) ˆ is binary crosswhere BCE(𝑝, entropy with a continuous target. At convergence, 𝑝𝑖 ≈ 𝑝𝑖★ on every training document: the proxy’s predicted probability tracks the oracle’s. On documents the oracle was unsure about, 𝑝𝑖 stays near 1 1 2 , so the derived score 𝑠𝑖 = 2|𝑝𝑖 − 2 | stays small and the calibration cascades them to the oracle. The ablation in §8.7 confirms that this matters at the deployment cost: hard-label BCE on the same architecture regresses by ∼8% on end-to-end cost. (b) SLA-aware constraint: how do we keep the proxy honest at the deployment target? The proxy’s training loss does not know the deployment target accuracy 𝛼. Without a coupling between the two, training can leave the proxy at a calibration that the deploy-time
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
cosine
Conference’17, July 2017, Washington, DC, USA
sum
relevance
MLP
MLP
query embedding
doc embedding
(a) Bi-encoder
MLP query embedding
doc embedding
(b) Cross-encoder
MaxSim MaxSim MaxSim MLP
MLP
query tokens
doc tokens
(c) ColBERT
Figure 4: Three model architectures. (a) Bi-encoder: query and document embeddings are projected by two independent MLPs; the cosine of the projected vectors is the relevance score. (b) Cross-encoder: the two embeddings are read jointly by one MLP that attends across them, producing a single relevance score. (c) ColBERT: query and document tokens are encoded independently by two MLPs into per-token output cells; for each query output cell, MaxSim picks the largest similarity against any document output cell, and the pertoken MaxSim values are summed into the final relevance score. Our proxy combines (b) and (c) through a small hybrid head, which turns the two relevance scores into the predicted probability 𝑝𝑖 (and thus the cascade score 𝑠𝑖 = 2|𝑝𝑖 − 21 |).
to the training loss. The training loop then alternates between two updates: a primal step, in which the proxy weights are moved by one gradient descent step on the loss with 𝜆 fixed; and a dual step, in which 𝜆 itself is updated at the end of each epoch with the proxy fixed. Intuitively, 𝜆 is an adaptive penalty weight – “how strongly does training currently have to push the proxy to keep 𝑅bC at or below 𝜖.” The mechanics are simple. (i) When the constraint is satisfied (𝑅bC ≤ 𝜖), the max(·) part of (4) is zero, so Lsla vanishes from the total loss for that epoch and gradients come only from terms (a) and (c) below; the dual step then decays 𝜆 slightly toward 0. (ii) When the constraint is violated (𝑅bC > 𝜖), the max(·) part becomes positive, and the next primal step moves the proxy toward predictions that lower 𝑅bC – i.e., either making the proxy more confident on documents it actually gets right, or making it less confident on documents it gets wrong; the dual step then raises 𝜆 in proportion to how badly the constraint was violated, so a persistent violation steadily inflates the penalty’s share of the total loss until training restores 𝑅bC ≤ 𝜖. In practice 𝜆 is clipped to [0, 300] and starts near 0, so the penalty contributes to the loss only when the proxy’s auto-accept error genuinely exceeds the user’s target. The ablation in §8.7 shows it is the SLA-stabilizer: removing it drops the SLA hit count from 16/17 to 13/17 on PubMed at only a modest cost increment of ∼4%. (c) Coverage regularizer: how do we stop the constraint from being trivially satisfied? The constraint in (b) is normalized by the proxy’s score 𝑠: if the proxy reports a low score on everything (every 𝑝𝑖 ≈ 12 ), the constraint is met by cascading every document to the LLM. We discourage that trivial corner with a single, lightweight term that pushes the average training-set score up: ∑︁ Lcov = 1 − | T1 | 𝑠𝑖 , (5) 𝑑𝑖 ∈ T
Figure 5: T-SNE visualization on raw embedding space (left) and bi-encoded space (right), with blue dots = yes-docs, orange dots = no-docs, star = query, for a PubMed query over ∼3K sampled docs.
threshold then has to fix by sending more documents to the LLM. We mitigate this by adding a single soft constraint to the training loss, evaluated on a held-out calibration sample C. The constraint asks that the proxy’s expected error on the documents it would auto-accept be at most the user’s error budget 𝜖 = 1 − 𝛼. Recall the score 𝑠𝑖 = 2|𝑝𝑖 − 12 | from the architecture above, and define the score-weighted soft error Í 𝑑𝑖 ∈ C 𝑠𝑖 𝑝𝑖 (1 − 𝑦𝑖 ) + (1 − 𝑝𝑖 )𝑦𝑖 b Í 𝑅C = . (3) 𝑑𝑖 ∈ C 𝑠 𝑖 + 𝜀 The constraint is 𝑅bC ≤ 𝜖. We enforce this constraint softly with a standard primal–dual scheme [4, 6, 12]. The scheme attaches a non-negative weight 𝜆 ≥ 0 – the Lagrange multiplier [4, 6] – to the constraint and adds a single penalty term Lsla = 𝜆 · max 0, 𝑅bC − 𝜖 (4)
The ablation in §8.7 shows this term is the cost-tightener: removing it inflates the end-to-end latency by ∼24% on PubMed. Total loss. The total loss is (𝛽 cov = 0.35): L = Lsoft + 𝛽 cov · Lcov + 𝜆 · max(0, 𝑅bC − 𝜖).
(6)
The three terms in (6) are not all applied to every component of the proxy; CE and CB are trained with (a) only, and the hybrid head is trained with all three, as this is the component that produces the final probability 𝑝𝑖 and therefore the score 𝑠𝑖 the cascade thresholds. This separation matches what each component is responsible for: (b) and (c) shape the score 𝑠𝑖 that determines actual auto-label vs. cascade; the CE and CB backbones learn to align their relevance scores with the oracle’s probability and can even be reused across different SLA targets (which we disable by default).
5
Per-Score-Range Adaptive Calibration
The calibration step turns the proxy’s score into a threshold 𝜏: documents with score above 𝜏 are auto-classified, the rest are sent to the oracle. We argue that prior calibrations leave cost on the table for a single, fixable reason: they treat all documents the same when deciding how much to trust the labeled sample, even though that sample covers some documents much better than others.
Conference’17, July 2017, Washington, DC, USA
5.1
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
Why prior calibrations are conservative
Algorithm 2 Threshold selection.
To decide 𝜏, a calibration estimates the proxy’s error rate as a function of the proxy’s score. The estimation is done on a small labeled sample C. ScaleDoc [49] groups documents in C into 64 score ranges and uses the measured error rate inside each range to decide 𝜏 so that the expected accuracy on auto-accepted documents meets the target 𝛼. This works well on score ranges that contain enough labeled documents: the measured error rate is close to the true error rate on all remaining docs. It fails on score ranges where the labeled sample is sparse: this may set 𝜏 too optimistic (calls the oracle less often than necessary — an SLA violation) or too conservative (calls the expensive oracle more often than needed). The simplest fix is to add a worst-case safety margin to every measured error rate, the way BARGAIN [48] does. It replaces the measurement with a high-confidence upper bound that holds with high probability over the labeled sample. This is safe but expensive. The safety margin is added uniformly, including on score ranges where the labeled sample is enough and the measurement is already reliable. Instead, we add safety margin to score ranges where the sample is sparse, and stay close to zero on score ranges where the sample is enough.
Input: labeled C, pool scores {𝑠𝑖 }𝑖 ∈ R , target accuracy 𝛼, number of score ranges 𝐵, mixing weight 𝜆. Output: score threshold 1 1: for candidate 𝜏 ∈ Quantiles(𝑠 C , 200) ∪ {0, 2 , 1} do 2: Form A C (𝜏), partition into 𝐵 equal-frequency ranges. 3: for 𝑏 = 1, . . . , 𝐵 do 4: Compute 𝑒ˆ𝑏 , CP𝑏 , 𝑢ˆ𝑏 by Eq. (7). 𝑝 5: Count 𝑛𝑏 = |{𝑖 ∈ A R (𝜏) : 𝑠𝑖 ∈ range 𝑏}|. 6: end for c 7: Compute Err(𝜏) by Eq. (8). 8: end for 9: return 𝜏 ★ by Eq. (9).
With 𝑢ˆ𝑏 in hand, the rest of the procedure is the standard one. We project the labeled-set rates onto the remaining unlabeled pool. 𝑝 At threshold 𝜏, let 𝑛𝑏 be the number of pool documents the proxy auto-accepts whose score falls in range 𝑏. The expected number of errors on the auto-accepted pool is c Err(𝜏) =
𝐵 ∑︁ 𝑝 𝑛𝑏 · 𝑢ˆ𝑏 ,
(8)
𝑏=1
5.2
Per-range mix of the empirical rate and a Clopper–Pearson upper bound
and the deployed threshold is the one that calls the oracle as little as possible subject to the accuracy target:
We partition the labeled auto-accept set A C (𝜏) = {𝑑 𝑗 ∈ C : 𝑠 𝑗 ≥ 𝜏 } into 𝐵 = 20 equal-frequency score ranges. Let 𝑛𝑏𝑐 be the number of labeled documents in range 𝑏 and 𝑘𝑏 the number whose proxy prediction disagrees with the oracle. ScaleDoc uses the empirical (measured) error rate 𝑒ˆ𝑏 = 𝑘𝑏 /𝑛𝑏𝑐 . We replace this single number with a combination of 𝑒ˆ𝑏 and a one-sided high-confidence upper bound CP𝑏 on the same quantity (computed by the standard Clopper– Pearson formula [7, 11, 46], which is the standard tool for upperbounding a binomial rate from a small sample; see [1, 3, 18, 43, 47] for related distribution-free conformal calibrations): 𝑢ˆ𝑏 = (1 − 𝜆)𝑒ˆ𝑏 + 𝜆 CP𝑏 ,
𝜆 = 0.06 (default).
(7)
The mixing weight 𝜆 is small on purpose. Here is why 𝑢ˆ𝑏 does what we wanted in the previous subsection. • When the labeled sample is plentiful in range 𝑏 (large 𝑛𝑏𝑐 ), Clopper–Pearson collapses to the measured error rate: CP𝑏 ≈ 𝑒ˆ𝑏 , hence 𝑢ˆ𝑏 ≈ 𝑒ˆ𝑏 . • When the labeled sample is sparse in range 𝑏 (small 𝑛𝑏𝑐 ), Clopper– Pearson widens, so CP𝑏 ≫ 𝑒ˆ𝑏 , and 𝑢ˆ𝑏 inflates above 𝑒ˆ𝑏 . The threshold becomes less willing to auto-accept documents in that range. In summary, 𝑢ˆ𝑏 tracks the measured rate on well-covered ranges and inflates only where the measured rate is not reliable.
c 𝜏 ★ = arg min{|R \ A R (𝜏)| : 1 − Err(𝜏)/𝑁 ≥ 𝛼 }. Algorithm 2 states the procedure.
5.4
Threshold selection
We write R for the deployment pool (the unlabeled documents to classify) and C ⊂ R for the small labeled calibration sample. For any candidate threshold 𝜏, A R (𝜏) = {𝑑𝑖 ∈ R : 𝑠𝑖 ≥ 𝜏 } is the subset of R that the proxy auto-accepts. We reuse the score ranges from §5.2: range 𝑏 (𝑏 = 1, . . . , 𝐵) is the 𝑏-th bucket of the calibration scores {𝑠 𝑗 } 𝑗 ∈ C .
Why is it tighter?
At a quick read, replacing 𝑒ˆ𝑏 with 𝑢ˆ𝑏 ≥ 𝑒ˆ𝑏 looks more conservative than ScaleDoc, not less. The key point is that ScaleDoc’s calibration is not just the bare empirical rate either: it smooths the per-bin counts before selecting the threshold [49]. The smoothing is a deliberate safety choice – it prevents an optimistic threshold on ranges that the labeled sample covers only sparsely – but it increases the per-bin error estimate uniformly across all 𝐵 ranges, including the ones the labeled sample already covers densely. In effect, it pays a (small) safety surcharge on every range, whether the measurement on that range needs it or not. In contrast, we add safety margin only where it is needed.
5.5
Assumptions
The calibration assumes (a) within each score range, the per-document error is an independent Bernoulli draw, which is what the Clopper– Pearson bound [11] needs; and (b) the labeled sample C is drawn from the same distribution as the deployment pool.
6 5.3
(9)
Adaptive Two-Phase: Model-Free Then Proxy
Our proxy in §4 with the calibration in §5 is the fastest plan on noneasy predicates, but not on every query: on queries where dense embeddings already group documents the way the predicate does, CSV is able to classify all documents confidently without many oracle calls. However, neither method is the fastest on every case. We propose Two-Phase that composes the two so that each query is answered by whichever phase is enough.
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
6.1
Why a composition, not a choice
On a typical evaluation, no single method is the per-query winner across the whole query mix (§8.5). CSV is the fastest when the predicate aligns with an embedding clustering — for example, on PubMedQ1 (“pediatric participants under 18”), a few clusters of pediatric documents are nearly all positive and the rest are nearly all negative, so a small per-cluster sample is enough to label the entire cluster. Our proxy is the fastest when the predicate cuts across the clustering — for example, on PubMedQ19 (“methodological detail to assess internal validity”), positives and negatives are mixed inside every cluster, so the per-cluster vote never agrees, the algorithm keeps splitting clusters and re-sampling until it runs out of labeling budget, and most documents end up cascaded to the oracle anyway. The composition idea. A user does not know in advance which method is faster in the end. What we do know is each method’s predetermined must-pay labeling cost – the cost the method pays before it can produce any prediction at all, regardless of how easy or hard the query turns out to be. Our proxy must label a fixed ∼7% of the corpus for training plus ∼5% for calibration before it can be deployed; CSV must label only its first per-cluster sample (∼0.5% of the corpus per cluster), and pays more only when a cluster does not agree and has to be split. On a query where CSV’s clusters agree quickly, its must-pay cost is well below the proxy’s; on a query where every cluster is mixed, the per-cluster samples accumulate linearly with the corpus until CSV catches up with – and typically overshoots – the proxy’s 12% budget. We use this ordering to compose the two: run CSV first (its must-pay cost is smaller), and escalate to the proxy phase only when the samples accumulate enough. The Phase-1 labels are reused as the Phase-2 training set, so the escalation adds no extra oracle calls for training – only the calibration set is sampled afresh.
6.2
Procedure: model-free first, proxy only when needed
The labeled sets. The deployment pool R is the full corpus the filter must classify (§5), and C ⊂ R is the labeled calibration sample; T ⊂ R is for the labeled training sample used by the proxy. T and C are sampled disjointly, but both are subsets of the same R. Phase 1: CSV. Cluster the 𝑁 documents into 𝑘 = 4 groups on dense embeddings. In each round, for every cluster: (1) Draw a sample of size max(⌈0.005 𝑁 ⌉, 100); (2) Call the oracle on the sample, collecting 𝑦 and 𝑝 ★; (3) Vote: if the agreement on the sample is at least the vote threshold 𝜌 vote , propagate the majority label to the entire cluster; (4) Otherwise, split the cluster into two via 𝑘-means and revisit it in the next round. Phase 1 stops when the cumulative labeled fraction reaches 𝜆p1 = 0.07 of the corpus. The vote threshold is set to the user’s deployment target, 𝜌 vote = 𝛼. This couples the two phases through a single user knob. A higher target accuracy 𝛼 makes Phase 1 more conservative about declaring a cluster homogeneous (and so more aggressive about reclustering), while the same 𝛼 tightens the Phase-2 calibration in §5.
Conference’17, July 2017, Washington, DC, USA
Early exit. If at the end of Phase 1 every cluster has agreed (which may happen before the labeling budget 𝜆p1 is reached), the predictions are already known and Phase 2 is skipped. The only oracle cost is the Phase-1 sample. Phase 2 with reused training labels. If at least one cluster remains mixed at the end of Phase 1, we escalate. The Phase-1 labels become the Phase-2 training set: every document the oracle labeled during Phase 1 (𝜆p1 of the corpus) goes into T , with no extra oracle calls. The calibration set C (0.05 of the corpus) is sampled separately from the pool minus T , by stratified sampling on the proxy score, and does require its own oracle calls. (1) Train CE, CB, and the hybrid head on T (and C for the primal– dual constraint). (2) Score all 𝑁 documents with the proxy (including documents in agreed Phase-1 clusters — Phase 2 does not trust propagated labels since the predicate is now determined non-easy). (3) Compute the score 𝑠𝑖 for every document in the remaining pool R, run Phase-2 (Algorithm 2) to select 𝜏, and cascade documents with 𝑠𝑖 < 𝜏 to the oracle.
6.3
Two design choices
The two non-obvious knobs in the procedure are the vote threshold 𝜌 vote that gates Phase-1 propagation, and the decision not to sample the Phase-2 calibration set C from the documents Phase-1 has already labeled. Vote threshold = user target. The original CSV paper [20] does not say how to set 𝜌 vote from a user accuracy target; if anything, it explains the opposite – the vote threshold and the realized accuracy are not directly coupled, and tuning them by hand is part of the cost of using CSV. We sidestep that tuning by taking 𝜌 vote = 𝛼. The reading is simple: if Phase-1 propagates a cluster’s majority label only when the sample agrees on at least an 𝛼 fraction of its documents, then – in expectation – the propagated label is wrong on at most 1 − 𝛼 fraction of the cluster, i.e., the per-cluster expected error rate is at most the user’s error budget. Coupling the two settings through one knob also makes the two phases pull in the same direction: a stricter target accuracy makes Phase-1 more conservative about declaring a cluster homogeneous (and so more aggressive about reclustering), and the same 𝛼 tightens Phase 2. Calibration set is sampled separately from Phase-1. A naturallooking optimization is to run Phase-1 a little longer and reuse those extra labels as the calibration set C, so that the transition between phases costs zero oracle calls. However, it violates the Clopper–Pearson assumption in §5.5 that C is drawn from the same distribution as the deployment pool. Phase-1’s sampling is anything but uniform: it concentrates on whichever clusters turned out to be mixed first, so a C taken from Phase-1’s labels is a biased view of the remaining pool, the per-bin Clopper–Pearson bound is no longer a valid upper bound on the pool’s error rate, and the safety guarantee from §5 no longer holds. We pay the cost of sampling C separately (stratified on the proxy score over the pool minus T ) so that the calibration set is representative of the documents the threshold will actually act on.
Conference’17, July 2017, Washington, DC, USA
7
BER as Difficulty Compass and Lower Bound
We propose two uses of the oracle’s own per-document confidence. (i) A difficulty compass. The same per-document confidence that we use as a soft training label in §4 is also a method-independent measure of how hard a query is: it predicts, without running any method, which one will be fastest on that query. (ii) A lower bound on cascade cost. The same per-document confidence places a hard floor on how few documents any cascade plan can send to the oracle while still meeting the accuracy target, regardless of how smart the proxy is. We use the floor as a benchmark for headroom and as a research target for future work; we do not claim it is attainable.
7.1
Bayes error [15, 33]: the oracle’s own uncertainty
For document 𝑑𝑖 , let 𝑝𝑖★ ∈ [0, 1] be the oracle’s predicted probability that the document satisfies the predicate. The per-document Bayes error is 𝜂𝑖 = min(𝑝𝑖★, 1 − 𝑝𝑖★): the probability that the oracle’s own hard label is wrong if we were to re-sample it from the oracle. A document the oracle is sure about (𝑝𝑖★ close to 0 or 1) has 𝜂𝑖 close to 0; a document the oracle is unsure about (𝑝𝑖★ close to 12 ) has 𝜂𝑖 close to 12 . So the ceiling is 12 . Í The query BER BER𝑞 = 𝑁1 𝑖 𝜂𝑖 averages this over the corpus. It is a property of the query and the oracle; it does not depend on which proxy or which calibration the user picks. It is also free to measure: the oracle returns 𝑝𝑖★ together with the hard label on every call, so BER𝑞 can be estimated from any pilot sample at no extra oracle cost.
7.2
Why query BER predicts which family wins
The two prior families behave very differently as BER𝑞 grows. CSV labels a small per-cluster sample and propagates the majority label to the whole cluster. When BER𝑞 is small, most documents are confidently in one class, so most clusters are homogeneous and propagation gets the answer for free. As BER𝑞 grows, more documents fall near the predicate boundary, more clusters become mixed, and CSV must split them and label larger samples; the cost grows quickly. An online proxy (ScaleDoc or our Phase-2) pays a fixed training and calibration cost regardless of BER𝑞 , and a deployment cost that grows much more slowly with BER𝑞 because the proxy can resolve documents the cluster vote could not. The two cost curves cross at a query-specific BER (§8.6): below the crossing, CSV is the fastest plan; above it, our proxy is the fastest. Two-Phase tracks the faster of the two on each query by running CSV first and escalating only when CSV is not enough.
7.3
A lower bound on deployed cascade calls
The Bayes error 𝜂𝑖 is a lower bound on the probability that any classifier, on any representation, makes a mistake on 𝑑𝑖 [15], because the oracle is the most accurate label we have. Suppose a cascade plan auto-classifies a subset A of the pool and sends the rest to the oracle. The expected number of auto-classification errors is at Í least 𝑑𝑖 ∈ A 𝜂𝑖 . To meet the corpus error budget (1 − 𝛼)𝑁 , the plan must keep this sum below the budget, which means it can only auto-classify documents whose 𝜂𝑖 values are small enough to fit inside the budget when summed.
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
BER-LB. The cheapest set A that fits inside the budget is found by sorting documents in increasing order of 𝜂𝑖 and including them one by one until the budget is full: Definition 1 (BER-LB). Sort documents by ascending Bayes error Í 𝜂 (1) ≤ 𝜂 (2) ≤ · · · ≤ 𝜂 (𝑁 ) , and let 𝑘 ★ be the largest 𝑘 with 𝑘𝑖=1 𝜂 (𝑖 ) ≤ (1 − 𝛼)𝑁 . Then the minimum number of cascade oracle calls any cascade plan can make at the target accuracy 𝛼 is at least 𝑁 − 𝑘 ★. BER-LB does not count the oracle calls needed to learn the perdocument 𝜂𝑖 (it assumes them as given), so it is a strict lower bound on the cost of deployed cascade calls, not a target a real method can reach. We use it to quantify headroom: the gap between the cheapest deployable plan and BER-LB tells us how much further any future cascade method could possibly cut the cost.
7.4
Headroom
Our deployable methods sit closer to BER-LB than the baselines, but the floor itself is still ∼4–20× below our cost on the three corpora (§8.2).Two costs explain the remaining gap: (i) the oracle calls needed to learn the per-document 𝜂𝑖 values, which BER-LB does not pay; and (ii) the safety margins needed to certify the accuracy target under a finite labeled sample. Closing the gap is a concrete research target for future cascade plans. Summary. Oracle Bayes error gives the user, for free, both a prediction of which cascade family will be cheapest on a query and a hard lower bound on the cost of any cascade family on that query. Neither use has been isolated in prior semantic-filter work; we use both inside Two-Phase and as headroom in the evaluation.
8
Experiments
This section asks whether the four contributions of §1 other than the unified framework (C1)—the revisited online proxy (C2, §4), the tighter calibration (C3, §5), the adaptive two-phase method (C4, §6), and the BER compass and lower bound (C5, §7)—translate into measurable wins for the semantic-filter operator: lower end-to-end latency at the user’s accuracy target SLA. We organize the evaluation around six research questions. RQ1– RQ3 establish empirical efficacy at the user’s target accuracy, under a sweep of that target, and at a per-segment level of cost breakdown; RQ4–RQ5 explain which queries our methods win and why; RQ6 isolates which ingredient of the proxy and calibration contributions binds via controlled ablations. • RQ1 (efficiency at the SLA). At target accuracy 𝛼 = 0.9, how much do Phase-2 and Two-Phase reduce mean end-toend latency and oracle-call count relative to ScaleDoc/ CSV/ BARGAIN, and how close do they get to the BER-LB lower bound? • RQ2 (target-accuracy robustness). As the user tightens 𝛼 from 0.70 to 0.95, do our methods’ cost advantages widen, hold, or collapse? • RQ3 (cost decomposition). On a per-query basis, where does the time go—proxy training/scoring, sample labeling for voting, training set labeling, calibration set labeling, or oracle cascade?
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
• RQ4 (per-query competitiveness). The mean numbers hide which plan is the fastest per query. Does a single method track the per-query lower envelope? • RQ5 (BER as compass). Does the query’s Bayes error rate BER𝑞 , computed from the oracle’s free logprobs, predict which family wins on that query? • RQ6 (ablations). Inside the proxy, is the binding ingredient the architecture, the training loss, the primal–dual update, or the coverage regularizer? Inside the calibration, where do our single-knob per-bin blend, ScaleDoc’s histogram band, and a naive empirical calibration sit relative to the non-deployable omniscient bound that knows every label in advance?
8.1
Setup
Datasets and queries. We use the three corpora used in [49], each with 𝑁 ≈ 10 000 documents, along with 20 queries we generated: PubMed (biomedical abstracts, mean 510 prompt tokens), BigPatent (patent specifications, 233 tokens) and GovReport (government reports, 718 tokens). Document features are NV-Embed embeddings (4096-D) [25, 31, 44] shared across methods. The oracle is Llama-3.1-70B-Instruct [40, 42] run on two A100 80GB GPUs. Methods compared. • CSV [20]: 𝑘-means 𝑘 = 4, sample max(⌈0.005 𝑁 ⌉, 100) documents per cluster, vote threshold 𝜌 vote = 𝛼, same as described in §6.2. • BARGAIN [48]: Llama-3.1-8B-Instruct is used as a prebuilt proxy. • ScaleDoc [49]: bi-encoder over frozen NV-Embed trained with multi-stage contrastive learning, 7% train / 5% calibration split. • Phase-2 (ours): Our proxy in §4 and calibration in §5. Same 7%/5% splits as ScaleDoc. • Two-Phase (ours): CSV → Phase-2, with Phase-1 labels reused for Phase-2 training (§6). • BER-LB (Def. 1): non-deployable lower bound based on the oracle’s per-document Bayes errors. Primary metrics. We use four metrics: (i) end-to-end latency, (ii) oracle-call count (the dominant cost), (iii) SLA hit count (number of queries whose realized corpus accuracy Acc𝑞 meets the target Í 𝛼), and (iv) SLA-violation magnitude ( 𝑞 max(0, 𝛼 − Acc𝑞 ) summed over queries; lower is better, 0 means no query violated the SLA). The hit count and the violation magnitude together distinguish a calibration, e.g., that misses on 4/20 queries by a hair from one that misses on 1/20 by a wide margin. Hyperparameters. Unless otherwise stated, Phase-2 uses 𝐵 = 20 bins, 𝜆cp = 0.06 as the weight for the CP blend in our calibration. Two-Phase uses vote threshold 𝜌 vote = 𝛼 as CSV and training sample ratio 𝜆p1 = 0.07 as ScaleDoc; the proxy trains the CE for 60 epochs, CB for 15, hybrid head for 120.
8.2
RQ1: Efficiency at the SLA
Table 2 summarizes mean end-to-end latency, mean oracle-call count, SLA hit count, and SLA-violation magnitude on three corpora
Conference’17, July 2017, Washington, DC, USA
at the standard target 𝛼 = 0.9. The non-deployable BER-LB row quantifies remaining headroom. Three findings stand out. Phase-2 cuts E2E and oracle calls by roughly 2×. On PubMed Phase-2 is 2.03× faster than ScaleDoc (264.6 s vs. 536.8 s); on GovReport it is 1.85× faster (412.7 vs. 765.0); on BigPatent it is 1.87× faster (116.1 vs. 217.3). The reduction is driven almost entirely by the cascade segment of the cost: the mean oracle-call count falls from 3,468 to 1,890 on PubMed (a ∼45% cut), from 3,694 to 2,141 on GovReport (∼42%), and from 2,292 to 1,672 on BigPatent (∼27%). RQ3 traces this reduction to the per-query breakdown. Two-Phase is the strongest SLA-compliant plan. Two-Phase pays a small E2E increment over Phase-2 on PubMed (8.6%) and is in fact faster than Phase-2 on GovReport (4.0%) and BigPatent (9.4%). In exchange, Two-Phase delivers 19/20 SLA hits on every corpus, which is the highest among the deployable plans on BigPatent and ties or trails on the others. The SLA-violation magnitude tells the more important story. Two-Phase’s violation is at most 0.008 on every corpus (0.0071, 0.0051, 0.0081), roughly 7× smaller than Phase2’s 0.0551 on PubMed and below the 0.012–0.023 range of ScaleDoc. The efficiency of Two-Phase comes from bypassing Phase 2 on queries that Phase 1 alone resolves – 3/20 on PubMed, 0/20 on GovReport, 5/20 on BigPatent. The improved SLA hit count comes from setting the Phase-1 vote threshold to the user target 𝛼 (§6.3), which keeps Phase-1 propagation conservative. Headroom: the BER-LB floor is still 4–20× ahead. The BER-LB row of Table 2 reports the minimum oracle-call count and the corresponding E2E that any proxy-based plan could attain at 𝛼 = 0.9 (Def. 1). The deployable-to-floor gap is ∼ 20.0× on PubMed (287.4 s vs. 14.4 s), ∼ 7.5× on GovReport (396.1 vs. 53.1), and ∼ 3.9× on BigPatent (105.2 vs. 26.7). The gap is largest where the BER distribution is most skewed toward zero (PubMed) and smallest where CSV’s cluster-vote already exploits much of the easy regime (BigPatent), consistent with the floor analysis of §7.
8.3
RQ2: Target-Accuracy Robustness
The 𝛼 = 0.9 comparison of RQ1 is a single slice of a curve. A calibration that wins at 𝛼 = 0.9 can lose at 𝛼 = 0.95 if its bound is loose on small samples; a method that wins at 𝛼 =0.85 can become dominated when the target tightens. RQ2 sweeps 𝛼 ∈ [0.70, 0.95] and asks whether our advantage widens, holds, or collapses as the target tightens (Fig. 6). The pattern is consistent across corpora. Two-Phase and Phase-2 form the cheapest, steepest pair at every 𝛼 on every corpus: they are furthest to the left (cheapest at a given 𝛼) and most vertical (cheapest delta as 𝛼 tightens), because the Phase-2 bound only inflates the grey zone where the calibration sample is sparse. ScaleDoc and BARGAIN trail by a roughly constant horizontal offset across the entire 𝛼 range, indicating that their calibrations do not adapt to the tightening budget. CSV is competitive at low 𝛼 where Phase1 alone suffices, then degrades sharply as 𝛼 tightens because the cluster-vote residual stops fitting the budget. At 𝛼 = 0.95 all perquery proxy methods converge toward a common wall: the global BER floor of Def. 1 leaves no headroom on the high-BER queries, an empirical manifestation of the BER-LB lower bound.
Conference’17, July 2017, Washington, DC, USA
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
PubMed
GovReport
BigPatent
Method
E2E (s) Oracle acc≥ 0.9 SLA viol. E2E (s) Oracle acc≥ 0.9 SLA viol. E2E (s) Oracle acc≥ 0.9 SLA viol.
CSV [20] BARGAIN [48] ScaleDoc [49] Phase-2 (ours) Two-Phase (ours)
914.8 579.8 536.8 264.6 287.4
6933 3144 3468 1890 2080
19/20 20/20 17/20 16/20 19/20
0.0039 0.0000 0.0231 0.0551 0.0071
1250.6 639.4 765.0 412.7 396.1
6737 2194 3694 2141 2052
19/20 20/20 19/20 18/20 19/20
0.0107 0.0000 0.0123 0.0065 0.0051
255.5 209.1 217.3 116.1 105.2
4239 2218 2292 1672 1555
20/20 17/20 17/20 18/20 19/20
0.0000 0.0635 0.0192 0.0171 0.0081
BER-LB (Def. 1)
14.4
109
20/20
0.0000
53.1
286
20/20
0.0000
26.7
444
20/20
0.0000
Table 2: End-to-end comparison at target accuracy 𝛼 =0.9 (SLA). Per dataset: mean end-to-end latency (s), mean total oracle-LLM Í call count per query, number of queries (out of 20) meeting the SLA, and the SLA-violation magnitude 𝑞 max(0, 𝛼 − Acc𝑞 ) (lower is better; 0 means no query violated the SLA). BER-LB is the non-deployable greedy lower bound of Def. 1.
8.4
(a) PubMed.
(b) GovReport.
8.5
(c) BigPatent.
Figure 6: Target accuracy vs. end-to-end cost. Curves further to the left are cheaper at a given target; curves steeper upward hold their cost as the target tightens. Two-Phase and Phase-2 form the cheapest, steepest pair; ScaleDoc and BARGAIN trail by a roughly constant offset; CSV is competitive only at low 𝛼. The dashed BER-LB curve is the non-deployable greedy lower bound (Def. 1) at each 𝛼.
RQ3: Per-Query Cost Decomposition
RQ1 and RQ2 reported aggregate cost; RQ3 asks where the time actually goes on each individual query. Fig. 7 decomposes the perquery latency at 𝛼 =0.9 into the five cost segments of the unified template: proxy training/scoring, Phase-1 sample labeling, trainingset labeling, calibration-set labeling, and cascade. Three patterns hold across all 60 query slots. (i) Cascade dominates. The cross-hatched cascade segment is by far the largest contributor on every method that deploys a proxy. Improvements in mean E2E come almost entirely from shrinking this segment; proxy training time is a flat ∼15 s for Phase-2 and TwoPhase, regardless of corpus or query. (ii) Phase-2 and Two-Phase shrink the cascade consistently. The cascade reduction is visible per query, not just on average, which rules out the possibility that the mean gain is driven by a few easy outliers. (iii) Two-Phase additionally removes the training labeling segment on Phase-1-resolved queries. The bars for short-circuit queries (left side of each panel) show only the Phase-1 sample bar; the train and calibration bars vanish because Phase-2 is bypassed. Absolute scale grows with prompt length (GovReport’s prompts are ∼3× longer than BigPatent’s), but the relative ordering of methods is unchanged across panels.
RQ4: Per-Query Competitiveness
The aggregates of Table 2 hide that the cheapest plan changes per query. A plan that is 2× better on average but 5× worse on a small fraction of queries can still hurt the user. The right diagnostic is a lower-envelope view: per query, take the cheapest deployable plan; sort the queries by this per-query minimum; then plot each method’s per-query cost against the sorted axis. A method that “tracks the envelope” across the whole axis is the plan a singlesystem optimizer should reach for as a default. Fig. 8 confirms the prediction on PubMed. Two-Phase tracks the envelope across all 20 queries, being competitive to CSV on the Phase-1-resolved low-BER queries and competitive to Phase-2 on the rest. CSV is the per-query winner where it wins, but its loss on the right half of the axis (5–25× over the envelope) is what a deployer pays for picking it as a single-system default. ScaleDoc and BARGAIN sit far above the envelope almost everywhere, consistent with the mean numbers of RQ1. The picture is qualitatively the same on GovReport and BigPatent.
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
Conference’17, July 2017, Washington, DC, USA
(a) PubMed.
(b) GovReport.
(c) BigPatent.
Figure 7: Per-query end-to-end time breakdown at 𝛼 =0.9. Bars stack proxy time and oracle-LLM time across five segments: proxy train/score, Phase-1 sample labeling, training-set labeling, calibration labeling, and cascade. The cross-hatched cascade segment dominates everywhere; Phase-2 and Two-Phase shrink it on essentially every query; Two-Phase additionally removes the training-labeling segment on Phase-1-resolved queries. Right-axis markers indicate per-query accuracy: ◦ for Acc𝑞 ≥ 0.9, × for Acc𝑞 < 0.9.
8.6
RQ5: BER as a Compass
Section 7.2 predicts that the per-query ordering of methods seen in Fig. 8 is explained by the query’s Bayes error rate BER𝑞 computed from the oracle’s free logprobs. We test this directly. For every query we record (BER𝑞 , winner), where the winner is the deployable method with the smallest E2E, and fit a logistic regression of 𝑃 (𝐶𝑆𝑉 wins | BER𝑞 ). The crossover BER (where the fit crosses 0.5)
is the BER above which our proxy family takes over; AUC quantifies how cleanly BER𝑞 alone separates the regimes. Two observations: (i) AUC ≥ 0.81 on every corpus. BER𝑞 alone is a strong predictor of whether CSV is the cheapest plan. The remaining gap to 1.0 comes from a handful of “CSV beats its compass” edge cases (PubMedQ11 at BER = 0.090 and BigPatent Q14 at 0.110) where clustering happens to align with the predicate despite high oracle noise. (ii) The crossover BER is consistent within 2× across corpora (0.020 on PubMed, 0.040 on GovReport and BigPatent).
Conference’17, July 2017, Washington, DC, USA
Figure 8: Per-query lower-envelope on PubMed at 𝛼 = 0.9. Queries are sorted along the 𝑥-axis by the per-query cheapest deployable plan (the dashed envelope); 𝑦-axis is end-to-end time. Two-Phase (gray) tracks the envelope across the whole axis; CSV (red) wins outright on the leftmost ∼6 queries and then explodes by 5–25× on the rest; ScaleDoc (teal) and BARGAIN (orange) sit far above the envelope almost everywhere.
Figure 9: BER as a compass for which family wins, PubMed. Each point is one of the 20 queries plotted at its query BER BER𝑞 ; the point’s color marks the deployable method that achieved the smallest E2E on that query. The S-curve is a logistic fit of 𝑃 (𝐶𝑆𝑉 wins | BER𝑞 ); the dashed vertical line is the crossover BER at which the fit equals 0.5. The fitted crossovers are 0.020 on PubMed, 0.040 on GovReport, 0.040 on BigPatent; AUC of the BER-only predictor is 0.881, 0.810, 0.859 respectively.
The crossover is not a corpus-specific artifact; it is a property of the operator. It also justifies the design of Two-Phase: the Phase-1 cluster-vote agreement is itself a cheap, in-pipeline signal that the query is in the low-BER regime where CSV is the cheaper plan with enough confidence: a query whose clusters all agree at the user target 𝛼 is effectively a low-BER query and the rest of the budget can be saved by skipping Phase 2. Two-Phase therefore does not need to estimate BER𝑞 or consult any router; the Phase-1 step is the per-query plan selector.
8.7
RQ6: Ingredient Ablations
RQ6 isolates which ingredient binds inside the two ablatable contributions of our method: the proxy (RQ6a, four ingredients) and the calibration (RQ6b, three ingredients). All ablations are run on PubMed; both tables restrict to the 17/20 queries where Two-Phase fires Phase 2 so that all rows share the same train/cal split. Each table holds one contribution fixed and varies the other.
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
RQ6a: Proxy ingredients. The proxy contribution has four ingredients: the architecture (CE+CB+hybrid head vs. ScaleDoc’s biencoder), the training loss on CE+CB (soft-label BCE vs. hard-label BCE vs. contrastive), the primal–dual (PD) update that enforces the SLA constraint during training of the hybrid head, and the coverage penalty (cov) that pushes the selection gate to cover as many documents as possible. Table 3 sweeps each ingredient with the calibration held fixed at the full Phase-2. The ablation shows that the four ingredients have different binding strengths. (i) The architecture is the dominant ingredient. Holding training fixed at soft-BCE + PD + cov, swapping our CE+CB+hybrid for a bi-encoder over frozen embeddings regresses the E2E from 328.7 s to 415.4 s and drops the SLA hits from 16/17 to 13/17. The architecture is the binding constraint, as predicted by §4.1: cosine over dense embeddings fuses token-level evidence away, and any training scheme we evaluate cannot recover what the representation has discarded. (ii) The training loss for the backbones matters at the margin. Holding the architecture at our CE+CB+hybrid, swapping soft-BCE for contrastive on CE/CB is within noise on E2E (329.6 s vs. 328.7 s) but drops 2 SLA hits; swapping for hard-BCE regresses E2E by ∼8% and drops 4 SLA hits because the proxy becomes over-confident near the cascade boundary. (iii) PD and cov each contribute, in different ways. Removing the coverage regularizer inflates the cascade to 406.4 s but holds 15/17 SLA hits; removing PD gives a tighter 340.5 s but drops to 13/17 SLA hits. PD is the SLA-stabilizer; cov is the cost-tightener. (iv) Reference. ScaleDoc’s bi-encoder + contrastive without PD/cov sits at 543.1 s and 13/17, which is the gap we attribute to the combined proxy contribution. RQ6b: Calibration. The calibration step holds the proxy fixed at our full CE+CB+hybrid + soft-BCE + PD + cov and asks which procedure turns the proxy’s score into a deployable SLA. Table 4 compares four calibrations: Naive empirical (the bare empirical error rate on C with no conservative refinement); ScaleDoc’s smoothed histogram band; ours (the single-knob per-bin Clopper–Pearson blend of §5.2); and a non-deployable omniscient bound that assumes the proxy’s score and every document’s true label are known in advance and picks the threshold that minimises cascade calls subject to the SLA. The omniscient bound is the smallest cascade cost any calibration could possibly achieve at this proxy and this target; it is a within-proxy analogue of the BER-LB lower bound, restricted to the calibration. The four ablated calibrations split cleanly along the deployable / non-deployable axis and along the SLA-safe / SLA-violating axis. (i) The omniscient bound sets the floor at 297.2 s. This is the smallest cascade cost any calibration could possibly achieve on this proxy at 𝛼 = 0.9. Any deployable calibration must sit above it. (ii) The naive empirical calibration dips below the floor at 295.3 s, but only by violating the SLA. It meets the target on only 12/17 queries and incurs the largest violation magnitude (0.0602) in the table. The 295.3 < 297.2 gap is exactly the cost of those violations: the calibration saves cascade calls on queries it should have cascaded. (iii) ScaleDoc’s smoothed band pays the opposite cost. It meets the SLA on all 17/17 queries but at 363.2 s, ∼22% above the floor, because its uniform smoothing inflates the per-bin error estimate on every score range – including the well-covered ones whose empirical rate is already reliable – and the threshold cascades more documents
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
Architecture
Training
Conference’17, July 2017, Washington, DC, USA
E2E (s)
acc≥ 0.9
Architecture sweep (training fixed at ours) CE+CB+hybrid (ours) soft-BCE + PD + cov Bi-encoder (ScaleDoc) soft-BCE
328.7 415.4
16/17 13/17
Training loss for CE/CB CE+CB+hybrid contrastive + PD + cov CE+CB+hybrid hard-BCE + PD + cov
329.6 355.8
14/17 12/17
Training loss for Hybrid CE+CB+hybrid soft-BCE + PD CE+CB+hybrid soft-BCE + cov
406.4 340.5
15/17 13/17
Baseline reference Bi-encoder (ScaleDoc)
543.1
13/17
contrastive
Table 3: Proxy ablation on PubMed. The 3 Phase-1-only queries are removed, so all rows are reported on the same 17 queries; all proxies are trained on the same Phase-1 labeled documents. PD: primal–dual update on C; cov: coverage regularizer on T . Calibration is fixed to our full per-bin Clopper– Pearson blend throughout.
Calibration
E2E (s)
mean acc
min acc
acc≥ 0.9
SLA viol.
Ours ScaleDoc Naive empirical Omniscient bound
328.7 363.2 295.3 297.2
0.919 0.931 0.910 0.910
0.893 0.916 0.876 0.900
16/17 17/17 12/17 17/17
0.0071 0.0000 0.0602 0.0000
Table 4: Calibration ablation on PubMed. Proxy held fixed at our CE+CB+hybrid with soft-BCE, primal–dual, and coverage regularizer. Naive empirical: bare per-bin error rate on C, no safety margin. ScaleDoc: 64-bin histogram band with smoothed yes/no counts. Ours: single-knob per-bin Clopper– Pearson blend of §5.2. Omniscient bound: non-deployable; assumes every document’s true label is known and picks the threshold that minimizes cascade calls subject to the SLA.
than the target actually requires (cf. §5.4). (iv) Our calibration sits within ∼10% of the floor. 328.7 s (+10.6% over 297.2) with 16/17 SLA hits and violation magnitude 0.0071: one missed query by a hair. The single-knob per-bin blend (§5.2) is the only deployable calibration in the table that sits close to the within-proxy floor on both axes simultaneously – E2E cost (column 1) and SLA hit count (column 4).
9
Conclusion
This paper optimizes the semantic filtering operator, unifies existing cascade plans into a single six-step framework with four design knobs, and shows that prior cascades have been optimizing the wrong knobs. Within the framework, we propose a token-aware hybrid proxy, trained for the first time with the oracle’s per-document confidence as a soft label, so the proxy’s probability tracks the oracle’s at the deployment target; a single-knob blend of the per-bin
empirical rate with a Clopper–Pearson upper bound, increasing the safety margin only on the sparse score ranges that actually need it; an adaptive two-phase model-free and online-proxy composition that reuses the Phase-1 labels as the Phase-2 training set to reduce the transition cost; and the first use of the oracle’s own perdocument Bayes Error Rate as both a per-query difficulty compass and a non-deployable lower bound on the cost any proxy-based cascade can achieve. The combined plan is 1.6–2.0× faster than the previous deployable state of the art on every corpus and within ∼10% of a non-deployable omniscient calibration that knows every label in advance. Further remaining ∼4–20× gap to the strict performance upper bound could be achieved with a better proxy and probably through adaptive per-query labeling budgets for training and calibration.
References [1] Anastasios N. Angelopoulos and Stephen Bates. 2021. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. CoRR abs/2107.07511 (2021). arXiv:2107.07511 https://arxiv.org/abs/2107.07511 [2] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. CoRR abs/1611.09268 (2016). [3] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. 2021. Distribution-free, Risk-controlling Prediction Sets. J. ACM 68, 6 (2021), 43:1–43:34. doi:10.1145/3478535 [4] Dimitri P. Bertsekas. 1999. Nonlinear Programming (2nd ed.). Athena Scientific, Belmont, MA. [5] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. 2017. Adaptive Neural Networks for Efficient Inference. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (Proceedings of Machine Learning Research). PMLR, 527–536. http://proceedings.mlr.press/v70/bolukbasi17a.html [6] Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press. [7] Lawrence D. Brown, T. Tony Cai, and Anirban DasGupta. 2001. Interval Estimation for a Binomial Proportion. Statist. Sci. 16, 2 (2001), 101–133. [8] Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Trans. Mach. Learn. Res. 2024 (2024). https://openreview.net/forum?id=cSimKw5p6R [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning (ICML). [10] Yeounoh Chung, Rushabh Desai, Jian He, Yu Xiao, Thibaud Hottelier, YvesLaurent Kom Samo, Pushkar Kadilkar, Xianshun Chen, Sam Idicula, Fatma Özcan, Alon Halevy, and Yannis Papakonstantinou. 2026. 100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models. CoRR abs/2603.15970 (2026). arXiv:2603.15970 https://arxiv.org/abs/ 2603.15970 [11] C. J. Clopper and E. S. Pearson. 1934. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika 26, 4 (1934), 404–413. doi:10. 2307/2331986 [12] Andrew Cotter, Heinrich Jiang, Maya R. Gupta, Serena Wang, Taman Narayan, Seungil You, and Karthik Sridharan. 2019. Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals. Journal of Machine Learning Research 20 (2019), 1–59. [13] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 Deep Learning Track. In The TwentyEighth Text REtrieval Conference (TREC) Proceedings. [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 4171–4186. doi:10.18653/V1/N19-1423 [15] Keinosuke Fukunaga. 1990. Introduction to Statistical Pattern Recognition (2 ed.). Academic Press. doi:10.1016/C2009-0-27872-X [16] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Conference’17, July 2017, Washington, DC, USA
[17] Yonatan Geifman and Ran El-Yaniv. 2017. Selective Classification for Deep Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS) 30. arXiv:1705.08500 https://proceedings.neurips.cc/paper_files/paper/2017/hash/ 4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html [18] Isaac Gibbs and Emmanuel J. Candès. 2021. Adaptive Conformal Inference Under Distribution Shift. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. 1660–1672. https://proceedings.neurips.cc/ paper/2021/hash/0d441de75945e5acbc865406fc9a2559-Abstract.html [19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531 (2015). https://arxiv.org/abs/1503.02531 [20] A. Hou et al. 2026. Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm (CSV / UniCSV / SimCSV). Proceedings of the VLDB Endowment (2026). https://arxiv.org/abs/2603.04799 To appear; preprint arXiv:2603.04799. [21] Daniel Kang, Edward Gan, Peter Bailis, Tatsunori Hashimoto, and Matei Zaharia. 2020. Approximate Selection with Guarantees using Proxies. In Proceedings of the VLDB Endowment, Vol. 13. 1990–2003. doi:10.14778/3407790.3407811 [22] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for OpenDomain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [23] Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020. ACM, 39–48. doi:10. 1145/3397271.3401075 [24] Ferdi Kossmann, Ziniu Wu, Alex Turk, Nesime Tatbul, Lei Cao, and Samuel Madden. 2026. KEN: An Execution Engine for Unstructured Database Systems. Proc. VLDB Endow. 19, 5 (2026), 902–916. doi:10.14778/3796195.3796204 [25] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=lgsyLSsDRe [26] Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021. Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool Publishers. [27] Chunwei Liu et al. 2024. Palimpzest: Optimizing AI-Powered Analytics with Semantic Operators. In Conference on Innovative Data Systems Research (CIDR). [28] Shichen Liu, Fei Xiao, Wenwu Ou, and Luo Si. 2017. Cascade Ranking for Operational E-commerce Search. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017. ACM, 1557–1565. doi:10.1145/3097983.3098011 [29] Yao Lu, Aakanksha Chowdhery, Srikanth Kandula, and Surajit Chaudhuri. 2018. Accelerating Machine Learning Inference with Probabilistic Predicates. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10-15, 2018. ACM, 1493–1508. doi:10.1145/3183713.3183751 [30] Qiuyang Mang, Yufan Xiang, Hangrui Zhou, Runyuan He, Jiaxiang Yu, Hanchen Li, Aditya Parameswaran, and Alvin Cheung. 2026. PLOP: Cost-Based Placement of Semantic Operators in Hybrid Query Plans. CoRR abs/2604.09944 (2026). arXiv:2604.09944 https://arxiv.org/abs/2604.09944 [31] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL). [32] Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. CoRR abs/1901.04085 (2019). arXiv:1901.04085 http://arxiv.org/abs/1901.04085 [33] Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. 2021. Confident Learning: Estimating Uncertainty in Dataset Labels. J. Artif. Intell. Res. 70 (2021), 1373–1411. doi:10.1613/JAIR.1.12125 [34] Liana Patel et al. 2024. LOTUS: Enabling Semantic Queries with LLMs over Tables. In Proceedings of the VLDB Endowment. [35] Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing Neural Networks by Penalizing Confident Output Distributions. In Workshop Track of the 5th International Conference on Learning Representations (ICLR). [36] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). [37] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). [38] Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2025. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. Proc. VLDB Endow. 18 (2025). arXiv:2410.12189 doi:10.
Kyoungmin Kim, Martin Catheland, and Anastasia Ailamaki
14778/3746405.3746426 [39] Panagiotis Sioulas, Viktor Sanca, Ioannis Mytilinis, and Anastasia Ailamaki. 2021. Accelerating Complex Analytics using Speculation. In Conference on Innovative Data Systems Research (CIDR). https://www.cidrdb.org/cidr2021/papers/ cidr2021_paper03.pdf [40] Llama Team. 2024. The Llama 3 Herd of Models. CoRR abs/2407.21783 (2024). arXiv:2407.21783 doi:10.48550/ARXIV.2407.21783 [41] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. [42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. CoRR abs/2302.13971 (2023). [43] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer. doi:10.1007/b106715 [44] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text Embeddings by Weakly-Supervised Contrastive Pre-training. CoRR abs/2212.03533 (2022). [45] Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-Shot Information Extraction via Chatting with ChatGPT. CoRR abs/2302.10205 (2023). arXiv:2302.10205 doi:10.48550/ARXIV.2302.10205 [46] Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference. J. Amer. Statist. Assoc. 22, 158 (1927), 209–212. [47] Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. 2022. Adaptive Conformal Predictions for Time Series. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA (Proceedings of Machine Learning Research). PMLR, 25834–25866. https://proceedings.mlr.press/v162/zaffran22a.html [48] Sepanta Zeighami, Shreya Shankar, and Aditya Parameswaran. 2025. BARGAIN: Budget-Aware Risk-Adaptive Guarantees for Accuracy-Improved LLM Cascades over Documents. Proceedings of the ACM SIGMOD International Conference on Management of Data (2025). https://arxiv.org/abs/2509.02896 Preprint arXiv:2509.02896. [49] Hengrui Zhang, Yulong Hui, Yihao Liu, and Huanchen Zhang. 2025. ScaleDoc: Scaling LLM-based Predicates over Large Document Collections. CoRR abs/2509.12610 (2025). arXiv:2509.12610 doi:10.48550/ARXIV.2509.12610