Improving BM25 Code Retrieval Under Fixed Generic Tokenization
arXiv:2605.18561v1 [cs.IR] 18 May 2026
Adaptive q-Log Odds as a Drop-In BM25 Fix
Santosh Kumar Radha∗ AgentField [email protected]
Oktay Goktas AgentField [email protected]
Abstract In retrieval-augmented coding, failures often begin when the relevant file is absent from the retrieved context. Under frozen generic tokenization, where a BM25 index has been built by a search system whose analyzer the practitioner does not control, this failure is routine: BM25’s logarithmic RSJ-odds IDF under-separates the identifier tail that distinguishes one function from another. We replace the outer logarithm of the Robertson–Spärck-Jones odds with a q-logarithm. At q = 1 the transform recovers BM25 exactly by L’Hôpital’s rule, and for q < 1 it is a BoxCox transform of the RSJ odds with λ = 1 − q. On CoIR [Li et al., 2024] CodeSearchNet [Husain et al., 2019] Go (182K documents), oracle-tuned NDCG@10 rises from 0.2575 to 0.4874 (absolute +0.2299; +89.3% relative; zero sign reversals in 10,000 paired-bootstrap resamples, reported as p ≤ 10−4 ). The effect is graded across code languages and is near-zero on BEIR [Thakur et al., 2021] text. A one-parameter closed form estimates a corpus-level q from hapax density and stays near q = 1 on corpora where BM25 is already optimal. The index-time cost is a single pass over the sparse score matrix and query latency is unchanged. A tokenizer ablation shows that identifier-aware tokenization largely removes the incremental gain from q-IDF.
1
Introduction
A coding agent asked to patch a bug first has to find the gold file. When the function is called handleWebSocketUpgrade in a repository of fifty thousand files, the agent’s retriever should return one file, not a thousand. In practice the retriever is a commodity BM25 served through Lucene, Tantivy, Elasticsearch, or a similar indexer whose analyzer was tuned for natural-language text. Under frozen generic tokenization, the agent gets distractor files in its context, the model produces a wrong patch, and the failure is attributed to the model. Given an inherited generic tokenizer, the remaining under-separation is in the weighting. When analyzer changes are allowed, tokenization is the first fix and often the only one needed; this paper studies the regime where they are not. BM25 scores a query by summing term-level weights of the form log((N −nt +0.5)/(nt +0.5)) [Jones, 1972, Robertson et al., 1994], where nt is the document frequency of term t and N the corpus size. The logarithm grows so slowly at the tail that it flattens the weight gap between ultra-rare and rare identifiers. A df=1 hapax like handleWebSocketUpgrade and a df=50 identifier that recurs across many files end up with IDF weights that differ by a small constant rather than an order of magnitude. The agent-relevant consequence is that the gold file is ranked below distractor files which happen to share a few of those rare-but-not-unique tokens, and the context window fills with code the agent does not need. ∗
Corresponding author. Email: [email protected]
Preprint. Under review.
When the retrieval system is owned end-to-end, the right fix is to change the analyzer: emit both the whole identifier and its sub-tokens, building on a decade of work on automatic identifier splitting in the software-engineering community [Caprile and Tonella, 2000, Enslen et al., 2009, Hill et al., 2014], so that BM25 has the discriminative signal available. Many deployments do not have that option. The tokenizer is fixed by an infrastructure decision; practitioners who wire a retriever into an existing search cluster inherit a tokenizer built for English and are asked to improve retrieval without touching it. This is the same regime in which repository-scale code-retrieval benchmarks such as RepoBench [Liu et al., 2024] and Long Code Arena [Bogomolov et al., 2024] situate their evaluations, and the regime this paper addresses. The mechanism we propose is a one-parameter deformation of the outer logarithm in the RSJ IDF. Writing the classical RSJ IDF as log((N − nt + 0.5)/(nt + 0.5)), we replace the log by the Tsallis q-logarithm lnq (x) = (x1−q − 1)/(1 − q), giving the q-log RSJ-odds form. At q = 1 the transform recovers the classical BM25 IDF exactly via L’Hôpital’s rule; for q < 1 the IDF grows as a power law in the very-rare regime, amplifying df=1 hapaxes that the log would flatten against their lessrare neighbours. Mathematically this is a Box-Cox transform of RSJ odds with λ = 1 − q [Box and Cox, 1964]: we are not proposing a new function family, only identifying the right base (the RSJ odds) to apply a classical power transform to, and re-parameterising the exponent so that q = 1 is the identity. Under the same frozen generic tokenization used by out-of-the-box BM25, the q-log transform at q = 0.05 improves NDCG@10 on CoIR CodeSearchNet Go (182K documents) by +89.3% over BM25 (from 0.2575 to 0.4874; zero sign reversals in 10,000 paired-bootstrap resamples, reported as p ≤ 10−4 ). The signal is graded across languages, scales with corpus size, and is near-zero on BEIR text. It is also estimated from corpus statistics alone: a one-parameter closed form q = 1−7.28 htok, fit on the six CoIR code languages at three subset sizes, attains a leave-one-language-out grand-mean recovery of 0.72 on held-out languages, and falls back to plain BM25 on corpora where qopt ≈ 1. A tokenization confound qualifies the result. If the analyzer is changed to emit both whole identifiers and sub-tokens, BM25 already carries the distinguishing evidence and the q-log gain collapses. This is a property of the method’s scope: the tokenizer ablation shows that identifier-aware tokenization largely removes the incremental gain from q-IDF, and the q-log form is the path available when the tokenizer is not. The frozen-tokenizer regime is not a corner case: it covers managed Elasticsearch and OpenSearch deployments, hosted code-search products built on Lucene or Tantivy, and any internal corpus indexed once by a platform team and queried by many downstream agents. In each case the analyzer is set by infrastructure rather than by the practitioner wiring in the retriever, and changing it requires a reindex coordinated across consumers; a one-line IDF rescale at index-load time does not.
Contributions. 1. The q-log RSJ-odds IDF, a one-line change to BM25 that recovers BM25 exactly at q = 1 by L’Hôpital and is a Box-Cox transform of the RSJ odds with λ = 1 − q (section 3). 2. An oracle-sweep upper bound of +89.3% NDCG@10 on CoIR CodeSearchNet Go (182K) at the best tested q = 0.05 under frozen generic tokenization, with 95% paired-bootstrap CI strictly above zero and a monotone scaling curve from 1K to 182K documents (section 4). 3. A one-parameter closed-form predictor q = 1 − 7.28 htok fit on 18 labeled points (six languages, threesizes) with leave-one-language-out grand-mean recovery 0.72, median 0.57 across all twenty 63 three-train/three-test splits, and minimum predicted q ≥ 0.88 on oracle-flat corpora (section 6). 4. A tokenizer ablation showing that identifier-aware tokenization largely removes the incremental gain from q-IDF; on Python, whose snake_case identifiers are dictionary-like, sub-token decomposition reduces signal and the whitespace-preserving tokenizer performs best (section 7). 5. Two agent-relevant retrieval proxies, Recall@K-tokens (a context-window analogue of Recall@K) and RepoBench-R, with significant gains on benchmarks with larger candidate pools or lower BM25 baselines and secondary regression checks on smaller-pool settings (section 8). 2
16 q = 0.5 (amplify) q = 0.8 q = 1 (BM25) q = 1.2 q = 1.5 (saturate)
14 12
idf RSJ (t) q
10 8 6 4 1/(q−1)
2 0 10−4
10−3
10−2
10−1
Document frequency fraction nt /N (log scale)
(t) = Figure 1: The q-log deformation of RSJ odds. Each curve is idf RSJ q 1−q lnq ((N − nt + 0.5)/(nt + 0.5)) with lnq (x) = (x − 1)/(1 − q), plotted against the document-frequency fraction nt /N . At q=1 (dashed gray) the expression recovers the classical RSJ log-IDF exactly via L’Hôpital. For q < 1 the IDF amplifies ultra-rare tokens into a power-law tail x1−q /(1 − q); for q > 1 it saturates at 1/(q − 1) and the rarest tokens stop contributing new discrimination. The one-parameter lever is curvature at the tail.
2
Background: BM25, RSJ odds, and a classical transform
This section fixes the BM25 form used throughout the paper and locates the transform we apply in the statistics literature. BM25 with RSJ-odds IDF. The BM25 scoring function [Robertson and Zaragoza, 2009] for a query q against a document d is a sum of term-level products of an IDF weight and a saturating term-frequency factor, X ft,d · (k1 + 1) BM25(q, d) = idf BM25 (t) · (1) ¯, ft,d + k1 (1 − b + b · |d|/d) t∈q
where ft,d is the within-document frequency and k1 , b are saturation parameters. The IDF is the logarithm of a smoothed Robertson–Spärck-Jones (RSJ) odds ratio [Jones, 1972, Robertson et al., 1994], N − nt + δ , δ = 0.5. (2) idf BM25 (t) = log nt + δ The smoothing δ is a standard term-specificity prior; the logarithm is the specific functional form we will generalise. 2 Divergence from randomness. An alternative lexical family is the divergence-from-randomness framework [Amati and van Rijsbergen, 2002], which scores terms via the divergence between ob2
A tokenizer implementation note. Equation (2) is the classical RSJ-odds IDF, log((N − nt + 0.5)/(nt + 0.5)), to which our lnq substitution reduces exactly at q = 1. Modern Lucene ships the shifted variant log(1 + (N −nt +0.5)/(nt +0.5)). Our bit-identity claim is stated against the classical RSJ definition. Modern systems often differ in this detail [Kamphuis et al., 2020]. In our implementation the baked-in denominator is the strictly positive Lucene shifted IDF from bm25s [Lu, 2024]; for q ̸= 1, the sparse score matrix is rescaled from that Lucene denominator to the q-log RSJ weight. At q = 1 the implementation skips the rescale, preserving the original bm25s/Lucene score matrix bit-for-bit. No document-rank equivalence between the classical and shifted IDF conventions is assumed.
3
served and expected frequencies under a null model. We include the DPH variant, a later parameterfree instance of the DFR family, as a baseline in section 4. No entropy claim. The probabilistic-relevance derivation of BM25 is independent of any information-theoretic functional, and our modification touches only the outer transform in eq. (2). The method is a classical power transform of a probabilistic odds ratio, not a replacement of one entropy by another. Relation to Box-Cox. Writing y = (N − nt + δ)/(nt + δ) for the smoothed RSJ odds, the BoxCox transform [Box and Cox, 1964] at parameter λ is (y λ −1)/λ, with the λ = 0 limit equal to log y by L’Hôpital’s rule. Our lnq (y) = (y 1−q − 1)/(1 − q) with λ = 1 − q is this standard transform in disguise. The q-notation makes BM25 recovery explicit (at q = 1 the transform is the identity on the log-IDF) and connects to q-generalised logarithms in non-extensive statistics, which we return to only in section 9.
3
Method: q-Log Deformation of RSJ Odds
We define the q-log RSJ-odds IDF as N − nt + δ , idf RSJ (t) = ln q q nt + δ
lnq (x) =
x1−q − 1 , 1−q
δ = 0.5,
(3)
and call BM25 with the IDF in eq. (2) replaced by eq. (3) the q-log method. No other component of BM25 is changed: the term-frequency saturation factor and length normalisation in eq. (1), and tokenisation, are left intact. Recovery of BM25 at q = 1 via L’Hôpital. At q = 1 the expression (x1−q − 1)/(1 − q) is 0/0 indeterminate. Differentiating numerator and denominator with respect to q, d 1−q x − 1 = −x1−q log x, dq
d 1 − q = −1, dq
so limq→1 lnq (x) = log x. Applied pointwise to eq. (3), this recovers BM25 exactly at q = 1 via L’Hôpital. Monotonicity and asymptotic behaviour. For every q ∈ R, lnq is strictly increasing on x > 0, and the RSJ odds (N −nt +0.5)/(nt +0.5) is strictly decreasing in nt , so idf RSJ is strictly decreasing q in nt for every q. The ordering of the vocabulary by document frequency is preserved for every q; only the curvature of the weighting at the tail changes. When nt ≪ N , the RSJ odds x → N/nt grows large, and 1−q x q < 1 (power-law amplification), 1 − q lnq (x) ≈ log x q = 1 (BM25), 1 q > 1 (bounded saturation). q−1 The three regimes are illustrated in fig. 1. Logarithmic rarity weighting can under-separate ultra-rare identifiers because the log grows so slowly that a term appearing in a single file and a term appearing in fifty both receive very similar weight. The q < 1 regime in eq. (3) restores discrimination at the tail; the q > 1 regime explicitly throws it away. What is new. Equation (3) is the Box-Cox transform of the smoothed RSJ odds under λ = 1 − q. The contribution is to apply the Box-Cox/q-log transform to RSJ odds and to select its exponent from corpus statistics. Numerical stability. The closed form (x1−q −1)/(1−q) has a removable 0/0 singularity at q = 1 and catastrophic cancellation in a neighbourhood. We use log x directly whenever |q − 1| < ϵ with ϵ = 10−9 , which matches the L’Hôpital limit to machine precision. The bit-identity gate at q = 1.0 4
exactly is verified to max absolute score difference 0.0 on toy, CoIR Go 5K, and BEIR NFCorpus corpora, and top-k rankings are identical. For very common terms, the classical RSJ odds can fall below 1 and therefore produce a negative q-log weight. We retain that RSJ sign convention. The rescale denominator, however, is the Lucene shifted IDF already baked into the bm25s matrix, which is strictly positive for observed terms; empty columns are absent from the sparse matrix. Implementation. Working from a built BM25 index, the method rescales each CSC column of the per-term score matrix by the ratio of the new IDF to the baked-in Lucene IDF: idf RSJ (t) q (new) (BM25) . st,d = st,d · idf Lucene (t) At q = 1 we skip the rescale entirely, preserving the original index. Index build is unchanged up to a single O(|V | + nnz) sparse column rescale (118 ms on the V = 144,938 CoIR-Go vocabulary); query latency is unchanged to within measurement noise (p50 +0.18%, p95 +2.29%; see section A).
4
Main Results
We evaluate the q-log method against BM25 and two additional lexical baselines on CoIR CodeSearchNet (six languages at full corpus scale) and BEIR (three text datasets, as a negative control). NDCG@10 [Järvelin and Kekäläinen, 2002] is the primary metric; the main oracle and scaling deltas are accompanied by 95% paired-bootstrap CIs over queries with 10,000 resamples [Smucker et al., 2007]. We report p ≤ 10−4 when no centered bootstrap resample is at least as extreme as the observed mean difference; this is the empirical resolution of 10,000 resamples, not a smaller exact p-value. Three operating points appear in the paper and should not be conflated: the perlanguage oracle qopt (upper bound, table 1); a conservative q = 0.10 used for the Go scaling curve (table 3), chosen to be robust across corpus sizes; and the deployable closed-form predictor qpred = 1 − 7.28 htok of section 6 (table 2), which uses no labels or queries. Tokenisation is the default bm25s pipeline [Lu, 2024] (lower, stopword filter, \b\w\w+\b regex), held fixed across methods. This is the “frozen generic tokenization” regime of the introduction. Baselines. Plain BM25 [Robertson and Zaragoza, 2009, Lu, 2024] (Lucene parameters k1 = 1.5, b = 0.75). Two additional lexical families test whether the gain reduces to any monotone rarity rescaling: idf γ , a power-of-IDF transform directly comparable to lnq at the tail, and DPH, the divergence-from-randomness variant of Amati and Van Rijsbergen [Amati and van Rijsbergen, 2002]. Both are evaluated at full corpus scale on CoIR-Go and CoIR-Python. 4.1
Multi-language NDCG@10 with confidence intervals
Table 1 reports the central result. On CoIR-Go 182K under frozen generic tokenization, q-log at q = 0.05 improves NDCG@10 from 0.2575 to 0.4874, an absolute gain of +0.2299 with 95% pairedbootstrap CI [+0.2203, +0.2395] and no sign reversals in 10,000 centered bootstrap resamples. Java, Ruby, and JavaScript are significantly positive at their per-language qopt ; PHP is not (CI spans zero); Python selects qopt = 1.00 and is bit-identical to BM25 by construction. For oracle-selected rows, CIs condition on the selected grid value and therefore should be read as conditional uncertainty, not as a selection-adjusted significance test. The size of the gain tracks how much of the retrieval signal is concentrated in the identifier tail. Deployable adaptive mode. Table 2 reports the label-free deployment setting. Using the corpusadaptive predictor of section 6 (qpred = 1−7.28 htok, fit once and frozen), a single-pass deployment recovers most of the oracle gap on every code corpus. Go captures 82.7% of the oracle gap at qpred = 0.54 (+0.190 absolute). Java, PHP, and JavaScript land within 4% of the oracle setting, so the predictor realises almost the full oracle delta. Python sits at qpred = 0.88, with a predicted delta within implementation noise of zero. The adaptive row corresponds to deployment without relevance labels; the oracle row is an upper bound. Negative control on text. On three BEIR text datasets the q-log deltas are indistinguishable from zero at 95% paired bootstrap (table 1, bottom block). Text vocabularies are not driven by an identifier tail, the oracle q sits at or above 1, and the method collapses to plain BM25 by the same mechanism that makes it ineffective on Python. 5
Table 1: Main results. Multi-language NDCG@10 at full corpus scale under frozen generic tokenization. BM25 baseline vs. q-log at the per-language oracle qopt , with 95% paired-bootstrap CIs over 10,000 resamples. Oracle rows are upper bounds selected on the same relevance data, so the CIs are conditional on the selected grid value; for Go, qopt = 0.05 is the best tested value at the lower grid boundary. Shaded columns mark the proposed q-log method (ours). Go, Java, and Ruby serve as the original predictor-development split (section 6); Python, PHP, and JavaScript are the corresponding held-out display split. Bold cells mark significant wins at p < 0.05. htok is reported to four digits so that the predictor formula qpred = 1 − 7.28 htok in table 2 is reproducible from the displayed values. N
Corpus CoIR-Go CoIR-Java CoIR-Ruby
htok qopt BM25 q-log
182,440 .0630 0.05 180,866 .0156 0.90 27,570 .0244 0.70
CoIR-Python 280,310 .0160 1.00 CoIR-PHP 267,725 .0133 0.90 CoIR-JavaScript 64,854 .0206 0.70 BEIR-NFCorpus BEIR-SciFact BEIR-ArguAna
3,633 5,183 8,674
— 1.10 — 1.10 — 1.10
∆ 95% CI
Display
.258 .487 +.230 [+.220, +.240] .371 .383 +.012 [+.010, +.014] .345 .370 +.024 [+.014, +.035]
DEV DEV DEV
.727 .727 0.000 [0, 0] HELD .371 .373 +.002 [−.001, +.004] HELD .352 .362 +.010 [+.003, +.017] HELD .306 .305 −.002 spans zero .662 .656 −.006 spans zero .361 .360 −.001 spans zero
control control control
Table 2: Deployable adaptive mode under frozen generic tokenization. BM25, oracle q-log at qopt (upper bound), and the corpus-adaptive predictor qpred = 1−7.28 htok (single-pass deployment, no labels or queries). Shaded columns mark the proposed q-log method (ours). Recovery is the fraction of the BM25→oracle gap that the predictor captures, computed from full-precision NDCG values; displayed cells are rounded to three decimals so visually equal cells (e.g. Java NDCG@qopt = .383 and NDCG@qpred = .383 from underlying 0.3832 and 0.3828) can correspond to a non-trivial recovery. PHP is included for completeness even though its oracle delta is not significant (table 1).
4.2
Corpus
BM25 qopt NDCG@qopt qpred NDCG@qpred Recovery
CoIR-Go CoIR-Java CoIR-Ruby CoIR-Python CoIR-PHP CoIR-JavaScript
.258 .371 .345 .727 .371 .352
0.05 0.90 0.70 1.00 0.90 0.70
.487 .383 .370 .727 .373 .362
0.54 0.89 0.82 0.88 0.90 0.85
.448 .383 .364 .726 .373 .360
82.7% 96.7% 75.0% flat ∼ 100% 80.8%
BEIR text (3)
—
≥1
near BM25
1.00
= BM25
n/a
Scaling with corpus size on Go
The Go result is monotone in corpus size. Figure 2 plots a q-sweep on CoIR-Go at 50K and at the full 182K corpus. Table 3 reports the 95% paired-bootstrap CIs along the full 1K → 182K scaling curve at q = 0.10. Every CI is strictly above zero; the mechanism does not saturate at the scales we test. 4.3
Comparison against rarity-rescale baselines
We next compare against monotone IDF rescalings that amplify rare terms. Replacing the Lucene IDF column by idf γ with γ ∈ {0.5, 0.8, 1.0, 1.2, 1.5, 2.0} (γ = 1 is bit-identical to BM25), the best tested setting on CoIR-Go is γ = 2.0, which yields NDCG@10 = 0.3517 (+36.6% over BM25). The q-log method at q = 0.05 yields 0.4874, a further +38.5% over best-γ (table 4). The DFRDPH family loses outright on code: NDCG@10 = 0.1968 on Go (−24%) and 0.5538 on Python (−24%), reflecting a tf/length normalisation that is miscalibrated for short code documents. Our DPH implementation is validated bit-for-bit against PyTerrier / Terrier 5.113 on a matched 5K-doc 3
https://github.com/terrier-org/pyterrier
6
Go 50K Go 182K
NDCG@10
0.5
0.4
BM25 50K (q=1)
0.3
BM25
BM25 182K (q=1)
0.2 0.1
0.3
0.5
0.7
1
1.3
q-log exponent q
Figure 2: q-log sweep on CoIR-CSN Go under the q-log RSJ IDF, idf RSJ (t) = lnq (N − nt + q 1−q 0.5)/(nt + 0.5) with lnq (x) = (x − 1)/(1 − q). At q=1 this recovers standard BM25 exactly (L’Hôpital limit). With the IDF written on the RSJ-odds base, rare-term amplification lives at q<1: NDCG@10 rises monotonically from 0.31 (BM25, 50K) to 0.53 at q≈0.05 and saturates. Beyond q=1 the lnq is bounded above by 1/(q − 1) and NDCG@10 decays. The 182K sweep (dashed) confirms the effect scales: +88.3% at q=0.1, compared with the 50K +72.3%.
Table 3: Go scaling at q = 0.10 under frozen generic tokenization; this is the conservative operating point, with the non-conservative main result at qopt = 0.05 in table 1. Every row is a significant win (p ≤ 10−4 at the empirical bootstrap resolution, CI strictly above zero). The absolute delta grows monotonically from +0.061 at 1K to +0.227 at full corpus, consistent with non-saturating gains at the scales we test. N 1,000 2,000 5,000 10,000 20,000 50,000 182,440
nq BM25 q-log (ours) 1,000 2,000 5,000 8,122 8,122 8,122 8,122
.532 .501 .417 .392 .363 .309 .258
.594 .579 .574 .575 .560 .531 .485
∆ (abs) 95% CI +0.061 +0.078 +0.157 +0.183 +0.198 +0.222 +0.227
[+0.043, +0.079] [+0.064, +0.092] [+0.147, +0.168] [+0.175, +0.192] [+0.189, +0.207] [+0.213, +0.231] [+0.218, +0.237]
Go subset (median Kendall τ = 1.00, mean abs NDCG@10 diff 3e−3); the DPH deficit is a real scoring-model effect, not a port artefact.
5
Per-Query Mechanism
The per-query decomposition in fig. 4 locates the gain at the level of individual queries, and fig. 3 walks through one concrete Go example. On the CoIR-Go full corpus at q = 0.10, ∆NDCG@10 correlates positively with the fraction of query tokens at df ≤ 5 (OLS slope +0.10, Spearman ρ = +0.10 over N = 8,122 queries). The correlation is weak, so we treat it as descriptive rather than causal evidence. Panel (d) analyzes the mechanism with a df-bin occlusion: for each bin we drop every query token in that bin and record the resulting NDCG@10 loss. Dropping df= 1 hapaxes alone costs +0.4351 of NDCG@10 on Go, roughly 4× the next-largest bin (df= 2 at +0.1127) and 54× the mean of the four middle bins (df= 6–1000, mean +0.0080), with the combined tip (df= 1 + df= 2) accounting for +0.5478 of the occlusion loss. At q < 1 the q-log transform overamplifies df= 1 hapaxes, the unique identifiers where rare-token mass concentrates, and the Go gain is concentrated toward larger separation among df=1 and df=2 terms. The occlusion is diagnostic rather than a causal proof, because dropping tokens also changes query length and semantics. 7
Table 4: Lexical-family comparison at full corpus scale under frozen generic tokenization. BM25 is the Lucene baseline, idf γ sweeps γ ∈ {0.5, . . . , 2.0} (best reported), DPH is the parameterfree divergence-from-randomness variant of Amati and Van Rijsbergen [Amati and van Rijsbergen, 2002], and our q-log method (shaded column) is at the per-language qopt . Percentage gains are computed from full-precision NDCG; displayed NDCG cells are rounded to three decimals. On Go, the q-log gain is not reproduced by the tested idf γ family on this benchmark; because the best tested γ is at the edge of the grid, this comparison does not rule out every possible monotone rarity transform. On Python every rarity-rescale method ties with BM25, because there is no rare-identifier signal to amplify. idf γ (best γ)
BM25 CoIR-Go full (182K) CoIR-Python full (280K)
Query:
.258 .727
q-log (ours)
DPH
.352 (γ=2.0, +36.6%) .197 (−23.6%) .487 (+89.3%) .727 (γ=1.2, +0.0%) .554 (−23.9%) .727 (+0.0%)
handleWebSocketUpgrade auth middleware
(CoIR-Go q174632; 4 default tokens)
q-log top-5 (q = 0.10)
BM25 top-5 (log-IDF) 1. auth/middleware/jwt.go
⋆ 1. ws/gateway/upgrade.go (gold)
2. net/http/handler_test.go
2. auth/middleware/jwt.go
3. auth/session/cookie.go
3. net/http/handler_test.go
4. middleware/logging.go
4. auth/session/cookie.go
5. auth/oauth2/provider.go
5. middleware/logging.go
. . . gold file at rank 23 (NDCG@10 = 0.00).
Gold lifted from rank 23 to rank 1 (NDCG@10 = 1.00).
Per-token IDF weight (why the reshuffle happens) token handleWebSocketUpgrade middleware auth handle
df
idf BM25
idf q=0.10
ratio q-log / BM25
1 1,820 3,714 14,203
11.7 4.6 3.9 2.5
41,933 68.8 35.2 9.2
3578× 15× 9× 4×
Figure 3: A qualitative walkthrough of the mechanism on a CoIR-Go full-corpus query. Under log-IDF the df= 1 identifier handleWebSocketUpgrade carries weight only about 3× that of the middle-df tokens auth and middleware, so documents sharing two or three middle-df tokens outrank the single gold file that carries the hapax. The q-log rescale at q = 0.10 amplifies the df= 1 weight by roughly 3,500× while amplifying middle-df weights by only one or two orders of magnitude; the ranking inverts and the gold moves to rank 1. Per-token df values and both IDF columns are exact values from the CoIR-Go 182K index; the gold ∆NDCG@10 is taken from query q174632. The displayed file paths are representative of the CoIR-Go corpus structure; the ranking flip reflects the df-bin mechanism of section 5.
The same analysis on CoIR-Python at q = 1.00 (q-log is identity to BM25 by construction, so this is the BM25 baseline’s own df signature) shows an inverted picture: the signal is concentrated in the middle df bins (df= 201–1000 carries +0.0703, df= 1001–5000 carries +0.0528), not at the tip. Python’s ecosystem reuses dictionary-like tokens across hundreds of thousands of files, so BM25’s log-IDF is already using the middle-df bins optimally. Any deviation from q = 1 in either direction distorts those bins and degrades ranking. Python’s oracle q is therefore exactly 1, and the q-log gain is close to zero on any corpus with a similar df profile. An orthographic identifier heuristic (CamelCase or dotted tokens) predicts the per-query gain much less well than df-based features (panel (c), OLS slope +0.002, Spearman ρ = −0.003); the transform operates on document frequency rather than on surface syntax. A query can look like code and be dominated by common tokens, while a prose-like query can sit in the hapax tail. 8
(a) low-df mass
(b) median df 1
∆NDCG@10
∆NDCG@10
1 0.5 0 −0.5
0.5 0 −0.5
0
0.2
0.4
0.6
100
101
low_df_mass(q)
102
103
104
median query df
(c) identifier frac.
(d) df-bin occlusion
1
mean ∆ loss
∆NDCG@10
0.4 0.5 0
0.2
−0.5 0 0
0.2
0.4
0.6
0.8
1
2
identifier fraction Go (q=0.10, N =8,122)
3-5 6-10 11-5051-200201-1K1K-5K 5K-20K20K+
occluded df bin Python (q=1.00, N =14,918, null)
Go rolling median (window 500)
Figure 4: Per-query mechanism of q-log BM25 on CoIR-CodeSearchNet at each corpus’s working point: Go full 182K at q=0.10 (blue, N =8,122 queries; mean ∆NDCG@10 = +0.23) and Python full 280K at q=1.00 (orange, N =14,918 queries; ∆ ≡ 0 by construction — q-log BM25 is identical to BM25 at q=1, which we use as a null control). Panels (a)–(c) scatter per-query ∆NDCG@10 against three features of the query: low-df mass (fraction of tokens with df ≤ 5), median query df, and a syntactic identifier-fraction heuristic. The red line in each panel is a window-500 rolling median over Go queries sorted by x. Panel (a) shows the mechanism directly: the Go rolling median rises from ≈ 0 at low_df_mass = 0 to > +0.3 at low_df_mass > 0.5 (OLS slope +0.10, Spearman ρ= + 0.10). Panel (b) uses a log10 x-axis because median_df is heavy-tailed; the rolling median drops with increasing median df as expected. Panel (c) shows that the purely orthographic identifier heuristic carries little additional signal beyond what df already captures — the mechanism is frequency-based, not syntax-based. Panel (d) is the df-bin occlusion: for each df decile we drop every query token in that bin and measure the resulting NDCG@10 loss, so a tall bar marks where the retrieval signal at the operating point lives. On Go the loss is concentrated at df = 1 (hapaxes), followed by df = 2 and df = 3−5, confirming that q-log BM25 at q=0.10 works by over-amplifying genuinely-unique identifiers. On Python the same analysis under BM25 (q=1) finds the signal in the middle df bins (df = 51−5000), not at the tip — which is why q-log BM25 with q̸=1 hurts Python: whichever direction q deviates, it distorts the bins that BM25 is already using correctly.
6
Corpus-Adaptive Predictor
Grid search requires held-out queries and relevance labels, so we test a label-free corpus-level predictor. To separate post-hoc fitting from an actual predictor, model forms are evaluated with heldout-language diagnostics; after selecting the form, the displayed deployment coefficient is refit once on all 18 labeled development points. Features. For each CoIR-CSN language we compute six corpus-level statistics, all computed without any access to queries or relevance labels: token count Ntok , vocabulary size V , hapax token density htok (fraction of tokens whose type appears exactly once in the corpus), type-token ratio TTR, median document frequency, fraction of vocabulary with df ≤ 5, and the Zipf exponent α fit via the powerlaw package4 with KS-minimised xmin . The Zipf α turns out to be non-discriminative (α ∈ [1.65, 1.71] across all six CoIR code languages) and is dropped. 4
https://github.com/jeffalstott/powerlaw
9
(a) Leave-one-language-out 1
Go Python
Java PHP
(b) Calibration (RMSE = 0.18)
0.75
Oracle qopt
0.75
Oracle qopt
qpred ≥ 0.85
1
Ruby JS
0.5
0.25
0.5
0.25
0
0 0
0.25
0.5
0.75
1
0
Predicted q (LOLO)
0.5
0.25
0.75
1
Predicted q (final fit)
Figure 5: The q-predictor, two views. (a) Leave-one-language-out: for each of the six code languages, the form q = 1 − c htok is fit on the other five languages at all three corpus sizes and applied to the held-out language’s three sizes; grand-mean recovery 0.72. Shaded band is ±1.96σ from the eighteen LOLO residuals (σ = 0.183). (b) Final fit on all eighteen points, qpred = 1 − 7.28 htok (clipped to [0.01, 1]), with RMSE vs. oracle 0.18. The dotted box marks the region where qopt ≥ 0.95 ⇒ qpred ≥ 0.85; minimum predicted q on oracle-flat corpora is 0.88, so the worst-case regression on a BM25-optimal corpus is under 1%. A full distribution over can didate predictor forms across all 63 = 20 three-train / three-test splits is deferred to fig. 12 in the supplement. Label. The oracle qopt is the argmax of NDCG@10 over q ∈ {0.05, 0.10, 0.20, 0.30, 0.50, 0.70, 0.90, 1.00} on the corpus. For CoIR-Python full corpus the argmax is q = 1 exactly; for CoIR-Go full the best tested value is q = 0.05, which lies at the lower edge of the grid. The label spans the whole tested range. Fit protocol. We evaluate candidate forms across the full 6 × 3 = 18 labeled points (6 CoIR code languages at subset sizes 1K, 10K, and full). Five candidate forms are fit: two one-parameter variants (1 − c · htok; 1 − c · frac_df ≤ 5; 1 − c/α) and two two-parameter variants that add a scale term 1/ log T . Selection is driven by three criteria: (i) the distribution of mean-test recovery across all 63 = 20 possible three-train/three-test partitions; (ii) leave-one-language-out (LOLO) across corpus sizes; (iii) predicted q remains at or above 0.85 when qopt is near 1. Parsimony breaks ties in favour of fewer parameters. Selected predictor. qpred = 1 − 7.28 · htok,
clipped to [0.01, 1.0].
(4)
The coefficient 7.28 is the final coefficient refit on all 18 labeled points after selecting the oneparameter htok form. Across the 20 three-train/three-test splits, where the coefficient is refit inside each training split, the median mean-test recovery is 0.566 with interquartile range [−0.574, +0.638]; the one-parameter htok form is the only form whose median is positive (all dfonly forms tip to −0.06 or below). LOLO across the six languages, with the predictor fit on the other five at all three sizes and applied to the held-out language at all three sizes, gives a grand-mean recovery of 0.722 (range: 0.488 on Python to 0.890 on Go). Calibration RMSE vs oracle q across the 18 points is 0.18. On oracle-flat corpora (qopt ≥ 0.95), the minimum predicted q is 0.884, above the 0.85 floor. Full candidate-form statistics are in the supplement. Graceful behaviour at the BM25 boundary. Figure 6 shows the predictor’s behaviour along the CoIR-Python scaling curve, where the oracle q drifts from qopt = 0.50 at 1K to qopt = 1.00 at 280K as hapax density falls from 0.042 to 0.016. A naive static pick of q = 0.70, which is what the 5K-subset sweep would suggest, beats BM25 at small sizes but regresses by −4.6% at full scale. The closed-form predictor of eq. (4) rises from q = 0.70 at 1K to q = 0.88 at 280K, stays at or above 10
0.9
NDCG@10
0.85
0.8
0.75
BM25 (q=1) Static q=0.7 Adaptive q=predict_q(stats)
0.7 103
104
105
CoIR-CSN Python corpus size (docs, log scale)
Figure 6: Adaptive q-log gate on the CoIR-CSN Python scaling curve. BM25 (q=1) loses NDCG@10 monotonically as the corpus grows from 1k to 280k docs. The naive static pick q=0.7 (argmax on the 5k subset sweep) helps at small sizes but collapses to −4.6% at full scale where the oracle argmax has moved back to q=1. The deployed one-parameter adaptive predictor qpred = 1 − 7.28 htok of eq. (4) tracks the oracle: it beats BM25 at every subset and loses only 0.23% at full scale, because the predicted q rises from 0.70 (1k) to 0.88 (280k) as hapax density drops from 0.042 to 0.016. Static q=0.7 on the same sweep loses 4.6% at 280k, a 20× larger regression. Data: bench/zir/results/adaptive_q_python.csv.
BM25 at every subset, and regresses by only −0.23% at full scale — a 20× smaller regression than the static pick. The method is gated from corpus statistics, and predicted q remains at or above 0.85 when qopt is near 1. df df-only companion. A df-only companion predictor qpred = 1 − c1 · frac_df ≤ 5 − c2 / log T , using only document-frequency statistics of the corpus vocabulary, attains grand-mean LOLO recovery −0.65, driven by a single catastrophic hold-out (PHP) where the fit extrapolates off the labeled range. We report the comparison but deploy eq. (4): the hapax-density feature is cheaper to compute, more stable across languages, and self-consistent with the per-query mechanism analysis of section 5, which shows the q-log gain is carried by df=1 hapaxes specifically.
7
Tokenizer Substitution: identifier-aware tokenization removes most incremental q-IDF gain
An alternative to the q-log method is to change the tokenizer: emit both the whole identifier and its sub-tokens so that BM25 sees both handleWebSocketUpgrade and its camelCase-split parts, along the lines of the identifier-splitting procedures long studied in program comprehension [Caprile and Tonella, 2000, Enslen et al., 2009, Hill et al., 2014]. Figure 7 reports a four-tokenizer ablation on CoIR-Go 50K (panel (a)): T0 (default bm25s pipeline, no stem), T1 (whitespace-only, no stem), T2 (identifier-aware, emits both whole and sub-tokens), T3 (sub-tokens only). Table 5 gives the four corner values. From the anchor cell T0 + BM25 = 0.309, the q-log method alone (T0 + q-log) reaches 0.531, a +72% gain, and the tokenizer alone (T2 + BM25) reaches 0.563, a +82% gain. T2+BM25 reaches 0.563, while T2+q-log reaches 0.564, so q-log adds little once identifier-aware tokenization is used. Applying q < 1 to sub-tokens only overshoots into the T3 regime where short, frequent sub-tokens dominate and the wrong mass is amplified (−37%). 11
Table 5: CoIR-Go 50K NDCG@10 under four tokenizers, BM25 vs. q-log at qopt = 0.10 (the conservative operating point of table 3, not the q = 0.05 main result). T0: default bm25s pipeline (lowercase, stopword filter, no stem); T1: whitespace-only, no stem; T2: identifier-aware (emits both whole identifier and camelCase / snake_case sub-tokens); T3: sub-tokens only. Percentage deltas are computed from full-precision NDCG; displayed NDCG cells are rounded to three decimals. Tokeniser T0 default (no stem) T1 whitespace T2 ident-aware T3 sub-tokens only
BM25 NDCG@10
q-log NDCG@10
∆
0.309 0.149 0.563 0.469
0.531 0.258 0.564 0.295
+71.9% +73.8% +0.2% −37.0%
Extending to Java, Ruby, and Python. The same four-tokenizer ablation on three non-Go CoIR languages at 50K documents each shows the substitution pattern is language-dependent and tracks identifier orthography (fig. 7). Reading the key cells: • Java (camelCase, qopt = 0.90): T2+BM25 vs T0+BM25 gives +27.8%; T0+q-log vs T0+BM25 gives +2.4%; T2+q-log vs T2+BM25 gives +0.2%. The interaction is strongest on Go and Java. • Ruby (snake_case, qopt = 0.70): T2+BM25 vs T0+BM25 gives +22.5%; T0+q-log vs T0+BM25 gives +7.1%; T2+q-log vs T2+BM25 gives +3.6%. The interaction is weaker on Ruby: both changes help individually and a small residual stack remains. • Python (snake_case, qopt = 1.00): T1 (whitespace-only, no decomposition) gives +17.5% NDCG@10 over T0; T2 gives +0.3%; T3 gives +0.0%. The two changes point in opposite directions on Python: the q-IDF change is null (qopt = 1.00, identity to BM25 by construction), but tokenizer choice is non-null and points away from sub-token decomposition. On CoIR-Python the tokenizer that preserves whole identifiers performs best; the code-aware tokenizer that decomposes snake_case reduces signal. On a corpus whose identifier morphology is snake_case and whose naming conventions are conventional dictionary-like words (self, value, data), the agent-relevant signal is concentrated in the whole identifier, not on its parts. The interaction is therefore reversed on Python. Operationally, an analyst who controls tokenisation should adopt a code-aware analyzer matched to the language’s identifier morphology (camelCase-aware splitting on Java and Go, whitespacepreserving on snake_case Python) and leave q = 1. Under frozen tokenisation, q-log with a predicted q is the available lever. Figure 8 summarises the deployment procedure.
8
Agent-Relevant Retrieval Proxies
For a coding agent, the operational metric is whether the gold file appears inside a bounded context window. We therefore report two proxies that evaluate retrieval under that constraint.
8.1
Recall@K-tokens under a context budget
For each query we rank the corpus top-down, tokenise each document with tiktoken5 (cl100k_base), accumulate tokens until the cumulative count exceeds the budget K, and declare the query “recalled at K” iff the gold document appears within that prefix. Figure 9 reports the budget-recall curves. On CoIR-Go full corpus, BM25 recalls the gold within an 8K-token budget on 48.1% of queries; the q-log method at q = 0.10 raises this to 68.8%, an absolute +20.7 percentage points. Most of the axis is flat because Go documents are short (median ∼ 15–27 tokens), and the method wins by ranking the gold higher rather than by packing more aggressively. On CoIR-Python, a flat-gain language at corpus NDCG, we still observe +2.1–+2.4 percentage points at every budget, because rank improvements that are too small to move corpus-level NDCG still reshape the context-budget curve. 12
Python (q=1.00, BM25 = q-log)
Go (q=0.10)
0.8
0.56 0.56
0.6 0.53
0.47
0.4 0.31
0.30
0.26 0.15
0.2
NDCG@10
NDCG@10
1
0
0.91
0.78
0.78
T2
T3
0.5
0 T0
T1
T2
T3
T0
0.53 0.53
NDCG@10
0.55 0.55
0.43 0.44
0.4
T1
Ruby (q=0.70)
Java (q=0.90) 0.6
NDCG@10
0.78
0.33 0.35
0.2
0.42 0.44
0.41 0.42
T2
T3
0.37 0.4 0.35 0.24 0.20
0.2
0
0 T0
T1
BM25 (q=1)
T2
T0
T3
q-log BM25 (q=qopt )
T1
BM25 = q-log (Python, q=1)
Figure 7: Tokenizer ablation across CoIR CSN languages (Go, Python, Java, Ruby; 50K docs per language). NDCG@10 for BM25 (blue) vs q-log BM25 (orange) at each language’s oracle qopt across four tokenizers: T0 default (no stem, bm25s pipeline), T1 whitespace-only (no stem), T2 identifier-aware (whole identifier and sub-tokens), T3 sub-tokens only. Python panel shows a single gray bar per tokenizer because at qopt =1 q-log BM25 is identical to BM25 by construction. Identifier-aware tokenization (T2) largely removes the incremental gain from q-IDF on Go and Java, leaves a smaller residual gain on Ruby, and reverses direction on Python, where the winning tokenizer is T1 because Python’s snake_case identifiers are dictionary-like words that should not be decomposed.
8.2
RepoBench-R
On RepoBench-R [Liu et al., 2024], a small per-row candidate pool benchmark (5–9 candidates for _easy, ≥ 10 for _hard), six of eight cells sit inside a ±3.7% band (fig. 10). Two cells exceed the ±3.7% relative band: Python cfr/hard at +9.2% (relative NDCG@10, true cross-file retrieval) and Java cff/hard at −7.6% (completion-from-file with locally shadowed identifiers). Languagemean deltas are Python +1.8% and Java −1.6%. We read RepoBench-R as a secondary regression check rather than primary evidence of improvement. 8.3
Long Code Arena
We evaluated bug-localisation on Long Code Arena [Bogomolov et al., 2024], treating the issue title and body as query and the patch-changed files as gold. After excluding repositories whose on-disk size exceeded 500 MB, n=45 Python and n=44 Java test-split queries remained. Point estimates on NDCG@10 were positive on Python (BM25 .576 → q-log q=0.5 .596, +3.5%) and flat on Java (+0.5%), but paired-bootstrap 95% confidence intervals straddled zero in both cases ([−0.032, +0.076] Python; [−0.040, +0.051] Java). The per-query tie rate approaches 55%, which reflects the small working set. LCA is consistent in sign but underpowered at this sample size, and we report it as a secondary regression check rather than primary evidence of improvement. 5
https://github.com/openai/tiktoken
13
Can you change the tokenizer? no yes
Is hapax density htok low?
Use a code-aware analyzer • camelCase split + whole identifier • whitespace-preserving on snake_case • leave q = 1 (BM25)
yes
no Apply q-log rescale
Keep BM25 • htok ≲ 0.02 ⇒ qpred ≥ 0.85 • predicted q stays near BM25
• compute htok once (O(|V |)) • set qpred = 1 − 7.28 htok • ∼50 LoC over bm25s • index +7%, query latency unchanged
0.6 0.4 0.2 0 2K
4K
8K
CoIR-Python
0.9
Recall@K-tokens
CoIR-Go (q=0.10)
0.8
Recall@K-tokens
Recall@K-tokens
Figure 8: A practitioner decision chart. When the analyzer is under the operator’s control, the first fix is a code-aware tokenizer matched to the language’s identifier morphology: camelCase splitting on Go and Java, whitespace-preserving on snake_case Python. Under such a tokenizer, BM25 at q = 1 already captures the rare-identifier signal. When the tokenizer is fixed by infrastructure, hapax density htok is the single statistic that decides whether to intervene: htok ≲ 0.02 gives qpred ≥ 0.85, and htok ≳ 0.02 motivates a q-log rescale at qpred = 1 − 7.28 htok. The predictor returns q near 1 on corpora where BM25 is already optimal.
0.88 0.86 0.84 0.82 0.8 2K
16K
4K
8K
0.6 0.58 0.56
16K
Context budget K (tokens)
Context budget K (tokens) BM25
CoIR-Java
0.62
2K
4K
8K
16K
Context budget K (tokens)
q-log BM25 (shown operating point)
Figure 9: Recall@K-tokens on CoIR CodeSearchNet full-corpus splits under frozen generic tokenization. Each language is ranked with BM25 (gray) and the q-log BM25 method (blue) at the shown operating point (Go uses the conservative q=0.10 setting); we walk the ranked list top-down and mark a query as recalled at budget K iff the cumulative tiktoken (cl100k_base) count through the gold document is ≤ K. Note the different y-ranges per panel (Go is zoomed to [0, 0.80]; Python to [0.80, 0.90]; Java to [0.55, 0.62]) — Go docs are short enough that all top-50 fit well inside 2K tokens, so its K-axis is flat and the 20.7-pp gap is a rank win, not a budget win; Python shows the genuine K-axis slope because its long-doc tail extends past 8K. On Go, the gold document appears in the first 8K tokens for 69% of queries under q-log versus 48% under BM25.
9
Discussion
A physics reading. The function lnq (x) originates in non-extensive statistical mechanics, where the q-generalised logarithm [Tsallis, 1988, 2009] is the natural pairing to power-law distributions in the same way that the natural logarithm pairs with exponentials. Code vocabularies follow heavytailed, Zipf-like distributions at multiple levels of granularity (identifiers, method names, file sizes, dependency graphs [Louridas et al., 2008, Zhang, 2009]), and the transform matched to power-law statistics reshapes the IDF toward larger separation among df=1 and df=2 terms. We record this as 14
0.6
NDCG@10
NDCG@10
cff (completion-from-file) 0.4
0.2
0 Py-easy
Py-hard
Java-easy
0.4
0.2
0 Py-easy
Java-hard
BM25
cfr (completion-from-retrieval)
Py-hard
Java-easy
Java-hard
q-log BM25 (q=0.1)
Figure 10: RepoBench-R NDCG@10 on the four “{Python, Java} × {easy, hard}” cells, split by setting: left panel is completion-from-file (cff, local in-file candidates, 5–9 per query on easy / ≥10 on hard); right panel is completion-from-retrieval (cfr, cross-file candidates). Bars show BM25 (gray) vs. q-log BM25 at the primary operating point q=0.1 (blue). Six of eight cells sit inside a ±3.7% relative band, reflecting the benchmark’s small per-row candidate pool which compresses IDF differences. The two hard-split cells exceed the ±3.7% relative band: Python cfr/hard (+9.2% NDCG@10, 0.369 → 0.403) and Java cff/hard (−7.6% NDCG@10, 0.279 → 0.258). We read RepoBench-R as a secondary regression check rather than primary evidence of improvement.
interpretation, not derivation. The empirical claims in this paper depend on the Box-Cox transform framing only; nothing in section 4 or section 6 requires any particular entropy functional. Language dependence. Languages differ in how much of their discriminative retrieval signal is concentrated in df= 1 hapaxes. Go has heavy package-qualified identifier tails (htok = 0.063), and the occlusion analysis in section 5 localises the Go signal on df= 1 tokens, which is exactly where q < 1 amplifies. Python has a dictionary-like identifier vocabulary (htok = 0.016) with BM25 signal concentrated in middle-df bins that the log already handles correctly, so any q ̸= 1 distorts those bins. The predictor of section 6 encodes this relationship quantitatively. Text corpora. BEIR queries are carried by common content words rather than rare identifiers. With little to amplify at q < 1 and an oracle q at or above 1, the method collapses to plain BM25 on text for the same reason it collapses on Python. Relation to divergence-from-randomness. The DFR-DPH family [Amati and van Rijsbergen, 2002] reshapes term weights through a different axis, tf/length normalisation, and loses cleanly on short code documents where that normalisation is miscalibrated. The q-log approach leaves tf and length normalisation untouched and modifies only the outer IDF transform, an axis that matches code retrieval but not the natural-language corpora that motivated DFR.
10
Limitations
Substitutability. The largest boundary condition is that identifier-aware tokenization largely removes the incremental gain from q-IDF. Stacking them does not help and can hurt (−37% at T3 in section 7). If an operator controls tokenisation, the q-log contribution is small; this paper’s value is highest where the tokeniser is frozen by infrastructure. Language-dependence of q. The per-language qopt spans a wide range (from 0.05 on Go to 1.00 on Python); neither a single global q nor a single family’s recommendation is adequate. The predictor of section 6 addresses this at the cost of requiring hapax density (htok) to be computable once per index, which is an O(|V |) pass over the vocabulary. Where that single statistic is not computable, the operational fallback is a grid search over q ∈ {0.1, 0.3, 0.5, 0.7, 0.9, 1.0} on a small held-out query set, fewer than ten calls per language. 15
Small-pool benchmarks. RepoBench-R’s 5–9-candidate pools compress IDF differences, and six of eight cells sit within noise. We treat this setting as a secondary regression check rather than primary evidence of improvement. Held-out agent benchmarks. CrossCodeEval [Ding et al., 2023] gold sets are constructed by running BM25 on oracle chunks (the positives are the BM25 top-k by construction), so any departure from BM25 ranking looks like a departure from the gold. CrossCodeEval is therefore unsuitable as a gain benchmark for this method and useful only as a secondary regression check. Long Code Arena bug-localisation gave positive point estimates on both Python and Java but was underpowered at n≈45 queries per language (both 95% CIs crossed zero). We include it for completeness but not as primary evidence of improvement. Python and PHP at full corpus scale. At full corpus scale, Python’s oracle q is exactly 1 and PHP’s is 0.90 with a delta of +0.4% indistinguishable from noise. The method offers neither significant loss nor significant gain on these languages. With a thin identifier tail there is no rare-token signal to recover, and every baseline considered (idf γ , DPH, q-log) converges to the same point. Scope of evaluation. The full corpus multi-language scaling curve (1K → 182K) is reported only for Go. Python, Java, Ruby, PHP, JS have full-corpus single points at qopt and three-point scaling ladders (1K, 10K, full) but not the dense Go-style curve. The tokenizer ablation covers Go, Java, Ruby, Python at 50K; C#, C++, and domain-specific languages are out of scope for this submission.
References Gianni Amati and C. J. van Rijsbergen. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems, 20(4): 357–389, 2002. doi: 10.1145/582415.582416. Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, and Timofey Bryksin. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612, 2024. G. E. P. Box and D. R. Cox. An analysis of transformations. Journal of the Royal Statistical Society. Series B (Methodological), 26(2):211–243, 1964. doi: 10.1111/j.2517-6161.1964.tb00553.x. B. Caprile and P. Tonella. Restructuring program identifier names. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM 2000), pages 97–107, 2000. doi: 10. 1109/ICSM.2000.883022. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. doi: 10.48550/arXiv.2310. 11248. URL https://openreview.net/forum?id=wgDcbBMSfh. Eric Enslen, Emily Hill, Lori Pollock, and K. Vijay-Shanker. Mining source code to automatically split identifiers for software analysis. In 6th IEEE International Working Conference on Mining Software Repositories (MSR 2009), pages 71–80, 2009. doi: 10.1109/MSR.2009.5069482. Emily Hill, Dawn Lawrie, Lori Pollock, and K. Vijay-Shanker. An empirical study of identifier splitting techniques. Empirical Software Engineering, 19(6):1754–1780, 2014. doi: 10.1007/ s10664-013-9261-0. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint, 2019. doi: 10.48550/arXiv.1909.09436. Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422–446, 2002. doi: 10.1145/582415.582418. 16
Karen Spärck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21, 1972. doi: 10.1108/eb026526. Chris Kamphuis, Arjen P. de Vries, Leonid Boytsov, and Jimmy Lin. Which BM25 do you mean? a large-scale reproducibility study of scoring variants. In Advances in Information Retrieval, volume 12035 of Lecture Notes in Computer Science, pages 28–34. Springer, 2020. doi: 10.1007/ 978-3-030-45442-5_4. Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. CoIR: A comprehensive benchmark for code information retrieval models. arXiv preprint, 2024. doi: 10.48550/arXiv.2407.02883. Tianyang Liu, Canwen Xu, and Julian McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. doi: 10.48550/arXiv.2306.03091. Panagiotis Louridas, Diomidis Spinellis, and Vasileios Vlachos. Power laws in software. ACM Transactions on Software Engineering and Methodology, 18(1):1–26, 2008. doi: 10.1145/ 1391984.1391986. Xing Han Lu. BM25S: Orders of magnitude faster lexical search via eager sparse scoring. arXiv preprint, 2024. doi: 10.48550/arXiv.2407.03618. Santosh Kumar Radha and Oktay Goktas. RareCode: Code and artifacts for adaptive q-log bm25 code retrieval. https://github.com/santoshkumarradha/rarecode, 2026. Public code repository. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. doi: 10.1561/1500000019. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994, number 500-225 in NIST Special Publication, pages 109–126. National Institute of Standards and Technology (NIST), 1994. URL https://trec.nist.gov/pubs/trec3/papers/city.ps.gz. Mark D. Smucker, James Allan, and Ben Carterette. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM 2007), pages 623–632, 2007. doi: 10.1145/1321440. 1321528. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS 2021), 2021. doi: 10.48550/arXiv.2104.08663. Constantino Tsallis. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52(1–2):479–487, 1988. doi: 10.1007/BF01016429. Constantino Tsallis. Introduction to Nonextensive Statistical Mechanics: Approaching a Complex World. Springer, New York, 2009. doi: 10.1007/978-0-387-85359-8. Hongyu Zhang. Discovering power laws in computer programs. Information Processing & Management, 45(4):477–483, 2009. doi: 10.1016/j.ipm.2009.02.002.
A
Systems Overhead
The q-log method rescales the per-term IDF vector once at index-load time and leaves the query path unchanged. We measure the end-to-end overhead on CoIR-Go (182,440 documents, V = 144,938 vocabulary tokens) across 5 trials of 1,000 queries with top-100 retrieval; values reported as median ± MAD with gc.disable(). Figure 11 visualises the overhead breakdown summarised in the table below. 17
time (ms, log scale)
BM25 q-log BM25
103
101
10−1 index build
q-log rescale
query p50
query p95
Figure 11: Systems overhead on CoIR-Go (182,440 docs, V=144,938). q-log BM25 adds a single O(|V | + nnz) sparse-matrix pass at index time (median 117.7 ms, MAD 1.4 ms over 5 trials). Index-build wall clock is within +7.06% of BM25. Query latency (top-100, 1000 queries) is within measurement noise (p50 ∆ = +0.18%, p95 ∆ = +2.29%). q-log BM25 and BM25 share the exact same CSC scoring code path at query time—the only change is the per-term IDF factor baked into the CSC data array at index time.
Mean test recovery
1 0.5 0 −0.5 −1
↓
−c
1 A:
ok ht B:
1
c df fra −c
↓
↓
/α
5 ≤
C:
c 1−
D:
+1
og /l
T
on
A
E:
og /l +1
T
B on
Figure 12: Distribution of mean-test recovery across all 63 = 20 three-train / three-test partitions of the six CoIR code languages, per candidate predictor form. The winning form A (q = 1 − c htok, blue) is the only one-parameter form whose IQR sits entirely above zero; df-only forms B, C, and E tip below zero on a full quartile or more. Arrows mark whiskers clipped off-scale (true minima between −15 and −26). Form A is the deployed predictor in eq. (4).
Index build (s) q-log rescale (ms) Index size (MB, pickled) Query p50 (ms) Query p95 (ms) Peak RSS at query (GiB)
BM25
q-log BM25
∆
1.82 — 21.7 0.693 ± 0.00 1.489 ± 0.01 0.37
1.95 117.74 ± 1.41 21.7 0.694 ± 0.00 1.523 ± 0.01 0.37
+7.06% — +0.00% +0.18% +2.29% ≈0
Index build adds +7% wall-clock, dominated by the single O(|V |+nnz) pass over the CSC indptr and data arrays (118 ms). Pickled index size is identical to BM25: the rescale overwrites per-term score values in place. Query p50 and p95 are within measurement noise; BM25 and q-log BM25 share the exact same bm25s.BM25.get_scores code path, so query-time compute is identical by construction. Peak RSS at query time is unchanged because the CSC matrix is already allocated at build time, and q-IDF changes its values rather than its shape. 18
B
Predictor: Candidate-Form Distribution
The main-text fig. 5 reports leave-one-language-out and calibration behaviour of the deployed predictor. For completeness, fig. 12 shows the distribution of mean-test recovery across all 63 = 20 three-train / three-test partitions of the six CoIR code languages, for each of five candidate predictor forms. The winning form A is the only one-parameter form whose IQR sits entirely above zero; df-only forms (B, C, E) tip below zero on at least a full quartile and were dropped on that basis, with the parsimony tie-breaker then selecting the one-parameter A over the two-parameter D.
C
Reproducibility
The main text reports the method definition, scaling results, confidence intervals, predictor protocol, and candidate-form distribution. The full q-sweep grids per language and per subset size, raw NDCG/MRR/Recall traces, and the bit-identity audit are released in the public RareCode repository associated with this submission [Radha and Goktas, 2026]. The implementation of the q-log rescale is approximately fifty lines of Python over bm25s [Lu, 2024]; the predictor adds another fifty lines of corpus-statistics computation. The retriever uses no learned scoring model or additional index, and the query path is unchanged; the predictor coefficient is fit offline.
19