arXiv:2605.18732v1 [cs.CL] 18 May 2026
Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency
Matthew L. Smith∗ International Development Research Centre Canada Samuel T. Segun Global Center on AI Governance Canada [email protected]
Jonathan P. Shock University of Cape Town South Africa Iyiola E. Olatunji SnT, University of Luxembourg Luxembourg
Tegawendé F. Bissyandé SnT, University of Luxembourg CITADEL AI Centre of Excellence, Burkina Faso [email protected]
Abstract While scaling laws govern aggregate large language model performance, no scaling law has linked factual recall to both model size and training-data composition. We evaluated 38 models on over 8,900 scholarly references evaluated by an automated reference verification system. Recall quality follows a sigmoid in the log-linear combination of model parameter count and topic representation in training data. These two variables alone explain 60% of the variance across 16 dense models from four families, rising to 74– 94% within individual families. The form matches a superposition-inspired account in which recall is gated by a signal-to-noise ratio: signal strength scales with concept frequency and the noise floor with model capacity.
1
Introduction
Factual recall errors, or confabulations, remain one of the most consequential failure modes of large language models (LLMs)[1]. Prior work has linked these failures to both model scale and the distribution of knowledge in training data, but no single framework combines both factors into a predictive scaling law for factual recall quality. We measured recall quality across 38 LLMs and 24 topics spanning five orders of magnitude in training-data representation. Using academic references as the factual recall being measured, we find that recall quality scales with the log of a signal-to-noise ratio (SNR), where signal strength is set by how frequently a concept appears in training data and the noise floor is set by model capacity. Under a superposition-inspired account, both contributions enter the SNR multiplicatively, yielding a sigmoid in their log-sum: quality = σ(α log10 P + β log10 S + γ) ∗ Corresponding author: [email protected]
Preprint.
(1)
where P is model parameter count, S is a proxy for concept frequency in training data, and σ is the logistic function. Fitted to 16 dense models across 24 topics (N = 384 model–topic observations from 3,661 evaluated references; R2 = 0.599), the relationship holds across independent model families. Across all 38 models and 8,913 evaluated scholarly references, topic representation frequency is positively associated with recall quality independently of architecture or parameter count. The remainder of this paper is organised as follows. Section 2 reviews the relevant literature. Section 3 develops the theoretical framework that motivates the sigmoid functional form. Section 4 describes the experimental design, verification pipeline, and statistical methods. Section 5 presents the empirical findings, and Section 6 interprets them. Section 8 summarises our contributions and outlines directions for future work.
2
Related Work
Scaling laws for language models. Kaplan et al.[2] established that aggregate language model performance improves as a power law in model size, dataset size, and compute. Hoffmann et al.[3] refined these relationships to show that model and dataset size should be scaled in tandem for compute-optimal training. These aggregate scaling laws describe loss on held-out text but do not decompose performance into specific capabilities such as factual recall. Knowledge capacity and superposition. Allen-Zhu and Li[4] showed that knowledge capacity scales at roughly two bits per parameter, providing a quantitative link between model size and the number of facts that can be stored. Mechanistically, Elhage et al.[5] introduced the concept of superposition—the encoding of more features than a network has dimensions—and demonstrated it in toy models. Bricken et al.[6] and Templeton et al.[7] confirmed that superposition operates in production language models via sparse autoencoder analysis. Bereska et al.[8] formalised superposition as lossy compression. Liu et al.[9] provided empirical evidence across four open-source LLM families that models operate in a strong-superposition regime where representation vectors form near-equiangular tight frames with squared overlaps scaling as 1/d (where d is model width). Knowledge frequency and the long tail. A complementary line of work has established that rarer concepts are disproportionately susceptible to recall failure. Kandpal et al.[10] showed that LLMs struggle to learn long-tail knowledge, with question-answering accuracy strongly correlated with the number of training documents containing the relevant entity. Mallen et al.[11] found that parametric memory is reliable only for popular entities, while retrieval augmentation is needed for the long tail. Sun et al.[12] confirmed this head-to-tail degradation across a range of LLMs, and Lu et al.[13] derived scaling laws for fact memorisation as a function of model size and training epochs. Yin et al.[14] formalised the phenomenon as a “law of knowledge overshadowing,” in which high-frequency knowledge suppresses retrieval of related low-frequency facts. Yuan et al.[15] developed holistic benchmarks for factual knowledge recall. Moayeri et al.[16] demonstrated systematic geographic biases in LLM factual recall, consistent with frequency-driven disparities. Memorisation and training data. Carlini et al.[17] quantified memorisation across neural language models, showing that larger models memorise more training data and that memorisation increases with the number of times a sequence appears in training. Hernandez et al.[18] established scaling laws for learning from repeated data, finding that data repetition and model size interact in predictable ways. Xiong et al.[19] provide a comprehensive survey of memorisation mechanisms, measurement, and mitigation in LLMs. This work. While these strands have established model size and knowledge frequency as independent drivers of factual recall, no prior work has combined them into a single predictive functional form. We provide this unification by showing that a sigmoid in the log-linear combination of both factors—motivated by a superposition-inspired SNR account—fits the data across four independent model families. 2
3
Theoretical Framework
The functional form fitted in Section 5 is motivated by two ingredients: the geometry of superposition in neural representations, and the statistics of concept frequencies in naturallanguage training corpora. We present this as a superposition-inspired latent signal-to-noise model rather than a derivation: each step involves an explicit assumption that we flag below, and the data are consistent with the resulting sigmoid form rather than uniquely selecting it. 3.1
Setup: superposition, interference, and the recall threshold
We use the following notation throughout. On the model side, P is the total parameter count and Neff the effective number of dimensions available for factual encoding, with Neff ≪ P and Neff ∝ P γe for some γe ≤ 1. M is the number of factual concepts encountered in training. On the data side, fi is the training-corpus frequency of concept i. The encoded signal strength of concept i is si (fi ), a monotonically increasing function of fi for which we adopt the simplest scale-free form s(f ) = f βs . (Note that βs and γe are theoretical exponents distinct from the fitted parameters β and γ.) S denotes the OpenAlex scholarly work count for a topic, serving as an empirical proxy for training-data representation at the topic level, with individual concept frequencies scaling as f ∝ S δ for some δ ≤ 1. We note that S also reflects the number of distinct facts associated with a topic (its inventory size), so the f ∝ S δ assumption with δ < 1 partially absorbs the breadth-versus-reinforcement ambiguity but does not formally separate signal strength from concept count; the framework should accordingly be read as a coarse-grained model in which S acts as a single composite proxy. When M > Neff , the network must encode concepts in overlapping directions, a phenomenon termed superposition[5] and confirmed by sparse-autoencoder studies[6, 7] as well as work formalising it as lossy compression[8]. Let vi ∈ RNeff denote the representation vector for concept i. Retrieving concept i produces a readout X ri = ⟨vi , vi ⟩ + ⟨vi , vj ⟩, (2) j̸=i
where ri is the retrieval signal, the first term is the true signal, and the second term is interference from the remaining concepts. For near-random vectors in RNeff ,√the expected squared inner√product is ∼ 1/Neff , so the typical pairwise overlap scales as ∼ 1/ Neff . We use ε(Neff ) ∼ 1/ Neff as a coarse summary of the interference floor; the aggregate interference at retrieval depends additionally on the effective number of active competitors and their relative signal strengths, so this expression should be read as a typical-case scaling rather than a tight bound. Liu et al.[9] provide empirical support across four LLM families, showing that open-source LLMs operate in a “strong superposition” regime where representation vectors form near-equiangular tight frames with squared overlaps scaling as 1/d (where d is model width). A concept is recalled when its signal exceeds this interference floor, 1 , (3) si (fi ) > ε(Neff ) ∼ √ Neff so the signal-to-noise ratio SNRi = si (fi )/ε(Neff ) controls recall. Increasing Neff (via more parameters) lowers the floor; increasing fi (via more training data on a topic) raises the signal. 3.2
Scaling law for the recall fraction
Natural-language concept frequencies are well approximated by a power law (Zipf’s law): if concepts are ranked by frequency, the k-th concept has frequency fk ∝ k −αZ , where αZ > 1 is the Zipf exponent. With this assumption and the power-law signal mapping s(f ) = f βs , −1/2 the threshold condition (Eq. 3) becomes k −αZ βs > Neff , which is satisfied for all ranks 1/(2α β ) k < k ∗ where k ∗ ∝ Neff Z s . With Neff ∝ P γe , the recall fraction Q = k ∗ /M scales as Q ∝ P γe /(2αZ βs ) . 3
(4)
A worked estimate from Liu et al.[9] is illustrative: they measure P ∼ d2.52 across LLM families, giving γe ≈ 0.40 if Neff is identified with model width d. (Total encoding capacity likely also involves depth, so this is a lower bound on γe .) For the topic axis, we incorporate the empirical proxy S by writing the frequency of the k-th concept within a topic as fk = c0 · S δ · k −αZ , where δ < 1 because additional papers on a topic typically introduce new facts rather than reinforce existing ones. Substituting into the threshold condition: −1/2
(c0 · S δ · k −αZ )βs > Neff
=⇒
1/2
k αZ βs < cβ0 s · S δβs · Neff .
Taking the S dependence at fixed P (and hence fixed Neff ), we get k ∗ ∝ S δβs /(αZ βs ) = S δ/αZ . The exponent βs cancels in this S dependence (though it remains in the P dependence of Eq. 4), so Q ∝ S δ/αZ . (5) Combining the two axes, Q = C · P γe /(2αZ βs ) · S δ/αZ .
(6)
The three power-law assumptions (s ∝ f βs , Neff ∝ P γe , f ∝ S δ ) cannot be independently verified from our data: only the compound exponents are observable, absorbed into the fitted α and β of Equation 1. All three must be power laws for the log-linear form to hold, and the high R2 of within-family fits constrains how far they can deviate. At very small model sizes the Neff ∝ P γe assumption likely breaks down: the floor regime, in which models produce template fabrications rather than noisy recall, is consistent with Neff falling effectively to zero below a family-specific threshold. 3.3
Mapping to continuous quality
The recall fraction Q = k ∗ /M is bounded in [0, 1] by definition, but Equation 6 can exceed 1 for sufficiently large P and S, indicating a regime where capacity exceeds the concept inventory and quality saturates. Our experiment measures a continuous quality score on [0, 1] via free recall: each model produces 10 references per topic, and we score field-byfield accuracy. The measured quantity is therefore an order-statistic over the model’s most-confident outputs, not the population recall fraction Q itself; if the model can rank concepts by internal signal strength, easy/canonical references will be over-represented and the empirical mapping inherits any selection-induced bias. We use a logistic link as a tractable, bounded mapping between the latent log Q and observed quality, motivated by but not derived from the superposition account: quality = σ a · log10 Q + b , (7) where σ(x) = 1/(1 + e−x ) and a, b are empirical constants. This produces three regimes. For log Q ≪ 0 (the floor), σ(x) ≈ ex , so quality ∝ Q a/ ln 10 —a power law approaching zero. Near the inflection point (the ramp), σ is approximately linear, yielding the tangent-line form quality ≈ m · log10 (P ) + n · log10 (S) + c,
(8)
which is the small-curvature limit of Equation 1. The slope of the logistic at the inflection point is σ ′ (0) = 1/4, so m = α/4, n = β/4, and c = 1/2 + γ/4. For log Q ≫ 0 (the ′ ′ ceiling), σ(x) ≈ 1 − e−x , so the distortion 1 − quality ∝ P −m S −n , where m′ , n′ inherit the factor a/ ln 10 from the sigmoid argument. This recalls the rate–distortion framework of Shannon[20], in which distortion decays exponentially with rate (D ∝ 2−2R for a Gaussian source); the ceiling regime of our logistic mapping takes the same form with an effective exponent set by the underlying scaling parameters. The fitted parameters α and β in Equation 1 therefore absorb both the theoretical exponents (γe , αZ , βs , δ) and the linearisation factor a. The sigmoid derivative σ ′ (z) = q(1 − q), with q denoting quality, predicts an inverted-U in the effective content sensitivity neff ≡ ∂ quality/∂ log10 S: it peaks at q ≈ 0.5 and declines toward both extremes. 4
3.4
Empirical constraints on the Zipf exponent and the reference slope
From the OpenAlex concept hierarchy (65,026 concepts), OLS regression yields αZ = 1.23 with bootstrap 95% CI [1.18, 1.30], confirmed by maximum-likelihood estimation[21] at αZ = 1.24. Independently, Michaud et al.[22] measured αZ ≈ 1.24 for the rank–frequency distribution of learned “quanta” in a Pythia model—a strikingly close agreement, given that one estimate is derived from external scholarly concept frequencies and the other from internal model behaviour. A rolling-window analysis reveals that αZ varies from ∼1.0 at broad fields to ∼3–4 at niche topics. The threshold derivation (Eq. 4) yields a slope γe /(2αZ βs ) for log Q vs log P . Setting the canonical reference values γe = βs = 1 gives a benchmark slope mmax = 1/(2αZ ) ≈ 0.41. We emphasise that this is a reference value rather than a true upper bound: γe < 1 is plausible (the Liu et al. width-only estimate gives γe ≈ 0.40), and βs is unconstrained. The logistic mapping introduces the additional linearisation factor a, which the fitted α also absorbs. With these caveats, three of four within-family log-linear slopes cluster tightly at 0.22–0.24 (Mistral 0.218, Llama 0.224, Gemma 0.238), or 54–59% of the canonical-reference mmax = 0.407, consistent with models using roughly half their parameter budget for nonfactual capabilities relative to that reference. The Qwen3 series is an outlier (slope 0.366, or 90% of mmax ), but this reflects its smallest member (Qwen3-8B, q = 0.054) sitting in the floor regime in which the Neff ∝ P γe assumption breaks down; the steep apparent slope is partly the climb out of that floor rather than a clean within-family scaling estimate (Table 1). In summary, the Zipf exponent αZ sets the reference slope against which encoding efficiency is measured. Table 1: Reference-slope quantities under different Zipf-exponent estimates. Empirical comparators: mLlama = 0.224, n = 0.080 (cross-family OLS). Efficiency is m/mmax with mmax = 1/(2αZ ) at the canonical reference γe = βs = 1.
3.5
Scenario
αZ
mmax
Efficiency
Canonical Zipf 24-topic OLS Global MLE
1.00 1.23 1.24
0.500 0.407 0.403
44.8% 55.0% 55.6%
Predictions
The framework yields five predictions, each consistent with the data: 1. Within-family log-linear scaling (R2 = 0.74–0.94 across four families with ≥ 3 model sizes each: Llama n = 6, Gemma n = 4, Mistral n = 3, Qwen n = 3; treated as descriptive rather than inferential given the small per-family n). 2. Cross-topic log-linear scaling: a positive content slope n in Eq. 8 across all four families (Llama, Gemma, Mistral, Qwen each show n > 0). 3. Gradual degradation across verification categories as capacity decreases. 4. Empirical slope below the canonical reference: m < 0.41. 5. Sigmoid saturation: quality-vs-log P traces a logistic curve with floor and ceiling regimes (Fig. 4).
4
Methods
4.1
Model selection and reference generation
The primary analysis uses 16 dense models spanning 1B to 405B parameters across four families (Llama, Gemma, Qwen, Mistral), yielding 3,730 evaluated references. An additional 22 model runs (MoE architectures, chain-of-thought variants, and closed-source models with unknown parameter counts) were collected for supplementary analyses, bringing the total to 38 runs and 8,913 evaluated references (Table 2; Appendix Table A3). Each model produced 10 scholarly references for each of 24 topics drawn from six thematic groups, each containing four topics at decreasing specificity (Appendix Table A4). This 5
creates a gradient in training-data representation from millions of scholarly works (broad topics) to fewer than 50 (specific topics). All non-thinking models were run at temperature 0 for deterministic output. Thinking models (Qwen3 think, DeepSeek R1) do not accept a temperature parameter and use the model’s default reasoning mode; these variants are excluded from the primary analysis. The prompt template was identical across all model runs (verbatim, with {topic} substituted per request): List 10 different relevant scholarly references (journal papers, conference papers, technical reports, or dissertations) about {topic} ## RULES: 1. Use standard APA citation format: Author(s) (Year). Title. Journal/Publisher. 2. Provide 10 distinct references –- no duplicates. 3. Only provide the list. No commentary, questions, or explanations.
The Qwen3 nothink variants additionally prepend the Qwen-specific control token /no_think on its own line to disable the model’s default reasoning mode; the rest of the prompt is unchanged. 4.2
Reference collection and deduplication
Reference counts are summarised in Table 2. Of 9,120 candidate references requested (38 × 24 × 10), 8,913 were produced and submitted to SourceVerify for authentication (Section 4.3); the 207-reference shortfall reflects cells where the model returned fewer than 10 separable APA citations (concentrated in the smallest models, plus one refusal cell—GPT-5 Nano on “Biometric voter registration”—and one zero-output cell—DeepSeek V3 on “School dropout”). A further 84 normalized-title duplicates within model×topic were collapsed during matching, yielding 8,829 analysed references across 910 of the 912 populated model–topic cells. Deduplication is performed only within (model, topic) cells: each cell corresponds to a single prompt, so a title repeated within one model’s response to one topic is one recall event scored once, whereas the same title produced under a different topic prompt is treated as a separate recall event because it responds to a different question. Title normalization for the dedup match strips case, punctuation, and whitespace, so minor formatting drift is collapsed. For the 16 dense models used in the primary sigmoid fit, 3,730 of 3,840 candidates (97.1%) were produced and all 16 models covered all 24 topics, so the N = 384 model–topic cell structure is preserved. Table 2: Reference counts from request to analysis. Counts are exact and reconcile in both columns: full set 9,120 − 207 − 84 = 8,829; dense-16 subset 3,840 − 110 − 69 = 3,661. The dense-16 subset is the basis for the primary sigmoid fit (Fig. 4). Stage
Full set (38)
Dense-16 subset
9,120
3,840
Produced and evaluated
8,913 (−207)
3,730 (−110)
Analysed
8,829 (−84)
3,661 (−69)
Requested
4.3
Loss source 38 (16) models × 24 topics × 10 references Cells returning < 10 separable APA citations; 1 refusal cell (GPT-5 Nano, biometric voter registration); 1 zero-output cell (DeepSeek V3, school dropout) Normalized-title duplicates collapsed within model×topic
Authentication
Each reference was submitted to SourceVerify[23], which (i) extracts the key reference details, (ii) searches OpenAlex, Google Scholar, Google Books, and DOI registries for candidate matches, and (iii) returns field-level comparisons for title, authors, year, venue, and identifier. 6
Each field is classified as match, abbrev, contains, contradiction, unconfirmed, or absent. We compute a continuous authenticity score as P wi · vi i authenticity = max 0, P (non-absent fields) (9) i wi with field weights title (0.25), identifier (0.25), authors (0.20), year (0.15), venue (0.15) and verdict scores match = +1.0, abbrev = +0.75, contains = +0.5, unconfirmed = 0.0, contradiction = −1.0. Absent fields are excluded from both numerator and denominator. Thus, a contradicted field counts negatively, whereas a field SourceVerify could not assess does not penalize the score. 4.4
Authentication validation
A total of 288 references were rated by four human reviewers (301 ratings; 13 double-rated, with 100% pairwise binary agreement). Human verdicts were real (yes or yes-with-errors), ambiguous, or fake (no); treating ambiguous as fake, SourceVerify agreed with humans on 94.4% of ratings (Cohen’s κ = 0.887; recall 88.9%, precision and specificity both 100%; Fig. 1). The four SourceVerify status categories separate cleanly by authenticity (Fig. 1): verified and verified-with-error contain only real papers, unverified contains almost exclusively non-real papers, and needs-human is the genuine grey zone where human adjudication adds value. All 17 human–SourceVerify disagreements were false negatives (real papers SourceVerify could not confirm). All 17 of these references also had at least one factual defect (wrong venue, year, coauthor, or drifted title). SourceVerify never certified a fabricated reference as real, so reported authenticity and quality scores cannot be inflated by retrieval false positives.
SourceVerify status vs human authenticity verdict (n = 301 ratings, 4 reviewers) verified
72
1.0
3
75
verified-w-err
37
24
needs-human
2
12
61
2
50
66
0.6
share of row
SourceVerify status
0.8
0.4
0.2
unverified
1
2
6
90
99 0.0
real
real (errors)
ambiguous
Human verdict
fake
Figure 1: SourceVerify status versus human authenticity verdict. Confusion matrix for 301 independent ratings (288 unique references; 13 double-rated) by four human reviewers across four SourceVerify status categories. Both verified (75/75) and verified-with-error (61/61) contain exclusively real papers (100%); unverified is 97% not real with 3 real papers missed; needs-human is the genuine grey zone (52/66 not real). Treating ambiguous human verdicts as not real (n = 301), binary precision is 100% and specificity is 100%; all 17 disagreements are SourceVerify false negatives.
4.5
Relevance scoring
Each reference was judged for topic relevance by Kimi K2 (32B active MoE, temperature 0) on a three-level scale: YES (1.0), PARTIAL (0.50), or NO (0.0). Two independent human raters (n = 347 ratings across 295 stratified references) achieved 96.3% binary agreement with the automated judge (Cohen’s κ = 0.394 three-level, weighted 7
κ = 0.432). Nearly all disagreement occurred on the YES-vs-PARTIAL boundary; K2 never rated YES when a human rated NO (0/347). A sweep of PARTIAL weights from 0.0 to 1.0 confirmed robustness to this choice: the model rank order against the PARTIAL= 0.50 baseline is preserved (Spearman ρ ≥ 0.997 across the full range; Appendix Table A6), with sigmoid R2 ranging from 0.504 to 0.606 around the main-fit value of 0.599 (log-linear R2 from 0.481 to 0.573). Combined quality is the product of authenticity and relevance: quality = authenticity × relevance. 4.6
Statistical methods
OpenAlex scholarly work counts per topic (queried 8 March 2026 via the title_and_abstract.search filter) served as the proxy for training-data representation (S in Equation 1). Log-linear fits use ordinary least squares in log10 space; the sigmoid fit uses nonlinear least squares (Levenberg–Marquardt). Within-family fits used n = 6 model sizes (Llama, including two 70B variants and the 405B base together with its Hermes fine-tune), n = 4 (Gemma), or n = 3 (Mistral, Qwen). Cross-family fits included only dense models to avoid confounding by MoE routing variance. Reported standard errors and R2 values from the sigmoid fit assume independent model–topic residuals; clustering by model and by topic group would yield wider intervals, and we accordingly treat the reported uncertainties as point-estimate descriptive summaries rather than inferential bounds. The continuous authenticity score and a binary title-match alternative (a single yes-or-no question requiring no field-level judgements) yield near-identical sigmoid fits (R2 = 0.599 and 0.604 respectively), suggesting that the moderate explained variance reflects the underlying phenomenon rather than measurement choice. The remaining ∼40% of unexplained variance is attributable to factors beyond parameter count and topic frequency: models with the same parameter count but different training (e.g. Llama 405B base vs Hermes fine-tune, or Llama 8B vs Qwen3 8B nothink) show quality differences of up to 0.23 that neither P nor S can capture. 4.7
Citation-count analysis
For correctly recalled references, the number of times each has been cited (its citation count) provides a within-topic test of the training-data-representation hypothesis: if recall is driven by frequency in training data, then as model capacity grows, the marginal recalled reference should sit further down the citation tail. Higher-cited (better-represented) papers are recalled first, lower-cited (less-represented) ones only once the noise floor drops. Inclusion was restricted to references with SourceVerify status verified or verified-with-error (per the IRR audit, both buckets are 100% real papers). For each such reference, we extracted the paper title from the model’s APA-formatted output and queried the OpenAlex /works endpoint, accepting the top-ranked result only when content-word overlap (a small stopword list stripped before comparison) was at least 50%. The reported citation count is OpenAlex’s cited_by_count field; OpenAlex undercounts books and grey literature relative to Google Scholar, but the relative ranking of works by citation impact is reliable. Two exclusions apply: ∼ 10% of candidate references could not be confidently matched in OpenAlex (no hit, or low-overlap top hit), which biases reported medians upward (unmatched papers are disproportionately obscure) and makes the small-vs.-large-model contrast a conservative lower bound; and models contributing fewer than 50 matched references were excluded to keep the per-model median estimable. Per-model 95% confidence intervals on the median were obtained by 10,000-resample bootstrap, and the cross-model log–log fit weighted each point by the inverse variance of its bootstrap-derived standard error.
5
Results
5.1
Log-linear scaling with model size
Factual recall quality increases log-linearly with parameter count across 16 dense models spanning 1B to 405B (R2 = 0.794; Fig. 2a). The relationship holds across four independent model families with clear systematic offsets. Llama models consistently exceed the cross8
family trend while Gemma and Qwen models fall below it, indicating that factors beyond parameter count, such as training procedure or data curation, shift the intercept without changing the slope. Extending to 28 models across dense and Mixture of Expert (MoE) architectures, with MoE models plotted on total rather than active parameter count, preserves the log-linear trend (R2 = 0.712; Fig. 2b), consistent with total parameters, rather than active parameters, dominating the interference noise floor under a superposition account. (a) 16 dense models
Quality (authenticity x relevance)
0.8
(b) 28 dense + MoE on total params 1.0
linear fit, R 2 = 0.794 Llama Gemma Mistral Qwen
0.8
Quality (authenticity x relevance)
1.0
0.6
0.4
0.2
0.0
linear fit, R 2 = 0.711 Llama Gemma Mistral Qwen MoE
0.6
0.4
0.2
0.0 100
101
Total parameters (billions)
102
100
101
102
Total parameters (billions)
103
Figure 2: Factual recall quality scales log-linearly with model size across architectures and training generations (e.g., Llama 3.1, 3.2, & 3.3). (a) Quality versus parameter count for 16 dense models from four independent families (Llama, Gemma, Mistral, Qwen). Dashed line shows log-linear fit (R2 = 0.794). Systematic offsets between families reflect differences in training procedure rather than scale. (b) The relationship extends to 28 models across all architectures when plotted against total parameter count (R2 = 0.712). MoE models (diamonds) follow the same trend as dense models, suggesting total rather than active parameters dominate the noise floor.
5.2
Topic representation and recall quality
Topic representation is associated with recall quality across nearly all 38 models tested, independent of architecture or parameter count. 37 of 38 models show a positive Spearman correlation between quality and log10 S (median ρ = 0.55); the single outlier is Grok 3 (ρ = −0.04). The near-zero correlation for DeepSeek V4 Pro (ρ = 0.02) is consistent with ceiling compression: at quality = 0.847, nearly all references are correct regardless of topic frequency, attenuating the correlation (Fig. 3). 5.3
Sigmoid fit
We fit a sigmoid to N = 384 model–topic observations from 3,661 references generated by 16 dense models spanning 1B to 405B parameters across 24 topics. α = 1.48 ± 0.09, β = 0.46 ± 0.04, γ = −5.19 ± 0.31 (all |t| > 11, p ≈ 0); R2 = 0.599. Model size alone (log10 P ) accounts for 42.1% of variance; topic frequency (log10 S) adds a further 17.8% (F1,381 = 169.6, p ≈ 0), so both axes contribute independently to recall quality. We caution that the standard errors and F -statistic above treat the 384 model–topic observations as independent; clustering by model and by topic group would yield wider intervals (see Section 4). 5.4
Robustness: binary title-match metric
To test whether the unexplained variance lies in the measurement rather than the phenomenon, we simplified the scoring to a single binary question: does the model recall the exact title of a real paper? Fitting the same sigmoid to this binary title-match metric yields α = 2.08 ± 0.15, β = 0.52 ± 0.06, γ = −6.05 ± 0.44 (all |t| > 9) with R2 = 0.604, similar to the continuous score despite using strictly less information. The near-identical fit suggests that the moderate explained variance reflects the underlying phenomenon rather than measurement noise. The marginally steeper α (2.08 vs 1.48) is consistent with binary title recall being more sensitive 9
Quality (authenticity x relevance)
0.8
0.6
0.4
0.2
np r Bio ogra me ms Ins tric in r ec u tic Cli ide M vote ral a ma i -tre ni- r re rea teate grid gist s sm art M d b ele rati ag icro ed ctri on ric fin ne fic ult an ts ati ure ce for on for loa ma sm n re laria all pa ho ym Ru lder ent ral fa ele rme ctr rs if V ica Ma oter tion Cli l ari tur ma a p nou te r ch Mo even t an bil tio ge eb n ad ap Sch ank ta oo in Dig tion l dro g ita in a po l fi gr ut n ic De ancia ultu mo l s re cra erv tic ice e s En Gir lecti vir ls e on on du s me ca nta tio Po l Sc n lit ie Inf ical S nce ec cie t Re ious nce ne dis wa ea ble se en Ec ergy Cli ono ma mi te cs ch a Ed nge uc ati on He alt h En erg y
0.0
Sc
ho
ol
dro
po
ut
pre
ve n
tio
Model (bottom -> top: increasing mean quality)
1.0
Llama 3.1 405B Hermes Llama 3.1 405B base Llama 3.1 70B Mistral Medium 3.1 Llama 3.3 70B Mistral Large 2 Gemma 4 31B Llama 3.1 8B Mistral Small 3.2 Qwen3 32B nothink Qwen3 14B nothink Gemma 3 27B Gemma 3 12B Llama 3.2 1B Gemma 3 4B Qwen3 8B nothink
Topic (left -> right: increasing OpenAlex representation)
Figure 3: Topic representation drives recall quality across model families. Recall quality across 16 dense models and 24 topics, sorted by topic representation frequency (left to right) and model quality (bottom to top). Colour encodes quality (0–1), with red indicating low recall, yellow the 0.5 threshold, and green high recall. The gradient is consistent with topic frequency being positively associated with recall quality across model families.
to model size than the continuous composite score, as expected for an all-or-nothing retrieval metric. 5.5
Citation-count gradient
If highly cited works are referenced more often in training data, their bibliographic details should be encoded with stronger parametric redundancy. Recall gated by signal strength then predicts that small models will recall only the most-cited papers while larger models extend recall down the citation tail. Among correctly recalled references (SourceVerify status verified or verified-with-error) that matched a unique OpenAlex record, the median citation count decreases log-linearly with model size across 10 dense models from four families with n ≥ 50 matches each (weighted log–log slope = −0.35, R2 = 0.59, Spearman ρ = −0.79, p = 0.007). Llama 3.1 8B recalls papers with median 2,419 citations, while Llama 3.1 405B drops to 806 (base) or 589 (Hermes fine-tune), with bootstrap 95% CIs on every per-model median (Fig. 5). 5.6
Three regimes of the sigmoid
The sigmoid fit consists of three qualitatively distinct regimes. In the floor regime, models produce templated fabrications. In the ramp regime, recall quality increases log-linearly with both model size and topic representation. In the ceiling regime, gains saturate as capacity exceeds the concept inventory. At the bottom of the sigmoid, models stop producing references and begin producing templates. Llama 3.2 1B produced 215 references, of which only 22 were verifiable; eight of its nine Political Science references are the same work, Dahl’s Polyarchy (1971), regenerated with different publication years. Qwen3 8B draws from a pool of only 62 unique first-author surnames across 236 references, with “Smith” appearing in all 24 topics. These are not 10
Sigmoid fit, 16 dense models x 24 topics 1.0
Quality (authenticity x relevance)
0.8
model x topic cells (N = 384) sigma(z), R^2 = 0.599 Llama Gemma Mistral Qwen
0.6
0.4
0.2 alpha = 1.48 beta = 0.46 gamma = -5.19
0.0 5
4
3
2
1
z = log10 P + log10 S +
0
1
2
Figure 4: Factual recall quality follows a sigmoid in the log-linear combination of model size and topic representation frequency. Light blue points show individual model×topic observations (N = 384); coloured points show per-model averages by family (Llama, Gemma, Mistral, Qwen). The dashed curve is the fitted logistic σ(z), where z = α log10 P + β log10 S + γ (R2 = 0.599). Residual variance is greatest in the mid-range where models are near the recall threshold, compressing toward zero at the floor and ceiling as predicted by the sigmoid geometry.
necessarily researchers being recalled, but high-frequency surnames in the training corpus recycled as slot-fillers. The transition across regimes is gradual rather than binary: Llama 405B (base) produces 72% fully verified references and 13% unverified; at 70B these shift to 63% and 18%; at 8B, only 30% are verified while 42% are entirely unverified. Intermediate categories persist across all sizes, indicating a regime where models retrieve real papers but corrupt specific fields. This mid-range is also genuinely ambiguous because the continuous score conflates corrupted recall of a real source with coincidental overlap with an unrelated one, introducing measurement noise precisely where the sigmoid is steepest. Residual variance is greatest in the middle prediction third (SD = 0.181, 1.7× the low band at 0.108) and the high band (0.200), consistent with the steepest part of the sigmoid having the most measurement-induced noise and ceiling-side compression flattening the high band. At the top of the sigmoid, the largest closed-source models (Claude Opus 4.6, GPT-5, GPT-5.4) plateau at 0.76–0.81 quality, while DeepSeek V4 Pro, an open-weights MoE model with 1.6T total parameters, exceeds this range at 0.847. This suggests that the ceiling regime may not yet have been reached for the largest open-weights models. Closed-source comparisons are necessarily exploratory because exact parameter counts are undisclosed.
6
Discussion
These results extend the scaling-law framework of Kaplan et al. [2] and Hoffmann et al. [3] from aggregate performance to factual recall, providing a unified functional form and quantifying the role of topic frequency as a second independent predictor. That aggregate gains reflect improved factual encoding is expected under superposition theory[5, 9], since larger models sustain lower interference floors, bringing progressively more concepts into reliable recall. What was not previously established is that, across four independent model 11
Median citation count of recalled refs
weighted fit, slope = -0.35, R 2 = 0.59 Llama Gemma Mistral Qwen3
103
101
102
Total parameters (billions)
Figure 5: Larger models recall less-cited papers. Median citation count of correctly recalled references (SourceVerify status verified or verified-with-error) plotted against model size on a log–log scale, for 10 dense models from four families with n ≥ 50 matched references each. Error bars are bootstrap 95% CIs on the per-model median (10,000 resamples). The dashed line is a weighted log–log fit using the bootstrap-derived standard errors as inverse-variance weights (slope = −0.35, R2 = 0.59, Spearman ρ = −0.79, p = 0.007). Larger models extend recall further down the citation tail, consistent with a superposition account in which stronger parameter redundancy is required to encode low-frequency information.
families, the resulting recall quality follows a sigmoid in the latent log signal-to-noise ratio. This treats factual confabulation as a tractable consequence of compressing non-uniformly distributed knowledge into finite representational capacity. As model capacity decreases, the noise floor rises and concept representations are progressively overwhelmed, with rarer concepts falling below the recall threshold first. This account also explains the systematic intercept offsets between model families. Models trained on cleaner or more domain-relevant data produce stronger parametric representations, shifting the sigmoid leftward without changing its slope. Confabulations are therefore not random but reflect a structural inequality in how knowledge is encoded, consistent with the formal result that hallucination is an inherent limitation of LLMs[24] and with recent evidence of systematic geographic disparities in LLM factual recall[16]: high-frequency concepts sit above the interference threshold and are consistently retrieved, while low-frequency concepts remain near or below it. Scaling lowers the noise floor, bringing progressively rarer concepts into reliable recall, but this is coverage expansion, not uniform improvement. Gains for already well-represented topics saturate, while extending reliable recall into the long tail requires disproportionate increases in capacity. The topics in this study span more than five orders of magnitude in training-data representation, from climate change (S ≈ 1,200,000) to school dropout prevention in rural areas (S = 32). DeepSeek V4 Pro, the largest model tested at 1.6T total parameters, achieves a quality score of 0.90 on the former but only 0.43 on the latter. Solving Equation 1 for P at fixed quality suggests that, under the fitted dense-model relationship, raising the score on the low-frequency topic to 0.90 would require approximately 50T parameters, roughly 30× the scale of the largest model tested, and likely more in practice, since MoE architectures in our experiments underperform the dense sigmoid at equivalent total parameter counts. Aggregate benchmarks that average across the frequency spectrum therefore overestimate reliability for long-tail concepts and mask systematic disparities across topics, geographies, and languages. Data curation, targeted pre-training, and inference-time interventions should show the largest gains for concepts near the noise floor, where a modest increase in signal suffices to cross the recall threshold. For very low-frequency concepts below the floor, retrieval augmentation that 12
bypasses parametric recall entirely is the appropriate response. A direct test of the causal role of training data frequency would require controlled pre-training experiments in which concept representation is varied independently of model capacity, something the present observational design cannot support. The sigmoid parameterisation nonetheless offers a quantitative way to estimate the increase in log10 S required to push a concept across the recall threshold (σ(z) = 0.5) at a given model size, computed as ∆ log10 S = −(α̂ log10 P + γ̂)/β̂, from the fitted parameters.
7
Limitations
Single factual domain. Scholarly references are well-structured, externally indexed, and amenable to automated verification. Whether the sigmoid scaling relationship generalises to other forms of factual knowledge—commonsense facts, numerical quantities, procedural knowledge—remains untested. References may also be overrepresented in LLM training corpora relative to other factual domains, so the fitted parameters (α, β, γ) should not be taken as universal constants. Training-data opacity. The topic representation proxy S (OpenAlex work counts) is an indirect measure of training-data frequency. We do not have access to the training corpora of any model tested, so the assumption f ∝ S δ is empirically motivated but not directly verified. Models with different data-curation pipelines may induce different effective δ values, contributing to the observed family-level intercept offsets. Observational design. The study is correlational: model size and training-data quality co-vary across model generations, and we cannot manipulate concept frequency independently of model capacity. Establishing a causal role for training-data representation would require controlled pre-training experiments. Within-family sample sizes. The primary sigmoid fit pools 16 models, but within-family fits rely on as few as n = 3 model sizes (Mistral, Qwen). These slopes are therefore indicative rather than precise, and the Qwen3 outlier slope illustrates how a single model in the floor regime can dominate a small-sample fit. English only. All topics and references are in English. For low-resource languages, trainingdata representation is orders of magnitude lower, and the scaling relationship could shift substantially in both slope and intercept. Fixed prompt and decoding. All non-thinking models were evaluated with a single prompt template at temperature 0. Different prompting strategies (few-shot, chain-ofthought) or non-zero temperature could shift the sigmoid parameters, particularly for models near the recall threshold.
8
Conclusion
We have shown that factual recall quality in large language models follows a sigmoid in the log-linear combination of model parameter count and topic representation in training data. The functional form, motivated by a superposition-inspired signal-to-noise account, fits data from 16 dense models across four independent families (R2 = 0.599) and generalises across 38 models spanning dense, MoE, and closed-source architectures. The framework reveals three regimes—a floor of templated fabrication, a ramp of log-linear improvement, and a ceiling of saturation—and predicts that larger models extend recall into the citation tail rather than uniformly improving across all topics. These findings reframe factual confabulations as a predictable consequence of compressing non-uniformly distributed knowledge into finite representational capacity, rather than as random or idiosyncratic failures. Natural extensions include other factual domains beyond scholarly references, multilingual settings, and the interaction between parametric recall and retrieval augmentation. 13
Acknowledgements This work was carried out with support from the Artificial Intelligence for Development (AI4D) Africa programme, with financial support from Canada’s International Development Research Centre (IDRC) and the UK’s Foreign, Commonwealth & Development Office (FCDO). The views expressed herein do not necessarily represent those of IDRC or its Board of Governors, or those of FCDO.
Data availability. Verified reference data for all open-source model runs, relevance judgements, and topic metadata are available at https://github.com/matthewlongshore/ predictable-confabulations. Closed-source model outputs (GPT-5 family, Claude family) are included as aggregate scores but raw outputs are not re-distributable under API terms.
Code availability. All analysis scripts for verification, scoring, sigmoid fitting, and figure generation are available at the same repository.
Author Contributions M.S. designed the study, collected model outputs, ran verifications, performed the primary data analysis, and helped to write the manuscript. J.S. developed the theoretical framework, performed the sigmoid and scaling analysis, and wrote the manuscript. S.T.S., T.F.B., and I.E.O. contributed to study design and manuscript revision. All authors reviewed and approved the final manuscript. Use of AI tools. During preparation of this manuscript, the authors used Anthropic’s Claude (Opus and Sonnet models, accessed November 2025–May 2026) to assist with manuscript editing, consistency checks across drafts, simulated peer review of theoretical claims, and LaTeX debugging. The model was also used to draft analysis scripts that the authors reviewed, executed, and verified independently. All scientific claims, the experimental design, the data collection and verification pipeline, and the theoretical framework were developed by the authors. The authors reviewed and edited all AI-assisted text and take full responsibility for the content of the manuscript.
Competing Interests M.S. is a co-founder of SourceVerify, the reference verification service used to grade the model outputs in this study, and therefore has a financial interest in its success. To mitigate this conflict, the inter-rater reliability evaluation (Methods, “Authentication validation”) was performed independently by the other four authors (J.S., S.T.S., T.F.B., and I.E.O.); M.S.’s ratings were excluded from all reported precision, recall, specificity, accuracy, and Cohen’s κ figures. The downstream sigmoid fitting and scaling analyses were conducted on the verified reference outputs without further input from SourceVerify, and the full reference-level dataset (including raw verdict fields and human-review labels) is released alongside the paper to permit independent re-analysis. The remaining authors declare no competing interests. 14
A
Supplementary Tables
Table A3: All 38 models tested. Active parameters listed for MoE; total in parentheses. Quality = authenticity × relevance. Scores use contradiction penalty (−1.0). Model
Family
Params
Auth.
Quality
Llama 3.2 1B Gemma 3 4B Llama 3.1 8B Qwen3 8B (nothink) Qwen3 8B (think) Gemma 3 12B Qwen3 14B (nothink) Qwen3 14B (think) Mistral Small 3.2 Gemma 3 27B Gemma 4 31B Qwen3 32B (nothink) Qwen3 32B (think) Llama 3.1 70B Llama 3.3 70B Mistral Large 2 Mistral Medium 3.1 Llama 3.1 405B (base) Llama 3.1 405B Hermes
Llama Gemma Llama Qwen Qwen Gemma Qwen Qwen Mistral Gemma Gemma Qwen Qwen Llama Llama Mistral Mistral Llama Llama
1B 4B 8B 8B 8B 12B 14B 14B 24B 27B 31B 32B 32B 70B 70B 123B 250B 405B 405B
0.087 0.100 0.425 0.067 0.139 0.143 0.274 0.219 0.396 0.273 0.365 0.362 0.263 0.724 0.570 0.449 0.678 0.779 0.879
0.078 0.082 0.322 0.054 0.107 0.113 0.227 0.174 0.309 0.220 0.326 0.285 0.213 0.607 0.446 0.364 0.560 0.638 0.703
Dense Dense Dense Dense Dense+CoT Dense Dense Dense+CoT Dense Dense Dense Dense Dense+CoT Dense Dense Dense Dense Dense Dense (ft)
GPT-OSS 120B Qwen3 30B-A3B MiniMax M2.5 Mixtral 8x7B Llama 4 Scout Llama 4 Maverick Qwen3.5 Kimi K2 Mixtral 8x22B DeepSeek R1 DeepSeek V3 DeepSeek V4 Pro
— Qwen — Mistral Llama 4 Llama 4 Qwen — Mistral DeepSeek DeepSeek DeepSeek
5.1B (120B) 3B (30B) 10B (230B) 13B (47B) 17B (109B) 17B (400B) 17B (397B) 32B (1T) 39B (141B) 37B (671B) 37B (671B) 49B (1600B)
0.162 0.215 0.354 0.304 0.269 0.735 0.711 0.672 0.571 0.758 0.851 0.891
0.136 0.195 0.309 0.266 0.209 0.605 0.665 0.570 0.493 0.626 0.754 0.847
MoE MoE MoE MoE MoE MoE MoE MoE MoE MoE+CoT MoE MoE
GPT-5 Nano GPT-5 Mini Grok 3 GPT-5 GPT-5.4 Claude Sonnet 4.6 Claude Opus 4.6
GPT GPT — GPT GPT Claude Claude
unknown unknown unknown unknown unknown unknown unknown
0.548 0.795 0.749 0.876 0.884 0.861 0.924
0.432 0.623 0.690 0.762 0.761 0.693 0.806
unknown unknown unknown unknown unknown unknown unknown
15
Arch.
Table A4: The 24 research topics organised by thematic group and specificity level. OpenAlex work counts (queried 8 March 2026) serve as proxy for training-data representation. Group
Topic
OpenAlex Works
Economics
Economics Digital financial services Mobile banking Microfinance loan repayment
Education
Education Girls education School dropout School dropout prevention in rural areas
5,921,253 71,707 19,413 32
Energy
Energy Renewable energy Rural electrification Mini-grid electrification
8,631,334 501,950 9,391 747
Environ. Sci.
Climate change Environmental science CC adaptation in agriculture Climate-smart agriculture for smallholders
1,222,665 247,835 25,815 1,406
Health
Health Infectious disease Malaria prevention Insecticide-treated bed nets for malaria
8,504,910 469,401 13,463 1,331
Political Sci.
Political science Democratic elections Voter turnout Biometric voter registration
767,282 34,424 17,971 1,341
285,942 48,258 10,283 171
Table A5: Geometric interpretation of the scaling law. Term P S m n c
Interpretation √ Noise floor 1/ N Signal amplitude Encoding efficiency Signal sensitivity Baseline SNR offset
Determined by Architecture Training data Training procedure ≈Constant (dense) Data quality + procedure
Table A6: Relevance threshold robustness. Key metrics across PARTIAL weight values (N = 384 model–topic observations, 16 dense models with known parameter counts). Spearman ρ is the rank correlation of model orderings against the PARTIAL = 0.50 baseline. PARTIAL wt 0.00 0.25 0.50 0.75 1.00
sigmoid R2
log-linear R2
Spearman ρ
0.504 0.565 0.599 0.606 0.594
0.481 0.536 0.566 0.573 0.564
1.000 1.000 1.000 0.997 0.997
References [1] Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint, 2025. arXiv:2509.04664. 16
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint, 2020. arXiv:2001.08361. [3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint, 2022. arXiv:2203.15556. [4] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint, 2024. arXiv:2404.05405. Published at ICLR 2025. [5] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. URL https://transformer-circuits.pub/2022/toy_model/index.html. [6] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac HatfieldDodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/monosemantic-features. [7] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity. [8] Leonard Bereska, Zoe Tzifa-Kratira, Reza Samavi, and Efstratios Gavves. Superposition as lossy compression: Measure with sparse autoencoders and connect to adversarial vulnerability. arXiv preprint, 2024. arXiv:2512.13568. [9] Yizhou Liu, Ziming Liu, and Jeff Gore. Superposition yields robust neural scaling. In Advances in Neural Information Processing Systems 38 (NeurIPS), 2025. Best Paper Runner-Up. arXiv:2505.10465. [10] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv:2211.08411. [11] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. arXiv:2212.10511. [12] Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2024. arXiv:2308.10168. [13] Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling laws for fact memorization of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. arXiv:2406.15720. [14] Yuji Zhang, Sha Li, Cheng Qian, Jiateng Liu, Pengfei Yu, Chi Han, Yi R. Fung, Kathleen McKeown, Chengxiang Zhai, Manling Li, and Heng Ji. The law of knowledge overshadowing: Towards understanding, predicting, and preventing LLM hallucination. In Proceedings of the Eighth FEVER Workshop at ACL 2025, 2025. arXiv:2502.16143. 17
[15] Jiaqing Yuan, Lin Pan, Chung-Wei Hang, Jiang Guo, Jiarong Jiang, Bonan Min, Patrick Ng, and Zhiguo Wang. Towards a holistic evaluation of LLMs on factual knowledge recall. arXiv preprint, 2024. arXiv:2404.16164. [16] Mazda Moayeri, Elham Tabassi, and Soheil Feizi. WorldBench: Quantifying geographic disparities in LLM factual recall. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 1211–1228, Rio de Janeiro, Brazil, 2024. ACM. doi: 10.1145/3630106.3658967. [17] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint, 2023. arXiv:2202.07646. Published at ICLR 2023. [18] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability of learning from repeated data. arXiv preprint, 2022. arXiv:2205.10487. [19] Alexander Xiong, Xuandong Zhao, Aneesh Pappu, and Dawn Song. SoK: The landscape of memorization in LLMs: Mechanisms, measurement, and mitigation. arXiv preprint, 2025. arXiv:2507.05578. [20] Claude E. Shannon. Coding theorems for a discrete source with a fidelity criterion. IRE National Convention Record, 7(4):142–163, 1959. [21] Aaron Clauset, Cosma Rohilla Shalizi, and Mark E. J. Newman. Power-law distributions in empirical data. SIAM Review, 51(4):661–703, 2009. doi: 10.1137/070710111. [22] Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. arXiv:2303.13506. [23] SourceVerify. https://sourceverify.ai/, 2026. [24] Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint, 2024. arXiv:2401.11817.
18