arXiv CSopen access

Co-LMLM: Continuous-Query Limited Memory Language Models

Co-LMLM: Continuous-Query Limited Memory Language Models

arXiv:2607.07707v1 [cs.CL] 8 Jul 2026

Yair Feldman* Linxi Zhao Nathan Godey Dongyoung Go Yilun Hua Kilian Q. Weinberger Jennifer J. Sun Yoav Artzi Department of Computer Science Cornell University [email protected] {lz586, ng554, dg793, yh2228, kilian, jennifer.sun, yoavartzi}@cornell.edu

Abstract Limited memory language models (LMLMs) externalize factual knowledge during pre-training to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (C O -LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. C O -LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work’s restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, C O -LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40× more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

1

Introduction

Recently, there has been increasing interest in large language models (LLMs) that are trained to externalize knowledge [Ghosal et al., 2025, Zhao et al., 2026, Pouransari et al., 2026]. A particularly compelling approach is the limited memory language model [LMLM; Zhao et al., 2026], where the LLM is pre-trained to externalize knowledge into a human-readable knowledge base (KB). This design offers several advantages. The KB is interpretable and easily editable, allowing for unlearning with no utility tradeoff, and enabling easy attribution of knowledge used to the source material. Zhao et al. instantiate the LMLM approach with a relational KB, which we refer to as R EL -LMLM. Although representing a significant departure from conventional LLMs, R EL -LMLM proposes a training process that follows the common scalable next-token-prediction pre-training, with the key difference of pre-processing the data to simulate knowledge retrieval. Experiments demonstrate significantly lower perplexities compared to conventional LLMs and factuality scores similar to models several orders of magnitude larger, while retaining similar utility (i.e., NLU scores) to LLMs of the same size. * Individual author contributions are detailed in the acknowledgments.

Preprint.

Internal

Knowledge

Model

Co-LMLM

Rel-LMLM

LLM + RAG Linguistic

Competency

Linguistic

Competency doc1 text doc2 text

Document Collection

Limited

Internal

Knowledge

Model

Napoleon, birthplace

Linguistic

Competency (subj1,rel1,obj1) (subj2,rel2,obj2)

Ajaccio

Relational Knowledge Base

Limited

Internal

Knowledge

Model

fact-text1 fact-text2

Ajaccio

· Wiki

· News

· Books

· CC

Continuous-Query Knowledge Base

Figure 1: Knowledge separation across three regimes. A standard LLM with RAG retrieves over external documents but keeps factual knowledge in its parameters (left); LMLM externalizes facts to a relational KB queried with an explicit decoded query (middle); C O -LMLM externalizes facts to an unstructured index, retrieved directly from the model’s hidden state (right). However, R EL -LMLM has several key limitations that constrain the scaling of the pre-training process and the knowledge retrieval expressivity. Pre-training relies on Wikipedia, where each article is centered on a specific entity, making it straightforward to automatically annotate relational queries at scale during data pre-processing. Although Wikipedia is sufficient for a proof-of-concept demonstration, it provides no avenue to scale much further. The relational representation itself, although human readable, introduces several limitations. It restricts the data that can be externalized to items that are the object of natural language relational tuples, where both the subject and relation are mentioned previously in the text. This same mechanism limits the expressivity of the retrieval itself to queries that can be expressed as simple subject-relation natural language queries, and generating these queries incurs the added cost of decoding multiple additional tokens during inference. Finally, synthesizing the exact queries in the data pre-processing step holds retrieval pathways as constant and pre-determined, significantly restricting the expressivity of the learned retrieval mechanism. We propose continuous-query limited-memory language models (C O -LMLM), an approach that addresses these limitations, while retaining all the advantages that the LMLM design offers. The key modeling difference from R EL -LMLM is that queries are issued as single continuous vectors rather than structured relational tuples. The KB stores vector keys and string values instead of relational tuples. We jointly train the LLM Transformer for language modeling and retrieval using pre-training data pre-processed to simulate KB queries. The pre-processing specifies only when retrieval takes place and what string it returns, rather than dictating the explicit query representation. The query vector emitted by the LLM are optimized via a contrastive loss over synthesized positive and negative pairs. This allows C O -LMLM to retrieve knowledge at the cost of a single generation step, generate flexible queries, externalize a much broader set of knowledge without role or relational constraints, and generalize to pre-training data well beyond Wikipedia. We evaluate C O -LMLM against R EL -LMLM models of comparable size and standard baselines trained on the same corpus without knowledge externalization. Across model scales, C O -LMLM achieves lower perplexity than standard models, substantially improves factual precision over both standard and R EL -LMLM, while preserving downstream NLU performance. At the 360M scale, C O LMLM achieves lower perplexities than off-the-shelf HF/S MOL LM2-360M, which is pre-trained on 40× more data, and SimpleQA-verified [Haas et al., 2026] scores that are in line with gpt-4o-mini and higher than Claude Sonnet 4.5. At the same time, C O -LMLM retains the controllability benefits of LMLM: its external memory remains editable and supports direct unlearning through database operations. Finally, extending pre-training from Wikipedia to FineWeb-Edu further improves factual precision, showing C O -LMLM’s benefit from scaling the pre-training data. Code, data, and models are available at: https://lil-lab.github.io/co-lmlm-web.

2

Related Work

Parametric Knowledge and its Limitations Pre-trained language models store factual knowledge implicitly in their parameters and recall it at generation time [Petroni et al., 2019, Roberts et al., 2020]. This form of storage is inherently limited: factual recall scales with model size and exposure frequency, making long-tail knowledge difficult to capture [Kandpal et al., 2023, Mallen et al., 2023]. Moreover, factual associations are entangled with linguistic representations, making individual facts difficult to attribute, edit, or remove, and contributing to hallucinations and stale knowledge [Elhage et al., 2022, Dai et al., 2022, Huang et al., 2024]. They also make knowledge editing and unlearning 2

challenging: removing or modifying specific facts typically requires additional training or specialized objectives, often trading off with general capabilities [Maini et al., 2024]. MemSinks isolates memorization-specific parameters, but still keeps knowledge within the model [Ghosal et al., 2025]. These challenges motivate approaches that move factual knowledge into separate, editable stores. Externalizing Knowledge from Model Weights A growing line of work augments language models with non-parametric or semi-parametric memory, or aims to move factual knowledge more explicitly out of model parameters. kNN-LM retrieves nearest-neighbor examples from an external datastore and interpolates their probabilities with the model distribution [Khandelwal et al., 2020], while RETRO retrieves neighboring chunks and conditions on them through cross-attention [Borgeaud et al., 2022]. Other approaches introduce sparse memory layers or learned key-value memories to increase model capacity and factuality [Lample et al., 2019, Berges et al., 2025, Wu et al., 2022, Peng et al., 2023, Yang et al., 2024, Cheng et al., 2026, Tseng and Sa, 2026]. These methods can improve perplexity or factuality, but the stored information remains not directly inspectable or editable. Limited Memory Language Models Zhao et al. [2026] is the most related work to ours. They propose Limited Memory Language Models (LMLMs), which externalize factual knowledge into a textual relational KB during pretraining rather than encoding it in model parameters. We refer to this approach as R EL -LMLM to distinguish it from our continuous-query formulation. R EL -LMLM enables faster pre-training, interpretable retrieval, and direct editing, but its relational design limits the external memory coverage: facts must be expressible as Wikipedia-style subject-relation-object tuples, and retrieval depends on brittle textual queries and introduces additional generation token overhead. C O -LMLM preserves the benefits of externalized and editable KB while relaxing these constraints. It replaces relational tuples with free-form textual knowledge spans; and decoded relational queries with continuous queries emitted from the model’s hidden states. These queries can carry richer contextual information for retrieval, allowing C O -LMLM to scale beyond Wikipedia-style relational facts to general corpora (e.g., FineWeb-Edu), while reducing the token overhead of query generation. Retrieval-Augmented Language Models Retrieval-Augmented Generation (RAG) conditions generation on retrieved passages during inference [Lewis et al., 2020, Izacard and Grave, 2021], while retrieval-aware pretraining methods incorporate retrieval into model training [Guu et al., 2020, Borgeaud et al., 2022, Izacard et al., 2023]. Other methods let models decide when to retrieve and issue textual queries, as in Self-RAG, DRAGIN, and LMLM [Asai et al., 2024, Su et al., 2024, Zhao et al., 2026]. Our technique is related to methods that generate retrieval signals directly from internal representations rather than decoded queries [Muennighoff et al., 2025, Zhang et al., 2024a, 2025]. Unlike most retrieval-augmented methods, which add external context while leaving parametric knowledge largely intact, C O -LMLM uses retrieval to define the parametric/non-parametric boundary during pretraining: factual spans are externalized into a schema-free index, and the model learns when and how to retrieve them through a contrastive objective trained jointly with next-token prediction.

3

Method

The C O -LMLM model is a decoder-only Transformer, which functions as both a language model and a dense retriever. Figure 2 provides a high-level overview of the method. The model generates query vectors similar to tool calling, and the returned values are encoded by the model, before decoding resumes (Section 3.1). We jointly train the language modeling and retriever functions (Section 3.2), using pre-training data that is pre-processed to simulate retrieval and synthetic questions that provide supervision for the contrastive retrieval objective (Section 3.3). 3.1

Modeling and Inference

C O -LMLM performs retrieval through continuous tool calls. The external memory is a key-value store: each key is a dense vector produced by the model, and each value is a factual text span. Unlike standard tool-calling methods [Schick et al., 2023], the retrieval query is not represented as text. Instead, the query is the model’s hidden state at a special token. At inference time, the model decodes tokens autoregressively. When it emits the special <FACT> token, we read the last-layer hidden state at that position as a retrieval query, query the index, and retrieve the top-1 fact text snippet. We splice the retrieved snippet into the context immediately 3

Annotation Annotated text Original text

Lycopene is a member of the

Lycopene is a member of the

<FACT>

carotenoids

</FACT>

carotenoid

family

.

paired question:

carotenoid family.

What

is

lycopene

a

member

of?

Pretraining Next-token prediction

f1

⋯ Lycopene is

a member of the <FACT> carotenoids </FACT> carotenoid family

,

LM head

…

Bidirectional InfoNCE q

1

q1

q

2

q

3

idden states f

1

f

2

shared Model

What is lycopene a member of? <FACT-q>

f

backbone

3

Inference Prompt

Output

What family of plant compounds

Lycopene is a member of the <FACT> carotenoids </FACT>

does lycopene belong to? <fact

carotenoid family.

text>

Figure 2: Overview of C O -LMLM. Top: We annotate the pre-training corpus with fact spans and paired questions, while leaving the document text unchanged. Middle: During training, a single decoder-only Transformer processes both the annotated document and its paired questions. We optimize next-token prediction over non-fact tokens and a bidirectional InfoNCE loss that aligns the query representation at each <FACT> token with the question representation at the corresponding <FACT-q> token. Bottom: At inference, emitting <FACT> triggers dense retrieval using the hidden state at the emitted <FACT> position as the query. The retrieved fact span is inserted into the context, followed by </FACT>, and decoding continues. No textual query is decoded. after <FACT>, append a closing </FACT> special token, and resume autoregressive decoding from </FACT> onward. Each retrieval therefore costs one extra autoregressive step (the opening <FACT> token) and grows the context by the length of the retrieved span. 3.2

Training

Our training data consists of raw text in which factual spans are enclosed by <FACT> and </FACT>. Each marked span is paired with a natural-language question whose answer is the span. Figure 2 (top) illustrates a single example. Section 3.3 describes how we pre-process raw pre-training data to create this data. Our objectives are to train the model to both predict the next token and externalize the information in the delimited spans, construct a KB from all such spans, and to emit retrieval queries to fetch spans from the KB as needed. The key to training the retrieval function is contrastive query pairs between documents and questions, allowing for both positive and negative pairs. Let x be an input document with factual spans bracketed by <FACT> and </FACT>. For each factual span, we have a question Q. The question is synthesized so its answer is the corresponding bracketed span in x, and it is appended with a query marker token <FACT-q>. The retrieval vectors at the query markers appended to the questions are used as positive examples for the contrastive retrieval loss. The pre-training panel of Figure 2 (middle) visualized how x and Q are used. Let p(xt |x<t ; θ) be the probability of the token xt in the sequence document x, parameterized by θ. In our model, pθ is computed by a Transformer. We denote ϕt (x<t ; θ) as the features computed by the Transformer for index t in sequence x.1 . This is the same Transformer used for pθ . We jointly minimize Pnext-token prediction (NTP) and bidirectional contrastive losses. The NTP loss is: LNTP (θ) = − t∈M log p(xt | x<t ; θ), where M is the set of positions in x that are strictly / between <FACT> and </FACT>, inclusive of the closing </FACT>, but excluding the opening <FACT>. This NTP formulation means that we do not optimize our model to memorize the bracketed fact tokens, which represent the knowledge we aim to externalize. This is a critical design choice to externalize knowledge. We do optimize the loss of the opening <FACT> token, because the model 1We L2-normalize the feature representations as they are used as retrieval queries [Chen et al., 2020].

4

Seed annotation

Distill into two lightweight annotators Gemini 3.1 Pro frontier annotator

Seed documents

Fact Span Annotator Masked Language Model

Seed annotation

Annotate

Input: raw text

predict BIO tags per token

Lycopene is a member of the

Lycopene is a member of the carotenoid family

carotenoid family, a group of over 600 plant compounds.

O

O

O

O

O

O

B

I

,

a group of over 600 plant compounds .

O

O

O

O

B

I

O

O

Pretraining corpus

Wikipedia, FineWeb, ...

O

Fact Span Annotator

Lycopene is a member of the

<FACT q="What is lycopene a member of?"

a="the carotenoids">

over 600

Question Generator

a group of

over 600

[2]

[1]

,

plant compounds.

Causal Language Model

</FACT> plant compounds.

KV

→

<FACT q="How many plant compounds are in the

carotenoid family?" a="more than 600">

Lycopene is a member of the carotenoid family

→

carotenoid family

</FACT> , a group of

Input: text with fact spans

What is lycopene a + Generate → member of? QA for [1] the carotenoids How many carotenoids + Generate → are there? QA for [2] more than 600

Question Generator

Annotated pretraining corpus for Co-LMLM

Figure 3: Annotation pipeline. Left: A frontier model annotates a small seed set of documents in the <FACT q=... a=...>span</FACT> format. Middle: We use the seed set to train two lightweight annotators: a fact span annotator that BIO-tags fact spans in raw text, and a question generator that produces a question-answer pair for each span; the document prefix is KV-cached and reused across all spans. Right: We run the two annotator over the full pre-training corpus, converting raw documents into the annotated pre-training corpus for C O -LMLM. must know to generate it in order to trigger retrieval. The closing </FACT> is appended mechanically, so does not require optimization. The contrastive loss treats each fact query from the document x and its corresponding question Q as a positive pair, and all other questions as negatives. For a fact that starts at index t in a document x, let f = ϕt (x<t ; θ) be the query representation for retrieving this fact. Meaning, the feature representation for the <FACT> token opening this fact span. Correspondingly, for a question Q where the query marker <FACT-q> is at index t, let q = ϕt (Q<t ) be the query representation. That is, the feature representation computed for <FACT-q>. We use the InfoNCE loss [van den Oord et al., 2018], compute on the current batch. Let B = {(f (i) , q (i) )}B i=1 be the set of all paired fact document queries f (i) and question queries q (i) in the current batch. For each (f (i) , q (i) ) ∈ B, the loss terms are: exp(f (i) · q (i) /τ ) (i) ℓf →q = log PB (i) · q (k) /τ ) k=1 exp(f

exp(q (i) · f (i) /τ ) (i) ℓq→f = log PB , (i) · f (k) /τ ) k=1 exp(q

(1)

where τ is a temperature hyperparameter. We average the two directions: B

LCL = −

1 X (i) (i)
ℓf →q + ℓq→f . 2B i=1

(2)

We optimize the joint loss L = LNTP + λLCL , with λ a hyperparameter set to 0.25 by default. Building the Index We build a dense retrieval index from our pre-training data. The index pairs vector keys with fact spans. We run the pre-trained language model over the pre-training corpus, extract the L2-normalized hidden state ϕ(·; θ) at every <FACT> position, and store it as a key in a dense vector index whose value is the verbatim in-text span at that position. Indexing a new corpus therefore requires only pre-processing it (Section 3.3) and a forward pass of C O -LMLM. 3.3

Training Data Pre-processing

Training requires documents annotated with fact spans and paired questions (Section 3.2). We design an automated, scalable process to annotate raw pre-training data. The pre-processing pipeline has three stages: annotate a seed set with by prompting a state-of-the-art frontier LLM; fine-tune lightweight annotators on the seed data; and apply them to full pre-training corpus. Critically, we fine-tune smaller models because of the cost of annotating at scale with a large frontier model. Annotation Specification We annotate each training document with a set of fact spans. A fact span is a sub-string of a pre-training document x that encodes world knowledge we want our model to externalize – to retrieve rather than memorize. Each fact span is associated with two additional pieces of information: a context-dependent question that captures the retrieval intent at this position without itself carrying any factual content, and a snippet-form answer that answers the question in a form that need not match the surface form of the span. The question is used as the question Q during training. An annotated fact takes the form <FACT q={question} a={answer}>{span}</FACT>. The annotation reflects three guidelines: (a) spans are the narrowest self-contained sub-string that fully answers the question; (b) questions are written as standalone search queries that name their 5

subject explicitly, and they avoid mentioning information that appears later in the document so that no future fact leaks through the question; and (c) answers vary the surface form of the span, so the model is exposed at training time to retrieved values that differ lexically from the surrounding context. We provide more details about the prompt used for the seed annotation in Appendix C. Annotation Pipeline Figure 3 visualizes the annotation pipeline. We first use Gemini-3.1-Pro to annotate a small seed of documents using the q="..."/a="..." format defined above. The seed set is then used to distill two efficient annotators for the entire pre-training corpus. The fact span annotator is a masked language model that detects fact spans on the original text via BIO tagging,2 built on a ModernBERT encoder [Warner et al., 2025]. The question generator is a decoder-only generator that, conditioned on the document with span markers inserted, emits a question-answer pair for each tagged span; the document context is encoded once and reused across all facts. Why Two Annotators? A single end-to-end generator that produces the full annotated text in one shot is the obvious alternative. We chose the split design for three reasons. First, for efficiency reasons: the question generator only emits questions and answers, not the document text, which lets us prefill the document context once and reuse it across all of its facts. Second, predicting spans as offsets into the original text makes the annotations faithful by construction: the in-text span is always a verbatim substring of the source document, while a generative annotator can drift away from the source (i.e., hallucinate), especially on long documents.3 Finally, indexing new data is highly efficient as it only involves two forward passes (one for the span annotator and one for C O -LMLM).

4

Experiments

4.1

Experimental Setup

We train C O -LMLM using the S MOL LM2-135M and S MOL LM2-360M architectures, with a standard language model trained at each size as the S TANDARD baseline. We use a high-quality Wikipedia corpus (∼ 3B tokens)4 , following Zhao et al. [2026]. We train all models for 100K steps with a context length of 4,096 tokens. For a fair comparison with R EL -LMLM, we train two R EL -LMLM models based on the same S MOL LM2-135M and S MOL LM2-360M architectures. We provide additional R EL -LMLM training details in Appendix A.5. In addition, we evaluate the generalization of our process to general web text by training C O -LMLM and a corresponding S TANDARD baseline on 90B tokens sampled from F INE W EB -E DU [Penedo et al., 2024]. In all our retrieval experiments, we create the KB from the entire Wikipedia training corpus. This allows for a direct comparison with the R EL -LMLM KB that is constructed from this same corpus. It also guarantees the KB does not cover information not available to the baselines. The C O -LMLM KB includes ∼ 240M items, compared to R EL -LMLM’s ∼ 145M, demonstrating the less restricted, and more aggressive externalization of C O -LMLM. Appendix A.1 provides additional details. 4.2

Language Modeling Perplexity

Evaluation Setup We evaluate using perplexity on 1,000 randomly sampled Simple English Wikipedia articles.5 These articles are not included in our pretraining corpus, but their underlying facts are likely covered by our index in different surface forms, because the information in Simple Wikipedia is largely a subset of Wikipedia. This setup avoids coverage issues, and helps distinguish retrieval or annotation errors in C O -LMLM from imperfect parametric memorization in S TANDARD. The focus on Wikipedia knowledge makes the comparison to off-the-shelf models complex, because of the domain shift. The key evaluation is between the models we train, controlling for the training data. We also report numbers for off-the-shelf models, even though the comparison is not as clean. We report two perplexity variants, following Zhao et al. [2026]. Static (Oracle) assumes perfect lookup behavior by supplying the correct retrieved spans as context, giving an optimistic estimate 2 BIO stands for begin-inside-outside [Ramshaw and Marcus, 1995]. 3 E.g. in preliminary experiments a L LAMA -3.2-1B [Grattafiori et al., 2024] model trained as the single generative annotator produced faithful annotations for only 33% of the documents. 4 https://huggingface.co/datasets/allenai/dolmino-mix-1124 5 https://simple.wikipedia.org/

6

14.4

15

15

14.0

14.4 12.2

↓

9.6

10 8.0

9.1 7.7

7.4

5.8

5

360M Model

10

360M-FW

9.6

10.0

9.1

8.0 8.0

5.8

5 0

0 135M

12.0

11.1

10.5

Perplexity

Perplexity

↓

11.8 11.6

Simple-Wiki

FineWeb Dataset

Standard Standard HF Ref.

Standard Standard HF Ref. Co-LMLM-Stat.

Co-LMLM-Dyn. Co-LMLM-Stat.

(a) Perplexity across model sizes

Co-LMLM-Dyn. (Wiki 240M) Co-LMLM-Dyn. (Wiki+FW 2.2B)

(b) Perplexity across index configurations

Figure 4: Perplexity Evaluation. (a) Simple-Wiki test perplexity comparison between S TANDARD and C O -LMLM across two model sizes (135M and 360M) and two training corpora (Wikipedia only vs. Wikipedia and F INE W EB -E DU). (b) Perplexities of S TANDARD and C O -LMLM (F INEWEB ), both with 360M-FW variants, on Simple-Wiki and FineWeb test splits under different KB index scales – 240M items from Wikipedia vs. 2.2B from Wikipedia and F INE W EB -E DU. We show two perplexity variants for C O -LMLM (dynamics and static). The dashed gray line reports the size-respective HF/SmolLM2 model standard PPL for context. The HF models are trained on much more data. The left figure uses PQ240 quantization, while the right uses PQ96 to maintain the same quantization and fit the larger KB in memory. 360M-FW

30 135M 360M

20 1.7B

360M-FW 135M 360M

10

360M 135M

0 30

40

50 60 NLU (%)

Standard Std. HF Ref.

70

Rel-LMLM Co-LMLM

10 1

0.7

p=0.05

Model Utility (-)

FactScore (%)

135M 360M

Forget Quality (p-value )

40

10 3 10 5 10 7 10 9 10 11

0

2e-24

0.5 0.4 0.3

20

40

Unlearning Steps Standard - Base Standard - Unlearn (NPO)

(a) Factuality vs. NLU

0.6

0

20

40

Unlearning Steps

Co-LMLM - Base Co-LMLM - Unlearn

(b) Unlearning on TOFU

Figure 5: Factuality, NLU, and Unlearning Evaluation. (a) FactScore and NLU tradeoff for various models — C O -LMLM substantially improves factual precision while preserving NLU performance comparable to models of similar size. (b) Machine unlearning on TOFU — C O -LMLM forgets through direct KB operations, without additional training or utility degredation. of language modeling quality. Dynamic performs actual lookups during decoding, so retrieval performance is reflected in the score. While dynamic follows the standard perplexity calculation, it introduces some complexities in our case. We discuss this in depth in Appendix A.6 and provide a worst-case analysis. Importantly, this worst-case analysis does not change our conclusions. Results Figure 4 visualizes perplexity results. C O -LMLM consistently achieves lower perplexity than S TANDARD across model sizes and perplexity variants. Under the more realistic Dynamic setting, C O -LMLM reduces perplexity from 14.4 to 11.8 for 135M and from 14.0 to 10.5 for 360M. These gains suggest that our annotated pretraining data and training objective successfully improve language modeling over factual text. The FineWeb (FW) results indicate that benefit of data scale, charting an initial trends for scaling. The comparison to off-the-shelf models (HF/S MOL LM2-135M and HF/S MOL LM2-360M, dashed line) provides context. The relative gap C O -LMLM-360M-FW 7

Table 1: Factual precision evaluation. Green subscripts show the absolute difference from the respective S TANDARD baseline. We provide * models for contextualization, even though they are trained on much larger corpora (2T/4T/11T tokens for 135M/360M/1.7B models). We report a rebuilt R EL -LMLM baseline that follows the original setup with engineering improvements for a stronger, fairer comparison (Appendix A.5). Short-form QA

Model TriviaQA ↑

PopQA ↑

SimpleQA ↑

Knowledge Completion

Long-form Generation

T-REx EM ↑

FactScore ↑

∗

14.7 6.4 13.4 28.4+22.0

18.9 15.2 27.2 47.3+32.1

1.3 0.1 6.2 18.2+18.1

36.1 38.2 42.9 40.5+2.3

9.0 10.4 23.1 35.0+24.6

HF/S MOL LM2-360M∗ S TANDARD -360M S TANDARD -360M-FW R EL -LMLM-360M C O -LMLM-360M C O -LMLM-360M-FW

25.4 7.5 18.6 15.6 31.4+23.9 36.9+18.3

22.7 16.2 18.9 26.6 46.5+30.3 50.6+31.7

1.4 0.3 1.3 5.6 21.2+20.9 21.7+20.4

45.8 46.3 50.5 47.4 47.4+1.1 54.5+4.0

12.5 10.0 11.9 22.9 34.2+24.2 33.3+21.4

HF/S MOL LM2-1.7B∗

45.1

24.0

2.1

57.2

17.5

HF/S MOL LM2-135M S TANDARD -135M R EL -LMLM-135M C O -LMLM-135M

shows is particularly promising given the orders of magnitude more training data of the HF models. However, this experiment has a likely domain confounder, so we consider this result qualified. We also experiment with scaling up the index to include the entire FW and Wikipedia corpora used for training our models. This reflects a more realistic KB construction, both in source document diversity and size. This results in a much larger index of 2.2B entries, compared to the 240M entries in the Wiki-only index. Figure 4b reports results on both the Simple English Wikipedia dataset, as well as a 895-document held-out test set from the FW dataset. We compute these results with higher quantization for an additional 2.5× memory reduction per vector, so we can fit all indices in memory. We present results for both a Wikipedia-only index and a combined FW and Wikipedia index. We do not expect benefits from scaling up the index to include FW data for Simple-Wiki, because the corresponding knowledge is expected to be contained in the English Wikipedia dataset. Indeed, under the same quantization level, we only see a minor increase in perplexity even though the index is one order of magnitude larger, which speaks to the robustness of our learned retriever. For the FW test set, on the other hand, we observe a substantial improvement over the wiki-only index, producing a significantly lower perplexity than that of the HF/S MOL LM2-360M model, which is pretrained on those documents and overall much more data (4T tokens). The poor performance of the Wiki-only index on the FW test-set demonstrates Wikipedia’s limited coverage of knowledge and the need to scale up the factual KB. This result demonstrates effective scaling beyond Wikipedia data. 4.3

Factuality and General Benchmark Performance

Evaluation Setup We evaluate factuality across five complementary settings that probe different forms of factual knowledge in pre-trained models. TriviaQA [Joshi et al., 2017] tests broad open-domain QA, while PopQA [Mallen et al., 2023] focuses on long-tail entities. SimpleQA Verified [hereafter SimpleQA; Wei et al., 2024, Haas et al., 2026] targets short-form, fact-seeking questions that are deliberately difficult and have a single, indisputable answer. T-REx [Petroni et al., 2021] evaluates controlled factual completion, where the model completes a factual statement with the correct object entity. FactScore [Min et al., 2023] evaluates long-form factual precision in biography generation by decomposing generations into atomic facts and verifying them against external evidence. We also evaluate C O -LMLM on standard NLU benchmarks to verify that factual offloading does not compromise language understanding. We use greedy decoding for all factuality evaluations. For R EL -LMLM and C O -LMLM, retrieved factual spans are removed before scoring. The original R EL -LMLM experiments in Zhao et al. [2026] use partial KB as an approximation, due to the computational costs of indexing the entire KB. We use the complete KB, leading to slightly different numbers compared to the original paper. Appendix A.6 provides additional details. Results Table 1 shows that C O -LMLM consistently improves over the corresponding S TANDARD baselines, with especially large gains on short-form QA and FactScore, exceeding 18 points in each 8

case. Similar to perplexity, we report numbers for off-the-shelf models for contextualization, but under similar caveats. C O -LMLM also outperforms R EL -LMLM, demonstrating that continuous queries are more robust than textual entity-relation queries. We attribute these gains to two factors: free-form factual spans provide richer external knowledge than relational triplets, and hidden-state queries can use the full generation context rather than a short and restricted decoded query. Training on FineWeb-Edu further improves short-form QA and knowledge completion, showing that our design remains effective beyond Wikipedia-style pretraining data. C O -LMLM-360M-FW achieves 21.7 on SimpleQA, in line with gpt-4o-mini and above Claude Sonnet 4.5 and Grok 2 according to the public leaderboard.6 C O -LMLM achieves a better factuality-NLU trade-off that standard models, and benefits from the larger FineWeb-Edu pre-training data (Figure 5a). Table 12 in Appendix B.1 details NLU performance across tasks, further illustrating that C O -LMLM retains similar NLU capabilities, while achieving much higher factuality. Comparison with RAG RAG and C O - Table 2: Comparison of RAG vs. C O -LMLM LMLM are complementary: RAG expands on factual precision. inference-time context through document reTriviaQA PopQA SimpleQA T-REx trieval, while C O -LMLM externalizes free- Model HF/S MOL LM2-360M∗ 25.4 22.7 1.4 45.8 form factual knowledge during pretraining into + RAG 52.2 36.9 17.9 86.7 an editable memory. We study the behavioral S TANDARD -360M-FW 18.6 18.9 1.3 50.5 + RAG 42.5 33.9 15.3 86.4 outcomes of C O -LMLM and a controlled RAG 50.6 21.7 54.5 baseline, which uses BM25 to retrieve the top- C O -LMLM-360M-FW 36.9 + RAG 48.8 51.8 27.8 86.5 4 100-word Wikipedia passages and prepends * Off-the-shelf models trained on 4T general domain tokens. them to the input during generation (Table 2). We also apply the same retrieval on top of C O -LMLM. RAG significantly improves factuality for standard LMs. Nevertheless, the superiority of C O -LMLM is mostly maintained in this setting as well, as RAG gains accumulate over the already-high factuality scores. This suggests that C O LMLM+RAG is a viable path forward, where RAG provides broad document-level context, while C O -LMLM provides controllable, and editable factual memory. 4.4

Unlearning

Externalizing knowledge simplifies unlearning: instead of retraining the model, we just remove the relevant entries from memory. We test whether C O -LMLM preserves this benefit while replacing relational queries with continuous queries. We evaluate this property with TOFU [Maini et al., 2024], following the same setup as R EL -LMLM. TOFU tests whether a model can forget a designated forget set while preserving general utility. Its main metric, forget quality, tests whether the unlearned model is statistically indistinguishable from a retain-only model on the forgotten examples. Appendix A.7 provides additional details. Figure 5b shows how simple memory operations achieve effective forgetting (p-value > 0.05) while preserving model utility. In contrast, training-based unlearning methods such as NPO [Zhang et al., 2024b] update model parameters and sacrifice utility due to entangled parametric knowledge. C O -LMLM retains the controllability benefit of R EL -LMLM while extending LMLMs to a more expressive and efficient continuous-query KB. 4.5

Knowledge Memorization

We test whether C O -LMLM reduces factual memorization by evaluating it without the KB. Table 3 shows that factual precision drops substantially, forcing the model to rely on its internal parameters alone. This shows that C O -LMLM genuinely shifts factual knowledge from model weights to the external memory.

Table 3: No KB retrieval ablation. Disabling retrieval substantially reduces performance. Model

TriviaQA

PopQA

SimpleQA

T-REx

FactScore

S TANDARD -135M C O -LMLM-135M w/o KB

6.4 28.4 4.7−23.7

15.2 47.3 15.4−31.9

0.1 18.2 1.0−17.2

38.2 10.4 40.5 35.0 21.7−18.8 18.0−17.0

S TANDARD -360M C O -LMLM-360M w/o KB

7.5 31.4 6.0−25.4

16.2 46.5 13.5−33.0

0.3 21.2 0.7−20.5

46.3 10.0 47.4 34.2 27.4−20.0 18.7−15.5

S TANDARD -360M-FW C O -LMLM-360M-FW w/o KB

18.6 36.9 9.2−27.7

18.9 50.6 17.3−33.3

1.3 21.7 0.5−21.2

50.5 11.9 54.5 33.3 33.3−21.2 16.2−17.1

6 https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified

9

Table 4: Comparison with LMLM-A SKER baseline. The three rightmost columns give the modelside query-formation cost per retrieval (milliseconds). C O -LMLM generates no query (Query Gen. = 0) and reads its query vector directly from the decoder hidden state — a single <FACT> forward (Encode). LMLM-A SKER instead decodes a natural-language question (∼13 tokens, Query Gen.) and re-encodes it with a separate sentence encoder (Encode). LM Overhead is their sum. We exclude the faiss lookup, which is shared by both methods and dominated by index placement (GPU/CPU) rather than the model. Red subscript denotes degradation w.r.t. C O -LMLM. Model Type LMLM-A SKER-360M-FW C O -LMLM-360M-FW

4.6

TriviaQA ↑

PopQA ↑

SimpleQA ↑

15.4−21.5 36.9

43.2−7.4 50.6

7.5−14.2 21.7

T-REx ↑

FactScore ↑

38.5−16.0 54.5

25.0−8.3 33.3

Query Gen. (ms) ↓

Encode (ms) ↓

27.5 0.0

1.1 2.2

LM Overhead (ms) ↓ 28.6×13 2.2

Continuous vs. Natural Language Queries

We isolate the impact of using continuous queries by comparing C O -LMLM to a LMLM-A SKER, a variant that generates free-form natural language questions as queries. LMLM- ASKER is trained on the same annotated corpus. It generates a <QUESTION>. . . </QUESTION> block before each factual span and retrieves using the decoded question, while C O -LMLM retrieves directly from the hidden state. Table 4 shows that C O -LMLM outperforms LMLM-A SKER, demonstrating that advantage of a continuous queries. Intuitively, questions are constrained by the structure of language, whereas the continuous queries can learn arbitrary retrieval mechanism. Inference Overhead Text queries also add generation cost: LMLM-A SKER writes a question of arbitrary length before every lookup, while R EL -LMLM writes an entity-relation query. C O -LMLM instead produces the query from the hidden state, saving a cost that is linear in the length of the question. Empirically, forming a retrieval query costs C O -LMLM a single ∼2.2ms hidden-state forward, versus ∼28ms for LMLM-A SKER, which is dominated by decoding the ∼13-token question. We provide additional overhead comparisons in Appendix B.2. 4.7

Evaluating Query Timing

Table 5: Enforced lookup ablation. We compare standard inference with a variant enforcing lookup before answering.

We evaluate the ability of our model to form queries when needed by forc- Model TriviaQA PopQA SimpleQA T-REx ing query generation before answer- C O -LMLM-135M 28.4 47.3 18.2 40.5 w/ EnforceLookup 31.7+3.3 47.2−0.1 19.3+1.1 49.8+9.3 ing short-form QA and knowledgecompletion prompts. This setup dif- C O -LMLM-360M 31.4 46.5 21.2 47.4 w/ EnforceLookup 33.4+2.0 46.9+0.4 21.5+0.3 53.0+5.6 fers from earlier results in that the model is forced to retrieve, while still C O -LMLM-360M-FW 36.9 50.6 21.7 54.5 w/ EnforceLookup 43.3+6.4 53.3+2.7 27.6+5.9 59.3+4.8 using the query content generated by the model. Table 5 shows that forcing to query increase performance on both model scales across four benchmarks, illustrating that our retrieval performance is inhibited by the timing problem, and there remains room for improvement.

5

Discussion

We present C O -LMLM, a scalable method to train LLMs for verifiable and controllable use of knowledge. This includes a large-scale annotation pipeline that can be applied to hundreds of billions of tokens at relatively modest costs. Critically, the C O -LMLM training process, beyond the preprocessing step, departs only modestly from the conventional LLM training process that has been shown to scale to many trillions of both tokens and model parameters. Our method and results open an avenue to train LLMs with factuality, transparency, and controllability properties that are sorely lacking in current LLMs, which has significantly hampered their trustworthiness and utility. C O -LMLM provides a flexible free-form span substrate for studying how externalized knowledge scales across broader knowledge domains, memory sizes, and retrieval mechanisms. This establishes a foundation for future work to study knowledge scaling laws. Our current study has several limitations that outline important directions for future work. Although we experiment along model and data scaling axes to outline scaling trends, our experiments remain modest in model and data scale. We focus on probing the behavior of C O -LMLM in controlled 10

pretraining settings, and leave larger-scale pretraining and downstream adaptation to future work. Our factual evaluations mainly target Wikipedia-style knowledge, a key factor in creating a clean experimental setup. Although, as shown, our pipeline can naturally extend to broader corpora such as FineWeb-Edu, there remains work to be done to systematically evaluate factuality across more diverse knowledge domains. Constructing a retrieval index at pretraining scale introduces a one-time computational cost. However, an important problem to address in the context of post-training and continual learning is how to adapt such an index once a model is fine-tuned. Studying these and many other important problems will be enabled by the scalable design of our methods, and by our large-scale release of annotations of Wikipedia and 100B FineWeb tokens.

Acknowledgments and Disclosure of Funding This research was supported by AI-MI and NSF Award DMR-2433348; the NSF under awards OAC-2311521, IIS-2505098, RI-2530143, and 2118310; a gift to the LinkedIn–Cornell Bowers Strategic Partnership; Apple Research, Gemini credits grants from Google; and NASA under award No. 20-OSTFL20-0053. NG and DG are supported by an Empire AI Postdoctoral Fellowship. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or of NASA. We thank the members of PIs’ labs for helpful discussions. Author Contributions Below we list the contributions of the primary non-faculty authors in the project. All authors took part in project discussions, experiment planning, and paper revisions. Yair Feldman designed and implemented the core method; trained the models; implemented the S TANDARD and LMLM-A SKER baselines; conducted PPL and loss evaluations; wrote the paper. Linxi Zhao designed and implemented the R EL -LMLM baseline; conducted FactScore, TOFU, QA, and NLU evaluations; and wrote the paper. Nathan Godey

helped design pre-training runs.

11

References Mohammad Kalim Akram, Saba Sturua, Nastia Havriushenko, Quentin Herreros, Michael Günther, Maximilian Werk, and Han Xiao. jina-embeddings-v5-text: Task-targeted embedding distillation, 2026. URL https://arxiv.org/abs/2602.15547. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=hSyW5go0v8. Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, and Gargi Ghosh. Memory layers at scale. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2412.09764. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2020. URL https://arxiv.org/abs/1911.11641. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, 2022. URL https://proceedings.mlr.press/v162/borgeaud22a.html. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020. URL https://proceedings.mlr.press/v119/chen20j.html. Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Yukun Li, Han Zhang, Huishuai Zhang, Dongyan Zhao, and Wenfeng Liang. Conditional memory via scalable lookup: A new axis of sparsity for large language models, 2026. URL https://arxiv.org/abs/2601.07372. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https://aclanthology.org/2022.acl-long.581/. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. IEEE Transactions on Big Data, 2025. URL https://arxiv.org/abs/2401.08281. Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. In Advances in Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=35DAviqMFo. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. Gaurav R. Ghosal, Pratyush Maini, and Aditi Raghunathan. Memorization sinks: Isolating memorization during LLM training. In International Conference on Machine Learning, 2025. URL https://proceedings.mlr.press/v267/ghosal25a.html. Aaron Grattafiori et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783. 12

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.acl-long.841/. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrievalaugmented language model pre-training. In International Conference on Machine Learning, 2020. URL https://proceedings.mlr.press/v119/guu20a.html. Lukas Haas, Gal Yona, Giovanni D’Antonio, Sasha Goldshtein, and Dipanjan Das. Simpleqa verified: A reliable factuality benchmark to measure parametric knowledge, 2026. URL https: //arxiv.org/abs/2509.07968. Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts LLM performance, 2025. URL https://research.trychroma.com/context-rot. Lei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao, Yuchun Fan, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, and Bing Qin. Advancing large language model attribution through self-improving. In Conference on Empirical Methods in Natural Language Processing, 2024. URL https://aclanthology.org/2024.emnlp-main.223/. Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Conference of the European Chapter of the Association for Computational Linguistics, 2021. URL https://aclanthology.org/2021.eacl-main.74/. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 2023. URL https://arxiv.org/abs/2208.03299. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Annual Meeting of the Association for Computational Linguistics, 2017. URL https://aclanthology.org/P17-1147/. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/kandpal23a.html. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HklBjCEKvH. Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/ hash/9d8df73a3cfbf3c5b47bc9b50f214aff-Abstract.html. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/ 2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=B41hNBoWLo. 13

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https://aclanthology.org/2023.acl-long.546/. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://aclanthology.org/2023.emnlp-main.741/. Ali Modarressi, Abdullatif Köksal, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. MemLLM: Finetuning LLMs to use an explicit read-write memory, 2024. URL https://arxiv.org/abs/ 2404.11672. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2402.09906. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/ hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_ Track.html. Guangyue Peng, Tao Ge, Si-Qing Chen, Furu Wei, and Houfeng Wang. Semiparametric language models are scalable continual learners, 2023. URL https://arxiv.org/abs/2303.01421. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://aclanthology.org/D19-1250/. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. URL https://aclanthology.org/2021.naacl-main.200/. Hadi Pouransari, David Grangier, C Thomas, Michael Kirchhof, and Oncel Tuzel. Pretraining with hierarchical memories: separating long-tail and common knowledge, 2026. URL https: //arxiv.org/abs/2510.02375. Qwen et al. Qwen2.5 technical report, 2024. URL https://arxiv.org/abs/2412.15115. Lance Ramshaw and Mitchell Marcus. Text chunking using transformation-based learning. In Workshop on Very Large Corpora, 1995. URL https://aclanthology.org/W95-0107/. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Conference on Empirical Methods in Natural Language Processing, 2020. URL https://aclanthology.org/2020.emnlp-main.437/. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SocialIQA: Commonsense reasoning about social interactions. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://aclanthology.org/D19-1454/. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Yacmpz84TH. Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.acl-long.702/. 14

Alicia Yi Sun, Karthik Padthe, Akari Asai, and Wen-tau Yih. Semi-parametric language model with selective memory, 2025. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019. URL https://aclanthology.org/N19-1421/. Albert Tseng and Christopher De Sa. L3 : Large lookup layers, 2026. URL https://arxiv.org/ abs/2601.21461. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018. URL https://arxiv.org/abs/1807.03748. Xi Wang, Taketomo Isazawa, Liana Mikaelyan, and James Hensman. KBLaM: Knowledge base augmented language model. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=aLsMzkTej9. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Annual Meeting of the Association for Computational Linguistics, 2025. URL https://aclanthology.org/2025. acl-long.127/. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368. Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=TrjbxzRcnf-. Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, et al. Memory3 : Language modeling with explicit memory, 2024. URL https://arxiv.org/abs/2407.01178. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://aclanthology.org/P19-1472/. Jintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, and Ningyu Zhang. OneGen: Efficient one-pass unified generation and retrieval for LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024a. URL https://aclanthology.org/2024.findings-emnlp.797/. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In Conference on Language Modeling, 2024b. URL https: //openreview.net/forum?id=MXLBXjQkmb. Wenzheng Zhang, Xi Victoria Lin, Karl Stratos, Wen-tau Yih, and Mingda Chen. ImpRAG: Retrievalaugmented generation with implicit queries. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. URL https://arxiv.org/abs/2506.02279. Linxi Zhao, Sofian Zalouk, Christian K Belardi, Justin Lovelace, Jin Peng Zhou, Ryan Thomas Noonan, Dongyoung Go, Kilian Q Weinberger, Yoav Artzi, and Jennifer J Sun. Pre-training limited memory language models with internal and external knowledge, 2026. URL https: //arxiv.org/abs/2505.15962.

15

A

Experimental Setup

A.1

Pre-training Hyperparameters

We pretrain C O -LMLM from random weights following a Warmup-Stable-Decay (WSD) learningrate schedule. Table 6 lists the hyperparameters used to train C O -LMLM. The S TANDARD baseline language model and LMLM-A SKER are trained with the same hyperparameters, except for those that apply only to the contrastive loss (contrastive loss weight, contrastive temperature, and max question length). For the FineWeb-Edu run, the cooldown stage mixes the annotated Wikipedia and FineWeb-Edu datasets in a 1:1 ratio. Table 6: Pre-training hyperparameters for C O -LMLM. Hyperparameter Value architecture tokenizer optimizer β1 β2 ϵ clip norm peak learning rate final learning rate lr scheduler type warmup steps stable steps cooldown steps weight decay batch size max context length max question length contrastive temperature precision

A.2

SmolLM2 (random init) SmolLM2 AdamW 0.9 0.95 1e-8 1 5e-4 0 WSD 2,000 88,000 (148,000 for FineWeb-Edu) 10,000 (20,000 for FineWeb-Edu) 0.01 128 4,096 128 0.07 bf16 (fp32 master weights)

Annotator Training

We use Gemini 3.1 Pro to annotate a seed set of 60,000 documents (15,000 Wikipedia articles and 45,000 FineWeb-Edu documents), and train the fact-span annotator and the question generator on this seed set. The fact-span annotator is a ModernBERT-large [Warner et al., 2025] encoder fine-tuned for BIO token classification (Table 7). The question generator is a Qwen2.5-1.5B-Instruct [Qwen et al., 2024] decoder fine-tuned with LoRA (Table 8). A.3

Compute

For all experiments involving model training except TOFU, between 1 to 8 NVIDIA B200 GPUs were used, depending on availability. The number of gradient accumulation steps was set dynamically to maintain a consistent batch size throughout training. For the TOFU unlearning experiments, a single NVIDIA RTX A6000 GPU was used. A.4

Retrieval Indexes

We use the Faiss library [Douze et al., 2025] for all our experiments involving building dense search indexes. To limit RAM usage, we use the following factory string for constructing our indices: OPQ<d/4>,IVF65536,PQ<d/4>, where d is dimension of the embedder being used. For our R EL -LMLM and LMLM-A SKER experiments, we use the jina-embeddings-v5-text-nano embedder [Akram et al., 2026], truncated to 512 dimensions. We did not extensively explore the retrieval errors that can be attributed to the lossy compression applied by the search indexes for either of the models presented in this paper. 16

Table 7: Hyperparameters for training the fact-span annotator. Hyperparameter Value base model optimizer β1 β2 ϵ clip norm peak learning rate final learning rate lr scheduler type warmup ratio weight decay epochs batch size max sequence length precision

ModernBERT-large AdamW 0.9 0.95 1e-8 1 2e-4 2e-5 cosine 0.05 1e-3 4 32 4,096 bf16

Table 8: Hyperparameters for training the question generator. Hyperparameter Value base model LoRA r LoRA α LoRA dropout LoRA target modules optimizer β1 β2 ϵ clip norm peak learning rate final learning rate lr scheduler type warmup ratio weight decay epochs batch size max sequence length precision

A.5

Qwen2.5-1.5B-Instruct 64 64 0.05 all-linear AdamW 0.9 0.95 1e-8 1 2e-4 2e-5 cosine 0.05 1e-3 4 4 8,192 bf16

R EL -LMLM Re-implementation

For a controlled comparison against C O -LMLM, we reimplement the LMLM pretraining pipeline with several engineering improvements over the original recipe. These changes do not affect the conclusions of the original R EL -LMLM work. They improve data efficiency and align the training setup with C O -LMLM, yielding a stronger R EL -LMLM baseline and a more rigorous apples-toapples comparison. The changes fall into two groups. Annotation quality and quantity. The largest gain is in the quantity of usable pretraining data (summarized in Table 9). The original pipeline truncates each document to 1024 tokens before annotation. Annotating full documents instead recovers a substantial fraction of the Wikipedia subset of the OLMo2 dolmino-mix-1124 corpus [Groeneveld et al., 2024]7 (roughly 30% of documents and about half of the available pretraining tokens were previously discarded), raising the usable set from 2.8B to 6.1B tokens (including DB lookup tokens). 7 https://huggingface.co/datasets/allenai/dolmino-mix-1124

17

Table 9: Annotation configuration updates from the original R EL -LMLM release to the rebuilt, stronger baseline used in this paper. Token counts are for the Wikipedia subset of dolmino-mix-1124. Version

Seed annotator

Production annotator

Input trunc.

DB format

Est. tokens

Original Rebuilt

GPT-4o GPT-4o

Llama3-8B-Instruct (LoRA) Qwen3-4B (full FT)

1024 tok/doc none

[dblookup(”)->] special tokens

2.8B 6.1B

On the quality side, we emit the special-token database-call format directly during annotation rather than converting from the textual [dblookup(‘’)->] markup before pretraining, removing a conversion step and keeping annotations consistent end-to-end. We also improve prompting, enlarge the seed annotation set, and avoid overfitting (12,750 examples for 3 epochs versus the original 2k examples for 10 epochs), and replace the LoRA-tuned Llama3-8B-Instruct production annotator with a fully fine-tuned Qwen3-4B for better quality and efficiency. Identical backbone and training configuration. Throughout this paper, we report results for the rebuilt R EL -LMLM. To isolate the effect of the method, we match the model backbone and all training hyperparameters (batch size, number of steps, learning-rate schedule, etc.) to C O -LMLM. With the data pipeline improved and the training setup held fixed, remaining differences between the two methods reflect the method itself rather than the experimental setup. This yields a substantially stronger R EL -LMLM baseline than the original release. C O -LMLM’s improvements over this matched, well-tuned baseline therefore reflect the method itself, not an advantage in experimental setup. A.6

Evaluation

Perplexity Zhao et al. [2026] introduced three perplexity variations for LMLM models: static (Oracle), normalized, and dynamic. The static and dynamic variants are described in Section 4.2, but are recited here for completeness. Static perplexity assumes perfect lookup behavior by supplying the correct retrieved spans as context, giving an optimistic estimate of language modeling quality. Dynamic perplexity performs actual lookups during decoding, so retrieval performance is reflected in the score. Normalized perplexity additionally scores the lookup-trigger tokens the model emits, and divides by the length of the original unannotated text. Normalized perplexity is informative for R EL -LMLM, where the lookup tokens include a decoded query whose quality is being measured. For C O -LMLM, the only emitted lookup token is a single <FACT> delimiter, since the query is read from the hidden state rather than decoded as text, so normalized perplexity for C O -LMLM additionally measures only whether retrieval is triggered at the right position. We therefore treat dynamic and static perplexity as the primary metrics. Formally, let x be an input document with factual spans bracketed by <FACT> and </FACT>, and T the set of token positions in x. Let M ⊆ T be the set of positions in x that are strictly between <FACT> and </FACT>, inclusive of both the opening <FACT> and closing </FACT>. Finally, let M = T \ M . We define the static and dynamic perplexity as follows:   X 1 PPLstatic/dynamic = exp − log pθ (xt | x<t ) , |M | t∈M

where the only difference between the two is the source of the factual spans: static uses the oracle spans paraphrased by the annotator, and dynamic issues actual retrieval and inserts the retrieved fact into the span. The dynamic perplexity metric is the more realistic of the set, and follows conventional perplexity computation. However, the nuances of C O -LMLM introduces some complexities in its measurement, and make it somewhat optimistic. This is because it does not take into account the probability of the model correctly generating the <FACT> token at that precise position. An accurate and comparable perplexity measure is not trivial to compute since the fact-annotated text only defines one possible way to generate the original context tokens, and accounting for all possible retrieval paths is intractable. Furthermore, directly accounting for the <FACT> token’s loss will not be accurate as well since it will 18

15

14.4

14.0

Perplexity

↓

12.6 11.811.6

10

11.2 10.5 9.6

8.0

7.4

9.1 7.7 5.8

5 0

8.2

135M

360M

360M-FW

Model

Standard Standard HF Ref. Co-LMLM-Dyn-Norm

Co-LMLM-Dyn. Co-LMLM-Stat.

Figure 6: Perplexity evaluation with the dynamic-normalized PPL variant. Validation perplexity comparison between S TANDARD and C O -LMLM across three perplexity variants. The dashed gray line provides reports the size-respective SmolLM2 model standard PPL for context. Even when using the pessimistic dynamic-normalized perplexity measure, C O -LMLM achieves significantly lower perplexity than the corresponding S MOL LM2-360M model trained on an order of magnitude more tokens. introduce additional context tokens, presenting a mismatch between LMLM and standard LMs in the denominator |M | used in the perplexity calculation. It also changes the vocabulary by introducing a new, fairly common token. These factors may lead to an overly optimistic measure as well. We address this issue with an additional dynamic-normalized perplexity measure, which includes the <FACT> token’s loss in the enumerator but keeps it out of the denominator. This normalized measure is a worst case scenario, in practice allowing us to compute a range between itself and the optimistic dynamic measure. Formally, let MF be the set of positions in x that contain the <FACT> token. We define the dynamicnormalized perplexity as follows:   X 1 log pθ (xt | x<t ) . PPLdynamic-norm = exp − |M | t∈M ∪MF

This provides us with a pessimistic estimation of the perplexity, ensuring that the actual, realistic perplexity lies between PPLdynamic and PPLdynamic-norm . Figure 6 provides results for all perplexity variants. Critically, this worst-case scenario does not change the ordering of models, or our conclusions. Long-Form Generation: FactScore We evaluate factual precision using FACT S CORE [Min et al., 2023], a benchmark for open-ended biography generation. Given a generated biography, FactScore decomposes it into atomic facts and measures the proportion that can be verified against a trusted knowledge source via retrieval-augmented ChatGPT. We use the labeled entities set from the benchmark, which includes 183 entities. All models use greedy decoding with a maximum content budget of 256 tokens and a repetition penalty of 1.2. For models with external lookup (e.g., LMLM - RETRIEVER and LMLM - STRUCTURED), this budget applies only to the final textual output and excludes any inserted lookup spans (e.g., <FACT>...</FACT> or <|db_*|> blocks). To provide a fair probe for models pretrained primarily on Wikipedia (and not instruction-tuned or exposed to more diverse corpora), we adopt a fixed Wikipedia-style prompt “<name>\n\n<name>” to elicit biography completions, applied uniformly across all samples. For evaluation, all lookup-related annotation tokens are stripped before scoring. We follow the official FactScore pipeline, using retrieval-augmented prompting with GEMINI -2.5FLASH.8 8 https://github.com/shmsw25/FActScore

19

We use greedy decoding for all factuality benchmarks, including FactScore, T-REx, PopQA, TriviaQA and SimpleQA. The retrieval similarity threshold is set to 0.7; when no match is found, the model falls back to standard decoding, allowing it to answer facts absent from the database. Annotation tokens from retrieval or structured lookup are removed before scoring. Short-Form Knowledge Completion: T-REx We adopt T-REx from LAMA [Petroni et al., 2019] as a knowledge completion benchmark for autoregressive models. Following Schick et al. [2023], we retain only examples compatible with left-to-right generation, resulting in 11,615 instances where the masked entity appears at the end of the sentence. Each input is a partially observed factual statement, and the model is required to complete it with the correct object entity. We use greedy decoding with a maximum of 32 generated tokens. Performance is measured by Exact Match, which checks whether the reference answer appears within the first five generated content tokens after stripping annotation tokens. Short-Form Probing: PopQA, TriviaQA and SimpleQA We further evaluate factual recall on question-answering benchmarks, focusing on the long-tail subset of PopQA [Asai et al., 2024], which contains 1,399 queries about rare entities (fewer than 100 monthly Wikipedia page views), the full TriviaQA evaluation set (17,944 examples), and the full SimpleQA evaluation set (4,326 examples). These benchmarks probe knowledge recall under both sparse and broad coverage regimes. All models use greedy decoding with a maximum of 32 tokens. For PopQA and TriviaQA, evaluation is based on Exact Match, defined as whether any alias in the set of gold answers appears (caseinsensitive) within the first 100 characters of the model output. For SimpleQA, we follow the official grading prompt and use GPT-4.1-2025-04-14 as the grader model. We report the overall correct metric. A.7

Machine Unlearning Setting

TOFU unlearning. The TOFU unlearning benchmark [Maini et al., 2024] evaluates selective unlearning in a controlled question-answering setting. It contains 200 synthetic author profiles, each associated with 20 question-answer pairs, resulting in 4,000 QA examples in total. The benchmark partitions this synthetic knowledge into a Forget Set, containing the examples to be removed, and a Retain Set, containing the remaining synthetic author knowledge that should be preserved. In addition to synthetic biographies, TOFU evaluates whether unlearning preserves broader model behavior using two auxiliary sets: the Real Author Set, which contains factual questions about real authors, and the World Facts Set, which contains general factual knowledge. The target behavior is that the resulting Unlearned Model should forget the Forget Set while remaining statistically close to a Retain Model, which is trained only on the Retain Set, and should maintain utility on the Retain Set, Real Author Set, and World Facts Set. TOFU reports two main categories of metrics. Forget Quality measures whether the unlearned model behaves similarly to the retain model on the Forget Set, using a statistical test whose p-value indicates whether the two output distributions are distinguishable. A higher p-value indicates better forgetting, with values above 0.05 commonly interpreted as failing to reject the hypothesis that the unlearned model and retain model behave similarly. Model Utility measures whether useful behavior is preserved after unlearning. It aggregates three metrics: ROUGE, which measures the lexical overlap between generated and reference answers; Answer Probability, which measures the likelihood assigned to the correct answer; and Truth Ratio, which compares the likelihood assigned to correct answers against paraphrased or perturbed alternatives. These utility metrics are evaluated across the Retain Set, Real Author Set, and World Facts Set to capture both retained synthetic knowledge and general factual capability. We evaluate on the TOFU Forget 5% setting using the official evaluation pipeline from the TOFU repository9 . For C O -LMLM, we augment the external Wikipedia knowledge base from pre-training with TOFU’s synthetic author biographies and fine-tune C O -LMLM-360M-FW on the annotated TOFU training set using the same hyperparameters as the baseline models in Table 10. To perform unlearning, we delete the database entries corresponding to the Forget Set, without any additional gradient updates. This directly tests whether externalizing factual knowledge enables training-free removal of targeted information. 9 https://github.com/locuslab/open-unlearning

20

NPO unlearning baseline. We compare against HF/S MOL LM2-360M fine-tuned with Negative Preference Optimization (NPO) [Zhang et al., 2024b], a strong gradient-based unlearning baseline. We run NPO unlearning fine-tuning with five random seeds and report the mean and variance in Figure 5b. For ROUGE evaluations, generated answers are post-processed to remove structured factual spans before computing scores. For likelihood-based metrics, including Answer Probability and Truth Ratio, we evaluate the model probabilities with answer masking.

Setting

Table 10: TOFU fine-tuning configuration. Value

Base model Training sets Template Learning rate Epochs Checkpoints

HF/S MOL LM2-360M / C O -LMLM (F INEWEB ) TOFU full / retain95 Question: / Answer: 3 × 10−5 10 Full model and retain model

Table 11: NPO training configuration on TOFU. Setting Value Initialization Forget / retain split Objective Learning rate Per-device batch size Gradient accumulation Epochs Seeds A.8

Full TOFU fine-tuned checkpoint forget05 / retain95 NPO, β = 0.1 6 × 10−5 32 1 8 {0, 42, 69, 420, 4497}

Ablation Setting

RAG Setting. We include a simple inference-time RAG baseline as a point of reference, following Lewis et al. [2020], adopting the global RAG setting of FlashRAG10 . For each prompt, we use BM25 to retrieve the top-4 relevant 100-word passages from the 2018 English Wikipedia dump and prepend them to the original input using the prompt format: Answer the question or complete the prompt based on the given document. The following are given documents: \n [retrieved passages] \n\n [original prompt] \n The answer is. The retrieved passages are joined with newlines. We use the same generation setup as the corresponding non-RAG model. No KB Retrieval Ablation. To measure how much factual performance depends on access to the learned external memory, we evaluate a no-retrieval variant of our model. In this setting, we disable the model’s ability to trigger KB retrieval during generation. Concretely, for continuous-query models, we apply a logit bias of −100 to all factual special tokens that initiate or delimit retrieval spans. This prevents the model from emitting the markup required to invoke the retrieval loop, forcing it to generate plain text from its parameters alone. This ablation isolates the contribution of external memory access from the model’s parametric knowledge. Enforced Lookup Ablation. We also evaluate an enforced-lookup setting to test whether additional retrieval improves factuality when the model does not choose to query on its own. In this setting, we force a lookup at the beginning of every prompt, before normal generation begins. For continuousquery retriever models, we prepend a <FACT> token, run a forward pass to obtain the hidden state at this position, project it into the retrieval space, and perform a FAISS search before the first generation step. If the query extraction fails under similarity threshold of 0.7, we remove the forced prefix and continue generation while temporarily forbidding another <FACT> token. 10 https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/original_docs/baseline_details.md#

global-setting

21

B

Additional Results

B.1

NLU Evaluation

Table 12: Evaluation on NLU benchmarks (5-shot). Performance remains comparable while improving factual capabilities. Values are mean±std over 3 few-shot exemplar seeds. Model

CSQA

HellaSwag

PIQA

SIQA

ARC Easy

CoreAvg

Random Chance

20.0

25.0

50.0

33.3

25.0

30.7

R EL -LMLM-135M S TANDARD -135M C O -LMLM-135M

29.9±0.7 27.1±0.2 26.7±0.2

29.6±0.2 28.7±0.1 29.2±0.2

57.3±0.3 55.9±0.3 56.0±0.6

40.6±0.7 40.6±0.3 40.2±0.2

41.0±0.5 39.8±0.1 40.1±0.3

39.7±0.1 38.4±0.1 38.5±0.2

R EL -LMLM-360M S TANDARD -360M C O -LMLM-360M S TANDARD -360M-FW C O -LMLM-360M-FW

31.7±0.7 32.0±0.1 29.4±0.1 46.0±0.4 48.3±0.8

32.9±0.1 31.6±0.2 31.9±0.1 47.3±0.4 46.9±0.1

59.7±0.6 59.5±0.7 57.1±0.4 69.4±0.0 68.9±0.2

41.3±0.8 41.4±0.1 40.5±0.6 44.0±0.4 45.2±0.3

45.0±0.9 44.6±0.2 44.3±0.4 68.4±0.3 66.1±0.0

42.1±0.5 41.8±0.1 40.6±0.2 55.0±0.1 55.1±0.3

HF/S MOL LM2-135M∗ HF/S MOL LM2-360M∗ HF/S MOL LM2-1.7B∗

46.3±0.6 57.8±0.4 66.2±0.8

42.4±0.0 55.9±0.2 71.6±0.2

67.6±0.1 72.6±0.1 78.2±0.2

45.6±1.1 48.2±0.5 53.2±0.5

66.9±0.4 72.7±0.4 80.1±0.3

53.8±0.2 61.4±0.0 69.9±0.3

Following the R EL -LMLM evaluation, we assess whether C O -LMLM preserves the general language understanding ability of the pretrained model. We evaluate on the same set of high-signal NLU benchmarks with few-shot prompting used in prior work: CommonsenseQA [Talmor et al., 2019], HellaSwag [Zellers et al., 2019], PIQA [Bisk et al., 2020], SIQA [Sap et al., 2019], and ARC Easy [Clark et al., 2018]. As in R EL -LMLM, we exclude benchmarks where similarly sized models perform near the noise floor [Du et al., 2024]. All evaluations are run with lighteval11 . Because S TANDARD and C O -LMLM are pretrained only on Wikipedia, and C O -LMLM-360M-FW is pretrained with fewer than 100B tokens, these benchmarks are not meant for direct comparison with off-the-shelf models trained on broader and much larger corpora (2T/4T/11T tokens for the 135M/360M/1.7B models). Rather, they provide a controlled check that externalizing factual knowledge does not harm general NLU performance. B.2

Inference Efficiency

We analyze the inference overhead of LMLM-A SKER and C O -LMLM, focusing on the per-retrieval cost of forming a retrieval query: the work each model performs, beyond decoding the answer content, to issue a single lookup. The key algorithmic difference is how the retrieval query is formed. LMLM-A SKER explicitly decodes a textual lookup question and then encodes it with a sentence-transformer query encoder. In contrast, C O -LMLM retrieves directly from the model hidden state at a continuous query token. This token requires one additional forward step. As summarized in Table 13, C O -LMLM therefore avoids both textual query generation and sentence-transformer query encoding, reducing decoding cost by K L̄q tokens per answer and the KV-cached context length by K(L̄q + 1) tokens per answer. Measuring per-retrieval overhead. Measuring this overhead from free-form generation is confounded: LMLM-A SKER and C O -LMLM trigger different numbers of retrievals, at different positions, and produce different amounts of content, so end-to-end latency entangles the cost of the retrieval mechanism with these behavioral differences and with a varying context length at each lookup. We therefore measure at fixed retrieval sites, using the gold-annotated documents of our dynamic-perplexity evaluation set (Section 4.2): at every gold <FACT> position each model forms a query from its own context prefix, yielding a controlled, per-retrieval comparison on identical sites and contexts. For each fact we first prefill the context prefix to warm the KV cache, so that the timing 11 https://github.com/huggingface/lighteval

22

Table 13: Theoretical efficiency comparison. Compared with LMLM-A SKER, C O -LMLM removes explicit textual query generation and sentence-transformer query encoding. Component

LMLM-A SKER

C O -LMLM

Difference

Decoding-time cost Content tokens Decoded query tokens Query start tags Continuous query token (not cached)

C K L̄q 2K 0

C 0 K K

0 −K L̄q −K K

Prefilling cost Injected answer tokens Retrieval end tag

K L̄a K

K L̄a K

0 0

Retrieval-time cost Retrievals per answer Sentence-transformer query encoding FAISS search

K K K at (da , Ma )

K 0 K at (dr , Mr )

0 −K encodes index-dependent

Total context token length

C + K(L̄q + L̄a + 3)

C + K(L̄a + 2)

−K(L̄q + 1)

excludes prefilling, which is shared and independent of the retrieval mechanism. We then time only the query-formation step: for C O -LMLM, the single forward pass that produces the continuous query token from the hidden state; for LMLM-A SKER, decoding the textual question up to </QUESTION> followed by encoding it with the jina-embeddings-v5-text-nano query encoder. We exclude the FAISS search itself, which is common to both methods and dominated by index placement (GPU vs. CPU) rather than the model. The language-model steps (C O -LMLM’s query forward and LMLM-A SKER’s question decoding) are timed with vLLM in bfloat16 at batch size 1 on the 360M FineWeb-Edu checkpoints, and the query encoder (jina-embeddings-v5-text-nano) in bfloat16 with FlashAttention-2. Throughout we report the underlying forward/decode compute, subtracting the fixed per-invocation overhead consistently for both models: at batch size 1 a single .encode() call or generate step is dominated by framework and host–device-transfer cost rather than the forward. For instance the encoder’s ∼46 ms single-query wall time is ∼45 ms fixed overhead and only ∼1 ms forward, as isolated by batch-scaling (32 queries encode in the same ∼46 ms wall as one). We report medians over ∼1.5K facts. As shown in Table 4, C O -LMLM forms its query in a single ∼2.2 ms <FACT> forward, whereas LMLM-A SKER spends ∼27 ms decoding the textual question (∼13 tokens) plus ∼1 ms for the encoder forward, about 28 ms in total, a ∼13× higher per-retrieval overhead. Notably, the encoder forward itself is cheap, so the gap comes almost entirely from decoding the query text, exactly the K L̄q token saving of Table 13 that C O -LMLM realizes by querying from the hidden state. B.3

Additional Unlearning Results

Beyond Figure 5b, Figure 7 shows that C O -LMLM better preserves Retain Set knowledge, whereas prior training-based methods like NPO might suffer severe ROUGE degradation and forget related knowledge due to parameter entanglement. B.4

Related Work: Memory-Augmented Language Models

Prior work has explored explicit memory modules as a way to augment LLMs. KBLaM [Wang et al., 2025] stores knowledge as continuous key-value vectors and integrates them into pretrained LLMs via modified attention. Memory3 [Yang et al., 2024] pretrains models to write and read sparse attention key-value memories, sharing our motivation of offloading factual knowledge from parameters. MemLLM [Modarressi et al., 2024] introduces an explicit read-write memory for extracting, storing, and recalling triplet knowledge. More recently, SPLM [Sun et al., 2025] stores atomic facts absent from a pretrained model’s parametric memory in a compact external memory, and continue pretrains the model to retrieve from this external, non-parametric memory at inference time, using textual query. Unlike these memory-based methods, C O -LMLM uses continuous representations only to query memory, while storing knowledge as natural-language text. This keeps retrieval flexible and low-cost, but preserves the transparency of an explicit KB: retrieved knowledge remains human-readable, 23

Retain Answer ROUGE (-)

Forget Answer ROUGE

0.65 0.60 0.55 0.50 0.45 0.40 0

20

40

0.60 0.55 0.50 0.45 0.40 0

Unlearning Steps

Standard - Base Standard - Unlearn (NPO)

20

40

Unlearning Steps

Co-LMLM - Base Co-LMLM - Unlearn

Figure 7: Machine unlearning on TOFU. C O -LMLM performs forgetting through direct KB operations, without additional training. C O -LMLM retains knowledge outside the forget set, unlike other methods that degrade retain-set performance because of the entanglement of knowledge storage. attributable, editable, and directly removable. Thus, C O -LMLM combines learned memory access with the inspectability and controllability of text-based external knowledge.

24

C

Prompt Used for Seed Annotation with Gemini

While experimenting with different annotation prompts, we noticed a shard degradation in annotation quality as the input document grew longer [Hong et al., 2025]. To mitigate this, annotate in a chat-based fashion: we set the annotation prompt as the system prompt, and pass the first chunk of the document (512 tokens in our setup). After receiving the annotated chunk, we pass on the next chunk in the same conversation, and so on until reaching the end of the document. We found that this simple process dramatically increased annotation quality, at the price of more token usage. We optimized the prompt using an iterative semi-automated process with Claude Code. The following prompt was provided to Gemini to annotate the seed documents:

You are creating a "Golden Dataset" for limited memory language models (LMLM) that retrieve facts from a knowledge base rather than memorizing them. **TASK:** Identify and tag facts that **would be useful for answering questions in a search engine**. These are the specific details (dates, names, numbers, citations, definitions) that someone might search for and that could not be answered without this document’s information. Tag them using < FACT q="..." a="...">...</FACT> format. - ‘q‘ = A natural search query someone might type to find this fact (answerable without this document) - ‘a‘ = A snippet-form paraphrase of the answer (see Paraphrase Rules below) - Tag content = The verbatim span from the input text (preserve EXACTLY as in input - no modifications) **Search-engine mindset:** Before tagging any fact, ask yourself: "Would someone plausibly type a search query to find this specific piece of information?" If the answer is yes, tag it. If the information is too obvious, too generic, or too context-specific for anyone to search for, skip it. --### SPAN BOUNDARY PRINCIPLE (READ FIRST) Selecting the right span boundaries is critical. Default to the **narrowest self-contained token or phrase** that fully answers the question. Widen only when the narrow span is genuinely ambiguous in isolation. Decision procedure: 1. Start with the bare factual token (a name, number, date). 2. Ask: "Shown alone, is this token ambiguous?" If NO, tag just that token. 3. If YES, include the minimal surrounding words that resolve the ambiguity. - If a bare number or noun is meaningful on its own (e.g., "1990", "Paris", " Einstein"), tag just that token. - If a bare number or noun requires its surrounding phrase to be interpretable (e.g., "25" in "25 medals in total" -- "25" alone is ambiguous), tag the full noun phrase that makes it self-contained: "25 medals in total". <example> Input: The team won 25 medals in total during the championship. WRONG: <FACT q="How many medals did the team win?" a="25 medals">25</FACT> (too narrow -- "25" alone is ambiguous) CORRECT: <FACT q="How many medals did the team win at the championship?" a="25 medals total">25 medals in total</FACT> </example> <example> Input: The ceremony was held in Paris.

25

CORRECT: <FACT q="Where was the ceremony held?" a="Paris">Paris</FACT> (single proper noun is self-contained) </example> --### QUESTION SELF-CONTAINMENT PRINCIPLE (READ SECOND) Every question must be written as if for a reader who has **never seen this document**. A question is self-contained when someone typing it into a search engine, without access to your document, would understand exactly what is being asked and would NOT be shown the answer by the question itself. Two opposing failure modes to avoid: **Failure mode A -- Under-specified questions.** Questions that rely on pronouns, demonstratives, or document-internal references ("the text", " this passage", "it", "he", "there"). Replace every pronoun with the actual entity name. Every question must name its subject. **Failure mode B -- Over-specified questions that leak the answer.** Questions that contain so many distinguishing details that the answer becomes almost deducible from the question alone, or that name upcoming entities that will themselves be annotation targets later. - If the question is "Which newspaper published an article about Narrabri in July 1913?" and "July 1913" only comes from a later sentence (and may itself be a future annotation), the question leaks both chronology AND a future answer. - If the question is "Who won the 2018 World Cup hosted by Russia?" and both "2018" and "Russia" are planned annotation targets later in the document, this question reveals both of them. **Self-containment test (apply to every question):** 1. Cover up the tagged span with your finger. Read the question alone. 2. Ask: could this question be typed into Google by someone who has NEVER read my document? Is it clear what they are asking? (If no: under-specified.) 3. Ask: does the question itself contain or strongly hint at information that appears LATER in this document, especially information that will become another annotation answer? (If yes: over-specified / leaking.) 4. A well-formed question names the subject (what/who/where) and includes just enough non-answer, non-future context to disambiguate -- nothing more. <example> Input: Mount Kenya is the second-highest peak in Africa. WRONG q="What is it the second-highest of?" (under-specified -- "it" has no referent outside this document) WRONG q="What is the second-highest peak in Africa, located on the equator in Kenya?" (over-specified with later details that leak location) CORRECT q="What is the second-highest peak in Africa?" a="Mount Kenya" </example> --### COVERAGE PRINCIPLE (READ THIRD) Salient facts often hide mid-sentence, not as subjects. Before moving past a sentence, sweep for these frequently-missed fact types and tag them if they pass the other rules: **Named entities embedded mid-sentence.** Any specific person, organization, place, product, or work of art that names a unique referent deserves

26

consideration even when it is in an oblique position (object of a preposition, apposition, embedded clause). Examples of easily-missed cases: - "...the search by Vyacheslav Slysh for Dyson spheres around Moscow..." -- tag "Vyacheslav Slysh". - "...the track was written with co-producer Yeimer Lopez..." -- tag "Yeimer Lopez". - A scientist, athlete, author, director, or company name buried in a subordinate clause is still a who-question’s answer. **Short numeric and temporal modifiers.** Durations, counts, ages, distances, short date phrases. These are easy to overlook when they are not the sentence’s headline number. Examples: - "...trained for six days before the event..." -- tag "six days" (how long). - "...scheduled for Easter Wednesday..." -- tag "Easter Wednesday" (when). - "...priced at around $4.50 per unit..." -- tag "$4.50" or "$4.50 per unit" as appropriate. **Concise action/process definitions for proper-noun subjects.** When the document describes what a named protocol, function, or system does in a short phrase, tag that phrase as the answer to "What does X do?". Example: - "SMTP transfers email to the email client’s computer." -- tag "transfers email to the email client’s computer" with q="What does SMTP do?". **Coverage heuristic:** For each sentence, list every plausible who / what / when / where / how-many question a search user might ask about the sentence ’s content. For each such question, check whether the unique answer appears in the sentence. If yes, verify it passes Rule 1 (no forward references) and Rule 9 (salience) and tag it. This Coverage Principle does NOT override the salience bar: decorative color and roster enumeration items remain skippable. It is a reminder to not skip central facts just because they are grammatically embedded. --### THE 9 RULES **Rule 1: NO FORWARD REFERENCES (STRICT SENTENCE-BOUNDARY TEST)** Questions can only use info that appears BEFORE the answer in the text. **Verification procedure -- apply this for EVERY question you write:** 1. Identify the sentence containing the tagged span. 2. Collect all text from the beginning of the document up to and including that sentence. This is the "available context." 3. Every entity, date, descriptor, and detail mentioned in the question MUST appear in the available context. If any word or phrase in your question comes from a later sentence, the question is invalid. 4. **Cross-contamination check:** Re-read your question and confirm it does not reveal or strongly imply the answer to any OTHER fact that appears later in the document. If it does, rephrase to remove the leaking detail. 5. **Self-containment check:** Apply the Question Self-Containment test above. The question must work for a reader who has never seen the document. If two facts define each other, tag only ONE (the rarer fact). <example> Input: Cuyamaca Reservoir is located on Boulder Creek Good: Cuyamaca Reservoir is located on <FACT q="Where is Cuyamaca Reservoir located?" a="Boulder Creek">Boulder Creek</FACT> Cannot tag both - each would need the other’s info. </example>

27

COMMON VIOLATION - referencing later info in the question: <example> Input: "an article by Sydney Evening News journalist about work at Narrabri in July 1913" WRONG: <FACT q="Which newspaper published an article about Narrabri in July 1913?" a="Sydney Evening News">Sydney Evening News</FACT> (The question mentions "July 1913" which appears AFTER "Sydney Evening News" in the text) CORRECT: <FACT q="Which newspaper published an article about the work at Narrabri?" a="Sydney Evening News">Sydney Evening News</FACT> </example> **Structured / tabular documents:** When a document has repeating sections (e.g ., race results, treaty timelines, year-by-year summaries), treat each section or paragraph as an independent unit. Do NOT let details from one section bleed into questions about an earlier section. Verify each question against only the text up to the span’s position, not the entire document. If no single fact can stand alone, tag a larger phrase: <example> Beginning in 1991, Darcy Frey <FACT q="What did Darcy Frey start doing in 1991?" a="spent nine months with the Abraham Lincoln High School basketball team">spent nine months with the Abraham Lincoln High School basketball team</FACT>. </example> **Rule 2: COMPLETE DATA TUPLES** If a number appears before its description, tag the entire phrase: <example> In 2016, <FACT q="What was the linguistic breakdown of Albertans in 2016?" a ="76% </example> **Rule 3: STANDALONE QUESTIONS** Replace pronouns with entity names. Don’t reference "the text" or "the passage". Frame questions as natural search queries someone would actually type. Refer to the Question Self-Containment Principle above. For lists of items (X and Y), keep them together rather than splitting with " another": <example> WRONG: <FACT q="Name a tributary" a="Disang">Disang</FACT> and <FACT q="Name another tributary" a="Dikhou">Dikhou</FACT> CORRECT: <FACT q="What were the tributaries?" a="the Disang and the Dikhou"> Disang and Dikhou</FACT> </example> **Rule 4: DEFINITIONS - Proper Nouns Only** Tag definitions of Proper Nouns. Do NOT tag common noun/dictionary definitions. When text provides examples of a category, annotate the entity name asking " What is an example of [category]?": <example> Input: NaCl is an example of an ionic substance. CORRECT: <FACT q="What is an example of an ionic substance?" a="NaCl">NaCl</ FACT> is an example of an ionic substance. </example> <example> Masuzawa Station is a <FACT q="What type of facility is Masuzawa Station?" a=" railway station">railway station</FACT>. <- TAG (proper noun) "A multi-tool is a device that combines tools." <- NO TAG (common noun)

28

</example> **Rule 5: SPLIT COMPOUND FACTS** If a span answers multiple questions (Who, When), split it unless it violates Rule 1: <example> The award was presented by <FACT q="Who presented the X award?" a="the Queen"> the Queen</FACT> in <FACT q="When was the X award presented?" a ="1990">1990</FACT>. </example> **Rule 6: NO DEDUCIBLE INFO -- INCLUDING COMMON KNOWLEDGE** Don’t tag what can be inferred from common sense or is universally known. This includes: - Properties that follow from a category: "Made of silk, the fabric is soft." -> NO TAG - Logical deductions: "The market is busy in summer." -> NO TAG - Facts any educated adult would know without looking up (e.g., "digestion begins in the mouth", "the Earth orbits the Sun", "water boils at 100C at sea level") -> NO TAG - Facts that are self-evident from the document’s title or topic statement -> NO TAG **Key test:** Would someone actually search for this fact? If not, do NOT tag it. <example> "The human digestive process begins in the mouth." -> NO TAG (no one would search for this) "Paris is the capital of France." -> NO TAG (no one needs to look this up) "The Battle of Gettysburg was fought in 1863." -> TAG (someone might search " when was the Battle of Gettysburg") </example> **Rule 7: NO GENERIC SCRIPTS** Don’t tag actions implied by someone’s role. Only tag specific details: - "The firefighter extinguished the blaze." -> NO TAG (expected job) - "Led Zeppelin performed songs." -> NO TAG (bands perform songs) - "Led Zeppelin performed <FACT q="Which song did Led Zeppelin play?" a=" Stairway to Heaven">Stairway to Heaven</FACT>." -> TAG (specific) **Rule 8: NO CONTEXT-SPECIFIC KNOWLEDGE** Don’t tag knowledge only meaningful within this document (e.g., "formula_11"). **Rule 9: ANNOTATION IMPORTANCE THRESHOLD** Only tag facts that carry **high information density** -- facts that a reader would genuinely need to look up or that would be difficult to recall from memory. Do NOT tag: - Incidental or secondary details that merely elaborate on an already-tagged primary fact (e.g., if you tagged the venue name, do not also separately tag its general category like "circus" or "arena"). - Facts that are immediately obvious from the surrounding context or document title. - Minor descriptors or qualifiers that add little retrievable value. - Well-known facts that appear in the document merely as background context rather than as the document’s informational contribution. **Calibration guide:** Aim for roughly the same density as a well-edited encyclopedia article’s hyperlinks. If you find yourself generating significantly more annotations than there are sentences in a passage, reexamine whether each annotation truly passes the importance test. Conversely, if a fact would be hyperlinked in a Wikipedia article (named

29

entities, specific dates, specialized terms), it almost certainly deserves a tag. --### PARAPHRASE RULES FOR THE ‘a‘ ATTRIBUTE The ‘a‘ value must be a **snippet-form** paraphrase of the tagged span. It must NOT be a full sentence or clause that answers the question. **Core principle:** The paraphrase must be the kind of raw text that could appear inline in another document -- as a caption, list entry, table cell, headline fragment, or mid-sentence noun phrase. It must NOT be a subject+ verb clause framed as "X did Y", "the race was Y", "Y was at/on Z", etc. **Acceptance test:** Could this exact ‘a‘ string appear verbatim as a caption, table cell, list entry, or mid-sentence noun phrase in an unrelated article ? If no, compress to the minimal noun phrase that carries the same factual content. Rules: 1. The ‘a‘ value must contain the answer to the question in ‘q‘. 2. It must be snippet form -- a phrase, name, number, date, or short noun phrase. Not a complete sentence. 3. It should convey the fact without looking copy-pasted from the span, but must not inflate into a complete sentence. 4. Length may be slightly longer or shorter than the span. A single word may become a short phrase; a wordy span may be condensed. Length must stay in the snippet range. 5. Proper nouns, numbers, and dates must be preserved exactly in factual content. Only surface form can change (e.g. "21 May 1925" -> "May 21, 1925"). Additional information may be added but not removed. 6. Lists of items should be shuffled -- do not maintain the original ordering unless the order is inherently meaningful. 7. If there is no meaningful way to paraphrase the span, keeping ‘a‘ identical to the span text is acceptable. Proper nouns and venue names in particular should usually be kept as-is. 8. Be diverse -- vary your paraphrasing strategies. Sometimes keep dates/names as-is; sometimes use a surface variant (e.g. "Sep" for "September"); sometimes swap the order of a date’s components. Apply this variety across all fact types. --### COMPLETE EXAMPLES <example> Input: Stefan Gierowski Stefan Gierowski (21 May 1925 -- 14 August 2022) was a Polish painter and an avant garde artist of post-war Poland. Output: Stefan Gierowski Stefan Gierowski (<FACT q="When was Stefan Gierowski born?" a="May 21, 1925">21 May 1925</FACT> -- <FACT q="When did Stefan Gierowski die?" a="August 14, 2022">14 August 2022</FACT>) was a <FACT q="What was Stefan Gierowski’s nationality?" a="Polish">Polish</FACT> <FACT q="What was Stefan Gierowski’s profession?" a="painter">painter</FACT> and <FACT q="Stefan Gierowski is considered a representative of which artistic movement?" a="the avant-garde movement of post-war Poland">an avant garde artist of post-war Poland</ FACT>.

30

</example> <example> Input: Professionalism/The Over Prosecution of Kurt Mix On April 22, 2010 the BP-operated Deepwater Horizon oil rig exploded in the Gulf of Mexico. The explosion killed 11 men who worked on the rig. Kurt Mix, a BP engineer, was a first responder working on operation "Top Kill", a plan to stop the oil spillage. Output: Professionalism/The Over Prosecution of Kurt Mix On <FACT q="When did the Deepwater Horizon explosion occur?" a="April 22, 2010">April 22, 2010</FACT> the <FACT q="What event happened on April 22, 2010?" a="the BP-operated Deepwater Horizon oil rig exploded">BP-operated Deepwater Horizon oil rig exploded</FACT> in <FACT q="Where did the Deepwater Horizon explosion occur?" a="the Gulf of Mexico">the Gulf of Mexico</FACT>. The explosion killed <FACT q="How many people were killed in the Deepwater Horizon explosion?" a="11 men">11 men</FACT> who worked on the rig. Kurt Mix, a <FACT q="Who was Kurt Mix’s employer?" a="BP">BP</FACT> <FACT q=" What was Kurt Mix’s profession?" a="engineer">engineer</FACT>, was a <FACT q="What was Kurt Mix’s role in the Deepwater Horizon disaster?" a="first responder">first responder</FACT> working on <FACT q="What operation did Kurt Mix work on?" a="operation Top Kill">operation "Top Kill"</FACT>, a plan to <FACT q="What was the objective of operation Top Kill?" a="stop the oil spillage">stop the oil spillage</FACT>. </example> <example> Input: Linear Algebra/Vector Spaces A vector space is a way of generalizing... Further results may be applied to more general spaces which may have infinite dimension, such as in Functional Analysis. Output: Linear Algebra/Vector Spaces A vector space is a way of generalizing... Further results may be applied to more general spaces which may have infinite dimension, such as in <FACT q=" Which field involves spaces with infinite dimension?" a="Functional Analysis">Functional Analysis</FACT>. *(Note: Most content is common math knowledge - only tag specific proper nouns) * </example> --## Quality Focus When unsure about whether to tag: - Common knowledge -> don’t tag - Question needs info from AFTER answer -> don’t tag - Can’t make question standalone -> don’t tag - Named person/place/organization or short numeric/temporal modifier that IS the unique answer to a plausible search query -> tag, even if embedded midsentence Apply these rules uniformly across all document types -- formal articles, informal web text, lists, transcripts, and technical documents all contain extractable facts. Preserve original text exactly within FACT tags.

31

Output only the annotated text.

32