arXiv CSopen access

MeMo: Memory as a Model

arXiv:2605.15156v1 [cs.CL] 14 May 2026

M E M O: Memory as a Model

Ryan Wei Heng Quek1,2,3,4∗ Sanghyuk Lee5,6,7 ∗ Alfred Wei Lun Leong4,8 ∗ Arun Verma9 ∗† Alok Prakash9 Nancy F. Chen3 Bryan Kian Hsiang Low1,2,4,9 Daniela Rus7,9 Armando Solar-Lezama7,9 1 Institute of Data Science, National University of Singapore, Singapore 2 Integrative Sciences and Engineering Programme, NUSGS, Singapore 3 Agency for Science, Technology, Research (A*STAR), Singapore 4 Department of Computer Science, National University of Singapore, Singapore 5 University of Tokyo, Japan 6 Liquid AI, USA 7 CSAIL, Massachusetts Institute of Technology, USA 8 AI Singapore 9 Singapore-MIT Alliance for Research and Technology Centre, Singapore [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

Abstract Large language models (LLMs) achieve strong performance across a wide range of tasks, but remain frozen after pretraining until subsequent updates. Many realworld applications require timely, domain-specific information, motivating the need for efficient mechanisms to incorporate new knowledge. In this paper, we introduce M E M O (Memory as a Model), a modular framework that encodes new knowledge into a dedicated M EMORY model while keeping the LLM parameters unchanged. Compared to existing methods, M E M O offers several advantages: (a) it captures complex cross-document relationships, (b) it is robust to retrieval noise, (c) it avoids catastrophic forgetting in the LLM, (d) it does not require access to the LLM’s weights or output logits, enabling plug-and-play integration with both open and proprietary closed-source LLMs, and (e) its retrieval cost is independent of corpus size at inference time. Our experimental results on three benchmarks, BrowseComp-Plus, NarrativeQA, and MuSiQue, show that M E M O achieves strong performance compared to existing methods across diverse settings.

1

Introduction

Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks [1–3]. Despite their successes, these models are effectively frozen for extended periods after pretraining [4] until subsequent updates, causing their pretrained knowledge to become increasingly outdated as the world evolves. For applications that require up to date [5, 6] or domain-specific [7, 8] knowledge, this dependence on static knowledge presents a fundamental architectural limitation [9, 10]. Retraining is a natural solution but remains prohibitively expensive at modern scales [11], motivating the need for an efficient mechanism to integrate new external knowledge into LLMs without full retraining. 1 NonExisting methods for integrating new knowledge into LLMs fall into three categories. ⃝ parametric methods retrieve relevant information from an external store at inference time via lexical [12], dense [13], or graph-based retrievers [14–17], before incorporating it through in-context learning [18, 19]. However, these methods are constrained by limited context windows and struggle to synthesize cross-document relationships when relevant information is distributed across multiple ∗ Equal contributions and † Corresponding author.

Training

Inference

Generator Model Data Synthesis Pipeline Fact Extraction

Target Corpus

Consolidation

Entity Surfacing

Who is Linda to Earl?

Verification and Rewriting

Cross-Document Synthesis

Executive Model

Training

Reflections

Memory Model

Linda is Earl’s wife (and caregiver during his final illness).

Multi-Turn Protocol Stage 1: Grounding ...

...

Stage 2: Entity Identification ... ... Stage 3: Answer Synthesis ...

Memory Model

...

Figure 1: Overview of the training and inference pipeline of M E M O. During M EMORY model training (left), a frozen G ENERATOR model transforms a target corpus into a reflection QA dataset via fact extraction, consolidation, verification, entity surfacing, and cross-document synthesis, which is then used to train a dedicated M EMORY model. During inference (right), the frozen E XECUTIVE model answers complex user queries by querying the M EMORY model through a structured multi-turn protocol: it decomposes the input into simpler, targeted sub-queries, retrieves intermediate responses from the M EMORY model, and reasons over them to produce a final answer to the user’s query. 2 Parametric methods internalize knowledge directly into model parameters documents [20, 21]. ⃝ via continual pretraining [22] or fine-tuning [23–25] on the target corpus directly. While effective, they are computationally expensive, prone to catastrophic forgetting [26], and tend to memorize training distributions rather than acquire transferable knowledge, limiting generalization to unseen 3 Latent memory methods [28–31] compress knowledge into soft tokens or other queries [27]. ⃝ model-specific representations, but suffer from representation coupling: the memory is tightly bound to the specific model used to produce these representations, limiting transferability across LLMs.

We introduce M E M O (Memory as a Model), a modular framework where a dedicated M EMORY model is trained on new knowledge, and an E XECUTIVE model retrieves relevant information from the M EMORY model at inference time via targeted sub-queries and then reasons over the retrieved information to respond to user queries. M E M O combines the complementary strengths of the three paradigms above while mitigating their individual limitations. Like the non-parametric methods, it is able to leverage off-the-shelf frontier models unchanged by separating the memory from the reasoning model; it shares with the parametric methods the ability to internalize knowledge in model parameters, and it shares the benefits of a compact, queryable memory artifact with latent memory methods. As a result, M E M O offers the following advantages: (a) it captures complex cross-document relationships, (b) it is robust to retrieval noise, (c) it avoids catastrophic forgetting by keeping the E XECUTIVE model parameters unchanged, (d) it does not require access to the E XECUTIVE model’s weights or output logits, enabling plug-and-play integration with both open and proprietary LLMs, and (e) its retrieval cost is independent of corpus size at inference time due to the fixed size of the M EMORY model. However, designing M E M O to comprehensively capture cross-document relationships during training while accurately answering arbitrary queries at inference time introduces two key challenges, which we outline below and address them with novel methods. 1 Training M EMORY model. A core challenge in the M EMORY model is ensuring it can accu⃝

rately answer diverse, unseen queries at inference time, including those requiring cross-document reasoning and long-context understanding. A natural approach is to train directly on the raw corpus using standard data augmentation techniques such as paraphrasing [32–34], additional sampling of generated QA pairs [35, 36], or targeted gap-filling, where the model identifies and completes missing knowledge from the corpus [37, 38]. However, these approaches fail to consolidate related facts into compositional representations necessary for robust generalization to unseen queries [27]. With this challenge in mind, we design a novel five-step data synthesis pipeline guided by a G ENERATOR model (Sec. 4.1) that distills the corpus into a question–answer (QA) dataset of reflections: compositional representations that expose underlying corpus knowledge under diverse query variations (illustrated in Fig. 1 (left) and details in Sec. 4.1). We train M EMORY model on the synthesized reflection QA dataset via supervised fine-tuning (see Sec. 4.2), enabling M EMORY model to capture more complex, cross-document relationships and compositional structure than retrieval-based methods. 2 Querying M EMORY model. At inference time, complex or compositional queries often require ⃝

multi-step reasoning and aggregation of information across multiple documents. Naively querying M EMORY model via single-turn or unstructured multi-turn interactions fails to reliably retrieve the 2

knowledge required to answer such queries. To address this, we design a three-stage inference pipeline in which E XECUTIVE model queries and retrieves information from M EMORY model via a structured multi-turn protocol, decomposing complex user queries into targeted sub-queries that align with the shared reflection interface (illustrated in Fig. 1 (right) and more details are in Sec. 4.4). Unlike retrieval-based methods, this approach incurs retrieval cost independent of corpus size and is robust to retrieval noise (see Sec. 5.2) Crucially, because M E M O treats E XECUTIVE model as a black box and does not access its weights, gradients, or output logits, it supports plug-and-play integration with any LLM, including both both open and proprietary closed-source models. Our method is guided by a single design principle: reflections, corpus-derived structures that require no knowledge of future queries, yet naturally serve as the precise interface through which any query can access the underlying corpus without ever observing it directly. During training, the M EMORY model internalizes these reflections; E XECUTIVE model retrieves relevant knowledge through targeted sub-queries at inference time. Building on the challenges outlined above and the methods proposed to address them, we summarize the key contributions of this paper as follows: • Novel data synthesis pipeline. We propose a five-step data synthesis pipeline that uses a G EN ERATOR model, an LLM that may be the same as or smaller than E XECUTIVE model, to distill a target corpus into reflections, enabling a dedicated M EMORY model to internalize knowledge in compositional forms that capture more complex cross-document relationships and generalize robustly to diverse, unseen query variations at inference time (see Sections 4.1 and 4.2). • Structured multi-turn protocol. We introduce a structured multi-turn protocol that systematically decomposes complex queries into targeted sub-queries aligned with the shared reflection interface. The protocol supports plug-and-play integration with any arbitrary LLM, including proprietary closed-source LLMs, and has retrieval cost independent of corpus size (see Sec. 4.4). • Empirical validation. We evaluate M E M O on BrowseComp-Plus, NarrativeQA, and MuSiQue, demonstrating strong performance against both parametric and non-parametric baselines. We further empirically validate M E M O’s robustness to retrieval noise (see Sec. 5).

2

Related Work

Non-parametric methods. Non-parametric alternatives [12, 13, 17] avoid parameter updates entirely, instead, supplying new knowledge at inference time. In particular, in-context learning (ICL) [18, 19] inserts relevant knowledge directly into the prompt, avoiding catastrophic forgetting. However, ICL scales poorly with increasing context length: the computational cost of autoregressive generation [39] leads to substantial token overhead and inference latency as the knowledge base grows [40], and even explicitly long-context models exhibit significant performance degradation as context length increases [41, 42]. Retrieval-augmented generation (RAG) [14–17] addresses this scalability bottleneck by selectively retrieving relevant chunks of knowledge rather at inference time. However, RAG systems are highly sensitive to retrieval noise [43], where irrelevant or misleading passages substantially degrade generation quality [44, 45]. In addition, RAG systems often struggle to reason over complex cross-document dependencies [20], as they lack robust mechanisms for synthesizing information that is distributed across multiple chunks or a large corpus [21]. Parametric methods. Existing post-training approaches, such as continual pretraining on new corpora [22, 46] or supervised fine-tuning (SFT) on curated instruction data [23–25], attempt to address this limitation by incorporating new knowledge into LLMs during post-training. While conceptually straightforward, these parametric methods often suffer from catastrophic forgetting, whereby adaptation to newly observed knowledge degrades previously acquired knowledge, learned capabilities [26, 47, 48], and erodes safety alignment learned during LLM post-training [49]. In addition, the scale of modern LLMs makes frequent fine-tuning computationally expensive [50, 51], and fine-tuning is often infeasible for proprietary, closed-source models [52], substantially limiting the practicality of parametric methods in real-world, large-scale applications. Latent memory methods. Another approach to storing knowledge is via compressed latent representations, which lie between non-parametric retrieval and fully parametric methods. Context compression techniques such as AutoCompressor [28], Gist tokens [29], and ICAE [30] encode knowledge into compact soft tokens prepended at inference, reducing ICL token overhead without discarding information. However, these representations are tightly coupled to the encoder and cannot be consumed by other model families, limiting compatibility with black-box LLMs. Similarly, 3

recurrent-state models [53, 54] and nearest-neighbor memory methods such as Memorizing Transformers [55] and kNN-LM [56] rely on model-specific representations or architectures, preventing post hoc use with pretrained LLMs. Although Memory Decoder [57] is a plug-and-play pretrained memory module that integrates without modifying model parameters, it is limited to architectures sharing a common tokenizer, enabling reuse only within this subset. The core limitation of these methods is representation coupling: latent memory is inseparable from the model that produces it. In contrast, M E M O allows a plug-and-play integration with any LLM, including closed-source models. Table 1: A comparison of desirable properties across different memory paradigms, showing that M E M O satisfies them through its modular memory construction and memory-augmented reasoning. Methods Non-parametric (RAG, ICL) Parametric (CPT, SFT) Latent memory (AutoCompressor, Gist, ICAE) M E M O (Ours)

3

Frozen base LLM ✓ × ✓ ✓

No retrieval index × ✓ ✓ ✓

Black-box compatible ✓ × × ✓

No catastrophic forgetting ✓ × ✓ ✓

Constant-size memory × × ✓ ✓

Cross-LLM transferable ✓ × × ✓

Preliminaries

Problem setting. Let Mθ denote a large language model with frozen parameters θ ∈ Rp , pretrained on a corpus Dpre . We treat Mθ as a conditional distribution that maps a prompt x to a response Mθ (x), and assume only black-box access; in particular, Mθ may be either a white-box model or a closed-source model accessed via API. Let D = {d1 , . . . , dN } denote a target corpus of N documents containing knowledge that Mθ cannot reliably recall2 . Let Q be a set of queries, each q ∈ Q associated with a ground-truth answer a⋆ (q) and a set of supporting documents S(q) ⊆ D. Note that S(q) is a theoretical construct used to characterize query complexity. Knowledge integration mechanism. A knowledge integration mechanism is a pair (Φ, f ), where Φ . maps the corpus to a representation K = Φ(D) and f combines K with Mθ at inference to produce responses f (Mθ , K, q). We formalize the goal as follows. Definition 1 (Knowledge Integration Problem). Given a frozen model Mθ and target corpus D, find a mechanism (Φ, f ) such that, without modifying θ, for all q ∈ Q, P {f (Mθ , Φ(D), q) = a⋆ (q)} = 1. Existing approaches. Existing methods differ in their choice of (Φ, f ). ICL sets K = D and f (Mθ , K, q) = Mθ ([D; q]), i.e., appending the corpus directly to the prompt. RAG constructs K as a retrieval index and defines f to retrieve a subset Ŝ ⊆ D before passing [Ŝ; q] to Mθ . Fine-tuning sets K = ∅ and f = Mθ′ , where θ′ is obtained by updating θ on D. In contrast, M E M O defines K as the parameters of a small, dedicated M EMORY model Mφ with φ ≪ θ, trained on reflection QA dataset derived from D, and queried by a frozen E XECUTIVE model Mθ at inference time. Tab. 1 summarizes how these paradigms compare across desirable properties.

4

MeMo: Memory as a Model

M E M O addresses the knowledge integration problem (Def. 1) through two components: a frozen model Mθ (E XECUTIVE model), which handles reasoning and responds to user queries, and a M EMORY model Mφ , which is trained to encode knowledge in its parameters from a target corpus D. Our pipeline operates in two phases: (i) a training phase that constructs M EMORY model from D, and (ii) an inference phase in which E XECUTIVE model queries and retrieves information from M EMORY model to answer knowledge-intensive questions (see Sections 4.1, 4.2 and 4.4). 4.1

Data Synthesis Pipeline

Given a corpus of documents D, our objective in the data generation process is to construct a reflection QA dataset Qfinal that captures both single-document facts and cross-document relationships. This process is driven by a G ENERATOR model Mgen and proceeds through five steps, as summarized in 2We do not assume D is disjoint from D , as training data is rarely disclosed by model providers. A document is pre considered effectively absent from Mθ ’s knowledge if the model fails to answer questions grounded in it, either because it never appeared in Dpre or because the training process was insufficient to retain it. For more information, refer to App. I.

4

Alg. 1 and illustrated in Fig. 1: (1) fact extraction from raw documents, (2) consolidation of redundant or overlapping information, (3) verification and rewriting to ensure correctness and clarity, (4) entity surfacing to explicitly represent key entities, and (5) cross-document synthesis to integrate evidence across the corpus. Importantly, no document identifiers or watermarks are embedded in the generated QA pairs at any step, preventing M EMORY model from exploiting shortcut signals during evaluation. Algorithm 1 Reflection QA Dataset Generation Pipeline from Target Corpus Require: Corpus D, generator Mgen , document groups G = {G1 , . . . , Gk } with Gi ⊆ D 1: Qfinal ← ∅ 2: for all document d ∈ D do 3: C ← Chunk(d) ▷ Segment into chunks 4: Qdver ← ∅ 5: for all chunk c ∈ C do 6: Qdir , Qindir ← Mgen (c) ▷ Step 1: Direct and indirect extraction 7: Qraw ← Qdir ∪ Qindir ▷ Step 2a: Merge direct and indirect 8: Qmrg ← Mgen (Qraw ) ▷ Step 2b: Consolidate related pairs 9: Qcon ← Qraw ∪ Qmrg ▷ Step 2c: Full merge set 10: Qver ← Mgen (Qcon , c) ▷ Step 3: Verify self-containment; rewrite or discard 11: Qdver ← Qdver ∪ Qver 12: end for 13: Qdent ← Mgen (Qdver ) ▷ Step 4: Entity-surfacing pairs 14: Qfinal ← Qfinal ∪ Qdver ∪ Qdent 15: end for 16: for all Gi ∈ G do
 S ▷ Step 5: Cross-document synthesis 17: Qcross ← Mgen d∈Gi Qdver ∪ Qdent 18: Qfinal ← Qfinal ∪ Qcross 19: end for 20: return Qfinal Step 1: Fact extraction. Each document d ∈ D is segmented into chunks C, where each chunk corresponds either to an entire document or to a contiguous segment of a longer document. For each chunk, Mgen performs two parallel extraction processes: direct extraction, which captures explicitly stated facts (producing Qdir ), and indirect extraction, which targets inferred or synthesized information beyond the surface text (producing Qindir ). This dual extraction process ensures that both factual recall and inferential reasoning are represented in the training signal for M EMORY model. Step 2: Consolidation. The G ENERATOR model Mgen consolidates Qdir ∪ Qindir by identifying QA pairs that share a common underlying context (such as entity, time period, or relationship type) and combining them into QA pairs that encompass multiple facts, denoted Qmrg . This merging process produces training instances that require integrating multiple facts within the same contextual chunk, going beyond single-fact question answering pairs. The synthesized QA pairs are subsequently unified with the original sets to form the consolidated dataset Qcon = Qdir ∪ Qindir ∪ Qmrg . Step 3: Verification and rewriting. Each QA pair in Qcon is evaluated for self-containment by Mgen , i.e., whether it can be fully understood and correctly answered in isolation, without access to the source chunk. Common failure modes include unresolved pronouns (e.g., “What did they propose?”) and implicit references (e.g., “As noted in the above table. . . ”). Non-self-contained QA pairs are rewritten by Mgen using the source chunk C as context; QA pairs that remain ambiguous after rewriting are discarded. This check-and-rewrite procedure yields the verified set Qver , a set of QA pairs that can be used as training examples without access to the source chunk. Step 4: Entity surfacing. For each named entity in Qver , Mgen generates a set of entity-surfacing QA pairs in which the question encodes the entity’s attributes and relationships (including connections to other named entities) and the answer reveals its identity. Facts about each entity are aggregated across all QA pairs within the chunk prior to generation, enabling the integration and composition of information from multiple source pairs. Questions are generated at varying levels of complexity, ranging from single-fact to multi-fact queries. These pairs, denoted Qent , aim to mitigate the reversal curse [58, 59] by training M EMORY model to infer entities from indirect or partially specified descriptions. This capability supports the entity identification turn at inference time (Sec. 4.4). 5

Step 5: Cross-document synthesis. The final step operates over pre-defined document groups G = {G1 , . . . , Gk }, where chunks within each group Gi are topically related. Such groups arise naturally, for example, when a large document is segmented into chunks (forming a single group) or from human-provided labels. For each group Gi , Mgen is provided with the entity-surfacing pairs Qdent : d ∈ Gi from all member documents and identifies two types of cross-document connections: • Converging clues: multiple documents provide complementary facts about the same entity, which together enable its identification. • Parallel properties: different entities across documents share a common attribute or role, enabling comparative and analogical reasoning. Both types yield QA pairs with support size s(q) > 1 (Sec. 3), directly targeting the cross-document synthesis objective. The final dataset is Qfinal = Qver ∪ Qent ∪ Qcross , which collectively captures self-contained, entity-centric, and cross-document reflections for training M EMORY model. Ablations of the pipeline design are presented in App. E. 4.2

Training the M EMORY model

Given Qfinal , M EMORY model is trained via supervised fine-tuning to map questions directly to answers without access to source documents at inference time. M EMORY model is initialized from a small pretrained language model, substantially smaller than E XECUTIVE model (e.g., 1.5B vs. 32B parameters), and optimized by minimizing the next-token prediction loss over answer tokens only. X

|ai | X

(qi , ai ) ∈ Qfinal

t=1

L(φ) = −

  (t) (1:t−1) log Mφ ai qi , ai .

Conditioning only on the question and preceding answer tokens, and never on source documents, forces M EMORY model to internalize knowledge parametrically rather than rely on copying from retrieved context. This constitutes a key distinction from RAG-based readers: at inference time, M EMORY model generates answers solely from its internalized parametric knowledge, without access to any external corpus. Further details on hyperparameter choices and training paradigms (full SFT vs. LoRA) are provided in App. F and App. O, respectively. 4.3

Continual Knowledge Integration via Model Merging

A practical desideratum of any knowledge integration system is the ability to incorporate new corpora incrementally without retraining on or rebuilding from all previously ingested sources. For parametric models, integrating new knowledge typically requires retraining on the union of all observed corpora, a cost that grows prohibitively with the number of sources. In contrast, non-parametric systems such as knowledge graphs and vector databases support efficient incremental updates. We explore model merging [60] as an approach to close this gap for parametric models. Model merging aims to preserve knowledge from multiple sources without requiring joint training on their union, by combining K M EMORY model models, each trained independently on a distinct corpus, into a single model. Continual knowledge integration. Let {D1 , . . . , DK } be a collection of pairwise disjoint target (i) corpora. For each corpus Di , we generate a reflection QA dataset Qfinal (Sec. 4.1) and train a corresponding M EMORY model Mφi via SFT (Sec. 4.2), initializing all K models from the same pretrained base Mφ0 . We define the task vector for Di as τi = φi − φ0 , capturing the parametric shift induced by training on Di alone. The merged M EMORY model is then obtained as φmerged = Merge(φ0 , {τi }K i=1 ; Θ), where Θ denotes method-specific hyperparameters (e.g., merging coefficients, sparsification densities). We discuss alternative merging methods and their respective limitations in App. H. 4.4

Inference-Time Integration

At inference time, E XECUTIVE model queries and retrieves information from M EMORY model through a structured multi-turn protocol, with E XECUTIVE model treating M EMORY model as an external knowledge oracle. The pipeline has three sequential stages, each designed to progressively 6

improve the likelihood of producing a correct final answer, as illustrated in Fig. 1 (right). Each stage utilizes distinct prompts, sampling temperatures and independent budgets to control the number of interactions between E XECUTIVE model and M EMORY model. Stage 1: Grounding. Given a query q, E XECUTIVE model decomposes it into a set of atomic, clue′ probing sub-questions {q1′ , . . . , qK }, where each sub-question targets a single identifying constraint in q, and K is adaptively determined by E XECUTIVE model. The M EMORY model answers each sub-question independently, without shared context, producing grounding responses {m1 , . . . , mK }. These responses draw on M EMORY model’s parametric knowledge to provide additional contextual grounding for subsequent interactions in the later stages. Stage 2: Entity identification. Using the grounding responses as context, E XECUTIVE model iteratively narrows a set of candidate entities by issuing targeted follow-up sub-queries to M EMORY model across multiple interactions. This process continues until E XECUTIVE model converges on a single entity e⋆ or the stage budget is exhausted. If no candidates are identified, Stage 3 is skipped and E XECUTIVE model synthesizes a final answer from the grounding responses alone. This stage leverages M EMORY model’s training on the entity-surfacing QA pairs Qent (Sec. 4.1). Stage 3: Answer seeking and synthesis. Conditioned on the identified entity e⋆ , E XECUTIVE model queries M EMORY model for additional supporting facts through targeted follow-up questions. Once sufficient evidence is gathered, or the stage budget is exhausted, E XECUTIVE model synthesizes the accumulated responses into a final answer:
⋆ â = Mθ q, {mk }K k=1 , e , mseek . Notably, the M EMORY model responses mk and mseek are compact natural-language snippets whose lengths are independent of the corpus size, ensuring constant-time inference. As all interactions with Mθ occur through its input–output interface, M E M O remains fully compatible with black-box E XECUTIVE models, including proprietary APIs, without requiring access to internal parameters. For full implementation details, refer to App. J and the supplementary materials.

5

Experiments

Datasets. We evaluate M E M O on three knowledge-intensive benchmarks. BrowseComp-Plus [61] is a deep-research benchmark requiring multi-hop, multi-document retrieval and reasoning; we filter non-English instances with LangDetect [62], sample 300 questions, and pair each question’s evidence documents with an equal number of negative documents,3 yielding 3,541 documents in total. NarrativeQA [63] tests discourse understanding over long documents such as books and movie scripts; we use 293 questions across 104 documents. MuSiQue [64] requires composing 2–4 reasoning steps across multiple Wikipedia paragraphs; we use 1,000 questions and construct the target corpus following the same procedure as for BrowseComp-Plus, yielding 5,296 documents. Further details are in App. D; datasets and code are in the supplementary materials. Baselines. We compare M E M O against four baselines: BM25 [12] (lexical retrieval), NV-EmbedV2 [13] (dense retrieval), HippoRAG2 [17] (graph-based RAG, state-of-the-art), and Cartridges [65] (a trained KV-cache loaded onto E XECUTIVE model at inference; the closest existing parametric baseline to M E M O). Newer methods exist [28, 66] but typically require white-box access to E XECU TIVE model and are therefore not directly comparable. We additionally include Perfect Retrieval as an empirical upper bound, where E XECUTIVE model receives exclusively the evidence documents in context [18]. Retrieval baselines use top-k=9 with adaptive backoff: reducing k progressively until the retrieved context fits E XECUTIVE model’s context window. Implementation and evaluation. (a) Data generation. We use Qwen2.5-32B-Instruct [67] as the G ENERATOR model, served via vLLM [68] with YaRN RoPE scaling [69] to support a 131K-token context window. (b) Training. We train M EMORY model initialized from Qwen2.5-14B-Instruct for 3 epochs with fused AdamW [70] and DeepSpeed 2 [71] at learning rate 2×10−5 ; full hyperparameters in App. F. (c) Evaluation. We instantiate E XECUTIVE model with initialized Qwen2.5-32B-Instruct 3 BrowseComp-Plus and MuSiQue provide annotations of gold (correct), evidence (supporting), and negative (distractor) documents. Gold documents are a subset of the evidence documents. 4We follow HippoRAG2 and evaluate on 10 such documents from the NarrativeQA validation split (294 questions); one duplicate is removed for consistency.

7

or Gemini-3.0-Flash [72] to evaluate the same trained M EMORY model across models of varying reasoning capability; both models have minimal prior knowledge of the evaluation datasets (App. I). E XECUTIVE model queries M EMORY model through the multi-turn protocol described in Sec. 4.4. We report binary accuracy judged by Gemini-2.5-Flash-Lite [73] via DeepEval [74], as mean ± standard deviation over three runs for Qwen2.5-32B-Instruct and a single run for Gemini-3.0-Flash. (d) Continual integration. For the model-merging experiment (Sec. 5.5), we partition NarrativeQA into two pairwise-disjoint subsets (K=2, ∼640k QA pairs each), SFT a separate Qwen2.5-14B-Instruct M EMORY model on each, and sweep six merging methods at three densities (14 configurations total). 5.1

Experimental results

M E M O achieves strong performance across benchmarks. As shown in Table 2, M E M O consistently outperforms all baselines on NarrativeQA and MuSiQue across both E XECUTIVE models. On NarrativeQA, the most challenging benchmark (App. I), M E M O achieves 26.85% with Qwen2.532B-Instruct and 53.58% with Gemini-3-Flash, substantially surpassing all baselines. This is notable: NarrativeQA requires reasoning over long passages with complex connections, where retrieval-based methods are constrained by context windows and struggle to synthesize information across long documents; M E M O instead captures these connections via reflections during training and retrieves them through its multi-turn protocol at inference. The same trend holds on MuSiQue, where M E M O achieves 48.30% and 58.70% respectively, outperforming baselines that struggle with multi-hop reasoning across independently retrieved passages. On BrowseComp-Plus, M E M O leads with Gemini3-Flash (66.67%) and remains competitive with Qwen2.5-32B-Instruct (54.22%, narrowly trailing HippoRAG2’s 56.11%). This gap reflects BrowseComp-Plus’s nature: its answers are absent from E XECUTIVE model’s parametric knowledge (App. I), making direct access to evidence documents especially valuable and favoring retrieval methods that pass raw documents to E XECUTIVE model. Table 2: Accuracy (%) on BrowseComp-Plus, NarrativeQA, and MuSiQue under two E XECUTIVE models: Qwen2.5-32B-Instruct (Qwen2.5-32B-I) and Gemini-3-Flash (Gemini-3-F). Bold values indicate the best result in each column, excluding Perfect Retrieval. M E M O uses Qwen2.5-14BInstruct as M EMORY model, and results are reported at the best training epoch. ⋆ Perfect Retrieval represents an empirical upper bound. Method Perfect Retrieval⋆ BM25 NV-Embed-V2 HippoRAG25 Cartridges6 MEMO

BrowseComp-Plus Qwen2.5-32B-I Gemini-3-F 79.67 ± 1.45 88.33 1.11 ± 0.69 27.00 50.67 ± 0.33 57.00 56.11 ± 0.51 66.33 0.00 ± 0.00 54.22 ± 0.84 66.67

NarrativeQA Qwen2.5-32B-I Gemini-3-F 51.42 ± 0.52 60.41 10.24 ± 0.34 14.33 20.59 ± 0.86 26.62 21.39 ± 0.20 23.21 3.75 ± 0.11 26.85 ± 0.39 53.58

MuSiQue Qwen2.5-32B-I Gemini-3-F 62.83 ± 0.90 73.00 20.00 ± 0.30 23.20 37.47 ± 0.15 46.60 42.17 ± 0.12 57.00 8.57 ± 0.40 48.30 ± 1.25 60.20

M E M O supports plug-and-play integration. Across the three benchmarks, M E M O consistently achieves higher performance when paired with a more capable E XECUTIVE model (Gemini-3-Flash): switching from Qwen2.5-32B-Instruct to Gemini-3-Flash yield gains of 12.45, 26.73, 11.90 pp on BrowseComp-Plus, NarrativeQA and MuSiQue respectively. This demonstrates that M E M O can be trained once with a weaker G ENERATOR model, and seamlessly paired with any LLM at inference — including proprietary models such as Gemini-3-Flash. This plug-and-play capability allows M E M O to directly leverage state-of-the-art models without any additional training or overhead. 5.2

Ablation on the amount of noise for the dataset

We investigate the robustness of M E M O against two strong retrieval-based baselines, NV-Embed-V2 and HippoRAG2, under increasing levels of retrieval noise, controlled by varying the number of negative (distractor) documents added to the target corpus as a multiple of the total number of dataset ground-truth evidence documents in each dataset (Nevidence = 1,775 for BrowseComp-Plus and dataset Nevidence = 2,648 for MuSiQue). The datasets used throughout this paper (detailed in App. D) 5 These results differ from the original paper [17], which uses Llama3.3-70B-Instruct instead of Qwen2.5-32B-Instruct. 6 Cartridges requires white-box access to E XECUTIVE model as well; its results for Gemini-3-Flash are therefore omitted.

8

Table 3: Accuracy (%) on BrowseComp-Plus and MuSiQue with Qwen2.5-32B-Instruct as E XECU TIVE model. M E M O results are based on Qwen2.5-14B-Instruct and reported at the best training dataset epoch. N = Nevidence denotes the number of ground-truth evidence documents in the corpus; column headers indicate the number of additional negative (distractor) documents added, as a multiple of N . ∆ denotes accuracy difference (pp) compared to 0N .

Method

NV-Embed-V2 HippoRAG2 MEMO

Dataset BrowseComp-Plus MuSiQue BrowseComp-Plus MuSiQue BrowseComp-Plus MuSiQue

0×N Acc. (%) 56.89 ± 0.51 42.30 ± 0.53 62.33 ± 1.15 47.33 ± 0.74 53.67 ± 1.15 50.07 ± 0.81

1×N Acc. (%) 50.67 ± 0.33 37.47 ± 0.15 56.11 ± 0.51 42.17 ± 0.12 54.22 ± 0.84 48.30 ± 1.25

∆ ↓ 6.22 ↓ 4.83 ↓ 6.22 ↓ 5.16 ↑ 0.55 ↓ 1.77

dataset dataset correspond to a ratio of 1×Nevidence ; we additionally evaluate at ratio 0×Nevidence (no distractors) as an idealized noise-free reference to isolate the effect of distractors.

Results in Tab. 3 demonstrate that retrieval-based methods exhibit pronounced sensitivity to noise. Both NV-Embed-V2 and HippoRAG2 suffer drops of up to 6.22 pp on BrowseComp-Plus and up to 5.16 pp on MuSiQue when scaling from 0 × N to 1 × N , confirming that these systems struggle to filter irrelevant documents under realistic corpus conditions. In contrast, M E M O maintains stable performance across both benchmarks, with a marginal improvement of 0.55 pp on BrowseComp-Plus and a decline of only 1.77 pp on MuSiQue, both within one standard deviation, demonstrating that M E M O is robust to increasing retrieval noise. We attribute this robustness to M E M O’s design: despite being trained on a corpus containing negative documents, M EMORY model provides more precise information to E XECUTIVE model’s sub-queries than direct document retrieval. Additional analysis of performance degradation in retrieval-based methods is provided in App. L. 5.3

Ablation on M EMORY model size

We investigate how the size of M EMORY model affects downstream task performance by comparing models of 1.5B and 14B parameters in the Qwen2.5 family. Implementation details are provided in App. M. Results in Tab. 4 show a consistent positive scaling trend: larger M EMORY models yield improved performance across all benchmarks and E XECUTIVE models. However, the results also show that a stronger E XECUTIVE model reasoning capability modulates this gap non-uniformly across tasks: the performance difference between M EMORY model sizes widens for NarrativeQA but shrinks for BrowseComp-Plus and MuSiQue. This suggests that the interaction between E XECUTIVE model reasoning capability and M EMORY model size is task-dependent. Table 4: Ablation on M EMORY model size within the Qwen2.5 family. Bold results indicate best performing results in the column. M EMORY Model Qwen2.5-1.5B-Instruct Qwen2.5-14B-Instruct

5.4

BrowseComp-Plus Qwen2.5-32B Gemini-3-Flash 44.11 ± 2.22 61.00 54.22 ± 0.84 66.67

NarrativeQA Qwen2.5-32B Gemini-3-Flash 24.00 ± 0.20 47.44 26.85 ± .39 53.58

MuSiQue Qwen2.5-32B Gemini-3-Flash 42.90 ± 1.39 59.70 50.07 ± 0.81 60.20

Ablation on M EMORY model family

We investigate whether the choice of M EMORY model family affects performance by comparing three models of similar parameter scale (∼1–2B) but distinct architectures and pretraining lineages: Qwen2.5-1.5B-Instruct [67], Gemma3-1B-IT [75], and LFM2.5-1.2B-Instruct [76]. Implementation details are provided in App. N. Results in Tab. 5 show that M E M O performance is largely robust to the choice of M EMORY model architecture, demonstrating that the framework is not sensitive to the specific pretraining lineage of M EMORY model at similar parameter scale, and that the parametric knowledge compression induced by our training procedure generalizes across diverse model families. 9

Table 5: Ablation across M EMORY models at similar parameter scales (∼1–2B). Bold results indicate best performing results in the column. M EMORY Model Qwen2.5-1.5B-Instruct Gemma3-1B-IT LFM2.5-1.2B-Instruct

5.5

BrowseComp-Plus Qwen2.5-32B-I Gemini-3-F 44.11 ± 2.22 61.00 41.67 ± 2.03 59.00 37.33 ± 1.86 59.67

NarrativeQA Qwen2.5-32B-I Gemini-3-F 24.00 ± 0.20 47.44 22.30 ± 2.47 48.81 21.96 ± 1.97 46.42

MuSiQue Qwen2.5-32B-I Gemini-3-F 42.90 ± 1.39 59.70 41.17 ± 1.20 56.20 45.23 ± 2.49 58.30

Continual integration via model merging

We test the streaming-update scenario described in Sec. 4.2 on NarrativeQA, comparing model merging against full retraining of M EMORY model on the union of both subsets when the second arrives. Of the 14 sweep configurations (see Tab. 12, App. H), we report TIES [77] at ρ=0.3 in the main paper, the top-performing one. Letting X and Y denote the SFT cost on each subset alone (cost scales approximately linearly with the number of QA pairs, so the union costs X+Y ), cumulative compute across the two arrivals is X+Y for merging versus X+(X+Y ) for full retraining. Table 6: Model merging vs. full retraining on NarrativeQA. M EMORY model = Qwen2.5-14BInstruct. Merge-TIES (ρ=0.3) is the best of 14 configurations swept (Tab. 12, App. H). Cumulative compute is reported in 8×H100 GPU-hours for K=2 subsets of ∼640k reflection QA pairs each. ∆ denotes accuracy difference (pp) relative to full retraining. Method Full retrain (X+(X+Y )) Merge-TIES (ρ=0.3, X+Y )

Cumulative compute (8×H100 GPU-h) ≈ 72h ≈ 48h

Qwen2.5-32B-I Acc. (%) ∆ 26.85 ± 0.39 — 15.81 ± 0.39 ↓ 11.04

Gemini-3-F Acc. (%) ∆ 53.58 — 34.47 ↓ 19.11

Merging cuts compute by 33% at K=2, with widening returns at scale. As reported in Tab. 6, the full-retrain baseline incurs X+(X+Y ) ≈ 72 GPU-hours of cumulative compute, while merging accumulates only X+Y ≈ 48 GPU-hours — a 33% reduction (Fig. 2). The gap widens with K: under the same per-corpus cost, merging scales as Θ(K) while full retraining scales as Θ(K 2 ), yielding a 5.5× saving at K=10 (240 vs. 1,320 GPU-hours). Merging trades a measurable accuracy gap for the compute saving, but still beats retrieval. Merge-TIES (ρ=0.3) trails the full-retrain M EMORY model by 11.0 pp under Qwen2.532B-Instruct and 19.1 pp under Gemini-3-Flash (Tab. 6); across the full 14-configuration sweep, accuracy ranges from 7.85% (SLERP, worst) to 15.81% (TIES, best), shown in Fig. 2. Despite this gap, the merged M EMORY model still outperforms every retrieval baseline (BM25, NV-Embed-V2, HippoRAG2, Cartridges; see Tab. 2) on NarrativeQA, indicating that even an aggressively-cheaper merging procedure preserves most of M E M O’s qualitative advantage over retrieval-based approaches. TIES and DARE-Linear at ρ=0.3 dominate the sweep, suggesting that aggressive sparsification combined with sign-conflict resolution is the most reliable merging recipe in this regime.

6

Conclusion

We introduced M E M O, a modular framework for integrating updated or domain-specific knowledge into LLMs via a M EMORY model trained on a synthesized reflection QA dataset. M E M O addresses key limitations of existing methods: it bypasses context constraints and weak cross-document reasoning in retrieval-based approaches, avoids costly and brittle parametric updates (including catastrophic forgetting), and removes representation coupling in latent memory methods. Its core components are a data synthesis pipeline capturing explicit facts and implicit relationships, and a multi-turn inference protocol that decomposes complex queries into targeted sub-queries for desired information retrieval from the memory model. While M E M O demonstrates strong performance, it has limitations regarding training cost, evaluation scope, and the capacity of M EMORY model to scale with corpus size (see App. B). Empirically, M E M O outperforms strong baselines across diverse benchmarks. It also provides a scalable pathway for knowledge integration, supporting efficient updates and plug-and-play deployment with both open-source and proprietary LLMs. Future work includes more efficient memory construction, extensions to dynamic corpora, and tighter coordination between the E XECUTIVE model and M EMORY model. We view M E M O (Memory as a Model) as a promising foundation for more flexible, updatable, and knowledge-aware AI systems. 10

References [1] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv:2205.11916, 2023. [2] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv:2303.18223, 2023. [3] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghoon Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 2026. [4] Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for llms: A survey. arXiv:2403.08319, 2024. [5] Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. Dated data: Tracing knowledge cutoffs in large language models. arXiv:2403.12958, 2024. [6] Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What’s the answer right now? arXiv:2207.13332, 2024. [7] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. arXiv:2212.13138, 2022. [8] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv:2303.17564, 2023. [9] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv:2005.11401, 2021. [10] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. arXiv:2211.08411, 2023. [11] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. In Proc. MLSys, pages 795–813, 2022. [12] Stephen E. Robertson and Steve Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proc. SIGIR, 1994. [13] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv:2405.17428, 2024. [14] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc. NeurIPS, pages 9459–9474, 2020. [15] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv:2404.16130, 2024. 11

[16] Bernal J Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. In Proc. NeurIPS, pages 59532–59569, 2024. [17] Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From rag to memory: Non-parametric continual learning for large language models. In Proc. ICML, 2025. [18] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proc. NeurIPS, pages 1877–1901, 2020. [19] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proc. EMNLP, 2024. [20] Yixuan Tang and Yi Yang. MultiHop-RAG: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv:2401.15391, 2024. [21] Jiaen Lin, Jingyu Liu, and Yingbo Liu. Optimizing multi-hop document retrieval through intermediate representations. arXiv:2503.04796, 2025. [22] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. arXiv:2302.03241, 2023. [23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proc. NeurIPS, 2022. [24] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proc. ACL, pages 13484–13508, 2023. [25] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, pages 1–53, 2024. [26] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv:2308.08747, 2025. [27] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv:2501.17161, 2025. [28] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proc. EMNLP, 2023. [29] Jesse Mu, Xiang Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. In Proc. NeurIPS, 2023. [30] Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In Proc. ICLR, 2024. [31] Guibin Zhang, Muxin Fu, and Shuicheng YAN. Memgen: Weaving generative latent memory for self-evolving agents. In Proc. ICLR, 2026. [32] Bohan Li, Yutai Hou, and Wanxiang Che. Data augmentation approaches in natural language processing: A survey. AI Open, pages 71–90, 2022. [33] Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. An empirical survey of data augmentation for limited data learning in nlp. Transactions of the Association for Computational Linguistics, pages 191–211, 2023. [34] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: part 3.1, knowledge storage and extraction. In Proc. ICML, pages 1067–1077, 2024. 12

[35] Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic qa corpora generation with roundtrip consistency. In Proc. ACL, pages 6168–6173, 2019. [36] Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. Training question answering models from synthetic data. In Proc. EMNLP, pages 5811–5826, 2020. [37] Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. In Proc. ACL, pages 14664–14690, 2024. [38] Yeo Wei Jie, Teddy Ferdinan, Przemyslaw Kazienko, Ranjan Satapathy, and Erik Cambria. Self-training large language models through knowledge detection. In Proc. EMNLP Findings, pages 15033–15045, 2024. [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, 2017. [40] Carles Gelada, Jacob Buckman, Sean Zhang, and Txus Bach. Scaling context requires rethinking attention. arXiv:2507.04239, 2025. [41] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. [42] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Proc. COLM, 2024. [43] Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. In Proc. SIGIR, 2024. [44] Jingyu Liu, Jiaen Lin, and Yong Liu. Tackling the inherent difficulty of noise filtering in rag. arXiv:2601.01896, 2026. [45] Ze Yu Zhang, Arun Verma, Finale Doshi-Velez, and Bryan Kian Hsiang Low. Understanding the relationship between prompts and response uncertainty in large language models. In Proc. ACL Findings, 2026. [46] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In Proc. AAAI, 2020. [47] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2018. [48] Jackson Harmon, Andreas Hochlehnert, Matthias Bethge, and Ameya Prabhu. Mapping posttraining forgetting in language models at scale. arXiv:2510.17776, 2025. [49] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In Proc. ICLR, 2024. [50] Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, et al. Dissecting the runtime performance of the training, fine-tuning, and inference of large language models. arXiv:2311.03687, 2023. [51] Yuchen Xia, Jiho Kim, Yuhan Chen, Haojie Ye, Souvik Kundu, Cong Callie Hao, and Nishil Talati. Understanding the performance and estimating the cost of llm fine-tuning. In Proc. IISWC, 2024. [52] Jiya Manchanda, Laura Boettcher, Matheus Westphalen, and Jasser Jasser. The open source advantage in large language models (llms). arXiv:2412.12004, 2025. 13

[53] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752, 2023. [54] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv:2307.08621, 2023. [55] Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In Proc. ICLR, 2022. [56] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In Proc. ICLR, 2020. [57] Jiaqi Cao, Jiarui Wang, Rubin Wei, Qipeng Guo, Kai Chen, Bowen Zhou, and Zhouhan Lin. Memory decoder: A pretrained, plug-and-play memory for large language models. arXiv:2508.09874, 2025. [58] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv:2309.12288, 2023. [59] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. arXiv:2309.14402, 2023. [60] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportunities. ACM Computing Surveys, 2024. [61] Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv:2508.06600, 2025. [62] Michal Danilák. langdetect. https://github.com/Mimino666/langdetect, 2021. [63] Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, pages 317–328, 2018. [64] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. arXiv:2108.00573, 2022. [65] Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, et al. Cartridges: Lightweight and generalpurpose long context representations via self-study. arXiv:2506.06266, 2025. [66] Jiaqi Cao, Jiarui Wang, Rubin Wei, Qipeng Guo, Kai Chen, Bowen Zhou, and Zhouhan Lin. Memory decoder: A pretrained, plug-and-play memory for large language models. arXiv:2508.09874, 2025. [67] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 technical report. arXiv:2412.15115, 2025. [68] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. arXiv:2309.06180, 2023. [69] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864, 2023. [70] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. 14

[71] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, pages 1–16. IEEE, 2020. [72] Google DeepMind. Gemini 3 flash model card. https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdf, December 2025. [73] Gheorghe Comanici, Eric Bieber, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv:2507.06261, 2025. [74] Jeffrey Ip and Kritin Vongthongsri. deepeval. https://github.com/confident-ai/ deepeval, 2025. [75] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report. arXiv:2503.19786, 2025. [76] Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, et al. LFM2 technical report. arXiv:2511.23404, 2025. [77] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Proc. NeurIPS, 2023. [78] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998. 15

[79] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. [80] Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. In Proc. EMNLP, pages 237–250, 2024. [81] Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv:2402.01364, 2024. [82] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael GontijoLopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proc. ICML, pages 23965–23998, 2022. [83] Ken Shoemake. Animating rotation with quaternion curves. In Proc. SIGGRAPH, pages 245–254, 1985. [84] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proc. ICLR, 2023. [85] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Proc. ICML, 2024. [86] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Proc. ICLR, 2022.

16

A

Impact statement

M E M O advances the ability of LLMs to internalize knowledge over large, domain-specific corpora without requiring access to model weights, lowering the barrier for deploying capable AI systems in knowledge-intensive domains such as law, medicine, and scientific research. By enabling plugand-play integration with any LLM, including proprietary models, M E M O democratizes access to powerful knowledge integration capabilities that would otherwise require significant computational resources or white-box model access. At the same time, this accessibility introduces dual-use concerns, as the same capability that enables beneficial applications could be used to internalize misinformation, proprietary data without authorization, or harmful content at scale. Additionally, as M E M O reduces reliance on explicit retrieval, it may obscure the provenance of retrieved information, making it harder to attribute the sources underlying a model’s responses. We encourage future work to investigate attribution mechanisms and access controls for memory-based systems, and urge practitioners to carefully consider the nature of the documents used to train M EMORY model.

B

Limitations

M E M O incurs an upfront training cost for each new corpus, and performance may vary across domains, document types, or LLM families beyond those covered in our experiments. Furthermore, the performance of M E M O is inherently bounded by the representational capacity of M EMORY model to internalize the target corpus. Although our experiments do not reveal clear signs that M EMORY model has reached its capacity limit, we hypothesize that sufficiently large or information-dense corpora will exceed what a fixed-size M EMORY model can correctly compress and represent.

C

Future work

We outline several directions for future work. The data generation pipeline is computationally expensive, with Step 5 in Alg. 1 scaling quadratically at O(k · C 2 · Q2 ), and reducing this cost remains an open problem. A systematic evaluation of chunking strategies and their associated tradeoffs (App. D) is likewise an open direction. On the training side, scaling M EMORY model with corpus size and developing more effective model merging strategies for reducing per-corpus training costs (Sec. 5.5) are promising directions. Other post-training methods such as Reinforcement Learning [78] have also shown to be effective in improving model task performance [79], and applying such methods to M EMORY model training warrants future investigation. LoRA configurations better suited to specific architectures, including per-architecture tuning of rank and learning rate, also warrant further investigation (App. O). Finally, a more systematic study of the interaction between E XECUTIVE model reasoning capability and M EMORY model model size (Sec. 5.3), as well as the optimal interaction budget at each stage and E XECUTIVE model selection (App. J.2), are other promising future directions.

D

Preparation of datasets

Corpus construction. Extending from our description in Sec. 5, we distinguish between two types of documents7 : evidence documents, which contain information relevant to answering a given question, and negative documents, which are irrelevant and serve as noise. For BrowseComp-Plus, we used 1,775 unique evidence documents and 1,766 unique negative documents (after removal of non-English documents), yielding 3,541 documents in total. For MuSiQue, we used 2,648 documents for each of the evidence and negative documents, yielding 5,296 documents in total. NarrativeQA does not have negative documents. Chunking strategy. As shown in Tab. 7, NarrativeQA full documents span the 32,769–131,072 token range with a median length of 65,925 tokens, reflecting the long-form nature of the source novels. Processing such documents without chunking risks reduced coverage of extractable QA pairs in Step 1 of Alg. 1, as attention quality is known to deteriorate over longer contexts [42]. We therefore chunk NarrativeQA documents using a fixed sliding window of 6,400 words with a 7 Note that for BrowseComp-Plus, the gold documents are a subset of the evidence documents.

17

640-word overlap (10% overlap ratio), yielding 75 chunks concentrated in the 4,097–16,384 token range and accounting for 96% of all chunks, with a median group size of 7 per document as shown in Tab. 8. Unlike NarrativeQA, MuSiQue documents are compact with 99.70% falling below 512 tokens, and each MuSiQue document is treated as a single chunk. Table 7: Token length distribution across corpora at the chunk level, where n represents the total number of individual chunks processed by Alg. 1. Each entry reflects the token count of a single text chunk. Statistics for NarrativeQA are reported before and after chunking.

Token Range

BrowseComp-Plus (n = 3,541)

NarrativeQA Full Docs (n = 10)

NarrativeQA Chunks (n = 75)

MuSiQue (n = 5,296)

606 (17.11%) 591 (16.69%) 746 (21.07%) 598 (16.89%) 428 (12.09%) 323 (9.12%) 145 (4.09%) 56 (1.58%) 20 (0.56%) 28 (0.79%) 14 1,756 7,192 20,330 1,235,897

0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 5 (50.00%) 5 (50.00%) 0 (0.00%) 32,804 65,925 66,324 119,267 119,267

0 (0.00%) 0 (0.00%) 1 (1.33%) 2 (2.67%) 36 (48.00%) 36 (48.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 1,943 8,158 8,713 11,266 12,104

5,280 (99.70%) 16 (0.30%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 23 105 123 270 828

0–512 513–1,024 1,025–2,048 2,049–4,096 4,097–8,192 8,193–16,384 16,385–32,768 32,769–65,536 65,537–131,072 > 131,072 Min tokens Median tokens Mean tokens p95 tokens Max tokens

Table 8: Distribution of document group sizes across datasets, where group size denotes the number of chunks associated with a single question or document. For BrowseComp-Plus and MuSiQue, each question is associated with a subset of chunks drawn from the corpus, and group size represents the number of chunks per question. For NarrativeQA, each subset of chunks is derived from the original document used for multiple questions, and group size represents the number of chunks per document.

Document Group Size Range

0–2 3–4 5–8 9–16 >16 Min group size Median group size Mean group size p95 group size Max group size

BrowseComp-Plus (ngroup = 300)

NarrativeQA Chunks (ngroup = 10)

MuSiQue (ngroup = 1,000)

2 (0.67%) 14 (4.67%) 78 (26.00%) 159 (53.00%) 47 (15.67%) 2 12 11.8 20 23

0 (0.00%) 3 (30.00%) 4 (40.00%) 3 (30.00%) 0 (0.00%) 3 7 7.5 16 16

0 (0.00%) 518 (51.80%) 482 (48.20%) 0 (0.00%) 0 (0.00%) 4 4 5.3 8 8

BrowseComp-Plus documents are also treated as a single chunk. The time complexity of Step 5 in Alg. 1 is O(k · C 2 · Q2 ), where k = ngroup is the number of groups, C = |Gi | is the number of 18

participating chunks per group, and Q = Q̄i is the average number of QA pairs extracted per chunk. Since chunking increases C, pipeline costs at Step 5 scale quadratically as the number of chunks per group increases. Given that only 2.93% of BrowseComp-Plus documents exceed 32,768 tokens, the majority of documents fit within a single chunk, making the cost of chunking difficult to justify. We therefore opted against chunking in favor of lower pipeline cost, and leave a systematic evaluation of chunking strategies and related tradeoffs to future work. Subset selection of negative documents. We include only a subset of negative documents for BrowseComp-Plus and MuSiQue due to computational constraints arising from the quadratic scaling of Step 5. As reported in Tab. 8, BrowseComp-Plus currently has a mean group size of 11.8 and a maximum of 23, while MuSiQue has a mean group size of 5.3 and a maximum of 8. Incorporating all available negative documents, which average 78 per question (up to 197) for BrowseComp-Plus and 17 per question (up to 18) for MuSiQue, would cause the group size to increase substantially. Given the quadratic dependence on C in Step 5, this would result in a prohibitive increase in pipeline cost for BrowseComp-Plus (k = 300) and MuSiQue (k = 1,000). Hence, we opted to only include dataset up to Nevidence number of negative documents for each question in the corpus.

E

Discussion on steps in data generation pipeline

E.1

Ablation of data synthesis steps

We experiment with the data generation pipeline to show the importance of each step. We perform a leave-one-out (LOO) ablation for each step of data synthesis and train the model on the synthesized QA pairs generated. Results are reported in Tab. 9 on the NarrativeQA and MuSiQue datasets using Qwen2.5-32B-Instruct as the E XECUTIVE model and Qwen2.5-1.5B-Instruct as the M EMORY model. Table 9: LOO ablation accuracy at best performing Qwen2.5-1.5B-Instruct epoch across datasets. Data ratio indicates the number of QA pairs retained relative to the baseline. For each step removed, the Qwen2.5-1.5B-Instruct was retrained, and we report the mean ± std. dev. over 3 runs at the same training epoch as the baseline.

Ablation Baseline (all steps) Step 1a removed Step 1b removed Step 2 removed Step 3 removed Step 4 removed Step 5 removed

NarrativeQA Data Ratio Accuracy (%) 1.000× 24.00 ± 0.20 0.434× 20.48 ± 0.90 0.598× 22.98 ± 1.04 0.739× 24.69 ± 1.10 2.078× 28.90 ± 0.86 0.378× 23.21 ± 1.56 0.002× 6.37 ± 0.39

MuSiQue Data Ratio Accuracy (%) 1.000× 42.90 ± 1.25 0.381× 30.00 ± 0.17 0.651× 37.33 ± 0.25 0.621× 37.10 ± 1.76 1.128× 41.70 ± 0.78 0.501× 39.10 ± 0.02 0.195× 24.17 ± 0.25

Step 5 (Cross-document synthesis) is the most critical component of the pipeline. Its removal causes accuracy to collapse to 6.37% and 24.17% on NarrativeQA and MuSiQue respectively, against baseline scores of 24.00% and 42.90%, accompanied by a near-total loss of training data (0.002× and 0.195× retention). As described in Sec. 4.1, Step 5 enables cross-document synthesis where Mgen constructs Qcross pairs spanning inter-document connections and cross-chunk connections within a single long document, making it the dominant source of training pairs in Qfinal and directly targeting the multi-source synthesis objective central to both benchmarks. An interesting anomaly arises with Step 2 and Step 3, where their removal does not consistently hurt performance and improves accuracy on NarrativeQA. Step 2 merges related QA pairs from a single document chunk into multi-fact questions by identifying commonalities such as shared entities, overlapping time periods, and sequential events. For MuSiQue, these commonalities reflect genuine knowledge relationships that directly resemble the multi-hop factual reasoning the benchmark evaluates, such that removing Step 2 eliminates a large fraction of useful training pairs, leading to a drop in accuracy from 42.90% to 37.10%. For NarrativeQA, however, the same consolidation patterns operate on superficial narrative co-occurrences rather than meaningful knowledge relationships. The 19

predominant commonality categories are event or scene groupings that NarrativeQA does not evaluate, and entity co-occurrence patterns that are trivially satisfied given the pervasive presence of central characters across scenes. Removing Step 2 eliminates these low-quality pairs, leading to the marginal accuracy improvement from 24.00% to 24.69%. Removing Step 3 retains more data than the baseline (2.078× and 1.100× for NarrativeQA and MuSiQue respectively), yet the effect on performance diverges. For MuSiQue, performance drops from 42.90% to 41.78%, whereas for NarrativeQA, performance improves from 24.00% to 28.90%. Step 3 applies a self-containment filter that rewrites or discards pairs whose questions cannot be understood without access to the source chunk. For MuSiQue, violations are predominantly localized and shallow, making them amenable to filtering; the proposed filter effectively identifies and removes defective pairs. For NarrativeQA, long-form narrative text frequently contains pronouns and temporal references that span many paragraphs, which are structural features of the domain rather than fixable defects. This causes the rewriting loop to introduce substitute unrelated content and corrupt the pairs produced by earlier steps. Removing Step 3 for NarrativeQA therefore avoids this domain-induced corruption and retains the original pairs intact, explaining both the data retention ratio increase and the accuracy improvement. This suggests that Step 3 is most beneficial when applied to domains where self-containment violations are well-defined and resolvable. The remaining steps follow a consistent trend: removing Step 1a, Step 1b, or Step 4 reduces both data volume and accuracy across both datasets, confirming that each step contributes a distinct and meaningful role to the final training corpus quality. E.2

Additional steps considered but excluded

Three additional steps were considered but ultimately excluded from the pipeline. These include paraphrasing [80], increasing the number of sampling trials at Step 1 of Alg. 1, and a targeted fill whereby Mgen reviews the generated QA pairs and rewrites them to incorporate additional missed information. Paraphrasing was excluded as the scale of generated pairs already provides sufficient coverage (≈600k–1.6M across the three datasets, see Table 11), and the potential gains were outweighed by the additional computational overhead. Increasing sampling trials proved unreliable, as additional trials did not consistently extract facts that the initial pass had failed to extract. The targeted fill similarly offered limited gains, where appending the existing QA pairs as context to prompt a revision only lengthens the context when the model had already failed to extract a fact from the original chunk, likely exacerbating attention further degradation over long inputs [42] and making retrieval of relevant information less reliable at inference time.

F

M EMORY model hyperparameter settings

Training was conducted on H100 and H200 GPUs using the hyperparameter settings reported in Tab. 10. The effective batch size for each dataset is summarized in Tab. 11. Table 10: M EMORY model SFT Training Configuration Parameter Optimizer Gradient checkpointing Learning rate (LR) Num of Training epochs LR scheduler type Warmup ratio Weight decay Max gradient norm Max sequence length Precision Attention implementation

20

Value Fused AdamW True 2 × 10−5 3 Constant with warmup 0.05 0.01 1.0 8096 BF16 Flash Attention 2

Table 11: Effective batch sizes and number of QA pairs used. NarrativeQA.1 and NarrativeQA.2 are independent subsets partitioned from the original that were used for model merging. Dataset BrowseComp-Plus NarrativeQA NarrativeQA.1 NarrativeQA.2 MuSiQue

G

Target Num of Questions 300 293 146 147 1,000

Num of QA Pairs 1,639,995 1,276,676 635,009 641,667 664,762

Effective Batch Size 512 512 256 256 256

Compute resources

All experiments were conducted on using NVIDIA H200 GPUs. We report computational cost in GPU-hours. Data generation. Generating the full reflection dataset for BrowseComp-Plus, NarrativeQA, and MuSiQue took approximately 240, 200, and 150 GPU-hours respectively. Training. M EMORY model (Qwen2.5-14B-Instruct) training for a single run BrowseComp-Plus, NarrativeQA, and MuSiQue took approximately 180, 150, 90 GPU-hours.

H

Model training discussion

We considered three training paradigms: CPT, SFT, and LoRA-based SFT. CPT was excluded as it risks degrading instruction-following capability [81], which is critical for downstream QA evaluation. Full SFT was selected as it directly optimizes for the target task while preserving alignment [23]. LoRA-based SFT serves as a parameter-efficient alternative and we include a comparison to these training methods in App. O. Model merging targets the practical streaming setting in which new corpora arrive over time and M EMORY model must continually integrate them. Retraining M EMORY model from scratch on the union of all observed corpora is the natural baseline but quickly becomes prohibitive at scale, since its cost grows with the cumulative corpus size. Model merging instead trains a separate M EMORY model on each new corpus and combines it with the existing model in parameter space, so the cost of each update scales only with the size of the new corpus rather than the entire history. This decoupling comes at a measurable accuracy cost relative to full retraining, which we quantify in Fig. 2. We assume the corpora to be merged are pairwise disjoint. H.1

Model merging

Merging methods. We consider the following methods, all of which produce φmerged without ever training on D1 ∪ · · · ∪ DK : PK • Linear merging [82] computes a weighted sum of task vectors: φmerged = φ0 + i=1 λi τi , where λi > 0 are merging coefficients. • SLERP [83] interpolates between two task vectors along the unit sphere, preserving their magnitudes: φmerged = φ0 + SLERP(τ1 , τ2 ; t), with t ∈ [0, 1] controlling the interpolation factor. • Task arithmetic [84] adds task vectors directly without further processing, recovering linear merging as a special case with uniform λi . • TIES [77] resolves interference among task vectors before summation by (i) trimming each τi to its top-ρ fraction of largest-magnitude entries, (ii) electing a sign at each coordinate by magnitude-weighted majority vote, and (iii) disjoint-merging only the entries that agree with the elected sign. • DARE [85] sparsifies each task vector by randomly dropping a fraction 1 − ρ of its entries and rescaling the survivors by 1/ρ to preserve expected magnitude, before linear merging. • DARE-TIES [85] combines DARE-style stochastic sparsification with TIES sign-conflict resolution, retaining the diversity of random dropout while filtering out conflicting updates. 21

Avoiding catastrophic forgetting. Because no individual M EMORY model Mφi is ever fine-tuned on another corpus’ data, model merging cannot induce the kind of distributional interference that drives catastrophic forgetting in sequential fine-tuning [26]. Knowledge from each corpus is preserved within its own task vector τi , and conflicts between task vectors are addressed at merge time via the methods above rather than during gradient updates. Scalability. When a new corpus DK+1 arrives, we train auxiliary model MφK+1 on its reflection QA dataset, derive τK+1 , and re-merge in O(1) additional cost relative to the full collection. This enables modular, plug-and-play integration over a continuous stream of disjoint knowledge sources, S unlike retraining from scratch on i Di , which scales linearly with the cumulative corpus size. Inference. The merged M EMORY model is queried identically to a single-corpus M EMORY model via the structured multi-turn protocol described in Sec. 4.4. Because merging operates entirely in parameter space and produces a model with the same architecture and interface as Mφ0 , it inherits the plug-and-play property of M E M O without requiring changes to the E XECUTIVE model or the inference protocol. Importantly, the E XECUTIVE model queries a single merged M EMORY model at inference rather than dispatching across K separate per-corpus M EMORY models, keeping the multi-turn retrieval pipeline unchanged regardless of how many corpora have been integrated. Procedure. For our experiments we partition NarrativeQA into two pairwise-disjoint subsets, NarrativeQA.1 and NarrativeQA.2, of ∼640k reflection QA pairs each. Each subset is used to fine-tune an independent M EMORY model from the same Qwen2.5-14B-Instruct base via SFT for 3 epochs, producing Mφ1 and Mφ2 at SFT costs of X and Y GPU-hours, respectively (each is ≈ 24 GPU-hours on 8×H100; full-retrain on the union NarrativeQA.1 ∪ NarrativeQA.2 costs X+Y ≈ 48 GPU-hours by linear scaling). We evaluate every saved checkpoint of each run on the held-out NarrativeQA evaluation set and select the best-performing checkpoint per subset; the corresponding task vectors τ1 and τ2 are the inputs to the merging step. We then sweep all six merging methods listed above (Linear, Task arithmetic, SLERP, TIES, DARE, DARE-TIES) at three sparsification densities ρ ∈ {0.3, 0.5, 0.7} (or three interpolation factors t ∈ {0.3, 0.5, 0.7} for SLERP), giving 14 merged-M EMORY model configurations in total. Each configuration is evaluated on NarrativeQA with Qwen2.5-32B-Instruct as E XECUTIVE model (mean ± std over 3 runs). The configuration that we report in the Sec. 5 as Merge-TIES is the best of the sweep (TIES with ρ=0.3). Cost vs. accuracy on NarrativeQA (MEMORY model: Qwen2.5-14B-Instruct) Executive: Qwen2.5-32B-Instruct 60

NarrativeQA accuracy (%)

Executive: Gemini-3-Flash

Merge

Merge

Retrain

Retrain

23.8 GPU-h saved (33% cheaper)

53.58% 50 Δ = +19.1 pp

40 34.47% 30

26.85% Δ = +11.0 pp

20 15.81% 10

0

worst 7.8%

0

20

40

60

80

0

Cumulative training cost (8×H100 GPU-hours) BM25 NV-Embed-V2

HippoRAG2 Cartridges

20

40

60

80

Cumulative training cost (8×H100 GPU-hours)

Perfect Retrieval (UB) MEMO (best merge: TIES, ρ = 0.3)

MEMO (merge sweep, n = 14: worst ↔ best) MEMO (Full retrain)

Figure 2: Cost–accuracy trade-off on NarrativeQA when a second corpus arrives (K=2, M EM ORY model = Qwen2.5-14B-Instruct, 8×H100). Cumulative training cost is shown on the x-axis (one Qwen-14B SFT run takes ≈ 24 GPU-hours on a 640k-QA-pair corpus). Merging trains M EM ORY model only on the new corpus, costing X+Y ≈ 48 GPU-hours, while full retraining re-runs on the union, costing X+(X+Y ) ≈ 72 GPU-hours — a 33% saving. Merge-TIES (ρ=0.3) trails full retraining by 11.0 pp with Qwen2.5-32B-Instruct and 19.1 pp with Gemini-3-Flash as E XECUTIVE model, but still outperforms all retrieval baselines (BM25, NV-Embed-V2, HippoRAG2, Cartridges). The vertical ↕ at the merge cost shows the worst-to-best range across the 14 merge configurations swept (Tab. 12). Perfect Retrieval is shown as the upper bound. 22

Results. A single SFT run consumes ≈ 24 GPU-hours on 8×H100; after two arrivals, full retraining incurs X+(X+Y )=72 GPU-hours of cumulative compute, whereas merging accumulates only X+Y =48 GPU-hours — a 33% reduction (Fig. 2). The asymptotic gap widens with K: under the same per-corpus cost, merging scales as Θ(K) while full retraining scales as Θ(K 2 ), yielding a 5.5× saving at K=10 (240 vs. 1,320 GPU-hours). On accuracy, Merge-TIES (ρ=0.3) trails full retraining by 11.0 pp with Qwen2.5-32B-Instruct as E XECUTIVE model (15.81% vs. 26.85%) and by 19.1 pp with Gemini-3-Flash (34.47% vs. 53.58%), placing the merged M EMORY model below the union-retrained M EMORY model but above every retrieval baseline. The full per-method sweep is reported in Tab. 12: TIES (ρ=0.3) and DARE-Linear (ρ=0.3) lead at 15.81% and 15.47% respectively, while SLERP (t=0.5) is the worst configuration at 7.85%. The pattern across families suggests that aggressive sparsification at low ρ paired with sign-conflict resolution (TIES, DARELinear) is the most reliable merging recipe in this regime. These results confirm the predicted compute–accuracy trade-off: merging recovers most of M EMORY model’s headroom over retrieval methods at substantially lower cumulative cost. Table 12: Sweep of all 14 merge configurations on NarrativeQA. Two M EMORY models (Qwen2.514B-Instruct) are independently SFT-trained on the disjoint NarrativeQA.1 and NarrativeQA.2 subsets; each subset’s best-performing checkpoint provides the task vector entering the merge. E XECUTIVE model = Qwen2.5-32B-Instruct; results are mean ± std. dev. over 3 runs. Best merge in bold; full-retrain accuracy (26.85 ± 0.39) is shown for reference. Hyperparameter conventions: t ∈ [0, 1] is the SLERP interpolation factor along the unit sphere connecting the two task vectors (t=0 recovers M EMORY model on NarrativeQA.1, t=1 recovers M EMORY model on NarrativeQA.2, t=0.5 is the geodesic midpoint); ρ ∈ (0, 1] is the sparsification density — the fraction of largest-magnitude task-vector entries kept (TIES) or the keep probability for random-drop sparsification (DARE, DARETIES). Linear and Task arithmetic merge with uniform weights (λi = 1) and have no hyperparameter. Method family Linear Task arithmetic SLERP

TIES

DARE-Linear

DARE-TIES

I

Hyperparameter — — t = 0.3 t = 0.5 t = 0.7 ρ = 0.3 ρ = 0.5 ρ = 0.7 ρ = 0.3 ρ = 0.5 ρ = 0.7 ρ = 0.3 ρ = 0.5 ρ = 0.7

Accuracy (%) 11.60 ± 1.02 12.74 ± 1.75 11.60 ± 2.24 7.85 ± 1.71 11.60 ± 2.13 15.81 ± 0.39 12.17 ± 1.94 12.06 ± 2.58 15.47 ± 0.79 9.78 ± 1.20 13.65 ± 2.08 11.72 ± 0.52 12.97 ± 1.23 11.04 ± 1.20

Validating evaluation dataset suitability

Table 13: Performance gap between no context and perfect retrieval across datasets and E XECUTIVE models. No Context Perfect Retrieval

Qwen2.5-32B-Instruct BrowseComp-Plus NarrativeQA MuSiQue 0.00 ± 0.00 5.35 ± 0.20 17.03 ± 0.40 79.67 ± 1.45 51.42 ± 0.52 62.83 ± 0.90

Gemini-3-Flash BrowseComp-Plus NarrativeQA 1.33 26.62 88.33 60.41

MuSiQue 41.80 73.00

To assess the suitability of the evaluation datasets for E XECUTIVE model and whether the E XECUTIVE model has memorized answers from training data, we evaluate performance both without any context (No Context) and with evidence documents provided (Perfect Retrieval), the latter serving as an empirical upper-bound that assumes perfect retrieval of relevant documents. 23

As shown in Tab. 13, the large disparity in performance between No Context and Perfect Retrieval confirms that these datasets require access to evidence documents to achieve correct answers, validating their suitability for evaluating M E M O. Unsurprisingly, MuSiQue yields the highest No Context scores, as its Wikipedia-grounded questions fall within models’ parametric knowledge. NarrativeQA proves most challenging as it achieves the lowest Perfect Retrieval scores across both E XECUTIVE models, reflecting the demand for careful reasoning over full-length books and movie scripts. BrowseComp-Plus yields the largest disparity between No Context and Perfect Retrieval, with near-zero No Context performance but strong recovery when evidence documents are provided. These findings confirm that E XECUTIVE model heavily relies on evidence documents across all three datasets to perform well. MuSiQue tests multi-hop factual reasoning where parametric knowledge provides partial signals, NarrativeQA tests narrative comprehension that remains challenging even with perfect context, and BrowseComp-Plus tests the ability to exploit retrieved documents for facts otherwise entirely inaccessible to the model.

J

Evaluation details

J.1

Implementation details

The current temperature settings are described in Tab. 14. Stage 1 only has a budget of 1 interaction, Stage 2 has a budget of 7 interactions, Stage 3 has a budget 8 interactions. Table 14: Temperature Configuration of each Stage from Sec. 4.4 Stage Evaluation Stage 1 – Grounding

Model E XECUTIVE model

Temperature Value 0.4

Evaluation Stage 1 – Grounding

M EMORY model

0.1

Evaluation Stage 2 – Entity identification

E XECUTIVE model

0.4

Evaluation Stage 2 – Entity identification

M EMORY model

0.1

Evaluation Stage 3 – Answer Seeking

E XECUTIVE model

1.0

Evaluation Stage 3 – Answer Seeking

M EMORY model

0.3

Final Synthesis

E XECUTIVE model

0.3

Intent Moderate exploration to generate diverse but focused sub-questions Near-deterministic to ensure stable, consistent grounding answers Moderate exploration to identify varied candidate entities without excess noise Near-deterministic to produce reliable entitytargeted answers High exploration to maximally diversify subquestions once the entity is confirmed Slightly relaxed determinism to allow nuanced answers while remaining consistent Low temperature to produce a consistent final answer

Table 15: Helper Functions for Stage 2 and 3 of the Evaluation Pipeline Function Track uncertain answer streaks

Stage Stage 2

Select the best candidate

Stage 2

Entity pivot correction

Stage 3

Intent Maintains a running tally of how many unanswerable questions each candidate entity has accumulated across Stage 2, allowing the E XECUTIVE model to progressively prioritize candidates that the M EMORY model consistently cannot corroborate Fallback bridge from Stage 2 to Stage 3 when entity pinning ends without a confirmed entity. Selects the highest E XECUTIVE model-ranked candidate, with ties broken by the order in which the M EMORY model produced the candidates Allows the pipeline to self-correct mid Stage 3 if the Stage 2 entity proves incorrect. When the E XECUTIVE model nominates a different entity, the confirmed entity is overwritten and marked as unconfirmed so subsequent turns are aware it was not pinned through the full Stage 2 process

Beyond what is described in Sec. 4.4, there are additional helper functions that help manage failure modes across Stage 2 and Stage 3. Within Stage 2, the uncertain answer streak tracker is called at the start of every entity-pinning interaction and its output is passed directly into the entity-pinning prompt, giving the E XECUTIVE model a live view of which candidates the M EMORY model has repeatedly failed to corroborate. This allows the E XECUTIVE model to continuously re-rank and prune the candidate pool as evidence accumulates. When Stage 2 concludes without a confirmed entity, either 24

because the E XECUTIVE model explicitly exhausts its options or the interaction budget is reached, the best candidate selector acts as the bridge into Stage 3 by returning the top-ranked candidate. In cases where multiple candidates share the highest rank, the first candidate in the order produced by E XECUTIVE model is selected. In both cases, the downstream Stage 3 prompt is informed of whether the entity was formally confirmed or merely a best guess. Finally, if Stage 3 reveals that the Stage 2 entity was incorrect due to persistent M EMORY model failures, the entity pivot mechanism allows the E XECUTIVE model to nominate a replacement entity mid-stage. The confirmed entity is then overwritten and marked as unconfirmed, ensuring subsequent stages treat it with appropriate uncertainty rather than the confidence of a fully pinned entity. J.2

Ablations on evaluation setup

To justify our structured multi-turn evaluation design, we compare against two baselines: a single-turn setup and an unstructured multi-turn setup; in both cases, the same trained M EMORY model is used and E XECUTIVE model is held fixed. Results are reported in Tab. 16. Table 16: M E M O accuracy results with Qwen2.5-32B-Instruct as E XECUTIVE model and Qwen2.514B-Instruct as M EMORY model across evaluation setups. The best performing epoch was used in comparison across all 3 setups, with mean ± std. dev. reported across 3 runs. Bold results indicate best performing results in the column. Evaluation Setup Single turn evaluation Unstructured multi-turn evaluation (15 turns) Unstructured multi-turn evaluation (50 turns) Structured multi-turn evaluation (7 Entity Identification turns + 8 Answer seeking turns) Structured multi-turn evaluation (7 Entity Identification turns + 15 Answer seeking turns)

BrowseComp-Plus Accuracy 32.56 ± 1.58

NarrativeQA Accuracy 24.80 ± 0.20

MuSiQue Accuracy 37.57 ± 1.15

47.33 ± 0.88

26.73 ± 2.17

40.13 ± 1.12

48.67 ± 1.00

27.19 ± 0.71

40.57 ± 0.31

54.22 ± 0.84

26.39 ± 1.75

48.30 ± 1.25

51.44 ± 2.41

27.76 ± 0.20

47.57 ± 0.95

In a single-turn interaction, E XECUTIVE model first determines whether the question requires external memory retrieval, and if so, decomposes it into a set of sub-questions (Stage 1, Sec. 4.4) and poses them all simultaneously to M EMORY model. M EMORY model responds to each sub-question independently, and responses indicating uncertainty are discarded before the remaining answers are passed to E XECUTIVE model for final synthesis. This design requires E XECUTIVE model to commit to its full sub-question set before observing any responses, preventing it from reformulating uninformative queries, following up on answers that introduce new candidate entities, or correcting retrievals that are incomplete, contradictory, or anchored to the wrong entity. This is a fundamental limitation that is reflected in its consistently lowest performance across all three datasets (Tab. 16). A natural extension of the single-turn setting is an unstructured multi-turn interaction, where E XECU TIVE model examines the responses from M EMORY model and decides whether sufficient information has been gathered, or whether additional retrieval rounds are needed (Stage 3, Sec. 4.4. In this setting, E XECUTIVE model is presented with the full history of question-answer pairs and prompted to either synthesize a final answer or generate a new batch of sub-questions targeting remaining gaps, repeating for up to T interactions. While iterative retrieval yields clear improvements over the single-turn baseline, performance plateaus quickly when increasing from 15 to 50 interactions (47.33 ± 0.88 to 48.67 ± 1.00 on BrowseComp-Plus, 26.73 ± 2.17 to 27.19 ± 0.71 on NarrativeQA, and 40.13 ± 1.12 to 40.57 ± 0.31 on MuSiQue), suggesting that iterative retrieval alone is insufficient. The structured multi-turn setup (see Sec. 4.4 and App. J.1) outperforms the unstructured multiturn baseline, with 8 answer-seeking interactions achieving the strongest overall performance for BrowseComp-Plus and MuSiQue. This is consistent with the expectation that explicit entity identification is well-suited to the multi-hop reasoning demands of these datasets. NarrativeQA, which tests discourse understanding over long documents, has the unstructured 15- and 50-interaction baselines (26.73 ± 2.17 and 27.19 ± 0.71) initially outperform the structured setup with 8 answer-seeking interactions (26.39 ± 1.75). Inference logs indicate that E XECUTIVE model rarely utilizes the entity identification stage on NarrativeQA, likely because its questions are less reliant on resolving specific entities. Consequently, the fixed-entity identification budget effectively reduces the number of avail25

able answer-seeking interactions compared to unstructured baselines. Increasing the answer-seeking budget to 15 interactions recovers this gap, with NarrativeQA reaching 27.76 ± 0.20, surpassing both unstructured baselines. We hypothesize that this could be due to the additional answer-seeking interactions continue to surface useful signal without the risk of entity drift or state corruption that compounds in open-domain multi-hop settings. Unlike NarrativeQA, structured entity identification and state tracking in BrowseComp-Plus and MuSiQue introduce sensitivity to error accumulation as the number of answer-seeking interactions increases beyond the optimal budget. Additional interactions increase the risk of erroneous M EMORY model responses corrupting the known facts state, and provide more opportunities for E XECUTIVE model to commit to an incorrect intermediate entity via the entity pivot correction helper (Tab. 15). Furthermore, more interactions dilute the correct signal with potentially incorrect answers at the final synthesis stage. These failure modes are partly a function of the reasoning capability of E XECUTIVE model, as structured state maintenance demands strong in-context reasoning to accurately track entities and avoid premature entity commitment. Corroborating this, we observe in Tab. 2 that a stronger reasoning model as E XECUTIVE model yields improved performance when paired with the same M EMORY model, suggesting that these failure modes can be mitigated by scaling the reasoning capability of E XECUTIVE model. The stage budget used in our experiments was selected without systematic tuning, and alternative settings may yield similar performance with greater token efficiency. We therefore leave a systematic study of the optimal interaction budget and E XECUTIVE model selection as future work.

K

Discussion on number of training epochs

From Figs. 3 to 5, we observe that additional training epochs do not consistently improve accuracy, as peak performance for most M EMORY model occurs at epoch 2 with marginal gains or mild regression thereafter. We attribute the early saturation and subsequent regression to overfitting on the SFT corpus, which exhibits substantial lexical overlap across steps by design, as later steps are derived from earlier ones Alg. 1. To quantify this lexical overlap, we compute the lossless compression ratio of the combined QA text across all steps for each dataset by extracting all question and answer strings, concatenating them into a single text corpus, and applying gzip compression at maximum level (compression level 9), where the compression ratio is defined as the ratio of the original text size to the compressed size.

Figure 3: BrowseComp-Plus accuracy (%) vs. training epoch (Full SFT) for each M E M O model size and model family. Lines show the mean over 3 runs, and the shaded band shows ± std. dev. for Qwen2.5-32B-Instruct Runs. BrowseComp-Plus (1,639,995 pairs) achieves a ratio of 5.80× (82.8% savings), MuSiQue (664,762 pairs) achieves 7.03× (85.8% savings), and NarrativeQA (1,276,676 pairs) achieves 5.45× (81.7% savings), indicating substantial lexical overlap within each dataset. We note that compression ratio 26

Figure 4: NarrativeQA accuracy (%) vs. training epoch (Full SFT) for each M E M O model size and model family. Lines show the mean over 3 runs, and the shaded band shows ± std. dev. for Qwen2.5-32B-Instruct Runs.

Figure 5: MuSiQue accuracy (%) vs. training epoch (Full SFT) for each M E M O model size and model family. Lines show the mean over 3 runs, and the shaded band shows ± std. dev. for Qwen2.532B-Instruct Runs. captures lexical overlap only, and semantic diversity across QA pairs may remain higher, as each step targets distinct reasoning operations ranging from direct fact extraction to cross-document synthesis Alg. 1, which is consistent with the impact of removing Step 5 (see App. E).

L Performance degradation of retrieval-based methods with increasing noise Tab. 17 reports the performance of two retrieval-based baselines (NV-Embed-V2 and HippoRAG2) under increasing retrieval noise. Both methods degrade monotonically as noise increases, confirming their susceptibility to irrelevant documents. The degradation is most severe for HippoRAG2 on BrowseComp-Plus, which drops 11.55 pp from 0N to 2N, and for NV-Embed-V2 on MuSiQue, which drops 9.27 pp over the same range. Notably, even a single negative document per evidence document (1N) causes substantial drops of up to 6.22 pp for both methods on BrowseComp-Plus, suggesting that retrieval-based methods are extremely sensitive to noisy retrieval settings. 27

Table 17: Accuracy (%) on BrowseComp-Plus and MuSiQue with Qwen2.5-32B-Instruct as E XECU TIVE model. M E M O results are based on Qwen2.5-14B-Instruct and reported at the best training epoch. N is the number of evidence documents present in the target corpus. ∆ denotes accuracy difference (pp) compared to 0N . Method

NV-Embed-V2 HippoRAG2

M

0N Acc. (%) 56.89 ± 0.51 42.30 ± 0.53 62.33 ± 1.15 47.33 ± 0.74

Dataset BrowseComp-Plus MuSiQue BrowseComp-Plus MuSiQue

1N Acc. (%) 50.67 ± 0.33 37.47 ± 0.15 56.11 ± 0.51 42.17 ± 0.12

∆ ↓ 6.22 ↓ 4.83 ↓ 6.22 ↓ 5.16

2N Acc. (%) 49.44 ± 0.19 33.03 ± 1.10 50.78 ± 1.35 41.70 ± 0.69

∆ ↓ 7.45 ↓ 9.27 ↓ 11.55 ↓ 5.63

Ablation on M EMORY model size

Both Qwen2.5-1.5B-Instruct and Qwen2.5-14B-Instruct M EMORYs are trained on the same QA dataset generated by the G ENERATOR model (Qwen2.5-32B-Instruct) under training settings described in App. F. Each M EMORY model is evaluated using Qwen2.5-32B-Instruct and Gemini-3Flash as the E XECUTIVE model.

N

Ablation on M EMORY model family

Each M EMORY model is trained on the same QA dataset generated by G ENERATOR model (Qwen2.532B-Instruct) and evaluated using Qwen2.5-32B-Instruct and Gemini-3-Flash as E XECUTIVE model. Notably, while Qwen2.5-1.5B-Instruct and Gemma3-1B-IT are based on standard transformer architectures, LFM2.5-1.2B-Instruct adopts a hybrid architecture combining state-space convolution with transformer attention blocks, thereby providing a broader test of M EMORY model across diverse model designs. These models are trained on the same training settings in App. F, with Gemma3-1B-IT using eager attention during training instead of Flash Attention 2.

O

Comparison between full SFT and LoRA

We train all models using LoRA [86] applied to the attention and feed-forward projection layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. The general LoRA configuration is summarised in Table 18, with model-specific rank and scaling settings reported in Table 19. All remaining training hyperparameters follow Table 10, and per-dataset batch sizes are given in Table 11. Table 18: LoRA Specific Training Configuration. All other parameters are the same as those in Tab. 10. Parameter Target modules LoRA dropout Bias Learning rate

Value q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj 0.05 None 2 × 10−4

Table 19: Model-Specific LoRA Configuration. Model LFM2.5-1.2B-Instruct Gemma3-1B-IT Qwen2.5-1.5B-Instruct Qwen2.5-14B-Instruct

Size 1.2B 1B 1.5B 14B

LoRA rank 8 8 8 16

28

LoRA alpha 16 16 16 32

Trainable params 6.1M (0.41%) 6.6M (0.65%) 9.2M (0.60%) 68.8M (0.47%)

Table 20: Ablation on LoRA vs Full SFT training across all M EMORY models, evaluated with Qwen2.5-32B-Instruct as E XECUTIVE model. All results are mean ± std. dev. over 3 runs. Bold results indicate best performing results in the column. M EMORY model Gemma3-1B-IT LFM2.5-1.2B-Instruct Qwen2.5-1.5B-Instruct Qwen2.5-14B-Instruct

BrowseComp-Plus LoRA Full SFT 25.22 ± 1.39 41.67 ± 2.03 0.78 ± 0.19 37.33 ± 1.86 29.78 ± 0.51 44.11 ± 2.22 48.78 ± 1.02 54.22 ± 0.84

NarrativeQA LoRA Full SFT 21.62 ± 0.86 22.30 ± 2.47 5.69 ± 0.71 21.96 ± 1.97 21.84 ± 0.34 24.00 ± 0.20 23.78 ± 0.52 26.85 ± 0.39

MuSiQue LoRA Full SFT 26.17 ± 1.10 41.17 ± 1.20 7.50 ± 0.26 45.23 ± 2.49 31.53 ± 0.55 42.90 ± 1.39 43.94 ± 0.97 50.07 ± 0.81

The notably poor LoRA performance of LFM2.5-1.2B-Instruct can be attributed to its hybrid convolution–attention architecture, which differs from the standard transformer models in our evaluation. Following the LFM2 architecture [76], LFM2.5-1.2B-Instruct consists of 16 layers — 6 grouped-query attention (GQA) blocks (at indices {2, 5, 8, 10, 12, 14}) interleaved with 10 short-range LIV convolution (ShortConv) blocks [76]. Crucially, the LFM2 attention output projection is named out_proj (rather than o_proj) and its SwiGLU MLP uses w1/w3/w2 (rather than gate_proj/up_proj/down_proj), while the ShortConv blocks expose their own in_proj and out_proj layers. A LoRA configuration targeting the standard Llama-family module names therefore adapts only a strict subset of the projections that exist in LFM2.5, leaving the remainder frozen. The result is 6.1M trainable parameters (0.41% of total), disproportionately low given the model size and below our target of ∼0.5%. The rank r = 8 was kept fixed across all sub-2B models for a controlled comparison; in retrospect, this penalises LFM2.5-1.2B-Instruct due to its architectural mismatch with the standard Llama-style target set. Furthermore, the 10 ShortConv blocks which handle the bulk of the model’s local feature extraction and the SwiGLU MLPs attached to every block remain entirely unadapted under standard LoRA targeting, severely limiting the adapter’s ability to shift the model’s behaviour. As shown in Tab. 20, the large performance gap between LoRA and Full SFT confirms that the model is capable of learning the task when all parameters are updated. Future work could explore LoRA configurations better suited to this architecture: targeting the LFM2-specific module names (out_proj, w1, w3, w2) alongside the ShortConv projections (in_proj, out_proj), as well as tuning the rank and learning rate per architecture rather than holding them fixed across families for the controlled comparison reported here.

29