Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework [Experiment, Analysis & Benchmark] Yanchen Wu∗
Tenghui Lin∗
Yingli Zhou
Fangyuan Zhang
Qintian Guo
CUHK-Shenzhen
CUHK
CUHK-Shenzhen
HITSZ
BIT
Xun Zhou
Sibo Wang
Xilin Liu
Yuchi Ma
Yixiang Fang
HITSZ
CUHK
Huawei Cloud
Huawei Cloud
CUHK-Shenzhen
PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/Yanchen398/Memory-in-the-LLM-Era.
1
INTRODUCTION
The development of Large Language Models (LLMs) like GPT-5 [79], Qwen3 [96], and Claude Sonnet 4.6 [3] has sparked a revolution in the field of artificial intelligence [19, 29, 43, 51, 63, 86, 88, 102]. Building on this success, LLM-powered agents have rapidly emerged and are being deployed across a wide range of domains, from industrial automation to personal assistance. For example, SWE-agent systems for software engineering tasks [98] and personal assistant ∗ The first two authors contributed equally to this research.
This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 19, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX
Naive Long-context Prompting co ove ntext rflo • token-intensive w!
Message History ?
• high-latency • unreliable
Prompt LLM Response
Current Query
Memory-augmented Prompting Relevant Information • token-efficient ...
PVLDB Reference Format: Yanchen Wu, Tenghui Lin, Yingli Zhou, Fangyuan Zhang, Qintian Guo, Xun Zhou, Sibo Wang, Xilin Liu, Yuchi Ma, and Yixiang Fang. Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework [Experiment, Analysis & Benchmark]. PVLDB, 19(1): XXX-XXX, 2026. doi:XX.XX/XXX.XX
...
arXiv:2604.01707v1 [cs.CL] 2 Apr 2026
ABSTRACT Memory emerges as the core module in the large language model (LLM)-based agents for long-horizon complex tasks (e.g., multiturn dialogue, game playing, scientific discovery), where memory can enable knowledge accumulation, iterative reasoning and selfevolution. A number of memory methods have been proposed in the literature. However, these methods have not been systematically and comprehensively compared under the same experimental settings. In this paper, we first summarize a unified framework that incorporates all the existing agent memory methods from a high-level perspective. We then extensively compare representative agent memory methods on two well-known benchmarks and examine the effectiveness of all methods, providing a thorough analysis of those methods. As a byproduct of our experimental analysis, we also design a new memory method by exploiting modules in the existing methods, which outperforms the state-of-the-art methods. Finally, based on these findings, we offer promising future research opportunities. We believe that a deeper understanding of the behavior of existing methods can provide valuable new insights for future research.
Memory System Message History
…
• low-latency • reliable
Prompt
LLM Response
?
Current Query
Figure 1: Overview of naive long-context prompting and memory-augmented prompting. agents such as OpenClaw [64] illustrate how LLM-based agents can autonomously plan, reason, and execute complex multi-step workflows. These agents are increasingly expected to operate autonomously, adapt to diverse environments, and support personalized interactions tailored to user needs. A key capability underlying such intelligent behavior is the memory mechanism [67, 101]. As illustrated in Figure 1, memory mechanisms allow agents to move beyond naive long-context prompting by maintaining and leveraging relevant information from past interactions. By equipping agents with memory mechanisms, they can accumulate experience over time, maintain contextual knowledge, and make more informed decisions—analogous to how humans rely on memory to learn from past experiences and guide future actions. In recent years, a growing number of memory methods have been proposed to enhance the capability of LLM-based agents to retain, organize, and utilize historical information across interactions. These methods aim to enable agents to move beyond stateless reasoning and instead support long-term planning, personalization, and adaptive decision-making. Motivated by the limitations of stateless LLM agents and the growing need for persistent contextual reasoning, researchers from multiple communities—including databases, data mining, machine learning, and natural language processing—have begun developing efficient and scalable memory mechanisms for intelligent agents [66, 67, 73, 87, 90]. In Table 1, we summarize ten representative agent memory methods. We categorize them according to four key dimensions: underlying storage structure, information extraction mechanism, memory management strategy, and retrieval method. After a careful literature review, we make the following observations. First, there is a lack of a unified framework for abstracting and systematically analyzing agent memory methods. Second, most previous studies
Table 1: Classification of representative agent memory methods. Method
Information Extraction
Management Operations
Storage Structure
Retrieval Mechanism
A-MEM [93]
Direct archive, Summarization-based extract
Connect, Update
Flat, Vector
Vector-Based
MemoryBank [103]
Direct archive
Integrate, Update, Filter
Flat, Vector
Vector-Based
MemGPT [66]
Direct archive
Integrate, Transform, Update
Hierarchical, Vector
Lexical-Based, Vector-Based
Mem0 [11]
Direct archive, Summarization-based extract
Integrate, Update, Filter
Flat, Vector
Vector-Based
Mem0𝑔 [11]
Graph-based extract
Connect, Update, Filter
Flat, Graph
Vector-Based, Structure-Based
MemoChat [56]
Direct archive
Integrate
Flat
LLM-Assisted
Zep [72]
Direct archive, Graph-based extract
Connect, Transform, Update
Hierarchical, Graph
Lexical-Based, Vector-Based, Structure-Based
MemTree [73]
Direct archive
Connect, Integrate, Update
Flat, Tree
Vector-Based
MemoryOS [32]
Direct archive
Connect, Integrate, Transform, Update, Filter
Hierarchical, Vector
Lexical-Based, Vector-Based
MemOS [44]
Direct archive, Summarization-based extract
Connect, Integrate, Update
Hierarchical, Tree
Lexical-Based, Vector-Based
2.1
report overall performance results, but rarely examine the roles and effects of individual components within these methods. Third, comprehensive and systematic comparisons among different methods—especially regarding their accuracy and efficiency—are still lacking. Our work. We address these gaps by proposing a unified, modular framework and conducting an in-depth experimental study of representative agent memory methods. The framework decomposes memory mechanisms into four stages, including ❶ Information Extraction, ❷ Memory Management, ❸ Memory Storage, and ❹ Information Retrieval. Under this framework, we compare representative methods on two typical long-term conversational benchmarks, LOCOMO [57] and LONGMEMEVAL [91]. Beyond overall performance, we analyze practical robustness dimensions, including context scalability and positional sensitivity. Based on these analyses, we further design a new agent memory method that achieves the best performance and cost efficiency. In summary, our principal contributions are listed as follows: • We propose a unified framework that decomposes representative agent memory methods into four modular components, enabling systematic comparisons of their differences. • We conduct comprehensive experimental studies on LOCOMO and LONGMEMEVAL, together with analyses of token cost, context scalability, evidence position sensitivity, and LLM backbone dependence. • Based on the above analyses, we propose a new agent memory method that achieves state-of-the-art performance. We further derive several key insights and highlight promising research directions for future work. Roadmap. Section 2 introduces preliminaries. Section 3 presents the unified framework. Sections 4–7 characterize representative design choices within this framework. Section 8 reports experimental results and analyses. Section 9 summarizes lessons and opportunities. Section 10 reviews related work, and Section 11 concludes.
2
LLM-related Concepts
We introduce two fundamental LLM-related concepts below. LLM Prompting. Prompting [6, 12, 54] specifies an LLM task by constructing an input context (prompt) that contains task instructions, the current input, and optionally demonstrations or auxiliary evidence. The model’s output is then generated conditioned on this context. Prompting is particularly important in LLM-based systems because it provides a lightweight, training-free interface for task adaptation [54]. Model behavior can be redirected by revising the input context without modifying model parameters. This property has enabled a wide range of practical applications in which task specifications and constraints are expressed directly in natural language [1, 65]. In addition to final response generation, prompting is often used to drive intermediate sub-tasks in an LLM-centered pipeline [4, 33, 76, 99]. Typical examples include extracting key information, consolidating intermediate results, and so on. LLM-based Agents. An LLM-based agent utilizes an LLM as a core decision model for sequential action selection [40, 78, 99]. In contrast to single-turn prompting, an agent operates in a closed loop: it receives observation, then performs reasoning or planning, executes an action (possibly via tools), obtains feedback from the environment, and further proceeds to the next step [97]. The interaction history and available state are combined into textual context, based on which the agent predicts the next action. The action space of an LLM-based agent typically includes both natural-language responses and structured tool calls [44, 66] (e.g., information search, API invocation and memory read/write operations), which enable multi-step task completion beyond one-shot generation. Therefore, the LLM-based agent must retain and reuse information across conversation turns and sessions, which motivates effective memory mechanisms.
2.2
Agent Memory
Memory is incorporated into LLM-based agents to compensate for the bounded context window [26, 52, 57]. Since the model conditions only on a limited number of tokens, information that comes from earlier turns and lies outside the current prompt can be easily lost, which degrades performance in long-horizon dialogue and multi-session tasks [77, 91, 94]. An explicit memory module [5, 21, 66, 67, 83] allows the system to persist interaction-derived information—such as user preferences, salient events, intermediate
PRELIMINARIES
In this section, we go through some important concepts and the typical workflow of existing memory methods in the LLM era. The relationship between RAG and memory is also discussed. 2
decisions, and task constraints—and to reintroduce it when relevant, thereby improving consistency and enabling reasoning that depends on long-term context [21, 67]. The typical workflow of memory-augmented systems begins by selectively extracting important information from ongoing interactions—such as facts, user preferences, or important events—and storing them as memory entries. These entries then undergo a series of management operations: consolidation [14, 70, 103], which integrates similar memories to improve coherence and reduce redundancy; updating [27, 53, 66], which modifies stored content to maintain accuracy and reflect the latest knowledge; filtering [41, 67, 103], which removes outdated, redundant, or low-utility memories to preserve the efficiency and relevance of the system; and enhancement [28, 84], which marks or surfaces important memories for easy identification and retrieval. This structured process ensures that the memory system remains organized, scalable, and aligned with ongoing user needs. When additional context is required for reasoning or generation, the memory system retrieves and supplies the most relevant information to the LLM, supporting both short-term adaptation and long-term continuity across evolving interactions. Memory and retrieval-augmented generation (RAG) [18, 23, 36, 87] are related but distinct mechanisms. Memory primarily targets stateful, interaction-dependent information that evolves over time and is required for personalization and cross-session continuity [66, 103]. In contrast, RAG primarily targets external knowledge grounding, retrieving evidence from document collections or knowledge bases to supplement domain knowledge and reduce hallucinations [18, 36]. In practice, they are complementary: memory supplies user- and session-specific context, while RAG provides task-relevant factual evidence from external corpora.
3
message history
response relevant information
current query
…
?
LLM Agent Memory System Information Extraction
Information Retrieval Ø lexical-based retrieval Ø vector-based retrieval Ø structure-based retrieval Ø LLM-assisted retrieval
Ø direct archiving Ø summarization-based Ø graph-based
Memory Storage flat
hierarchical
vector-based
graph-based
Memory Management Connecting
Integrating Filtering
Transforming
Updating
Figure 2: An overview of the unified framework for agent memory systems.
❹ Information retrieval. When a new query arrives, this component governs how the agent system retrieves the most relevant information from H to support reasoning or response generation. The above four components capture the key functional distinctions among different agent memory methods.
A UNIFIED FRAMEWORK 4
In this section, we decompose existing agent memory systems into modular components under a unified framework, as illustrated in Figure 2. The framework comprises four key components: ❶ Information extraction, ❷ Memory management, ❸ Memory storage, and ❹ Information retrieval. Taken together, these components capture how existing agent memory systems operate in practice. Given the current user message M and the existing memory H , the agent system operates through four key components: ❶ Information extraction. This component describes how the agent system identifies and extracts essential information from M that is useful for updating the memory. It filters out redundant details and transforms the relevant content into different types of knowledge (e.g., triples derived from text, informational summaries) suitable for downstream processing. ❷ Memory management. This component illustrates how the agent system integrates the newly extracted information with the existing memory H through operations such as consolidation, updating, filtering, and enhancement. The goal is to maintain a coherent, consistent, and up-to-date memory state that accurately reflects accumulated knowledge. ❸ Memory storage. This component specifies how the agent system organizes and persists the processed memory. Depending on the system design, it may employ vector-based, graph-based, or hybrid storage formats to accommodate diverse information retrieval requirements.
INFORMATION EXTRACTION
This component serves to identify and extract the information from M that is both useful and necessary for downstream memory processing. As illustrated in Figure 4, existing agent systems adopt different information extraction methods, which can be broadly categorized as follows: ❶ Direct archiving. This method represents the most straightforward form of information extraction, where the agent system simply archives raw messages and timestamps without any processing. ❷ Summarization-based extraction. This method employs LLMs to generate concise informational summaries from one or more dialogue turns. Memory methods such as A-MEM and Mem0 extract keywords and contextual tags from M, or prompt the LLM to produce an abstracted summary of the raw text. Figure 3 illustrates a representative prompt used for this extraction method. ❸ Graph-based extraction. This method leverages LLMs to extract fine-grained entities and relations from M, forming subject–predicate–object triples for knowledge graph construction (e.g., Mem0𝑔 , Zep). Additionally, temporal metadata such as creation or invalidation timestamps are recorded to support dynamic updates and temporal reasoning within the graph-based memory. A concrete example of the prompt designed for graph-based extraction is provided in Appendix A. 3
or continuity, enabling synchronous updates and alignment across connected memories. Separately, graph-based methods such as Zep and Mem0𝑔 focus on connecting individual episode or entity nodes to support reasoning and retrieval across conceptually or temporally aligned memories. ❷ Integrating Fragmented Memories. Humans tend to summarize daily experiences, retaining only key events while discarding details. Agent systems achieve similar integration through abstraction or summarization. For example, MemoryBank aggregates repetitive daily records into event summaries and refines a global user profile as experiences accumulate. Likewise, MemoChat groups related dialogues under shared topics and produces topic-level summaries. This process reduces redundancy, distills essential information, and transforms scattered memories into concise high-level representations suitable for long-term storage. ❸ Transforming Across Memory Levels. Human memory gradually transfers important information from low-level to high-level storage, reinforcing what is repeatedly recalled. Agent systems adopt similar hierarchical migration mechanisms. For instance, MemoryOS implements a two-stage migration strategy: short-term memories are first moved to mid-term storage following a First-In, First-Out (FIFO) policy, and mid-term memories are then promoted to long-term storage using a heat-based score that jointly considers access frequency and recency. Additionally, memory methods such as Zep organize semantically related memories into communities, forming structured, interconnected long-term representations. This stage strengthens persistent knowledge while maintaining efficiency. ❹ Updating Existing Memories. Humans constantly revise their memories by integrating new experiences and correcting inconsistencies. Agent systems follow a similar principle through three main updating paradigms: (1) Rule-based updating, where existing memories are updated according to predefined rules. For example, MemoryBank adopts Ebbinghaus’s Forgetting Curve theory to adjust memory strength over time. In MemoryOS, new memories are integrated into existing structures based on semantic and keyword similarities; (2) LLM-based updating, where large language models are prompted to summarize, merge, or resolve conflicts between entries. To give an example, MemTree updates its memory by relying on the LLM to execute a specialized Aggregate Operation, where the prompt and the count of descendants guide the LLM to appropriately compress and generalize information before the new content is written back to the parent node. Taking another example, the update process in Zep requires the LLM to execute resolution tasks by strictly following detailed semantic constraints and procedural guidelines given in the prompts. (3) Agent-based updating, where agents autonomously decide which operations (e.g., revise, merge, prune) to apply, as in MemGPT and MemOS. To be more specific, agents are granted access to current context as well as historical or archival memory entries and learn to utilize specialized system tools for managing memories efficiently and flexibly. These strategies ensure that memory remains accurate, consistent, and aligned with evolving knowledge. ❺ Filtering Obsolete Information. Finally, the memory system must remain compact and relevant by filtering outdated or redundant information, which can be achieved by either directly removing the memories, lowering their assigned weight score, or
Prompt for Summarization-based Extraction. Generate a structured analysis of the following message by extracting salient keywords, identifying core themes and contextual elements, and assigning relevant categorical tags. Format the response as a JSON object: { "summary": "<one-sentence gist>", "keywords": [ <several keywords capturing key concepts> ], "tags": [ <several broad themes for classification> ] } Message for analysis: {message} Figure 3: A sample prompt for summarization-based extraction.
Messages Messages
Timestamp Direct archiving
Summary Messages
Keywords
Triples Graph-based extraction
Tags Summarization -based extraction
Figure 4: Methods of information extraction.
5
MEMORY MANAGEMENT
The memory management process governs how an agent system maintains, refines, and evolves its memory over time. As illustrated in Figure 5, it mirrors the human memory lifecycle, encompassing five core operations: connecting related experiences, integrating fragmented information, transforming short-term into long-term memory, updating outdated content, and filtering obsolete knowledge. Table 2 summarizes how representative agent memory methods instantiate these operations through different implementation paradigms. Through this process, the system maintains a coherent, efficient, and adaptive memory state that supports continual learning and reasoning. ❶ Connecting Related Experiences. Humans naturally associate related events across time and context; agent systems emulate this through connection. This mechanism establishes explicit connections between memory entries that share semantic similarity, temporal proximity, or contextual relevance, realized through either structural edges within a graph or associative links across discrete records. For example, memory methods such as A-MEM and MemoryOS leverage associative links based on semantic similarity 4
Structural Edges
specific prompts
Associative Links
(summarize/merge/…)
LLMs
graph
semantic similarity
temporal proximity
Existing Memories
contextual relevance
revise? prune?
Connecting Related Experiences Agents
Specific rules
Fragment memories
Updating Existing Memories
integrate
compress summarization
Usage-based
abstraction distill Integrating Fragmented Memories
access frequency
High memory level
time-based decay
Low memory level long-term storage transform
Filtering
Content-based
synthesized structure
outdated
Transforming Across Memory Levels
removing / lower score / invalid tags
duplicated
Filtered Information
Filtering Obsolete Information
Figure 5: Workflow of the memory management process. applying a status label (such as “invalid”). This filtering process parallels human forgetting, which selectively fades unused or irrelevant memories. (1) Usage-based filtering, as seen in MemoryOS and MemoryBank, relies on access frequency and time-based decay. Memories created a long time ago and rarely retrieved are filtered first. (2) Content-based filtering examines semantic similarity and leverages LLMs to detect and filter duplicated or outdated knowledge, such as in Mem0 and Mem0𝑔 , reducing noise and improving retrieval precision. Together, these mechanisms sustain an efficient, lightweight memory that supports ongoing adaptation and learning.
6
long-term memory for user preferences. By applying different, synergistic management and retrieval strategies to respective storage components, hierarchical storage effectively optimizes the trade-off between computational overhead and knowledge persistence. ❸ Vector-based storage. This approach encodes textual memory into high-dimensional embeddings, subsequently indexed in dedicated vector libraries or databases, such as FAISS [13] and Qdrant, to enable the agent to perform efficient semantic similarity search. Vector-based storage can function as a standalone repository or serve as a foundational building block frequently integrated into more complex storage architectures. ❹ Graph-based storage. This approach utilizes diverse graph topologies, such as trees, knowledge graphs, and temporal graphs, to preserve the rich structural information inherent in memory. For instance, MemTree organizes memory into a hierarchical tree where each node encapsulates aggregated textual content, providing varying levels of abstraction along the tree’s depth; Zep employs a layered temporal knowledge graph that concurrently organizes memory by representing raw messages as nodes, extracting subject–predicate–object triples, and clustering entities into communities. These graph-based storage methods capture intricate relationships and multi-hop associations that lie beyond the reach of simple vector similarity metrics.
MEMORY STORAGE
The memory storage component governs how the processed memory is organized and persisted across two primary dimensions: organization-centric and representation-centric. The former determines the architectural depth of the storage system, encompassing flat storage and hierarchical storage, and the latter characterizes the employed technological paradigm, primarily comprising vectorbased storage and graph-based storage. ❶ Flat storage. This approach represents a unified, single-tier storage that aggregates all information within a homogeneous space, such as a FIFO queue or a JSON file, defined relative to hierarchical storage within the organization-centric dimension. ❷ Hierarchical storage. This approach partitions memory into specialized, multi-tiered architectures, allowing individual storage components to fulfill distinct functional roles and operate at different levels of granularity. For example, MemoryOS organizes memory into a three-tier hierarchical structure: short-term memory for timely conversations, mid-term memory for topic summaries, and
7
INFORMATION RETRIEVAL
This component governs how the agent system identifies and extracts the most relevant information from memory storage to support informed reasoning or context-aware response generation. Existing information retrieval strategies can be broadly categorized 5
Table 2: Comparison of implementation paradigms for memory management operations; “N/A” means that this operation is not explicitly implemented. Method A-MEM MemoryBank MemGPT Mem0 Mem0𝑔 MemoChat Zep MemTree MemoryOS MemOS
Connecting
Integrating
Transforming
Updating
Filtering
Associative Links N/A N/A N/A Structural Edges N/A Structural Edges Structural Edges Associative Links Structural Edges
N/A Summarization Summarization Summarization N/A Abstraction N/A Summarization Abstraction + Summarization Abstraction + Summarization
N/A N/A Stage-wise Transfer N/A N/A N/A Community Formation N/A Stage-wise Transfer N/A
LLM-based Rule-based Agent-based Agent-based LLM-based N/A LLM-based LLM-based Rule-based Agent-based
N/A Usage-based N/A Content-based Content-based N/A N/A N/A Usage-based N/A
into four paradigms based on the fundamental mechanisms they employ: ❶ Lexical-Based Retrieval. This paradigm relies on the overlap of surface-level tokens or terms, typically implemented through some representative techniques such as set-based matching via the Jaccard similarity coefficient or scoring models like BM25 [74]. Lexical-based retrieval provides a strong baseline for exact term matching, which can be particularly effective for retrieving names, specific entities, or phrases where precise wording is critical. ❷ Vector-Based Retrieval. This paradigm leverages semantic similarity in a continuous vector space to address the vocabulary mismatch problem inherent in exact keyword matching. By encoding both the query and memories into high-dimensional vectors via embedding models, vector-based retrieval is formulated as a top-𝑘 search for the most relevant entries using distance metrics like cosine similarity. This approach excels at capturing latent semantic nuances, ensuring that relevance is determined by semantic content rather than surface-level lexical form. To maintain efficiency within the massive scale of memory storage, Approximate Nearest Neighbor (ANN) search algorithms, such as HNSW [58] or PQ [30] (Product Quantization), are frequently employed. ❸ Structure-Based Retrieval. This paradigm exploits the explicit relational connections between memory entities, often operating on graph-based or hierarchical storage, performing graph traversal, neighborhood expansion, or multi-hop reasoning to retrieve interconnected clusters of information instead of simple queryto-item matching. For example, Mem0𝑔 explores the relationships starting from nodes identified through similarity search to construct a comprehensive subgraph that captures relevant and multi-faceted information. Similarly, Zep utilizes a BFS-based graph traversal algorithm to enhance initial search results by identifying additional nodes and edges. ❹ LLM-Assisted Retrieval. This paradigm integrates LLMs as an active reasoning component to guide or refine the retrieval process. In addition to directly deciding which specific information should be retrieved, LLMs can also be utilized to transform ambiguous user prompts into precise search queries or to identify key entities within a query to facilitate more targeted retrieval. By leveraging
the reasoning capabilities of LLMs, this paradigm excels at uncovering latent semantic dependencies, thereby ensuring a more closer alignment between queries and retrieved knowledge.
8
EXPERIMENTS
We now present the experimental results. We discuss the setup in Section 8.1, and then report the evaluation results across several experiments in Section 8.2.
8.1
Setup
Workflow of our evaluation. We conduct a systematic experimental study of agent memory mechanisms along three dimensions: (1) we collect and reimplement 10 representative methods within the unified framework described in Section 3; (2) we perform a comprehensive evaluation on two widely-used long-term memory benchmarks using multiple complementary metrics; (3) we carry out multi-dimensional analyses to assess architectural trade-offs and robustness, covering token cost efficiency, ground-truth position sensitivity, context scalability, and LLM backbone dependence. Benchmark Datasets. We employ two benchmark datasets, LOCOMO and LONGMEMEVAL, to evaluate the performance of each memory mechanism. Both datasets are designed to assess longterm conversational memory capabilities but represent two distinct interaction scenarios: LOCOMO is grounded in dialogues between two human users, whereas LONGMEMEVAL is based on user–AI interactions. • LOCOMO. The LOCOMO benchmark [57] comprises ten longterm conversations for question-answering evaluation. Each conversation features an average of 198.6 questions spanning 27.2 sessions and approximately 588.2 dialogue turns between two speakers. Questions are categorized into four types: SingleHop Retrieval, Multi-Hop Retrieval, Temporal Reasoning, and OpenDomain Knowledge. • LONGMEMEVAL. The LONGMEMEVAL benchmark [91] contains 500 high-quality questions designed to evaluate four core long-term memory abilities: Information Extraction, Multi-Session Reasoning, Knowledge Updates, and Temporal Reasoning. Each question is grounded in a dedicated conversation history based 6
Table 3: Comparison of methods on LONGMEMEVAL, where Purple denotes the best result, and Orange denotes the best result excluding the best one. Information Extraction user
Method F1
BLEU-1
assistant
preference
F1
F1
BLEU-1
Multi-Session
BLEU-1
F1
BLEU-1
Temporal
Knowledge Updates
Overall
F1
BLEU-1
F1
BLEU-1
F1
BLEU-1
19.37 17.80 21.65 28.86 29.16 17.49 — 29.97 26.34 21.34
15.54 13.15 14.81 21.19 21.60 11.35 — 21.77 21.61 15.65
25.59 31.51 28.09 41.28 40.43 6.51 — 41.49 30.62 33.99
21.63 27.31 24.85 36.37 35.51 4.28 — 38.56 27.97 27.75
25.53 27.65 29.16 32.46 30.66 12.16 — 36.92 32.50 32.48
21.24 23.03 24.08 26.95 25.38 8.26 — 31.05 28.31 26.38
22.93 25.32 24.48 30.85 33.39 27.33 — 34.65 32.31 30.47
19.40 21.73 18.47 24.34 28.97 19.24 — 24.30 23.63 19.51
39.19 36.02 29.21 43.35 47.15 21.33 — 48.90 53.46 48.51
28.84 29.34 23.16 40.14 43.23 17.70 — 43.63 48.96 40.85
31.32 35.79 32.63 37.85 38.47 22.09 — 44.25 46.04 39.88
25.79 30.70 25.82 32.62 33.25 17.18 — 37.02 39.42 32.02
Qwen2.5-7B-Instruct A-MEM MemoryBank MemGPT Mem0 Mem0𝑔 MemoChat Zep MemTree MemoryOS MemOS
46.75 46.28 52.82 56.51 53.21 6.44 — 68.48 56.85 65.48
41.26 42.19 47.54 51.23 47.39 3.24 — 61.85 53.49 56.02
43.21 49.31 52.26 32.35 18.18 16.95 — 57.79 61.32 49.40
36.74 43.08 44.56 25.99 13.82 12.74 — 46.73 52.71 42.44
9.76 13.96 13.61 11.44 10.23 7.18 — 9.50 12.40 12.17
A-MEM MemoryBank MemGPT Mem0 Mem0𝑔 MemoChat Zep MemTree MemoryOS MemOS
54.43 52.96 54.81 67.48 62.63 18.98 — 66.21 73.00 69.06
46.92 48.64 49.39 60.81 54.37 15.29 — 58.77 68.23 62.64
49.05 57.41 63.23 47.38 40.21 26.38 — 68.09 75.10 53.88
39.91 51.63 47.84 39.08 35.82 20.21 — 58.35 64.12 44.09
10.38 12.53 4.78 11.89 10.75 7.80 — 11.09 12.43 12.37
0.52 3.79 2.90 0.69 0.79 0.12 — 1.00 1.18 0.77
16.61 19.41 18.61 23.02 24.42 12.27 — 21.96 19.35 22.81
14.32 16.42 16.71 20.72 22.04 10.09 — 19.88 17.80 19.72
Qwen2.5-72B-Instruct 0.55 4.91 0.03 0.71 0.53 0.09 — 0.17 1.20 0.70
19.07 28.21 19.29 28.13 31.47 20.46 — 37.02 36.56 29.17
on long-term user–AI interactions, averaging 50.2 sessions and approximately 115,000 tokens in length.
17.04 24.97 16.04 26.41 27.00 18.39 — 33.74 32.67 25.24
adopted in existing agent memory studies. The maximum context length is set to 20,000 tokens, and we employ greedy decoding to ensure deterministic outputs. For all methods that involve top-𝑘 retrieval, following the previous work [32, 73, 93], we set 𝑘 = 10 to fit within the context length budget. We adopt all-MiniLM-L6v2, a representative and widely-used sentence-transformer model, as the unified embedding model across all methods. All remaining method-specific hyperparameters follow the original settings reported in their respective papers and codebases.
Further dataset details are provided in Appendix B.1. Due to the configurable structure of LONGMEMEVAL, we construct specific variants to evaluate context scalability and position sensitivity. The detailed variant construction methodology is outlined in Appendix B.2. Evaluation Metric. Following the evaluation protocols of the two benchmarks and prior studies on long-horizon conversational memory, we adopt two complementary metrics. F1 measures tokenlevel overlap by balancing precision and recall. BLEU-1 captures unigram-level modified precision with a brevity penalty, reflecting lexical fidelity to the reference answer. These two metrics are reported for each capability category on both datasets, consistent with the standard convention in representative agent memory studies. Implementation. We implement all methods in Python under the proposed unified framework, ensuring faithful and consistent reimplementation based on original papers and publicly available code. All experiments are conducted on 8 NVIDIA A100 (80 GB) GPUs. If a method cannot finish in two days, we mark its result as “— ” in the tables. Additionally, since F1 and BLEU-1 metrics are highly sensitive to answer verbosity, we apply a uniform simplification step to all generated answers. The specific prompt for this process is detailed in Appendix A. Hyperparameter Settings. Unless otherwise stated, we use Qwen2.5-7B-Instruct as the default LLM backbone, as it is widely
8.2
Evaluation
Exp.1. Overall performance. We first report the performance of all agent memory methods on both benchmarks across two model scales (Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct). Results on LONGMEMEVAL and LOCOMO are shown in Table 3 and Table 4, respectively. Based on these results, we draw the following observations: (1) Tree-based memory methods (e.g., MemTree and MemOS) generally achieve strong performance by organizing memory in a multilayered, multi-granularity fashion. Specifically, MemTree attains the highest F1 score of 36.92 on LONGMEMEVAL at the 7B scale, while MemOS achieves the highest F1 scores of 37.05 and 42.79 on LOCOMO at the 7B and 72B scales, respectively. Tree structures provide high-level conceptual summaries at upper layers while preserving fine-grained details at leaf nodes. A similar advantage can be realized through well-designed hierarchical architectures that 7
Table 4: Comparison of methods on LOCOMO.
Method
Single-Hop F1 BLEU-1
Multi-Hop F1 BLEU-1
Temporal F1 BLEU-1
Open-Domain F1 BLEU-1
Overall F1 BLEU-1
Qwen2.5-7B-Instruct A-MEM MemoryBank MemGPT Mem0 Mem0𝑔 MemoChat Zep MemTree MemoryOS MemOS
33.11 15.34 21.31 25.92 27.90 7.21 40.42 33.36 33.14 39.55
27.59 11.92 16.57 21.47 23.45 5.84 34.59 27.91 28.33 33.58
25.38 16.63 18.92 19.43 21.14 9.74 30.97 26.40 28.52 34.04
18.36 12.41 13.64 14.76 13.98 6.59 20.31 17.59 19.90 23.38
14.69 11.57 17.37 37.49 35.70 5.33 22.64 25.62 17.65 37.55
12.19 9.49 13.61 31.26 30.54 4.26 17.82 21.45 14.09 31.90
14.69 16.09 11.14 16.93 18.67 14.31 21.79 20.67 22.05 22.55
12.55 11.16 8.06 10.58 13.44 10.81 17.57 16.23 16.95 16.51
26.71 14.84 19.42 26.58 27.71 7.72 33.82 29.68 28.34 37.05
21.75 11.46 15.51 21.60 22.57 5.96 27.42 23.95 23.11 30.30
23.27 19.12 16.42 21.04 20.37 15.43 15.87 23.02 22.57 24.88
18.55 14.35 13.42 14.09 12.81 11.65 11.47 17.46 18.22 17.87
30.35 21.61 25.40 32.38 32.07 13.83 37.45 34.85 39.63 42.79
25.46 17.50 20.30 27.51 27.06 10.78 30.46 28.49 32.62 35.16
Qwen2.5-72B-Instruct A-MEM MemoryBank MemGPT Mem0 Mem0𝑔 MemoChat Zep MemTree MemoryOS MemOS
42.24 24.54 28.78 32.83 29.58 12.79 42.46 38.73 43.56 44.30
38.68 20.26 21.15 27.89 25.19 10.02 35.99 32.42 37.00 36.85
27.87 23.28 24.21 25.35 30.22 13.16 33.11 33.01 36.32 36.67
21.97 18.62 19.11 21.14 24.67 9.19 23.39 25.07 27.91 26.95
28.01 13.21 20.28 40.76 43.89 16.64 34.58 29.81 37.36 49.59
facilitate efficient information flow and transformation across different levels of abstraction, as evidenced by the highly competitive results of MemoryOS and Zep. (2) Preserving information completeness is crucial for effective memory persistence—specifically, retaining raw messages during the information extraction phase and incorporating original conversations during final response generation. For example, methods that exclusively extract graph-based triples may suffer from information loss compared to those that preserve raw dialogue fragments, which may explain why Mem0 outperforms Mem0𝑔 in many cases. (3) Effective memory organization requires mechanisms that build explicit or implicit connections among related pieces of information, thereby enhancing the coherence of stored memories. This is particularly important for multi-hop reasoning tasks. Methods lacking such associative operations, such as MemoryBank, MemGPT, and MemoChat, perform poorly on the Multi-Session tasks of LONGMEMEVAL and the Multi-Hop tasks of LOCOMO. In contrast, although Mem0 does not feature an explicit “connecting” operator, its strategy of concurrently updating similar memories during ingestion achieves a comparable integration effect. Notably, on the Multi-Session tasks of LONGMEMEVAL, Mem0 achieves 18.60% F1 and 26.19% BLEU-1 improvements over MemoryBank. (4) Due to the complexity and inherent difficulty of temporal reasoning, these tasks remain highly sensitive to the reasoning capacity of the backbone model. For instance, MemoryOS and MemoChat exhibit a performance leap exceeding 2× on LOCOMO as the backbone LLM scales from 7B to 72B. To mitigate this heavy dependence on LLM reasoning and improve system robustness, it is essential to
22.64 10.18 16.37 36.03 38.44 13.91 27.89 24.48 29.58 43.13
Average Token Costs
2500 2000
+ MemOS
A-MEM
+
+ +
Mem0
1500
MemTree
Mem0g + MemGPT
1000
+
+ MemoryOS
500 0
MemoChat MemoryBank +
+
10
20 30 F1 Score
40
Figure 6: Overall trade-off between performance and token costs on LOCOMO. design specialized architectural components for temporal information processing rather than relying solely on the model’s in-context reasoning during response generation. Exp.2. Token cost analysis. In this experiment, we evaluate the computational overhead of each method from two perspectives: (1) we analyze the overall trade-off between performance and cost. Figure 6 illustrates the relationship between the average token cost per dialogue (Y-axis) and the overall F1 score (X-axis); (2) we examine the scalability of each method during the memory ingestion phase. Figure 7 shows how the average token consumption evolves as the volume of ingested memory increases. Based on these results, we make the following observations: 8
Zep
MemoryBank
A-MEM
Mem0 Mem0
MemGPT
g
MemoChat
MemoryOS
MemTree
MemOS
Table 5: Position sensitivity analysis of representative memory mechanisms across Information Extraction sub-tasks on LONGMEMEVAL (F1 Score).
8000
Average Token Costs
7000
Method
Position
user
assistant
preference
Mem0𝑔
Early Middle Late Improvement
53.95 48.88 63.48 +9.53
19.62 17.02 17.91 -1.71
9.93 10.52 11.01 +1.08
MemTree
Early Middle Late Improvement
52.85 60.03 63.59 +10.74
49.26 60.82 58.18 +8.92
9.31 10.58 9.72 +0.41
MemOS
Early Middle Late Improvement
60.65 64.24 69.88 +9.23
43.00 49.26 51.30 +8.30
11.20 11.97 11.86 +0.66
6000 5000 2000 1500 1000 500 0
1
4
7
10
Sessions Figure 7: Average token costs per dialogue across sessions on LOCOMO. (1) In general, higher performance correlates with increased token consumption, reflecting the benefit of extensive LLM utilization. However, the design of the memory framework remains the primary determinant of cost efficiency. While MemTree and MemOS achieve high accuracy, they incur substantial token overhead. In contrast, MemoryOS maintains a competitive balance, achieving strong performance with significantly lower token cost. Simpler methods such as MemoChat and MemoryBank minimize token usage but fail to deliver adequate accuracy. Notably, although both MemGPT and MemOS are inspired by operating systems and employ agent-based updates to autonomously decide which operations to apply, MemOS allocates more tokens to sophisticated decision-making, yielding superior performance compared to MemGPT. (2) The granularity of information processing significantly influences token cost. During the information extraction phase, this granularity is determined by whether information is extracted from individual dialogue turns or collectively from multiple turns. For instance, MemoryOS organizes dialogues into segments for mid-term storage, while MemoryBank compiles historical messages into summaries at a daily granularity. Owing to the strong reasoning capabilities of modern LLMs, such coarsening of granularity does not necessarily compromise performance and may even improve it, offering a viable path toward enhancing memory efficiency. (3) As memory volume grows, certain methods exhibit poor scalability in terms of average token cost. For MemTree, the cost of processing each turn escalates as the tree depth increases with the accumulated dialogue history, since every top-down insertion of a new dialogue node requires updating all nodes along the path. Similarly, the graph complexity of Zep grows with the number of dialogue turns, leading to rising costs for deduplication and consistency maintenance. Exp.3. Context scalability analysis. In this experiment, we investigate the context scalability of various memory architectures by expanding the context length of LONGMEMEVAL from 50% to 200%. In long-horizon interactions, the primary challenge for memory systems shifts from simple retrieval to robust noise suppression as information density grows. We evaluate various architectures across different context sizes to see how well they maintain retrieval
precision as context grows. Figure 8a illustrates the overall trends in context scalability, while comprehensive experimental results are detailed in Appendix B.3. (1) As the context scale expands from 50% to 200%, nearly all memory architectures exhibit a steady decline in F1 scores. This performance attrition is primarily driven by the increased density of irrelevant information, which lowers the signal-to-noise ratio during retrieval. (2) The scaling results reveal a performance divide rooted in operational complexity. Methods such as MemOS and MemGPT implement an “LLM-as-OS” paradigm, requiring the LLM to autonomously manage memory via complex tool calls. As the scale increases to 200%, the expanded candidate space significantly raises the difficulty for the LLM to accurately reason over and execute these management instructions, leading to higher rates of tool-call failures and indexing conflicts. In contrast, MemoryOS reduces the agent’s cognitive load by employing explicit rule-based hierarchical management. By offloading organizational logic from the LLM to a deterministic framework, MemoryOS maintains high stability, demonstrating that simplifying the agent’s internal management overhead through deliberate design is critical for robust scaling. (3) Different task categories exhibit varying sensitivity to scaling pressure. As illustrated in Figure 9, Knowledge Update (KU) is particularly sensitive, showing sharp performance attrition because an increased memory volume raises the density of conflicting records. Since KU requires the model to identify the latest fact among mutually exclusive versions, the presence of more obsolete candidates directly increases retrieval interference. Conversely, temporal tasks remain relatively stable, as they rely on the relative ordering of events, which remains structurally distinct even as the background volume grows. Unlike the versioning conflicts in KU, the chronological precedence of events is not easily compromised by the addition of context. Exp.4. Position sensitivity analysis. In this experiment, we evaluate how the placement of key evidence affects retrieval and reasoning on variants of LONGMEMEVAL by positioning the evidence in the early (first 1/3), middle (middle 1/3), or late (last 1/3) sections of the context. As evidence appears earlier in the context, 9
F1 Score
40
50%
100%
150%
200%
MemGPT
Mem0
Mem0g
MemTree
35 30 25 20
A-MEM
MemoryBank
MemoryOS
MemOS
MemoryOS
MemOS
(a) Context scalability
Early
F1 Score
40
Middle
Late
35 30 25 20
A-MEM
MemoryBank
MemGPT
Mem0g
Mem0
MemTree
(b) Position sensitivity
Figure 8: Robustness analysis of memory mechanisms on LONGMEMEVAL. (a) illustrates the context scalability as the input scale varies from 50% to 200%, while (b) shows the position sensitivity regarding ground-truth information placed at different relative positions: Early (first 1/3), Middle (middle 1/3), and Late (last 1/3).