Beyond Retrieval: Analytic Memory for Multimodal Agents Zhoujin Tian1 , Yao Tian2 * , Hao Zhang2 , Cheng Chen2 Yakun Li2 , Lei Zhang2 , Xiaofang Zhou1 1 HKUST, 2 ByteDance {ztianaf, zxf}@cse.ust.hk {yao.tian, zhanghao.ai, chencheng.sg, liyakun.hit, zhanglei.michael}@bytedance.com
arXiv:2607.29440v1 [cs.AI] 31 Jul 2026
Abstract Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions. Existing systems largely emphasize retrieval memory, organizing interaction histories through summaries and indexes to return query-relevant information at multiple granularities, from high-level abstractions to underlying records. In this paper, we formulate analytic memory as a complementary abstraction that organizes recurring multimodal observations into queryable structures supporting filtering, aggregation, ranking, and temporal comparison. We present A DA MM, a framework that jointly supports retrieval and analytic memory. Rather than relying on application-defined schemas, A DA MM extracts provenance-linked attribute-value observations from dialogue, images, and contextual metadata, discovers recurring field structures, and materializes them for analytical access. At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools. Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that A DA MM improves performance by up to 11.3% and 7.3%, respectively.
1
Introduction
As Large Language Model (LLM) agents interact with users over increasingly long horizons, they accumulate extensive multimodal histories – including dialogues, screenshots, images, documents, and contextual observations – that quickly exceed the finite context windows of their backbone models (He et al., 2024; Koh et al., 2024; Xie et al., 2024; Wang et al., 2024). Multimodal memory systems, which retain and organize such histories to support future reasoning (Wang and Chen, 2025; Feng et al., 2026), have therefore become essential * Corresponding author.
Long-term Historical Multi-modal Conversation Sessions Apr 23 Good morning!
Apr 24
May 1
May 2
A Bad sleep.
Shopping at …
Good sleep !
…
Good morning! I ordered … Starbucks Americano
$3.45
Any reason …
Coffee can …
…… ……
Good news …
…
Some exercise
I am having lunch with …
…
……
Access
User Queries Average sleep duration over the past month?
Which market last time?
…
… …… ……
Lots of things !
Tools
Results Retrieve
Top-k
Memory
Retrieval Memory
…
…
Traditional System
… Analytic Memory
Our AdaMM
Figure 1: Long-term multimodal interaction histories contain observations distributed across sessions, modalities, and time. Answering user queries may require either retrieving specific past events or performing analytical operations, such as temporal selection and aggregation, over multiple observations.
and are widely adopted in applications such as personal assistance (Li et al., 2024; Feng et al., 2026), embodied interaction (Driess et al., 2023; Wang et al., 2023), and tool-augmented workflows (Yao et al., 2022; Schick et al., 2023). Recent multimodal memory systems have explored a broad range of designs, including textual abstraction (Feng et al., 2026), specialized memory types (Nguyen et al., 2026), hybrid stores (Wang and Chen, 2025), and cross-modal retrieval (Lu et al., 2026). Most existing systems follows a retrieve-then-answer paradigm, conditioning an LLM on a bounded set of relevant memories, which we denote as retrieval memory. Such systems are effective at selecting relevant memories from long interaction histories. However, long interaction histories also accumulate recurring observa-
tions that collectively form an append-only log. Analytical questions over such histories require complete, correctly scoped records and operations such as filtering, aggregation, ranking, and temporal selection. Relevance-based approach adopted by retrieval memory systems therefore creates a coverage–context trade-off: a small retrieval set may omit required observations, whereas a larger retrieval set consumes the limited context budget and introduces redundant or distracting evidence. For example, as illustrated in Figure 1, computing a user’s average sleep duration over the past month requires collecting all relevant measurements across multiple screenshots before aggregation; incomplete retrieval may therefore produce a biased estimate. We call this gap between relevance-based retrieval and analytic the retrieval– analysis mismatch. To bridge this gap, our key insight is that a memory system should couple two complementary subsystems: retrieval memory, which identifies and supplies relevant historical records as context, and analytic memory, which proactively organizes interaction histories into reusable and queryable structures and adaptively executes operations over them. Together, they support both flexible semantic recall and structured analysis, rather than requiring the LLM to repeatedly reconstruct the needed organization from retrieved records. Realizing analytic memory, however, presents two key challenges. First, interaction histories do not come with an explicit organizing schema, making it unclear what information should be preserved and how they should be structured to support future queries. Second, user queries rarely specify memory access strategies, the system therefore must reconcile what each query requires with what the evolving memory can currently support. Even questions over the same underlying records may require fundamentally different operations (e.g., aggregation, ordering, filtering), posing challenges to the design of adaptive execution plans. To address these challenges, we introduce A DA MM, an Adaptive Multi-view Memory framework that couples data-driven structure induction with memory-aware query planning. It maintains two complementary subsystems: retrieval memory for flexible semantic access and analytic memory for executable analysis over recurring observations. A DA MM extracts provenance-linked attribute–value pairs from multimodal interactions (e.g., {Sleep Time: 5.5 hrs}), induces recurrent
schemas, and materializes them as structured tables, which is accessible via tools, while preserving unstructured evidence in a hierarchical semantic graph. At query time, a planner adaptively composes different analytic and retrieval tools based on the query and the memory structures currently available. This design enables grounded semantic recall and structured analysis over long-term interaction histories. Our contributions are summarized as follows: • We identify a retrieval–analysis mismatch in long-term multimodal memory: retrievaloriented interfaces support returning queryrelevant information at multiple granularities, but cannot reliably answer queries requiring complete-range filtering, aggregation, ranking, or temporal selection over append-only interaction histories. • We propose A DA MM, a multimodal memory framework that complements retrieval memory with schema-induced analytic memory. A DA MM discovers recurrent structures from multimodal observations, materializes them as queryable tables, and uses a memory-aware planner to select or compose semantic retrieval with native analytical operations. • We evaluate A DA MM on two multimodal longterm memory benchmarks, MemEye and MemGallery, where A DA MM consistently outperforms strong memory baselines, and improves accuracy by up to 11.3% and 7.3%, respectively.
2
Related Work
Agent memory. Long-term agent memory preserves information across interactions, allowing agents to reuse preferences, observations, decisions, and task states beyond a finite context window (Hatalis et al., 2023; Li et al., 2024; Zhang et al., 2025). Existing systems explore personalized stores and tiered memory management (Zhong et al., 2024; Packer et al., 2023), scalable or agentic memory construction (Chhikara et al., 2025; Xu et al., 2025), and temporal or relational organization (Rasmussen et al.; Huang et al., 2025). Recent schema-grounded memory moves beyond semantic recall by transforming textual interactions into validated, queryable records, but relies on application-defined schemas (Petrov et al., 2026). Despite these advances, existing methods remain largely text-centric, leaving underexplored how to
induce reusable analytical structures from heterogeneous multimodal histories and adaptively execute retrieval and analytical operations over them. Multimodal agent memory. Multimodal agent memory extends text-centric memory to histories containing heterogeneous visual observations. Existing approaches broadly follow two directions: MIRIX and M2A translate visual content into captions, summaries, or semantic abstractions for textbased retrieval (Wang and Chen, 2025; Feng et al., 2026), whereas MMA, VisualMem, and OmniSimpleMem retain native visual evidence and retrieve multimodal or image-backed memories (Lu et al., 2026; Nguyen et al., 2026; Liu et al., 2026). Although these methods improve visual evidence preservation, they still primarily organize histories as semantic memories. MemEye further reveals their difficulty in capturing fine-grained details and evolving visual states (Guo et al., 2026). In contrast, A DA MM adaptively organizes multimodal histories into complementary analytical and semantic memories, enabling both precise analysis and context-aware retrieval for complex queries.
3
Method
3.1
Problem Formulation
Let H = {Si }M i=1 denote a multimodal interaction i history, where each session Si = {Ri,j }nj=1 consists of multiple interaction rounds. Each round Ri,j = (di,j , Vi,j , τi,j ) comprises the user–agent dialogue di,j , visual observations Vi,j , and temporal information τi,j . A multimodal agent memory system transforms the interaction history into memory representations, M = Fbuild (H), and aims to generate an accurate answer ŷ = Fanswer (M, q) for a user query q based on M. 3.2
Overview
Figure 2 presents the overview of the proposed A DA MM framework. Given a long-term multimodal interaction history, A DA MM constructs two complementary memories: analytic memory Mana induces recurring patterns as adaptive schemas for analytical queries while retrieval memory Mret hierarchically organizes interactions according to their semantics to enable flexible retrieval. At query time, A DA MM jointly reasons over the query and the memories instantiated from the current history, then constructs an execution plan by adaptively selecting and composing access tools to gather sufficient evidence for grounded answer generation.
3.3
Analytic Memory Construction
Analytic memory aims to uncover and organize recurring patterns across fragmented observations in multimodal interaction histories for analytical operations. The key challenge is that the meaningful attributes, which is in the form of key-value pair, and their co-occurrence patterns are unknown in advance, and each round reveals only a sparse fragment of the latent structure. To address this challenge, A DA MM treats H as a chronologically ordered interaction rounds {Rt }N t=1 , performs Attribute Extraction to adaptively identify attribute evidence within each round, then conducts Schema Induction to discover stable structures from recurring cross-round patterns, and finally applies Memory Materialization to transform the induced schemas into executable analytic memory. 3.3.1
Attributes Extraction
Recovering recurring structure must begin from the evidence available within individual rounds, yet the attributes that will form a useful schema are unknown at extraction time. Record fragment extraction therefore aims to preserve explicit analytic observations from each round without imposing a predefined and unobserved pattern. Given Rt , A DA MM jointly examines its dialogue and visual content to identify every grounded attribute–value correspondence: t Ot = Extractor(Rt ) = {(atℓ , xtℓ , ptℓ )}m ℓ=1 , (1)
where atℓ and xtℓ are the observed attribute and its source-faithful value, and ptℓ points to the supporting dialogue span or image. Extractor is an LLM-based extractor that targets at identifying all possible pairs from each round. The resulting collection {Ot }N t=1 captures these observations and their within-interaction co-occurrence patterns for subsequent schema induction. 3.3.2
Schema Induction
Given the extracted record fragments, it remains challenging to induce reusable schemas without assuming a predefined schema. New observations may reveal either a previously unseen structure or evolution to an existing one. To address this challenge, A DA MM first mine candidate patterns and then induces schemas through two processes: Schema Discovery creates new schemas, while Schema Evolution extends existing ones with consistently co-occurring attributes.
Historical Sessions
Memory Construction
…… Apr 24
{sleep: 4.8h, heart: 78}
Analytic Memory
……
… Apr 26
{sleep, heart}
{hospital: A, doctor: Bob}
{hospital, doctor}
{sleep: 8.2h, heart: 58}
……
⋯
{sleep: 7.9h, heart: 60}
I went to hospital this morning …
…… ……
May 1
date
sleep
heart
×
Apr 24
4.8 h
78
May 1
8.2 h
58
✓
May 2
7.9 h
60
⋯
⋯
⋯
Hierarchical Retrieval Memory Topic → Episode → Event
Open-ended Events Retrieval Memory
✓
⋯
⋯
…
Memory Materialization
Schema Induction
Attribute Extraction
Topic
⋯
…… ……
Episode 1
Episode 2
Episode N
…
Adaptive Query Processing
May 2
High-level Plan Generation
……
Query conditioned metadata retrieval
User Queries My average sleep after going to the hospital ?
Table Metadata schema: {sleep, heart}
Episode Metadata
Access Tools
episodes: {Episode 1, …}
Query Plan
Lookup · Filter · Compute · Rank SemanticMatch · EventLocate
1 EventLocate: go to hospital LLM Planner
2 Compute: average sleep time after hospital
Progressive Tool Execution 8.1 h
8.1 hours.
Compute(AVG ( sleep ) where date > Apr 26)
Apr 26
EventLocate (go to hospital)
Figure 2: Overview of A DA MM. It complements recall-oriented retrieval memory with schema-induced analytic memory over recurrent multimodal observations. At query time, an operation planner jointly considers the query and the instantiated memories to select designed access tools for answer generation.
Candidate Pattern Mining. For each extracted record fragment Ot , let At = {atℓ : (atℓ , xtℓ , ptℓ ) ∈ Ot }
(2)
denote the set of attributes contained in Ot . A reliable candidate pattern C is a set of attributes that consistently recur sufficient interactions together. Following the Apriori framework for frequentitemset mining (Agrawal and Srikant, 1994), we use support to quantify information recurrence. Specifically, the support of a candidate pattern C after processing round t is suppt (C) =
t X
I[C ⊆ As ] ,
(3)
s=1
A DA MM retains C as a candidate pattern if suppt (C) ≥ θs ,
(4)
which ensures that C recurs across sufficient interaction rounds to yield a populated schema. Schema Discovery. For each recurrent candidate C, A DA MM first compares it with the active schemas before round t. If C exhibits an attribute
structure distinct from all existing schemas, it is evaluated for reliable new-schema discovery. Following a variant of Apriori algorithm (Omiecinski, 2003), we use all-confidence to measure withinpattern co-occurrence consistency, defined as all-conf t (C) =
suppt (C) . maxa∈C suppt ({a})
(5)
A candidate is admitted as a new schema if all-conf t (C) ≥ θa .
(6)
This constraint requires the complete candidate to occur reliably whenever any of its constituent attributes occurs, thereby preventing a ubiquitous attributes from being merged with most patterns. To avoid admitting a qualified candidate together with its qualified subsets, A DA MM retains only inclusion-maximal novel candidates. Schema Evolution. For an existing schema A, if C ⊆ A, then C is already subsumed by it. If C contains A, A DA MM treats C as a potential extension of A. Let ∆ = C \ A is the newly added attributes. We use extension confidence to measure
how consistently ∆ accompany occurrences of A, defined as ext-conf t (C | A) =
suppt (C) suppt (A)
(7)
When ext-conf t (C | A) ≥ θe , the candidate provides sufficient evidence that ∆ have become stable components of A, and we update the schema A to C. Otherwise, A remains unchanged. 3.3.3
Memory Materialization
Considering that each schema specifies a stable set of attributes and each matched interaction provides their co-occurring values, the induced patterns naturally align with relational tables. A DA MM therefore materialize each schema A into a table TA , using its attributes as columns and the matched interactions as rows. Specifically, for a newly discovered schema A, its attributes first define the data columns. Given these attributes and their sampled values from the corresponding extracted observations {Ot }, a language model generates the table metadata, including its name and description, and infers the type of each column. We then populate the table at the interaction level. For each associated round Rt , every extracted tuple (atℓ , xtℓ , ptℓ ) ∈ Ot whose attribute belongs to A is mapped to the corresponding column, and values extracted from the same round jointly form one row. Two auxiliary columns, order and time, record the position and occurrence time of the source interaction, respectively. Any value not observed in that round is left empty. When a schema evolves from A to A′ = A∪∆, the attributes in ∆ are appended as new columns to the existing table TA . A DA MM then updates the table information and rematerializes the rows from their associated observations following the same procedure. The resulting analytic memory is represented as the collection of tables Mana = {T }. 3.4
Retrieval Memory Construction
Beyond structured analytic queries, many user requests can be addressed through direct semantic matching against past interactions. Retrieval memory is designed for such queries by preserving openended events, relations, and visual details that may not exhibit recurring schemas. It complements analytic memory by enabling flexible semantic retrieval and context-aware reasoning over interaction histories.
Following prior work on hierarchical memory organization (Yue et al., 2026), we construct retrieval memory Mret into three levels: topic → episode → event. Events preserve fine-grained evidence extracted from individual interactions, episodes group temporally adjacent and semantically coherent events, and topics aggregate related episodes into broader semantic contexts. Each level maintains a concise description, multimodal representation, temporal information, and provenance. This hierarchy allows the planner to first identify relevant topics or episodes and then locate supporting events, preserving both high-level context and fine-grained evidence for complex queries. 3.5
Adaptive Query Processing
Different queries place different demands on memory: some require semantic retrieval over openended events, whereas others require structured computation over analytic memory. No single retrieval primitive can adequately support this full range of needs, calling for an adaptive mechanism that selects and composes operations according to query intent and the structures available in memory. A DA MM therefore abstracts the native capabilities of analytic and semantic memory as operationspecific tools, and employs a joint query–memory planner to generate an executable procedure for answering each query. 3.5.1
Memory Access Tools
Analytic and semantic memory differ in their underlying structures and consequently provide distinct native capabilities. To expose these capabilities at query time, A DA MM equips each memory with structure-specific tools. Specifically, each tool is represented as ξi = (di , Θi , Ωi ),
(8)
where di textually specify its tool function, while Θi and Ωk define its typed arguments and return schema. Table 1 summarizes the access tools designed for analytic and semantic memory. For example, for the analytic tool C OMPUTE, di specifies a deterministic computation over an induced table, Θi includes the target table, computation operator, target column, and optional constraints, and Ωi returns the computed result and matched records. For the semantic tool S EMANTIC M ATCH, dk specifies relevance-based memory retrieval, Θi includes a semantic query content and retrieval budget, and
Table 1: Access tools for analytic and retrieval memory. Memory
Tools
Capability
Analytic
L OOKUP, F ILTER, C OMPUTE, R ANK
Exact record access and deterministic computation. Relevance-based retrieval and event localization.
retrieval
S EMANTIC M ATCH, E VENT L OCATE
Ωi returns the ranked memory units with their relevance scores. 3.5.2 Memory-Aware Joint Query Planning The access tools specify how each memory can be queried. However, users describe desired outcomes rather than access procedures, and a feasible operation also depends on what the current memory contains. To bridge this gap, A DA MM first constructs a query-conditioned planning context that exposes relevant memory structure, and then employs an LLM-based planner to jointly reason over the query and this context, composing an executable procedure for retrieving and analyzing the required information. The resulting evidence is subsequently provided to the answer model for response generation. Planning Context Construction. Valid query planning requires knowledge of the current memory structures and their supported operations. Exposing the complete memory contents, however, introduces irrelevant information and expands the planning space. A DA MM therefore abstracts analytic and retrieval memory into compact metadata, retrieves query-relevant entries, and associates them with their available tools to form the effective planning context. Specifically, we treat each analytic table T ∈ Mana and retrieval episode P ∈ Mret as an individual planning candidate represented by textual metadata µT and µP , respectively. For each table, µT summarizes its name, description, columns, and sampled values, while each episode directly uses its description as µP . Together, these candidates form the metadata search space U. Given a query q, A DA MM ranks each µ ∈ U using hλ (q, µ) = λ cos(E(q), E(µ)) + (1 − λ) TokSim(q, µ),
(9)
where E(·) is a semantic encoder and TokSim measures token-level similarity. After retaining the
top-ranked µ, A DA MM augments their metadata with designed access tool descriptions, forming the planning context Dq to expose relevant memory candidates and available operations to the planner. Progressive Tool-Execution Planning. Complex queries often require multiple dependent operations, with the arguments of later tool calls determined by earlier outputs. Instantiating all tool calls upfront is therefore unreliable. To address this challenge, A DA MM separates high-level planning from progressive tool instantiation. High-level plan generation. Conditioned on the query q and planning context Dq , A DA MM employs an LLM-based planner to generate an abstract plan specifying the information goals and their execution order, without binding them to concrete tools or arguments: L π̄(q, Dq ) = Planner(q, Dq ) = (gℓ , ξℓ ) ℓ=1 , (10) where gℓ is a textual description of the information required at step ℓ, and ξℓ specifies the access tool designated to obtain it. Progressive tool instantiation. Guided by the above high-level plan, A DA MM instantiates tool calls sequentially. At step ℓ, it conditions on the preceding results Z<ℓ = {z1 , . . . , zℓ−1 }: Θℓ = Planner(q, gℓ , ξℓ , Dq , Z<ℓ ), zℓ = Invoke(ξℓ , Θℓ ),
(11)
where ξℓ and Θℓ denote the selected tool and its instantiated arguments. This allows later calls to consume intermediate results (e.g., a timestamp returned by E VENT L OCATE can serve as a temporal constraint for a subsequent C OMPUTE call). Question Answering. Finally, the answer model generates response conditioned on the query and an evidence context comprising the instantiated plan and its tool outputs.
4
Experimental Evaluation
4.1
Experimental Setup
Benchmarks. We evaluate A DA MM on two complex multimodal memory benchmarks. MemEye evaluates visual-memory granularity and reasoning through paired multiple-choice and openended queries (Guo et al., 2026), while MemGallery assesses long-term conversational memory management (Bei et al., 2026). MemEye uses EM, BLEU-1 and LLM-judge for evaluation,
Benchmarks
Sessions
Rounds
Images
QA pairs
A-Mem MemoryOS
MemEye MemGallery
221 240
848 3,962
438 1,003
742 1,711
MemEye
M2A MMA
MIRIX MM-RAG
MemGallery AR
Brand
Social
Table 2: Benchmark statistics.
80
UniversalRAG AdaMM
VS
Card
CD 80
50
60
VR
20
whereas MemGallery reports F1, BLEU-1, and LLM-Judge scores. Dataset statistics are provided in Table 2.
Cartoon
Health
Implementation details. We use GPT-4.1-nano and GPT-5.4-min as answer and memory construction backbones for all methods. Text and image representations are produced by MiniLM-L6-v2 and siglip2-base-patch16-384, respectively. We use Top-10 retrieved memory units for retrieval memory baselines, and our method constrain the planner to generate at most three execution steps under a shared budget of 10 evidence units. Results are averaged over 3 runs. 4.2
Main Results
Overall performance. Table 3 presents a comprehensive comparison of A DA MM with representative unimodal and multimodal memory frameworks across two benchmarks and answer backbones. A DA MM consistently achieves the best result on every metric. With GPT-4.1-nano, it surpasses the strongest competing result on MemEye by 7.3% and 5.8% percentage points for MCQ and open-ended questions, respectively, and improves MemGallery F1, BLEU-1, and LLM-Judge by 4.2%, 4.3%, and 7.0%. The advantage persists with GPT-5.4-mini, yielding gains of 3.2% and 11.3% points on MemEye and 2.6%, 2.2%, and 5.2% on the three MemGallery metrics. The consistant improvements on different settings demonstrate that A DA MM generalizes across answer formats, benchmark settings, and backbone capacities.
KR
TTL Home
Outdoor Multi
Baselines. We compare against unimodal memory agents, including A-Mem and MemoryOS (Xu et al., 2025; Li et al., 2025), which organize longterm textual interaction histories. Multimodal baselines include dedicated memory agents (M2A, MMA, and MIRIX) (Feng et al., 2026; Lu et al., 2026; Wang and Chen, 2025), which construct persistent memories from dialogue and visual observations, and retrieval-based systems (MM-RAG and UniversalRAG), which directly retrieve relevant multimodal evidence for answer generation.
FR
40
TR
MR
Figure 3: LLM-as-Judge performance across finegrained tasks with GPT-5.4-mini.
Fine-grained task analysis. To further examine where the improvement arises, Figure 3 reports the LLM-Judge breakdown using GPT-5.4-mini. On MemEye, the largest gains over the strongest baseline occur on Card Playlog and Personal Health, with improvements of 18.8% and 16.7%, respectively. Both tasks require exact operations over recurring records, such as filtering observations, comparing values, or tracking changes, highlighting the benefit of analytic memory. A DA MM also achieves substantial gains on Multi-scene (11.1%), Brand Memory (9.0%), Outdoor Navigation (8.6%), and Social Chat (8.2%), where relevant evidence must be retrieved and connected across interactions. The smaller gains on Cartoon Entertainment (6.1%) and Home Renovation (5.2%) are consistent with their greater reliance on direct visual-semantic recall. On MemGallery, the largest margins appear in Conflict Detection (10.6%), Knowledge Resolution (10.5%), and Factual Retrieval (8.1%), demonstrating the value of structured access for reconciling and precisely locating information. A DA MM further improves Multi-entity Reasoning, Visual-centric Reasoning, and Visual-centric Search, while remaining comparable to the strongest baselines on Answer Refusal, Temporal Reasoning, and Test-time Learning. Together, these results suggest that combining retrieval with operation-specific analytic access is particularly beneficial when answering requires more than semantic relevance alone. 4.3
Ablation Study
We evaluate four ablated variants on MemEye to examine the contributions of the designed memories and the memory-aware planning mechanism. The Health task emphasizes numerical and tempo-
Backbone
MemEye
Method
Unimodal
A-Mem MemoryOS
Multimodal
M2A MMA MIRIX MM-RAG UniversalRAG A DA MM (Ours)
Unimodal
A-Mem MemoryOS
Multimodal
M2A MMA MIRIX MM-RAG UniversalRAG A DA MM (Ours)
GPT-4.1-nano
GPT-5.4-mini
MemGallery
EM
BLEU-1
LLM-Judge
F1
BLEU-1
LLM-Judge
39.1 42.4
17.1 15.8
32.0 27.6
53.2 53.4
47.2 47.1
67.4 65.9
34.1 39.0 40.9 42.7 43.1 50.4(+7.3)
5.8 10.6 5.7 16.6 16.5 21.2(+4.1)
13.8 32.0 13.6 42.2 40.2 48.0(+5.8)
50.2 55.9 56.3 58.3 57.7 62.6(+4.2)
44.3 50.7 51.1 52.4 52.4 56.7(+4.3)
61.5 64.1 66.3 67.3 67.0 74.3(+7.0)
48.0 48.7
22.4 21.8
35.2 33.3
63.0 64.5
58.6 59.0
74.4 75.5
40.1 53.9 46.7 61.8 62.4 65.5(+3.2)
12.3 25.7 18.0 29.7 30.4 35.9(+5.5)
33.5 43.3 33.2 49.4 48.6 60.7(+11.3)
60.8 64.0 65.7 66.5 65.6 69.1(+2.6)
56.0 59.6 60.9 61.9 60.8 64.1(+2.2)
71.7 73.3 78.2 78.7 77.4 83.9(+5.2)
Table 3: Main results on MemEye and MemGallery. Higher is better for all metrics. bold values denote the best result within a backbone and red values denote improvements over the strongest baseline. Full
w/o Analytic
w/o Retrieval
LLM-Judge Accuracy
Overall
58.9 57.4 56.1
35
48
32.7
48.3
32.1
45.1
29.6
30
43.6
25
42.0
38.4
21.4
20
dation patterns confirm that analytic and retrieval memory provide complementary capabilities.
Brand
42 53.2
52
w/o Progressive
36.3
60.7
60
56
w/o Context
Health
36
Figure 4: Ablation results on MemEye. Overall and task-level LLM-as-Judge accuracy using GPT-5.4-mini.
ral analysis over multiple personal-health records, whereas the Brand task primarily evaluates visually grounded recall of brand-related information. In w/o Analytic and w/o Retrieval, we disable the analytic memory and retrieval memory, respectively. In w/o Planning Context, the planner receives the entire memory metadata. In w/o Progressive Execution, all tool calls and their arguments are instantiated before execution. Contributions of Complementary Memories. As shown in Figure 4, removing either memory component consistently degrades performance. Without analytic memory, overall accuracy decreases by 4.6%, with a substantially larger 14.9% drop on Health. This result highlights the importance of organizing recurring records into executable structures for analytical queries. Removing retrieval memory instead causes a 7.5% overall drop and is particularly detrimental to Brand, where performance decreases by 9.9%. The distinct degra-
Contributions of Memory-Aware Planning. Removing the planning context reduces performance by 3.3% overall and by 6.7% and 6.3% on Health and Brand, respectively. This demonstrates that effective tool selection requires knowledge of both the query intent and the memory structures currently available. Instantiating all calls before execution also produces consistent degradation, including drops of 4.2% on Health and 3.2% on Brand. These results indicate that progressive execution is useful when later operations depend on information obtained from preceding calls.
5
Conclusion
In this work, we identify a retrieval–analysis mismatch in long-term multimodal agent memory, highlighting the need for executable operations over accumulated experience beyond retrieval. To address this issue, we introduced A DA MM, which couples hierarchical retrieval memory with schemainduced analytic memory and exposes their distinct capabilities through operation-specific tools. A memory-aware planner further grounds tool selection in the current memory state and progressively composes retrieval and analytic operations. Experiments on MemEye and MemGallery across two answer backbones demonstrate consistent improvements from combining flexible recall with executable analysis.
Limitations While A DA MM demonstrates promising results in long-term multimodal agent memory, two limitations remain. First, analytic memory depends on record fragments extracted from multimodal interactions. Incorrect or missing fields may propagate to schema induction, table construction, and downstream computations. Future work could incorporate confidence-aware extraction and cross-round consistency checks to improve robustness. Second, A DA MM uses a predefined set of access tools and thus requires manual extension for unseen, domainspecific operations. Future work could develop a self-evolving tool framework that identifies emerging capability gaps and safely synthesizes, validates, and integrates new tools.
References Rakesh Agrawal and Ramakrishnan Srikant. 1994. Fast algorithms for mining association rules in large databases. In Proceedings of the 20th International Conference on Very Large Data Bases, pages 487– 499. Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, and Hanghang Tong. 2026. Mem-gallery: Benchmarking multimodal long-term conversational memory for mllm agents. arXiv preprint arXiv:2601.03515. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, and 3 others. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. Junyu Feng, Binxiao Xu, Jiayi Chen, Mengyu Dai, Cenyang Wu, Haodong Li, Bohan Zeng, Yunliu Xie, Hao Liang, Ming Lu, and Wentao Zhang. 2026. M2a: Multimodal memory agent with dual-layer hybrid memory for long-term personalized interactions. arXiv preprint arXiv:2602.07624. Minghao Guo, Qingyue Jiao, Zeru Shi, Yihao Quan, Boxuan Zhang, Danrui Li, Liwei Che, Wujiang Xu, Shilong Liu, Zirui Liu, Mubbasir Kapadia, Vladimir Pavlovic, Jiang Liu, Mengdi Wang, Yiyu Shi, Dimitris N. Metaxas, and Ruixiang Tang. 2026. Memeye: A visual-centric evaluation framework
for multimodal agent memory. arXiv:2605.15128.
arXiv preprint
Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos-Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. 2023. Memory matters: The need to improve long-term memory in llm-agents. In Proceedings of the AAAI Symposium Series, volume 2, pages 277–280. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-toend web agent with large multimodal models. arXiv preprint arXiv:2401.13919. Zhengjun Huang, Zhoujin Tian, Qintian Guo, Fangyuan Zhang, Yingli Zhou, Di Jiang, Zeying Xie, and Xiaofang Zhou. 2025. Licomemory: Lightweight and cognitive agentic memory for efficient long-term reasoning. arXiv preprint arXiv:2511.01448. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649. Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459. Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, and 3 others. 2025. Memos: An operating system for memoryaugmented generation (MAG) in large language models. CoRR, abs/2505.22101. Jiaqi Liu, Zipeng Ling, Shi Qiu, Yanqing Liu, Siwei Han, Peng Xia, Haoqin Tu, Zeyu Zheng, Cihang Xie, Charles Fleming, Mingyu Ding, and Huaxiu Yao. 2026. Omni-simplemem: Autoresearch-guided discovery of lifelong multimodal agent memory. arXiv preprint arXiv:2604.01007. Yihao Lu, Wanru Cheng, Zeyu Zhang, and Hao Tang. 2026. Mma: Multimodal memory agent. arXiv preprint arXiv:2602.16493. Viet Nguyen, Thao Nguyen, Vishal M. Patel, and Yuheng Li. 2026. Personal visual memory from explicit and implicit evidence. arXiv preprint arXiv:2605.28806. Edward R. Omiecinski. 2003. Alternative interest measures for mining associations in databases. IEEE Transactions on Knowledge and Data Engineering, 15(1):57–69.
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. Memgpt: Towards llms as operating systems. arXiv preprint arXiv:2310.08560. Alex Petrov, Alexander Gusak, Denis Mukha, and Dima Korolev. 2026. From unstructured recall to schema-grounded memory: Reliable AI memory via iterative, schema-aware extraction. arXiv preprint arXiv:2604.27906. Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep: A temporal knowledge graph architecture for agent memory, 2025. URL https://arxiv. org/abs/2501.13956. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, and Furong Huang. 2024. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529. Yu Wang and Xi Chen. 2025. Mirix: Multi-agent memory system for llm-based agents. arXiv preprint arXiv:2507.07957. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972. Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie Liang, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Juwei Yue, Chuanrui Hu, Jiawei Sheng, Zuyi Zhou, Wenyuan Zhang, Tingwen Liu, Li Guo, and Yafeng Deng. 2026. Hypermem: Hypergraph memory for long-term conversations. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35237– 35254.
Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2025. A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems, 43(6):1–47. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In AAAI, pages 19724–19731. AAAI Press.