arXiv CSopen access

VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions

VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions

Yuxin Chen1,2 , Yi Zhang2,3 , Zhengzhou Cai2,4 , Yaorui Shi2,3 , Zhiyuan Yao2,5 , Chenhang Cui1,2 , Jingnan Zheng1,2 , Yaqi Huo2 , Xi Su2 , Qi Gu2,† , Xunliang Cai2 , Xiang Wang3 , An Zhang3,† , Tat-Seng Chua1

arXiv:2605.27141v1 [cs.AI] 26 May 2026

1 National University of Singapore, 2 Meituan, University of Science and Technology of China, 4 Beijing University of Posts and Telecommunications, 5 Zhejiang University † Corresponding authors: [email protected], [email protected] 3

Abstract Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements. Code is available at https://github.com/meituan-longcat/vitabench-2.0.

1

Introduction

Recent advances in large language models (LLMs) have improved their capabilities in reasoning and tool use [1–4], enabling them to evolve from passive text generators into interactive agents operating in real-world environments [5–7]. As these agents move from single-turn interactions to sustained collaboration with users, effective assistance increasingly depends on understanding user intent beyond what is explicitly stated [8]. In real-life scenarios, such intent is often reflected implicitly through fragmented interactions [9–11], making personalization central to user–agent collaboration. However, this growing need for personalization in human–agent collaboration remains insufficiently captured by existing agent benchmarks. Existing benchmarks primarily focus on evaluating multi-step Preprint.

reasoning and tool orchestration, where tasks are well-specified and the context required for successful completion is clearly stated within the context [12–17]. As a result, they mainly evaluate agents’ ability to follow explicit instructions and execute correct action sequences. In contrast, emerging realworld agent systems increasingly operate in settings where user intent is under-specified and must be inferred from prior interactions [18]. In such scenarios, effective assistance requires agents to maintain a consistent representation of user preferences, adapt to their evolution over time, and proactively acquire missing information when necessary. This shift introduces a fundamentally different source of complexity, moving beyond reasoning over explicit instructions to decision-making grounded in implicit and evolving user preferences. This gap highlights the need for agent benchmark that explicitly evaluates personalization and proactiveness in realistic user-agent interaction settings. Toward this end, we introduce VitaBench 2.0, an agent benchmark for evaluating personalized and proactive behavior in real-world long-term user interactions. Beyond tool use and reasoning ability, VitaBench 2.0 also evaluates personalization along three dimensions: (1) preference extraction, where agents infer implicit preferences from fragmented interactions; (2) preference utilization, where agents leverage these preferences for user-specific decision-making; and (3) preference updating, where agents capture preference drift and revise their understanding as user behavior evolves. Building on this formulation, we further evaluate proactiveness, which arises when user preference is conditional and requires agents to actively acquire missing information before making decisions. Following the general setup of existing agent benchmarks [15–17], VitaBench 2.0 is constructed as an interactive agent benchmark, where agents interact with environments to fulfill user needs. Tasks in VitaBench 2.0 are organized as temporally ordered sequences for individual users, where each task sequence spans multiple domains, and each task is paired with a dedicated set of tools and an executable environment to support realistic interaction. To evaluate personalization, we curate a series of fine-grained preferences for each user and embed them into fragmented interactions, including both dialogues and behaviors. As agents continuously interact with users over time, user preferences may evolve, which is reflected in newly observed interactions, requiring agents to maintain and update a consistent representation of preferences within task sequences. To capture long-term user dynamics in realistic interaction settings, we allow agents to maintain a memory module for each user. Building on this, VitaBench 2.0 provides an extensible memory interface that supports flexible implementations and enables controlled comparison across representative memory mechanisms [19–21]. We conduct extensive evaluations on a wide range of frontier proprietary and open-source language models. Our results show that real-world personalization tasks remain highly challenging for current agents, revealing a substantial gap between existing capabilities and practical requirements. We further analyze the role of memory and find that, while memory mechanisms are essential for longterm user modeling, existing approaches often fail to consistently translate stored information into improved performance, and different memory designs lead to markedly different outcomes. Through systematic analysis, we identify key failure patterns and primary bottlenecks of current agents, providing insights into why current models struggle with personalization. VitaBench 2.0 highlights a gap between current LLM agents and realistic personalized assistants and provides a testbed for future research on memory, personalization, and proactive agent behavior.

2

Related Work

Personalized LLM. As large language models are increasingly deployed in user-facing applications, personalization has become a critical capability for aligning model outputs with individual user needs and preferences [8, 18, 22]. Achieving personalization requires models to capture userspecific characteristics and incorporate them into the generation process. Existing methods can be broadly understood from three alignment perspectives: input-level alignment, model-level alignment, and objective-level alignment. Input-level alignment enriches prompts with user-specific context. Retrieval-augmented methods obtain such context from interaction histories or external knowledge stores [23–25], while profile-based approaches explicitly summarize and inject user preferences into the prompt [26, 27]. Model-level alignment adapts the model itself to generate outputs conditioned on user preferences, through parameter adaptation for white-box models [28, 29] or model factorization frameworks for black-box models [30]. Objective-level alignment incorporates personalization into training objectives, including personalized reward modeling [31], multi-objective preference optimization [32], and causal preference modeling [33]. As user interactions become increasingly long-term and informative, memory-augmented personalization has gained growing attention, supported by 2

advances in memory systems and the increasing capability of LLMs to utilize them. This line of work augments LLMs with external memory mechanisms that support the storage, retrieval, and updating of user-relevant information over time [34, 19, 35]. Benchmarks for LLM Personalization. As personalized LLMs become increasingly complex, there is a growing need for systematic evaluation benchmarks. Existing work can be broadly categorized along two dimensions: the form of user-specific information and the evaluation setting. From the input perspective, prior benchmarks assess personalization using various forms of user information, including explicit profiles [36–38], user-authored documents [39, 40], and interaction histories with implicit or evolving preferences [41–43, 10, 9, 44]. From the evaluation perspective, early benchmarks mainly consider relatively static personalization scenarios, where user information is explicitly provided or derived from a fixed set of documents or profile attributes [39, 40, 36– 38]. More recent efforts place greater emphasis on long-term memory and dynamic user modeling, evaluating whether models can retain user-related information across extended interactions, infer implicit preference signals from conversational histories, and adapt to preferences that evolve over time [41–43, 10, 9, 44]. However, these benchmarks remain largely confined to passive text-intext-out settings, where personalization is evaluated primarily through generation rather than action, leaving a gap toward realistic assistant scenarios involving tool use and decision-making. Benchmarks for LLM Agents. LLMs have evolved from text generators into autonomous agents capable of interacting with external tools and environments [45]. Existing agent benchmarks have progressed from evaluating isolated tool-use capability to assessing increasingly realistic forms of interactive task execution. Early benchmarks mainly focus on API invocation and tool-use accuracy, evaluating whether models can select appropriate tools and generate valid arguments for a given user request [46, 47]. Subsequent benchmarks move toward more interactive and stateful settings, where agents must reason over multiple turns, track intermediate states, and respond to evolving context and feedback during execution [48–52]. More recent efforts further place agents in realistic execution environments, including web searching [13, 14], computer using [53], software engineering [12], and user-agent interaction [54, 15–17, 55–57], to evaluate end-to-end task completion under real-world constraints. However, existing agent benchmarks largely overlook personalization and typically assume that all task-relevant information is explicitly available in the current context, creating a gap with real-world assistant scenarios. Our work addresses this gap by jointly evaluating personalization and agentic execution in realistic interactive settings.

3

VitaBench 2.0

VitaBench 2.0 is designed to simulate long-term, user-agent collaboration scenarios for personalization and proactiveness evaluation, where agents are required to continuously satisfy user needs. Figure 1 provides an overview. Each user u is associated with a profile Pu , evolving preferences Nu = (n1 , n2 , . . . , nL ), and a temporal task sequence Tu = (t1 , t2 , . . . , tN ), designed to evaluate the agent’s ability to infer, maintain, and leverage user preferences over time. Between tasks ti−1 and ti , the agent is exposed to newly introduced interaction histories that reflect emerging preferences or preference drift, and enabled to maintain a memory module M to store user information and support future decisions. We describe the task formulation and the key modules in benchmark below. We also provide a detailed analysis of our curated user profile and preferences in Appendix C.2 3.1

Task Set

Tasks in VitaBench 2.0 are organized as temporally ordered sequences for individual users, where each sequence spans multiple domains. Each individual task ti is an agentic task in which the agent interacts with domain-specific tools and an executable environment to fulfill a user request. Concretely, each task can be modeled as a partially observable Markov decision process (POMDP): Pi = (Si , Ai , Oi , Ti , ri ),

(1)

where Si denotes the environment state, Ai the action space, Oi the observation space, Ti : Si ×Ai → Si the state transition function, and ri the task reward or evaluation function. We design task complexity to arise from both tool-use and personalized user understanding, requiring agents to reason over explicit constraints from the user query and implicit signals derived from 3

User Information User Profile

Preferences

Interaction History

Embed

Interaction History

Personalized Memory

Agentic Task

Past Interactions

Ordered: a mid-range hotel near the airport

Infer

Searched: indoor swimming pools nearby, good books to read at home Dialog:

Order some takeout for me.

Flight: evening departure, China Eastern Airline

I noticed you’ve recently been focusing on low-fat eating. I’ll prioritize light, healthy options.

Hotel: near airport, mid-range price

Can you find a spicy place for dinner tonight?

Tool Calling: Search(restaurants)

Leisure: feeling good —> swimming feeling down —> reading

Sure! I’ll look for nearby Sichuan or hotpot options.

I ‘ve been going to the gym lately and want to reduce my body fat

Update

I want to relax tonight. Any Suggestion? I remember that when you’re feeling good you prefer swimming; otherwise, you prefer reading at home.

Memory Interface

How are you feeling today ?

Yes, especially high-protein, low-fat meals after training

Agent retrieves a conditional preference and proactively asks for clarification

Searched: weight-loss diet, high protein snacks, low-fat meal

Delivery

Agent retrieves Updated preference from Memory

Proactive Personalization

Food: light, low-fat healthy

Food: Spicy, Salty Sichuan

Recommend: Light Salad

Retrieve

Preference Drift

New Interactions

Got it ! Should I prioritize healthier meals and workoutrelated suggestions

Personalization

Food： light，low-fat， healthy （Preference Shift）

Browed: evening flights to Qingdao on China Eastern

Task Sequence

Context

In-store Consumption

Summary

VS

RAG

Online Travel Agency

66 Tools

56 Users

2000+ Preferences

Figure 1: Overview of VitaBench 2.0. The agents are required to operate over temporal task sequences for each user, infer evolving user preferences from fragmented interactions, maintain these preferences via a memory mechanism, and make personalized and proactive decisions. fragmented user interactions. Specifically, a task instance is specified as: ti = (qi , Fi , Ei , Gi , Hi ),

(2)

where qi is the user query, Fi is the set of available tools, Ei is the executable environment with underlying states, Gi is a set of evaluation rubrics, and Hi denotes the interaction histories exposed to agent between tasks ti−1 and ti , simulating fragmented user interactions over time. Successful task execution requires the agent to identify user intent from qi , select appropriate tools, and infer relevant user preferences from H1:i to make consistent and personalized decisions. Before solving task ti , the agent is allow to updates its memory if enabled based on Hi : Mi = U PDATE(Mi−1 , Hi ).

(3)

At each step t within task ti , the agent receives an observation ot ∈ Oi consisting of the user query, dialogue history, and environment feedback from previous actions. The agent then selects an action conditioned on the current observation and updated memory state: at ∼ π(at | ot , Mi ),

(4)

Ai = Atool ∪ Adialogue ,

(5)

where the action space is given by where Atool denotes tool invocations and Adialogue denotes natural-language responses to the user simulator. After executing at , the environment transitions to a new state st+1 and returns a new observation ot+1 . The agent iterates between tool use and user interaction until the task is completed or a maximum number of steps is reached, producing a trajectory: τi = (o0 , a0 , o1 , a1 , . . . , oT , aT ).

(6)

The task accuracy is evaluated at both the trajectory level and outcome level by applying an evaluator LLM to τi and aT using the rubric set Gi , which decomposes task success into a set of atomic criteria. Inheriting from VitaBench [17], we construct VitaBench 2.0 through systematic abstraction of real-world life-serving scenarios across three domains—Delivery, In-store Consumption, and Online Travel Agency—with a total of 66 tools. Detailed descriptions of the task pipeline and environment construction are provided in Appendix A.3 and Appendix A.2. 4

3.2

Key Module

VitaBench 2.0 evaluates personalization by requiring agents to infer user preferences from fragmented historical interactions and leverage these preferences to collaborate with users. To support this evaluation, we carefully curate 56 users with more than 2,000 fine-grained preferences, covering diverse preference types and interaction contexts. The construction of user profiles and preference distributions is data-driven, drawing inspiration from real-world user scenarios to better reflect realistic preference diversity and behavioral heterogeneity. To reflect realistic long-term interaction scenarios, we allow the agent to maintain an external memory module that stores and updates userspecific information over time. We next describe the construction of user profiles, user preferences, interaction histories, and the memory interface in detail. User Profiles. Each user u is associated with a manually curated profile Pu , constructed in a data-driven manner to reflect realistic user characteristics. To ensure both diversity and realism in the user population, we model users along multiple dimensions, including demographics, geographic and socioeconomic attributes, occupation, and social context, with distributions aligned to real-world scenario statistics. A comprehensive analysis of the curated profiles is provided in Appendix C.2.1. User Preferences. Each user u is also associated with a set of preferences Nu = {n1 , n2 , . . . , nL }, spanning multiple aspects of daily life (e.g., dining, leisure and entertainment, shopping, travel, hobbies, and lifestyle habits). Preferences are expressed as natural language statements grounded in the user profile (e.g., “avoids spicy food due to a stomach condition”). User preferences in real life are inherently dynamic. To simulate realistic evolution, we introduce temporally grounded preference drift events throughout each user’s task sequence. Between selected consecutive tasks, a subset of preferences may undergo one of three changes: (1) addition, where a new preference emerges; (2) deletion, where an existing preference becomes inactive; and (3) modification, where an existing preference shifts. In total, we manually curate 56 users with over 2,000 unique preferences. Detailed descriptions and illustrative examples are provided in Appendix A.1. A comprehensive analysis of the curated preference is provided in Appendix C.2.2. Interaction History. User preferences are not explicitly provided to the agent, but are instead encoded in fragmented interaction histories accumulated over time. As the agent progresses from task ti−1 to ti , it is exposed to newly introduced interaction histories Hi , which may reflect changes in the user’s underlying preferences. Inspired by information accessibility in real-world scenarios, Hi contains two types of records: (1) dialogues, consisting of multi-turn user–agent conversations; and (2) behaviors, consisting of user behavior logs such as browsing, ordering, reviewing, and searching histories. Among these, not all interactions are preference-relevant. Instead, Hi can be viewed as comprising both signal interactions that reflect the user’s underlying preferences and noise interactions that are irrelevant, ambiguous, or contextually misleading. This requires agents to distinguish consistent user preferences from irrelevant actions. Detailed construction of interaction history and illustrative examples are provided in Appendix A.1. Memory Module. To capture long-term user dynamics across temporal task sequences, we allow agents to maintain an external memory module M for each user as a persistent representation of user-specific information. When memory is enabled, the agent interacts only with the memory module and does not have direct access to the full interaction histories. Formally, before executing each task ti , the agent is exposed to any newly available interaction history Hi and updates its memory: Mi = U PDATE(Mi−1 , Hi ).

(7)

During task execution, the agent conditions its actions on both the current observation and memory: at ∼ π(at | R ETRIEVE(Mi , qi ), ot ),

(8)

where R ETRIEVE(Mi , qi ) returns task-relevant information from memory. To systematically study the role of memory in personalization, VitaBench 2.0 defines an extensible memory interface through two operations—U PDATE and R ETRIEVE—allowing different memory architectures to be plugged in. Also, we implement two representative memory mechanisms: • Agentic Memory. The agent maintains a structured representation of user information and actively controls the memory content by deciding what information to retain, update, or discard. The 5

Table 1: Performance of non-thinking and thinking models under different memory settings. The leaderboard is sorted by the Avg@4 on Full Context setting. The best performance is in bold. Full Context

Models

Avg@4

Pass@4

Agentic Memory Passˆ4

Avg@4

RAG Memory

Pass@4

Passˆ4

Avg@4

Pass@4

Passˆ4

0.229 0.467 0.537 0.569 0.576 0.623 0.674 0.664 0.665 0.658

0.008 0.056 0.105 0.112 0.129 0.084 0.145 0.182 0.198 0.207

0.094 0.205 0.290 0.316 0.351 0.317 0.383 0.383 0.406 0.424

0.227 0.409 0.471 0.523 0.543 0.555 0.621 0.585 0.625 0.618

0.011 0.059 0.136 0.152 0.174 0.123 0.163 0.200 0.208 0.247

0.533 0.567 0.599 0.508 0.638 0.609 0.625 0.596 0.646 0.556 0.712 0.669 0.642 0.647 0.656 0.650 0.645

0.073 0.098 0.091 0.088 0.138 0.124 0.107 0.106 0.123 0.150 0.118 0.154 0.178 0.204 0.255 0.225 0.259

0.261 0.309 0.315 0.303 0.320 0.314 0.336 0.336 0.375 0.328 0.390 0.362 0.374 0.410 0.430 0.339 0.430

0.452 0.544 0.519 0.511 0.579 0.518 0.574 0.555 0.591 0.485 0.643 0.587 0.573 0.591 0.584 0.496 0.566

0.091 0.107 0.134 0.118 0.109 0.143 0.135 0.147 0.179 0.185 0.153 0.158 0.186 0.236 0.271 0.205 0.299

Non-thinking Models GPT-4o-mini (w/o thinking) GPT-3.5-Turbo (w/o thinking) LongCat-Flash-Chat (w/o thinking) GLM-4.5 (w/o thinking) Doubao-Seed-1.6 (w/o thinking) GLM-4.6 (w/o thinking) Kimi-K2.6 (w/o thinking) GLM-5.1 (w/o thinking) Doubao-Seed-2.0-pro (w/o thinking) DeepSeek-V4-Pro (w/o thinking)

0.067 0.140 0.298 0.307 0.326 0.342 0.378 0.420 0.428 0.456

0.180 0.314 0.510 0.529 0.512 0.612 0.632 0.654 0.649 0.652

o4-mini (w/ thinking) Gemini-2.5-Flash (w/ thinking) Qwen3-Max (w/ thinking) Kimi-K2.6 (w/ thinking) Gemini-2.5-Pro (w/ thinking) MiniMax-M2.7 (w/ thinking) GLM-4.6 (w/ thinking) GLM-4.5 (w/ thinking) Doubao-Seed-1.6 (w/ thinking) GLM-5.1 (w/ thinking) DeepSeek-R1-0528 (w/ thinking) o3 (w/ thinking) Claude-4.5-Sonnet (w/ thinking) GPT-5 (w/ thinking) DeepSeek-V4-Pro (w/ thinking) Doubao-Seed-2.0-pro (w/ thinking) Claude-Opus-4.6 (w/ thinking)

0.210 0.282 0.284 0.293 0.331 0.345 0.359 0.364 0.373 0.394 0.396 0.403 0.417 0.441 0.472 0.474 0.503

0.433 0.556 0.499 0.533 0.605 0.584 0.612 0.623 0.599 0.587 0.691 0.653 0.658 0.658 0.649 0.683 0.664

0.006 0.019 0.123 0.127 0.171 0.113 0.147 0.204 0.218 0.267

0.084 0.231 0.302 0.330 0.340 0.336 0.397 0.423 0.426 0.427

Thinking Models 0.047 0.063 0.105 0.099 0.109 0.145 0.116 0.156 0.176 0.213 0.131 0.169 0.197 0.226 0.295 0.270 0.337

0.270 0.312 0.324 0.280 0.378 0.351 0.351 0.311 0.383 0.352 0.412 0.401 0.397 0.421 0.449 0.428 0.454

memory is incrementally updated with each new history batch, and R ETRIEVE returns all or a selective memory representation. This design requires the agent to perform selective abstraction, resolve conflicts across observations, and maintain long-term consistency. • RAG Memory. Interaction records are stored in a memory bank with vector embeddings. U PDATE indexes new records, and R ETRIEVE performs similarity-based retrieval given the task query. This design follows a fixed pipeline, where memory access is determined by retrieval without explicit control over what information is retained or discarded. We provide a detailed discussion of memory mechanisms for agent systems in Appendix B.1.

3.3

Proactiveness

Beyond leveraging stored user preferences, an effective personalized agent should also know when its current knowledge is insufficient and proactively seek user clarification or conduct environment exploration. We evaluate this capability through proactive tasks, where successful task completion depends not only on retrieving the relevant user preference, but also on recognizing missing contextual information that cannot be inferred from memory or the current query alone. Building on this idea, proactive tasks are constructed around missing but necessary information, where the correct action depends on contextual factors that are not directly observable to the agent. Solving such tasks requires the agent to capture the relevant conditional preference, recognize the unresolved ambiguity, and query the user or explore the environment before acting. These tasks are interleaved with standard personalization tasks, requiring the agent to adaptively decide when additional context is necessary rather than making decisions under incomplete information. 6

Figure 3: Average performance across tasks at each temporal task index.

Figure 2: Average performance versus number of turns across models under full-context setting.

4

Experiment

4.1

Experimental Setups

Models. We evaluate a diverse set of state-of-the-art proprietary and open LLMs, covering both non-thinking and thinking configurations when available. The evaluated models include OpenAI family, including GPT-3.5-Turbo, GPT-4o-mini, GPT-5, and o-series models such as o3 and o4mini [58, 3, 59–61]; the DeepSeek family, including DeepSeek-R1 and DeepSeek-V4 variants [2, 62]; Anthropic’s Claude series, including Claude Sonnet and Claude Opus variants [63, 4, 64]; Google’s Gemini series, including Gemini-2.5-Flash and Gemini-2.5-Pro [65–67]; Qwen3-Max [7]; GLM variants, including GLM-4.5, GLM-4.6, and GLM-5.1 [68–70]; ByteDance Seed series, including Seed-1.6 and Seed-2.0-Pro [71, 72]; Kimi-K2.6 [73]; LongCat-Flash [74]; and MiniMax-M2.7 [75]. To ensure fair comparison, we distinguish between reasoning-enhanced (thinking) and non-reasoning (non-thinking) models. For hybrid architectures that support both modes, we evaluate the think-on and think-off configurations separately. We exclude smaller models due to the difficulty of the benchmark. The leaderboard is accordingly divided into thinking and non-thinking categories. Implementations. All agents are implemented as function-calling agents based on the OpenAI tool schema. Interactions proceed without a predefined step limit and terminate either when the agent emits the token “###STOP###” or upon failure. We use gpt-4.1-2025-04-14 as the user simulator and evaluator. Each task is run four times with a temperature of 0.0 for deterministic evaluation. Prompt templates for all components are detailed in Appendix A.3. For memory mechanisms, we compare MemAgent [21] as agentic memory and the traditional RAG system as agent RAG memory. For detailed implementation and configuration settings, please refer to Appendix C.2.4. Metrics. We report Avg@4, Pass@4, and Passˆ4, computed from four independent runs and averaged over all tasks. Avg@4 measures the mean performance across the four runs. Pass@4 denotes the probability that at least one of the 4 i.i.d. trials successfully completes the task. Passˆ4 represents the probability that all 4 i.i.d. trials are successful. 4.2

Main Results

Table 1 presents evaluation results on VitaBench 2.0. We have the following observations. Real-world personalization tasks remain highly challenging for current agents. Even under the Full Context setting where full interaction history is accessible, state-of-the-art models achieve only Avg@4 of around 0.5 and Passˆ4 of around 0.3. This indicates that current agents struggle to reliably infer and utilize user preferences, despite already simplified task settings where tool complexity and instruction difficulty are intentionally controlled. Compared to traditional reasoning-intensive domains such as coding or mathematics, improvements from stronger base models are noticeably less pronounced, suggesting that personalization has emerged as a new bottleneck. As LLM agents are increasingly deployed in real-world user-facing applications, this gap highlights a fundamental limitation in their ability to support personalized decision-making. 7

Figure 4: Analysis of model behavior on VitaBench 2.0. Left: average performance on proactive tasks across model series. Right: performance on VitaBench 2.0 given ground-truth user preferences.

Figure 5: Failure pattern statistics for DeepSeek-V4-Pro and DeepSeek-R1. Category A denotes tool-related errors, category B denotes preference-related errors. Memory mechanisms play a critical but under-explored role. In realistic scenarios, interactions are often long-term and fragmented across sessions, making memory mechanisms essential for maintaining user representations. However, we observe that most models experience performance degradation when relying on memory, compared to the Full Context setting. This trend holds for both agentic memory (where the model decides what to store and retrieve) and pipeline-based RAG memory. These results suggest that current agents are not yet capable of effectively utilizing memory, and that memory design remains a key challenge for improving long-term personalization. Reasoning improvements do not directly translate to personalization gains. Unlike tasks that primarily depend on multi-step reasoning, enabling “thinking” modes does not consistently lead to better performance on VitaBench 2.0. While some models benefit from reasoning enhancements, the overall gains are modest and inconsistent across settings. As illustrated in Figure 2, enabling thinking mode does not consistently yield higher effectiveness (Avg@4) nor improved efficiency (Number of Turns). This suggests that personalization requires capabilities beyond general reasoning, including robust preference extraction, long-term consistency, and the ability to handle noisy and incomplete observations. Consequently, advances in reasoning alone are insufficient to address the challenges of real-world personalized decision-making. 4.3

Analysis & Discussion

Accumulated long-term user interactions pose a fundamental challenge for context handling and memory management. In VitaBench 2.0, tasks are organized as temporally ordered sequences for each user, spanning multiple domains. As the sequence progresses, interactions accumulate, leading to increasingly long and complex contexts for later tasks. To study this effect, we analyze the average performance at each temporal index across all users, reporting mean Avg@4 over all evaluated models. We report the task index of up to 10, as users have at least 10 tasks in their task sequences. As shown in Figure 3, performance generally degrades with increasing task index in 8

both settings. Under full-context, this indicates a limitation in handling long interaction histories, where agents struggle to extract relevant signals from long context. Under memory-based settings, the degradation is further amplified by imperfect memory management: repeated U PDATE and R ETRIEVE operations introduce information loss and error accumulation, causing early inaccuracies to propagate to later tasks. These results highlight that both long-context reasoning and effective memory utilization remain key bottlenecks for current agent systems. Current agents struggle to recognize missing information and engage in proactive interactions with users. We evaluate proactive capabilities by measuring performance on tasks that require agents to actively identify missing information and query users or explore the environment before making decisions. As shown in Figure 4 (left), proactive performance is consistently lower than personalization performance across all model families. For example, while Claude achieves an average personalization score of 46.0, its proactive score drops to 27.4; similar gaps are observed for DeepSeek (44.1 vs. 27.8) and GLM (36.4 vs. 19.3). These results indicate that current agents often fail to recognize when their knowledge is insufficient, and instead proceed with incomplete information rather than initiating clarification. This limitation suggests that proactive interaction remains underdeveloped in current agent systems. Even given ground-truth user preferences, effectively leveraging them remains challenging for current agents. To isolate the difficulty of preference utilization, we provide models with ground-truth user preferences for each task and evaluate their performance. As shown in Figure 4 (right), performance improves compared to the standard setting but remains far from optimal. For instance, DeepSeek and Claude achieve 52.7 and 51.2 under ground-truth preferences, while most other model families remain below 50 (e.g., Seed 49.3, Kimi 43.8, MiniMax 42.6). These results suggest that the challenge of personalization is not solely due to preference extraction and maintain, but also arises from difficulties in reasoning over, prioritizing, and consistently applying preference information during decision-making. Even when provided with accurate user profiles, current agents often fail to translate this information into effective actions. Failure pattern analysis shows that personalization emerges as the primary bottleneck in agent performance. We analyze failure patterns of agents on VitaBench 2.0. Specifically, we use Claude-Opus-4.6 as an external analyzer to examine the full trajectories of DeepSeek family models, including DeepSeek-V4-Pro and DeepSeek-R1, and categorize their errors into fine-grained types. The results are summarized in Figure 5. We observe that the majority of errors in VitaBench 2.0 stem from failures in capturing and utilizing user preferences. In many cases, even state-of-the-art agents fail to correctly infer user preferences from historical interactions, or neglect them during decision-making. For example, agents often default to selecting high-rated or popular items, instead of aligning with user-specific preferences inferred from prior behavior. Moreover, we observe a shift in failure patterns across model generations. Earlier models tend to suffer more from tool-related errors due to limitations in their base capabilities, whereas stronger models exhibit fewer tool failures but still struggle with personalization, making it the dominant bottleneck. This suggests that, as foundational reasoning and tool-use abilities improve, personalization becomes the next critical challenge for advancing agent performance.

5

Limitation

While VitaBench 2.0 provides a principled benchmark for studying personalization and proactiveness in LLM agents, it has several limitations. First, user preferences and interaction histories are programmatically constructed to allow precise control over preference dynamics and task difficulty. While this facilitates reproducible evaluation, it may not fully capture the full diversity of real-world user behavior. Second, the memory interface abstracts memory into update and retrieval operations, enabling controlled comparison across different designs. This abstraction focuses on isolating the role of memory, and does not aim to cover all possible end-to-end architectures. Third, evaluation is based on rubric-driven assessment over task trajectories, providing structured and interpretable signals. More open-ended measures of user satisfaction are beyond the scope of this work. Overall, these design choices are intended to prioritize controllability and comparability, and we view VitaBench 2.0 as a complementary testbed for studying core challenges in personalized and proactive agents. 9

6

Conclusion

In this work, we introduce VitaBench 2.0, a benchmark for evaluating personalization and proactiveness in LLM-based agents. VitaBench 2.0 organizes tasks as user-centric sequences, embeds evolving preferences into fragmented interaction histories, and incrementally exposes these histories to the agent, capturing the key challenge of inferring and updating user preferences over time. To support systematic analysis, we design an extensible memory interface that enables controlled comparison of different memory mechanisms within a unified framework. Through extensive experiments on a diverse set of frontier models, we find that current LLM agents struggle to reliably infer, utilize, and update user preferences, especially when preferences evolve or when information is incomplete. Further analysis provides insights into the failure modes of current agents and the difficulties of long-term preference modeling. Overall, VitaBench 2.0 reveals a significant gap between existing LLM agents and realistic personalized assistants, and provides a testbed for advancing research on memory, personalization, and proactive behavior.

References [1] DeepSeekAI. Deepseek-v3.1 model card. deepseek-ai/DeepSeek-V3.1.

2025.

URL https://huggingface.co/

[2] Daya Guo et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [3] OpenAI. Introducing gpt-5. introducing-gpt-5/.

2025.

URL https://openai.com/index/

[4] Anthropic. Claude sonnet 4.5 model card. 2025. URL https://www.anthropic.com/ news/claude-sonnet-4-5. [5] Meituan LongCat Team. Longcat-flash-thinking-2601 technical report. CoRR, abs/2601.16725, 2026. [6] Aixin Liu et al. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. [7] Qwen Team. qwen3-max.

Qwen3-max model card.

2025.

URL https://qwen.ai/blog?id=

[8] Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Wenhao Yu, Jieming Zhu, Minda Hu, Menglin Yang, Tat-Seng Chua, and Irwin King. A survey of personalized large language models: Progress and future directions. arXiv preprint arXiv:2502.11528, 2025. [9] Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, et al. PersonaMem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory. arXiv preprint arXiv:2512.06688, 2025. [10] Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J Taylor, and Dan Roth. Know me, respond to me: Benchmarking LLMs for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504.14225, 2025. [11] Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, et al. Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation. arXiv preprint arXiv:2604.08455, 2026. [12] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. [13] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, et al. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. 10

[14] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024. [15] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. [16] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. [17] Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, et al. VitaBench: Benchmarking LLM agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490, 2025. [18] Zhehao Zhang, Ryan Lutz, Aidan Mao, Tianyue Bao, Zijian Wang, Zhoujian Zhao, Kaixin Xiang, Liwei Ding, Le Tong, Jiaxin Zhuo, et al. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027, 2024. [19] Mem0. Mem0: The memory layer for personalized AI. https://mem0.ai, 2024. [20] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: agentic memory for LLM agents. CoRR, abs/2502.12110, 2025. [21] Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. Memagent: Reshaping long-context LLM with multi-conv rl-based memory agent. CoRR, abs/2507.02259, 2025. [22] Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Huang, et al. Two tales of persona in LLMs: A survey of role-playing and personalization. arXiv preprint arXiv:2406.01171, 2024. [23] Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Optimization methods for personalizing large language models through retrieval augmentation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024. [24] Sheshera Mysore, Zhuoran Lu, Mengting Wan, Julian McAuley, and Hamed Zamani. PEARL: Personalizing large language model writing assistants with generation-calibrated retrievers. In Proceedings of the 1st Workshop on Customizable NLP, 2024. [25] Jesse Richardson, Kristen Bloom, Aggeliki Founta, and Brendan Mathew. Integrating summarization and retrieval for enhanced personalization via large language models. arXiv preprint arXiv:2310.20081, 2023. [26] Cheng Li, Mingyang Chen, Haoping Wang, Bin Zhu, Haoyu Luo, et al. Teach LLMs to personalize–an approach inspired by writing education. arXiv preprint arXiv:2308.07968, 2023. [27] Ostap Wu, Max Haim, Tanmay Dey, et al. Understanding the role of user profile in the personalization of large language models. arXiv preprint arXiv:2406.17803, 2024. [28] Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democratizing large language models via personalized parameter-efficient fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. [29] Yuting Zhang, Yuliang Ding, et al. PLoRA: Personalized low-rank adaptation for humancentered text understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. [30] Tao Zhuang, Xin Wang, Zhirui Yuan, et al. HYDRA: Model factorization framework for black-box LLM personalization. arXiv preprint arXiv:2406.02888, 2024. 11

[31] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Shafran, Yejin Choi, et al. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023. [32] Zhanhui Zhou, Jie Liu, Jing Dong, Jiaheng Yang, et al. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023. [33] Xiaoyan Zhao, Juntao You, Yang Zhang, Wenjie Wang, Hong Cheng, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. NextQuill: Causal preference modeling for enhancing LLM personalization. arXiv preprint arXiv:2506.02368, 2025. [34] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, Ion Stoica, and Joseph E Gonzalez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. [35] Wujiang Xu, Zujie Liang, Kai Mei, et al. A-MEM: Agentic memory for LLM agents. arXiv preprint arXiv:2502.12110, 2025. [36] Thomas P Zollo, Andrew Weidinger, et al. PersonalLLM: Tailoring LLMs to individual preferences. arXiv preprint arXiv:2409.20296, 2024. [37] Xiaoyan Zhao, Yang Zhang, Juntao You, Wenjie Wang, Fuli Feng, et al. Do LLMs recognize your preferences? evaluating personalized preference following in LLMs. In International Conference on Learning Representations, 2025. [38] Zhaoxuan Tan et al. PersonaBench: Evaluating AI models on understanding personal information through accessing (synthetic) private user data. In International Conference on Learning Representations, 2025. [39] Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. [40] Ishita Kumar, Snigdha Viswanathan, et al. LongLaMP: A benchmark for personalized longform text generation. arXiv preprint arXiv:2407.11016, 2024. [41] Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. [42] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Wu, Kai Yu, et al. LongMemEval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813, 2024. [43] Zeyu Zhang et al. MemSim: A Bayesian simulator for evaluating memory of personal assistants. arXiv preprint arXiv:2409.20163, 2024. [44] Jianfei Xiao, Xiang Yu, Chengbing Wang, Wuqiang Zheng, Xinyu Lin, Kaining Liu, Hongxun Ding, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. AlpsBench: An LLM personalization benchmark for real-dialogue memorization and preference alignment. arXiv preprint arXiv:2603.26680, 2026. [45] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, 2023. [46] Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. [47] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive APIs. In Advances in Neural Information Processing Systems, 2024. 12

[48] Nicholas Farn and Richard Shin. ToolTalk: Evaluating tool-usage in a conversational setting. arXiv preprint arXiv:2311.10775, 2023. [49] Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In International Conference on Learning Representations, 2024. [50] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. arXiv preprint arXiv:2307.16789, 2023. [51] Jiarui Lu, Thomas Zhu, Hao Jiang, Marta Skreta, Arun Sai Rawat, et al. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. arXiv preprint arXiv:2408.04682, 2024. [52] Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen, Fuli Feng, Xueyuan Hao, Xi Su, Qi Gu, Hui Su, Xunliang Cai, et al. Aj-bench: Benchmarking agent-as-a-judge for environment-aware evaluation. arXiv preprint arXiv:2604.18240, 2026. [53] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, 2024. [54] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Reshef Manber, Vinty Baber, David Fishi, et al. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. [55] Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, et al. SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026. [56] Ruipeng Wang, Yuxin Chen, Yukai Wang, Chang Wu, Junfeng Fang, Xiaodong Cai, Qi Gu, Hui Su, An Zhang, Xiang Wang, et al. Agentnoisebench: Benchmarking robustness of tool-using llm agents under noisy condition. arXiv preprint arXiv:2602.11348, 2026. [57] Jingnan Zheng, Yanzhen Luo, Jingjun Xu, Bingnan Liu, Yuxin Chen, Chenhang Cui, Gelei Deng, Chaochao Lu, Xiang Wang, An Zhang, et al. Risky-bench: Probing agentic safety risks under real-world deployment. arXiv preprint arXiv:2602.03100, 2026. [58] OpenAI. Introducing gpt-4.1 in the api. 2025. URL https://openai.com/index/ gpt-4-1/. [59] OpenAI. Introducing gpt-5.1. 2025. URL https://openai.com/index/gpt-5-1/. Introducing gpt-5.2. [60] OpenAI. introducing-gpt-5-2/.

2025.

[61] OpenAI. Introducing o3 and o4-mini. introducing-o3-and-o4-mini/.

2025.

URL https://openai.com/index/ URL https://openai.com/index/

[62] DeepSeekAI. Deepseek-v4 model card. 2026. URL huggingface.co/deepseek-ai/ DeepSeek-V4-Pro. [63] Anthropic. Claude sonnet 4 system card. 2025. URL https://www.anthropic.com/news/ claude-4. [64] Anthropic. Claude opus 4.6 system card. 2026. URL https://www.anthropic.com/ claude-opus-4-6-system-card. [65] Gheorghe Comanici et al. Gemini 2.5: Advanced reasoning, multimodality, and agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [66] Google. Gemini 2.5 pro model card. 2025. URL https://modelcards.withgoogle.com/ assets/documents/gemini-2.5-pro.pdf. 13

[67] Google. Gemini 2.5 flash model card. 2025. URL https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-2-5-Flash-Model-Card.pdf. [68] Aohan Zeng et al. Glm-4.5: Agentic, reasoning, and coding foundation models. arXiv preprint arXiv:2508.06471, 2025. [69] Z.ai. Glm-4.6 technical blog. 2025. URL https://z.ai/blog/glm-4.6. [70] Z.ai. GLM-5.1 model card. 2026. URL https://huggingface.co/zai-org/GLM-5.1. [71] ByteDance. Seed 1.6 technical introduction. 2025. URL https://seed.bytedance.com/ en/seed1_6. [72] ByteDance Seed. Seed 2.0 model card: Towards intelligence frontier for real-world complexity. 2026. URL seed.bytedance.com/en/seed2. [73] Moonshot AI. Kimi-K2.6 model card. moonshotai/Kimi-K2.6.

2026.

URL https://huggingface.co/

[74] Meituan LongCat Team. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025. [75] MiniMax. MiniMax-M2.7: Model self-improvement, driving productivity innovation through technological breakthroughs. 2026. URL https://www.minimax.io/models/text/m27. [76] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024. [77] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [78] Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. Recursively summarizing enables long-term dialogue memory in large language models. arXiv preprint arXiv:2308.15022, 2023. [79] Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan O. Arik. Chain of agents: Large language models collaborating on long-context tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [80] Xixi Wu, Kuan Li, Yida Zhao, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou, et al. ReSum: Unlocking long-horizon search intelligence via context summarization. arXiv preprint arXiv:2509.13313, 2025. [81] Weiwei Sun, Miao Lu, Zhan Ling, et al. Scaling long-horizon LLM agent via context-folding. arXiv preprint arXiv:2510.11967, 2025. [82] Rui Ye, Zhongwang Zhang, Kuan Li, et al. AgentFold: Long-horizon web agents with proactive context management. arXiv preprint arXiv:2510.24699, 2025. [83] Minki Kang, Wei-Ning Chen, Dongge Han, et al. ACON: Optimizing context compression for long-horizon LLM agents. arXiv preprint arXiv:2510.00615, 2025. [84] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [85] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In ACM Symposium on User Interface Software and Technology (UIST), 2023. [86] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. 14

[87] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In AAAI Conference on Artificial Intelligence, 2024. [88] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. MemoryBank: Enhancing large language models with long-term memory. In AAAI Conference on Artificial Intelligence, 2024. [89] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. [90] Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, and Julian McAuley. MemoryLLM: Towards self-updatable large language models. In International Conference on Machine Learning (ICML), 2024. [91] Hongli Yu, Tinghong Chen, Jiangtao Feng, et al. MemAgent: Reshaping long-context LLM with multi-conv RL-based memory agent. arXiv preprint arXiv:2507.02259, 2025. [92] Yu Wang, Ryuichi Takanobu, Zhiqi Liang, et al. Mem-α: Learning memory construction via reinforcement learning. arXiv preprint arXiv:2509.25911, 2025. [93] Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. MEM1: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841, 2025. [94] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In International Conference on Learning Representations (ICLR), 2015. [95] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2020. [96] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. In International Conference on Machine Learning (ICML), 2020. [97] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, et al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning (ICML), 2022. [98] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research (JMLR), 2023. [99] Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. SCM: Enhancing large language model with self-controlled memory framework. In International Conference on Database Systems for Advanced Applications (DASFAA), 2025. [100] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations (ICLR), 2024. [101] Zafeirios Fountas, Martin A. Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-inspired episodic memory for infinite context LLMs. In International Conference on Learning Representations (ICLR), 2025. 15

[102] Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. HippoRAG: Neurobiologically inspired long-term memory for large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [103] Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAG to memory: Non-parametric continual learning for large language models. In International Conference on Machine Learning (ICML), 2025. [104] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024. [105] Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. LightRAG: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779, 2024. [106] Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep: A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956, 2025.

16

Appendix 19

A Benchmark Construction A.1 User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

A.1.1 User Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

A.1.2 User Preference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

A.1.3 Interaction History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 A.1.4 Prompt Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

A.2 Task Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

A.2.1 Toolset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

A.2.2 Environment Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

A.2.3 Prompt Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

A.3 Benchmark Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

A.3.1 Task Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

A.3.2 User Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

A.3.3 Task Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 A.3.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 A.3.5 Prompt Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 34

B Discussion B.1 Memory in LLM Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.1.1

34

Context Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34

B.1.2 Agentic Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

B.1.3

RAG Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

B.1.4

Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

B.2 Code of Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

B.3 Broader Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

B.4 Safeguards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36 36

C Analysis C.1 Experiments Compute Resources . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

C.2 Benchmark Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

C.2.1

User Profile Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

37

C.2.2

User Preference Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . .

38

C.2.3

Interaction Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

40

C.2.4

Implementation Configurations . . . . . . . . . . . . . . . . . . . . . . . . 41

D Trajectory

42 17

D.1 Case Study: Memory-Sensitive Delivery for User A891207 . . . . . . . . . . . . .

42

D.1.1 Part 1. One Subtask, Three Backends . . . . . . . . . . . . . . . . . . . .

42

D.1.2 Part 2. Preference Drift Across Three Subtasks . . . . . . . . . . . . . . .

44

18

A

Benchmark Construction

VitaBench 2.0 is designed to model long-term, user-centric interaction scenarios, where agents are required to continuously satisfy long-term user needs. The benchmark is built around 56 curated users, each associated with a temporally ordered sequence of tasks spanning diverse real-world domains. This design enables systematic evaluation of preference inference, preference evolution, and proactive decision-making in realistic settings. In the following, we describe the construction of user profiles, preferences, interaction histories, and task environments in detail. A.1

User

Modeling realistic users is critical in our setting, as agents are required to infer and adapt to evolving user preferences over time. However, due to well-known biases and hallucination issues, large language models alone are insufficient for generating high-quality user data. To ensure realism and consistency, we rely on manual annotation for user profiles and preferences, complemented by controlled synthesis for interaction histories. A.1.1

User Profile

Each user is associated with a detailed profile describing their demographic attributes and background information. These profiles are manually curated and inspired by our real-world application scenarios to ensure diversity and realism. The profiles serve as the foundation for preference construction and downstream task generation. We provide an illustrative example of user profile below. Example User Profile user_id: T850685 occupation: Teacher gender: Female date_of_birth: 1991-08-16 current_city: Leshan, Sichuan Province home_address: Building 1, Unit 602, Nvshen Yuan, Shizhong District, Leshan, Sichuan Province work_address: Leshan Normal University, Leshan, Sichuan Province hometown: Yibin, Sichuan Province education: Doctoral degree family_status: [“Unmarried, no children”] chronic_conditions_allergies: [“Chronic rhinitis”]

A.1.2

User Preference

Each user is associated with a set of preferences expressed as natural language statements (e.g., dietary restrictions, spending habits, travel styles), grounded in the corresponding user profile. To ensure realism and consistency, all user preferences are manually annotated. Preferences cover diverse aspects of daily life and vary significantly across users, resulting in a rich and heterogeneous preference space. In total, we curate over 1,000 user-specific preferences, with each user exhibiting multiple fine-grained constraints that jointly influence decision-making. User preferences in real life are inherently dynamic. To simulate realistic evolution over time, we introduce temporally grounded changes, including preference addition (emergence of new preferences), modification (shifts in existing preferences), and deletion (disappearance of previously relevant preferences). These changes are distributed across the task sequence to reflect long-term user dynamics. In addition, we explicitly distinguish conditional preferences, where the correct decision depends on context that is not directly observable from the current query or maintained user preference (e.g., time, companion, or situational constraints). Such preferences require agents to recognize ambiguity and actively acquire missing information from the user, forming the basis for proactive tasks in our benchmark. We provide an illustrative example of user preference below.

19

Example User Preference user_id: T850685 dietary_preferences: “Likes Sichuan cuisine” “Likes sushi” “Dislikes Cantonese cuisine” “Likes fruit tea” “Prefers drinks half-sugar” “Prefers drinks ice-free” “Likes to try newly opened restaurants on weekends” shopping_preferences: “Likes clothing with pocket designs” “Unwilling to pay a premium for luxury brands” “Likes flat shoes” “Likes to buy practical items” “Likes cool-toned colors” travel_and_accommodation_preferences: “Prefers high-speed rail for travel” “Prefers chain hotel brands” “Prefers sea-view hotel rooms” “Prefers hotel rooms with windows” leisure_preferences: “Likes playing mahjong” “Likes reading” preference_tag_change_history (selected entries across the task sequence): {“content”: “Prefers watermelon-flavored fruit tea”, “type”: “update”, “source”: “Likes fruit tea”} {“content”: “Prefers fresh-milk–based desserts”, “type”: “update”, “source”: “Prefers local brands for desserts and snacks”} {“content”: “Likes sushi”, “type”: “delete”, “source”: “”} {“content”: “Likes to buy practical items”, “type”: “delete”, “source”: “”} {“content”: “Prefers small private rooms in mahjong parlors”, “type”: “update”, “source”: “Prefers large private rooms in mahjong parlors”} {“content”: “Wants to travel abroad”, “type”: “update”, “source”: “Does not like overseas trips”} {“content”: “Dislikes Cantonese cuisine”, “type”: “delete”, “source”: “”} {“content”: “Likes clothing with pocket designs”, “type”: “delete”, “source”: “”} {“content”: “Prefers high-speed rail for travel”, “type”: “delete”, “source”: “”} {“content”: “Likes playing mahjong”, “type”: “delete”, “source”: “”} {“content”: “Prefers executive-floor hotel rooms”, “type”: “update”, “source”: “Prefers sea-view hotel rooms”}

Example Proactive Preference user_id: B865629 conditional_preference: “Prefers high-caffeine coffee in the morning; low-caffeine in the afternoon” historical_conversation (source of the conditional preference): “I find I’m quite sensitive to caffeine — high-caffeine in the morning is totally fine, gives a real kick, but in the afternoon I can only handle low-caffeine.” user_query: “I have a meeting on the 26th, order me a coffee in advance to stay alert.” unresolved_ambiguity: The correct caffeine level depends on the time of the meeting, which is not specified in the query. The

20

agent should proactively ask the user when the meeting is scheduled. user_clarification: “The meeting is in the afternoon.” expected_outcome: “The ordered product should be coffee” “The ordered product should be low-caffeine” “Delivery address: user’s workplace”

A.1.3

Interaction History

User preferences are not directly exposed to the agent, but are instead implicitly encoded in fragmented interaction histories. Inspired by information accessibility in real-world scenarios, we construct interaction histories consisting of two modalities: (1) dialogues, including multi-turn conversational interactions between users and agents, and (2) behaviors, including user logs such as browsing, ordering, reviewing, and searching. To ensure both realism and sufficient difficulty, interaction histories are constructed through a controlled synthesis process guided by manually designed preference signals. Specifically, given initial user preferences, we first manually design how these preferences can be embedded into fragmented interactions and then generate a large set of interactions that implicitly reflect these preferences. As tasks are temporally ordered, additional interaction histories are generated between consecutive tasks, capturing preference evolution over time. These histories may encode preference addition, preference drift, or preference disappearance. We distinguish between two types of preference changes. In some cases, preference changes are explicitly reflected through interactions (e.g., a user repeatedly ordering vegetarian meals). In other cases, preference changes are not directly observable from interactions and are instead modeled as implicit state transitions. For example, a user may temporarily prefer lighter meals due to illness, or stop exhibiting pregnancy-related preferences after the corresponding period ends. Such implicit changes introduce additional difficulty, as agents cannot rely solely on observable interaction signals and must maintain a consistent and adaptive representation of user preferences over time. To further increase difficulty and realism, not all interactions are preference-relevant. We deliberately introduce noise, including irrelevant actions, ambiguous signals, and short-term preference fluctuations that may appear inconsistent. This requires agents to distinguish stable preferences from noisy observations. All interaction histories are manually reviewed and refined to ensure that they are coherent, free of contradictions, and do not introduce unintended preference leakage. We provide an illustrative example of interaction history below. Example User Interaction History User ID: T850685 Period: 2026-04-14 to 2026-04-15 Preference State Before: likes fruit tea; prefers drinks half-sugar and no ice. Preference Drift: update from “likes fruit tea” to “prefers watermelon-flavored fruit tea”, triggered on 2026-04-14 by the user’s statement: “Fruit teas feel sweeter these days; only watermelon still tastes refreshing. From now on, prioritize watermelon flavor for fruit tea.” Timeline 2026-04-14 Behaviors Order: Mango pomelo sago, half-sugar and no ice, from A Little Tea (Wanda Store), 18 RMB. Dialogue User: What model are you? Assistant: I am your AI life assistant. What can I help you with? User: Just asking. Help me order mango pomelo sago from A Little Tea, the usual way. Assistant: Sure. Mango pomelo sago from A Little Tea Wanda Store, 18 RMB, half-sugar and no ice. Should I place the order? User: Go ahead.

21

Assistant: Ordered. It should arrive in about 30 minutes. User: I feel like fruit teas are getting sweeter these days; only watermelon still tastes refreshing. Assistant: Mango and coconut-based drinks can indeed taste rich even at half sugar, while watermelon tea is usually lighter. Should I prioritize watermelon flavors for fruit tea from now on? User: Yes, prioritize watermelon flavor for fruit tea in the future. Assistant: Got it. I will prioritize watermelon-flavored fruit tea for you from now on. 2026-04-15 Behaviors Search: “is pearl cream actually made from pearls?” Order: Watermelon boba fruit tea, half-sugar and no ice, from Shuyi Tealicious (Teachers College Store), 14 RMB. Dialogue User: Is pearl cream really made from pearls? Assistant: Traditional pearl cream often contains pearl powder, but the amount varies a lot across products today. User: Oh, I was just curious. User: By the way, help me order a fruit tea from Shuyi. Assistant: Shuyi Tealicious Teachers College Store has “Watermelon Boba Fruit Tea” for 14 RMB, half-sugar and no ice, delivered to Leshan Teachers College. Does that work? User: Yes, place the order. Assistant: Ordered. It should arrive in about 20 minutes. 2026-04-17 ......

A.1.4

Prompt Template

We provide the prompt templates used for synthesizing user interaction histories below. User Interaction Synthesis Prompt Template # General Prompt I will provide you with information about a real human user and a period of time during which the user’s personalized preferences have changed. The user fulfills everyday needs on a comprehensive life-service platform, either by interacting with an AI agent or by taking platform actions independently. Based on the input fields, synthesize a temporally ordered interaction history from start_date to end_date. The history should contain user-agent dialogues, platform behavior logs, preference drift signals, and preference-irrelevant noise. The central goal is to make the preference changes naturally observable through the generated dialogues and behaviors, while keeping all personalized content grounded in the provided user profile and current preferences. # Input Fields 1. start_date: the first date covered by the generated history. 2. end_date: the last date covered by the generated history. 3. user_profile: the user’s demographic background, lifestyle context, and other basic information. 4. current_preferences: the user’s complete preference state after the drift events in this period. 5. preference_changes: the preference additions, deletions, or modifications that occur during this period. These changes are the core signals that must be reflected. 6. historical_chat_list: user utterance fragments that must all appear in the generated dialogues. They may be placed on different dates and adapted to fit the context, but their core meaning must be preserved. 7. historical_behavior_text: behavior records that must all appear in the generated platform behavior logs. They may be distributed across different dates. 8. noise_text: preference-irrelevant content that must be naturally incorporated into the generated dialogues or behaviors. # Preference Drift Preference drift is the primary target of this history. The generated dialogues and behaviors should naturally reflect how the user’s preferences change during this period. Newly added preferences should appear through attempts, discoveries, explicit interest, or repeated related actions. Modified preferences

22

should show a plausible transition from the previous tendency to the new one. Deleted preferences should be reflected through reduced interest, explicit rejection, or behavior that indicates the preference is no longer active. These drift signals should be distributed across the history rather than stated mechanically in one place. # Dialogues Dialogues represent multi-turn interactions between the user and the AI agent. Each topic or intent should be sufficiently developed: task-oriented dialogues such as ordering food, booking tickets, shopping, or travel planning should usually contain 4–8 user-assistant turns, while casual conversations should usually contain 3–6 turns. The user’s language style should match the user profile and preference descriptions, and all required utterance fragments in historical_chat_list must be included as user messages. The agent’s responses should be varied and natural, and a dialogue should not end before the user’s intent has been fully expressed or resolved. # Behaviors Behavior records represent actions taken by the user on the platform, including actions completed independently and actions related to prior agent interactions. All required behaviors in historical_behavior_text must be included. Not every behavior needs a corresponding dialogue; independent searches, browsing, favorites, comments, ratings, and orders may appear as standalone platform activity. Supported behavior types include search, order, rate, comment, high_freq_browse, complaint, add_to_cart, and favorite. The order type covers delivery, ticket, instore, travel_ticket, hotel, attraction, local_trip, and errand scenarios. # Noise Noise refers to content that is unrelated to personalization but makes the history more realistic. All items in noise_text must be naturally incorporated into the generated dialogues or behavior logs. Noise may include casual chat, ordinary one-off actions, ambiguous records, or preference-irrelevant interactions. It should not introduce new user preferences, contradict the user profile, or distract from the preference drift signals. # Notes All records must occur between start_date and end_date, be organized by day, and appear in increasing chronological order. Each day may contain any number of dialogues or behaviors, including days with only one type of record. All personalized content must be traceable to current_preferences, preference_changes, or user_profile. Do not invent preferences, habits, interests, or consumption patterns that were not provided. You may add extra dialogues and behaviors to reflect unchanged preferences and make the history richer, but they must remain grounded. Merchant, restaurant, and hotel names may be plausibly fabricated, but real-world attractions, cuisines, aircraft models, and well-known brands must not be fabricated. Return only a JSON array. The response must start with [ and end with ]. Do not wrap the output in a code block. If quotation marks are needed inside a JSON string value, use Chinese quotation marks such as [], not unescaped English double quotation marks. # Output Format Return a list of daily records. Each record contains a date, a behavior list, and a dialogue list. Use the following structure: [{"date":"2023-07-07", "behavior":[ {"behavior_type":"search", "content":{"keyword":"swimming pool"}}, {"behavior_type":"order", "content":{"scenario":"instore", "merchant_name":"a fitness center", "tags":["heated pool"], "items":[{"product_name":"annual pass", "price":5888, "quantity":1}]}}], "dialogue":[ {"role":"user","content":"Where is a nearby swimming pool?"}, {"role":"assistant","content":"I found three options ..."}, {"role":"user","content":"I prefer a heated pool."}, {"role":"assistant","content":"The first option has one ..."}]},

23

{"date":"2023-07-15", "behavior":[ {"behavior_type":"rate", "content":{"target_name":"a fitness center", "target_type":"merchant", "score":4.5}}], "dialogue":[]}] # Inputs {"start_date": "{start_date}", "end_date": "{end_date}", "user_profile": {user_profile}, "current_preferences": {current_preferences}, "preference_changes": {preference_changes}, "historical_chat_list": {historical_chat_list}, "historical_behavior_text": {historical_behavior_text}, "noise_text": {noise_text}}

A.2

Task Environment

The environment in VitaBench 2.0 provides the execution space in which the agent interacts with tools to fulfill user requests. As each task is formulated as an agentic task, the environment must expose a structured and realistic candidate space that supports tool-based decision making. We construct domain-specific environments that simulate real-world service scenarios, together with executable tool interfaces and structured data. A.2.1

Toolset

We adopt the toolset design from VitaBench [17], covering three representative domains: Delivery, In-store Consumption, and Online Travel Agency. In total, the benchmark includes 66 tools that expose structured APIs for retrieving and manipulating environment states. The tools are designed to be consistent across domains and sufficiently expressive to support multi-step interactions, while avoiding unnecessary complexity. This ensures that agents must correctly invoke tools to access relevant information, but that task difficulty primarily stems from preference inference and decisionmaking rather than tool usage itself. We provide an illustrative example of toolset below. Example Toolset Delivery (12 tools). delivery_distance_to_time, get_delivery_store_info, get_delivery_product_info, delivery_store_search_recommand, delivery_product_search_recommand, create_delivery_order, pay_delivery_order, get_delivery_order_status, cancel_delivery_order, modify_delivery_order, search_delivery_orders, get_delivery_order_detail. In-store (16 tools). instore_shop_search_recommend, instore_product_search_recommend, create_instore_product_order, pay_instore_order, instore_cancel_order, instore_book, pay_instore_book, instore_cancel_book, instore_reservation, instore_modify_reservation, instore_cancel_reservation, get_instore_orders, get_instore_reservations, get_instore_books, search_instore_book, search_instore_reservation. OTA (37 tools). get_ota_hotel_info, get_ota_attraction_info, get_ota_flight_info, get_ota_train_info, hotel_search_recommand, attractions_search_recommend, flight_search_recommend, train_ticket_search, create_hotel_order, create_attraction_order, create_flight_order, create_train_order, pay_hotel_order, pay_attraction_order, pay_flight_order, pay_train_order, search_hotel_order, search_attraction_order, search_flight_order, search_train_order, get_hotel_order_detail, get_attraction_order_detail,

24

get_flight_order_detail, get_train_order_detail, modify_train_order, modify_flight_order, cancel_hotel_order, cancel_attraction_order, cancel_flight_order, cancel_train_order, get_taxi_info, taxi_search, create_taxi_order, pay_taxi_order, cancel_taxi_order, search_taxi_order, get_taxi_order_detail. Personalization (2 tools). read_preference_memory, query_preference_memory. Illustrative tool schemas (taken verbatim from the codebase; we keep the original spelling recommand for consistency with the data files). delivery_store_search_recommand: description: “In delivery scenario, can extract keywords describing stores from user expressions, search or recommend multiple stores” preconditions: “In delivery scenario, get keywords describing stores” postconditions: “Return store list, guide user to select and confirm store” args: {“keywords”: “Keywords describing stores”} returns: “Structured store information output” tool_type: “READ” create_delivery_order: description: “Create delivery order, only supports single store ordering, single store can order multiple items” preconditions: “In delivery scenario, determine unique store id and one or more product ids, determine user’s dietary restrictions, reflect in the order” postconditions: “Return order information, ask user whether to pay the order” args: {“user_id”, “store_id”, “food_ids”, “food_cnts”, “address”, “dispatch_time”, “attributes”, “note”} returns: “Order information” tool_type: “WRITE” query_preference_memory: description: “Query user preference memory with a specific question” preconditions: “Use when you need to query specific user preference information” postconditions: “Returns preference information relevant to the query” args: {“query”: “Natural language question about user preferences”} returns: “Preference information relevant to the query” tool_type: “READ”

A.2.2

Environment Synthesis

As each task in VitaBench 2.0 is formulated as an agentic task, it requires an executable environment in which the agent can invoke tools to interact and fulfill user needs (e.g., placing a food-delivery order). Our benchmark contains thousands of tasks spanning diverse domains (e.g., food delivery, in-store services, and travel booking), and ensuring sufficient task difficulty requires each environment to present a rich and structured candidate space. Manually constructing such environments at scale is therefore impractical. To address this, we design a synthesis pipeline that generates structured, executable environments for each task, complemented by programmatic validation and manual refinement to ensure correctness and consistency. We adopt a multi-agent design that decomposes environment synthesis into a sequence of specialized components, each responsible for a well-defined subtask. Given a user profile, user query, and evaluation rubric, the environment generator materializes a database of merchants and items consistent with the task specification, including both valid candidates and carefully constructed distractors. Importantly, our benchmark evaluates not only basic tool-use capabilities but also the ability to infer and leverage user preferences. Accordingly, we construct environments whose difficulty arises from two complementary dimensions. First, conditioned on the user query, we generate merchants and items that are relevant to the domain but do not satisfy the task constraints, requiring the agent to correctly invoke tools and filter out irrelevant candidates. Second, conditioned on the evaluation rubric, we generate candidates that satisfy the user query but violate user preferences, requiring the agent to infer and apply preference information to eliminate such distractors. 25

To improve realism and control complexity, environments are synthesized in a top-down manner: we first generate a set of merchants and then populate each merchant with items. We explicitly control the number of items that satisfy all rubric constraints to ensure that each task admits a well-defined solution while remaining sufficiently challenging. To ensure evaluation correctness, we first apply a strong model to verify the logical consistency of the environment and detect potential conflicts, followed by human expert review to further refine the data and ensure that the resulting environments are both valid and non-trivial. We provide an illustrative example of task environment below. Example Environment The following example is adapted from the first subtask of user M793481. The user asks the agent to book a ticket from Chengdu to Kunming for the coming Saturday. The environment contains weather and location context, together with a train-candidate database. Only one weather record, one distractor train, and one target train are shown; omitted entries are denoted by .... { "task_turn_num": "M793481_01", "domain": "ota", "instruction": "I want to visit Kunming. Please book a ticket for this Saturday.", "current_time": "2024-07-02", "task": "OTA / ticket booking", "rubric": [ "the recommended product should include a ticket for 2024-07-06", "the recommended product should be a bullet-train ticket", "the recommended product should depart from Chengdu", "the recommended product should arrive in Kunming" ], "environment": { "time": "2024-07-02 14:30:00", "weather": [ {"city": "Kunming", "category": "sunny", "datetime": "2024-07-06", "temperature": [18, 26], "humidity": 58}, "..." ], "location": [ {"address": "Chunxi Road, Jinjiang District, Chengdu", "longitude": 104.0665, "latitude": 30.5728} ], "trains": { "S17775666762126018_T00002": { "train_number": "G2815", "departure_city": "Chongqing", "arrival_city": "Kunming", "departure_time": "2024-07-06 08:20", "arrival_time": "2024-07-06 13:45", "tags": ["high-speed rail", "fast", "comfortable"], "train_type": "distraction", "distraction_reason": "departure city is Chongqing, violating the Chengdu-departure requirement", "products": [ {"seat_type": "second class", "price": 398, "quantity": 68, "date": "2024-07-06", "product_type": "distraction", "product_id": "S17775666762126018_P00003"}, {"seat_type": "first class", "price": 628, "quantity": 25, "date": "2024-07-06", "product_type": "distraction", "product_id": "S17775666762126018_P00004"} ] }, "...": "other query-relevant distractor trains are omitted" } } }

26

A.2.3

Prompt Template

We provide the prompt templates for the environment generator, noise injector, and verifier below. Environment Synthesis Prompt Template # General Prompt Produce a structured database of merchants and items that is consistent with the task specification, forming an executable environment for the agent. You are an environment synthesizer for a {domain} agentic benchmark. Your job is to produce a complete JSON environment database that turns the given task into an executable testbed, where an AI agent must use tools to find the correct answer. # Task Specification - User profile: {user_profile} - Current time: {current_time} - Task type: {task_type} - User query: {instruction} - Evaluation rubric: {rubric_formatted} # Output Structure Produce a JSON database in the following abbreviated schema. The field {entity_key} should be one of stores, shops, trains, hotels, flights, or attractions, depending on the domain. { "time": "...", "weather": [...], "location": [...], "{entity_key}": { "entity_001": { "id": "...", "name": "...", "...": "domain attributes such as score, location, tags", "{entity_type}_type": "target | distraction", "distraction_reason": "...", "products": [ { "name": "...", "price": ..., "...": "product attributes, options, tags", "product_type": "target | query_distractor | preference_distractor | irrelevant", "distraction_reason": "..." } ] } } } # Top-Down Synthesis Protocol Pass 1 — Merchant layer. Generate a realistic set of merchants relevant to the user query. Merchant names, tags, scores, and locations should resemble a real search-result page in the target domain, e.g., natural names such as “Dianwei Crossing-Bridge Rice Noodles (High-Tech Branch)” rather than generic test names. Pass 2 — Item layer. For each merchant, populate items. The full environment must contain exactly one globally valid target product that satisfies both the user query and all rubric constraints, multiple distractors along the two axes below, and a small number of plainly irrelevant items for realism. # Rubric Decomposition Before generating the database, parse the evaluation rubric into atomic constraint dimensions. For example, if the rubric requires the ordered product to be crossing-bridge rice noodles and the flavor to be tomato, the dimensions are category = crossing-bridge rice noodles and flavor = tomato. Also infer the keywords that the agent is likely to search for from the user query. # Axis A: Query-Relevant but Rubric-Violating Distractors Generate items that look domain-appropriate and satisfy the user’s literal request, but violate one or

27

more rubric constraints. These items force the agent to issue the right tool calls and filter the candidate space correctly. Label them with product_type = "query_distractor". For each rubric dimension, design at least five “only-one-thing-off” distractors. Each such distractor should violate exactly one dimension while satisfying all other dimensions, and should be almost identical to the target in name, tags, attributes, and price range. In addition, include 3–5 multi-dimensional violations, each violating 2–3 dimensions simultaneously. # Axis B: Rubric-Satisfying but Preference-Violating Distractors Generate items that satisfy the user’s literal query and most rubric constraints, but violate latent preferences inferred from the user profile. These items force the agent to infer and apply user preferences rather than selecting the first plausible candidate. Label them with product_type = "preference_distractor". Distribute these distractors across multiple merchants rather than clustering them in one place. # Distractor Annotation For every non-target item, the field distraction_reason must explicitly state which rubric or preference constraint is violated and how. For plainly irrelevant items, label them with product_type = "irrelevant" and include a short reason. # Complexity Control - Target uniqueness: the target product must be globally unique. If any other item could also satisfy all rubric constraints, relabel it as a distractor. - Scale: generate 10–15 merchants, 4–8 products per merchant, at least 40 products in total, and at least 30 distractors. - Placement: the target product must not appear in the first five merchants; place it in the middle or later portion of the list. - Ranking: the target merchant must not be the highest-scored merchant. Its score should be in [4.0, 4.6], and at least two other merchants should have higher scores. - Search coverage: the names and tags of all merchants should include the main search keywords inferred from the user query, so lexical uniqueness alone is insufficient. # Address and Location Constraints If the rubric involves a delivery address or location, populate the location field with coordinates consistent with that constraint. For delivery-type tasks, all merchants should lie within a reasonable range of the specified address. # Output Requirements Return the JSON database only. Do not include comments, explanation text, or markdown fences outside the JSON body. Use placeholder IDs such as store_001 and prod_001; the system will rewrite them in post-processing. The weather field should cover the current day ± 1–2 days.

Noise Injector Prompt Template # General Prompt Given an already-generated environment, inject additional adversarial distractors so that an agent without preference memory cannot solve the task reliably, while an agent with preference memory still can. You are a noise injector for an agentic benchmark. Given a task and its initial environment, inject additional distractor entries so that an agent relying only on the literal user query cannot reliably select the correct answer, while an agent with full preference memory still can. # Core Objective The null agent has no user preference memory and acts purely on the literal user query. The injected distractors must appear perfectly reasonable at the query level, satisfying the user’s explicit request, but fail at the preference level, which is the dimension the null agent cannot recover. # Inputs - User query: {instruction} - Evaluation rubric: {rubric} - Current target item: {target_text} - Non-empty environment keys: {keys_list} - Tool signatures: {tool_section} - Retrieval and schema information: {keys_section}

28

- Null-agent queries: {null_q_text} - Groundtruth-agent queries: {gt_q_text} # Rubric Decomposition Partition every rubric constraint into two sets before generating distractors. Set A contains query-derivable constraints, namely constraints the agent can recover directly from the user query. For example, if the query asks for a 6-person combo, then the rubric constraint “serves 6” belongs to Set A; if the query asks for coffee, then “product is coffee” belongs to Set A; if the query asks for delivery to the office, then the delivery-address constraint belongs to Set A. Set B contains preference-specific constraints, namely constraints that require user profile or historical behavior and are not mentioned in the query. For example, if the query asks the agent to pick a board game, then “card-based” may belong to Set B; if the query asks for coffee to stay awake, then “sugar-free” may belong to Set B; if the query asks to buy flowers for home, then “pink roses” may belong to Set B. # Adversarial Invariants Every injected distractor must jointly satisfy the following conditions. 1. It must pass all Set A constraints. The item should be a plausible answer to the literal user query, so a preference-free agent has no lexical or query-level reason to reject it. 2. It must clearly and unambiguously violate at least one Set B constraint. The violation must be crisp, with no borderline or partial match that an evaluator could reasonably accept. For instance, if the preference-specific constraint is “card-based”, then ambiguous games such as Catan, Werewolf, or Undercover are bad distractors because each can be argued to involve cards. Pure strategy, board, or puzzle games such as chess, Go, jigsaw, or Jenga are better distractors because they contain no cards. 3. It must be retrievable. Because the search tool ranks candidates by fuzz.partial_ratio(name + tags, query) with a large top_k, the distractor’s name and tags must contain the null agent’s query keywords. For hotels, attractions, flights, and trains, the relevant locations or cities must also match the query. # Absolute Prohibitions - Do not produce distractors that violate Set A, because the agent would filter them out trivially. - Do not produce distractors whose Set B violation is ambiguous or partially satisfiable, because the evaluator may score them as correct and corrupt the measurement. # Where the Target Gets Its Score The target satisfies both Set A and Set B. Each injected distractor should satisfy Set A but crisply violate Set B. As a result, a preference-free agent may choose a plausible-looking wrong answer, while a preference-aware agent can still separate the target from the distractors. # Generation Requirements For each environment key in {keys_list}, generate {NOISE_PER_KEY} distractor entries. Each entry must satisfy the following requirements. 1. Satisfy every Set A constraint through its name, tags, attributes, or structured fields. For example, if the query asks for a 6-person board game, the product name should indicate “6-person combo” or “6-player”; if the query asks for coffee to the office, the product should be coffee and the delivery address should match. 2. Crisply violate at least one Set B constraint, with no hedging. If the Set B constraint is “sugar-free”, use “full sugar” or “standard sweetness”, not “light sugar” or “low sugar”. If the Set B constraint is a specific brand, use a completely different brand. 3. Be retrievable. Names and tags must contain the null agent query’s core keywords. For hotels and attractions, location.address must lie in the query city. For flights and trains, departure and arrival cities must match the query. 4. Be highly confusable. Make the distractor more attractive than the target to an agent without preference memory by using well-known brands, popular choices, high ratings, competitive prices, and names such as “Deluxe”, “Hot Pick”, “Classic”, or “Best-Seller”. 5. Match the exact field structure of the real samples. All required top-level and product-level fields must be present. 6. Leave the id field as an empty string, because post-processing fills it. 7. For stores products, store_name must equal the parent store’s name, and attributes must be a list of strings. 8. Ensure diversity across the {NOISE_PER_KEY} entries for each environment key, including different brands, styles, and price points. Each entry’s distraction_reason must explicitly state which Set B constraint is violated and how.

29

# Output Return pure JSON only. Do not include explanation text or markdown fences. Begin directly with {, using the following abbreviated top-level structure. Only include keys that are present in {keys_list}. Each list should contain {NOISE_PER_KEY} distractor entries, each following the full schema of the corresponding environment key. { "stores": [ {...}, {...}, ... ], "shops": [ {...}, {...}, ... ], "hotels": [ {...}, {...}, ... ], "trains": [ {...}, {...}, ... ], "flights": [ {...}, {...}, ... ], "attractions": [ {...}, {...}, ... ] }

A.3

Benchmark Pipeline

We formulate VitaBench 2.0 as a sequential user-agent interaction process, where the agent is required to continuously fulfill user needs over a temporally ordered sequence of tasks. Each task corresponds to a concrete user request issued by a user simulator and is solved by the agent through interaction with domain-specific tools and an executable environment. Between consecutive tasks, the agent is exposed to newly generated interaction histories and may update its internal memory to maintain an evolving understanding of user preferences. During task execution, the agent integrates current observations, retrieved memory, and tool feedback to make decisions. This setting enables unified evaluation of tool-use ability, preference inference and utilization, and proactive behavior under incomplete information. A.3.1

Task Set

The task set is manually constructed and grounded in the corresponding user profile to ensure realism and consistency. Each task is designed to evaluate one or multiple user preferences, requiring the agent to capture, utilize, and maintain these preferences over time. Tasks span diverse real-world domains, reflecting a wide range of everyday user needs. Specifically, tasks require agents to reason over two complementary sources of difficulty. The first arises from explicit constraints specified in the user query, which must be satisfied through appropriate tool use. The second arises from implicit signals derived from user interaction histories, requiring the agent to correctly infer and apply user preferences. Compared to existing agent benchmarks that primarily emphasize multi-step reasoning and complex tool orchestration, our benchmark introduces an additional dimension of difficulty through implicit preference modeling. To better isolate this capability, we intentionally reduce the complexity of explicit reasoning and tool usage, avoiding overly intricate tool chains. This design ensures that task success depends primarily on the agent’s ability to capture, utilize, and maintain user preferences. In addition, a subset of tasks involves conditional preferences, where the correct decision depends on context not directly observable from the current query. These tasks form the basis for evaluating proactive behavior. A.3.2

User Simulator

Following prior work on complex tool-use benchmarks [15–17], we formulate VitaBench 2.0 as a user-agent interactive benchmark to simulate realistic assistant scenarios. In this setting, the evaluated model acts as an assistant that must fulfill user needs, while the user is instantiated as a simulator within the environment. The user simulator is responsible for issuing task instructions and providing interaction feedback during task execution. A key challenge in designing such a simulator lies in controlling its available context: it must possess sufficient information to generate realistic interactions, while avoiding direct exposure of information that would trivialize the task. In practice, user simulators are typically implemented using large language models, which are inherently difficult to control and may exhibit unintended behaviors such as information leakage. Prior work has reported that such leakage can significantly compromise the validity of evaluation in interactive benchmarks [4]. To address this issue, our simulator is deliberately restricted. It does not have access to the underlying user preferences and only provides user queries solely based on our predefined to-do list. During interaction, it provides only minimal feedback required for task completion, 30

without revealing preference-related signals. This design is critical in our setting, as task difficulty in VitaBench 2.0 primarily arises from the agent’s ability to infer and utilize user preferences. Any unintended leakage of preference information from the simulator would significantly reduce task difficulty and undermine evaluation validity. In proactive tasks, the simulator may provide additional information upon request, but such responses are predefined and strictly controlled. A.3.3

Task Agent

The task agent is responsible for fulfilling user requests through interaction with tools and the environment. All evaluated large language models in our benchmark are instantiated as the task agent. Tasks are presented sequentially, and between consecutive tasks, the agent is exposed to newly generated interaction histories that reflect fragmented user behaviors and evolving preferences. The agent may maintain an external memory module to accumulate user-specific information over time. Upon receiving new interaction histories, the agent updates its memory representation, which is subsequently used during task execution. When memory is enabled, the agent does not have direct access to the full interaction histories and must rely solely on its memory for user modeling. During task execution, the agent integrates current observations, retrieved memory, and tool feedback to make decisions. This setup requires the agent to continuously infer, utilize, and update user preferences across tasks, while operating under incomplete and noisy observations. A.3.4

Evaluation

Task outcomes are evaluated based on manually curated rubric sets that specify the necessary conditions for successful completion. Each rubric decomposes task success into a set of atomic constraints (e.g., item attributes, price range, or temporal conditions), ensuring structured and interpretable evaluation aligned with underlying user preferences. Given an agent’s interaction trajectory, we employ a strong evaluator model to assess performance based on these rubrics. Following VitaBench [17], we adopt a window-based evaluation scheme, where the trajectory of each task is segmented into multiple interaction windows. The evaluator assigns scores to each window, which are then aggregated to produce a trajectory-level reward. This design allows us to capture not only final decision correctness but also the quality of intermediate actions, such as tool usage and clarification behavior. Beyond trajectory-level reward, we additionally introduce an outcome-level reward that explicitly evaluates whether the final decision aligns with user preferences. This complements window-based evaluation by ensuring that the agent’s behavior leads to preference-consistent outcomes, rather than merely exhibiting locally correct interactions. This combination of rubric-based evaluation, window-level scoring, and outcome-level assessment provides a comprehensive and reliable measure of agent performance in personalized and proactive settings. An illustrative example is provided below. A.3.5

Prompt Template

We provide the prompt templates for the user simulator, task agent, and evaluator below. User Simulator — User Prompt Template # Role Setting You are playing the role of a user interacting with an intelligent agent. Your character is described in the <persona> tag. <persona> {persona} </persona> # Current Task You need to make the following request to the agent: <instruction> {instruction} </instruction> # Conversation Style Rules: - Generate only one line of content each time to simulate user messages

31

- Keep your requests vague and concise, don’t proactively provide too many details - When the agent asks about your preferences, respond with vague answers like “whatever”, “you decide”, “I’m not sure” # Acceptance or Rejection Rules: - Only express acceptance or rejection after the agent gives a clear recommendation - Judgment criterion: whether the recommendation satisfies the explicit conditions stated in <instruction> (e.g., size, quantity, time, price limit, etc.) - If the recommendation meets all explicit conditions in the instruction, accept it (do not evaluate other choices the agent made based on preferences) - If the recommendation fails any explicit condition in the instruction, briefly express dissatisfaction without giving a reason # Information Disclosure Rules: - Don’t proactively reveal your preference details, let the agent infer through memory - If the agent asks for your identity information (such as user ID, phone number, etc.), reply with your user ID as identity information {proactive_section} # When You CAN End the Conversation: - When the agent has completed your request and you are satisfied with the result - Generate ###STOP### mark to end the conversation

Task Agent Prompt Template # Environment - Current time: {time} # Role Setting You are a personalized intelligent service assistant. You need to provide more considerate and customized services based on the user’s preference memory. # Tool Usage Guidelines: - When the user’s needs require using tools to complete, first determine whether all parameter information is known. If it is known, extract the corresponding parameters, otherwise ask the user for the relevant parameter values - When the user cannot provide relevant information, first check the user preference memory, then obtain relevant information through tools - When the user’s request is vague (e.g., “order some food for me”), prioritize consulting user preference memory to infer likely needs - Complete tasks based on Precondition and Postcondition # Preference Memory Usage Guidelines - Prioritize completing tasks based on preference memory; preference match takes precedence over store ratings, distance, and other factors - When the user’s request is vague, use information from the preference memory to make reasonable inferences and recommendations - Confirm recommendations based on preferences with the user, but don’t excessively question things that are already answered in memory # Conversation Guidelines - Only use information from the above context and preference memory, prohibit constructing information without basis and replying to users - Focus on completing user needs, prohibit divergent guidance to users to propose new needs - After completing the user’s task requirements, ask if there are any other needs. If the user indicates no, generate ###STOP### mark to end the conversation

Memory Update Prompt Template You are a user preference memory manager. Your task is to maintain an accurate and concise summary of user preferences based on their interaction history.

32

## Current Memory Content: {current_memory} ## New User Interaction Records: {new_interactions} ## Task Requirements: Please merge the new interaction records with the existing memory and generate an updated user preference summary. Requirements: 1. Retain valid preference information from existing memory 2. Update old preferences that contradict new interactions 3. Add new preferences discovered from new interactions 4. Structure the preference information, including but not limited to: - Food preferences (taste, cuisine, ingredient preferences, etc.) - Spending habits (price range, frequency, etc.) - Time preferences (meal times, travel times, etc.) - Location preferences (frequent areas, distance preferences, etc.) - Service preferences (delivery requirements, seating preferences, etc.) - Other preferences Please output the updated preference summary directly, without any explanatory text.

Evaluator Prompt Template # System Information {env_info} # User Complete Instruction {user_instruction} # Background - This is a conversation scenario between a user and an assistant, where the assistant can call tools to retrieve information and complete operations. Tool return results will start with “tool” - You need to evaluate whether the user instruction has been completed. The user’s complete instruction has been broken down into several scoring points (rubrics), and you only need to judge whether each scoring point is satisfied - <trajectory_content> contains the complete conversation content between the user and assistant - <current_rubrics> contains the current status of all scoring points (true means satisfied, false means not satisfied, all scoring points have an initial status of false) # Task - Based on the conversation content, update the scoring point rubric status - You can update the status from false to true, if and only if the assistant completed the goal in the conversation # Important Notes - Important: All evaluations are based on whether the assistant’s responses and tool call requests complete the goals in the rubrics. User expressions in the conversation are only considered as guidance for the assistant and do not directly affect evaluation standards. Everything is based on the rubric fields! - Important: Query tool return results are only visible to the assistant and do not represent content recommended by the assistant to users, so they do not directly affect evaluation results either. Everything must be based on the assistant’s responses to users after obtaining information! Also note that the Assistant cannot fabricate Tool return results! - Important: For order-related rubrics (involving order details that must generate orders), you must confirm whether the assistant actually completed the ordering operation. The assistant may mistakenly believe they completed the ordering operation when in fact the tool call failed; or situations where the user states they “can place the order themselves,” etc., should all be considered as not meeting the requirements - For rubrics involving order details such as product quantity or delivery time, the original rubric requirements must be strictly met (no deviation in product quantities, delivery must not be later than the expected time). User compromise behavior does not affect evaluation results (for example, when a user

33

states “fewer items is okay”, “I have no objections to the order content” or “later delivery is fine”, etc.). These situations should still be considered as not meeting the requirements - For rubrics involving text content matching of addresses or order notes, apply the functional equivalence principle: as long as the actual content can achieve the same function (such as roughly locating the delivery location or conveying the customer’s main needs), it is considered to meet the requirements even if the expression is not completely consistent or lacks some details - In the justification, record key information related to the current rubric and its corresponding round [x] in an appending manner. If status modifications occur, record the reason using concise language # Format Requirements - Your response should be a JSON object containing the following fields: - rubric_idx: Unique identifier for the rubric - rubric: Restatement of the rubric - justification: Explanation of status changes - meetExpectation: Updated status (true or false) # Example Input Structure: <trajectory_content></trajectory_content> <current_rubrics></current_rubrics> # Example Response Structure: [ {{ "rubric_idx": "rubric_0", "rubric": "<restate the rubric>", "justification": "<brief explanation of status change, recorded in appended format>", "meetExpectation": <true or false> }}, ... ]

B

Discussion

B.1

Memory in LLM Agents

As LLM-based agents are deployed on increasingly long-horizon tasks—such as web research, software engineering, multi-session dialogue, and embodied control—memory has become a firstclass design component rather than a byproduct of context length. We organize prior work along three axes that differ in where memory is stored and how it is accessed: (i) Context memory, which keeps memory within the model’s working context via compression or summarization; (ii) Agentic memory, where the agent explicitly controls memory operations such as writing, updating, and retrieval; and (iii) RAG memory, which externalizes memory to an embedding or graph store and retrieves it on demand. B.1.1

Context Memory

Context memory treats the model’s active context window as the primary storage medium. However, practical deployment is limited by both computational cost and the model’s ability to process long sequences. As a result, a line of work focuses on preserving task-relevant information while maintaining a bounded context. Early approaches extend effective context length through architectural or retrieval augmentation. Xiao et al. [76] leverage the attention-sink phenomenon combined with sliding windows for stable long-context generation, while Wang et al. [77] augments a frozen backbone with a retrieval-based side network over cached key-value states. More recent work explicitly treats summarization as a memory operation in agent settings. Recursive summarization [78] compresses dialogue into cumulative memory, while Chain-of-Agents [79] replaces full attention with collaborative message passing. ReSum [80] periodically compresses ReAct trajectories into compact reasoning states and trains policies with reward broadcasting. Context-Folding [81] and AgentFold [82] extend this idea by allowing agents to branch and fold sub-trajectories into concise representations, treating trajectories as dynamic workspaces rather than static logs. ACON [83] 34

further optimizes compression prompts using failure cases. MemGPT [34] lies at the boundary between context and external memory, introducing OS-style memory paging abstractions. Despite their effectiveness, context-based approaches fundamentally rely on lossy compression, which may discard information required at later stages. B.1.2

Agentic Memory

Agentic memory treats memory operations as part of the agent’s action space. Rather than passively compressing context, the agent actively decides what to store, update, retrieve, or discard. Early work adopts prompted memory updates. Reflexion [84] stores verbal feedback across trials; Generative Agents [85] maintain a memory stream enriched with reflections; Voyager [86] builds a library of reusable skills; and ExpeL [87] extracts reusable insights from trajectory comparisons. MemoryBank [88] introduces a forgetting mechanism inspired by human memory, while Mem0 [89] and A-MEM [35] formalize structured memory operations over note-like representations. More recent work learns memory policies via reinforcement learning. MemoryLLM [90] introduces latent memory tokens updated end-to-end during inference. MemAgent [91] learns overwrite policies over fixed-length memory under long-horizon rewards. Mem-α [92] trains agents to operate structured memory through tool APIs, while MEM1 [93] learns to compress history into a compact state at each step. The key advantage of agentic memory lies in its adaptivity: agents can selectively preserve information that fixed compression might discard. However, it introduces challenges in credit assignment, training complexity, and evaluation of memory quality. B.1.3

RAG Memory

RAG-based memory externalizes storage to an embedding or graph-based memory system and retrieves relevant information on demand. This paradigm traces back to Memory Networks [94] and is widely adopted in modern retrieval-augmented language models such as RAG [95], REALM [96], RETRO [97], and Atlas [98]. Extending RAG to agent settings introduces additional control mechanisms. SCM [99] adds a memory controller to decide when to retrieve, while Self-RAG [100] integrates retrieval decisions into the model’s generation process. EM-LLM [101] segments token streams into episodic events for scalable retrieval. Graph-based approaches further enrich memory structure. HippoRAG [102] and its successor [103] use graph traversal for multi-hop reasoning, while GraphRAG [104] constructs hierarchical summaries for global queries. LightRAG [105] supports dual-level retrieval, and Zep [106] introduces temporal knowledge graphs with validity intervals. Compared to context and agentic memory, RAG memory scales to large corpora and supports continual updates. However, its effectiveness depends heavily on retrieval quality, and bridging the gap between similarity-based retrieval and task-relevant reasoning remains an open challenge. B.1.4

Position

In this work, we focus on systematically understanding the role of memory in personalized agent behavior. To this end, we provide a unified and extensible memory interface that supports different classes of memory mechanisms, including context-based, agentic, and retrieval-based memory. This design allows us to isolate and compare how different memory paradigms influence the agent’s ability to infer, utilize, and update user preferences over time. Our goal is to study memory as a key factor in personalization and proactive decision-making. By placing different memory mechanisms under a shared evaluation framework, VitaBench 2.0 enables controlled and interpretable analysis of how memory design affects long-horizon user modeling and agent performance. B.2

Code of Ethics

This work complies with the NeurIPS Code of Ethics. Our research focuses on the design and evaluation of benchmark datasets for personalized and proactive agent behavior, without involving human subjects, sensitive personal data, or real-world deployment. All data used in VitaBench 2.0 are either manually annotated or synthetically generated. For manually annotated data, we follow strict internal guidelines to ensure that no personally identifiable or sensitive information is included. For synthetically generated data, all content is reviewed and refined by human annotators to ensure quality, consistency, and the absence of harmful or inappropriate content. We do not release any private or user-identifiable data, and the benchmark is constructed to simulate realistic scenarios without exposing real individuals or proprietary information. 35

Table 2: Per-user statistics of tasks, environment entities, and preferences in VitaBench 2.0.

B.3

Metric

Min

Max

Avg

Total

Tasks Envs Preferences

10 405 25

20 1,051 68

14.6 739.5 40.8

819 41,414 2,286

Broader Impacts

This work aims to advance the evaluation of personalized and proactive agents in realistic settings, which can benefit the development of more reliable and user-aligned AI assistants. Improved personalization and robustness may enhance user experience in applications such as recommendation systems, digital assistants, and decision support tools. However, such capabilities may also introduce potential risks. For example, more effective personalization could be misused to manipulate user behavior, reinforce existing biases, or enable overly persuasive systems. In addition, errors in preference inference may lead to inappropriate or misleading recommendations, particularly in highstakes scenarios. To mitigate these risks, our benchmark is designed as an evaluation framework rather than a deployable system. It does not include real user data, and all scenarios are constructed through controlled simulation. We encourage future work to incorporate safeguards such as transparency, user control over personalization, and monitoring mechanisms to prevent misuse. B.4

Safeguards

VitaBench 2.0 is constructed using a combination of manual annotation and large language modelbased synthesis. To ensure responsible data release, we adopt a multi-stage quality control process. For manually annotated data, we follow strict guidelines to ensure that all content is free from sensitive, personal, or harmful information. For synthetically generated data, we first generate candidate samples using large language models, and then apply human verification and refinement to ensure correctness, consistency, and safety. All interaction histories, user profiles, and task environments are reviewed to remove unintended biases, sensitive content, or unrealistic artifacts. The final dataset does not contain real user data and is designed to minimize risks related to privacy, misuse, or harmful content generation.

C

Analysis

C.1

Experiments Compute Resources

Each evaluation on Vitabench 2.0 covers 56 tasks, amounting to approximately 819 subtask-level interactions (with an average of 14.6 subtasks per task, including multi-turn follow-ups). For statistical rigor, we repeat the full evaluation over 4 independent trials, yielding roughly 3,276 interactions per (model, memory) configuration in total. Since all models are accessed through commercial closed-source APIs, our evaluation imposes no intensive local CPU or GPU demands. The wall-clock cost is instead bounded by API latency and rate limits. Specifically, we cap the request rate at 200 requests per minute (RPM) and run at most 20 concurrent asynchronous user-agent rollouts per evaluation. Under this configuration, and in the most demanding setting—full context, where the agent is provided with the full dialogue history, i.e., the complete sequence of all prior interactions across turns. The wall-clock time per (model, memory) configuration is as follows: • Non-reasoning models (e.g., , LongCat-Flash-Chat): approximately 4.3–4.6 hours; • Reasoning-enabled models (e.g., Claude-Sonnet-4.5, DeepSeek-V4-Pro, GLM-5.1, Gemini-2.5-Pro): approximately 3.7–8.6 hours C.2

Benchmark Data Analysis

VitaBench 2.0 is designed to evaluate personalized and proactive agents in realistic daily-life scenarios. To ensure the benchmark reflects practical deployment settings, the construction of user 36

Figure 6: Overview of user profile statistics in VitaBench 2.0. profiles, preferences, and interaction histories is guided by detailed statistics derived from real-world application scenarios. The benchmark consists of 56 users with a total of 819 tasks, 41,414 environment entities, and 2,286 preference annotations (Table 2). On average, each user is associated with 14.6 tasks, 739.5 environment entities, and 40.8 preferences, indicating a rich and structured user representation. In the following, we analyze the dataset from three aspects: user profiles, user preferences, and interaction patterns. C.2.1

User Profile Analysis

User profiles in VitaBench 2.0 are constructed to approximate the structural properties of real-world users in online life-service applications. As illustrated in Figure 6, each user is described along multiple dimensions, including demographic attributes, geographic distribution, socioeconomic status, occupation, and social context. Rather than relying on simplified or synthetic distributions, these attributes are designed to follow statistics observed in real-world scenarios, enabling the benchmark to capture realistic heterogeneity in user characteristics. We analyze the resulting user population from three perspectives: demographic distribution, regional and socioeconomic structure, and occupation and social context. Demographic distribution. The dataset reflects a population structure dominated by active users of modern online platforms. In terms of gender, 62.5% of users are female and 37.5% are male. The age distribution is concentrated in the 20–29 (62.5%) and 30–39 (26.8%) groups, with smaller proportions in ≤19 (1.8%), 40–49 (3.6%), and ≥50 (5.4%). The average age is approximately 31, and users born after 1990 account for 87.5% of the dataset, among which Generation Z (1995–2009) alone contributes 71.4%. This reflects the fact that younger users constitute the primary participants in online consumption scenarios, while the inclusion of younger and older groups ensures coverage of less frequent but behaviorally distinct segments. Regional and socioeconomic structure. Users are distributed across different levels of urban development to capture heterogeneous consumption behaviors. Approximately 50% of users are from emerging first-tier regions, 28.6% from first-tier cities, and 21.4% from lower-tier regions. This distribution introduces variation in consumption frequency, spending power, and price sensitivity, ranging from high-frequency urban consumption to more conservative decision patterns in lower-tier regions. In addition, 59% of users exhibit cross-region mobility (i.e., mismatch between registered 37

Figure 7: Overview of user preference statistics in VitaBench 2.0.

location and residence), which creates scenarios where user preferences are influenced by multiple geographic contexts. Such mobility further increases the complexity of preference modeling, as agents must generalize across location-dependent behaviors. Occupation and social context. The dataset covers a wide range of occupational and social backgrounds. Users include 60.7% white-collar workers, 16.1% blue-collar workers, and 23.2% gray-collar workers, spanning industries such as technology, education, administration, healthcare, manufacturing, and service sectors. This occupational diversity introduces variation in lifestyle patterns, time constraints, and consumption habits. In addition, users are associated with diverse family structures, including single individuals, couples, and families with children (12.5%), as well as different generational settings such as single-child (9%) and multi-child (14.3%) families. We further incorporate life-stage transitions (e.g., from student to employee, or from single to married), which affect both preferences and decision contexts. Implications for personalization. Overall, the combination of demographic concentration (toward active user groups) and structural diversity (across regions, occupations, and family contexts) results in a user population that is both representative and heterogeneous. This design ensures that personalization cannot be reduced to simple demographic heuristics, but instead requires agents to model fine-grained, context-dependent user characteristics and adapt their decisions across diverse user profiles. C.2.2

User Preference Analysis

User preferences are the core source of personalization difficulty in VitaBench 2.0. To support finegrained user modeling, we construct 17,928 preference annotations in total, covering 2,048 distinct preference types across five major categories. As illustrated in Figure 7, each user is associated with 40.8 preferences on average, spanning food, shopping, travel, leisure and entertainment, and special long-tail preferences. We analyze the preference space from four aspects: category distribution, temporal evolution, personalization diversity, and consumption structure. 38

Table 3: Distribution of user preference categories in VitaBench 2.0. Category Food-related Shopping Travel Entertainment Special preferences

Proportion

Example dimensions

35.0% 18.1% 20.2% 21.0% 5.0%

Cuisine, taste, beverages, dietary restrictions Product category, brand, price sensitivity Transportation, accommodation, travel style Leisure activities, social activities Long-tail or context-specific preferences

Table 4: Consumption distribution across major service scenarios. Metric Order proportion Avg. order amount

Online delivery

In-store

Hotel

Travel ticket

Attraction

Other

57.7% 99.7

23.2% 239.4

5.0% 1567.3

4.9% 2820.0

3.8% 164.2

5.4% –

Preference scale and category coverage. Table 3 summarizes the distribution of preference categories. Food-related preferences account for the largest proportion (35%), reflecting the central role of dining scenarios in daily-life service platforms. These preferences cover fine-grained dimensions such as cuisine type, taste, beverage choices, and dietary restrictions. Shopping (18.1%), travel (20.2%), and entertainment (21.0%) preferences are more evenly distributed, each accounting for roughly one fifth of all preferences. This balanced coverage ensures that the benchmark does not collapse into a single dominant scenario, but instead evaluates personalization across diverse daily activities. In addition, 5% of preferences correspond to special long-tail cases, such as uncommon dislikes or context-specific habits, which further increase the difficulty of user modeling. Dynamic preference evolution. Preferences in VitaBench 2.0 are not static labels, but evolve over the user life cycle. On average, each user experiences 48.38 preference changes, including 36.07 additions, 7.52 deletions, and 4.79 modifications. Preference additions dominate the evolution process, reflecting the fact that user interests usually expand as new life events, consumption scenarios, and habits emerge. Meanwhile, deletions and modifications introduce non-monotonic changes, requiring agents to avoid treating all historical preferences as permanently valid. This design evaluates whether an agent can maintain an up-to-date user representation rather than merely accumulating all historical signals. Preference diversity and sparsity. The preference space is highly individualized. Approximately 92.4% of preferences are user-specific, while only 7.6% are shared across users. Shared preferences mainly correspond to generic habits, such as common beverage preferences, whereas most preferences are tied to a particular user’s lifestyle, health conditions, social context, or consumption habits. This long-tail structure prevents agents from relying on population-level shortcuts and requires them to infer fine-grained personalized signals from each user’s own interaction history. Consumption structure. We further analyze whether the constructed preference distribution aligns with realistic consumption patterns. Online delivery and in-store consumption account for 57.7% and 23.2% of orders, respectively, together contributing over 80% of user activities. This reflects the high-frequency daily consumption structure of real-world service platforms. Lower-frequency scenarios such as hotels (5.0%), travel tickets (4.9%), and attractions (3.8%) appear less often but involve substantially higher average order amounts, reaching 1,567.3 for hotels and 2,820 for travel tickets. This design creates a mixture of high-frequency low-value scenarios and low-frequency highvalue scenarios, allowing the benchmark to evaluate preference modeling across different decision contexts. Consumption-level diversity. Within each scenario, we further divide users into high-, medium-, and low-consumption groups to ensure sufficient variation in price sensitivity and decision behavior. For example, in online delivery, 19.6% of users fall into the high-consumption group, 62.5% into the medium-consumption group, and 17.9% into the low-consumption group. For in-store consumption, the corresponding proportions are 14.3%, 75.0%, and 10.7%. For hotels, the distribution is 33.9%, 46.4%, and 19.6%, while travel tickets show a stronger skew toward low-consumption users (44.6%). 39

Table 5: Distribution of consumption levels within major service scenarios. Consumption level High Medium Low

Online delivery

In-store

Hotel

Travel ticket

Attraction

11 (19.6%) 35 (62.5%) 10 (17.9%)

8 (14.3%) 42 (75.0%) 6 (10.7%)

19 (33.9%) 26 (46.4%) 11 (19.6%)

13 (23.2%) 18 (32.1%) 25 (44.6%)

16 (30.8%) 17 (32.7%) 19 (36.5%)

This layered consumption design ensures that agents must model not only categorical preferences, but also price sensitivity and consumption level within each domain. C.2.3

Interaction Analysis

Interaction histories constitute the primary source from which agents must infer user preferences. We therefore construct interaction data to exhibit realistic long-horizon, high-density, and noisy behavioral patterns, closely matching real-world user activity. Long-horizon interaction patterns. User interactions span extended time horizons, with timelines covering more than 10 years in total. For each user, the average interaction duration is 1,580 days (approximately 4.3 years), with the longest reaching 2,974 days (8.1 years). The lifecycle distribution follows a realistic pattern: medium-term users (3–6 years) account for approximately 75% of the population, while short-term and long-term users each account for 12.5%. Such long-horizon coverage enables the benchmark to evaluate whether agents can track preference formation, stabilization, and drift over extended periods, rather than relying on short-term signals. Behavioral complexity and density. User interactions are highly dense and heterogeneous. On average, each user generates 2,093 interaction events, including browsing, searching, consulting, comparing, and purchasing behaviors. Among these, 221 interactions correspond to successful purchase conversions, indicating that most interactions are exploratory rather than goal-completing. Importantly, decision-making processes are not uniform: some tasks require multiple rounds of information gathering and comparison before conversion, while others are resolved with minimal interaction. The same underlying intent may also span multiple sessions, leading to fragmented and non-contiguous evidence for preference inference. This diversity in decision trajectories significantly increases the difficulty of modeling user intent from interaction histories. Cross-domain and multi-scenario behavior. User interactions naturally span multiple service domains, including online delivery, in-store consumption, hotels, travel booking, and attractions. Different scenarios exhibit distinct behavioral patterns: high-frequency, low-cost activities (e.g., food delivery) coexist with low-frequency, high-cost decisions (e.g., travel booking). This cross-domain structure requires agents to generalize preference signals across heterogeneous contexts, rather than relying on domain-specific heuristics. Noise and uncertainty. To further reflect real-world conditions, interaction histories include approximately 20% noisy behaviors that do not directly correspond to true user preferences. These noisy interactions are carefully designed to mimic realistic but misleading signals, including: (i) irrelevant interactions, such as casual conversations or browsing unrelated to consumption intent; (ii) exploratory behavior, where users browse or search without a clear objective; (iii) proxy actions, such as placing orders on behalf of others; (iv) impulsive or short-lived interests, which do not persist over time; and (v) inconsistent or corrective actions, such as cancellations or repeated submissions. These noise patterns are structurally similar to genuine interactions in form (e.g., search, click, purchase), but are uninformative or even misleading with respect to true preferences. As a result, agents must perform robust signal extraction under uncertainty, distinguishing stable preference signals from transient or irrelevant behaviors. Implications for evaluation. The combination of long temporal span, high interaction density, cross-domain coverage, and structured noise results in a challenging setting for preference inference. Agents cannot rely on single interactions or short-term patterns, but must aggregate fragmented evidence over time, handle conflicting signals, and maintain consistent user representations under 40

uncertainty. This design enables rigorous evaluation of long-term memory, preference tracking, and robustness to noisy observations. C.2.4

Implementation Configurations

Agentic Memory. Following the settings of MemAgent [21], our agentic memory backend treats UPDATE as an LLM-driven rewrite rather than an append-only log: at each update step the model is shown the current memory and the newly arrived interaction batch, and is asked to produce a single consolidated preference summary that supersedes the previous one. This forces the agent to explicitly decide what to retain, merge, or discard, giving it active control over the long-term representation. We cap the rewritten memory at a 4,096-token buffer, which we found sufficient to cover the preference state of a user across the full subtask sequence while remaining small enough to fit in the agent’s system prompt. Crucially, this budget is not only enforced at the decoding API level but is also explicitly stated inside the prompt, so the model is aware of the output length it must target when consolidating memory. The preference prompt is a structured template that instructs the model to (i) retain valid information from the existing memory, (ii) update entries that conflict with new observations, and (iii) add newly discovered preferences, organized along canonical axes such as food taste, spending habits, time and location preferences, and service-specific requirements. RETRIEVE simply returns the current memory blob, since the rewrite step has already produced a selective abstraction. The full prompt template is shown in Box C.2.4. Agentic Memory Update Prompt [System] You are a preference memory manager. Your job is to maintain an accurate, concise summary of user preferences based on their interaction history. [User] You are a user preference memory manager. Your task is to maintain an accurate and concise summary of user preferences based on their interaction history. ## Current Memory Content: {current_memory} ## New User Interaction Records: {new_interactions} ## Task Requirements: Please merge the new interaction records with the existing memory and generate an updated user preference summary. Requirements: 1. Retain valid preference information from existing memory 2. Update old preferences that contradict new interactions 3. Add new preferences discovered from new interactions 4. Structure the preference information, including but not limited to: - Food preferences (taste, cuisine, ingredient preferences, etc.) - Spending habits (price range, frequency, etc.) - Time preferences (meal times, travel times, etc.) - Location preferences (frequent areas, distance preferences, etc.) - Service preferences (delivery requirements, seating preferences, etc.) - Other preferences Please output the updated preference summary directly, without any explanatory text. (Please keep the updated summary within about 4096 tokens.)

RAG Memory. For a fair comparison across runs, the RAG backend uses a single fixed embedding and retrieval configuration. All interaction records are embedded with text-embedding-3-large, chunked into fixed-size windows of 512 tokens with zero overlap using the cl100k_base tokenizer. UPDATE embeds and indexes each new chunk asynchronously; no LLM summarization 41

is performed, so the pipeline is fully deterministic once embeddings are computed. At query time, RETRIEVE embeds the task instruction, ranks stored chunks by cosine similarity, and returns the top k = 8 chunks after filtering out any chunk whose similarity falls below a threshold of 0.3. The threshold is set as a conservative lower bound: scores from text-embedding-3-large cluster above 0.25 for only loosely related text and above 0.5 for directly relevant content, so 0.3 removes unambiguous distractors while preserving recall. Unlike the agentic backend, the RAG backend has no explicit control over what is kept or discarded – memory access is entirely determined by the retrieval score.

D

Trajectory

D.1

Case Study: Memory-Sensitive Delivery for User A891207

The boxes below use a simple colour scheme: blue for user utterances, green for assistant turns, orange for tool calls, purple for task metadata, grey for environment / preference blocks. To save space we summarise tool arguments and responses rather than dumping raw JSON, and we keep only the target store and two representative distractors from the environment. All Chinese strings have been translated to English; the field structure and numeric values are verbatim from the benchmark record. User A891207 is a 26-year-old Party affairs officer living in Jinzhou (Liaoning). Across her 20subtask trajectory her home address at Hongye Fengjing No. 2, Apt. 101 is stable, but her dietary preferences drift meaningfully over the year: she switches from spicy to light Cantonese food in March 2027, explicitly states a love of durian in the summer, and later narrows her meat preference to exclude pork. We pick three subtasks from this trajectory (sub_A891207_13, sub_A891207_17, sub_A891207_20) where the three memory settings in our matrix (Full Context, Agentic Memory, RAG Memory) diverge, to illustrate (i) how a single preference error inside memory translates into a rubric failure, and (ii) how each setting represents a drifting preference over time. D.1.1

Part 1. One Subtask, Three Backends

The focal subtask is sub_A891207_17, an evening dessert delivery order at home. The instruction is deliberately preference-laden and under-specified: the rubric contains seven criteria, most of which must be recovered from memory rather than from the instruction. task subtask_id: sub_A891207_17 current_time: 2027-09-04 domain: delivery instruction: “Not sure if my period is coming soon, I’m craving sweets!! Please order me a bowl of tong sui, Cantonese-style, room-temperature is fine, deliver to home.”

rubric (seven binary criteria) R1. the ordered item must be a Cantonese-style tong sui R2. the ordered item must be served at room temperature R3. the ordered item must contain durian R4. the ordered quantity must equal 1 R5. the merchant’s delivery time must be ≤ 30 min R6. the merchant’s rating must be ≥ 4.3 R7. delivery address must equal the user’s home address

R3 (durian) and R6 (rating ≥ 4.3) are the two criteria that are not derivable from the instruction text at all: the user does not mention durian, and there is no numeric rating threshold in the message. Both come from her preference memory (“likes durian”, “for delivery, prefers merchants rated ≥ 4.3”). Preference Memory and Environment. The ground-truth preference snapshot at t = 2027-09-04 contains a dietary block with ten tags; the three relevant to this subtask are italicised below.

42

user_scenario.personalized_preference_memory.current.dietary (GT) prefers light/fresh flavours; carnivore; likes sugar-and-oil-free French bread; skips breakfast; for delivery, prefers merchants delivering within 30 min; dislikes chain bakeries; for delivery, prefers merchants rated ≥ 4.3; for dine-in, prefers merchants within 5 km; likes Cantonese cuisine; likes durian

The environment for this subtask contains 42 candidate stores in Jinzhou, each labelled target or distraction with an explicit distraction_reason. For readability we show only the target and two representative distractors: environment.stores (3 of 42 shown) S00027 — Gangshi Sweet-Dessert Shop (Linghe). rating 5.0, delivery 25 min. Products include Cantonese durian tong sui (room-temperature) at ¥26. STORE _ TYPE : TARGET. S00017 — Guangfu Tong-Sui Store (Flagship). rating 4.0, delivery 26 min. Products include Cantonese durian tong sui (room-temperature) at ¥24. STORE _ TYPE : DISTRACTION. Distraction reason: rating 4.0 < 4.3, violates R6. S00011 — Traditional Cantonese Tong-Sui Shop. rating 4.2, delivery 23 min. Products include mango tong sui (no durian). STORE _ TYPE : DISTRACTION. Distraction reason: rating 4.2 < 4.3 and no durian, violates R3 & R6.

Crucially, S00027 (target, 5.0) and S00017 (distraction, 4.0) sell a product with the same name and description (“Cantonese durian tong sui, room-temperature”). They differ only in merchant rating. Separating the two requires the agent to remember the ≥ 4.3 rating preference and to actually read the rating field in the tool output. Rollouts. We replay the same subtask under the three backends and report the trajectory in condensed form. Tool arguments that repeat verbatim across calls (e.g. the fixed home-address geocoding) are omitted after the first appearance; every create_delivery_order call is shown in full. (a) Agentic Memory — reward 1.0. user Not sure if my period is coming soon, I’m craving sweets!! Please order me a bowl of tong sui, Cantonesestyle, room-temperature is fine.

assistant → tool calls address_to_longitude_latitude(home) → (121.16, 41.13); delivery_store_search_recommand(keywords=[“Cantonese tong sui”, “double-skin milk”, “mango pomelo sago”, . . . ]) → 6 stores returned including S00027 (target, 5.0) and S00003 (Guangji Tong-Sui, 4.9); get_delivery_store_info on the three top-rated stores → menus including durian tong sui at S00027.

assistant (turn 23) “Three high-rated choices: Gangshi (5.0, 25 min, durian tong sui ¥26), Guangji (4.9, 26 min, ginger-milk ¥23), Zhengzong (4.8, 26 min). Which one?”

user You pick, any is fine.

assistant (proactively recalls “likes durian”) “The durian one, no question — you’re a durian fiend.”

create_delivery_order store_id=S00027, product=“Cantonese durian tong sui (room-temperature)”, qty=1, address=home → order ¥26, delivery 25 min. All seven rubrics satisfied, reward 1.0.

(b) Full Context — reward 1.0. The Full Context backend behaves analogously: it enumerates five stores with rating ≥ 4.9, compares delivery times, and selects S00027. The final order is identical to (a). Reward 1.0. (c) RAG Memory — reward 0.0.

43

user Not sure if my period is coming soon, I’m craving sweets!! Please order me a bowl of tong sui, Cantonesestyle, room-temperature is fine.

assistant → tool calls delivery_product_search_recommand(keywords=[“Cantonese tong sui”, . . . ]); address_to_longitude_latitude(home). No get_delivery_store_info call is issued — the agent skips rating verification and proceeds directly to ordering.

assistant (turn 4; fabricated rating) “Found several Cantonese tong sui shops! Recommended: Guangfu Tong-Sui Store — Cantonese durian tong sui, ¥24, delivery ≈ 20 min. You’re a durian fiend, I remember!” (Note: Guangfu’s actual rating is 4.0. It does not appear in the shown top-three, and the agent never retrieved its rating.)

create_delivery_order store_id=S00017, product=“Cantonese durian tong sui (room-temperature)”, qty=1, address=home → order ¥24, delivery 26 min. R1–R5, R7 satisfied; R6 (rating ≥ 4.3) violated because S00017 is rated 4.0. Reward 0.0.

Takeaway. The three backends diverge on a single latent preference (“delivery merchants rated ≥ 4.3”). Agentic Memory and Full Context both surface this preference before the decision step: Agentic Memory has it in the consolidated summary as a “hard threshold ≥ 4.3”, and Full Context carries every past order (all of which were placed at ≥ 4.3 merchants), which lets the agent apply the constraint implicitly. RAG Memory retrieves only a handful of past interaction chunks by cosine similarity to the query “Cantonese tong sui”: the relevant chunks mention durian but not the rating threshold, so the agent commits to the first plausible store without a rating check. The resulting order is coherent and well-motivated but fails R6. D.1.2

Part 2. Preference Drift Across Three Subtasks

Subtask sub_A891207_17 is not an isolated failure: the rating-threshold preference, along with several other dietary tags, drifts through the user’s trajectory and different memory backends preserve the drift to different degrees. Table 6 traces the dietary preference block at three timestamps (subtasks 13, 17, 20 from the user’s sequence) and aligns each ground-truth tag against what Agentic Memory and RAG Memory actually surfaced at that timestamp. Green marks a GT preference that the backend represents faithfully; Red marks a GT preference that is missing, contradicted, or drowned in stale content. All rows reflect snapshots taken immediately before the corresponding subtask. Reading the table. The drift pattern explains why Agentic Memory consistently outperforms RAG Memory on this user. Agentic Memory carries an evolving, deduplicated summary: stale preferences (spicy food, pork) are explicitly crossed out, newly created preferences (Cantonese, durian, pork exclusion) are added, and stable thresholds (rating ≥ 4.3, delivery ≤ 30 min) are kept even when they are not relevant to the current instruction. RAG Memory, in contrast, returns the chunks most similar to the current query and has no mechanism to promote stable-but-off-topic thresholds or to demote outdated chunks. When the query keyword matches an old record (“spicy Sichuan”, “pork trotter hotpot”) the RAG snapshot can actively mislead the agent; when it does not match any record (sub20’s “scissors” query) the RAG snapshot simply loses the dietary context altogether. The net effect on this user is a reward gap of 0.18 between Agentic Memory and RAG Memory aggregated over her 20 subtasks, driven primarily by memory slots that were present but not retrieved.

44

Table 6: Preference drift for user A891207 across three subtasks. GT = ground-truth dietary preferences active at that timestamp. Agentic Memory shows what the LLM-consolidated summary lists in its dietary section. RAG Memory shows the top retrieved records (or whether the preference is derivable from them). Relevant GT tag

Subtask 13 (t=2027-06-05) Agentic / RAG

Subtask 17 (t=2027-09-04) Agentic / RAG

Subtask 20 (t=2027-12-25) Agentic / RAG

Light/fresh flavour ! Agentic: explicit transi(switched from spicy tion note, “fully switched to in Mar. 2027) light Cantonese” % RAG: top chunks are old spicy orders (hotpot, sourand-spicy noodles) from 2026

! Agentic: “absolute light flavour, completely stopped spicy food” % RAG: retrieves a 2023 spicy Sichuan order as the top match for “dessert craving”

! Agentic: “light flavour, no pork as an exception” ◦ RAG: no relevant retrieval (query is about scissors)

Delivery merchant rating ≥ ! Agentic: “rating ≥ 4.3 4.3 (stable across all 3 sub- hard gate” tasks) % RAG: retrieved chunks mention ratings of 4.5 / 4.7 but no threshold statement

! Agentic: “rating ≥ 4.3 hard gate, unchanged” % RAG: threshold not retrieved; agent orders from a 4.0 merchant (sub17 failure)

! Agentic: “≥ 4.3” % RAG: not retrieved; agent picks a 4.7 store by luck, still passes

! Agentic: “≤ 30 min” ! Agentic: “≤ 30 min” ◦ RAG: one retrieved chunk ◦ RAG: one retrieved chunk explicitly says “within mentions 28 min delivery 30 min”

! Agentic: “flash-purchase ≤ 35 min” % RAG: only 30-min threshold retrieved; the 35-min flash-purchase rule created later is not retrieved

Likes Cantonese cuisine ! Agentic: “Cantonese (created Mar. 2027) dominates, all delivery / travel / dining” % RAG: top chunks are 2026 Sichuan / Hunan hotpot orders

! Agentic: Cantonese as core cuisine ! RAG: retrieves a 2027-07 chunk with “I still prefer Cantonese light flavour”

! Agentic: Cantonese, with pork exclusion ◦ RAG: not relevant to the scissors query

Likes durian (created sum- — not yet in GT mer 2027; directly tested by R3 in sub17)

! Agentic: “durian fiend, high-frequency purchase” ! RAG: retrieves a 2027-0901 chunk “durian fiend, watch your intake”

! Agentic: “durian, continued” ◦ RAG: not retrieved (query is about scissors)

Carnivore, no pork — GT was still “carnivore” (updated in late 2027)

— GT was still “carnivore”

! Agentic: “carnivore excluding pork, updated” % RAG: retrieved chunks include old pork-trotter hotpot orders; the pork-exclusion update is not surfaced

Delivery time ≤ 30 min (sub13, sub17); ≤ 35 min for flash-purchase (sub20)

45