RUMBA: Russian User Memory Benchmark Elizaveta Shevtsova1 , Inna Glebkina1 , Mark Baushenko1 , Pavel Gulyaev1 , Alena Fenogenova1 1 DAIMLD Correspondence: [email protected]
arXiv:2607.21447v1 [cs.CL] 23 Jul 2026
Abstract
vide incomplete coverage of memory behaviours required in realistic long-horizon interactions, including memory updates and consolidation, stale or obsolete information, and forgetting (Du, 2026; Hu et al., 2026; Uddin et al., 2026). As summarizes in Table 1, existing benchmarks also rely on synthetic dialogue construction and offer limited diagnostic granularity through relatively coarse question categories. This makes it difficult to isolate failure factors such as temporal dependencies, conversational span, or query semantics.
The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) — a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user–assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.
1
Meanwhile, Russian benchmarks such as MERA (Fenogenova et al., 2024) and LIBRA (Churin et al., 2025) evaluate broader model capabilities, including general language understanding and long-context processing, but not conversational memory in a targeted way. This leaves a clear need for a benchmark focused on memory, temporal awareness, and reasoning in long Russian dialogues. To address this gap, we introduce RUMBA (Russian User Memory Benchmark), a benchmark for evaluating long-term conversational memory in Russian. RUMBA is built around long-form dialogue scenarios and a taxonomy of question types covering contextual recall, entity tracking, temporal reasoning, and multi-step reasoning. We provide two test sets (Russian and an aligned English translation) and conduct evaluations under both RAG and full-context scenarios.
Introduction
Large language models (LLMs) are increasingly deployed as conversational assistants that must retain user-specific information across sessions and remain consistent over time. This requires more than simple retrieval: models must preserve salient details, resolve long-range references, and reason over temporal updates and multi-step dependencies.
Our contributions are as follows: • We create and release RUMBA as an opensource benchmark1 , the first benchmark specifically designed to evaluate long-term conversational memory in Russian, accompanied by an aligned English translation to
Recent English-centric memory benchmarks (Maharana et al., 2024; Wu et al., 2025; Bei et al., 2026) have advanced long-term memory evaluation. However, recent work on memory-agent evaluation notes that existing memory benchmarks are still largely recall- or retrieval-oriented and pro-
1
https://huggingface.co/datasets/ai-forever/ RUMBA
1
enable cross-lingual diagnostics and reproducibility. • We present a fine-grained taxonomy of memory-centric question types along multiple axes, enabling detailed model comparison beyond aggregate accuracy and simple retrieval. • We implement a unified evaluation pipeline that supports both retrieval-based memory systems and full-context baselines for both languages, which standardizes the evaluation workflow while allowing each method family to operate under its natural memory-access setting2 . • We conduct an extensive empirical analysis of different families of memory solutions and a single-model case study, demonstrating RUMBA as a diagnostic tool for identifying strengths and failure modes across benchmark slices.
dates as isolated task types rather than dimensions that can interact; and (3) ignoring memory control behaviors such as intentional forgetting or validity constraints. Several benchmarks (e.g., TimeBench (Chu et al., 2024), temporal NLI (Vashishtha et al., 2020), and the three-dimensional temporal QA framework (Piryani et al., 2026)) evaluate temporal reasoning in LLMs, covering event ordering, duration, and temporal commonsense. However, they are not designed for long-context settings nor grounded in realistic user–assistant interactions, and thus fail to capture temporally dependent reasoning across extended multi-session conversations. The gap is even more pronounced for Russian: no dedicated benchmark for persistent long-term memory exists. LIBRA (Churin et al., 2025) evaluates long-context understanding, but does not test structured memory usage over time. MERA (Fenogenova et al., 2024) focuses on general reasoning and knowledge tasks, without isolating memoryspecific abilities. The only partially relevant dataset, ruTiE (Chervyakov et al., 2026), includes conversational evaluation, but operates on short dialogues and does not capture persistent, multisession memory phenomena.
We hope that RUMBA will serve both as a diagnostic benchmark for evaluating memory in Russianlanguage conversational systems and as a practical resource for developing models that can reliably store, update, and use user-specific information over time.
2
Related work
Recent work on evaluating memory capabilities in language models has been shaped by benchmarks such as LoCoMo (Maharana et al., 2024), LongMemEval (Wu et al., 2025), Mem-Gallery (Bei et al., 2026). These datasets provide a structured starting point for studying long-term memory in conversational systems. In particular, LoCoMo organizes evaluation around reasoning-oriented question families (e.g., single-hop, multi-hop, temporal, commonsense, adversarial), while LongMemEval defines a set of core memory abilities (e.g., recall, personalization, multi-session reasoning, updates, temporal reasoning, abstention) and operationalizes them through specific question types. Together, these benchmarks establish a useful conceptual and methodological baseline for memory evaluation and, particularly, for the RUMBA taxonomy.
To address all of the above limitations (flat taxonomies, lack of interactive temporal dimensions, absence of forgetting scenarios, inapplicability to long-context dialogues, and the complete gap for Russian), we propose a benchmark for Russian with an extended taxonomy covering forgetting and broader reasoning types, multi-axis annotations for fine-grained diagnostics, and human-written user turns to reduce synthetic interaction bias.
RUMBA Methodology
3.1
Overview
RUMBA was designed around a set of diagnostic research questions that directly address the limitations identified in prior work. The goal is not only to produce an aggregate score but to make model evaluation informative about specific memory capabilities and failure modes. Accordingly, dialogues, questions, answers, and evidence configurations were created to support controlled comparisons along the main dimensions of long-term conversational memory. The benchmark supports the following research questions:
However, they have three key limitations (Table 1 shows the overview): (1) flat taxonomies that entangle reasoning complexity, temporal structure, and memory access within single question categories; (2) treating temporal reasoning and knowledge up2
3
https://github.com/ai-forever/RUMBA
2
Figure 1: Overview of the proposed benchmark and annotation framework. The figure summarizes dataset-level statistics, representative user–assistant dialogues, example question–answer instances, and the multi-dimensional annotation scheme used to categorize memory questions. Feature Modality Language support Context length (chars) # Annotated Q-A pairs Question taxonomy User utterances origin Temporal reasoning as independent dimension Explicit support for "forgetting" Memory scenario coverage Date for each dialogue’s utterances Date for questions Failure diagnosis granularity
LoCoMo
LongMemEval
Mem-Gallery
RUMBA
Text/Image English only 44k–90k 1,542 Flat (5 types) Generated
Text English only S: 455k–514k / M: 4.5M–5.2M 500 Flat (5 types) Generated
Text/Image English only 42k–86k 1,711 Flat (3 types) Mixed
Text Russian + English RU: 297k–569k / EN: 234k–528k 1,543 Orthogonal axes (17 × 2 × 2 types) Human-written
No No retention, abstention
No No retention, abstention, updating
No No retention, abstention, updating
Yes (✓) Yes (✓) retention, abstention, updating, forgetting
Yes No
Yes Yes
Yes No
Yes Yes
Limited
Limited
Limited
Finer-grained (temporal vs. atemporal)
Table 1: Comparison of memory benchmarks: LoCoMo, LongMemEval, Mem-Gallery, and RUMBA. Modality: content types present in the dialogues; Language support: languages of the dialogues and questions; Context length: dialogue length measured in characters; # Annotated Q-A pairs: number of question-answer pairs associated with each dialogue; Question taxonomy: structure of the question taxonomy, distinguishing Flat taxonomies, where each question is assigned a single type, from Orthogonal axes, where each question receives multiple tags; RUMBA uses semantic × quantitative × temporal axes; User utterances: the primary source of user-side dialogue utterances, distinguishing LLM-generated, mixed-source, and human-written data; RUMBA uses manually written user utterances. Temporal reasoning as an independent dimension: whether one of the assigned tags indicates that a question is temporal or atemporal; Explicit support for “forgetting”: whether the benchmark includes scenarios involving deletion requests, such as “delete this” or “forget this”; Memory scenario coverage: high-level task categories used for memory evaluation; Dates for dialogue utterances and questions: whether each utterance and question is timestamped; Failure diagnosis granularity: the ability to identify which benchmark dimension contributes to question complexity.
1. Memory scope and semantics. How does performance vary with session scope (single vs. multiple sessions) and with semantic supergroups (extraction, reasoning)? 2. Temporal reasoning. How does performance differ between temporal and atemporal questions, and between cases where temporal information is explicit vs. implicit? 3. Retrieval vs. full-context How do retrievalaugmented memory systems, which trade off perfect recall for scalability, compare with full-context baselines that preserve all information within a finite window but cannot scale indefinitely?
3.2
Taxonomy
The RUMBA taxonomy is designed to address the limitations discussed in Sec.2 by moving from a flat, benchmark-specific categorization to a multidimensional, compositional framework for memory evaluation. Instead of assigning each question to a single class, RUMBA decomposes tasks along several orthogonal axes, including: • the semantic operation required (e.g., recall, update tracking, comparison), • the temporal dimension (temporal vs. atemporal), • the number of sessions involved, • the reasoning complexity, • and the memory validity state (e.g., active vs. forgotten information).
Fig.1 shows the overall structure of the proposed benchmark.
The questions are categorized according to three 3
axes: (1) Semantic axis represents the classification of questions with respect to their meaning and assessment focus. Broadly, the questions are classified into Information Extraction and Reasoning classes. (2) Quantitative axis indicates the number of sessions required to answer a given question. Questions are divided into single-session and multi-session classes. (3) Axis of temporality denotes whether a question is temporal. Questions are divided into temporal and atemporal classes. The summary of the axes and corresponding types of questions is presented in Table 2.
memory-enabled assistant systems. In particular, its structure was informed by observations from a deployed proprietary conversational assistant with long-term memory functionality, including naturally occurring user behaviors related to storing, updating, correcting, and deleting memories. Grounding the taxonomy in authentic interaction patterns allowed us to better capture the diversity and practical characteristics of real conversational memory queries beyond synthetic or narrowly constructed benchmark settings. Table 2: RUMBA Taxonomy Summary. Example of question’s full annotation: Question = What do I like about my job? T ypes = StaticUser, multi, atemporal. Full taxonomy description is available in Table 26 of Appendix.
This axis-based design enables a more precise characterization of model behavior. For example, the same underlying operation (e.g., recall) can be evaluated in both temporal and atemporal settings, revealing different failure modes. Similarly, reasoning is treated not as a separate category, but as a property that can be combined with different types of memory access, allowing finer-grained analysis of model capabilities.
Axis
Categories / Types
Semantic
Extraction class (6 types): StaticUser, UpdatingInfo, DeleteInfo, AsstQuery, OpenDomainType, DateExtraction Reasoning class (10 types): SocialRelationship, Ordering, Arithmetic, Comparison, UserQA, CalendarUnderstanding, TemporalCommonsense, TemporallyModifiedGeneralReasoning, ComplexRelations, OtherReasoning Abstention type: no answer in dialogue Types with subtypes: OtherReasoning, Arithmetic, DateExtraction, CalendarUnderstanding, TemporalCommonsense, TemporallyModifiedGeneralReasoning, ComplexRelations
A key strength of RUMBA is its treatment of temporality as an independent dimension. Unlike prior benchmarks, where temporal reasoning is isolated as a specific task type, RUMBA allows temporality to interact with all other axes. This makes it possible to systematically study how models handle time-dependent versus time-independent memory queries within the same semantic framework.
Quantitative
Single-session, Multi-session
Temporality
Atemporal, Temporal Temporal Expression Tags: explicit / implicit / no temporal expression
3.3
Dataset characteristics
Table 3 summarized the main characteristics and statistics of the Russian version of the dataset. The instances that comprise the dataset represent textual dialogues between an assistant (LLM) and a user, as well as questions associated with each dialogue that relate to its content.
Another important contribution is the explicit modeling of memory control and validity. In addition to standard abilities such as recall and update tracking, RUMBA introduces evaluation of intentional forgetting, where models must correctly exclude previously stored information after explicit user instructions. The forgetting scenario is supported through user behavior and a dedicated question type, DeleteInfo. Users may trigger it with requests such as “delete this” or “forget it.” The correct response is “There is no such information” (or similar), indicating that the information has been successfully forgotten and is no longer retained by the system.
Sessions. Each dialogue consists of timestamped sessions, each representing one day of interaction with the assistant. We distinguish evidence sessions, which contain information relevant to at least one question, from filler sessions, which include both conversations about the user that are irrelevant to questions and sessions with no user-related factual information. Evidence sessions constitute 52.77% of all sessions, micro-averaged across dialogues. This proportion does not alone characterize retrieval difficulty because most content in an evidence session may be irrelevant to a particular question.
This reflects realistic requirements for memoryenabled assistants and introduces a new class of failure modes not captured in prior work. A detailed taxonomy is provided in the Appendix A.
Questions. As described above, the question typology is based on three axes and covers different semantic question types. The dataset consists of
The taxonomy was designed with close consideration of real-world production user behavior in 4
Language RU EN
both open-ended and closed-ended questions. The question-level timestamp is of equal importance to the session-level timestamp, given that the correct interpretation of the former is critical to generating an accurate response.
Value / Description
Total dialogues Total questions Average dialogue length Utterances per dialogue Session definition
85 1,543 ≈ 340,000 characters 180 – 998 Single interaction with the assistant on a specific day 12 – 85 52.77% of all sessions (micro-average) 14 – 22 17 base types Contemporary standard Russian (user specific) Include language errors Reflect Russian sociolinguistic worldview None (no offensive, insulting, or threatening data)
Sessions per dialogue Evidence sessions Questions per dialogue Number of Question Types Language Dialogue specifics Offensive content
+ roles 344,758 324,437
+ answer prompt 346,751 326,430
Table 4: Context lengths in the Russian and English versions of RUMBA, measured in characters.
Dialogues Characters. The dialogue characters represent diverse behavioral scenarios, which increases the realism of the user–assistant interactions and reflects authentic user behavior. The dialogues also incorporate sociocultural features characteristic of the Russian language and culture. Information about the characters and topics is presented in Subsection B.2 of the Appendix. Feature
Dialogue text 341,103.13 320,782.08
thors, followed by a validation phase conducted by other contributors to ensure quality and consistency. More detailed information about the validation process can be found in Section B of Appendix. Before the start of data collection, all participants received comprehensive instructions (See Fig. 8 in Appendix), were acquainted with the question taxonomy, and were provided with the matrix. The dialogues were created in real time. User utterances were written by human participants, while GigaChat LLM was selected to generate the assistant’s responses, as it is specifically designed for the Russian language and Russian-speaking users, and is the only model pretrained on Russian data (Mamedov et al., 2025). The dialogue authors used the web version of GigaChat Max dated July 1–31 and October 21–November 10, 2025.
Note: Dialogues are independent of each other.
English version. We additionally provide an aligned English diagnostic split. Given the large size of the dialogue data, controlled manual translation would be too resource-intensive. Therefore, the English version of the dataset was produced using automated LLM translation, followed by human verification of the translation quality. This follows common practice in machine translation evaluation and multilingual benchmark construction, where full human assessment is costly and machine-translated data is often validated or postedited on selected subsets (Lo and Knowles, 2023; Singh et al., 2025b; Rajaee et al., 2025).
Table 3: RUMBA Dataset Features and statistics.
3.4
Dataset creation
Russian version. The dataset was developed in multiple stages, including taxonomy design and dialogue construction. Initially, information extraction questions — constituting the majority of the dataset — were created, subsequently, reasoning questions and a category of temporal questions were incorporated. The dialogue creation process involved a total of 26 contributors, comprising staff members and crowd workers (ABC Elementary3 and Rambler&Co companies4 ). Each participant received monetary compensation according to the terms and conditions specified in their respective employment or contractual agreements. Information about the participants (gender, age, and field of expertise) is presented in Fig.7 of Appendix.
For the assessment of translation quality, 10% of the translated dataset (i.e., 9 dialogues or a total of 185 sessions) was uniformly sampled and evaluated by human annotators. Translation quality was assessed using the Critical and Task-specific criteria defined in POLLUX (Martynov et al., 2025), with results summarized in Table 8. All 3,543 translated pairs RU-utterance — EN-utterance of the 9 dialogues were validated by domain-qualified specialists. Each evaluator compared the original Russian text with its English translation according to the POLLUX criteria, and every RU-EN pair was assessed by three annotators. The overall translation
Data were collected over 20 working days. The process comprised two main stages: the creation of dialogues and associated questions by the au3 4
https://elementary.center https://rambler-co.ru
5
quality score was estimated at 0.88 (Table 9).
etary and open-source long-context models. The proprietary models are gpt-4.1-mini, gpt-5.4, gemini-3.1-flash-lite, grok-4.1-fast, and claude-sonnet-4.6; the open-weight models are llama-4-maverick and minimax-01.
The next stage involved improving the quality of the translated set through a careful validation of the question–answer pairs and the corresponding evidence sessions. A total of 1,543 question–answer pairs (3,086 utterances in total) and 2,232 evidence sessions containing responses to the questions were each validated by three annotators. Where necessary, corrections were made to the English versions of the question–answer pairs and sessions. This helped preserve the logical and semantic coherence of the question, answer and evidence sessions to the same extent as in the original Russian version. More details on translation and its validation are provided in Appendix C.
4
Evaluation
4.1
Baselines
Memory Agent/RAG baselines. The second family represents retrieval-based memory systems (Lewis et al., 2020). Unlike full-context baselines, these methods construct a persistent memory representation from the dialogue history during ingestion stage. At question-answering time, a small set of relevant memories is retrieved and inserted into the answer prompt. The final answer is generated by the same fixed answering model, openai/gpt-4.1-mini, across all memory-based baselines. Thus, whenever we refer to an Agent/RAG or memory-based method, we compare memory layers rather than answergeneration models. Performance differences within this family primarily reflect how well each method stores, updates, structures, and retrieves memories for the fixed answering model.
We evaluate two families of baselines that correspond to two common ways of using long interaction histories in LLM-based assistants: full-context inference and memory-based retrieval-augmented inference.
Our simple RAG baseline stores original messagepair chunks from the dialogue history and retrieves them directly from a vector database. The other systems go beyond this plain setup: they use LLMbased components to extract, transform, consolidate, or structure memories before retrieval. We therefore use Agent/RAG as a compact label for retrieval-based memory systems with different degrees of LLM-mediated memory processing, even though not all of them are agents in the narrow sense. (Singh et al., 2025a; Wang et al., 2024).
Full-context baselines. The first family represents long-context models that receive the complete dialogue history directly in the input context and generate the answer from this full evidence. This setting provides a classic upper-bound-style baseline for memory use. It also tests whether the model can locate and integrate relevant evidence in very long conversational contexts. In our experiments, we selected models with context windows around or above 1M tokens: fitting the largest dialogue requires context lengths roughly above 400K tokens, while the next practical tier among available long-context models is around 1M tokens. This constraint excluded Russian-oriented models, for which we did not find suitable context lengths, as well as available open-source models from the Qwen and DeepSeek families. Although Qwen3.5-Flash and Qwen3.5-Plus nominally satisfy the context-length requirement, providerside content inspection rejected benchmark inputs with DataInspectionFailed errors in our setup, so we excluded them from the final evaluation. Prior long-term memory benchmarks similarly use long-context LLMs as a natural comparison point for evaluating implicit memory mechanisms (Maharana et al., 2024; Wu et al., 2025). We therefore evaluated both propri-
In our experiments, this family includes simple vector-based RAG as well as dedicated memory frameworks: mem0 (vector-based) and mem0g (graph-based) (Chhikara et al., 2025), graphiti (open-source zep) (Rasmussen et al., 2025), cortex (Bhattacharjee, 2025), and memOS (Li et al., 2025). We followed the setup described in the corresponding papers and repositories. For reproducibility, we integrated the open-source frameworks as git submodules, while mem0(g) was accessed through its hosted commercial API. When a framework allowed replacing the embedding backend, we used language-specific embedding models of comparable dimension and parameter scale: FRIDA (ai-forever, 2024) for Russian and Qwen3-Embedding-0.6B (Zhang et al., 2025) for English, selected as strong retrieval-oriented em6
bedding backbones on MTEB benchmark (Muennighoff et al., 2023). 4.2
tary one. The judge assigns labels 0–3; labels 0 and 1 are mapped to incorrect, while labels 2 and 3 are mapped to correct for binary accuracy. F1 is the mean per-question bag-of-words F1 between the generated and reference answers, using the builtin Lighteval implementation: words are split on whitespace and treated as sets, and the maximum F1 is used for questions with multiple references. The full evaluation criteria and judge prompt templates are provided in Appendix E.2.
Evaluation Pipeline
In addition to the dataset, we provide an evaluation methodology and an accompanying pipeline (Fig.10) as a reference protocol for validating memory-based systems. The proposed pipeline follows evaluation practices commonly used in long-context dialogue and memory benchmarks referenced in Sec.2.
The analysis includes: (1) a family-level comparison of retrieval-based memory systems against fullcontext baselines; and (2) a matched-answerer comparison between the gpt-4.1-mini full-context baseline and Agent/RAG systems; and (3) withinfamily slice contrasts over different benchmark scopes. Confidence intervals are estimated by bootstrap resampling over questions, and p-values are Holm-corrected within each comparison block. All results are reported separately for Russian and English as well. Because mem0g is atemporal in our setup, we exclude it from Agent/RAG family aggregates for temporal, temporal-expression, and scope–temporal analyses.
RUMBA is evaluated using a two-stage pipeline: add and eval. In the first add stage, the memory system processes the dialogue data sequentially as user–assistant pairs and populates its internal storage by extracting and storing memories (along with their timestamps) from the conversations. Dialogues belonging to different users are processed independently, and the resulting memories are stored in isolated user-specific memory spaces, e.g., database collections. The Russian and English subsets are ingested separately, resulting in distinct language-specific memory stores. This stage is skipped in the full-context evaluation setup. In the eval stage, the system is evaluated using the open-source Lighteval framework (Habib et al., 2023). For each benchmark question, the system first retrieves the top-k = 10 relevant memories from the corresponding store. The retrieved timestamped5 memories, along with timestamped question, are then provided as context to an answer-generation model, which produces a response based on a fixed language-specific prompt (Fig. 11). The generated answer is evaluated using an LLM-as-Judge procedure that compares the predicted answer with the reference answer. Different judge models are used depending on the evaluation language. For Russian evaluation, we use POLLUX (Martynov et al., 2025), a pretrained judge model for Russian language, which was selected based on a reported correlation of ρ = 0.704 with expert judgments. For English evaluation, the same scale and criteria are applied, while DeepSeek-R1 is used as a judge model. We discuss the choice of judges in Appendix E.1. The pipeline produces LLM-as-Judge accuracy as the primary metric, with F1 reported as a complemen-
5
Results
We report descriptive overall scores for each method in Table 5 and then analyze performance across the main benchmark axes: session scope, temporal reasoning, temporal-expression explicitness, and semantic type. A detailed description of the slice-level analysis and additional results are provided in Appendix F. An example of a single model case study is provided in Appendix G. With the answerer controlled, the gpt-4.1-mini Full-context baseline exceeds the overall Agent/RAG score by +4.28 points in Russian and +8.64 in English. Method-level results are mixed: cortex, memOS, and simple RAG score significantly higher in Russian; in English, memOS scores significantly higher, while simple RAG is not significantly different (Table 6). Semantic supergroup analysis shows that the observed Full-context advantage is concentrated in Reasoning and Extraction questions, but not in Abstention. Across both Agent/RAG and Fullcontext systems, Extraction questions are significantly easier than Reasoning questions in both languages, with within-family accuracy gaps ranging from +9.35 to +14.24 points (Table 7).
5
The mem0g API did not provide timestamps for graph memories. To evaluate the graph-based implementation as is, we did not augment it with timestamped vector memories, resulting in an atemporal graph-based setup.
7
Method
Russian
Agent/RAG
Embedder
mem0g (atemporal) graphiti (open-source zep) mem0 cortex simple RAG memOS
– FRIDA | Qwen3-Embedding-0.6B – FRIDA | Qwen3-Embedding-0.6B FRIDA | Qwen3-Embedding-0.6B FRIDA | Qwen3-Embedding-0.6B
Full context
Context size
llama-4-maverick minimax/minimax-01 gpt4.1-mini gemini-3.1-flash-lite x-ai/grok-4.1-fast claude-sonnet-4.6 gpt-5.4
1M tokens 1M tokens 1M tokens 1M tokens 2M tokens 1M tokens 1.05M tokens
English
LLM-as-Judge
F1
LLM-as-Judge
F1
34.93 ± 1.21 42.71 ± 1.26 53.21 ± 1.27 68.24 ± 1.19 68.89 ± 1.18 66.82 ± 1.20
20.52 ± 0.81 27.00 ± 0.91 30.10 ± 0.95 36.32 ± 0.99 36.17 ± 0.98 38.27 ± 1.00
35.26 ± 1.22 45.11 ± 1.27 54.12 ± 1.27 55.80 ± 1.26 65.46 ± 1.21 69.99 ± 1.17
24.27 ± 0.83 29.13 ± 0.90 35.05 ± 0.95 34.26 ± 0.94 39.00 ± 0.96 42.59 ± 0.96
48.35 ± 1.27 56.71 ± 1.26 60.08 ± 1.25 70.06 ± 1.17 76.22 ± 1.08 78.61 ± 1.04 83.60 ± 0.94
27.18 ± 0.97 31.99 ± 1.02 27.26 ± 0.87 39.30 ± 1.06 34.40 ± 0.92 43.03 ± 1.02 40.43 ± 0.95
50.81 ± 1.27 57.42 ± 1.26 62.93 ± 1.23 69.09 ± 1.18 79.59 ± 1.03 81.98 ± 0.98 83.99 ± 0.93
28.58 ± 0.95 31.68 ± 0.90 33.68 ± 0.89 39.93 ± 0.97 23.34 ± 0.76 40.98 ± 1.01 40.75 ± 0.90
Table 5: Overall RUMBA results for retrieval-based memory systems and full-context baselines. All scores are reported on a 0–100 scale; uncertainty is reported as standard error. Agent/RAG
Lang.
GPT-4.1-mini FC
Agent/RAG
∆ [95% CI]
cortex cortex mem0 mem0 mem0g mem0g memOS memOS simple RAG simple RAG graphiti (zep) graphiti (zep)
RU EN RU EN RU EN RU EN RU EN RU EN
60.08 62.93 60.08 62.93 60.08 62.93 60.08 62.93 60.08 62.93 60.08 62.93
68.24 55.80 53.21 54.12 34.93 35.26 66.82 69.99 68.89 65.46 42.71 45.11
-8.17 [-10.89, -5.44] +7.13 [4.41, 9.85] +6.87 [4.02, 9.66] +8.81 [6.03, 11.60] +25.15 [22.29, 28.06] +27.67 [24.89, 30.46] -6.74 [-9.46, -4.08] -7.06 [-9.66, -4.47] -8.81 [-11.47, -6.22] -2.53 [-5.25, 0.13] +17.37 [14.52, 20.22] +17.82 [14.91, 20.74]
Overall Overall
RU EN
60.08 62.93
55.80 54.29
+4.28 [2.08, 6.43] +8.64 [6.42, 10.77]
Table 6: Direct matched-answerer comparison of the
Contrast
Family / Language
A
B
∆ [95% CI]
Single-session − Multi-session Single-session − Multi-session Single-session − Multi-session Single-session − Multi-session
Agent/RAG Russian Agent/RAG English Full context Russian Full context English
63.92 62.92 76.65 78.21
43.18 40.87 53.69 55.70
+20.73 [17.71, 23.83] +22.06 [19.02, 25.14] +22.96 [20.07, 25.88] +22.51 [19.57, 25.50]
Atemporal − Temporal Atemporal − Temporal Atemporal − Temporal
Agent/RAG English Full context Russian Full context English
60.02 69.39 70.91
52.87 62.96 65.30
+7.15 [3.54, 10.70] +6.43 [3.10, 9.67] +5.61 [2.23, 8.93]
Explicit time evidence − Implicit time evidence Explicit time evidence − Implicit time evidence Explicit time evidence − Implicit time evidence Explicit time evidence − Implicit time evidence
Agent/RAG Russian Agent/RAG English Full context Russian Full context English
63.33 60.20 76.19 76.89
44.35 38.55 46.17 47.00
+18.99 [9.68, 28.61] +21.65 [11.82, 31.34] +30.02 [21.90, 37.77] +29.89 [21.61, 38.35]
Extraction − Reasoning Extraction − Reasoning Extraction − Reasoning Extraction − Reasoning
Agent/RAG Russian Agent/RAG English Full context Russian Full context English
56.12 53.82 69.20 69.85
41.88 41.67 55.02 60.51
+14.24 [10.26, 18.11] +12.15 [8.32, 16.02] +14.18 [10.22, 18.19] +9.35 [5.48, 13.33]
Note: Agent/RAG temporal contrasts exclude mem0g.
Table 7: Main statistically significant within-family slice-
gpt-4.1-mini Full-context baseline with each Agent/RAG system and the overall Agent/RAG family mean. Scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = gpt-4.1-mini FC − Agent/RAG; confidence intervals are paired bootstrap 95% CIs over the same 1,543 questions.
difficulty contrasts. Scores are family-level means of binary LLM-as-judge correctness across methods within each family; scores and differences are reported on a 0–100 accuracy-point scale.
Among the three main benchmark axes, session scope is the strongest and most consistent difficulty driver. Across both method families and both languages, single-session questions are easier than multi-session questions by 20.73–22.96 accuracy points, with all confidence intervals excluding zero (Table 7). Full-context systems stay more accurate on multi-session questions but are not immune to this difficulty.
Implicit temporal expressions are a particularly strong failure mode. Questions whose supporting evidence contains explicit temporal expressions are substantially easier for models than questions whose evidence requires resolving implicit temporal expressions, for both method families and both languages (Table 7). The gap is especially large for Full-context systems: +30.02 accuracy points in Russian and +29.89 accuracy points in English.
6
Temporal questions are also harder on average, although the effect is smaller and less uniform than the single-to-multi degradation. Among statistically significant contrasts, the atemporal-totemporal gap ranges from 5.61 to 7.15 accuracy points depending on language and method family; the Russian Agent/RAG contrast is not significant after correction (Table 7). In English, the Fullcontext–Agent/RAG gap is +12.43 on temporal and +10.89 on atemporal questions.
Conclusion
We introduced RUMBA, the first benchmark for evaluating long-term conversational memory in Russian, together with an aligned English version. Unlike existing English-centric benchmarks, RUMBA accounts for the cultural and linguistic specifics of Russian, including natural interaction patterns and temporal expressions. We proposed a multilingual methodology with a fine-grained taxonomy covering contextual recall, entity tracking, 8
temporal reasoning, and multi-step integration of evidence. We released RUMBA as an open-source resource6 and evaluated state-of-the-art LLMs under both RAG and full-context scenarios. Our results show that current models struggle significantly with tasks requiring simultaneous memory retention, temporal reasoning, and multi-step integration in Russian. RUMBA thus serves as both a diagnostic tool and a catalyst for developing culturally aware conversational agents with robust longterm memory. Future work includes scaling the dataset, enriching reasoning-heavy examples, and probing internal memory representations beyond QA.
a systematic diagnostic analysis of retrieval design choices, such as the choice of embedder, ranking method, retrieval depth, etc. This is outside the main scope of the paper: our goal is to compare memory approaches under a unified pipeline and to show how RUMBA can be used for slice-level diagnostics, rather than to optimize a particular retrieval architecture. Future work. The rapid development of longterm memory benchmarks and memory-augmented agents suggests several natural directions for future work. Recent benchmark extensions, such as LongMemEval-V2 (Wu et al., 2026), and ongoing updates to memory frameworks7 , appeared after RUMBA creation, illustrate the increasing importance of larger-scale, more agentic, and more diagnostic memory evaluation. Looking ahead, future versions of RUMBA can (i) scale the dataset by adding more dialogues and extending session lengths, (ii) enrich the benchmark with a broader variety of reasoning-intensive and temporally entangled examples to improve class diversity, (iii) develop complementary diagnostics for retrieval behavior, memory retention, consistency, and update tracking across sessions, thereby further strengthening RUMBA’s utility as a comprehensive evaluation framework.
Limitations Data. The dataset size, while adequate for initial benchmarking, is not large enough to introduce substantial difficulties for very long-context models. Scaling up both the number of dialogues and their length would further stress-test long-context capabilities. In addition, extraction-oriented questions dominate, whereas complex reasoning and temporally entangled queries are underrepresented. This imbalance reflects practical considerations and ensures a solid foundation, but it limits evaluation of higher-order memory abilities. Finally, the English version is obtained through automatic translation rather than manual rewriting, with no explicit localization applied. This may make the English split less culturally natural and introduce translation artifacts that can affect model understanding and embedding-based retrieval.
Ethical consideration Data collection and annotation All dialogues, questions, and answers were collected from scratch, without reusing any existing copyrighted or restricted datasets. Annotations were performed by native Russian speakers with prior experience in linguistic tasks. All annotators were compensated at a rate exceeding the local minimum wage, and participation was voluntary. Written informed consent was obtained from all annotators, and no personally identifiable information was collected.
Evaluation. Our Russian and English evaluations use different LLM judges. Consequently, part of the observed cross-lingual difference may stem from judge-specific calibration rather than model performance alone (Thakur et al., 2025; Fu and Liu, 2025). RUMBA also follows a standard QA pipeline, which enables transparent and reproducible evaluation of output correctness but does not directly probe the model’s internal memory representations. Nevertheless, the benchmark targets memory capabilities by construction: unlike opendomain QA, RUMBA centers on user-specific information that evolves over time, including repeated updates, corrections, and changes in personal facts.
Licensing and accessibility The resulting benchmark is released under a public license (MIT). The Russian and English subsets are both distributed under the same terms. AI assistant usage During the preparation of this manuscript, the authors used AI assistants solely for editing and improving the English language of the text, including grammar correction, phrasing refinement, and vocabulary suggestions. All intellectual contributions, benchmark design decisions,
Retrieval diagnostics. Our experiments include retrieval-based memory systems, but do not provide 6
7
https://github.com/ai-forever/RUMBA
9
https://mem0.ai/research
and interpretations remain the sole responsibility of the authors.
Panchenko, Alexander Kapitanov, and Alena Fenogenova. 2026. Multimodal evaluation of Russian-language architectures. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2114–2161, Rabat, Morocco. Association for Computational Linguistics.
Acknowledgments We thank all those who contributed to the RUMBA dataset. The dialogues were written, edited, and reviewed by Irina Shamova, Ksenia Kuleshova, Olga Borodina, Maria Avilova, Angelina Pryadko, Elena Dorofeeva, Tatiana Chabanyuk, Anna Plotnikova, Margarita Khurramova, Maksim Anoshin, Pavel Prosyanik, Ksenia Kondrakova, Yulia Piliguzova, Ekaterina Kuryachaya, Allam Atabaev, Inessa Kurevleva, Zakhar Chaichenko, Victoria Nikolaeva, Olga Borisova, Vsevolod Alipov, Danil Vyazovov, Evgenia Efremova, Timur Shenturk, and Konstantin Gridin. Final validation was performed by Olga Borisova, Taisiya Metelkina, Maria Saburbaeva, Yulia Piliguzova, Ekaterina Kuryachaya, and Allam Atabaev, and we are especially grateful to Artemy Stankevich for making the validation process possible. The English version was worked on by Ekaterina Kuryachaya, Allam Atabaev, Natalia Chukicheva, Ksenia Kuleshova, Dmitry Kosourov, Eleonora Petrikova, Alena Gubar, Maria Petrova, Lada Kovalchuk, and Daria Panaiotti. Finally, we appreciate the management efforts of Elvira Budaeva, Pavel Kovalev, Anna Kostikova, and Dmitry Bocharov, as well as the overall coordination by Yuliya Vasilieva.
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. TimeBench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1204–1228, Bangkok, Thailand. Association for Computational Linguistics. Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergei Averkiev, and Alena Fenogenova. 2025. Long context benchmark for the Russian language. In Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025), pages 1–13, Suzhou, China. Association for Computational Linguistics. Pengfei Du. 2026. Memory for autonomous llm agents: Mechanisms, evaluation, and emerging frontiers. arXiv preprint arXiv:2603.07670. Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia Minaeva, Denis Dimitrov, Alexander Panchenko, and Sergey Markov. 2024. MERA: A comprehensive LLM evaluation in Russian. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9920–9948, Bangkok, Thailand. Association for Computational Linguistics.
References ai-forever. 2024. FRIDA: Full-scale finetuned retrieval model inspired by denoising architecture based on T5. https://huggingface.co/ai-forever/FRIDA. Hugging Face model card. Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, and Hanghang Tong. 2026. Memgallery: Benchmarking multimodal long-term conversational memory for MLLM agents. Preprint, arXiv:2601.03515.
Xiyan Fu and Wei Liu. 2025. How reliable is multilingual LLM-as-a-judge? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11040–11053, Suzhou, China. Association for Computational Linguistics.
Biswaroop Bhattacharjee. 2025. Cortex: Advanced memory system for ai agents.
Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval: A lightweight framework for llm evaluation.
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025. JudgeLRM: Large reasoning models as a judge. Preprint, arXiv:2504.00050.
Yuanzhe Hu, Yu Wang, and Julian McAuley. 2026. Evaluating memory in llm agents via incremental multi-turn interactions. In International Conference on Learning Representations.
Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria Tikhonova, Alexander Kharitonov, Yulia Lyakh, Petr Surovtsev, Denis Shevelev, Vildan Saburov, Vasily Konovalov, Elisei Rykov, Ivan Sviridov, Amina Miftakhova, Ilseyar Alimova, Alexander
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel,
10
Sebastian Riedel, and Douwe Kiela. 2020. Retrievalaugmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33.
Proceedings of the Tenth Conference on Machine Translation, pages 1–30, Suzhou, China. Association for Computational Linguistics. Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956.
Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, and 3 others. 2025. MemOS: An operating system for memoryaugmented generation (MAG) in large language models. arXiv preprint arXiv:2505.22101.
Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025a. Agentic retrieval-augmented generation: A survey on agentic RAG. arXiv preprint arXiv:2501.09136. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025b. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
Chi-kiu Lo and Rebecca Knowles. 2023. Data sampling and (in)stability in machine translation evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13064–13074, Toronto, Canada. Association for Computational Linguistics. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870, Bangkok, Thailand. Association for Computational Linguistics.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2025. JudgeBench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations.
Valentin Mamedov, Evgenii Kosarev, Gregory Leleytner, Ilya Shchuckin, Valeriy Berezovskiy, Daniil Smirnov, Dmitry Kozlov, Sergei Averkiev, Lukyanenko Ivan, Aleksandr Proshunin, Ainur Israfilova, Ivan Baskov, Artem Chervyakov, Emil Shakirov, Mikhail Kolesov, Daria Khomich, Daria Latortseva, Sergei Porkhun, Yury Fedorov, and 14 others. 2025. GigaChat family: Efficient Russian language modeling through mixture of experts architecture. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 93–106, Vienna, Austria. Association for Computational Linguistics.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 404–430, Vienna, Austria and virtual meeting. Association for Computational Linguistics.
Nikita Martynov, Anastasia Mordasheva, Dmitriy Gorbetskiy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Victoria Berestova, Nikolay Ivanov, Valeriia Zanina, and Alena Fenogenova. 2025. Eye of judgement: Dissecting the evaluation of russian-speaking LLMs with POLLUX. Preprint, arXiv:2505.24616.
Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, and Gengyu Wang. 2026. From recall to forgetting: Benchmarking long-term memory for personalized agents. arXiv preprint arXiv:2604.20006. Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. Temporal reasoning in natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4070–4078, Online. Association for Computational Linguistics.
Niklas Muennighoff, Nouamane Tazi, Loïc Lorentz, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. Bhawna Piryani, Abdelrahman Abdallah, Jamshid Mozafari, Avishek Anand, and Adam Jatowt. 2026. It’s high time: A survey of temporal question answering. Preprint, arXiv:2505.20243.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.
Sara Rajaee, Rochelle Choenni, Ekaterina Shutova, and Christof Monz. 2025. An empirical analysis of machine translation for expanding multilingual benchmarks. In
Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, and Kai-Wei Chang. 2026. Longmemeval-v2: Evaluating long-term agent
11
memory toward experienced colleagues. arXiv:2605.12493.
set of questions required approximately 2 working days to complete.
Preprint,
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2025. LongMemEval: Benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations.
To standardize the dialogue creation process, a matrix was developed containing the required question types for each dialogue. Authors used this matrix during dialogue construction, which ensured diversity of question types within each dialogue.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
A
Dialogue Validation Process. The validation process consisted of three stages. (1) Authors crossvalidated dialogues, checking question adequacy, taxonomy alignment, and consistency of dialogue and question dates. Necessary corrections to questions and dates were introduced at this stage. (2) Questions and corresponding sessions were validated using an LLM (see the paragraph below Evidence annotation). Based on LLM answers, validators — including the seven highest-performing authors and three independent validators — evaluated whether (a) the question matched its taxonomy category; (b) all dates were explicit and temporally consistent; (c) the answer was contained in the designated sessions and absent from unrelated ones; and (d) the dialogue was internally consistent and provided sufficient information to answer the question. Validators also revised questions and sessions when necessary and occasionally added new sessions. (3) Final validation was performed after all revisions. Each dialogue was reviewed by three validators who had not previously reviewed that dialogue.
Taxonomy details
Semantic axis. Questions are broadly classified into information-extraction and reasoning questions, based on the information available about the user. A detailed classification of question types, along with their descriptions and illustrative examples, is presented in Table 26. Quantitative axis. Questions are classified into two types. Single-session questions: the answer to the question is contained within a single session, whereas in multi-session questions, the answer is distributed across multiple sessions. The distribution of questions on the quantitative axis is depicted in Fig.4. Axis of temporality. Questions are classified as atemporal, for which comprehension of the dialogue’s temporal framework is not required, or temporal, for which such temporal understanding is essential to answer. The distribution of questions along axis of temporality is presented in Fig.5. The taxonomy of temporal questions is presented in Fig.3.
Finally, the complete set of question–answer pairs, session dates, question dates, and taxonomy labels was independently validated by human annotators with an overlap of three. Evidence annotation. We additionally constructed an utterance-level evidence annotation to identify where answer-relevant information appears in the dialogues. The input consisted of spreadsheet files with written dialogues, where each row contained a speaker role, an utterance, and a session-boundary marker, together with question-level metadata: the question, its reference answer, and its type. We first reconstructed session identifiers from the boundary markers and extracted the set of questions with their associated answers, types, and source sessions.
Each temporal question is accompanied by a tag indicating how temporal information was presented in the dialogue: explicit temporal expression (time was indicated explicitly, e.g., a specific day and/or month, year, etc.), implicit temporal expression (time was indicated implicitly, e.g., with words yesterday, 2 days ago, etc.), or no temporal expression (no temporal information was provided). Fig.6 presents the distribution of temporal expression tags.
B
Annotation details
B.1
Dataset Creation
Then, for each utterance–question pair, we queried GigaChat-Max and asked whether the utterance contained explicit or implicit information relevant
On average, the development (creation and validation) of a single dialogue and its corresponding 12
Figure 2: Overview of RUMBA Taxonomy.
Figure 3: Overview of temporal questions Taxonomy: distribution by type, with corresponding counts and percentages.
Figure 4: Distribution across the quantitative axis, reflecting the ratio between single-session and multisession questions, including both the number of questions and their percentage representation.
Figure 5: Distribution across the axis of temporality, reflecting the ratio between temporal and atemporal questions, including both the number of questions and their percentage representation.
13
Figure 6: Temporal tags: distribution of explicit, implicit, non-expressed temporal expressions in the dialogues, including both the number of expressions and their percentage representation.
Figure 7: Information about the dialogue authors including gender, age and field of expertise distribution, reported in terms of percentages.
Figure 8: Author instructions describing the required manuscript components and the requirement to avoid personal data in dialogues.
to answering the question. If no relevant information was present, the model returned None; otherwise, it returned a concise excerpt or comment describing the relevant evidence.
Fig. 9. B.2
Users characters
Dialogues Characters. Users characters represented in the dialogues are heterogeneous and multifaceted, and their interactions encompass a range of thematic domains, which enhances the realism of the constructed interactions and brings them closer to authentic user–assistant conversations. The behavioral scenarios assigned to these characters likewise reflect real-world user patterns, ranging from memory-related operations (such as storing, forgetting, and modifying information) to various strategies for expressing intent. These include explicit requests to memorize a fact, introducing a fact within a single utterance or anaphorically, and linking facts across different sessions, etc. Information about the characters and topics is presented in
The output was an augmented spreadsheet for each dialogue. It preserved the session_id, speaker role, and utterance text, and added one evidence column per question. Each evidence column encoded the question, question type, reference answer, and associated source sessions in its header, while its cells contained the model-produced evidence for each utterance or an empty value when no evidence was detected. This produced a dense utterance-by-question evidence map for subsequent human inspection and diagnostic analysis. The LLM validation prompt for GigaChat Max in Russian (and translated into English) is shown in 14
Figure 9: Evidence annotation prompt template used during validation of dialogue sessions and their associated questions. The original Russian prompt is shown together with an English translation provided for visualization in the paper.
Figure 10: RUMBA validation pipeline.
Fig.12.
a single request to preserve conversational context and message ordering. gpt-4.1-mini8 was used as a translator model. Fig. 13 reports the system prompts and compact user-message schemas used for machine translation.
Sociolinguistic Portrait of the Characters. The dialogues reflect sociocultural realities characteristic of Russia and Russian culture, as they include grammatical constructions typical of Russian, along with culture-specific references such as idioms, proverbs, folk sayings, anecdotes, and allusions to canonical and popular Russian literary and cultural works. Dialogues intentionally include spelling, grammar, lexical, stylistic errors to approximate authentic user-generated input.
C
All remaining dataset fields were carried over unchanged, as they were either originally in English or language-independent. C.1
Translation validation details
The translation validation was carried out by specialists who possess relevant academic qualifications in the field (civil law contract workers and crowd workers (ABC Elementary9 and Rambler&Co companies10 ).
Translation
Two translation granularities were used. Questions and answers were translated independently at the sentence level. Dialogue messages were translated at the session level, where the full sequence of messages within a session was provided to the model in
8
https://openai.com/index/gpt-4-1/ https://elementary.center 10 https://rambler-co.ru 9
15
Figure 11: Answer-generation prompt templates used for Russian and English evaluation. The templates condition the retrieved memories (full dialogues in case of full context evaluation), memory timestamps, the benchmark question, and the question timestamp with its weekday.
Figure 12: High-level information about RUMBA dialogues. Dialogue characters include male and female adults as well as children. The top 10 conversation topics range from health to music and cinema. The dialogues also extensively reflect Russian discourse, including Russian history and art, social realities, notable figures, and broader societal characteristics.
16
Validation of translation quality: 10% of the translated dataset. Each evaluator received the original Russian text and its English translation for comparison and assessment of translation quality. Evaluators were provided with translation quality assessment criteria implemented in POLLUX (Martynov et al., 2025). The names of the criteria, their descriptions, and the scoring system are presented in Table 8.
Judge details
E.1
English judge choice
Our goal was to select an English judge that could be used with the same rubric as POLLUX, so that the English scores would be produced under an evaluation procedure as close as possible to the Russian one. We based the selection on JudgeBench (Tan et al., 2025) and its Hugging Face leaderboard12 . JudgeLRM additionally reports 78.67% agreement between DeepSeek-R1 and human judgments on PandaLM (Chen et al., 2025). Higher-ranked alternatives were not suitable for this setup: several fine-tuned judge or reward models are designed for pairwise preference comparison rather than scoring a single answer against a reference; Prometheusstyle judges rely on their own scoring scale; and prompted GPT/o-series judges were avoided because GPT-family models are included among our answer-generation baselines. We therefore selected DeepSeek-R1 as a strong non-GPT prompted judge compatible with our rubric. We found no single judge with reported human alignment for both languages under comparable conditions and thus used language-specific judges.
The average time required to complete the evaluation of a single RU-EN pair of utterances was approximately 3 minutes. Question-Answer-Evidence Sessions Validation. Experts evaluated a triplet: the evidence sessions and associated question–answer pairs. At this stage, it was essential to ensure that: (1) the translated question preserves the meaning and intent of the original question; (2) the translated gold answer to the question is consistent with the original answer; (3) the translated gold answer corresponds to the translated evidence session, i.e., the session indeed contains the information required to answer the question; and (4) the translated question–answer–session triplet is free of errors and contradictions and remains logically and semantically coherent to the same extent as the original Russian version. All issues were corrected.
E.2
Non-evidence sessions (irrelevant to questions, filler sessions) were not manually validated by experts. For these sessions, we assume a translation quality score of 0.88, which was obtained through 10%-dataset validation.
The judge assigned one of 4 rubric labels. This rubric set was selected empirically. Labels 0–2 correspond to the standard POLLUX rubrics for the Answer Correctness criterion: incorrect or missing answer, incomplete answer, and answer matching or equivalent to the reference answer. We added label 3 for a frequent borderline case in our benchmark: the model answers the question correctly in terms of the key meaning, but adds extra details that are not supported by the question or the reference answer and would require checking the original dialogue. For the final binary accuracy score, labels 0 and 1 were treated as incorrect, while labels 2
The evaluators received compensation for each completed task in accordance with the terms stipulated in their employment contracts. The validation procedure was carried out using the ABC Elementary 11 platform.
Evaluation details
Fig. 10 shows the RUMBA evaluation pipeline. Language-specific answer-generation prompts are provided in Fig. 11.
12
11
Judge prompts and verdicts
To obtain comparable scores for the Russian and English subsets, we used the same POLLUX-style evaluation format in both languages. The only difference is that POLLUX was evaluated with the prompt template and response format it was trained for, while DeepSeek-R1, being a general-purpose model, required a short task description and a JSON output format. Full templates are presented in Fig. 14.
The average time required to complete the evaluation of the triplet was approximately 5 minutes.
D
E
https://huggingface.co/spaces/ScalerLab/ JudgeBench
https://elementary.center/
17
Table 8: Translation Evaluation Criteria, their descriptions, and the scoring system Criteria name
Description
Scores
(a) This criterion evaluates whether it is necessary to further grade the LLM’s output based on other criteria or the LLM’s output is unreadable and cannot be graded. (b) The criterion evaluates whether this is the LLM’s output or there is just a placeholder with the text "Unfortunately, generative models cannot discuss such topics" or similar standard text where the LLM avoids answering.
(a) 0: We read the model’s response and it can be further evaluated based on other criteria. 1: The model’s response shows excessive looping, or there are many artifacts, or the model’s response is entirely in a different language, or the model simply copied the user’s query. It does not make sense to further evaluate the model’s response. (b) 0: The censor did not trigger, the LLM’s output can be evaluated further based on other criteria. 1: The censor triggered, there is nothing to evaluate.
Critical criteria (a) Format violation or (b) Censor block
Task-specific criteria Original goal
This criterion evaluates whether the communicative purpose of the source text is preserved in the translation: for example, to inform, advertise, accuse, tease, and so on.
Original tone
This criterion evaluates whether the tone of the source text is preserved in the translation. It refers to that part of the expressive plan which consists of the connotatively marked elements of the source text. If events are described with irony, the translator should (at least partially) retain this irony. If the information in the source text is presented neutrally, the translator should convey the same tone. Conversely, if the tone of the source text is not neutral, the translation should (at least partially) convey this as well.(Source: S. V. Tyulenev “Translation Theory”) This criterion evaluates whether the translation distorts the author’s position. The translation should not contradict the author’s position. The author’s position may include the author’s point of view (for example, they may be subtly criticizing a product between the lines); ideological stance (often, the ideological position can be inferred from indirect signs); degree of confidence/uncertainty (if the author speaks about something uncertainly, it should not become a statement in the translation), etc. This criterion evaluates whether the functional style is preserved in the translation. When translating, the text should maintain its functional style: literary, journalistic, official business, scientific, or colloquial-everyday, including the linguistic features of each style.
Author viewpoint
Compliance with functional style
Language norms
This criterion evaluates the compliance with lexical, grammatical, syntactic and stylistic norms of the target language. The translation should “sound natural”, meaning it should contain appropriate vocabulary and grammar, syntax, and stylistic devices characteristic of the target language.
Factual accuracy
This criterion evaluates the accuracy of conveying the factual information presented in the source text. The LLM should not distort factual (precision) information from the source text. An exception is made for factual information that would definitely not be understood by a native speaker of the target language: in such cases, the factual information should be adapted to convey the tone of the source text and the author’s intent.
18
0: In the translation, the source text intent is completely lost. 1: (a) The source text intent is fully preserved in the translation. (b) In the translation, the source text intent is preserved but slightly blurred (for example, the aim was to teasingly poke fun at the reader, but only the humorous effect remains without the mockery). 0: The emotional tone of the text was completely lost in the translation. 1: (a) The LLM completely preserved the tone of the source text. (b) In general, the translation preserved the tone of the source text, but it is somewhat diluted; it could have been handled better.
0: The author’s position was completely distorted during the translation. 1: (a) The author’s position is completely preserved, and the LLM conveyed the subtext, if there was any. (b) The author’s position is generally preserved, but there are some issues (for example,the LLM overlooked certain veiled messages).
0: The LLM completely changed the functional style. 1: (a) The translated text fully corresponds to the functional style in which the source text is written. (b) The LLM retained the functional style, but some linguistic features not typical for this functional style appeared in the translation. 0: The LLM made 3 or more errors in compliance with lexical, grammatical, syntactic and stylistic norms of the target language. 1: (a) The LLM made no errors in compliance with lexical, grammatical, syntactic and stylistic norms of the target language. (b) The LLM made one or two errors in compliance with lexical, grammatical, syntactic and stylistic norms of the target language. 0: The LLM made factual errors, incorrectly conveying precise information from the source text. 1: (a) The LLM conveyed the facts and their related context/subtext completely accurately. (b) The LLM did not make any factual errors but made a slip related, for example, to the subtext concerning some phenomenon or event.
Table 9: Criterion-specific scores obtained from human evaluation of translation quality. Criteria
Value
Critical criteria (Format violation or Censor block)
0.99
Task-specific criteria Original goal Original tone Author viewpoint Compliance with functional style Language norms Factual accuracy
0.99 0.98 0.99 0.99 0.9 0.97
General score
0.88
QA-pair translation SYSTEM: You are a professional translator. Translate Russian to natural English. You are given a question-answer pair. Translate both together. Rules: - Preserve exact meaning and the relationship between question and answer. - Keep punctuation and sentence type where possible. - Preserve grammatical cues affecting meaning (gender, number, etc.). - Do not omit, generalize, or add information. - Keep names, titles, and quoted text consistent. - If the answer is an alphabetical list, reorder it to be correct in English while preserving the items. USER: {"question": question_ru, "answer": answer_ru} Dialogue-session translation SYSTEM: You are a professional translator. Translate Russian to natural English. This is a conversation session; messages are context-dependent. Rules: - Preserve messages order and meaning exactly. - Preserve formatting inside each message text: keep newlines, markdown, bullets, code blocks, etc - if any. - Translate ONLY the message text; do not change dia_id. - Output must match the provided schema. USER: {"dia_id": dia_id, "messages": [{"speaker": speaker, "text": text_ru}, ...]}
Figure 13: Translation message templates used to construct the English split. System prompts are shown verbatim; user messages are shown as compact payload schemas.
Figure 14: LLM-as-Judge prompt templates used for Russian and English evaluation. Placeholders indicate the inserted benchmark question, reference answer, and model answer.
19
and 3 were treated as correct. E.3
failure mode is treating over-informative answers as incomplete, which occurred in 13 cases.
Qualitative Analysis
Judge error type
POLLUX
DeepSeek-R1
Correct answer judged as over-informative Correct answer judged as wrong Correct answer judged as incomplete Over-informative answer judged as correct Over-informative answer judged as wrong Over-informative answer judged as incomplete Wrong answer judged as correct Wrong answer judged as over-informative Wrong answer judged as incomplete Incomplete answer judged as correct Incomplete answer judged as over-informative
2 (2.6%) 2 (2.6%) 19 (24.7%) 8 (10.4%) 3 (3.9%) 13 (16.9%) 2 (2.6%) 19 (24.7%) 9 (11.7%) 0 (0.0%) 0 (0.0%)
16 (16.7%) 1 (1.0%) 9 (9.4%) 3 (3.1%) 0 (0.0%) 1 (1.0%) 21 (21.9%) 11 (11.5%) 30 (31.2%) 3 (3.1%) 1 (1.0%)
77
96
Total
DeepSeek-R1 shows the opposite tendency. Among binary-relevant errors, 11 cases correspond to underestimation, whereas 36 cases correspond to overestimation. Thus, 76.6% of DeepSeek-R1 binary-relevant errors are overestimations. The largest category is wrong answers judged as incomplete, with 30 cases. Although this category does not necessarily change the final binary decision if both wrong and incomplete answers are mapped to incorrect, it indicates that the judge often recognizes that the answer is imperfect but fails to identify that the semantic content is fundamentally wrong. More importantly, DeepSeek-R1 also frequently assigns positive judgments to incorrect answers: 21 wrong answers were judged as correct and 11 wrong answers were judged as overinformative.
Table 10: Manual qualitative classification of LLM judge errors for POLLUX and DeepSeek-R1. The analyzed errors account for 5.0% and 6.2% of all 1,543 benchmark questions, respectively; row percentages are computed within each judge’s error set.
In addition to the aggregate agreement metrics, we conducted a manual qualitative analysis of judge errors. The goal of this analysis was to identify recurring failure modes of the LLM-as-a-judge setup and to better understand how these failures may affect the interpretation of the benchmark results.
We further grouped reasoning-level failures into broader categories. For both judges, the dominant source of error is incorrect identification of the semantic core of the answer. This accounts for 50 cases for POLLUX and 87 cases for DeepSeek-R1. In these cases, the judge fails to correctly determine whether the model answer preserves the key meaning of the reference answer. POLLUX also shows a relatively larger share of cases where the judge reasoning is mostly plausible but the final verdict is inconsistent with that reasoning. This occurred in 21 cases. In contrast, DeepSeek-R1 rarely distorts the rubric explicitly, but more often fails at semantic equivalence judgments.
A linguist manually inspected cases in which the automatic judge verdict did not match the human annotation and assigned each case to an error category. The annotation focused on the direction of the judge error: whether the judge underestimated the answer quality, overestimated it, or made a score-level distinction that did not affect the final binary correctness decision. Table 10 summarizes the resulting error taxonomy. For POLLUX, the analysis includes 77 judge error cases, corresponding to 5.0% of all questions. For DeepSeek-R1, the analysis includes 96 judge error cases after filtering out cases that were excluded from the qualitative analysis, corresponding to 6.2% error rate.
These findings suggest that the judge errors are not random. The Russian judge is biased toward stricter scoring and tends to penalize answers for incompleteness or unsupported extra details. The English judge is more permissive and more often overestimates semantically incorrect answers.
F
The qualitative analysis shows that the two judges have different error profiles. POLLUX tends to be more conservative: among binary-relevant errors, 37 cases correspond to underestimation of answer quality, while 21 cases correspond to overestimation. Thus, 63.8% of POLLUX binary-relevant errors are underestimations. The most frequent POLLUX error types are judging correct answers as incomplete and judging wrong answers as over-informative, each accounting for 19 cases. Another frequent
Detailed Slice-Level Result Analysis
This appendix provides the detailed slice-level analysis underlying the summary in Section 5. The purpose of this analysis is to use the full set of benchmark annotations to obtain more fine-grained insights into method behavior. These annotations capture different properties of memory-oriented question answering, including the semantic operation required by the question, the distribution of supporting evidence across sessions, and the need 20
F.1
for temporal grounding. Each question belongs to three main axes: semantic type, single-session versus multi-session evidence, and temporal versus atemporal setting. We additionally analyze temporal-expression explicitness, language effects, evidence-position effects, and semantic supergroups. Unless stated otherwise, all analyses use LLM-as-Judge accuracy as the primary metric.
Table 11 reports the non-direct family-level comparison across the main benchmark slices, without matching the answer model. Full-context scores significantly exceed Agent/RAG scores in all listed slices for both languages. The overall gap is larger in English (+15.11 accuracy points) than in Russian (+11.86 accuracy points). The difference between these two gaps is also significant: (∆EN − ∆RU ) = +3.25 accuracy points, 95% CI [1.82, 4.72]. The matched gpt-4.1-mini comparison in Table 6 yields smaller overall gaps.
ℓ Let ym,q ∈ {0, 1} denote the binary correctness score of method m on question q in language ℓ. For a method family F , we define the per-question family score as
F.2
1 X ℓ sℓF,q = ym,q . |F |
Single-Session vs Multi-Session Difficulty
Table 12 shows that multi-session questions are consistently harder than single-session questions. The single-to-multi degradation ranges from +20.73 to +22.96 accuracy points across languages and families. This makes session multiplicity the strongest difficulty axis in the benchmark.
m∈F
For a benchmark slice A with question set QA , the slice-level family score is s̄ℓF,A =
Full Context vs Agent/RAG as Method Families
1 X ℓ sF,q . |QA |
Importantly, the multi-session penalty is not specific to retrieval-based systems. Full-context systems also show a large degradation from singlesession to multi-session questions. Additional interaction tests show that the degradation difference between full-context and Agent/RAG families is not statistically significant: in Russian, the difference is +2.23 accuracy points with 95% CI [−0.69, 5.13]; in English, it is +0.46 accuracy points with 95% CI [−2.42, 3.42].
q∈QA
Family-level comparisons within a slice are computed as ∆ℓFC−Agent/RAG,A = s̄ℓFC,A − s̄ℓAgent/RAG,A , whereas slice-difficulty contrasts within a family compare two slices A and B as ∆ℓA−B,F = s̄ℓF,A − s̄ℓF,B .
F.3
Temporal vs Atemporal Difficulty
Table 13 shows that temporal questions are harder on average, but the effect is smaller than the single-to-multi degradation. The atemporal-totemporal gap ranges from +2.14 to +7.15 accuracy points. The largest temporal penalty appears for English Agent/RAG systems, while the Russian Agent/RAG contrast is not significant after correction.
Thus, slices of different sizes are compared through their own mean per-question scores rather than raw counts. Confidence intervals are estimated by bootstrap resampling over questions; p-values are Holmcorrected over planned comparisons. Slice-level results are reported separately for Agent/RAG and full-context systems, and all analyses are reported independently for Russian and English. We also provide descriptive method-level results to compare individual baselines within and across the two method families. For temporal, temporal-expression, and scope–temporal family aggregates, the Agent/RAG family excludes mem0g, whose graph-based setup is atemporal; all other aggregates retain the full family.
In English, the Full-context–Agent/RAG gap is +12.43 accuracy points on temporal questions and +10.89 on atemporal questions. In Russian, the corresponding gaps are +4.55 and +8.84, respectively. F.4
Joint Scope–Temporal Intersections
Table 14 reports the four intersections of the scope and temporal axes. The hardest setting is the multisession + temporal intersection. The multi-session
All code used to produce slice-level analyses, statistical tests, tables, and figures is open source. 21
Slice
Lang.
Full context
Agent/RAG
∆
95% CI
Holm p
Overall Overall
RU EN
67.66 69.40
55.80 54.29
+11.86 +15.11
[10.56, 13.17] [13.80, 16.42]
<0.001 <0.001
Single Single Multi Multi
RU EN RU EN
76.65 78.21 53.69 55.70
63.92 62.92 43.18 40.87
+12.73 +15.29 +10.51 +14.83
[11.12, 14.36] [13.66, 16.91] [8.27, 12.75] [12.61, 17.10]
<0.001 <0.001 <0.001 <0.001
Atemporal Atemporal Temporal Temporal
RU EN RU EN
69.39 70.91 62.96 65.30
60.55 60.02 58.41 52.87
+8.84 +10.89 +4.55 +12.43
[7.27, 10.39] [9.32, 12.40] [1.77, 7.32] [9.54, 15.28]
<0.001 <0.001 0.001 <0.001
Single + atemporal Single + atemporal Multi + atemporal Multi + atemporal Single + temporal Single + temporal Multi + temporal Multi + temporal
RU EN RU EN RU EN RU EN
78.61 80.11 55.49 57.05 71.54 73.29 48.42 51.76
69.00 70.03 47.82 44.93 67.20 60.38 43.51 40.13
+9.62 +10.08 +7.67 +12.11 +4.34 +12.91 +4.92 +11.63
[7.77, 11.50] [8.18, 11.91] [4.91, 10.39] [9.40, 14.80] [0.83, 7.93] [9.28, 16.53] [0.56, 9.30] [6.75, 16.57]
<0.001 <0.001 <0.001 <0.001 0.027 <0.001 0.029 <0.001
Table 11: Family-level comparison between Full context and Agent/RAG across the main benchmark slices. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = Full context − Agent/RAG. Agent/RAG temporal and scope–temporal aggregates exclude mem0g. Family
Lang.
Single
Multi
∆
95% CI
Holm p
Agent/RAG Agent/RAG Full context Full context
RU EN RU EN
63.92 62.92 76.65 78.21
43.18 40.87 53.69 55.70
+20.73 +22.06 +22.96 +22.51
[17.71, 23.83] [19.02, 25.14] [20.07, 25.88] [19.57, 25.50]
<0.001 <0.001 <0.001 <0.001
Table 12: Single-session versus multi-session difficulty within method families. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = Single − Multi.
penalty is significant within both atemporal and temporal questions. Within single-session questions, the temporal penalty is significant for English Agent/RAG and both full-context languages, but not for Russian Agent/RAG. Within multi-session questions, it is not significant for Agent/RAG in either language or for English full context after correction. F.5
Questions with no explicit temporal expression occupy an intermediate regime: they are significantly easier than questions with implicit temporal evidence, but for full-context systems they are significantly harder than questions with explicit temporal evidence. F.6
Language Effects
Table 18 reports English–Russian differences. These comparisons should be interpreted cautiously because Russian and English runs use language-specific judges. At the family level, language effects are statistically detectable but small: Agent/RAG systems perform slightly better in Russian overall, whereas full-context systems perform slightly better in English overall. The clearest language-conditioned difference for Agent/RAG appears on temporal questions and implicit/noexpression temporal tags, where Russian scores are higher. For full-context systems, the largest language effect appears in the Reasoning semantic supergroup.
Temporal-Expression Explicitness
Table 16 first reports the Full context versus Agent/RAG comparison (in a non-answerer-fixed setting) within each temporal expression tag. The gap is significant for explicit expressions in both languages and for all English tags; the Russian implicit and no-expression gaps are small and not significant after correction. Table 17 shows that implicit temporal expressions are much harder than explicit temporal expressions. This is one of the strongest slice-level findings. For full-context systems, the explicit-to-implicit gap is approximately 30.02 accuracy points in Russian and 29.89 accuracy points in English. Thus, simply providing the full dialogue does not eliminate the difficulty of implicit temporal grounding.
F.7
Semantic Supergroups
We collapse the first category axis into three semantic "supergroups": Extraction, Reasoning, and 22
Family
Lang.
Agent/RAG Agent/RAG Full context Full context
RU EN RU EN
Atemporal
Temporal
∆
95% CI
Holm p
60.55 60.02 69.39 70.91
58.41 52.87 62.96 65.30
+2.14 +7.15 +6.43 +5.61
[-1.55, 5.74] [3.54, 10.70] [3.10, 9.67] [2.23, 8.93]
0.259 <0.001 <0.001 0.004
Table 13: Atemporal versus temporal difficulty within method families. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = Atemporal − Temporal. Agent/RAG aggregates exclude mem0g. Family
Lang.
Agent/RAG Agent/RAG Full context Full context
RU EN RU EN
Single + Atemporal
Multi + Atemporal
Single + Temporal
Multi + Temporal
69.00 70.03 78.61 80.11
47.82 44.93 55.49 57.05
67.20 60.38 71.54 73.29
43.51 40.13 48.42 51.76
Table 14: Family-level accuracy on the 4 scope–temporal intersections. Scores are family-level means of binary LLM-as-judge correctness and are reported on a 0–100 accuracy-point scale. The table shows how session scope and temporal reasoning jointly affect performance within each method family and language. Agent/RAG aggregates exclude mem0g.
Abstention (Table 2).
effect in this comparison, the two groups differ conceptually and may still reveal model-specific patterns for individual model/methods.
Table 19 shows that Abstention is comparatively saturated: Agent/RAG systems and full-context systems perform similarly (in a setting without a matched answer model), and the gap is not statistically significant in either language. In contrast, full-context systems significantly outperform Agent/RAG systems on both Reasoning and Extraction. Reasoning is the hardest supergroup for both families.
F.9
Figures 15 and 16 show method-level heatmaps by semantic type. These figures are intended as descriptive visualizations rather than statistical comparisons, since semantic categories vary in sample size.
Table 20 reports the within-family semanticsupergroup difficulty contrasts. Extraction is consistently easier than Reasoning. Abstention is easier than both Extraction and Reasoning, especially relative to Reasoning. These contrasts show that the semantic grouping captures a robust difficulty gradient rather than only a difference between method families. F.8
Semantic-Type Heatmaps
F.10
Lost-in-the-Middle Proxy
We also test whether performance drops when the evidence is located in the middle of the dialogue. Since exact token positions in the final prompt are not used here, this is a session-level proxy (thus, it cannot adjudicate token-level lost-in-the-middle effects): questions with evidence near the beginning or end are grouped as edge, and compared against questions with evidence in the middle.
Reasoning vs Temporal Reasoning
As a more focused reasoning analysis, we compare non-temporal Reasoning categories against Temporal Reasoning categories from Fig.2. Table 21 reports the planned contrasts.
Table 22 shows no statistically significant edgeversus-middle effect for either method family or language. Thus, we do not detect a lost-in-themiddle pattern under this proxy.
The observed differences are small and not statistically significant after Holm correction. Moreover, the direction of the difference is not consistent across languages and method families: Temporal Reasoning is slightly higher for Agent/RAG in Russian, whereas general Reasoning is slightly higher in the other three settings.
Because evidence-position effects can be confounded by multi-session questions, we also repeat the same edge-versus-middle analysis on singlesession questions only. Table 23 shows the same conclusion: no statistically significant lost-in-themiddle effect is detected after restricting to singlesession questions.
Although we do not observe a significant aggregate 23
Contrast
Family / Lang.
A
B
∆
95% CI
Holm p
Single+Atemporal − Multi+Atemporal Single+Atemporal − Multi+Atemporal Single+Atemporal − Multi+Atemporal Single+Atemporal − Multi+Atemporal
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
69.00 70.03 78.61 80.11
47.82 44.93 55.49 57.05
+21.17 +25.10 +23.12 +23.06
[17.18, 25.03] [21.13, 28.97] [19.62, 26.42] [19.53, 26.54]
<0.001 <0.001 <0.001 <0.001
Single+Temporal − Multi+Temporal Single+Temporal − Multi+Temporal Single+Temporal − Multi+Temporal Single+Temporal − Multi+Temporal
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
67.20 60.38 71.54 73.29
43.51 40.13 48.42 51.76
+23.70 +20.25 +23.12 +21.53
[17.67, 29.77] [14.10, 26.33] [17.50, 28.64] [15.69, 27.33]
<0.001 <0.001 <0.001 <0.001
Single+Atemporal − Single+Temporal Single+Atemporal − Single+Temporal Single+Atemporal − Single+Temporal Single+Atemporal − Single+Temporal
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
69.00 70.03 78.61 80.11
67.20 60.38 71.54 73.29
+1.79 +9.65 +7.08 +6.82
[-2.30, 6.02] [5.33, 14.05] [3.32, 10.67] [3.29, 10.47]
0.406 <0.001 <0.001 <0.001
Multi+Atemporal − Multi+Temporal Multi+Atemporal − Multi+Temporal Multi+Atemporal − Multi+Temporal Multi+Atemporal − Multi+Temporal
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
47.82 44.93 55.49 57.05
43.51 40.13 48.42 51.76
+4.32 +4.80 +7.07 +5.29
[-1.58, 10.15] [-0.96, 10.57] [1.61, 12.38] [-0.46, 10.84]
0.207 0.207 0.045 0.206
Table 15: Within-family contrasts over the scope–temporal intersections. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ computed as the first slice minus the second slice in each contrast. Agent/RAG aggregates exclude mem0g. Temporal-expression tag
Lang.
Explicit Explicit Implicit Implicit No temporal expression No temporal expression
RU EN RU EN RU EN
Full context
Agent/RAG
∆
95% CI
Holm p
76.19 76.89 46.17 47.00 62.18 65.63
63.33 60.20 44.35 38.55 60.33 53.85
+12.86 +16.69 +1.82 +8.45 +1.85 +11.78
[8.07, 17.79] [11.93, 21.60] [-3.60, 7.16] [1.41, 15.40] [-1.89, 5.70] [7.88, 15.71]
<0.001 <0.001 0.692 0.017 0.692 <0.001
Table 16: Family-level comparison between Full context and Agent/RAG by temporal expression tags. Scores are familylevel means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = Full context − Agent/RAG. Agent/RAG aggregates exclude mem0g.
F.11
Best-Observed Methods
versions of our evaluation, reaching 78.61 accuracy points in Russian and 81.98 accuracy points in English. Using the same diagnostic axes and statistical procedure as in the preceding appendix analyses, we examine where this model is robust, where it degrades, and which benchmark dimensions explain its main failure modes.
For practical interpretation, we also report the best observed method in each family. This is an upperbound comparison and is not used as the main statistical evidence for family-level claims. Across most full-context slices, the strongest observed model is GPT-5.4. For Agent/RAG systems, the strongest Russian results are typically obtained by simple RAG, cortex, or memOS depending on the slice; in English, memOS is the strongest Agent/RAG method across most major slices. These comparisons characterize the strongest observed upper bounds, not a controlled effect of the memory-access mechanism; the matched-answerer results in Table 6 show that individual Agent/RAG systems can exceed the gpt-4.1-mini full-context baseline.
G
Table 24 reports descriptive slice-level scores. Claude Sonnet 4.6 is strongest on Abstention and Extraction-oriented questions, while its weaker regions are multi-session questions, implicit temporal expressions, and Reasoning-oriented questions. This profile is broadly consistent across languages, although English is generally stronger than Russian. Table 25 summarizes the statistically significant contrasts. All deltas are reported in accuracy points on a 0–100 scale. We omit non-significant contrasts from the table to keep the case study focused on the model’s main diagnostic failure modes.
Case study: Claude Sonnet 4.6
In this section, we demonstrate how RUMBA can be used to profile a single selected model rather than for benchmarking different methods. For this case study, we choose anthropic/claude-sonnet-4.6, the second-best system in the full-context setting in both language
The clearest degradation is caused by session scope. As shown in Table 24, accuracy drops from 83.49 to 71.03 in Russian and from 88.60 to 71.69 in English when moving from single-session to multi-session 24
Contrast
Family / Lang.
A
B
∆
95% CI
Holm p
Explicit − Implicit Explicit − Implicit Explicit − Implicit Explicit − Implicit
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
63.33 60.20 76.19 76.89
44.35 38.55 46.17 47.00
+18.99 +21.65 +30.02 +29.89
[9.68, 28.61] [11.82, 31.34] [21.90, 37.77] [21.61, 38.35]
<0.001 <0.001 <0.001 <0.001
No expression − Explicit No expression − Explicit No expression − Explicit No expression − Explicit
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
60.33 53.85 62.18 65.63
63.33 60.20 76.19 76.89
-3.01 -6.34 -14.01 -11.26
[-9.69, 3.82] [-13.05, 0.61] [-19.94, -7.71] [-17.02, -5.20]
0.387 0.153 <0.001 <0.001
No expression − Implicit No expression − Implicit No expression − Implicit No expression − Implicit
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
60.33 53.85 62.18 65.63
44.35 38.55 46.17 47.00
+15.98 +15.30 +16.01 +18.63
[7.33, 24.95] [6.73, 24.30] [8.75, 23.20] [11.05, 26.59]
<0.001 <0.001 <0.001 <0.001
Table 17: Temporal-expression explicitness analysis within method families. Scores are family-level means of binary LLM-asjudge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ computed as the first tag minus the second tag in each contrast. Agent/RAG aggregates exclude mem0g. Slice
Family
English
Russian
EN–RU
95% CI
Holm p
Overall Overall
Agent/RAG Full context
54.29 69.40
55.80 67.66
-1.51 +1.74
[-2.78, -0.24] [0.72, 2.74]
0.019 0.001
Single Single Multi Multi
Agent/RAG Full context Agent/RAG Full context
62.92 78.21 40.87 55.70
63.92 76.65 43.18 53.69
-0.99 +1.57 -2.32 +2.01
[-2.59, 0.57] [0.27, 2.85] [-4.42, -0.19] [0.35, 3.62]
0.217 0.042 0.040 0.040
Atemporal Atemporal Temporal Temporal
Agent/RAG Full context Agent/RAG Full context
60.02 70.91 52.87 65.30
60.55 69.39 58.41 62.96
-0.53 +1.52 -5.54 +2.34
[-2.16, 1.06] [0.35, 2.66] [-8.05, -3.08] [0.31, 4.41]
0.524 0.025 <0.001 0.023
Implicit time Implicit time No temporal expression No temporal expression
Agent/RAG Full context Agent/RAG Full context
38.55 47.00 53.85 65.63
44.35 46.17 60.33 62.18
-5.80 +0.83 -6.48 +3.45
[-10.72, -0.87] [-3.73, 5.38] [-9.75, -3.20] [0.88, 6.09]
0.049 0.762 <0.001 0.010
Extraction Extraction Reasoning Reasoning Abstention Abstention
Agent/RAG Full context Agent/RAG Full context Agent/RAG Full context
53.82 69.85 41.67 60.51 83.01 84.14
56.12 69.20 41.88 55.02 81.60 82.28
-2.30 +0.65 -0.22 +5.48 +1.41 +1.86
[-3.83, -0.77] [-0.62, 1.89] [-3.12, 2.63] [3.18, 7.83] [-1.62, 4.44] [-0.56, 4.36]
0.005 0.325 0.884 <0.001 0.382 0.294
Table 18: Language comparison. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = English − Russian. Agent/RAG temporal and temporal-expression aggregates exclude mem0g. Note that Russian and English evaluations use different LLM judges.
questions. The corresponding contrasts in Table 25 are significant in both languages, with gaps of 12.47 and 16.92 accuracy points. This degradation also remains significant inside both atemporal and temporal subsets. The hardest scope–temporal intersection is multi-session temporal questioning, where the model reaches 66.88 in Russian and 66.23 in English. This suggests that the main structural weakness is not temporal grounding alone, but the combination of temporal grounding with evidence distributed across multiple sessions.
Russian and 84.31 in English, but drops sharply on implicit temporal expressions, reaching only 57.97 and 60.87. The explicit–implicit gaps are large and significant in both languages: 27.32 accuracy points in Russian and 23.44 in English. Implicit temporal expressions are also significantly harder than questions with no temporal expression. The semantic profile shows another major weakness. Claude is strong on Abstention, especially in Russian, where it reaches 90.26 accuracy points, and it is also relatively strong on Extraction. Reasoning is substantially weaker: 65.16 in Russian and 76.77 in English. The Extraction–Reasoning contrast is significant in both languages, while Reasoning–Abstention is even larger, especially in Russian, where the gap is -25.10 accuracy points. This indicates that the model is reliable when it must retrieve or reject unknown user information,
Temporal questions as a broad category are not the strongest explanation of the model’s failures. The atemporal–temporal contrast is not significant after Holm correction in either language. However, the temporal-expression analysis reveals a more specific failure mode. Claude handles explicit temporal expressions well, with accuracy of 85.29 in 25
Full context
Agent/RAG
∆
95% CI
Holm p
RU EN
82.28 84.14
81.60 83.01
+0.68 +1.13
[-2.69, 4.13] [-1.73, 4.00]
0.694 0.455
Reasoning Reasoning
RU EN
55.02 60.51
41.88 41.67
+13.14 +18.84
[10.21, 16.12] [15.71, 22.07]
<0.001 <0.001
Extraction Extraction
RU EN
69.20 69.85
56.12 53.82
+13.09 +16.04
[11.45, 14.70] [14.43, 17.60]
<0.001 <0.001
Semantic supergroup
Lang.
Abstention Abstention
Table 19: Family-level comparison between Full context and Agent/RAG by semantic supergroup from Figure 2. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ = Full context − Agent/RAG. Contrast
Family / Lang.
A
B
∆
95% CI
Holm p
Extraction − Reasoning Extraction − Reasoning Extraction − Reasoning Extraction − Reasoning
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
56.12 53.82 69.20 69.85
41.88 41.67 55.02 60.51
+14.24 +12.15 +14.18 +9.35
[10.26, 18.11] [8.32, 16.02] [10.22, 18.19] [5.48, 13.33]
<0.001 <0.001 <0.001 <0.001
Extraction − Abstention Extraction − Abstention Extraction − Abstention Extraction − Abstention
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
56.12 53.82 69.20 69.85
81.60 83.01 82.28 84.14
-25.48 -29.19 -13.08 -14.28
[-29.65, -21.31] [-33.52, -24.73] [-17.17, -9.01] [-18.47, -9.95]
<0.001 <0.001 <0.001 <0.001
Reasoning − Abstention Reasoning − Abstention Reasoning − Abstention Reasoning − Abstention
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
41.88 41.67 55.02 60.51
81.60 83.01 82.28 84.14
-39.72 -41.34 -27.26 -23.63
[-44.85, -34.48] [-46.58, -36.15] [-32.25, -22.08] [-28.71, -18.46]
<0.001 <0.001 <0.001 <0.001
Table 20: Within-family difficulty contrasts between semantic supergroups from Figure 2. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ computed as the first supergroup minus the second supergroup in each contrast.
but less reliable when the answer requires combining, comparing, or transforming it. Temporal Reasoning is not the uniquely problematic group for Claude Sonnet 4.6. In Russian, Temporal Reasoning is numerically higher than general Reasoning, 69.57 versus 60.40 accuracy points. In English, the direction reverses: General Reasoning reaches 78.52, while Temporal Reasoning reaches 75.16. The within-language Reasoning–Temporal Reasoning contrast is not significant after correction in either language. However, the English– Russian gap is significant for general Reasoning, with English outperforming Russian by 18.12 accuracy points, while the corresponding language gap for Temporal Reasoning is not significant. This suggests that Claude’s Russian weakness is concentrated more in general memory reasoning operations than in Temporal Reasoning specifically. The language comparison also reveals a targeted rather than uniform gap. English is significantly better overall by 3.37 accuracy points, but the largest language differences occur in Reasoningheavy slices. The English–Russian gap is 11.61 accuracy points for the Reasoning supergroup and 18.12 accuracy points for General Reasoning. By contrast, multi-session questions, temporal ques26
tions as a whole, implicit temporal expressions, Abstention, and Temporal Reasoning do not show significant English–Russian differences. Therefore, Claude’s Russian weakness is concentrated primarily in General Reasoning rather than in memory retrieval or temporal grounding overall. Finally, the evidence-position analysis does not support a clear lost-in-the-middle explanation for this model. The edge–middle contrast is not significant in either language, including when restricting the analysis to single-session questions. For all questions, edge–middle differences are -0.30 accuracy points in Russian and 3.54 in English; for single-session questions only, they are -2.78 and 0.83 accuracy points, respectively. Consequently, the dominant failure modes are better explained by multi-session integration, implicit temporal inference, and Reasoning operations than by the absolute position of the supporting evidence in the dialogue.
Contrast
Family / Lang.
A
B
∆
95% CI
Holm p
Reasoning − TR Reasoning − TR Reasoning − TR Reasoning − TR
Agent/RAG RU Agent/RAG EN Full context RU Full context EN
39.49 42.62 55.99 62.42
44.10 40.79 54.13 58.74
-4.61 +1.83 +1.87 +3.68
[-11.54, 2.35] [-4.86, 8.68] [-5.16, 8.95] [-2.94, 10.18]
0.792 1.000 1.000 0.868
Table 21: General Reasoning versus Temporal Reasoning (TR) categories defined in Figure 2. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ computed as General Reasoning minus Temporal Reasoning. Family
Lang.
Edge
Middle
Edge–Middle
95% CI
Holm p
Agent/RAG Agent/RAG Full context Full context
RU EN RU EN
55.62 55.37 68.08 69.97
56.13 52.36 66.90 68.37
-0.51 +3.01 +1.18 +1.60
[-3.68, 2.66] [-0.20, 6.26] [-1.96, 4.30] [-1.60, 4.83]
0.986 0.270 0.986 0.986
Table 22: Lost-in-the-middle proxy analysis by evidence position in the dialogue. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ computed as Edge minus Middle. Edge combines questions whose evidence is located near the beginning or end of the dialogue.
Figure 15: Agent/RAG heatmaps by semantic type.
27
Figure 16: Full-context heatmaps by semantic type.
Family
Lang.
Edge
Middle
Edge–Middle
95% CI
Holm p
Agent/RAG Agent/RAG Full context Full context
RU EN RU EN
63.87 64.11 76.11 77.83
64.01 60.41 77.79 79.02
-0.14 +3.70 -1.68 -1.19
[-4.04, 3.64] [-0.28, 7.72] [-5.13, 1.90] [-4.73, 2.47]
1.000 0.289 1.000 1.000
Table 23: Lost-in-the-middle proxy analysis restricted to single-session questions. Scores are family-level means of binary LLM-as-judge correctness; scores and differences are reported on a 0–100 accuracy-point scale, with ∆ computed as Edge minus Middle.
28
Axis
Slice
n
Russian
English
Overall
All questions
1543
78.61
81.98
Scope
Single-session Multi-session
939 604
83.49 71.03
88.60 71.69
Temporal grounding
Atemporal Temporal
1128 415
79.79 75.42
83.24 78.55
Scope × time
Single + atemporal Multi + atemporal Single + temporal Multi + temporal
678 450 261 154
84.66 72.44 80.46 66.88
89.68 73.56 85.82 66.23
Temporal expression
Explicit Implicit No temporal expression
102 69 244
85.29 57.97 76.23
84.31 60.87 81.15
Semantic supergroup
Extraction Reasoning Abstention
1079 310 154
80.82 65.16 90.26
82.58 76.77 88.31
Reasoning supergroup
General Reasoning Temporal Reasoning
149 161
60.40 69.57
78.52 75.16
Table 24: Single-model diagnostic profile for Claude Sonnet 4.6 across benchmark axes and slices. Values are LLM-as-judge accuracy points on a 0–100 scale; n denotes the number of questions in each slice.
Block
Contrast
Lang.
∆
95% CI
Holm p
Scope
Single − multi Single − multi
RU EN
12.47 16.92
[8.21, 16.77] [12.82, 21.02]
< 0.001 < 0.001
Scope × time
Single+atemporal − multi+atemporal Single+atemporal − multi+atemporal Single+temporal − multi+temporal Single+temporal − multi+temporal
RU EN RU EN
12.22 16.12 13.58 19.59
[7.40, 17.02] [11.47, 20.71] [4.93, 22.39] [11.27, 28.12]
< 0.001 < 0.001 0.003 < 0.001
Temporal expression
Explicit − implicit Explicit − implicit No expression − implicit No expression − implicit
RU EN RU EN
27.32 23.44 18.26 20.28
[13.68, 40.79] [9.85, 36.62] [5.43, 31.30] [7.67, 32.91]
< 0.001 0.001 0.005 0.003
Semantic supergroup
Extraction − Reasoning Extraction − Reasoning Extraction − Abstention Reasoning − Abstention Reasoning − Abstention
RU EN RU RU EN
15.65 [9.84, 21.62] < 0.001 5.80 [0.55, 11.06] 0.030 -9.44 [-14.55, -3.97] 0.003 -25.10 [-32.20, -17.66] < 0.001 -11.54 [-18.33, -4.42] 0.002
English − Russian
Overall Single-session Atemporal Single+atemporal Single+temporal Reasoning supergroup General Reasoning
– – – – – – –
3.37 5.11 3.46 5.01 5.36 11.61 18.12
[1.23, 5.44] [2.77, 7.56] [1.15, 5.76] [2.21, 7.82] [0.38, 10.34] [6.77, 16.45] [10.07, 25.50]
< 0.001 < 0.001 0.006 < 0.001 0.049 < 0.001 < 0.001
Table 25: Significant single-model diagnostic contrasts for Claude Sonnet 4.6 across benchmark axes. Differences are reported as LLM-as-judge accuracy points on a 0–100 scale; ∆ is computed as the first slice minus the second slice in each contrast, except for the language block where ∆ = English − Russian.
29
Table 26: Question taxonomy: detailed description of question types and subtypes according to the semantic axis, with illustrative examples. Category&Type Type Description
Example
NON-TEMPORAL INFORMATION EXTRACTION StaticUser
(a) The fact to which the question refers remains unchanged across all sessions, both qualitatively and quantitatively. (b) The facts to which the question refers are complementary and do not constitute a qualitative or quantitative change of the same fact across sessions.
User: Our dog’s name is Losyash. Question: What’s my dog’s name? Answer: Losyash.
UpdatingInfo
The fact to which the question refers undergoes qualitative or quantitative changes within one or more sessions.
Condition: Pick the most up-to-date answer within one or across two or more sessions. User: I love roses. [...] User: I really love cacti now, I’m tired of roses. [...] User: I don’t like cacti anymore, lilies are my favorite now. Question: What are my favorite plants? Answer: Lilies.
DeleteInfo
The fact to which the question refers is deleted by the user within one or more sessions.
Condition: Pick the most up-to-date answer within one or across two or more sessions. AnswerStandard: The correct answer is “No such information” (or similar). User: I don’t want to be a poor peasant woman, I want to be a noble lady. ... User: Delete the information about what I want to be. Question: What do I want to be? Answer: No such information.
AsstQuery
The fact to which the question refers is contained in the assistant’s response. In this case, the response constitutes advice, a recommendation, or an answer to an instructional request. Important: the fact does not concern the assistant’s persona and does not disclose attributes such as age, gender, personality, name, etc.
User: Who should I read from the 20th century? Assistant: [List]... Bulgakov, Pasternak, Dovlatov, Orwell, Joyce... Question: Which Russian 20th-century writers did you recommend? Answer: Bulgakov, Pasternak, Dovlatov. Continuation
30
Category&Type Type Description
Example
OpenDomain Type
The fact to which the question refers pertains to widely known entities (events—past or recent—objects, public figures, etc.) and is characterized by the fact that a correct or approximate answer can be obtained from publicly available sources without consulting the session history.
User: In Attack on Titan, my favorite pairing is Jean and Mikasa. Question: What’s my favorite Attack on Titan pairing? Answer: Jean and Mikasa.
Social Relationship
The fact to which the question refers reveals the nature of the relationship between individuals. Important: the social role is not explicitly stated in the dialogue and must be inferred through reasoning.
User: I don’t have many relatives, my mom only has one sister, Lera. Question: Who is Lera to me? Answer: Your aunt.
Ordering
The entities to which the question refers share a common attribute. To answer the question, these entities (objects, features, etc.) must be arranged in a specific order according to a given parameter.
User: My sisters’ names are Ira, Vasilisa, and Alisa. Question: List my sisters in alphabetical order. Answer: Alisa, Ira, Vasilisa.
Arithmetic
The fact to which the question refers requires arithmetic computation: (a) sum — calculation of the total; (b) less / more — calculation of the difference; (c) average — calculation of the mean value; (d) other — other mathematical operations.
AnswerStandard: The answer should include a number. User: I have a guitar, a domra, and my grandma’s piano at home. But I only play the guitar and domra. Question: How many musical instruments do I have? Answer: Three.
Comparison
The fact to which the question refers requires qualitative or quantitative comparison of two or more entities. The question reflects an understanding of comparative, superlative, and equivalence relations between entities: (a) comparative — direct comparison (A is more X than B); (b) superlative — identification of the most X within a set; (c) equivalence — A is equal to B.
User: We have two cats: Musya (5 years old) and Kusya (5 months old). Question: Which pet is older? Answer: Musya.
UserQA
The fact to which the question refers requires an understanding not only of the information about the user contained in the dialogue but also of specific or common world knowledge.
User: I professionally study German folklore. Question: Am I a Germanist? Answer: Yes.
REASONING
OtherReasoning Other categories of reasoning tasks. Causal Reasoning
The fact to which the question refers requires an understanding of causal relationships: why / due to what / what was the cause / what was the effect. Important: both the cause and the effect are present in the text across different sessions and must be correlated. Condition: To answer a question about the user, it is necessary to integrate information from multiple sessions.
User: I got into a car accident, the car is wrecked. [...] User: I’m in the hospital, good thing the airbag worked. Question: Why was I in the hospital? Answer: Because you got into a car accident.
MultiStep
The fact to which the question refers requires multi-step reasoning.
User: I bought my first houseplants — four violets. [...] User: One violet died, Lena gave me her ficus (she had two), and my grandma gave me one big aloe and one small one. [...] User: My grandma gave Lena two snake plants because she only had ficuses. Question: How many more plants do I have than Lena? Answer: By 3.
TEMPORAL INFORMATION EXTRACTION DateExtraction Types of questions involving the extraction of an event date and/or questions containing a specific date. Continuation
31
Category&Type Type Description
Example
ExplicitTE Condition Question
(a) The user’s utterance explicitly contains a date, month, year, or time interval related to an event. (b) The question either includes a specific date (day, month, year) and/or refers to such a date. The corresponding subsubtypes: date-question, month/yearquestion, span-question, date-answer, month/yearanswer, span-answer
[16.09.2025] User: I worked at a swimming pool on July 15. [20.12.2025] Question: Where did I work on July 15? [20.12.2025] Answer: At a swimming pool.
NonTE Condition Question
(a) The user’s utterance does not explicitly mention the time of the event and contains no temporal reference. (b) The question either includes a specific date (day, month, year) and/or refers to such a date. The corresponding subsubtypes: date-question, month/yearquestion, span-question, date-answer, month/yearanswer, span-answer
[01.03.2025] User: Come up with a rap nickname for me. [01.03.2025] Assistant: [List]... Vupsen... [06.06.2025] Question: Do you remember what nickname you gave me in March? Answer: Vupsen.
REASONING CalendarUnderstanding Types of questions in which it is necessary to correctly understand and interpret calendrical information: terminology and structure, as well as the units used in calendars. ImplicitTE Condition Question
(a) The user’s utterance contains an implicit temporal reference (yesterday, today, this month, recently, the other day, etc.). (b) A question aimed at identifying and extracting the time of an event. The corresponding subsubtypes: date-question, month/year-question, span-question, dateanswer, month/year-answer, span-answer
[23.11.2025] User: Yesterday I had a singing lesson, today I have a domra lesson. [24.11.2025] Question: What lesson did I have on November 23? Answer: Domra.
RelativeTime Understanding
A question aimed at understanding relative time, expressed in relation to the moment of speaking (i.e., when the question is asked).
[23.11.2025] User: I’m baking a charlotte, but it fell apart. [24.11.2025] Question: What did I bake yesterday? Answer: Charlotte.
Weekday Understanding
A question concerning the correspondence between a date and a day of the week.
[24.11.2025] User: I baked a charlotte on November 20. [25.11.2025] Question: What did I bake on Thursday? Answer: Charlotte
DateQA
A question concerning calendar periods, production calendars, and the properties of the calendar as a system.
[23.11.2025] User: I baked a charlotte on November 13, and today I’m making muffins. [05.12.2025] Question: What did I bake in the second half of November? Answer: Muffins.
TemporalCommonsense A class of questions that fundamentally requires an understanding of the temporal structure of the world. (Event)Ordering (a) A question concerning the sequence of events: what occurred before/after/between, as well as ordinal position: first/second/last. (b) A question involving entities sharing a common attribute, based on an understanding of event timing and/or the time of questioning. To answer, these entities must be ordered according to a specified parameter.
[21.05.2025] User: I have graduation coming up, and my thesis defense already happened. [...] [11.09.2025] User: I started working as a waitress before my thesis defense. [11.12.2025] Question: Did I start working as a waitress before graduation? Answer: Yes.
EventFrequency A question concerning event frequency: how often does an event occur? The minimum unit of measurement is one day.
[06.04.2025] User: Cucumber for breakfast. [...] [17.04.2025] User: Salad with cucumbers. [...] [25.04.2025] User: Smashed cucumbers for dinner. [01.05.2025] Question: How often do I eat cucumbers? Answer: 3 times a month.
EventDuration
[20.10.2025] User: I started painting the walls. [...] [25.10.2025] User: I got divorced. [30.12.2025] Question: How many days before the divorce did I start painting? Answer: 5 days.
A question concerning event duration: how long did X last? how long before did Y occur? how many days after did A take place? The minimum unit of measurement is one day.
TemporallyModifiedGeneralReasoning Reasoning questions complicated by temporal constraints. Continuation
32
Category&Type Type Description
Example
Arithmetic
The fact to which the question refers requires arithmetic computation based on an understanding of event timing and/or the time of questioning.
[11.09.2025] User: I weigh 65 kg. [11.10.2025] Question: How much do I weigh now if last month I weighed 5 kg less? Answer: 70 kg.
Comparison
The fact to which the question refers requires qualitative or quantitative comparison based on an understanding of event timing and/or the time of questioning.
[11.09.2025] User: Today I leg pressed 145 kg, last week it was 148 and 150 kg. [30.10.2025] Question: What was my best result that month? Answer: 150 kg.
UserQA
The fact to which the question refers requires an understanding not only of dialogue-based information but also of general or domain-specific knowledge, in the context of event timing and/or the time of questioning.
[15.03.2025] User: It’s my birthday today. [...] [15.07.2025] User: My husband’s 40th birthday is tomorrow. [25.10.2025] Question: What zodiac signs are me and my husband? Answer: Pisces and Cancer.
Causal Reasoning
The fact to which the question refers requires an understanding of causal relationships in the context of event timing and/or the time of questioning. Condition: To answer a question about the user, it is necessary to synthesize information from multiple sessions.
[15.10.2025] User: I argued with my mom, I’m so mad, how do I leave home? [...] [22.10.2025] User: We argued again about homework, same as last week. [22.12.2025] Question: Why did we argue on October 15? Answer: Because of homework.
MultiStep
The fact to which the question refers requires multi-step reasoning in the context of event timing and/or the time of questioning.
[01.09.2025] User: I got back from vacation, I was there from the 15th. [...] [15.11.2025] User: I’ll have a short vacation next month (5 days), and a small one this month from the 1st to the 4th. [30.11.2025] Question: How long was my second vacation? Answer: 4 days.
EventEvent Time Understanding
The fact to which the question refers is revealed only through another event and requires an understanding of event timing and/or the time of questioning. At least two interrelated events are required.
[15.11.2025] User: Vasya and Petya came over. [...] [17.11.2025] User: My birthday was the day before yesterday. [30.11.2025] Question: Who came over on my birthday? Answer: Vasya and Petya.
During Reasoning
A question concerning the overlap of events. At least two events are required.
[20.10.2025] User: I started painting the walls. [...] [21.10.2025] User: I went for a walk with my nephew Vanya. [...] [22.10.2025] User: I’m hanging out with Vanya. [...] [25.10.2025] User: I finished painting the walls. [...] [30.10.2025] User: I’m with my nephew again, everything’s good. [20.11.2025] Question: How many times did I see my nephew while painting the walls? Answer: 2 times.
ComplexRelations
Continuation
33
Category&Type Type Description
Example
Trends
The fact to which the question refers changes over the course of the dialogue. The question is posed to determine how a particular event or fact evolved over time.
[20.01.2025] User: I smoke 15 cigarettes a day. [...] [21.02.2025] User: I smoke 10 cigarettes a day. [...] [22.03.2025] User: I smoke 5 cigarettes a day. [...] [25.04.2025] User: I smoke 1 cigarette a day. [...] [30.05.2025] User: I smoke 15 cigarettes a day. [21.09.2025] Question: How did my daily smoking change? Answer: First it decreased from 15 cigarettes to 1, then increased sharply back to 15 cigarettes.
The dialogue does not contain an answer to this question. This is a control (trick) question intended to prevent hallucinations. The answer to the question is: No such information and similar.
User: I’m 17 and I have a boyfriend. Question: How old is my boyfriend? Answer: There’s no information about that.
OTHER TYPES Abstention
34