Supra Cognitive Modes: A Routed Architecture for Agent Memory Joshua Tobkin* and David Yang *
Corresponding author: [email protected]
Supra Research 17 July 2026
arXiv:2607.19096v1 [cs.AI] 21 Jul 2026
Abstract Agent-memory workloads mix direct factual lookup, relation-chain and current-state reasoning, and broad synthesis over long histories. We describe Supra Cognitive Modes (SCM), an architecture that maps explicit or automatically selected per-query modes to retrieval and synthesis payloads over one shared ingest substrate. A frozen semantic classifier and runtime gates dispatch queries among fused lexical and dense lookup, graph or iterative multi-hop handling, and stratified long-form synthesis. The substrate combines multi-granularity embeddings, extracted triples, fact-version metadata, and optional asynchronous enrichments. We characterize the deployed configuration on three benchmarks: Long-term Conversational Memory (LoCoMo; n = 1,986), MemoryAgentBench (MAB; n = 3,671), and LongMemEval (n = 500). The reference run records 84.87% on LoCoMo factoid categories and 68.61% on adversarial abstention, 61.49% on MAB across two repetitions, and 86.00% on LongMemEval. A repository-backed reproduction produces similar aggregate scores and supports task- and modeconditioned failure analysis. Raw baseline outputs, aligned end-to-end timing for LoCoMo and LongMemEval, and complete token ledgers are unavailable; stored rows also omit some final runtime decisions. The results characterize one implemented routed configuration and its diagnostic failure patterns, while source inspection verifies the per-query control interface and shared-substrate design. Causal routing effects, efficiency gains, and statistical significance remain outside the available evidence.
1
Introduction
Language-model agents accumulate conversations, retrieved evidence, and user-specific state over long periods. Their memory layer must answer heterogeneous questions: a direct lookup may need one recent fact, a relation-chain question may require several linked records, and a broad request may require synthesis across hundreds of conversations. A single retrieval and synthesis policy is unlikely to serve all three shapes equally well. This heterogeneity creates a deployment tension among answer quality, response time, and cost. Aggressive ingest-time extraction can reduce query-time work but pays before the future query volume is known. Broad multipass synthesis can improve coverage but adds latency and model usage to questions that may need only one fact. We use this memory trilemma as a design lens, not as a claim that the evaluated configurations establish a universal accuracy–latency–cost frontier.
1.1
Approach
Supra Cognitive Modes (SCM) treats the memory operating point as a per-query control. An application can pass an explicit mode or request automatic selection. A mode is a semantic label, a payload is the concrete retrieval and synthesis configuration derived from that label, and a procedure is the execution family that answers the query. Modes and procedures are
2
Supra Cognitive Modes
not one-to-one: several labels can map to the same initial payload, and runtime gates can change the executed procedure. The evaluated configuration uses four frozen semantic labels for single-fact lookup, longform synthesis, time-anchored lookup, and latest-version resolution. A runtime route handles in-context-learning corpora, and query-shape gates can force multi-hop handling. These controls dispatch among direct fused retrieval, graph or iterative relation handling, and stratified long-form synthesis. All procedures read from one substrate containing multigranularity embeddings, extracted relations, fact-version metadata, and optional cached enrichments. Source and configuration inspection establish that this per-query control surface and the procedure families exist. Completed benchmark runs characterize one deployed configuration and expose diagnostic task and mode patterns. Because stored rows do not preserve every final runtime decision and fixed-policy controls were not run, the results do not identify routing or an individual procedure as the cause of a benchmark difference.
1.2
Contributions
This paper makes three bounded contributions: 1. It defines a mode–payload–procedure interface that exposes an agent-memory operating point per query rather than fixing one policy for an entire deployment. 2. It describes a source-visible implementation of that interface over a shared asynchronous substrate, including direct, graph-capable, and long-form procedure families. 3. It provides a repository-traceable characterization of the deployed configuration across three memory benchmarks, together with reproducible task- and mode-conditioned failure strata and an explicit audit of the available timing, cost, judge, and provenance evidence. The architectural novelty is the composition and control interface, not the underlying retrieval primitives. Hybrid lexical and dense retrieval, reciprocal-rank fusion, graph traversal, temporal metadata, and multipass synthesis come from established lines of work discussed next.
2
Related Work
2.1
Agent memory systems
Foundational agent architectures established that stored experience can shape planning and behavior. Generative Agents stores observations and synthesized reflections for later planning [18]; CoALA organizes language agents around modular memory, actions, and decision processes [22]; Reflexion reuses verbal self-critique [21]; and Voyager accumulates a reusable skill library [26]. MemoryBank adds long-term conversational memory, updating, and forgetting [33], while MemGPT and Letta make memory an explicit resource across main-context, recall, and archival tiers [17, 12]. Recent systems place more work in memory construction and organization. Mem0 extracts persistent user-specific memories at ingest [4]; Zep/Graphiti uses a temporal knowledge graph with provenance [19]; A-Mem organizes memories as dynamically linked notes [30]; and MIRIX uses several typed stores under a controller [27]. MemOS frames memory as an operating-system resource [14], while LightMem moves consolidation into lightweight and offline stages [7]. These systems show that memory quality depends not only on retrieval but also on where construction, organization, and synthesis costs are paid.
J. Tobkin and D. Yang
2.2
Adaptive retrieval and routing
Routing among computation paths is well established. Adaptive-RAG classifies questions into no-retrieval, single-step, and multi-step tiers [9]; Self-RAG learns when to retrieve and critique evidence [1]; IRCoT interleaves retrieval with multi-hop reasoning [25]; and FLARE retrieves when generation becomes uncertain [10]. FrugalGPT and RouteLLM apply a parallel idea to model and cascade selection under cost–quality trade-offs [3, 16]. SCM differs primarily in the routed unit. Rather than exposing only retrieval depth or model choice, it maps a per-query semantic label to a payload that can select a retrieval strategy, substrate reads, prompt family, and synthesis procedure. This paper characterizes the deployed routed configuration but does not compare it with fixed-policy controls.
2.3
Retrieval, graph, and synthesis primitives
The procedure families reuse established components. Retrieval-augmented generation and dense passage retrieval provide the dense-retrieval foundation [13, 11]; sparse lexical retrieval and reciprocal-rank fusion combine complementary rankings [5]. GraphRAG and RAPTOR demonstrate graph- and hierarchy-based synthesis over large text collections [6, 20]. Temporal knowledge graphs preserve changing relations and provenance [2, 19]. The contribution here is not a new primitive, but the organization of these mechanisms into procedure-owned paths over one shared substrate, including latest-revision precedence for ordinary current-state queries.
2.4
Benchmarks and judge context
The evaluation uses three complementary suites. Long-term Conversational Memory (LoCoMo) studies very long conversations [15]; LongMemEval covers extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention [29]; and MemoryAgentBench (MAB) covers retrieval, test-time learning, long-range understanding, and selective forgetting [8]. Newer suites broaden the surface toward factual and reflective memory, milliontoken conversations, web-agent experience, and long-horizon trajectories [23, 24, 28, 31]. Several benchmark metrics use language-model judges. Prior judge evaluations document prompt sensitivity, position and verbosity effects, and imperfect human agreement [32]. We preserve the benchmark prompt families and model versions, separate lexical and judge-based cuts, and interpret judge results as benchmark-metric agreement rather than independently validated correctness.
2.5
Positioning
The closest prior systems already contain memory tools, graphs, consolidation policies, or internal routers. The narrower distinction studied here is an application-visible perquery selector whose procedures share one ingest substrate. The evaluation contribution is correspondingly bounded: it reports descriptive accuracy and diagnostic traces while separating those observations from unavailable fixed-policy, aligned timing, complete cost, and mechanism-level evidence.
3
4
Supra Cognitive Modes
3
System Design
3.1
Design objective
The architecture treats accuracy, query latency, and cost placement as coupled design dimensions. A direct lookup should not require the same evidence pool and synthesis plan as a cross-session report, while a current-state query may need revision handling that an ordinary lookup does not. SCM therefore exposes the operating point as a per-query selector over one shared memory substrate. The design separates four concepts. The mode source is either an application-selected label or the benchmark classifier. A payload map turns that label into concrete retrieval and synthesis flags. Runtime gates can revise the initial tier using query or retrieved-context signals. Finally, a procedure executes direct lookup, graph or iterative multi-hop handling, or long-form synthesis. This separation matters because semantic modes and executed procedures are not one-to-one. Direct lookup
Query
Mode source explicit or auto
Payload map + runtime gates
Graph / iterative multi-hop
Synthesis / answer policy
Answer or abstain
Long-form synthesis
Shared substrate embeddings, triples, supplements
Figure 1 Mode-to-procedure architecture. An explicit or automatic mode selects an initial payload; runtime gates may change the tier before a procedure reads from the shared substrate.
3.2
Mode and runtime dispatch
The frozen classifier emits one of four semantic labels and defaults to single-fact lookup when its output cannot be parsed. It was selected from five prompts on a synthetic off-test calibration set and then checked on a consensus-labeled development slice. Table 1 shows that the semantic label set is larger than the initial payload action set: three labels share the same forced-simple configuration. Appendix C records the classifier evidence and complete mapping. The eval route can further use a lightweight tier classifier, intent rules, and a relation-chain detector. Multiple-choice questions can be forced to the simple tier, summary requests to the summary tier, and queries with several relation markers to multi-hop handling. Retrievalshape detection can also reroute in-context-learning corpora after inspecting returned chunks. The dispatch logic is summarized as mode = explicit_mode or classify(query) payload = mode_to_payload(mode) payload = runtime_gates(payload, query, retrieved_context) procedure = procedure_for(payload.final_tier) return execute(procedure, query, payload) The active long-form payload has a provenance mismatch: its JSON field records graph_routed, while an amendment note records a change to graph_then_iterative. Stored outputs do not retain the final graph strategy. We therefore describe the supported procedure family without assigning an individual score to either strategy.
J. Tobkin and D. Yang
5
Mode source
Retrieval config
Initial execution
Role in the reviewed run
Single-fact, time-anchored, or latest-version label Long-form label
h40-temporal-synth
h17-narrative
Forced simple tier; top 100; Sonnet synthesis Graph-capable; top 100; Sonnet synthesis
Examples route
h40-temporal-synth
Three semantic labels sharing one initial payload Narrative retrieval and bullets-expand flags Runtime PATTERN_ FROM_EXAMPLES route for in-context-learning corpora
Forced simple tier; top 100
Table 1 Compact mapping from semantic modes to initial runtime payloads. Runtime gates and fallbacks can still change the final procedure.
3.3
Procedure families
Direct lookup. The direct path embeds the query and can add two lightweight paraphrases before running retrievers in parallel. The active configurations combine Okapi BM25 with dense retrieval over chunk, sentence, and paragraph indexes, then merge candidates with reciprocal-rank fusion. A deterministic paper-grade setting disables wall-clock recency boosting; the narrative configurations retain an entity-frequency reranker. The final depth is capped at 100 chunks. A single Sonnet call synthesizes the answer and receives an instruction to emit “No information available” when the queried entity is unsupported. That abstention behavior is prompt-directed rather than a deterministic post-retrieval filter. Graph and iterative multi-hop handling. The multi-hop family targets relation chains and current-state questions. Extracted triples store normalized subject, predicate, and object fields together with source memory, confidence, version number, active status, and optional temporal fields. A planner can select a seed entity and hop sequence over a closed predicate vocabulary; a grounded walker then chooses among actual incoming and outgoing edges. Candidate triples are ordered by descending version number, giving newer values precedence for ordinary current-state retrieval while retaining older evidence for provenance. Depending on the active strategy, failed graph planning can fall back to iterative dense retrieval or standard fused retrieval. Long-form synthesis. The summary tier samples the corpus chronologically, targeting 200 chunks, and applies a multipass bullets-expand procedure. A cached or online sourceshape classifier distinguishes narrative, research, technical, log, and mixed corpora. The selected shape controls a coverage-planning prompt and an expansion prompt. The active payload enables shape-aware wrapping, broad research prompts, narrative name fidelity, and soft keypoint hints while leaving document-summary, canonical-entity, event-chain, and question-pattern reranking disabled.
3.4
Shared asynchronous substrate
All three procedure families read from one ingest substrate. The mandatory Tier-1 path writes chunks, optional sentence and paragraph segments, embeddings, lexical-index fields, and extracted triples. Once that path completes, fused lookup is available immediately;
6
Supra Cognitive Modes
graph quality depends on the retained triples, and long-form synthesis can operate directly from stratified chunks. Optional supplement jobs then cache higher-level signals such as document shape, anticipated keypoints, canonical entities, event chains, question patterns, structural importance, and entity types. These fields are nullable, and query code reads them only when the active payload enables the corresponding flag. Missing enrichments therefore degrade to online detection or omission rather than blocking query readiness. query-ready boundary Corpus chunks
Tier-1 ingest
Queryable substrate BM25, embeddings, triples
Mode-routed query path
Async supplement jobs
Cached signals doc shape, keypoints, entity types
Nullable procedure reads
Answer
read only when enabled
Figure 2 Shared-substrate lifecycle. Mandatory ingest creates the queryable indexes; asynchronous supplements later add nullable signals read by selected payloads.
The source establishes the presence of these procedures and reads, but the completed benchmark outputs do not isolate their individual effects. In particular, final runtime procedures and fallbacks are not fully persisted, and no component-removal controls are part of the retained evaluation.
4
Evaluation Protocol and Evidence Boundaries
4.1
Evaluated answer paths
The evaluation package supplies results for three configurations: SCM in its deployed moderouted configuration, a production comparator, and Mem0 v2 OSS with its language-model reranker. The repository contains SCM outputs but not raw baseline outputs, so baseline parity cannot be independently verified. The table records the declared models and scoring settings used by the reproduction workflow. Answer generation is benchmark-specific: MAB scores the online route answer, whereas LoCoMo and LongMem score a later offline synthesis answer. Benchmark
Retrieved evidence
MemoryAgentBench Online mode-routed eval path LoCoMo
Mode-routed retrieval snapshot
LongMemEval
Mode-routed retrieval snapshot
Scored answer
Timing and usage aligned?
Answer returned by the Yes for SCM; same request per-request wall time under concurrency Offline Stage-3 No; stored latency synthesis belongs to the earlier online answer Offline Stage-3 No; Stage-3 duration synthesis and usage were not persisted
Table 2 Benchmark-specific answer paths. Only the MAB output aligns the scored answer with the retained request timing.
J. Tobkin and D. Yang
4.2
7
Declared settings and non-invariants
Table 3 Declared evaluation settings. Component
Value
Synthesis model
Claude Sonnet 4.5 (claude-sonnet-4-5-20250929, temperature 0) SCM synthesis Online mode-aware answer generation for MAB; offline intent- and role-routed Stage-3 synthesis for LoCoMo and LongMem Embeddings OpenAI text-embedding-3-large at 1,024 dimensions Memory-agent task suite scorer Byte-faithful upstream substring-and-exact-match scorer Memory-agent task suite gpt-4o-2024-05-13 at temperature 0.1, three-prompt synthesis judge keypoint-recall F1 Long-conversational-memory gpt-4o-mini-2024-07-18 with the comparator’s accuracy factoid judge prompt Long-conversational-memory Benchmark authors’ paper-lexical substring scorer abstention scorer Longitudinal personal-memory gpt-4o-2024-08-06 with the benchmark authors’ six judge task-type prompts Mem0 v2 SDK mem0ai==2.0.3, top_k=100, rerank=True, language-model reranker using gpt-4o-mini at temperature 0 Mem0 v2 extraction model Claude Haiku 4.5 Vector store ChromaDB (Mem0 v2); pgvector (Production Comparator and SCM)
The declared settings are intended to reduce several obvious confounds, but the absence of baseline rows prevents an audit of whether model, prompt, and generation parity held in the supplied baseline runs. Retrieval outputs, prompt inputs, vector stores, and pipeline structure can also interact with the generator and judges. We therefore report supplied configuration-level differences and avoid attributing them to individual memory components.
4.3
Timing boundary
The desired latency boundary begins when a question is dispatched and ends when the scored answer is ready. The stored MAB output meets that boundary for SCM because it scores the online answer and retains the same request’s latency. The driver launches concurrent requests (concurrency 12 in the runbook), so the value is per-request wall time under load rather than a serial benchmark. The stored LoCoMo and LongMem outputs do not meet the desired boundary. They preserve latency for an online answer, then score a different offline Stage-3 answer without recording Stage-3 duration. An internal ten-call Sonnet probe reports a 4.22-second mean (median 3.96, p90 7.81), but its raw rows are absent and it does not reconstruct per-question Stage-3 time. We therefore omit cross-system end-to-end latency comparisons for LoCoMo and LongMem rather than pair accuracy and timing from different answer paths.
4.4
Modeled monetary cost
Dollar figures are model-based operational budgets, not measured billing totals. The package does not persist complete provider usage fields or include billing exports, and the scored
8
Supra Cognitive Modes
LoCoMo/LongMem Stage-3 path cannot be reconstructed from the old per-query estimates. We therefore omit exact per-query cost comparisons and retain only the coarse per-run budget in Appendix B.
4.5
Benchmark metrics and judge scope
The memory-agent task suite uses its canonical macro over accurate retrieval (AR), test-time learning (TTL), long-range understanding (LRU), and selective forgetting (SF). LoCoMo factoid categories use the vendored language-model judge, while the adversarial abstention category uses the benchmark’s lexical scorer. LongMemEval uses its task-type-aware binary judge. No additional human-agreement study was performed. Language-model-judge scores therefore measure agreement with the specified benchmark judge, not human-validated correctness. The LoCoMo prompt originated with one compared system, creating a potential circularity; using the same prompt for all outputs standardizes scoring but does not eliminate that concern. Appendix B states these limits explicitly.
4.6
Repetition and statistical scope
The two MAB reference repetitions score 61.72% and 61.26%, with arithmetic mean 61.49% and sample standard deviation 0.32 percentage points. Reporting the runs explicitly avoids making two observations resemble a confidence interval. The other reference-system cells are single-run unless stated otherwise. A local reproduction with retained outputs and a separately reported teammate run appear as consistency checks; neither is pooled with the reference because their provenance, environments, and output snapshots differ. We report descriptive percentages and percentage-point differences. We do not report p-values, confidence intervals, or claims of statistical significance. Terms such as “higher,” “lower,” and “equal at displayed precision” describe the observed outputs only. Appendix B summarizes this statistical boundary.
4.7
Router evidence
The frozen classifier was selected on 1,065 synthetic question-shape examples, reaching 96.06% overall and 95.80% macro accuracy. A read-only real-development sanity check on 568 consensus-labeled questions reached 90.14% micro and 89.37% macro accuracy. Appendix C reports the per-mode evidence and the runtime overrides. These data show that the classifier carries question-shape signal. Measuring operational routing efficiency would additionally require a held-out procedure-level confusion matrix and misroute penalties. Stored outputs also lack enough payload provenance to resolve every strategy amendment. The supported claim is therefore implementation of automatic and explicit per-query dispatch, not router optimality or an advantage over fixed policies.
4.8
Intervention and observational analyses
Section 6 distinguishes three evidence levels. Configuration-level benchmark comparisons vary complete memory systems. Route-conditioned task and error breakdowns reuse completed outputs and are observational. The reported abstention-on/off comparison is an author-provided configuration difference; the available package contains the with-abstention scored output but not the paired base output, so even this mechanism comparison is not independently auditable as a strict one-variable ablation.
J. Tobkin and D. Yang
4.9
9
Threat from benchmark contamination
The public benchmarks may occur in the pretraining data of the synthesis and judge models. Sharing model versions across systems reduces one source of variation but does not guarantee that contamination affects all retrieved contexts equally. Absolute scores, and potentially relative differences, should therefore be interpreted with this caveat.
5
Results
This section reports completed configuration-level accuracy measurements. The evaluation package includes baseline scores but not the corresponding raw outputs. Timing and cost are discussed separately because the scored LoCoMo and LongMem answers were produced by an offline synthesis path whose latency and token usage were not persisted. We therefore do not present an aligned three-axis comparison.
5.1
Three-benchmark accuracy summary
Table 4 Reported aggregate scores and derived cross-benchmark summaries. Raw baseline rows are unavailable. Benchmark or summary
Mem0 v2 OSS
Long-conversational memory, mixed 13.08% full set (n = 1,986) Memory-agent task suite (n = 3,671) 34.03%
a
Production Comparator
SCM
63.10%
81.22%a
52.22%
Longitudinal personal memory (n = 500)
24.00%
57.00%
61.49%; runs 61.72% and 61.26% 86.00%
Minimum of three benchmark aggregates (“floor”) Unweighted geometric mean
13.08%
52.22%
61.49%
22.02%
57.27%
75.45%
Corrected from 81.17% by repairing a one-row duplicate-question category collision in the completed scored output; Section 5.4 documents the correction. Baseline values retain the externally reported scorer outputs because raw baseline rows are unavailable. The two MAB observations are shown explicitly; their arithmetic mean rounds to 61.49%. The minimum and geometric mean equalweight the three reported benchmark percentages. Because the underlying metrics differ, these are descriptive summaries rather than pooled accuracy, formal robustness statistics, or routing-effect estimates.
10
Supra Cognitive Modes
GM 75.45 SCM min 61.49 GM 57.27 Production comparator min 52.22 GM 22.02 Mem0 v2 OSS min 13.08 0
20
40
60
80
100
Reported benchmark score (%) Benchmark aggregate
Long-conv. memory (mixed)
MAB task suite
Longitudinal memory
Figure 3 Cross-workload profiles derived from Table 4. Benchmark aggregates are shown with distinct point shapes, the leftmost point is each configuration’s minimum, and the diamond is its unweighted geometric mean. Lines connect each configuration’s observed minimum and maximum; they do not imply continuity or an ordering among benchmarks. The underlying metrics differ, and the summaries are descriptive.
The pattern is descriptive. SCM records the largest displayed aggregate on all three benchmarks and the largest observed minimum, but the memory-agent suite’s sub-dataset results remain uneven. Because baseline rows are unavailable and most cells are single-run, the numerical differences are not presented as significance-tested wins or as evidence that routing caused the cross-workload profile.
5.2
Reference and reproduction runs
Table 5 keeps the reported SCM runs separate because they differ in environment and provenance. The author reference is used in Table 4. The local reproduction is preserved in the repository; the teammate values are externally reported without row-level output in the package. Table 5 Reported SCM runs and their artifact provenance. Run source
Date / build MAB v3
Author reference
2026-06-15, deployed v4 fast build
Local 2026-06-16 reproduction local rep
Teammate report
LongMem
61.49% overall; 86.0% runs 61.72% and overall, n = 61.26% 500
62.19% overall; 85.4% AR 78.15%, overall TTL 50.32%, LRU 57.94%, SF 62.38% Externally 61.43% overall; 86.4% reported run AR 78.92%, overall TTL 50.33%, LRU 53.72%, SF 62.75%
LoCoMo
Notes
84.87% factoid; Reference 68.61% row used abstention; in Table 4 81.22% mixed 85.78% factoid; Raw 68.83% scored abstention; output 81.97% mixed included 84.86% factoid; Original 68.46% scorer abstention; cut; no 81.17% mixed row-level output
The author reference and local reproduction with retained outputs fall within roughly one point of one another on each headline accuracy metric. This is consistency evidence, not
J. Tobkin and D. Yang
11
a formal reproducibility or variance estimate: only the MAB author reference contains two repetitions, and the environments are not exchangeable. The externally reported values are numerically similar but provide context only because their underlying rows are unavailable.
5.3
Memory-agent task-suite breakdown
Table 6 reports the author-reference competency breakdown. Accurate retrieval (AR), test-time learning (TTL), long-range understanding (LRU), and selective forgetting (SF) were defined in Section 4.5. The sub-dataset aliases are: single- and multi-hop document question answering (SH-Doc-QA and MH-Doc-QA), event question answering (EventQA), LongMemEval-S* (LME-S*), multi-class classification (MCC), movie recommendation (MovieRec), InfiniteBench summarization (InfBench-Sum), detective question answering (DetectiveQA), and single- and multi-hop fact consolidation (FC-SH and FC-MH). The MCC group averages BANKING77, CLINC150, natural-language understanding (NLU), and the TREC coarse- and fine-label tasks. Table 6 Author-reference MAB competency means across two runs and sub-dataset accuracy from the retained fast-build repetition. Competency
Mean
Per-dataset detail
AR
78.62%
TTL LRU SF Overall
50.05% 54.67% 62.62% 61.49%
SH-Doc-QA 96.0%; MH-Doc-QA 85.0%; EventQA 86.3%; LME-S* 45.7% MCC group 86.40%; Movie-Rec 13.9% InfBench-Sum 27.5%; Detective-QA 80.3% FC-SH 82.7%; FC-MH 42.8% Macro over AR, TTL, LRU, and SF
The aggregate masks substantial heterogeneity. SH-Doc-QA, MH-Doc-QA, EventQA, and FC-SH are the strongest displayed cells; Movie-Rec, InfBench-Sum, FC-MH, and LME-S* are substantially lower. Route-conditioned failure counts in Section 6 use these cells to formulate engineering hypotheses, not to assign causal credit to a procedure.
5.4
Long-conversational-memory scoring cuts and one-row correction
The benchmark contains four factoid categories and one adversarial abstention category. Factoids use the vendored language-model judge; abstention uses the benchmark paper’s lexical scorer. We report their scores separately and also give a mixed full-benchmark average for continuity with the supplied evaluation. The original scoring join keyed duplicated questions by question text. One question— “What did Gina receive from a dance contest?”—appears once as a category-4 factoid with gold answer “a trophy” and once as a category-5 adversarial item with gold answer “undefined.” The join assigned both scored rows the adversarial metadata, yielding category counts 1,539/447 rather than the fixture’s 1,540/446 split. In both repository-backed SCM outputs, the prediction for the displaced factoid is “A trophy with a glass globe on top,” an exact-match-safe correct answer. Reclassifying that existing row, without issuing a new model call, gives:
12
Supra Cognitive Modes
Table 7 Corrected LoCoMo scoring cuts for repository-backed SCM outputs. Run
Factoid categories Abstention (n = (n = 1,540) 446)
Mixed full set (n = 1,986)
Author reference Local reproduction
84.87% 85.78%
81.22% 81.97%
68.61% 68.83%
The mixed score combines two different metrics and should not be interpreted as a homogeneous accuracy measure. We use it only as an end-to-end summary and give the component cuts alongside it. When a comparison refers specifically to SCM factoid recall, the relevant result is 84.87%; 81.22% is reserved for the explicitly labeled mixed full set. We do not compare 84.87% directly with a supplied baseline unless that baseline is confirmed to use the same factoid cut. Because raw production-comparator and Mem0 rows are absent, their supplied values have not been reprocessed through this one-row correction.
5.5
Reported abstention configuration difference
The author-provided comparison reports 2.46% on the original 447-row adversarial cut with entity-grounded abstention disabled and 68.46% with it enabled, a 66.00-point difference under that original scorer output. The enabled reference output is present and becomes 68.61% after the category correction above. The paired disabled output is absent, so we cannot independently verify that the entity-grounding toggle was the only difference or recompute its corrected denominator. We therefore treat the 66-point change as a reported configuration difference, not an audited causal ablation. In the enabled factual path, the synthesis prompt instructs the model to identify the question’s main entity and answer “No information available” when that entity is unsupported by retrieved memories. This is model-directed prompt behavior, not a deterministic postretrieval filter. The missing disabled output prevents a mechanism-level effect estimate.
5.6
Longitudinal personal-memory task types
Table 8 Reported LongMemEval accuracy by task type. Task type
n
Mem0 v2 OSS
Comparator
SCM
Single-session-user Single-session-assistant Single-session-preference Multi-session Temporal-reasoning Knowledge-revision
70 56 30 133 133 78
34.29% 5.36% 66.67% 31.58% 15.04% 14.10%
77.14% 94.64% 53.33% 53.38% 31.58% 62.82%
95.71% 94.64% 83.33% 78.95% 83.46% 88.46%
The observed SCM–comparator differences are +18.57 points on single-session-user, 0 on single-session-assistant, +30.00 on single-session-preference, +25.57 on multi-session, +51.88 on temporal-reasoning, and +25.64 on knowledge-revision. The completed local trace records mode labels, not the final executed procedure. Temporal questions are distributed mainly across time-anchored, long-form, and single-fact labels; knowledge-update questions are mostly labeled single-fact, with smaller long-form, time-anchored, and latest-version groups.
J. Tobkin and D. Yang
13
No removal ablations were run. The task scores therefore cannot be assigned to a unique procedure.
+18.57
Single-session user
0.00
Single-session assistant
+30.00
Single-session preference
+25.57
Multi-session
+51.88
Temporal reasoning
+25.64
Knowledge revision 0
15
30
45
Accuracy difference (percentage points)
Figure 4 Observed longitudinal task-type score differences relative to the supplied comparator values. These descriptive differences should not be read as executed-procedure or causal effects.
5.7
Operational timing and cost scope
MAB uses the answer returned by the online evaluation route, so its stored request latency and scored answer refer to the same path. The reference reports 9.00 seconds per request under the benchmark launch configuration. The full-run driver uses concurrent requests (the runbook uses concurrency 12), so this is per-request wall time under load rather than a serial single-thread benchmark. LoCoMo and LongMem use a separate offline Stage-3 synthesizer for the scored answer. Their retrieval records retain latency for the earlier online answer and do not record Stage-3 call duration. Pairing those latencies with Stage-3 accuracy would mix two answer paths, so we omit end-to-end latency claims for these benchmarks. The same missing Stage-3 token records prevent a per-query cost comparison aligned to the scored answers. Appendix B reports only the coarse runbook budget. These evidence boundaries support a limited conclusion: the completed runs measure one implemented routed configuration and expose task- and mode-conditioned hypotheses. Comparisons against a fixed procedure, aligned end-to-end latency across systems, and component-level attribution require additional controls.
6
Diagnostic Analysis and Limitations
6.1
Evidence levels
The retained package supports three levels of interpretation. First, benchmark tables compare complete configurations; many components vary together and raw baseline rows are unavailable. Second, SCM traces retain semantic mode labels, retrieved chunks, and scored outcomes, enabling reproducible mode-conditioned diagnostics. They do not always retain the final procedure after runtime gates and fallbacks. Third, an author-provided abstention comparison changes a complete configuration, but the disabled per-question output is absent. Consequently, the evidence supports description of the routed configuration, not a comparison of automatic routing against a fixed policy.
14
Supra Cognitive Modes
6.2
Patterns in completed traces
Table 9 identifies high-mass error strata in the repository-backed local reproduction. Counts are useful for selecting cases to inspect, but they are not causal effects and should be interpreted alongside each route’s prevalence. Observed stratum
Largest recorded mode group
Diagnostic reading
LongMem temporal failures
Long-form (14/25), then time-anchored (9/25) Single-fact (5/10)
Errors cluster in two semantic routes Most failures are not labeled latest-version Relation-chain errors concentrate in this label Recommendation questions are poorly served by the recorded route Event errors concentrate in this label Residual refusal errors concentrate in lookup
LongMem knowledge-update failures MAB FC-MH failures
Long-form (214/229)
MAB Movie-Rec failures
Single-fact (179/179)
MAB EventQA failures
Long-form (188/191)
LoCoMo abstention failures
Single-fact (110/139)
Table 9 Mode-conditioned failure strata from completed outputs. The final executed procedure is not always known.
LongMemEval provides the clearest retrieval-versus-synthesis diagnostic. All 73 judged failures in the local 85.4% reproduction have hit_at_k=true. The largest residual cells are multi-session (26/133 failures) and temporal reasoning (25/133). Representative rows contain evidence-use errors: incomplete counting despite several relevant mentions, generic advice instead of a stored preference, incorrect ordering of retrieved events, and abstention when current and prior facts are present. The hit flag is coarse, but the pattern is consistent with errors after initial retrieval. MAB failures are similarly concentrated. FC-MH records 229/400 failures, Movie-Rec 179/200 recall misses, EventQA 191/1,500 failures, and LME-S* 164/300 failures. These strata motivate relation-chain, recommendation, and event-synthesis investigation; they do not identify a final executed procedure. On the corrected LoCoMo adversarial cut, the local reproduction has 139/446 residual failures. Many are near-miss refusals or semantically reasonable corrections that fail the lexical target, reinforcing the decision to report factoid and abstention metrics separately. The author-provided abstention comparison reports 2.46% with entity-grounded abstention disabled and 68.46% with it enabled on the original 447-row adversarial cut. The enabled output is present and becomes 68.61% after the category correction; the disabled output is not present. Source inspection finds a prompt instruction to emit “No information available” when the main entity is unsupported, not a deterministic entity filter. The prompt and reported configuration difference are therefore documented separately rather than combined into a mechanism-level effect estimate.
6.3
Threats to validity
Baseline provenance. Raw Mem0 and production-comparator predictions are unavailable, so prompt parity, row-level corrections, and paired uncertainty cannot be
J. Tobkin and D. Yang
audited. Answer-path alignment. MAB scores the online response whose latency is stored. LoCoMo and LongMemEval score offline Stage-3 synthesis while retaining timing for an earlier answer; exact Stage-3 usage is absent. Repetition and judges. The MAB reference has two repetitions and most other cells are single-run. Language-model judges may be prompt-, verbosity-, and contaminationsensitive, and no additional human-agreement study was performed. System non-invariants. Mem0 uses ChromaDB, while SCM and the production comparator use pgvector. Generator, retrieval, storage, and prompt differences remain part of the complete-configuration comparison. Routing provenance. Stored rows retain semantic labels but not every post-gate tier, graph strategy, fallback, or prompt family. The active graph field and amendment note also disagree, and the real-development router set contains almost no latest-version examples. Scoring correction. A question-text join displaced one LoCoMo factoid into the adversarial cut. The existing SCM rows are corrected by fixture identity, but raw baselines cannot be reprocessed in the same way.
6.4
Scope and next steps
The available evidence supports an implementation claim and a descriptive characterization of one routed configuration. Claims about routing efficiency, component causality, statistical significance, or human-validated judge accuracy require new controls. The highest-value next steps are to persist complete routing provenance, align scored answers with timing and token usage, compare routed and fixed procedures on identical corpus snapshots, and audit a stratified judge sample. Those controls remain outside the claims of this version.
7
Conclusion
Supra Cognitive Modes implements a per-query control interface over a shared agent-memory substrate. Explicit or automatically selected semantic modes map to retrieval and synthesis payloads, while runtime gates can dispatch among direct, graph-capable, and long-form procedures without rebuilding the memory store. Mandatory ingest creates a queryable lexical, embedding, and relation substrate; asynchronous supplements add nullable signals for selected procedures. Across three reported benchmark comparisons, the deployed configuration records the largest aggregate accuracy and the local reproduction with retained outputs yields similar headline scores. The rows also expose concrete weaknesses in recommendation, long-range summarization, multi-hop conflict resolution, temporal evidence use, and lexical abstention. Because the comparisons are configuration-level, they provide neither a routing-efficiency estimate nor component-level attribution. The next evaluation step is therefore not another broad benchmark table, but aligned and attributable measurement: persist the final procedure and fallback path, score the same answer whose latency and usage are retained, compare routed and fixed policies on identical snapshots, and validate a stratified sample of judge decisions. Until then, the central result is the implemented mode–payload–procedure interface and its evidence-bounded characterization over one shared substrate.
15
16
Supra Cognitive Modes
References 1
2
3
4
5
6
7
8
9
10
11
12 13
14 15
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In Proceedings of the International Conference on Learning Representations, 2024. URL: https://arxiv.org/abs/2310.11511. Borui Cai, Yong Xiang, Longxiang Gao, He Zhang, Yunfeng Li, and Jianxin Li. Temporal knowledge graph completion: A survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6545–6553, 2023. URL: https://doi.org/ 10.24963/ijcai.2023/734. Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023. URL: https://arxiv.org/abs/2305.05176. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. URL: https://arxiv.org/abs/2504.19413. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759, 2009. URL: https://doi.org/10.1145/1571941.1572114. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024. URL: https://arxiv.org/abs/2404.16130. Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, and Ningyu Zhang. LightMem: Lightweight and efficient memory-augmented generation. arXiv preprint arXiv:2510.18866, 2026. ICLR 2026. URL: https://arxiv.org/abs/2510.18866. Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. arXiv preprint arXiv:2507.05257, 2025. URL: https://arxiv.org/ abs/2507.05257. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, 2024. URL: https://aclanthology.org/2024.naacl-long.389/. Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023. URL: https://arxiv.org/abs/2305.06983. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6769–6781, 2020. URL: https://aclanthology.org/2020.emnlp-main.550/. Letta. Agent memory and architecture. https://docs.letta.com/guides/agents/ architectures/memgpt, 2026. Accessed 2026-06-09. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, 2020. URL: https://papers.nips.cc/ paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html. Zhiyu Li, Chenyang Xi, Chunyu Li, Ding Chen, et al. MemOS: A memory OS for AI system. arXiv preprint arXiv:2507.03724, 2025. URL: https://arxiv.org/abs/2507.03724. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Proceedings
J. Tobkin and D. Yang
16
17
18
19
20
21
22
23
24
25
26
27 28
29
30
31
of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. URL: https://aclanthology.org/2024.acl-long.747/. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data. arXiv preprint arXiv:2406.18665, 2024. URL: https://arxiv.org/abs/2406.18665. Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. URL: https://arxiv.org/abs/2310.08560. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. URL: https://arxiv.org/abs/2304.03442. Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep: A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956, 2025. URL: https://arxiv.org/abs/2501.13956. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval. In Proceedings of the International Conference on Learning Representations, 2024. URL: https://arxiv.org/abs/2401.18059. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023. URL: https://arxiv.org/abs/2303.11366. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. URL: https: //arxiv.org/abs/2309.02427. Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents. arXiv preprint arXiv:2506.21605, 2025. URL: https://arxiv.org/abs/2506.21605. Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani, and J Ross Mitchell. Beyond a million tokens: Benchmarking and enhancing long-term memory in LLMs. arXiv preprint arXiv:2510.27246, 2025. URL: https://arxiv.org/abs/2510.27246. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. URL: https://aclanthology.org/2023.acl-long.557/. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. URL: https://arxiv.org/abs/2305.16291. Yu Wang and Xi Chen. MIRIX: Multi-agent memory system for LLM-based agents. arXiv preprint arXiv:2507.07957, 2025. URL: https://arxiv.org/abs/2507.07957. Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, and Kai-Wei Chang. LongMemEval-V2: Evaluating long-term agent memory toward experienced colleagues. arXiv preprint arXiv:2605.12493, 2026. URL: https://arxiv.org/abs/2605.12493. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813, 2024. URL: https://arxiv.org/abs/2410.10813. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic memory for LLM agents. arXiv preprint arXiv:2502.12110, 2025. URL: https: //arxiv.org/abs/2502.12110. Yujie Zhao, Boqin Yuan, Junbo Huang, Haocheng Yuan, Zhongming Yu, Haozhou Xu, Lanxiang Hu, Abhilash Shankarampeta, Zimeng Huang, Wentao Ni, Yuandong Tian, and Jishen Zhao. AMA-Bench: Evaluating long-horizon memory for agentic applications. arXiv preprint arXiv:2602.22769, 2026. URL: https://arxiv.org/abs/2602.22769.
17
18
Supra Cognitive Modes
32
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023. URL: https://arxiv.org/abs/2306.05685. Wanjun Zhong, Lianghong Guo, Qiqi Gao, Ye He, and Yanlin Wang. MemoryBank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731, 2024. URL: https://doi.org/10.1609/ aaai.v38i17.29946.
33
A
Methodology details
This appendix records the executable configuration that is materialized in the reproduction artifact. The goal is to make the reported measurements traceable to files, command entrypoints, and pinned runtime parameters rather than to prose descriptions alone.
A.1
Artifact sources of truth
Table 10 Artifact sources of truth. Method component
Artifact path
What the artifact fixes
Question-shape classifier
supraos-memory-benchmark/ scripts/ classifier-rebuild/ frozen-classifier.json supraos-memory-benchmark/ scripts/ classifier-rebuild/ phaseB-mode-config.json evals/memory-agent-bench/ configs/*.json
Classifier prompt, model, temperature, mode set, synthetic-selection scores, and dev-sanity scores. Retrieval config selected per mode, final top_k, synth mode, force-tier flags, and amendment notes.
Mode-to-runtime payload map
Retrieval configs
Full-run retrieval driver
Offline synthesis driver
Benchmark scorers
Upstream judge prompts
Question fixtures
Retriever mix, BM25 parameters, RRF constant, post-processing stack, and disabled/non-functional comment keys. scripts/phaseB-cmPer-question owner-address fullrun-retrieval.py mapping, reference-date construction, classifier invocation, runtime fallbacks, checkpointing, and output provenance fields. scripts/run_memory_ Local Sonnet synthesis path used pipeline.py; for LoCoMo and LongMem scoring, scripts/memory_system.py role cache, intent classifier, prompt family, and max-token policy. scripts/score_mab.sh; End-to-end scoring entrypoints scripts/score_locomo.sh; and the ordered conversion steps scripts/score_longmem.sh required before judging. scripts/scoring/upstream_ Byte-vendored prompts for prompts/*.txt MemoryAgentBench LRU and the Mem0 LoCoMo judge. data/ The 6,157 benchmark questions gbrain-rerun-2026-05-16/ used by the retrieval driver. *.ndjson
J. Tobkin and D. Yang
A.2
19
Pinned model and call parameters
Table 11 Pinned model and call parameters. Component
Pinned value in the artifact
Frozen routing classifier Offline role and intent classification Offline synthesis
claude-haiku-4-5-20251001, temperature 0, max tokens 16. claude-haiku-4-5-20251001, temperature 0, max tokens 30.
claude-sonnet-4-5-20250929, temperature 0. Max tokens are 4,096 for summaries, 1,024 for recommendation lists, and 600 for other answers. DetQA resynthesis claude-sonnet-4-5-20250929, temperature 0, max tokens 1,024, canonical MemoryAgentBench detective prompt plus hardening clause. MAB LRU summarization gpt-4o-2024-05-13, temperature 0.1, max tokens 4,096, judge three prompt calls per InfBench-Sum question. LoCoMo judge gpt-4o-mini-2024-07-18, temperature 0, max tokens 300, JSON response format, categories 1–4 only. LongMemEval judge gpt-4o-2024-08-06, temperature 0, max tokens 10, task-type-specific prompt. OpenAI helper retry policy Six attempts, exponential backoff, 120-second timeout. Anthropic helper retry policy Six attempts, exponential backoff, 180-second timeout.
A.3
Retrieval and scoring parameters
Table 12 Retrieval and scoring parameters. Parameter
Value fixed by code or config
Embedding provider
openai-large, corresponding to OpenAI text-embedding-3-large at 1,024 dimensions in the runbook and ingest documentation. top_k = 100 in the /api/memory/eval-run body for the benchmark runs. Retrieval-config candidate fetch depths may be larger before fusion and post-processing. Okapi BM25 with k1 = 1.5, b = 0.75. Canonical config uses top_k = 100; h17/h40 configs use wider candidate pools. Chunk, sentence, and paragraph embedding retrievers in db_ann mode. Reciprocal-rank fusion with k_constant = 30. Present in h17/h40 configs with boost factor 0, because all chunks in the frozen paper-grade run are freshly ingested and a wall-clock recency boost would be non-deterministic. Enabled in h17/h40 configs with boost_factor = 0.5, location_weight = 0.5, and min_freq = 2; the driver rewrites the ingest_run_id at runtime. The runbook commands use concurrency 12 for full benchmark runs. The retrieval driver also defines a hard cap of 24 concurrent eval-run POSTs.
Final retrieval depth
BM25 retriever Dense retrievers Fusion Recency booster
Narrative rerank
Driver concurrency
20
Supra Cognitive Modes
Parameter
Value fixed by code or config
MAB canonical scoring
Overall is the mean of AR, TTL, LRU, and SF. TTL is the mean of in-context classification and recommendation recall; LRU is the mean of InfBench-Sum F1 and Detective-QA substring exact match after DetQA resynthesis. Headline uses the Mem0 lenient LLM judge for categories 1–4; category 5 is reported separately with the benchmark paper’s abstention lexical rule. Official task-type-aware binary judge, with abstention prompts selected for abstention question IDs.
LoCoMo scoring
LongMemEval scoring
A.4
Reproducibility guardrails
Two guardrails are methodologically important enough to call out. First, scripts/bolt_ from_stage3.py joins Stage 3 synthesis output back to source rows by question ID rather than by parallel-completion order. The runbook records that an enumerate-based join measured LongMem at 21.8%, while the corrected question-ID join measured 88.2% on the same rep. Second, the MAB scorer applies the Detective-QA resynthesis before computing the canonical LRU macro; without this step, Detective-QA rows are judged against summary-style predictions rather than the benchmark’s detective prompt. The current artifact snapshot is not fully portable: scripts/scoring/score_locomo.py and scripts/scoring/score_longmem.py use hard-coded absolute roots instead of deriving ROOT from their own file paths. This does not change the recorded results, but those scripts require path normalization before they can be reused in another environment.
B
Reproducibility and Evidence Scope
B.1
Judge-based metrics
LoCoMo factoids, LongMemEval task types, and the MemoryAgentBench long-range metric use benchmark-provided or comparator-derived language-model judge prompts. Exact prompts, model identifiers, temperatures, and output limits are recorded in Appendix A. No additional human-agreement annotation was performed. Reported values therefore measure agreement with the specified judge pipelines rather than independently validated correctness. This distinction matters for semantically plausible answers, verbosity-sensitive cases, and LoCoMo factoids, where the prompt originated with one compared system.
B.2
Comparator scope
The Mem0 reference is described as using mem0ai==2.0.3, top_k=100, language-model reranking with gpt-4o-mini, and Claude Haiku 4.5 extraction. The production comparator is treated as an evaluated configuration rather than a fully inspectable research artifact. The package contains aggregate baseline values and the reproduction command surface, but not raw baseline predictions, retrieval traces, complete logs, or billing records. Baseline comparisons are therefore reported values rather than independently recomputed measurements.
B.3
Cost and statistical reporting
The runbook contains planning estimates of approximately USD 172 and two to three hours for one full SCM reproduction, including provider calls and judges. These are budget
J. Tobkin and D. Yang
21
estimates, not observed bills: provider token usage and reconciled billing exports are not persisted consistently, and the LoCoMo and LongMemEval Stage-3 synthesis path cannot be reconstructed from earlier per-query estimates. The paper consequently reports no exact cross-system per-query cost. The MAB reference repetitions are 61.72% and 61.26%, with arithmetic mean 61.49% and sample standard deviation 0.32 percentage points. The tables show both observations instead of presenting a mean with a plus-or-minus standard deviation. Other reference cells are single-run unless stated otherwise. Raw baseline rows and sufficient independent repetitions are unavailable, so the manuscript reports no p-values, confidence intervals, or formal effect sizes. Percentage-point differences are descriptive arithmetic on the retained scores.
B.4
Artifact scope
The current artifact snapshot contains fixtures, scripts, configurations, prompts, and SCM outputs, but does not include row-level baseline outputs, the abstention-disabled output, complete token ledgers, probe rows, human annotations, or significance-test outputs. Credentials, private-infrastructure references, nonportable paths, and third-party licensing require review before distribution. No public artifact accompanies this version; a later release would require sanitization, licensing review, portable path configuration, and a file manifest linking every reported result to its source artifact.
C
Classifier routing accuracy
This appendix documents the routing evidence present in the repository. It distinguishes three layers that can otherwise be conflated: the frozen four-mode classifier, the mode-to-runtime payload mapping, and post-classifier runtime overrides that are visible in the full-run driver. Two classifier copies are present. The nested supraos-memory-benchmark package uses the semantic names reported in the benchmark outputs; the top-level legacy copy uses DIRECT_RECALL, LONG_RANGE_READER, TEMPORAL_RECALL, and SKEPTIC. This appendix cites the nested package explicitly. The copies are aliases from different packaging revisions and should not be treated as byte-identical provenance.
C.1
Frozen classifier artifact
Table 13 Frozen classifier artifact fields. Field
Value
Artifact
supraos-memory-benchmark/scripts/ classifier-rebuild/frozen-classifier.json. 2026-05-17T17:46:24+0800. claude-haiku-4-5-20251001. 0. c5-balanced. SINGLE_FACT_LOOKUP, LONG_FORM_SYNTHESIS, TIME_ANCHORED_LOOKUP, and LATEST_VERSION_RESOLUTION. 1,065 synthetic shape-based questions, described in the artifact as off-test and independently verified. Highest macro accuracy across five candidate prompts.
Frozen at Model Temperature Winner candidate Frozen modes
Selection set Selection criterion
22
Supra Cognitive Modes
Field
Value
Synthetic overall / macro Synthetic per-mode accuracy
96.06% overall, 95.80% macro. Single-fact lookup 91.73%, long-form synthesis 99.72%, time-anchored lookup 92.24%, latest-version resolution 99.54%. 616-question read-only slice; 568 consensus-labeled questions after two strong labelers. Inter-labeler agreement 93.88%; classifier micro 90.14%, macro 89.37% on the consensus subset. Single-fact lookup 88.13% (n = 278), long-form synthesis 95.10% (n = 204), time-anchored lookup 84.88% (n = 86). The dev slice contains almost no verbalized-conflict questions, so LATEST_VERSION_RESOLUTION is validated synthetically but not measured on the real-dev slice.
Real-dev sanity check Real-dev agreement Real-dev per-mode accuracy
Known limitation
The classifier always returns one of the four frozen modes after parsing; unparseable outputs default to SINGLE_FACT_LOOKUP. The full prompt is stored verbatim in the frozen classifier artifact rather than reprinted here.
C.2
Mode-to-runtime mapping
Table 14 Mode-to-runtime mapping. Runtime mode
Retrieval config
Eval-run flags fixed by the artifact
SINGLE_FACT_LOOKUP
h40-temporal-synth force_tier=simple, top_k=100, synthesizer=sonnet; payload also sets synth_mode to this mode. TIME_ANCHORED_LOOKUP h40-temporal-synth force_tier=simple, top_k=100, synthesizer=sonnet; payload also sets synth_mode to this mode. LATEST_VERSION_RESOLUTION h40-temporal-synth force_tier=simple, top_k=100, synthesizer=sonnet; payload also sets synth_mode to this mode. LONG_FORM_SYNTHESIS h17-narrative multi_hop_strategy: graph_routed in the JSON field, with an amendment note to graph_then_iterative; graph_planner_model: haiku; top_k=100; synthesizer=sonnet; payload also sets synth_mode to this mode. ICL_BROAD_CONTEXT h40-temporal-synth Runtime route added in the mode config for in-context-learning corpora; uses simple tier, top_k=100, and single-fact lookup payload metadata.
The inspected synthesizer source accepts a synth_mode field but does not branch directly on the frozen classifier labels. Prompt selection in the inspected code is driven by intent, question format, and long-form synthesis flags. The synth_mode values above should therefore
J. Tobkin and D. Yang
23
be treated as payload metadata unless the final run uses a source version that consumes them directly. The active mode-config file also carries an amendment note about changing LONG_FORM_SYNTHESIS from graph_routed to graph_then_iterative. The active JSON field still reads graph_routed, and the stored launch command contains no override. Perquestion outputs omit the final strategy. We retain this mismatch as a provenance limitation and do not identify either strategy as the one responsible for the reported scores.
C.3
Runtime overrides and observed routing distribution
The retrieval driver can alter or supplement the frozen classifier’s output using productionobservable signals only: A chain-shape detector can force long-form-synthesis routing before the Haiku classifier when the question text matches multi-hop chain patterns. A second chain-shape gate can force force_tier to multi_hop in the final /api/memory/ eval-run payload when the query has at least two chain markers. An ICL corpus-shape detector can reroute to ICL_BROAD_CONTEXT after inspecting retrieved chunks for repeated label: examples. Empty-retrieval fallbacks can retry an ICL route or use summary-tier retrieval to recover chunks. These rules explain why the full-run output can contain five classified_mode values even though the frozen Haiku classifier has four modes. In the completed MAB rep stored at data/mab-rep1-2026-06-09, the observed distribution was: Table 15 Observed MAB classified-mode distribution.
C.4
Observed classified_mode
Count
Share of 3,671 questions
LONG_FORM_SYNTHESIS SINGLE_FACT_LOOKUP ICL_BROAD_CONTEXT TIME_ANCHORED_LOOKUP LATEST_VERSION_RESOLUTION
2,192 867 496 73 43
59.71% 23.62% 13.51% 1.99% 1.17%
Validation boundary
The repository does not contain a held-out procedure-level gold assignment, a runtime confusion matrix, per-procedure precision and recall, or measured misroute penalties. The synthetic selection set and consensus real-development slice show that the frozen semantic classifier carries signal, but they do not validate the full classifier-plus-gates execution path. This paper therefore makes no operational router-accuracy or routing-efficiency claim.
D
Dataset Statistics and Trace Audit
This appendix reports dataset and run-shape statistics derived from bundled fixtures and completed SCM outputs. It also records evidence limitations for the vector-store comparison and internal synthesis-latency probe.
24
Supra Cognitive Modes
D.1
Benchmark fixture sizes
Table 16 Benchmark fixture sizes. Benchmark
Question count
Source fixture
Additional corpus statistic available in repo
Memory-agent task suite
3,671
Long-conversationalmemory
1,986
Longitudinal personal-memory
500
Upstream parquet files and source maps are bundled; the completed MAB rep records 100 retrieved chunks per question. phaseA-locomo. 10 conversation samples, 1,986 ndjson; QA items, 272 parsed sessions, locomo10.json and 5,882 message-like conversation entries. phaseA-longmem. 500 question records, 948 ndjson; haystack sessions, and 10,960 longmemeval_ message-like haystack entries. oracle.json phaseA-mab. ndjson
The question counts are the exact row counts consumed by the full-run retrieval driver. LoCoMo and LongMem corpus counts are computed from the bundled JSON fixtures. The memory-agent task suite’s upstream parquet files are included, but the package does not include a mechanically generated document and token-count table for that corpus.
D.2
MAB sub-dataset distribution
The completed MAB rep at data/mab-rep1-2026-06-09 records the following sub-dataset distribution after tagging: Table 17 MAB sub-dataset distribution. Sub-dataset
Count
Scoring role
EventQA
1,500
FC-MH
400
FC-SH
400
LME-S*
300
Movie-Rec
200
BANKING77
100
CLINC150
100
InfBench-Sum
100
Accurate-retrieval macro component. Conflict-resolution / compositional-factoid component. Conflict-resolution / single-hop fact-consolidation component. Accurate-retrieval macro component. Recommendation component of TTL. In-context classification component of TTL. In-context classification component of TTL. LRU summarization component, judged by three-prompt GPT-4o F1.
J. Tobkin and D. Yang
D.3
25
Sub-dataset
Count
Scoring role
MH-Doc-QA
100
NLU
100
SH-Doc-QA
100
TREC-Coarse
100
TREC-Fine
100
Detective-QA
71
Accurate-retrieval macro component. In-context classification component of TTL. Accurate-retrieval macro component. In-context classification component of TTL. In-context classification component of TTL. LRU detective component after canonical DetQA resynthesis.
Recorded mode distributions
Appendix C reports the MAB distribution. The existing local LoCoMo and LongMem retrieval traces yield the following semantic-mode counts without new model calls: Table 18 Recorded semantic-mode distributions. Recorded mode
LoCoMo (n = 1,986)
LongMem (n = 500)
SINGLE_FACT_LOOKUP TIME_ANCHORED_LOOKUP LONG_FORM_SYNTHESIS LATEST_VERSION_RESOLUTION
1,286 (64.75%) 475 (23.92%) 222 (11.18%) 3 (0.15%)
301 (60.20%) 121 (24.20%) 74 (14.80%) 4 (0.80%)
These are recorded classifier labels, not final procedure counts. Runtime gates and fallbacks can change execution after the label is written.
D.4
Existing-output error-analysis frame
The completed 2026-06-16 local reproduction contains enough per-question data to support the diagnostic analysis in Section 6 without running a new benchmark condition. The joined fields are: question text, gold answer, prediction, benchmark task type or category, scorer pass/fail field, classified mode, latency, and retrieved chunks. For LongMemEval, the retrieval trace also records hit_at_k. File names in the table are relative to the corresponding codex-*-rep1-20260616 output directory unless otherwise noted. Table 19 Files and joins used for existing-output error analysis. Benchmark
Scored-output file
Memory-agent task suite scored-mab-lexical. json; scored-mab-lru.json Long-conversationalresults/scored-locomo. memory json
Retrieval-trace file
Join convention
mab-retrievaltagged.ndjson
_idx row index.
locomo-retrieval. scorer row order / ndjson _idx.
26
Supra Cognitive Modes
Benchmark
Scored-output file
Retrieval-trace file
Longitudinal personal-memory
results/scored-longmem. longmemjson retrieval.ndjson
Join convention question_id.
The following table gives the scored-failure strata used in Section 6. For LoCoMo categories 1–4, failure means the Mem0-style language-model judge label is not correct. For LoCoMo category 5, failure means the benchmark paper’s lexical abstention scorer does not return full credit. For LongMemEval, failure means the task-type binary judge returns 0. For MAB rows, failure means the row’s headline metric is 0 for binary or recall-style metrics; InfBench-Sum is reported as a continuous summarization-F1 score rather than converted to a binary failure count. Table 20 Scored-failure strata in completed outputs. Benchmark stratum
Scored failures
Denominator
Largest associated route strata
LoCoMo category 1
41
282
LoCoMo category 2 LoCoMo category 3
58 26
321 96
LoCoMo category 4
94
841
LoCoMo category 5
139
446
LongMemEval multi-session
26
133
LongMemEval temporal-reasoning
25
133
LongMemEval knowledge-update
10
78
LongMemEval remaining task types
12
156
MAB FC-MH
229
400
MAB Movie-Rec MAB EventQA MAB LongMemEval-S
179 191 164
200 1500 300
MAB FC-SH MAB Detective-QA
72 12
400 71
30 SINGLE_FACT_LOOKUP; 10 LONG_FORM_SYNTHESIS. 49 TIME_ANCHORED_LOOKUP. 15 SINGLE_FACT_LOOKUP; 11 LONG_FORM_SYNTHESIS. 56 SINGLE_FACT_LOOKUP; 21 LONG_FORM_SYNTHESIS; 17 TIME_ANCHORED_LOOKUP. 110 SINGLE_FACT_LOOKUP; 19 TIME_ANCHORED_LOOKUP. 13 SINGLE_FACT_LOOKUP; 9 TIME_ANCHORED_LOOKUP; all 26 have hit_at_k=true. 14 LONG_FORM_SYNTHESIS; 9 TIME_ANCHORED_LOOKUP; all 25 have hit_at_k=true. 5 SINGLE_FACT_LOOKUP; 4 LONG_FORM_SYNTHESIS; all 10 have hit_at_k=true. Single-session-user 2/70; single-session-assistant 3/56; single-session-preference 7/30; all 12 have hit_at_k=true. 214 LONG_FORM_SYNTHESIS; 15 SINGLE_FACT_LOOKUP. 179 SINGLE_FACT_LOOKUP. 188 LONG_FORM_SYNTHESIS. 110 SINGLE_FACT_LOOKUP; 36 TIME_ANCHORED_LOOKUP. 65 SINGLE_FACT_LOOKUP. 11 SINGLE_FACT_LOOKUP.
J. Tobkin and D. Yang
27
Benchmark stratum
Scored failures
Denominator
Largest associated route strata
MAB InfBench-Sum
continuous
100
Mean summarization F1 32.77%; min 0.00%; max 91.30%.
These counts are not a human-labeled root-cause taxonomy. They are a reproducible sampling frame for manual failure inspection. The immediate use is to select examples from high-mass cells, inspect the retrieved chunks and predictions, and label each example as retrieval miss, evidence present but synthesis wrong, routing/procedure mismatch, abstentionnormalization issue, or judge disagreement.
D.5
Vector-store and synthesis-probe limitations
The evaluation description uses pgvector-backed retrieval for SCM and the production comparator and ChromaDB for Mem0. No paired ChromaDB-versus-pgvector control holds embedding model, top_k, corpus scale, and scoring path constant. Vector-store choice therefore remains part of the configuration-level comparison. The internal Sonnet probe is summarized as mean 4.22 seconds, median 3.96, p90 7.81, n = 10. Its inputs, per-call rows, script, timestamp, and model provenance are absent. More importantly, a constant offset cannot recover the missing per-question Stage-3 time for LoCoMo and LongMem. The main results therefore omit aligned latency claims for those scored paths.