arXiv CSopen access

TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management

1

TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management

arXiv:2606.06337v1 [cs.AI] 4 Jun 2026

Shweta Mishra

Abstract—Large language model (LLM) deployments for longhorizon tasks face a fundamental constraint: context windows are finite while productive work sessions are not. When session history exceeds the Maximum Effective Context Window (MECW), critical structured information—architectural decisions, task status transitions, file modification histories, and resolved errors—is silently discarded. Existing mitigations (truncation, summarization, retrieval augmentation) treat history as flat text, destroying the typed, relational structure that makes sessions resumable. We present TokenMizer, an open-source proxy system that models LLM session history as a typed knowledge graph. The graph schema defines 14 node types with status lifecycles and 7 semantic edge types. A hybrid extraction pipeline (heuristic with LLM upgrade path) populates the graph incrementally. A three-tier checkpoint system serializes the graph into compact, structured resume blocks. An 8-layer compression pipeline reduces context overhead, and a sentence-embedding semantic cache reduces repeated-query latency. We evaluate TokenMizer on a controlled benchmark of 21 sessions spanning 5 application domains (software engineering, data science, DevOps, research/writing, and debugging) with manually annotated ground truth. All reported results are measured on this benchmark; no estimated values are presented. The benchmark is a synthetic but carefully constructed corpus; evaluation on live developer sessions is identified as the highest-priority future work. The primary result is token economy: TokenMizer produces resume blocks averaging 78 tokens (σ = 21.4, range: 42–124)— 2× smaller than any evaluated baseline (159–170 tokens)—while achieving higher decision recall than all three baselines (+9– 17 percentage points). Decision recall is the key structural metric: no evaluated baseline preserves why a technology was chosen, only that it was mentioned. Across 21 sessions and 5 domains, TokenMizer achieves mean task recall 51.0% (σ = 32.7%), decision recall 46.6% (σ = 32.0%), and file recall 58.7% (σ = 46.7%). High variance reflects genuine domain heterogeneity: sessions with explicit imperative phrasing (software engineering) score substantially higher than sessions with implicit reasoning (research, planning). This distinction is addressed in the Limitations section. Ablation studies establish that fuzzy label matching contributes +33 percentage points of task recall improvement—the dominant single improvement factor. The heuristic compression pipeline achieves 47.3% token reduction without external dependencies. A novel token efficiency metric—recall per 100 resume tokens— reveals debugging sessions as the highest-efficiency domain (mean η = 0.91). TokenMizer is released as open-source software (MIT licence) at https://github.com/Shweta-Mishra-ai/tokenmizer. Index Terms—large language models, context management, session memory, knowledge graph, prompt compression, semantic caching, long-horizon tasks, developer tools, information extraction Independent Researcher, India. E-mail: [email protected]. Code and benchmark: https: //github.com/Shweta-Mishra-ai/tokenmizer. Manuscript received June 4, 2026.

I. I NTRODUCTION

L

ARGE language model (LLM) deployments for software engineering [12], [13], data science [18], and research assistance increasingly involve long-horizon sessions: multiturn interactions spanning hours, where each turn builds on the accumulated context of previous turns. These sessions develop rich internal structure: technology choices made in turn 3 constrain implementation options in turn 12; errors resolved in turn 7 inform test strategies in turn 15; files modified in turn 4 must be updated consistently in turn 18. The fundamental constraint is context window capacity. Paulsen [1] defines the Maximum Effective Context Window (MECW) as the context length at which model task accuracy degrades below an acceptable threshold. Critically, MECW is substantially lower than the advertised maximum context window (MCW) for complex, multi-step tasks. Liu et al. [9] further demonstrate that content placed in the middle of long contexts is systematically under-attended, amplifying the effective degradation. At approximately 950 tokens per turn, a development session exhausts a 16,000-token MECW after only 16 turns. A. Limitations of Existing Approaches Three categories of mitigation exist, each with a structural limitation. Truncation-based methods [19] discard the oldest messages when the context budget fills. This preserves recency but discards exactly the content that matters most: architectural decisions, initial technology choices, and goal definitions established early in a session. Summarization-based methods [10], [7] produce free-text summaries via a secondary LLM call. These compress well but are structurally imprecise: a natural language summary cannot reliably distinguish a completed task from a pending one, nor can it preserve the rationale for a technology decision. The summary of “We decided to use Redis for session storage because it supports TTL natively” and “We considered Redis but chose PostgreSQL” may be informationally indistinguishable. Retrieval-augmented methods [3], [2] retrieve relevant passages on demand using vector similarity. These may miss contextually relevant but semantically distant information: an early environment decision may be critical to a late deployment question yet fail to retrieve because the embedding distance between the two turns is large. A key distinction between TokenMizer and MemGPT [2] is scope. MemGPT targets long-term factual recall across sessions using an OS-inspired hierarchical memory model controlled by the LLM itself. TokenMizer targets per-session

2

structural continuity through a transparent proxy requiring zero application-level changes. Both approaches address different failure modes of the same underlying constraint. B. Key Insight and Contributions The core insight motivating TokenMizer is that session history is not flat text—it is a structured knowledge artifact. Every developer session contains: • Tasks with explicit status lifecycles (pending → inprogress → completed/failed); • Decisions with rationales that constrain future choices; • Files whose modification history defines the codebase state; • Errors and their resolutions; • Environment facts (runtime versions, infrastructure choices) that scope all other nodes. These entities are connected by typed relationships (IMPLEMENTS, FIXES, DEPENDS - ON, etc.) that encode causal and structural dependencies. A representation that preserves this structure enables efficient, queryable context preservation: discard the raw conversation tokens; retain the typed graph; serialize to a compact resume block. The structural encoding also enables a qualitatively different kind of recall: knowing that a decision node encodes a technology choice with its rationale is not equivalent to knowing that the word “Redis” appears somewhere in retained text. Contributions: 1) A formal typed knowledge graph schema for LLM sessions with 14 node types, 7 edge types, and a welldefined status transition system (Section IV-B). 2) A hybrid extraction pipeline with V1/V2 heuristic and LLM upgrade path, with a full ablation study identifying the contribution of each improvement (Section V). 3) A three-tier checkpoint system with graph diffs and token-budgeted serialization (Section VI). 4) An 8-layer heuristic compression pipeline achieving 47.3% token reduction at zero inference cost, with optional neural stages (Section VII). 5) A semantic embedding cache with measured hit rate on a controlled repeated-query workload (Section VIII). 6) A 21-session benchmark spanning 5 domains with ground-truth annotations and a novel token efficiency metric, released publicly (Section X). 7) A correlation analysis of session properties with extraction quality, yielding actionable design implications (Section XI-F). II. BACKGROUND AND R ELATED W ORK A. Context Window Degradation Paulsen [1] introduces the MECW concept, demonstrating empirically that all tested frontier models degrade in accuracy on multi-step tasks before reaching their advertised MCW. For a model with MCW = 128,000 tokens, the MECW may be as low as 16,000 tokens for complex coding tasks. This motivates the 16k-token MECW used as the running example throughout this paper.

Liu et al. [9] demonstrate the “lost in the middle” phenomenon: retrieval and multi-document QA tasks show Ushaped accuracy profiles, with content in the middle of long prompts systematically under-used. This amplifies the effective degradation: even content that nominally fits within MCW may not be reliably attended. B. Memory Systems for LLMs MemGPT [2] proposes an OS-inspired hierarchical memory model distinguishing main context (fast, limited) from external storage (slow, unlimited). Memory management is explicit, controlled by the LLM itself, with paging operations inserted as function calls. MemGPT targets long-term factual recall across sessions; TokenMizer targets per-session structural continuity at zero inference cost. LangChain Memory [10] provides several memory backends including ConversationKGMemory, which is architecturally most similar to TokenMizer. The key differences are: LangChain’s KG memory uses an untyped, schema-free graph; extraction quality and node lifecycle management are developer responsibilities; and the backend requires explicit integration rather than a transparent proxy. TokenMizer provides a formal schema with 14 typed node classes, status lifecycle enforcement, and a validation layer. Active Context Compression (ACC) [7] compresses context by autonomous LLM calls, achieving 22.7% in-session compression. TokenMizer’s heuristic pipeline achieves 47.3% without any inference calls, making it substantially more efficient before neural stages are applied. The tradeoff is coverage: LLM-based extraction addresses implicit phrasing that heuristics cannot reach. C. Prompt Compression LLMLingua [4] and LongLLMLingua [5] use small proxy LLMs to compute per-token importance scores, removing lowimportance tokens at 2–6× compression ratios. TokenMizer incorporates LLMLingua-2 as layers 7–8 in its compression pipeline, applied after heuristic preprocessing has removed structurally trivial content. RECOMP [6] applies selective summarization for retrievalaugmented generation (RAG), retrieving only relevant passages. TokenMizer applies structural extraction rather than textual retrieval: the resume block is generated from graph state, not from retrieved text passages, enabling typed queries (“what decisions were made about authentication?”) that passage retrieval cannot natively support. D. Graph-Based Knowledge Representation GraphRAG [8] builds knowledge graphs from document corpora to improve retrieval quality. The schema is documentoriented (entities, claims, relationships) and designed for static corpora. TokenMizer builds session-oriented graphs with tasklifecycle nodes (TASK status transitions, ERROR/FIX pairs) that are absent from GraphRAG’s schema and cannot be straightforwardly added without redesigning the extraction pipeline.

3

TABLE I Q UALITATIVE C OMPARISON WITH R ELATED S YSTEMS . L ATENCY IS FOR CONTEXT MANAGEMENT OPERATIONS ; COST REFERS TO PER - OPERATION INFERENCE API COST. T OKEN M IZER ’ S 0.5 MS REFLECTS HEURISTIC EXTRACTION ; THE OPTIONAL LLM EXTRACTOR INCURS API COST. System

Struct. graph

Mgmt. latency

Transparent proxy

API cost

Sliding window LangChain KG Mem. MemGPT

None Untyped Vector DB

Yes No No

$0 Yes Yes

ACC RECOMP TokenMizer

None None Typed, 14-class

<1 ms >200 ms 50– 200 ms >100 ms 200 ms 0.5 ms (heuristic)

No No Yes

Yes Yes $0 (heuristic)

III. P ROBLEM F ORMULATION Definition 1 (LLM Session). A session S = ⟨m1 , m2 , . . . , mn ⟩ is an ordered sequence of messages, each with token countPτ (mi ) > 0. The cumulative t token count at turn t is T (t) = i=1 τ (mi ). Definition 2 (Context Overflow). A session overflows at the first turn n∗ such that: T (n∗ ) > TMECW

(1)

For τ̄ = 950 tokens/turn and TMECW = 16,000: n∗ = ⌊16,000/950⌋ = 16. Definition 3 (Structured Resume Block). A resume block R is a compact text representation of the session knowledge graph G, serialized to at most Tbudget tokens. After a checkpoint at turn n∗ − 2, the new context is: C ′ = R ∪ {mn∗ −1 , mn∗ }

(2)

satisfying |C ′ | ≤ TMECW by construction. Definition 4 (Information Loss). L(R) = 1 −

1 Rectask (R) + Recdec (R) + Recfile (R) (3) 3

where each recall term is the fraction of ground-truth entities correctly recovered by fuzzy matching (Definition 6). Equal weighting of the three recall dimensions is a simplifying assumption; domain-specific weighting is identified as future work (Section XIII). Definition 5 (Token Efficiency). 1

η(S) = 3

(Rectask + Recdec + Recfile ) |R|/100

(4)

Token efficiency normalizes mean recall by resume block size (per 100 tokens), enabling cross-session comparison of information density independent of session length. Proposition 1. For any fixed recall level ρ ∈ (0, 1), a lower |R| implies a higher η, motivating the use of structural (graphbased) rather than textual compression.

Fig. 1. TokenMizer system architecture. The proxy sits transparently between any OpenAI-compatible client and the LLM provider. When session_id is present, the five-component pipeline is activated; otherwise the request passes through with no overhead. Persistent storage comprises a SQLite graph database and a JSON checkpoint store.

IV. S YSTEM A RCHITECTURE A. Deployment Model Figure 1 shows the system architecture. TokenMizer operates as a transparent HTTP reverse proxy implementing the OpenAI Chat Completions API. The client configures base_url to the TokenMizer server; no application code changes are required beyond this endpoint substitution. An optional session_id parameter in the request body activates the full pipeline; its absence produces a transparent passthrough with zero overhead, making adoption incremental. The proxy intercepts each request, routes it through the five-component pipeline (graph memory, hybrid extractor, checkpoint manager, compression engine, semantic cache), then forwards the processed request to the configured LLM provider. Nine providers are supported via a common adapter interface. B. Knowledge Graph Schema Figure 2 illustrates a representative session graph. The schema defines 14 node types across three functional categories. 1) Node Types: Table II defines the 14 node types. Action nodes carry status lifecycles; context nodes are static. 2) Node Attributes: Each node carries: Label: normalized surface form (≤120 chars). Status σ ∈ {Pending, In-Progress, Completed, Failed, Modified} (action nodes only). • Importance score ρ ∈ [0, 1]: increases on reference, decays with inactivity. Governs checkpoint serialization priority. • Confidence score κ ∈ [0, 1]: assigned by GraphValidator; nodes with κ < 0.50 are rejected. • Timestamps: creation and last-update turn. • •

4

TABLE III E DGE T YPES (7 TOTAL ). Type

Semantics

DEPENDS_ON RELATED_TO IMPLEMENTS FIXES BLOCKS PART_OF SUPERSEDES

A cannot function without B A and B address the same concern A is the realization of B A resolves error B A cannot proceed until B resolves A is a component of B A replaces or overrides B

V. H YBRID E XTRACTION P IPELINE A. V1 Heuristic Extractor Fig. 2. Session knowledge graph for a FastAPI authentication session (8 representative nodes). Edge labels show typed relationships; node colors indicate type. The GOAL node anchors the hierarchy; TASK nodes carry status (completed/in-progress); the DECISION node records the rationale for choosing bcrypt + JWT.

The baseline extractor applies 34 compiled regular expressions (11 for tasks, 8 for decisions, 5 for files, 4 for errors, 6 for environment and endpoints). Mean extraction latency across the 21-session benchmark: 0.5 ms per session. Inference cost: $0.

TABLE II N ODE T YPES (14 TOTAL ). ACTION NODES CARRY STATUS LIFECYCLES (P ENDING , I N -P ROGRESS , C OMPLETED , FAILED , M ODIFIED ). C ONTEXT NODES ARE STATUS - FREE .

B. V2 Improvements

Type

Category

Primary trigger

TASK FILE ERROR TEST SCHEMA METRIC

Action Action Action Action Action Action

Imperative verb + object Extension or path pattern Error / exception / failure Test / spec / assert Schema / table / model Metric / score / benchmark

DECISION DEPENDENCY API

Decision Decision Decision

Decided / Using / Chose Install / package / library URL / API / endpoint

GOAL ENVIRONMENT PROJECT CONCEPT AGENT

Context Context Context Context Context

First-turn imperative Version / runtime / OS Repository / project name Abstract domain term Service / CI / orchestrator

3) Edge Types: Table III defines all 7 edge types. Edges are directed and labelled; weight encodes co-occurrence frequency. 4) Graph Maintenance: Deduplication uses identity-based addressing: id(n) = sha1(τ (n) : ℓnorm )[: 12], where ℓnorm is the lowercase, whitespace-normalized label. Re-extraction of an existing entity updates importance score and status; it does not create a duplicate node. Status transitions obey monotonic ordering: Pending → In-Progress → Completed/Failed; downgrades are rejected to prevent extraction noise from reverting confirmed progress. Pruning evicts nodes with importance score < 0.1 when the graph exceeds 500 nodes. DECISION, ENVIRONMENT, GOAL, and SCHEMA nodes are unconditionally retained regardless of importance score, as these encode session-defining context.

Error analysis on the 21-session benchmark identifies four systematic failure modes: 1) Incomplete trigger vocabulary. V1 misses 9 common completion verbs: Fixed, Deployed, Resolved, Migrated, Running, Launched, Shipped, Finished, Updated. Sessions with implicit phrasing (“Running: Prometheus + Grafana”) score 0% decision recall on V1. 2) Label verbosity mismatch. The dominant failure: extracted labels are verbose (“Missing NSMotionUsageDescription in Info.plist”) while ground-truth annotations are concise (“ios crash fix”). Exact-string matching yields 0% recall on this pair, even though the extracted label correctly identifies the event. 3) Compound sentences. Single LLM sentences encode multiple tasks: “Completed: API auth, tests, deployment.” V1 extracts one compound node; V2 splits on comma boundaries. 4) CSV-encoded decisions. “Decided: Prometheus, Grafana, Jaeger” should produce 3 decision nodes; V1 produces 1. C. Fuzzy Label Matching Definition 6 (Fuzzy Match). Labels a and b fuzzy-match if: (a ⊆ b) ∨ (b ⊆ a) ∨

|Wa ∩ Wb | ≥ 0.50 min(|Wa |, |Wb |)

(5)

where Wx = {w : w ∈ tokens(x), |w| ≥ 3}. The ≥3-character token filter prevents high-frequency short tokens (“is”, “a”, “in”) from inflating overlap scores. The 50% threshold was determined empirically on a held-out 5-session development split to balance precision (avoiding false matches between unrelated labels) against recall (matching verbose extractions to concise ground-truth).

5

Algorithm 1 Checkpoint Serialization Require: Graph G, budget Tbudget Ensure: Resume block R 1: R ← [[CHECKPOINT: session_id]] 2: N ← sort by importance(G.nodes) 3: for n ∈ N do 4: if τ (R) + τ (serialize(n)) > Tbudget then 5: break 6: end if 7: R.append(serialize(n)) 8: end for 9: ∆G ← G − Gprev 10: store(R, ∆G) to checkpoint store 11: return R

TABLE IV C HECKPOINT R ESUME T IERS WITH E XAMPLE C ONTENTS . Content

Critical

GOAL + in-progress TASKs + top DECISION All TASKs + all DECISIONs + FILEs + ENV Complete graph incl. DEPENDENCIEs, SCHEMAs

Standard Full

Budget ≤ 100 tok ≤ 300 tok ≤ 600 tok

C. Graph Diffs Each checkpoint stores a graph diff : nodes added, modified, or removed since the previous checkpoint. This enables incremental resume block updates across multiple context boundaries without re-extracting the full session history, reducing per-checkpoint cost to O(∆) rather than O(|G|).

D. GraphValidator Every candidate node passes through the GraphValidator before graph insertion. The scoring function is: κ(n) = 0.50 + 0.20 · 1[file path pattern] + 0.10 · 1[|ℓ| > 15] + 0.10 · 1[version number present] − 0.20 · 1[|ℓ| < 8] − 0.40 · 1[single generic verb]

Tier

(6)

Nodes with κ < 0.50 are rejected. Type correction is applied after scoring: extension-matched labels are forced to FILE; URL-pattern labels to API. The scoring weights were set by heuristic design and not optimized; a learned validation model is identified as future work. E. LLM Extractor (Upgrade Path) The codebase includes an LLM extractor that sends the last 3 messages to a small inference model (Claude Haiku or GPT-4o-mini) with a structured JSON prompt requesting task/decision/file/error extraction. This path addresses indirect phrasing (“we’ll go with Redis”, “turns out the issue was. . . ”) that regex patterns cannot capture. The LLM extractor is not evaluated in the present paper: all reported results use only the heuristic pipeline. This conservative choice maintains experimental integrity at the cost of coverage; LLM extractor evaluation is the primary technical future work (Section XIII). VI. C HECKPOINT S YSTEM A. Trigger Condition Checkpoints trigger when cumulative token count exceeds 85% of the configured MECW. This threshold leaves sufficient headroom for the resume block and at least two additional turns before the next overflow event. B. Serialization Algorithm 1 defines checkpoint serialization. Nodes are ordered by importance score; each tier has a hard token budget.

VII. C OMPRESSION P IPELINE The 8-layer compression pipeline is applied to large inputs (files, long API responses, historical messages) before they enter the context window. 1) Layer 1: Filler Removal: Fifteen compiled patterns target AI-generated hedging language: apology prefixes (“Certainly! I’d be happy to...”), discourse markers (“As an AI language model...”), and uncertainty hedges (“Please note that...”). 31.2% reduction on the representative session (Fig. 3). This is the dominant compression layer, confirming that a substantial fraction of LLM output in iterative sessions consists of structurally contentless preambles. 2) Layer 2: Deduplication: Exact line deduplication (orderpreserving) removes repeated code blocks, log lines, and error messages common in iterative development sessions. 16.1% additional reduction. 3) Layers 3–6: Heuristic Normalization: Whitespace normalization (normalize to single spaces; remove blank lines), comment stripping (language-aware: Python #, JS //, SQL --), history pruning (retain last occurrence of assistant messages exceeding 200 tokens), and file-type smart truncation (CSV: schema + 3 sample rows; JSON: non-null fields only; log files: first + last N lines). 4) Layers 7–8: Neural Compression (Optional): LLMLingua-2 [4] (for inputs >300 tokens) and LongLLMLingua [5] (for inputs >4,000 tokens) provide additional 5.3% reduction. A quality gate rejects compressions with semantic similarity below 0.55 (measured by sentencetransformer cosine similarity), preventing over-compression artifacts. VIII. S EMANTIC C ACHE The semantic cache stores LLM responses keyed by sentence-transformer embeddings [11] (all-MiniLM-L6-v2, 384 dimensions). Cache lookup computes cosine similarity between the incoming query embedding and all stored keys; a hit is declared at threshold θ = 0.92. Eviction follows LRU semantics via OrderedDict; TTL is 3,600 s.

6

TABLE V B ENCHMARK C ORPUS (21 S ESSIONS , 5 D OMAINS ). Domain

Sessions

N

Software Eng.

FastAPI, React, GraphQL, Flutter, Microservices, Rust Churn ML, BERT fine-tuning, Spark ETL, Forecasting, Recommender Kubernetes EKS, CI/CD, DB Migration, Security Hardening LLM Survey, Technical Blog, NSF Grant Production Incident, Memory Leak, Performance Regression

6

Data Science DevOps Research Debugging Fig. 3. Compression pipeline staged token reduction on a representative 747token session. Filler removal (Layer 1) contributes 31.2% reduction—the dominant stage. Layers 1–6 require zero external dependencies (heuristic only). LLMLingua-2 (Layer 7) adds an optional 5.3%. Total heuristic reduction: 47.3%.

Preliminary hit rate measurement: 70% across a controlled 10-query workload with three semantic clusters (cluster size 3–4 queries). This represents a proof-of-concept measurement on a small synthetic workload; the 70% figure should not be interpreted as a production hit rate estimate. Evaluation on real developer query logs at scale is identified as future work. IX. I MPLEMENTATION TokenMizer is implemented in Python 3.10+, comprising ∼4,500 lines of code across seven modules: graph_memory, checkpoints, compression, cache, providers, validator, and api. The HTTP proxy uses FastAPI [14] and Uvicorn. Token counting uses tiktoken [16] (cl100k_base). The knowledge graph is persisted in SQLite [15] using three tables: nodes, edges, and checkpoints. Security: all extraction inputs are scanned for API keys, private keys, and credential patterns before graph insertion; matched content is redacted to [REDACTED]. Testing: 306 tests across unit, integration, and chaos test categories; 100% coverage of graph mutation operations. Configuration: YAML-based, with all thresholds (MECW percentage, confidence floor, cache similarity, compression minimum) exposed as user-configurable parameters. X. E XPERIMENTAL S ETUP A. Benchmark Corpus All results in this paper use a single, consistent 21-session benchmark. Table V details the domain distribution. Sessions are manually constructed to reflect realistic developer workflows, covering project initialization, iterative implementation, error resolution, architectural decision points, and deployment. We note a key scope constraint: the benchmark is synthetic. Sessions were constructed by the author to be structurally representative of each domain, not collected from live developer interactions. Linguistic diversity—particularly in implicit phrasing and domain-specific jargon variation—may therefore be underrepresented. Evaluation on real developer sessions is the highest-priority future work; results on this benchmark

Total

5 4 3 3 21

should be interpreted as controlled proof-of-concept evidence, not production performance estimates. B. Ground Truth Annotation Each session is annotated with: • Completed tasks: actions explicitly marked as finished (e.g., “tests pass”, “deployed”). • Pending tasks: actions initiated but not completed within the session. • Decisions: technology choices, algorithm selections, infrastructure decisions. • Files: source files created or modified. Annotation was performed by the paper author. A secondannotator inter-rater reliability study is planned as future work; reported recall values should be interpreted with this limitation in mind. C. Evaluation Metrics Recall (task, decision, file): for each category, fraction of ground-truth labels that fuzzy-match at least one extracted label (Eq. (5)). Information Loss: Eq. (3). Resume Tokens: token count of the standard-tier checkpoint block (tiktoken cl100k_base). Compression Ratio: original session tokens / resume tokens. Token Efficiency: Eq. (4). Extraction Latency: wall-clock time for heuristic graph extraction. D. Baselines We compare against three text-retention baselines that represent the current practice for context management in production LLM applications: 1) Naive Truncation (NT): retain last 300 tokens of concatenated history. 2) Sliding Window (SW-10): retain last 10 messages. 3) Naive Summary (NS): first 300 tokens of fullyconcatenated history (approximating a manual summary).

7

TABLE VI E XTRACTION R ECALL BY D OMAIN (V2, 21 S ESSIONS ). VALUES ARE MEAN ± STANDARD DEVIATION . W IDE STANDARD DEVIATIONS WITHIN DOMAINS ( ESPECIALLY SE TASK RECALL : ±47%) REFLECT GENUINE INTER - SESSION LINGUISTIC VARIABILITY, NOT MEASUREMENT ERROR . Domain

Task R

Dec R

File R

Info Loss

SE (n=6) DS (n=5) Ops (n=4) Res (n=3) Dbg (n=3)

47±47% 69±24% 45±21% 44±39% 43±29%

70±20% 48±33% 38±17% 44±51% 11±19%

72±46% 40±55% 50±58% 33±58% 100±0%

37% 48% 56% 59% 49%

All (n=21)

51±33%

47±32%

59±47%

48%

TABLE VII T OKEN M IZER V2 VS . BASELINES (21 S ESSIONS ). B OLD : BEST PER COLUMN . “T OKENS ”: MEAN RESUME BLOCK SIZE . D IFFERENCES ARE DESCRIPTIVE ; THE SMALL EVALUATION SET (n=21) LIMITS INFERENTIAL CONCLUSIONS . Method

Task R

Dec R

File R

Tokens

Naive Truncation Sliding Window (10) Naive Summary

45% 50% 42%

35% 30% 38%

55% 60% 48%

165 159 170

TokenMizer V2

51%

47%

59%

78

Fig. 5. Recall metrics (left) and token overhead (right) for all four methods across 21 sessions. TokenMizer achieves the highest decision recall across all evaluated baselines (+9–17 pp) while requiring 2× fewer tokens. File recall is comparable to Sliding Window (59% vs. 60%), which is expected: recent messages frequently name the files they modify. Fig. 4. Per-domain recall with 95% confidence intervals (t-distribution, df = n − 1). Wide CIs for software engineering (task recall) and research (decision recall) reflect high inter-session variability. Small per-domain n (3–6) limits statistical precision; these results should be read as descriptive rather than inferential.

Baseline recall is computed by applying the same fuzzymatching protocol to the retained text, counting matched ground-truth entities. These baselines represent practical current-practice approaches, not structured competitors. The key comparison is not accuracy parity but structural capability: baselines can match surface-form mentions of entities but cannot encode their status (task completed vs. pending) or rationale (why a technology was chosen). This qualitative difference in representation is the primary motivation for the graph-based approach. XI. R ESULTS A. Extraction Quality by Domain Table VI presents extraction results with standard deviations. Figure 4 visualizes per-domain recall with 95% confidence intervals. Domain-level analysis reveals four patterns. Software engineering achieves the highest decision recall (70%) because developers habitually state choices explicitly (“Decided: bcrypt for hashing”). Data science leads in task recall (69%) due to outcome-oriented statements (“MAPE: 3.6%”, “F1=0.71”) that uniquely match ground-truth labels. Debugging achieves 100% file recall (log messages and stack traces consistently

name files) but only 11% decision recall, because diagnostic reasoning is typically expressed implicitly (“the culprit was...”) rather than as explicit choice statements. Research and writing sessions score lowest overall: academic prose avoids the imperative sentence patterns that trigger heuristic extraction.

B. Baseline Comparison Table VII and Figure 5 compare TokenMizer V2 against three baselines. TokenMizer’s advantage is most pronounced for decision recall (+9 pp over naive summary, +12 pp over truncation, +17 pp over sliding window). No baseline preserves decision rationale: they retain text that may contain decision keywords, but the decision is not structurally encoded. This means a baseline system cannot distinguish “We chose Redis because it supports TTL” from “We rejected Redis in favor of PostgreSQL”; a graph node of type DECISION does. Task recall is marginally superior to all baselines (+1–9 pp); file recall is comparable to sliding window (59% vs. 60%), which is expected because recent messages tend to explicitly mention the files they modify. The token efficiency advantage is the most robust result. At 78 tokens average, TokenMizer’s resume blocks are 2.0–2.2× smaller than any baseline (159–170 tokens). This advantage is structural: graph serialization is concise by design, independent of session verbosity.

8

TABLE VIII A BLATION : C UMULATIVE C ONTRIBUTION OF E ACH V2 F IX (10- SESSION SAMPLE ). VALUES ARE MEANS ; ERROR BARS OMITTED FOR CLARITY. E VALUATED ON THE SAME 10 SESSIONS THROUGHOUT TO ENSURE COMPARABILITY. Configuration V1 baseline + expanded trigger vocab. + fuzzy label matching + compound task splitting + decision CSV splitting

Task R

Dec R

Info Loss

22% 20% 55% 55% 55%

63% 63% 68% 68% 64%

50% 51% 38% 38% 39%

Fig. 6. Ablation study: cumulative contribution of each V2 improvement on a 10-session sample. Fuzzy matching dominates (+33 pp task recall), addressing the label verbosity mismatch. Decision CSV splitting introduces a −4 pp regression in decision recall; the mechanism is discussed in Section XI-C.

C. Ablation Study Table VIII and Figure 6 report the cumulative contribution of each V2 fix, evaluated on a 10-session sample. The dominant result is that expanded trigger vocabulary alone provides no measurable benefit (±0 pp) or slight harm (−2 pp task recall). Expanded patterns match more candidates but introduce false positives that exact matching penalises. Fuzzy matching then recovers and exceeds the V1 task recall (+33 pp), confirming that the fundamental bottleneck is the matching function, not extraction coverage. Decision CSV splitting marginally decreases decision recall (−4 pp). Analysis reveals the mechanism: individual split tokens (“Prometheus”, “Grafana”) are shorter than the original compound string (“Prometheus + Grafana”) and match fewer ground-truth labels under the current fuzzy-match threshold. Applying fuzzy matching per-token rather than per-node would address this regression; this is identified as a concrete future improvement (Section XIII). D. Resume Token Analysis Figure 7 shows per-session resume block sizes. Mean: 78.1 tokens (σ = 21.4, range: 42–124). The resume block size correlates moderately with total session tokens (r = 0.655), confirming that more complex sessions produce more structured extractable content. The correlation is not strong (r < 0.70), indicating that session length and information density are partially independent: a

Fig. 7. Per-session resume block size (colored by domain). The minimum (42 tokens, security hardening) reflects few explicit files and implicit decision phrasing. The maximum (124 tokens, FastAPI auth) reflects many explicit tasks, decisions, and named files. Dashed line: 78-token benchmark mean.

Fig. 8. Cumulative token count without (red) and with (blue) TokenMizer for a 20-turn session at 950 tokens/turn. At turn 14 (85% MECW), the 13,300-token history is replaced by a 78-token resume block, resetting cumulative context and allowing the session to continue beyond the MECW without overflow.

verbose session may produce few structured entities, and a concise session may encode many. E. Context Window Recovery Figure 8 illustrates the recovery mechanism for a representative 20-turn session. F. Token Efficiency and Correlation Analysis Figure 9 presents the per-session token efficiency vs. information loss scatter plot. The Pearson correlation between efficiency and information loss is r = −0.21 (weak negative), indicating that sessions with lower information loss tend to have higher token efficiency—but the relationship is noisy, driven by domain-level variation rather than a strong systematic effect. Additional correlation findings from the 21-session dataset: • Task recall vs. session length (r = 0.15): weak positive— longer sessions use more explicit completion phrasing on average. • Information loss vs. session length (r = −0.59): moderate negative—longer sessions accumulate more structured content relative to the resume budget, suggesting that TokenMizer provides proportionally greater benefit for extended sessions.

9

XII. D ISCUSSION A. The Structural Recall Argument

Fig. 9. Token efficiency (Eq. (4)) vs. information loss per session, colored by domain. Pearson r = −0.21 (weak negative). Debugging sessions (red) achieve high efficiency due to file-path specificity in stack traces. Research sessions (purple) are low-efficiency due to implicit phrasing that heuristic extraction cannot capture.

The 47% mean decision recall reported here should be read carefully. On one reading, it means the system misses more than half of all decisions—a significant limitation. On another reading, the 47% that is captured is typed: the system knows that a DECISION node records a technology choice with rationale, not merely that a technology name appears somewhere in retained text. A baseline achieving 35–38% decision recall is not merely 9–12 pp worse; it is qualitatively different in what it preserves. Whether this structural distinction matters in practice depends on how the resume block is used. For a downstream system that only needs to re-establish lexical context (what tools were mentioned), baselines may suffice. For a system that needs to query session state (what decisions are locked in and why, which tasks are still open), the graph representation provides capabilities that no text-retention baseline can offer. B. When TokenMizer Helps Most The correlation analysis in Section XI-F suggests that TokenMizer provides proportionally greater benefit for longer sessions (r = −0.59, information loss vs. session length). The domain analysis suggests it works best for sessions with explicit imperative phrasing: software engineering and data science outperform research and planning workflows substantially. Practitioners should expect lower benefit for sessions dominated by exploratory discussion, academic writing, or high-level planning without concrete action statements.

Fig. 10. Node type distribution (left, 21 sessions, N =123 total nodes) and GraphValidator mean confidence by type (right). All 8 primary node types exceed the 0.50 rejection threshold. GOAL achieves the highest confidence (0.95) because first-turn imperatives match extraction patterns reliably. ERROR scores lowest (0.74) due to high phrasing variability from stack traces to natural language.

•

Compression ratio vs. total tokens (r = −0.03): nearzero—compression ratio is effectively independent of session length, confirming that the resume block size is governed by information structure rather than session verbosity.

C. Token Cost and Efficiency The 78-token average resume block is a structural property of the graph serialization format, not an artifact of benchmark construction. Appendix B shows that block sizes range from 42 tokens (minimal structured content) to 124 tokens (dense, explicit workflows) across the 21 sessions. The 2× token advantage over baselines is thus consistent across all sessions in the corpus and represents a reliable, design-level property of the approach. To illustrate the practical implication without overstating it: at current LLM input pricing, the 78-token average versus 165token baseline average means roughly half the token cost per resume injection. Actual cost savings depend on usage volume and pricing, which vary across providers and over time.

G. Graph Statistics and Validator Performance XIII. L IMITATIONS AND F UTURE W ORK Figure 10 presents node type distribution and validator confidence across the benchmark. TASK (30%) and FILE (25%) nodes are the most frequent types, consistent with the task-oriented nature of developer sessions. GOAL nodes achieve the highest confidence (0.95), as first-turn imperative sentences reliably match extraction patterns. ERROR nodes score lowest (0.74) because error descriptions vary from highly structured stack traces to vague natural language descriptions (“it just crashes on startup”).

L1: Synthetic benchmark. Sessions were manually constructed by the author; they were not collected from live developer workflows. Linguistic diversity, phrasing variability, and domain jargon may be underrepresented. Evaluation with real annotated sessions is the highest-priority future work. L2: Single annotator. Ground-truth annotations were performed by the paper author. An inter-annotator agreement study is planned; until then, reported recall values should be interpreted as estimates subject to annotation subjectivity.

10

L3: Decision recall ceiling (47%). Indirect phrasing (“let’s go with Redis”, “I think PostgreSQL makes more sense here”) is not captured by heuristic patterns. The LLM extractor upgrade path is implemented and is expected to substantially improve decision recall on implicit phrasing; its evaluation is deferred to future work. L4: Equal metric weighting. Equation (3) weights task, decision, and file recall equally. Domain-specific weighting (decision recall matters more in architecture sessions; file recall in debugging sessions) is not explored. A weighted or learned metric formulation would more accurately reflect deployment priorities. L5: Decision CSV splitting regression. The −4 pp decision recall regression from CSV splitting should be recoverable by applying fuzzy matching to individual split tokens rather than per-node strings. L6: Shallow edge linking. Current edges use vocabulary overlap for node co-occurrence detection. Embedding-based semantic edge linking is architecturally supported (sentencetransformers is a project dependency) but not yet implemented. L7: No cross-session memory. Each graph is scoped to a single session. The SQLite backend would support crosssession retrieval (“what did I decide about Redis last month?”) but the retrieval layer is not yet implemented. L8: Zero-recall outliers in implicit-phrasing sessions. Several sessions exhibit 0% task recall: flutter_fitness_app and grant_proposal_nsf in particular. These reflect a fundamental limitation of heuristic pattern matching against highly implicit language, not extraction failures in the traditional sense. flutter_fitness_app describes completed work through consequence (“iOS build working now”) rather than explicit completion (“Completed: iOS crash fix”); grant_proposal_nsf uses academic prose throughout (“We propose to...”, “The objective is...”) with no trigger phrases matching any V2 pattern. Research and planning workflows represent a domain where heuristic extraction is structurally insufficient. The LLM extractor upgrade path is expected to address this class of failure; all zero-recall outliers are reported transparently in Appendix C and included in all aggregate statistics. L9: Significance testing. With n=21 and per-domain n = 3– 6, the observed differences between TokenMizer and baselines are not assessed for statistical significance. Formal hypothesis testing (e.g., Wilcoxon signed-rank tests for paired recall differences) requires a larger evaluation set. This is a key priority for future work alongside real-session data collection. XIV. C ONCLUSION This paper presents TokenMizer, a system that addresses LLM context loss by modeling session history as a typed knowledge graph and serializing it into compact, structured resume blocks. Evaluated on a controlled 21-session benchmark across 5 domains—all values measured, no estimates reported— TokenMizer achieves: • 51% task recall, 47% decision recall, 59% file recall with 78-token resume blocks;

+9–17 pp decision recall advantage over all three evaluated baselines at 2× fewer tokens; • 47.3% heuristic compression with zero external inference dependencies; • 0.5 ms extraction latency. Ablation analysis establishes that fuzzy label matching is the dominant technical contributor (+33 pp task recall). Token efficiency analysis identifies debugging as the highestefficiency domain and research/writing as the lowest, pointing to domain-adaptive extraction as a productive future direction. The moderate correlation between information loss and session length (r = −0.59) suggests that longer sessions benefit proportionally more from structured checkpointing. The core contribution is representational rather than merely quantitative: sessions have structure, not just content. A typed graph of TASK status transitions, DECISION rationales, and FILE modification histories enables compact, queryable, and reproducible session state that text-retention approaches cannot match—at half the token cost of any evaluated baseline, and at zero inference overhead. •

ACKNOWLEDGMENT The author thanks the open-source communities behind FastAPI, SQLite, sentence-transformers, and LLMLingua for the foundational tools on which TokenMizer is built. R EFERENCES [1] N. Paulsen, “Context Is What You Need: The Maximum Effective Context Window,” arXiv preprint arXiv:2509.21361, 2025. [2] C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez, “MemGPT: Towards LLMs as Operating Systems,” arXiv preprint arXiv:2310.08560, 2023. [3] P. Lewis et al., “Retrieval-Augmented Generation for KnowledgeIntensive NLP Tasks,” in Proc. NeurIPS, 2020, pp. 9459–9474. [4] H. Jiang et al., “LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models,” in Proc. EMNLP, 2023, pp. 13358–13376. [5] H. Jiang et al., “LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios,” arXiv preprint arXiv:2310.06839, 2023. [6] F. Xu et al., “RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation,” in Proc. ICLR, 2024. [7] C. Smith and J. Park, “Active Context Compression for Long-Horizon LLM Sessions,” arXiv preprint arXiv:2601.07190, 2026. [8] E. S. Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” arXiv preprint arXiv:2404.16130, 2024. [9] N. F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” Trans. ACL, vol. 12, pp. 157–173, 2024. [10] H. Chase, “LangChain,” https://github.com/langchain-ai/langchain, 2022. [11] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks,” in Proc. EMNLP, 2019, pp. 3982– 3992. [12] J. Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” in Proc. NeurIPS, 2024. [13] GitHub, “GitHub Copilot,” https://github.com/features/copilot, 2024. [14] S. Ramı́rez, “FastAPI,” https://fastapi.tiangolo.com, 2018. [15] D. R. Hipp, “SQLite,” https://www.sqlite.org, 2000. [16] OpenAI, “tiktoken: Fast BPE Tokeniser,” https://github.com/openai/ tiktoken, 2023. [17] Z. Feng et al., “CodeBERT: A Pre-Trained Model for Programming and Natural Languages,” in Proc. EMNLP Findings, 2020, pp. 1536–1547. [18] X. Hou et al., “Large Language Models for Software Engineering: A Systematic Literature Review,” arXiv preprint arXiv:2308.10620, 2024. [19] T. Brown et al., “Language Models are Few-Shot Learners,” in Proc. NeurIPS, 2020, pp. 1877–1901.

11

[20] D. Jin et al., “GemFilter: Fast and Accurate Context Selection via Filtering Layers,” arXiv preprint arXiv:2409.17422, 2024.

A PPENDIX # V2: 9 additional completed triggers _COMPLETED = re.compile( r’(?:Completed|Implemented|Fixed|Added’ r’|Deployed|Resolved|Migrated|Launched’ r’|Shipped|Published|Done|Finished’ r’|Updated)[:\s]+(.+?)(?:\.|$)’, re.I | re.M) # V2: 3 additional decision triggers _DECISION = re.compile( r’(?:Decided|Using|Running|Tools?’ r’|Chose|Selected)[:\s]+(.+?)(?:\.|$)’, re.I | re.M) # Fuzzy match (Definition 5) def fuzzy_match(a: str, b: str) -> bool: a, b = a.lower(), b.lower() if a in b or b in a: return True wa = set(re.findall(r’\w{3,}’, a)) wb = set(re.findall(r’\w{3,}’, b)) if not wa or not wb: return False shorter = wa if len(wa) <= len(wb) else wb return len(wa & wb) / len(shorter) >= 0.5

FastAPI Auth (122 tokens — standard tier): [CHECKPOINT: fastapi_auth] DONE: user model, auth endpoints, fix 422 error, 12 tests passing PENDING: refresh tokens, rate limiting DECIDED: python 3.12, bcrypt, jwt, redis, postgresql FILES: api/auth.py, api/models.py, tests/test_auth.py ENV: FastAPI, Python 3.12

Security Hardening (42 tokens — minimum in corpus): [CHECKPOINT: security_hardening] DONE: security scan, critical fixes, cve remediation, waf setup PENDING: 3 remaining cves

Table IX presents complete per-session results for full reproducibility. Zero-recall sessions (flutter_fitness_app, grant_proposal_nsf) are highlighted and discussed in Section XIII.

12

TABLE IX F ULL P ER -S ESSION R ESULTS (V2, 21 S ESSIONS ). TR: TASK RECALL ; DR: DECISION RECALL ; FR: FILE RECALL ; IL: INFO LOSS ; RT: RESUME TOKENS ; CR: COMPRESSION RATIO ; EFF: TOKEN EFFICIENCY; MS: EXTRACTION LATENCY ( MS ). ROWS WITH TR = 0% REPRESENT IMPLICIT- PHRASING SESSIONS WHERE HEURISTIC EXTRACTION IS STRUCTURALLY INSUFFICIENT ( SEE S ECTION XIII, L8). Session

Domain

TR

DR

FR

IL

RT

CR

EFF

MS

fastapi auth react dashboard graphql ecommerce flutter fitness app microservices migration rust cli tool

SW Eng SW Eng SW Eng SW Eng SW Eng SW Eng

100% 20% 100% 0% 0% 60%

60% 100% 67% 80% 71% 40%

100% 100% 33% 0% 100% 100%

13% 27% 33% 73% 43% 33%

122 124 104 66 95 70

1.7x 1.7x 1.8x 2.3x 1.8x 2.1x

0.71 0.59 0.64 0.40 0.60 0.95

1.4 0.5 0.4 0.4 0.3 0.4

ml churn prediction bert sentiment spark etl pipeline timeseries forecast recommendation system

Data Sci Data Sci Data Sci Data Sci Data Sci

40% 50% 80% 100% 75%

60% 20% 40% 100% 20%

50% 50% 100% 0% 0%

50% 60% 27% 33% 68%

90 87 98 75 59

2.0x 1.9x 1.7x 1.9x 2.4x

0.56 0.46 0.75 0.89 0.54

0.5 0.4 0.3 0.5 1.3

kubernetes eks cicd github actions db migration security hardening

DevOps DevOps DevOps DevOps

40% 40% 75% 25%

20% 40% 33% 60%

100% 100% 0% 0%

47% 40% 64% 72%

72 71 67 42

2.2x 2.3x 2.2x 3.4x

0.74 0.85 0.54 0.68

0.9 0.4 0.5 0.8

llm agents survey technical blog rag grant proposal nsf

Research Research Research

67% 67% 0%

100% 0% 33%

0% 100% 0%

44% 44% 89%

71 74 77

2.3x 2.0x 1.8x

0.78 0.75 0.14

0.8 0.3 0.3

production incident memory leak nodejs perf regression api

Debugging Debugging Debugging

75% 20% 33%

0% 33% 0%

100% 100% 100%

42% 49% 56%

46 67 63

4.3x 2.6x 2.5x

1.27 0.76 0.71

0.4 0.4 0.3

51% 33%

47% 32%

59% 47%

48% 18%

78 21

2.2x 0.6x

0.66 0.24

0.5 0.3

Mean Std dev