PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM Agents

PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM Agents

arXiv:2611.10240v1 [cs.LG] 6 Nov 2026

Suman Chatterjee Amazon Web Services

Abstract Long-horizon LLM agents accumulate sessions of up to millions of tokens, far more key-value (KV) cache than one GPU holds, so serving systems keep a small budget of tokens and evict the rest. Eviction methods differ in which tokens they keep but share one goal: that the tokens a later step needs are still cached. We show this is not enough. In 100K-token coding sessions on Qwen3-8B with a 2,048-token budget, a planner-guided policy that keeps over 99% of the needed tokens resident still fails late steps increasingly often: its accuracy falls from 93% to 64% as the gap between a definition and its use grows from 2K to 90K tokens, at residency 0.996. The needed tokens are resident but unretrievable: what grows is their rotary-position distance to the query. We present PACE (Planner-Aware Context Engine), a KV-cache engine that couples plan-aware retention, in which the agent’s planner tells the cache which context future steps will read, with bounded positions, which renumber kept tokens so their rotary distance to the query stays below 8K at any session length. In a pre-registered evaluation on fresh tasks, PACE passes 97% of hidden unit tests at 100K tokens, against 42% for H2 O and 3% for SnapKV under the same budget, and needs both components: either alone reaches 63% (the combination beats each, p < 10−4 ). The result holds with length: in 1M- and 2M-token sessions on one 24 GB GPU, with definitions written 0.9M and 1.8M tokens before use, PACE passes 99% and 100% of checks and recalls every session fact, while bounded positions without foresight falls from 33% to 13%; PACE holds under 0.7 GB of KV state where the full cache would need 147–295 GB. Renumbering is exact, costs no measurable perplexity, and adds 9% wall time. On Mistral-7B, whose coding ability caps it at 40% even with nothing evicted, PACE stays at that ceiling while retention alone falls below it.

1

Introduction

LLM agents that write code, operate tools, or carry out multi-step plans do not answer one prompt; they run a session. Every step appends an instruction, retrieved material, and the model’s own output, and later steps depend on definitions made much earlier: a data model declared at step 1, a business rule fixed at step 3, an interface agreed at step 10. Sessions of 105 –106 tokens are routine for repository-scale coding or long tool-use traces. The key-value (KV) cache that attention needs grows linearly with the session: for an 8B-parameter model with grouped-query attention (Qwen3-8B), it costs 144 KiB per token, so 14.7 GB at 100K tokens and 147 GB at 1M. A single 24 GB accelerator cannot hold it, so serving systems fix a KV budget and evict. A large body of work studies which tokens to keep. Attention-based policies retain “heavy hitters” that past queries attended to [15, 31], query-aware policies score tokens against the current prompt [3, 12, 25], and streaming policies keep attention sinks plus a recent window [28]. More recently, agent-aware systems exploit knowledge of the workflow to predict which context will be reused 1

[16, 19, 26, 30, 32]. These methods differ in how they choose, but they share one objective: when a later step needs a token, the token should still be in the cache. Keeping is not enough. We find that this objective, even when met perfectly, does not deliver correct behavior in long sessions. Figure 1 shows coding sessions of 100K tokens on Qwen3-8B with a budget of 2,048 tokens (2% of the session). A retention policy guided by the agent’s own plan keeps 99.6% of the needed tokens resident at every gap we test, yet its accuracy falls from 93% to 64% as the definitions recede from 2K to 90K tokens before their use. Pooled across configurations, accuracy at full residency declines from 96–100% when the needed context is within 8K rotary positions of the query to 69–77% beyond 16K (Figure 3). The needed tokens are resident but unretrievable: retention fixed which tokens survive, but not where they sit relative to the query in rotary-position space. Standard eviction suffers both problems at once: under the same budget, H2 O reaches 40–55% across our runs and SnapKV 3%. PACE. We present PACE (Planner-Aware Context Engine), a KV-cache engine that addresses both. Plan-aware retention turns the dependency graph an agent planner already produces into a per-span foresight bonus that dominates attention scores, so the context future steps will read is kept, soonest-needed first. Bounded positions renumber the kept tokens in order whenever positions approach a limit L, re-rotating their keys exactly, so the rotary distance between any kept token and the query never exceeds L, at any session length. With L = 8,192, the smallest limit the budget allows, needed context stays within 2–7K positions of the query whether it was written 2K or 1.8M tokens earlier. PACE also makes sessions persistent, so each step sends only its new tokens, and spills evicted context to host memory, local storage, and a remote cache from which the planner can restore it. Results. In a pre-registered evaluation on tasks the method had never seen, PACE passes 97% of hidden unit tests at 100K tokens, against 42% for H2 O; it recalls every session fact; and it beats each of its two components alone (p < 10−4 ), each of which reaches only 63%. The distance sweep isolates the cause: retention alone decays with distance at full residency, while PACE stays at 97–100% from 2K to 90K. With a production-style LLM planner instead of the true dependency graph, PACE still reaches 92% against 41%. The result holds with length: in 1M-token sessions on one 24 GB GPU, with definitions written about 900K tokens before their use, PACE passes 99% of checks (74/75) and recalls every fact (125/125), against 33% without foresight and 0% for a StreamingLLM-style cache, with under 0.7 GB of KV state where the full cache would need 147 GB. At 2M tokens, with definitions 1.8M tokens back, PACE passes all 75 checks and recalls all 125 facts, while bounded positions without foresight falls further, to 13% of checks and 50% recall: the margin from foresight widens with length. On Mistral-7B, whose coding ability caps it at 40% of checks even with nothing evicted, PACE stays at that ceiling from 2K to 90K and recalls 96–100% of facts, while retention alone drops below it as the gap grows and H2 O collapses to 1%. Renumbering is token-identical to the baseline when inactive, costs no measurable perplexity, and adds 9% wall time. Scope. PACE is not a million-token context window. At every step the model attends to about 2K kept tokens, chosen by the plan and by attention; the rest of the session is evicted to host memory and storage. PACE targets agent workloads whose planner can state what later steps depend on, such as coding, multi-step tool use, data pipelines, and multi-agent hand-offs. Without a plan it reduces to attention-based eviction with bounded positions, and full-context serving remains preferable for unplanned recall of arbitrary details from an entire history. Contributions. 1. A measured finding: in long-horizon agent sessions under a fixed KV budget, keeping the 2

Hidden-test checks passed (%)

100 75

PACE (L = 8K) retention only H2 O

50 25 0

retention only: needed tokens resident 0.996 at every gap

2K 10K 25K 50K 90K Gap between definitions and their use (logical tokens)

Figure 1: Keeping the right tokens is not enough. 100K-token coding sessions, Qwen3-8B, KV budget 2,048 tokens. Planner-guided retention keeps 99.6% of the tokens each late step needs, yet its accuracy falls as those tokens recede; H2 O loses them outright. PACE adds bounded positions, holding the rotary distance to the query below 8K, and stays at 97–100%. Five randomized sessions per point, three probes each. needed tokens is necessary but not sufficient. At residency above 0.99, accuracy falls with the rotary-position distance between the needed context and the query (Section 5.3). 2. PACE, which couples plan-aware retention with bounded, exactly re-rotated positions, plus persistent sessions and tiered rehydration, implemented in an open-source inference engine (Section 3). We identify and fix a compounding-rounding failure of naive re-rotation that appears only in very long sessions (Section 3.2, Appendix D). 3. An execution-scored, pre-registered long-horizon benchmark protocol with randomized specifications, leak checks, a no-eviction control, and a recall metric that separates forgetting from coding errors (Section 4), and an evaluation showing at 100K tokens that both components are required and, up to 2M tokens, that PACE keeps its quality (Section 5).

2

Background and Problem

KV eviction under a budget. An autoregressive transformer caches, for every past token and every layer, a key and a value vector. Under a budget of B tokens per layer, an eviction policy decides after each prefill chunk or every few decoded tokens which B entries to keep. H2 O [31] keeps the tokens with the largest cumulative attention received, plus a recent window; SnapKV [12] ranks tokens by the attention of the last few prompt tokens; StreamingLLM [28] keeps the first few “sink” tokens and a sliding window. All of them are retrospective: they judge a token by how it has been used, or by how the current query uses it. Rotary positions. Most open LLMs encode position with rotary embeddings (RoPE) [24]. A query at position m and a key at position n are rotated, pairwise per frequency θi , by angles mθi and nθi , so their dot product depends on positions only through the difference: ⟨Rm q, Rn k⟩ = ⟨q, Rn−m k⟩,

Ra Rb = Ra+b .

(1)

Engines cache keys after rotation. Eviction removes keys but leaves the survivors’ positions untouched, so in a long session the query’s position keeps growing while a kept definition stays 3

Agent planner dependency DAG

blueprint

Serving layer blueprint → token spans PACE (KV-cache manager)

Plan-aware retention foresight bonus, Eq. (2)

Bounded positions renumber ≤ L, exact re-rotation

Persistent session suffix-only turns

keep top-B

GPU KV cache budget B tokens

evict restore (planned)

Evicted context host RAM → disk → remote

Figure 2: PACE inside the engine’s KV-cache manager. The planner’s dependency graph becomes a per-span foresight bonus that decides which tokens stay; bounded positions renumber the kept tokens so their rotary distance to the query stays below L; persistent sessions carry the cache across steps; evicted context is tiered and restored when the plan needs it. where it was written. Two consequences follow. First, the session cannot outgrow the model’s positional range even though only B tokens are resident; context-extension methods such as YaRN [20] enlarge that range by rescaling frequencies. Second, the rotary gap between the query and a kept token grows with the session. Agent plans. Agent frameworks commonly decompose a task into steps with explicit dependencies before executing it, for example a directed acyclic graph in which step 7 depends on steps 1 and 3. That graph is a statement about the future that the cache never sees: it says which context each upcoming step will read. Problem statement. Let step t issue its query at logical (session) position mt , which the model sees at physical rotary position m̂t , and let Nt be the set of earlier tokens it needs (the spans its dependencies point to). With Ct the resident set, define the residency ρt = |Nt ∩ Ct |/|Nt | (minimum over layers), and the rotary gap ∆t = m̂t − p(Nt ), the distance in physical rotary positions, as the model sees them, from the query to the start of the needed span (where without renumbering, m̂t = mt and p(Nt ) = pos(Nt )). Eviction policies, including agent-aware ones, maximize ρt . This paper asks whether ρt ≈ 1 suffices, and shows that it does not: at ρt ≥ 0.99, accuracy still falls with ∆t (Section 5.3). PACE therefore controls both quantities.

3

PACE

PACE sits inside the inference engine’s KV-cache manager (Figure 2). It changes three things: which tokens are kept (Section 3.1), which positions the kept tokens carry (Section 3.2), and how a session’s state lives across steps and memory tiers (Sections 3.3 and 3.4). With all three disabled it reduces to H2 O token for token (Section 5.7), which makes every ablation a controlled comparison.

4

3.1

Plan-aware retention

Blueprint. Before each step, the agent sends the planner’s dependency information as a blueprint: a list of segments, each naming earlier messages of the session and the future steps that will read them. The serving layer renders the chat template, locates each message’s token span, and converts the blueprint into a forward-reference table of spans (a, b, F ), where F is the set of future steps reading tokens [a, b). No model change is required; the blueprint travels as an extra request field. Foresight bonus. At step t, a token in span (a, b, F ) with next reference s⋆ = min{s ∈ F : s ≥ t} receives   1 1 −λ(s⋆ −t) βt = σ + e , (2) 2 2 and tokens with no future reference receive 0. The bonus is added to the token’s cumulative attention score, and eviction keeps the top-B by the sum. The scale σ (105 , giving a floor of σ/2 = 50K) exceeds every cumulative attention score we observe (at most 25K, in 2M-token sessions), so every referenced token outranks every unreferenced one, the core commitment of foresight. Among referenced tokens, the one needed soonest ranks highest, which is Belady’s rule [1] when protected spans exceed the budget; attention still breaks ties and ranks everything unreferenced. We use λ = 0.35. When a span’s last planned reader has run, its bonus drops to zero and it becomes evictable. In persistent sessions (Section 3.3) the bonus of every resident token is re-computed at each turn boundary.

3.2

Bounded positions

Renumbering. PACE keeps, for every resident slot, a physical position p, and for every sequence a shift c with m̂ = m − c for newly written tokens. Initially c = 0, so physical and logical positions coincide and PACE behaves exactly like H2 O. When the next physical position would come within a headroom H of a limit L (checked after each eviction), the resident row of each layer is renumbered to p′ ∈ {0, . . . , R − 1} in logical order, every kept key is re-rotated from p to its new position p′ , and the shift is advanced so that the next token continues at m̂ = R. Physical positions therefore never exceed L, regardless of session length. Proposition 1 (Exact re-rotation). Let a cached key be k̃ = Rp k for a pre-rotation key k and physical position p. Then Rp′ −p k̃ = Rp′ k: re-rotating the cached key by the position difference yields exactly the key the engine would have cached at p′ . Consequently a query at physical position m̂ scores the renumbered key as if it were m̂ − p′ positions away. The proof is Equation (1). We compute Rp′ −p from the engine’s own cosine and sine tables per frequency θi : cos((p′ − p)θi ) = cos(p′ θi ) cos(pθi ) + sin(p′ θi ) sin(pθi ), sin((p′ − p)θi ) = sin(p′ θi ) cos(pθi ) − cos(p′ θi ) sin(pθi ), normalized to unit length, so re-rotation matches the engine’s rotation convention and any frequency scaling it applies; it extends to partial and interleaved rotary layouts by applying the same rotation to the rotated dimensions only. Semantics. Renumbering is order-preserving and gap-collapsing: kept tokens keep their relative order, and the gaps left by evicted tokens disappear. The model therefore sees a kept definition at most L positions away instead of tens of thousands. Between renumbers, ordinary relative distances apply. This is a deliberate change of what the model observes; Section 5.7 measures its cost. 5

Choosing L. The limit must leave room for the resident set and for the tokens appended between two checks: L > Bmax + H, where Bmax is the largest resident row (the budget plus the eviction interval) and H the prefill chunk. With B = 2,048 and 4,096-token chunks, L = 8,192 is the smallest admissible power of two. We show in Section 5.3 that smaller is better, so the rule is: set L as small as the budget allows. Setting L to the model’s full positional range (40,960 for Qwen3-8B) bounds positions too, but leaves rotary gaps of ∼24K and is measurably worse. Non-compounding re-rotation. Keys are stored in bf16. Re-rotating the stored key at every renumber adds one rounding each time its position changes, and planner-protected keys are precisely the long-lived ones. At 100K tokens (about 23 renumbers per session with L = 8K) the accumulated error is small; emulating the rounding of a 1M-token session degraded accuracy from 100% to 84% on the same tasks, and the fix below restores 100% under the same stress (Appendix D). PACE therefore copies each resident key into a reference store at its first renumber, together with the position pref it is valid at, and derives every later position from that copy in fp32. The error stays at about two roundings however many renumbers a key survives (in a CPU simulation of eviction churn, maximum relative key error 0.003 after 60 position-changing renumbers, against 0.031 when re-rotating the stored key). The store is sized to the resident set (4,096 tokens per layer, 0.3 GB for Qwen3-8B); entries are invalidated when a slot is reused, moved with the slot when the engine compacts rows, and checked against the stored key before use.

3.3

Persistent sessions

An agent typically re-sends its full transcript at every step. Without persistence, each step re-prefills that transcript; tokens evicted in earlier steps are silently re-inserted, so eviction quality never matters across steps, and the prefill cost grows quadratically with the session. PACE maps the transcript onto a persistent engine sequence: each step sends only its new messages, appended to the engine’s own token history, and the KV state, including every past eviction decision, carries over. Message spans are tracked in session coordinates, so blueprints refer to the right resident tokens. For chat templates that rewrite earlier turns (for instance, ones that move the system prompt into the latest user turn), we use a prefix-stable variant of the same format so that appending is exact.

3.4

Tiered rehydration

Planners make mistakes, and some workloads protect more context than the budget holds. At every compaction PACE offloads the evicted rows to pinned host memory, spills to local storage under pressure, and asynchronously replicates them to a remote Redis-compatible cache (Valkey). Before a step, the planner’s lookahead identifies needed spans that are no longer resident; PACE restores them from the nearest tier, splices them into the row, and renumbers so positions stay ordered and bounded. Writes to the remote tier run off the decode path.

3.5

Implementation

PACE is implemented as a subclass of the H2 O cache manager in SparseEngine [23], an open-source inference engine, and is enabled by a configuration switch, so PACE and H2 O share the same prefill, eviction schedule, and decode path. An OpenAI-compatible server accepts the blueprint as an extra request field and exposes persistent sessions. Bounded positions currently cover full-rotary, split-half RoPE with static frequency scaling (default, linear, YaRN, Llama-3 style), which includes the Qwen and Llama-architecture families; nothing in the method is specific to a model family. 6

Table 1: Arms. All share the KV budget (B = 2,048 tokens, including 4 sink and 128 recent tokens), the engine, greedy decoding, persistent sessions, and the eager decode path. Arms without bounded positions use YaRN × 4 so that 100K-token sessions fit the positional range. Arm

Retention

Positions

Role

H2 O SnapKV retention only positions only PACE PACE (L = 40K)

cumulative attention current-query attention H2 O + foresight bonus H2 O H2 O + foresight bonus H2 O + foresight bonus

as written (YaRN) as written (YaRN) as written (YaRN) bounded, L = 8K bounded, L = 8K bounded at the model’s range

standard eviction standard eviction ablation: no bounded positions ablation: no foresight full system limit ablation

4

Experimental Setup

4.1

A long-horizon coding benchmark

Public long-context benchmarks mostly pose a single question over a long document [6]. Agent sessions differ in two ways that matter for eviction: context arrives over many turns, and the step that needs a definition comes long after it, with unrelated work in between. We therefore built an execution-scored session benchmark in which recall of early definitions is the only path to a correct late answer. Session. Each session is a sequence of user steps answered by the model under test (Appendix A lists every template). Three definition steps fix a specification: step 1 a dataclass with randomized class and field names, two fee constants (basis points and a cap) and an exception class with a constructor code; step 2 a status class with three randomized members and values; step 3 a fee rule given as exact code, with a randomized rounding mode and quantum. Then follow filler steps, each pasting a ∼2,000-token chunk of Python standard-library source [21] (PSF License) and asking for the name of its first function, so the session grows with realistic code while the answers stay short. In the 100K experiments of Sections 5.1 to 5.5 (except the replication in Section 5.1), each session draws its filler from a fixed pool in which files recur; later runs (the longer sessions and the drift checks) use non-repeating chunks and exclude chunks on the task’s own topic (Appendix B). All arms of a run see identical sessions. Three probe steps ask for code that is correct only if the definitions are recalled: fee_for(entry) (p1 ), settle(entries) (p2 ), and the complete module (p3 ). Specifications are sampled per seed (30 seeds give 30 distinct specifications), so values cannot be guessed, and an automated check verifies that no probe text reveals any of them. Scoring. Each probe’s code is executed against hidden unit tests in a subprocess: p1 has 7 checks (one fee per amount), p2 3 (total, status update, duplicate detection with the right exception code), p3 5 (definitions, fees, total, status, duplicates). For p1 and p2 the reference classes are injected before and after the answer, so only recall of names, values and rules is tested; p3 is scored standalone with a fixed standard-library import prelude identical for every arm. Independently of the tests, a recall metric checks whether the answer contains each session fact (class, field, constant names and values, rounding mode, quantum, status names and values). Together they classify each probe as pass, failed with all facts recalled (a coding error), or failed with facts missing (forgetting). A control run with no eviction (an 8K-token session within an 8K budget) scores 100% on Qwen3-8B, so failures in the long sessions are attributable to the cache. Layouts. In the single layout the definitions sit 20% into a 100K-token session and the probes at

7

roughly 52K, 74K and 98K tokens. In the distance layout the session length is fixed at 100K tokens and the definitions are placed D tokens before the three probes at the end, so only the gap varies.

4.2

Systems compared

Table 1 lists the arms. SnapKV receives the same total budget, split into sink, scored and recent tokens. The arms with foresight also run tiered rehydration; it was rarely triggered, because in these sessions the protected set fits the budget (Section 5.7 evaluates rehydration directly). Unless stated otherwise, the blueprint comes from the session’s true dependency graph (an oracle planner); Section 5.5 replaces it with an LLM planner. Models and hardware. Qwen3-8B [29] (native positional range 40,960) is the primary model. Mistral-7B-Instruct-v0.3 [7] (32,768; run through the engine’s Llama code path, which its architecture matches exactly) is the second. All runs use one NVIDIA L4 GPU (24 GB) with 128 GiB of host memory. Statistics and integrity. The unit of analysis is a (session, probe) pair, scored as the fraction of its checks passed. Comparisons between arms are paired on identical sessions and use a two-sided sign test; full-pass rates use Fisher’s exact test; the two-factor interaction uses a sign-flip permutation test with bootstrap confidence intervals. The headline comparison (Section 5.1), the second-model analysis, the numerical-drift verification, and the 1M- and 2M-token evaluations were pre-registered: the limit, arms, seeds, primary measures and decision rules were committed before the runs, and are reported whatever the outcome. Result files carry a fingerprint of every instruction template, and the merge tool refuses to pool runs whose wording or engine settings differ. When we changed one probe’s wording after an early run, the earlier trials were kept separate rather than merged (Appendix B).

5

Results

5.1

Headline: pre-registered comparison on fresh tasks

After the distance sweep of Section 5.3 suggested L = 8K, we fixed the limit, the four arms of a 2 × 2 design and ten new seeds in advance, and ran them once. Table 2 reports the result. PACE passes 145 of 150 checks (97%) and fully passes 28 of 30 probes; H2 O passes 42% and fully passes 7. Paired on identical sessions, PACE is better than H2 O on 22 probes, worse on none. PACE recalls all 250 session facts and has no failure with a fact missing; its two failures are coding errors. A later replication on five further fresh sessions, with non-repeating filler that excludes chunks on the task’s topic (Appendix B), gives the same picture: PACE 75/75 checks (100%, all 15 probes fully passed, every fact recalled) against H2 O’s 33/75 (44%); 13 better, 0 worse (p = 0.0002).

5.2

Both components are required

Each component alone helps: positions only and retention only both beat H2 O (Table 2) and tie each other (8 better, 6 worse; p = 0.79). Neither comes close to the combination, which beats each of them on every comparison in the lower panel of Table 2. The two components fix different failures, which the failure taxonomy makes visible. With positions only, the needed tokens are close to the query but often gone: H2 O’s policy evicts the details (the quantum is missing in 20 of its 30 answers, the rounding mode in 14). With retention only, the tokens are resident (residency 1.000 at 8

Table 2: Pre-registered comparison (Qwen3-8B, 100K-token sessions, B = 2,048, ten fresh sessions, three probes each). Checks: hidden-test checks passed. Full: probes passing every check. Paired: (session, probe) pairs better/worse/tied against the reference arm, two-sided sign test. Residency: fraction of needed tokens resident at the probes. Arm

Foresight

L = 8K

Checks

Full

Recall

Paired vs. H2 O

– – ✓ ✓

– ✓ – ✓

63/150 (42%) 94/150 (63%) 94/150 (63%) 145/150 (97%)

7/30 10/30 12/30 28/30

83% 86% 96% 100%

– 14/2/14, p = 0.004 15/2/13, p = 0.002 22/0/8, p < 10−6

H2 O positions only retention only PACE PACE vs.

Paired

retention only positions only

17/0/13, p = 1.5 × 10 19/1/10, p = 4.0 × 10−5 −5

Full-pass p1

Full-pass p3

9/10 vs. 1/10, p = 0.001 9/10 vs. 0/10, p < 0.001

9/10 vs. 1/10, p = 0.001 9/10 vs. 0/10, p < 0.001

Table 3: Distance sweep (Qwen3-8B, 100K-token sessions, definitions D tokens before the probes, five sessions per D, same seeds at every D). Hidden-test checks passed; the rotary gap is the mean distance, in the positions the model sees, from the query to the definitions, reconstructed from the position ledger recorded at each probe. Residency is 0.992–0.996 for every PACE-family arm at every D. Checks passed

Rotary gap

D

H2 O

retention only

PACE L = 40K

PACE L = 8K

L = 40K

L = 8K

2K 10K 25K 50K 90K

44% 55% 45% 53% 52%

93% 76% 76% 77% 64%

100% 97% 56% 64% 93%

97% 97% 100% 100% 100%

2.9K 10.7K 24.4K 24.7K 23.9K

2.7K 5.5K 5.5K 5.8K 6.5K

every probe) but 30–80K positions away; 8 of its 18 failed probes contain every session fact and still apply them wrongly. Only when the tokens are both kept and close does the model use them reliably. Is the combination more than additive? At the limit of the model’s own positional range (L = 40K), it is: in ten earlier sessions, foresight added 5.9 points with positions as written and 33.5 points with bounded positions, an interaction of +27.6 points (95% CI [+12.0, +42.1], p = 0.0014; Appendix C). At L = 8K both single components are stronger (69% each on the per-probe mean, vs. 63% on pooled checks), and the interaction, +8.3 points (CI [−8.0, +23.8], p = 0.32), is not significant: the additive prediction is already 89%, leaving at most 11 points below the ceiling. We therefore claim that both components are required, which the paired tests establish, and not that they are super-additive at the recommended setting.

5.3

Mechanism: residency is not retrievability

The distance layout fixes the session at 100K tokens and moves only the definitions (Table 3, Figure 1). H2 O fails at every gap, including 2K, because a 2,048-token budget cannot hold the definitions through even one filler file; its failures are forgetting (recall 76–83%). Retention only keeps 99.6% of the needed tokens at every gap, yet its accuracy falls from 93% to 64%. Its recall 9

Checks passed (%)

100

96

100

75

87 77

71

76

69

50 25 0

0-4K 4-8K 8-16K 16-24K 24-32K 40-64K 64-128K Rotary gap between query and needed definitions (positions the model sees)

Figure 3: Accuracy tracks rotary distance at full residency. All 225 probes of the three PACE-family arms in the Qwen3-8B distance sweep, binned by the reconstructed distance from the query to the definitions. Bars: checks passed. Residency of the needed tokens is 0.995–0.996 in every bin. Probes per bin, left to right: 47, 58, 30, 38, 22, 15, 15. No probe fell in 32–40K. stays at 98–100%, so the facts are present in the answers; what degrades is how reliably they are applied. PACE with L = 8K keeps the rotary gap at 2–7K positions regardless of the logical gap and passes 97–100% everywhere; it beats H2 O at every gap (p ≤ 0.0034). Figure 3 pools every probe of every PACE-family arm by its reconstructed rotary gap. Residency is flat at 0.995–0.996 across bins, while the pass rate falls from 96–100% below 8K to 87% at 8–16K and 69–77% beyond 16K. Where arms overlap in a bin they score alike: the L = 40K arm’s packed definitions and the retention-only arm’s distant ones fall in the same 16–32K bins with similar pass rates. The gap, rather than the retention policy, is what the pass rate follows. Because the largest gaps arise only at the longest D, this pooled view cannot fully separate gap from session position; the distance sweep, where L = 8K holds the gap small at every D and stays at 97–100%, is the controlled comparison. This is consistent with RoPE’s long-range attenuation [24] and with position effects on retrieval reported for single long prompts [13]; frequency rescaling (YaRN) extends the range a model can address but, as the retention-only arm shows, does not keep distant content as usable as near content. The L = 40K arm shows why the limit should be small. Once its definitions are packed by a renumber, they sit about 24K positions from the query, and its probes in the 16–32K bins pool with retention only at the same gaps. Its spread across D at a matched ∼24K gap (56% at 25K, 64% at 50K, 93% at 90K) is not explained by the gap; with five sessions per point and correlated probes we report it as observed. At L = 8K the spread disappears.

5.4

Standard eviction baselines

On five further fresh sessions with the same budget, SnapKV passes 2 of 75 checks (3%) against H2 O’s 30 (40%; 11 worse, 1 better, p = 0.006). Its answers are well-formed Python with invented names, and its recall of session facts is 5%: none of the session’s class, field or constant names survive. SnapKV ranks tokens by the attention of the current prompt’s last tokens; in an agent session the current prompt is an unrelated step, so the definitions score low at the first turn after they are written and are evicted. Query-aware eviction is myopic across turns: the query that needs the definitions has not been asked yet. SnapKV was designed for compressing a single long prompt whose question comes last; we run it as the engine implements it, in multi-turn sessions 10

Table 4: Mistral-7B-Instruct-v0.3, distance layout, primary measures (p1 + p2 ). D = 2K and 25K: five sessions; D = 90K: ten (a pre-declared single extension). Residency 0.997–0.998 for both PACE-family arms. Checks passed (p1 + p2 )

Recall of facts

D

H2 O

retention only

PACE L = 8K

H2 O

retention only

PACE L = 8K

2K 25K 90K

14% 0% 0%

72% 40% 36%

56% 64% 46%

55% 45% 67%

100% 91% 83%

100% 91% 98%

with decode-time eviction. H2 O’s cumulative attention partly preserves the definitions but loses details: across runs it misses the quantum or the rounding mode in most of its answers.

5.5

With an LLM planner

Foresight is only as good as the plan. We replaced the oracle with a production-style multi-agent planner: an LLM (Qwen3-8B, full context) prompted to emit a dependency graph over the session’s steps, whose output is converted to blueprints by the same parser and builder an agent would use. Its edges had precision 1.0 and recall 0.9 in every session; the one missed edge was always from the final module to the fee rule. With these blueprints and L = 40K (this experiment preceded the limit study), PACE passed 69 of 75 checks (92%) against H2 O’s 41% (11 better, 0 worse, p = 0.001), and retention only reached 60% (not significant against H2 O). The miss cost little because the final module also depends on the earlier fee_for answer, which restates the rule, and the planner kept that edge.

5.6

A second model

We repeated the distance study on Mistral-7B-Instruct-v0.3 under an analysis plan committed beforehand (Table 4). A no-eviction control first establishes the model’s own ceiling: it recalls 99% of facts but passes only 40% of checks, failing mostly with every fact present, so for this model hidden tests largely measure coding ability. The plan therefore made p1 + p2 (scored with reference classes, insensitive to module structure) and recall the primary measures. For a model that passes 40% of checks with perfect memory, the meaningful question is how close each method comes to that ceiling. Across all probes, PACE stays at the model’s own ceiling at every gap (45%, 49% and 37% at D = 2K, 25K and 90K, against 40% with no eviction) and recalls 96–100% of facts; retention only falls below the ceiling as the gap grows (53%, 32%, 30%; recall 98%, 94%, 88%); H2 O collapses (9%, 1%, 1%). The ceiling comes from a different, shorter session, so this comparison is descriptive rather than paired. PACE beats H2 O significantly (at D = 90K, 8 better and 0 worse on p1 + p2 , p = 0.008; on all probes 12 and 0, p < 0.001). Most of PACE’s remaining failures recall every fact and break the code, as in the control: memory is no longer the limit, the model’s coding is. The pre-declared paired test of PACE against retention only at D = 90K is not significant (46% vs. 36% on p1 + p2 ; 6 better, 4 worse; p = 0.75; the five added sessions, run with the non-repeating filler, favored retention only). With a 40% ceiling, the two arms can differ on only a handful of probes, so the test-pass measure has little room to separate them; the separation appears in recall, which has no coding ceiling. We therefore state the second-model result as: PACE reaches the model’s no-eviction ceiling and beats standard eviction on both models, while a statistically 11

established test-pass advantage of bounded positions over retention alone is shown on Qwen3-8B.

5.7

Exactness, cost, and scale

Exactness. With no blueprint and no renumbering, PACE reproduces H2 O’s output token for token on a multi-chunk prompt, and so does PACE with bounded positions whose limit is not reached; this holds on both models. Persistent sessions produce the same tokens as re-prefilling the full history. Quality cost of renumbering. Teacher-forced continuations after a 12K-token prefix, with L = 8K forcing 16 renumbers, change the mean negative log-likelihood by −0.003 nats (95% CI [−0.019, +0.015], eight windows) on Qwen3-8B and by +0.007 ([−0.002, +0.016], four windows, 8 renumbers) on Mistral-7B. A 48K-token prompt beyond Qwen3-8B’s 40,960-position range runs with physical positions below the limit and recalls a protected needle at token 0, with continuation likelihood equal to H2 O on the last 30K tokens (0.515 vs. 0.516 nats). Rehydration. In a controlled test where the planner flags a definition only at the step that needs it, after it was evicted, rehydration restores it and recall is 5/5 against 1/5 without rehydration (p ≈ 0.05); with no flag at all, nothing is restored and PACE equals H2 O. After deleting every host-side copy, all restores came from the remote tier and recall stayed 5/5. A restore from host memory took 2.3 ms; one from the remote tier about 2.7 s. Time and memory. Over ten 100K-token sessions, PACE took 1,047 s against H2 O’s 963 s (+9%) on the same eager decode path. Persistent sessions reduce prefilled tokens 5.5× and wall time 4.3× against re-sending the transcript in a 16K-token session; since re-sending costs prefill quadratic in the number of steps and persistence linear, the saving grows with session length. The resident KV state is the budget (2,048 tokens, 0.30 GB for Qwen3-8B) plus the reference store (at most 0.30 GB), independent of session length; the full cache would need 14.7 GB at 100K tokens and 147 GB at 1M. In a 300K-token smoke run, the latency of filler turns (a 2,000-token chunk plus a one-line answer) stayed at 0.9–1.0 s from the first turn to the last; the per-step average at 1M tokens below also includes probe turns that generate up to 900 tokens. At 2M tokens filler turns do slow down, to 1.7–3.2 s by the end of the session, in every arm alike (see Sessions of 2M tokens below). Sessions of 300K tokens. Only bounded-position arms can run past the positional range, so at 300K we compare PACE with positions only and with a StreamingLLM-style baseline that spends the budget on sink and recent tokens with in-cache positions. On three fresh sessions with the definitions 270K tokens before the probes, PACE passes 45 of 45 checks and recalls every fact; positions only passes 58% and recalls 79% (it loses the rounding mode and quantum), and the StreamingLLM-style arm passes none and recalls 11%. PACE is better on 9 of 9 probes against the StreamingLLM-style arm (p = 0.004) and on 7 of 9, with 2 ties, against positions only (p = 0.016). Sessions of 1M tokens. Under the same design, with criteria committed before the run, we ran five 1M-token sessions on the single L4, each placing the definitions about 894K tokens before the probes (Table 5). PACE passes 74 of 75 checks (99%), fully passes 14 of 15 probes, and recalls all 125 session facts; its one missed check is a coding error with every fact present. Positions only passes 33% and recalls 74% (losing the quantum and rounding mode), and the StreamingLLM-style arm passes none and recalls 8%. PACE is better than the StreamingLLM-style arm on all 15 probes (p = 6 × 10−5 ) and than positions only on at least 13, with none worse (p < 0.001). All three pre-registered criteria (p3 ≥ 80%, recall ≥ 90%, significant wins over both baselines) are met. Inside the cache, the 2,050–2,170 resident tokens at each probe comprise 4 sinks, 410–1,070 needed tokens, about 470 old unprotected tokens and recent text; no unprotected token’s attention score 12

Table 5: Quality at length (Qwen3-8B, one 24 GB GPU, B = 2,048, filler excluding task-topic chunks, fresh sessions). Definitions are placed 90% of the session before the probes. H2 O and the YaRN arms cannot run past 131K positions; the StreamingLLM-style arm, at 0% at 1M, was not re-run at 2M. Session

Sessions

Arm

Checks

Full

Recall

100K

5

H2 O PACE

44% 100%

2/15 15/15

81% 100%

300K

3

StreamingLLM-style positions only PACE

0% 58% 100%

0/9 2/9 9/9

11% 79% 100%

1M

5

StreamingLLM-style positions only PACE

0% 33% 99%

0/15 1/15 14/15

8% 74% 100%

2M

5

positions only PACE

13% 100%

0/15 15/15

50% 100%

ever reached the smallest foresight bonus (maximum 14.6K against a floor of 50K), so accumulated heavy hitters never displaced planned context. After 242 renumbers per session, the definitions sat 3.8–7.1K positions from the query. A 1M-token session took about 13.6 minutes (about 1.7 s per step); the resident KV state was about 0.3 GB plus at most 0.3 GB of reference keys, where the full cache would need about 147 GB (both computed from the model’s dimensions). Sessions of 2M tokens. With the same code, settings, seeds and decision rule, committed before the run, we doubled the length: five 2M-token sessions on the same L4, definitions 1.8M tokens before the probes. The StreamingLLM-style arm was dropped, having passed nothing at 1M. PACE passes all 75 checks, fully passes all 15 probes, and recalls all 125 facts. Positions only falls from 33% at 1M to 13% (10/75), fully passes no probe, and recalls 50%; every one of its failures has facts missing, most often the rounding mode and quantum. PACE is better on all 15 probes (p = 6 × 10−5 ; full-pass Fisher p = 0.008 at each probe), and all three pre-registered criteria are met again. The protection margin thus widens with length: without foresight, attention-ranked retention loses more of the definitions the longer they must survive, while the plan keeps them regardless. The cache looks as it did at 1M: 2,050–2,170 resident tokens, 410–1,090 of them needed, 460–490 old unprotected tokens, and needed-span residency at least 0.992. After 480 renumbers per session (twice the 1M count), the definitions sat 4.4–7.1K positions from the query (mean 6.3K) against a logical gap of about 1.79M. Positions only sees the same rotary gaps (mean 6.1K), so at this length the difference between the arms is retention, not distance. No unprotected token reached the foresight floor, but the largest unprotected attention score rose from 14.6K at 1M to 25.1K, half the 50K floor (Section 7). The resident KV state is unchanged, where the full cache would need about 295 GB (computed). Per-turn time, however, is not: the median filler turn rose from about 0.9 s at the start of each session to 1.7–3.2 s at the end, by the same factor in both arms, and a 2M-token session took about 31 minutes (2.3× the 1M time). Since attention, eviction and renumbering act on the fixed resident set, the growth comes from per-turn work that scales with the session elsewhere in the serving path; we have not yet isolated it.

13

6

Related Work

KV eviction and compression. H2 O [31] and Scissorhands [15] keep tokens that past queries attended to; SnapKV [12] and PyramidKV [3] score tokens by the attention of the prompt’s final window, the latter with per-layer budgets; Quest [25] keeps the full cache but reads only queryrelevant pages. These methods target single requests, where the question is known when compression happens. In multi-turn agent sessions it is not, which is why query-aware compression collapses in our benchmark (Section 5.4). PACE is complementary: its foresight bonus is added to H2 O’s score here, and could be added to any score-based policy. Agent-aware KV management. A recent line of serving systems uses knowledge of agent workflows. KVFlow [19] abstracts the schedule as an agent step graph and evicts and prefetches prefix-cache entries by steps-to-execution; PBKV [32] predicts future agent invocations for dynamic workflows; CacheScout [30] learns agent transitions online; TokenCake [2] co-schedules offloading during function calls; Continuum [11] pins caches across tool calls with a time-to-live; WAKMan [17] adapts the eviction policy to the traffic mix. These systems decide which requests’ prefixes stay cached across invocations, a throughput problem. Closer to us, ActKV [26] and MemDecay [16] evict within an agent’s context, by contribution to action generation and by semantic region respectively. All of them optimize residency. Our finding concerns what happens once residency is achieved: retained context can still be unusable because of where it sits in position space. Plan-aware retention in PACE is one instance of this family; bounded positions are what the family lacks. Positions beyond the training range. Position interpolation [4], YaRN [20], LongRoPE [5] and Self-Extend [8] change how positions map to rotations so that a model can address longer inputs. StreamingLLM [28] assigns positions within the cache rather than in the original text, so that a sink-plus-window working set can stream indefinitely, and InfLLM [27] retrieves blocks of distant context into a bounded window with its own position assignment; these are the closest precedents for bounded positions. PACE differs in three ways: positions are renumbered only when needed, by exact re-rotation of the cached keys, so the model sees ordinary distances between renumbers; renumbering preserves the order of planner-selected, non-contiguous context instead of a fixed sink-plus-window; and we show that the limit should be set as small as the budget allows, because accuracy depends on the resulting rotary gap. Studies of long-context use, such as “lost in the middle” [13] and RULER [6], document that models use distant context less reliably; we measure this at fixed residency inside a KV-evicting server and act on it. Offloading and memory tiers. FlexGen [22] and InfiniGen [10] move KV state between GPU and host; CacheGen [14] compresses and streams it over the network; PagedAttention [9] manages it in pages. PACE’s rehydration tiers follow this line, but restores are scheduled by the plan rather than by a recency or attention heuristic. Agent memory. Application-level memory systems such as MemGPT [18] decide what text to put into the prompt; PACE acts after that decision, on the KV state the prompt produces, and the two compose.

7

Limitations

• One task family. The benchmark is a synthetic coding session with randomized specifications. It isolates recall of early definitions by design; other agent workloads (tool traces, multi-document research, multi-agent hand-offs) may weigh retention and distance differently.

14

• Two models at 7–8B, one GPU type. The statistically established test-pass advantage of bounded positions over retention alone is on Qwen3-8B; on Mistral-7B, whose coding ceiling is 40%, PACE reaches that ceiling and the advantage shows in recall (Section 5.6). The ceiling comparison uses a different session and is descriptive. Larger models, which may use distant context better, are untested. • Planned workloads only. PACE’s advantage comes from knowing future dependencies. In open-ended conversation without a plan it falls back to attention-based eviction with bounded positions (63% at 100K, against 42% for H2 O and 97% with foresight); questions about arbitrary unflagged details of a long history need a full context window or a retrieval-driven restore, which we leave to future work. • Mostly an oracle planner. One experiment uses an LLM planner (recall 0.9). How accuracy degrades with planner errors beyond that is not yet characterized. • A fixed foresight scale bounds session length. The bonus floor (50K) is a constant, while the largest cumulative attention score of an unprotected token grows with the session: 8.0–14.6K at 1M tokens, 11.5–25.1K at 2M. Extrapolated, it reaches the floor at about 5M tokens, after which accumulated heavy hitters could outrank planned context. Longer sessions need a relative bonus, scaled to the current unprotected maximum at each eviction; we have not yet evaluated it. • Gap collapse changes what the model sees. Renumbering preserves order but not distance between kept tokens. Perplexity shows no measurable cost, but tasks that depend on how long ago something happened could be affected. • Rehydration evaluated in controlled tests. In the 100K sessions the protected set fits the budget, so rehydration rarely fires, and it is disabled in the 300K, 1M and 2M runs; its benefit is shown in a scripted late-flag test. • Latency. All session arms use the engine’s eager decode path; we report relative overhead, not absolute serving throughput, and do not evaluate concurrent sessions. KV memory is independent of session length, but per-turn time is not: in 2M-token sessions the median filler turn grows 1.9–3.7× from start to end, equally with and without foresight. The source, outside the bounded attention path, is not yet isolated. • No full-context reference at 100K. We do not report the same model with its full cache at 100K tokens on this hardware; the no-eviction control is an 8K-token session. • Filler content matters. Filler is Python standard-library source. In a 300K-token session, a probe asking for decimal-based fee code was derailed into reproducing the standard library’s own decimal implementation, while the same session’s final probe recalled every definition. Topical lookalikes of the answer test a model’s resistance to copying rather than memory; our long-session runs therefore exclude filler on the task’s topic, and real agent sessions offer no such filter. The 100K experiments also span two filler regimes (recurring vs. non-repeating files); comparisons are only ever made within one run. • Small samples per cell. Five to ten sessions per configuration, three probes each. The headline comparison is pre-registered and strongly significant; some secondary patterns (the L = 40K arm’s spread across distances) are reported as observed.

15

8

Conclusion

The working assumption behind KV eviction, including its agent-aware variants, is that a token kept is a token usable. In long-horizon agent sessions it is not: at full residency, accuracy falls with the rotary distance between the needed context and the query. PACE keeps the context the plan says will be needed and keeps it close, by renumbering kept tokens so their distance to the query stays bounded. Under a budget of 2% of a 100K-token session, this lifts hidden-test accuracy from 42% to 97% on fresh tasks, and neither half of the design achieves it alone. The effect does not fade with length: on a single 24 GB GPU, PACE uses definitions written 900K tokens earlier correctly in 99% of checks at one million tokens and definitions written 1.8M tokens earlier in all checks at two million, with a KV footprint under 0.7 GB where the full cache would need 147 GB and 295 GB, while bounded positions without foresight falls from 33% to 13%. We expect the distinction between keeping and retrieving to matter for any system that bounds an agent’s working memory, and we hope the residency-gap view and the benchmark protocol help evaluate them.

References [1] L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems Journal, 5(2):78–101, 1966. [2] Zhuohang Bian, Feiyang Wu, Zhuoran Li, Teng Ma, et al. TokenCake: A KV-cache-centric serving framework for LLM-based multi-agent applications. arXiv preprint arXiv:2510.18586, 2025. [3] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, et al. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024. [4] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [5] Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, et al. LongRoPE: Extending LLM context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024. [6] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, et al. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. [7] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. [8] Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, et al. LLM maybe LongLM: Self-extend LLM context window without tuning. arXiv preprint arXiv:2401.01325, 2024. [9] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al. Efficient memory management for large language model serving with PagedAttention. In ACM Symposium on Operating Systems Principles, 2023. arXiv:2309.06180. 16

[10] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In USENIX Symposium on Operating Systems Design and Implementation, 2024. arXiv:2406.19707. [11] Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, et al. Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live. arXiv preprint arXiv:2511.02230, 2025. [12] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, et al. SnapKV: LLM knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024. [13] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, et al. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2024. arXiv:2307.03172. [14] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, et al. CacheGen: KV cache compression and streaming for fast large language model serving. In ACM SIGCOMM, 2024. arXiv:2310.07240. [15] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, et al. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Advances in Neural Information Processing Systems, 2023. arXiv:2305.17118. [16] Venkatesha Matam and Keon Kim. MemDecay: Region-aware KV cache eviction for efficient LLM agent inference. arXiv preprint arXiv:2607.10582, 2026. [17] Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, and Wesley Harris. WAKMan: Workload aware KVCache manager. arXiv preprint arXiv:2610.36355, 2026. [18] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, et al. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. [19] Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, et al. KVFlow: Efficient prefix caching for accelerating LLM-based multi-agent workflows. arXiv preprint arXiv:2507.07400, 2025. [20] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations, 2024. arXiv:2309.00071. [21] Python Software Foundation. The Python 3.12 standard library (CPython 3.12.14 source, Lib/). https://github.com/python/cpython/tree/v3.12.14/Lib, 2026. Python Software Foundation License Version 2. [22] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al. FlexGen: Highthroughput generative inference of large language models with a single GPU. In International Conference on Machine Learning, 2023. arXiv:2303.06865. [23] SparseEngine contributors. SparseEngine: an inference engine for sparse attention and KV-cache methods. https://github.com/CURRENTF/SparseEngine, 2026. Apache License 2.0. [24] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, et al. RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. 17

[25] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, et al. Quest: Queryaware sparsity for efficient long-context LLM inference. In International Conference on Machine Learning, 2024. arXiv:2406.10774. [26] Zihan Wang, Cheng Tang, Lei Gong, Chao Wang, et al. ActKV: Efficient LLM agents through action-guided KV cache management. arXiv preprint arXiv:2609.31395, 2026. [27] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, et al. InfLLM: Trainingfree long-context extrapolation for LLMs with an efficient context memory. arXiv preprint arXiv:2402.04617, 2024. [28] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024. arXiv:2309.17453. [29] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [30] Rui Zhang, Chaeeun Kim, Shaoting Feng, Kuntai Du, Yuhan Liu, et al. Learning agent execution for KV-cache management in agentic serving. arXiv preprint arXiv:2608.14624, 2026. [31] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, et al. H2 O: Heavyhitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, 2023. arXiv:2306.14048. [32] Haoyu Zheng, Fangcheng Fu, Jia Wu, Binhang Yuan, Yongqiang Zhang, et al. Efficient serving for dynamic agent workflows with prediction-based KV-cache management. arXiv preprint arXiv:2605.06472, 2026.

18

A

Session templates

Placeholders in braces are sampled per seed. {k} is the step number. STEP 1: Define the core data model. Write a Python dataclass ‘{cls}‘ with fields ‘{f_ref}: str‘ (unique reference), ‘{f_amt}: int‘ (amount in minor units), ‘{f_ccy}: str‘ (ISO currency) and ‘status: str = ’new’‘. Give the class two class-level constants: ‘{fee_name} = {fee_bps}‘ (fee in basis points) and ‘{cap_name} = {fee_cap}‘ (maximum fee in minor units). Also define an exception ‘{err}(Exception)‘ whose constructor takes an integer ‘code‘ and stores it as ‘self.code‘. Return one python code block. STEP 2: Define the status values. Write a class ‘{status_cls}‘ with three string constants: ‘{s_new} = ’{v_new}’‘ is the NEW state, ‘{s_done} = ’{v_done}’‘ is the SETTLED state and ‘{s_void} = ’{v_void}’‘ is the VOIDED state. Return one python code block. STEP 3: Record the fee rule exactly; reply ’noted’. For an entry with amount A, fee rate BPS and cap CAP (the two class constants from step 1): fee = (Decimal(A) * Decimal(BPS) / Decimal(10000) / Decimal({quantum})) .quantize(Decimal(1), rounding=decimal.{rounding}) * Decimal({quantum}) fee = int(min(fee, Decimal(CAP))) STEP {k}: Here is a source file from the codebase. <~2,000-token stdlib file> Reply with one line: the name of the first function it defines.

(filler)

STEP {k}: Write a function ‘fee_for(entry)‘ that returns the fee, as an int in minor units, for an instance of the data class defined in step 1. Use the fee constants defined on that class and apply the rounding rule from step 3 exactly. Return only the function (with its imports) in one python code block. (p1) STEP {k}: Write a function ‘settle(entries)‘ that takes a list of instances of the step-1 data class. If two entries share the same unique reference field, raise the exception class defined in step 1 with code={dup_code}. Otherwise set every entry’s status to the SETTLED-state constant of the step-2 status class and return the total, over all entries, of each entry’s amount field (from step 1) minus fee_for(entry). Assume fee_for already exists. Return only the function in one python code block. (p2) STEP {k}: FINAL STEP. Write the complete module in one python code block: the step-1 dataclass and exception exactly as specified there, the step-2 status class, and the functions fee_for(entry) and settle(entries) from the earlier steps. Reuse exactly the names, fields, constants and rules defined earlier. (p3)

The system prompt states that definitions made in earlier steps stay in force for the session. Sampled values: class name, field names, constant names, fee in [17, 49] basis points, cap in {300, . . . , 900}, exception name and duplicate code, status class, members and values, rounding mode, quantum in {5, 10, 25}.

B

Integrity protocol

• Hidden, execution-based scoring. Tests are never shown to the model; answers are executed in a subprocess with a timeout. The p3 import prelude (decimal, dataclasses, typing) is fixed and identical for all arms and models; it was not widened after a Mistral answer failed on an unimported collections.defaultdict. • Leak and guessability checks. Automated tests assert that no probe contains any sampled value and that 30 seeds yield 30 distinct specifications. 19

• Control. With no eviction pressure, Qwen3-8B passes every check, so the hidden tests accept a model that remembers. • Wording changes. After the first 100K run, the p2 instruction was reworded because answers wrote a bare amount variable; trials run with the old wording are reported separately, never merged. A fingerprint of all templates is stored in every result file, and the merge tool refuses to pool runs with different fingerprints or engine settings (drift diagnostic, re-rotation mode). • Pre-registration. The headline comparison, the Mistral analysis (including a single predeclared extension at D = 90K), the drift decision rule, the fix verification, and the 1M- and 2M-token evaluations were committed before the corresponding runs, together with their decision rules, and are reported regardless of outcome. • Filler regimes. A change to how filler chunks are built altered the sessions of runs made after it: earlier runs (most 100K results, including the headline comparison) draw each session’s filler from a pool in which files recur; later runs use non-repeating chunks. We discovered this by checksumming the prompts of paired runs; every comparison reported is within one run, hence on identical sessions, and PACE with L = 8K scores 97–100% at 100K under both regimes. Result files now record a checksum of every session’s prompts. • Topical-distractor filter. For the 100K re-check, 300K, 1M and 2M runs we exclude filler chunks that mention Decimal, decimal, quantize or a ROUND_* mode, a decision committed before those runs after a 300K session showed a probe copying the standard library’s decimal implementation. Earlier results were not re-scored or re-run under the filter. • Design changes made after seeing data. The 8K limit was chosen after the distance sweep; the headline run then tested it on fresh seeds. The earlier 40K-limit results are reported as measured.

C

Additional results

Table 6: Earlier 2 × 2 at L = 40K (Qwen3-8B, 100K tokens, ten sessions, single layout). The interaction is computed on per-probe fractions of checks. Arm H2 O retention only positions only (L = 40K) PACE (L = 40K)

Checks

Paired vs. H2 O

Recall

74/150 (49%) 82/150 (55%) 84/150 (56%) 138/150 (92%)

– 10/7, p = 0.63 12/7, p = 0.36 19/2, p = 0.0002

80% 95% 82% 98%

Interaction +27.6 points, 95% CI [+12.0, +42.1], sign-flip p = 0.0014 (n = 30)

D

Numerical drift of re-rotation

Each renumber that changes a key’s position re-rotates it; computing in fp32 and storing in bf16 adds one rounding. In a CPU simulation with Qwen3-8B’s rotary frequencies, the mean relative key error grows from 0.011 after 23 renumbers (a 100K-token session at L = 8K) to 0.019 after 71 (300K) and 0.037 after 240 (1M). To test whether this matters, we added extra rotate-and-return passes 20

Table 7: Recall of session facts in the distance sweep (Qwen3-8B). D

H2 O

retention only

PACE L = 40K

PACE L = 8K

2K 10K 25K 50K 90K

81% 79% 76% 81% 83%

98% 98% 98% 99% 100%

100% 100% 88% 100% 100%

98% 100% 100% 100% 100%

Table 8: Mistral-7B-Instruct-v0.3, all probes (p3 included; the no-eviction control passes 40% of checks). Checks passed

Recall

D

H2 O

retention only

PACE

H2 O

retention only

PACE

2K 25K 90K (n = 10)

9% 1% 1%

53% 32% 30%

45% 49% 37%

56% 46% 70%

98% 94% 88%

100% 96% 99%

per renumber, calibrated so that a 100K-token session accumulates the rounding of a 1M-token one, without changing any position. On the five sessions of the headline run where PACE scored 75/75 checks, the stressed arm scored 63/75 (84%; 4 probes worse, none better), failing the pre-registered criterion of 90%; one session lost every p1 check. Drift affects only the arms that renumber, so it cannot explain their advantage at 100K, where they score 97–100%; it does limit very long sessions. The reference store of Section 3.2 removes the compounding. We verified it on the GPU under the same pre-registered criterion, on the same five sessions: with the identical stress, the reference-store implementation passes 75/75 checks (0 worse), and without stress also 75/75. Every resident key fit in the store (no overflow), and the consistency check never found a reference that disagreed with its stored key. All results at 300K, 1M and 2M tokens use this implementation.

21

Related documents

• QAOA Inside ADMM: Subproblem Wins Do Not Become Decomposition Wins at Matched Budgets#1605226
• EmbodiedIntentBench: Benchmarking Intent-Driven Runtime Adaptation for Mobile Robotic Missions#1605227
• Reproducible LoRA Inference Through Deterministic Split-K Reduction#1605228
• Continual Instruction Tuning for LLM-ased Management of Continuum Resources#1605229
• FirstSwap: Pre-Consensus Trade Approvals for Decentralized Market Design#1605230