InferScale: GPU-Native KV Injection for Personalized LLM Serving

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributedcomputing parallelcomputing

distributed computing, parallel computing, cloud

InferScale: GPU-Native KV Injection for Personalized LLM Serving Peter Li

Prashant Pandey

Northeastern University Boston, Massachusetts, USA [email protected]

arXiv:2607.27090v1 [cs.DC] 29 Jul 2026

Abstract Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user’s many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact’s KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM’s paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact’s KV. InferScale is implemented through vLLM’s KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases. On Llama3.1-8B, TTFT increases by only 4% from 𝑘 =5 to 𝑘 =50 (16.6–17.3 ms), compared with 106% for Mem0, a state-of-the-art memory system (33.2–68.3 ms). At 𝑘 = 50, InferScale reduces TTFT by 72–79% (3.6–4.8×), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7–4.5× the throughput under concurrent load. These results demonstrate that reusable KV state decouples memory-conditioned serving latency from retrievedcontext size while preserving application quality.

Introduction

Production LLM applications increasingly maintain persistent userspecific context, such as accumulated memories or long conversation histories, that is reused across many requests. Because prepending a user’s entire history to every request incurs prefill cost quadratic in its length, production memory systems such as Mem0 [3], Zep [19], and MemGPT/Letta [16] manage this context by retrieving a relevant top-𝑘 subset of memory facts, short natural-language statements distilled from the user’s history, per query and injecting them as prompt text. Retrieval avoids prefilling an entire memory store, but it does not eliminate the cost of conditioning: every retrieved fact is still re-prefilled on every request. Consequently, time-to-first-token (TTFT) grows with the amount of memory retrieved, even when the retrieved memories are identical across many requests.

Existing KV-reuse techniques such as prefix caching [6] reduce repeated prefilling for identical prompt prefixes, but they assume reused tokens occupy fixed positions in the prompt. Persistentmemory systems violate this assumption: different subsets of memory are assembled dynamically for each request, and their positions depend on the retrieval result. Therefore, today’s memory systems continue to pay the full prefill cost of retrieved context despite extensive reuse across requests.

Inject at the attention layer, not the token layer. Our key observation is simple: under causal attention, the key–value (KV) representation of a static memory fact depends only on the fact itself and is independent of the query. Therefore, a fact can be encoded once and reused across arbitrarily many future requests instead of being repeatedly re-prefilled. Concretely, this collapses the per-request attention cost of conditioning from 𝑂 ((𝑚+𝑞) 2 ) to 𝑂 (𝑞(𝑚+𝑞)) for 𝑚 memory and 𝑞 query tokens, eliminating the 𝑂 (𝑚 2 ) term a memory system otherwise re-pays on every one of a user’s requests. This observation suggests a different serving primitive. Rather than injecting retrieved memory as prompt tokens (prompt injection), we inject its KV representation directly into the attention cache (KV injection). We show that, for a fixed context injected at its intended positions, this is not an approximation but an exact equivalence (Theorem 3): KV injection produces the same hidden states and output distribution at all query positions as prompt injection, for any positional encoding, attention variant, and feed-forward architecture. However, achieving KV injections comes with two challenges. The first challenge is positional encoding. Retrieved facts must often appear at positions different from where they were originally encoded, making naïve KV reuse incorrect under rotary positional embeddings (RoPE). We address this challenge with chunked RoPE: storing keys before rotary position encoding and applying the appropriate rotation at insertion time allows a fact encoded once to be injected at any prompt position while preserving exactly the same attention behavior as prompt injection (Theorem 6). The second challenge is preserving the accuracy of prompt injection. Encoding each fact independently eliminates repeated prefilling, but it also removes the interactions between neighboring facts that would naturally arise if they were jointly encoded as part of a prompt. We address this with context-window encoding, which encodes each fact together with a small window of preceding conversation turns while caching only the target fact’s KV representation. This recovers nearly the accuracy of prompt injection without requiring serving-time recomputation, cache re-encoding, or model fine-tuning. In short, the injection mechanism itself, i.e., KV injection with chunked RoPE, is exact; the only approximation InferScale introduces is encoding facts independently, which context-window encoding largely recovers.

Peter Li and Prashant Pandey LLM Inference Memory System: Mem0 (Prompt Injection) vs. InferScale (KV Injection) InferScale

Mem0 CPU

GPU

CPU

Memory Store Vector DB

GPU

Offline (Once)

Persistent GPU Memory

Conversation History

Jasper Index

NOVEL

PCIe

Retrieved Text

Prompt Injection

Context-Window KV Store

Context-Window Encoding

Prefill Memory + Query

Online (Every Request) PCIe

Query

LLM

Retrieve KV NOVEL

Chunked RoPE

KV Injection

Prefill Query Only

TTFT grows with retrieved memory

TTFT independent of retrieved memory

Figure 1: Where memory lives, and what crosses the PCIe bus. (a) In Mem0 (prompt injection) the vector database and memory store sit on the CPU. Each request ships the retrieved memory to the GPU as prompt text, and the GPU re-prefills all 𝑚 retrieved tokens, so TTFT grows with the amount retrieved. (b) InferScale keeps the vector index (Jasper) and the pre-RoPE KV store on the GPU. Only the 𝑞 query tokens cross to the GPU and only they are prefilled, while retrieved memory is injected as KV and never leaves the device. InferScale. We realize these ideas in InferScale,1 a GPU-native retrieve-and-inject serving engine for persistent memory. InferScale is a data-systems co-design: it maintains two GPU-resident indices over the same memory, keyed by a shared identifier, an approximate-nearest-neighbor index over semantic embeddings for retrieval and a pre-RoPE KV store for injection, and manages them across an GDDR/host-DRAM hierarchy. Keeping both on-device means retrieved memory never crosses the PCIe bus and only the 𝑞 query tokens are prefilled. At serving time, it embeds the query, retrieves the relevant facts, applies chunked RoPE to assign their serving-time positions, and injects the resulting KV directly into vLLM’s [6] paged KV cache through the KV-connector interface. InferScale requires no model retraining and no modifications to the serving engine. Figure 1 shows the high-level design of InferScale and how it differs from existing LLM memory systems such as Mem0. Contributions. Our contributions are: • Retrieve-and-inject inference. We show that retrieved memory need not be re-prefilled on every request. Instead, its key– value (KV) representation can be injected directly into the attention cache. We prove that KV injection is equivalent to prompt injection at query positions for any positional encoding scheme, attention variant, and feed-forward architecture (Theorem 3). • Chunked RoPE for position-independent KV reuse. Retrieved memory must often be placed at prompt positions different from where it was encoded. We introduce Chunked RoPE, which stores keys before rotary positional encoding and re-rotates them at insertion time, allowing a fact encoded once to be injected at arbitrary prompt positions while preserving the attention produced by prompt injection (Theorem 6). • Context-window encoding for accurate KV reuse. Encoding retrieved facts independently sacrifices the attention between 1 Source code at https://github.com/saltsystemslab/InferScale.

neighboring facts and reduces model accuracy. We introduce Context-Window Encoding, which encodes each fact together with a small window of preceding conversation turns while caching only the target fact’s KV representation. This recovers nearly the accuracy of prompt injection without serving-time recomputation, cache re-encoding, or model fine-tuning. • InferScale: a GPU-native retrieve-and-inject serving engine. We design and implement InferScale as a vLLM KVconnector plugin that combines GPU-resident semantic retrieval with reusable KV storage, requiring no modifications to the serving engine, no model fine-tuning, and no changes to existing retrieval pipelines (Section 3). Results. By replacing repeated prompt prefilling with KV injection, InferScale makes serving latency nearly invariant to the amount of retrieved memory. On the LoCoMo benchmark [10], as retrieval grows from 𝑘=5 to 𝑘=50 the vLLM engine TTFT for Llama-3.1-8B rises by only 4% (16.6 to 17.3 ms), whereas Mem0 rises by 106% (33.2 to 68.3 ms). This trend is consistent across three open-weight models, giving 72–79% lower TTFT at 𝑘=50, a 3.6–4.8× speedup. Encoding facts independently introduces an accuracy trade-off, as interactions between neighboring retrieved facts are not captured during encoding. Context-window encoding largely eliminates this trade-off by encoding each fact together with a small window of preceding conversation turns while caching only the target fact. It restores monotonic accuracy as more memory is retrieved, achieving 60.3% on LoCoMo at 𝑘=50 compared to Mem0’s 63.3%, while matching or exceeding Mem0 at smaller retrieval budgets. Under concurrent load, InferScale’s throughput exhibits near-linear scaling with the number of users, reaching 3.7–4.5× that of Mem0 at 100 users. InferScale’s per-conversation KV store (1.8–4.8 GB) can also be offloaded from GPU memory to host DRAM and streamed over PCIe on demand, lifting the HBM capacity bound at negligible cost: at𝑘=50 on Llama-3.1-8B, offloading adds only ∼3 ms to engine TTFT and keeps end-to-end query-to-first-token (≈100 ms) still 2.3× faster than Mem0 (236 ms), while leaving throughput essentially unchanged. These results distinguish InferScale from three adjacent lines of work. Production memory systems (Mem0 [3], MemGPT/Letta [16], Zep [19]) retrieve relevant context but still inject it as prompt text, so latency grows with retrieved-context size. Prefix- and block-level KV-reuse techniques (Block-Attention [9], CacheBlend [24], LazyAttention [23], LMCache [8]) reuse cached KV but only for fixed prompt layouts, and typically require model fine-tuning, selective reencoding, or cache recomputation when the context changes. Sparse and retrieval attention (RetroInfer [2], RetrievalAttention [7]) instead reduce the decode-time cost of attending over a single long context. In contrast, InferScale couples GPU-native semantic retrieval with position-independent KV injection to eliminate the repeated prefill of shared context, with no fine-tuning, engine changes, or attention recomputation, while preserving nearly the accuracy of prompt injection.

Background and Motivation

In this section, we introduce the components of modern LLM serving that InferScale builds upon. We first review the prefill/decode execution model and KV cache management, which explain why repeatedly injecting retrieved memory incurs high serving latency. We

InferScale : GPU-Native KV Injection for Personalized LLM Serving

then describe retrieval-based memory systems, whose token-layer interface motivates InferScale’s attention-layer injection. Finally, we review rotary position embeddings and GPU-native vector indexing, which enable our two key optimizations—Chunked RoPE for position-independent KV reuse and Context-Window Encoding for accurate memory representations.

2.1

LLM inference: prefill and decode

An autoregressive LLM generates a response one token at a time, each token attends to the full sequence of tokens before it. Serving a request proceeds in two distinct phases. In the prefill phase, the model consumes the entire input prompt in a single parallel forward pass: for each of the 𝑛 prompt tokens and each layer it computes query, key, and value (KV) projections, runs self-attention, and produces the first output token. Because every token attends to all preceding ones, prefill performs Θ(𝑛 2 ) attention work and is compute-bound. Prefill is a major model-execution component of time-to-first-token (TTFT). In the decode phase, the model emits the remaining tokens one at a time. Each new token attends to all previous tokens and appends its own KV, so per-step work grows with the running context length and the phase is bound by memory bandwidth rather than compute. To avoid recomputing the KV of earlier tokens at every decode step, serving engines store them in a KV cache: prefill populates the cache for the prompt, and decode reads it and appends one entry per generated token. Caching makes each decode step cheap, but the prefill phase bears the cost of input context that scales quadratically when no reusable prefix-cache entry is available.

2.2

Paged attention and KV cache management

Because the KV cache grows with sequence length and differs across concurrent requests, its memory management is a central concern in LLM serving. Modern engines such as vLLM [6] use paged attention: each request’s KV cache is divided into fixed-size blocks (e.g., 16 tokens) allocated on demand from a shared pool. This avoids the fragmentation of contiguous allocation and enables prefix caching, i.e., reusing blocks across requests that share a common prompt prefix, and continuous batching. Prefix caching reuses contiguous prompt prefixes but cannot reuse dynamically retrieved memory assembled at different positions. InferScale builds on this substrate: rather than let prefill produce the memory tokens’ KV, it inserts precomputed KV directly into a request’s paged blocks (Section 3), so those blocks are populated from the cache instead of by computation.

2.3

Retrieval-based memory and its serving cost

Production memory systems share one architecture. Mem0 [3] extracts facts from conversations, stores them in a vector database, and retrieves the top-𝑘 relevant memories by embedding similarity. Zep [19] maintains a temporal knowledge graph and retrieves relevant subgraphs. Letta/MemGPT [16] pages facts between a main context and archival storage. In every case the retrieved memory is serialized into prompt text, concatenated with the query, and processed through full prefill. All these systems operate at the token layer. Retrieval is both effective and necessary: because prefill cost is quadratic, prepending a user’s entire history to every request is infeasible, so keeping the prompt short is the point. But token-layer

injection carries two serving costs. First, retrieved memories are reprefilled on every request, causing latency to grow with the amount of retrieved context. Second, retrieved text must be transferred from CPU memory to the GPU before prefilling. InferScale eliminates both costs by storing the retrieval index and reusable KV representations on the GPU and injecting memory directly into the KV cache.

2.4

Rotary position embeddings

Most modern decoders encode position with rotary position embeddings (RoPE) [20], which rotate the query and key vectors at position 𝑝 by a fixed block-diagonal rotation 𝑅𝑝 . Because 𝑅𝑎⊤ 𝑅𝑏 = 𝑅𝑏 −𝑎 , the attention score ⟨𝑅𝑎 𝑞,𝑅𝑏 𝑘⟩ = ⟨𝑞,𝑅𝑏 −𝑎 𝑘⟩ depends only on the relative position 𝑏 −𝑎. Two things to note here: the rotation is applied after the key projection 𝑊𝐾 ℎ, so a key can be stored before rotation and rotated later; and the score a key receives depends only on where it sits relative to the query, not on its absolute index. InferScale’s chunked RoPE (Section 5) rests on both.

2.5

Contextual encoding of memory

Retrieved memories are commonly stored as independently retrievable units. Mem0, for example, extracts discrete facts from conversation turns. While this organization supports efficient semantic retrieval, transformer representations are contextual, i.e., a token’s hidden state depends on preceding tokens through self-attention. Encoding a fact independently removes conversation context that can disambiguate its meaning, reducing downstream answer quality after KV injection. This tension between retrieval granularity and contextual encoding motivates our context-window encoding technique, which preserves local context while maintaining independently reusable fact representations.

2.6

Jasper: A GPU-native vector index

Retrieval over embeddings is an approximate nearest-neighbor (ANN) search problem, conventionally served from a CPU-resident vector index such as HNSW [11] or Faiss [5]. InferScale instead uses Jasper [12], a GPU-resident graph-based ANN index (in the spirit of GPU indices such as CAGRA [14]): it builds a proximity graph over the fact embedding vectors in GPU memory and answers top-𝑘 queries with a beam search on the device. Keeping both the vectors and the search on the GPU lets retrieval run on the same device as the pre-RoPE KV store and the serving engine, so a request never leaves the GPU between embedding its query and injecting the retrieved KV. We access Jasper as the vector backend of Mem0; as Section 6 shows, because Mem0 runs Qdrant in exact-search mode the two indices return near-identical results and answer quality changes only marginally, so the choice of index affects serving latency rather than accuracy. Jasper’s device memory footprint is negligible compared to the space required by model weights and KV cache. We present and discuss the memory footprint of Jasper index in Section 6.

InferScale Design

InferScale is a GPU-native retrieve-and-inject serving engine implemented as a vLLM KV connector [6], allowing it to integrate with stock vLLM without modifying the serving engine. The design separates memory processing into two phases. An offline phase, executed once when memory is created, constructs reusable KV

Peter Li and Prashant Pandey

Retrieval path - retrieval embedding space

Injection path - answer-model KV space (pre-RoPE)

Offline Encoding per user - Mem0 extracts top-5 facts per turn; each fact is encoded with w preceding turns of context

Vector store

embeddings

retrieval

(Jasper GPU index) (fact_id, embedding)

embedding model

Fact extraction

LoCoMo conversation

facts

(Mem0) top-5 facts / turn

fact-ID map Context-Window Encoding fact + w preceding turns (answer-model, pre-RoPE)

Serving Time

embedding -> KV (shared fact_id)

pre-RoPE K, raw V + IDs

NOVEL

KV fact store (HBM) fact_id -> pre-RoPE K, raw V

(vLLM) top-k ids

Incoming query

retrieval embed (query)

Vector search

(Jasper top-k)

fact-ID map

-> k fact IDs

fetch k facts pre-RoPE K, raw V

query tokens (prefill) NOVEL rotated K,V

Paged KV cache

scatter-copy KV

[ prefix | fact_1 | ... | fact_k | query ]

into paged blocks

Chunked RoPE positions 0 ... m-1 rotate keys; query at m

paged attention (vLLM) query attends all KV

decode -> response

Figure 2: InferScale architecture; the preprocessing pipeline follows Mem0 [3]. Offline, each extracted fact is encoded twice under a shared fact_id, a retrieval embedding in the GPU vector store (retrieval space) and a pre-RoPE key/value tensor from the answer model (injection space), with a window of 𝑤 preceding turns. At serving time, the top-𝑘 retrieved facts are resolved to their KV, positioned by chunked RoPE, and scatter-copied into the paged cache ahead of the query. Section 3 details each stage. representations that are stored on the GPU. An online serving phase, executed for every request, retrieves the relevant facts, adapts them to their serving-time positions, and injects them directly into the attention cache before the query is prefilled. This separation eliminates repeated prefilling while preserving compatibility with existing retrieval pipelines and serving infrastructure. Figure 2 gives an overview of both phases. The offline phase consists of Context-Window Encoding and construction of a persistent GPU memory store. The online phase performs semantic retrieval, Chunked RoPE composition, and KV injection through the vLLM KV-connector interface. We describe each stage below and defer the correctness proofs to Section 4.

3.1

Offline context-window encoding

InferScale preprocesses a user’s memory once before serving, following Mem0’s extraction pipeline [3]: each conversation turn is distilled into up to five salient facts, and each fact becomes a memory chunk encoded with the same LLM employed during inference. Simply encoding each fact independently, however, loses the conversational context that helps disambiguate it and leads to reduced retrieval quality after KV injection.

To address this, InferScale employs context-window encoding. Each fact is encoded together with a configurable window of the 𝑤 conversation turns preceding the turn it was extracted from, but only the KV corresponding to the fact is retained. This preserves local contextual information while allowing every fact to remain independently retrievable and reusable. To enable reuse at arbitrary prompt positions, the encoder stores keys before rotary position encoding. We intercept the output of the key projection (𝑊𝐾 ℎ) before apply_rotary_pos_emb and cache these unrotated keys, together with the corresponding value tensors. The resulting reusable KV is stored in GPU memory keyed by fact_id. Because encoding is performed offline, the encoder model is unloaded before serving begins and does not consume GPU memory during inference.

3.2

Persistent memory store and retrieval

Each fact is represented in two complementary forms linked by a shared fact_id: a semantic embedding optimized for approximate nearest-neighbor search, and a reusable KV representation that conditions the LLM during inference. This separation is deliberate: nearestneighbor search over raw attention keys is poorly behaved, whereas

InferScale : GPU-Native KV Injection for Personalized LLM Serving

a dedicated embedding model retrieves well, so each subsystem uses the representation it needs while the fact_id map keeps them referring to the same memory object. Each fact’s text is embedded (OpenAI text-embedding-3-small) and indexed for similarity search. We manage this through Mem0 [3], configured with Jasper (§2.6), a GPU-resident vector index, as its backend; the embedding is stored under the same fact_id as the fact’s KV. At serving time InferScale embeds the query with the same embedding model and issues a top-𝑘 search, yielding the fact_ids of the most relevant facts. Retrieval and injection thus operate in two different spaces, a contrastively trained embedding space for search and the model’s own KV space for conditioning, linked only by the shared identifier.

3.3

Chunked-RoPE composition

The retrieved KV cannot be injected directly because it was encoded at a different position than where it will appear in the serving prompt. Since RoPE encodes token position directly into the keys, naïvely reusing stored KV would produce incorrect attention scores. Given the retrieved fact_ids, InferScale gathers the corresponding pre-RoPE facts, prepends a fixed instruction prefix, and composes them into a single memory segment. Composition assigns the concatenated tokens contiguous virtual positions 0,...,𝑚−1 and applies RoPE to the stored keys at those positions on the fly, using the model’s own rotary tables. Theorem 6 guarantees that relocating a fact’s stored KV to a chosen position yields exactly the attention scores that KV would receive if prefilled at that position, so the composed KV is a valid drop-in for the paged cache. The query is placed immediately after the memory, at position 𝑚. Because facts are encoded independently rather than as one joint prefill, the composed segment omits the cross-chunk attention that a full prefill would include; we quantify the resulting accuracy cost in Section 6.

prefills only the query tokens and attends to the injected memory as if it had been prefilled. The connector is loaded as an external plugin (kv_connector_module_path) and touches no vLLM internals, which keeps InferScale compatible with stock releases.

3.5

Storage cost

Each fact stores 2×𝐿×𝑐 ×ℎ×𝑑 ×𝑏 bytes of KV (layers 𝐿, fact length 𝑐, KV heads ℎ, head dimension 𝑑, byte width 𝑏), plus one embedding vector for retrieval. For Llama-3.1-8B-Instruct with GQA (𝐿=32, ℎ=8, 𝑑=128, bfloat16), 1,024 memory tokens occupy ≈ 128 MB on HBM, and the retrieval embeddings add a few kilobytes per fact. Because facts are encoded once and reused across all of a user’s requests, this cost is amortized over the request stream.

Theoretical Analysis

We now establish the correctness foundation for KV injection: the conditions under which it produces exactly the same outputs as prompt injection, and the computational separation it induces. The base result assumes memory is served at the same positions at which it was encoded. Section 5 then lifts it to the position-reassignment case that selective retrieval requires, and we are explicit below about which parts of InferScale the theorem covers exactly and which it covers only approximately.

4.1

Definitions and setup

Definition 1 (Causal Decoder Transformer). A causal decoder transformer T with 𝐿 layers maps an input token sequence (𝑡 1,...,𝑡𝑛 ) to output hidden states via the recurrence: h𝑖(0) =Embed(𝑡𝑖 ), 𝑖 =1,...,𝑛 ) (ℓ ) h𝑖(ℓ ) =U (ℓ ) h𝑖(ℓ −1) , {(k (ℓ 𝑗 ,v 𝑗 )} 𝑗 ≤𝑖 ,

(1) ℓ =1,...,𝐿

(2)

) (ℓ ) where each key/value pair (k (ℓ 𝑗 ,v 𝑗 ) is a (position-encoded) pro-

3.4

KV injection via the vLLM KV connector

Once composed, the retrieved memory is injected through vLLM’s KV connector in four steps. (1) The connector registers the composed memory and its corresponding token sequence. (2) During scheduling, vLLM recognizes that the initial prompt tokens are already available and allocates paged KV blocks without scheduling them for prefill. (3) Before execution, the connector copies the composed KV directly into those blocks using GPU-to-GPU memory copies. (4) The model prefills only the query tokens, after which decoding proceeds unchanged. The composed memory, its per-layer KV tensors and the token IDs that produced them, is registered in a GPU-resident store, and the request prompt is formed as [memory token IDs | query token IDs]. Our connector implements vLLM’s KVConnectorBase_V1 interface with two sides. Scheduler side: get_num_new_matched_tokens() checks whether the prompt begins with a registered memory token sequence; on a match it reports those tokens as “externally available,” so vLLM allocates paged blocks for them without scheduling them for prefill. Worker side: before the forward pass, start_load_kv() scatter-copies the composed KV into the allocated blocks via the slot mapping, a GPU-to-GPU copy on the same device, after which vLLM

−1) jection of h (ℓ , and the per-layer update U (ℓ ) subsumes causal 𝑗 self-attention over the preceding key/value pairs together with the layer’s residual connections, normalization, and feed-forward transformation. Causal masking enforces that position 𝑖 depends on position 𝑗 only for 𝑗 ≤𝑖. The analysis uses only two properties of this recurrence: (i) causality, and (ii) that preceding positions enter the update solely through their key/value pairs.

Definition 2 (Prompt Injection and KV Injection). Let 𝑀 = (𝑡 1,...,𝑡𝑚 ) be a memory token sequence and 𝑄 = (𝑡𝑚+1,...,𝑡𝑚+𝑞 ) a query token sequence. • Prompt injection (PI) runs the full forward pass of T on the concatenation [𝑀;𝑄], producing hidden states h𝑖(ℓ ) for all positions 𝑖 ∈ {1,...,𝑚+𝑞} and layers ℓ ∈ {0,...,𝐿}. • KV injection (KV) first runs the forward pass of T on 𝑀 alone, ) (ℓ ) producing KV pairs ( k̃ (ℓ 𝑗 ,ṽ 𝑗 ) for 𝑗 ∈ {1,...,𝑚} at each layer ℓ. At serving time, the KV pairs are injected into the cache at positions 1,...,𝑚, and the forward pass runs on [𝑀;𝑄] but skips the computation of hidden states for positions 1,...,𝑚, using the pre-computed KV pairs instead. Both methods assign position index 𝑗 −1 to token 𝑡 𝑗 (i.e., positions 0,...,𝑚 − 1 for memory, 𝑚,...,𝑚 +𝑞 − 1 for query) for the purpose of positional encoding.

Peter Li and Prashant Pandey

4.2

Exact equivalence

Theorem 3 (Exact Eqivalence for Causal Decoder Transformers). Let T be a causal decoder transformer (Definition 1) with 𝐿 layers, using any position encoding scheme that assigns embeddings based solely on position index (including RoPE [20], learned absolute embeddings). Let 𝑀 and 𝑄 be memory and query sequences as in Definition 2, with identical effective key/value tensors at each query position and deterministic inference (i.e., no dropout, stochastic routing, training mode). Then for every layer ℓ ∈ {1, ... , 𝐿} and every query position 𝑖 ∈ {𝑚+1,...,𝑚+𝑞}: h𝑖(ℓ ) KV =h𝑖(ℓ ) PI That is, the hidden states at all query positions are identical under KV injection and prompt injection. Proof. We proceed by induction on the layer index ℓ. Base case (ℓ = 0). The embedding layer is position-wise: h𝑖(0) = Embed(𝑡𝑖 ) depends only on the token 𝑡𝑖 , which is the same under both methods. Thus h𝑖(0) KV =h𝑖(0) PI for all 𝑖. Inductive step. Assume that for layer ℓ −1, the hidden states agree at all positions: −1) −1) h (ℓ =h (ℓ , ∀ 𝑗 ∈ {1,...,𝑚+𝑞}. 𝑗 𝑗 KV PI

We must show the same holds at layer ℓ. We consider memory positions (𝑗 ≤𝑚) and query positions (𝑗 >𝑚) separately. ) Memory positions (𝑗 ≤ 𝑚). Under PI, h (ℓ 𝑗 is computed from {h𝑘(ℓ −1) }𝑘 ≤ 𝑗 via causal attention. Since 𝑗 ≤ 𝑚, the set {𝑘 : 𝑘 ≤ 𝑗 } contains only memory positions. Under KV injection, the forward (ℓ −1) ) pass on 𝑀 alone computes h̃ (ℓ }𝑘 ≤ 𝑗 , exactly the same 𝑗 from { h̃𝑘 set of positions (causal masking ensures no query token is visible). By the inductive hypothesis, h̃𝑘(ℓ −1) = h𝑘(ℓ −1) PI for all 𝑘 ≤ 𝑚. Since U (ℓ ) is a deterministic function of its inputs, and the position indices are identical: ) (ℓ ) h̃ (ℓ 𝑗 =h 𝑗 PI , ∀ 𝑗 ≤𝑚. Consequently, the pre-computed KV pairs satisfy ) (ℓ ) (ℓ ) (ℓ ) ( k̃ (ℓ 𝑗 ,ṽ 𝑗 ) = (k 𝑗 ,v 𝑗 ) PI .

Query positions (𝑖 > 𝑚). Under KV injection, the attention at position 𝑖 uses: ) (ℓ ) • The injected KV pairs ( k̃ (ℓ 𝑗 , ṽ 𝑗 ) for 𝑗 ≤ 𝑚 (pre-computed from memory). ) (ℓ ) • The freshly computed KV pairs (k (ℓ 𝑗 ,v 𝑗 ) for 𝑚 < 𝑗 ≤𝑖 (from the query forward pass).

From the memory-position argument above, the injected KV pairs are identical to what PI would compute. From the inductive hy−1) pothesis, the query hidden states h (ℓ for 𝑗 >𝑚 are also identical. 𝑗 Therefore the layer update at position 𝑖 receives identical inputs under both methods, and since U (ℓ ) is deterministic: ) (ℓ ) h𝑖(ℓ ) KV =U (ℓ ) h𝑖(ℓ −1) , {(k (ℓ 𝑗 ,v 𝑗 )} 𝑗 ≤𝑖 KV ) (ℓ ) =U (ℓ ) h𝑖(ℓ −1) , {(k (ℓ 𝑗 ,v 𝑗 )} 𝑗 ≤𝑖 PI

=h𝑖(ℓ ) PI .

By induction, the claim holds for all layers ℓ ∈ {1,...,𝐿} and all query positions 𝑖 ∈ {𝑚+1,...,𝑚+𝑞}. □ From the mechanism to the system. Theorem 3 makes two assumptions that InferScale’s retrieval pipeline does not literally satisfy, and it is worth stating exactly how each is handled. First, it assumes memory is injected at the same positions 0, ...,𝑚−1 at which it was encoded. Selective retrieval instead composes chunks and places them at request-time positions; Theorem 6 shows that storing keys pre-RoPE and re-rotating them at injection makes this relocation exact, so position reassignment, which naively breaks the equivalence, is recovered rather than lost. Second, the theorem encodes the memory 𝑀 as a single block, so memory tokens attend to one another. InferScale instead encodes each fact independently, so a fact’s KV omits the cross-fact left context that a joint prefill would provide. KV injection is therefore exact for a jointly encoded context and an approximation for independently encoded facts; the residual is the accuracy cost we measure in Section 6, and it is the price of encoding each fact once and reusing it across a user’s requests. Corollary 4 (Output Distribution Eqivalence). Under the conditions of Theorem 3, the next-token probability distribution 𝑃 (𝑡𝑚+𝑞+1 | 𝑡 1,...,𝑡𝑚+𝑞 ) is identical under KV injection and prompt injection. Proof. The output distribution is a deterministic function (the (𝐿) language model head) of h𝑚+𝑞 , which is identical under both methods by Theorem 3. □ Remark. Theorem 3 provides an exact equivalence, not an approximation. This is a consequence of the causal structure: memory tokens cannot attend to future query tokens, so their KV representations are query-independent. The result holds regardless of the specific attention variant (vanilla, multi-head, grouped-query, multi-query) and regardless of the positional encoding scheme, provided position assignments are consistent.

4.3

Complexity separation

Proposition 5 (Complexity Separation). For a single attention layer with𝑛ℎ heads of dimension𝑑, memory length𝑚, and query length 𝑞: (1) Prompt injection requires Θ (𝑚+𝑞) 2 ·𝑛ℎ 𝑑 attention FLOPs for the prefill pass. (2) KV injection requires Θ 𝑞 · (𝑚 + 𝑞) · 𝑛ℎ 𝑑 attention FLOPs at serving time (excluding the one-time encoding cost). (3) The per-request FLOP savings is: Δ =Θ(𝑚 2 ·𝑛ℎ 𝑑 +𝑚·𝑞 ·𝑛ℎ 𝑑) =Θ(𝑚 2 ·𝑛ℎ 𝑑)

when 𝑚 ≫𝑞.

The total savings across 𝐿 layers and 𝑅 requests is Θ(𝑅 · 𝐿 ·𝑚 2 ·𝑛ℎ 𝑑), while the one-time encoding cost is Θ(𝐿 ·𝑚 2 ·𝑛ℎ 𝑑)—amortized to zero as 𝑅 → ∞. Under compute-bound prefill, TTFT is proportional to attention FLOPs. Interpretation under retrieval. In InferScale the injected memÍ ory is the concatenation of the 𝑘 retrieved chunks, so 𝑚 = 𝑖 |𝐶𝑖 | grows with the retrieval budget 𝑘. Under prompt injection the perrequest prefill is Θ((𝑚 + 𝑞) 2 ) and thus grows with 𝑘; under KV injection it is Θ(𝑞(𝑚+𝑞)), grows linearly rather than quadratically with 𝑘. KV injection eliminates the quadratic memory-prefill term.

InferScale : GPU-Native KV Injection for Personalized LLM Serving Running example - turn t: "He walks" (prior turn: "Adopted Rex")

Context-Window Encoding

offline, once per turn

prior turn (context, discarded)

Adopted

target turn t (K,V kept)

Rex

walks

Transformer prefill keep only turn t's K,V

KV store (HBM)

serving-time injection

top-k retrieved chunks - stored pre-RoPE, no positions t

walks

Vet

Fri

compose: contiguous virtual positions 0 .. m-1, rotate keys on the fly (R_p k_pre) 0

walks

Vet

Fri

R_0

R_1

R_2

R_3

R_4

Paged KV cache (vLLM) [ prefix | He walks | Vet Fri | query ]

score(q@m, key@v) = < R_m q, R_v k_pre > = < q, R_(v-m) k_pre > depends only on the relative distance (v - m) -> any placement is exact

Figure 3: The two techniques on one minimal example (memory turn 𝑡: “He walks,” with prior turn “Adopted Rex”). Top: context-window encoding (offline). Turn 𝑡 is encoded behind a window of preceding turns in a single forward pass, but only 𝑡’s pre-RoPE 𝐾 and raw 𝑉 are kept: the stored slice can condition ‘He‘ on ‘Rex‘, without adding the context tokens to each request. Bottom: chunked RoPE (serving). Retrieved chunks are stored without positions (𝑘 pre =𝑊𝐾 ℎ); at injection they receive contiguous virtual positions 0..𝑚−1 and their keys are rotated on the fly. Because the score ⟨𝑅𝑚 𝑞, 𝑅𝑣 𝑘 pre ⟩ = ⟨𝑞, 𝑅𝑣−𝑚 𝑘 pre ⟩ depends only on the relative distance 𝑣 −𝑚, any placement is exact.

Its remaining online attention cost grows linearly with retrieved memory length.

Position-agnostic storage

Theorem 6 (Chunked RoPE Injection). Let M be a transformer with rotary position embedding (RoPE), trained on sequences with relative positions in [−𝐷 max,𝐷 max ]. Let C = {(𝑘𝑝 0 +𝑗 ,𝑣 𝑝 0 +𝑗 ) : 0 ≤ 𝑗 <𝑐} denote a contiguous span of 𝑐 tokens from the source context, stored in pre-RoPE form (i.e., 𝑘𝑝 := 𝑊𝐾 ℎ𝑝 before any positional rotation). Suppose C is virtually injected at offsets [𝑣 0,𝑣 0 +𝑐) and queried by 𝑞 𝑁 at virtual position 𝑁 . Then:

turn_id t -> pre-RoPE K, raw V

Chunked RoPE

5.1

InferScale stores keys in pre-RoPE form, intercepting 𝑘 pre =𝑊𝐾 ℎ before apply_rotary_pos_emb. The following theorem shows that the virtual position assigned to a chunk at injection is then independent of any choice made at encoding. A chunk stored once may be injected by different requests at different virtual positions 𝑣, with attention scores given exactly by ⟨𝑅𝑣⊤𝑞,𝑘 pre ⟩ for the pre-rotated query (Figure 3, bottom).

Chunked RoPE

The equivalence theorem assumes every memory chunk is encoded at the same position where it will later be served. However, selective retrieval violates this assumption because a chunk may appear at different prompt locations for different requests. In this section, we present chunked RoPE wherein we store the keys before applying RoPE and remove this dependency. Relocating stored keys is exact for fixed hidden states, but independently encoding facts omits leftcontext interactions and can reduce answer quality. We address this separate source of approximation with context-window encoding.

(1) Shift equivalence. The attention score between 𝑞 𝑁 and the injected key at virtual position 𝑣 0 + 𝑗 equals the score that M would produce for the same key at its original position 𝑝 0 + 𝑗 when queried by 𝑞 𝑁 ′ at 𝑁 ′ := 𝑁 − (𝑣 0 −𝑝 0 ). That is, ⟨𝑅𝑁 𝑞,𝑅𝑣0 +𝑗 𝑘𝑝 0 +𝑗 ⟩ = ⟨𝑅𝑁 ′ 𝑞,𝑅𝑝 0 +𝑗 𝑘𝑝 0 +𝑗 ⟩ for all 𝑗. (2) Within-chunk preservation. The attention pattern over C at virtual offset 𝑣 0 is structurally identical to a contiguous span seen at training time, provided the per-token relative distances {𝑁 − (𝑣 0 + 𝑗) : 0 ≤ 𝑗 <𝑐} lie within [−𝐷 max,𝐷 max ]. No intra-chunk relative distance is altered. (3) Cross-chunk independence. For 𝐾 injected chunks C1, ..., C𝐾 at offsets 𝑣 1,...,𝑣 𝐾 , the cross-attention scores depend only on the Ð per-token relative distances {𝑁 −𝑣 :𝑣 ∈ 𝑖 C𝑖 }. Inter-chunk virtual gaps 𝑣𝑖 −𝑣 𝑗 enter only through the softmax normalization, never through a key–key RoPE interaction. Chunked RoPE introduces no additional interaction beyond standard attention normalization. Proof sketch. RoPE acts as a block-diagonal rotation 𝑅𝑝 with ⊤ frequencies {𝜃 𝑖 }𝑑/2 𝑖=1 , and satisfies 𝑅𝑎 𝑅𝑏 = 𝑅𝑏 −𝑎 . Hence ⟨𝑅𝑎 𝑞,𝑅𝑏 𝑘⟩ = ⟨𝑞, 𝑅𝑏 −𝑎 𝑘⟩, i.e., scores depend on 𝑏 − 𝑎 alone. For (1), the chunked score is ⟨𝑞, 𝑅 (𝑣0 +𝑗 ) −𝑁 𝑘⟩; substituting 𝑁 ′ = 𝑁 − (𝑣 0 − 𝑝 0 ) gives (𝑣 0 + 𝑗) − 𝑁 = (𝑝 0 + 𝑗) − 𝑁 ′ , recovering the full-context score. (2) −1 is an arithmetic progression with follows because {𝑁 − (𝑣 0 + 𝑗)}𝑐𝑗=0 unit step, identical in form to any training-time chunk. (3) follows because cross-attention from 𝑞 𝑁 to memory keys involves√no key–key Í RoPE interaction; the softmax denominator 𝑣 exp(·/ 𝑑) couples chunks only through individual {𝑁 −𝑣 } terms. □ Corollary 7 (In-distribution injection budget). For any retrieval scheme, chunked injection remains in-distribution iff every injected token’s virtual position 𝑣 satisfies |𝑁 −𝑣 | ≤ 𝐷 max . Equivalently, the total budget of virtual positions consumable by retrieved chunks is at most min(𝑁 ,𝐷 max ), occupying positions to the left of the query. A fact therefore carries no positional metadata in storage: only its content embedding (for retrieval) and its raw 𝐾,𝑉 tensors are persisted, and RoPE is applied at injection for whatever positions the request assigns. This is exactly what lets Section 3.3 compose independently encoded facts at request time.

Peter Li and Prashant Pandey

5.2

Context-window encoding

A fact read in isolation is often underspecified, e.g., a pronoun, an ellipsis, or a deictic reference (“he,” “there,” “the same one”) resolves only against earlier turns. Encoding a fact without cross-turn attention would bake that ambiguity into its𝐾,𝑉 , and no choice of injection position at serving time can recover a referent the keys never saw. InferScale therefore encodes each fact in context. To capture a fact extracted from turn 𝑡, the encoder prepends a window of up to 𝑤 turns preceding 𝑡 and runs a single forward pass over the concatenation, but retains only the 𝐾,𝑉 slice belonging to the fact itself (Figure 3, top), the prefix is discarded. Because keys are intercepted before rotation (𝑘 pre =𝑊𝐾 ℎ, Section 5.1), the retained slice is still position-agnostic and composes under chunked RoPE exactly as Theorem 6 requires. The prefix changes only the content of the fact’s hidden states, which now encode the disambiguating context, not the positions at which the fact can later be served. The stored slice is thus simultaneously (1) context-aware, i.e., the fact’s keys already reflect the referents established by its prefix (in Figure 3, “He” resolves to Rex), and (2) relocatable to any serving-time position. The cost of context-window encoding is paid once, offline, at serving time the prefix is neither retrieved nor prefilled, so a fact carries its context for free. Together, chunked RoPE and context-window encoding decouple where a fact is served from how it is encoded. The former guarantees exact position relocation, while the latter preserves the contextual information needed for accurate conditioning. This separation enables InferScale to retrieve, compose, and inject facts independently at serving time without re-encoding. Theorem 8 (Context-Window Encoding). Let 𝑃 = (𝑥 1,...,𝑥 𝑤 ) denote a prefix consisting of the 𝑤 conversation turns immediately preceding a target fact 𝑇 = (𝑥 𝑤+1,...,𝑥 𝑤+𝑐 ) (the 𝑐 tokens of the fact). Suppose the transformer is executed once on the concatenated sequence 𝑃 ◦𝑇 . Let 𝐾𝑇 ,𝑉𝑇 denote the key and value tensors corresponding only to the tokens of 𝑇 . Then: (1) The stored (𝐾𝑇 ,𝑉𝑇 ) are exactly the tensors that would have been produced for 𝑇 during a standard prefill over 𝑃 ◦𝑇 . (2) Every token in 𝑇 incorporates information from every token in 𝑃 through the transformer’s causal attention. (3) No information from tokens occurring after𝑇 contributes to (𝐾𝑇 ,𝑉𝑇 ). Consequently, context-window encoding stores the exact representation of the target fact conditioned on the chosen encoding window. Proof. Consider the standard autoregressive forward pass over the sequence 𝑃 ◦𝑇 . Under causal attention, the hidden state of every token depends only on earlier tokens in the sequence. Since every token of 𝑃 precedes every token of 𝑇 , the hidden states computed for tokens in 𝑇 incorporate all information contained in the prefix 𝑃 exactly as they would during ordinary inference. Because keys and values are deterministic projections of these hidden states, the tensors (𝐾𝑇 ,𝑉𝑇 ) extracted from the forward pass are exactly those that would appear in the KV cache after prefilling 𝑃 ◦𝑇 . Finally, causal masking prevents every token occurring after the end of 𝑇 from influencing the hidden states of tokens within𝑇 . Therefore the computed (𝐾𝑇 ,𝑉𝑇 ) depend only on 𝑃 and 𝑇 , establishing (1)–(3). □ Corollary 9 (Exactness as the Window Grows). Suppose the encoding window contains every retrieved fact that precedes the target

fact in the serving prompt. Then the stored KV for the target fact is identical to the KV that would be produced by a joint prompt injection over the retrieved memory. Context-window encoding therefore introduces a single approximation parameter: the size of the encoding window. Increasing the window monotonically enlarges the portion of the joint-prefill dependency graph captured by each fact. The limiting case, where the window contains the entire retrieved prefix, exactly reproduces prompt injection. Position policy. We adopt the simplest assignment consistent with the theorem. The retrieved facts 𝐶 1,...,𝐶𝑘 are placed contiguously Í starting at virtual position zero, so fact 𝐶𝑖 begins at 𝑏𝑖 = 𝑗 <𝑖 |𝐶 𝑗 | and Í the query follows immediately after the memory at 𝑏𝑞 =𝑚 = 𝑖 |𝐶𝑖 |. (Facts are variable-length, so the offsets are cumulative lengths rather than multiples of a fixed chunk size.) This preserves each fact’s internal relative positions exactly (Theorem 6(2)) while discarding the original inter-fact gaps. Furthermore, we order the retrieved facts in reverse order of proximity and position the most relevant fact adjacent to the query.

Evaluation

We now evaluate InferScale as a memory system for LLMs. In our evaluation, we compare InferScale against the state-of-the-art production memory system Mem0 [3]. InferScale contributes two techniques over a Mem0-style memory pipeline: GPU-native retrieval, which serves the vector index from GPU memory instead of a CPU vector store, and KV Injection (KV), which conditions on retrieved facts by inserting their precomputed KV tensors directly into the cache rather than re-prefilling them as prompt text. Mem0, by contrast, uses a CPU vector store and Prompt Injection (PI), serializing the retrieved facts into the input prompt and recomputing their KV on every request. Both systems extract and retrieve the same top-𝑘 facts using Mem0’s pipeline. Unless noted otherwise, Mem0 denotes its default production configuration (a Qdrant CPU vector store with prompt injection), so that InferScale-vs-Mem0 comparisons reflect the combined effect of both contributions; the vector-backend ablation (Table 1) isolates the GPU-native-retrieval contribution on its own. Our evaluation answers four questions: • RQ1 (serving latency). Is serving latency nearly independent of the amount of retrieved memory, and what is the GPU-memory cost of the KV store? • RQ2 (accuracy). Does context-window encoding recover the accuracy lost to independent chunk encoding? • RQ3 (serving throughput). How does serving throughput scale with concurrent users? • RQ4 (CPU offloading). How does offloading the pre-computed KV embeddings to CPU impact the serving latency?

6.1

Experimental setup

Hardware and software. All experiments run on a single NVIDIA RTX PRO 6000 Blackwell Server Edition with 96GB of GDDR7 memory and peak bandwidth of 1597GB/s. The GPU is connected through PCI Express 5.0 x16 with peak read throughput of 64GB/s. InferScale is implemented as a KV connector plugin in vLLM v0.19.1. The

InferScale : GPU-Native KV Injection for Personalized LLM Serving

Latency (ms)

Llama-3.1-8B

Qwen2.5-7B

Mistral-7B

320

160

20 5

Top-𝑘 InferScale TTFT

Top-𝑘 Mem0 TTFT

Top-𝑘

InferScale query-to-first-token

Mem0 query-to-first-token

Figure 4: Serving latency on LoCoMo (log scale, ms) vs. retrieval budget 𝑘 across three models: vLLM engine TTFT (solid) and end-to-end query-to-first-token (dashed) for InferScale and Mem0 (default configuration), with prefix caching enabled. InferScale’s latency is nearly flat in 𝑘 and independent of the encoding window (curves for 𝑤 ∈ {0,5,20,50} overlap within 1 ms; a representative 𝑤=5 is shown), whereas Mem0 grows with 𝑘. KV cache uses bfloat16 precision with a 16-token page size. We enable vLLM’s prefix caching in all benchmarks so that prompt injection can receive any available speed ups if input tokens are repeated.

and streams the retrieved KV to the GPU over PCIe to scale capacity beyond GPU memory (§6.7, §6.8); we report both where the distinction matters. We sweep the retrieval budget 𝑘 ∈ {5,10,20,50} and, for InferScale, the offline encoding window 𝑤 ∈ {0,5,20,50} (measured Models. We evaluate three open-weight instruction-tuned modin complete conversation turns preceding a fact’s source turn; 𝑤=0 els: Llama-3.1-8B-Instruct [21] (our primary target), Mistralencodes each fact in isolation). Note that increasing 𝑤 changes only 7B-Instruct-v0.3 [13], and Qwen2.5-7B-Instruct [18]. They span different attention and positional-encoding configurations (e.g., grouped- the information captured during offline encoding and its one-time preprocessing cost. It does not affect serving-time retrieval, KV inquery head counts [1] and RoPE parameterizations), testing that jection, or GPU memory consumption. Both systems retrieve with InferScale’s mechanism is model-agnostic rather than tuned to one Mem0’s pipeline using the same embeddings and budget 𝑘; the GPU architecture. InferScale requires no fine-tuning; each model is used and CPU indices return comparable facts (Table 1). as released. To evaluate the robustness of InferScale on higherparameter models, we also evaluate Qwen3-14B [22] (Section 6.5). Metrics. We evaluate InferScale across three axes. Accuracy is Dataset. Our workload is long-conversation question answering the fraction of answerable questions judged correct by an indepenon LoCoMo [10]. We use the ten-conversation subset, which after dent LLM judge, Gemma-2-9B-Instruct [4], prompted zero-shot for excluding the adversarial/unanswerable category comprises 1,540 a binary verdict against the reference answer (a refusal on an ananswerable QA pairs across four categories: 282 multi-hop, 321 temswerable question counts as incorrect). For latency, we report two poral, 96 open-domain, and 841 single-hop. Following Mem0’s prequantities. TTFT is the vLLM engine’s time to first token. It excludes processing pipeline [3], we extract up to five salient facts from each retrieval and KV composition and isolates what injection changes dialogue turn; each fact is stored as one memory chunk, and its inside the engine. Query-to-first-token is the end-to-end latency text is embedded for retrieval. All accuracy numbers are microand additionally includes the cached question-vector lookup, ANN averaged over the 1,540 answerable questions. We precompute 1,536search, KV composition and registration, and prompt assembly. Fidimensional text-embedding-3-small [15] embeddings for both nally, we report throughput as the number of queries served per memory facts and questions and cache them on disk, so the measured second (QPS) under concurrent load. Starting from a single user, request path performs a cached vector lookup rather than a live emwe increase the number of users from 10 to 100 (10, 25, 50, 75, 100) bedding call. The Jasper index [12] uses inner-product distance with issuing LoCoMo queries simultaneously and record the sustained 64 graph neighbors and beam width 64. All one-time preprocessQPS at each concurrency level. ing—fact extraction, embedding generation, per-fact KV encoding, Jasper index construction, and vLLM startup, is treated as an offline setup cost and excluded from all reported latency and throughput. Conditions. We compare two end-to-end systems. Mem0 [3], the production baseline, serializes the top-𝑘 retrieved facts into the prompt and re-prefills them on every request (prompt injection). Mem0 uses Qdrant [17] as the vector index for storing and retrieving memory facts. InferScale retrieves from a GPU-resident index and injects the retrieved facts as pre-RoPE KV composed with chunked RoPE (KV injection). We evaluate InferScale in two configurations that differ only in where its pre-RoPE KV store resides: GPU-resident (the default) and CPU-offloaded, which keeps the store in host DRAM

6.2

Serving latency

We report two latencies: engine TTFT, which isolates what injection changes inside vLLM, and end-to-end query-to-first-token, which additionally includes retrieval, KV composition, and prompt assembly. Figure 4 plots both for InferScale and Mem0. Prefix caching is enabled, giving Mem0 its best case. The full per-window breakdown is in Table 8 of Appendix A. Note that InferScale’s latency is nearly flat in 𝑘 and independent of the encoding window (curves for 𝑤 ∈ {0,5,20,50} overlap within 1 ms; a representative 𝑤=5 is shown).

Peter Li and Prashant Pandey

Table 1: Retrieval backend: end-to-end query-to-first-token (ms) for Mem0 with the GPU-native Jasper index vs. a conventional CPU store (Qdrant). The two backends return comparable results; the backend’s large effect is on retrieval latency. Model

Backend

𝑘=5

Llama-3.1-8B

Jasper (GPU) Qdrant (CPU)

47.92 186.15

53.13 190.22

60.86 206.79

86.56 236.48

Mistral-7B

Jasper (GPU) Qdrant (CPU)

39.72 129.37

45.78 147.41

56.56 162.01

88.41 198.07

Qwen2.5-7B

Jasper (GPU) Qdrant (CPU)

51.30 178.81

54.87 182.25

63.36 190.83

85.24 224.76

TTFT. InferScale’s engine TTFT is nearly flat with increasing top-𝑘 (≈16–18 ms across 𝑘 on every model), because retrieved memory is injected rather than prefilled. Mem0 re-prefills the retrieved facts, so its TTFT grows with 𝑘, reaching 68/82/63 ms at 𝑘=50 on Llama/Mistral/Qwen, 3.6–4.8× InferScale’s. TTFT is also independent of the encoding window (InferScale’s 𝑤 curves overlap within 1 ms), confirming that 𝑤 is an offline-only cost. End-to-end. The same performance gap holds end-to-end, and the gap widens. InferScale’s query-to-first-token rises only slightly with 𝑘 (e.g., 52 → 60 ms on Llama) because retrieval (Jasper) and injection both run on the GPU. Mem0 additionally pays CPU-side vector search and host-to-device transfer, so its query-to-first-token is 3.4–3.9× higher and grows faster, 236/198/225 ms at 𝑘=50 versus InferScale’s ≈60/53/66 ms. Vector backend. The retrieval index determines chiefly where nearestneighbor search runs. The two backends return comparable top-𝑘 sets with only minor differences in accuracy; the backend’s large effect is on latency. Table 1 runs Mem0 (prompt injection) with either the GPU-native Jasper index or a conventional CPU store (Qdrant). The CPU index adds a large, roughly fixed cost due to host-side ANN search plus a host-to-device transfer of the retrieved text, inflating query-to-first-token by 100–150 ms (e.g., 187 vs. 48 ms at 𝑘=5 on Llama). This is precisely the overhead the Mem0 baseline in Figure 4 pays; InferScale avoids it by keeping retrieval on the same device as the KV store and the engine.

6.3

End-to-end memory QA accuracy

Table 2 reports overall judged accuracy for InferScale (per window 𝑤) and Mem0 in its default configuration. Encoding each fact in isolation (𝑤=0) trails Mem0 and degrades as more facts are retrieved, from 62.2% to 53.8% on Llama and, most steeply, 54.0% to 37.2% on Mistral as 𝑘 grows from 5 to 50. This is because independently encoded facts share no cross-fact attention and add noise. A context window reverses this: with 𝑤≥20, accuracy is flat-to-rising in 𝑘 and comes within a few points of Mem0 (on Llama, 60.3% vs. 63.3% at 𝑘=50) while often exceeding it at small 𝑘. Prompt injection (Mem0), which performs full cross-fact attention over the retrieved set, remains the accuracy ceiling, most visibly on Mistral and Qwen. Because the window is a purely offline cost, this accuracy is bought with no serving-time penalty.

Table 2: Judged LoCoMo accuracy (%) vs. retrieval budget 𝑘 for InferScale (per encoding window 𝑤) and Mem0 (prompt injection), micro-averaged over the answerable questions. Per-category results are in Appendix B. Model

Method

𝑘=5

Llama-3.1-8B

InferScale (𝑤=0) InferScale (𝑤=5) InferScale (𝑤=20) InferScale (𝑤=50) Mem0

62.21 60.00 60.13 60.06 56.95

62.14 59.81 62.08 61.36 59.29

57.66 59.87 61.62 62.66 61.49

53.77 56.82 59.22 60.26 63.25

Mistral-7B

InferScale (𝑤=0) InferScale (𝑤=5) InferScale (𝑤=20) InferScale (𝑤=50) Mem0

54.03 56.04 53.96 55.78 61.30

55.13 57.60 55.97 56.49 63.96

52.84 58.09 56.95 58.70 65.00

37.22 56.45 58.24 58.30 64.35

Qwen2.5-7B

InferScale (𝑤=0) InferScale (𝑤=5) InferScale (𝑤=20) InferScale (𝑤=50) Mem0

59.74 59.29 60.00 58.83 60.06

56.75 57.53 59.68 58.25 63.64

54.22 58.05 58.25 58.18 65.13

50.45 57.40 58.64 58.77 64.61

The per-category breakdown (Appendix B) localizes this residual gap. It concentrates in multi-hop questions, which require reasoning across several retrieved facts, exactly the cross-fact attention that a joint prefill provides and independent encoding omits (Theorem 6(3)), where Mem0 leads by as much as 16 points at 𝑘=50 (steepest on Mistral). On open-ended questions the ordering reverses on Llama and Mistral, where InferScale exceeds Mem0 by 3–9 points; single-hop and temporal accuracy are closer, with Mem0 generally holding a slight edge. The gap to prompt injection is therefore not a general loss but a targeted one, confined to the multi-hop reasoning that most depends on cross-fact attention.

6.4

Serving throughput

Setup. We measure sustained serving throughput under concurrent load on LoCoMo dataset. 𝑁 users share one conversation’s memory corpus, each user issues two distinct queries sampled from that conversation’s question set, and all 2𝑁 requests are submitted to the vLLM engine as a single batch. Decoding is greedy and every request generates exactly 50 output tokens, so conditions differ only in how retrieved memory reaches the model. We sweep 𝑁 ∈ {10,25,50,75,100} and report QPS as completed requests divided by the engine’s batch generation time. The engine is warmed with length-matched random-token batches and the prefix cache is reset before measurement, so no measured prompt is cache-seeded. Both systems retrieve top-50 facts per query: Mem0 re-prefills them as prompt text, while InferScale injects the pre-composed KV. Results. Figure 5 shows that InferScale’s throughput scales nearlinearly with the number of concurrent users, while Mem0 saturates early. At 100 users, InferScale reaches 100/104/135 QPS on Llama/Mistral/Qwen versus Mem0’s 27/23/32 QPS, a 3.7–4.5× speedup, and the gap widens with concurrency (Mem0 gains only ∼2× from 10 to 100 users, whereas InferScale gains ∼4×). The reason is that InferScale injects precomputed KV tensors and prefills

InferScale : GPU-Native KV Injection for Personalized LLM Serving

Throughput (QPS)

Llama-3.1-8B

Qwen2.5-7B

Mistral-7B

120

50 75 Concurrent users

100

InferScale (GPU KV)

50 75 Concurrent users

100

Concurrent users Mem0

InferScale (CPU-offloaded KV)

Figure 5: Sustained serving throughput on LoCoMo (queries/s) vs. the number of concurrent users, for InferScale (GPU-resident and CPU-offloaded KV) and Mem0. InferScale’s throughput scales near-linearly with concurrency, whereas Mem0 saturates; the GPU and CPU-offloaded curves overlap. Table 3: Robustness to model scale: Qwen3-14B on LoCoMo at 𝑘=50, 𝑤=50. Judged accuracy (%) and latency (ms) for InferScale (GPU-resident and CPU-offloaded KV) vs. Mem0 in its default configuration (Qdrant vector store with prompt injection). Full sweeps over 𝑘 and 𝑤 and the per-category breakdown are in Appendix D. Method InferScale (GPU KV) InferScale (CPU KV) Mem0

Accuracy (%)

TTFT (ms)

Q-to-first-tok. (ms)

69.42 69.42 79.09

28.06 31.11 140.52

88.02 129.55 257.76

Table 4: Average offline model KV-tensor precomputation time in seconds per conversation. Each value averages the complete 𝑘 ∈ {5,10} runs, with 10 LoCoMo conversations per run. Model

𝑤 =0

𝑤 =5

𝑤 =20

𝑤 =50

Llama 3.1 8B Mistral 7B v0.3 Qwen2.5 7B

38.91 15.51 30.91

48.25 20.76 41.56

105.59 41.19 80.77

206.45 78.24 166.50

Table 5: Average per-conversation storage footprint in decimal MB. Jasper graph and fact-KV tensors are GPU metrics; the fact–ID map is a CPU metric. Model

only each request’s short query, so many more requests fit in a batch, Mem0 re-prefills the retrieved facts on every request and bottlenecks on prefill compute. Offloading the KV store to CPU costs no throughput loss: the CPU and GPU curves are indistinguishable, because the one-time PCIe transfer is overlapped and amortized across the batch’s decode.

6.5

Robustness to model scale

To test whether InferScale’s techniques hold beyond the 7–8B regime, we repeat the evaluation on Qwen3-14B. Table 3 reports the main results for (𝑘=50, 𝑤=50). The full sweeps over 𝑘 and 𝑤 and the per-category breakdown are in Appendix D. The results follow the same pattern as with other models evaluated. The latency advantage, grows with model size. KV injection holds TTFT flat at ∼28 ms while Mem0 climbs to 141 ms (5.0× lower), and lowers end-to-end query-to-first-token by 2.9×. This is due to the fact that a larger model’s prefill is costlier for prompt injection to repeat on every request. CPU offloading remains inexpensive, adding only ∼3 ms to TTFT. Regarding accuracy, context-window encoding is essential in recovering accuracy loss due to chunked RoPE encoding (lack of cross attention in memory facts). Encoding facts in isolation (𝑤=0) collapses accuracy as 𝑘 grows (to 24.7% at 𝑘=50), while a window restores it to 69.4%, higher in absolute terms than on the 7–8B parameter models. InferScale’s mechanism is therefore scale to larger models, and its serving benefit strengthens at scale.

Llama 3.1 8B Mistral 7B v0.3 Qwen2.5 7B

6.6

Jasper GPU

Map CPU

Fact KVs GPU

13.50 13.50 13.50

7.97 3.02 6.48

4796.67 2257.44 1780.10

Cost of context-window encoding

Context-window encoding improves accuracy (Section 6.3) at a purely offline cost. However, encoding each fact behind its 𝑤 preceding turns lengthens the one-time preprocessing forward pass. Table 4 reports this cost per conversation. It grows with the window, on Llama, from 39 s at 𝑤=0 to 206 s at 𝑤=50 (∼5×), with the same trend on Mistral and Qwen. This is because a larger prefix means more tokens per encoding pass. The cost is paid once per conversation and amortized across all of that user’s requests. Crucially, 𝑤 does not affect the serving latency. It changes neither the stored nor the injected token count, so TTFT is flat across 𝑤 (Figure 4) and the GPU footprint is unchanged (Section 6.7). The window is therefore a pure offline-compute knob that trades one-time preprocessing time for accuracy, with no serving-time or memory penalty.

6.7

GPU memory footprint

InferScale keeps three structures resident per conversation (Table 5): the Jasper proximity graph [12], a CPU-side fact–ID map, and the pre-RoPE fact KV store. The retrieval structures are negligible in size. The Jasper graph and the map together cost under 25 MB per conversation on every model. The footprint is dominated by the KV store

Peter Li and Prashant Pandey

Table 6: Latency impact of offloading the pre-RoPE KV store to pinned host DRAM (streamed to the GPU over PCIe) vs. keeping it GPU-resident, at 𝑤=5 and 𝑘=50 (the largest budget). All values in ms, averaged over 1,540 queries. Offloading adds ∼1.5–3 ms to engine TTFT but a larger, PCIe-bound ∼33–38 ms to the end-to-end latencies at this budget. Model

KV store

TTFT

Q-to-first-tok.

Q-to-full-answer

Llama-3.1-8B

GPU CPU

17.52 20.45

62.12 100.29

274.02 312.66

Mistral-7B

GPU CPU

17.02 18.93

53.10 86.42

583.65 618.32

Qwen2.5-7B

GPU CPU

17.82 19.29

67.37 103.42

482.43 519.98

(1.8–4.8 GB per conversation), which is also the only term that scales with model architecture, growing with the number of layers, KV heads, and head dimension: Qwen2.5-7B’s aggressive grouped-query attention makes it the cheapest (1.78 GB) and Llama-3.1-8B the most expensive (4.80 GB). The KV store therefore bounds how many users can stay resident at once. After model weights and vLLM’s paged cache, a single 96 GB GPU holds the memory of roughly ten (Llama) to several tens (Qwen) of concurrent conversations, but because each fact is encoded once and reused across all of a user’s requests, this is a one-time, amortized cost rather than a per-request one.

6.8

Offloading KV embedding to CPU

The per-conversation KV embedding is the biggest component that resides in GPU memory and can become a bottleneck for supporting larger memory context. At 1.8–4.8 GB it limits how many users’ memory fits in GDDR. However, to overcome this bottleneck we can keep the pre-RoPE KV in pinned host DRAM and stream only the retrieved facts’ tensors to the GPU over PCIe on each request. Table 6 measures what this costs at serving time. Offloading carries a minor cost due to PCIe data transfer latency. Engine TTFT rises by only ∼1.5–3 ms (e.g., 17.5 → 20.5 ms on Llama at 𝑘=50), since the injection step itself is unchanged. The end-to-end latencies, however, absorb the host-to-device transfer of the retrieved facts’ KV over PCIe: query-to-first-token grows by ∼33–38 ms at 𝑘=50 (e.g., 62 → 100 ms on Llama), and query-to-full-answer by a similar amount. This overhead scales with the retrieval budget, from ∼8 ms at 𝑘=5 to ∼38 ms at 𝑘=50 on Llama, because more retrieved KV must be streamed. Even so, CPU-offloaded query-to-first-token (≈100 ms on Llama at 𝑘=50) is 2.3× faster than Mem0’s (236 ms). Offloading therefore lifts the GDDR capacity bound at a per-request cost of tens of milliseconds that scales with 𝑘 (full sweep in Appendix C).

6.9

Empirical equivalence

Theorem 3 guarantees identical outputs under exact arithmetic. We test whether this survives bf16 paged attention, where GPU reductions are non-deterministic. On each model we build three users (89– 143 memory tokens) and issue a factual-recall and an open-ended query each (18 cases). Both paths consume the identical prompt tokens and differ only in whether the memory prefix’s KV is supplied by

Table 7: Empirical-equivalence summary: outcome counts over the six (user × query-type) cases per model; factual accuracy is 100% in all cases. Full per-case breakdown in Table 19 of Appendix E. Model

Identical

Sem. equiv.

Different

Llama-3.1-8B Mistral-7B Qwen2.5-7B

3 2 4

3 4 1

0 0 1

Total

InferScale’s connector or recomputed by prefill, with greedy decoding. Table 7 summarizes the outcome: most cases are token-for-token identical or semantically equivalent, one differs, and factual accuracy is 100% throughout. This is exactly what bf16 predicts: the two paths reach the same prefix KV through different reduction orders, so a near-tie between logits can flip the greedy argmax and cascade without changing factual grounding, confirming Theorem 3 in practice.

Conclusion

We presented InferScale, a GPU-native memory system that replaces the repeated prefill of retrieved memory with reusable KV state: it precomputes each fact’s KV once and injects it into vLLM’s paged cache, made relocatable by chunked RoPE and made accurate by context-window encoding, with no fine-tuning, and vLLM engine changes. We observe that InferScale can extend to any workload that repeatedly conditions on a largely static retrieved context such as RAG over a stable corpus, cached system prompts, tool descriptions, and we expect attention-layer injection to become a default primitive for conditioning LLMs on static context. Across three open-weight models on LoCoMo, InferScale keeps engine TTFT nearly constant as the retrieval budget grows—3.6– 4.8× lower than Mem0 at 𝑘=50—and raises throughput by 3.7–4.5× at 100 concurrent users, while context-window encoding recovers accuracy to within a few points of prompt injection (60.3% vs. 63.3%). Offloading the KV store to host DRAM further lifts the GPU-memory capacity bound at only a few-millisecond TTFT cost. InferScale opens several directions. First, its GPU-resident KV store is the capacity bottleneck; scaling to larger contexts and more users calls for eviction and paging policies that keep hot memory on the GPU, spill cold memory to host DRAM or disk, and prefetch along the retrieval path. Second, our current store is built once and treated as read-only; supporting online memory updates, incrementally inserting, revising, and deleting facts, with consistent updates to both the Jasper index and the pre-RoPE KV store, would let InferScale track memory that evolves as conversations continue. Third, extending InferScale to multi-node, multi-GPU deployments raises questions of sharding the KV store and retrieval index across devices and routing requests to where a user’s memory resides, so that capacity and throughput scale beyond a single accelerator.

Acknowledgments This research is funded in part by NSF grant OAC 2339521 and 2517201.

InferScale : GPU-Native KV Injection for Personalized LLM Serving

References [1] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). [2] Yaoqi Chen, Jinkai Zhang, Baotong Lu, Qianxi Zhang, Chengruidong Zhang, Jing Liu, Jingjia Luo, Di Liu, Huiqiang Jiang, Qi Chen, Bailu Ding, Xiao Yan, Jiawei Jiang, Chen Chen, Mingxing Zhang, Cheng Li, Yuqing Yang, Fan Yang, and Mao Yang. 2026. RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference. Proceedings of the VLDB Endowment 19, 5 (2026), 1016–1031. doi:10.14778/3796195.3796212 [3] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413 doi:10.48550/arXiv.2504.19413 [4] Gemma Team. 2024. Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118 doi:10.48550/arXiv.2408.00118 [5] Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data (2019). [6] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [7] Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and Lili Qiu. 2025. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval. In Advances in Neural Information Processing Systems, Vol. 38. Curran Associates, Inc., Red Hook, NY, USA, 28 pages. https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 4e36d4049fb0fea195a8267c8dcd0824-Abstract-Conference.html [8] Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665 doi:10.48550/arXiv.2510.09665 [9] Dongyang Ma, Yan Wang, and Tian Lan. 2025. Block-Attention for Efficient Prefilling. In The Thirteenth International Conference on Learning Representations. OpenReview.net, Singapore, 15 pages. https://proceedings.iclr.cc/paper_files/paper/ 2025/hash/a03037317560b8c5f2fb4b6466d4c439-Abstract-Conference.html [10] Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. In Proceedings of the 62nd Annual Meeting of the Association for

Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 13851–13870. doi:10.18653/v1/2024.acl-long.747 [11] Yu A. Malkov and Dmitry A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 4 (2020), 824–836. [12] Hunter McCoy, Zikun Wang, and Prashant Pandey. 2026. GPU-Accelerated ANNS: Quantized for Speed, Built for Change. arXiv:2601.07048 doi:10.48550/arXiv.2601.07048 [13] Mistral AI Team. 2024. Mistral-7B-Instruct-v0.3 Model Card. Hugging Face model repository. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 Accessed 17 July 2026. [14] Hiroyuki Ootomo, Akira Naruse, Corey Nolet, Ray Wang, Tamas Feher, and Yong Wang. 2023. CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs. arXiv preprint arXiv:2308.15136 (2023). [15] OpenAI. 2024. New Embedding Models and API Updates. OpenAI product announcement. https://openai.com/index/new-embedding-models-and-apiupdates/ Published 25 January 2024; accessed 17 July 2026. [16] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560 doi:10.48550/arXiv.2310.08560 [17] Qdrant Solutions GmbH. 2026. Qdrant: Vector Database and Vector Search Engine. Software repository. https://github.com/qdrant/qdrant Accessed 17 July 2026. [18] Qwen Team. 2024. Qwen2.5 Technical Report. arXiv:2412.15115 doi:10.48550/arXiv.2412.15115 [19] Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956 doi:10.48550/arXiv.2501.13956 [20] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing 568 (2024), 127063. doi:10.1016/j.neucom.2023.127063 [21] Llama Team and AI @ Meta. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 doi:10.48550/arXiv.2407.21783 Qwen3 Technical Report. arXiv:2505.09388 [22] Qwen Team. 2025. doi:10.48550/arXiv.2505.09388 [23] Haocheng Xia, Mihir Pamnani, Hanxi Fang, Supawit Chockchowwat, and Yongjoo Park. 2026. LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding. arXiv:2606.04302 doi:10.48550/arXiv.2606.04302 [24] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 94–109. doi:10.1145/3689031.3696098

Peter Li and Prashant Pandey

Table 10: Per-category judged LoCoMo accuracy (%) vs. retrieval budget 𝑘 for Mistral-7B: InferScale (per encoding window 𝑤) vs. Mem0 (prompt injection).

Appendix

Full Serving-Latency Results

Figure 4 plots serving latency for a representative encoding window; Table 8 gives the full per-window breakdown—engine TTFT and end-to-end query-to-first-token for InferScale at every window 𝑤 and for Mem0, across all three models. InferScale’s latency is essentially unchanged across 𝑤 (within 1 ms), confirming that the encoding window is an offline-only cost.