arXiv CS open access

When Cache Pressure Becomes a Trap: Metastability in Agent Serving

Chengzhi Lu et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

LLM agents repeatedly send a growing conversation to a model, making prefix reuse central to efficient serving. We show that a brief surge can disrupt this reuse and trigger metastable failure in prefill--decode (PD) disaggregated engines: service remains degraded after arrivals return to a load the

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Chengzhi Lu

Boxi Zhou

Nanyang Technological University Singapore

Yanying Lin

Cambridge, MA, USA

Nanyang Technological University Singapore

chengzhi.lu@ntu.edu.sg

bzhou011@e.ntu.edu.sg

yanying_lin@seas.harvard.edu

dmitrii.ustiugov@ntu.edu.sg

Harvard University

surge

Abstract LLM agents repeatedly send a growing conversation to a model, making prefix reuse central to efficient serving. We show that a brief surge can disrupt this reuse and trigger metastable failure in prefill–decode (PD) disaggregated engines: service remains degraded after arrivals return to a load the engine previously sustained. On SGLang, a ten-minute surge reduced goodput by about 88% over the following hour. Removing the residual burst sessions restored service within minutes. The sustaining mechanism is population–cache feedback. Slower calls prolong sessions and enlarge their aggregate context, or working set; the resulting cache misses increase recomputation and further delay departures. A fluid model identifies the resource conditions for this feedback and explains how cache capacity, tool time and base load affect vulnerability. Across 36 simulated deployments, it flags all 17 cache-first cases. Controlled simulations hold the post-surge state fixed and show that changing subsequent arrivals alone reverses the recovery outcome. Once the cache collapses, low-reuse departure capacity sets the scale of the load reduction needed for recovery. Guided by these dynamics, we build BASIN, which gives each session a fixed priority, reserves their predicted peak contexts against an adaptive admission budget, extends the cache with idle decode memory, and sheds excess demand. On real H200 1P3D deployments under surges, including longcontext SWE-bench Pro sessions, an arrival-time prototype achieves 1.20–1.70× the goodput of Continuum and 2.35– 3.59× that of HiCache alone. In simulation, the predictordriven policy delivers 1.20–2.05× the strongest baseline’s goodput across six surge scenarios.

Dmitrii Ustiugov

more sessions active

slower calls, sessions stay

loop

working set > cache

cache misses: re-prefill

(a) Why: a feedback loop

loop active

cache fits

completions / min

arXiv:2611.02543v1 [cs.DC] 2 Nov 2026

When Cache Pressure Becomes a Trap: Metastability in Agent Serving

tipping point

healthy arrivals recovers

piles up

surge

active sessions (b) Effect: completions fall

Fig. 1: Population–cache feedback (schematic). (a) Cache misses extend session lifetimes, increasing the working set and further reducing reuse. (b) Departures rise with active sessions until cache pressure reduces throughput. A surge crossing the unstable intersection with base arrivals leaves departures below arrivals and active sessions accumulating.

disaggregated engines [65, 67, 95] use prefix caching to reuse this earlier computation. A returning call then prefills only its new tokens, reducing prefill work by up to 50× when the prefix is available. The problem. This reliance on prefix reuse also creates a vulnerability: a brief surge can leave the engine persistently degraded even after arrivals return to a rate it previously sustained. On SGLang, a ten-minute burst reduces goodput by about 88% over the following hour (§3). Before the burst, prefill and decode both operate well below saturation. Utilization, queue length and KV usage, commonly used for scaling and overload control [42, 67, 74, 92], all show ample headroom and give little warning of the failure. The degradation persists after the trigger ends, a defining feature of metastable failure [7, 31]. Why it happens. The degradation outlasts the burst because slower calls keep sessions active and competing for the cache. A surge enlarges the engine’s working set: the combined context of active sessions, including those executing tools. Cache contention evicts prefixes, forcing returning calls to recompute more tokens. The extra prefill work slows calls and keeps sessions active longer, since a session ends only after its final turn. New arrivals during this delay enlarge the working set, causing more eviction and recomputation (Fig. 1). To

Introduction

LLM agents build long conversations as they alternate between model calls and tool executions. Each call resends much of the preceding context: in the Terminal-Bench traces we study, the median prompt contains 25.6K tokens, 98% of which appeared in the preceding call. Prefill–decode (PD) 1

completions

deeper fall

declines slowly

too tight

too loose

vulnerability, and why preserving reuse can still leave a long backlog. Recovery also depends on cache history: in simulation, hit rates remain low after collapse at working sets that sustain high reuse in healthy runs. This hysteresis limits throughput during recovery. We use this analysis to design BASIN, which tracks session contexts across calls and tool executions (Fig. 2d). It assigns each session a fixed priority from its arrival time and predicted peak context, using the same order for admission and both engine queues. Idle decode memory preserves prompts between calls, reducing recomputation. Running decode requests reclaim the lent memory on demand (§6). An adaptive working-set budget regulates admission, and shedding removes persistent excess demand. On real H200 1P3D deployments, an arrival-time prototype delivers up to 3.59× HiCache’s goodput and 1.70× that of the strongest baseline, Continuum. In simulation with the working-set predictor, BASIN achieves the highest goodput in all six end-to-end scenarios, 1.20–2.05× the strongest baseline’s. In the simulated 3–6× surge-size sweep on 1P3D, BASIN avoids trapping and completes at least 95% of sessions while shedding at most 2.2% (§7). This paper makes three contributions:

2 surge

working set

3 working set

(a) Capacity

(b) Scheduling

(d) Ours

Fig. 2: Existing remedies (a–c) and our design (d) on the curve of Fig. 1b (gray: unchanged engine). (a) More capacity moves the cliff; vulnerability remains if the cache still limits the engine first. (b) Serving sessions already in progress first limits entry into the working set, but a surge can leave a long queue. (c) A fixed load or capacity threshold can admit too much or hold sessions out unnecessarily. (d) BASIN combines fixed session priority (2), lent cache capacity (1), and admission and shedding for excess load (3).

test whether these accumulated sessions sustain the failure, we terminate the sessions introduced by the burst; service recovers within minutes. In a matching run that lets those sessions finish, active sessions keep accumulating after the last burst session departs, with arrivals at the original base rate. Once this feedback develops, the engine can complete fewer sessions than arrive at the original base rate, so active sessions continue to accumulate after the surge ends. Recovery requires completions to exceed new arrivals, making the outcome depend on both the state left by the surge and the load that follows it. Controlled simulations confirm this dependence: runs with the same post-surge state recover under lighter arrivals and remain trapped under heavier arrivals. For a given deployment, the recovery boundary therefore depends on the base load.

• We identify population–cache feedback: slower calls keep sessions active, and continuing arrivals sustain cache thrashing after a finite surge. Real-system interventions establish its causal role in metastable failure in PD-disaggregated agent serving (§3). • We develop a fluid model that identifies resource conditions for cache-driven vulnerability and explains the effects of capacity, load and entry control. Across 36 simulated deployments, it flags every cache-first case. Controlled interventions reveal how continuing arrivals and cache hysteresis govern recovery (§4, §5).

Existing remedies. To improve the balance between session arrivals and completions, existing systems change capacity, service order or admission. Capacity expansion adds cache tiers or borrows idle decode resources to raise prefill throughput [48, 67, 84, 85]. More cache supports more active sessions before the cliff, but feedback remains possible if cache pressure still limits service before decode saturates (Fig. 2a). Scheduling changes request or session priorities to preserve reuse and accelerate completion [1, 43, 93]. Prioritizing sessions already in progress can preserve reuse but leave new sessions waiting when completions barely exceed arrivals (Fig. 2b). Admission control bounds entry using KV usage, cache feedback or tier capacity [9, 37]. KV usage can remain low while tool-idle sessions accumulate prefix demand, whereas a fixed cache-capacity budget can delay admission even while reuse remains high (Fig. 2c).

• We design BASIN to preserve reuse through a fixed session priority, admission by predicted peak context, and decode-memory lending. A SGLang prototype establishes the benefits of session ordering and lending on hardware; simulation evaluates the full predictor-driven policy across deployments and loads (§6, §7).

Background

Prefill–decode disaggregation. Each model call begins with prefill, which processes the prompt in parallel, followed by decode, which generates one token per request at each step. These phases have different compute and memory demands, so PD engines place them on separate instances and transfer the request’s key–value (KV) state from prefill to decode [65, 67, 95]. In SGLang, a request first reserves KV space on decode, then runs prefill, transfers its state and begins decoding. Decode retains this KV throughout generation, often requiring several decode instances per prefill

Our approach. To explain how capacity, scheduling and admission change population–cache feedback, we model the active session population as a fluid, with departures determined by closed-loop demand, cache reuse and decode capacity (§5). The model identifies deployments where cache pressure limits service before decode saturates, flagging all 17 cache-first cases in a 36-deployment comparison. It explains how cache capacity, tool time and base load change 2

0 10

−2

−1

Prompt (K tokens)

Gap between calls (s)

(a) Prompt length per call

(b) Gap between calls

101 100 0

Fig. 3: Workload: 216 Terminal-Bench trajectories, 7,108 calls.

100

0.5 0.0

Calls / min

50 40

100

(b) Prefix hit rate

100 20

Time (min)

150

Prefix caching. The KV state computed during prefill can also be reused by later calls that share the same prefix. SGLang retains prompt KV in a radix tree on prefill, allowing a request that extends a cached prefix to prefill only the uncached suffix [93]. When memory fills, the engine evicts least-recently-used prefixes. Hierarchical caches such as HiCache extend capacity with finite host-memory or remote-storage tiers [67, 84]. Hybrid models also maintain fixed-size recurrent state in Mamba, lightning or gated-delta layers [45, 56, 68]. Reusing a prefix requires retaining this state alongside its KV, so recurrent-state slots impose another capacity constraint.

100

10 5 2 1 0.5

Time (min)

Session start (min)

(d) Session slowdown

Fig. 4: Same arrivals at a base load of 3.3 sessions/min, with and without a surge in minutes 15–25 (shaded). (d) compares sessions by start time against the no-surge run.

These sessions reuse their contexts for minutes, making prefill cost sensitive to whether their prefixes remain cached. We serve this workload with Qwen3.6-35B-A3B (FP8), a hybrid linear-attention MoE model, using SGLang in PD disaggregation on one prefill and one decode instance, each on an H100 NVL GPU and connected by NIXL. The prefill instance holds a 1.09M-token prefix cache and the decode instance 1.66M tokens of KV. Sessions arrive as a Poisson process and replay their recorded calls and tool gaps. Each run starts from steady operation with 42 active sessions joined mid-trajectory.

Agentic workloads. Agents revisit these prefixes as they alternate between model calls and tool executions. In the ReAct pattern, each action’s result is appended to the context before the next call [89], often for tens of turns in coding tasks [35, 54, 86]. Serving systems expose these sequences as programs that pause during tool execution [1, 43, 52]. Their contexts remain useful for subsequent calls even while the tools are running [39]. The resulting workload is partly-open [70]: new sessions arrive independently of engine progress, but each session issues its next call only after the preceding call and tool execution finish. Slower service therefore delays subsequent calls and keeps sessions active longer, while new sessions continue to arrive.

3.2

A Transient Surge, a Persistent Failure

The engine can remain degraded long after a surge ends, even at a base rate it previously sustained. Fig. 4 compares the same base arrival stream at 3.3 sessions/min, with and without ten-minute surges of +33% and +100%. Without a surge, the engine uses a twelfth of its saturated prefill throughput and maintains a median time to first token (TTFT) of about 0.2 s. It recovers from the +33% surge within twenty minutes. The +100% surge, however, leaves the engine deteriorating after arrivals return to 3.3 sessions/min: median TTFT reaches about 60 s and prefix hit rate falls from 0.95 to 0.27. From minute 25 onward, it completes 0.80× as many calls as the surge-free run. Under the goodput definition of §7.1, sessions arriving in the following hour achieve 17 sessions/h against 138 without the surge, and goodput over the whole run falls from 123 to 43 sessions/h. Sessions arriving after the surge take 1.8–6.1× longer (Fig. 4d), and the degradation persists for the remaining hour.

Metastable failures. Slow service keeps more sessions active, and their increased cache demand can, in turn, sustain the slowdown through additional cache misses. This follows the pattern of metastable failure: a trigger drives a vulnerable system into a degraded state, and a sustaining effect preserves that state after the trigger ends [7]. Retries, garbage collection and cold look-aside caches are known sustaining effects [31].

3.1

surge

(a) Median TTFT

instance. Reported deployments range from 2.25 and 3 decode nodes per prefill node for DeepSeek and Kimi K2 on SGLang [77, 78] to a 4-node prefill unit paired with a 40-node decode unit in DeepSeek’s deployment [14].

+100%, 10 min 1.0

Time (min) Slowdown ratio

TTFT p50 (s)

+33%, 10 min

Prefix hit rate

No surge

CDF

Metastable Failures in Agent Serving

3.3

Workload and Setup

Residual Sessions Sustain the Failure

To test whether sessions left active after a burst sustain the degradation, we inject short sessions at four times the base rate during minutes 15–25. These sessions complete within five minutes on a healthy engine. We compare two runs: one lets the burst sessions finish on their own, and the other terminates them all at minute 25. Both runs retain the same base arrivals. By the end of the burst at minute 25, both runs have about

Our real-system experiments replay 216 Terminal-Bench trajectories containing 7,108 calls. Prompts grow incrementally: the median contains 25.6K tokens and the 90th percentile 77K, with 98.2% of prompt tokens repeated from the preceding call (Fig. 3a). Outputs are short, at a median of 160 tokens, and the median gap from a response to the next call is 0.42 s (Fig. 3b). 3

burst

burst, sessions ended at 25

burst period

mean lifetime T as λT , where T includes serving and tool time.

1.00

Prefix hit rate

Median TTFT (s)

no burst

101 10

100

0.75

Cache pressure. As sessions accumulate, so does their working set: the aggregate context of all active sessions, including those executing tools. The growing working set causes LRU to evict prefixes before their sessions return. Across the two-minute windows of our real runs, hit rate declines from 0.95 to 0.80 before dropping sharply to about 0.3 when the working set reaches roughly twice the cache capacity (Fig. 6b). We call this transition the cache cliff. The resulting recomputation raises uncached prefill work in the +100% run from 2.5K to 28–35K tokens/s, near the saturated throughput of 31K tokens/s (Fig. 6c). Prefill delays also tie up decode memory because SGLang reserves decode KV before prefill begins. After the +100% surge, requests awaiting prefill hold two thirds of decode KV (Fig. 6d). Decode memory is 97% full, yet only half as many requests decode as in the surge-free run. New requests wait 25 s for a reservation, further extending session lifetimes. Together, these effects form a feedback loop: cache misses prolong sessions, and the additional active sessions sustain cache misses. The +33% surge leaves the working set below the cliff and the engine recovers. The +100% surge crosses it, reducing departures below base arrivals. In the short-session experiment, removing the burst sessions reduces the working set and breaks this loop. The cliff raises service cost sharply; recovery then depends on whether the engine can complete sessions faster than new ones arrive. Section 5 models this balance and tests the role of continuing arrivals.

0.50 0.25 0.00

Time (min)

(a) Median TTFT

(b) Prefix hit rate

100

1.00

burst

100 50

Uncached (K tok/s)

100

0.00

window median

(a) Active sessions, trapped run

(b) Hit rate vs. working set

+100%, 10 min

prefill capacity

0.25

Working set / cache size

20 0

0.50

Time (min) No surge

0.75 cache size

base

Prefix hit rate

total

Time (min) (c) Uncached prefill

100

Decode KV used

Active sessions

Fig. 5: The same arrivals at a base load of 3.3 sessions/min: without a burst, with a burst of short sessions at 4× the base rate in minutes 15–25 (shaded), and with the same burst whose sessions all end at minute 25. Dotted line: last call of the burst sessions that are left in place.

decoding

reserved, idle

1.0 0.5 0.0

100

Time (min)

(d) Decode KV, +100% run

Fig. 6: The sustaining loop in real runs; shaded: the surge. (a) Active sessions after a burst of short sessions at 4× the base rate, split into base and burst sessions. (b) Each 2-minute window of all runs and their binned median; dashed: cache size. (c) Prefill tokens that miss the cache. (d) Decode KV of decoding requests vs. reserved for requests awaiting prefill.

Observation 1. A transient surge can sustain prefix-cache thrashing by extending session lifetimes and enlarging the working set. Removing the residual burst sessions breaks this feedback; recovery depends on both the working set and the base arrival rate.

110 active sessions, including 65 burst sessions, and median TTFT is about 9 s. Terminating the burst sessions restores service within two minutes. When left to finish on their own, the burst sessions remain active for up to 25 more minutes, while the number of active base sessions grows to 1.7× that in the no-burst run by minute 50. The hit rate recovers only briefly when the last burst sessions finish. Active base sessions continue to accumulate, reaching 2.4× the no-burst count by minute 90 (Figs. 5 and 6a). Recovery after removing the burst sessions shows that the accumulated sessions sustain the failure. At the same base arrival rate, the engine either remains healthy or degrades persistently depending on its earlier exposure to the burst. This dependence on past load is a defining feature of metastable failure (§2).

3.4

Characterizing Vulnerability

The feedback identified in §3 creates a vulnerable region below steady-state overload. We map this region and show how deployment resources change it using a calibrated discrete-event simulator. Plots in this section and §5 report simulation results unless explicitly identified as real measurements. Methodology. The simulator replays the real driver’s trajectories, arrival process and warm start. Each call is released only after its predecessor completes and the recorded tool gap elapses. Slower service therefore keeps sessions active longer while new sessions continue to arrive, preserving the session-level feedback. The simulator models prefill queues and cache tiers, decode KV reservation, and decode costs as functions of batch size and running context. H100 and H200 use separate engine calibrations (§7.10). We summarize persistent latency in these sweeps using Llate , the maximum reported window-median TTFT in the final 30 minutes. A value above 10 s identifies persistent high

Population–Cache Feedback

Session lifetime. Slower calls keep sessions active longer because each session departs only after its final turn. Meanwhile, new arrivals add more active sessions. In steady state, Little’s law relates mean population to arrival rate λ and 4

2 of 3

1 of 3

Load / overload

trapped, 3 of 3 runs 1 0.85 0.7 0.4 25

100

200

recovers

overloaded

100

saturates with either GPU-only caching or a host tier twice the GPU cache at every evaluated tool time. With tool time scaled by 30, the cache remains the first limit even with a host tier ten times the GPU cache. A trapped deployment with a 125 GB host tier reaches a median TTFT of 335 s, 3.3× the GPU-only H100 value. Larger GPUs raise the absolute triggering load while leaving substantial vulnerability below overload. The H200 profiles provide roughly twice the cache and decode memory and 1.6–1.75× the nominal prefill throughput of H100. On 1P3D, the first load trapped by a +300% surge rises from 3.5 sessions/min on H100 to 5.5 on H200. Across the GPU and host-tier sweep, this trapping load remains between 0.46 and 0.70 of the steady-state overload rate. Proportional expansion of prefix-cache and decode memory preserves their ratio. The model in §5 relates this ratio to the fraction of active sessions in service to identify which resource limits the engine first.

1 0.85 0.7 0.4

300

200

Surge (+%, 10 min)

(a) 1P1D, GPU cache

(b) 1P3D, HiCache

300

Fig. 7: How often a ten-minute surge traps the engine, by base load (relative to steady-state overload) and surge size.

x40

3.25

x10

3.25

GPU

3.25

2.75

x10

x30

decode saturates first

Cache tier

cache cliff first

x40

19.5

x10

19.5

14.5

12.5

9.5

GPU

8.5

6.5

3.5

x10

x30

Tool time

(a) 1P1D

(b) 1P3D

Fig. 8: Highest load (sessions/min) that recovers from a +100% surge, by cache tier (GPU only, or a host tier of 2–40× the GPU cache) and tool time. Color: the first resource to limit service.

Observation 2. Finite surges trigger cache-driven failure well below steady-state overload. More cache raises the recoverable load; additional decode capacity and longer tool execution shift the first limit toward the cache.

latency. The controlled post-surge intervention in §5.2 uses the full trapped-state definition of §7.1.

4.1

The model tracks how arrivals and completions change the active session population. It connects this balance to prefix reuse and prefill–decode capacity, explaining the vulnerable regions in §4 and how entry control and added resources change them.

Vulnerability Below Saturation

Finite surges trigger persistent cache thrashing well below steady-state overload. Following prior metastability studies [31], we vary base load and the size of a ten-minute surge, from +25% to +300%. On 1P3D with HiCache, a +300% surge traps the engine at half the overload rate measured without a surge (Fig. 7). As base load approaches that rate, the surge needed to trigger failure decreases to +50% on 1P3D and +25% on GPU-only 1P1D. Trigger sizes in this range also occur in production arrivals: after thinning two Alibaba Bailian traces [80] to a single deployment’s rate, their busiest ten-minute windows exceed the hourly mean by 16–49%.

4.2

A Fluid Model of Session Dynamics

5.1

Population and Resource Limits

Population dynamics. Let N(t) count active sessions, including those executing tools; we call this the session population. Initially, assume every arriving session becomes active. Sessions arrive at rate λ(t) and issue K calls on average, separated by tool gaps of mean g. We approximate the session departure rate by call throughput divided by calls per session: dN X(N) = λ(t) − , dt K

More Cache Moves the Boundary

Adding cache raises the recoverable load until decode becomes the first limit. Fig. 8 varies HiCache host capacity, the ratio of prefill to decode instances (P:D), and tool time. For each configuration, we report the highest base load that recovers from a +100% surge and the limiting resource above that load. On 1P3D at the default tool time, expanding from GPU-only caching to a host tier 40 times GPU capacity raises this recoverable load from 9 to 19.5 sessions/min. More decode instances and longer tool execution both shift the bottleneck toward the prefix cache. Additional decode instances expand generation capacity relative to the prefill cache, while tool execution prolongs demand for cached contexts between decode requests. On 1P1D with a host tier, decode is the first observed bottleneck except at the longest tool times. On 1P3D, cache reuse deteriorates before decode

(1)

where X(N) is call throughput at population N. Throughput is constrained by closed-loop demand, prefill work and decode capacity; we combine their bounds with a smooth minimum. At small N, the closed loop limits throughput to N/(g + L0 ), where L0 is unloaded call latency. As cache misses increase prefill work, the prefill throughput bound µ p /u(N) falls. Here µ p is saturated prefill throughput and u(N) = max nnew , (1−h(W /C))P is uncached work per call. The mean prompt length is P, and nnew is the new suffix length. The hit rate h(W /C) depends on working set W = N c̄ relative to prefix-cache capacity C, where c̄ is mean context length across active sessions. The model uses the real-system hit-rate curve in Fig. 6b. Decode capacity bounds throughput by the number of requests whose KV fits in memory divided by their decode 5

budget B

100

Active sessions N

(a) Equilibria and tipping

150

0.75 0.50

safe

1P1D 1P3D

flagged

missed: none

0.25 0.25

0.50

0.75

1.00

Model: cache / decode bound (b) Model vs. simulator

1.0

103 102 101 0.1

Fig. 9: (a) Departure rate in the fluid model for 1P3D with HiCache: arrivals meet it at a stable equilibrium Ns and an unstable one Nu , beyond which the population grows. (b) Resource-based vulnerability classification across simulated deployments. The model flags those left of the dashed line; all 17 cache-first deployments fall in this region.

host 2×, +100%

GPU-only, +100%

Prefix hit rate

trigger

over-cautious

pre-surge base

2.5 0.0

host 2×, +300%

GPU-only, +300%

1.00

Sessions at 2 h

5.0

trapped: dN/dt > 0 arrivals λ

Sim. hit rate at overload

Departures / min

N∗

7.5

0.3

0.5

0.7

0.9

Base rate after surge (× overload) (a) Recovery vs. base rate

same working set, two states

0.5

no surge after +300%, trapped

0.0

Working set / cache size

(b) Hit rate, host 2×

Fig. 10: Post-surge arrival intervention on 1P3D with HiCache alone (three seeds). (a) Mean active sessions at the end of the two-hour run, by post-surge arrival rate; the dashed line marks the pre-surge rate. Filled markers: no seed trapped; hollow: all three trapped; gray: some trapped, using the state definition of §7.1. (b) Host-tier hit rate at comparable working sets, with and without a prior collapse. Trapped-run samples cover minutes 40–120; shading marks the overlap.

time. Comparing this bound with the cache-dependent prefill bound determines whether falling reuse can reduce throughput before decode limits service. Equilibria and tipping. When cache misses make prefill the first limit, the departure curve D(N) = X(N)/K rises with the session population, peaks at N ∗ , and then falls. After reuse collapses, departures approach a low-reuse rate Dlow . A base arrival rate between Dlow and the peak D(N ∗ ) intersects the curve twice: at a stable equilibrium Ns and an unstable equilibrium Nu (Fig. 9a). Populations below Nu return toward Ns ; beyond Nu , departures remain below arrivals and the population grows. The same base load thus supports healthy service before a surge and sustained population growth afterward. Raising base load moves Ns and Nu closer together, so a smaller population increase triggers failure. Arrivals below Dlow let the population drain while reuse remains poor.

It also flags eight of the 19 decode-first deployments, giving 28 correct classifications (78%).

5.2

Continuing Arrivals Sustain Failure

For a fixed post-surge state, continuing arrivals determine whether active sessions drain or keep growing. We isolate this effect by running HiCache alone on H200 1P3D with GPU-only caching or a host tier twice the GPU cache. Each run starts at half the deployment’s reference overload rate λref and receives a ten-minute surge in minutes 15–25. We then set arrivals to 0.1–0.9 λref . Within each seed and surge setting, the runs have identical histories and engine states through minute 25. We apply the trapped-state definition of §7.1 to the final two 20-minute windows, comparing against no-surge controls with the same arrival schedule. The reference rates are 12.0 sessions/min with GPU-only caching and 16.5 with the host tier. Changing only subsequent arrivals reverses the outcome (Fig. 10a). After the +300% surge on GPU-only 1P3D, every seed recovers at 0.1 λref and remains trapped at 0.2 λref . With the host tier, every seed recovers at 0.2 λref and remains trapped at 0.5 λref ; two of three seeds are trapped at each of 0.3 λref and 0.4 λref . After the smaller +100% surge, these same deployments remain healthy throughout the tested rates. Larger surges therefore leave states that require a greater reduction in arrivals to recover. Recovery depends jointly on the state left by the surge and the arrival rate that follows.

Vulnerability condition. The cache cliff and decode-memory saturation impose competing limits on the active population. The measured cliff is near Nc ≈ 1.9C/c̄. Let DKV be decode KV capacity and f the fraction of active sessions in service. Decode memory then fills near Nd ≈ DKV /( f c̄). Comparing these population estimates gives the condition for the cache cliff to precede decode-memory saturation: DKV . (2) C ≲ 2f More host capacity raises Nc , moving the cache cliff to a larger population. More decode memory raises Nd , and longer tool gaps raise Nd by reducing f : more sessions are executing tools at a given decode occupancy. These shifts explain why added decode capacity and longer tool execution favor cache-first failure. Scaling C and DKV together preserves the criterion at fixed f ; GPU service rates can also change f .

Cache history and recovery. After cache collapse, reuse remains low even at working sets that support high hit rates in healthy runs (Fig. 10b). On host-tier 1P3D, the simulated hit rate remains at 0.26–0.37 after collapse at working sets 1.6–1.7 times cache capacity; no-surge runs at comparable working sets retain 0.81–0.93. This is cache hysteresis: similar demand for cached context produces different hit rates because the cached contents differ. Prefixes evicted during collapse must be recomputed before they become reusable

Deployment classification. To account for service rates as well as memory, we compare the model’s peak throughput under the closed-loop and prefill bounds with its decode throughput bound. Fig. 9b plots their ratio for 36 simulated deployments spanning P:D ratios, cache tiers and tool times. With engine costs matched to the simulator and a 1% tolerance at the crossover, the model flags all 17 cache-first deployments. 6

Table 1: Design principles and their realization in BASIN. W = N c̄: working set; Ŵi : predicted peak context of session i; Wc : cliff; Ceff : cache capacity; B0 = min(mWc ,Ceff ): reference budget; B: adapted budget.

again. The low-reuse state limits the departure capacity available for recovery. In trapped runs, measured departures are 0.20–0.31 of λref on GPU-only 1P3D and 0.20–0.24 with the host tier. At the pre-surge arrival rate of 0.5 λref , arrivals exceed these departures and keep sessions accumulating. Recovery requires a departure surplus while reuse remains low. Throughput measured in this collapsed state sets the scale of the arrival reduction needed to drain active sessions.

5.3

Implications for Mitigation

In Eq. 1, every base rate between Dlow and the departure peak is vulnerable to a finite surge. Mitigations act on this balance by regulating entry into the active population and raising the departure curve. Serving admitted sessions first controls entry. Sessions waiting for their first call have yet to create private cached context. Let Q(t) count these sessions and I(t) be their admission rate into the active population. Then dN = I(t) − D(N), dt

dQ = λ(t) − I(t). dt

Model term

Mechanism

Engaged when

Priority

entry balance: Eq. 3

fixed session priority

Capacity

C, hence Wc and D(N ∗ )

Entry

lend decode memory as a cache tier Denning admission: ∑ Ŵi ≤ B shedding

persistent prefill backlog, wait for decode memory, or W ≥ αO B0 W ≥ αL B0 of the prefill tiers or admission active, decode has spare memory W ≥ B0

Q when λ > D(N ∗ )

long-run arrivals exceed the peak

λ − D(N). For a fixed departure curve and sustained λ > D(N ∗ ), this total grows at least at rate λ − D(N ∗ ). Bounding the backlog therefore requires added departure capacity or long-run shedding of at least this excess rate.

(3)

Basin Design

BASIN follows three principles from the model: preserve reuse, raise departure capacity, and regulate entry. It applies Denning’s working-set principle [17] at session granularity, reserving for each admitted session the largest context it is predicted to reach. Each session receives a fixed priority from its arrival time and predicted demand. This priority orders admission and both engine queues, favoring smaller sessions while limiting how long later arrivals can overtake earlier ones. Decode-memory lending extends the cache and raises departures on cache-limited deployments, while shedding removes persistent excess demand. Table 1 maps these principles to the model terms they change and their activation conditions. A session gateway runs the controller, and extensions to the prefill and decode engines apply its decisions. The gateway combines a deployment profile with measurements of outstanding prefill work QP , requests waiting for decode memory QD , prefix hit rate h, working set and long-run arrival ¯ It evaluates these signals every control interval ∆, rate λ. and more often while backlog or an active control persists. Releasing a control requires its condition to remain clear for a cool-down.

Under resident-first scheduling, new sessions enter using the service capacity left after resident calls. As cache misses increase resident prefill work, less capacity remains for entry, opposing further population growth. This feedback explains how program ordering with KV retention across tool calls [1, 43] preserves reuse while new sessions accumulate in an external queue. When ordering maintains the population near the departure peak, N ≈ N ∗ , the queue evolves as dQ ≃ λ(t) − D(N ∗ ). dt

Control

(4)

At constant base load λ < D(N ∗ ), a backlog of S sessions takes approximately S/(D(N ∗ ) − λ) to drain. The drain time therefore diverges as base arrivals approach the departure peak from below, even with healthy cache reuse. Capacity raises the departure peak. More cache supports a larger resident population before recomputation limits service. A host tier, a larger GPU or idle decode memory can supply this capacity. Borrowing decode compute for prefill [85] raises the prefill bound directly. A falling departure branch leaves base rates between Dlow and the new peak vulnerable to finite surges after capacity expands. Once decode limits service, decode capacity caps the departure peak. Under resident-first order, the larger departure surplus also drains the external queue faster. Admission explicitly bounds entry. A working-set budget scaled to the cache cliff regulates admission according to the working-set principle [17]. Parking first calls when the budget is full regulates the entry rate I(t) into the active population. Admission divides unfinished sessions between N and Q. Adding their balances in Eq. 3 gives d(N + Q)/dt =

6.1

Working-Set Estimation

A session’s latest context remains reusable while its tools run, so BASIN counts it throughout the session’s lifetime. To decide when controls engage, it estimates aggregate demand as W = N c̄, where N counts active admitted sessions and c̄ is a smoothed mean context length. New sessions immediately contribute c̄ to this estimate, allowing the controller to anticipate their demand before their contexts fully develop. Admission reserves demand per session. Since context grows across turns, the first prompt alone understates a 7

Online controller (every interval Δ)

Algorithm 1 BASIN’s control loop, run every control interval ∆. Parameter values: §6.6.

Offline profile

Monitor

ramp and predictor cliff scale κ

Policy

telemetry

actions

Require: profile (κ, λc ); capacities (CG ,CH ); predictor of Ŵ Require: interval ∆; margin m; thresholds α; lending cap φ; shedding horizon Ts ; priority rate r ¯ W ← N c̄ 1: read QP , QD , h, N, c̄, λ; 2: for each new session i: predict Ŵi ; pi ← ai + Ŵi /r ▷ fixed Capacity 3: C ← max(CG ,CH ); B00 ← min(mκxkC,C) 4: lend ← (W ≥ αL B00 ∨ admit)∧ spare decode memory 5: L ← held-unit capacity, at most φ of decode memory; Ceff ← max(C,CG + L) 6: B0 ← min(mκxkCeff ,Ceff ) Entry 7: if W ≥ B0 or (admit ∧W ≥ αR B0 ) then 8: admit ← true; adapt B to h within a bounded range around B0 9: else if parked queue nonempty then 10: admit ← false; raise B gradually 11: else 12: admit ← false; B ← ∞; update plateau from h Scheduling and shedding 13: order ← prefill backlog or decode-memory wait persists 14: order ← order ∨ (W ≥ αO B0 ) ∨ admit 15: order both engine queues by p when order 16: admit queue heads by p while they fit B or no residents remain ¯ > λc and highest-priority parked wait > Ts then 17: if λ 18: shed the lowest-priority parked session 19: release actions after their conditions clear for a cool-down

PD engine

Agent sessions

admit / shed

fixed priority

Router

Prefill

session queue

prefix cache

lent units KV

Decode run

lent

tokens

Fig. 11: BASIN’s online controller and offline profile.

session’s eventual footprint. At the first call, BASIN predicts Ŵi , the session’s peak context length, including input and output tokens. The predictor uses the task text and first-prompt length visible at that call (§6.5). The prediction stays fixed until completion, and the sum of these reservations over residents governs further admission. This sum accounts for future growth; W = N c̄ supplies the live demand signal that activates controls. The deployment profile gives a cliff Wc = κxkCeff , where xk ≈ 1.9 is the reference hit-curve knee and κ scales this curve to the deployment. Let CG and CH denote aggregate prefill GPU and inclusive host capacity, and L the estimated lent capacity. We use C = max(CG ,CH ) and Ceff = max(C,CG + L) to account for overlap between tiers. The prototype estimates L by smoothing the capacity of retained units. The reference budget is B0 = min(mWc ,Ceff ), combining a margin m < 1 below the profiled cliff with the physical cache capacity. The capacity term anchors reservations to available memory; the cliff term accounts for locality lost through routing or fragmentation. Counting each session’s peak separately reserves space for its later growth, while shared prefixes and tool-idle intervals can make actual occupancy smaller. Hit-rate feedback adapts the admission budget B around B0 to exploit this reuse. Thus B0 provides a reference for control, and the current B sets the reservation limit when admission is active. The profile also supplies the peak departure rate λc used for shedding.

6.2

lent pages. On the next turn, prefill compares its local prefix coverage with the unit. If the unit covers more context, decode writes the missing state into prefill’s KV pages in layer order, overlapping transfer with execution (§6.6). Reclamation. Decode retains units only with headroom beyond queued and active decode demand: adoption requires spare tokens beyond that demand, and reclamation begins at a lower margin. Lending is capped at a fraction φ of decode memory. Units also held by prefill are reclaimed first, followed by LRU units. Decode allocations reclaim lent pages before retracting running requests. Reserving decode KV at prefill completion prevents requests awaiting prefill from occupying this headroom.

Memory Lending for Prefix Reuse

Lending expands the cache with idle decode memory while prefill computation remains on prefill instances. On cache-limited deployments, retaining more prefixes reduces recomputation and raises departure capacity. BASIN enables lending once W ≥ αL B0 , with B0 computed from the prefill tiers alone, and keeps it enabled while admission is active. To preserve decode headroom, it stops adopting units when requests are waiting for decode memory or running requests occupy most of that memory.

6.3

Session Priority for Cache Packing

To fit more sessions within the cache budget, BASIN gives smaller predicted footprints a priority advantage. At its first call, session i receives the priority pi = ai + Ŵi /r, where ai is its arrival time and r converts tokens of predicted demand into seconds; smaller values go first. A larger r gives arrival time more weight; a smaller r favors smaller footprints more strongly. BASIN uses the same key in the parked queue, the prefill queue and the decode-memory queue. Returning calls retain their key even after eviction, allowing the session to recompute and finish under its original priority. Smaller

Units. A decode instance retains the prompt state already delivered by the PD transfer as a unit after the call completes. The unit contains the transferred KV pages and, for hybrid models, a copy of the prompt-end recurrent state stored in 8

sessions can precede larger residents; admission controls the active cohort through the entry rate I(t) in Eq. 3. Keeping the key fixed avoids reshuffling service order as a session waits or its context grows. The arrival term limits overtaking: if a new session j arrives after ai + Ŵi /r, then p j > pi even for a zero-size prediction. Later arrivals therefore stop passing session i after a finite interval; its actual wait also depends on the work already ahead of it. With equal estimates, the key reduces to arrival order. Ordering engages when a prefill backlog or a wait for decode memory persists. It also engages at W ≥ αO B0 or whenever admission is active (Table 1). When the active cohort operates near peak departure capacity, Eq. 4 estimates how quickly completions drain the remaining external backlog.

6.4

design; the head of the queue waiting Ts shows that admission itself has stalled.

6.5

Offline Profiling

BASIN locates the cache cliff with a stepped load ramp. At each step, it lets the number of active sessions settle and records working set and hit rate. It fits κ by aligning a reference curve, whose knee is xk ≈ 1.9, with the observed transition from the healthy plateau to the low-reuse floor. Engine rates and session statistics are measured separately, and BASIN uses the lower quartile of κ across repeated ramps. For several prefill instances, it profiles each router separately to account for prefix distribution, then applies a single budget to aggregate capacity (§7.6). To estimate each session’s peak context, BASIN fits a separate ridge regression [28] for each workload. The regression predicts log peak context length from term frequency–inverse document frequency (TF-IDF) features [69] of the task text and the log lengths of the task text and first prompt. We regularize the text-feature coefficients and fit the two length coefficients freely. Predictions are cross-fitted: each trajectory is evaluated using a model trained on the remaining trajectories, with vocabulary, IDF weights and feature centering computed from that training set. On Terminal-Bench and SWE-bench Pro, these predictions have rank correlations of 0.64 and 0.56 with peak context, and log-error standard deviations of 0.54 and 0.34, respectively.

Admission for Working-Set Control

Admission complements the fixed service order by limiting the combined reservations of active sessions, following the working-set principle [17]. Parked and newly arriving sessions form one queue in priority order. With a finite budget B, BASIN admits its head i when ∑ j∈res Ŵ j + Ŵi ≤ B. The sum reserves each resident’s predicted peak from the start, covering subsequent growth up to that prediction. When the head’s reservation exceeds the remaining budget, it stays parked at the gateway, as in ThunderAgent’s router port [37, 51]. If no residents remain, the head enters even when its prediction exceeds B, ensuring progress. Admitted sessions continue to completion; prediction errors and capacity changes affect subsequent entry through hit-rate feedback.

6.6

Implementation

BASIN adds a session gateway before SGLang’s PD router and extends the engine’s allocation and cache paths for unit lending (Fig. 11). The hardware prototype implements arrival-time ordering, current-context admission, lending and shedding. The simulator evaluates the complete predictordriven controller, including peak-context reservations and the fixed size-aware key, on the same calibrated engine model.

Activation. Admission engages when W ≥ B0 and remains active until W < αR B0 has held for the cool-down. The lower release threshold and cool-down keep control active through short-lived drops in demand. The reference B0 determines activation, while the adapted B determines which first calls enter. When control is inactive and no parked sessions remain, the budget is unrestricted. Budget adaptation and shedding. Following page-faultfrequency control [12], BASIN raises B when sessions are parked but reuse remains on its healthy plateau, and lowers it when reuse deteriorates. Adaptation stays within a bounded range around B0 ; downward steps account for measured resident demand, and admitted sessions continue running. Lent capacity remains in the budget during a healthy-hit-rate prefill backlog, allowing the engine to use that cache while computation catches up. After admission is released, gradual budget growth spreads the return of parked sessions over time. BASIN uses shedding to remove persistent excess demand. ¯ exceeds the profiled peak If the long-run mean arrival rate λ departure rate λc and the highest-priority parked session has waited Ts , BASIN sheds the lowest-priority parked session, one per interval (Algorithm 1). Under a size-aware priority the oldest parked session is often a large one that waits by

Parameters. The predictor-driven controller uses ∆ = 30 s (5 s while backlog or a control persists), m = 0.9, (αL , αO , αR ) = (0.8, 0.9, 0.8) for lending, ordering and release, φ = 0.5, Ts = 15 min and r = 30 tokens/s. Section 7.9 varies these parameters, together with the implementation settings for cool-down, backlog persistence, context smoothing, reopening and lending headroom. Gateway. The gateway identifies each call’s session, holds parked first calls and runs the controller. It polls existing Prometheus metrics every 5 s to derive hit rate, outstanding prefill work and requests awaiting decode memory after prefill completion. The hardware prototype uses session arrival times to order both engine queues; the simulator supplies the predicted size-aware key to the corresponding queues. When lending is active, the gateway marks prompts for retention on decode. 9

CONCUR

500 R OO

250 0

ThunderAgent

O O O

O O R T T

×5 surge

GPUonly

O T

tools ×3

(a) Goodput

O O O T T TT

SWEPro

Continuum

Median wait (s)

Goodput (sess./h)

HiCache

Crossflow

Composition

Basin

We simulate 1P1D, 1P2D and 1P3D, plus 2P6D in §7.6. Unless specified as GPU-only, HiCache has a host tier twice the GPU prefix capacity. Base load is relative to the deployment’s profiled critical rate. The main comparison starts at half the deployment’s profiled critical rate and applies a ×5 surge during minutes 15–35 on 1P1D and 1P3D, using the default workload, tool time scaled by 3, or GPU-only caching. Further sweeps vary trigger size and duration, output length, cache capacity and routing.

100 ×5 surge

GPUonly

tools ×3

SWEPro

(b) Wait of new sessions

Fig. 12: Real H200 1P3D results: three ×5 surge scenarios on Terminal-Bench, and the same surge on SWE-bench Pro from 2.6 sessions/min. The prototype uses arrival-time ordering and current-context admission. (a) Goodput; letters above bars give the run state: T trapped, O overloaded, R recovering; unmarked bars are healthy. (b) Median first-call wait.

Workloads. We replay the Terminal-Bench trajectories of §3 and 70 SWE-bench Pro [15] sessions as a second coding-agent workload on hardware. On Terminal-Bench and SWE-bench Pro, median prompts contain 25.6K and 83K tokens, median peak contexts reach 40K and 107K tokens, and median tool gaps are 0.42 s and 48 ms, respectively. Each call becomes eligible after its predecessor completes and the recorded tool gap elapses. The deployment and policy sweeps use TerminalBench unless specified; sensitivity experiments scale tool time or output length, and Bailian supplies alternative arrival patterns for these trajectories.

Reservation at prefill completion. Decode KV reservation follows prefill completion, while prefill retains the request’s KV until transfer. We extended this allocation path to NIXL, the decode radix cache, write-through HiCache on prefill and hybrid-model state. Unit lending. Decode stores completed prompt pages by unit identifier, with Mamba state copied before the first decode step. Allocation and retraction paths adopt and reclaim units under the headroom rules. Prefill reports which units it still holds so decode can reclaim redundant copies first, and maintains a per-session index of the latest unit’s decode instance and length. When prefill’s local radix match covers fewer tokens than the latest unit, prefill allocates pages and requests NIXL writes of the missing state in layer order. A 25K-token unit transfers within tens of milliseconds over NVLink. To overlap transfer with execution, decode publishes completion flags after each group of layers, and a Triton kernel waits on the corresponding flag before each layer executes. Attention and Mamba layers run outside SGLang’s captured piecewise-prefill segments, keeping the captured graphs intact. The wait kernel is launched only for batches fetching unit data. A CPU monitor detects timeouts or failed decode instances and falls back to recomputation.

7 7.1

Real deployments. We implement the arrival-time BASIN prototype, Continuum, ThunderAgent and CONCUR in SGLang with NIXL PD transfers and page size 64 (§6). The hardware deployments are one four-H200 node in 1P3D and two H100 NVL GPUs in 1P1D. Both serve Qwen3.6-35B-A3B in FP8, with 30 linear-attention layers among its 40 layers. The replay client uses the same session trajectories, arrival schedules and tool gaps as the corresponding simulated configuration. Runs last 2 h, or 2.5 h with tool time scaled by 3; session metrics cover arrivals in the first 90 minutes. Baselines. We compare HiCache alone with concurrency control in CONCUR [9] and ThunderAgent [37, 51], ordering and KV retention in Continuum [43], and decode-side elastic prefill in Crossflow [85]. A composition combines ThunderAgent’s admission bound, Crossflow’s borrowing, and Continuum’s ordering and TTL pinning. CONCUR is included in the main end-to-end and hardware comparisons and in the steady-load and production-arrival sweeps; each later figure shows the systems its comparison concerns, in the same order and colors. ThunderAgent bounds admitted footprint by the larger cache tier. Our CONCUR port uses decode occupancy for KV usage and enables turn-boundary pausing. We raise the hit-rate threshold from 0.2 to 0.7 to place it within the observed range. Scheduling priorities apply to both prefill and decode-memory queues. Crossflow shares the mixed-step token budget between prefill and decode and adapts its lease to a 50 ms ITL SLO. BASIN uses the same SLO and is configured as in §6.

Evaluation Methodology

Hardware experiments compare BASIN with existing policies on H200 and H100 (§7.2). Simulation covers the broader deployment, load and policy sweeps. The comparisons in §7.2 report real measurements; the subsequent performance sweeps use simulation. Section 7.10 reports calibration and hardware agreement. Simulator and deployments. The closed-loop simulator uses separate H100 and H200 calibrations (§7.10). It accounts for token-position-dependent prefill cost, batchand context-dependent decode steps, KV reservation and retraction, recurrent-state slot limits, and host and decode-lent cache tiers.

Metrics. Goodput is the number of completed sessions per hour with slowdown at most 4 and at least 95% of decode steps meeting the ITL SLO. Slowdown divides elapsed session 10

ThunderAgent

103

10−1 0

Time (min)

120

(a) Prefix hit rate

Time (min)

120

(b) Wait of new sessions (s)

Fig. 13: The ×5 surge (minutes 15–35, shaded) on the real H200 1P3D deployment, in 5-minute bins of arrival time: (a) prefix hit rate of the calls; (b) median wait of the sessions arriving in the bin.

100 75 50 25 0

GPU only

worst scenario

100 41

(a) Goodput (higher is better)

800 400

316

290

234

200

382

287

224

100

e ach

HiC

time, including first-call wait, by an unloaded reference computed from tool gaps, call overheads, new-suffix prefill and batch-one decode using the engine’s calibrated costs. Rejected and shed sessions count as failed, and completion fraction covers all evaluated arrivals. Warm-start sessions are admitted initially and excluded from session metrics. Latency metrics include median and p90 TTFT, first-call wait and session slowdown. A 20-minute window is degraded if median TTFT exceeds 5 s, median first-call wait exceeds 5 minutes, or at least 25% of arriving sessions never start. A run is trapped if its final two windows remain degraded, improve by less than 25%, and both show prefix hit rate at least 0.10 below the pretrigger rate. These windows end at least 35 minutes after the trigger. Persistent degradation without improvement but with retained cache reuse is labeled overloaded; an improving run is recovering. Labels are unchanged for hit-rate thresholds from 0.10 to 0.20.

7.2

×5 surge tools ×3

1P3D (left) 1P1D (right)

Basin

Median slowdown, % of lowest Goodput, % of best

Continuum

t gen

n itio

low

uum

n asi

n ssf os erA nti mp Cro Co und Co Th (b) Median slowdown (lower is better)

Fig. 14: Each marker is one scenario on 1P3D or 1P1D: a ×5 surge, the same surge with tool time scaled by 3 or a GPU-only cache. Color marks the deployment (1P3D filled, left; 1P1D hollow, right), shape the scenario, and bars each system’s worst scenario. (a) Goodput as a percentage of the best system in the scenario. (b) Median slowdown of completed sessions, including the wait before the first call, as a percentage of the lowest in the scenario.

Sessions

HiCache

Continuum

250 0 0

Crossflow

Median TTFT (s)

CONCUR

100 0

Time (min) (a) Sessions

(b) Median TTFT 1

CDF 0 100

0 101

102

Session slowdown (c) Session slowdown

The hardware prototype uses arrival-time ordering and current-context admission (§6.6); the predictor-driven controller is evaluated in simulation from §7.3 on. On hardware, BASIN achieves 1.20–1.70× the goodput of Continuum and 2.35–3.59× that of HiCache alone across the four H200 1P3D scenarios (Fig. 12). HiCache remains trapped in all four, completing only 0.28–0.54 of sessions. On Terminal-Bench its prefix hit rate falls from 0.97 to 0.33–0.35. Crossflow is trapped in all four as well, completing 0.26–0.53 of sessions. On Terminal-Bench, ThunderAgent and the composition maintain reuse but impose median first-call waits of 26–39 minutes; on SWE-bench Pro both are trapped. CONCUR also preserves reuse except with GPU-only caching, where its hit rate falls to 0.33 before its concurrency window contracts and recovers slowly. On SWE-bench Pro, whose sessions reach a median of 107K tokens against 40K on Terminal-Bench, BASIN completes 126.7 sessions/h against 106.0 for Continuum and 86.0 for CONCUR, with a median first-call wait of 68 s. Across the three Terminal-Bench scenarios, Continuum avoids the trap, but its hit rate remains 10–15 points below the pre-surge rate and new sessions wait 3–6 minutes. Under BASIN, median first-call wait is at most 18 s. The GPU-only

Time (min)

Real Deployments

Basin

102

CDF

HiCache

10−1

100

101

102

Wait of new session (s) (d) Wait of new sessions

Fig. 15: Response to a ×3 surge during minutes 15–25 (shaded) on 1P3D with output lengths scaled by 3, where decode saturates first: (a, b) time series and (c, d) session distributions. Unfinished sessions appear at the right edge of (c).

scenario gives the largest gap to Continuum, because lending provides the only cache tier beyond prefill GPU memory. In the default surge (Fig. 13), HiCache’s hit rate collapses within ten minutes of surge onset and remains low through the end of the run. BASIN drains the backlog by minute 75, while Continuum drains it near the end of the run. On H100 1P1D, decode saturates first (Eq. 2). Under the ×5 surge, BASIN, Continuum and the composition achieve 56.7, 55.3 and 55.3 sessions/h. For the arrival-time prototype, limited lending headroom and the absence of cache-driven vulnerability leave little throughput benefit over ordering alone; the size-aware priority adds 1.20–1.48× in simulation (§7.3).

7.3

Surge Response

BASIN achieves the highest goodput in all six scenarios of Fig. 14. Each baseline loses at least 51% in one or more scenarios. On cache-first 1P3D, BASIN exceeds Continuum, 11

×5

×4

×3

×6

Surge (20 min)

overloaded

HiCache CONCUR

500 0

×5

×4

×3

×6

Surge (20 min)

ThunderAgent

500 0 10

Offered load (sessions/min) (a) Goodput

HiCache, Poisson

10.0

Continuum

Crossflow

Basin

102 101 100 10

12.5

15.0

17.5

20.0

overloaded

102 101 100 5

Mean load (sessions/min) (b) 1P3D, outputs ×3

(a) 1P3D

HiCache ThunderAgent

Composition Basin

Offered load (sessions/min) (b) TTFT, p90

Fig. 17: Steady load on 1P3D with tool time scaled by 3.

the strongest baseline in all three cases, by 1.64× under the default surge, 1.33× with longer tools and 2.05× with GPUonly caching. Relative to HiCache, the gain is 3.13–4.17×. Median slowdown falls from 6.6 to 3.6 under the default surge and from 6.6 to 3.0 with GPU-only caching. On 1P1D, BASIN achieves 1.20–1.48× the strongest baseline’s goodput. Under the default surge, it completes 92% of sessions and reduces p90 first-call wait to 0.56 of the composition’s. Fig. 15 follows a surge on 1P3D with output length scaled by 3, making decode the first bottleneck. BASIN, HiCache and Continuum achieve 232, 231 and 229 sessions/h, respectively, with p90 first-call wait of 4 s under BASIN. Crossflow’s median TTFT rises to about 100 s as prefill competes with decoding. In 15 healthy control scenarios, BASIN retains at least 99.6% of the best baseline’s goodput. At most 1.3% of its sessions violate the ITL SLO on more than 5% of their steps across the main end-to-end scenarios.

7.4

beyond capacity

not trapped trapped

Fig. 18: Session arrivals shaped by the Bailian coder trace, two hours. Shaded: offered load beyond capacity. (a) Dotted: HiCache under Poisson arrivals; marker fill as in Fig. 16. (b) Median TTFT in the second hour, where the first call of a parked session includes its wait at the router.

p90 wait (s)

TTFT p90 (s)

Goodput (sess./h)

CONCUR

Crossflow Basin

Mean load (sessions/min)

Fig. 16: Surges of increasing size, 20 minutes long. End-state markers follow the legend and §7.1. HiCache

1000

(b) 1P3D, tools ×3

(a) 1P3D

ThunderAgent Continuum

Median TTFT (s)

500

not trapped trapped

103 101 1×

2×

4×

8×

Host tier / GPU prefix cache (a) Steady load, wait of new sessions

not trapped trapped

Goodput (sess./h)

Composition Basin

Goodput (sess./h)

Continuum Crossflow

Goodput (sess./h)

HiCache ThunderAgent

overloaded

500

1×

2×

4×

8×

Host tier / GPU prefix cache (b) ×5 surge, goodput

Fig. 19: ThunderAgent, Composition and BASIN under increasing host-cache capacity on 1P3D. Marker fill as in Fig. 16.

from 15.5 sessions/min onward. At the highest load, p90 TTFT is 11 s under BASIN, compared with 53 s for Continuum and 225 s for Crossflow. ThunderAgent keeps p90 TTFT at 3.1 s by holding sessions at admission, trading throughput for shorter admitted-call queues. Replaying Bailian’s arrival patterns with Terminal-Bench session trajectories produces the same goodput advantage. At 18.2 sessions/min under the coder pattern, just below the profiled critical rate of 18.5, BASIN achieves 1.41× Continuum’s goodput, 2.98× ThunderAgent’s and 3.4× CONCUR’s. At the highest mean load, 20.4 sessions/min, BASIN achieves 645 sessions/h under the coder pattern and 724 under trace A, compared with 315 and 428 for Continuum and 169 for CONCUR under the coder pattern (Fig. 18). With output lengths scaled by 3, the profiled critical rate is 9.9 sessions/min. At the highest coder load, 9.2 sessions/min, BASIN’s second-hour median TTFT is 0.2 s, compared with 17.8 s for HiCache and 187.8 s for Crossflow. Across the two highest loads, its p90 first-call wait is 20–34 minutes, compared with 45–54 under ThunderAgent and 46–59 under CONCUR. Maintaining reuse shifts part of the delay to admission; session slowdown includes both components.

Trigger Size and Sustained Load

Fig. 16 varies 20-minute surge factors from 3 to 6 on 1P3D and 1P3D with tool time scaled by 3. Their base loads are 0.55 and 0.66 of the profiled critical rate, respectively. Crossflow remains trapped throughout this range. Under BASIN, runs are healthy or recovering at the end, with 2.71–4.19× HiCache’s goodput and 1.21–1.63× the strongest baseline’s. At least 95% of sessions complete, compared with 32–48% under HiCache, while at most 2.2% are shed. Admission-bound systems retain much longer queues: sessions arriving 35 minutes after the two largest surges still wait about 35–48 minutes under ThunderAgent and the composition. In the steady-load sweep (Fig. 17), BASIN peaks at 948 sessions/h at an offered rate of 16.8 sessions/min and retains 769 at 19.4 sessions/min, the highest load shown. Continuum, CONCUR and ThunderAgent achieve 449, 312 and 84 sessions/h there. HiCache completes fewer than 20 sessions/h

7.5

Sources of Performance Differences

Resource borrowing. Crossflow runs prefill on decode instances, sharing compute with token generation. BASIN lends cache memory while keeping prefill computation on prefill instances, with decode allocations taking priority over lent pages (§6.2). Across the 1P1D scenarios, Crossflow achieves 0.30–0.40 of the best system’s goodput (Fig. 14). Crossflow is also sensitive to per-chunk overhead. Charging 12

cache-aware

affinity

least loaded

Crossflow

HiCache

Basin

1000 500 0

cache-aware

affinity

least loaded

(b) ×5 surge, 20 min

(a) Steady load

Fig. 20: 2P6D under three prefill routers. HiCache ThunderAgent Continuum

Crossflow Basin Composition

surge start / end Basin lends

Sessions

Time (min) (a) Basin from the start

100

103 102 101 100

100

Basin

250

0.1

0.3

0.5

λr /λref

0.7

0.9

0.1

(a) +300%, 10 min

switched on

Basin, priority only

0.3

0.5

λr /λref

0.7

0.9

(b) ×6, 20 min

Fig. 22: GPU-only 1P3D under varying post-surge arrivals: goodput (top, sessions/h) and p90 first-call wait (bottom, s; below 1 s drawn at 1 s) after (a) a +300% surge for 10 minutes and (b) a ×6 surge for 20 minutes. Arrivals before the surge are 0.5 λref ; results average three seeds. Hollow markers: Llate > 10 s in every seed.

1000 0

Continuum

500

priority order Denning admission

2000 1000

ThunderAgent

Goodput

1000

Continuum

p90 wait (s)

ThunderAgent

Goodput (sess./h)

HiCache

150

Time (min)

7.6

(b) Each system switched on at minute 45

Fig. 21: A ×5 surge for 20 minutes (minutes 15–35) on 1P3D; sessions include those parked at the router. (a) All systems; the bands at the top mark when BASIN lends decode memory, orders queues by its priority and limits admission. (b) Each system switched on at minute 45, after HiCache alone is trapped.

Multiple Prefill Instances

Routing changes usable cache capacity (Fig. 20). On 2P6D, the profiled critical rates are 39 sessions/min with cache-aware routing, 41 with affinity and 29 with least-loaded routing. The router-specific profiles give lower-quartile cliff scales of 0.81, 1.38 and 0.55, respectively. Under a ×5 surge, BASIN exceeds the strongest baseline by 1.58×, 1.53× and 2.18×, shedding at most 1.2% of sessions. Least-loaded routing scatters prefixes across instances: in its steady control, BASIN achieves 1742 sessions/h, while HiCache and ThunderAgent fall to 53 and 58.

a fixed 20 ms per prefill chunk reduces its goodput from 206 to 174 sessions/h under a ×3 surge and from 76 to 5 in the high-load tool-time-scaled case, relative to zero fixed charge. BASIN’s goodput is unchanged in this sensitivity test. Admission signals. In our CONCUR port, decode KV usage covers requests reserved, transferring or running on decode. Tool-idle sessions retain prefix-reuse demand without an in-flight decode reservation. At equal decode occupancy, different numbers of tool-idle sessions impose different cache demand, and the hit-rate trigger responds only after reuse deteriorates. CONCUR achieves 0.52 of BASIN’s goodput under the default 1P3D surge and 0.24 with GPU-only caching. Admission threshold. ThunderAgent ties its bound to the larger prefill cache tier. With a host tier equal to GPU capacity, BASIN achieves 2.33× ThunderAgent’s goodput and 2.30× the composition’s (Fig. 19). At host tiers of four and eight times GPU capacity, the ratios narrow to 0.98–1.00 against ThunderAgent and 1.22–1.36 against the composition. At this cache size, decode limits service: at eight times GPU capacity BASIN completes 14% more sessions within the slowdown SLO than the earlier arrival-time controller and 20% more than ThunderAgent, but 35% of them exceed the ITL allowance, against 18% and 24%. The cache-based budget allows decode concurrency to grow beyond the ITL target. Eviction policy. Across surge factors 3–6, next-call-oracle eviction achieves 1.08–1.18 of LRU’s goodput and 0.28–0.41 of BASIN’s. Tool-gap pinning remains within 1% of LRU. Largest-first eviction yields substantially lower goodput even in the steady controls.

7.7

Recovering a Trapped Engine

Fig. 21b activates each policy at minute 45, after HiCache is trapped. BASIN reduces the total number of resident and parked sessions to at most 60 by 103–128 minutes after activation across five seeds, completing 99.9% of measured sessions. Its goodput is 301.1 sessions/h, compared with 167.8 for Continuum, 136.0 for the composition and 108.9 for HiCache. HiCache completes 36.1% of sessions. The long drain time reflects the accumulated backlog even after service improves; Fig. 21a shows the corresponding run with controls active before the surge.

7.8

Systems under Post-Surge Arrivals

Fig. 22 applies the same post-surge arrival intervention to the evaluated systems on GPU-only 1P3D and 1P3D with a host tier. It uses the evaluation’s engine options while retaining the reference rates of §5.2. Goodput and first-call wait use the metrics of §7.1; the late-latency marker uses Llate from §4. The systems differ in both departure capacity and the delay imposed on new sessions. Following the +300% surge, ThunderAgent’s p90 first-call wait is 275–467 s at post-surge rates up to half the reference rate, while BASIN stays below 17 s across the sweep. Following the larger surge, Continuum leaves new sessions waiting 1.4–2.0K s. Across GPU-only and host-tier settings, BASIN achieves 1.00–1.38× Continuum’s goodput after the +300% surge and 1.29–2.77× 13

0.5 0.0

×5 ×8 ×6 late GPU GPU GPU ×4 ×5 ×8 late 1P1D 40m

lower value

102 101 100

(a) Goodput

×5 ×8 ×6 late GPU GPU GPU ×4 ×5 ×8 late 1P1D 40m

1.2 1.0 0.8

r impl.

(a) Controller parameters

(b) TTFT, p90

1.5

1P3D ×5

GPU-only ×5

1P1D ×4

1.0 0.5 0.0

prior pred. σ1.0 σ0.7 σ0.5

true

(b) Working-set estimate

Fig. 24: Sensitivity of Basin’s goodput (five seeds). (a) One controller parameter set lower or higher, relative to the default: ∆ 15/60 s, m 0.7/1.0, α 0.5–0.7/0.95–1 (three thresholds pooled), φ 0.3/0.7, Ts 7.5/30 min, r 10/100 tokens/s, and the nine implementation settings (“impl.”). Bars span the five ablation triggers and an hour of sustained overload on 1P3D; the white tick marks the median. m leaves the bound unchanged (diamond), as the capacity term binds. (b) Relative to the predictor-driven controller: the earlier arrival-time controller (“prior”), and the new controller supplied with true peaks perturbed by lognormal error σ or with exact peaks.

Fig. 23: Ablation of BASIN’s actions, relative to the full BASIN, on 1P3D with a host tier and with a GPU-only cache (“GPU”). “40m”: a 40-minute surge; “late”: BASIN switched on at minute 45 in a trapped engine; “1P1D ×4”: a ×4 surge on 1P1D; “prior”: the earlier controller, with arrival-time ordering and current-context admission.

after the ×6 surge, and 1.09–3.08× ThunderAgent’s. The larger-surge gain over Continuum is 1.66–2.77× with GPU-only caching and 1.29–2.10× with the host tier. Without lending and admission, the priority alone retains 0.95–1.05 of Continuum’s goodput after the +300% surge and reaches 1.19–1.64× after the ×6 surge: the size-aware priority and lent capacity both contribute to the gain over program ordering.

7.9

higher value

Goodput / Basin

1.0

w/o admission

Goodput / default

w/o priority order lending only

TTFT p90 / Basin

Goodput / Basin

w/o lending prior

Controller parameters. We vary the controller parameters of §6.6 to a lower and a higher value, one at a time, on the five ablation triggers and an hour of sustained overload on 1P3D (five seeds). Of the 204 variant–scenario pairs, 81% stay within 3% of the default goodput (median change 0.5%), and all lie within 0.80–1.33. The margin m has no effect in these sensitivity configurations, where the capacity term sets B0 . Each component of α changes goodput by at most 8%. A shorter interval (∆ = 15 s) raises goodput by up to 1.17× under sustained overload and costs up to 6% under the ×8 surge; 60 s costs up to 4%. The lending cap φ changes goodput by 0.97–1.07×, and φ = 0.7 lowers the GPU-only p90 firstcall wait to 0.77×. A shorter shedding horizon (Ts = 7.5 min) raises goodput by up to 1.07× under sustained overload by shedding earlier; a longer one (30 min) lowers it to 0.93×. The priority rate r trades goodput against tail admission delay. At r = 10 tokens/s, a stronger preference for small footprints raises goodput by 7–33%, while p99 first-call wait rises by 19–43%. At r = 100, closer to arrival order, goodput falls to 0.80–0.96×. We use r = 30 to balance goodput and tail wait. Among the implementation settings, a wide budget range ([0.25, 4] B0 ) costs up to 16% under sustained overload, slower context smoothing (90 min) up to 12% under the ×8 surge, and doubled budget steps up to 7% on 1P1D; the others stay within 0.96–1.03 across their tested ranges (Fig. 24a).

Mechanism Ablations

Fig. 23 removes each mechanism and includes lending alone. Returning to the earlier arrival-time controller retains 0.67–0.77 of the full policy’s goodput under large surges on 1P3D, 0.84 on 1P1D and 0.94–0.95 during recovery. This comparison changes both service order and demand accounting. To isolate engine ordering, we keep predicted admission but remove the priority from both engine queues: goodput falls to 0.81–0.95 with the host tier, 0.31–0.34 with GPU-only caching, 0.36 during recovery and 0.55 under sustained overload. Removing lending retains 0.94–1.03 with the host tier and 0.50–0.89 in the GPU-only and recovery cases. Lending alone achieves 0.32–0.56. Admission matters where the cache tier is large: without it, BASIN retains 0.62–0.82 under host-tier surges and 0.90 on 1P1D, while the GPU-only, recovery and sustained-overload runs stay within 1.00–1.03. Bounding at the profiled cliff alone, without the capacity term, retains 0.82–0.92 with the host tier, and dropping lent capacity from the bound under a compute backlog retains 0.83–0.87 with GPU-only caching. Under sustained overload with the host tier, admission reduces the mean number of active sessions from 852 to 252 and the prefill queue from 620 to 14 during minutes 60–75; it sheds 82 sessions and improves goodput by 1.25×, or 1.08× with GPU-only caching. An always-on variant achieves 1.05–1.11× the adaptive controller’s goodput under the two largest finite surges but 0.70–0.92 under steady and sustained load, where it holds sessions the adaptive controller admits. Scaling the profiled cliff by 0.5–2× changes goodput by at most 1%, as the capacity term sets the bound.

Working-set estimate. Fig. 24b varies predictions within the new controller and includes the earlier arrival-time controller as a reference. The latter reaches 0.76–0.84 of the full policy’s goodput in the three plotted scenarios and 0.67–0.95 across ablation triggers. Supplying exact peak contexts raises goodput by 1.27–1.35× in the plotted scenarios and 1.10–1.51× across triggers, showing further benefit from improved predictions. Exact peaks perturbed by independent lognormal error with σ = 1.0 also outperform the predictor (1.15–1.19× in the plotted scenarios), indicating that error structure and session ranking matter alongside 14

error magnitude. Under sustained overload, the variants reach 0.98–1.00 of the default goodput.

Metastability and overload control. Metastable failures [7, 19,26,31,32] belong to a broader class of feedback-driven performance failures, including congestion collapse [33], receive livelock [57], bistable loss networks [8,24], thrashing [16] and cache stampedes [59, 79]. Overload controllers limit work in progress through admission, dropping or degradation [10, 11, 20,30,55,64,81,96]. Related KV-cache planning uses working sets for capacity sizing [44]. BASIN applies the working-set principle [17] and page-fault-frequency control [12] to control the number of active sessions. EfficientAgent relates queueing to reuse distance in a fixed agent pool and sizes the host tier and writes accordingly [71]. With open session arrivals, slow service keeps more sessions active and enlarges their aggregate context, allowing cache thrashing to outlive its trigger. We describe this coupling with a population-dynamics approximation [13, 40] for a partlyopen workload [70].

The first-call features of §6.5 give leave-one-trajectory-out rank correlations of 0.64 on Terminal-Bench and 0.56 on SWE-bench Pro. Additional prediction checks on GitHub Copilot production traces [47] and two Bailian traces train on earlier sessions and test on later ones: first-call promptsegment lengths and model label give a rank correlation of 0.60 on Copilot, and first-request length gives 0.71–0.74 on Bailian. Log-error standard deviations across these checks range from 0.34 to 0.69.

7.10

Related Work

Simulator Fidelity

These comparisons use the earlier arrival-time controller for BASIN, matching the hardware implementation; the engine and cache models are shared with the predictor-driven sweeps. H100. Calibration covers five 1P1D configurations: surge runs of HiCache, Continuum, ThunderAgent and Crossflow, plus the GPU-only reproduction. Across five simulated seeds per configuration, mean goodput differs from the corresponding real run by at most 8.9%. All five real runs and their seed-0 simulations are classified healthy. In the later three-system H100 comparison, goodput error is within 3%. The simulator underestimates BASIN’s ITL-violating session share, at 0.5% against 13.6% measured.

Capacity management. Batching and memory management improve per-GPU throughput [3, 41, 90, 93], PD disaggregation sizes prefill and decode independently [60, 65, 67, 95], and cluster managers add, migrate or switch instances with demand [21, 74, 75, 83, 92]. Elastic designs lend idle decode resources to prefill [6,85]. KV systems expand capacity across host, storage and disaggregated memory [2, 48, 67, 84], retain conversation state [23,91], reduce loading and reuse costs [49, 88], and adapt routing or eviction to prefix locality [62,73,80]. Recent work also studies the effectiveness of replacement policies for LLM prefix reuse [50], and Yao et al. derive the LRU hit ratio of multi-turn conversations [87]. Our model explains how added cache changes vulnerability and identifies when decode becomes the first limit. AMPD and Dynamo retrieve session KV from the decode worker that produced it [27, 61], while MemServe pools memory across instances for context caching [29]. ConServe places a conversation’s later turns on the decode instance holding its state [18]. BASIN retains prompts in idle decode memory under explicit headroom and reclamation rules, with the cache index on prefill and running decode requests taking allocation priority.

H200. We fit decode-step cost to batch size and running context using two-minute windows from eighteen calibration runs, including the main trace and SWE-bench Pro. The latency fit has RMSE 0.57 ms. Prefill rate and the SWE-Pro prompt-length scale are also calibrated. Under the evaluation metrics, simulated goodput averaged over five seeds differs from the fifteen main-trace 1P3D runs in this calibration set by 8.8% on average and at most 32%. The largest BASIN error is a 21% underestimate in the GPU-only case. For the earlier arrival-time controller used by the hardware prototype, simulated BASIN/Continuum goodput ratios of 1.18–1.61 are below the measured 1.29–1.70. We assess accuracy beyond calibration using thirteen additional H200 1P3D runs: Crossflow and the composition in the three Terminal-Bench scenarios, and all seven systems on SWE-bench Pro. These runs were excluded from fitting. For the six Terminal-Bench runs, simulated goodput is within 2–12% of the measurement. On SWE-bench Pro it is within 5–11% for HiCache, CONCUR, Continuum and BASIN, which it ranks as measured, but it overestimates ThunderAgent and Crossflow by 33% and 37%, respectively. For the composition, it predicts 2.6× the measured goodput.

Agent scheduling and admission. Prior systems expose program structure [1, 25, 46, 76], schedule programs [52, 66, 72, 82], and retain or prioritize agent KV [34, 43, 63]. Continuum’s age order is the case of BASIN’s fixed priority in which all working-set estimates are equal. Length-aware request schedulers use output-length predictions for batching and memory allocation [36, 94], or learn relative lengths to approximate shortest-job-first scheduling [22]. BASIN predicts each session’s peak context across turns and uses this estimate to assign a fixed session priority and reserve cache capacity at admission. TOPAS coordinates prefix residency with request admission [58]. Admission policies use predicted latency [67], KV

Transfer-cost sensitivity. Replacing the fixed 10 ms PD transfer charge with a prompt-proportional charge changes BASIN’s goodput by at most 1.4% across five comparisons. 15

usage and hit rate [9], or GPU and tier capacity [37, 51]. Ao et al. derive request-level admission from a fluid model with a unique equilibrium [5]. Our working-set model explains how these controls affect session accumulation and backlog drainage under continuing arrivals.

post-surge arrival interventions, 2P6D and broader policy sweeps use simulation. SWE-bench Pro supplies a second hardware workload; tool-time and output-length scaling vary Terminal-Bench trajectories, while Bailian varies their arrival process (§7.10).

Instabilities in agent serving. Prior work studies middlephase cache thrashing in offline agent batches [9], collapse under sustained agent load [53], instability from KV growth [4], and feedback between hit-rate collapse and re-prefill [38]. We isolate how continuing session arrivals sustain population–cache feedback after a finite trigger, identify resource conditions for this vulnerability, and show that changing subsequent arrivals alone reverses recovery. The post-surge intervention holds the triggering state fixed, establishing the role of continuing load in sustaining failure.

Conclusion

Population–cache feedback gives PD-disaggregated agent serving a load-dependent recovery boundary. A finite surge can cross this boundary and leave service degraded at a base rate the engine previously sustained. Real-system interventions show that accumulated active sessions sustain the failure, and controlled simulations show that changing only subsequent arrivals changes recovery. The fluid model identifies resource conditions for vulnerability and explains how entry control and departure capacity change the feedback. BASIN applies these dynamics through a fixed session priority, admission by predicted peak context, decode-memory lending and shedding. On real H200 1P3D deployments, the arrival-time prototype achieves 1.20–1.70× Continuum’s goodput and 2.35–3.59× HiCache’s.

Discussion and Limitations

Generality. The feedback follows from closed-loop sessions, continuing arrivals and context-dependent miss cost. The population balance and resource comparison apply to fullattention and hybrid models through architecture-specific hit-rate curves and engine bounds; the common per-token KV size cancels from the memory comparison (Eq. 2). In colocated deployments, prefill and decode share compute and memory; the corresponding model bounds would reflect that shared resource pool.

References [1] A BHYANKAR , R., H E , Z., S RIVATSA , V., Z HANG , H., AND Z HANG , Y. InferCept: Efficient intercept support for augmented large language model inference. In Proceedings of the 41st International Conference on Machine Learning (ICML) (2024), vol. 235 of Proceedings of Machine Learning Research, pp. 81–95.

Capacity changes. BASIN assumes fixed capacity between profiles. Adding [42], switching or losing an instance changes the cliff, with new cache capacity becoming useful only as it warms. Hit-rate feedback adapts the budget between profiles, and reprofiling updates the cliff estimate.

[2] AGARWAL , S., H U , B., M AO , A., A KELLA , A., AND V ENKATARA MAN , S. SYMPHONY: Enabling compute-memory disaggregation in LLM serving systems. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26) (2026).

Dependent sessions. The evaluated trajectories consist of independent sessions. Parent–child workflows couple their lifetimes: a parent can retain context while its child waits for service. When sessions share cached pages, those pages contribute once to their aggregate footprint. These workflows therefore call for dependency-aware ordering, admission and prefix accounting.

[3] AGRAWAL , A., K EDIA , N., PANWAR , A., M OHAN , J., K WATRA , N., G ULAVANI , B. S., T UMANOV, A., AND R AMJEE , R. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 117–134.

Decode-first deployments. The arrival-time hardware prototype achieves goodput similar to Continuum and the composition on the real H100 1P1D surge (§7.2): with decode setting the departure rate, protecting the cache adds little throughput, and ordering by arrival only redistributes waiting time between resident and new sessions. The size-aware priority still helps there. In simulation it reaches 1.20–1.48× the strongest baseline’s goodput on 1P1D (§7.3), because packing smaller sessions first completes more of them within the slowdown SLO under the same decode and cache capacity.

[5] AO , R., L UO , G., S IMCHI -L EVI , D., AND WANG , X. Optimizing LLM inference: Fluid-guided online scheduling with memory constraints, 2025.

[4] AO , R., D ONG , J., L UO , G., AND S IMCHI -L EVI , D. Serviceinduced congestion in memory-constrained LLM serving, 2026. arXiv:2606.15555.

[6] A RUN , S., PARAYIL , A., B HARADWAJ , S., S T. A MANT , R., AND R ÜHLE , V. Towards load-aware prefill deflection for disaggregated LLM serving, 2026. arXiv:2607.02043. [7] B RONSON , N., AGHAYEV, A., C HARAPKO , A., AND Z HU , T. Metastable failures in distributed systems. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS) (2021), pp. 221–227. [8] C ARLEIAL , A. B., AND H ELLMAN , M. E. Bistable behavior of ALOHA-type systems. IEEE Transactions on Communications 23, 4 (1975), 401–410.

Evaluation scope. All hardware experiments serve the hybrid MoE model Qwen3.6-35B-A3B on SGLang. Hardware interventions identify the sustaining feedback, and H100 and H200 runs compare selected operating points. Vulnerability maps, the 36-deployment model comparison,

[9] C HEN , Q., Y E , Z., TANG , T., S UN , P., T IAN , B., WANG , G., L I , S., W EN , Y., H AN , Z., AND Z HANG , T. Concur: Proactive agent-level admission control for efficient agentic batch inference, 2026. arXiv:2601.22705.

[10] C HO , I., S AEED , A., F RIED , J., PARK , S. J., A LIZADEH , M., AND B ELAY, A. Overload control for µs-scale RPCs with Breakwater. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) (2020), pp. 299–314.

[27] H E , W., J IANG , Y., Z HAO , P., X U , Q., YONEKI , E., C UI , B., AND F U , F. Efficient multi-round LLM inference over disaggregated serving. In Proceedings of the 43rd International Conference on Machine Learning (ICML) (2026).

[11] C HO , I., S AEED , A., PARK , S. J., A LIZADEH , M., AND B ELAY, A. Protego: Overload control for applications with unpredictable lock contention. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) (2023), pp. 725–738.

[28] H OERL , A. E., AND K ENNARD , R. W. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12, 1 (1970), 55–67. [29] H U , C., H UANG , H., H U , J., X U , J., C HEN , X., X IE , T., WANG , C., WANG , S., BAO , Y., S UN , N., AND S HAN , Y. MemServe: Context caching for disaggregated LLM serving with elastic memory pool, 2024. arXiv:2406.17565.

[12] C HU , W. W., AND O PDERBECK , H. The page fault frequency replacement algorithm. In Proceedings of the AFIPS Fall Joint Computer Conference, Part I (1972), pp. 597–609. [13] DAI , J. G. On positive Harris recurrence of multiclass queueing networks: A unified approach via fluid limit models. The Annals of Applied Probability 5, 1 (1995), 49–77. [14] D EEP S EEK -AI. arXiv:2412.19437.

DeepSeek-V3

technical

[30] H U , Y., Z HANG , Z., L IU , Y., G U , Y., L EI , S., K ASIKCI , B., AND H UANG , P. Mitigating application resource overload with targeted task cancellation. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP) (2025).

report, 2024.

[31] H UANG , L., M AGNUSSON , M., M URALIKRISHNA , A. B., E STYAK , S., I SAACS , R., AGHAYEV, A., Z HU , T., AND C HARAPKO , A. Metastable failures in the wild. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 73–90.

[15] D ENG , X., DA , J., PAN , E., H E , Y. Y., I DE , C., G ARG , K., L AUFFER , N., PARK , A., PASARI , N., R ANE , C., S AMPATH , K., K RISHNAN , M., K UNDURTHY, S., H ENDRYX , S., WANG , Z., B HARADWAJ , V., H OLM , J., A LURI , R., Z HANG , C. B. C., JACOBSON , N., L IU , B., AND K ENSTLER , B. SWE-Bench Pro: Can AI agents solve longhorizon software engineering tasks?, 2025. arXiv:2509.16941.

[32] I SAACS , R., A LVARO , P., M AJUMDAR , R., M UNISWAMY-R EDDY, K.-K., S ALAMATI , M., AND S OUDJANI , S. Analyzing metastable failures. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS) (2025), pp. 172–178.

[16] D ENNING , P. J. Thrashing: Its causes and prevention. In Proceedings of the AFIPS Fall Joint Computer Conference, Part I (1968), pp. 915– 922. [17] D ENNING , P. J. The working set model for program behavior. Communications of the ACM 11, 5 (1968), 323–333.

[33] JACOBSON , V. Congestion avoidance and control. In Proceedings of the ACM SIGCOMM Symposium on Communications Architectures and Protocols (1988), pp. 314–329.

[18] D ING , J., H OSSEINI , R., G HOLAMI , P. M., X IANG , M., AND H OFFMANN , H. Observation, not prediction: Conversation-level disaggregated scheduling for agentic serving, 2026.

[34] J IANG , Y., F U , F., Y UAN , B., M ALLADI , K., A RDESTANI , E. K., S HU , Z., A ZIZ , A., AND X U , Y. Coda: Exploiting admission flexibility for coding-agent serving, 2026. arXiv:2610.03088.

[19] D ONG , J. Metastability in queues. Queueing Systems 100, 3 (2022), 413–415.

[35] J IMENEZ , C. E., YANG , J., W ETTIG , A., YAO , S., P EI , K., P RESS , O., AND NARASIMHAN , K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR) (2024).

[20] E LNIKETY, S., NAHUM , E., T RACEY, J., AND Z WAENEPOEL , W. A method for transparent admission control and request scheduling in e-commerce web sites. In Proceedings of the 13th International Conference on World Wide Web (WWW) (2004), pp. 276–286.

[36] J IN , Y., W U , C.-F., B ROOKS , D., AND W EI , G.-Y. S3 : Increasing GPU utilization during generative inference for higher throughput. In Advances in Neural Information Processing Systems (NeurIPS) (2023).

[21] F U , Y., X UE , L., H UANG , Y., B RABETE , A.-O., U STIUGOV, D., PATEL , Y., AND M AI , L. ServerlessLLM: Low-latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 135–153.

[37] K ANG , H., L I , Z., X U , W., YANG , X., C HEN , Y., WANG , J., C HEN , B., K RISHNA , T., X U , C., AND A RORA , S. ThunderAgent: A simple, fast and program-aware agentic inference system, 2026. arXiv:2602.13692.

[22] F U , Y., Z HU , S., S U , R., Q IAO , A., S TOICA , I., AND Z HANG , H. Efficient LLM scheduling by learning to rank. In Advances in Neural Information Processing Systems (NeurIPS) (2024).

[38] K IM , J., K WON , M., M IN , Y., K IM , J., AND Z HAO , J. Dynamic HBM repartitioning for multi-turn MoE serving, 2026. arXiv:2609.13537.

[23] G AO , B., H E , Z., S HARMA , P., K ANG , Q., J EVDJIC , D., D ENG , J., YANG , X., Y U , Z., AND Z UO , P. Cost-efficient large language model serving for multi-turn conversations with CachedAttention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24) (2024), pp. 111–126.

[39] KONDRASHOV, L., L IU , H., PARK , J., Z HOU , B., L IU , Z., L U , C., M ANCINI , R., C HOUKSE , E., JAVAID , H., S VIRIDOV, G., P ENG , T., Z HAO , C., AVDEEVA , A., G USEV, A., KOGIAS , M., M AI , L., AND U STIUGOV, D. Rethinking AI cloud infrastructure for agentic serving systems with the Aries experimentation framework, 2026. arXiv:2607.29069.

[24] G IBBENS , R. J., H UNT, P. J., AND K ELLY, F. P. Bistability in communication networks. In Disorder in Physical Systems, G. R. Grimmett and D. J. A. Welsh, Eds. Oxford University Press, 1990, pp. 113–128.

[40] K URTZ , T. G. Solutions of ordinary differential equations as limits of pure jump Markov processes. Journal of Applied Probability 7, 1 (1970), 49–58. [41] K WON , W., L I , Z., Z HUANG , S., S HENG , Y., Z HENG , L., Y U , C. H., G ONZALEZ , J. E., Z HANG , H., AND S TOICA , I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP) (2023), pp. 611–626.

[25] G IM , I., M A , Z., L EE , S.- S ., AND Z HONG , L. Pie: A programmable serving system for emerging LLM applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP) (2025). [26] H ABIBI , F., L ORIDO -B OTRAN , T., S HOWAIL , A., S TURMAN , D. C., AND NAWAB , F. MSF-Model: Queuing-based analysis and prediction of metastable failures in replicated storage systems. In 43rd International Symposium on Reliable Distributed Systems (SRDS) (2024). arXiv:2309.16181.

[42] L AI , R., L IU , H., L U , C., L IU , Z., C AO , S., S HAO , S., Z HANG , Y., M AI , L., AND U STIUGOV, D. TokenScale: Timely and accurate autoscaling for disaggregated LLM serving with token velocity, 2025. arXiv:2512.03416.

[43] L I , H., H E , R., M ANG , Q., Z HANG , Q., M AO , H., C HEN , X., Z HOU , H., Z HANG , H., C HEUNG , A., G ONZALEZ , J., AND S TOICA , I. Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live, 2025. arXiv:2511.02230.

[61] NVIDIA. Full-stack optimizations for agentic inference with NVIDIA Dynamo. NVIDIA Technical Blog, https://developer.nvidia.c om/blog/full-stack-optimizations-for-agentic-inferen ce-with-nvidia-dynamo/, Apr. 2026.

[44] L I , L., WANG , S., RUAN , Z., L I , D., AND G ONG , B. The KV cache working set: Online capacity planning for LLM inference systems, 2026. arXiv:2609.27746.

[62] PAN , R., WANG , Z., J IA , Z., K ARAKUS , C., Z ANCATO , L., DAO , T., WANG , Y., AND N ETRAVALI , R. Marconi: Prefix caching for the era of hybrid LLMs. In Proceedings of Machine Learning and Systems (MLSys) (2025).

[45] L IEBER , O., ET AL . Jamba: A hybrid Transformer-Mamba language model, 2024. arXiv:2403.19887.

[63] PAN , Z., PATEL , A., S HEN , Y., H U , Z., G UAN , Y., L I , W.-L., Q IN , L., WANG , Y., AND D ING , Y. KVFlow: Efficient prefix caching for accelerating LLM-based multi-agent workflows. In Advances in Neural Information Processing Systems 38 (NeurIPS) (2025), pp. 139912– 139931. arXiv:2507.07400.

[46] L IN , C., H AN , Z., Z HANG , C., YANG , Y., YANG , F., C HEN , C., AND Q IU , L. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 929–945. [47] L IU , B., Q IU , H., G OIRI , Í., F ONSECA , R., B IANCHINI , R., AND C HOUKSE , E. Agentic coding in the wild: Characterizing GitHub Copilot traces at production scale, 2026. arXiv:2608.00101.

[64] PARK , J., PARK , J., J UNG , Y., L IM , H., Y EO , H., AND H AN , D. TopFull: An adaptive top-down overload control for SLO-oriented microservices. In Proceedings of the ACM SIGCOMM 2024 Conference (2024).

[48] L IU , Y., C HENG , Y., YAO , J., A N , Y., C HEN , X., F ENG , S., H UANG , Y., S HEN , S., Z HANG , R., D U , K., ET AL . LMCache: An efficient KV cache layer for enterprise-scale LLM inference, 2025.

[65] PATEL , P., C HOUKSE , E., Z HANG , C., S HAH , A., G OIRI , Í., M ALEKI , S., AND B IANCHINI , R. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st ACM/IEEE Annual International Symposium on Computer Architecture (ISCA) (2024), pp. 118–132.

[49] L IU , Y., L I , H., C HENG , Y., R AY, S., H UANG , Y., Z HANG , Q., D U , K., YAO , J., L U , S., A NANTHANARAYANAN , G., ET AL . CacheGen: KV cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference (2024), pp. 38–56.

[66] PATKE , A., R EDDY, D., J HA , S., Q IU , H., P INTO , C., NARAYANASWAMI , C., K ALBARCZYK , Z., AND I YER , R. Queue management for SLO-oriented large language model serving. In Proceedings of the ACM Symposium on Cloud Computing (SoCC) (2024), pp. 18–35.

[50] L IU , Y., Y U , M., AND YANG , J. When fancy eviction fails: Rethinking cache replacement for LLM prefix reuse, 2026. arXiv:2609.28870. [51] LLM - D CONTRIBUTORS. Session-level admission control for agentic workloads (thunderagent plugin). llm-d router issue #2964, https: //github.com/llm-d/llm-d-router/issues/2964, Sept. 2026.

[67] Q IN , R., L I , Z., H E , W., C UI , J., R EN , F., Z HANG , M., W U , Y., Z HENG , W., AND X U , X. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25) (2025), pp. 155–170. Overload-oriented scheduling discussed in the extended version, arXiv:2407.00079, Sec. 7.

[52] L UO , M., S HI , X., C AI , C., Z HANG , T., W ONG , J., WANG , Y., WANG , C., H UANG , Y., C HEN , Z., G ONZALEZ , J. E., AND S TOICA , I. Agentix: An efficient serving engine for LLM agents as general programs. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26) (2026), pp. 2443–2459. Preprint titled “Autellix: An Efficient Serving Engine for LLM Agents as General Programs”, arXiv:2502.13965.

[68] Q WEN T EAM. Qwen3.6-35B-A3B. Hugging Face model card, https: //huggingface.co/Qwen/Qwen3.6-35B-A3B, 2026. [69] S ALTON , G., AND B UCKLEY, C. Term-weighting approaches in automatic text retrieval. Information Processing & Management 24, 5 (1988), 513–523.

[53] M ENG , J., AND L I , B. Metronome: Bound the cache, keep the beat for real-time interaction model serving, 2026. arXiv:2607.02640. [54] M ERRILL , M. A., S HAW, A. G., C ARLINI , N., L I , B., R AJ , H., B ERCOVICH , I., S HI , L., S HIN , J. Y., WALSHE , T., B UCHANAN , E. K., ET AL . Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026.

[70] S CHROEDER , B., W IERMAN , A., AND H ARCHOL -BALTER , M. Open versus closed: A cautionary tale. In 3rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 06) (2006), pp. 239–252.

[55] M EZA , J. J., G OWDA , T., E ID , A., I JAWARE , T., C HERNYSHEV, D., Y U , Y., U DDIN , M. N., DAS , R., NACHIAPPAN , C., T RAN , S., ET AL . Defcon: Preventing overload with graceful feature degradation. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23) (2023).

[71] S HAO , K., C HEN , J., Y U , J., L I , X.-H., TAO , C., WANG , Y., L IN , H., C HENG , K.-T., T SUI , C. Y., AND BAI , H. EfficientAgent: What makes KV cache offloading work for concurrent agents?, 2026. arXiv:2609.33762. [72] S HENG , Y., C AO , S., L I , D., Z HU , B., L I , Z., Z HUO , D., G ONZALEZ , J. E., AND S TOICA , I. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 965–988.

[56] M INI M AX. MiniMax-01: Scaling foundation models with lightning attention, 2025. arXiv:2501.08313. [57] M OGUL , J. C., AND R AMAKRISHNAN , K. K. Eliminating receive livelock in an interrupt-driven kernel. ACM Transactions on Computer Systems 15, 3 (1997), 217–252.

[73] S RIVATSA , V., H E , Z., A BHYANKAR , R., L I , D., AND Z HANG , Y. Preble: Efficient distributed prompt scheduling for LLM serving. In International Conference on Learning Representations (ICLR) (2025).

[58] N I , H., T IAN , H., Z HANG , C., L I , G., AND TAN , H. TOPAS: Workflow-aware prefix-state scheduling for multi-agent LLM serving, 2026.

[74] S TOJKOVIC , J., Z HANG , C., G OIRI , Í., T ORRELLAS , J., AND C HOUKSE , E. DynamoLLM: Designing LLM inference clusters for performance and energy efficiency. In IEEE International Symposium on High Performance Computer Architecture (HPCA) (2025), pp. 1348–1362.

[59] N ISHTALA , R., F UGAL , H., G RIMM , S., K WIATKOWSKI , M., L EE , H., L I , H. C., M C E LROY, R., PALECZNY, M., P EEK , D., S AAB , P., ET AL . Scaling memcache at Facebook. In 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13) (2013), pp. 385–398.

[75] S UN , B., H UANG , Z., Z HAO , H., X IAO , W., Z HANG , X., L I , Y., AND L IN , W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 173–191.

[60] NVIDIA. NVIDIA Dynamo: A datacenter-scale distributed inference serving framework. https://github.com/ai-dynamo/dynamo, 2025.

[76] TAN , X., J IANG , Y., YANG , Y., AND X U , H. Teola: Towards endto-end optimization of LLM-based applications. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Volume 2 (2025), pp. 1302–1316.

[92] Z HANG , D., WANG , H., L IU , Y., W EI , X., S HAN , Y., C HEN , R., AND C HEN , H. BlitzScale: Fast and live large model autoscaling with O(1) host caching. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25) (2025). arXiv:2412.17246.

[77] T HE M OONCAKE T EAM. Deploying Kimi K2 with PD disaggregation and large-scale expert parallelism on 128 H200 GPUs. LMSYS Org blog, https://www.lmsys.org/blog/2025-07-20-k2-large-s cale-ep/, July 2025.

[93] Z HENG , L., Y IN , L., X IE , Z., S UN , C., H UANG , J., Y U , C. H., C AO , S., KOZYRAKIS , C., S TOICA , I., G ONZALEZ , J. E., BARRETT, C., AND S HENG , Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 37 (NeurIPS) (2024), pp. 62557–62583.

[78] T HE SGL ANG T EAM. Deploying DeepSeek with PD disaggregation and large-scale expert parallelism on 96 H100 GPUs. LMSYS Org blog, https://www.lmsys.org/blog/2025-05-05-large-scale-ep/, May 2025.

[94] Z HENG , Z., R EN , X., X UE , F., L UO , Y., J IANG , X., AND YOU , Y. Response length perception and sequence scheduling: An LLM-empowered LLM inference pipeline. In Advances in Neural Information Processing Systems (NeurIPS) (2023).

[79] VATTANI , A., C HIERICHETTI , F., AND L OWENSTEIN , K. Optimal probabilistic cache stampede prevention. Proceedings of the VLDB Endowment 8, 8 (2015), 886–897.

[95] Z HONG , Y., L IU , S., C HEN , J., H U , J., Z HU , Y., L IU , X., J IN , X., AND Z HANG , H. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 193–210.

[80] WANG , J., H AN , J., W EI , X., S HEN , S., Z HANG , D., FANG , C., C HEN , R., Y U , W., AND C HEN , H. KVCache cache in the wild: Characterizing and optimizing KVCache cache at a large cloud provider. In 2025 USENIX Annual Technical Conference (USENIX ATC 25) (2025), pp. 465–482.

[96] Z HOU , H., C HEN , M., L IN , Q., WANG , Y., S HE , X., L IU , S., G U , R., O OI , B. C., AND YANG , J. Overload control for scaling WeChat microservices. In Proceedings of the ACM Symposium on Cloud Computing (SoCC) (2018), pp. 149–161.

[81] W ELSH , M., C ULLER , D., AND B REWER , E. SEDA: An architecture for well-conditioned, scalable internet services. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP) (2001), pp. 230–243. [82] W U , B., Z HONG , Y., Z HANG , Z., L IU , S., L IU , F., S UN , Y., H UANG , G., L IU , X., AND J IN , X. Fast distributed inference serving for large language models, 2023. [83] W U , Y., L IU , T., Z ENG , Y., W U , S., X IONG , J., D ONG , X., YANG , H., Z HANG , K., AND L I , J. Arrow: Adaptive scheduling mechanisms for disaggregated LLM inference architecture, 2025. arXiv:2505.11916. [84] X IE , Z. SGLang HiCache: Fast hierarchical KV caching with your favorite storage backends. LMSYS Org blog, https://www.lmsys. org/blog/2025-09-10-sglang-hicache/, Sept. 2025. [85] X U , Y., A RDESTANI , E. K., F U , W., S CHATZ , M., M ALLADI , K., S HU , Z., A ZIZ , A., K ANAUJIA , S., M ATHEWS , A., AND TANG , C. Crossflow: Prefill-decode elasticity for agentic LLM serving, 2026. arXiv:2609.27085. [86] YANG , J., J IMENEZ , C. E., W ETTIG , A., L IERET, K., YAO , S., NARASIMHAN , K., AND P RESS , O. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS) (2024). [87] YAO , H., G AO , C., LYU , Y., G ROSOF, I., AND S IMCHI -L EVI , D. Multi-turn LLM conversations under the least-recently-used policy: Mean-field asymptotics and hit ratio approximation, 2026. arXiv:2609.02027. [88] YAO , J., L I , H., L IU , Y., R AY, S., C HENG , Y., Z HANG , Q., D U , K., L U , S., AND J IANG , J. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys) (2025). [89] YAO , S., Z HAO , J., Y U , D., D U , N., S HAFRAN , I., NARASIMHAN , K., AND C AO , Y. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR) (2023). [90] Y U , G.-I., J EONG , J. S., K IM , G.-W., K IM , S., AND C HUN , B.-G. Orca: A distributed serving system for Transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 521–538. [91] Y U , L., L IN , J., AND L I , J. Stateful large language model serving with Pensieve. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys) (2025), pp. 144–158.

When Cache Pressure Becomes a Trap: Metastability in Agent Serving

Analyze & explore related research in Conceptio

Related documents