Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributedcomputing parallelcomputing

distributed computing, parallel computing, cloud

Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs Utopia Meng

Unicornt Zhao

Derek Li

Goalen Gao

Frank Du

[email protected]

arXiv:2607.17181v1 [cs.DC] 19 Jul 2026

Abstract

paServe, MuxServe, and Aegaeon multiplex or fast-switch models on pooled resources [5, 7, 8]. These techniques make model residency a first-class concern, but their placement and scheduling abstractions do not explicitly preserve continuity across calls in an agent session. Large models intensify this gap: when the aggregate working set exceeds HBM capacity, a placement miss can trigger weight loading together with KV restoration or prefix recomputation. Tool-using agents expose why a request alone is an insufficient scheduling unit. A user task alternates between model calls and tool execution, carrying state from one call to the next [9–12]. Across 445 SWE-Bench Verified sessions [13] (7,386 calls), median session depth is 11 model calls (p95: 49). Prompt-length measurements (n = 5,697) have a median of 25K tokens (p95: 105K), and 96.8% of measured continuations (n = 5,270) reuse more than half of the prior call’s prefix. The median tool-use gap is only 0.39 s—over an order of magnitude shorter than the 4.3 s needed to recompute a median-length prefix on Qwen3-235B/TP=8. These timescales create a continuity window: continuations return quickly, so preserving their large, mostly reusable prefixes can avoid seconds of recomputation. Treating each return as a fresh request repeatedly incurs avoidable state-recovery cost across a session. Per-call scheduling therefore misaligns with session completion time along two coupled dimensions. Spatial mismatch: a router using only load or queue length may send a continuation to the least-loaded instance, which may hold neither the target weights nor the session KV; for long-context sessions, KV movement or prefix recomputation can exceed the queueing delay it avoids. Temporal mismatch: even when routing is correct, a multi-model scheduler that rotates through models in fixed-length rounds [5] may defer a returning session until the next slot for that model—KV locality is preserved spatially but squandered on the time axis. Under prefill/decode disaggregation, the same locality problem spans tiers: the returning session enters through a prefill path, while reusable KV is valuable on the decode side. These failure modes compound. KV-aware routing can preserve the right state yet still leave a continuation waiting until the target model’s next slot; mid-slot admission can exploit

Serverless multi-model LLM systems multiplex popularityskewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model’s next slot. Both failures are especially costly for hundred-billionparameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multimodel serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance’s admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench modelsessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3× and 2.6×.

Introduction

Modern LLM serving platforms host popularity-skewed model catalogs on shared GPU clusters: a few hot models carry stable traffic, while a long tail is invoked only intermittently [1–5]. Statically reserving GPUs for every model wastes accelerator capacity on the tail, motivating serverless multi-model serving: demand-driven opening, closing, and time-sharing of models over pooled GPUs. ServerlessLLM optimizes cluster placement and cold-start loading [6]; Al1

Cluster layer

Router

Instance 1

Instance layer

turn n out

Instance 3

Instance 2

Model A

KV Cache

slot

Model C

turn n+1 in

Instance 4 SP

slot

Model D (activate)

Model E

KV Cache

Model F (activate)

KV Cache

KV migration mooncake backend

Hot Pool

Cold Pool

stable hot models

multi-model slots + SP

dynamic placement

Memory substrate

Multi-model Memory Substrate (HKVR · HBM Allocator · D2D Accelerate)

request

telemetry updates

KV migration

soft reservation

kv cache hit

Figure 1: Talaria architecture. The router preserves return affinity across hot and cold pools; SP admits eligible mid-slot returns, while each cold-pool instance preserves restorable KV across model switches. preserved state only when routing has placed the continuation on the right instance. The cluster layer must land a return on a useful instance, and the instance layer must expose a timely execution opportunity.

cause policies change completion times. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria’s instance-local mechanisms cut p50 session completion time from 1000 s to 189 s and p95 from 2296 s to 867 s—speedups of 5.3× and 2.6×. On a separate two-worker testbed, residency-aware routing reduces high-load avoidable model opens from 37 to 1 and TTFT p95 from 8.07 s to 5.28 s versus least-pressure routing. This paper makes three contributions.

We present Talaria, a serverless multi-model LLM inference system for agentic workloads (Figure 1). Talaria realizes session continuity through coordinated placement, admission, and state restoration. At deployment time, it configures a hot pool of pinned single-model instances and a cold pool whose instances token-level time-share a set of long-tail models. A session-aware router uses soft reservations to account for likely returns in the serving instance’s next eligible model-slot budget (§3.3); calls without a usable reservation are placed by ranking candidate instances according to model residency, KV residency, and pressure. Within each cold-pool slot, colocated prefill/decode execution enables session-prefill (SP) to admit eligible returns mid-slot, reusing device-resident KV or restoring host-restorable KV while the model remains active. A per-instance memory substrate coordinates HBM allocation, host-side KV state, and weight staging to reduce the switching and restoration costs charged to each round.

• Session continuity as a scheduling abstraction. We identify session continuity as the missing unit in serverless multi-model serving for agents. From 445 agent sessions, we quantify the depth, return timing, and reusable state that make request-level scheduling costly, then isolate spatial placement and temporal admission with controlled experiments. • Joint session-continuity scheduling. At the cluster layer, soft reservation accounts for likely returns without pinning device KV; at the instance layer, SP provides budgeted mid-slot admission. The two mechanisms address complementary failure modes along the same return path.

On a single TP=8 server, we replay ten SWE-Bench issues across Qwen3-235B, GLM5-nvfp4, and Qwen3.5-122BA10B, yielding 30 model-sessions and 960 calls. For each model-session, fixed replay holds request bodies, model and session IDs, per-session call order, completion-token counts, and return gaps constant across configurations. This prevents agent-path divergence; global interleaving may still differ be-

• Switch-resilient instance substrate. A stable HBM layout and host-restorable KV preserve usable session state across model switches; opportunistic D2D staging reduces the switching cost charged to each round. 2

CDF (%)

100

(a) Session depth

(b) Inter-call gap

p50=11

p50=0.39s

therefore dominate the latency of an entire session turn. Taken together, these properties mean that agent serving is not scheduling a stream of independent requests. It is scheduling a stateful session whose KV state, target model, and return timing persist across calls and must be tracked explicitly by the serving system.

75 50

p95=49

25 0

p50=25K

p95=1.3s

100 0

LLM calls per session

p95=105K

Tool gap (s)

100

150

200

Prompt tokens (K)

SWE-Bench Verified · 445 sessions, 7,386 calls · prompt n=5,697 · prefix n=5,270

2.2 Spatial mismatch: routing to the wrong instance

Figure 2: SWE-Bench Verified sessions are deep (median 11 calls), fast-returning (median gap 0.39 s), and long-context (median 25K tokens). Axes truncate at 100 calls, 5 s, and 200K tokens; the gap CDF excludes five >30 s environmentstartup intervals.

Current serverless routers schedule at request granularity, commonly using load, queue length, or model-load cost as the routing signal [5, 6]. This is a poor proxy for a returning agent call because the fastest instance for the cluster may be the slowest instance for that session. Consider two instances in the same cold pool. Instance A is currently serving the target model and still holds the session’s reusable KV prefix, but has a short queue. Instance B is idle, but may hold neither the target model nor the session KV. A least-load or round-robin router sends the returning call to B because its queue is shorter. For this session, B must first make the target model resident if needed, then reconstruct the prefix. The recomputation alone costs 4.3 s TTFT on Qwen3235B/TP=8. Instance A may incur queueing or wait until the target model’s next slot, whereas B must additionally activate the model and recover the prefix. Because recovery cost grows with context length, a shorter queue on B does not imply a faster return. A load-only router cannot see this asymmetry. KV-aware routing [14–16] narrows the gap, but it is still incomplete for multi-model serving. Knowing where the session KV lives is not enough if the router cannot also tell whether the target model is resident and whether the instance can accept a prefill soon. Good placement therefore requires joint visibility into model residency, KV residency, and instance pressure. This visibility must also be session-scoped: the router must remember which instance just served a session and treat the next call as a likely continuation, not as an unrelated arrival.

Background and Motivation

We first characterize agent sessions in a real agent trace (§2.1), then show how request-level placement loses spatial locality (§2.2) and how round-based multi-model scheduling loses temporal locality (§2.3). We close with the memorymanagement requirements that make session-aware scheduling implementable (§2.4).

2.1

Agent sessions are stateful

We define an agent session as a single user task: the agent issues a sequence of model calls interleaved with tool invocations until the task completes. This request pattern follows the agent execution model introduced by tool-use and softwareengineering agents [9–11]. We characterize 445 completed agent executions of SWE-Bench Verified tasks [13], totaling 7,386 LLM calls. Prompt length is available for 5,697 calls and reusable-prefix overlap for 5,270 continuations. We combine these traces with measured Qwen3-235B and GLM5nvfp4 profiles to quantify recovery cost. Figure 2 summarizes session depth, tool gaps, and prompt lengths. Session depth. The median session issues 11 model calls, with a p95 of 49 and a maximum of 187. A request-level scheduler therefore sees one user task as many unrelated arrivals, hiding the prefix reuse and timing dependence between consecutive calls. Short inter-call gaps. The median tool-use gap is only 0.39 s, with a p95 of 1.32 s. This is more than ten times shorter than recomputing the median-length prefix, as measured below. A return therefore often arrives while its prior state remains worth preserving, whether it is still in HBM or restorable from host memory. Continuation inputs. Prompts are long and highly repetitive: the median call has 25K prompt tokens, the p95 reaches 105K, and 96.8% of measured continuations reuse more than half of the previous call’s prefix. Recomputing a 25K-token prefix on Qwen3-235B/TP=8 costs 4.3 s TTFT at p50; at 100K tokens, it reaches 10.9 s. A single misplaced return can

2.3 Temporal mismatch: admitting at the wrong time Correct placement is not sufficient if the instance scheduler exposes no time window in which the returning session can run. A cold-pool instance that time-shares models in fixed rounds may preserve the session KV in space while failing to use it in time. Consider a round-based token-level scheduler such as Aegaeon [5]. The decode side commits to one active model for a slot. After a session finishes a decode step and enters a tool call, the scheduler may move on to another model. When the tool returns 0.39 s later, the session’s KV can still be resident in HBM, but its model window may have closed. 3

The session then waits for that model’s next slot—up to one round—before it can resume. Systems that disaggregate prefill and decode [17–19] expose an analogous cross-tier mismatch: the return enters through a prefill path while its reusable state is valuable on the decode side. Talaria focuses on co-located prefill/decode; extending its admission contract across tiers requires coordinated KV ownership (§7). The failure is temporal rather than spatial: the state may be in the right place, but the scheduler cannot admit the session while it is still useful. Request-level autoscaling does not solve this problem because it requeues the returning session as a fresh request, losing the continuity established by the router. What is needed is mid-slot admission: a returning session should be able to re-enter the current model window and reuse preserved KV before the scheduler rotates away.

2.4

slot admission to matter. The memory layer must therefore manage device memory, host KV state, and weight movement as one substrate rather than as independent mechanisms.

3 3.1

Design System Overview

At a high level, Talaria separates multi-model agent serving into a control plane and a data plane. The control plane—a unified router—decides where each session call should land. The data plane—a set of hot and cold inference instances— executes those calls while maintaining the local budget and telemetry contracts that the router’s decisions assume. The router creates a soft reservation when a completed call enters tool execution. It records the serving instance, model, and reusable-prefix handle and charges that instance’s admission budget for a likely return. If the session returns before the lease expires and the target remains healthy, the router sends it back to that instance, where SP can admit it while the KV is device-resident or host-restorable. Requests without a usable reservation take the normal cost path, which weighs instance pressure, KV recovery, and model activation. The router never touches token-level execution.

Multi-model memory management

The router and scheduler above assume that model and KV state can be moved, kept valid, and restored within a bounded time. Existing LLM serving systems already show that KV layout, paging, and virtual memory are central to serving performance [20–22]. In large multi-model serving, the requirement is tighter: freeing HBM for one model can invalidate reusable KV, while slow weight movement consumes the same round budget that mid-slot admission relies on. Memory management becomes part of the scheduling substrate. HBM capacity. Weights are only part of a model’s device footprint. Each model also brings runtime structures such as CUDA graphs, FlashInfer workspaces, and KV pool regions. If each model allocates these regions independently, the footprints accumulate even when request load is low, leaving too little HBM for useful KV or for staging the next model. Host-side KV consistency. Cold-pool switching often requires session KV to leave HBM and later return. The system must know which model owns each KV block, whether the block is still valid, and whether the current copy is on device or host. Without this metadata, the serving stack can restore stale KV and produce incorrect output, or miss a valid host-side copy and recompute a long prefix unnecessarily. Switch latency. Model switching directly consumes the round budget that the scheduler relies on for TTFT and TPOT. A large-model switch may include weight loading, kernel reinitialization, and KV-pool reorganization, each of which can be second-scale. In our fixed-replay measurements, a logical TP=8 H2D-only switch has a 1.50 s p50 and a 1.52 s mean (§5.6). This consumes a substantial fraction of a 10 s TTFT budget. If switching occupies a large fraction of the round, time-sharing stops being a multiplexing benefit and becomes a throughput tax. These constraints are coupled. Bounded switching requires spare HBM for staging; KV restoration requires host-side consistency; and both must complete quickly enough for mid-

Instances are split into a hot pool and a cold pool. Hotpool instances obey a single invariant: each pins one model and never switches, eliminating switching tail latency for stable traffic. Cold-pool instances time-share multiple long-tail models using round-based scheduling. Each model slot has the structure P → D1 → (SP → D)∗ → Dfin : queued prefill, initial decode, zero or more interleaved SP (session-prefill) / decode pairs, and final decode. Each SP step batch-admits eligible returns that fit the remaining prefill budget, enabling preserved-prefix reuse without a model switch; excess returns remain queued for a later opportunity. Beneath the instance scheduler, a multi-model memory substrate manages the HBM and host memory of each inference instance, addressing the layout, host-KV, and switch-latency constraints identified in §2.4. It provides a stable HBM layout, persists session KV across model switches, and reduces switch cost through weight staging. The router observes this substrate through telemetry such as memory pressure and staging slack; the instance scheduler uses it to restore KV and switch models within the round budget. One ownership boundary holds across the design: the router reasons about cluster placement but never executes tokens, while each instance enforces local execution and memory budgets without reconstructing cluster-wide demand. Pinned hot instances avoid the switch path; cold instances expose the telemetry and actuation needed for multiplexing. 4

model call Timeline

Model Call 1 (Instance A)

tool execution

is near the p90 tool gap in our traces. The reservation is an admission lease, not a GPU-KV pin or an execution guarantee: while live, it charges one configured admission unit against the serving instance’s next eligible model-slot budget. When the next call for the same session and model arrives, the router checks that the lease is live, the instance is healthy, and the prefix remains device-resident or host-restorable. A successful lookup consumes the lease and routes the call to that instance. A successor lease is installed only after this call completes; routing does not refresh the old lease. If any check fails, the router discards the lease and falls back to normal residency-aware placement. The lease reduces the chance that unrelated arrivals consume all predicted admission capacity during a tool gap, but it cannot guarantee same-slot admission or a TTFT deadline: the next prompt suffix and concurrent returns are unknown when the lease is created. KV may also move from device to the host registry under memory pressure. Reservation therefore preserves bounded affinity and accounts for likely demand; it does not guarantee device residency. Longer leases preserve more affinity but hold budget longer and can reduce model residency hit rate; §5.4 measures this tradeoff.

state recovery

session returns tool execution

Model Call 2 (Instance A, KV hit)

soft reservation τ

(a) With soft reservation router sees A as loaded → routes to Instance B Model Call 1 (Instance A)

tool execution

state recovery (KV migration / recompute)

Model Call 2 (Instance B)

(b) Without soft reservation

Figure 3: Soft reservation preserves return affinity to an instance with device-resident or host-restorable state; it does not pin device KV.

3.2

Configured Hot-Cold Pool Organization

Hot-pool instances pin a single model and never switch; this eliminates switching tail latency on stable, high-throughput traffic. Cold-pool instances time-share long-tail models through round-based scheduling (§3.4). Our prototype configures pool membership at deployment time. Within the cold pool, the router prefers instances that already hold the requested model and session prefix, subject to admission and HBM constraints. This concentrates demand on useful state and avoids needless model opens; if the preferred instance fails admission, cost ranking selects another feasible cold instance or invokes the configured overload policy. These are per-call placement decisions and do not reclassify instances. This configured split separates stable high-demand service from the multi-model tail while keeping the continuity mechanisms independent of a particular pool-sizing controller. The telemetry used for placement could also drive promotion and demotion, but dynamic pool reallocation is outside the current prototype and evaluation.

3.3

Placement for non-reserved requests. Requests without an active reservation—new sessions and sessions whose reservation has expired—are placed by a cost-ranking policy. The router scores each candidate instance i by req Cost(r, i) = ∆Rbi (r) + CbiKV (r) + Cbiact (mr ),

(1)

where all terms are in latency units. The three terms charge incremental queue/prefill/decode work, KV recovery, and model req activation exactly once. ∆Rbi contains only the incremental queue, prefill, and decode work introduced by r. State recovery and model activation are excluded from this term and charged once by the remaining terms. CbiKV is zero for a device-resident prefix, the measured H2D restore cost for a host-restorable prefix, and the profiled recomputation cost otherwise. Cbiact is zero when mr is active, the advertised switch cost when its state is resident or staged, and the measured fullopen cost otherwise. Partial staging changes the activation estimate rather than adding a second model-open penalty. The router refreshes its per-model latency profiles from recent round summaries using an exponentially weighted moving average. The in-flight overlay in §4.1 adds calls and reservations forwarded since the latest snapshot. For each candidate, the router simulates the resulting round load and HBM use, retaining it only if both configured admission checks pass. These are predictive overload controls rather than hard TTFT guarantees under stale telemetry. Miscalibration affects placement quality, while deadline-risk telemetry makes an underestimated instance less attractive in subsequent decisions. KVi (s) is valid when instance i reports session s’s prefix as either device-resident or host-restorable through HKVR with

Session-Aware Residency Routing

The router assigns each session call to an inference instance with two objectives: preserve KV locality from the previous call, and respect model and memory capacity constraints. The two objectives have different time horizons. A session prefix is reusable when a model call completes, but it may move to host or be evicted while the session executes a tool. Preserving or restoring that locality is therefore time-sensitive. Capacity constraints depend on model residency and aggregate round pressure, which evolve over scheduling windows rather than a single routing decision. Soft reservation for returning sessions. The router creates a soft reservation only after a model call completes. From the completion metadata, it records the session id, model, serving instance, reusable-prefix handle, and expiry time texp = tdone + τ. We choose τ = 1 s from the sensitivity sweep in §5.4; it 5

Algorithm 1 Session-aware residency routing.

Require: Request r for session sr , model mr ; instance snapshots and gateway queue/reservation overlay 1: // Consume a lease created by the previous completion 2: if sr has live lease (sr , mr , h, k,texp ) then 3: if h is healthy and KVh (sr ) is valid then 4: consume lease; route r to h 5: return 6: else 7: discard lease 8: end if 9: end if 10: // Cost-based placement 11: C ← 0/ 12: for all instances i do 13: simulate request, KV, activation, round, and HBM costs

Timeline

TTFT

SP TTFT

Model A

Model B

slot boundary

P switch

TTFT

(a) With SP admission.

tool call returns P

Decode

tool call returns

next round

switch

TTFT waits full round

P switch

(b) Without SP (round-level scheduler).

Figure 4: Cold-pool slot structure. SP opens mid-slot prefill windows for eligible continuations before the scheduler rotates away.

14: if round-admission and HBM checks pass then 15: if i needs no fresh replica or Eq. (2) passes then 16: C ← C ∪ {i} 17: end if 18: end if 19: end for

useful instances, while the feasibility filter redirects overflow to nodes with sufficient round budget and staging slack (remaining capacity for D2D pre-staging, defined in §3.5).

3.4 Cold-Pool Scheduler with Session-Prefill (SP) Admission

20: if C = 0/ then 21: invoke the configured gateway overload policy 22: return 23: end if 24: i∗ ← arg mini∈C Cost(r, i) 25: route r to i∗

Cold-pool instances must share a GPU group across multiple models while still serving agent sessions with short tool gaps. The evaluated design co-locates prefill and decode on the same GPU group, avoiding a cross-tier KV handoff on each turn. A round-based multi-model scheduler then faces a second problem: a session returning from a tool call after its model’s slot has closed must wait until the next full rotation even when its prefix remains reusable. The SP stage resolves both: by staying non-P/D-disaggregated it avoids per-turn KV handoff, and by opening a mid-slot admission window it captures eligible returns before the active model slot closes. Talaria implements a non-P/D-disaggregated token-level scheduler for the cold pool. A cold-pool instance serves several models on the same tensor-parallel group, executing only one model at any instant. Time is organized into rounds; each active model receives one slot per round, and the instance switches models only at slot boundaries. A slot begins with a queued-prefill phase P, which admits ordinary work already waiting when the slot opens, and an initial decode phase D1 . Zero or more session-prefill/decode pairs may follow before a final decode phase: ∗ P → D1 → SP → D → Dfin .

26: return

an agreed prefix. A new model replica is opened on instance j only when the predicted demand benefit exceeds the opening cost: gain(m, j) > Tswap (m, j) + µ j ∆Wm + νm ,

Prefill

(2)

where gain(m, j) is the latency-equivalent benefit of opening the replica: the predicted reduction in queued or spillover demand for model m over the next control window, computed from the router’s observed arrivals and reserved returns. Tswap is the switch latency, ∆Wm is the added HBM footprint, µ j is the instance-specific HBM pressure price, and νm is a perreplica penalty against unjustified expansion. µ j has units of latency per GiB and converts added HBM footprint into a latency-equivalent cost using instance j’s current memory pressure and staging slack. Algorithm 1 handles call arrival only. The completion hook in §4.1 installs a lease for a possible subsequent call. The lease accounts for a likely return and preserves instance affinity; SP provides a budgeted opportunity to consume that return while the model slot remains open. Neither mechanism alone guarantees same-slot admission. Within the configured cold pool, model-residency preference concentrates demand on

An SP opportunity occurs only at a decode boundary; it does not preempt an executing kernel. A call is SP-eligible when it targets the active model, is a continuation that arrived after P, and has a device-resident or host-restorable prefix on this instance. At the k-th opportunity, the scheduler forms the eligible set Em,k and selects a bounded batch Bm,k ⊆ Em,k 6

whose predicted prefill, restore, and HBM costs fit the budget remaining after protected decode. Calls that do not fit remain queued for a later SP opportunity or the next model slot. After the batch completes, decode resumes for all in-flight requests. The final decode phase Dfin is the drain interval after the last SP opportunity in the slot: the scheduler stops admitting new prefill but spends the remaining protected budget on inflight decode before switching models. New sessions do not bypass P through SP. The slot closes when the round budget is exhausted or the scheduler decides to switch models. A round on instance i is therefore Ri = ∑ Pm + D1,m + ∑k (SPm,k + Dm,k ) + Dm,fin

remaining slot budget. The key departure from a standard round-based scheduler is the SP step: whenever an eligible continuation returns during a slot, it is queued for the next SP opportunity rather than immediately deferred until the next slot for that model. At that opportunity, the scheduler admits only the bounded subset that fits the remaining budget; it never delays protected decode to admit an unbounded return batch. Decode then resumes for all in-flight sessions, including newly admitted continuations. This interleaving can repeat multiple times within a slot as successive tool calls complete, preserving useful locality and avoiding an unnecessary next-slot wait. A continuation that arrives after the slot closes, or does not fit before the final decode phase, waits for the next slot of the same model. Best-effort prefill does not mean silently deferring work. When the protected decode budget leaves insufficient prefill time, or when cumulative pressure pushes the round past the TTFT bound, the instance reports the shortfall through structured telemetry—staging slack (remaining capacity for D2D pre-staging, detailed in §3.5) and deadline-risk counters— rather than hiding overload by rolling work into later rounds. The router reacts by spilling traffic to another instance, opening a replica, or applying the configured admission-control policy.

m∈Miactive

∑ Sm→m′ ,

(3)

(m→m′ )

where SPm,k = Tbp f (Bm,k ) + Tbrestore (Bm,k ) is the batched prefill and restore cost at the k-th opportunity, and Sm→m′ is the switch cost exposed by the memory substrate (§3.5). The switch estimate is updated from staging coverage and dirtychunk telemetry rather than treated as a static model constant. Decode-share budget. The round scheduler controls average token cadence; it does not claim a hard deadline for every inter-token gap. Let dm be the number of decode iterations reserved for model m in a predicted round, and let tmD be its profiled decode-step time. Define

3.5

The router and cold-pool scheduler decide where and when model weights and session KV should reside. They rely on a memory substrate that makes these placement decisions executable within the round budget. The substrate exposes three capabilities to the rest of Talaria. Stable HBM layout. Cold-pool instances switch among models whose weights, KV pools, runtime buffers, and graph state all compete for HBM. The substrate gives each instance a stable multi-model layout: structural regions can be released, reused, and rebound across switches without fragmenting device memory or invalidating the runtime state needed to serve the next slot. Persistent session KV. Switching away from a model may reclaim its device KV region, but a returning session still needs the prefix it produced before the tool call. The substrate therefore maintains a host-side KV registry that stores valid session prefixes across model switches and restores them to device memory when the session returns. The registry is part of the correctness boundary: it must not expose a prefix unless all tensor-parallel ranks agree on the same valid blocks. Reduced switch cost. Model switching consumes the same round budget that SP admission depends on. The substrate stages future model weights while the current model is serving, then uses device-to-device movement for the staged portion at switch time and falls back to host-to-device loading only for uncovered chunks. The router sees the remaining staging capacity as staging slack; when that slack is low, the

Um = Ri − dmtmD as all predicted time in the round not spent on m’s decode iterations, including its own P/SP phases, other model slots, and switches. For a sequence that remains active throughout the round, the predicted average time per token is Ri /dm . To keep this average below T TPOT , the scheduler requires dm T TPOT − tmD ≥ Um . (4) If T TPOT ≤ tmD , the target is infeasible for that model profile. The scheduler reserves the corresponding dm decode iterations before assigning the remaining slot budget to P and SP. Intuitively, protected decode establishes the target average token cadence; prefill and SP share only the time left after that reservation. The round itself is bounded by the TTFT budget with a safety margin: Ri ≤ γTTTFT ,

γ < 1.

Multi-Model Memory Substrate

(5)

This round-length constraint bounds predicted model revisit time; it is not by itself a hard TTFT guarantee because queueing across rounds and estimation error can still violate the target. Prefill arrivals and prompt lengths are unknown when a round begins, so prefill is treated as best-effort within the 7

Self-managed vram

instance becomes a worse target for additional traffic. Staging slack is therefore both a memory signal and a switch-cost signal: lower slack predicts a larger uncovered H2D fraction on the next switch. These capabilities form the contract between scheduling and implementation: the instance reports measured restore and switch estimates, and the router and SP scheduler charge those estimates to their admission budgets. Section 4 describes how Talaria implements the lease and memory substrate.

Bump model region

kv cache region

Pytorch runtime region

TMS / VMM CUDA graph pool

other

token-to-slot table pytorch inductor

gather kernel D2D

activation

layer first kv cache buffer

NCCL buffer

L0-k

chunk

L1-k

chunk

CUDA runtime

chunks can be preempted (valid D2D, dirty H2D)

Implementation

L0-v

chunk

L1-v

chunk

fallback H2D

We implement Talaria on SGLang [23] as a stateful Go gateway (5.5K lines) and an extended inference engine (15K lines) that adds session caching, HKVR, SP scheduling, and staged switching. We additionally adapt 1.4K lines of SGLang’s existing TMS support for per-model memory reclamation. The gateway maintains session leases and routing telemetry; each engine instance implements the instance-local cache, scheduling, and memory contracts.

layer head first

flashinfer workspace decode input buffer

layer tail first

dispatch kernel H2D

Host model weight

Switch accelerate

Figure 5: Bump-managed HBM layout. Stable regions preserve pointers across switches; clean staged chunks move by D2D and dirty chunks reload from host.

finishes, the wrapper saves the request-pool slot, committed KV length, radix lock state, and any architecture-specific cache state. On the next turn, it restores that slot before prefix matching, so the existing scheduler sees a prefix hit without a separate token path. The same finish path checkpoints aligned KV blocks to HKVR when host KV is enabled. Instance telemetry reports the fields the gateway needs for routing: loaded models, cached sessions, model queues, round remaining time, deadline risk, KV occupancy, and free staging bytes.

4.1 Router State and Session Cache Integration The gateway extracts a stable session id from the agent runtime and keeps a session table keyed by that id. Each entry records the last serving instance, model, observed KV length, last-use time, and a soft-reservation lease. After a request finishes, the gateway updates the entry from response usage metadata and installs a lease on the serving instance. Before routing the next call, it overlays its local in-flight queue on top of the latest instance snapshot, so recent forwards and reservations affect admission immediately instead of waiting for the next telemetry poll. Instance snapshots are versioned and refreshed on the control-plane heartbeat; if a snapshot is stale or the instance misses a heartbeat, the gateway can still honor live reservations only when the instance passes the health check, but it excludes that instance from new non-reserved placements until fresh telemetry arrives. The lease is represented as (s, m, i, k,texp ): session s, model m, reserved instance i, prefix handle k, and expiry time texp . A live lease contributes one configured admission-unit charge to the gateway’s local load overlay. On a matching return, the gateway consumes the lease before forwarding the call to i; the completion hook may later install a successor lease. If the lease expires, the instance becomes unhealthy, or the prefix is no longer restorable, the call falls back to normal placement. Because the next suffix length is unknown at reservation time, this charge is capacity accounting rather than a hard latency guarantee. Each SGLang instance makes the prefix handle executable through a session-aware prefix-cache wrapper. When a turn

4.2

Bump-Managed HBM Layout

A cold-pool instance switches among models whose weights, KV tensors, CUDA graphs, runtime workspaces, and bookkeeping state all occupy HBM. Rebuilding these objects on every activation would recapture CUDA graphs, reinitialize attention backends, rebuild KV-pool shells, and fragment PyTorch’s caching allocator. Talaria therefore removes the large structural regions from the caching allocator and places them in one managed HBM buffer. For PyTorch-owned per-model objects, it uses SGLang’s existing torch_memory_saver (TMS) integration, a VMM-based physical-page manager that preserves virtual addresses while reclaiming physical pages. Talaria adapts that mechanism with per-model tags and switch-time pause/resume calls, so CUDA-graph and runtime allocations can be parked with the model that owns them. TMS is a memory mechanism below the scheduler, not the scheduling abstraction itself. The bump buffer uses a fixed layout: weights and KV data grow upward from the low-address end, runtime buffers are anchored at the high-address end, and the gap between 8

them forms the KV pool. This layout lets buffer-like regions be shared across models without static per-model partitions. TMS covers the PyTorch-managed objects that cannot simply be rebound into the bump buffer, including CUDA-graph pools and per-model runtime metadata. When a model is parked, TMS releases their physical pages while preserving virtual addresses; on restore, the same addresses are remapped, so captured kernel pointers remain valid. Switching from model m to m′ then reduces to releasing the active kv_cache region, resizing the weights region for m′ , and rebinding m′ ’s parameter tensors to views in the bump buffer. The engine caches each model’s CUDA graph, attention backend, and KV-pool shell, and recaptures only if a restore-time pointer check detects an address change. The check compares the base addresses of the weight, KV, graphpool, and runtime buffer regions. Once the bump layout is stable, the common switch path reuses captured kernels directly and avoids cudaFree/cudaMalloc churn. The stable addresses keep cached CUDA graphs and runtime objects valid across model switches.

4.3

the agreement. Third, every model switch drains outstanding D2H events before freeing the active kv_cache region. Since D2H is issued as turns finish rather than batched at switch time, this barrier usually waits only for the final tail blocks.

4.4

D2D-Staged Switching

Reloading a multi-hundred-GB model from host at the switch boundary would spend the same round budget that SP relies on. Talaria instead pre-stages the next model’s weights into unused HBM while the current model is serving, then uses D2D movement for the staged fraction at switch time. The staging surface is the high-address tail of each KV layer block; the KV allocator reuses low addresses first, keeping those tails clean under normal allocation pressure. Staging is opportunistic and correctness-preserving. The staging manager writes weights into fixed-size chunks and marks a chunk dirty if a concurrent prefill allocation takes rows that overlap the chunk’s staging range. During an active H2D write, the allocator skips the chunk’s row range to avoid reading partially written data. At switch time, the gather plan copies only clean staged chunks with D2D and reloads dirty or missing chunks from host. This protocol makes the expensive H2D component scale with the uncovered weight fraction; the total switch still includes D2D gathering for staged chunks. It also gives the router a simple signal: staging slack, the remaining HBM space that can be used to pre-stage a future switch. When slack falls, the instance becomes a worse target for more traffic in the residency cost model; Section 5.6 separates the full-D2D, partial, and H2D fallback paths. Together, the router lease, bump layout, HKVR, and D2D staging realize the substrate contract exposed to the design: stable residency, restorable session state, and measured switch-cost telemetry.

Host KV Registry

Switching away from a model may reclaim its device KV region, but returning sessions still need the prefix produced before the tool call. HKVR stores this state in a unified pinnedmemory pool shared across attention layouts. MHA models allocate paired K/V slabs, MLA models allocate latent-state slabs, and Mamba-style models allocate recurrent-state slabs. All slabs return to the same pool when evicted, so host memory is not statically partitioned by model or architecture. HKVR checkpoints aligned KV blocks asynchronously when a request finishes or a session enters a tool gap. Each block is keyed by the model name and cumulative prefix hash, matching the radix tree’s chunk granularity. On return, the scheduler allocates new device slots, restores the agreed prefix through H2D layer-first/page-first transfer kernels, and recomputes only the unconfirmed tail. This path reuses SGLang’s KV movement kernels for MHA. MLA and Mamba share the registry metadata and slab allocator, while architecture-specific completeness checks may decline a restore rather than publish partial state. The router sees a common device/host/missing state; the instance decides whether the architecture-specific state is restorable. Three invariants keep HKVR safe under concurrent serving. First, transfer pins are separated from radix ownership: DMA pins prevent reclaiming blocks in use, while radix references track whether prefix metadata still points to a block. Second, HKVR exposes a prefix only after TP ranks agree on the same valid common prefix. A disagreement is handled conservatively: the restore tracker truncates the published prefix to the longest common block range shared by all ranks and marks the inconsistent tail for recomputation. Partial or inconsistent tails remain private until a later checkpoint extends

Evaluation

We evaluate Talaria with fixed replay as the primary evidence and targeted mechanism measurements for effects that a single replay cannot separate. Fixed replay holds the applicationlevel call trace constant, so policy differences are not confounded by the agent choosing a different tool path. We ask five questions: (1) do the mechanisms improve end-to-end session completion, (2) does residency-aware routing avoid misplaced model opens, (3) what is the cost/benefit of soft reservation, (4) does SP capture returns within an active slot, and (5) does the memory substrate reduce switch cost without exceeding HBM capacity?

5.1

Methodology

The main replay is captured from Claude Code running ten Astropy issues from SWE-Bench Verified [13]. We replay 9

each issue once on Qwen3-235B, GLM5-nvfp4, and Qwen3.5122B-A10B, yielding 30 model-session executions and 960 calls. Reusing the ten task identities across three model profiles enables paired, within-trace policy comparisons; the persession plot shows every observation.

TP=8 fixed replay

1.0

CDF

0.8

For each model-session, every configuration replays the same request bodies under the same model and session IDs, in the same per-session call order, and with the same completiontoken count for each call. The replay is closed-loop: after a call finishes, the next waits for the same recorded tool gap after a 10 s cap. The cap affects 20 of 960 calls. Global interleaving may differ across configurations because completion times differ. The resulting trace targets the long-context, shortreturn-gap regime for which session continuity matters most. Large-model runs use one TP=8 server with eight 183 GiB GPUs, 2 TB host memory, dual-socket x86 CPUs, and eight 400 Gb/s network adapters. Each run performs the same prewindow warmup; all measurements exclude warmup.

0.6 0.4 0.2 0.0 102

103

Session completion time (s) Talaria no-HKVR Round-only no-SP

H2D-only

Figure 6: End-to-end SCT on the TP=8 fixed replay. Talaria cuts p50 SCT from 1000 s to 189 s relative to the Round-only ablation.

Session completion time (s)

Reverse ablations on the same trace

We compare Talaria’s instance-local configuration (SP, HKVR, and D2D staging) with reverse ablations. no-SP disables mid-slot session-prefill admission while keeping HKVR and D2D staging. no-HKVR keeps SP and D2D staging but disables host-restorable KV. H2D-only keeps SP and HKVR but disables D2D weight staging, so switches reload from host memory. Round-only disables SP, HKVR, and D2D staging and serves the same trace with the otherwise identical round scheduler and host-to-device model reloads. Round-only is the controlled all-off baseline within the same engine.

2400

1800

1200

1000s 623s

600

486s 194s

189s 0 Talaria

no-SP

no-HKVR

H2D-only

Round-only

Figure 7: SCT for all 30 model-sessions on the same replay; boxes show the interquartile range under the nearest-rank empirical definition. Each point is one of ten issues executed on one model.

Router-policy experiments use two TP=4 workers to create real placement choices. They replay a 120-call, 30-modelsession subset under three relative offered-load regimes. The trace preserves session ids, return gaps, model order, and reusable-prefix lengths, but maps the large models to Qwen332B, Qwen2.5-32B-Instruct, and Qwen3.5-35B-A3B. This split is intentional: TP=8 evaluates instance-local mechanisms under large-model memory pressure, while TP=4 isolates placement decisions. Together, the two testbeds separate instance execution from cluster placement without conflating their effects.

5.2

End-to-End Session Completion

Figures 6 and 7 show the SCT distribution for the TP=8 fixedreplay configurations. All rows replay the same 960 calls over the same 30 sessions. Relative to Round-only, Talaria’s instance-local configuration reduces p50 SCT from 1000 s to 189 s and p95 SCT from 2296 s to 867 s, delivering 5.3× and 2.6× speedups. The p50 TTFT falls from 13.44 s to 0.55 s. The reverse ablations show each mechanism’s marginal effect with the others enabled; these effects are not additive. Round-only combines next-slot waiting with prefix recomputation after reusable KV is lost. SP provides the largest p50 marginal contribution, reducing SCT from 623 s to 189 s by admitting eligible returns within the active model slot. HKVR reduces p50 SCT from 486 s to 189 s and TTFT p95 from 26.33 s to 14.17 s. D2D staging leaves p50 SCT nearly unchanged (194 s to 189 s) but reduces p95 from 933 s to 867 s; §5.6 isolates its switch path. Talaria improves 29 of 30 paired model-sessions.

The primary metric is session completion time (SCT), measured from a model-session’s first call to its final response. We also report TTFT, switch latency, round composition, and HBM footprint. TTFT is schedule-aware: it starts at the intended enqueue time and therefore includes admission delay. For raw replay distributions, percentiles use the nearest-rank empirical definition: for N observations, pq is the observation at rank ⌈qN/100⌉; targeted TP=4 experiments retain their original harness summaries. The main trace intentionally runs at high pressure to expose differences in how the policies preserve session state while per-session call order, bodies, and recorded gaps remain fixed. 10

(a) High-load TTFT

0.4 0.2 0.0 0

30 20

medium

Session-sticky

p95

1.75

-35% p50

1.50 1.25 1.00

Reservation timeout τ (s) same instance model-residency hit

Full router

p50

2.00

0.75 0 0.5 1

high

0 0.5 1

Reservation timeout τ (s)

unused-lease proxy

Figure 9: TP=4 reservation-timeout sweep. A 1 s lease gives the lowest returning-call p95; extending to 5 s trades model residency for higher return locality.

Figure 8: Two-worker TP=4 placement over 120 calls. The full router nearly eliminates avoidable model opens while controlling high-load TTFT.

5.4 5.3

40 20

TTFT (s) Least-pressure

low

Returning TTFT (s)

0.6

(b) Returning TTFT

2.25

Rate (%)

CDF

0.8

(a) Locality tradeoff

(b) Cold opens Avoidable cold opens

1.0

Soft-Reservation Tradeoff

Soft reservation adds a temporal decision to router placement: after a session leaves for a tool call, the gateway leases capacity for a bounded interval τ instead of making the next call compete as an unrelated arrival. The TP=4 sensitivity run sweeps τ ∈ {0, 0.5, 1, 2, 5} s and reports same-instance return rate, model-residency hit rate, returning-call TTFT, model opens, and an unused-lease proxy. The proxy counts reserved returns that did not produce a timely same-instance hit; it does not measure how long a live lease occupied capacity. With τ = 0, returning sessions still use residency-aware cost ranking, but no capacity is held: 56.2% return to the same worker and 43.8% miss the device-KV proxy. A 1 s lease raises same-worker returns to 67.7% and lowers returning-call TTFT p50/p95 from 1.51/2.11 s to 0.99/1.91 s, the lowest p95 in the sweep. This gain uses additional placement capacity: relative to τ = 0, model-residency hits fall from 72.5% to 60.8%, model opens rise from 33 to 47, and the unused-lease proxy is 50%. Extending the lease to 5 s raises same-worker returns further to 79.2%, but provides similar TTFT while model-residency hits fall to 53.3% and model opens rise to 56. We therefore use 1 s as the operating point that best balances return locality and model residency for this workload; τ is a workload-dependent policy knob.

Spatial Placement: Router Policy

Figure 8 compares three routing stacks on the same 120call trace. Least-pressure chooses the least-loaded worker for each call. Session-sticky pins a session to its first worker. The full router combines residency cost ranking with soft reservations, matching the prototype policy; §5.4 separately sweeps the reservation timeout. We count an avoidable cold open when the selected worker must open the requested model even though another worker already has that model resident. Least-pressure frequently selects a worker that must reopen the requested model. Session-sticky preserves affinity but cannot trade it against model residency and queue pressure. The full router eliminates avoidable opens at low and medium load and leaves one at high load. At medium load, TTFT p50 is 0.36 s, versus 1.67 s for least-pressure and 2.29 s for session-sticky. At high load, it retains 97.8% of returning calls on the same worker, reduces avoidable opens from 37 (leastpressure) and 5 (session-sticky) to 1, and cuts TTFT p95 from 8.07 s and 6.23 s to 5.28 s. The placement gains translate most clearly once model opens and queue pressure matter. At medium load, full routing improves SCT p50/p95 to 6.28/12.74 s from 11.01/19.59 s for least-pressure and 10.97/19.28 s for session-sticky. At high load, it lowers p95 to 21.31 s from 31.29 s for leastpressure and 23.03 s for session-sticky; its 17.44 s p50 is slightly above sticky’s 16.33 s. At low load, eliminating avoidable opens does not translate into an SCT gain: full-router p50/p95 is 7.39/15.81 s, versus 6.82/11.31 s for least-pressure and 5.62/13.68 s for session-sticky. Thus residency-aware placement yields its clearest SCT benefit at medium load; at high load it improves p95 but not p50, while at low load it eliminates model opens without an SCT gain.

5.5

Temporal Placement: SP Admission

SP addresses the temporal mismatch inside a model slot: a returning session should not wait for the next full prefill window when the current slot can admit its reusable prefix. Figures 10 and 11 compare Talaria with no-SP on the same fixed replay, keeping request bodies, arrival gaps, HKVR, D2D staging, and the router fixed. With all other mechanisms fixed, SP lowers p50 TTFT from 4.60 s to 0.55 s and p95 from 20.5 s to 14.2 s. Within Talaria’s 200 rounds, SP admits 316 return-handling events into the active slot and parks 345, an admission rate of 47.8%. 11

(a) Admission

(b) Handling

40 30 20 10

no-SP

(a) Latency

1.0

Admitted Deferred

36%

1497 ms 1.7

0.6

1.8

0.2

H2D-only

Talaria

28%

Talaria no-SP

0.0

Talaria

0.0 H2D-only

103

Talaria

Switch latency (ms)

Figure 10: SP handling events on the TP=8 fixed replay. Talaria admits 47.8% of its recorded return-handling events within the active model slot.

D2D

Partial

H2D

Figure 12: Logical TP=8 switch latency, computed as the maximum across ranks. Each configuration contains 600 logical switches; paths use the worst-rank classification. Staging cuts p50 by 34%.

Per-call TTFT

0.8 0.6

(a) Composition

0.4

(b) Aggregate

no-SP (p95=20.5s)

0.2

Talaria (p95=14.2s) 0.0 0.1

TTFT (s)

Figure 11: SP TTFT impact on the same replay. Mid-slot admission lowers per-call TTFT.

35% 52% 0.5

65% 48%

Aggregate round time (s)

1.0

Fraction of wall time

CDF

36%

984 ms

1.6

p95

100%

0.5 0.4

102

1.0

(b) Paths 1.0

0.8

CDF

48%

Handling events/round (not unique calls)

Within-policy events admitted (%)

2000

1500

1000

500

0.0

0 H2D-only

Talaria records none of the 123 host-cache capacity misses and 2,532 architecture-specific restore skips observed with no-SP. These are schedule-dependent diagnostics, not uniquecall outcomes; all 960 calls complete in both configurations. A parked request may also appear in multiple rounds, so we normalize handling events within each policy and use TTFT, measured over the common call set, as the direct cross-policy result.

Switch -38% 1753s 1632s

Talaria

H2D-only

Active

Talaria

Switch

Figure 13: Round-time composition for Talaria and H2D-only. D2D staging reduces aggregate switch time by 38%.

ing 36.0% falls back to H2D at a 2011 ms p50. The staged common path therefore delivers the largest speedup, while partial coverage retains a proportional benefit. Active writes dirty some staged chunks, causing the correctness guard to reload them from host and accounting for the heavier tail. H2D-only also records 39 host-cache capacity misses and 1,704 Mamba restore skips, versus zero for Talaria. These schedule-dependent diagnostics capture downstream cache interactions; the logical-switch distribution is the direct path comparison. Figure 13 shows the same effect at round granularity. H2Donly spends 51.7% of round wall time in switching, while Talaria spends 34.6%; total switch time drops from 907 s to 565 s. Talaria’s aggregate round wall time is also lower (1632 s versus 1753 s). Because SP and HKVR already preserve the common return path, the 38% switch-time reduction has a larger effect on tail SCT than on p50: p50 changes from 194 s to 189 s, while p95 falls from 933 s to 867 s.

5.6 Memory Substrate: Switch and HBM Cost Across the replay, D2D staging reduces aggregate switch time by 38% and switch share from 51.7% to 34.6%. Figure 12 isolates the switch path by comparing Talaria with H2D-only, which keeps SP and HKVR enabled but reloads weights from host memory. Each logical switch emits one record per TP rank; we report the maximum rank latency because the scheduler resumes only after all ranks finish. At this granularity, D2D staging improves the center of the distribution but exposes a tail tradeoff under high KV pressure: p50 falls from 1497 ms to 984 ms and mean from 1521 ms to 1071 ms, while p95 rises from 2133 ms to 2679 ms. Full D2D switches, 27.7% of the observed mix, complete in 106 ms at p50; partial D2D plus H2D accounts for 36.3% at a 1000 ms p50, and the remain12

Round time (ms)

Active

System

(a) Talaria

20000 10000 0 25

100

125

150

Model route KV route Yes No No Yes Yes

No Yes Yes No Yes

Agent return next model slot request arrival storage restore cold start reservation + SP

Table 1: Placement signals and cross-call return handling.

175

(b) Round-only

30000

Related Work

20000

Table 1 distinguishes KV-aware placement (“KV route”) from internal KV synchronization: only the former lets residency change which instance receives a return.

10000 0 0

100

150

200

250

300

Round index

Serverless and multi-model LLM serving. Classical prediction-serving systems focus on model lookup, batching, replication, pipeline provisioning, and interference control under request-level latency objectives [24–29]. Serverless systems similarly hide placement behind a request/function abstraction whose state is short-lived relative to model execution [30]. LLM serving makes weights and KV cache persistent serving state. ServerlessLLM optimizes checkpoint loading and migration for on-demand LLM inference, and SpotServe adapts serving to preemptible GPU availability [6, 31]; AlpaServe and MuxServe multiplex multi-model or modelparallel inference on shared GPUs [7, 8]; Aegaeon targets long-tail LLM marketplaces with token-level auto-scaling [5]. These systems multiplex models and requests. Talaria instead treats the session as the scheduling object: a router must preserve model and KV residency across tool gaps, and a cold-pool instance must admit the returning session before the locality expires.

Figure 14: Per-round composition for Talaria and Round-only. The combined mechanisms reduce switch share and round count.

183 GiB cap 180

HBM used (GiB)

Unit

Aegaeon [5] request/token Preble [16] prompt Mooncake [15] KV object ServerlessLLM [6] request Talaria session

30000

Round time (ms)

Switch

peak 177.1 GiB

170

Talaria

H2D-only

no-SP

Round-only

no-HKVR

183 GiB cap

160 0

100

150

200

250

300

Round index

Adapter and fine-tuned-model serving. S-LoRA, Punica, and dLoRA make large catalogs of LoRA adapters practical by sharing a resident base model, paging adapter state, or dynamically migrating requests and adapters [3, 4, 32]. These systems reduce the cost of serving many fine-tuned variants of a base model. Talaria targets the complementary full-model setting: the active model itself may switch, and a returning session’s KV must remain usable across tool gaps and model residency changes.

Figure 15: HBM footprint under the bump allocator. Runs remain below the 183 GiB device cap. Curves show per-round averages; the annotation is the maximum across all configurations.

Figure 14 shows the combined effect over time. Round-only carries a persistent switch band because every slot-boundary model change reloads from host memory. Talaria’s SP, HKVR, and staging mechanisms jointly reduce switch share and the number of rounds; this comparison does not attribute the difference to staging alone. Curves use an 11-round moving average for visual clarity and retain separate round-index axes because the configurations execute 200 and 339 rounds.

Memory management for LLM serving. PagedAttention, vAttention, FlexGen, LoongServe, InfiniGen, and InfiniteLLM improve LLM serving by reducing KV fragmentation, exploiting virtual memory, offloading cache state, or distributing long-context KV across memory tiers [20–22, 33–35]. These systems optimize the memory hierarchy for active requests or long contexts. Talaria uses memory management for a different scheduling contract: a model can leave the GPU without losing a returning session’s prefix, and a future model switch can be prepared while the current model is still serving.

Figure 15 shows device footprint across configurations. The peak observed HBM use is 177.1 GiB, below the 183 GiB device cap. This confirms that the managed layout, restorable KV, and weight staging remain within the device-memory budget throughout the replay.

Execution engines and phase-aware scheduling. Orca, vLLM, Sarathi-Serve, FastServe, DeepSpeed-FastGen, and 13

NanoFlow improve utilization of a loaded model through iteration-level scheduling, KV paging, chunked prefill, preemption, dynamic split-fuse scheduling, and intra-device overlap [20, 36–40]. Splitwise, DistServe, and TetriInfer separate prefill and decode resources, and TokenScale autoscales such deployments using token velocity [17–19, 41]. Talaria is complementary: it coordinates model switching, session returns, and memory pressure across many models. Its evaluated colocated design avoids adding a cross-tier KV handoff to each session turn.

paired model-sessions. SP lowers p50 TTFT from 4.60 s to 0.55 s, while D2D staging cuts p50 logical-switch latency by 34% and aggregate switch time by 38%. On the twoworker placement testbed, the router eliminates every avoidable model open at low and medium load and leaves only one at high load. The design intentionally keeps the router out of the token data path. Weight and KV transfers are executed by instances, while the router uses telemetry to decide when those transfers are worth paying. This separation is what lets Talaria combine cluster-level placement with instance-level execution contracts: the router selects an instance using model residency, KV availability, and pressure; that instance restores KV and activates or stages models as needed.

KV reuse and cache-aware routing. CachedAttention and SGLang reuse conversation or structured-program prefixes through KV hierarchy and RadixAttention [14, 23]. Hydragen accelerates shared-prefix attention, CacheGen and CacheBlend reduce the cost of moving or composing reusable KV, and LMCache provides a KV-cache layer for enterprisescale serving [42–45]. Mooncake and MemServe expose KV cache as a disaggregated cluster resource, while Preble, DualMap, and Llumnix use cache-aware routing or migration to balance locality and load [15, 16, 46–48]. Talaria builds on the same principle that KV is schedulable state, but the routing decision also depends on model residency, round pressure, and observed session-return timing.

Limitations and future work. Our current evidence combines controlled large-model execution on one TP=8 server with isolated placement experiments on two workers. Broader workload and cluster-scale evaluation are the next empirical steps. On the mechanism side, workload-adaptive lease and host-cache policies, remote-KV integration, dynamic hot-cold pool reallocation, and P/D-disaggregated admission are natural extensions. D2D staging improves the p50 and mean switch path under high KV pressure; reducing dirty-chunk fallbacks can extend these gains to p95.

Serving agentic programs. ReAct, Toolformer, ReWOO, Reflexion, and AutoGen popularized tool-interleaved, reflective, and multi-agent language-model execution, and SWEagent shows that software-engineering agents repeatedly call models while interacting with a repository and test environment [9–11, 49–51]. SGLang, Parrot, Pie, Autellix, and Continuum then show that such programs expose reusable prefixes, control flow, inter-request dataflow, and KV lifetimes that should be scheduled above the single-request level [12, 23, 52–54]. Talaria consumes that session-level structure at a complementary layer: placement and execution across a serverless multi-model pool. Here, models may be hot or cold, and the router must jointly decide whether to reuse a model replica, restore session state, or recompute a prefix.

References [1] Stanford Institute for Human-Centered Artificial Intelligence. The 2025 ai index report. Technical report, Stanford University, 2025. [2] Avijit Ghosh, Lucie-Aimée Kaffee, Yacine Jernite, and Irene Solaiman. State of open source on hugging face: Spring 2026. Hugging Face Blog, 2026. [3] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-LoRA: Serving thousands of concurrent LoRA adapters. In Proceedings of Machine Learning and Systems (MLSys), 2024.

Conclusion

This paper presented Talaria, a serverless multi-model serving system for agentic workloads. Serving an agent session requires tracking both session state—KV locality and return timing—and instance state—model residency and pressure. Request-level placement and slot-boundary admission lose continuity in different ways. Talaria addresses them with soft reservation at the router and session-prefill (SP) at the cold-pool scheduler. An instance-local substrate preserves restorable KV and stages weights across model switches. On the fixed TP=8 replay, the instance-local mechanisms deliver 5.3× and 2.6× speedups in p50 and p95 SCT relative to the internal Round-only ablation, and improve 29 of 30

[4] Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. dLoRA: Dynamically orchestrating requests and adapters for LoRA LLM serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 911–927, 2024. [5] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective gpu pooling for concurrent llm serving on the market. In Proceedings 14

and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with CachedAttention. In Proceedings of the 2024 USENIX Annual Technical Conference (USENIX ATC), 2024.

of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP), 2025. [6] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Low-latency serverless inference for large language models. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.

[15] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation – a KVCache-centric architecture for serving LLM chatbot. In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST), 2025.

[7] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2023.

[16] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving, 2024. [17] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Iñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), 2024.

[8] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. MuxServe: Flexible spatial-temporal multiplexing for multiple LLM serving, 2024. [9] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.

[18] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodputoptimized large language model serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.

[10] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[19] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference: Disaggregate LLM inference for mixed downstream workloads, 2024.

[11] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023.

[12] Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for llm agents as general programs, 2025.

[21] Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. vAttention: Dynamic memory management for serving LLMs without PagedAttention, 2024.

[13] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.

[22] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th

[14] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, 15

[31] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. SpotServe: Serving generative large language models on preemptible instances. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024.

International Conference on Machine Learning (ICML), 2023. [23] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[32] Lequn Chen, Zhiqiang Ye, Yilong Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy, and Tianqi Zhang. Punica: Multi-tenant LoRA serving. In Proceedings of Machine Learning and Systems (MLSys), 2024.

[24] Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. TensorFlow-Serving: Flexible, high-performance ML serving, 2017.

[33] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. LoongServe: Efficiently serving long-context large language models with elastic sequence parallelism, 2024.

[25] Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017.

[34] Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.

[26] Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. Nexus: A GPU cluster engine for accelerating DNN-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP), pages 322–337, 2019.

[35] Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. Infinite-LLM: Efficient LLM service for long context with DistAttention and distributed KVCache, 2024.

[27] Daniel Crankshaw, Gur-Eyal Sela, Corey Zumar, Xiangxi Mo, Joseph E. Gonzalez, Ion Stoica, and Alexey Tumanov. InferLine: Latency-aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing (SoCC), 2020.

[36] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022.

[28] Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less inference serving. In Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC), pages 397–411, 2021.

[37] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.

[29] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: Performance predictability from the bottom up. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020.

[38] Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. FastServe: Fast distributed inference serving for large language models. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2023.

[30] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC), 2020.

[39] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. DeepSpeed-FastGen: High-throughput text generation for LLMs via dynamic splitfuse, 2024. 16

[40] Kan Zhu, Yilong Zhao, Liang Zhao, Gan Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. NanoFlow: Towards optimal large language model serving throughput, 2024.

[50] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [51] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation, 2023.

[41] Ruiqi Lai, Hongrui Liu, Chengzhi Lu, Zonghao Liu, Siyu Cao, Siyang Shao, Yixin Zhang, Luo Mai, and Dmitrii Ustiugov. TokenScale: Timely and accurate autoscaling for disaggregated LLM serving with token velocity, 2025. [42] Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput LLM inference with shared prefixes, 2024.

[52] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of LLM-based applications with semantic variable. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 929–945, 2024.

[43] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM Conference, 2024.

[53] In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A programmable serving system for emerging LLM applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP), 2025.

[44] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion, 2024.

[54] Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph E. Gonzalez, and Ion Stoica. Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live, 2025.

[45] Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang. LMCache: An efficient KV cache layer for enterprise-scale LLM inference, 2025.

Trace and Replay Details

Session boundaries. A session is one agent task. The trace starts at the first model call issued for that task and ends at the final model response before the task terminates. We define the inter-call gap as the wall-clock interval between a model response and the next model enqueue. This interval includes tool execution and agent/runtime orchestration, so it is not a pure tool-execution measurement. Harness startup and teardown fall outside the session boundary. Across the 6,941 possible consecutive-call intervals, 6,294 have a complete timestamp pair; the remaining 647 do not. The characterization plot further excludes five >30 s environment-startup intervals, leaving 6,289 gaps for the reported distribution.

[46] Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. MemServe: Context caching for disaggregated LLM serving with elastic memory pool, 2024. [47] Ying Yuan, Pengfei Zuo, Bo Wang, Zhangyu Chen, Zhipeng Tan, and Zhou Yu. DualMap: Enabling both cache affinity and load balancing for distributed LLM serving, 2026. [48] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.

Prefix reuse. For consecutive calls within a session, prefix reuse is measured after tokenization as the longest common prefix between the new prompt and the prior prompt plus generated response that remains in the session context. The resulting prefix length quantifies the amount of reusable work. Whether a return actually reuses device KV, restores host KV, or recomputes the prefix also depends on cache residency, validity, and capacity at return time.

[49] Binfeng Xu, Zhendong Peng, Bowen Lei, Subhabrata Mukherjee, Yu Liu, and Dongkuan Xu. ReWOO: Decoupling reasoning from observations for efficient augmented language models, 2023. 17

Replay inputs. The 445-session characterization trace supports the distributions and recovery-cost estimates in §2.1. Each available record contains the session id, call index, model id, enqueue timestamp, prompt and output tokens, reusable-prefix tokens, and inter-call gap. The recovery-cost estimates combine these records with model-specific prefill profiles measured on the testbed. The TP=8 fixed replay is a separate 960-call workload. Across configurations, each model-session uses the same request bodies, model and session ids, per-session call order, completion-token counts, and recorded inter-call gaps. The replay caps each gap at 10 s, affecting 20 of 960 calls. It is closed-loop: each next call is released after the preceding call completes and its capped gap elapses, so global interleaving may differ as configurations finish calls at different times. The TP=4 router experiments use a separate 120-call, 30-modelsession subset and map its model ids to the models on that testbed (§5.1).

whose first generated tokens are known; after a forced model switch and HKVR restore, the client compares the generated prefix with the expected sequence and records whether the cache hit came from device or host state. Large-model stress runs enable the same runtime checks under round scheduling: TP-prefix disagreement truncates the exposed prefix, stale or uncommitted DMA blocks are never published to radix metadata, and staged weight chunks marked dirty are reloaded from host rather than copied through D2D. These tests exercise the guards; performance claims use the workloads in §5.

HKVR separates the scheduler-visible cache state from the physical layout used to move KV bytes. The router only observes whether a prefix is device-resident, host-restorable, or missing. The restore path below that interface packs and unpacks KV pages through layout-aware transfer kernels. Layer-first and page-first movement. SGLang’s KV movement path supports two equivalent views of the same logical prefix. In a layer-first (LF) layout, host slabs are grouped by model layer, then by page within that layer; this matches the device-side KV cache layout used by attention kernels and makes a layer’s pages contiguous for DMA. In a pagefirst (PF) layout, the host representation is grouped by logical prefix page, with the per-layer fragments for that page stored under one page descriptor. PF is useful for registry operations because the prefix can be committed, invalidated, or restored at page granularity. HKVR records the logical prefix as pages and invokes LF/PF transfer kernels to translate between the registry view and the active device layout. LF/PF are datamovement layouts; HKVR is the consistency and naming layer above them. Attention-layout metadata. Each HKVR entry carries a compact layout descriptor: model id, TP rank, prefix hash, committed token length, page size, dtype, attention family, and slab shape. MHA models store paired K/V slabs, while MLA models use latent-state slabs. Supported Mamba-style paths register recurrent and auxiliary state through architecturespecific descriptors rather than treating that state as KV slabs. The router-visible states remain device-resident, hostrestorable, or missing across these families, but transfer and completeness checks are architecture-specific. HKVR publishes restorable state only when that state is complete and the agreement rule in §B holds; otherwise the unconfirmed suffix is recomputed.

Timing telemetry. Replay logs keep request-local TTFT and schedule-aware TTFT in the same schema. The latter starts at the replay scheduler’s intended enqueue time, so it includes any replay-side admission delay. Unless stated otherwise, the paper reports schedule-aware TTFT. Round summaries use a mutually exclusive decomposition, wall_ms = active_ms + switch_ms. Phase counters such as prefill, decode, H2D, and D2H are diagnostic subviews inside active slots; they can overlap and are not summed into wall time.

HKVR Transfer Layouts

Correctness Conditions

Talaria relies on three implementation-level conditions when moving state across HBM and host memory. Prefix agreement. Each tensor-parallel rank reports its committed prefix length and cumulative prefix hash at page boundaries. HKVR exposes only the longest page-aligned prefix on which all ranks agree for the same model. If a rank lags or hashes diverge, HKVR publishes the last agreed boundary and marks the remaining suffix for recomputation. This makes TP disagreement a reuse loss rather than a correctness risk. Lifetime separation. Radix-tree ownership and DMA transfer pins are tracked separately. A block can remain pinned until an asynchronous copy finishes even if the radix tree no longer references it; conversely, radix metadata cannot expose a block whose DMA copy has not completed. Switch barrier. Before a model’s active kv_cache region is released, the scheduler drains outstanding D2H events for prefixes that may be restored later. The barrier is scoped to the outgoing model, so unrelated models do not block the switch path.

Switch Timeline Instrumentation

The implementation records one SWITCH_TIMING line per TP rank for every model switch. Each rank’s timer starts after pre-switch diagnostic probes. The instrumented latency of a

Validation protocol. We exercise these guards at two levels. Synthetic switch/restore tests use deterministic prompts 18

logical switch is the maximum across its TP ranks; the phase breakdown below instead reports rank-level records so that per-rank work remains visible. A switch follows five ordered phases. Phase 1: save and teardown. The instance drains outstanding HKVR D2H writes for the outgoing model, cancels stale H2D restores, saves the CUDA-graph and KV-pool handles, releases runtime buffers, parks the outgoing radix tree, and pauses the outgoing model’s per-model TMS tags. Phase 2: open target weights. The instance waits for any outstanding pre-staging work, verifies the staging bitmap, updates the active model configuration, resolves the CPU-side model entry, and opens the target weights either through D2D gather, partial D2D plus H2D top-up, or full H2D fallback. Phase 3: restore the KV shell. The scheduler checks whether the target model’s KV-pool shell can be reused under the current weight-region size, page size, and capacity. A hit rebinds the cached shell; a miss rebuilds the pool under the target model’s TMS tag. Phase 4: restore runtime and graph state. The instance restores a cached attention backend and CUDA graph when the KV shell and graph pointers match; otherwise it rebuilds runtime buffers and recaptures graphs. Phase 5: publish references. The instance propagates the new model, allocator, KV, radix, and telemetry references back to the scheduler and worker. This phase is intentionally small: large memory movement has already completed in Phase 2. Phase

D2D full p50 p95

D2D + H2D p50 p95

P1 P2 P3 P4 P5

26.2 34.7 109.6 244.8 0.5 2.0 8.6 20.3 0.2 0.3

Total

149.8 298.9 1041.2 1854.6 2608.6 3170.6

rank-level records: 14,952 D2D-full, 102 partial D2D+H2D, and 16 full-H2D fallbacks. These records hit the cached KV shell and CUDA graph, so the tail is driven by uncovered weight bytes rather than runtime recapture. D2D-full switches have a p50 D2D gather-sync time of 18.3 ms; partial switches add H2D synchronization with a p50 of 696.2 ms for uncovered chunks. Dirty chunks arise when active KV allocations overlap the staging range, and the guard reloads those chunks from host rather than copying unsafe D2D sources. Model-specific weight postprocessing. Some models expose runtime weights that are not a byte-for-byte copy of the raw checkpoint tensors. In our GLM runs, the quantized postload path materializes packed and derived tensors on the GPU after the first open. The switch path therefore treats the postprocessed physical layout as the stable representation: after finalization, it writes the processed tensors back to the hostside model entry and uses that layout for later H2D or D2D reloads. This keeps region sizing and staged-byte accounting aligned with the physical parameter and persistent-buffer order used by the active model.

Policy Knobs

Reservation timeout τ. We select τ = 1 s from the sensitivity sweep in §5.4; it is near the p90 inter-call gap in the characterization trace. Expired sessions fall back to normal residency-aware placement. Bump memory fraction. The component experiments use mem-fraction-bump=0.8, reserving most HBM for the managed layout while leaving headroom for runtime allocations and a safety margin. Admission margin γ. Cold-pool rounds are bounded below the configured TTFT target with γ < 1, after accounting for switching and restore overhead. This knob is an admissioncontrol margin; absolute target attainment still depends on the offered load and provisioning level. Replica penalty νm . The router uses a per-model replica penalty to avoid opening replicas for transient bursts whose benefit does not cover switch and memory cost.

H2D-only p50 p95

26.9 30.9 21.9 31.7 995.8 1819.8 2570.6 3142.2 0.5 1.2 0.8 1.9 8.3 20.0 7.7 16.4 0.2 0.3 0.2 0.3

Table 2: Per-rank five-phase switch timing in a separate component stress run (ms). Table 2 reports a separate TP=8 component stress run, not the 600 logical switches per configuration in Figure 12. D2Dfull and D2D+H2D use steady-state rank-level records after excluding warm-up and instrumentation outliers (n = 14,952 and n = 102); H2D-only uses rank-level records from a noD2D stress run (n = 2,104). The table shows why staging helps: phases other than weight opening are already small under cache hits, so the switch path is dominated by Phase 2. Full D2D staging reduces the p50 Phase 2 cost from 2.57 s to 110 ms, while partial staging lands between the two according to the uncovered H2D fraction. Path classification. Each rank-level record also identifies the load path, staged-byte coverage, dirty staging bytes, and D2D/H2D gather times. In the TP=8 staged run used for the first two path groups of Table 2, the filtered set contains 15,070 19

Record · ID 386837 · SHA-256 012328c43d1c773d
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.