Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
Fangyue Liu* 1 2 Hua Liu* 2 Xinyuan Lyu* 2 Shuo Ai 2 Hao Liang 2 Lingpeng Chen 2 Ziqian Hu 2 Chong Zha 2 Xin Jin 1 Hanmei Luo 2 Peng Chen 2
arXiv:2604.07874v1 [cs.OS] 9 Apr 2026
Abstract
(1) Kernel-Level Preemption
LLM inference powers latency-critical production services nowadays. The bursty nature of inference traffic results in over-provisioning, which in turn leads to resource underutilization. While onlineoffline colocation promises to utilize idle capacity, broad production deployment must overcome two major challenges: (i) large online interference due to slow or frequent preemptions, and (ii) extensive frameworks and drivers modifications, to colocate different models and support preemptions. We present Valve, a production-friendly colocation system that jointly bounds preemption latency and preemption rate. Specifically, Valve enables submillisecond compute preemption at most once per online request, and rate-limited sub-layer memory reclamation. These guaranties are provided by a GPU runtime that combines channel-controlled compute isolation, page-fault-free memory reclamation, and dynamic memory reservation. Critically, Valve is practical to deploy, requiring one line of driver modification and 20 lines of framework patch. Deployed on 8,054 GPUs in production, Valve improves cluster utilization by 34.6%, which translates to a 2,170 GPU save. This efficiency gains is achieved with minimal online interference, incurring < 5% TTFT increase and < 2% TPOT increase across workloads.
(2) Layer-Level Preemption
OFF:1
P:1
OFF:2
P:2
P:3
D D D D D
OFF:3 Slow P:1
OFF:1
(3) Time-Slice Preemption
OFF:1
(4) Valve
OFF:1 Offline : Layer-id
P:2
Preemption D D D D D
P:3
OFF:2 P:1
P:2
P:3
D
D
P:1
P:2
P:3
D D D D D
Online Prefill : Layer-id
D
D
D
Frequent Preemption Fast, Infrequent Preemption
Online Decode
Figure 1. Comparison of online-offline colocation approaches.
user-facing applications, inference has also become a building block for training workflows, including data curation, post-training actor rollouts, and critic scoring (Guo et al., 2025; Lee et al., 2023). Despite their importance, LLM inference clusters in production still suffer from low utilization. This is primarily because operators must provision for bursty demand under strict latency SLAs, resulting in significant idle capacity off-peak. In practice, the burstiness comes from two main sources. First, online services may experience unpredictable traffic spikes (Xiang et al., 2025), which are further amplified by customized or fine-tuned models. Second, inference in training workflows typically arrives in periodic large batches, causing volatile GPU memory usage (Zhong et al., 2025; Wu, 2025). Harvesting idle capacity to improve utilization remains one of the central challenges in LLM inference deployments.
1. Introduction
A promising direction is to colocate latency-critical online serving with offline inference on the same GPU, which utilizes the idle capacity to run preemptible offline workloads. In practice, however, we find two key obstacles that limit broad deployments of existing approaches (Wu et al., 2023; Qiao et al., 2024; Fan et al., 2025): (i) interference with online workloads caused by high preemption latency or frequency, and (ii) extensive modifications to GPU drivers or inference frameworks.
Large language model (LLM) inference powers a growing set of workloads. These include latency-critical production services, such as conversational assistants, code generation, multimodal tasks (OpenAI, 2022; Anthropic, 2023; Liu et al., 2024; Jimenez et al., 2023; Jain et al., 2024). LLM also serves batch processing workloads, such as document processing, data analysis (Lai et al., 2023). Beyond *
Online arrival
Equal contribution. 1 Peking University 2 Tencent.
Coarse-grained preemption—e.g., kernel-level (Wu et al., 1
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate GPU Util
Percentage (%)
2023) (which degrades to iteration-level when CUDA graphs are enabled) and transformer-layer-level (Qiao et al., 2024)—incurs preemption latencies of up to tens of milliseconds. Immediate preemption (Fan et al., 2025; Harris, 2017) can react quickly, but may trigger preemptions frequently. Moreover, many existing approaches require extensive framework or driver modifications (Ruan et al., 2023; Qiao et al., 2024), which hinders deployment in production environments.
KV Cache Util
60 40 20 0
.5 .5–1 0
0–0
1–2
2–4
4–8
≥8
(a) CV distribution of utilization
Figure 2. Coefficient of Variation (CV) analysis of GPU utilization and KV cache utilization.
To address these challenges, we present Valve, a productionfriendly online-offline colocation system that jointly bounds preemption latency and preemption rate, while imposing negligible interference with online services. Valve builds on three key ideas:
2. Background We first analyze why GPU utilization is low in production by characterizing workload burstiness and SLA requirements. We then derive key requirements for production-friendly online–offline inference colocation, review existing systems and their limitations, and outline the main challenges.
(1) Channel-controlled compute isolation. Valve uses GPU channel control to preempt and recover offline execution within sub-millisecond latency. Combined with online request lifecycle awareness, Valve gates offline kernels outside the lifetime of an online request, ensuring each online request is preempted at most once.
2.1. Low GPU Utilization of Production Workloads Production LLM inference is bursty in both compute and KV-cache usage. To meet strict latency SLAs, online services reserve peak headroom, leaving GPUs underutilized on average.
(2) Sub-layer memory reclamation with dynamic reservation. Valve reclaims KV cache promptly by coordinating memory reclamation with compute preemption. After preempting compute, Valve remaps reclaimed pages to a quarantine page and exposes invalidated page IDs to the framework for recomputation, preventing unrecoverable page faults. Valve further regulates reclamation rate by dynamically reserving memory for online workloads.
Burstiness in compute and KV-cache memory. Each request uses GPU compute and allocates KV cache throughout its lifecycle. As a result, compute utilization often switches between idle and fully busy. Meanwhile, KV cache grows with the number of concurrent context tokens, and can spike under batch arrivals. Figure 2 measures burstiness across workloads using CV and Figure 3 shows two typical patterns: some workloads are bursty in both compute and KV cache, while other workloads are bursty mostly only in compute.
(3) Throughput-aware scheduling. A burst and multi-GPU aware scheduler models offline throughput on harvested GPUs and assigns offline workloads, meeting their SLAs. We build Valve as a production-friendly system consisting of a node-level runtime and a cluster-level scheduler. Valve requires only a one-line driver modification and 20-line framework patching, making it easy to deploy in production. The main contributions of this paper are summarized as follows:
Production workloads and their SLAs. Production inference includes both online and offline workloads. Online inference is user-facing (or latency-critical stages in posttraining) and must meet strict latency SLAs, so it can tolerate almost no interference. Offline inference often requires only throughput SLAs, or has no SLA. To meet online SLAs under bursty demand, operators overprovision GPUs, which leads to low average utilization.
• We design a production-friendly runtime that enables sub-millisecond compute preemption at most once per online request, and sub-layer memory reclamation with a rate-bounded reclamation frequency.
2.2. Key Requirements for Inference Online-Offline Colocation in Production
• We develop a burst- and multi-GPU-aware scheduler that places offline jobs smartly on harvested GPUs to meet throughput SLAs while improving utilization.
Extremely low interference for online workloads. To meet strict SLAs for real-time online inference, the system must introduce almost no extra delay. Interference comes from both compute and memory effects, and is driven by how long each preemption lasts and how often preemptions happen. Thus, the key goal is to bound both preemption latency and preemption rate.
• We deploy Valve in a production cluster with 8,054 GPUs, improving average utilization by 34.6%, which translates to saving 2,170 GPUs. Across workloads, Valve incurs < 5% TTFT increase and < 2% TPOT increase. 2
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate Table 1. Comparison of schemes for online-offline colocation.
Utilization
1.0
GPU Util.
Conserve
Gpreempt
Valve
Iteration-level Frequent 0 LOC 0 LOC
Layer-level Layer-level >5000 LOC 0 LOC
Frequent Not handling 0 LOC >200 LOC
< 1ms per-request Sub-layer, limited rate <20 LOC 1 LOC
KV Cache Util.
0.4
0.5
0.2
0.0 1.0
0.0 0.4
0.5
0.2
0.0 0
5 Time (min)
10 0
5 Time (min)
CDF
Compute Interference Memory Interference Framework Modifications Driver Modifications
TGS
1.0 0.8 0.6 0.4 0.2 0.0
10
1
10
2
Gap interval CDF (us)
Figure 4. Distribution of gap interval between decode iterations.
0.0 10
Figure 3. GPU utilization and KV cache utilization time series for user-facing inference (bottom) and reward model inference (top). They show different degrees of burstiness in KV cache utilization.
offline throughput. Gpreempt (Fan et al., 2025) uses a CUDA-driver timeslice for automatic switching, but the decode phase has short gaps between iterations (Figure 4), so it may run offline kernels once per iteration, causing frequent preemptions and increased queue length.
Minimal modifications to drivers and frameworks. Production deployment requires minimal modifications to GPU drivers and inference frameworks. Extensive modifications increase maintenance burden and limit broader adoption.
Memory-side KV-cache isolation. vAttention, Conserve, vTensor, and Prism virtualize KV-cache placement via VMM indirection and resize memory footprints on demand (Prabhu et al., 2025; Qiao et al., 2024; Xu et al., 2024; Yu et al., 2025). However, they do not fully solve how to reclaim offline KV memory quickly and safely when online demand spikes. For example, Conserve (Qiao et al., 2024) can reclaim KV cache only at transformer-layer boundaries, which can delay reclamation by up to hundreds of milliseconds during long prefills. Moreover, prior work largely does not discuss how to control the reclamation frequency. In practice, inference workloads change their memory regions and sizes over time; naively relying on UVM (Harris, 2017) or aggressively sharing memory can trigger reclamation repeatedly, causing severe interference to online workloads.
Throughput SLAs for offline workloads. The system should place offline workloads on suitable nodes to meet throughput SLAs, while maximizing cluster throughput. 2.3. Existing Solutions for Online-Offline Colocation A complementary line of work colocates latency-critical online inference with offline jobs on the same GPU, to backfill idle capacity while preserving strict online SLAs (Wu et al., 2023; Han et al., 2022; Qiao et al., 2024; Fan et al., 2025; Shen et al., 2025; Prabhu et al., 2025; Xu et al., 2024; Yu et al., 2025). Table 1 compares these systems. In production, they commonly face two obstacles: (i) noticeable interference with online workloads (due to slow or frequent preemptions), and (ii) extensive driver/framework modifications that make deployment and maintenance hard.
Deployability. Many systems also fall short in deployability. Conserve (Qiao et al., 2024) requires extensive inference-framework changes, often involving thousands of lines of code (e.g., injecting checkpoints into inference code and adding new scheduling modules). REEF (Han et al., 2022) requires replacing the compiler toolchain with a custom compiler, which in turn demands major changes to the user container image. XSched (Shen et al., 2025) Lv3 and REEF only support idempotent operators, but all-reduce in multi-GPU inference and some linear attention kernels are not idempotent, making the whole CUDA graph unpreemptible. As a result, no existing approach simultaneously achieves low-interference, LLM-compatible preemption and production-friendly deployability.
Compute-side preemption. TGS (Wu et al., 2023), XSched (Shen et al., 2025) Lv2 intercepts kernel launches, but LLM serving often relies on CUDA Graphs, which bundle kernels of one inference iteration into a single graph, so preemption degrades to graph-level granularity. Conserve (Qiao et al., 2024) inserts checkpoints into inference code and preempts at the transformer-layer level, but long prefills in production batch inference (e.g., 32k tokens) can stretch layer-level preemption delay to hundres of milliseconds. The preempted iteration is duplicated, decreasing 3
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate Online User
Offline User
Injected Runtime
Offline Workload
Online Workload
GPU Colocation Runtime
Kernel
Mem Alloc
Channel-Controlled Compute Isolation (§4)
Sub-layer Memory Reclamation (§5)
Channel Op.
VMM Op.
GPU 1
GPU 2
Application streams
Offline Cluster Scheduler (§6)
GPU 3
Launch kernel
User-mode Driver API
Injected Runtime Kernel on stream
Enabled Channels Engines
runlist
Stream-channel mapping Kernel
ioctl
on channel
l l
Launch kernel on hardware Enable/disable channels
Disabled Channels
GPU 4
Figure 6. Channel control in the kernel launch path.
Figure 5. Valve architecture overview.
with fast sub-layer memory reclamation coordinated with compute preemption, and controls reclamation frequency via MIAD(Multiple Increase, Addictive Decrease)-style online reservation (§5). These mechanisms also preserve high offline throughput by harvesting most idle compute cycles; during memory reclamation, selective eviction affects fewer offline requests and safe sub-layer reclamation avoids terminating offline workloads.
2.4. Challenges We face three challenges in designing a production-friendly online-offline inference colocation system in production. Challenge 1: Gap-aware sub-millisecond compute interference. The system must achieve sub-millisecond compute preemption while not inserting offline wake-ups between online decode iterations. This requires controllable, fast offline switch, with awareness of online request lifecycles.
These mechanisms also preserve high offline throughput by harvesting most idles compute cycles, impacting less offline requests during memory preemptions, without terminating offline workloads during memory preemptions.
Challenge 2: Rate-limited sub-layer memory reclamation. Online bursts may require reclaiming offline KV cache immediately in sub-layer granualrity. However, swapping KV pages to CPU is too slow, while invalidating KV pages without coordination can cause illegal accesses. The system must reclaim memory promptly while not killing offline applications. Restricting the reclamation frequency with framework transparency is also critical.
At the cluster level, online workloads are submitted directly to GPUs, while offline workloads are submitted to the Cluster Scheduler (§6). The scheduler builds a comprehensive performance model of offline workloads on harvested GPUs, and schedules them to satisfy their throughput SLAs, which is specified as the fraction of the standalone throughput.
Challenge 3: Precise offline performance modeling. Offline throughput varies with online burstiness and multiGPU behaviors, requiring precise modeling and scheduling.
4. Channel-Controlled Compute Isolation We use workload-aware channel control to achieve two goals: sub-millisecond preemption latency and at most one preemption per online request. Channel control provides a fast and precise way to pause and resume offline execution, while workload awareness triggers preemption in time and avoids frequent preemptions.
3. Valve Overview We introduce Valve, an industrial system for online-offline inference colocation. Valve is designed to meet three goals: (1) low compute and memory interference to online workloads, (2) reliable throughput SLAs for offline workloads, and (3) minimal framework/driver modifications. Figure 5 shows the overall architecture.
4.1. GPU Channel Control Channels in the kernel launch path. Figure 6 shows the kernel launch path and where GPU channels fit in. A CUDA stream issues kernel launches through the user-mode driver, which submits work to a channel managed by the kernelmode driver (KMD) and the GPU. A process typically owns one or more channels, which are bound to specific compute engines. The GPU schedules channels using a hardwaremaintained runlist. Importantly, the KMD exposes standard ioctl interfaces to create, enable, and disable channels, and to submit work to channels. These ioctls (I/O control commands) make channels a practical control point for control-
At the node level, the GPU Colocation Runtime enables compute and memory sharing with low interference by jointly bounding preemption latency and rate. For compute, it limits online impact by providing sub-millisecond, infrequent kernel preemptions via channel control and workloadaware offline execution control (§4). For memory, it follows prior work (Prabhu et al., 2025) to share GPU memory through a global pool with coarse-grained handles and an allocate–release interface. It bounds memory interference 4
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
Compute Preemption
Framework Block Table
① Preempt Compute
Global Memory Pool
R1
R1
R2
R2
R2
R2
R3
R3
R3
R3
④ Notification of Invalidated Page
Handle 1 R1
② Remap to Shared Quarantine Page
R1
R1 R1
Handle 2
Handle 3
R2 R2
R2 R2
R3 R3
R4
R4
R5
R5
track whether the online workload is active. When the online workload transitions to busy, the runtime immediately issues channel-disable commands for all offline workloads on the node, so offline execution is paused promptly.
Allocate/Release
Handle 4
Handle 5 R4 R5
R3 R3
Waking up offline workloads. To bound preemption to at most once per online inference request, we do not re-enable offline workloads immediately when the online workload becomes idle, since short idle gaps can appear between decode iterations. Instead, we wake up offline workloads only after a cooldown interval Tcool during which the online workload stays continuously idle. We set Tcool to twice the maximum gap G between decode iterations, as measured by our runtime instrumentation. This avoids waking offline work in per-iteration gaps and ensures at most one preemption over the lifetime of an online request.
R4 R5
③ Selective Memory Reclamation
I. Sub-layer Memory Reclamation
II. Dynamic Online Memory Reservation
Figure 7. Sub-layer memory reclamation with dynamic online memory reservation.
ling offline compute. Channel control for portable compute isolation. Valve uses this ioctl-level control point to preempt and resume offline workloads with low latency. Specifically, it disables the offline workload’s channel to preempt execution and later reenables it to resume. On Pascal+ GPUs, disabling a channel triggers a hardware context save to an on-GPU contextsave buffer, so in-flight kernels can be safely restored after re-enable (e.g., registers and other on-chip state). These operations take effect within 1 ms, enabling fast preemption without waiting for kernel boundaries.
5. Sub-layer Memory Reclamation with Dynamic Reservation The main bottleneck of online-offline memory sharing is KVcache reclamation. When online memory demand spikes, the system may need to unmap and remap offline VMMmapped KV cache (Prabhu et al., 2025) on the online critical path. Valve addresses this with three techniques: (i) sublayer memory reclamation to make preemption fast, (ii) dynamic online memory reservation to reduce interference rate, and (iii) selective handle reclamation for higher offline throughput without adding interference.
A practical challenge is that these KMD ioctls require drivermanaged identifiers (e.g., GPU-client and channel handles) that are not exposed through CUDA APIs. We obtain these identifiers without modifying the driver by intercepting CUDA initialization ioctls: their arguments contain the GPU-client and channel identifiers, which consistently appear as matched pairs on Pascal+ drivers. Our colocation runtime records a mapping from each application to its (GPU-client, channel) handles, and then issues the corresponding disable/enable ioctls on demand.
Sub-layer memory reclamation. We first focus on reducing reclamation latency. A natural baseline (Xiang et al., 2025; Qiao et al., 2024) reclaims KV memory only at iteration boundaries or layer boundaries, because unmapping KV pages during kernel execution can lead to unrecoverable memory faults. However, this coarse granularity can make reclamation slow, especially when online bursts arrive during a long prefill iteration. prefill phase when online bursts arrive during offline prefill iterations.
Optimizing preemption latency for multi-GPU preemption. Naively issuing preemption ioctls on a multi-GPU node leads to latency that grows roughly linearly with the number of GPUs. The bottleneck is a shared write lock that the KMD holds while handling these ioctls across GPUs on the same node. We find this synchronization is not required for inference tasks. On Turing+ GPUs, the preemption command can be offloaded directly to the target device without taking the global lock, reducing kernel-space overhead. Accordingly, we apply a one-line driver modification which changes the flag used to bypass the lock and offload the command to the channel. With this modification, preemption latency on an 8-GPU node drops from > 5 ms to < 1 ms.
Valve enables safe sub-layer reclamation by coordinating compute and memory preemption (Figure 7). When online workloads need memory, we always disable offline compute first, ensuring no in-flight kernel can access pages being reclaimed. We then select a set of offline “evictor” memory handles, remap the virtual pages in those handles to a shared quarantine page, and finally reclaim the memory handles for online workloads. This avoids faults and allows the offline workload to resume later. Accesses to reclaimed pages can lead to incorrect intermediate tokens. To keep the behavior correct, Valve records the reclaimed KV page (block) IDs and exposes them through a small patch: a single callback that returns the invalidated IDs for each request after a decode step. Across vLLM, SGLang, and TensorRT-LLM, this integration touches at
4.2. On-GPU Offline Workload Scheduling Preempting offline workloads. Our runtime, injected into the online process, intercepts kernel-launch commands to 5
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
Algorithm 1 Valve selective memory reclamation
of online workloads. We formulate it as
Require: Number of handles k; handle set H (equal size); request cost C OST(r); impacted requests R EQS(h) Ensure: A handle subset S to reclaim 1: S ← ∅, E ← ∅ 2: for i = 1 to k do P 3: h∗ ← arg minh∈H\S r∈R EQS(h)\E C OST(r) 4: S ← S ∪ {h∗ } 5: E ← E ∪ R EQS(h∗ ) 6: end for 7: return S
Thrput(w,N ) = Pcompute,(w,N ) ·Pmemory,(w,N ) ·Pmulti,(w,N ) . Thrput(w,max) (1) Here, w denotes an offline workload and N denotes a node. Thrput(w,N ) is the effective throughput of w on N , and Thrput(w,max) is its throughput on the monopolized GPU. We define the three performance factors as follows. Idle compute fraction. We measure the idle compute fraction using the colocation runtime as the fraction of GPU timeslices available to run the offline workload.
most two scheduler-side functions and requires fewer than 20 lines of code changes. The framework then discards intermediate data for affected requests, returns them to the waiting state with only the input and previously generated tokens, and later resume them by recomputing. This achieves sub-layer reclamation without crashes.
Burstiness and average of memory. GPU memory determines both feasibility and throughput. For each workload w, we profile it once at submission to obtain a memory– throughput curve Thrputw (memory). Let M be the available memory on node N . Without eviction, the effective throughput is the time average of Thrputw (M ) over the node’s memory trace. When M dips below the workload’s required memory Mreq , the shirnk ∆M = max(0, Mreq − M ) introduces throughput loss. We use a workload-specific coefficient MACw to map the expected deficit to throughput loss. The memory factor is formulated as:
Dynamic MIAD-style memory reservation. To reduce reclamation frequency within a given budget while maximizing offline memory, Valve maintains a dynamic online KV-cache headroom H as pre-mapped VMM handles. Valve adapts H using MIAD (Multiplicative Increase, Additive Decrease): on an online pressure event (i.e., when H reaches 90% utilization), it multiplicatively increases H by a factor α to reserve more mapped handles in advance; when pressure is absent, it shrinks conservatively by releasing one handle every interval T . The release interval T is also MIAD-controlled: if the pressure-event rate over a sliding window exceeds the user-specified target, Valve multiplicatively increases T ; otherwise, it decreases T . This reservation drives the reclamation rate toward the target.
Pmemory,(w,N ) =
E[Thrputw (M )] − MACw ·E[∆M ] . Thrputw (Mmax ) (2)
The multi-card behavior. Online multi-GPU services often use GPUs asynchronously, so activity can be misaligned across cards; in our trace, 32% of instances show only partial overlap. In contrast, model-parallel offline inference runs in lockstep. Misalignment across cards then creates stragglers and idle gaps, reducing throughput and risking SLA violations. We quantify cross-card alignment with a pairwise ) score Pmulti,(w,N ) = TT∩∪ (N (N ) , where T∩ is overlapping busy time and T∪ is union busy time. At placement, we admit a k-GPU job only if all pairs satisfy Pmulti,(w,N ) ≥ 0.95.
Selective handle reclamation. The KV cache is not allocated continuously on the memory handle due to the memory fragment problem, so one handle can be shared by different numbers of offline requests. Reclaiming a handle blindly may preempt more requests than necessary. Valve uses selective handle reclamation (Algorithm 1) to minimize the number of affected offline requests. Specifically, it greedily selects handles with the lowest marginal token cost, defined as the total number of extra tokens incurred by the additional requests affected by reclaiming that handle. This improves offline throughput without increasing online interference.
Scheduling. Building on this model, our scheduler schedules offline workloads to nodes with guaranteed throughput SLA. Besides, a monitor periodically checks the past throughput of each offline throughput and evicts those that persistently violate their SLA for rescheduling.
6. Valve Cluster Scheduling 7. Evaluation
To satisfy offline throughput SLAs, we build a comprehensive performance model for offline LLM inference on harvested GPUs. We characterize a harvested GPU along three aspects: (i) idle compute fraction; (ii) the burstiness and average of memory usage; and (iii) the multi-GPU behavior
In this section, we first quantify the production impact of Valve (§ 7.1), then compare it with existing online-offline colocation approaches in terms of interference to online workloads and throughput of offline workloads (§ 7.2). 6
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
0 0
24
48
72
96 120 Hour (a) Improved GPU utilization
144
168
Figure 8. Cluster GPU utilization with Valve.
Deployment Scale
9000 7200 5400
Workload Percentage (%)
20
Workload Percentage (%)
40
100
100
Workload Percentage (%)
Precentage (%)
60
100
80
0
Nov 01
Dec 01 Jan 01 Date (b) Deployment scale growth
Figure 9. Deployment scale with Valve.
0–5% 5–10% 10–20% ≥20%
60 40 20 0 (b) TPOT increased percentage distribution
80
0–2% 2–5% 5–20% ≥20%
60 40 20 0
3600 1800
(a) TTFT increased percentage distribution
(c) Offline throughput distribution
80 40 20
0 m M M M sm Me UV UV UV Pri t el+ p+ pt+ atic el+ t n m n m n S a l+ ee an ree Ch ne lPr Ch GP an rne Ch Ke
7.1. Production Impact of Valve Deployment. Valve is deployed in production clusters with 8045 GPU cards. The clusters serve both online and offline inference workloads. Valve has been in production for more than three months, with the number of managed GPU cards increasing over time, illustrated by Figure 9.
≥0.95 0.8–0.95 0.5–0.8 <0.5
60
e
lv Va
Figure 10. Distribution over 10 workloads of (a) TTFT increase percentage (b) TPOT increase percentage and (c) offline throughput under each strategies. Stacked segments indicate the workload percentage in each bin.
Metrics. We use two metrics to quantify Valve’s end-to-end impact: (i) improved GPU utilization, which is the fraction of time when GPUs execute offline compute; and (ii) saved GPU cards. The amount of GPU cards saved by each colocated offline workload is computed as the throughput normalized by standalone offline throughput.
throughput, which is the throughput divided by the throughput under no-memory-preemption, i.e. Prism with our compute preemption (Channel+Prism). Methodology. We sample 10 online workload and offline workload pairs from production deployments, and replay them in our test cluster with different colocation strategies.
Results. Figure 8 shows improved GPU utilization in the production cluster over one week, with an average of 34.6% improvement. The inference work done by the offline workloads translates to a saving of 2170 GPU cards.
Baseline techniques. We consider two orthogonal design dimensions—compute preemption and memory preemption. Compute preemption. (1) KernelPreempt: kernel-level preemption—switch at kernel boundaries, adopted by TGS (Wu et al., 2023). With CUDA graphs, kernel boundaries align with iteration steps, resulting in coarse-grained preemption. (2) GPreempt: immediate online workload preemption via setting short time slice for offline workloads and long time slice for online workloads, proposed by GPreempt (Fan et al., 2025). (3) Channel: our channel-based preemption with workload awareness (§4).
7.2. Comparison with Alternative Colocation Approaches Metrics. We evaluate both the interference to online workloads and the throughput of offline workloads using three metrics: • TTFT increase percentage: The increase percentage of online workloads under each strategy, compared with standalone running TTFT.
Memory preemption. (1) UVM (Harris, 2017): allocate normal device memory for online workloads and use CUDA Unified Virtual Memory for offline workloads, which allows the online workloads to reclaim offline memory as needed. (2) Prism (Yu et al., 2025): share memory between online and offline workloads via CUDA VMM. (3) StaticMem: statically allocate unused memory to offline workloads via the CUDA VMM. The minimum free memory over the past
• TPOT increase percentage: The increase percentage of online workloads under each strategy, compared with standalone running TPOT. • Offline throughput: To compare the throughput of different strategies clearly, we report the normalized 7
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
Throughput (token/s)
hour is used as the offline limit; online bursts above this kill offline workloads immediately. (4) OurMem: our memory preemption(§5)—sub-layer reclamation and MIAD-style dynamic memory reservation for low reclamation rate. Baseline combinations. We evaluate combinations of the above techniques as baselines to assess how Valve’s preemption mechanisms reduce interference while sustaining high offline throughput. We (i) compare KernelPreempt+UVM, Gpreempt+UVM, and Channel+UVM to show the effects of our compute preemption; (ii) compare Channel+UVM, Channel+Prism, Channel+StaticMem, and Valve to show the effects of our memory preemption.
FIFO
1000
Valve
750 500 250 0
10G 20G 10G 20G 10G 20G 0.2/m 0.2/m 0.4/m 0.4/m 0.8/m 0.8/m
Figure 11. Effectiveness of the eviction selection.
8. Related Work Autoscaling. Serverless autoscaling systems (e.g., BlitzScale, ServerlessLLM, and HydraServe) unload model weights when idle and reload them on demand (Zhang et al., 2025; Lou et al., 2025; Fu et al., 2024). This reduces steadystate GPU memory footprint, but the on-demand reload can introduce cold-start delays (e.g., TTFT spikes) under highly bursty traffic, making it hard to satisfy strict latency SLAs. These approaches are complementary to Valve and are suitable when bursts are milder or SLOs are less stringent.
Results. Figure 10 illustrates the results. Valve keeps the online TTFT increase within 5% and TPOT increase within 2% across all workloads, which is significantly lower than all baselines. Meanwhile, Valve maintains similar offline throughput to Channel+Prism where offline KV cache is not reclaimed, and significantly outperforms UVM-based baselines and static memory allocation baselines. For compute preemption, KernelPreempt suffers from large single preemption latency in all cases since CUDA graphs make preemption iteration-level; Gpreempt incurs frequent preemptions because an offline wakeup and preemption happens after each inference iteration. This harms TPOT and increases queue length, which in turn increases TTFT. In contrast, Valve’s channel-based control with workload awareness bounds both preemption latency and rate, achieving low compute interference.
Multiplexing. Several systems multiplex multiple models on one GPU via temporal or spatial sharing (NVIDIA, 2026a; Li et al., 2023; Duan et al., 2024; Xiang et al., 2025; Patke et al., 2024; NVIDIA, 2026b; Ghodrati et al., 2020). They typically target relaxed or best-effort SLAs: under contention, workloads queue for compute and memory, leading to higher latency. Valve instead targets online–offline colocation by jointly bounding interference latency and rate.
For memory preemption, Channel+UVM preempts often: UVM lets offline workloads fill spare online memory, which is reclaimed whenever online demand spikes; it also cannot use memory already allocated by online workloads, limiting offline throughput. Channel+Prism does not reclaim memory, forcing online batch-size reduction and more queueing, increasing TTFT significantly in 4 sampled workloads. Channel+StaticMem shows low interference but static allocation underutilizes memory, yielding 9%–100% lower offline throughput in the 4 sampled online workloads with bursty memory usage. Valve utilizes MIAD-style dynamic reservation and sub-layer fast reclamation for low memory interference while keeping high offline throughput.
LLM inference systems. Prior work improves online LLM serving via scheduling and memory management (Kwon et al., 2023; Agrawal et al., 2023; Sun et al., 2024; Sheng et al., 2024). These systems focus on optimizing the serving stack (e.g., batching, KV-cache efficiency, and tail-latency control) and are orthogonal to them by providing boundedinterference for online–offline inference colocation. Cluster scheduling. Interference-aware CPU schedulers (Verma et al., 2015; Schwarzkopf et al., 2013; Delimitrou & Kozyrakis, 2013; 2014) and ML schedulers (Xiao et al., 2018; Gu et al., 2019; Xiao et al., 2020; Crankshaw et al., 2017; Gujarati et al., 2020; Crankshaw et al., 2020) improve cluster utilization. Valve is orthogonal by solving new challenges in inference colocation for high utilization.
Effectiveness of Valve eviction policy. We further evaluate our eviction policy under varying reclamation rate and reclaimed memory size. We compare against a FIFO baseline, which evicts offline KV cache blocks in first-in-first-out order. We colocate a 7B online model with a 7B offline model. As shown in Figure 11(a), by targeting blocks tied to fewer in-flight offline requests, our policy consistently reduces throughput loss by 22.9%–40.1% over FIFO.
9. Conclusion We present Valve, a production-friendly online–offline colocation system that jointly bounds preemption latency and preemption rate. Valve combines channel-controlled compute isolation with safe, rate-limited sub-layer memory reclamation, achieving both low preemption latency and rate. Deployed on 8,054 GPUs, Valve improves average 8
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
cluster utilization by 34.6% and saves 2,170 GPUs, while incurring < 5% TTFT increase and < 2% TPOT increase across workloads.
Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving {DNNs} like clockwork: Performance predictability from the bottom up. In USENIX OSDI, 2020.
References
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023. Anthropic. Introducing Claude, https://www.anthropic.com/index/ introducing-claude.
Han, M., Zhang, H., Chen, R., and Chen, H. Microsecondscale preemption for concurrent gpu-accelerated dnn inferences. In USENIX OSDI, 2022.
2023.
Harris, M. CUDA Unified Memory, 2017. https://developer.nvidia.com/blog/ unified-memory-cuda-beginners/.
Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., and Stoica, I. Clipper: A {Low-Latency} online prediction serving system. In USENIX NSDI, 2017.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Crankshaw, D., Sela, G.-E., Mo, X., Zumar, C., Stoica, I., Gonzalez, J., and Tumanov, A. Inferline: latency-aware provisioning and scaling for prediction serving pipelines. In ACM Symposium on Cloud Computing, 2020.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Delimitrou, C. and Kozyrakis, C. Paragon: Qos-aware scheduling for heterogeneous datacenters. ACM SIGPLAN Notices, 48(4):77–88, 2013. Delimitrou, C. and Kozyrakis, C. Quasar: Resourceefficient and qos-aware cluster management. ACM Sigplan Notices, 49(4):127–144, 2014.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In ACM SOSP, 2023.
Duan, J., Lu, R., Duanmu, H., Li, X., Zhang, X., Lin, D., Stoica, I., and Zhang, H. Muxserve: Flexible spatialtemporal multiplexing for multiple llm serving. arXiv preprint arXiv:2404.02015, 2024.
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319–18345. PMLR, 2023.
Fan, R., Ren, T., Xie, M., Gao, S., Shu, J., and Lu, Y. {GPREEMPT}:{GPU} preemptive scheduling made general and efficient. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), pp. 263–272, 2025.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
Fu, Y., Xue, L., Huang, Y., Brabete, A.-O., Ustiugov, D., Patel, Y., and Mai, L. {ServerlessLLM}:{Low-Latency} serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 135–153, 2024.
Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In USENIX OSDI, 2023.
Ghodrati, S., Ahn, B. H., Kim, J. K., Kinzer, S., Yatham, B. R., Alla, N., Sharma, H., Alian, M., Ebrahimi, E., Kim, N. S., et al. Planaria: Dynamic architecture fission for spatial multi-tenant acceleration of deep neural networks. In IEEE/ACM MICRO, 2020.
Liu, Z., Fang, F., Feng, X., Du, X., Zhang, C., Wang, N., Zhao, Q., Fan, L., GAN, C., Lin, H., et al. Ii-bench: An image implication understanding benchmark for multimodal large language models. Advances in Neural Information Processing Systems, 37:46378–46480, 2024.
Gu, J., Chowdhury, M., Shin, K. G., Zhu, Y., Jeon, M., Qian, J., Liu, H., and Guo, C. Tiresias: A GPU cluster manager for distributed deep learning. In USENIX NSDI, 2019. 9
Valve: Production Online–Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
Lou, C., Qi, S., Jin, C., Nie, D., Yang, H., Liu, X., and Jin, X. Towards swift serverless llm cold starts with paraserve. arXiv preprint arXiv:2502.15524, 2025.
Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., and Wilkes, J. Large-scale cluster management at Google with Borg. In EuroSys, 2015.
NVIDIA. CUDA Multi-Process Service, 2026a. https://docs.nvidia.com/deploy/mps/ index.html (accessed 2026-04-09).
Wu, B., Zhang, Z., Bai, Z., Liu, X., and Jin, X. Transparent {GPU} sharing in container clouds for deep learning workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 69– 85, 2023.
NVIDIA. ”nvidia multi-instance gpu (mig)”, 2026b. ”https://docs.nvidia.com/datacenter/ tesla/mig-user-guide/ (accessed 2026-04-09)”. OpenAI. Introducing ChatGPT, 2022. https://openai. com/blog/chatgpt.
Wu, C. Hybridflow: A flexible and efficient rlhf framework. EuroSys 2025 (30/03/2025-03/04/2025, Rotterdam), 2025. Xiang, Y., Li, X., Qian, K., Yang, Y., Zhu, D., Yu, W., Zhai, E., Liu, X., Jin, X., and Zhou, J. Aegaeon: Effective gpu pooling for concurrent llm serving on the market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pp. 1030–1045, 2025.
Patke, A., Reddy, D., Jha, S., Qiu, H., Pinto, C., Narayanaswami, C., Kalbarczyk, Z., and Iyer, R. Queue management for slo-oriented large language model serving. arXiv preprint arXiv:2407.00047, 2024. Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., and Panwar, A. vattention: Dynamic memory management for serving llms without pagedattention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pp. 1133–1150, 2025.
Xiao, W., Bhardwaj, R., Ramjee, R., Sivathanu, M., Kwatra, N., Han, Z., Patel, P., Peng, X., Zhao, H., Zhang, Q., et al. Gandiva: Introspective cluster scheduling for deep learning. In USENIX OSDI, 2018. Xiao, W., Ren, S., Li, Y., Zhang, Y., Hou, P., Li, Z., Feng, Y., Lin, W., and Jia, Y. Antman: Dynamic scaling on GPU clusters for deep learning. In USENIX OSDI, 2020.
Qiao, Y., Anzai, S., Yu, S., Ma, H., Wang, Y., Kim, M., and Xu, H. Conserve: Harvesting gpus for low-latency and high-throughput large language model serving. arXiv preprint arXiv:2410.01228, 2024.
Xu, J., Zhang, R., Guo, C., Hu, W., Liu, Z., Wu, F., Feng, Y., Sun, S., Shao, C., Guo, Y., et al. vtensor: Flexible virtual tensor management for efficient llm serving. arXiv preprint arXiv:2407.15309, 2024.
Ruan, Z., Park, S. J., Aguilera, M. K., Belay, A., and Schwarzkopf, M. Nu: Achieving {Microsecond-Scale} resource fungibility with logical processes. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 1409–1427, 2023.
Yu, S., Xing, J., Qiao, Y., Ma, M., Li, Y., Wang, Y., Yang, S., Xie, Z., Cao, S., Bao, K., et al. Prism: Unleashing gpu sharing for cost-efficient multi-llm serving. arXiv preprint arXiv:2505.04021, 2025.
Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., and Wilkes, J. Omega: flexible, scalable schedulers for large compute clusters. In EuroSys, 2013.
Zhang, D., Wang, H., Liu, Y., Wei, X., Shan, Y., Chen, R., and Chen, H. {BlitzScale}: Fast and live large model autoscaling with o (1) host caching. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pp. 275–293, 2025.
Shen, W., Han, M., Liu, J., Chen, R., and Chen, H. {XSched}: Preemptive scheduling for diverse {XPUs}. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pp. 671–692, 2025.
Zhong, Y., Zhang, Z., Wu, B., Liu, S., Chen, Y., Wan, C., Hu, H., Xia, L., Ming, R., Zhu, Y., et al. Optimizing {RLHF} training for large language models with stage fusion. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pp. 489–503, 2025.
Sheng, Y., Cao, S., Li, D., Zhu, B., Li, Z., Zhuo, D., Gonzalez, J. E., and Stoica, I. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 965–988, 2024. Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., and Lin, W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 173–191, 2024. 10