DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs Jiaxuan Chen
Jianshu She
Ye Yuan
McGill University Canada
MBZUAI United Arab Emirates
McGill University Canada
Rajat Ghosh
Karan Gupta
Qirong Ho
Nutanix, Inc. USA
Nutanix, Inc. USA
MBZUAI United Arab Emirates
Xue Liu
Oana Balmau
McGill University Canada
McGill University Canada
Nutanix Production Inference Workload Trace
1
20
1000
10
0
Requests/sec
Tokens/sec
LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. DeltaServe encapsulates its co-serving mechanisms behind a compact hook interface that requires only multi-LoRA batching support. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Nutanix, DeltaServe on vLLM delivers 2.9× higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.
2000
GPU Utilization (%)
arXiv:2607.28848v1 [cs.DC] 30 Jul 2026
Abstract
0
100
200
400
600
800
1000
GPU Utilization Under Production Workload
60.6%
50 0
0 1200
0
200
400
600
Time (s)
800
1000
1200
Figure 1. Inference fluctuation in a production workload (Nutanix). On average, 40% of the GPU is unused.
a substantial fraction of GPU compute idle when inference is the only workload running. Prior work has approached this underutilization from two angles. The first improves the inference pipeline itself: Splitwise [26] disaggregates prefill and decode across machines and Orca [38] mixes prefill and decode requests within the same batch. These techniques raise efficiency under a fixed request stream but do not recover GPU cycles that sit idle when the request stream itself falls below peak. The second co-locates an independent workload on the same GPU using classic sharing primitives such as temporal time-slicing or spatial partitioning (e.g., NVIDIA MIG and MPS). However, these mechanisms operate at granularities far too coarse to track the sub-second fluctuations of inference demand without violating latency targets. A more promising direction is to use idle inference capacity for a workload that many deployments already need but cannot always provision separately: continual model adaptation. LLM services often need to incorporate new data, evaluate model or adapter variants, and specialize behavior for different products, users, or tenants. For teams with limited GPU budgets, dedicating separate accelerators to this
Introduction
Production LLM serving infrastructure is provisioned for peak inference load so that interactive latency targets can be met during traffic spikes [18]. Consequently, outside of those spikes the same hardware sits substantially underutilized. For instance, Figure 1 shows a representative twenty-minute window from the production inference trace of Nutanix, running a co-pilot style coding service. The average load is roughly 60% of provisioned capacity, and the same pattern persists across six months of collected traces. More broadly, the combination of bursty request arrivals [38] and the memory-bandwidth-bound decode phase [14, 30] leaves 1
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau
fine-tuning pipeline can directly reduce the capacity available for serving, forcing a trade-off between keeping models up to date and meeting inference latency targets. Parameter-efficient fine-tuning (PEFT) [21], and Low-Rank Adaptation (LoRA) [13] fine-tuning in particular, is well matched to this setting because it shares most of its execution state with inference. With LoRA, the base model remains frozen and only a small adapter is updated, so inference and fine-tuning share frozen weights, kernels, and memory. The forward pass of a LoRA fine-tuning step is structurally the same as inference prefill, except that fine-tuning must preserve activations for the later backward pass [23]. Two recent systems explore this opportunity. LLMStation [12] co-executes PEFT forward work with inference decode iterations to exploit decode’s memory-bandwidth-bound compute slack. FlexLLM [25] interleaves the two workloads at the level of token chunks, slicing fine-tuning sequences and running each chunk alongside inference tokens. Both demonstrate that LoRA co-serving is feasible while preserving inference SLOs, but each leaves room for improvement: LLMStation’s decode fusion is constrained by tight per-step time-peroutput-token (TPOT) budgets and a latency cache that does not distinguish CUDA-graph from eager execution, while FlexLLM’s chunked training pays substantial per-chunk datamovement overhead. Exploiting this opportunity well requires fine-grained control across scheduling, execution, and memory management. The first challenge is that fine-tuning forward and backward computation consume the same compute and memory bandwidth that inference relies on, so uncontrolled admission inflates time-to-first-token (TTFT) for new requests and timeper-output-token (TPOT) for ongoing decodes [14, 30, 38]. Second, backward computation introduces long-running GPU work that must be interleaved with inference without disrupting the kernel-launch and execution efficiency of the latency-critical serving pipeline. Third, inference load fluctuates at sub-second timescales, so fine-tuning must be admitted, throttled, paused, and resumed at a finer granularity than existing co-serving systems support. A fourth challenge is practical rather than algorithmic: the technique should be easy to add to an existing inference engine, not delivered as a new serving system that replaces it. Production inference already runs on mature, heavily optimized engines, and these engines differ widely in how they schedule and execute requests. A co-serving technique that is tied to one engine’s internals, or that requires substantial re-engineering to adopt, is unlikely to be used in practice. We present DeltaServe, a host-agnostic co-serving design that augments an existing inference engine with LoRA fine-tuning under explicit SLO control. DeltaServe is not a standalone serving system but a design that can be incorporated into an existing engine. Its co-serving logic, comprising the SLO latency model, the admission policy, and the decoupled backward executor, is host-agnostic and is
exposed through a compact interface of integration hooks. The only capability DeltaServe requires of the host engine is multi-LoRA batching, which modern serving systems already support in order to serve many adapters over a single frozen base model. We integrate DeltaServe into three such engines spanning the range of serving stacks it might attach to: vLLM and SGLang, two state-of-the-art inference engines widely deployed in industry, and S-LoRA, a popular research serving system built around multi-LoRA batching, the one capability DeltaServe requires. Each integration reuses the same host-agnostic core and rewrites only the engine-specific hooks. DeltaServe exploits the structural identity between inference prefill and the LoRA fine-tuning forward pass: the two perform the same computation, so fine-tuning folds into ongoing inference work rather than running as an unrelated job. At the core of DeltaServe is an SLO-aware admission scheduler built on this observation, driven by an analytical model that predicts how much latency the fine-tuning would add to the inference workload at the batch-granularity before it is admitted. This prediction gives DeltaServe the fine-grained control prior co-serving systems lack: it admits, throttles, and pauses fine-tuning as load shifts, always within the latency budget of in-flight and queued requests, thereby preserving TTFT and TPOT targets. The backward pass runs in a separate GPU process that gives way to inference whenever inference demands more compute, to let fine-tuning consume only otherwise-idle capacity. In summary, the contributions of this paper are: • A host-agnostic co-serving design. DeltaServe decomposes SLO-aware co-serving of inference and fine-tuning into a reusable, host-agnostic core and a compact set of integration hooks, requiring only multi-LoRA batching from the host engine; we showcase the same core on vLLM [14], SGLang [41], and S-LoRA [29]. • Design and implementation of DeltaServe, consisting of the following three novel technical contributions: (1) SLO-aware fine-tuning admission. Building on the structural identity between inference prefill and the LoRA forward pass, DeltaServe incorporates fine-tuning into ongoing inference batches and admits it against per-request TTFT and TPOT budgets, preserving inference SLOs. (2) A CUDA-graph-aware latency model. A light analytical model, calibrated offline and refined online, predicts per-step execution time under both graph and eager execution and provides the basis for admission decisions. (3) A decoupled backward executor. Fine-tuning backpropagation executes in a separate GPU subprocess that yields to inference at each transformer-layer boundary and is preempted upon the arrival of new requests, consuming only otherwise-idle compute. • Evaluation. We implement DeltaServe on three serving engines, vLLM, SGLang, and S-LoRA, confirming that the 2
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
same host-agnostic core ports across them, and we release it as open source. On a production trace from Nutanix, DeltaServe-vLLM sustains 2.9× the fine-tuning throughput of LLMStation, the state-of-the-art co-serving system, and 39.84% more than a split-pool baseline that dedicates a separate GPU to fine-tuning, all at full SLO compliance and without additional hardware.
2
Background and Related Work
2.1
ML Serving and Inference Systems
layers of the base model, typically the attention and feedforward projections. Given a frozen weight matrix𝑊 ∈ R𝑑 ×𝑘 , LoRA represents the trainable update as Δ𝑊 = 𝐵𝐴, where 𝐴 ∈ R𝑟 ×𝑘 and 𝐵 ∈ R𝑑 ×𝑟 with rank 𝑟 ≪ min(𝑑, 𝑘). The effective weight becomes 𝑊eff = 𝑊 + 𝐵𝐴, while only the lowrank factors 𝐴 and 𝐵 are updated during fine-tuning. Since these factors are much smaller than 𝑊 , LoRA substantially reduces the memory required for gradients and optimizer states compared with full-parameter fine-tuning. PEFT also changes model serving: instead of deploying one full model per task, a serving system can host many lightweight adapters over a shared frozen base model. A naive implementation runs one forward pass per adapter, preventing efficient batching. Punica [4] and S-LoRA [29] address this with multi-LoRA batching, where tokens using different adapters share one base-model forward pass while applying adapter-specific low-rank weights. This makes multi-adapter LoRA serving a natural substrate for co-serving: fine-tuning samples can be represented as adapter-bound requests and processed through the same adapter-aware batching and forward-execution pipeline as inference.
LLM inference is autoregressive: output tokens are generated one at a time, each conditioned on the prompt and all previously generated tokens. Serving engines advance generation in discrete steps, or iterations, where each step runs one forward pass over the current batch and emits one token per active request. Each request proceeds through two phases with distinct resource profiles [14, 26]. In prefill, the engine processes the full prompt to produce the first output token and initialize the key–value (KV) cache. This full-sequence computation is primarily compute-bound [8]. In decode, each step processes only the newest token but reads the accumulated KV cache, making it lower in arithmetic intensity and often memory-bandwidth-bound [30]. These phases correspond to the main latency objectives: time-to-first-token (TTFT), largely driven by prefill, and time-per-output-token (TPOT), determined by repeated decode steps. Inference serving engines are part of a large body of systems research targeting efficient ML serving at scale. Earlier frameworks [7, 10, 11, 15, 27] address latency modeling, dynamic scaling, memory swapping [2], preemptive scheduling [6], workload prediction [40], and cost-oriented provisioning [39], while Nexus [28] and Gpulet [5] add finergrained batching and GPU virtualization for small-model deployments. More recent LLM serving stacks, including vLLM [14], Sarathi [1], SGLang [41], AlpaServe [18], and Punica [4], optimize decoder-heavy traffic through continuous batching, KV-cache management, and customized PEFT kernels, and others add fairness-aware [35] or speculativedecoding [3, 17, 20, 22, 24] scheduling. DeltaServe does not compete with these stacks but builds on them, treating engines such as vLLM and SGLang as hosts, reusing their inference pipeline, and adding only the machinery to co-serve fine-tuning. 2.2
2.3 GPU Resource Sharing for ML Efficient sharing of GPU resources across heterogeneous ML workloads has also been widely studied. Systems such as Lyra [16] focus on dynamically resizing GPU allocations to match workload behavior, whereas Gandiva [36] and AntMan [37] employ time-slicing to improve cluster-level utilization. GSLICE [9] introduces adaptive partitioning mechanisms to accommodate different latency SLOs. While these approaches substantially improve multiplexing for classic training and inference pipelines, they do not directly address the unique characteristics of PEFT-based LLM services, where numerous lightweight models share a large frozen backbone, nor do they fully resolve the resource inefficiencies observed in modern multi-adapter serving environments. 2.4
Co-Serving Systems
Co-serving systems for LLMs execute latency-sensitive inference and fine-tuning concurrently on the same GPU resources. For LoRA-based LLMs, this is attractive because both workloads share the frozen base model, while fine-tuning updates only lightweight adapter parameters. However, supporting fine-tuning inside an inference serving system requires more than admitting extra GPU work. It introduces training data queues, activation storage, backward execution, optimizer updates, and adapter-weight synchronization, requiring scheduling and execution structures absent from inference-only pipelines. LLMStation [12] is the closest concurrent co-serving system to DeltaServe, differing in both scope and mechanism. In scope, it is a serving system built on vLLM, whereas DeltaServe is a host-agnostic co-serving design that reuses the host’s existing serving pipeline as much as possible
Parameter-Efficient Fine-Tuning
Full-parameter fine-tuning of large language models is expensive because gradients and optimizer states must be maintained for billions of parameters. Parameter-efficient finetuning (PEFT) [21] reduces this cost by freezing the base model and training only small additional modules. Low-Rank Adaptation (LoRA) [13] is a widely used PEFT method that injects trainable low-rank updates into selected 3
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau
and adds fine-tuning through a compact set of hooks, so it adapts to mainstream serving engines; we realize it on vLLM, SGLang, and S-LoRA. In mechanism, both co-serve LoRA fine-tuning under inference SLOs, but differ in how fine-tuning is integrated and how its latency is predicted. LLMStation confines fine-tuning to decode-phase headroom, where each admission must fit decode’s tight pertoken budget. DeltaServe instead admits fine-tuning in any phase, modeling each sample as a single-step, prefill-only request (Section 3), so it can admit against the full inference latency budget. The two also predict latency differently: LLMStation relies on a per-shape latency cache that ignores the graph-versus-eager distinction, whereas DeltaServe uses an execution-mode-aware analytical model that needs no cache and prices that distinction directly. FlexLLM [25] is a system that interleaves inference and fine-tuning at token granularity [19], slicing fine-tuning sequences into chunks and running each chunk’s forward and backward computation alongside inference tokens. This finegrained overlap also incurs substantial data movement: each chunk reloads the full model from GPU memory, and when a chunk is left unsplit to avoid that cost, an in-flight finetuning step can occupy the pipeline and stall the TTFT of newly arrived requests. DeltaServe takes a different position, retaining full-sequence fine-tuning forward passes and treating them as prefill-like work rather than decomposing them across tokens; it thus operates at a coarser but more latency-stable granularity that keeps inference transparent while still harvesting idle GPU capacity. These systems show that LoRA fine-tuning can be colocated with online inference, but also expose a practical limitation: co-serving often requires substantial changes to the serving pipeline. DeltaServe targets this integration problem by reusing the existing LoRA serving path for fine-tuning forward computation, deferring backward execution outside the critical inference path, and using an accurate SLO-aware scheduler that is independent of a particular host-engine structure. It admits fine-tuning only when the predicted impact remains within TTFT and TPOT budgets. More broadly, unlike LLMStation and FlexLLM, which define their own co-serving execution structures, DeltaServe attaches to existing engines through compact hooks while preserving the host scheduler and optimized execution path.
2.5
as a full graph because its per-step shape is relatively stable. Prefill is more difficult because prompt lengths vary, so these systems use piecewise graph execution: shape-stable parts of the forward pass are captured and replayed, while variable-length attention runs eagerly.
3
System Design
The background discussion shows that LoRA co-serving is appealing because inference and fine-tuning share the frozen base model, but realizing it inside an existing serving engine is difficult: fine-tuning needs training-specific control, activation capture, backward execution, and optimizer updates that inference engines are not designed to provide. DeltaServe addresses this gap by separating what must be added for finetuning from what should remain under the host system’s control. Its design keeps the host responsible for request handling, batching, KV-cache management, sampling, and optimized forward execution, while DeltaServe adds only the mechanisms needed to turn admitted fine-tuning samples into prefill-like batch entries and to process their backward pass outside the critical inference path. This section describes how DeltaServe implements this separation. Section 3.1 first presents the host interface and the add-on components that attach to a generic inference engine. The following sections then describe how fine-tuning samples are executed through the host forward path, how the scheduler estimates SLO headroom and admits fine-tuning work, and how the backward subprocess performs training updates while yielding to latency-critical inference. 3.1
DeltaServe Overview
Figure 2 illustrates DeltaServe as an extension around a generic LLM inference engine. The white boxes represent the common structure of modern serving systems: a client-facing request queue, a scheduler, and a GPU execution engine that executes the model forward pass. We use this abstraction because these components appear across serving engines despite their internal implementations differ. DeltaServe therefore defines its interface at the boundary of these components rather than depending on a particular runtime. Under this interface, DeltaServe does not replace the serving pipeline. The host remains responsible for request handling, scheduling, and GPU execution, and DeltaServe reuses the engine’s optimized serving path, including KVcache management, inference kernels, batching logic, sampling, and inference optimizations such as CUDA-graph replay when available. The only semantic capability DeltaServe requires from the host is multi-LoRA batching, which allows a reserved training adapter to share the same forward path as inference adapters. The blue components in Figure 2 mark the DeltaServe components that extend the host pipeline:
CUDA Graphs
A CUDA graph records a sequence of GPU kernel launches and replays it with a single host-side submission. This removes per-kernel launch overhead, which is significant for short or memory-bound kernels and especially important for LLM decode, where each step performs limited computation but invokes many kernels. Modern LLM serving systems such as vLLM [14] and SGLang [41] therefore use CUDA graphs to accelerate inference. Decode can often be captured
• Fine-tuning Manager 1 . The fine-tuning manager is an add-on module that supplies the training side of co-serving 4
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs Host Components
Client (HTTP)
engine, such as gradient computation and adapter weight updates; its mechanics are detailed in Section 3.2.
DeltaServe Add-ons
CPU Request Queue
Host Scheduler
Fine-tuning Manager ①
③
DeltaServe Scheduler ②
3.2 ⑧
GPU Host Execution Engine
Figure 3 illustrates an example workflow by which DeltaServe admits and executes fine-tuning samples during co-serving. DeltaServe folds the LoRA fine-tuning forward into the host engine’s existing forward pass 1 and adds a separate GPU subprocess for the fine-tuning backward 2 . The finetuning forward shares the host’s batch rather than running as an independent pass: as established earlier, the forward pass of LoRA fine-tuning is structurally identical to inference prefill, so the cost of folding a fine-tuning sample into an ongoing inference batch is close to that of one additional prefill sample rather than that of a separate forward pass. DeltaServe scheduler 3 exploits this property in its latency model (Section 3.3), which directly prices the added cost and uses it to guide SLO-aware admission.
DeltaServe Hooks ④
Shared GPU Memory (Zero-copy CUDA IPC) ⑦
Base Model
Fine-tuning Adapter
Co-Serving Model
⑥
Activation Buffer
Backward Subprocess ⑤
Figure 2. DeltaServe architecture and host interface. White boxes are generic inference serving system components; blue boxes are DeltaServe add-ons. Numbered markers indicate the attachment points and their relation described in Section 3: CPU-side admission control, GPU-side activation capture, shared GPU memory, and the backward subprocess.
Mixed forward batch. DeltaServe admits each fine-tuning sample as an ordinary host-batch entry: a single-step, prefillonly request routed to a reserved fine-tuning adapter and configured to produce no client-visible output. The sample executes through the host’s existing multi-LoRA forward path, using the same pipeline as inference requests. DeltaServe ’s only addition to this path is activation capture 4 . Hooks on the host model record residual-stream activations for fine-tuning rows into a buffer shared with the backward subprocess, while loss and gradient computation are left entirely to the backward path. The amount of activation state to retain is a design choice that trades GPU memory for backward recomputation. By default, DeltaServe saves only the residual-stream input to each transformer layer, since each layer can be recomputed from this input during backward. For a model with 𝑁𝐿 layers, hidden size 𝑑, and activation precision 𝑝, this costs roughly 𝑁𝐿𝑑𝑝 bytes per fine-tuning token; for a typical 8B model with 𝑁𝐿 = 32, 𝑑 = 4096, and fp16 activations, this is about 0.25 MB per token. If GPU memory permits, one may cache additional intra-layer intermediates, such as attention projections and feed-forward activations, to reduce recomputation. If memory is constrained, DeltaServe can offload captured activations to host memory and stream them back on demand. After the step completes, DeltaServe retires the fine-tuning request before any output token is emitted, keeping fine-tuning invisible to inference clients.
outside the host engine’s inference request queue. It loads and tokenizes the fine-tuning corpus, orders candidate samples for admission, and tracks training progress across epochs. This separation lets DeltaServe introduce finetuning work into the serving loop without modifying the host’s client-facing request path. • DeltaServe Scheduler 2 . The DeltaServe scheduler is an extension that intercepts the host’s scheduling loop before the host scheduler forms a new batch and dispatches it to its execution engine. It inspects the host’s proposed batch, estimates its execution cost with an analytic latency model, and admits as many fine-tuning samples as the inference request’s SLO permits 3 . The host then builds a single batch combining both inference and fine-tuning. The admitted fine-tuning samples are retired immediately after a single forward pass, ensuring that fine-tuning outputs never enter the client-facing serving path. • Inference-Engine Hooks 4 . The inference-engine hooks are lightweight insertions into the host’s forward path and engine loop. Model-level forward hooks 6 save the activations needed for fine-tuning backpropagation into the shared GPU memory 7 . They also record per-step execution time to refine the latency model online 9 . The same hook interface also supports an offline profiling pass that initializes the model before serving begins. • Backward Subprocess 5 . The backward subprocess is an add-on process launched alongside the inference engine under a CUDA MPS control daemon. It supplies the training-specific computation absent from the inference
Backward subprocess. The fine-tuning backward pass runs in a dedicated GPU subprocess under CUDA MPS, separate from the host forward engine. We use a subprocess rather 5
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau admission suppressed (backward running) ❼
FT admission open
Pipeline Client
R1
Host Scheduler
R1
❸ DeltaServe Scheduler
+
F1
Host Engine
R1
F1
R2
R3
❶
R1
R2
F2
F3
+
F4
F2
F3
R2
R3
Activation buffer
❹
Backward ❷ Subprocess new request
R3
R1
R1
F4
R1
R2
R2
R3
R3
R4
R1
R3
R4
R3
resume
R2 R3
R4
R3
R4
R4
CPU
R3
R4
+
F5
F6
R3
R4
F5
F6
GPU
❺
Layer N-1
Layer N-2
complete request
Paused
...
❽
prefill
Layer 0
❻ time
decode
fine-tuning
DeltaServe Add-on
Figure 3. Co-serving workflow in DeltaServe. The vertical axis shows the serving pipeline, and time advances from left to right. Each box represents one request or fine-tuning sample: yellow boxes are new arrivals, white boxes are inference prefill, orange boxes are inference decode, dashed gray boxes are completed requests, and blue boxes are fine-tuning samples. Dashed blue outlines mark DeltaServe add-on components. Analytical latency model. At deployment time, the model architecture is fixed, so the latency of a forward step is determined primarily by the batch composition. The dominant compute comes from the transformer attention and feed-forward layers: attention cost depends on the sequence lengths processed in the step, while feed-forward cost scales with the number of tokens passing through the layers. In addition to computation, step latency includes memory traffic from key–value cache reads for decode requests and activation-buffer writes for admitted fine-tuning tokens. Thus, for a mixed batch, DeltaServe estimates step latency from the prompt lengths 𝑛𝑖 of prefill requests, the number of active decode requests 𝐵𝑑 , the total key–value cache size 𝐾, and the number of admitted fine-tuning tokens 𝑇ft . DeltaServe predicts the step time from them as
than an additional CUDA stream so that inference and backward execution maintain distinct CUDA contexts [31], reducing launch-path contention and interference with latencycritical inference while allowing backward kernels to execute concurrently with inference. At startup, the subprocess maps the base model, the finetuning adapter, and the activation buffers into its address space using inter-process shared GPU memory. It can therefore access all tensors required for training without interprocess copies. The subprocess then waits in a blocking event loop until the scheduler issues a backward task. For each task, it reads the shared activations, reconstructs logits from the saved hidden states 5 , recomputing unsaved intermediates, computes the loss and LoRA gradients, and applies optimizer updates directly to the shared adapter weights. After the update completes, it releases the corresponding activation-buffer entries and returns to the idle state 6 . While a backward task is in flight, the scheduler admits no new fine-tuning samples, ensuring that activations are not overwritten before consumption 7 . Once the task completes, the scheduler records the trained samples and reopens fine-tuning admission.
3.3
𝑇 ≈𝛼
∑︁
(𝑛𝑖 + 𝐵𝑑 ) 2 + 𝛽 (𝑇in + 𝐵𝑑 ) + 𝛾 𝑇ft + 𝜀 𝐾 + 𝑐, (I )
𝑖
Í where 𝑇in = 𝑖 𝑛𝑖 is the total prefill length. Here 𝛼 prices the self-attention compute, whose quadratic span combines each prefill request’s 𝑛𝑖 prompt tokens with the 𝐵𝑑 decode queries sharing the step; 𝛽 prices the feed-forward compute; 𝛾 captures the cost of saving the fine-tuning activations; 𝜀 captures key–value cache memory access for decode requests; and 𝑐 is a fixed per-step overhead. For a step that carries only decode work, the prefill and fine-tuning terms vanish, leaving the feed-forward cost of the 𝐵𝑑 decode tokens and the key–value cache access:
SLO Budgeting
A co-serving scheduler must determine, before dispatching a batch, how much inference latency headroom remains. DeltaServe formulates this headroom as a per-batch budget: the difference between the batch’s predicted execution time and the latency deadline imposed by either the earliestarriving request’s TTFT constraint or the TPOT constraint of ongoing decode requests. DeltaServe’s SLO estimator supplies the prediction, and the scheduler (Section 3.4) consumes the budget to decide how much fine-tuning work can be safely co-scheduled with inference.
𝑇 ≈ 𝛽 ′ 𝐵𝑑 + 𝜀 ′ 𝐾 + 𝑐 ′ .
(II )
The two models are fit independently, with the coefficients of each seeded by an offline profiling pass and refined online during serving (Section 3.4). This split matches hosts that form a prefill-decode mixed batch, a single step that combines prefill and decode requests and is priced by Equation I ; it 6
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
does not require fusing, however: a host that runs prefill and decode as separate steps uses Equation I for its prefill steps and Equation II for its decode steps, so the model transfers across host engines unchanged.
unchanged. The rest of this section describes how the scheduler determines the amount of fine-tuning work to admit at each opportunity. Fine-tuning admission. Algorithm 1 shows the fine-tuning admission procedure for one scheduling step. The input is the inference batch 𝐵 inf that the host is about to dispatch, together with the SLO estimator, fine-tuning pool, backwardprocess state, remaining activation-buffer capacity, and host batching mode. The algorithm is designed around engines that form a prefill-decode mixed batch, in which prefill and decode requests are processed together in one step, and extends to engines that schedule the two phases separately. DeltaServe first rejects admission when fine-tuning cannot safely proceed: the backward subprocess is running, the activation buffer is full, the fine-tuning pool is empty, or, for hosts that do not form mixed batches, the pending step contains only decode work (line 5). It then computes the available latency budget Δ as the tighter of the TTFT slack and TPOT slack. The TTFT slack follows Equation III ; the TPOT slack is 𝑇tpot minus the predicted cost of any decode work deferred by this step. When the host forms a prefilldecode mixed batch, no decode batch is deferred and 𝐷 wait is empty (line 8-10).
Execution-mode coefficients. As discussed in Section 2.5, production inference systems often capture CUDA graphs to remove per-kernel launch overhead. One forward pass step can therefore run in one of two modes, replayed from such a captured graph or eagerly, with its kernels dispatched one at a time and no graph to replay. Which mode a step uses changes its latency by nearly an order of magnitude at small batch sizes, so the model for a prefill-carrying step must account for both. An inference-only prefill or mixed step can be replayed from the host’s captured graph, whereas a co-serving step, which carries fine-tuning tokens, is forced eager because the activation-capture hooks cannot run inside a replayed graph without major modification to the host engine. DeltaServe therefore fits Equation I with two sets of coefficients, one for graph execution and one for eager, and selects between them at prediction time using the host’s own graph-versus-eager decision. TTFT budget. With the latency model in place, DeltaServe determines how much fine-tuning can be admitted without violating latency objectives. For the upcoming step, it inspects the earliest-arriving inference request and computes its remaining TTFT budget, Δbudget = 𝑇SLO − (𝑡 now − 𝑡 arrival ),
Algorithm 1 Fine-tuning admission for one step. Inputs: Host proposed inference batch 𝐵 inf for the next step; SLO estimator 𝐸; fine-tuning pool 𝑄 ft ; Backward process state 𝑃𝐵 ; remaining buffer 𝑆 max ; host capability mixedBatch Outputs: Fine-tuning samples 𝑅ft to enqueue for the next step 1: 𝑅ft ← [ ] 2: if 𝑃𝐵 is running or 𝑆 max ≤ 0 or 𝑄 ft = ∅ then 3: return 𝑅ft 4: end if 5: if not mixedBatch and 𝐵 inf has no prefill tokens then 6: return 𝑅ft cannot merge fine-tuning into a decode-only step 7: end if 8: Δtt ← TTFT budget of 𝐵 inf Eq. III , with queue wait 9: Δtp ← 𝑇tpot − 𝐸.predict(𝐷 wait ) 𝐷 wait empty if decode shares the step 10: Δ ← min(Δtt, Δtp ) 11: if 𝐸.predict(𝐵 inf ) > Δ then 12: return 𝑅ft no slack even without fine-tuning 13: end if 14: for 𝑓 𝑡 ∈ 𝑄 ft in increasing token length do 15: 𝐵 ′ ← 𝐵 inf ∪ 𝑅ft ∪ {𝑓 𝑡 } 16: 𝑐 1 ← tokens(𝑅ft ) + tokens(𝑓 𝑡) > 𝑆 max activation buffer full 17: 𝑐 2 ← 𝐸.predict(𝐵 ′ ) > Δ would violate an SLO 18: if 𝑐 1 ∨ 𝑐 2 then 19: break 20: end if 21: 𝑅ft .append(𝑓 𝑡) 22: end for 23: Claim(𝑅ft ) reserve buffer, remove from pool 24: return 𝑅ft
(III )
where 𝑇SLO is the request’s TTFT target. This conversion of an SLO target into a dynamic per-step budget is the central admission lever; Section 3.4 describes how the scheduler turns it into a fine-tuning token count. 3.4
DeltaServe Scheduler
Guided by the SLO estimator, the DeltaServe scheduler serves as the control point between the host engine and DeltaServe ’s fine-tuning components. It operates inside the host’s scheduling loop, intercepting each step before dispatch. At this point, it draws candidate samples from the fine-tuning manager (Section 3.2), evaluates the latency impact of augmenting the pending inference batch, and admits fine-tuning only when the SLO budget and activation-buffer capacity permit. The scheduler also coordinates with the backward subprocess: it suppresses new fine-tuning admission while a backward batch is running and updates finetuning progress as completed backward batches are reported. At a high level, DeltaServe does not construct a separate fine-tuning batch. Instead, the scheduler enqueues admitted samples into the host’s scheduling path so that the host forms a single combined batch. After the step completes, DeltaServe retires the fine-tuning entries before results are returned to clients, leaving the host’s inference semantics 7
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau
Given Δ, DeltaServe first checks whether the inferenceonly batch already exhausts the budget (line 11-13). If not, it greedily considers fine-tuning samples in increasing token length and admits each sample only while the resulting mixed batch remains within both the latency budget and the activation-buffer capacity. The admitted samples are then claimed by reserving buffer space and removing them from the fine-tuning pool, preventing duplicate selection or buffer overcommitment under pipelined scheduling (line 14-23). All costs are predicted by the estimator in the execution mode used by the step. The inference-only baseline is priced in the host’s native mode, which may use CUDAgraph replay, whereas any mixed batch is priced with eagermode coefficients because activation hooks require leaving graph replay. For hosts that do not fuse prefill and decode, DeltaServe applies the same rule only to prefill steps and charges any delayed decode work against the TPOT budget, preserving the host’s batching semantics while exploiting prefill-phase headroom.
checked by the backward subprocess at every transformerlayer boundary. The inference process clears the flag before any forward step containing prefill tokens and restores it afterward. As a result, prefill can reclaim the GPU within one layer’s worth of backward kernels, while decode can continue to co-run with background training. Fine-tuning forward interruption. When no inference work is pending, DeltaServe may issue a forward step containing only fine-tuning samples. If an inference request arrives during such a step, DeltaServe sets an abort signal checked by the activation hooks at transformer-layer boundaries. The fine-tuning forward then stops, discards any partial activations, returns the interrupted samples to the fine-tuning pool, and immediately schedules inference. The interrupted samples are re-admitted later, so the cost of interruption is limited to the layers already executed.
4
Estimator calibration. The coefficients of the latency models in Equations I and II are initialized by an offline profiling pass before serving begins and refined online during execution. DeltaServe does not interfere with host engine initialization, such as CUDA-graph capture when available. Instead, it calibrates against the execution path the host will use online: after the host reports that initialization is complete, but before it begins accepting requests, DeltaServe issues representative synthetic batches through the same scheduling and forward path used during serving. For each observed step, DeltaServe records the batch composition, execution mode, and measured latency, and assigns the measurement to the corresponding coefficient set. The profiling shapes are chosen to make the coefficients separately identifiable. DeltaServe sweeps total prefill length to estimate linear prefill cost, varies how a fixed prefill length is split across requests to separate quadratic self-attention from linear projection and feed-forward cost, samples a grid over decode batch size and key–value cache length, and augments inference batches with varying numbers of finetuning tokens to isolate activation-capture overhead. During serving, DeltaServe continues refining the estimator from host-reported step times, periodically refitting each execution mode so the model tracks changes in load, batching behavior, and sequence-length distribution.
Evaluation
In this section, we evaluate DeltaServe’s ability to fold LoRA fine-tuning into a host inference engine under diverse conditions, harvesting the capacity inference leaves idle without violating the host’s inference SLOs, guided by the following key questions: • End-to-end effectiveness. Does DeltaServe exploit idle capacity left by gaps and low-load periods in inference workloads, converting it into fine-tuning throughput while preserving inference SLOs? How does this compare with LLMStation and with a split-pool deployment in finetuning throughput, average latency, and SLO compliance (Section 4.2)? • Portability across host engines. Can DeltaServe’s coserving design provide benefits across different inference serving systems, rather than being tied to a single host engine? We evaluate this by integrating DeltaServe with vLLM, SGLang, and S-LoRA (Section 4.3). • Scheduling and tunability. How do the scheduler’s perstep admission decisions shape the resulting trade-off between inference latency and fine-tuning throughput, and how does DeltaServe keep inference within its SLO during burst inference load (Section 4.4)?
4.1 Backward preemption. The backward pass is computeintensive and can interfere with inference when executed concurrently. This contention is significant during prefill, which also requires substantial compute, whereas decode is memory-bound and leaves more headroom. DeltaServe therefore allows backward execution to overlap with decode steps, but preempts it when prefill work arrives ( 8 in Figure 3). This is implemented with a shared GPU-grant flag
Experimental Setup
Hardware. Our experiments run on two setups that bracket the hardware where it is likely to be deployed: a consumergrade GPU and a datacenter-class multi-GPU server. The consumer setup is a single NVIDIA RTX 5090 (32 GB), paired with a 16-core AMD Ryzen 9 9950X CPU (Zen 5, 4.3 GHz) and 64 GB of DRAM. The datacenter setup is a 4×NVIDIA A100 (40 GB) server with a 24-core Intel Skylake Xeon CPU at 2.20 GHz and 340 GB of DRAM. 8
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
Burst-light
6000
40
3000
Baselines. We compare against two baselines. LLMStation [12] is the state-of-the-art inference–fine-tuning co-serving system and our primary point of comparison; it exploits decodeside opportunities to co-schedule fine-tuning with inference. The second baseline is a split-pool deployment, vLLM+torchtune, which reflects a common production approach: vLLM serves inference on one GPU pool, while torchtune 0.6.1 [33] runs LoRA fine-tuning on a separate dedicated GPU, with no coordination between the two workloads. Fine-tuning Workload. For all co-serving experiments, we fine-tune a dedicated LoRA adapter on the Llama 3-8B base model. The adapter uses a standard configuration with rank 𝑟 = 16, scaling factor 𝛼 = 32, dropout 0.05, and is applied to all attention projection matrices. Fine-tuning is performed on the Alpaca instruction-tuning dataset [32], a corpus of instruction–response pairs that is widely used as a LoRA fine-tuning benchmark [12, 25]. For all DeltaServe-enabled systems, training uses the AdamW optimizer with a learning rate of 1 × 10−3 and weight decay 0.01. The backward batch size is set to 256 tokens, meaning activations are accumulated until 256 fine-tuning tokens are processed before a backward pass is triggered.
requests / s
tokens / s
9000
80 0
Implementations. Our primary implementation, DeltaServevLLM, realizes the host-agnostic DeltaServe core on vLLM version 0.21.0 [14]. The core is a self-contained module of roughly 4,000 lines that implements the four components described in Section 3. Integrating it with vLLM requires only a compact set of in-tree hooks, confined to about a dozen host files, that expose the host’s scheduling loop, forward path, and worker startup interface to DeltaServe. All other serving functionality is reused unchanged, including vLLM’s continuous-batching scheduler, paged key–value cache, multi-LoRA forward kernels, sampler, and inference CUDA-graph capture. To demonstrate portability beyond vLLM, we also build DeltaServe-SGLang on SGLang [41], whose scheduler and execution path differ from vLLM’s, and DeltaServe-S-LoRA on S-LoRA [29], a minimal multiLoRA serving engine. The SGLang integration shows that DeltaServe can be adapted to another production-oriented serving stack, while the S-LoRA integration shows that the same core can be ported to a lightweight engine that provides little beyond multi-LoRA batching. Together, the three integrations show that DeltaServe depends only on multiLoRA batching as a host capability, rather than on a particular serving-engine architecture.
Burst-dense
120
0
10
20
30
time (s)
40
50
60
0
10
20
Nutanix Trace
30
time (s)
40
50
0 60
120
9000
80
6000
tokens / s
requests / s
Model. We evaluate on Llama 3-8B [34], a widely deployed open-weight model with grouped-query attention, representative of the model sizes served in interactive LLM applications. The same base model is shared by inference and fine-tuning across all systems.
40 0
3000 0
100
200
300
400
500
600
time (s)
700
800
900
0 1000 1100 1200
Figure 4. Inference workloads for evaluation: the synthetic burst-light and burst-dense patterns (top) and the 20-minute Nutanix production trace (bottom). In each panel the grey shaded bars report the incoming request rate and the blue line reports the resulting output-token rate.
Inference Workloads. Figure 4 summarizes the three inference workloads used in our evaluation: two one-minute synthetic burst patterns, burst-light and burst-dense, and a 20-minute production trace from Nutanix. In each panel, the x-axis shows wall-clock time, the shaded grey area reports incoming request rate on the left y-axis, and the blue curve reports output-token rate on the right y-axis. The synthetic workloads differ in both burst intensity and burst duration. Burst-light uses short 2-second peaks at 80 requests per second (RPS), leaving frequent low-load intervals and exposing substantial headroom for co-serving. Burst-dense uses longer 4-second peaks at 120 RPS, increasing the fraction of time spent under high load and reducing the scheduling slack available for fine-tuning. The Nutanix trace is irregular, with multi-scale bursts, spikes above 100 RPS, and extended near-zero intervals, providing a production-like test of DeltaServe under realistic load variation. These profiles are used as the primary four-GPU workloads; for singleGPU experiments, we replay the same temporal patterns with request rates scaled down proportionally, preserving burst structure while matching the capacity of one GPU. We set the service-level objectives to 400ms TTFT and 120ms maximum TPOT on A100 systems, and to 200ms TTFT and 100ms maximum TPOT on RTX 5090. 4.2
End-to-End Evaluation
We first evaluate DeltaServe in the multi-GPU setting targeted by production deployments. DeltaServe-vLLM coserves inference and fine-tuning on all four GPUs under analytic SLO-aware admission control. LLMStation also coserves the two workloads, but admits fine-tuning through decode-side opportunities using a cache-based SLO estimator. The split-pool reference, vLLM+torchtune, dedicates three GPUs to vLLM inference and one GPU to torchtune finetuning. Since all systems process the same arrivals, their 9
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau
0
Burst-dense latency (s)
10
0
20
30
40
50
DeltaServe-vLLM avg: 2.268s LLMStation avg: 4.704s vLLM+torchtune avg: 2.832s
8 6 4
60
0
10
20
30
50
DeltaServe-vLLM avg: 2.731s LLMStation avg: 3.708s vLLM+torchtune avg: 1.879s
8 6 4 2 0
40
0
200
400
600
time (s)
60
800 1000 1200
LLMStation throughput
FT avg: 1797 tok/s Total avg: 3088 tok/s SLO Satisfaction rate: 100.0%
95% SLO target vLLM+torchtune throughput
FT avg: 519 tok/s Total avg: 1810 tok/s SLO Satisfaction rate: 100.0%
FT avg: 1014 tok/s Total avg: 2305 tok/s SLO Satisfaction rate: 100.0%
100
5000
50
0
10000
20
40
60
FT avg: 1233 tok/s Total avg: 4593 tok/s SLO Satisfaction rate: 100.0%
20
40
60
FT avg: 483 tok/s Total avg: 3843 tok/s SLO Satisfaction rate: 83.3%
20
40
60
FT avg: 1014 tok/s Total avg: 4374 tok/s SLO Satisfaction rate: 100.0%
12500 10000 7500 5000 2500 0
0
100 50
5000 0
20
40
60
FT avg: 1418 tok/s Total avg: 2724 tok/s SLO Satisfaction rate: 100.0%
20
40
60
FT avg: 489 tok/s Total avg: 1795 tok/s SLO Satisfaction rate: 85.5%
20
40
60
FT avg: 1014 tok/s Total avg: 2320 tok/s SLO Satisfaction rate: 100.0%
0
100 50
250
500
750
time (s)
1000
250
500
750
time (s)
1000
250
500
750
time (s)
1000
SLO Satisfaction %
10000
15000
2 0
Nutanix Trace latency (s)
10
tokens / s
2
tokens / s
Burst-light latency (s)
4
15000
tokens / s
DeltaServe-vLLM avg: 2.025s LLMStation avg: 2.775s vLLM+torchtune avg: 1.653s
6
DeltaServe-vLLM throughput
SLO Satisfaction %
Request E2E latency vs time
LLMStation vLLM+torchtune Total throughput SLO satisfaction
SLO Satisfaction %
DeltaServe-vLLM Finetune throughput
Inference throughput
0
Figure 5. End-to-end comparison of DeltaServe-vLLM, LLMStation, and vLLM+torchtune on the 4-GPU deployment, one row per workload (top: burst-light; middle: burst-dense; bottom: Nutanix). The left column overlays per-request end-to-end latency against arrival time, with per-system averages annotated. The three right columns show each system’s per-second token throughput as a stacked band: fine-tuning (orange hatched) below inference (blue), the black line tracing total throughput (3s rolling mean). The purple curve (right axis) tracks the rolling fraction of requests meeting the TTFT/TPOT SLO, against a dotted 95% target. Each panel reports mean fine-tuning throughput, mean total throughput, and overall SLO satisfaction. inference token throughput is comparable within each workload; the key differences are how much fine-tuning throughput they obtain and how that throughput affects inference latency and SLO compliance. Figure 5 reports the comparison, one row per workload, against the timelines of Figure 4. The leftmost column overlays per-request end-to-end latency for all three systems, with per-system averages boxed; the remaining three columns give each system its own throughput panel, stacking finetuning tokens beneath inference tokens with the rolling TTFT-SLO satisfaction overlaid against a 95% guide. Burst-light. The burst-light trace has the shortest and lowestintensity peaks, giving co-serving systems frequent opportunities to use otherwise idle GPU capacity. All systems satisfy the SLO for every request on this workload, but DeltaServe-vLLM exploits the available headroom most effectively, reaching the highest fine-tuning throughput at 1797 tok/s: 3.5× that of LLMStation and 77% above the splitpool baseline. This throughput comes at modest inference cost. DeltaServe-vLLM’s average end-to-end latency is only 22% higher than the vLLM+torchtune baseline, whereas LLMStation incurs 68% overhead because its decode-phase finetuning inflates latency during bursts. The split-pool baseline
attains the lowest latency because one GPU is dedicated to fine-tuning and the remaining three vLLM GPUs are still sufficient for this light workload. However, its fine-tuning rate is fixed by the dedicated training GPU and cannot adapt to inference slack. In contrast, DeltaServe-vLLM’s fine-tuning band contracts near burst peaks and expands as bursts drain, showing the admission scheduler throttling fine-tuning to preserve the SLO and reopening capacity as inference pressure subsides. This throttling also explains why DeltaServevLLM’s latency approaches the split-pool baseline near burst tails: once fine-tuning admission is reduced, late-arriving requests experience less co-serving interference. Burst-dense. The burst-dense trace increases both burst intensity and duration, sustaining 120 RPS peaks for longer periods than burst-light. Under this heavier load, preserving the inference SLO becomes the primary challenge. LLMStation fails to do so, satisfying the SLO for only 83% of requests, because its co-serving policy continues to consume capacity for fine-tuning when inference is already constrained. DeltaServe-vLLM maintains 100% SLO satisfaction by giving inference priority under peak load. This also lets DeltaServe-vLLM outperform the split-pool baseline: vLLM+torchtune serves inference on only three GPUs, 10
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
4.3
Scheduled Request Timeline
E2E latency (s)
Output tokens / s
2000 20
1500 1000
10 0
1.50 1.25 1.00 0.75 0.50 0.25 0.00
500 0
200
400
600
Time (s)
800
1000
0 1200
FT throughput: 972 tok/s
1500 1000 500
0
200
400
600
Time (s)
2.0
E2E latency (s)
2500
800
1000
0 1200
FT throughput: 484 tok/s
1500
1.5
1000
1.0
500
0.5 0.0
FT throughput (tok/s)
Requests / s
30
FT throughput (tok/s)
whereas DeltaServe-vLLM can use the full four-GPU pool when fine-tuning is curtailed, yielding 20% lower average end-to-end latency. The throughput timeline explains how DeltaServe-vLLM still obtains substantial fine-tuning throughput under burstdense load. During peak intervals, fine-tuning throughput drops sharply, showing that the scheduler suppresses finetuning when inference pressure is high. Once each burst is absorbed, the scheduler reopens fine-tuning admission. These recovery periods account for most of the 1233 tok/s that DeltaServe-vLLM achieves. On average, DeltaServevLLM achieves 2.6× the fine-tuning throughput of LLMStation and 21% more than the split-pool baseline. Nutanix. The Nutanix production trace is irregular and heavy-tailed, with long stretches of light load punctuated by sharp, heavy bursts. DeltaServe-vLLM again achieves the highest fine-tuning throughput at 1418 tok/s, satisfying the SLO for every request: it outpaces LLMStation by 2.9× and exceeds the dedicated fine-tuning GPU of the split-pool reference by 39%. Harvesting this fine-tuning raises its average end-to-end latency to 45% above the split-pool reference as expected since DeltaServe is designed to harvest the gap between execution time and SLO requirement. LLMStation incurs a larger inference cost, running 97% above the reference and meeting the SLO for only 85% of requests. The latency and throughput timelines show DeltaServevLLM adapting to the trace’s changing load. During heavy bursts, such as the spike around 60s and the sustained interval between 970 and 1100s, the scheduler quickly throttles both fine-tuning admission and backward execution to protect inference. Requests arriving in these windows therefore see latency close to the split-pool reference, lower during the 60s spike and comparable during the later sustained burst, while LLMStation violates the SLO. During the longer, lowerrate intervals between roughly 300 and 1000s, where the request rate remains below the burst-light peak, DeltaServevLLM uses the available headroom more aggressively. This increases fine-tuning throughput and occasionally raises latency above LLMStation’s within the same window, but still remains within the SLO.
0
200
400
600
Time (s)
800
1000
0 1200
Figure 6. Portability of DeltaServe to SGLang and S-LoRA on a single RTX 5090, replaying the scaled-down Nutanix trace. Top: request timeline. Middle and bottom: per-request end-to-end latency as dots (left axis) for the unmodified host engine and its DeltaServe-enabled version, with fine-tuning throughput as the filled curve (right axis).
includes optimizations that reduce inference overhead and leave more usable headroom under the same SLO. On this host, DeltaServe sustains 972 tok/s of fine-tuning throughput while preserving the inference SLO for every request. The fine-tuning throughput follows the workload shape: it increases during low-load intervals, drops during request bursts, and resumes as the burst drains, mirroring the Nutanix behavior observed in the multi-GPU results of Section 4.2 and Figure 5. S-LoRA provides a complementary portability point. Unlike SGLang, it is a lightweight research serving engine centered on multi-LoRA batching, with fewer inference-side optimizations, making it close to the minimum host capability DeltaServe requires. On this host, DeltaServe still sustains 484 tok/s of fine-tuning throughput with full SLO compliance. The lower throughput is expected: with a less optimized host inference pipeline, the same inference trace consumes more of the latency budget, leaving less headroom for fine-tuning admission. DeltaServe-enabled S-LoRA slightly reduces average end-to-end latency. This improvement does not come from co-serving itself, but from the CUDA-graph capture added as part of the integration. The result shows that DeltaServe can operate even on a minimal multi-LoRA host while preserving the host’s inference semantics.
SGLang & S-LoRA
We next evaluate whether DeltaServe ’s co-serving design transfers across host engines with different serving architectures. We evaluate DeltaServe-SGLang and DeltaServeS-LoRA on a single RTX 5090, using the scaled-down Nutanix trace from Section 4.1. For each host, we compare the DeltaServe-enabled version against the unmodified inference-only engine. Figure 6 shows the request timeline, per-request end-to-end latency, and fine-tuning throughput over time. SGLang represents a production-oriented inference engine with an optimized scheduling and execution path. Its runtime is designed for high-throughput LLM serving and 11
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau
4.4
Ablation Study
These finer-grained opportunities appear around 40s and throughout 150–200s, where DeltaServe-Temp continues to produce fine-tuning throughput despite nonzero inference load. Since fine-tuning-only steps never share a batch with inference, their impact on inference performance is minimal: average end-to-end latency increases by only 2%. Thus, when inference efficiency is prioritized, DeltaServe closely matches vLLM’s latency behavior while obtaining 507 tok/s of fine-tuning throughput from otherwise unused capacity. The few requests for which DeltaServe-Temp’s latency exceeds bare vLLM’s coincide with fine-tuning-throughput peaks, indicating that they arrived while a fine-tuning-only step was already executing and waited briefly for it to yield. The resulting delay is small because DeltaServe interrupts fine-tuning-only execution at fine granularity, as described in Section 3.3. We evaluate the effect of this interruption mechanism separately later. Enabling forward batch co-serving shifts DeltaServevLLM toward higher fine-tuning throughput. By allowing fine-tuning tokens to be appended to inference batches, the full system increases fine-tuning throughput from 507 to 934 tok/s, an 84% improvement over DeltaServe-Temp, while still preserving the SLO. This additional throughput comes at higher inference latency because fine-tuning tokens lengthen shared forward steps. The trade-off is controlled by the SLO budget used for admission: a tighter budget would move DeltaServe-vLLM closer to the DeltaServe-Temp operating point, reducing latency overhead at the expense of fine-tuning throughput.
We next evaluate DeltaServe in isolation to separate the effects of its individual mechanisms. Using scaled-down versions of the same traces on a single RTX 5090, we study two components without the averaging effect of multi-GPU replication. First, we measure the benefit and latency cost of admitting fine-tuning into inference-carrying batches rather than restricting it to idle steps (forward batch co-serving). We then evaluate how DeltaServe protects inference latency when a request arrives during an in-flight fine-tuning-only forward pass (fine-tuning forward interruption).
200
4
100
2 00
25
50
75
100
Time (s)
125
150
175
1.5
Fine-tuning forward interruption. This experiment isolates the fine-tuning-only step interruption mechanism in Section 3.3. When an inference request arrives during a finetuning-only step, DeltaServe aborts the forward batch so inference does not wait for the full fine-tuning forward pass to finish. We compare bare vLLM, DeltaServe-Temp with interruption enabled, and DeltaServe-No-INTR, which disables interruption.
2000
1.0
1000
0.5 0.0 0
2000
25
50
75
100
Time (s)
125
150
175
2000
The orange curve in the lower panel shows that DeltaServeTemp sustains substantial fine-tuning throughput even when forward batch co-serving is disabled. In this mode, DeltaServe exploits not only the long idle intervals in the request timeline, such as 0–15s, around 75s, and around 125s, but also short gaps between inference steps within busy periods.
Figure 8. Effect of fine-tuning-only step interruption on one RTX 5090 over a scaled-down burst-light trace. Dots show per-request end-to-end latency; filled curves show finetuning throughput. Disabling interruption forces inference requests arriving during a fine-tuning-only step to wait until that step completes.
2000 1500
0.8
1000
0.4 0.0
12
DeltaServe-vLLM-Temp : 5% tail 0.686s (+8.1% vs vLLM) DeltaServe-vLLM-No-INTR : 5% tail 0.811s (+27.8% vs vLLM)
1.2
FT throughput (tok/s)
Figure 7. Forward batch co-serving on a single RTX 5090 using the 600–800s window of the Nutanix trace, scaled to one RTX 5090. Top: scheduled request timeline. Bottom: perrequest end-to-end latency for vLLM, DeltaServe-Temp, and DeltaServe-vLLM, with fine-tuning throughput for the two DeltaServe variants. DeltaServe-Temp restricts finetuning to inference-free steps.
E2E latency (s)
E2E latency (s)
300
6
FT throughput (tok/s)
Requests / s
8
Output tokens / s
Forward Batch Co-serving. This experiment isolates the contribution of forward batch co-serving, that is, folding fine-tuning tokens into a batch that already carries inference workload. We compare three configurations on a single RTX 5090 over the 600–800s window of the Nutanix trace, with the request rate scaled down to fit the capacity of one RTX 5090. The configurations are: bare vLLM, which serves inference only; DeltaServe-Temp, which disables forward batch co-serving and admits fine-tuning only when the host would otherwise issue no inference work; and DeltaServevLLM, the full design, which also admits fine-tuning tokens into inference-carrying batches. Figure 7 reports per-request end-to-end latency and fine-tuning throughput against the same scheduled request timeline.
500 0
5
10
15
20
Time (s)
25
30
35
0 40
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
Figure 8 shows that without interruption, latency spikes at burst onsets, where new inference requests arrive during an in-flight fine-tuning step. With interruption enabled, DeltaServe-Temp closely tracks bare vLLM and limits this tail amplification: average latency increases by only 0.7%, while the 5% tail increases by 8%. Disabling interruption raises the 5% tail overhead significantly to 27%. The finetuning throughput difference is small: DeltaServe-No-INTR reaches 869 tok/s on average, only 2% above DeltaServeTemp. Thus, layer-boundary interruption protects inference tail latency while sacrificing little fine-tuning throughput.
5
New York, NY, USA. [10] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association. [11] Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R. Das. 2022. Cocktail: A Multidimensional Optimization for Model Serving in Cloud. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA. [12] Yongjun He, Haofeng Yang, Yao Lu, Ana Klimović, and Gustavo Alonso. 2025. Resource multiplexing in tuning and serving large language models. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA). USENIX Association, Usa. [13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. [14] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany). Association for Computing Machinery, New York, NY, USA. [15] Yunseong Lee, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, and Matteo Interlandi. 2018. PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA. [16] Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, and Cong Wang. 2023. Lyra: Elastic Scheduling for Deep Learning Clusters. In Proceedings of the Eighteenth European Conference on Computer Systems (Rome, Italy). Association for Computing Machinery, New York, NY, USA. [17] Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuoming Chen, Sean Lai, Xinhao Cheng, Xupeng Miao, and Zhihao Jia. 2025. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding. arXiv:2501.12162 [cs.CL] [18] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA. [19] Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models. In Proceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang (Eds.), Vol. 139. Pmlr. [20] Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2025. TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput. arXiv:2406.14066 [cs.AI] [21] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-ofthe-art Parameter-Efficient Fine-Tuning methods. https://github.com/ huggingface/peft. [22] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna
Conclusion
This paper introduced DeltaServe, a host-agnostic co-serving design that folds LoRA fine-tuning into an existing inference engine under SLO control, driven by a CUDA-graph-aware latency model and realized on vLLM, SGLang, and S-LoRA. On a Nutanix production trace it delivers 2.9× the fine-tuning throughput of LLMStation at 100% inference SLO compliance versus LLMStation’s 85%, harvesting idle GPU capacity without compromising interactive latency.
References [1] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369 [cs.LG] [2] Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020. PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association. [3] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318 [cs.CL] [4] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. [5] Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX Association, Carlsbad, CA. [6] Yujeong Choi and Minsoo Rhu. 2020. PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). [7] Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: a low-latency online prediction serving system. In Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation (Boston, MA, USA). USENIX Association, Usa. [8] Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691 [cs.LG] [9] Aditya Dhakal, Sameer G Kulkarni, and K. K. Ramakrishnan. 2020. GSLICE: controlled spatial sharing of GPUs for a scalable inference platform. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA). Association for Computing Machinery, 13
Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau
Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (La Jolla, CA, USA). Association for Computing Machinery, New York, NY, USA. [23] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri). Association for Computing Machinery, New York, NY, USA. [24] Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. 2025. SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [25] Gabriele Oliaro, Xupeng Miao, Xinhao Cheng, Vineeth Kada, Mengdi Wu, Ruohan Gao, Yingyi Huang, Remi Delacourt, April Yang, Yingcheng Wang, Colin Unger, and Zhihao Jia. 2026. FlexLLM: TokenLevel Co-Serving of LLM Inference and Finetuning with SLO Guarantees. In The 23rd USENIX Symposium on Networked Systems Design and Implementation. [26] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2025. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (Buenos Aires, Argentina). IEEE Press. [27] Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association. [28] Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: a GPU cluster engine for accelerating DNN-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (Huntsville, Ontario, Canada). Association for Computing Machinery, New York, NY, USA. [29] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. 2024. SLoRA: Scalable Serving of Thousands of LoRA Adapters. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. [30] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: high-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA). JMLR.org. [31] Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece). Association for Computing Machinery, New York, NY, USA. [32] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https:// github.com/tatsu-lab/stanford_alpaca. [33] torchtune maintainers and contributors. 2024. torchtune: PyTorch’s finetuning library. https://github.com/pytorch/torchtune [34] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, MarieAnne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] [35] Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. 2025. Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, New York, NY, USA. [36] Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018. Gandiva: Introspective Cluster Scheduling for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA. [37] Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. 2020. AntMan: Dynamic Scaling on GPU Clusters for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association. [38] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA. [39] Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). USENIX Association, Renton, WA. [40] Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA. [41] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
14