arXiv:2606.22983v1 [cs.DC] 22 Jun 2026
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs Xiangyu Zhi∗
Peiqi Yin∗
Sheng Guan
The Chinese University of Hong Kong [email protected]
The Chinese University of Hong Kong [email protected]
The Chinese University of Hong Kong [email protected]
Chenguang Zheng
James Cheng
Xiao Yan
The Chinese University of Hong Kong [email protected]
The Chinese University of Hong Kong [email protected]
Wuhan University [email protected] Turn 1
Abstract Realtime omni-modal LMs support speech-centric conversations where users stream inputs, hear generated audio, and interrupt freely. Existing Omni-LM serving systems still rely on throughput-oriented LLM scheduling and LRU KV offloading. These policies ignore audio playback and multiturn reuse: they may generate tokens far beyond what users hear, wasting work after barge-in, and evict KV state needed in the next turn. LiveServe is an interaction-aware serving system for realtime Omni-LM interaction. It exposes playback progress, speech activity, and barge-in events to the serving pipeline. The scheduler prioritizes first-audio and near-underrun sessions while limiting generation beyond the playback frontier. The KV manager uses next-use-aware eviction and preloads likely-needed KV during user speech to hide reload latency. On vLLM-Omni, LiveServe improves realtime serving across two Omni-LMs and mixed workloads. It lowers P90 audio TTFP by 1.55× on average and up to 2.21×, while improving completed-request throughput by 1.15× on average and up to 1.56×, and moves most KV reload work off the next-turn critical path. Keywords: Omni-modal Large Models, Model Serving, Realtime Interaction
1
Introduction
Omni-modal large models (Omni-LMs) [9, 30, 46, 47, 52] (also known as any-to-any models) extend large language models (LLMs) [1, 4, 53] to process and produce multi-modal contents (e.g., text, image, video, and audio). Representative Omni-LMs include the Qwen-Omni series [44, 51, 52], Ming-Omni [15, 30], and Nemotron VoiceChat [43]. These models typically adopts a multi-stage pipeline instead of a monolithic decoder: the upstream stages perform multimodal understanding, mapping heterogeneous inputs into shared token or embedding representations, while the downstream stages synthesize the user-facing outputs. For instance, Qwen3-Omni [52] utilizes modality-specific encoders
∗ These authors contributed equally to this work.
Speaking
Turn 2 Speaking
Turn 3 Speaking
Listening User User Barge-in Listening Omni-LM
Speaking
Listening
Discarded
Time
Figure 1. Interactive Omni-LM serving with multiple turns. to handle the inputs, a language backbone (thinker) for reasoning and response planning, and speech synthesis components (talker and vocoder) to produce audible user replies. As shown in Figure 1, Omni-LM serving is usually interactive with multiple turns (called a session). In particular, the user provides streaming speech, image, and video inputs to the Omni-LM and may barge in (i.e., interrupt) while the model is still generating responses. The user-model interaction can take multiple turns, and the model takes all previous turns as contexts to process the current turn. Such an interactive paradigm is adopted by applications like hands-free assistants [21, 49], real-time translation [29], customer support [10], and accessibility services [3]. Commercial offerings include Google’s Gemini Live [14], OpenAI’s Realtime API [31], and ByteDance’s Doubao Realtime Voice [38]. The community currently adapts high-throughput LLM serving systems to run the multiple stages (i.e., model components) of Omni-LMs in a stage-oriented manner. Representative stacks include vLLM-Omni [54] and SGLang-Omni [45], which extend the runtime of vLLM [20] and SGLang [58], respectively. In these systems, different model stages (e.g, encoders, thinker, and talker) run on separate inference engines with independent schedulers, batching policies, GPU placement, and possibly different parallelization strategies, and an orchestrator routes intermediate results between the stages. These systems optimize the individual stages for metrics like time-to-first-token (TTFT) and time-between-tokens (TBT). Lacking a global view at the session level with multiple interactive stages, these systems suffer from two critical problems that hinder efficiency.
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
• Excessive generation. LLM serving engines optimize metrics like TTFT and TBT for individual stages, which do not necessarily lead to better user experience during Omni-LM interactions. Specifically, for live audio reply, users care the most about audio time-to-first-packet (audio TTFP) and whether the playback is smooth. Once the audio starts, users are largely satisfied as long as the playback has no glitches, and further reducing the TBT does not improve user experience. However, existing Omni-LM serving systems conduct continuous token generation at each LLM stage without considering the playback of the audio stream, and many tokens could be generated ahead of the playback progress. If users barge in during the playback, the generated but not yet played tokens will be wasted. • Passive cache management. For each stage of Omni-LMs, a session needs to keep all KV caches of the previous turns for the current turn, stressing limited GPU HBM when serving multiple sessions concurrently. Audio and video inputs make the matter worse by producing large KV caches. LLM serving systems can offload the KV caches to CPU DRAM [32] but eviction is handled reactively via LRU without considering realtime interaction semantics. For instance, a session that is quiet during user playback may appear cold and be evicted, although its KV cache will likely be needed for the next interaction turn. Moreover, evicted KV cache is restored only when the current turn already starts, and thus reloading time boosts TTFP. To tackle the two problems above, we build LiveServe, an interaction-aware serving system for Omni-LMs. Compared with existing solutions that process the stages of Omni-LMs separately, LiveServe adopts a global view at the session level to make holistic request batching decisions via urgency based scheduling and handle KV cache swapping with proactive cache management. In particular, to avoid excessive generation while ensuring user experience, LiveServe adds an interaction layer in vLLM-Omni to track real-time session states, including playback progress, speech activity, and barge-in events. Using these signals, LiveServe schedules the inference requests for the next batch based on the playback progresses of their sessions. Sessions that have not produced the first output token and sessions whose playback buffer is close to under-run are prioritized to reduce audio TTFP and avoid glitches, respectively. In contrast, sessions whose generated tokens are far ahead of the playback progress are delayed to avoid wasting computation if the users barge in. Different from LRU that conducts eviction passively according to historical cache accesses, LiveServe proactively utilizes interaction patterns to infer future cache accesses. Specifically, LiveServe keeps the KV caches in GPU HBM for sessions whose audio playback is coming to an end since their KV caches will soon be used for the next interaction. Meanwhile, sessions that have just started a long playback are offloaded to CPU memory because they are likely to
Text
Text Image Audio Video
Modality Encoders
LLM Backbone
Modality Generators
Encode Stage
LLM Stage
Gen. Stage(s)
Image Audio Video
Figure 2. A common architecture of Omni-LMs. remain inactive for a while. Moreover, when users begin speaking or barge in, LiveServe preloads the CPU resident KV caches to GPU HBM as these sessions will become active for generation shortly. Such a design overlaps DRAM-toHBM transfer with user’s speaking time and moves cache reloading latency off the next-turn audio TTFP path. We implement LiveServe atop vLLM-Omni and evaluate its performance using multiple Omni-LMs models and realtime workloads. Experimental results show that LiveServe improves the maximum sustainable throughput by up to 55% over vLLM-Omni while maintaining the same audio TTFP. In the presence of random user barge-in, our urgency based scheduling reduces P90 audio TTFP by over 50% and cuts the calculated-but-unheard tokens by 75% on average. Meanwhile, proactive cache management reduces the peak LLM-stage KV residency by about 26% and moves most KV reload work off the next-turn critical path. To summarize, we make the following contributions: • We analyze the problems of existing systems when serving interactive Omni-LMs with continuous playback, possible barge-in, and multi-turn context reuse. • We propose and build LiveServe, an interaction-aware serving system for Omni-LMs, adopting a holistic sessionlevel view over individual stages to enhance efficiency. • We design urgency based scheduling to ensure user experience while avoiding wasting computation and proactive cache management to handle KV cache eviction and hide DRAM-HBM cache loading latency.
2
Background and Motivation
2.1
Omni-modal Models
Text-only LLMs excel at language understanding and generation, but are limited to textual inputs and outputs. Omnimodal large models (Omni-LMs) [9, 30, 46, 47, 52], also referred to as any-to-any multimodal models, extend this capability by jointly understanding and generation content across text, images, video, and audio within unified architectures. This unification supports flexible cross-modal reasoning and multimodal output beyond separate understanding and generation pipelines, including spoken responses as well as diffusion-based image or video generation [6, 9]. The emergence of any-to-any modeling leads to substantially more complex structures than conventional LLMs. As
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
Calculated tokens are wasted!
… Talker & Vocoder
…
Audio Stream Time
2
4
Talker
6
8
Vocoder
0
10
20
Player
30
40
50
60
Time Since Request Start (s) Time Since Request Start (s) (a) Short Context (b) Long Context Figure 4. Generation and playback completion over time.
User Barge-in
Figure 3. Interaction-unaware generation ahead of playback.
shown in Figure 2, modern Omni-LMs commonly organize inference into an encoding stage, an autoregressive (AR) LLM stage, and one or more generation stages. The encoding stage maps multimodal inputs into embedding representations consumed by the LLM backbone. The backbone performs semantic understanding and response planning, then drives modality generators such as speech talkers and vocoders, or diffusion transformers (DiT) [35] for visual synthesis. Recent Omni-LMs instantiate this template with different choices of encoders, language backbones, and output generators. Qwen3-Omni [52] is a representative speechoriented design. It accepts text, image, video, and audio inputs through dedicated encoders, uses a thinker LLM for understanding and response planning, and routes hidden states to a talker that generates speech tokens followed by a lightweight vocoder for waveform reconstruction. Many other Omni-LMs follow the same multi-stage organization while swapping in different generators for their target modalities, such as the Ming-Omni series [15, 30] and LongCat-FlashOmni [47] for spoken interaction, Baichuan-Omni-1.5 [22] for controllable speech output, or AR-plus-DiT pipelines for image and video outputs [6, 57]. The concrete modules differ across models, but the overall architecture remains modular, with encoders for ingestion, a shared backbone for reasoning, and specialized decoders for each output modality. To reduce response latency, recent models use different methods of incremental execution, but they differ in how much cross-stage overlap they expose. Some models only stream input or output tokens, while others propose asynchronous chunking across adjacent stages [43, 52]. Take Qwen3Omni as an example. The thinker can pass partial hiddenstate chunks to the talker before thinker decoding fully completes, and the talker can pass speech-token chunks to the vocoder before the full sequence is generated. This chunked execution lowers audio TTFP and improves user-perceived responsiveness in spoken interaction. 2.2
Completion (%)
Thinker
Thinker 100 80 60 40 20 00
Serving Systems for Omni-LMs
To deploy Omni-LMs in production, recent systems extend high-throughput LLM runtimes to multi-stage pipelines [45, 54]. vLLM-Omni [54] builds on vLLM [20] and introduces a
fully disaggregated serving stack. Users decompose an OmniLM into interconnected stages (e.g., encoders, thinker, talker, vocoder, or DiT modules); each stage runs as an independent engine with its own scheduler and KV cache manager. An orchestrator drives request progress across the stages, while inter-stage connectors route intermediate tensors and control metadata between engines. As stages have different compute and memory profiles, each can adopt an independent parallelization strategy, such as data parallelism or tensor parallelism, without constraining the rest of the pipeline. SGLang-Omni [45] pursues a similar stage-oriented design on top of SGLang [58]. Both systems treat Omni-LM inference as coordinated execution across multiple stages rather than a single monolithic forward pass, improving job completion time and resource utilization for diverse any-toany workloads. They primarily provide the serving support for general inference scenarios with multimodal generation, such as text-to-speech, audio chat, and image generation. 2.3
Serving Challenges
The interactive workload in Section 1 creates two serving problems. The server keeps generating far ahead of playback, and multi-turn KV is evicted and reloaded at the wrong time. We next examine both problems in turn. Interaction-unaware generation ahead of playback. During a spoken reply, the client plays audio at a fixed rate, while the thinker and talker continue decoding and synthesizing speech ahead of what has been heard. Stage-local schedulers still favor faster token production, so they keep extending this generation lead even once playback already has enough buffered audio. If the user barges in, the unplayed portion of that lead is wasted GPU work, as Figure 3 illustrates. Figure 4 quantifies this execution-time mismatch on Qwen3Omni. The thinker, talker, and vocoder reach full completion quickly, while client playback progresses much more slowly. Server-side stages therefore finish far ahead of what the user has actually heard, leaving a long interval in which the system keeps generating tokens that may never be played. A barge-in during this interval discards a large portion of already-computed work, and the waste grows when a longer context makes execution finish further ahead of playback. Interaction-unaware multi-turn KV management. A multi-turn session must keep prior-turn KV at each stage so the next request can reuse the full context, and audio or
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
456K 6.4s
400K
7.5 5.0
200K 0K
2.5 LRU
0.0 Interactive-aware
Reload time
KVCache size 100K
102 1K
100
10
Eviction policy
10 100 1K 10K 100K 1M Token count
(a) LRU eviction pressure
(b) KV DRAM reload cost
Request KVCache size (MB)
E2E latency Reload time (ms)
496K 7.8s
E2E p90 latency (s)
Evicted KV blocks
Evicted KV blocks 600K
Figure 5. Interaction-unaware multi-turn KV management. (a) LRU eviction under load increases evicted KV blocks and tail latency. (b) Reloading offloaded KV from host DRAM back to GPU HBM incurs latency that grows with KV size.
Response
Interaction Plane API Server Live Session Prefetch Endpoint
3
OmniCast Architecture
We propose LiveServe, an interaction-aware serving system for realtime Omni-LM interaction. As shown in Figure 6, LiveServe keeps the stage-oriented execution model of vLLM-Omni, but separates the runtime into an interaction plane and a data plane. The data plane preserves the orchestrator and stage-local engines, while the interaction plane tracks live session signals such as streaming arrivals, playback progress, and barge-in events. By exposing these signals to schedulers and KV managers, LiveServe allows engine-level control decisions to react to user interaction without changing the model pipeline. Below, we describe how this separation is realized through the session-facing interaction layer, interaction-aware execution engines, and stage-oriented orchestration.
Event Ring Pressure Snapshot
Data Plane Orchestrator Stage Graph
Engine Engine Engine (Stage-0)
video inputs make this resident state grow quickly against limited HBM. Existing engines spill idle session KV to host DRAM [32] and use least recently used (LRU) to decide which blocks remain in GPU memory. LRU reflects recent access rather than upcoming turn timing, so a session can be evicted while the user is still listening and then pay a reload penalty when the next turn begins. Figure 5 shows that this mismatch becomes more costly as load increases. First, a session that is temporarily quiet during playback may appear cold and be evicted even though its KV will likely be needed on the next turn, causing hot requests to be evicted and increasing both reload volume and E2E tail latency under concurrent load (Figure 5(a)). Second, reload timing is not coordinated with interaction (Figure 5(b)). When KV loading from host DRAM begins only after the user has already started the next turn, the transfer sits on the critical path to LLM prefill and directly inflates audio TTFP; this cost grows with the amount of offloaded KV that must be brought back to GPU HBM. Because loading is not interaction-aware, the system misses chances to preload KV during client-side idle intervals, so reactive reload at turn start makes TTFP much worse than necessary.
Runtime Monitor Record
Execution Engine Interaction-aware Scheduler (§4)
Interaction-aware KV Manager (§5)
Balancer (§4.2)
Eviction (§5.2) Pre-Load (§5.3)
Model Runner
Figure 6. System architecture of LiveServe.
• Session-facing interaction layer. The API server is the entry point for live multimodal sessions, forwarding streaming inputs to the orchestrator and returning generated audio to clients. It also exposes the prefetch endpoint for history preheat and current-turn prefill, labeling this preparatory work separately from latency-critical decoding. Alongside the API server, a lightweight runtime monitor tracks how each session is progressing at the client, including whether playback is advancing normally or has been interrupted. The monitor turns these client-side signals into a compact runtime view that engine policies can read without directly coupling to the session protocol. • Interaction-aware execution engines. Each model stage runs as an execution engine, such as the thinker, talker, and vocoder engines in a speech-oriented Omni-LM. Inside each engine, the scheduler manages the stage-local request queue and forms batches, while the KV manager keeps the cached conversation state used by that stage. LiveServe feeds session state from the runtime monitor into these two components, so scheduling and KV-residency decisions can be aware of client-side interaction. This keeps interaction-aware control inside the engine where the corresponding execution and memory decisions are made. • Stage-oriented orchestration. LiveServe adopts vLLM-Omni’s orchestrator and stage graph to connect these engines into an Omni pipeline. The orchestrator routes requests, intermediate data, and generated outputs across stages, while each stage run with its own engine and GPU allocation. Thus, LiveServe preserves the existing disaggregated execution substrate and focuses its changes on the interaction-aware control inside the engines.
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
Scheduler …
Waiting Queue
Defer Queue …
Balancer (§4.2) U-Aware Policy
Deferred Req Reschedule
Preload Handler KV Monitor KV Pressure Info
GPU
Request Meta: 1. Lead time 2. Prefill 3. KV length …
Req / Preload …
Running Queue KVCache HBM
Naive policy
400 200 0
10
20
30
40
Norm. value
No policy
Runtime Monitor
KV in HBM (MB)
API Server
Time since target KV first appears (s)
1.0 0.5 0.0 Time
KV resident
Figure 8. Motivating KV-pressure-aware scheduling with a long multi-turn dialogue. The left panel tracks the GPU KV-cache residency of one long-context request over time, while the right panel reports normalized residency duration and average resident KV footprint.
Figure 7. Overview of the interaction-aware scheduler. 4.1
Barge-in handling. During playback, the client reports how much generated audio has been consumed, while VAD detects whether the user starts a new utterance. When a bargein arrives, the orchestrator notifies the engines serving the current response; the engines abort in-flight computation, discard tokens beyond the playback point, and clear temporary stage state. In parallel, the runtime monitor records the interruption in the session state seen by later scheduling and KV-management decisions. This keeps abort and cleanup on the execution path, while letting interaction-aware policies observe the interruption through the monitor.
4
Interaction-aware Request Scheduling
Modern LLM serving systems commonly use first-come-firstserve (FCFS) request scheduling inside each execution engine [20, 58]. Requests wait until the engine has enough round budget to admit them into the running set, after which continuous batching repeatedly schedules active requests for decode steps. This design keeps GPU batches large and improves throughput. It also reduces time-between-tokens (TBT), which matters for text generation because users observe the response token by token. Realtime Omni-LM interaction changes what the scheduler should protect. Before playback starts, asynchronous chunking improves audio TTFP only if downstream stages are scheduled soon after their first chunks become ready. After playback starts, the primary goal is to keep audio playback smooth. Generating farther ahead of the client does not further improve the user’s immediate experience and can waste work if the user interrupts. FCFS misses both cases, since it may continue serving well-buffered sessions while other sessions are still waiting for first audio or are close to playback underrun.
Urgency-Aware Scheduling
LiveServe replaces FCFS-style ordering with an interactionaware policy inside each execution engine. At each scheduling round, the scheduler reads session state from the runtime monitor and chooses a resource-feasible subset from its ready set R𝑠 . Serving a request too late can delay its first playable audio or let playback drain, while serving a well-buffered request too aggressively may produce work that the user never hears. As shown in Figure 7, LiveServe first separates requests by realtime urgency and then ranks requests within the same urgency class. The scheduler assigns request 𝑖 at stage 𝑠 to an urgency class Γ𝑖𝑠 ∈ {0, 1, 2}, where smaller values indicate higher urgency. The main signal is the stage-aware playback buffer P𝑖𝑠 , which estimates how much work stage 𝑠 has produced beyond what downstream consumers currently need. A positive buffer means the stage has produced more units than the downstream consumer has consumed or is ready to consume. For audio-generation stages, this buffer corresponds to playable audio waiting at the client, computed from generated audio and client playback progress. For upstream stages, it is estimated from downstream work queues, such as talker𝑠 be the minimum safe ready or pending talker work. Let Psafe buffer used to identify playback risk. • U0: playback urgency. Requests that have started playback 𝑠 ) are most but have a small playback buffer (P𝑖𝑠 ≤ Psafe 𝑠 urgent (Γ𝑖 = 0), as delaying them may cause audible stalls. LiveServe sorts them by playback buffer in ascending order, serving the session closest to underrun first. • U1: first-audio urgency. Requests that have not produced their first playable audio packet are next (Γ𝑖𝑠 = 1), because they are still on the audio TTFP critical path. LiveServe sorts them by ready age A𝑖 , preserving FCFS-style aging within this class. • U2: efficiency scheduling. Requests that have already produced first audio and have enough playback buffer (Γ𝑖𝑠 = 2) fall into this class. LiveServe orders them with a utility that
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
balances KV-pressure relief against excessive generation ahead of playback. U2 efficiency policy. LiveServe orders U2 requests with a lightweight utility that combines KV-pressure relief and barge-in exposure. Figure 8 illustrates why the KV term is needed in a long multi-turn dialogue. We compare a nopolicy baseline with an urgency-only scheduler that protects U0 playback-risk requests but ignores the KV state of nonurgent resident requests. The urgency-only scheduler can leave the target request resident much longer after it leaves the playback-critical path, because it is repeatedly delayed behind U0 work while its large context remains allocated. This prolonged residency consumes allocatable KV blocks and increases HBM contention under memory pressure. The barge-in term addresses the other U2 goal by penalizing requests that already have enough playback headroom but keep generating far ahead of the client. For request 𝑖 at stage 𝑠, the utility is 𝑠 𝑠 U𝑖𝑠 = 𝛽𝑠 Ukv,𝑖 − 𝛼𝑠 Cbarge,𝑖 , (1) where 𝛼𝑠 and 𝛽𝑠 tune the relative weight of barge-in exposure and KV-pressure relief for stage 𝑠. LiveServe sets their ratio offline: for each stage, we sweep candidate 𝛼𝑠 /𝛽𝑠 values on a mock workload and choose the setting that gives the best tradeoff between discarded work and KV-pressure stalls. The barge-in exposure cost penalizes large playback buffers, since additional calculated-but-unheard work may be discarded if the user interrupts. LiveServe computes 𝑠 ) max(0, P𝑖𝑠 − Psafe 𝑠 Cbarge,𝑖 = . (2) 𝑠 Psafe This cost grows when the request exceeds the safe buffer, so requests with normal playback headroom are not penalized. The KV-pressure benefit favors requests that already occupy substantial KV when the stage KV pool is close to full. Let K𝑖𝑠 denote the GPU KV cache occupied by request 𝑖, and let R𝑠,occ denote the ratio between occupied GPU KV and allocatable GPU KV at this stage. LiveServe computes 𝑠 Ukv,𝑖 = K𝑖𝑠 · R𝑠,occ . (3) This benefit grows when a long resident request is holding KV in a crowded stage. LiveServe uses U𝑖𝑠 only as a per-round ordering heuristic for U2 requests after realtime urgency has been handled.
Algorithm 1: LiveServe scheduling procedure. Input: ready set R𝑠 , arrival ages A, round budgets M𝑠 Output: next batch B𝑠 Function Schedule(R𝑠 , A, M𝑠 ): B𝑠 ← ∅ 3 foreach 𝑖 ∈ R𝑠 do 4 P𝑖𝑠 ← EstimateBuffer(𝑖) 5 Γ𝑖𝑠 ← Classify(𝑖, P𝑖𝑠 ) 6 if Γ𝑖𝑠 = 2 then 7 U𝑖𝑠 ← ComputeUtility(𝑖, P𝑖𝑠 ) 8 C0 ← {𝑖 ∈ R𝑠 | Γ𝑖𝑠 = 0} sort P𝑖𝑠 ↑ // playback risk 9 C1 ← {𝑖 ∈ R𝑠 | Γ𝑖𝑠 = 1} sort A𝑖 ↓ // first audio 10 C2 ← {𝑖 ∈ R𝑠 | Γ𝑖𝑠 = 2} sort U𝑖𝑠 ↓ // U2 efficient 11 C ← Concat(C0, C1, C2 ) // priority order 12 foreach 𝑖 ∈ C do 13 if not FitsBudget(𝑖, B𝑠 , M𝑠 ) then 14 break 15 B𝑠 ← B𝑠 ∪ {𝑖} 16 M𝑠 ← UpdateBudget(M𝑠 , 𝑖) 17 return B𝑠 1
2
efficiency score U𝑖𝑠 , which is computed after classification from the current playback buffer and KV pressure. The scheduler forms the next batch by sorting each urgency class, joining the three lists in U0-U1-U2 order, and scanning the candidates. For each candidate, if admitting it would exceed the remaining round budgets M𝑠 , namely the token budget and available KV blocks, admission stops. Otherwise, LiveServe adds the request to B𝑠 and reduces M𝑠 by the token and KV capacity that the request consumes. Requests that are not selected remain in engine state and are reconsidered in the next scheduling round. This procedure gives realtime requests strict precedence over efficiency-oriented work. U0 protects playback continuity by serving the request closest to audio drain first, while U1 protects the first-audio path with FCFS-style aging. U2 is considered only after these classes, where U𝑖𝑠 acts as a lightweight ordering heuristic rather than a global optimum or an exact knapsack solution. The same procedure applies across stages with stage-specific playback-buffer estimates, utility weights, and resource budgets.
5 Interaction-aware KV Cache Management 4.2
Scheduling Procedure
Each execution engine runs the policy at the beginning of a scheduling round, as shown in Algorithm 1. The ready set R𝑠 contains requests that can make progress at stage 𝑠 if they are admitted into the next batch. Before selecting the batch, the scheduler refreshes the per-request interaction state and estimates the playback buffer P𝑖𝑠 for each ready request. It then assigns each request an urgency class Γ𝑖𝑠 ∈ {0, 1, 2} according to the rules above. Only U2 requests need the
Section 2.3 shows that multi-turn Omni realtime interaction turns LLM-stage KV1 into long-lived session state. Reusing LLM-stage KV avoids re-prefilling conversation history and shortens next-turn TTFP. As HBM capacity is limited, systems [12, 36] offload idle KV to host DRAM and use leastrecently-used (LRU) replacement to keep only a small working set on GPU. However, deciding which KV to keep on 1We use LLM-stage KV to refer to KV cache maintained by autoregressive
model stages, such as the thinker and talker in speech-oriented Omni-LMs.
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs CPU KVCache
Scheduler DRAM
Preload Req
Req Meta
KVCache Manager Eviction Policy (§5.2) Offload
Preload
Update Candidate Heap
KVConnector HBM
Req A
BlockPool
Free Block Queue
score
version
block id
s1
7
b42
s2
4
b17
Pop & Filter
Req B
GPU
Evicted Blocks
Figure 9. Overview of the LiveServe’s KV manager. It manages multi-turn KV residency across HBM and DRAM. GPU is different in realtime interaction. Evicting KV that will be reused soon forces a DRAM-to-HBM reload onto the next-turn critical path, while LRU ranks KV by past access time rather than by when the session is likely to speak again. LiveServe manages this state with the two paths shown in Figure 9. For eviction, it uses playback progress to choose idle multi-turn KV by predicted next-use time instead of last-use time. For loading, it uses speech start and barge-in to preload offloaded KV while the user is still speaking. In this design, GPU HBM holds pinned running KV, reusable multi-turn KV, and free KV space for active execution, while CPU DRAM stores evicted multi-turn KV. The scheduler decides which requests run in the current batch, whereas the KV manager decides where reusable cross-turn state resides. The next two subsections describe the eviction policy (Section 5.1) and preload policy (Section 5.2). 5.1
Next-use-aware KV Eviction
LRU’s weakness appears when playback progress differs across sessions. A session whose LLM stage just finished may still have a long audio response left to play, so its KV is unlikely to be reused soon unless the user barges in. Another session may have older KV but be near playback completion, making its next turn more likely. Thus, last-use order can evict near-future KV blocks while keeping KV blocks whose next use is farther away. Next-use estimate. LiveServe keeps block-level KV allocation, but ranks eviction candidates by an estimate of when the cached session state will next be needed. For session 𝑖, the KV manager estimates: Tnext,𝑖 = Tplay,𝑖 + Treply,𝑖 . (4) Here Tplay,𝑖 is the remaining playback horizon, and Treply,𝑖 estimates the interval from playback completion to the next completed user input using a per-session moving average when available and a workload-level prior otherwise. The estimate is used only to order eviction candidates, so it need not be an exact wall-clock prediction. When playback accounting is unavailable or noisy, LiveServe falls back to
observable progress counters, using longer unfinished responses as a sign of later reuse. If the monitor reports speech start or barge-in, LiveServe treats the session as immediate reuse and protects its resident KV from normal eviction. Eviction policy. When HBM pressure requires freeing KV capacity, the KV manager considers only idle-resident multiturn KV. Pinned running KV and sessions marked for immediate reuse are excluded. The manager orders eligible sessions by decreasing Tnext and scans from the session whose next use is farthest away. It evicts blocks from that session until enough HBM is released or the session has no evictable blocks left, then continues with the next session if needed. Thus, LiveServe changes the order of eviction targets while keeping allocation and eviction block-level. Within a selected session, LiveServe gives suffix blocks higher eviction priority than prefix blocks. Prefix blocks are shared by more future turns and are more expensive to reconstruct, so keeping them preserves prefix continuity and improves KV reuse. Suffix blocks lie near the tail of the cached conversation state, tend to have a shorter reuse horizon, and therefore impose lower reconstruction cost and smaller impact on subsequent turns when evicted. This policy preserves the block allocator used by existing serving engines while changing only the eviction order. As a result, LiveServe can free HBM for active requests while reducing short-interval evict-and-reload cycles for sessions close to their next turn. 5.2
Speech-triggered KV Preloading
Next-use-aware eviction reduces unnecessary KV swap-in and swap-out between GPU HBM and host DRAM, but it cannot remove the latency of a necessary reload. If offloaded KV is loaded only after the next-turn request reaches the LLM stage, the DRAM-to-HBM transfer delays prefill and increases TTFP. Thus, LiveServe starts preload at speech start or barge-in, before the full user input reaches the model. Preload trigger. Speech start is an early signal that the next turn is coming, not that the next turn is ready. After the user starts speaking, the utterance still needs to finish, be encoded, and be routed through the orchestrator. Barge-in creates a similar window because it interrupts the current response while the user continues speaking a new instruction. LiveServe overlaps this speaking interval with KV transfer, removing part of the DRAM-to-HBM latency from the nextturn critical path. Preload admission. When a preload trigger arrives, the KV manager first protects any resident KV of that session from eviction. If the session KV is offloaded, LiveServe treats preload as best-effort background work rather than foreground work. The manager admits an asynchronous DRAMto-HBM transfer only when the remaining time before LLMstage execution is enough to hide the transfer cost under current pressure. A preload that needs more HBM space may
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
evict later-use idle KV using the policy in Section 5.1. When the admission check fails, LiveServe skips the preload and lets the normal LLM-stage path load missing KV. LiveServe bounds background preload so it can’t interfere with live work. Under transient pressure, the admission may wait briefly and re-evaluate, but the time available before LLM-stage execution shrinks during the wait, so preload cannot extend the next-turn deadline. After a preload completes, LiveServe keeps the warmed KV on GPU for a short time so the next turn can reuse it, but caps the total amount of such protected KV. If the transfer is incomplete, canceled, or evicted before reuse, the next turn falls back to synchronous loading, so preloading affects latency but not correctness.
6
Implementation
We implement LiveServe on top of vLLM-Omni [54], a stageoriented extension of vLLM for multimodal serving. LiveServe adds three components to this substrate: an interactionaware scheduler, a monitor-driven request-state path, and a hierarchical KV manager for multi-turn KV residency. Together, they are implemented in ∼6000 lines of Python. The implementation preserves the engine’s original budget checks and default allocator as fallbacks, so missing metadata or disabled policies reduce to the original serving behavior. Scheduler and monitor integration. Each autoregressive stage keeps the original continuous-batching loop, but replaces FCFS admission order with the urgency hierarchy in Section 4. A scheduler mixin reads the monitor’s per-session interaction and pressure state at the start of each scheduling round. It estimates playback buffer, computes the U0/U1/U2 class and U2 utility, and then calls the engine’s existing feasibility checks for token budget and available KV blocks. This keeps the policy change local to request ordering, while memory allocation, preemption, abort handling, and decode execution remain delegated to the underlying engine. Preload requests use the same scheduling path but are tagged as cancellable background work. During bursts, the mixin can cancel running preloads or hold new ones before they compete with foreground U0/U1 work. A canceled preload falls back to synchronous KV loading when the next turn actually runs, preserving the correctness path. Hierarchical KV cache manager. The KV manager extends the paged KV block pool and the existing HBM-DRAM offload connector. Rather than changing the upstream block layout, LiveServe stores policy metadata in side tables attached to requests, sessions, and KV blocks. These side tables track next-use estimates, block position, and preload protection state as turns move between active, idle, offloaded, and preload-protected states. For eviction, the implementation maintains the indexed candidate heap described in Section 5.1, while keeping the original LRU allocator as a fallback on every allocation. The indexed path can run in logging-only mode or replace LRU
only when the metadata needed by the next-use policy is available. For loading, the entrypoint turns speech-start and barge-in events into best-effort preload requests and uses the offload connector’s asynchronous transfer path when admission succeeds. Cold misses, admission skips, and canceled preloads fall back to the normal foreground load path. Fail-closed operation and instrumentation. All LiveServe mechanisms are optional at runtime and fail back to the substrate’s original behavior. Missing playback-buffer telemetry disables interaction-aware ordering, while missing U2 utility inputs reduce only U2 ordering to ready-age ordering. Sparse eviction metadata returns the block pool to default LRU, and failed preload admission returns the next live request to synchronous KV loading. The prototype records counters for policy fallbacks and preload outcomes, which we use to validate in evaluation that the scheduler and KV manager exercise the intended design paths.
7
Experimental Evaluation
LiveServe is designed to improve the serving performance of Omni realtime interaction workload. We evaluate whether our interaction-aware policies improve user-facing latency and serving efficiency over existing serving systems. Our evaluation focuses on the following questions: • How does LiveServe compare with baselines under realtime interaction workloads, notably with user barge-in? • How much does interaction-aware scheduling improve system performance under the playback-continuity SLO? • How much does interaction-aware KV management reduce reload overhead for multi-turn sessions?
7.1
Experiment Settings
Datasets. We evaluate LiveServe with three complementary data sources that cover single-turn speech, multi-turn voice interaction, and mixed modality Omni requests. • ShareGPT conversations. The single-turn workload is constructed from both short and long conversations to stress first-token latency under different context lengths. We use the ShareGPT Chinese-English 90K corpus [39] as the source of realistic conversational prompts. • Interactive traces. The multi-turn workload is built from retained interaction traces [17]. Each record has a session ID, a request timestamp, query and response token lengths, and a turn index. This structure preserves repeated user turns and model speech replies within the same session. • StreamingBench and mixed Omni data. The video and multimodal portion uses StreamingBench [24] as the source of video questions and media assets. We combine these video events with retained interactive voice sessions to form mixed text, speech, and video workloads.
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
ShareGPT
throughput (req/s)
vLLM w/o offload
vLLM w/ offload
1.6
1.35
1.2
1.20
0.8
1.05
0.4
0.90 600
1200
1800
6000
Interactive
throughput (req/s)
audio TTFP.p90 (ms) 3.0
4
2.5
3
2 6000
9000
1.5
audio TTFP.p90 (ms)
Mixed
18000
15000
30000
45000
audio TTFP.p90 (ms) 3.2
4
2.8
3
2.4
2
2.0 4000
8000
12000
audio TTFP.p90 (ms) Qwen3-Omni
uses LRU to maintain the GPU HBM working set. Unless otherwise specified, KV offloading is enabled by default. Models and deployment. We use two representative speechoriented Omni-LMs: Qwen3-Omni [52] and Ming-Flash-Omni 2.0 [30]. We follow the official vLLM-Omni deployment configurations and scale them to an 8-GPU server. Qwen3-Omni uses a three-stage pipeline2 , with DP=4 for the thinker, DP=4 for the talker. Ming-Flash-Omni 2.0 uses a two-stage pipeline with a TP=2, DP=2 thinker and a DP=4 talker stage. All compared systems share the same hardware allocation, model weights, and serving configurations. Metrics. We evaluate LiveServe with metrics that reflect the user-facing goals of Omni realtime interaction: fast first audio, smooth playback, and serving throuhgput.
2.0
3000
throughput (req/s)
12000
audio TTFP.p90 (ms)
5
1
LiveServe
15000
30000
45000
audio TTFP.p90 (ms) Ming-Flash-Omni
Figure 10. End-to-end throughput-latency frontier across two Omni-LMs and three workloads. Each curve connects results over concurrency-pressure values 𝑐 ∈ {2, 4, 8, 12, 16}; higher and further-left points are better.
Workloads. We generate online request arrivals using both synthetic and trace-driven patterns. For the synthetic setting, requests arrive according to a Poisson distribution, and we evaluate a range of offered loads to measure the highest request rate each system can sustain under the realtime SLO. For trace-driven setting, we use BurstGPT [48] arrivals to capture short-term load spikes in interactive services. We model user barge-in as a client-side policy. Each request independently triggers a barge-in event with Bernoulli probability 𝑝 bi ; unless a barge-in experiment is explicitly reported, we set 𝑝 bi = 0. Barge-in sensitivity experiments use 𝑝 bi ∈ {0.0, 0.3, 0.7, 1.0} to cover different barge-in regimes. For requests with barge-in, the cut time is anchored at audio TTFP and then sampled from the dataset-derived distribution of output audio durations. Baselines. We compare LiveServe against two baselines. The first is vLLM-Omni [54] without KV offloading, denoted as vLLM-Omni-wo. It serves Omni-LM stages with the default scheduling and memory management policies. The second is vLLM-Omni with vLLM-style KV offloading enabled for multi-turn sessions, which offloads KV cache to DRAM and
• Audio time-to-first-packet (TTFP) is the elapsed time from the end of the replayed user turn, as observed by the benchmark client, to the first decodable audio fragment delivered by the server. By default we report P90 TTFP. • Real-time factor (RTF) is the ratio between audio generation time and the audio duration. RTF < 1 indicate that the system generates audio faster than real-time playback. • Playback continuity measures whether the streamed audio plays smoothly after the first chunk arrives. A request is continuous if playback gaps stay below 100ms, the default threshold in vLLM-Omni benchmark. Requests with bargein are excluded unless stated otherwise. • We report throughput as requests per second (RPS) over the steady-state window. Useful RPS is the highest offered load at which the system meets the continuity SLOs. Testbed. Experiments were conducted on a single H200 server with eight NVIDIA H200 GPUs. The host contains two Intel Xeon Platinum 8480C sockets, 56 physical cores per socket, and 2.0 TiB of DRAM. Our implementation is developed based on vLLM 0.20.0 and vLLM-Omni 0.20.1. 7.2
Main Results
Throughput-latency frontier. Figure 10 reports the end-toend throughput–latency frontier as the c-bound concurrency of online sessions changes. The top row uses Qwen3-Omni and the bottom row uses Ming-Flash-Omni 2.0; within each row, the panels correspond to ShareGPT, interactive, and mixed workloads. The x-axis is P90 audio TTFP and the y-axis is completed request throughput, so better systems move toward the upper-left region. Across the two Omni-LMs and three workloads, LiveServe consistently improves the frontier over both vLLM-Omni baselines. On ShareGPT, where the workload is less dominated by multi-turn KV reuse, LiveServe achieves comparable or higher peak throughput while reducing high-concurrency P90 audio TTFP by about 2× on both models. 2 By default, Qwen3-Omni colocates the encoder with the thinker. Its vocoder
is a lightweight CNN module and is colocated with the talker.
0
mean
p50
p90
Audio TTFP
90
70
p95
vLLM-Omni-wo vLLM-Omni Ours
80
2
4
8
12
16
Concurrency pressure c
Figure 11. Interactive playback continuity and generatedtoken waste under concurrency and barge-in pressure.
1500 TTFP -40%
1000 500 0
Poisson
BurstGPT
1
0
Ours 2000
1000
0
0 0.25 0.5 0.75 1 Barge-in probability
0 0.25 0.5 0.75 1 Barge-in probability
Figure 13. Sensitivity to configured barge-in probability on the ShareGPT audio workload.
Ours
TTFP -26%
RPS +44%
Effecitve RPS
P90 Audio TTFP (ms)
vLLM-Omni
P90 TTFP (ms)
500
vLLM-Omni 2
RPS +17%
1.0
vLLM-Omni
0.5 0.0
Poisson
BurstGPT
Figure 12. Effect of interaction-aware scheduling under Poisson and BurstGPT arrivals using Qwen3-Omni audio serving.
On interactive traces, repeated turns increase KV pressure and scheduling contention, making the gap larger. For Qwen3-Omni, LiveServe improves peak throughput by 56– 78% over the baselines while also reducing P90 audio TTFP at the same concurrency; at moderate concurrency, it further improves throughput by 28.5% and lowers P90 audio TTFP by 39.8% over the offloading baseline. Ming-Flash-Omni 2.0 shows the same upward-left shift, indicating that the benefit is not specific to one Omni-LM. On the mixed workload, LiveServe remains effective under heterogeneous stage demand. At the largest c-bound, it improves throughput by 12–16% over the best baseline and substantially reduces P90 audio TTFP, with the reduction reaching about 42% on Ming-Flash-Omni 2.0. Overall, simple KV offloading makes longer sessions feasible, but without interaction-aware scheduling and KV management, the saved memory does not translate into the same first-audio responsiveness. Tail-latency distribution at fixed concurrency. The left panel of Figure 11 drills into the Qwen3-Omni ShareGPT audio workload without configured barge-in at 𝑐 = 8. LiveServe lowers the whole visible latency distribution: the median decreases from 0.86 s to 0.53 s, while P90 and P95 fall from 1.38 s and 1.45 s to 0.84 s and 0.92 s. This shows that the scheduler compresses both typical and tail first-audio delays under the same concurrency pressure. Playback continuity. The right panel of Figure 11 evaluates whether the lower latency comes at the cost of streamedaudio continuity. Under heavier pressure, LiveServe degrades more gracefully: at 𝑐 = 12, continuity remains 91.8%, compared with 81.3% for vLLM-Omni with offloading; at 𝑐 = 16, LiveServe keeps 87.5% continuity, compared with 76.6% and 70.3% for the two baselines. Thus, the U0/U1/U2 scheduler
+Schedule
1.0 0.5 0.0
No barge-in pbi = 0.5
Norm. P90 TTFP
1000
100
Effecitve RPS
vLLM-Omni Ours
Norm. RPS
1500
Continuity (%)
Audio TTFP (ms)
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
+Preload
+Evict
1.0 0.5 0.0
No barge-in pbi = 0.5
Figure 14. Component ablation on the interactive workload.
preserves enough playback buffer for continuity while reducing work on well-buffered U2 sessions. Arrival distribution. Figure 12 compares LiveServe with the KV-offloading baseline under Poisson and trace-driven BurstGPT arrivals. This experiment uses an audio ShareGPTstyle workload with 𝑐 = 8, 32 requests, request rate 4 RPS for the Poisson case, and a BurstGPT manifest with the same peak request rate. Under Poisson arrivals, LiveServe reduces P90 audio TTFP from 1.13 s to 0.68 s and raises effective throughput from 0.82 to 1.18 RPS. Under BurstGPT arrivals, where short-term bursts leave less scheduling slack, LiveServe still lowers P90 audio TTFP from 1.63 s to 1.20 s and improves effective throughput from 0.96 to 1.12 RPS. This shows the scheduling policy remains robust even under bursty arrivals. Sensitivity to user barge-in. Figure 13 reports sensitivity to configured ShareGPT barge-in probability. The experiment uses Qwen3-Omni audio serving with 𝑐 = 8. On effective RPS, LiveServe outperforms the offloading baseline across the full range: at 𝑝 bi = 0.5 and 𝑝 bi = 0.75, throughput improves by 2.6× and 2.0×, respectively. This experiment shows the corresponding P90 audio TTFP, where LiveServe is also consistently lower, cutting latency by more than half at the same two barge-in probabilities. The improvement comes from the playback-buffer-aware urgency hierarchy: U0/U1 requests stay ahead of well-buffered U2 sessions, while the U2 barge-in exposure term limits discardable audio work before an abort arrives.
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
0.4
vLLM-Omni Ours
0.3 0.2
500
1000
40 Thinker Talker Vocoder
20 0
1500
Audio TTFP p90 (ms)
20
40
Player vLLM-Omni Ours
60
Time since request start (s)
Figure 15. Audio generation pacing. The left panel varies concurrency on a ShareGPT audio workload; the right panel illustrates generation and playback completion over time.
vLLM-Omni Ours
40
44.1
30
26.1
20
14.6
10 0
12.4
6.7 0
0
0
2.4
0.63
0.1
6.5 1.6
0.3
TTFT comp. (ms)
Waste ratio (%)
50
300
302.1 ms
0.5
0.7
0.9
200 127.8 ms
100 0 vLLM-Omni
Ours
Figure 16. Impact of barge-in and reload pressure. Left: wasted-token ratio under different barge-in probabilities. Right: latency breakdown of a reload-pressure target request.
7.3
30 20 10 0 0
10 20 Replay time (s)
1.00
30
0.75 0.50 0.25 0.00
Time
KV resident
Figure 17. Effect of KV-aware deferral on thinker KV residency. Left: thinker GPU KV timeline. Right: normalized completion time and resident-KV footprint.
wait load prefill decode
3.2
Barge-in probability
28.8s
60
KV-aware
Norm. value
0.5
40
15.8s
0.6
KV-unaware
80
KV cache (MB)
Completion (%)
P50 Audio RTF
100 0.7
Analysis
We next run controlled experiments to study where LiveServe’s gains come from and how robust they are across workload conditions. Unless otherwise stated, this analysis uses Qwen3-Omni in audio mode. Component ablation. Figure 14 adds LiveServe’s components one by one using Qwen3-Omni under two settings: without barge-in and with barge-in probability 𝑝 bi = 0.5. Without barge-in, the full system reduces P90 TTFP by 29.8% and improves RPS by 8.8%. With barge-in enabled, the gains increase to 39.8% lower P90 TTFP and 28.5% higher RPS. The staged improvements show that scheduling, preload, and eviction are complementary rather than redundant. RTF-latency tradeoff. Figure 15 shows why LiveServe does not simply generate speech as fast as possible. In the ShareGPT audio workload with 𝑝 bi = 0.5, both systems keep P90 RTF below 1, so playback can remain faster than real time. However, as concurrency grows, LiveServe converts work on well-buffered U2 sessions into lower first-audio latency. At 𝑐 = 8, P90 audio TTFP drops from 1.54 s to 0.91 s, while median RTF remains below real time. The long-horizon example explains the mechanism: the baseline finishes modelside generation in about 8.2 s while the player consumes the response over about 65.9 s, accumulating a large stageaware playback buffer that can be discarded after barge-in. LiveServe stretches generation to about 55.3 s, closer to playback progress, preserving enough buffer for continuity while freeing decode capacity for U1 first-audio requests.
Barge-in token waste. The left panel of Figure 16 profiles generated-but-unheard production tokens under different configured barge-in probabilities. With no barge-in, both systems waste no generated tokens. As the barge-in probability increases, the vLLM-Omni waste ratio rises to 44.06%, because generation can run far ahead of playback until an abort arrives. LiveServe’s U2 barge-in exposure term limits this discardable buffered work, reducing the waste ratio to at most 12.38% and eliminating about 72%–78% of wasted generated tokens across barge-in settings. First-token critical path under reload pressure. The right panel of Figure 16 shows a multi-turn request from an interactive workload, where interaction-aware KV management removes reload work from the user-visible path. The offloading baseline spends 71.0 ms on on-path KV reload and reaches 302.1 ms text TTFP. With a warm prefetch hit, LiveServe eliminates the reload segment and reduces text TTFP to 127.8 ms, a 57.7% reduction, by moving bulk HBM– DRAM transfer off the next-turn critical path. Together, these analyses show that LiveServe’s gains come from complementary mechanisms. Playback-aware scheduling protects U0/U1 requests and limits discardable U2 work, improving effective throughput and first-audio latency under barge-in. Interaction-aware KV management ensures that when a session returns for its next turn, the required state is already resident or prefetched. 7.4
Microbenchmarks
KV residency timeline. Figure 17 profiles KV-aware U2 scheduling on the Qwen3-Omni interactive workload. Under KV pressure, KV-aware ordering favors resident longcontext requests, allowing them to finish and release HBM earlier than KV-unaware ordering. The right panel summarizes this effect with a shorter replay and lower normalized resident-KV footprint, showing why efficiency-class scheduling should account for memory residency. Playback-continuity timeline. Figure 18 stress-tests playback continuity over time on ShareGPT with BurstGPT arrivals (𝑐 = 32, 12 RPS). Without barge-in, both systems start
100
Continuity (%)
Continuity (%)
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
80 60 40
vLLM-Omni Ours
20 0 0
25
50
75
Time (s)
100
100
125
(a) No barge-in
80 60 40
vLLM-Omni Ours
20 0 0
25
50
75
Time (s)
100
125
(b) 𝑝 bi = 0.5
Figure 18. Playback-continuity timeline. The right panel enables barge-in with triggers anchored after TTFP. Table 1. Effect of the eviction index optimization. System LiveServe w/o index
Avg. OH P90 OH Effective E2E P90 (ms) (ms) RPS (ms) 0.093 5.311
0.222 8.270
2.625 1.970
3069 3381
at full continuity, but LiveServe ends higher than vLLMOmni. With barge-in at 𝑝 bi = 0.5, LiveServe keeps a clearer advantage by limiting U2 barge-in exposure and avoiding obsolete in-flight audio work. Eviction-index overhead. Table 1 measures eviction overhead in the interactive multi-turn dialogue scenario without barge-in. Compared with tail scanning, LiveServe’s heapbased eviction index reduces average overhead from 5.31 ms to 0.093 ms and P90 overhead from 8.27 ms to 0.222 ms, while improving effective QPS from 1.97 to 2.63. This shows that indexed eviction avoids making next-use-aware eviction a scheduler bottleneck. Takeaway. The microbenchmarks validate the key design assumptions behind the end-to-end gains. For memory management, the goal is not to minimize KV residency at all times, but to keep the right session KV resident before its next turn. For audio scheduling, the scheduler should preserve enough playback buffer for continuity without accumulating excessive discardable audio. Together, next-use-aware eviction, and speech-triggered preload improve both memory stability and user-visible latency in multi-turn realtime interaction.
8
Related Work
Efficient LLM serving. Many systems optimize LLM serving from different parts of the inference pipeline [2, 7, 20, 25, 55, 58, 59]. vLLM [20] and SGLang [58] are the most popular open-sourced repositories, and they cooperated with system optimizations such as continuous batching [55], prefill/decode disaggregation [34, 59], and chunked prefill [2, 16] to improve the overall system performance of serving LLMs. Some systems [40, 42, 50] propose scheduling algorithms to handle dynamic workloads. Besides, Andes [25] and TokenFlow [7] observe that LLMs often generate text faster than
users can consume it, so continuously decoding can waste GPU time without improving perceived quality. They schedule by QoE and token-buffer status, pausing well-buffered requests to serve stall-prone ones. In contrast, LiveServe focuses on a more complex multi-stage Omni-LM pipeline, where scheduling follows audio playback progress to avoid barge-in waste from each LLM stage running too far ahead. Multi-turn KV management. Multi-turn interaction turns KV cache from a per-request workspace into reusable session state, motivating systems that preserve conversation history across turns instead of recomputing the full prompt [12, 13, 17, 18, 26, 27, 36, 56]. CacheAttention [12] and FlashGen [18] use multi-tier caching and request scheduling to accelerate multi-turn serving. Mooncake [36] builds a KV-centric disaggregated architecture that pools cache storage across inference nodes. LMCache [26] provides a KV cache layer spanning GPU, CPU, disk, and remote storage. Pensieve [56] manages GPU-CPU KV residency for conversation state, evicting cached chunks based on inactivity and recomputation cost while supporting non-contiguous cached context. These systems optimize KV reuse, capacity, and transfer cost, while LiveServe focuses on when session KV should reside in GPU memory under realtime interaction, where audio playback and barge-in determine the next-turn latency path. Serving systems for multimodal models. Multimodal serving spans both understanding models that consume nontext inputs and produce text, and generation models that also synthesize speech, images, or video. Early systems mainly focused on multimodal understanding workloads. EPD disaggregation [41], EPD-Serve [5], HydraInfer [11], and SpaceServe [23] disaggregate multimodal encoding, LLM prefill, and LLM decode into different workers to reduce encoderinduced interference. ModServe [37], ElasticMM [28], and TCM-Serve [33] allocate resources according to modalityand stage-level workload heterogeneity. Generation-oriented Omni serving further complicates the problem because heterogeneous stages also appear on the output side. Existing systems either expose Omni inference as stage-oriented runtimes [45, 54], optimize generic any-toany computation graphs [8], or target streaming speech models with speech-aware scheduling [19]. In contrast, LiveServe targets realtime Omni interaction with interaction-aware scheduling and KV management for playback, barge-in, firstaudio latency, and multi-turn sessions.
9
Conclusion
Existing stage-oriented Omni serving systems ignore live interaction signals, causing over-generation and poorly timed multi-turn KV management. We present LiveServe, which tracks playback progress and speech activity, and applies interaction-aware scheduling together with next-use-aware KV eviction and preloading. Across Omni-LM models and realtime workloads, LiveServe lowers P90 audio TTFP up
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
to 2.21×, while improving completed-request throughput up to 1.56×, and cuts wasted generated tokens by 72-78% under barge-in. LiveServe shows that making Omni serving aware of live interaction state can substantially improve both system efficiency and user-perceived responsiveness.
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [2] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369 (2023). [3] Aleksandr Algazinov, Matt Laing, and Paul Laban. 2025. MATE: LLMPowered Multi-Agent Translation Environment for Accessibility Applications. arXiv preprint arXiv:2506.19502 (2025). [4] Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). [5] Fan Bai, Pai Peng, Zhengzhi Tang, Zhe Wang, Gong Chen, Xiang Lu, Yinuo Li, Huan Lin, Weizhe Lin, Yaoyuan Wang, et al. 2026. EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascend. arXiv preprint arXiv:2601.11590 (2026). [6] Siyu Cao, Hangting Chen, Peng Chen, Yiji Cheng, Yutao Cui, Xinchi Deng, Ying Dong, Kipper Gong, Tianpeng Gu, Xiusen Gu, et al. 2025. HunyuanImage 3.0 Technical Report. arXiv preprint arXiv:2509.23951 (2025). [7] Junyi Chen, Chuheng Du, Renyuan Liu, Shuochao Yao, Dingtian Yan, Jiang Liao, Shengzhong Liu, Fan Wu, and Guihai Chen. 2026. TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling. In Proceedings of the 21st European Conference on Computer Systems. 497–513. [8] Jae-Won Chung, Jeff J Ma, Jisang Ahn, Yizhuo Liang, Akshay Jajoo, Myungjin Lee, and Mosharaf Chowdhury. 2026. Cornserve: A distributed serving system for any-to-any multimodal models. arXiv preprint arXiv:2603.12118 (2026). [9] Google Deepmind. 2026. Gemini Omni: Speak it. See it. Share it. https://gemini.google/overview/video-generation/ [10] Matendo Didas. 2026. A multi-agent artificial intelligence-powered architecture for customer experience management. International Journal of Advanced Computer Research 16 (2026), 76. [11] Xianzhe Dong, Tongxuan Liu, Yuting Zeng, Liangyu Liu, Yang Liu, Siyu Wu, Yu Wu, Hailong Yang, Ke Zhang, and Jing Li. 2025. Hydrainfer: Hybrid disaggregated scheduling for multimodal large language model serving. arXiv preprint arXiv:2505.12658 (2025). [12] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. {CostEfficient} large language model serving for multi-turn conversations with {CachedAttention}. In 2024 USENIX annual technical conference (USENIX ATC 24). 111–126. [13] Shiwei Gao, Youmin Chen, and Jiwu Shu. 2025. Fast state restoration in llm serving with hcache. In Proceedings of the Twentieth European Conference on Computer Systems. 128–143. [14] Google Gemini. 2025. Gemini Live – Ask AI a question in any mode you choose. https://gemini.google/overview/gemini-live/ [15] Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, et al. 2025. Ming-Omni: A Unified Multimodal Model for Perception and Generation. arXiv preprint arXiv:2506.09344 (2025).
[16] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. 2024. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference. arXiv preprint arXiv:2401.08671 (2024). [17] Shipeng Hu, Guangyan Zhang, Yuqi Zhou, Yaya Wei, Ziyan Zhong, and Jike Chen. 2026. Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness. In 24th USENIX Conference on File and Storage Technologies (FAST 26). USENIX Association, Santa Clara, CA, 101–116. https://www.usenix. org/conference/fast26/presentation/hu-shipeng [18] Jinwoo Jeong and Jeongseob Ahn. 2025. Accelerating llm serving for multi-turn dialogues with efficient resource management. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1–15. [19] Keisuke Kamahori, Wei-Tzu Lee, Atindra Jha, Rohan Kadekodi, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2026. VoxServe: Streaming-Centric Serving System for Speech Language Models. arXiv preprint arXiv:2602.00269 (2026). [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [21] Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Shijian Wang, Guanting Dong, Jiajie Jin, Hao Wang, Yinuo Wang, Ji-Rong Wen, Yuan Lu, et al. 2026. Omnigaia: Towards native omni-modal ai agents. arXiv preprint arXiv:2602.22897 (2026). [22] Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. 2025. Baichuan-Omni-1.5 Technical Report. arXiv preprint arXiv:2501.15368 (2025). [23] Zhicheng Li, Shuoming Zhang, Jiacheng Zhao, Siqi Li, Xiyu Shi, Yangyu Zhang, Shuaijiang Li, Donglin Yu, Zheming Yang, Yuan Wen, et al. 2026. SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs. Advances in Neural Information Processing Systems 38 (2026), 79272–79296. [24] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. 2024. StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding. arXiv:2411.03628 [cs.CV] doi:10.48550/arXiv.2411.03628 [25] Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. 2024. Andes: Defining and enhancing qualityof-experience in llm-based text streaming services. arXiv preprint arXiv:2404.16283 (2024). [26] Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al. 2025. Lmcache: An efficient KV cache layer for enterprise-scale LLM inference. arXiv preprint arXiv:2510.09665 (2025). [27] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. 2024. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference. 38–56. [28] Zedong Liu, Shenggan Cheng, Guangming Tan, Yang You, and Dingwen Tao. 2026. Elasticmm: Efficient multimodal llms serving with elastic multimodal parallelism. Advances in Neural Information Processing Systems 38 (2026), 94264–94289. [29] Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Xiaobo Xia, et al. 2026. Openomni: Advancing open-source omnimodal large language models with progressive multimodal alignment and real-time emotional speech synthesis. Advances in Neural Information Processing Systems 38 (2026), 158925–158953.
Xiangyu Zhi, Peiqi Yin, Sheng Guan, Chenguang Zheng, James Cheng, and Xiao Yan
[30] Bowen Ma, Cheng Zou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Dandan Zheng, Fudong Wang, Furong Xu, et al. 2025. Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation. arXiv preprint arXiv:2510.24821 (2025). [31] OpenAI. 2025. Introducing gpt-realtime and Realtime API updates for production voice agents. https://openai.com/index/introducing-gptrealtime/ [32] Danny Harnik Or Ozeri. 2026. Inside vLLM’s New KV Offloading Connector: Smarter Memory Transfer for Maximizing Inference Throughput. https://vllm.ai/blog/2026-01-08-kv-offloading-connector [33] Konstantinos Papaioannou and Thaleia Dimitra Doudali. 2026. TCMServe: Modality-aware Scheduling for Multimodal Large Language Model Inference. arXiv preprint arXiv:2603.26498 (2026). [34] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132. [35] William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 4195–4205. [36] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving {LLM} chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). 155–170. [37] Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, et al. 2025. Modserve: Modality-and stage-aware resource disaggregation for scalable multimodal model serving. In Proceedings of the 2025 ACM Symposium on Cloud Computing. 817–830. [38] ByteDance Seed. 2026. Doubao Realtime Voice Model. https://seed. bytedance.com/en/special/realtime_voice [39] shareAI. 2023. ShareGPT Chinese-English 90K. https://huggingface. co/datasets/shareAI/ShareGPT-Chinese-English-90k. [40] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E Gonzalez, and Ion Stoica. 2024. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 965–988. [41] Gursimran Singh, Xinglu Wang, Yifan Hu, Timothy Tin Long Yu, Linzi Xing, Wei Jiang, Zhefeng Wang, Bai Xiaolong, Yi Li, Ying Xiong, et al. [n. d.]. Efficiently Serving Large Multimodal Models Using EPD Disaggregation. In Forty-second International Conference on Machine Learning. [42] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). 173–191. [43] Nvidia Nemotron Team. 2026. Nemotron Voicechat Model. https: //build.nvidia.com/nvidia/nemotron-voicechat [44] Qwen Team. 2026. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804 (2026). [45] SGLang Team. 2026. SGLang Omni: High-Performance Multi-Stage Pipeline Framework for Omni Models. https://github.com/sgl-project/ sglang-omni [46] Xiaomi Mimo Team. 2026. Xiaomi MiMo-V2-Omni. https://mimo. xiaomi.com/mimo-v2-omni [47] Bairui Wang, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, et al. 2025. LongCatFlash-Omni Technical Report. arXiv preprint arXiv:2511.00279 (2025). [48] Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2025. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems.
In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. ACM, New York, NY, USA, 5831–5841. doi:10.1145/3711896.3737413 [49] Zichen Wen, Boxue Yang, Junlong Ke, Jiajie Huang, Chenfei Liao, Junxi Wang, Xuyang Liu, and Linfeng Zhang. 2026. EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant. arXiv preprint arXiv:2605.10343 (2026). [50] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2026. {FastServe}:{Iteration-Level} Preemptive Scheduling for Large Language Model Inference. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). 57–74. [51] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. 2025. Qwen2.5omni technical report. arXiv preprint arXiv:2503.20215 (2025). [52] Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. 2025. Qwen3-Omni Technical Report. arXiv preprint arXiv:2509.17765 (2025). [53] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [54] Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, et al. 2026. vLLM-Omni: Fully Disaggregated Serving for Anyto-Any Multimodal Models. arXiv preprint arXiv:2602.02204 (2026). [55] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538. [56] Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful large language model serving with pensieve. In Proceedings of the Twentieth European Conference on Computer Systems. 144–158. [57] Z.AI. 2026. GLM-Image: Auto-regressive for Dense-knowledge and High-fidelity Image Generation. https://z.ai/blog/glm-image [58] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583. [59] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210.