arXiv:2606.18741v1 [cs.DC] 17 Jun 2026
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving Haipeng Yuan
Kaining Zheng
Yongshu Bai
[email protected] Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
[email protected] Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
[email protected] Zhejiang Lab Hangzhou, China
Yuchen Zhang
Yunquan Zhang
Baodong Wu
[email protected] Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
[email protected] Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
[email protected] Infinigence AI Beijing, China
Xiang Gao
Daning Cheng
[email protected] Zhejiang Lab Hangzhou, China
[email protected] Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
Abstract
1
Current large language model (LLM) inference systems universally deploy ultra-large-scale models using a combination of Tensor Parallelism (TP) and Pipeline Parallelism (PP). However, existing systems treat the model parallelism topology as a static configuration that cannot be flexibly adjusted at runtime. This rigid design creates a fundamental contradiction with the dynamically changing inference workloads in real-world scenarios. State-of-the-art systems lack online reconfiguration capabilities and can only switch configurations by restarting the service, resulting in several minutes of service interruption, KV cache loss, and prohibitive recomputation overhead. To address this problem, this paper presents ReMP, a runtime model parallelism reconfiguration framework that supports low downtime. ReMP achieves dynamic adjustment through three key techniques: (1) decoupling the model parallelism topology from runtime state to avoid full service reconstruction; (2) designing a two-dimensional KV cache migration mechanism to preserve reusable cache states after TP/PP changes; and (3) implementing end-to-end online reconfiguration. Experiments demonstrate that ReMP can complete most topology switches within 1-7 seconds on models ranging from 7B to 70B parameters, achieving speedups of tens to over a hundred times compared to the restart approach. Moreover, ReMP significantly outperforms fixed configurations under dynamic workloads, delivering superior performance in terms of TTFT, TPOT, and output throughput.
Large language model (LLM) serving systems increasingly rely on model parallelism to deploy models exceeding singleGPU memory capacity. In practice, inference engines combine tensor parallelism (TP) and pipeline parallelism (PP) to distribute weights, computations, and key-value (KV) caches across multiple GPUs. However, existing systems treat model parallel topology as a static pre-deployment configuration: once launched with a given TP/PP degree, weight partitioning, KV cache layout, workers, and communication groups become tightly coupled to this topology. Unlike data parallelism (DP), which is loosely coupled to hardware and natively elastic, TP/PP face fundamental online adjustment challenges due to deep entanglement with model partitioning and state layout. This rigidity inflexibilizes the overall parallel configuration, becoming a critical bottleneck for LLM serving systems to adapt to dynamic workloads. However, LLM inference traffic in real-world production environments exhibits significant dynamism. Our analysis of online traffic data from two major service providers reveals that their workloads follow a distinct periodic pattern: there are stable and substantial disparities in traffic volume across different time periods of the day, shown in Figure 1. More critically, the optimal combination of Tensor Parallelism and Pipeline Parallelism varies across different traffic ranges. During low-load periods, the core objective should be to minimize per-request latency, prioritizing a higher TP degree to accelerate single-sample computation. In contrast, during high-load peaks, the focus must shift to maximizing system throughput, leveraging a higher PP degree to fully
Keywords: LLM inference serving; Runtime reconfiguration; Model parallelism; Adaptive serving
Introduction
Conference’17, July 2017, Washington, DC, USA
First Author et al.
1.0 0.8 0.6 0.4 0.2 0.0
0
8
16
Hour of day
Normalized total token usage
Normalized total token usage
exploit pipeline processing capacity while avoiding the communication overhead bottleneck introduced by excessive TP. Consequently, there exists a fundamental contradiction between the static and rigid model parallel topology and the dynamic workload characteristics of LLM serving systems. 1.0 0.8 0.6 0.4 0.2 0.0
0
8
16
Hour of day
(a) Token Usage Profile of IN- (b) Token Usage Profile of ZheFINIGENCE Model Service jiang Lab Model Service
Figure 1. Token Usage Trend Stability of Two Large Model Services During Different Time Periods on the Same Day for One Model.Blue bars denote the average traffic over all observed days, and colored dashed lines denote daily real traffic.Traffic is normalized by setting the value at 10:00 to 1, with other time points scaled proportionally. Current mainstream LLM serving systems largely lack the capability to dynamically adjust model parallelism in a workload-aware manner. The only viable workaround today is to terminate the running service upon detecting a shift in traffic patterns and then reload and restart the model with a new TP/PP configuration. Unfortunately, this restart-based switching approach incurs prohibitively high overhead. As a result, despite the fact that distinct traffic regimes demand significantly different optimal TP/PP configurations, the prohibitive cost of restart-based switching renders dynamic model parallel topology adjustment practically infeasible in real-world deployments. Consequently, the vast majority of production systems remain stuck with static, fixed parallel configurations We present ReMP—a low-downtime, runtime model-parallel reconfiguration framework tailored for large language model (LLM) inference serving. The core objective of ReMP is to transform model-parallel topologies from static deployment parameters into dynamically adjustable system resources, enabling the serving system to flexibly switch between different tensor parallelism (TP) and pipeline parallelism (PP) configurations in response to real-time workload fluctuations—without requiring a full service restart. To achieve this, ReMP restructures the LLM inference runtime along three key dimensions. First, ReMP decouples model-parallel topology from runtime state. ReMP re-abstracts the dependencies among these states, ensuring that model weights, cache states, worker processes, and communication groups are no longer permanently bound to a specific topology. As a result, when
switching between parallel configurations, the system only needs to update topology-dependent components affected by the change—eliminating the need to tear down and rebuild the entire service instance. Second, ReMP introduces a two-dimensional KV cache migration mechanism to preserve reusable request states after TP/PP topology changes. The placement of KV cache is governed by two orthogonal dimensions: pipeline parallelism determines which pipeline stage owns the cache for each model layer, while tensor parallelism dictates which tensor-parallel rank holds the cache for different attention heads within a layer. ReMP jointly remaps and migrates KV cache blocks across these two dimensions, allowing previously computed, reusable KV states to remain valid under the new topology. This avoids the cache loss and associated prefill overhead commonly incurred by restart-based reconfiguration approaches. Finally, ReMP implements end-to-end runtime reconfiguration capability within vLLM v1. We deeply integrate ReMP into vLLM’s core modules—including its executor, worker management, KV cache manager, and communication initialization pipeline—enabling seamless TP/PP configuration switches while the service remains online. Experimental results on two 8-GPU platforms, NVIDIA H100 and RTX 5090, demonstrate that ReMP makes runtime model-parallelism reconfiguration practical for mainstream LLM serving. Across models from 7B to 70B parameters, ReMP completes most TP/PP topology transitions within 1–3 seconds; for Llama2-70B, ReMP still finishes reconfiguration within several seconds. This corresponds to substantial reductions over restart-based reconfiguration, with speedups typically reaching tens of times and exceeding 100× for several transitions. Beyond reducing switching cost, ReMP also improves serving performance under dynamic request pressure: compared with two representative fixed baselines, TP1PP8 and TP2PP4, ReMP dynamically selects better TP/PP configurations and achieves lower TTFT, lower TPOT, and higher output throughput. These results show that ReMP turns model-parallel topology from a static launch-time parameter into a runtime-adjustable resource for adaptive LLM serving.
2
Related Work
Recent LLM inference systems advance serving efficiency. ORCA [19] uses iteration-level scheduling, vLLM [7] proposes PagedAttention for KV-cache. Systems optimize variously: SGLang [20] for structured generation, FastServe [18] for preemptive scheduling, DeepSpeed-FastGen [5] for prompt and decode composition, LightLLM [10] and TGI [6] as production runtimes. These optimize under fixed configurations; ReMP reconfigures TP/PP topology in live instances. Phaseaware systems exploit prefill/decode differences. SarathiServe [1] uses chunked prefill, Splitwise [13] and DistServe [21]
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
separate phases, Mooncake [15] explores KV-cache disaggregation. None address post-launch topology switching; ReMP provides low-downtime changes. Large-model inference uses parallelism and memory optimization. FasterTransformer [11], TensorRT-LLM [12], and AlpaServe [8] support multi-GPU inference. FlexGen [16] and DeepSpeed Inference [2] use heterogeneous memory. vAttention [14] optimizes KV-cache, Punica [4] improves LoRA serving. These treat topology as static; ReMP decouples components from launch-time topology. Dynamic serving systems show adaptability value. Llumnix [17] enables request migration, SpotServe [9] targets preemptible resources, PipeLive [3] studies pipeline reconfiguration. These operate at instance level or focus on pipelines; ReMP jointly reconfigures TP and PP in running vLLM instances.
3
System Design
The key idea of ReMP is to decouple runtime states that are traditionally bound to the launch-time topology. Specifically, ReMP decouples model weights from GPU shard layouts by persisting the full model state in CPU shared memory; decouples KV cache from a fixed TP/PP layout by redistributing cache tensors along both the layer dimension and the KVhead dimension; decouples communication groups from a single topology by preconstructing parallel-state snapshots; and decouples worker lifetimes from the active world size through a standby/wakeup mechanism. 3.1
Challenge
Runtime model-parallel reconfiguration requires more than parameter changes. LLM serving systems hardwire runtime states—weights, KV caches, workers, communication groups—to initial topology. Topology shifts disrupt interdependent components, demanding solutions to three challenges: state decoupling, KV cache migration, and low-disruption switching. First, topology tightly couples with execution state: weight sharding, operators, and dependencies change with TP/PP. Traditional engines lack runtime reinterpretation mechanisms. Second, KV caches bind to pipeline stages (layer) and tensor ranks (head). Changing TP/PP requires coordinated remapping. Discarding wastes prefill; naive copying causes ownership errors. Semantic-aware migration is essential. Third, reconfiguration must minimize disruption. Topology changes affect workers, mappings, schedulers, and groups. Blocking or dropping requests makes optimal configurations impractical. Frameworks must coordinate switching, migration, and scheduling for correctness with minimal downtime and latency. 3.2
Architecture Overview
Figure 2 shows the architecture of ReMP. ReMP consists of six main components: the Reconfiguration Controller, Shared
Conference’17, July 2017, Washington, DC, USA
Table 1. ReMP decouples runtime states that are traditionally bound to the launch-time topology. Runtime State
Restart-based Switch
ReMP
Model weights
Reload checkpoints
Reshard via CPU memory
KV cache
Discard
2D layer/head migration
Comm groups
Destroy & recreate
Switch prebuilt snapshots
Workers
Terminate & restart
Standby/wakeup reuse
Scheduler
Reinitialize
Update cache config
Weight Store, MPU State Space, Worker Lifecycle Manager, KV Migration Engine, and Scheduler Adapter. The Reconfiguration controller is the entry point of a topology-switching transaction. It receives the target TP/PP configuration, updates the parallel configuration, and coordinates the executor, workers, model runners, KV migration engine, and scheduler. The shared weight store persists the full model state in CPU shared memory. At service startup, MPClient loads the full state dictionary through CheckPointManager and converts it into a SharedStateDict. During reconfiguration, workers reconstruct their target GPU shards from the shared state dictionary instead of reloading checkpoint files. The MPU State Space stores parallel-state snapshots for multiple candidate TP/PP topologies. Each snapshot contains the TP groups, PP groups, rank mapping, and related parallel-state metadata for a specific topology. During switching, ReMP selects the snapshot associated with the target topology and applies it to vLLM’s global parallel state. The worker lifecycle manager maintains active, standby, and wakeup states for workers. If the target world size is the same as the current world size, the worker set remains unchanged. If the target world size is smaller, extra workers enter standby after KV migration. If the target world size is larger, standby workers are woken up and rejoin the active worker set. The KV migration engine computes and executes the two-dimensional KV cache migration plan. The PP dimension determines layer ownership, while the TP dimension determines KV-head slice ownership. During migration, ReMP transfers cache tensors layer by layer using batched P2P operations and releases old layer storage after each layer is migrated. The scheduler adapter regenerates the KV cache configuration after topology switching, updates the block manager, handles cache block expansion or shrinking, preempts running requests when necessary, and refreshes the pipeline-parallel batch queue. Table 1 summarizes the key decoupling mechanisms in ReMP.
Conference’17, July 2017, Washington, DC, USA
Current vLLM Runtime TP=a,PP=b Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
Worker 6
Worker 7
switching
Current MPU State
Model Shards on GPU old TP/PP layout
Reconfigured vLLM
Transaction
Runtime TP=c, PP=d
Reconfiguration Controller Switch Target TP/PP Command Plan
Checkpoint I/O avoided during
TP/PP process groups
ReMP Reconfiguration
1
Active Workers Worker 0
First Author et al.
KV cache preserved when capacity permits
Workers and communication groups reused
Distributed KV Cache layers × KV heads
old KV layout
Serving under TP=a,PP=b
standby/ wakeup
3
MPU State Space
Prebuilt parallel-state snapshots Apply target TP/PP process groups
4
KV Migration Engine
5
Worker Lifecycle Manager
in-memory shard reload snapshot apply
2D redistribution PP layer mapping + TP head slicing
Active/ Standby/Wakeup Reuse worker processes
Low-downtime switch window
Worker 1
Worker 2
Worker 3
Standby
Weight Store 2 CPU Shared shared-memory state_dict
Reload target shards without checkpoint I/O
Reused /Woken Workers Worker 0
Target MPU State switched process groups
Model Shards on GPU new TP/PP layout
2D KV transfer
Migrated KV Cache preserved logical blocks
Serving under TP=c,PP=d
Figure 2. ReMP architecture. ReMP decouples model weights, KV cache, communication groups, and worker lifetimes from a fixed TP/PP topology, enabling low-downtime runtime reconfiguration in vLLM. 3.3
Reconfiguration Transaction
ReMP organizes each TP/PP switch as a controlled reconfiguration transaction. The transaction takes as input the source topology 𝑇𝑜𝑙𝑑 , the target topology 𝑇𝑛𝑒𝑤 , the current scheduler state, KV cache metadata, the shared model state, and the prebuilt MPU State Space. After the transaction completes, the serving instance continues execution under the target topology with updated active workers, GPU model shards, KV cache layout, and scheduler state. Conceptually, a ReMP switch proceeds through a sequence of states: serving under 𝑇𝑜𝑙𝑑 , quiescing the scheduler, preparing the worker set, applying the target MPU state, migrating KV cache, reloading model shards, rebinding cache and scheduler state, and finally serving under 𝑇𝑛𝑒𝑤 . The scheduler resumes only after model shards, KV cache bindings, communication state, and scheduler metadata have all been made consistent with the target topology. A key optimization in ReMP is to overlap model-shard reloading with KV cache migration. These two operations have different data sources and synchronization dependencies. Model-shard reloading reads parameter slices from the CPU shared-memory weight store and materializes them into the target GPU model shard. KV cache migration transfers live cache slices from the old physical placement to
the new placement according to the target TP/PP topology. Since the model parameters and KV cache tensors are disjoint runtime states, ReMP can execute the two operations concurrently after the target worker set and MPU state have been prepared. This overlap reduces the switching critical path. A sequen𝑠𝑒𝑞 tial design would incur 𝑇𝑠𝑤𝑖𝑡𝑐ℎ = 𝑇𝑤𝑜𝑟𝑘𝑒𝑟 + 𝑇𝑚𝑝𝑢 + 𝑇𝑘𝑣 + 𝑇𝑚𝑜𝑑𝑒𝑙 + 𝑇𝑠𝑐ℎ𝑒𝑑 , where 𝑇𝑘𝑣 is KV cache migration time and 𝑇𝑚𝑜𝑑𝑒𝑙 is model-shard reloading time. With overlap, ReMP 𝑜𝑣𝑒𝑟𝑙𝑎𝑝 reduces the dominant state-movement portion to 𝑇𝑠𝑤𝑖𝑡𝑐ℎ = 𝑇𝑤𝑜𝑟𝑘𝑒𝑟 +𝑇𝑚𝑝𝑢 + max(𝑇𝑘𝑣 ,𝑇𝑚𝑜𝑑𝑒𝑙 ) +𝑇𝑠𝑐ℎ𝑒𝑑 . In practice, this is especially useful for large models, where model-shard reloading can dominate, and for cache-heavy workloads, where KV migration becomes significant. In the implementation, some steps may be fused or reordered depending on the executor and worker states, but the transaction boundary remains the same. The switching procedure handles three cases depending on the change in world size. If 𝑇 𝑃𝑜𝑙𝑑 × 𝑃𝑃𝑜𝑙𝑑 = 𝑇 𝑃𝑛𝑒𝑤 × 𝑃𝑃𝑛𝑒𝑤 , the active worker set remains unchanged; ReMP only needs to migrate KV cache, apply the target MPU state, and reload model shards. If the target world size is smaller, some old workers will become standby workers after the switch. Since these workers may still hold KV slices required by
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
ShardStateDict
PP maps layers to workers
CPU shared-memory full model state
Embedding
slice A
Layer 0
slice A
Layer 1
slice A
materialize target shard
1
PP selects layer range
2
TP selects tensor slices
slice B
...
... Layer N-1
slice C
slice D
LM Head
slice C
slice D
select required slices
TP maps tensor slices to workers
Worker 1 PP stage 0, TP rank 1
GPUModelRunner
GPUModelRunner
GPU Shard
Target Shard Planner slice B
Worker 0 PP stage 0, TP rank 0
GPU Shard
Layers 0-15,
Layers 0-15,
tensor slice 0
tensor slice 1
Worker 2
Worker 3
PP stage 1, TP rank 0
PP stage 1, TP rank 1
GPUModelRunner
GPUModelRunner
GPU Shard
GPU Shard
Layers 16-31,
Layers 16-31,
tensor slice 0
tensor slice 1
Figure 3. Shared model weight store. ReMP loads the full model state into CPU shared memory at startup and reconstructs target GPU shards from the shared state dictionary during topology switching. the target topology, ReMP migrates their useful KV state before removing them from the active worker set. If the target world size is larger, ReMP wakes up standby workers, synchronizes the message-queue ring index so that they can receive control and KV-transfer messages, and then adds them to the target topology. 3.4
Shared Model Weight Store
In conventional vLLM serving, checkpoint loading is tightly coupled with the launch-time TP/PP topology. Each worker loads or constructs its GPU-resident shard according to its pipeline rank and tensor-parallel rank. When the topology changes, the conventional path shuts down the old service and restarts a new instance with the target configuration, causing checkpoint loading, CUDA initialization, worker initialization, and communication-group construction to happen again. Modern LLM serving servers are often equipped with sufficient host memory, making it practical to maintain a global parameter view that can be reused across different parallel configurations. ReMP uses the Shared Weight Store to avoid checkpoint I/O on the switching critical path. At service startup, MPClient loads the full state dictionary through CheckPointManager and converts it into a SharedStateDict backed by CPU shared memory. This SharedStateDict is shared across worker processes, avoiding per-worker copies of the full state dictionary. During reconfiguration, GPUModelRunner performs model reconfiguration according to the target TP/PP topology, selects the parameters required by the current rank from the shared state dictionary, and materializes them on the GPU. Figure 3 shows the lifecycle of the shared weight store. Checkpoint files are read only at service startup. Subsequent topology switches reconstruct target GPU shards directly from CPU shared memory. For each parameter, whether it should be loaded by a target rank is determined by the target TP/PP placement rule. PP determines the layer range owned by the rank, while TP determines the slicing of tensors such as attention weights,
Conference’17, July 2017, Washington, DC, USA
MLP weights, and output projections. Replicated parameters can be read by multiple ranks from the same shared state, while sharded parameters are sliced according to the target tensor-parallel rank. Overall, the shared weight store decouples model parameters from topology-specific local runtime state. During reconfiguration, the target topology does not need to reload the full model from disk or remote storage. Instead, workers perform slicing, mapping, and reconstruction based on the shared model state. This design reduces repeated I/O and data-loading overhead on the switching path and provides the foundation for low-downtime model-parallelism switching. ReMP does not eliminate model loading entirely. Workers still need to read from CPU shared memory, construct the target GPU shards, and copy parameters to GPU memory. However, this loading path is independent of KV cache migration. ReMP exploits this independence by overlapping shared-memory model reloading with KV cache transfer during the switching window. As a result, model loading contributes to the switching critical path primarily when it exceeds the concurrent KV migration time. 3.5
Two-dimensional KV Cache Migration
KV cache migration is the central mechanism in ReMP. In LLM serving, KV cache layout is tightly coupled with the model-parallel topology. PP determines which transformer layers are assigned to each pipeline rank and therefore where the KV cache of a layer is stored. TP determines which KV heads are assigned to each tensor-parallel rank and therefore how the head dimension of each layer’s KV cache is partitioned. When TP or PP changes, the KV cache stored on one old rank may need to be split and migrated to multiple new ranks. 3.5.1 Logical KV Cache Model. ReMP abstracts a logical KV cache slice as 𝐾𝑉 [𝑙, 𝑏, ℎ𝑠 : ℎ𝑒 ], where 𝑙 denotes the layer id, 𝑏 denotes the cache block id, and ℎ𝑠 : ℎ𝑒 denotes the KV-head range. The token dimension is represented by the internal cache-block layout, while the head dimension and block size remain unchanged for the same model. For a topology 𝑇 = (𝑇 𝑃, 𝑃𝑃), we define two ownership functions: 𝑝𝑝_𝑜𝑤𝑛𝑒𝑟 (𝑙, 𝑃𝑃) → 𝑝𝑝_𝑟𝑎𝑛𝑘,
𝑡𝑝_𝑜𝑤𝑛𝑒𝑟 (ℎ,𝑇 𝑃) → 𝑡𝑝_𝑟𝑎𝑛𝑘.
Given a layer 𝑙 and head ℎ, the global rank holding the corresponding KV slice is 𝑟𝑎𝑛𝑘 (𝑙, ℎ,𝑇 ) = 𝑟𝑎𝑛𝑘 (𝑝𝑝_𝑜𝑤𝑛𝑒𝑟 (𝑙, 𝑃𝑃), 𝑡𝑝_𝑜𝑤𝑛𝑒𝑟 (ℎ,𝑇 𝑃)). Thus, topology switching must preserve the following logical mapping: 𝐾𝑉 [𝑙, 𝑏, ℎ] on 𝑟𝑎𝑛𝑘 (𝑙, ℎ,𝑇𝑜𝑙𝑑 ) → 𝐾𝑉 [𝑙, 𝑏, ℎ] on 𝑟𝑎𝑛𝑘 (𝑙, ℎ,𝑇𝑛𝑒𝑤 ).
Conference’17, July 2017, Washington, DC, USA
Algorithm 1 Build 2D KV Cache Migration Plan Require: Old topology 𝑇𝑜𝑙𝑑 , new topology 𝑇𝑛𝑒𝑤 , live layers 𝐿, live blocks 𝐵, number of KV heads 𝐻 Ensure: Send plan and receive blueprint 1: for all 𝑙 ∈ 𝐿 do 2: 𝑜𝑙𝑑_𝑝𝑝 ← 𝑝𝑝_𝑜𝑤𝑛𝑒𝑟 (𝑙, 𝑃𝑃𝑜𝑙𝑑 ) 3: 𝑛𝑒𝑤_𝑝𝑝 ← 𝑝𝑝_𝑜𝑤𝑛𝑒𝑟 (𝑙, 𝑃𝑃𝑛𝑒𝑤 ) 4: for 𝑛𝑡𝑝 = 0 to 𝑇 𝑃𝑛𝑒𝑤 − 1 do 5: 𝐻𝑡𝑎𝑟𝑔𝑒𝑡 ← ℎ𝑒𝑎𝑑_𝑟𝑎𝑛𝑔𝑒 (𝑛𝑡𝑝,𝑇 𝑃𝑛𝑒𝑤 , 𝐻 ) 6: for 𝑜𝑡𝑝 = 0 to 𝑇 𝑃𝑜𝑙𝑑 − 1 do 7: 𝐻𝑠𝑜𝑢𝑟𝑐𝑒 ← ℎ𝑒𝑎𝑑_𝑟𝑎𝑛𝑔𝑒 (𝑜𝑡𝑝,𝑇 𝑃𝑜𝑙𝑑 , 𝐻 ) 8: 𝐻𝑜𝑣𝑒𝑟𝑙𝑎𝑝 ← 𝐻𝑡𝑎𝑟𝑔𝑒𝑡 ∩ 𝐻𝑠𝑜𝑢𝑟𝑐𝑒 9: if 𝐻𝑜𝑣𝑒𝑟𝑙𝑎𝑝 ≠ ∅ then 10: 𝑠𝑟𝑐 ← 𝑟𝑎𝑛𝑘 (𝑜𝑙𝑑_𝑝𝑝, 𝑜𝑡𝑝,𝑇𝑜𝑙𝑑 ) 11: 𝑑𝑠𝑡 ← 𝑟𝑎𝑛𝑘 (𝑛𝑒𝑤_𝑝𝑝, 𝑛𝑡𝑝,𝑇𝑛𝑒𝑤 ) 12: 𝑠𝑒𝑛𝑑_𝑝𝑙𝑎𝑛[𝑠𝑟𝑐]+ = (𝑑𝑠𝑡, 𝑙, 𝐵, 𝐻𝑜𝑣𝑒𝑟𝑙𝑎𝑝 ) 13: 𝑟𝑒𝑐𝑣_𝑝𝑙𝑎𝑛[𝑑𝑠𝑡]+ = (𝑠𝑟𝑐, 𝑙, 𝐵, 𝐻𝑜𝑣𝑒𝑟𝑙𝑎𝑝 ) 14: end if 15: end for 16: end for 17: end for 18: return 𝑠𝑒𝑛𝑑_𝑝𝑙𝑎𝑛, 𝑟𝑒𝑐𝑣_𝑝𝑙𝑎𝑛
3.5.2 Migration Plan Construction. ReMP constructs the migration plan from both the receiver side and the sender side. For each new rank in the target topology, ReMP first computes the target layer range from its pipeline rank, and then computes the target KV-head range from its tensorparallel rank. It then intersects the target head range with the head ranges owned by old TP ranks. Each non-empty intersection corresponds to a KV slice that must be received from an old rank. Each receive item can be represented as
First Author et al.
items within the same layer can be batched using asynchronous send and receive operations to reduce synchronization overhead. This design makes the actual communication volume proportional to the live KV data whose ownership changes, rather than to the full KV cache size or the number of model parameters. For topology pairs with substantial ownership overlap, more data can be handled by local copy, reducing cross-rank transfer volume. 3.5.4 Layer-wise Streaming Migration. A naive design would allocate a full target KV cache before releasing the old KV cache, causing the peak memory footprint to approach the sum of the old and new cache sizes. This can easily lead to out-of-memory failures for long-context workloads or high cache occupancy. ReMP instead performs layer-wise streaming migration. For each layer to be migrated, ReMP allocates the target layer’s cache storage, issues local copies and remote P2P transfers according to the migration plan, binds the target storage to the corresponding attention layer after all transfers complete, and releases the old layer storage. Since the migration working set is limited to one or a small number of layers, the peak memory overhead is much lower than materializing the entire target cache at once. Figure 4 illustrates the two-dimensional KV cache migration process. PP changes remap layer ownership, while TP changes remap head-slice ownership. ReMP computes the layer/head slices required by each new rank and retrieves them from the corresponding old ranks.
3.5.5 Correctness Invariants. ReMP maintains the following invariants during KV cache migration. Layer coverage. For every layer assigned to a target PP rank, all live cache blocks that should be preserved for that layer must exist in the corresponding target cache storage 𝑅𝑒𝑐𝑣𝐼𝑡𝑒𝑚 = (𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑙, 𝐵, 𝐻𝑠𝑟𝑐∩𝑑𝑠𝑡 ) after migration. Head coverage. For each migrated (𝑙𝑎𝑦𝑒𝑟, 𝑏𝑙𝑜𝑐𝑘) pair, , where 𝑠𝑟𝑐 is the old rank, 𝑑𝑠𝑡 is the new rank, 𝑙 is the layer the union of head ranges owned by all target TP ranks must id, 𝐵 is the set of cache blocks to migrate, and 𝐻𝑠𝑟𝑐∩𝑑𝑠𝑡 is the equal the full KV-head range, without overlapping ownership intersection between the source and target head ranges. unless replication is required by the model semantics. ReMP also constructs the send plan for each old rank. The Logical block identity preservation. The mapping mainsend plan and receive blueprint are dual: if a new rank needs tained by the scheduler from requests, prefixes, and logical to receive a layer/head slice from an old rank, then the old block ids to KV contents must remain valid after migration, rank generates a corresponding send item to that new rank. Algorithm 1 shows a simplified version of the two-dimensional or be updated through a deterministic block remapping, so that migrated KV cache can continue to be referenced by the migration-plan construction. scheduler. 3.5.3 Local Copy and Remote transfer. For each migraCapacity constraint. If the target topology provides less tion item, if 𝑠𝑟𝑐 = 𝑑𝑠𝑡, the KV slice remains on the same KV cache capacity than the current live cache footprint, worker after topology switching. ReMP handles this case ReMP cannot preserve all KV blocks. In this case, the schedwith a local copy or view assignment, avoiding communicauler preempts selected running requests, reclaims their cache tion. If 𝑠𝑟𝑐 ≠ 𝑑𝑠𝑡, ReMP transfers the slice from the old rank blocks, and moves those requests back to the waiting queue to the new rank using P2P communication. Multiple transfer for later recomputation.
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
Conference’17, July 2017, Washington, DC, USA
local copy
New Topology
Old Topology TP0: Heads 0-H/2
TP1: Heads H/2-H
Old Rank 0
Old Rank 1
PP0: Layers 0-7
KV Migration Engine Text
KV blocks layer range × head range
KV blocks layer range × head range
Old Rank 2
Old Rank 3
PP demension layers
KV blocks layer range × head range
KV blocks layer range × head range
Old Rank 4
Old Rank 5
PP2: Layers 16-23 KV blocks layer range × head range
KV blocks layer range × head range
Old Rank 6
Old Rank 7
KV blocks layer range × head range
KV blocks layer range × head range
PP3: Layers 24-31
TP1:
TP2:
TP3:
Heads H/4-H/2
Heads H/2-3H/4
Heads 3H/4-H
New Rank 0
New Rank 1
New Rank 2
New Rank 3
Migrated KV blocks
Migrated KV blocks
Migrated KV blocks
Migrated KV blocks
new layer range × new head range
new layer range × new head range
new layer range × new head range
new layer range × new head range
New Rank 4
New Rank 5
New Rank 6
New Rank 7
Migrated KV blocks
Migrated KV blocks
Migrated KV blocks
Migrated KV blocks
new layer range × new head range
new layer range × new head range
new layer range × new head range
new layer range × new head range
1. PP remaps layer ownership
2. TP remaps KV-head slices
PP1: Layers 8-15
TP0: Heads 0-H/4
src rank
dst rank
layers
head slice
Old 0
New 0
L0-7
H0-H/4
Old 0
New 1
L0-7
H/4-H/2
Old 4
New 2
L16-23 H0-H/4
Old 5
New 3
L16-23 H/2-3H/4
Layer-wise streaming transfer release old layer storage after transfer
TP dimension: KV heads
P2P KV slice transfer
PP change moves layersacross pipeline stages
TP change splits ormerges KV-head ranges
Streaming by layer bounds peak GPU memory
Figure 4. Two-dimensional KV cache migration. PP changes layer ownership, while TP changes KV-head slice ownership. ReMP redistributes live KV cache along both dimensions. 3.6
MPU State Space
TP/PP topology determines not only model and KV cache placement, but also the communication structure. In conventional systems, TP groups, PP groups, world groups, rank mappings, and other parallel-state metadata are created at service startup and assumed to remain unchanged throughout the service lifetime. Destroying and recreating NCCL process groups during reconfiguration would add latency and introduce complex synchronization and failure-handling issues. ReMP introduces the MPU State Space to manage communication states for multiple candidate topologies. For each candidate topology 𝑇𝑖 = (𝑇 𝑃𝑖 , 𝑃𝑃𝑖 ), ReMP preconstructs a corresponding parallel-state snapshot: 𝑀𝑃𝑈 𝑆𝑝𝑎𝑐𝑒 = {𝑇𝑖 → 𝑆𝑛𝑎𝑝𝑠ℎ𝑜𝑡𝑖 }. Each snapshot contains TP groups, PP groups, rank mapping, tensor model-parallel rank, pipeline modelparallel rank, and other metadata required by vLLM’s parallel state. During switching, a worker looks up the snapshot for the target topology and applies it to the global parallel state.
This design moves process-group construction off the switching critical path and into the initialization phase. The trade-off is that the candidate topology set must be bounded and known in advance. Supporting arbitrary topologies would require a dynamic group cache or on-demand group construction.
3.7
Worker Lifecycle Management
The vLLM multiprocess executor normally creates a fixed set of workers according to the launch-time world size. If the world size changes through a restart-based path, the system must destroy old workers and create new ones, reinitializing the CUDA runtime, model objects, communication state, and message queues. ReMP avoids this overhead through explicit worker lifecycle management. ReMP classifies workers into active and standby states. Active workers participate in model execution, communication, and KV cache management under the current topology.
Conference’17, July 2017, Washington, DC, USA
Standby workers retain process resources but do not execute requests. When the target world size is smaller than the current world size, ReMP removes extra workers from the active set and places them into standby. When the target world size is larger, ReMP wakes standby workers and adds them back to the active set. During scale-down, ReMP must migrate KV cache before moving extra workers to standby. This is because a worker that will leave the active set may still hold KV slices required by the target topology. If such a worker were put into standby before migration, its KV state might become inaccessible to the target topology. During scale-up, ReMP first wakes standby workers and synchronizes the message-queue ring index so that the new workers can correctly receive executor messages and KV-transfer requests. The workers then apply the target MPU state, load their target model shards from the shared weight store, and receive migrated KV cache. The standby mechanism avoids repeated worker destruction and reconstruction during topology switching. A standby worker retains a lightweight runtime context between switches and keeps the necessary runtime handles for fast wakeup. Once a target TP/PP topology is selected, ReMP can directly wake the corresponding workers and reconfigure their communication state, without repeating process creation, runtime initialization, and device-context setup. This reduces process-management overhead on the switching path and contributes to low-downtime reconfiguration. 3.8
First Author et al.
all running requests continue without recomputation under insufficient target capacity. In addition, PP changes alter the pipeline-stage structure, so the old PP batch queue cannot be reused directly. ReMP refreshes the PP batch queue after the switch so that subsequent batch metadata is consistent with the target pipeline layout. 3.9
1. The target active worker set has been determined. 2. The target MPU state has been applied. 3. The KV cache to be preserved has been migrated and bound to the target cache storage. 4. Target model shards have been loaded from the shared weight store onto GPUs. 5. The scheduler’s KV cache configuration and PP batch queue have been updated.
Scheduler and KV Block Adaptation
Topology switching changes not only model and communication state, but also the KV cache capacity and tensor shapes on each worker. ReMP therefore adapts the scheduler to the target KV cache configuration. At the beginning of a switch, the scheduler enters a safe switching window, pauses new scheduling decisions, and freezes current KV cache metadata. This gives the KV Migration Engine a consistent view of the live block set and prevents concurrent request execution from modifying cache contents during migration. After switching, ReMP regenerates the KV cache configuration for the target topology, including the number of layers per rank, the number of cache blocks, tensor shapes, cache dtype, and attention-layer bindings. The Scheduler Adapter updates the block manager using the new configuration and performs cache block expansion or shrinking according to the target capacity. If the target topology provides more KV blocks, the scheduler adds the new blocks to the free list, and subsequent requests can use them directly. If the target topology provides fewer KV blocks, the scheduler first attempts to release idle blocks. If idle blocks are insufficient, the scheduler preempts selected running requests, moves them back to the waiting queue, and reclaims their blocks. Thus, ReMP preserves KV cache when possible, but does not guarantee that
Consistency Boundary and Commit Point
ReMP executes topology switching as a controlled transaction. Therefore, the system must define a consistency boundary and a commit point for the switch. Before the commit point, the scheduler remains paused and the old topology has not been fully replaced. If the switch fails in an early phase, such as during worker preparation or MPU state application, the system can in principle keep the old topology and resume the old serving path. After the commit point, model shards, KV cache bindings, parallel state, and scheduler cache configuration have been updated to the target topology, and the system continues serving under 𝑇𝑛𝑒𝑤 . ReMP defines the commit point as the moment when all of the following conditions hold:
Only after this commit point does the scheduler resume execution. This prevents requests from accessing inconsistent model shards or KV cache tensors during a partially completed migration.
4
Evaluation
In this section, we conduct two key experiments to evaluate the performance and effectiveness of ReMP. 1. Reconfiguration Experiment: This experiment aims to quantify the efficiency of ReMP when altering the model parallelism (TP/PP) topology. We focus on answering: How much faster is ReMP compared to traditional restart-based reconfiguration? Additionally, we analyze which operations constitute the dominant costs during the runtime switching process. 2. Performance Optimization Experiment: This experiment aims to verify whether fast runtime reconfiguration can practically improve the overall performance of the serving system. We investigate whether the system can achieve better serving performance than a fixed deployment configuration by flexibly switching to optimal TP/PP configurations in the face of different models and dynamically changing request pressures.
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
4.1
Experimental Setup
Hardware. We evaluate ReMP on two 8-GPU platforms. The first platform is equipped with 8 NVIDIA H100 GPUs, an AMD EPYC 7R13 CPU, and 2TB host memory. The second platform is equipped with 8 NVIDIA RTX 5090 GPUs, an Intel Xeon Gold 6530 CPU, and 960GB host memory. Software. We implement ReMP on top of vLLM V1. The implementation adds runtime TP/PP topology switching, shared-memory model state reuse, two-dimensional KV cache migration, prebuilt communication-state switching, and worker lifecycle management. All experiments use the same software stack for ReMP and the restart-based baseline, except that the baseline changes TP/PP topology by restarting the serving instance. Models. Table 2 summarizes the models used in our evaluation. We use four representative LLMs ranging from 7B to 70B parameters, including both dense and MoE architectures. These models differ in parameter size, number of layers, hidden dimension, attention layout, and KV-head configuration, which affect both model shard reconstruction and KV cache migration during TP/PP reconfiguration.Due to GPU memory limitations, Llama2-70B is not evaluated on the RTX 5090 platform. Workloads. For serving-performance experiments, we use BurstGPT-derived request traces and replay them under different request pressures. Each run uses the same request sequence across TP/PP configurations to ensure that performance differences come from the serving topology rather than workload variation. Baselines. For reconfiguration-cost experiments, we compare ReMP with a restart-based baseline. The restart baseline terminates the current serving instance and launches a new one under the target TP/PP topology, reloading model checkpoints, reconstructing runtime state, and discarding existing KV cache. For serving-performance experiments, we compare ReMP with fixed TP/PP baselines. Each fixed baseline uses one static TP/PP configuration throughout the workload. In contrast, ReMP uses its low-downtime switching capability to probe representative TP/PP configurations under the current load and then continues serving with the configuration that performs best during the probing window. In the servingperformance figures, we report this selected configuration as ReMP. Metrics. For reconfiguration cost, we report restart-based switching time, ReMP switching time, speedup over restart, and ReMP’s internal switching-time breakdown. The breakdown includes model loading from the shared weight store and KV cache transfer. For serving performance, we report throughput and latency metrics, and compare fixed TP/PP baselines against the topology selected by ReMP under each model and request pressure.
4.2
Conference’17, July 2017, Washington, DC, USA
Reconfiguration Cost
We first evaluate the cost of changing the TP/PP topology at runtime. This experiment is designed to answer two questions. First, can ReMP complete model-parallel topology switching within seconds? Second, how much faster is ReMP compared with the conventional restart-based reconfiguration approach? For each model and topology transition, we compare ReMP with a restart-based baseline. The restart-based baseline terminates the current serving instance and launches a new instance under the target TP/PP topology. This path reloads model checkpoints, reconstructs worker processes, rebuilds runtime and communication states, and discards the existing KV cache. In contrast, ReMP keeps the serving runtime alive and performs topology switching through in-memory state transformation. Specifically, ReMP reuses worker processes, applies prebuilt MPU-state snapshots, reloads target model shards from the CPU shared-memory weight store, and constructs or migrates KV cache state according to the target topology. 4.2.1 End-to-end Reconfiguration Time. Figures 5 reports the end-to-end switching time of ReMP across different source and target TP/PP topologies. Each cell represents one topology transition, where the row denotes the source topology and the column denotes the target topology. The diagonal entries are omitted because they correspond to unchanged configurations. The results show that ReMP can complete most topology transitions within only a few seconds. On the H100 platform, ReMP completes Llama2-7B topology switching in roughly 1–2 seconds across the tested transitions. For larger models, the switching time increases because more parameter shards need to be materialized under the target topology, but the total cost remains low. Qwen3-30BA3B and DeepSeek-R1-Distill-Qwen-32B complete switching in around 2–3 seconds for most transitions. Even for Llama270B, whose model-shard reconstruction cost is substantially higher, ReMP still completes runtime topology switching within several seconds rather than requiring a full service restart. We exclude transitions among 4-GPU topologies for Llama2-70B, because these configurations cannot provide sufficient GPU memory for valid serving runs. The RTX 5090 platform shows a similar trend. For Llama27B, Qwen3-30B-A3B, and DeepSeek-R1-Distill-Qwen-32B, ReMP also completes topology switching at second-level latency. Although the absolute time differs across models and topology transitions, the switching cost remains within a practical range for online serving. Due to GPU memory limitations, Llama2-70B is not evaluated on the RTX 5090 platform. The variation across cells mainly comes from the amount of topology-dependent state that must be transformed. Transitions that change PP degree remap transformer layers
Conference’17, July 2017, Washington, DC, USA
First Author et al.
Table 2. Model configurations used in the evaluation. Model
Arch.
Params
Active Params
Layers
Hidden Size
Attn. Heads
KV Heads
Llama-7B Llama-70B DeepSeek-R1-Distill-Qwen-32B Qwen3-30B-A3B
Dense Dense Dense MoE
7B 70B 32B 30.5B
7B 70B 32B 3.3B
32 80 64 48
4096 8192 5120 2048
32 64 40 32
32 8 8 4
ReMP End-to-end Switching Time
1.9
2.1
1.9
2.0
1.7
2.0
2.1
2.4
1.9
1.8
1.9
2.1
2.3
3.0
2.0
2.1
2.0
2.5
2.9
2.5
1.1
1.1
2.3
3.1
2.5
2.2
1.1
2.4
3.1
2.5
2.6
1.7
TP=2 PP=4
2.0
1.9
TP=1 PP=8
2.0
1.9
2.0
TP=4 PP=1
1.9
1.7
1.8
2.0
TP=2 PP=2
1.9
2.0
1.9
2.1
2.1
2.1
2.1
2.0
1.0 1.1
1.1
3.2
2.5
3.0
Llama2-7B
2.9
2.6
2.4
2.8
2.6
2.7
2.5
2.5
2.7
2.6
2.4
3.0
2.8
2.5
2.9
3.2
2.4
2.5
2.5
2.6
2.3
3.0 2.8
2.4 3.1
DeepSeek-R1-Distill-Qwen-32B 2.2
2.2
2.5
2.4
2.4
2.3
2.2
2.7
2.8
2.7
6.2
2.3
2.9
2.3
2.5
6.2
7.1
2.3
2.8
2.5
6.1
6.8
6.4
2.4
2.8
6.3
6.4
6.5
4.3
2.5
5.7
6.5
7.0
3.9
TP=4 PP=2
2.4
TP=2 PP=4
2.5
2.3
TP=1 PP=8
2.6
2.4
2.5
TP=4 PP=1
2.3
2.3
2.4
1.8
TP=2 PP=2
2.2
2.4
2.3
1.8
TP=1 PP=4
2.3
2.4
2.3
1.8
3.2 3.3
2.5
7.3
5.8
=1 P=2 P=4 P=8 P=1 P=2 P=4 PP P P P P P P =8 P=4 P=2 P=1 P=4 P=2 P=1 TP T T T T T T
6.7
4.1
5.5
6.1
6.0
7.2
4.3
6.0
6.5
5.9
4.7
7.4
7.2
6.6
7.1
6.9
6.8
6.6
1.3
2.3
2.1
2.0
2.7
2.7
2.2
2.9
2.8
2.8
2.6
2.8
2.7
2.9
2.6
TP=2 PP=4
2.0
2.2
TP=1 PP=8
2.4
2.1
2.6
90
80
7.1
TP=4 PP=2
140
100
Llama2-70B
2.2
1.9
110
70
TP=8 PP=1
1.3
TP=8 PP=1
Qwen3-30B-A3B
60
2.3
120
2.9 100
DeepSeek-R1-Distill-Qwen-32B 2.9
TP=8 PP=1
2.7
2.1
2.8
1.6
80
50 TP=4 PP=2
2.6
TP=2 PP=4
2.4
2.5
TP=1 PP=8
2.8
2.4
Speedup
1.9
Source topology
Source topology
2.2
TP=4 PP=2
TP=1 PP=4
Qwen3-30B-A3B
2.1
Speedup
Llama2-7B 1.9
TP=8 PP=1
ReMP End-to-end Switching Time
60
40 1.5
40
30
4.6
=1 P=2 P=4 P=8 P=1 P=2 P=4 PP P P P P P P =8 P=4 P=2 P=1 P=4 P=2 P=1 TP T T T T T T
1
=8 TP
= PP
2
=4 TP
= PP
2.5
4
=2 TP
= PP
8
=1 TP
= PP
1
=8 TP
= PP
2
=4 TP
= PP
4
=2 TP
Target topology
Target topology
(a) H100 platform.
(b) RTX 5090 platform.
= PP
8
=1 TP
= PP
Figure 5. End-to-end ReMP switching time and speedup across source and target TP/PP topologies. The text in each cell reports the time (in seconds) required to switch from the source topology on the row to the target topology on the column. The color of each cell indicates the corresponding speedup (𝑇restart /𝑇ReMP ). Diagonal cells are omitted as they correspond to unchanged configurations. across pipeline stages, while transitions that change TP degree remap tensor slices and KV-head ownership across tensor-parallel ranks. As a result, different source-target topology pairs induce different model-shard reconstruction and KV-cache transformation costs. Nevertheless, the heatmaps show that ReMP avoids the order-of-magnitude cost of full service reconstruction and consistently keeps the switching latency at the second scale. Figure 5 also compares ReMP against the restart-based baseline. The speedup is computed as Speedup = 𝑇𝑇restart . ReMP Across both platforms, ReMP achieves significant speedup over restart-based reconfiguration. On the H100 platform, the speedup is consistently tens of times and can exceed 100× for several topology transitions. On the RTX 5090 platform, the speedup is also substantial, reaching more than 100× for many transitions. These speedups demonstrate that most of the cost in restart-based switching is not intrinsic to changing TP/PP
topology itself. Instead, restart-based switching pays for heavyweight operations such as checkpoint loading, process destruction and creation, CUDA/runtime initialization, communication-group construction, and scheduler reinitialization. ReMP removes these restart-only costs from the critical path by keeping the runtime alive and transforming only the topology-dependent states. Therefore, changing model parallelism becomes an in-memory reconfiguration problem rather than a full service reconstruction problem. 4.2.2 Effect of Overlapping Model Reloading and KV Cache Construction. We next analyze the internal optimization that overlaps model-shard reloading with KV cache construction or migration. Figure 6 compares the sequential execution time of these two operations with the optimized overlapped time. Without overlap, this state-transformation phase would approximately execute model reloading and KV cache conseq struction or migration sequentially: 𝑇state = 𝑇model + 𝑇kv .
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving Model loading
Overlapped Time
KV transfer 4.0 15% saved
7 6 5 30% saved
4 3
32% saved
17% saved
2 1 0
Average switching time (s)
Average switching time (s)
KV transfer 8
Model loading
Overlapped Time 20% saved
3.5
27% saved
3.0 2.5
13% saved
2.0 1.5 1.0 0.5 0.0
Llama 7B
Qwen3 30B-A3B
DeepSeek 32B
(a) H100 platform.
Llama 70B
Llama 7B
Qwen3 30B-A3B
DeepSeek 32B
(b) RTX 5090 platform.
Figure 6. Effect of overlapping model-shard reloading with KV cache construction or migration. The overlapped time represents the optimized state-transformation phase.
ReMP exploits the fact that the two operations work on different runtime states and data paths. Model reloading reads parameter slices from the CPU shared-memory state dictionary and materializes target GPU shards, while KV cache construction or migration prepares the cache layout required by the target topology. Therefore, ReMP can exeoverlap cute them concurrently when dependencies allow: 𝑇state ≈ max(𝑇model,𝑇kv ). The overlap breakdown shows that concurrent execution effectively reduces the state-transformation portion of the switch on both platforms. The dominant component varies across models. For smaller models, model-shard reloading is relatively lightweight, so KV cache construction or migration can contribute a larger fraction of the remaining cost. For larger models, model reloading becomes more dominant because more parameter data must be sliced and copied into GPU memory under the target topology. In both cases, overlap reduces the optimized state-transformation time toward the slower of the two operations rather than their sequential sum. 4.3
Serving Performance under Dynamic Request Pressure
The previous experiment shows that ReMP can switch TP/PP topologies at second-level latency. We next evaluate whether this capability can translate into better serving performance under changing request pressure. This experiment answers the following question: can fast runtime reconfiguration help the serving system select a better model-parallel topology than representative fixed deployments? 4.3.1 Methodology. We replay BurstGPT-derived request traces under different request rates and compare ReMP with two fixed TP/PP baselines: TP1PP8 and TP2PP4. These two baselines represent two commonly used deployment choices. TP1PP8 is a pipeline-heavy configuration, which increases pipeline parallelism but may suffer from higher pipeline latency and lower per-request efficiency. TP2PP4 is a more balanced configuration, which reduces pipeline depth while still distributing the model across multiple stages.
Conference’17, July 2017, Washington, DC, USA
For ReMP, we do not assume a single static topology throughout the workload. Instead, ReMP uses its fast topologyswitching capability to probe candidate TP/PP configurations under the current request pressure and selects the configuration that provides the best overall serving performance. The selected configuration is then reported as the ReMP result in Figure 7 and Figure 8. The selection metric follows the weighted score used in our benchmark spreadsheet, combining output throughput, mean TPOT, and mean TTFT. Throughput is treated as a higher-is-better metric, while TTFT and TPOT are treated as lower-is-better metrics. This reflects the practical objective of simultaneously improving serving capacity and reducing user-visible latency. 4.3.2 Results on H100. Figure 7 shows the serving performance on the H100 platform across four models: Llama27B, Qwen3-30B-A3B, DeepSeek-R1-Distill-Qwen-32B, and Llama2-70B. Across all models, ReMP consistently achieves lower TTFT and TPOT than the two fixed baselines while also delivering higher output throughput under medium and high request pressure. For Llama2-7B, all configurations can sustain the low request-rate regime, but their behavior diverges as the request rate increases. TP1PP8 exhibits increasing TTFT and TPOT because the deeper pipeline introduces higher end-toend latency. TP2PP4 reduces latency compared with TP1PP8 and improves throughput, but it remains suboptimal under higher pressure. ReMP selects a better topology for each load level, keeping both TTFT and TPOT low while achieving the highest throughput at high request rates. The same trend becomes more pronounced for larger models. For Qwen3-30B-A3B and DeepSeek-R1-Distill-Qwen32B, TP1PP8 suffers from higher latency and limited throughput scalability, while TP2PP4 improves the latency-throughput trade-off but still cannot match the dynamically selected topology. ReMP achieves higher throughput as request pressure grows and avoids the sharp latency increase observed in the fixed baselines. This shows that the best TP/PP configuration depends on both the model architecture and the current request pressure, and therefore cannot be captured by a single static deployment choice. For Llama2-70B, the benefit of dynamic topology selection is even more visible. Because the model is larger, fixed configurations are more sensitive to inefficient parallelism choices. TP1PP8 shows rapidly increasing TTFT and TPOT as request pressure grows, indicating that the pipeline-heavy configuration becomes a latency bottleneck. TP2PP4 improves over TP1PP8, but ReMP still achieves the best overall result by selecting configurations that better balance model-shard computation, communication overhead, and pipeline utilization. As a result, ReMP maintains lower latency while providing higher throughput across the tested request rates. 4.3.3 Results on RTX 5090. Figure 8 reports the same comparison on the RTX 5090 platform for Llama2-7B and
Conference’17, July 2017, Washington, DC, USA
First Author et al.
TP1PP8 Qwen3-30B-A3B
ReMP DeepSeek-32B
Llama2-70B
Throughput
TPOT
TTFT
Llama2-7B
TP2PP4
1
2
3
4
5
1
2
Request rate
3
4
5
Request rate
1
2
3
4
Request rate
5
1
2
3
4
5
Request rate
Figure 7. Serving performance comparison on the H100 platform. ReMP dynamically selects a TP/PP configuration under each request rate and is compared with two fixed baselines, TP1PP8 and TP2PP4. TP1PP8
TP2PP4
relatively small models, the optimal topology changes with serving pressure, and runtime selection can improve the latency-throughput trade-off. For DeepSeek-R1-Distill-Qwen-32B, the fixed baselines show larger instability under increasing request pressure. TP1PP8 has high latency and limited throughput growth, while TP2PP4 performs better in some cases but can still become suboptimal or unstable under certain load levels. ReMP avoids these fixed-topology limitations by selecting a better configuration for the current load, resulting in consistently lower latency and higher throughput in the figure.
ReMP DeepSeek-32B
Throughput
TPOT
TTFT
Llama2-7B
1
2
3
Request rate
4
5
1
2
3
4
5
Request rate
Figure 8. Serving performance comparison on the RTX 5090 platform. ReMP maintains lower latency and higher throughput than representative fixed TP/PP baselines across request pressures. DeepSeek-R1-Distill-Qwen-32B. The results are consistent with the H100 platform: ReMP keeps TTFT and TPOT lower than the fixed baselines and improves throughput as request pressure increases. For Llama2-7B, TP1PP8 has the highest latency among the three methods, while TP2PP4 reduces latency but still falls behind ReMP. ReMP maintains the lowest TTFT and TPOT across the tested request rates and delivers the highest throughput at high load. This indicates that even for
4.3.4 Analysis. These results demonstrate that model-parallel topology has a direct impact on serving performance, and the best topology is not fixed across models or request pressures. Pipeline-heavy configurations can improve model capacity distribution but may increase TTFT and TPOT due to deeper pipeline execution. More tensor-parallel configurations can reduce per-request latency but may introduce communication overhead or underutilize pipeline capacity under high load. Therefore, a static TP/PP configuration must compromise between different traffic regimes. ReMP avoids this compromise by making TP/PP topology a runtime-adjustable resource. When request pressure changes, ReMP can quickly evaluate candidate configurations and move the serving instance to the topology that provides the best observed latency-throughput trade-off. The results in Figure 7 and Figure 8 show that this dynamic selection consistently outperforms representative fixed baselines. Importantly, this improvement is enabled by the lowdowntime switching mechanism evaluated in Section 4.2:
ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
without fast reconfiguration, the cost of restarting the service would outweigh the benefit of adapting the topology. Overall, the serving-performance experiment confirms that ReMP is not only a faster reconfiguration mechanism, but also a practical foundation for adaptive LLM serving. By decoupling TP/PP topology from the launch-time configuration, ReMP allows the system to respond to workload changes and maintain better throughput and latency than fixed model-parallel deployments.
5
Conclusion
This paper presents ReMP, a runtime model-parallelism reconfiguration framework that enables adaptive LLM serving without full service restart. ReMP decouples model weights, KV cache, communication state, workers, and scheduler metadata from a fixed TP/PP topology, and supports lowdowntime switching through shared-memory model reloading, two-dimensional KV cache migration, prebuilt parallelstate snapshots, and worker reuse. Our evaluation shows that ReMP completes most topology switches within seconds, substantially reduces reconfiguration cost over restart-based switching, and improves latency and throughput under dynamic request pressure. ReMP demonstrates that model-parallel topology can be managed as a runtime-adjustable resource for modern LLM serving systems.
References [1] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association. [2] Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. 2022. DeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. [3] Xu Bai, Muhammed Tawfiqul Islam, Chen Wang, and Adel N. Toosi. 2026. PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving. arXiv preprint arXiv:2604.12171 (2026). [4] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems. [5] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arXiv preprint arXiv:2401.08671 (2024). [6] Hugging Face. 2023. Text Generation Inference. https://github.com/ huggingface/text-generation-inference. [7] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626.
Conference’17, July 2017, Washington, DC, USA
[8] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 663–679. [9] Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. SpotServe: Serving Generative Large Language Models on Preemptible Instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. [10] ModelTC. 2023. LightLLM: A Lightweight and High-Performance Large Language Model Service Framework. https://github.com/ ModelTC/lightllm. [11] NVIDIA. 2021. FasterTransformer: Faster Transformer Inference with FasterTransformer and Triton Inference Server. https://developer.nvidia.com/blog/fastertransformer-fastertransformer-inference-with-nvidia-triton-inference-server/. [12] NVIDIA. 2023. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. https://github.com/NVIDIA/ TensorRT-LLM. [13] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture. [14] Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. [15] Rui Qin, Ziqian Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association. [16] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning. 31094–31116. [17] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association. [18] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast Distributed Inference Serving for Large Language Models. arXiv preprint arXiv:2305.05920 (2023). [19] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. ORCA: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 521–538. [20] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [21] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association.