PhOSv1: Efficient Scaling of Model Sleeping on a Single Server for Autoscaling with GPU-Context Deduplication

arXiv:2610.39999v1 [cs.DC] 29 Oct 2026

P H OS V 1: E FFICIENT S CALING OF M ODEL S LEEPING ON A S INGLE S ERVER FOR AUTOSCALING WITH GPU-C ONTEXT D EDUPLICATION

Tianle Sun Shanghai Jiao Tong University China

Xingda Wei Shanghai Jiao Tong University China

Rong Chen Shanghai Jiao Tong University China

A BSTRACT Autoscaling is a key mechanism in model-as-a-service systems, where fast scale-up is essential because the scale-up time directly determines how long a model service remains overloaded— which in turn governs serving quality. The state-of-the-art approach—model sleeping, adopted by mainstream serving engines including vLLM and SGLang—achieves near-zero-cost scale-up by pausing pre-spawned engines on the GPU. However, sleeping is not a free lunch: each sleeping model must keep its GPU context resident, and this cost grows linearly with the number of models, degrading serving performance. Because providers must support many models, they face a dilemma between fast scale-up for all models and high service performance. In this paper, we show that a single server can keep many models asleep without paying this GPU context tax. The key observation is that the contexts of sleeping models are largely duplicated: the driver (CUDA) context can be shared across models, and their GPU kernel binaries overlap heavily due to common operators and libraries. P H OS V 1 deduplicates this context by transparently delegating each engine’s GPU context allocation to a central daemon through API remoting; therefore, a single server can keep far more models asleep than current engines allow. Our evaluation shows that P H OS V 1 matches the scale-up time of vanilla vLLM sleep mode while substantially reducing the GPU memory footprint, improving end-to-end multi-model serving performance. Keywords GPU context, LLM serving, cold start, checkpoint and restore, model pooling

1

Introduction

Model as a service—deploying multiple models on one or multiple shared clusters to serve LLM requests from applications—has become an important component of modern computing infrastructure [1–3]. Autoscaling— dynamically scaling an engine up and down on spare GPUs to serve a model—is essential for model-as-a-service systems. The reasons are twofold. First, a service provider typically deploys models from many vendors (e.g., DeepSeek, Qwen, GLM, and MiniMax), each of which offers models of different sizes. Second, the request arrival rate of a model fluctuates over time [1, 4, 5], because applications choose the requested model dynamically, depending on their budgets and capability requirements. If we statically deploy each model, assigning it a fixed number of GPUs without autoscaling, the model service becomes overloaded because it is underprovisioned when the request rate exceeds the capacity of the deployed GPUs. Conversely, GPUs become idle because of overprovisioning when the request rate decreases. Autoscaling dynamically adjusts each model’s GPU allocation according to its changing request rate. Efficient scale-up is a key system requirement for autoscaling. Scale-up is triggered when the request rate increases, but during the scale-up interval, the newly provisioned GPUs still cannot serve requests, so the model service remains overloaded. As a result, the scale-up time directly determines the duration of this overload, so model-as-a-service providers seek to minimize it. Unfortunately, current model autoscaling has a long scale-up time because starting an engine capable of serving a model requires a complex control plane to create the necessary GPU contexts (§2). To reduce this delay, the state-of-the-art

method—adopted by standard engines including vLLM and SGLang—is model sleeping [1, 6]. To avoid creating GPU contexts during startup, the system spawns multiple engines offline, each capable of serving one model online. After all the engines finish initialization, they are paused on the host server. Thus, a scale-up during online serving directly wakes up the engine, enabling near-zero-cost scale-up. Model sleeping is not a free lunch. To keep one model asleep, each engine must keep a substantial amount of context on the GPU—including the driver context (CUDA context) and the model’s GPU kernel binaries. This places a heavy burden on the GPU because GPU memory is limited and precious: in our experiment, sleeping 24 models consumes up to 27 GB (34%) of an 80 GB GPU’s memory, and this occupancy scales linearly with the number of sleeping models. This memory footprint can further degrade throughput by up to 54% when serving one model, because modern models reserve GPU memory for intermediate results (KV cache) to accelerate computation. Sleep mode must preserve the costly parts of this context on the GPU because, unlike model parameters, which can be restored quickly [4], these context components cannot be restored quickly. This poses a dilemma for the service provider: to enable fast scale-up for all models, the server pays online service overhead, while to ensure high service performance, the provider can only sleep a few models on each server, so the service still pays the scale-up costs. In this paper, we show that we can increase the number of sleeping models on one server without paying a linear GPU-context memory tax. The key observation is that the context for each sleeping model contains many duplicate components (up to 99% similarity between two popular models, as shown in Fig. 5): the driver context can be naturally shared across models, i.e., engines created later can reuse the driver context of an earlier engine, and the GPU kernel binaries of different models also substantially overlap because current models share operators and libraries (§3). By deduplicating this context, we can dramatically reduce the memory required to support more sleeping models on one server. To realize GPU context deduplication, we adopt a separation design that delegates each engine’s GPU context allocation to a central system daemon, where we can identify and avoid creating unnecessary contexts. To be compatible with existing engines, we leverage API remoting [7, 8] to implement this delegation transparently. One challenge of deduplication is that many internal GPU buffers allocated during initialization cannot be effectively deduplicated. Fortunately, such buffers can be created on the fly much faster than the other context components, so we treat them differently: we leave them unallocated while the model sleeps and reconstruct them at wake-up time. We have implemented deduplication from scratch and integrated it with vLLM’s sleep mode. Our evaluation shows that P H OS V 1 achieves a scale-up time similar to that of vanilla vLLM sleep mode while reducing the GPU memory footprint by 98.1% when sleeping 23 models. This saving improves end-to-end serving performance: in a multi-model trace replay, P H OS V 1 reduces average time to first token by 87.0% over vanilla vLLM, which can sleep only a limited number of instances within the memory budget. P H OS V 1 instead sleeps all required models without incurring the GPU memory burden of vanilla sleep mode. Contributions.

In summary, this paper makes the following contributions:

• The first characterization of the GPU-context memory tax of model sleeping and the opportunity to deduplicate it across models (§2, §3). • P H OS V 1, a transparent GPU-context deduplication system with delegated context management in a service daemon, shares duplicate driver contexts and kernel binaries, and reconstructs non-deduplicable buffers only at wake-up time (§3). • An implementation integrated with vLLM sleep mode and an evaluation showing that P H OS V 1 retains its near-zero-cost scale-up while reducing GPU-context memory use and improving end-to-end multi-model serving performance (§4). P H OS V 1 is open-sourced at https://github.com/blitz-serving/PhoenixOSv1.

2

System Model and Problem Statement

Multi-model serving and autoscaling. AI service providers typically host many models of various sizes from different vendors simultaneously in a shared GPU cluster [1–3], because no single model dominates user demand: models differ in capability and cost. Each model occupies one or more dedicated GPUs due to its size. A key challenge for model providers is allocating GPU resources to each model. Static GPU allocation—allocating a fixed number of GPUs to each model at deployment time—is suboptimal because per-model request rates fluctuate over time [1, 4, 5]. Consider assigning a fixed GPU budget to each model. When the provisioned GPUs exceed the actual 2

(a) --- Workload requirement of a model --- Serving system performance

ui Req

ent rem

Time Impact serving performance

(b)

Scale :me

Vanilla start (e.g., vLLM cold start)

CPU context

GPU context

Model loading

6.6 s

3.5 s

<1s

Start w/ C/R

0.57 s

~=0

<1s

Start w/ sleeped model

~=0

~=0

<1s

(c)

Figure 1: (a) How autoscaling copes with dynamic workloads. (b) The impact of scale-up speed on serving performance. (c) A breakdown of autoscaling time across different approaches.

requirement, GPUs are wasted. On the other hand, if insufficient GPUs are provisioned, requests queue and experience high latency. Modern service providers therefore use autoscaling to adjust GPU allocation per model in response to traffic (Fig. 1(a)). Specifically, autoscaling makes GPU provisioning for each model elastic: when a model’s load rises, the scheduler picks one or more idle GPUs, starts one or more inference engines on the host servers, and loads the model weights to expand the model’s serving capacity (scale up). When a model’s load falls, the scheduler deallocates a subset of the model’s GPUs, releasing them back to the pool for other models to save GPU time (scale down). A key system requirement of autoscaling is fast startup during scale-up: until the newly launched inference engines become ready to serve, requests to the model being scaled up must queue; a slow startup therefore directly causes request stalls and SLO violations during the scaling window (Fig. 1 (b)). The faster the scale-up, the smaller the performance penalties. Goal and design space overview. Our design goal is to reduce the cold start overhead of autoscaling a serving instance. To understand how to achieve this, we first break down the cold start into two stages and then analyze the detailed steps involved to determine how much we can optimize them. Fig. 1(c) illustrates the two stages: a control plane that builds the serving engine’s runtime context on the CPU and GPU, and a data plane that loads the model weights into GPU memory. The control plane builds two contexts. Building the CPU context starts the Python interpreter that runs the inference engine for a model. The interpreter imports the serving stack (e.g., PyTorch and vLLM), loading tens of thousands of Python modules and instantiating engine components such as the request scheduler. Building the GPU context initializes the GPU driver—e.g., by creating the CUDA primary context [9] on the target device— loads the model’s GPU kernels into device memory, and constructs the runtime handles the engine uses to launch GPU work—CUDA streams, cuBLAS, cuDNN, and so on. The data plane copies the model weights from host memory or external storage into GPU memory. Its latency is bounded below by the weight size divided by the link bandwidth (e.g., PCIe or RDMA), and has been optimized to approach the hardware limit through recent work [4] on network-based multicast weight loading. If the server starts from a clean state (without any optimizations), these stages could take seconds or even minutes, as we describe in §4. As a result, prior work has proposed various methods to mitigate the slow startup issue. In this paper, we focus on techniques to reduce the control-plane overhead, as the data plane has been optimized to approach the hardware limit thanks to recent work [4, 10]. Current solutions: checkpoint/restore and model sleeping. One intuitive way to accelerate cold start is to use checkpoint/restore [11]. Specifically, we checkpoint an engine that has completed control-plane initialization into an image; on a future restart, we restore from the checkpoint image and thus bypass all the control-plane overhead. However, checkpoint/restore acceleration has two issues when applied to modern inference engines. 3

30 20 10 00

6

12

18

# sleeping models

Thpt (tok/s)

GPU mem (GiB)

Figure 2: (Top) An illustration of using model sleeping to multiplex one or more GPUs across multiple models. (Bottom) An illustration of the GPU memory requirement for model sleeping.

700

24

300 0

500 4

8

12

16

Occupied mem (GiB)

Figure 3: (Left) GPU memory used to cache sleeping models grows linearly with their number. (Right) Serving throughput drops as available GPU memory shrinks, evaluated with Qwen3.6-27B on an A800 GPU. First, performance: P HOENIX OS [11], the state-of-the-art system, still incurs 0.57 s of overhead when restoring an engine during scale-up, because it takes time to parse the checkpoint image and restore the process’s memory pages and kernel state, as shown in Fig. 1(c). In the worst case, we found that restoration can take 10–20 seconds to launch an application because of unexplained delays in the CPU-side context restoration, which is performed using CRIU [12]. Second, compatibility: modern inference engines adopt complex optimizations that our CPU-side checkpoint/restore tool (based on CRIU [12]) cannot support today, such as io_uring [13]. Supporting them would require large changes to the current serving engines, or a huge implementation effort in CRIU, which is impractical because serving engines continue to evolve rapidly. Another approach adopted by current systems—including Aegaeon [1] and the sleep mode of vLLM and SGLang [6]—is model sleeping. It resembles the warm-start technique in serverless computing for cold start acceleration [14], but applies it in the model-as-a-service setting. Specifically, to enable fast cold start, the sleep mode prewarms a few engines on the server, each capable of serving one model, and then uses OS utilities [15] to pause these engines before they receive requests. A later scale-up unpauses the required engine, bypassing all the control-plane overhead. After the model is scaled down, the engine is re-paused. In the per-GPU serving model considered here, each GPU may cache multiple sleeping engines, but at most one engine wakes up and executes on that GPU at a time. Key issue of model sleeping: memory scales linearly with the number of sleeping models and slows serving. Having discussed the issues of checkpoint/restore above, a natural question is whether model sleeping addresses the cold-start issue. One key problem of model sleeping is that it trades GPU memory for fast startup. This limits the number of models that can be prewarmed on each machine, because the memory cost grows linearly with the number of sleeping models. Specifically, to make a later scale-up’s control-plane cost near zero, a paused engine keeps its GPU context resident in GPU memory and its CPU context resident in host memory. The CPU context occupies a relatively small fraction of host memory (e.g., vLLM holds 2.9 GB on the CPU) and can be shared across processes using host-side shared memory, but the GPU context is substantial and currently cannot be shared. As a result, keeping 4

Time (ms) Mem. (MiB)

2000

CUDA ctx 1360

0 1000

1184 1238

Kernels

GPU buffers 1272

544

Handles 1272 1260 1338

1646

0

. .5B -3B -7B -7B .6B 14B 3B -In 32B -4B -8B -2B 27B -9B 27B 3B -20b -8B -1B -3B 4B 31B E2B E4B .5-1 en2.5 en2.5 .5-VL en3-0 en3- 30B-A -A3B en3- wen3 wen3 3-VL n3.5- en3.5 n3.6- 35B-A t-oss a-3.1 a-3.2 a-3.2 26B-A ma-4- a-4- a-42 n Q wen we Qw we 3.6- gp lam lam lam a-4- em emm emm e Qw Qw en2 Qw Qw n3- 30B Qw Q Q Q en L L L m e g g g Qw Q Qw Qw wen3 Qw gem Q

Figure 4: (Top) GPU memory used (without parameters) by each sleeping model, broken down by resource type, and (Bottom) the time required to restore this memory.

more sleeping models on the same GPU wastes more GPU memory. Fig. 3(left) measures this growth using vLLM v0.19.1: caching 24 models consumes about 27 GB of GPU memory. Note that we have already excluded model parameters from this measurement because they can be restored quickly through recent autoscaling techniques. This cost applies even when a GPU serves only one model at a time: as long as it keeps other models sleeping, their GPU contexts stay resident. P H OS V 1 focuses on reducing this resident GPU-context memory cost; concurrent sharing and scheduling of multiple active engines on the same GPU are outside its scope. This memory cost creates a trade-off for providers. Modern LLM providers serve many models [1–3]. Model sleeping removes the control-plane cost only for models already sleeping, so avoiding that cost across many models requires keeping many of them asleep. One may ask whether the memory cost of keeping many models asleep is affordable. We found it is not, because the resident memory hurts serving: the more models kept asleep, the more serving performance drops. This is because serving a model requires storing its KV cache—the intermediate results of computation—in GPU memory. With less GPU memory available for the KV cache, requests queue and slow down significantly [16], and the model’s context length is restricted. Fig. 3(right) quantifies this: serving throughput falls from 747 to 341 tokens/s (a 54% drop) as the memory occupied by sleeping models grows from 0 to 16 GB. Providers therefore keep only a few models asleep today and pay the full control-plane cost for models not kept asleep.

3

Design of P H OS V 1

P H OS V 1 therefore has two goals: (1) remove the control-plane cost, even when a GPU is likely to sleep many models at the same time; (2) avoid the GPU memory cost of keeping many models asleep. Key design opportunity: GPU context across sleeping models is redundant. P H OS V 1 builds upon model sleeping to remove the control-plane cost, which enables fast scale-up. To further address its memory expansion issues, we make a key observation: their GPU contexts contain substantial duplicate content. If we can deduplicate this content, we can dramatically reduce the GPU memory required to keep multiple models asleep. More specifically, we first break down the GPU memory used for each sleeping model in detail in Fig. 4 to assess how much of their context we can deduplicate. Four components contribute to GPU-resident memory: CUDA context—the GPU runtime environment containing control state for virtual-memory management and synchronization events [17]; GPU kernels—the compiled GPU code that runs the model’s inference; GPU buffers—memory allocated for the model’s internal use; and handles—registries for resources and libraries used to access the GPU, such as CUDA streams and cuBLAS [18]. Except for the GPU buffers, which are model-dependent, and the handles, whose memory usage is negligible, we found that most content can be deduplicated. First, the CUDA context can be shared across models. This is straightforward because the driver context itself is model-independent, i.e., the context used by one model can trivially be used for another. Moreover, the kernels used by models also have a high degree of overlap: Fig. 5 provides the pairwise similarity of GPU kernels across models excluding GPU buffers. We choose models with high download counts on Hugging Face [3] because they represent popular models typically served by model-as-a-service providers. Most model pairs have 59–99% kernel similarity. Despite differences in architecture and size, existing models share similar kernels because (1) many operations, such as matrix multiplication, are common across models, and (2) a narrow set of libraries (e.g., FlashInfer [19]) provides these operations because writing them efficiently for GPUs is non-trivial. As a result, 5

0.6

Similarity 0.8

1.0

1.00

0.99

1.00

Qw

en Qw -1.5 e B Qw Qwen-3B e n Qwn-VL-7B Qwen3- -7B en 0.6 Qw 3-A B e 3B Qw Qwen3-4 en n3 B Qw3-VL-8B en -2B Lla3.5-9 B Llama-8 B m L a Ge lam -1B m a Ge ma -3B m Ge ma 26B mm -31 Ge a B mm -E2 a-E B 4B

Qwen-1.5B Qwen-3B 1.00 Qwen-7B Qwen-VL-7B Qwen3-0.6B Qwen3-A3B Qwen3-4B 1.00 Qwen3-8B Qwen3-VL-2B Qwen3.5-9B Llama-8B Llama-1B Llama-3B 0.99 Gemma-26B Gemma-31B Gemma-E2B Gemma-E4B

Figure 5: Pairwise similarity of GPU kernels across popular models. Context service

vLLM process A

vLLM process B

GPU binary

CPU binary

CPU binary

Driver context Kernels

Parameter

Parameter

Qwen 2.5

Qwen 3.5

Per-process

1

GPU binaries

Parameter

vLLM proce

…

CPU binary Paramete Qwen 3

Others (runtime) e.g., KVCache

GPU GPU binary of shared context

GPU HBM

Figure 6: Architecture of P H OS V 1’s context service, which shares GPU context across sleeping models.

deduplicating CUDA context and kernels can dramatically reduce the GPU memory used to keep multiple models asleep. One may notice that the GPU buffers still account for much of the GPU memory used for some models (e.g., Qwen3.627B, as shown in the top panel of Fig. 4). Nevertheless, we found that such buffers can be dynamically restored quickly: as shown in the bottom panel of Fig. 4, creating them from scratch only takes ∼30 ms, compared with the ∼430 ms required to create a CUDA context. Such buffers are typically reserved for internal use at runtime, so we can safely avoid storing them on the GPU and create them on demand. System architecture. To realize deduplication, we need to ensure that all the GPU contexts from different sleeping models are managed together by a central service, so that it can detect duplicate components and avoid creating duplicate resources. Thus, P H OS V 1 deploys a per-machine system daemon for managing GPU contexts from different models, as shown in Fig. 6. Specifically, unlike the existing sleep mode, in which each model’s engine process manages its own GPU context (the top panel of Fig. 2), the context service manages GPU kernel binaries in a single CUDA context shared by all models assigned to a GPU, along with per-model resources, such as customized kernels. On a server with multiple GPUs, the daemon manages one such context per GPU, allowing wake-ups on different GPUs to proceed 6

independently. The kernels are properly deduplicated: before a model loads a kernel binary, we check whether the load is necessary, i.e., whether the same binary has already been loaded. If so, we skip the load to reduce the GPU memory footprint. Below, we describe P H OS V 1 in detail: how we construct the context service offline and how the online serving system uses it to support inference requests after an engine wakes up. Offline phase: sleep the models and initialize the context service. The offline phase initializes the sleeping engines to support efficient online scale-up. This phase is optional but recommended: otherwise, the system initializes models in the context service on demand, i.e., upon first startup, incurring a one-time scale-up delay. In practice, this process can also be performed incrementally, by reusing scale-up instances or starting models during request idle time, thereby amortizing its cost and avoiding subsequent scale-up delays. The offline phase works as follows. Given N models that need fast startup via sleep mode on one server, we first initialize a context service as a background daemon that manages the GPU resources. After the daemon is initialized, we start the engines (e.g., vLLM/SGLang) one by one, putting each engine into sleep mode once it finishes starting up. Once asleep, an engine’s per-process state is paused, while its GPU context is held by the context service rather than by the engine itself. To achieve this, the engine does not create GPU resources directly; instead, it delegates resource creation to the context service. We make this delegation transparent to the engine using a well-established GPU virtualization technique: API remoting [7, 8]. Specifically, we use LD_PRELOAD to hook all the GPU resource creation calls at the CUDA Driver API boundary and execute them in the context service, requiring no source-code changes to inference engines. After receiving a resource creation request, the context service examines whether the resource already exists. If the resource (e.g., a kernel) has not been created, the context service creates it and records the creation information, so that later requests can recognize it. We use a lookup table that encodes the API semantics: for example, cuModuleLoadData loads a kernel binary into the GPU. A subsequent call with the same binary will be recognized as a duplicate, so the service simply skips the creation. The table incorporates all CUDA APIs that inference engines use for GPU resource management and therefore covers all GPU-context allocations. GPU buffers require special handling. If an engine requests a buffer allocation, the service creates a virtual placeholder for it but delays the real allocation until the online phase. This is fine because, as shown in the bottom panel of Fig. 4 and discussed above, creating these resources online is fast. The GPU buffer’s contents are checkpointed in host memory to ensure correct restoration. Online phase: supporting model wake-up and inference requests. During the online phase, after an engine wakes up to serve requests, we only need to direct it to use the GPU context stored by the context service for inference. We use the same resource-management mechanism described above: API remoting. When an engine invokes a GPU kernel, the call is first redirected to the context service, which creates the required GPU buffers if necessary and then invokes the kernel. Remoting the GPU API may add performance overhead to inference. This overhead is negligible in the evaluated workloads (§4.1) thanks to optimizations including batching and synchronization, because inference is primarily bottlenecked by GPU execution rather than CPU-side kernel launches [7]. If the model’s context is not already held by the context service, we perform the sleep procedure described above to store its context there, so that future wake-ups avoid a cold start. Storing a context that the service does not yet hold typically reserves additional GPU memory to accommodate the new model. Left unchecked, this could lead to a large memory footprint as we keep more models asleep. To address the issue, we set a maximum amount of GPU memory that may be reserved for sleeping models, and evict lesser-used models from the context service if necessary. Specifically, we use a least-frequently-used (LFU) strategy to evict sleeping models. Whether other strategies can further improve system performance is beyond the scope of our work.

4

Evaluation

We have implemented P H OS V 1 in 10,825 lines of Rust. Our evaluation answers three questions: (1) Does P H OS V 1 improve end-to-end multi-model serving performance under autoscaling? (2) Does it remove the linear GPU-memory cost of sleeping many models? (3) Does it preserve the low scale-up latency of vanilla sleep mode? Testbed. We evaluate P H OS V 1 on a single server with eight NVIDIA A800 GPUs, each with 80 GB of HBM. All measurements reported in this section use a single GPU. The server has two Intel Xeon Platinum 8350C CPUs providing 64 cores in total and 1 TB of DRAM. The server runs CUDA 12.9 [20] and vLLM v0.19.1. 7

vLLM sleep

with PhOSv1 Latency (s)

GPU mem (GiB)

30 20 10 0

0

6

12

18

23

# cached models

PhoenixOS

Cold start

vLLM sleep

with PhOSv1

40 20 0

B B I n. B B B B B B B B B B B B B -1.5 n2.5-3 n2.5-7 -VL-7 n3-0.6 0B-A3 A3B- en3-4 en3-8 -VL-2 n3.5-9 -3.1-8 -3.2-1 -3.2-3 6B-A4 a-4-31 5 . 2 a a a 5 3 w w e e e e 2 3 . n m n B m m m 2 Q Q w w w w e Q e Q Lla Lla Lla a-4 em Q wen Q en3 3-30 g Qw Qw m Q Qw wen gem Q

Figure 7: (Left) GPU memory used to cache sleeping models under vanilla vLLM sleep mode and P H OS V 1. (Right) Startup latency across models for cold start, vanilla sleep, P HOENIX OS, and P H OS V 1. Weight-loading times for cold start and P HOENIX OS are normalized to remove storage I/O overhead.

vLLM PhOSv1 100

200

250

500

2k 00

Output tok/s

Input tok/s

00

1

TTFT w/o queue (s)

00 100

20

CDF

E2E TTFT (s)

CDF

1

00

250

500

Time (seconds) Figure 8: End-to-end serving performance. Baselines. We compare P H OS V 1 with three baselines. Cold start launches a vanilla vLLM inference engine from scratch when a model is requested. Vanilla sleep uses vLLM sleep mode: it discards the KV cache, keeps model parameters in host memory, and later wakes the same vLLM engine. P HOENIX OS uses checkpoint and restore to start the engine: after sleep, the process is checkpointed; during scale-up, the process is restored and then woken up. Workloads and metrics. We describe the end-to-end workload in §4.1. Unless stated otherwise, we use 16 Qwen and Qwen-VL models with high download counts on Hugging Face [3]. We report GPU memory footprint, startup latency, end-to-end time to first token (TTFT), TTFT without queueing delay, and input- and output-token throughput. End-to-end TTFT spans the interval from when a request is issued to when its first token is produced and may therefore include queueing behind earlier requests. TTFT without queueing delay spans the interval from when the system begins scheduling the target model to the first token; if the model is inactive, it includes its cold-start or wake-up time. Measurement methodology. We use NVML [21] to measure GPU memory currently occupied by the vLLM process. When reporting GPU-context memory, we measure GPU memory after putting the vLLM engine to sleep; consequently, model weights are excluded. 4.1

End-to-end experiments

Evaluation setup. To evaluate whether P H OS V 1 improves user-visible multi-model serving performance under autoscaling, we construct a multi-model end-to-end trace using Aegaeon’s trace methodology [1]. Specifically, we serve 10 popular Qwen models ranging from 0.6B to 32B parameters [3]. The 10-minute ShareGPT [22] trace is generated with Poisson arrivals, with each model receiving requests at a rate of 0.3 requests/min. We set this per-model arrival rate to represent scenarios with infrequent requests to many long-tail models, where model sleeping is required for fast startup [1]. The vanilla-sleep baseline uses a 6 GB sleep-cache budget, reserving the remaining GPU memory for the active model’s weights and KV cache; it keeps as many sleeping vLLM instances as possible under this budget, 8

puts a model back to sleep after inference, evicts models using LFU when the budget is exceeded, and cold-starts a model when it is not cached. This 6 GB budget is the largest that preserves vLLM’s default GPU-memory allocation; a larger budget would reduce the KV-cache capacity and thus the supported context length. We use the same vLLM configuration for both P H OS V 1 and vanilla sleep. Overall performance. Figure 8 shows that P H OS V 1 reduces the average end-to-end TTFT from 146.82 s to 19.09 s, an 87.0% improvement over vanilla vLLM instances limited by the sleep-cache budget. It also reduces p50 TTFT from 142.39 s to 14.65 s, while finishing the same workload in a shorter active serving window (578 s vs. 728 s). The improvement comes from caching and sleeping the required models without consuming the large linear GPU-context memory of vanilla sleep mode; the baseline can only keep a limited set of models warm under the memory budget, so requests for uncached models pay cold-start delay and create queueing that dominates end-to-end TTFT. Overhead of our technique. To evaluate the inference overhead introduced by our shared context, we report the throughput (tokens per second) of P H OS V 1 and vanilla vLLM in the bottom panels of Figure 8. We can see that P H OS V 1’s throughput is at least 97.3% of vanilla vLLM’s, indicating that the overhead introduced by the shared context is small. 4.2

GPU context memory reduction and startup performance

Figure 7 (left) evaluates whether P H OS V 1 removes the GPU-memory growth that prevents vanilla sleep mode from scaling to many cached models. We use 23 models from Figure 4; model weights are excluded. Figure 7 shows that vanilla sleep consumes 25.67 GiB of GPU memory after caching 23 models, whereas P H OS V 1+sleep uses only 0.49 GiB, reducing the memory footprint by 98.1%. The savings come from sharing the CUDA context and deduplicating common kernel binaries in the context service instead of letting each sleeping engine hold a private GPU context. Figure 7 (right) further evaluates whether P H OS V 1 preserves the low wake-up latency of sleep mode while adding a context service connection and API-remoting layer. We use 16 models and compare cold start, waking up a sleeping vLLM model, P HOENIX OS, and P H OS V 1. Figure 7 shows that P H OS V 1 keeps startup latency comparable to vanilla sleep, averaging 4.12 s across 16 models, compared with 4.89 s for vanilla sleep and 24.41 s for cold start. Startup remains low because the expensive control-plane context is already available in the service, and the per-model cost of connecting to the context service is small, averaging 72 ms.

5

Discussion

This section discusses several assumptions and limitations of P H OS V 1. First, P H OS V 1 assumes that engines sharing a CUDA context are trusted. This assumption applies to model-as-a-service deployments [1], where the service provider manages the engines that serve different models. Sharing a context across untrusted engines is outside the scope of our work. Second, P H OS V 1 uses API remoting to implement context sharing, which may introduce inference overhead. This implementation choice allows P H OS V 1 to share a context without modifying NVIDIA’s closed-source CUDA driver. Implementing context sharing directly in the driver could avoid the remoting overhead. Finally, P H OS V 1 does not support CUDA graphs. Checkpointing and restoring these graphs requires preserving or reconstructing the device pointers and kernel references embedded in graph nodes, as discussed in prior work [23, 24]. We leave support for CUDA graphs to future work.

6

Related Work

Fast startup for serverless and LLM serving. Prior work accelerates serverless container startup through caching, checkpoint/restore, and remote fork, reusing initialized execution environments [25–27]. Related techniques accelerate VM startup [28] and GPU checkpoint/restore [11, 29]. For LLM serving, ServerlessLLM exploits storage locality for fast parameter loading [30]. BlitzScale combines network transfer and cooperative execution for fast autoscaling [4], and FaaSwap swaps weights under SLO constraints [31]. Medusa and Foundry accelerate CUDA graph setup through offline materialization [23, 24]. These approaches reduce parameter-loading costs or reconstruct selected execution state, complementing P H OS V 1’s deduplication of resident GPU context across sleeping engines. Model sleeping in vLLM/SGLang and Aegaeon [1, 6] avoids startup overhead but retains a separate GPU context for each model (§2). 9

OS primitives for GPUs. NVIDIA MPS supports concurrent CUDA processes through shared scheduling resources [32]. P H OS V 1 instead targets GPU-context deduplication across sleeping engines for fast scale-up. MIG partitions a GPU into isolated compute and memory instances [33], whereas we target serving workloads that use a full GPU for the active model. TGS shares GPUs across containers through rate control and unified-memory oversubscription [34]. REEF provides microsecond-scale kernel preemption; Paella offers software-defined GPU scheduling; and XSched extends preemptive scheduling to diverse XPUs [35–37]. GPU API remoting and virtualization decouple applications from physical devices [7, 8]. P H OS V 1 builds on API remoting to provide GPU-context deduplication across sleeping models.

7

Conclusion

We contribute GPU-context deduplication, the first technique that lets a single server keep many models asleep for fast autoscaling without paying a linear GPU memory tax. The key insight is that the GPU contexts of sleeping models contain largely duplicate components that can be shared across models through a central daemon, transparently to existing inference engines. Our evaluation shows that P H OS V 1 preserves the fast scale-up of model sleeping while substantially reducing its memory footprint, improving end-to-end multi-model serving performance.

References [1] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective GPU pooling for concurrent LLM serving on the market. In Youjip Won, Youngjin Kwon, Ding Yuan, and Rebecca Isaacs, editors, Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP 2025, Lotte Hotel World, Seoul, Republic of Korea, October 13-16, 2025, pages 1030–1045. ACM, 2025. [2] OpenRouter. The unified interface for llms. https://openrouter.ai, 2026. [3] Hugging Face. The ai community building the future. https://huggingface.co, 2026. [4] Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. Blitzscale: Fast and live large model autoscaling with O(1) host caching. In Lidong Zhou and Yuanyuan Zhou, editors, 19th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2025, Boston, MA, USA, July 7-9, 2025, pages 275–293. USENIX Association, 2025. [5] Yuxing Xiang, Xue Li, Kun Qian, Wenyuan Yu, Ennan Zhai, and Xin Jin. Servegen: Workload characterization and generation of large language model serving in production. CoRR, abs/2505.09999, 2025. [6] vLLM. Sleep mode, 2026. [7] Tianxia Wang, Zhuofu Chen, Xingda Wei, Jinyu Gu, Rong Chen, and Haibo Chen. Characterizing network requirements for GPU API remoting in AI applications. CoRR, abs/2401.13354, 2024. [8] Michela Becchi, Kittisak Sajjapongse, Ian Graves, Adam M. Procter, Vignesh T. Ravi, and Srimat T. Chakradhar. A virtual memory based runtime to support multi-tenancy in clusters with gpus. In Dick H. J. Epema, Thilo Kielmann, and Matei Ripeanu, editors, The 21st International Symposium on High-Performance Parallel and Distributed Computing, HPDC’12, Delft, Netherlands - June 18 - 22, 2012, pages 97–108. ACM, 2012. [9] NVIDIA Corporation. CUDA driver API: Primary context management. https://docs.nvidia.com/cuda/ cuda-driver-api/group__CUDA__PRIMARY__CTX.html, 2026. [10] Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Yue Cheng, Wei Wang, Ao Wang, and Ruichuan Chen. lambdascale: Enabling fast scaling for serverless large language model inference, 2025. [11] Xingda Wei, Zhuobin Huang, Tianle Sun, Yingyi Hao, Rong Chen, Mingcong Han, Jinyu Gu, and Haibo Chen. Phoenixos: Concurrent os-level GPU checkpoint and restore with validated speculation. In Youjip Won, Youngjin Kwon, Ding Yuan, and Rebecca Isaacs, editors, Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP 2025, Lotte Hotel World, Seoul, Republic of Korea, October 13-16, 2025, pages 996–1013. ACM, 2025. [12] CRIU. Criu main page, 2026. [13] io_uring support in criu #2131, 2026. 10

[14] Mohammad Shahrad, Rodrigo Fonseca, Iñigo Goiri, Gohar Irfan Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In Ada Gavrilovska and Erez Zadok, editors, Proceedings of the 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, pages 205–218. USENIX Association, 2020. [15] The Linux Kernel. Cgroup freezer subsystem. https://www.kernel.org/doc/Documentation/cgroupv1/freezer-subsystem.txt, 2026. [16] Rongxin Cheng, Yuxin Lai, Xingda Wei, Rong Chen, and Haibo Chen. KUNSERVE: parameter-centric memory management for efficient memory overloading handling in LLM serving. In Antonio Barbalace, Luo Mai, Roxana Geambasu, and Peter R. Pietzuch, editors, Proceedings of the 21st European Conference on Computer Systems, EuroSys 2026, McEwan Hall/The University of Edinburgh, Edinburgh, Scotland, UK, April 27-30, 2026, pages 1244–1260. ACM, 2026. [17] NVIDIA. Cuda c++ programming guide. https://docs.nvidia.com/cuda/cuda-c-programming-guide/, 2024. [18] NVIDIA. Basic linear algebra on nvidia gpus. https://developer.nvidia.com/cublas, 2025. [19] FlashInfer: Kernel Library for LLM Serving. https://github.com/flashinfer-ai/flashinfer, 2026. [20] NVIDIA. Cuda toolkit 12.3 downloads. archive, 2024.

https://developer.nvidia.com/cuda-12-3-0-download-

[21] NVIDIA Corporation. NVIDIA Management Library (NVML) API Reference Guide, 2026. [22] ShareGPT. Sharegpt dataset. https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_ unfiltered, 2023. [23] Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. Medusa: Accelerating serverless LLM inference with materialization. In Lieven Eeckhout, Georgios Smaragdakis, Kaitai Liang, Adrian Sampson, Martha A. Kim, and Christopher J. Rossbach, editors, Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, pages 653–668. ACM, 2025. [24] Xueshen Liu, Yongji Wu, Yuncheng Yao, Danyang Zhuo, Ion Stoica, and Z. Morley Mao. Foundry: Templatebased CUDA graph context materialization for fast LLM serving cold start. CoRR, abs/2604.06664, 2026. [25] Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. SOCK: Rapid task provisioning with serverless-optimized containers. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 57–70, Boston, MA, July 2018. USENIX Association. [26] Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen. Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting. In James R. Larus, Luis Ceze, and Karin Strauss, editors, ASPLOS ’20: Architectural Support for Programming Languages and Operating Systems, Lausanne, Switzerland, March 16-20, 2020, pages 467–481. ACM, 2020. [27] Xingda Wei, Fangming Lu, Tianxia Wang, Jinyu Gu, Yuhan Yang, Rong Chen, and Haibo Chen. No provisioned concurrency: Fast rdma-codesigned remote fork for serverless computing. In Roxana Geambasu and Ed Nightingale, editors, 17th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2023, Boston, MA, USA, July 10-12, 2023, pages 497–517. USENIX Association, 2023. [28] Lixiang Ao, George Porter, and Geoffrey M. Voelker. Faasnap: Faas made fast using snapshot-based vms. In Yérom-David Bromberg, Anne-Marie Kermarrec, and Christos Kozyrakis, editors, EuroSys ’22: Seventeenth European Conference on Computer Systems, Rennes, France, April 5 - 8, 2022, pages 730–746. ACM, 2022. [29] Shaoxun Zeng, Tingxu Ren, Jiwu Shu, and Youyou Lu. GPU checkpoint/restore made fast and lightweight. In André Brinkmann and Philip Shilane, editors, 24th USENIX Conference on File and Storage Technologies, FAST 2026, Santa Clara, CA, USA, February 24-26, 2026, pages 239–254. USENIX Association, 2026. [30] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. Serverlessllm: Locality-enhanced serverless inference for large language models. CoRR, abs/2401.14351, 2024. [31] Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping. CoRR, abs/2306.03622, 2023. [32] NVIDIA Corporation. NVIDIA multi-process service documentation. https://docs.nvidia.com/deploy/ mps/index.html, 2026. 11

[33] NVIDIA Corporation. NVIDIA multi-instance GPU user guide. https://docs.nvidia.com/datacenter/ tesla/mig-user-guide/latest/index.html, 2026. [34] Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. Transparent GPU sharing in container clouds for deep learning workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 69–85. USENIX Association, 2023. [35] Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558. USENIX Association, 2022. [36] Kelvin K. W. Ng, Henri Maxime Demoulin, and Vincent Liu. Paella: Low-latency model serving with softwaredefined GPU scheduling. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace, editors, Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 595–610. ACM, 2023. [37] Weihang Shen, Mingcong Han, Jialong Liu, Rong Chen, and Haibo Chen. Xsched: Preemptive scheduling for diverse xpus. In Lidong Zhou and Yuanyuan Zhou, editors, 19th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2025, Boston, MA, USA, July 7-9, 2025, pages 671–692. USENIX Association, 2025.

12

Related documents

• PhOSv1: Efficient Scaling of Model Sleeping on a Single Server for Autoscaling with GPU-Context Deduplication#1491607
• No-Restart Elasticity in an Adaptive Runtime System for Cloud-Native HPC#1116548
• Beyond Pre-Training: The Full Lifecycle of Foundation Models on HPC Systems#13058
• GUMPIM: Unitary and Malleable Memory for Processing-in-Memory with Guaranteed PIM Pages#1387682
• GPU-Native Approximate Nearest Neighbor Search with IVF-RaBitQ: Fast Index Build and Search#785115