ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform Xiaoxiao Jiang†∗ , Suyi Li†∗# , Sheng Yao† , Tianyu Feng† , Lingyun Yang† , Dapeng Nie, Haoran Yang, Wei Wang† † Hong Kong University of Science and Technology Alibaba Group provisioning, elastic scaling, and accounting to the platform. Consequently, a serverless T2I platform must efficiently multiplex customized workflows under dynamic traffic, all while hiding infrastructure management details like GPU assignment and data movement from the user. However, current serverless cloud platforms fall short in T2I serving. As a common practice, users are required to compose workflows using third-party tools such as ComfyUI [22] or Diffusers [85], and then deploy the entire T2I workflow as a single, monolithic GPU function [5, 20, 21]. This practice obscures internal model executions and data exchanges, leading to three problems that burden users and limit platform efficiency. First, while existing works enable multi-GPU parallelism for T2I inference [29, 52, 55], they target limited workflows and lack abstractions for flexible model placement and tensor movement. This forces users building customized workflows to manually manage GPU assignment and inter-GPU communication, violating the serverless principle of hiding infrastructure details [74]. Second, T2I workflows contain large models of sizes up to tens of GiB, making scale-out bottlenecked by loading overhead (cold starts [13]). Standard techniques that overlap loading with inference fail when the loading overhead dominates; in a Flux [51] workflow, per-model loading is 3.2×–13.5× slower than the model’s single inference pass. Third, GPU scarcity makes multi-tenant T2I serving backlog-prone during peak hours [26, 79]: on our platform, tens of thousands of requests can queue, and major users can experience up to 10% of requests backlogged. This exposes a fair scheduling problem: because user-composed workflows have heterogeneous resource demands, treating them as uniform requests allows costlier workflows to consume a disproportionate share of GPU time. Despite these challenges, serverless T2I inference for customized workflows remains largely unexplored. Prior systems [2, 29, 52, 55, 58, 76, 84] optimize individual T2I workflows through kernel optimization, parallelization, or inworkflow caching, but do not address the multi-tenant challenge of serving thousands of distinct workflows under dynamic traffic. We propose ServerlessT2I, a serverless-native system for T2I workflow inference. ServerlessT2I represents workflows as model DAGs (directed acyclic graphs), making individual model invocations and their data dependencies the fundamental units for execution, scaling, and scheduling. ServerlessT2I introduces three system components, each addressing one challenge.
arXiv:2607.26566v1 [cs.DC] 29 Jul 2026
Abstract Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently. Existing platforms typically deploy each workflow as an opaque GPU function, provisioning, placing, and scaling all constituent models in the workflow together. This monolithic design obscures workflow structure, inflates scaling overhead, forces users to manage low-level GPU coordination, and limits fine-grained fairness in multi-tenant clusters. In this paper, we present ServerlessT2I, a serverless-native system that decomposes a T2I workflow into loosely coupled model functions that can be independently managed and scheduled. By explicitly managing individual model execution, ServerlessT2I enables per-model scaling, declarative workflow composition, transparent GPU-resident communication, and fairness-aware scheduling. To make this decomposition efficient, ServerlessT2I harvests slack GPU memory left idle by compute-bound T2I inference to build a data plane that reduces model loading and data communication overheads. ServerlessT2I further introduces a fair scheduler for multi-tenant serving. Using production traces, ServerlessT2I sustains up to 2× higher request rates than existing T2I workflow serving systems with the same GPU budget; for a fixed request rate, it saves up to 3× GPU resources while satisfying service level objectives (SLOs).
1
Introduction
Text-to-image (T2I) workflows built on diffusion models are a cornerstone of modern image generation [46, 62, 70, 71, 94, 112], underpinning commercial services [1, 55, 61, 71] that serve millions of users at more than 10K requests per second (RPS) in production [57]. In a mainstream public cloud platform, we observe growing demand for serverless T2I deployments driven by two workload characteristics. First, unlike conventional large language model (LLM) services exposed through standard APIs [69], T2I applications are highly customized. Professional creators compose unique workflows from diverse diffusion models and adapters (e.g., LoRAs [39]) based on their application needs; recent Alibaba production traces [55, 57] reveal 31,133 distinct workflows in a single 20-day period. Second, T2I demand is often ad hoc and bursty. Serverless deployment addresses these needs by letting users upload custom workflows while delegating * Equal contribution; # Corresponding author
1
Workflow DAG Representation. ServerlessT2I exposes a new serverless programming interface that defines a userplatform contract for customized T2I workflows. Through this interface, a T2I workflow is converted from a monolithic GPU function into an explicit DAG of model function invocations: users implement each model as a model function that encapsulates model loading and execution and declares its inputs and outputs; ServerlessT2I uses these declarations to infer the workflow DAG. This DAG gives ServerlessT2I the data dependency information needed to parallelize independent model functions and materialize intermediate tensors across GPUs, without requiring users to manage GPU placement or inter-GPU communication. The same abstraction also lets ServerlessT2I control model scaling and fine-grained resource accounting while preserving a familiar serverless programming style, allowing users to focus on the high-level application logic—a key benefit provided by serverless T2I deployment.
Iterative denoising Basic Workflow
Text encoders
attn Reference
attn
Prompt
attn
Diffusion model
Diffusion model attn attn
attn attn ControlNet
Decoder
Diffusion model LoRA
ControlNet with Parallelization
LoRA
Figure 1. Basic Workflow and Workflows augmented with ControlNet [112] and LoRA [39]. of GPU service each tenant receives, rather than by request counts. For each model function execution, ServerlessT2I measures the GPU time spent on model loading, computation, and tensor transfer, and charges this cost to the owning tenant as vTime. The scheduler prioritizes tenants with lower cumulative vTime, but strictly enforcing this order can rule out dispatch choices that would improve serving efficiency, e.g., reduce queuing delay. ServerlessT2I therefore uses a two-stage scheduler that bounds unfairness while preserving scheduling flexibility. In each scheduling round, the first stage admits only tenants whose cumulative vTime is within an operator-configured slack of the least-served tenant. The second stage ranks ready model functions from this eligible set using queuing delay, execution cost, and remaining workflow work. This design achieves fairness at model function granularity while allowing workload-aware decisions that improve efficiency. We prototyped ServerlessT2I and evaluated its decomposition, data plane, and scheduler under realistic workload variation. Our evaluation spans 20 representative T2I workflows, including SD3.5 [81], Z-Image [83], Flux [50, 51], and various adapters. Using production traces on a multi-GPU testbed, we compare ServerlessT2I against state-of-the-art serving systems [84, 85]. ServerlessT2I sustains up to 2× higher request rates at the same GPU budget, reduces GPU demand by up to 3× at a fixed rate, and meets up to 7× tighter SLOs while tolerating 2× higher burstiness. Microbenchmarks confirm that the data plane effectively hides loading and communication overheads, and that the scheduler successfully balances tenant fairness with throughput.
GPU-Resident Data Plane. Decomposing a T2I workflow into model functions, while enabling fine-grained scaling and resource accounting, puts model loading and inter-model data transfers on the critical path. ServerlessT2I introduces a unified GPU-resident data plane to absorb these costs. Our key insight is that T2I inference is typically compute-bound and leaves substantial GPU memory on the table: in our production cluster, the P95 GPU memory usage of a diffusion model is 35 GiB, only 36% of a modern NVIDIA H20 GPU. The system runtime can harvest this unused GPU memory for cached weights and inter-model tensor communication. ServerlessT2I introduces three mechanisms in the data plane to implement this insight (§5). First, because diffusionmodel loading is often much slower than one inference pass, ServerlessT2I caches the first few layers of each model and overlaps loading of the remaining layers with ongoing inference computation, using a lightweight profiling step to determine the number of layers needed to hide loading latency. Second, ServerlessT2I materializes inter-model dependencies through GPU-resident transfers and callbackbased data fetching, enabling efficient tensor exchange and flexible communication patterns while remaining transparent to users. Third, ServerlessT2I manages GPU memory as a unified logical address space shared by active inference, tensor communication, and cached weights. At its core is model weight virtualization, which decouples a model’s logical weights from their physical placement in GPU memory. This enables on-demand allocation of weights across disjoint memory regions. Also, ServerlessT2I treats weights as evictable, spilling them to host memory under pressure. Together, these mechanisms cut scale-out and communication overhead without exposing the runtime data path to users. Fair Scheduling. In view of the request backlogs during peak hours and the need for fair scheduling, ServerlessT2I defines fairness for multi-tenant T2I serving by the amount 2
2
Background and Motivation
2.1
A Primer on Text-to-Image Workflow
Basic Text-to-Image Workflows. As shown in Fig. 1top, a basic text-to-image generation workflow consists of three types of models: text encoder, base diffusion model, and decoder-only variational autoencoder (VAE). The process begins with the text encoder, which encodes a text prompt into a sequence of text embeddings. The system then initializes a latent tensor with random Gaussian noise. Conditioned on the text embeddings, the base diffusion model iteratively refines this tensor through a series of denoising steps. Finally,
2.2
1.0
Ours
0.5
0.0
2
4
6
8
Time (days)
10
12
14
0.5
0.0
10−2
100
102
104
Coefficient of Variation
Figure 2. Left: Invocations per hour, normalized to the peak. Right: CDF of CoV values across functions. whereas the Azure trace maintains a relatively stable baseline at roughly 60% of its peak, the T2I trace fluctuates over a much wider range. We further quantify workload variability using the coefficient of variation (CoV) of request inter-arrival times (IATs). A Poisson arrival process has an IAT CoV of 1, while values above 1 indicate burstier arrivals [77]. Fig. 2-right shows the distribution of IAT CoV values in our T2I trace and the Azure trace. Both workloads exhibit substantial request variability: 75% of functions in the T2I trace have an IAT CoV above 1, compared with 60% in Azure Functions. 2.3
Limitations of Current Practices
Develop and Deploy a T2I Workflow. Serverless T2I services are currently provided in the cloud [19, 42], where users first build a workflow with existing tools, such as ComfyUI [22] or Diffusers [85]. They then provision GPU resources, upload the entire workflow as a GPU function [20, 42], and deploy it in the cloud. After that, the platform follows the standard serverless computing paradigm: it manages a shared GPU cluster and automatically scales the GPU functions in response to request traffic. Users are billed on a pay-as-you-go basis by accounting GPU usage based on the GPU specification and execution duration [6, 18, 42, 54]. However, we identify following limitations, each of which undermines an essential property of serverless computing [74]. L1: High Scaling Overhead. Current serverless T2I platforms typically deploy each workflow as a single GPU function [19, 42]. This practice makes the workflow, rather than an individual model component, the unit of scaling. When load increases, the platform must therefore replicate the full workflow even if only one component is the actual bottleneck. This coarse granularity increases auto-scaling overhead* in both startup latency and GPU memory footprint. For example, scaling a basic Flux1-Schnell workflow on H800 with Diffusers [85] takes 1.1 seconds even when loading from pinned host memory, adding 110% overhead relative to model inference under default settings. It also consumes 42% more GPU memory than scaling only the base diffusion model, which is typically the bottleneck. We observe similar behavior in vLLM-Omni [84]. The root cause is common across existing systems [29, 52, 55, 76, 84, 85]: they reuse Diffusers’ design [85], which follows a “single-file” abstraction [25] and packages the entire generation workflow, including the base model, adapters, and control logic, as a monolithic unit.
Serverless T2I Workflows Serving in Production
T2I workflows are becoming a major serverless workload: in a one-month trace from our platform in March 2026, they account for 28% of total GPU usage, the largest share among GPU workloads. We identify two key reasons. Prevalence of Customized T2I Workflows. Unlike DNN and LLM services, which are typically exposed through standardized APIs [69], T2I applications are often built as customized workflows, especially by professional creators with specialized visual requirements. These workflows span diverse applications, such as virtual try-on [17] and image editing [44]. Recent Alibaba production traces [55, 57] show this diversity at scale, with 31,133 distinct workflows recorded over 20 days; we observe similar trends on our platform. In addition to aesthetic customization, users may incorporate T2I-specific parallelization techniques to accelerate workflow execution, such as ControlNet parallelization [55] and sequence parallelism [29, 52, 55]. These techniques exploit the data dependencies within T2I workflows to parallelize adapter execution and base model inference. For example, in Fig. 1-bottom, ControlNet produces intermediate results that must be transferred to specific layers of the base model at each denoising step; otherwise, the base model stalls waiting for unavailable inputs. Exploiting this parallelism therefore requires specialized system support [55, 82]. Dynamic Workload Traffic. Fig. 2-left plots the request volume per hour in our production cluster, normalized by the peak hourly load observed from March 1 to March 14, 2026. We include the Azure Functions trace [77] as a comparison baseline. Our T2I workload is visibly more bursty:
Azure Function
1.0
CDF
Relative Invocations
the denoised latent representation is passed to the VAE decoder, which reconstructs the output image in pixel space. Adapter-augmented Workflows. Production T2I workflows often augment the basic pipeline with adapter models to provide fine-grained control over visual attributes such as spatial structure, artistic style, and illumination [39, 55, 99, 111–113]. From a systems perspective, two classes of adapters are most relevant, as shown in Fig. 1-bottom. First, tandem adapters, such as ControlNet [112], execute alongside the base diffusion model at each denoising step and inject spatial conditioning signals such as edges or depth maps. These adapters complicate serving because their parameter sizes are often comparable to the base model, which increases both model-loading latency and inference latency; moreover, maximizing throughput often requires parallelizing the adapter and base model across GPUs, introducing nontrivial synchronization and data communication [55]. Second, weight-update adapters, such as LoRA [39], modify the base model’s weights before inference. While they do not invoke an additional model, they introduce loading overhead: as users may rely on many such adapters, they are typically fetched and applied before inference on demand [55, 57].
3
* Like [103], we exclude the delay of fetching a remote container image for
cold starts, which can take extra seconds to minutes to complete.
1.0
1.0
0.5 0.0
CDF
CDF
H200 H20
0
Model Workflow 5 10 15 0 Mean Latency (s)
H800
Motivation and System Overview
3.1
Key Insight
1 TiB
H800
0
50
100
0
100
0
1000
GPU Mem. Usage (GiB) Host Mem. Usage (GiB)
Figure 4. Resource utilization. Recent Alibaba trace [57] profiles GPU memory but omits SM util. and host memory. LLM serving, which reserves substantial GPU memory for runtime state such as the KV cache needed to serve a large batch of requests [49], T2I inference maintains much less runtime state and therefore leaves a substantial fraction of GPU memory unused. For example, in Flux workflows, runtime state accounts for only 7.6% of the total model memory footprint. Fig. 3-right further shows that both end-to-end workflows and individual diffusion models use only a small fraction of the memory available on a modern GPU. Meanwhile, we observe the same pattern in our production workloads. As shown in Fig. 4, GPU SM utilization during T2I function execution is high, with the P50 reaching 100%. In contrast, GPU memory usage is much lower: the P50 is only 16 GiB and the P95 is 35 GiB. Since modern GPUs provide 80–141 GiB of memory, a large fraction of GPU memory remains idle in production. Another production trace from Alibaba reports a similar pattern, with P50 and P95 memory usage of 30 GiB and 36 GiB, respectively [57]. We further observe that host memory is significantly underutilized on these servers, motivating the design described in §5.1. How does it work in ServerlessT2I? ServerlessT2I turns this slack memory into a backend data plane for serverless T2I serving (§5). At a high level, the data plane serves two roles. First, it caches model weights in otherwise idle GPU memory, reducing model loading overhead for fast scaling. Second, because ServerlessT2I decomposes a T2I workflow into a DAG of model functions (§4), the data plane provides GPU-resident buffers for fast data movement along DAG edges. This keeps critical-path inter-model communication efficient. As a benefit of DAG execution, the exposed function boundaries also enable fine-grained resource accounting for fair scheduling (§6). 3.2
Compute-bound T2I leaves harvestable GPU memory. T2I inference is typically compute-bound: even an inference batch of one can saturate a high-end GPU [52, 55, 57]. Unlike * Qwen-Image, Z-Image, Z-Image-Turbo, Flux1-Schnell, Flux1-Dev, SD3,
SD3.5, and SDXL with default settings in [85], e.g., resolutions (1024×1024).
1.5 TiB
H20
SM Util. (%)
Figure 3. CDF of latency and GPU memory usage of eight workflows* on H800 at model level and workflow level. L2: Exposed Communication Complexity. High performance T2I execution often relies on multi-GPU parallelization, but existing systems force serverless users to handle the resulting communication complexity when composing workflows. In existing systems [29, 52, 55], the communication logic is tightly coupled to framework internals. Adapting these techniques to a user’s customized workflow therefore requires substantial systems expertise and engineering [29, 52, 55, 84]. In practice, users must reason directly about GPU placement, synchronization, and data movement at runtime [35]. This requirement exposes lowlevel resource management during workflow development, conflicting with the serverless principle of hiding infrastructure from users [74]. For example, combining ControlNet parallelization with sequence parallelism requires users to understand framework internals, tensor sharding, and lowlevel distributed communication [82], although these details should be hidden behind a serverless abstraction [45]. L3: Limited Support for Multi-tenant Serving. In a serverless platform, multiple tenants share a GPU pool to execute their workflows. However, GPUs are scarce resources, so bursty demand can quickly create request backlogs during peak periods; in our production cluster, tens of thousands of requests can queue, and major users can see up to 10% of requests backlogged. Backlog makes fairness a scheduling requirement: one tenant should not consume a disproportionate share of GPU service and delay others. Existing per-tenant quotas, such as request-per-minute (RPM) limits [26, 72], provide isolation but are not work-conserving: they can throttle a tenant even when GPUs are idle. Worse, quotas account for requests rather than GPU consumption, which mismatches T2I workflows whose costs vary widely: as shown in Fig. 3-left, the inference latency of eight popular workflows* spans up to 16×. While T2I serving systems [29, 55, 76, 84, 85] can be efficient in single-instance deployments, they largely lack scheduling mechanisms that ensure both fairness and efficiency in a multi-tenant cloud.
3
H200
0.5 0.0
50 80 96 141 Peak GPU Allocated (GiB)
Ours Alibaba
4
System Overview
ServerlessT2I treats a T2I workflow as a model DAG: individual models are exposed as functions, and workflow execution is a sequence of function invocations with data dependencies. System Architecture. Fig. 5 shows the architecture of ServerlessT2I. At the frontend, users implement model functions, compose them into workflows, and register the workflows with the system (○). 1 ServerlessT2I provides a serverless-style programming interface with T2I-specific abstractions for workflow composition, hiding the complex data movement required by DAG execution from users and addressing L2. After registration, users invoke a workflow
Workflow Composition Fn
Control Plane (Scheduler)
Fn
Workflow Registry
Fn Fn
Fn
Fn
Request ID: ... Prompt: “A coffee …” Seed: 0
vTime Tracker
Fn Fn Fn Fn
1 # # Developers start here ## 2 def setup_io ( self ) -> None : 3 # define inputs 4 add_input ( " latents " )
Queue
5 6
Unified GPU
Unified GPU Runtime States Cached Model Layers Comm. Buffer Data Plane Memory Address Runtime States Cached Model Layers Comm. Buffer on Executors Memory Address Host Memory Host Memory
with inputs such as text prompts through OpenAI-compatible APIs [69] (○). 2 At the backend, ServerlessT2I maintains workflow state and uses a fair scheduler to dispatch ready function executions across distributed GPU executors (L3). Each executor manages its GPU memory as a unified address space for active model inference runtime, weight caching for fast scaling (L1), and efficient communication between dependent model functions (○). 5 Host memory acts as a secondary storage tier (§5.1). Life of a Request. When a request arrives, the control plane instantiates the corresponding workflow DAG (○) 3 and tracks the readiness of each model function. A vTime tracker accounts for each tenant’s resource share to support fair scheduling, as described in §6. The scheduler selects ready functions, i.e., functions whose inputs are available, and dispatches them to GPU executors (○). 4 After executing a function, the executor reports completion to the control plane and exposes the output through the data plane. The control plane then releases downstream functions once their inputs become ready; this process repeats until the workflow output is returned to the user.
4
7 8 9
Cached Model Layers Cached Model Layers
Figure 5. System Overview.
10 def load_model ( model_path ) : 11 transformer = FluxTransformer2DModel . from_pretrained ( 12 model_path , torch_dtype = torch . bfloat16 , 13 14
) return { " transformer " : transformer }
15 16 @torch . no_grad () 17 def execute ( model_components , ** kwargs ) : 18 19 20
transformer = model_components [ " transformer " ] noise_pred = transformer . forward (** kwargs ) return { " noise_pred " : noise_pred }
21 22 # # Invisible from developers ## 23 class ModelFn : 24 setup_io = setup_io 25 initialize = load_model 26
execute = execute
Figure 6. A simplified implementation of Flux function. implementation code from Diffusers [85] when implementing these methods, which also preserves compatibility with inference optimizations such as torch.compile(). Under the hood, ServerlessT2I organizes the three methods under a ModelFn class, the scaling and management unit at runtime. Assemble Models as a Workflow. ServerlessT2I uses an orchestrator function to assemble models as a workflow, similar to the design of Azure Durable function [9]. ServerlessT2I adopts a declarative workflow programming model. The I/O interfaces declared by setup_io in each ModelFn are sufficient for ServerlessT2I to parse the data dependencies and apply topological sorting to infer the workflow DAG. For example, in Fig. 7, prompt_embeds is the output of text_enc, which serves as the input for controlnet and flux, meaning both of them depend on the output of text_enc for inference. At runtime, ServerlessT2I materializes the data communication while hiding the implementation details from users, as elaborated in §5.2. Function-as-a-Service Paradigm. In ServerlessT2I, ModelFn corresponds to the notion of a function in the Function-asa-Service paradigm. It abstracts model components in a T2I workflow as small, single-purpose functions that are exposed to the serverless platform, which scales them on demand. ModelFn is also stateless and with random seeds fixed, the same method inputs always produce the same outputs, allowing safe retries for fault tolerance.
Programming Interface
Design Objectives. First, ServerlessT2I’s programming interfaces resemble those provided by commercial serverless platforms [8, 10, 34], which reduces the additional learning curve for users. Second, ServerlessT2I abstracts complex data communication behind intuitive APIs, making data communication transparent to users and hiding the underlying infrastructure as well as the complexity of operating it [74]. With ServerlessT2I, individual models are implemented as functions and users can use an orchestrator function to assemble them into a workflow in a declarative manner. ServerlessT2I’s runtime handles execution and communication in the shared resource pool while respecting the workflow’s data dependencies. Model Function Development. Fig. 6 illustrates how to implement a Flux1-Dev model in ServerlessT2I. Developers implement three methods: setup_io(), load_model(), and execute(). For setup_io(), ServerlessT2I provides two primitives, add_input and add_output, to specify the model’s input and output interfaces. The other two methods encapsulate model loading and inference, respectively. As in prior systems [29, 55, 76, 84], users can reuse model
add_input ( " prompt_embeds " ) add_input ( " control_outputs " , callback = True ) # define outputs add_output ( " noise_pred " )
5
5
Data Plane
ServerlessT2I’s data plane builds on the insight in §3.1: harvested GPU memory can reduce two costs on the critical path, model loading and data movement across functions.
GPU Memory
1 def compose_workflow () : 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
tensors Layer 0 Infer.
Compute Stream
01234567
Load Stream
5
6
7
metadata Control Plane
Executor 0 (ControlNet)
nvshmem Layer 0 Infer. Callback Fires
Executor 1 (Flux)
of model loads from host memory, as detailed in Appendix A. Accordingly, we focus on host to GPU loading. The key idea is to split each model across the two tiers: cache its first layers in GPU memory and keep the remaining layers in host memory, as shown in Fig. 8-left. Consider a model with 𝐿 uniform layers. ServerlessT2I preloads the first 𝐿𝑒 layers into GPU memory and keeps the remaining 𝐿 − 𝐿𝑒 layers in host memory. When a request arrives, inference starts immediately on the cached first layers, while loading the remaining layers from host memory asynchronously. Let 𝑇𝑙𝑜𝑎𝑑 denote the loading latency of one layer and 𝑇𝑐𝑜𝑚𝑝 denote its inference latency. Because 𝑇𝑙𝑜𝑎𝑑 ≫ 𝑇𝑐𝑜𝑚𝑝 , the end-to-end latency 𝑇𝑒2𝑒 as a function of 𝐿𝑒 is
We describe how to reduce model loading overhead (§5.1) and provide efficient data communication (§5.2), then explain how a unified logical address space manages GPU memory (§5.3). Model Loading
Challenge: Model loading remains on the critical path. Decomposing a T2I workflow into model functions gives the platform control over each model, but it does not remove model loading during scale out. As workflows have large models along the critical path, loading remains a bottleneck. Prior systems reduce loading overhead by overlapping model loading with execution in two ways. One approach transfers later layers of a model while its earlier layers are executing [11, 44, 53, 103]. Another approach loads later stage models in a workflow while earlier stage models are executing [59]. Both approaches require enough execution time to hide loading, which often does not hold for T2I workflows. For example, in a basic Flux1-Dev workflow, loading text_encoder, text_encoder_2, transformer, and the VAE decoder from pinned host memory to an NVIDIA H800 GPU takes 5×, 13.5×, 3.2×, and 6.3× their respective inference latencies. Even with ideal overlap, layer by layer loading reduces Flux1-Dev end-to-end latency by only 13%. Prewarming later stage models also leaves a large bottleneck: even with weights cached in host memory, model loading still accounts for 44% of end-to-end latency. Prewarming further interferes with multi-tenant scheduling by injecting bursts of loading traffic that may delay other tenants. Caching the first layers. ServerlessT2I reduces model loading overhead with a multi tier cache spanning GPU memory, host memory, and external storage. GPU memory is the fastest tier, caching model weights for immediate execution. Host memory is the second tier, holding weights that can be loaded into GPU memory on demand. External storage is the last tier with nearly unlimited capacity but much lower bandwidth. Consistent with prior observations [55, 57], workflow popularity is skewed. In our trace replay, caching the 15 most popular workflows in underutilized host memory serves 99%
567
Figure 8. Left: A simplified illustration of ServerlessT2I’s loading mechanism. Right: An example of data fetching.
Figure 7. A simplified Flux workflow composition.
5.1
Host Memory
Placement 0 1 2 3 4
# create model function instances text_enc = ModelFn ( model_path = model_path ) flux = ModelFn ( model_path = model_path ) controlnet = ModelFn ( model_path = controlnet_path ) vae = ModelFn ( model_path = model_path ) ... # connect function instances latents = random_latents_generator ( seed ) prompt_embeds = text_enc ( prompt ) for i in range ( num_steps ) : # iterative denoising control_outputs = controlnet ( latents , prompt_embeds ) noise_pred = flux ( latents , prompt_embeds , control_outputs ) latents = denoise ( noise_pred , latents ) output_img = vae ( latents , mode = " decode " ) ...
𝑇𝑒2𝑒 (𝐿𝑒 ) = max(𝐿𝑇𝑐𝑜𝑚𝑝 , (𝐿 − 𝐿𝑒 )𝑇𝑙𝑜𝑎𝑑 + 𝑇𝑐𝑜𝑚𝑝 ).
(1)
Eq. 1 gives the minimum prefix length 𝐿𝑒′ required to fully
hide model loading: 𝐿 𝑇load − 𝑇comp + 𝑇comp 𝐿𝑒′ = . (2) 𝑇load Computing 𝐿𝑒′ requires only two profiled quantities: the per-layer loading latency 𝑇𝑙𝑜𝑎𝑑 and the per-layer inference latency 𝑇𝑐𝑜𝑚𝑝 . For diffusion models and their ControlNets, profiling is lightweight because they are typically composed of uniform transformer blocks with similar costs. Eq. 1 and Eq. 2 can be extended to models with non-uniform layers. 5.2
6
Communication
Challenge: Model DAG execution requires efficient and flexible tensor communication. Once a workflow is decomposed into functions, intermediate tensors must be communicated across functions. This poses two challenges. First, each workflow execution transfers a large volume of data— on the order of GiBs [55]—and 99% of the transferred objects are tensors. Second, specialized parallelization techniques introduce complex communication patterns in which both correctness and performance depend on consuming data at the right time. For example, if computation blocks while waiting for data to be produced, the system may lose much of the performance benefit from parallel execution (§2). Existing serverless data planes are a poor fit for T2I workflow execution. Data planes that rely on host memory [102] perform poorly for CUDA tensors: the communication latency for an SD3 ModelFn is 28× its execution latency due to PCIe transfers, serialization, and socket overhead. Recent work builds data planes atop high-speed interconnects such
as NVLink and RDMA [92, 104], but exposing these links through collective-style APIs introduces multiple synchronization points to preserve correctness. In our measurements, these synchronization points add up to 20% inference latency when running Flux1-Schnell [51], a computation-intensive model, on two NVIDIA H800 GPUs, in both the basic workflow and the ControlNet-augmented workflow. Efficient and flexible data fetching. ServerlessT2I combines NVSHMEM [65] with callback-based fetching to provide efficient and flexible GPU communication. NVSHMEM is a natural substrate for ServerlessT2I because it provides one-sided GPU communication over NVLink and RDMA, supports GPU-initiated transfers, and exposes a symmetric heap abstraction that avoids explicit remote-address management. In ServerlessT2I, each executor reserves a fixedsize NVSHMEM arena on its GPU using nvshmem_malloc[7] and manages the resulting symmetric heap with a buddy allocator[47, 48], returning device pointers usable by both local CUDA kernels and remote NVSHMEM operations. On top of this substrate, ServerlessT2I allows users to mark a ModelFn input as a callback when the input is needed only partway through execution. For example, line 6 in Fig. 6 declares control_outputs with callback=True to support ControlNet parallelization (§2.1). The runtime wraps this input and fires the callback when downstream models consume it, overlapping communication with earlier computation. Compared with manually managed peer-memory access [117], this design better matches T2I workflows, where intermediate tensors are produced and consumed at finegrained, model-dependent points during inference. At runtime, a producer writes its output tensor into local NVSHMEM and publishes the tensor metadata, such as its pointer and shape, to downstream consumers. When a consumer first needs the tensor, ServerlessT2I allocates a destination block from the consumer’s local NVSHMEM arena and pulls the tensor bytes directly from the producer’s remote address using a one-sided NVSHMEM operation. The data path stays in GPU memory, avoiding PCIe round trips and socket communication. The callback abstraction controls when a transfer occurs: the fetch is issued only when execution reaches the program point where the tensor is needed. Fig. 8-right presents an example of data fetching in ControlNet parallelization, where ControlNet execution is interleaved with the base Flux model, as shown in Fig. 1-bottom. At runtime, the output of ControlNet layer 0 is consumed partway through Flux layer 0. Rather than blocking until this output is available, Executor 1 begins executing Flux layer 0 and registers a fetch callback. When Flux reaches this point, the callback fires. By then, Executor 0 has produced the ControlNet layer 0 output and placed it in its NVSHMEM communication buffer (○); 1 the corresponding tensor metadata is forwarded to Executor 1 (○), 2 which uses it to issue a one-sided NVSHMEM fetch into its local tensor store (○) 3
before Flux consumes the tensor (○). 4 This callback-based fetch path allows Flux to overlap its computation with ControlNet execution. Without it, Flux would have to wait until the relevant ControlNet output was materialized, eliminating the parallelism between the two models. Note that tensor metadata is tiny, on the order of KiB, and executors piggyback it on completion notifications, allowing the control plane to track tensors with little overhead.
7
5.3 Unified GPU Memory Management The shared use of GPU memory requires each executor to coordinate three memory consumers within its local GPU: active inference state, communication buffers (§5.2), and model weights (§5.1). Active inference state, such as inputs and intermediate activations, is framework-specific and allocated through the deep learning framework runtime, i.e., PyTorch in ServerlessT2I. In contrast, communication buffers are allocated and managed by NVSHMEM. As a result, local GPU memory is divided into two disjoint allocation domains, which we refer to as the PyTorch region and the NVSHMEM region. ServerlessT2I introduces model weight virtualization, a software indirection layer that decouples a model’s logical weights from their physical placement in GPU memory. Similar to how virtual memory in operating systems decouples a process’s logical address space from physical memory frames, ServerlessT2I decouples a model’s logical layers from the physical GPU memory that stores their weights. Model weights can transparently reside in either the PyTorch region or the NVSHMEM region, while model execution accesses them through the same logical interface. This indirection bridges the two otherwise isolated regions into a unified pool for cached weights, exploiting the underutilized GPU memory (§3.1). How does it work? We walk through an example in Fig. 9left, to show how ServerlessT2I executes model inference and manages GPU memory. From the perspective of model execution, the model still consists of a conventional sequence of logical layers, as seen by the PyTorch forward() pass. Before executing each layer, ServerlessT2I consults a Layer Lookup Table to translate the logical layer into the corresponding tensor objects that represent its weights. Each tensor object contains the necessary metadata, such as shape and data type, together with a data pointer to the underlying physical GPU memory. This physical memory may be allocated from either the PyTorch region or the NVSHMEM region. The CUDA kernels invoked by the framework then execute normally, using the data pointers of both active inference state and model weights to access the appropriate physical memory. Weights allocated by PyTorch are ordinary tensor objects and can be used directly, whereas weights stored in the NVSHMEM region require pointer rebinding. For each
A Tensors for Active Inference W Model Layer Weights C Tensors for Communication Logical Layers
Layer Lookup Table
CUDA Kernel Execution Physical Memory
Layer Region
Tensor Object
L0
L0
PyTorch
<ptr_0, …>
L0_infer (ptr_0, …)
L1
L1
NVSHMEM <ptr_1,…>
L1_infer (ptr_1, …)
A W W W W W C
PyTorch Region NVSHMEM Region
Figure 9. Left: An example of model weight virtualization. Right: Difference of service received for two backlogged tenants. NVSHMEM allocation that holds a layer’s weights, ServerlessT2I wraps the device pointer as a CUDA tensor using PyTorch C++’s torch::from_blob, creating a tensor view over the NVSHMEM-backed memory. ServerlessT2I then rebinds the model’s parameter entries to these tensors and records them in the Tensor Object entries of the Layer Lookup Table. Consequently, forward() calls access NVSHMEM-resident weights as ordinary PyTorch tensors, and CUDA kernels treat them identically to tensors allocated by PyTorch’s CUDA allocator. ServerlessT2I tracks model weights at layer granularity and treats cached weights as elastic state. When GPU memory pressure arises, for example because a new model must be loaded from host memory, ServerlessT2I evicts cached weights layer by layer until the demand is satisfied. An executor orders its GPU-resident models by recency and reclaims weights from the least recently used (LRU) models first, evicting layers in reverse layer order to preserve first layers that enable overlapping loading with inference. This eviction policy trades a modest latency increase for reclaimed memory while striving to preserve cached weights for fast scale-out. Can NVSHMEM region be elastic? The NVSHMEM region size should be specified at initialization [66], and resizing it at runtime is impractical because NVSHMEM maintains a symmetric heap with identical size and layout across all GPUs, making any adjustment require cluster-wide coordination [67]. This creates a fundamental tension: an undersized NVSHMEM region risks deadlock when in-flight operations cannot allocate space for data communication, stalling the entire system; an oversized region, however, squeezes the space available for cached weights without model weight virtualization, which we quantitatively analyze in §8.3. Backend-as-a-Service Paradigm. ServerlessT2I’s data plane follows the Backend-as-a-Service paradigm in serverless computing [74]. It exposes model caching and data communication as managed backend services, allowing users to benefit from them without building or operating these components themselves. In T2I workflow execution, all intermediate data is immutable: intermediate tensors produced are consumed once and never updated [55, 85], which obviates consistency protocols and simplifies fault tolerance. The data plane reclaims tensors when no downstream ModelFn
requires them. We use expandable_segment [64] in the PyTorch region and our buddy allocator in the NVSHMEM region to mitigate memory fragmentation. If an executor fails, ServerlessT2I reconstructs lost data by re-executing the affected ModelFns and loading model weights, following a similar approach to prior cluster computing frameworks [63, 102, 106].
6
8
Ensure Fairness in Multi-tenant Serving
Resource fairness is critical in multi-tenant serving systems because GPUs are scarce cloud resources, and tenant requests can remain backlogged during peak demand [79]. Yet existing T2I serving systems provide limited support for resource-aware scheduling [2, 29, 52, 55, 58, 76, 84]. This gap is especially problematic for T2I workloads: because workflows differ substantially in GPU demand (Fig. 3-left), two backlogged tenants that submit requests at the same rate may still receive very different amounts of service. We illustrate this effect by augmenting Diffusers [85], a representative T2I serving system, with three scheduling policies: FIFO, SJF, and an adapted version of VTC [79], a fairness-oriented scheduler originally designed for LLM serving. We run the experiment on a four-H800 testbed with two tenants, each issuing requests at 2 RPS. One tenant invokes a basic SD3-medium [28] workflow, while the other invokes a basic Flux1-Dev [51] workflow. We define the service received by a tenant as its cumulative GPU time. Fig. 9-right reports the service difference between the two tenants over the interval in which both remain backlogged. None of the policies provides satisfactory fairness. FIFO accounts for requests rather than service: despite identical request rates, the Flux1-Dev tenant receives substantially more GPU time because each Flux1-Dev request is much more expensive, with 10× the inference latency of SD3-medium. SJF exhibits the opposite bias, giving more service to the SD3-medium tenant because it consistently favors shorter requests. Adapting VTC reduces the imbalance by counting T2I workflow operations, analogous to its use of decoding steps in LLM serving [79], but operation counts remain an inaccurate proxy for GPU service because T2I models differ widely in per-operation cost (Fig. 3-left). Fine-grained Fairness Accounting with vTime. ServerlessT2I accounts for service at the granularity of each ModelFn
execution. Whenever a ModelFn runs on a GPU, ServerlessT2I charges its owner a vTime equal to the wall-clock GPU time consumed, including computation, model loading, and tensor transfer. At dispatch time, the scheduler charges an estimated vTime derived from historical measurements so that scheduling decisions can proceed immediately; once execution completes, the estimate is replaced with the measured GPU time, keeping accounting faithful to realized usage. If a denoising step launches parallel ModelFns (e.g., a base model and a ControlNet), the tenant is charged for both. Fairness Scheduling with vTime. ServerlessT2I tracks the cumulative vTime of each tenant and prioritizes those that have received less service. For a tenant that newly enters the system or returns after inactivity, ServerlessT2I performs a vTime lift, initializing its vTime to the minimum among active tenants. Without this lift, a returning tenant’s stale, artificially low vTime would grant it repeated priority until it catches up, converting past absence into a scheduling advantage. To balance fairness and serving efficiency, ServerlessT2I allows a configurable slack so that the scheduler retains scheduling flexibility. This is realized as a two-layer scheduler. Layer 1: Fairness filter. The scheduler maintains cumulative vTime 𝑣𝑢 for each tenant 𝑢. At dispatch time, it forms a candidate set 𝐶 of tenants: a tenant 𝑢 is included if 𝑣𝑢 − min𝑢 ′ 𝑣𝑢 ′ ≤ Δ, where Δ is an operator-configured slack. Only ModelFns belonging to tenants in 𝐶 proceed to Layer 2. A smaller Δ enforces stricter fairness, while a larger Δ gives Layer 2 a wider pool of candidates to optimize throughput. Layer 2: Priority scoring. Strict fairness alone can hurt serving efficiency. For example, the scheduler may interleave the execution of two workflows to ensure fair resource allocation, increasing the latency of both. To improve serving efficiency, among the eligible ModelFns, Layer 2 ranks each ModelFn 𝑛 by a normalized priority score score(𝑛) = 𝑤ˆ (𝑛) − 𝑟ˆ (𝑛) − 𝑒ˆ (𝑛), where 𝑤ˆ (𝑛), 𝑟ˆ (𝑛), and 𝑒ˆ (𝑛) denote its waiting time, the remaining critical-path time of its workflow, and its execution latency, respectively. The scheduler dispatches the highest-scoring ModelFn. The terms −ˆ𝑟 (𝑛) and −𝑒ˆ (𝑛) favor ModelFns with shorter remaining work or lower execution latency. The term 𝑤ˆ (𝑛) prevents starvation. The selected ModelFn is dispatched to an executor that can run it more efficiently, e.g., one with its model already cached to reduce loading overhead or its inputs available in the local GPU memory to reduce communication overhead. Algorithm 1 summarizes the scheduling process, including vTime lift, candidate-set construction, priority scoring, and vTime correction after execution. Fairness bound. Since ServerlessT2I serves DAG-structured workflows, a tenant may have outstanding requests but no schedulable work when all pending ModelFns are blocked on predecessors. We thus call a tenant eligible-backlogged over [𝑡 1, 𝑡 2 ) if it has at least one ready ModelFn at every point in 9
Algorithm 1: Scheduling Algorithm Input: Per-tenant cumulative vTime 𝑣𝑢 ; fairness slack Δ Function Schedule()// invoked when an executor is idle: // vTime lift for new/returning tenants 2 foreach tenant 𝑢 with ready ModelFns do 3 if 𝑢 is new or returning then 𝑣𝑢 ← min𝑢 ′ ∈Active 𝑣𝑢 ′
1
4 5 6
7 8
9 10 11 12 13
// Layer 1: fairness filter 𝑅 ← tenants with ready ModelFns 𝑣min ← min𝑢 ∈𝑅 𝑣𝑢 𝐶 ← {𝑢 ∈ 𝑅 | 𝑣𝑢 − 𝑣min ≤ Δ} // Layer 2: priority scoring 𝑁 ← ready ModelFns owned by tenants in 𝐶 𝑛★ ← arg max𝑛∈𝑁 𝑤ˆ (𝑛) − 𝑟ˆ (𝑛) − 𝑒ˆ (𝑛) // Dispatch and charge estimated vTime 𝑇ˆ𝑛★ ← estimated execution time of 𝑛★ 𝑣owner(𝑛★ ) += 𝑇ˆ𝑛★ Dispatch 𝑛★ to best-fit executor
Function OnComplete(𝑛, 𝑇 ) // 𝑇 : measured GPU time of 𝑛: 𝑣owner(𝑛) += 𝑇 − 𝑇ˆ𝑛 // correct vTime
the interval. This refines the backlogged condition in VTC for LLM serving [79], which only requires a queued request. This distinction is necessary for DAGs, where dependencies can leave outstanding work with no ready computation. Let 𝐿max denote the largest vTime charge of any single dispatch, and define 𝑈 = Δ + 𝐿max . Consider any two tenants 𝑓 and 𝑔 that remain eligible-backlogged during [𝑡 1, 𝑡 2 ). For tenant 𝑢, let 𝑊𝑢 (𝑡 1, 𝑡 2 ) = 𝑣𝑢 (𝑡 2 ) − 𝑣𝑢 (𝑡 1 ) denote the service it receives during this interval. Then ServerlessT2I guarantees |𝑊 𝑓 (𝑡 1, 𝑡 2 ) − 𝑊𝑔 (𝑡 1, 𝑡 2 )| ≤ |𝑣 𝑓 (𝑡 2 ) − 𝑣𝑔 (𝑡 2 )| + |𝑣 𝑓 (𝑡 1 ) − 𝑣𝑔 (𝑡 1 )| ≤ 2(Δ + 𝐿max ) = 2𝑈 .
7
Implementation
We have implemented ServerlessT2I with a FastAPI [30] frontend, which exposes a programming interface for users to compose and register T2I workflows (§4). Users invoke their workflows with image generation parameters, such as prompts and reference images, similar to the OpenAI API [69]. ServerlessT2I’s backend runtime consists of a control plane and distributed executors (Fig. 5), totaling 4,000 lines of Python code. The data plane is implemented in 3,000 lines of Python and C++/CUDA code, built on PyTorch and NVSHMEM [65]. Aside from CUDA tensors, communication between the scheduler and distributed executors is facilitated via ZeroMQ [108].
8
Evaluation
We evaluate ServerlessT2I with the following highlights: • ServerlessT2I outperforms state-of-the-art baselines in controlled end-to-end evaluation, sustaining up to 2× higher request rates, satisfying 7× more stringent SLOs, reducing GPU requirements by up to 3×, or tolerating 2× higher
burst traffic, all while maintaining 90% SLO attainment (§8.2). • Our microbenchmarks isolate the benefits of ServerlessT2I’s designs: it reduces model loading overhead, enables efficient parallelization with minimal effort, and explores a tradeoff between fairness and efficiency. (§8.3 & §8.4). • ServerlessT2I’s DAG execution adds negligible overhead (§8.5).
8.1
Setup
Testbed and Workloads. By default, we use a testbed of 32 NVIDIA H800 GPUs and a scaled real-world T2I production trace collected from our production cluster (Fig. 2-left). To evaluate under diverse conditions, we vary request rates, SLO targets, traffic burstiness, and testbed sizes, covering a broad range of traffic patterns and performance requirements. Metrics. Our primary metric is SLO attainment: the fraction of requests completed within their specified latency deadline. We set the default deadline to 3× the solo inference latency of each workflow (SLO Scale =3). Unlike prior works [2, 29, 52, 55], ServerlessT2I does not alter the computation of T2I inference and we have validated the identity of generated images. Baselines. We primarily compare ServerlessT2I with vLLMOmni and Diffusers , which are representative state-ofthe-art T2I serving systems [24, 84, 85]. Following current practices, we deploy each workflow as a monolithic GPU function (§2.2). Since these systems were originally designed as standalone inference frameworks, we adapt them to the serverless setting and evaluate three deployment variants: • NoCache (No caching) executes each request to a workflow in a GPU function without any caching on GPU. • WCache (Workflow cache) utilizes slack GPU memory to cache entire workflows. A GPU function is terminated when the cached workflow is evicted. • MCache (Model cache) utilizes slack GPU memory to cache individual models within workflows. Because caching is performed at a finer model granularity, this variant can accommodate more workflows in GPU memory. Note that WCache and MCache are augmented versions of NoCache that utilize slack GPU memory and use the same LRU caching policy as ServerlessT2I for a fair comparison. Workflows and Settings. We compose T2I workflows using six popular base models: SD3.5-Large [81], Z-Image [83], Z-Image-Turbo [83], Flux1-Dev [51], Flux1-Schnell [51], and Flux2-Klein [50]. They exhibit diverse computational characteristics, with parameter counts spanning 6B to 12B and denoising steps ranging from 4 to 50. We set up two settings, as detailed in Table 1, randomly assigning workflows to the top-tier tenant traffic from our production trace. In S1, baselines are based on vLLM-Omni, 10
Table 1. Evaluation settings: S1 includes basic workflows, where each workflow consists of text encoders, a diffusion model, and a decoder. S2 further includes adapter-augmented workflows, which extend the basic workflows with different ControlNet and LoRA adapters. Setting
Diffusion Models
No. Workflows
S1
SD3.5-Large, Z-Image, Z-Image-Turbo Flux1-Dev, Flux2-Klein
5
S2
SD3.5-Large, Z-Image, Z-Image-Turbo Flux1-Dev, Flux1-Schnell, Flux2-Klein
20
as it provides little support for adapters [84] and does not support Flux1-Schnell. In S2, baselines use Diffusers. 8.2
End-to-end Evaluation
As Fig. 10 and Fig. 11 show, ServerlessT2I consistently achieves higher SLO attainment than the baselines in S1 and S2. We use a controlled evaluation methodology: each experiment varies one workload or deployment factor while holding the others fixed. Overall, ServerlessT2I sustains up to 2× higher request rate, satisfying up to 7× stringent SLOs, saving up to 3× GPU resources, and tolerating 2× higher burst traffic, all while maintaining SLO attainment for 90% of the requests. These gains are partly driven by ServerlessT2I’s model-loading design: data-plane weight caching accelerates 99%/86% of model loads in S1/S2, leaving only 1%/14% to require full model loading. Among the baselines, MCache and WCache consistently outperform NoCache, confirming ServerlessT2I’s insight that fine-grained use of idle GPU memory improves serving efficiency. SLO Attainment vs. Rate Scale. We first vary the request rate scale while fixing the SLO scale, testbed size, and traffic burstiness as the default values. Fig. 10(a) and Fig. 11(a) show that ServerlessT2I preserves high SLO attainment over a wider load range than all baselines. In S1, ServerlessT2I maintains 100% attainment up to a rate scale of 1.5. By contrast, the strongest baseline, MCache, falls below 90% once the rate scale reaches 1.0, and NoCache drops to only 10% at rate scale 2.0. In S2, ServerlessT2I again remains above 90% through rate scale 1.5, whereas MCache reaches 85% at the same load and the other baselines are below 50%. These results show that whole-workflow caching alone is insufficient under increasing load: even when some requests hit in cache, monolithic deployments still pay high loading and scaling costs when the active working set changes. ServerlessT2I avoids this cliff by decomposing workflows and reusing GPU-resident model state across requests. SLO attainment vs. SLO Scale. We next vary the SLO scales while fixing the rate scale and testbed size, using the scaled original production trace. Fig. 10(b) shows that ServerlessT2I meets substantially tighter SLOs in S1: at an SLO scale of 3.0, ServerlessT2I completes all requests within deadline, whereas WCache and MCache require scale 24.0 to exceed 90% attainment, and NoCache never reaches 90%
100
50
50
0
0.5
1.0
1.5
(a) Rate Scale
2.0
0
WCache (vLLM-Omni)
MCache (vLLM-Omni)
100
100
50
0
10
20
0
(b) SLO Scale
GPU Mem. Usage (%)
SLO Attainment (%)
NoCache (vLLM-Omni) 100 90
50
8
16
24
(c) Number of GPUs
0
32
2
4
(d) CoV Scale
ServerlessT2I 100 80 60 40 20 0
0
50
100
150
(e) Time (s)
NoCache (Diffusers)
WCache (Diffusers)
MCache (Diffusers)
100 90
100
100
100
50
50
50
50
0
1
2
(a) Rate Scale
3
0
5
10
15
(b) SLO Scale
0
8
16
24
(c) Number of GPUs
0
32
2
4
6
(d) CoV Scale
GPU Mem. Usage (%)
SLO Attainment (%)
Figure 10. End-to-end evaluation of Setting 1 (S1). All baselines are implemented on top of vLLM-Omni [84]. ServerlessT2I 100 80 60 40 20 0
0
100
200
(e) Time (s)
Latency (s)
even at the loosest SLO. This gap reveals that the baselines are not merely short of compute capacity; their tail latency is dominated by model and workflow loading overheads, which only very loose deadlines can hide. The same trend holds in S2 (Fig. 11(b)): ServerlessT2I achieves 97% attainment at SLO scale 8.0, while MCache and WCache require scales of 16.0 and 18.0, respectively. Even as adapters increase workflow diversity, ServerlessT2I converts deadline slack into SLO attainment more efficiently than baselines. SLO Attainment vs. Testbed Size. We next vary the number of GPUs while keeping the workload fixed. In Fig. 10(c), ServerlessT2I meets the 90% SLO attainment target with only 8 GPUs; the strongest vLLM-Omni baseline requires 24 GPUs to match this, MCache requires 32, and NoCache remains below 90% even at 32 GPUs. In S2, ServerlessT2I reaches 98% attainment with 16 GPUs, whereas the strongest Diffusers baseline requires 24 GPUs to exceed 90% and other baselines remain far below target at 32 GPUs. These results show that ServerlessT2I’s gains extend beyond latency: scaling individual models rather than entire workflows reduces over-provisioning and allows the cluster to operate as a shared GPU pool. SLO Attainment vs. CoV. Finally, we evaluate robustness to bursty traffic by varying the coefficient of variation (CoV) of request arrivals while fixing the average rate scale, SLO scale, and testbed size. Following prior works [36, 56], we partition the original trace into time windows, fit arrivals to a Gamma process, and resample at scaled CoV values to control burstiness. Higher CoV increases short-term queue buildup, stressing autoscaling and cache replacement. As shown in Fig. 10(d), ServerlessT2I maintains at least 96% attainment up to CoV scale 4.0, while MCache falls below 90% between scales 2.4–4.0 and WCache is below 90% at the lowest CoV. In S2, ServerlessT2I stays above 90% through CoV scale 5.0, whereas MCache drops to 89% at scale 2.5. The comparison between S1 and S2 indicates that adapter-heavy workloads make burst handling more sensitive to cache granularity: workflow-level caching cannot react quickly when bursts
2
1.9 1.4
1.2
1 0.78
0
w/o ppl.
Loading Inference 1.1 1.0
0.29 0.11 0.03 ppl. ppl. ppl. (0) (Le′ /2) (Le′ )
w/o load
Normalized Latency
Figure 11. End-to-end evaluation of Setting 2 (S2). All baselines are implemented on top of Diffusers [85]. 1-GPU
2-GPU
Opt.
1.0
0.5
0.0
0.510.50
0.62 0.55
CFG parallel ControlNet parallel
Figure 12. Left: Latency breakdown of workflow execution. ppl.: pipeline. Right: Normalized inference latency w/ and w/o parallelization. Opt.: theoretically optimal latency. shift demand across variants, while ServerlessT2I absorbs these shifts by reusing shared model components. GPU Memory Utilization. Figures 10(e) and 11(e) show average GPU memory utilization at runtime. ServerlessT2I consistently achieves the highest utilization, followed by MCache, WCache, and NoCache. This ordering reflects their caching granularities: ServerlessT2I caches at the granularity of model layers, MCache at models, and WCache at entire workflows. Finer granularity reduces internal fragmentation by better utilizing residual memory. In S1 at rate scale 1.5, NoCache, WCache, MCache, and ServerlessT2I achieve 13%, 29%, 71%, and 88% utilization, respectively. In S2 at rate scale 1.75, they achieve 30%, 41%, 62%, and 84%, respectively. 8.3
11
Data Plane
Model Loading. We elaborate on the model loading design in §5.1. Fig. 12-left reports the latency breakdown of a basic Flux1-Schnell workflow request, including model loading and inference. We define two performance bounds: a lower bound where loading and inference execute serially without pipelining (w/o ppl.), and an upper bound where model loading latency is fully hidden (w/o load). We show a spectrum of caching configurations: ppl.(0) indicates no layers are pre-cached in the GPU, equivalent to existing layer-wise pipelined loading [11, 103]. By caching 𝐿𝑒′ layers in the GPU (ppl.(𝐿𝑒′ )), ServerlessT2I reduces loading latency by 90% and end-to-end latency by 21% relative to existing methods, rivaling the upper bound. Even when half of the 𝐿𝑒′ layers
w/ model weight virtualization
5 System System Stalls Stalls
0
3.73.4
12.1
5.9 4.2
3.8
2 4 8 16 32 Size of NVSHMEM Region (GiB)
vTime Diff. (s)
Volume of Model Loading (TiB)
w/o model weight virtualization
10
20 10 0 0
Slack=20 Slack=10 Slack=5 Slack=0
8.5
Execution Overhead. Decomposing a monolithic workflow into a model DAG introduces overhead from data communication and control-plane coordination. We quantify this overhead by comparing ServerlessT2I with monolithic baselines on S1’s workflows. For each workflow, we measure the execution latency of requests that generate the identical image. Compared with Diffusers, on which ServerlessT2I builds its model components, ServerlessT2I adds only 3% average latency overhead. While vLLM-Omni reduces execution latency by 8% relative to ServerlessT2I, it falls short in the end-to-end evaluation (§8.2). ServerlessT2I at scale. To show ServerlessT2I remains efficient at large scale, we conduct simulation-based experiments on a 256-GPU setup under high concurrency, with 600 inflight requests. The simulator models ServerlessT2I’s procedures with request latencies matching measured values. Across the workflows in S1, ServerlessT2I incurs only 3.3% overhead of total execution time, indicating that neither the control plane nor the data plane becomes a bottleneck at this scale.
50 100 Relative Clock Time (s)
Figure 13. Left: Effectiveness of model weight virtualization. Right: Fair scheduler with varying slacks.
are evicted (ppl.(𝐿𝑒′ /2)), ServerlessT2I still reduces loading latency by 62%. Communication. ServerlessT2I’s data plane enables efficient and flexible communication for T2I-specific parallelization (§2.2). We validate these properties by measuring the speedups achieved by ServerlessT2I’s parallelization, which would be substantially reduced if either property were absent. As shown in Fig. 12-right, ServerlessT2I exploits parallelism to accelerate Z-Image workflow executions on NVIDIA H800 GPUs. Its speedup approaches the theoretically optimized performance and is consistent with prior results [29, 52, 55], indicating that ServerlessT2I’s data plane supports efficient, timely data exchange. Model weight virtualization. As described in §5.3, model weight virtualization allows cached weights to reside in either GPU memory region, addressing the limitations of a statically sized NVSHMEM region. We validate this on an 8-GPU testbed using the ControlNet-augmented workflows in S2 at a rate scale of 0.5. As shown in Fig. 13-left, an undersized NVSHMEM region can stall the system when in-flight operations cannot allocate space for data communication, even after all weights have been evicted from the NVSHMEM region. An oversized NVSHMEM region, conversely, crowds out the space for cached model weights without model weight virtualization, increasing the volume of loaded weight by up to 3× and degrading SLO attainment by up to 50%. We also verified that repeated offloading and reloading of parameters does not fragment the PyTorch or NVSHMEM regions: even at 97% peak memory utilization, we observed zero allocation stalls, retries, or out-of-memory events. 8.4
9
Tradeoff between Fairness and Efficiency
ServerlessT2I’s scheduler exposes a configurable fairness slack to balance fairness and efficiency. We set a microbenchmark on an 8-GPU testbed with two tenants, issuing basic Z-Image-Turbo and Flux1-Schnell workflows, respectively. In Fig. 13-right, ServerlessT2I bounds the service difference according to the configured slack when requests are backlogged. Larger slack values permit greater transient imbalance, but improve scheduling flexibility and hence serving efficiency: SLO attainment increases from 62% with strict fairness (Slack=0) to 75%, 81%, and 84% with slack values of 5, 10, and 20, respectively.
System Overhead
12
Discussion and Related Work
Can ServerlessT2I serve video generation models? While video generation also uses diffusion models, its serving characteristics differ substantially from T2I, placing it outside ServerlessT2I’s scope. First, video models exhibit different loading and computation profiles. For the transformer of Wan2.2-14B [86], the one-step denoising inference latency for a 480P video on an H800 is 2× higher than its loading latency, so loading can be pipelined with computation. Second, video generation service is typically exposed through APIs rather than user-composed workflows, a poor fit for serverless execution. As evidence, Wan2.2-14B (Mar. 2025) has only 2 community adapters on HuggingFace, whereas ZImage (Nov. 2025) has 135, both released by the same Alibaba Tongyi team. Model Serving in Serverless Clouds. To our knowledge, prior serverless model-serving systems target DNN and LLM inference rather than T2I workflows [40]. To reduce model loading latency, PipeSwitch and Torpor [11, 103] overlap host-to-GPU model loading with inference, but this is less effective for T2I workflows, where loading can still dominate the execution time of individual stages. BlitzScale and FaaScale [104, 110] speed up loading by transferring parameters over high-speed GPU interconnects instead of PCIe. However, this design assumes a small set of repeatedly loaded model types and spare interconnect bandwidth, both of which are less suitable for serverless T2I serving, where workflows use diverse models and interconnects are also needed for intermediate tensor communication. ServerlessLLM [31] reduces LLM loading latency with a multi-tier path across disk, host memory, and GPU memory. As discussed in §5,
ServerlessT2I instead focuses on memory-to-GPU loading and GPU-resident communication, where pinned memory alone is insufficient. Its disk-to-memory optimization is complementary to ServerlessT2I. ServerlessLLM [31], DeepServe [40], and Medusa [107] also optimize LLM-specific state such as KV-cache management and CUDA-graph materialization. These techniques are valuable, but they do not address the workflow heterogeneity and data-movement bottlenecks of serverless T2I serving. T2I Workflow Serving Systems. Existing T2I workflow serving systems [12, 24, 76, 84] accelerate individual workflow execution, but they do not target serverless deployment. As a result, they inherit the limitations of deploying T2I workflows as monolithic GPU functions on serverless platforms (§2.3). Nirvana [2] reduces denoising steps through cached images; DistriFusion [52] and xDiT [29] exploit multi-GPU parallelism; Katz [55] parallelizes ControlNets and asynchronously loads LoRAs; TetriServe [58] and TridentServe [93] adapt sequence parallelism for latency SLOs. However, several of these systems [2, 52, 58, 93] do not support the adapters commonly used in production workloads [55, 57]. ServerlessT2I is complementary to these systems: it targets efficient T2I workflow serving on serverless platforms and focuses on serverless-specific challenges in scaling, data movement, and multi-tenant scheduling. Other Model Serving Systems. Prior work on model serving has improved latency [23, 43, 75, 78, 87, 89, 115, 116], throughput [4, 73, 96, 101], and resource efficiency [37, 88, 90, 95, 109] across DNNs and LLMs [3, 14, 15, 27, 32, 33, 38, 41, 60, 68, 80, 91, 97, 98, 100, 105, 114]. ServerlessT2I complements this line of work by focusing on T2I workflow serving, which has different computation characteristics. KunServe [16] is the closest to our data plane design, but it is specific to LLM serving: it observes redundant LLM parameters and selectively drops them to free memory for KV cache.
10
[2] Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. 2024. Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models. In Proc. USENIX NSDI. [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In Proc. USENIX OSDI. [4] Sohaib Ahmad, Hui Guan, Brian D. Friedman, Thomas Williams, Ramesh K. Sitaraman, and Thomas Woo. 2024. Proteus: A highthroughput inference-serving system with accuracy scaling. In Proc. ACM ASPLOS. [5] Amazon SageMaker AI. 2026. Deploy models with Amazon SageMaker Serverless Inference. https://docs.aws.amazon.com/ sagemaker/latest/dg/serverless-endpoints.html. [6] Amazon. 2025. AWS Lambda pricing. https://aws.amazon.com/ lambda/pricing/. [7] Nvidia NVSHMEM APIs. 2026. Memory Management. https://docs. nvidia.com/nvshmem/api/gen/api/memory.html. [8] AWS. 2025. Adapt your own inference container for Amazon SageMaker AI. https://docs.aws.amazon.com/sagemaker/latest/dg/adaptinference-container.html. [9] Microsoft Azure. 2026. Azure documentation. https://learn.microsoft. com/en-us/azure/durable-task/common/durable-task-sequence. [10] Azure-Samples. 2026. Azure Functions PyTorch ML multi-model image classification with Remote Build and Azure File integration. https://github.com/Azure-Samples/azure-functions-pytorchimage-identify/blob/master/classify/__init__.py. [11] Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020. PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications. In Proc. USENIX OSDI. [12] BentoML. 2025. comfy-pack: Serving ComfyUI Workflows as APIs. https://www.bentoml.com/blog/comfy-pack-serving-comfyuiworkflows-as-apis. [13] Xiaohu Chai, Tianyu Zhou, Keyang Hu, Jianfeng Tan, Tiwei Bie, Anqi Shen, Dawei Shen, Qi Xing, Shun Song, Tongkai Yang, Le Gao, Feng Yu, Zhengyu He, Dong Du, Yubin Xia, Kang Chen, and Yu Chen. 2025. Fork in the road: reflections and optimizations for cold start latency in production serverless systems. In Proc. USENIX OSDI. [14] Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, Yuening Zhu, Qingliang Ou, Jiaqi Liao, Xianglin Chen, Zhiyuan Ai, Yongwei Wu, and Mingxing Zhang. 2025. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models. In Proc. SOSP. [15] Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. 2025. Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference. In Proc. SOSP. [16] Rongxin Cheng, Yuxin Lai, Xingda Wei, Rong Chen, and Haibo Chen. 2026. KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving. In Proc. EuroSys. [17] Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. 2021. VITON-HD: High-Resolution Virtual Try-On via MisalignmentAware Normalization. In Proc. CVPR. [18] Alibaba Cloud. 2025. Alibaba Cloud Function Compute Billing Overview. https://www.alibabacloud.com/help/en/functioncompute/ fc/product-overview/billing-overview-of-fc. [19] Alibaba Cloud. 2026. Build a text-to-image service with ComfyUI and SD/FLUX using Function Compute. https://www.alibabacloud.com/help/en/functioncompute/fc/usecases/building-a-text-to-image-service-that-uses-comfyui-andsd-flux-through-function-compute?spm=a2c63.p38356.help-menu2508973.d_4_0_0.62b54f6bCLyXzC&scm=20140722.H_2872195._.
Conclusions
We presented ServerlessT2I, an efficient serverless inference system for T2I workflows. ServerlessT2I has three key designs: (1) a programming interface that allows users to compose workflows as a model DAG; (2) a unified data plane that harvests slack GPU memory for efficient model loading and data communication; and (3) a fairness-aware scheduler for multi-tenant serverless serving. Overall, ServerlessT2I substantially improves T2I workflow serving efficiency. Under the same GPU budget, it sustains up to 2× higher request rates than existing serving systems; at a fixed request rate, it reduces GPU requirements by up to 3× while meeting SLOs for more than 90% of requests.
References [1] Adobe. 2025. Create with Adobe Firefly generative AI. https://www. adobe.com/products/firefly.html.
13
OR_help-T_intl~en-V_1. Create a GPU function. https: [20] Alibaba Cloud. 2026. //www.alibabacloud.com/help/en/functioncompute/fc/userguide/creating-a-gpu-function/. [21] Google Cloud. 2026. Google Cloud Serverless Computing. https: //cloud.google.com/serverless. [22] ComfyUI. 2025. ComfyUI: The most powerful and modular visual AI engine and application. https://github.com/comfyanonymous/ ComfyUI. [23] Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A low-latency online prediction serving system. In Proc. USENIX NSDI. [24] HuggingFace Diffusers. 2025. Create a server. https://github. com/huggingface/diffusers/blob/main/docs/source/en/usingdiffusers/create_a_server.md. [25] HuggingFace Diffusers. 2025. Philosophy. https://huggingface.co/ docs/diffusers/en/conceptual/philosophy. [26] Alibaba Cloud Documentation. 2026. Set function quotas. https://www.alibabacloud.com/help/en/functioncompute/fc/userguide/overview-of-configuring-the-maximum-number-of-ondemand-instances. [27] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. In Proc. ICML. [28] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for HighResolution Image Synthesis. In Proc. ICML. [29] Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. 2024. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. arXiv preprint arXiv:2411.01738 (2024). [30] FastAPI. 2025. FastAPI. https://github.com/fastapi/fastapi. [31] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: LowLatency Serverless Inference for Large Language Models. In Proc. USENIX OSDI. [32] Shiwei Gao, Qing Wang, Shaoxun Zeng, Youyou Lu, and Jiwu Shu. 2025. WEAVER: efficient multi-LLM serving with attention offloading. In Proc. ATC. [33] Lingwen Gong, Kaixin Liu, Xiaolu Li, Shujie Han, Patrick P. C. Lee, Yuchong Hu, and Dan Feng. 2025. HyperGen: Optimizing Generative Inference with Long Prompts for Resource-Constrained Systems. In Proc. APSys. [34] GoogleCloudPlatform. 2026. Google Cloud Platform Python Samples. https://github.com/GoogleCloudPlatform/python-docssamples/blob/main/run/image-processing/main.py. [35] Grokipedia. 2026. Multi-GPU Support in ComfyUI. https://grokipedia. com/page/Multi-GPU_Support_in_ComfyUI. [36] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance predictability from the bottom up. In Proc. USENIX OSDI. [37] Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R. Das. 2022. Cocktail: A multidimensional optimization for model serving in cloud. In Proc. USENIX NSDI. [38] Yongjun He, Haofeng Yang, Yao Lu, Ana Klimović, and Gustavo Alonso. 2025. Resource multiplexing in tuning and serving large language models. In Proc. ATC. [39] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. ICLR.
[40] Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Jie Meng, Baoquan Zhang, Shining Wan, Gengyuan Dan, Zhiyu Dong, Zhihao Ren, Changhong Liu, Tao Xie, Dayun Lin, Qin Zhang, Yue Yu, Hao Feng, Xusheng Chen, and Yizhou Shan. 2025. DEEPSERVE: serverless large language model serving at scale. In Proc. ATC. [41] Zhengding Hu, Vibha Murthy, Zaifeng Pan, Wanlu Li, Xiaoyi Fang, Yufei Ding, and Yuke Wang. 2025. HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows. In Proc. SOSP. [42] HuggingFace. 2026. Using GPU Spaces. https://huggingface.co/docs/ hub/en/spaces-gpus. [43] Wenqi Jiang, Suvinay Subramanian, Cat Graves, Gustavo Alonso, Amir Yazdanbakhsh, and Vidushi Dadu. 2025. RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving. In Proc. ISCA. [44] Xiaoxiao Jiang, Suyi Li, Lingyun Yang, Tianyu Feng, Zhipeng Di, Weiyi Lu, Guoxuan Zhu, Xiu Lin, Kan Liu, Yinghao Yu, et al. 2026. FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling. In Proc. EuroSys. [45] Eric Jonas, Qifan Pu, Shivaram Venkataraman, Ion Stoica, and Benjamin Recht. 2017. Occupy the cloud: distributed computing for the 99%. In Proc. SoCC. [46] Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. 2024. BrushNet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. In Proc. ECCV. [47] Kenneth C. Knowlton. 1965. A fast storage allocator. Commun. ACM (1965). [48] Kenneth C. Knowlton. 1966. A programmer’s description of L6. Commun. ACM (1966). [49] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proc. SOSP. [50] Black Forest Lab. 2025. FLUX.2: Frontier Visual Intelligence. https: //bfl.ai/blog/flux-2. [51] Black Forest Labs. 2024. FLUX. https://github.com/black-forestlabs/flux. [52] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han. 2024. DistriFusion: Distributed parallel inference for high-resolution diffusion models. In Proc. IEEE/CVF CVPR. [53] Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2025. Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference. In Proc. USENIX ATC. [54] Suyi Li, Wei Wang, Jun Yang, Guangzhen Chen, and Daohe Lu. 2023. Golgi: Performance-Aware, Resource-Efficient Function Scheduling for Serverless Computing. In Proc. SoCC. [55] Suyi Li, Lingyun Yang, Xiaoxiao Jiang, Hanfeng Lu, Zhipeng Di, Weiyi Lu, Jiawei Chen, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, and Wei Wang. 2025. Katz: Efficient Workflow Serving for Diffusion Models with Many Adapters. In Proc. USENIX ATC. [56] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In Proc. USENIX OSDI. [57] Yanying Lin, Shuaipeng Wu, Shutian Luo, Hong Xu, Haiying Shen, Chong Ma, Min Shen, Le Chen, Chengzhong Xu, Lin Qu, and Kejiang Ye. 2025. Understanding Diffusion Model Serving in Production: A Top-Down Analysis of Workload, Scheduling, and Resource Efficiency. In Proc. ACM SoCC.
14
[79] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in Serving Large Language Models. In Proc. USENIX OSDI. [80] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. 2025. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. In Proc. ICLR. [81] stabilityai. 2025. stable-diffusion-3.5-large. https://huggingface.co/ stabilityai/stable-diffusion-3.5-large. [82] Suyi32. 2025. distrifuser-controlnet. https:// github.com/Suyi32/distrifuser-controlnet/commit/ e2099655ef0052ee1f5f030007f52bb25faf06b4. [83] Image Team, Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Xin Jin, Liangchen Li, Zhen Li, Zhong-Yu Li, David Liu, Dongyang Liu, Junhan Shi, Qilong Wu, Feng Yu, Chi Zhang, Shifeng Zhang, and Shilin Zhou. 2025. Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer. arXiv:2511.22699 [84] vllm project. 2026. vLLM Omni. https://github.com/vllm-project/ vllm-omni. [85] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. 2022. Diffusers: State-of-the-art diffusion models. https://github.com/ huggingface/diffusers. [86] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. 2025. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314 [cs.CV] https://arxiv.org/abs/2503.20314 [87] Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxin Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jingqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, and Congfeng Jiang. 2026. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation. Proc. ACM Manag. Data (2026). [88] Luping Wang, Lingyun Yang, Yinghao Yu, Wei Wang, Bo Li, Xianchao Sun, Jian He, and Liping Zhang. 2021. Morphling: Fast, near-optimal auto-configuration for cloud-native model serving. In Proc. ACM SoCC. [89] Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An efficient multi-level inference system for large language models. In Proc. ACM EuroSys. [90] Yuke Wang, Boyuan Feng, Zheng Wang, Tong Geng, Kevin Barker, Ang Li, and Yufei Ding. 2023. MGG: Accelerating graph neural networks with fine-grained intra-kernel communication-computation pipelining on multi-GPU platforms. In Proc. USENIX OSDI. [91] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proc. SOSP. [92] Hao Wu, Junxiao Deng, Minchen Yu, Yue Yu, Yaochen Liu, Hao Fan, Song Wu, and Wei Wang. 2026. Efficient Data Passing for Serverless Inference Workflows: A GPU-Centric Approach. In Proc. ACM EuroSys. [93] Yifei Xia, Fangcheng Fu, Hao Yuan, Hanke Zhang, Xupeng Miao, Yijun Liu, Suhan Ling, Jie Jiang, and Bin Cui. 2025. TridentServe: A
[58] Runyu Lu, Shiqi He, Wenxuan Tan, Shenggui Li, Ruofan Wu, Jeff J. Ma, Ang Chen, and Mosharaf Chowdhury. 2026. TetriServe: Efficiently Serving Mixed DiT Workloads. In Proc. ASPLOS. [59] Ashraf Mahgoub, Edgardo Barsallo Yi, Karthick Shankar, Sameh Elnikety, Somali Chaterji, and Saurabh Bagchi. 2022. ORION and the Three Rights: Sizing, Bundling, and Prewarming for Serverless DAGs. In Proc. USENIX OSDI. [60] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proc. ASPLOS. [61] Midjourney. 2025. Midjourney AI. https://www.midjourney.com/ explore. [62] Modal. 2025. How OpenArt scaled their Gen AI art platform on hundreds of GPUs. https://modal.com/blog/openart-case-study. [63] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In Proc. USENIX OSDI. [64] PyTorch Developer Notes. 2026. CUDA semantics. https://docs. pytorch.org/docs/2.12/notes/cuda.html. [65] NVIDIA. 2025. NVIDIA OpenSHMEM Library (NVSHMEM) Documentation. https://docs.nvidia.com/nvshmem/api/index.html. [66] NVIDIA. 2025. NVIDIA OpenSHMEM Library (NVSHMEM) Documentation, Environment Variables. https://docs.nvidia.com/ nvshmem/api/gen/env.html. [67] NVIDIA. 2025. NVSHMEM APIs, Memory Management. https://docs. nvidia.com/nvshmem/api/gen/api/memory.html. [68] Gabriele Oliaro, Xupeng Miao, Xinhao Cheng, Vineeth Kada, Ruohan Gao, Yingyi Huang, Remi Delacourt, April Yang, Yingcheng Wang, Mengdi Wu, Colin Unger, and Zhihao Jia. 2025. FlexLLM: A system for co-serving large language model inference and parameter-efficient finetuning. arXiv preprint arXiv:2402.18789 (2025). [69] OpenAI. 2020. OpenAI API. https://openai.com/index/openai-api/. [70] OpenAI. 2025. Introducing 4o Image Generation. https://openai.com/ index/introducing-4o-image-generation/. [71] OpenAI. 2025. OpenAI DALL·E 2. https://openai.com/index/dall-e-2/. [72] OpenAI. 2025. Rate limits. https://developers.openai.com/api/docs/ guides/rate-limits. [73] Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi, Yongji Wu, Jialin Li, and Cheng Li. 2026. Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic LLM Workloads. In Proc. NSDI. [74] Johann Schleier-Smith, Vikram Sreekanti, Anurag Khandelwal, Joao Carreira, Neeraja J. Yadwadkar, Raluca Ada Popa, Joseph E. Gonzalez, Ion Stoica, and David A. Patterson. 2021. What serverless computing is and should become: the next phase of cloud computing. Commun. ACM (2021). [75] Luis Gaspar Schroeder, Aditya Desai, Alejandro Cuadron, Kyle Chu, Shu Liu, Mark Zhao, Stephan Krusche, Alfons Kemper, Matei Zaharia, and Joseph E. Gonzalez. 2026. vCache: Verified Semantic Prompt Caching. In Proc. ICLR. [76] sgl project. 2026. SGLang Diffusion. https://github.com/sgl-project/ sglang/tree/main/python/sglang/multimodal_gen. [77] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. In Proc. ATC. [78] Jianshu She, Zonghang Li, Hongchao Du, Shangyu Wu, Wenhao Zheng, Eric P. Xing, Zhengzhong Liu, Huaxiu Yao, Jason Xue, and Qirong Ho. 2026. PLA-Serve: A Prefill-Length-Aware LLM Serving System. In Proc. MLSys.
15
Stage-level Serving System for Diffusion Pipelines. arXiv:2510.02838 [94] Yuhao Xu, Tao Gu, Weifeng Chen, and Arlene Chen. 2025. OOTDiffusion: Outfitting Fusion Based Latent Diffusion for Controllable Virtual Try-On. Proc. AAAI (2025). [95] Lingyun Yang, Yongchen Wang, Yinghao Yu, Qizhen Weng, Jianbo Dong, Kan Liu, Chi Zhang, Yanyi Zi, Hao Li, Zechao Zhang, Nan Wang, Yu Dong, Menglei Zheng, Lanlan Xi, Xiaowei Lu, Liang Ye, Guodong Yang, Binzhang Fu, Tao Lan, Liping Zhang, Lin Qu, and Wei Wang. 2025. GPU-disaggregated serving for deep learning recommendation models at scale. In Proc. USENIX NSDI. [96] Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. 2022. INFless: A native serverless system for low-latency, high-throughput inference. In Proc. ACM ASPLOS. [97] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proc. EuroSys. [98] Xiaozhe Yao, Qinghao Hu, and Ana Klimovic. 2025. DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs. In Proc. EuroSys. [99] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023). [100] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for transformer-based generative models. In Proc. USENIX OSDI. [101] Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful Large Language Model Serving with Pensieve. In Proc. EuroSys. [102] Minchen Yu, Tingjia Cao, Wei Wang, and Ruichuan Chen. 2023. Following the Data, Not the Function: Rethinking Function Orchestration in Serverless Computing. In Proc. NSDI. [103] Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, Haoran Yang, and Yu Ding. 2025. Torpor: GPU-enabled serverless computing for lowlatency, resource-efficient inference. In Proc. ATC. [104] Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Zirui Wang, Yue Cheng, Wei Wang, Ao Wang, and Ruichuan Chen. 2026. FaaScale: Unlocking Fast LLM Scaling for Serverless Inference. In Proc. MLSys. [105] Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Hank Levy, and David Culler. 2025. IC-Cache: Efficient Large Language Model Serving via In-context Caching. In Proc. SOSP. [106] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Proc. NSDI. [107] Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. 2025. Medusa: Accelerating Serverless LLM Inference with Materialization. In Proc. ASPLOS. [108] zeromq. 2025. ZeroMQ. https://github.com/zeromq/pyzmq. [109] Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. In Proc. USENIX ATC. [110] Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. Fast and Live Model Auto Scaling without Caching. In Proc. USENIX OSDI. [111] Lvmin Zhang. 2025. Fooocus. https://github.com/lllyasviel/Fooocus. [112] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In Proc. IEEE/CVF ICCV. [113] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2025. Scaling Inthe-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport. In Proc. ICLR.
[114] Wei Zhang, Ziyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2026. JITServe: SLO-aware LLM Serving with Imprecise Request Information. In Proc. USENIX NSDI. [115] Zongpu Zhang, Pranab Dash, Y. Charlie Hu, Qiang Xu, Jian Li, and Haibing Guan. 2026. Rethinking DVFS for Mobile LLMs: Unified Energy-Aware Scheduling with CORE. In Proc. MLSys. [116] Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, Fan Yang, Mao Yang, Lidong Zhou, Asaf Cidon, and Gennady Pekhimenko. 2022. ROLLER: Fast and Efficient Tensor Compilation for Deep Learning. In Proc. USENIX OSDI. [117] NVIDIA Developer Zone. 2026. Peer Device Memory Access. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART_ _PEER.html.
16
A
weights are not present in host memory and therefore must be fetched from remote storage. The miss rate drops quickly as cache capacity increases and then saturates near zero: caching the 10 most popular workflows reduces the miss rate to 0.37%, while caching the top 15 reduces it further to 0.09%. Thus, with only the top 15 workflows cached in host memory, more than 99% of model loads are served from host memory rather than remote storage. We also evaluate a least-frequently-used (LFU) policy and observe similar results, suggesting that the benefit comes primarily from the inherent popularity skew rather than from a specific eviction policy. Since host memory is largely underutilized in our production setting (Fig. 4), caching these popular workflows imposes little additional resource cost. Based on this observation, ServerlessT2I keeps popular workflow weights in host memory and focuses the remainder of the loading design on optimizing the host-to-GPU transfer path.
Caching Workflows in Host Memory
This appendix provides additional details on the trace replay described in §5.1. A T2I workflow is large, often totaling tens of GiBs, and loading its weights from external storage can introduce substantial latency. However, production workflow popularity is highly skewed: a small number of workflows account for the vast majority of requests. This skew makes host memory an effective intermediate cache tier. By storing the weights of only the most popular workflows in otherwise underutilized host memory, ServerlessT2I can avoid reading external storage for nearly all model loads, leaving only the much faster host-to-GPU transfer on the critical path. To quantify this effect, we replay our production trace using host memory caches of varying capacities. We use leastrecently-used (LRU) replacement and measure the cache miss rate, defined as the fraction of requests whose workflow
17