arXiv CSopen access

LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflows

LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflows Lingyun Yang†∗ , Suyi Li†∗# , Tianyu Feng† , Xiaoxiao Jiang† , Zhipeng Di, Weiyi Lu, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, Wei Wang† † Hong Kong University of Science and Technology Alibaba Group

arXiv:2604.08123v1 [cs.DC] 9 Apr 2026

Abstract

A portrait of a lovely shorthair golden-shaded cat, sitting on a windowsill, capturing every fur detail.

Text-to-image generation executes a diffusion workflow comprising multiple models centered on a base diffusion model. Existing serving systems treat each workflow as an opaque monolith, provisioning, placing, and scaling all constituent models together, which obscures internal dataflow, prevents model sharing, and enforces coarse-grained resource management. In this paper, we make a case for micro-serving diffusion workflows with LegoDiffusion, a system that decomposes a workflow into loosely coupled model-execution nodes that can be independently managed and scheduled. By explicitly managing individual model inference, LegoDiffusion unlocks cluster-scale optimizations, including permodel scaling, model sharing, and adaptive model parallelism. Collectively, LegoDiffusion outperforms existing diffusion workflow serving systems, sustaining up to 3× higher request rates and tolerating up to 8× higher burst traffic.

1

Prompt

Iterative denoising computation attn Text encoders

A portrait of … capturing every fur detail.

Text encoders Reference image

Image encoder

A portrait of … capturing every fur detail.

Text encoders

attn

attn Decoder

Diffusion model Diffusion model attn attn attn

attn

attn attn attn ControlNet Patch

Decoder

Diffusion model

ControlNet

Decoder

Image encoder

Figure 1. Top: a basic diffusion workflow using FluxDev [34]. Middle: add ControlNets [90] alongside the base diffusion model, which process an additional input reference image and pass the intermediaries to diffusion model to control the composition in image generation. Bottom: Further add LoRA [27] to change image styles by patching LoRA weights onto diffusion model weights.

Introduction

Text-to-image (T2I) generation using diffusion models enables the creation of high-quality, contextually accurate images from textual description [33, 46, 52, 53, 74, 90]. A typical T2I generation workflow integrates a base diffusion model with multiple adapter models to form a pipeline. Execution begins with text encoders that convert prompts into embeddings (Fig. 1-top). Conditioned on the embeddings, the diffusion model iteratively generates latent representations via a denoising process, which are subsequently decoded as the final image. To refine visual attributes such as composition or artistic styles, T2I workflows increasingly incorporate adapter models, such as ControlNet [90] and LoRA [27], alongside the base model [38, 41]. By augmenting the diffusion process (Fig. 1), these adapters enable fine-grained alignment with user intent and aesthetics [27, 79, 90, 91]. Despite the modular nature of diffusion workflows, existing inference systems, such as HuggingFace Diffusers [15] and ComfyUI [7], primarily employ a monolithic serving practice, where the entire workflow, comprising the base model and all associated adapters, is encapsulated into a monolithic instance for provisioning and scheduling. The system treats these workflow instances as opaque black boxes running on a fixed set of GPUs, while remaining oblivious to the workflow’s internal model executions and data exchanges.

While operationally simple, monolithic serving introduces fundamental inefficiencies. First, it forces coarse-grained scaling: if a single model becomes a bottleneck, the system must replicate the entire workflow. Second, isolated monoliths preclude model sharing. Production traces [38, 41] show that popular diffusion backbones and adapters are frequently reused across workflows. However, monolithic serving forces each workflow instance to maintain its own model copies, leading to redundant memory footprints. Third, treating workflows as opaque black boxes hides internal data dependencies, preventing the system from automatically optimizing resource allocation and pipelining. Finally, tight coupling increases fragility: a single model failure crashes the entire workflow. To address these limitations, we argue that the schedulable unit for inference should be the individual model execution node, not the entire diffusion workflow. Instead of monolithic instances, the system should decompose workflows into loosely-coupled microservices—each encapsulating a specific component like a text encoder, diffusion backbone, or adapter. This micro-serving architecture directly addresses the inefficiencies of monolithic serving. First, it enables finegrained scaling: individual models can scale elastically based on real-time demand, eliminating the resource waste of replicating the entire pipeline. Second, it facilitates cross-workflow

* Equal contribution; # Corresponding author

1

model sharing: distinct workflows can multiplex shared models, such as a common base model, avoiding redundant memory footprints. Third, by making model execution and data flow explicit, the system regains visibility into the computation graph, enabling automated optimization of resource allocation and pipelining. Decoupling the workflow also enables fault isolation and fast failure recovery. However, realizing micro-serving for diffusion workflows presents non-trivial systems challenges. While prior frameworks have successfully applied micro-serving to CPU-centric data analytics [60, 70, 81, 83, 87] and LLM-based agentic workflows [35, 43, 47, 63], applying this paradigm to diffusion pipelines requires overcoming hurdles that these systems cannot handle. First, diffusion workflows exhibit complex, iterative data dependencies between base models and adapters, which cannot be expressively defined or easily supported in existing frameworks. Second, decoupling these tightly integrated model workflows necessitates massive, latency-sensitive tensor communications across GPUs. Existing LLM or CPU micro-serving frameworks lack an efficient data plane to manage these high-bandwidth transfers. In this paper, we present LegoDiffusion, a system purposebuilt for the efficient micro-serving of diffusion workflow. LegoDiffusion introduces three key designs:

are transparent to developers: once the compiler produces the workflow DAG, the data engine automatically orchestrates all inter-node tensor transfers. Workflow Node Scheduling. The LegoDiffusion scheduler maps workflow nodes onto distributed executors using three strategies that exploit micro-serving’s decomposition. First, it enforces model-granular scaling: rather than replicating entire workflows, LegoDiffusion scales only the bottleneck models, avoiding redundant resource provisioning. Second, because a loaded model is workflow-agnostic, the scheduler preferentially dispatches nodes to executors that already hold the required model state, enabling multitenant model sharing. Third, LegoDiffusion employs adaptive parallelism: it dynamically adjusts model parallelism at scheduling time based on real-time cluster availability, rightsizing resource allocation to maximize throughput without incurring queuing delays. We prototyped LegoDiffusion and evaluated it across a diverse array of diffusion workflows, encompassing SD3 [18], SD3.5-Large [62], Flux-Dev, and Flux-Schnell [34], along with their respective adapters. Our experiments show that LegoDiffusion’s micro-serving architecture significantly outperforms state-of-the-art monolithic serving systems, sustaining up to 3× higher request rates and satisfying 6× more stringent SLO, and tolerating 8× higher burst traffic, while meeting latency SLOs for over 90% of requests. Crucially, we verify that LegoDiffusion maintains full compatibility with emerging diffusion-specific optimizations, such as approximate caching [4], achieving performance gains consistent with their original monolithic implementations. We will open-source LegoDiffusion after the double-blind review process.

Programming Interface & Compilation. LegoDiffusion provides a Python-embedded domain-specific language (DSL) for composing diffusion workflows. The DSL exposes primitives for model initialization, inference, and diffusion-specific operations for LoRA and ControlNet application [27, 90]. To provide a unified support for diverse community-developed models [30], LegoDiffusion wraps each model behind a standardized interface that encapsulates its native loading and inference logic. All primitives enforce strict input/output typing, making data dependencies explicit and catching errors at compile time. A graph compiler translates the workflow composition into a directed acyclic graph (DAG) of loosely coupled workflow nodes. Each node represents a discrete model inference operator that can be independently provisioned and scheduled–the fundamental unit of microserving.

2

Background and Problem Statement

2.1

A Primer on Image Generation Workflow

Basic Workflows. As illustrated in Fig. 1-top, a basic textto-image (T2I) generation workflow consists of three models: a text encoder, a base diffusion model, and a decoder-only variational autoencoder (VAE). The process begins with the text encoder, which encodes a text prompt into a sequence of semantic token embeddings. The system then initializes a latent tensor with random Gaussian noise. Conditioned on the text embeddings, the base diffusion model iteratively refines this tensor through a series of denoising steps. Finally, the denoised latent representation is passed to the VAE decoder, which reconstructs the output image in pixel space.

Runtime & Data Plane. At runtime, LegoDiffusion applies lazy execution: upon each request, it analyzes the workflow DAG and dynamically recomposes the compute graph— inserting or substituting nodes—to apply diffusion-specific optimizations. Decomposing workflows into distributed nodes, however, introduces high-bandwidth tensor transfers with complex synchronization requirements (e.g., forwarding ControlNet intermediates at specific denoising layers). To handle this, we design a distributed data engine atop NVSHMEM [49] that enables GPU-direct, zero-copy tensor movement over high-speed interconnects. The engine provides two fetch modes—eager and deferred—so that tensors arrive precisely when needed without stalling execution. These mechanisms

Workflows with Adapters. To achieve fine-grained control over visual attributes, such as spatial structure, diffusion models are frequently augmented with adapter models [27, 38, 79, 89–91]. These adapters can be categorized into two classes based on their execution patterns [38]: 2

Duplication Base D.M. Base D.M. Latent parallel

Sync & aggregate

model and becomes even more complex when multiple ControlNets are used, producing fan-in/fan-out transfers that are difficult to schedule efficiently. 3) Asynchronous LoRA loading. In production systems, LoRA adapters are often stored remotely and must be fetched on demand [38]. To hide this fetching cost, asynchronous LoRA loading overlaps adapter retrieval with the early stages of base-model inference. When the LoRA weights arrive, the system must pause execution, hot-patch the base model in GPU memory, and then resume computation [38]. This optimization introduces non-deterministic timing and dynamic state mutation: execution now depends on I/O completion, forcing the system to coordinate mid-inference weight updates without incurring synchronization stalls.

C.N. Base D.M. ControlNets parallel

Figure 2. Latent parallelism and ControlNets parallelism.

1) Parallel Execution Adapters. The first class includes adapters that operate in tandem with the base diffusion model during inference, such as ControlNet [90] (Fig. 1-middle). From a systems perspective, they introduce two complications: (1) their parameter sizes are often comparable to the base model, introducing substantial model loading latency; and (2) maximizing throughput often requires parallelizing the adapter and base model across GPUs, which necessitates intricate synchronization and data transfer patterns (§2.2). 2) Weight-Patching Adapters. The second class adapts the base model through parameter-efficient fine tuning, such as LoRA [27] and IC-Light [91] (Fig. 1-bottom). These adapters patch the base model’s weights before inference, incurring no additional computational overhead during subsequent denoising steps. The trade-off is state management: once patched, a diffusion model replica is specialized to a specific request until its weights are restored or replaced. Serving such workflows therefore requires fetching adapter weights from remote storage on demand [38], which can bottleneck loading and complicate sharing model replicas across requests.

2.2

Monolithic Serving and Its Inefficiency

Existing diffusion serving systems, such as HuggingFace Diffusers [15, 66], ComfyUI [11], SGLang-Diffusion [59], vLLM-Omni [65], and xDiT [19], operate on a monolithic paradigm. Whether employing Diffusers’ “single-file” abstraction* [16, 59, 65] or ComfyUI’s flexible node graph [12], these frameworks encapsulate the entire generation pipeline, comprising the base model, adapters, and control logic, into a monolithic execution unit. Consequently, the serving system provisions resources and schedules execution at the granularity of the entire workflow, without managing internal model invocations and data flows. While this monolithic serving simplifies deployment, it has four fundamental limitations:

Data Dependencies in Diffusion Workflows. Diffusion workflows exhibit intricate data dependencies induced by performance-oriented parallelization strategies. These strategies improve performance [19, 36, 38], but they also introduce model-specific data transfers and synchronizations that monolithic systems struggle to express or optimize (§4.3). We highlight three common cases: 1) Latent Parallelism. Diffusion models typically use classifier-free guidance (CFG) [26] to improve image quality by executing two denoising passes at each step: one conditioned on the prompt and one unconditional. Latent parallelism accelerates CFG by parallelizing these two computations on separate GPUs [19, 36, 38]. However, this approach introduces frequent “scatter-gather” synchronization, where partial results must be aggregated at every denoising step (Fig. 2). These high-frequency communication barriers can erode the benefit of parallelism if not handled efficiently. 2) ControlNet Parallelism. To reduce the overhead of ControlNets, serving systems often execute them in parallel with the base diffusion model on separate GPUs [38] (Fig. 2). This design introduces fine-grained data dependencies: ControlNet feature maps must be transferred to, and consumed by, specific layers of the base model during each denoising step. The exact communication pattern depends on the base

L1: Inefficient Scaling via Full Replication. Monolithic serving treats the entire workflow as a scaling unit, enforcing coarse-grained replication regardless of which component is the actual bottleneck. This indiscriminate scaling is particularly costly for diffusion workloads, where the base diffusion model is typically the sole bottleneck under load spikes. In standard pipelines [18, 34, 54], the full workflow footprint is often 1.7× to 4× larger than the base model alone. Consequently, scaling the entire monolith incurs significant overhead: our experiments on NVIDIA H800 GPUs reveal that monolithic replication using Diffusers [66] adds up to 80% in loading latency and wastes up to 75% of GPU memory compared to scaling only the bottlenecked component. Similarly, with vLLM-Omni [65] and SGLang-Diffusion [59], scaling an entire Flux-Dev pipeline adds up to 70% and 75% latency, respectively, compared to scaling only Flux-Dev model. L2: Inability to Share Common Models. Monolithic serving enforces strict isolation between workflow instances [7, * SGLang-Diffusion, vLLM-Omni and xDiT explicitly reuse the pipeline de-

sign from Diffusers [16] to serve diffusion workflows [59, 65]. However, they currently provide insufficient support to use adapters in their frameworks [57, 58, 65].

3

0.4

SD3

Workflows

Flux

T5 SD3

VAE C.N.

102 101

10−2

10−1 Latency (s)

14.7

1.0 11.9

10 3.0

0

100

Figure 3. Left: Loading time of workflow scaling and base diffusion model (DM) scaling. Right: Latency-throughput tradeoff of models in a SD3 workflow. Both use H800 GPUs.

0.5 Parallelism=1 Parallelism=2 Adaptive

1.8

SD3

Models

Flux

1.2

1.4 1.6 Latency (s)

1.8

Figure 4. Left: Model sharing reduces request latency. Right: Adaptive parallelization reduces request latency.

15], precluding model sharing. This design is fundamentally inefficient given that production T2I workloads exhibit highly skewed model popularity. Alibaba’s trace analyses [38, 41] indicate that popular backbones (e.g., SDXL [54], SD3 [18], and Flux-Dev [34]) appear in nearly all workflows, while the top 5 ControlNets serve 95% of generation requests. Under monolithic serving, each workflow instance must maintain independent replicas of these massive models (2–24 GiB in FP16 [38]). This redundancy prevents memory multiplexing, resulting in excessive GPU memory consumption, low GPU utilization, and load imbalance across replicas.

configuration. This design effectively addresses the inefficiencies of current monolithic serving systems (§3.1). However, it introduces new challenges for diffusion workflows (§3.2). 3.1

Benefits of Micro-Serving

Per-Model Management. Micro-serving makes each model, rather than the entire workflow, the unit of management. This lets the serving system scale only the bottlenecked model and choose resources according to each model’s latency– throughput tradeoff, directly addressing L1 and part of L3. As a result, the system avoids replicating non-bottleneck components, reduces model-loading overhead, and better matches heterogeneous models to available hardware. In Fig. 3-left, we compare full-workflow scaling with scaling only the base diffusion model on H800 GPUs. Because the diffusion model is the bottleneck, full replication loads other components unnecessarily. Scaling only the diffusion model therefore reduces scaling latency by up to 90%.

L3: Runtime Inefficiency. By encapsulating workflows as opaque black boxes, monolithic serving eliminates systemlevel visibility into internal model dependencies, data flow, and execution logic. This opacity compels the system to enforce rigid, workflow-level resource allocation, forfeiting opportunities for fine-grained runtime optimization. Specifically, because models within a workflow exhibit heterogeneous arithmetic intensities and distinct latency–throughput trade-offs (Fig. 3-right) [56], a static, per-workflow configuration is inherently suboptimal. Furthermore, monolithic systems typically enforce a fixed degree of model parallelism. Unlike automatic tuning strategies [39, 63], this static approach prevents the system from adapting to dynamic workloads or fluctuating GPU availability, leading to significant performance degradation as quantified in §3.1 (Fig. 4-right).

Model Sharing. When different workflows invoke common models [38], micro-serving lets the system share those loaded replicas across workflows, directly addressing L2. Instead of binding a model replica to the workflow that loaded it, the system can multiplex compatible requests onto any resident replica. This reduces redundant replicas and improves load balance, since requests can use identical models already loaded elsewhere in the cluster. To show these benefits, we serve a pair of workflows on two H800 GPUs: one with ControlNet and one without. This setup creates model-sharing opportunities for the text encoders and diffusion models. In Fig. 4-left, we compare request latency with and without model sharing for such workflow pairs, using SD3 and Flux as the base diffusion model in separate experiments. Compared with isolated workflow replicas, multiplexing alreadyloaded models reduces request latency by up to 40% and GPU memory footprint by up to 60%. Furthermore, base diffusion models patched with adapters (e.g., LoRA, see §2.1) can still be shared across requests requiring different LoRAs through efficient patch swapping [38]. We elaborate on this in §7.3.

L4: System Fragility and Maintenance Overhead. Monolithic serving imposes high maintenance overheads by violating modular systems principles. The tight coupling of independent components creates system fragility, where a failure in a single sub-component cascades into a complete workflow failure. This lack of fault isolation complicates debugging, forcing developers to check the entire monolith to identify root causes. Besides, under the monolithic architecture, updating a single component necessitates holistic validation and coordination across the entire workflow, unnecessarily prolonging development and deployment cycles.

3

w/o sharing w/ sharing

CDF

CLIP_G CLIP_L

103

Latency (s)

2.9

2.5 0.0

Base DM

5.2

4.0

Throughput (QPS)

Loading Time (s)

Workflow

5.0

A Case for Micro-Serving

We advocate micro-serving diffusion workflows. Instead of treating an entire workflow as a schedulable unit, microserving decomposes the workflow into independently managed model-execution components and gives the serving system per-model control over scaling, sharing, and runtime

Adaptive Resource Configuration. Micro-serving exposes model dependencies and execution choices to the runtime, which lets the system tune resource configurations per model instead of fixing one for the entire workflow. This directly addresses L3. Automatic model parallelism is one example. 4

We deploy three SD3 workflows [18] on four H800 GPUs under three settings: Parallelism=1 fixes the parallelism degree at 1 and leaves acceleration opportunities unused; Parallelism=2 always applies latent parallel proposed in [19, 38], which speeds up image generation without quality loss but requires a pair of GPUs (§2.1); and Adaptive selects the parallelism degree at runtime according to GPU availability. As shown in Fig. 4-right, the tradeoff is clear: Parallelism=1 yields consistently higher latency because it forgoes parallel speedup, whereas Parallelism=2 introduces queuing when later requests wait for an available GPU pair, producing a stepped CDF curve. Compared with the static configurations, Adaptive’s automatic parallelization tuning accelerates average request serving by 1.3× and 1.2×, respectively.

Workflow Composition

Coordinator Workflow Registry

Executor Data Store

Graph Compiler workflow_id: “sdxl_txt2img”, prompt : “A cat …”, seed: 0, …

Invoke

Scheduler Enqueue

Data Transfer

Dis pa tch

Workflow requests

Executor Data Store

Figure 5. An overview of LegoDiffusion. Finally, micro-serving only pays off if the scheduler can exploit diffusion-specific opportunities at runtime. Existing systems do not directly support model-granular scaling, cross-workflow model sharing, adaptive parallelization, or SLO-aware admission control. These challenges drive our design of a diffusion-specific programming interface, graph compiler, runtime/data engine, and scheduler in §4 and §5.

Modular Development. Micro-serving also improves modularity, directly addressing L4. By serving workflow models as independent components, it gives developers clearer failure boundaries and cleaner update paths. They can modify, validate, and debug one model without reasoning about the entire workflow, and they can test individual components in isolation. This reduces maintenance overhead and makes bugs easier to localize.

3.2

Register

4

System Design of LegoDiffusion

In this section, we present LegoDiffusion, an efficient serving system for micro-serving diffusion workflows. LegoDiffusion comprises four components: a programming interface for workflow composition and model integration (§4.1), a graph compiler that decomposes workflows into executable nodes and applies diffusion-specific optimizations (§4.2), a runtime with a GPU-native data engine for workflow execution (§4.3), and an orchestrator for scheduling and resource management (§5). Together, these components optimize cluster-level serving performance while accommodating existing model acceleration techniques [29].

Challenges of Micro-Serving

Despite these benefits, micro-serving diffusion workflows cannot be built by directly reusing existing microservice systems for analytics, general task runtimes, or LLM agents [14, 40, 47, 60, 63, 70, 81, 83, 87]. Diffusion workflows couple heterogeneous models through iterative denoising, adapter patching, and diffusion-specific parallelism, creating requirements these systems do not target. First, the system needs an abstraction that is expressive for developers yet structured for backend analysis. Analytics DAG systems such as Spark [83] and generic task runtimes such as Ray [47] can express tasks and dependencies, but not adapter–base-model interactions, deferred tensor dependencies, or patching operations (§2.1). Second, the system must compile a workflow into executable model-level tasks while preserving diffusion-specific optimizations. LLM-centric systems such as Parrot and Ayo [40, 63] primarily optimize autoregressive inference through prefix sharing and streamed decoding [93]. Diffusion workflows instead require a compiler that preserves denoising structure and supports caching, asynchronous LoRA loading, and specialized multi-GPU parallelization [4, 19, 36, 38]. Third, the runtime must be GPU-native and diffusionaware. Data analytics and microservice systems largely assume CPU execution and host-memory dataflow [14, 60, 70, 81, 83, 87], whereas diffusion workflows exchange CUDA tensors across GPUs, rely on asynchronous and interleaved data movement, and require a dedicated data engine.

System Overview. Fig. 5 presents an overview of LegoDiffusion. At the frontend, model developers integrate individual models by subclassing a Model base class, and workflow developers compose these models into workflows and register them with the system (○). 1 End users invoke registered workflows (○) 2 by submitting requests with inputs such as textual prompts and random seeds. At the backend, the graph compiler transforms a workflow into a set of loosely coupled workflow nodes (○), 3 which are dispatched by the scheduler across a cluster of distributed executors (○). 4 Each executor owns one GPU and uses an efficient data engine for inter-node communication (○). 5 4.1

Programming Model for Developers

LegoDiffusion’s programming model targets two audiences. Model developers integrate individual models and adapters by subclassing a Model base class; they implement modelspecific logic without reasoning about how their models are composed into a workflow. On the other hand, workflow developers assemble models into end-to-end workflows by instantiating models and invoking them; they do not manually wire a DAG (directed acyclic graph). Instead, LegoDiffusion adopts an implicit workflow programming model: model invocations and the I/O interfaces declared in each 5

Class

Model

Workflow

API

Description

1 # # No need to modify it , invisible to model developers ##

__init__() __call__() setup_io() load() execute() add_patch() rm_patch() __init__() add_input() add_output()

Create a model instance Express a model invocation in the frontend Define model inputs and outputs Load the model in the backend Execute model inference in the backend Attach a patchable adapter to a model Remove a patchable adapter from a model Create a workflow instance Add a workflow input placeholder Add a workflow output placeholder

2 class Model : 3 def __init__ ( self , ** kwargs ) : 4 self . setup_io ()

Table 1. Primitives in LegoDiffusion’s Python library for defining models and composing diffusion workflows. Model subclass are sufficient for the graph compiler (§4.2) to infer the workflow DAG and optimize execution automatically. This contrasts with explicit-DAG systems such as ComfyUI [11], where developers must manually specify every node and edge. Model Integration. To keep pace with the proliferation of models and acceleration techniques [29], LegoDiffusion provides a Model base class that standardizes model and adapter integration while encapsulating all workflow-facing logic. A model developer subclasses Model and implements three methods (Table 1): setup_io() declares the model’s typed inputs and outputs, which are visible to the compiler; load() initializes the model on a given device; and execute() runs inference. Because these are the only methods a model developer writes, model integration is decoupled from workflow construction: a model developer never reasons about how the model is wired into a larger workflow. The base class handles workflow integration automatically. Its __call__() method records each model invocation as a workflow node and derives data dependencies from the I/O interface declared in setup_io(). This separation captures a key design principle: model developers specify what a model consumes and produces; LegoDiffusion uses that specification to place the model into an inferred workflow graph. Fig. 6 illustrates this split with a simplified Flux integration: the base Model class (top) is provided by the framework, while the Flux subclass (bottom) contains only modelspecific code.

5 6 7 8 9 10 11 12 13 14

# store associated weight - patching adapters self . _patches = [] # Make the class callable to create workflow node def __call__ ( self , ** kwargs ) : workflow = WorkflowContext . get_current_workflow () workflow_node = WorkflowNode ( op = self , ** kwargs ) workflow . add_workflow_node ( workflow_node ) return workflow_node . get_outputs ()

15 16 17

@abstractmethod def setup_io ( self ) : pass

18 19 20

def add_patch ( self , patch ) : self . _patches . append ( patch )

21 22

def rm_patch ( self , patch ) :

23 self . _patches . remove ( patch ) 24 25 # # Model developers start here ## 26 class Flux ( Model ) : 27 def setup_io ( self ) : 28 29 30

# define inputs self . add_input ( " latents " , torch . Tensor ) self . add_input ( " prompt_embeds " , torch . Tensor ) # define " deferred " inputs , detailed in Sec . 4.3.2 self . add_input ( " controlnet_inputs " , torch . Tensor , deferred = True ) # define outputs self . add_output ( " noise_pred " , torch . Tensor )

31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

def load ( self , model_path , device ) : model = SD3Transformer2DModel . from_pretrained ( ... ) . to ( device ) return { " transformer " : model } @torch . no_grad () def execute ( self , model_components , ** kwargs ): transformer = model_components [ " transformer "] noise_pred = transformer (** kwargs ) [0] return { " noise_pred " : noise_pred }

Figure 6. A simplified case of integrating Flux with LegoDiffusion. calls within that scope are automatically recorded as workflow nodes. Workflow inputs are declared with add_input(), and intermediate values such as prompt_embeds flow directly between model calls. The loop (line 23) shows that iterative denoising is expressed naturally in Python, while LegoDiffusion captures the structure needed for backend execution. After construction, the workflow developer registers the workflow with LegoDiffusion for later invocation by end users.

Workflow Composition. Workflow developers compose workflows declaratively: they declare workflow inputs and outputs, instantiate models, and invoke them. They never explicitly wire a DAG; the graph compiler (§4.2) infers all data dependencies from model invocations and the I/O interfaces declared in setup_io(), then optimizes the resulting graph. Fig. 7 shows a workflow for the Flux example in Fig. 1bottom. Creating a Workflow instance (line 2) establishes a scope (maintained by WorkflowContext); subsequent model

4.2

Graph Compiler

The graph compiler (○) 3 lowers a registered workflow into a topologically sorted DAG of schedulable workflow nodes and applies optimization passes before execution. 6

inference (§2.1). When the compiler detects an add_patch() attachment on a model, it rewrites the workflow graph by inserting (1) an initial node that triggers asynchronous LoRA loading and (2) a check node after each diffusion-model node that tests whether the adapter is ready to be patched in. The workflow developer writes only add_patch(lora); the compiler can insert the asynchronous-loading machinery automatically.

1 # create a workflow instance 2 workflow = Workflow ( name = " flux_txt2img_workflow " ) 3 # initialize models . All inherit the Model class 4 latents_generator = LatentsGenerator () 5 text_enc = FluxTextEncoder ( model_path = model_path ) 6 flux = Flux ( model_path = model_path ) 7 controlnet = ControlNet ( model_path = controlnet_path ) 8 lora = LoRA ( model_path = lora_path ) 9 vae = FluxVAE ( model_path = model_path ) 10 # initialize input placeholders for the workflow 11 seed = workflow . add_input ( name = " seed " , data_type = int ) 12 prompt = workflow . add_input ( name = " prompt " , data_type = str ) 13 num_denoising_steps = workflow . add_input ( name = " 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

4.3

num_denoising_steps " , data_type = int ) ref_image = workflow . add_input ( name = " ref_image " , data_type = Image ) # add_patch () registers a LoRA adapter with the flux , flux . add_patch ( lora ) # Invoke models and establish their I / O dependencies . # Model invocation is expressed via __call__ () . latents = latents_generator ( seed ) prompt_embeds = text_enc ( prompt ) ref_image = vae ( image = ref_image , mode = " encode " ) # Perform iterative denoising computation . for i in range ( num_denoising_steps ) : controlnet_outputs = controlnet ( latents , prompt_embeds , ...) noise_pred = flux ( latents , prompt_embeds , controlnet_outputs , ...) latents = denoise ( noise_pred , latents ) output_img = vae ( latents , mode = " decode " ) workflow . add_output ( output_img , name = " output_img " )

LegoDiffusion’s Runtime

4.3.1 Micro-Serving Control Plane. Given the topologically sorted DAG from the compiler, the runtime executes each request through a node-level control plane. Under microserving, each workflow node is independently schedulable on any executor once its non-deferred inputs are satisfied; the runtime can also configure parallelism and resource allocation per node based on the node’s encapsulated model and available hardware. Request Execution Lifecycle. Workflows are compiled once at registration time; the compiled DAG is instantiated only when a request arrives [3, 48, 71, 83] (○ 2 in Fig. 5). The control plane enqueues all root nodes (those with no upstream dependencies) and enters a dispatch loop. In each cycle, the scheduler selects ready nodes and dispatches them to executors (○). 4 When an executor completes a node, it reports the result to the control plane, which marks downstream nodes whose inputs are now satisfied as ready. This loop continues until all nodes complete and the workflow output is returned to the end user. The scheduler’s placement and batching policies are detailed in §5.

Figure 7. A simplified diffusion workflow using Flux [34]. DAG Construction. As described in §4.1, each model invocation during workflow composition is recorded as a workflow node with typed I/O declared in setup_io(). The compiler resolves data dependencies among these nodes and produces a topologically sorted DAG. Topological order guarantees correct execution and exposes optimization opportunities: nodes without mutual dependencies can run in parallel to reduce latency, and nodes that invoke the same model can be batched to improve throughput (§4.3).

4.3.2 Distributed Data Engine. Micro-serving introduces frequent data movement between nodes. In typical diffusion workflows such as SD3 [18] and Flux models [34], CUDA tensors account for over 99% of transferred data (see Fig. 11right); for example, an SDXL workflow with a single ControlNet transfers 5.3 GiB [38]. Host-memory staging through PCIe is prohibitively slow at this scale. To avoid CPU staging, LegoDiffusion deploys a distributed data engine with a per-executor local data store (Fig. 5). The stores are built on NVSHMEM [49], which provides onesided GPU communication over NVLink and RDMA, enabling zero-copy sharing within an executor and high-speed transfers across executors.

Optimization Passes. After constructing the DAG, the compiler applies a series of graph-rewriting passes. Each pass pattern-matches on node properties (e.g., model type, adapter attachments) and may insert, remove, or replace nodes. This design makes the compiler extensible: adding a new optimization requires only a new pass, without modifying the core lowering logic. The compiler also applies per-model optimizations such as torch.compile() within individual nodes. Below, we illustrate two diffusion-specific passes and evaluate them in §7.4. 1) Approximate caching [4] reduces the number of denoising steps by initializing from a pre-cached image of a similar prompt instead of random noise (§2.1). When a prompt cache is configured, the compiler replaces the random-latentinitialization node with a cache-lookup node, requiring no changes to the workflow definition. 2) Asynchronous LoRA loading [38] overlaps LoRA adapter retrieval with the early stages of diffusion model

Data Fetch Modes. Diffusion workflows exhibit diverse data-movement patterns due to their parallelization strategies (§2.1). LegoDiffusion supports two fetch modes: eager, where an input must be ready before a node begins execution, and deferred, where a node starts execution and fetches the input at the point of consumption. 1) Eager data fetch. By default, inputs are fetched eagerly: a node cannot begin until all its eager inputs are available. In Fig. 8, the text-encoder node on Executor 1 produces a 7

Tensor metadata Metadata transfer

Tensors Tensor transfer

Coordinator Text encoder Executor 1 infer.

Executor 2

Deferred inputs Produce tensor

Data store Add tensor Time

5

Workflow Node Scheduling

The scheduler is the component that translates micro-serving into runtime actions (○ 4 in Fig. 5). It sits between the compiled workflow DAGs and the executor cluster, maintaining a global queue of workflow nodes and making three online decisions in each scheduling cycle: (1) which same-model nodes to batch together and which executors to route them to, exploiting model sharing across workflows; (2) how many GPUs to allocate per batch, adapting parallelism to current resource availability; and (3) whether to admit or reject incoming requests to preserve SLO attainment. Algorithm 1 summarizes the scheduling loop; the remainder of this section describes how the scheduler makes each decision. To make these decisions, the scheduler maintains two key data structures that the runtime keeps up to date. A model state table records, for every executor, which models are currently loaded in GPU memory. Executors piggyback their model states on node-completion notifications to the coordinator, so the table is updated without extra RPCs. A set of per-model latency profiles, collected offline, provides stable estimates [73] of data-fetch time, model-loading time, and inference time for each model under various batch sizes and parallelism degrees. Following prior work [10, 37, 39, 55, 88], the scheduler orders the ready queue by first-come-firstserve (FCFS). For nodes with the same arrival time (e.g., nodes from the same request), it further prioritizes those at shallower depths in the DAG. Since optimal ordering requires foreknowledge of future arrivals [23, 88], FCFS is a simple, neutral baseline that isolates LegoDiffusion’s gains from scheduling policy. The scheduler is a pluggable module; other policies can be substituted.

Flux infer.

Layer0 Layer0

C.N. infer.

Eager fetch: Executer 2 fetches prompt embedding( ) from Executer 1 Deferred fetch: Executer 1 fetches ControlNet output( )from Executer 2

Figure 8. Illustrating data fetch. For simplicity, we primarily illustrate tensor fetch process in data store. C.N.: ControlNet. prompt embedding (○) 1 and places it in its local data store (○). 2 When the coordinator schedules the downstream ControlNet node on Executor 2, it forwards the embedding’s metadata (○). 3 Executor 2 uses this metadata to fetch the tensor into its own store (○). 4 The ControlNet node then reads the embedding from its local store (○) 5 and begins execution—only after the fetch completes, a guarantee enforced by eager fetching. 2) Deferred data fetch. Model developers can mark an input as deferred (line 32 in Fig. 6), indicating that it needs not be ready when a node starts inference. A deferred input is implemented as a fetch function invoked at the point of consumption: it returns immediately if the data is available, or blocks until the data arrives. This mode is tailored to diffusion workflows where ControlNet computation is interleaved with the base model (Fig. 1middle). At runtime (Fig. 8), the ControlNet output consumed mid-way through Flux inference is marked as a deferred input (dashed tensor, ○). 6 Flux begins execution without it. When Flux reaches the consumption point, the ControlNet output has been produced and placed in Executor 2’s store (○); 7 its metadata is forwarded to Executor 1 (○), 8 which fetches the tensor into its local store (○) 9 for Flux to consume. Without deferred fetching, Flux could not start until ControlNet completes, eliminating the parallelism between the two models. Note that tensor metadata, including a tensor’s pointer, is tiny (on the order of KiB). Executors can piggyback it on node-completion notifications, allowing the coordinator to track global tensor placements with little overhead.

5.1

Cross-Workflow Batching and Model Sharing

In each scheduling cycle, the scheduler pops the FCFS-earliest node 𝑛ℎ𝑒𝑎𝑑 from the ready queue and inspects its model field. It then scans the remaining ready nodes for any that reference the same model, regardless of the originating workflow, and groups them into a single batch of up to 𝐵𝑚𝑎𝑥 entries. The per-model 𝐵𝑚𝑎𝑥 is determined offline by profiling batching efficiency: beyond a model-specific threshold, larger batches increase latency with diminishing throughput gain [10]. Because matching is by model identity rather than by workflow, a single batch may contain nodes from multiple workflows— this is how the scheduler realizes model sharing (§3.1). After forming a batch, the scheduler must select an executor (Line 13 –17). For each candidate executor 𝑒, it computes a latency score from three profiled components: (1) 𝐿𝑑𝑎𝑡𝑎 , the cost of fetching the batch’s input tensors from their producing executors via the data engine (§4.3.2); (2) 𝐿𝑙𝑜𝑎𝑑 , the cost of loading the required model into GPU memory; and (3) 𝐿𝑖𝑛𝑓 𝑒𝑟 , the estimated inference time for the batch. The model state table makes 𝐿𝑙𝑜𝑎𝑑 zero for any executor that already hosts the required model, so the scoring function naturally routes

Design Properties. The data engine is transparent to both model developers and workflow developers: once the compiler lowers a workflow into nodes, the engine automatically orchestrates all inter-node data movement. All intermediate data is immutable—tensors produced during diffusion workflow execution are consumed once and never updated [38, 66]—which obviates consistency protocols and simplifies fault tolerance. The engine reclaims tensors as soon as no downstream node requires them, reducing memory pressure. If an executor fails, LegoDiffusion reconstructs lost data by re-executing the affected nodes, following a similar approach to prior cluster computing frameworks [47, 81, 83]. 8

scheduler chooses the parallelism degree 𝑘 per batch by a work-conserving heuristic: it sets 𝑘 = min(|𝐸𝑎𝑣𝑎𝑖𝑙 |, 𝑘𝑚𝑎𝑥 ), where 𝑘𝑚𝑎𝑥 is the maximum useful parallelism for the model (determined offline). This rule uses all currently available GPUs without waiting for more to free up, maximizing parallelism while avoiding extra queueing delay [39, 63]. Because the scheduler makes this decision per batch, different invocations of the same model can run at different parallelism degrees depending on instantaneous cluster load. The scheduler then selects the 𝑘 lowest-scoring executors, dispatches the batch with a parallelism descriptor, and each executor processes its assigned input shard. We evaluate intra- and inter-node parallelism in §7.3.

Algorithm 1: Scheduling Algorithm while True do // Admission control runs asynchronously (§5.3) 2 Admit or reject arrived requests; // Identify ready nodes 3 𝑄𝑟𝑒𝑎𝑑 𝑦 ← nodes with satisfied dependencies from queue; 4 𝐸𝑎𝑣𝑎𝑖𝑙 ← currently available executors; 5 if 𝑄𝑟𝑒𝑎𝑑 𝑦 = ∅ or 𝐸𝑎𝑣𝑎𝑖𝑙 = ∅ then 6 continue;

1

7 8

9 10

11 12

13 14 15

16 17 18 19

Sort 𝑄𝑟𝑒𝑎𝑑 𝑦 by (arrival time, node depth); 𝑛ℎ𝑒𝑎𝑑 ← 𝑄𝑟𝑒𝑎𝑑 𝑦 .𝑝𝑜𝑝 (0); // Batch same-model nodes (§5.1) 𝐵𝑚𝑎𝑥 ← profiled max batch size for 𝑛ℎ𝑒𝑎𝑑 .𝑚𝑜𝑑𝑒𝑙; 𝐵𝑎𝑡𝑐ℎ ← {𝑛ℎ𝑒𝑎𝑑 } ∪ {𝑛 ′ ∈ 𝑄𝑟𝑒𝑎𝑑 𝑦 | 𝑛 ′ .𝑚𝑜𝑑𝑒𝑙 = 𝑛ℎ𝑒𝑎𝑑 .𝑚𝑜𝑑𝑒𝑙 and |𝐵𝑎𝑡𝑐ℎ | < 𝐵𝑚𝑎𝑥 }; // Choose parallelism degree (§5.2) 𝑘𝑚𝑎𝑥 ← max useful parallelism for 𝑛ℎ𝑒𝑎𝑑 .𝑚𝑜𝑑𝑒𝑙; 𝑘 ← min( |𝐸𝑎𝑣𝑎𝑖𝑙 |, 𝑘𝑚𝑎𝑥 ); // Score and select executors for 𝑒 ∈ 𝐸𝑎𝑣𝑎𝑖𝑙 do 𝐿𝑑𝑎𝑡𝑎 ← CalcDataFetchLatency(𝐵𝑎𝑡𝑐ℎ, 𝑒); 𝐿𝑙𝑜𝑎𝑑 ← 𝑒 hosts 𝑛ℎ𝑒𝑎𝑑 .𝑚𝑜𝑑𝑒𝑙 ? 0 : CalcLoadTime(𝑛ℎ𝑒𝑎𝑑 .𝑚𝑜𝑑𝑒𝑙); 𝐿𝑖𝑛𝑓 𝑒𝑟 ← CalcInferenceTime(𝐵𝑎𝑡𝑐ℎ, 𝑒, 𝑘); 𝑒.𝑠𝑐𝑜𝑟𝑒 ← 𝐿𝑑𝑎𝑡𝑎 + 𝐿𝑙𝑜𝑎𝑑 + 𝐿𝑖𝑛𝑓 𝑒𝑟 ;

5.3

Admitting requests beyond system capacity inflates queueing delays and causes cascading SLO violations. LegoDiffusion prevents this with an early-abort admission policy that leverages micro-serving’s per-node visibility into request progress. When a new request arrives, the scheduler estimates its end-to-end completion time. Because the control plane tracks which nodes of each inflight request have completed, the scheduler can compute, for every inflight request, the sum of profiled latencies along its remaining critical path. It admits the new request only if the estimated completion time—accounting for current queueing depth—satisfies the request’s latency SLO. Otherwise, the request is rejected immediately, preserving resources for already-admitted requests. This early-abort policy is feasible only under microserving: in monolithic serving, the system has no visibility into sub-workflow progress and cannot estimate remaining work at fine granularity. The admission control runs asynchronously and does not block the scheduling loop. We quantify its effect on SLO attainment in §7.3.

𝐸𝑡𝑎𝑟𝑔𝑒𝑡 ← top 𝑘 from 𝐸𝑎𝑣𝑎𝑖𝑙 with min scores; Dispatch 𝐵𝑎𝑡𝑐ℎ to 𝐸𝑡𝑎𝑟𝑔𝑒𝑡 (triggers model load if needed);

batches to executors with warm models. When no executor hosts the model, the scheduler selects the executor with the lowest total score and triggers a model load—loading only the single needed model, not the entire workflow, unlike monolithic scaling (L1 in §2.2). We evaluate the throughput and latency gains from model sharing in §7.3. 5.2

SLO-Aware Admission Control

Adaptive Parallelism

LegoDiffusion exploits two forms of diffusion-specific parallelism (§2.1) through scheduling decisions.

6

Inter-Node Parallelism. When the compiler produces a DAG in which two nodes have no eager data dependency— for example, a ControlNet and its corresponding base-model node connected only by a deferred input (§4.3.2)—both nodes enter 𝑄𝑟𝑒𝑎𝑑 𝑦 simultaneously once their non-deferred inputs are satisfied. The scheduler dispatches them to separate executors in the same or adjacent loop iterations, so they execute concurrently. Runtime data exchange between the two nodes is handled by deferred data fetch: the base model begins execution immediately, and retrieves the ControlNet output mid-inference when it becomes available (Fig. 2-right).

Implementation

We have implemented LegoDiffusion with a FastAPI [20] frontend and a distributed GPU-based inference engine. The frontend (approx. 1,000 LoC in Python) exposes an intuitive programming interface for users to compose and register diffusion workflows (§4.1). Users can invoke registered workflows with customized image generation parameters, such as prompts and reference images, similar to the OpenAI API [51]. We currently support diffusion workflows for popular models including the SD3 family [18] and Flux family [34]. LegoDiffusion’s backend runtime consists of a coordinator and distributed executors (Fig. 5), totaling 4,000 lines of Python code. The data engine is implemented in 1,000 lines of C++/CUDA code using NVSHMEM [49]. Aside from CUDA tensors, communication between the coordinator and distributed executors is facilitated via ZeroMQ [2].

Intra-Node Parallelism. A single model invocation can also be split across multiple GPUs via latent parallelism [19, 36, 38], which partitions the input tensor and distributes the shards to 𝑘 executors for parallel inference (Fig. 2-left). The 9

• Diffusers represents a static deployment strategy. Each workflow is executed monolithically and statically bound to dedicated GPUs [15, 66]. It cannot share models across workflows, adapt parallelism at runtime. • Diffusers-C implements a swap-based serving strategy by adapting Clockwork [23] to Diffusers. Leveraging the predictable end-to-end latency of diffusion workflows, it treats each monolithic workflow as a swappable DNN model unit, dynamically loading and unloading workflows into GPU memory on demand. Because the swap unit is an entire workflow, it cannot share individual models across workflows or adapt parallelism within a workflow. • Diffusers-S incorporates the planning-and-scheduling framework of Shepherd [88] to orchestrate instances, modeling each monolithic diffusion workflow as a distinct model unit. Like Diffusers-C, it schedules whole workflows and thus cannot exploit per-model sharing or adaptive parallelism.

Table 2. Evaluation Settings. We use workflows of representative diffusion models. S1–S4 represent single-model deployments, each including three workflow variants: a Basic workflow (text encoders, diffusion model, and decoder), plus two using adapters (Basic + C.N. 1 and Basic + C.N. 2). S5–S6 represent mixed-model deployments. C.N.: ControlNet. Setting

Diffusion Model

Workflow

Single-model Deployments (3 workflows each) S1 S2 S3 S4

SD3 [18] SD3.5-Large [62] Flux-Schnell [34] Flux-Dev [34]

(Basic, +C.N. 1, +C.N. 2) (Basic, +C.N. 1, +C.N. 2) (Basic, +C.N. 1, +C.N. 2) (Basic, +C.N. 1, +C.N. 2)

Mixed-model Deployments (6 workflows each) S5 S6

7

SD3 + SD3.5-Large Flux-Schnell + Flux-Dev

S1’s + S2’s S3’s + S4’s

Evaluation

We evaluate LegoDiffusion with the following highlights: • LegoDiffusion outperforms state-of-the-art baselines, sustaining up to 3× higher request rates, satisfying 6× more stringent SLOs, reducing GPU requirements by up to 3×, and tolerating 8× higher burst traffic, all while maintaining over 90% SLO attainment (§7.2). • Microbenchmarks isolate the contribution of each scheduling mechanism and validate compatibility with emerging diffusion optimizations (§7.3, §7.4). • LegoDiffusion introduces negligible system overhead (§7.5). 7.1

Workloads. We use a real-world T2I production trace [38]. To rigorously evaluate under diverse conditions, we vary request arrival rates, SLO targets, traffic burstiness, and testbed sizes, effectively simulating a wide spectrum of real-world traffic patterns and performance requirements. Metrics. Our primary metric is SLO attainment: the fraction of requests completed within their specified latency deadline. We set the default deadline to 2× the solo inference latency of each workflow (SLO Scale = 2), which is tight given that any queueing or resource contention will cause violations. Unlike prior works [4, 19, 36, 38], LegoDiffusion does not alter the computation performed during diffusion inference and therefore requires no evaluation of the image quality.

Experimental Setup

Diffusion Workflows and Testbed. We use 12 diffusion workflows composed from four popular base models: SD3 [18], SD3.5-Large [62], Flux-Dev [34], and Flux-Schnell [34]. They exhibit diverse computational characteristics, with parameter counts spanning 2.5B to 12B and denoising steps ranging from 4 to 50. In Table 2, we categorize these workflows into six evaluation settings (S1–S6) to assess system performance under varying degrees of workload heterogeneity. We use a real testbed of 8 to 32 NVIDIA H800 GPUs to evaluate performance and a 256-GPU simulator to analyze scalability.

7.2

End-to-End Performance

As Fig. 9 shows, LegoDiffusion consistently outperforms all baselines across six settings, four traffic dimensions, and a range of testbed sizes. At high request rates, LegoDiffusion achieves over 90% SLO attainment in settings where the strongest baseline drops below 3%. SLO Attainment vs. Rate. We evaluate LegoDiffusion and the baselines by varying the request rate while keeping the SLO scale (2.0) and traffic burstiness fixed. As shown in Fig. 9 (a)–(f) and (j), LegoDiffusion consistently achieves higher SLO attainment across varying rate scales. Compared to Diffusers-S, the strongest baseline, LegoDiffusion sustains up to a 3× higher request rate while meeting a 90% SLO attainment target. At low request rates, all systems achieve high SLO attainment; however, as the rate increases, baseline performance plunges due to coarse-grained workflow scaling and inability to share common models (§2.2). In contrast, LegoDiffusion’s gains stem from two mechanisms: model sharing (§5.1) enables batching nodes from all three workflows onto shared model replicas, avoiding redundant loading; adaptive parallelism (§5.2) further reduces

Baselines. We primarily compare LegoDiffusion with Diffusers, the most representative monolithic-serving system (§2.2)[15, 66], as our goal is to compare micro-serving with the prevailing monolithic design. While other systems [19, 59, 65] support more parallelism methods and high-performance kernels, they largely inherit Diffusers’s monolithic pipeline design and these optimizations are orthogonal to LegoDiffusion. To compare against a broader monolithic design space, we include monolithic-serving system variants by adopting techniques from multi-model serving systems [23, 88] for workflow orchestration. For a fair comparison, the baselines use FCFS scheduling and workflow-level admission control. 10

SLO Attainment (%) SLO Attainment (%)

Diffusers

(a) S1 @ 8 GPUs

100 90

(b) S2 @ 8 GPUs

Diffusers-C

Diffusers-S

LegoDiffusion

(c) S3 @ 8 GPUs

(d) S4 @ 8 GPUs

(e) S5 @ 16 GPUs

50 0

0.5

100 90

2.0 4.0 Rate Scale

8.0 0.25

(g) S6 @ 16 GPUs

0.5 0.7 Rate Scale

0.9 1.0

(f) S6 @ 16 GPUs

1.0

0.5

0.75 1.0 Rate Scale

2.0 0.1

(h) S6 @ 16 GPUs

0.3

0.5 0.75 Rate Scale

1.0 1.0 2.0 4.0 6.0 8.0 10.0 12.0 2.0 SLO Scale

4.0 8.0 CV Scale

16.0 8.0

1.5 2.0 Rate Scale

4.0 1.0

16.0 24.0 Number of GPUs

32.0 0.5

(i) S6 @ 8-32 GPUs

2.0 4.0 Rate Scale

8.0

(j) S6 @ 32 GPUs

50 0 0.25

1.0

2.0 4.0 Rate Scale

8.0

Normalized Latency

per-request latency at low-to-moderate rates by distributing inference across idle GPUs. Next, we present evaluations that vary the SLO scale, traffic burstiness, and testbed size, respectively. We focus on the Flux model family (S6), widely adopted models [31] representative of recent advances in the field.

1-GPU

1.0

2-GPU 0.75

0.53

0.5 0.0

0.90

0.56

SD3.5 Flux-S SD3.5 Flux-S

Intra-Node Parallel Inter-Node Parallel

SLO Attainment (%)

Figure 9. End-to-end performance across six settings (S1–S6 in Table 2), evaluated under varying request traffic rates (a–f, j), SLO requirements (g), traffic burstiness (h), and testbed sizes (i). W/ A.C. W/o A.C.

75

63 23.7

25 0

69

50

50 44

0.4

1.0

5.5

S1 S2 S3 S4 (RS=8) (RS=2) (RS=8) (RS=0.9)

Overhead (ms)

Create (NVLink) Create (RDMA) Fetch (NVLink) Fetch (RDMA)

100 10−1

6.6 1.6 25M

10−2 1K

8K 64K 512K 4M 32M256M Block Size (Bytes)

CDF

Figure 10. Left: Normalized latency of LegoDiffusion across different numbers of available GPUs with intra-/internode parallelism (§5). Flux-S: Flux-Schnell. Right: Effectiveness of admission control (A.C.) in settings S1-4. RS: Rate Scale.

SLO Attainment vs. SLO Scale (16 GPUs). In Fig. 9(g), we fix the rate scale at 1.0 and evaluate using the original production trace. Even at a strict SLO scale of 1.0, LegoDiffusion achieves substantially higher SLO attainment than the baselines. At this scale, the deadline is tight enough that only intra-node parallelism—splitting the base model across two GPUs (§5.2)—can bring per-request latency below the target; baselines, which run each workflow on a single GPU, cannot meet this deadline regardless of scheduling policy. At an SLO scale of 2.0, LegoDiffusion satisfies the SLO for over 90% of requests, whereas the baselines require an SLO scale of 12.0 to reach the same level. We observe a sharp increase in baseline SLO attainment when the SLO scale rises from 1.0 to 2.0, because this relaxation begins to absorb the inherent overheads of monolithic serving (§2.2). Beyond that point, baseline improvements are gradual. In contrast, LegoDiffusion benefits more effectively from relaxed SLOs, achieving 1.4× higher SLO attainment than the strongest baseline at an SLO scale of 4.0.

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

SD3 Flux-Dev

4

0.5M 25M 104 102 106 Tensor Size (Bytes)

Figure 11. Left: Data fetching latency of varying sizes of tensor blocks. Right: The distribution of tensor block sizes found in typical SD3 and Flux workflows. control (§5.3) then protects admitted requests during the spike itself by rejecting those that would violate their SLOs. SLO Attainment vs. Testbed Size. Finally, in Fig. 9 (i), we fix the rate scale at 0.5 and the SLO scale at 2.0, varying the testbed size under the original production trace. LegoDiffusion requires up to 3× fewer GPUs to achieve a 90% SLO attainment target. This high resource efficiency is driven by our micro-serving design. Unlike monolithic serving systems that rigidly partition resources at the workflow level, LegoDiffusion enables fine-grained model scaling and serving, as well as model sharing, which effectively treats all GPUs as a unified pool. This eliminates resource over-provisioning, and ensures every available GPU cycle is utilized efficiently.

SLO Attainment vs. CV (16 GPUs). In Fig. 9 (h), we fix the rate scale at 0.25 and the SLO scale at 2.0. Following prior work [23, 39], we slice the original trace into time windows and fit the arrivals to a Gamma Process parameterized by the coefficient of variation (CV). Scaling the CV and resampling allows us to control traffic burstiness. Higher CVs indicate burstier traffic, which exacerbates queuing delays and increases SLO violations. As shown, LegoDiffusion gracefully handles highly bursty traffic, sustaining high attainment even at an 8× larger CV compared to the baselines. When traffic subsides, the work-conserving parallelism heuristic (§5.2) drains the queue faster by assigning more GPUs per request, creating headroom before the next burst. Admission

7.3

Microbenchmarks

We isolate the benefits of LegoDiffusion’s micro-serving (§3). Model Sharing. Diffusion models patched with LoRAs can be shared across requests, significantly reducing the memory 11

respectively. The optimization achieves speedups of 1.13× and 1.43× with its original implementation on Diffusers, and comparable speedups of 1.17× and 1.42× on LegoDiffusion, evidencing LegoDiffusion’s effective support for the optimization. Async LoRA Loading. We implement Katz’s [38] asynchronous LoRA loading design in LegoDiffusion and evaluate it against the original Diffusers implementation, using SDXL [54] with a papercut-style LoRA [64]. Our implementation reduces LoRA loading overhead from 0.5 seconds to 0.05 seconds, matching the results reported in [38].

Table 3. Effective LOC and adaptive-runtime support. Technique

LOC / (Support adaptive adjustment?) Katz [38] xDiT [19] LegoDiffusion

Latent parallel ControlNet parallel Async LoRA loading

92 (No) 127 (No) 182 (Yes)

68 (No) N.A. N.A.

74 (Yes) 79 (Yes) 61 (Yes)

footprint and latency overhead compared to loading a new model. We validate this using SD3 and a typical LoRA [42]. While the LoRA occupies 886 MiB of memory and takes 100 ms for swapping [38], this saves the 3.9 GiB of memory and 430 ms of latency incurred by loading a fresh SD3 model. Intra-Node Parallelism. LegoDiffusion natively integrates latent parallelism [19, 36, 38] with intra-node parallelism (§5.2), enabling accelerated diffusion model inference across two GPUs. As shown in Fig. 10-left, our intra-node implementation achieves a speedup of up to 1.9×. This aligns with findings in [36, 38] and validates LegoDiffusion’s capability to support state-of-the-art optimizations. Inter-Node Parallelism. As discussed in §4.3.2, LegoDiffusion implements a deferred data fetch mechanism to enable ControlNet parallelization [38], a key form of internode parallelism described in §5.2. As shown in Fig. 10-left, LegoDiffusion’s inter-node parallelism accelerates workflow execution across different models by up to 1.3×, consistent with results in [38]. Note that the gains with Flux models are limited because their ControlNets are small (only 6% of the base model size) and have negligible latency compared to the base model. Admission Control. We evaluate the effectiveness of LegoDiffusion’s admission control (§5.3) in optimizing SLO attainment. In Fig. 10-right, enabling admission control across the four settings (Table 2) prevents system overload under high request rates. By proactively aborting requests that are destined to violate SLOs, LegoDiffusion increases SLO attainment from a mere 0.4% to 44% in setting S1. Programmability. LegoDiffusion provides an intuitive programming model for composing complex workflows. Following [93], we quantify developer productivity using effective Lines of Code (LoC). We compare LegoDiffusion against Katz [38] and xDiT [19], two popular diffusion model serving engines that support parallel acceleration. As shown in Table 3, LegoDiffusion requires comparable or lower implementation effort to express these optimizations, while additionally supporting adaptive runtime behavior that the baselines do not. 7.4

7.5

System Overhead

Execution Overhead. Micro-serving introduces additional overhead due to inter-node communication and controlplane coordination. We quantify this overhead by comparing LegoDiffusion against monolithic baselines on four workflows: SD3, SD3.5-Large, Flux-Dev, and Flux-Schnell. Across all cases, the maximum end-to-end overhead is 150 ms. Given that these diffusion workloads typically take 2–20 seconds to complete, this additional cost is negligible. Control-Plane Scalability. To show LegoDiffusion’s control plane remains efficient at large scale, we conduct simulationbased experiments on a 256-GPU setup under high concurrency, with 500 inflight requests. Across two representative workloads, Flux-Dev and SD3.5-Large, the coordinator accounts for only 3.4% and 2.7% of total execution time, respectively, indicating that the control plane does not emerge as the dominant bottleneck at this scale. Data Transmission Latency. We further isolate the cost of intermediate tensor movement in Fig. 11. The left panel shows the latency of tensor serialization and transmission over a range of tensor sizes, and the right panel reports the actual intermediate tensor sizes produced by SD3 and Flux-Dev workflows with ControlNet. Even for the largest intermediate tensors, transmission latency remains below 1 ms, confirming that the inter-GPU bandwidth is not a bottleneck for LegoDiffusion’s fine-grained execution model.

8

Discussion and Related Works

Scalability and Fault Tolerance. In LegoDiffusion, one coordinator manages 𝑁 executors (Fig. 5). To avoid a coordinator bottleneck as 𝑁 grows, LegoDiffusion shards executors across multiple coordinators, each managing a disjoint subset of workflows that share models, preserving sharing opportunities. A cluster management service [1, 32] handles coordinator discovery and failure detection. Executor failures are tolerated naturally: the coordinator reassigns affected nodes to other executors.

Case Study

We next show that LegoDiffusion supports emerging optimizations tailored for diffusion models described in §4.2. Approximate Caching. In LegoDiffusion, we implement Nirvana’s [4] approximate caching optimization (§2.1). Following prior work [4, 38], we use a SDXL workflow and configure to reduce 20% and 40% denoising computation,

Diffusion Model Serving Systems. Existing serving systems [7, 15, 59, 65] follow a monolithic design with limited adapter support [57, 58, 65] (§3.2). Several works accelerate 12

individual workflow execution: Nirvana [4] reduces denoising steps via cached images; DistriFusion [36] and xDiT [19] exploit multi-GPU parallelism; Katz [38] parallelizes ControlNets and asynchronously loads LoRAs; TetriServe [44] and TridentServe [73] adapt sequence parallelism for latency SLOs. However, several of these [4, 36, 44, 73] lack adapter support prevalent in production [38, 41], and none target cluster-level multi-workflow deployment. LegoDiffusion’s micro-serving approach is complementary (§4.2, §7.4).

[6] Sohaib Ahmad, Hui Guan, Brian D. Friedman, Thomas Williams, Ramesh K. Sitaraman, and Thomas Woo. 2024. Proteus: A highthroughput inference-serving system with accuracy scaling. In Proc. ACM ASPLOS. [7] BentoML. 2025. comfy-pack: Serving ComfyUI Workflows as APIs. https://www.bentoml.com/blog/comfy-pack-serving-comfyuiworkflows-as-apis. [8] Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, Yuening Zhu, Qingliang Ou, Jiaqi Liao, Xianglin Chen, Zhiyuan Ai, Yongwei Wu, and Mingxing Zhang. 2025. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models. In Proc. SOSP. [9] Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. 2025. Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference. In Proc. SOSP. [10] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2024. Punica: Multi-tenant LoRA serving. In Proc. MLSys. [11] ComfyUI. 2025. ComfyUI: The most powerful and modular visual AI engine and application. https://github.com/comfyanonymous/ComfyUI. [12] ComfyUI. 2025. Understand the concept of a node in ComfyUI. https: //docs.comfy.org/essentials/core-concepts/nodes. [13] Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A low-latency online prediction serving system. In Proc. USENIX NSDI. [14] Jeffrey Dean and Sanjay Ghemawat. 2004. MapReduce: Simplified Data Processing on Large Clusters. In Proc. OSDI. [15] HuggingFace Diffusers. 2025. Create a server. https://github. com/huggingface/diffusers/blob/main/docs/source/en/usingdiffusers/create_a_server.md. [16] HuggingFace Diffusers. 2025. Philosophy. https://huggingface.co/ docs/diffusers/en/conceptual/philosophy. [17] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. In Proc. ICML. [18] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In Proc. ICML. [19] Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. 2024. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. arXiv preprint arXiv:2411.01738 (2024). [20] FastAPI. 2025. FastAPI. https://github.com/fastapi/fastapi. [21] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: LowLatency Serverless Inference for Large Language Models. In Proc. OSDI. [22] Shiwei Gao, Qing Wang, Shaoxun Zeng, Youyou Lu, and Jiwu Shu. 2025. WEAVER: efficient multi-LLM serving with attention offloading. In Proc. ATC. [23] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance predictability from the bottom up. In Proc. USENIX OSDI. [24] Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R. Das. 2022. Cocktail: A multidimensional optimization for model serving in cloud. In Proc. USENIX NSDI. [25] Yongjun He, Haofeng Yang, Yao Lu, Ana Klimović, and Gustavo Alonso. 2025. Resource multiplexing in tuning and serving large language models. In Proc. ATC.

Other Model Serving Systems. Prior work on model serving has improved latency [13, 68], throughput [6, 76], and resource efficiency [24, 67, 69, 75, 85] across DNNs and LLMs [5, 8, 9, 17, 22, 25, 28, 45, 50, 61, 72, 77, 78, 80, 82, 84, 92]. LegoDiffusion complements them by focusing on text-toimage serving, which has distinct computational and workflow characteristics. Prior work on online multi-model serving has further explored model placement [39, 88], request scheduling [23, 88], and dynamic scaling [21, 86]. In §7, we integrate representative techniques from them with existing diffusion workflow serving systems [15]. Despite their effectiveness, LegoDiffusion outperforms with its micro-serving approach, which fundamentally addresses the limitations of monolithic serving. Also, LegoDiffusion is compatible with these optimizations.

9

Conclusions

We presented LegoDiffusion, an efficient micro-serving system for diffusion workflow. LegoDiffusion has three key designs: (1) a programming model that transforms workflow compositions into loosely coupled nodes; (2) a specialized runtime and data plane that streamline data communication to facilitate micro-serving; and (3) a scheduler that realizes the benefits of micro-serving at the cluster level. Collectively, LegoDiffusion outperforms existing monolithic serving systems, sustaining up to 3× higher request rates and tolerating 8× higher burst traffic, with the same performance requirements.

References [1] 2025. Apache ZooKeeper. https://zookeeper.apache.org/. [2] 2025. ZeroMQ. https://github.com/zeromq/pyzmq. [3] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proc. OSDI. [4] Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. 2024. Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models. In Proc. USENIX NSDI. [5] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In Proc. OSDI. 13

[26] Jonathan Ho and Tim Salimans. 2021. Classifier-Free Diffusion Guidance. In Proc. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications. [27] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. ICLR. [28] Zhengding Hu, Vibha Murthy, Zaifeng Pan, Wanlu Li, Xiaoyi Fang, Yufei Ding, and Yuke Wang. 2025. HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows. In Proc. SOSP. [29] HuggingFace. 2025. Accelerate inference of text-to-image diffusion models. https://huggingface.co/docs/diffusers/en/tutorials/fast_ diffusion. [30] HuggingFace. 2025. HuggingFace Models. https://huggingface.co/ models?pipeline_tag=text-to-image&sort=downloads. [31] HuggingFace. 2025. HuggingFace Models. https://huggingface.co/ models?pipeline_tag=text-to-image&sort=likes. [32] Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed. 2010. ZooKeeper: Wait-free Coordination for Internet-scale Systems. In Proc. ATC. [33] Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. 2024. BrushNet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. In Proc. ECCV. [34] Black Forest Labs. 2024. FLUX. https://github.com/black-forest-labs/ flux. [35] LangChain. 2025. LangChain. https://www.langchain.com. [36] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han. 2024. DistriFusion: Distributed parallel inference for high-resolution diffusion models. In Proc. IEEE/CVF CVPR. [37] Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2025. Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference. In Proc. USENIX ATC. [38] Suyi Li, Lingyun Yang, Xiaoxiao Jiang, Hanfeng Lu, Zhipeng Di, Weiyi Lu, Jiawei Chen, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, and Wei Wang. 2025. Katz: Efficient Workflow Serving for Diffusion Models with Many Adapters. In Proc. USENIX ATC. [39] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In Proc. OSDI. [40] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM-based Applications with Semantic Variable. In Proc. OSDI. [41] Yanying Lin, Shuaipeng Wu, Shutian Luo, Hong Xu, Haiying Shen, Chong Ma, Min Shen, Le Chen, Chengzhong Xu, Lin Qu, and Kejiang Ye. 2025. Understanding Diffusion Model Serving in Production: A Top-Down Analysis of Workload, Scheduling, and Resource Efficiency. In Proc. ACM SoCC. [42] linoyts. 2025. Yarn_art_SD3_LoRA. https://huggingface.co/linoyts/ Yarn_art_SD3_LoRA. [43] LlamaIndex. 2025. LlamaIndex. https://www.llamaindex.ai. [44] Runyu Lu, Shiqi He, Wenxuan Tan, Shenggui Li, Ruofan Wu, Jeff J. Ma, Ang Chen, and Mosharaf Chowdhury. 2026. TetriServe: Efficiently serving mixed DiT workloads. In Proc. ACM ASPLOS. [45] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proc. ASPLOS. [46] Modal. 2025. How OpenArt scaled their Gen AI art platform on hundreds of GPUs. https://modal.com/blog/openart-case-study. [47] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul,

Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In Proc. OSDI. [48] Dung Nguyen and Stephen B. Wong. 2000. Design patterns for lazy evaluation. In Proc. SIGCSE. [49] NVIDIA. 2025. NVIDIA OpenSHMEM Library (NVSHMEM) Documentation. https://docs.nvidia.com/nvshmem/api/index.html. [50] Gabriele Oliaro, Xupeng Miao, Xinhao Cheng, Vineeth Kada, Ruohan Gao, Yingyi Huang, Remi Delacourt, April Yang, Yingcheng Wang, Mengdi Wu, Colin Unger, and Zhihao Jia. 2025. FlexLLM: A system for co-serving large language model inference and parameter-efficient finetuning. arXiv preprint arXiv:2402.18789 (2025). [51] OpenAI. 2020. OpenAI API. https://openai.com/index/openai-api/. [52] OpenAI. 2025. Introducing 4o Image Generation. https://openai.com/ index/introducing-4o-image-generation/. [53] OpenAI. 2025. OpenAI DALL·E 2. https://openai.com/index/dall-e-2/. [54] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In Proc. ICLR. [55] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In Proc. FAST. [56] Pol G. Recasens, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Alaa Youssef, Jordi Torres, and Josep Ll. Berral. 2024. Towards Pareto Optimal Throughput in Small Language Model Serving. In Proc. EuroMLSys. [57] sgl project. 2026. diffusion: add-ons support (lora & controlnet). https: //github.com/sgl-project/sglang/issues/13790. [58] sgl project. 2026. [Roadmap] Diffusion (2025 Q4). https://github.com/ sgl-project/sglang/issues/12799. [59] sgl project. 2026. SGLang Diffusion. https://github.com/sgl-project/ sglang/tree/main/python/sglang/multimodal_gen. [60] Arjun Singhvi, Arjun Balasubramanian, Kevin Houck, Mohammed Danish Shaikh, Shivaram Venkataraman, and Aditya Akella. 2021. Atoll: A Scalable Low-Latency Serverless Platform. In Proc. SoCC. [61] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. 2025. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. In Proc. ICLR. [62] stabilityai. 2025. stable-diffusion-3.5-large. https://huggingface.co/ stabilityai/stable-diffusion-3.5-large. [63] Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards Endto-End Optimization of LLM-based Applications with Ayo. In Proc. ASPLOS. [64] TheLastBen. 2025. Papercut Style, SDXL LoRA. https://huggingface. co/TheLastBen/Papercut_SDXL. [65] vllm project. 2026. vLLM Omni. https://github.com/vllm-project/vllmomni. [66] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. 2022. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/ diffusers. [67] Luping Wang, Lingyun Yang, Yinghao Yu, Wei Wang, Bo Li, Xianchao Sun, Jian He, and Liping Zhang. 2021. Morphling: Fast, near-optimal auto-configuration for cloud-native model serving. In Proc. ACM SoCC. [68] Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An efficient multi-level inference system for large language models. In Proc. ACM EuroSys. [69] Yuke Wang, Boyuan Feng, Zheng Wang, Tong Geng, Kevin Barker, Ang Li, and Yufei Ding. 2023. MGG: Accelerating graph neural networks with fine-grained intra-kernel communication-computation pipelining on multi-GPU platforms. In Proc. USENIX OSDI. 14

[70] Zibo Wang, Pinghe Li, Chieh-Jan Mike Liang, Feng Wu, and Francis Y. Yan. 2024. Autothrottle: A Practical Bi-Level Approach to Resource Management for SLO-Targeted Microservices. In Proc. NSDI. [71] Wikipedia. 2025. Lazy evaluation. https://en.wikipedia.org/wiki/Lazy_ evaluation. [72] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proc. SOSP. [73] Yifei Xia, Fangcheng Fu, Hao Yuan, Hanke Zhang, Xupeng Miao, Yijun Liu, Suhan Ling, Jie Jiang, and Bin Cui. 2025. TridentServe: A Stagelevel Serving System for Diffusion Pipelines. arXiv:2510.02838 [74] Yuhao Xu, Tao Gu, Weifeng Chen, and Arlene Chen. 2025. OOTDiffusion: Outfitting Fusion Based Latent Diffusion for Controllable Virtual Try-On. Proc. AAAI (2025). [75] Lingyun Yang, Yongchen Wang, Yinghao Yu, Qizhen Weng, Jianbo Dong, Kan Liu, Chi Zhang, Yanyi Zi, Hao Li, Zechao Zhang, Nan Wang, Yu Dong, Menglei Zheng, Lanlan Xi, Xiaowei Lu, Liang Ye, Guodong Yang, Binzhang Fu, Tao Lan, Liping Zhang, Lin Qu, and Wei Wang. 2025. GPU-disaggregated serving for deep learning recommendation models at scale. In Proc. USENIX NSDI. [76] Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. 2022. INFless: A native serverless system for low-latency, high-throughput inference. In Proc. ACM ASPLOS. [77] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proc. EuroSys. [78] Xiaozhe Yao, Qinghao Hu, and Ana Klimovic. 2025. DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs. In Proc. EuroSys. [79] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023). [80] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for transformer-based generative models. In Proc. USENIX OSDI. [81] Minchen Yu, Tingjia Cao, Wei Wang, and Ruichuan Chen. 2023. Following the Data, Not the Function: Rethinking Function Orchestration in Serverless Computing. In Proc. NSDI. [82] Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Hank Levy, and David Culler. 2025. ICCache: Efficient Large Language Model Serving via In-context Caching. In Proc. SOSP. [83] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Proc. NSDI. [84] Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. 2025. Medusa: Accelerating Serverless LLM Inference with Materialization. In Proc. ASPLOS. [85] Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. In Proc. USENIX ATC. [86] Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. Fast and Live Model Auto Scaling without Caching, In Proc. OSDI. arXiv preprint arXiv:2412.17246. [87] Hong Zhang, Yupeng Tang, Anurag Khandelwal, Jingrong Chen, and Ion Stoica. 2021. Caerus: NIMBLE Task Scheduling for Serverless Analytics. In Proc. NSDI. [88] Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. Shepherd: Serving DNNs in the wild. In Proc. USENIX NSDI. [89] Lvmin Zhang. 2025. Fooocus. https://github.com/lllyasviel/Fooocus. [90] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In Proc. IEEE/CVF

ICCV. [91] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2025. Scaling In-theWild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport. In Proc. ICLR. [92] Wei Zhang, Ziyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2026. JITServe: SLO-aware LLM Serving with Imprecise Request Information. In Proc. USENIX NSDI. [93] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Proc. NIPS.

15