FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving Yaqi Qiao∗1 , Ping He∗2,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
arXiv:2607.12121v1 [cs.DC] 13 Jul 2026
Abstract Diffusion models have become the central backbone for modern image, video, and audio generation, but their efficient service remains a challenge. Unlike autoregressive decoding, diffusion inference repeatedly updates high-dimensional spatial or temporal latents over many denoising steps. This all-region execution pattern makes generation latency high and limits serving throughput. Existing multi-GPU parallelization methods (e.g., sequence parallelism) can reduce per-step computation, but often introduce substantial activation exchange overhead, causing communication to offset or even outweigh the benefits of parallel execution. This paper presents FlashDiff, a diffusion serving system that improves inference efficiency through adaptive regional execution and scheduling. FlashDiff is based on the observation that diffusion refinement is not uniform across latent regions or denoising steps: different regions often stabilize at different rates, while neighboring steps exhibit strong temporal correlation. FlashDiff leverages these properties to selectively execute only regions that require further refinement and to reallocate the resulting compute slack across concurrent serving requests. FlashDiff consists of three mechanisms. First, it decomposes the latent representation into coherent execution regions using early-stage attention signals, preserving semantic structure while exposing fine-grained parallelism. Second, it uses a lightweight runtime controller to estimate region activity and bypass low-impact updates when further refinement is unlikely to affect output quality. Third, it applies an affinity-aware online scheduler that co-locates dependent regions, balances residual load across GPUs, and reuses reclaimed compute capacity to improve serving efficiency. Across real-world image, video, and audio workloads, FlashDiff reduces end-to-end serving latency by 30–97% and improves throughput by 1.2–2.2×, by eliminating 24–66% of computation without degrading quality.
1
Introduction
Diffusion models underpin modern text-to-image (T2I) [42, 43], text-to-video (T2V) [20, 39], and text-to-audio (T2A) generation [10, 13, 30], powering large-scale services such as Adobe Firefly, which produces millions of images per day [2]. Unlike the token-by-token decoding of large language models (LLMs), diffusion models operate over highdimensional spatial and temporal latents through tens of 1 Equal Contribution
♦ Work was done when visiting at UIUC.
3 HKUST
4 NVIDIA
5 Texas A&M University
steps (e.g., often 50), where each step predicts noise and progressively refines the previous latent toward a coherent output [9, 24]. Despite their moderate parameter size, typically a few billion parameters [50], diffusion models impose substantial inference cost. Each denoising step updates the entire latent representation (e.g., spatial feature maps in T2I or temporal sequences in T2A), making execution fundamentally compute-bound. Even with recent advances in reducing denoising steps [14, 32, 56] and simplifying architectures [27], generating moderate-resolution outputs can take a dozen seconds on H100 GPUs [25]. Unfortunately, with existing multi-GPU parallelism strategies, such as tensor and sequence parallelism [7, 12, 28], the communication payloads (e.g., for exchanging activations) often outweighs the compute savings these strategies aim to provide (§2.2). This has led to severe latency and throughput bottlenecks in production [2]. This paper introduces FlashDiff, a diffusion serving engine that breaks the rigid, monolithic execution of denoising into semantic-aware, parallelizable subtasks. Although latent elements are globally coupled at each step, FlashDiff exploits two intrinsic properties of diffusion: (1) Spatial heterogeneity: different latent regions refine at different rates (e.g., smooth backgrounds versus detailed objects), allowing low-complexity regions to skip denoising steps without perceptual degradation; and (2) Temporal affinity: latent states across adjacent denoising steps are correlated [7, 25], enabling reuse of prior states when updates are skipped. These properties motivate a new execution principle, semantic patch parallelism, which decouples where computation is needed from when it must be performed. The key idea is to partition the latent into semantically coherent patches— contiguous spatial regions (for images and video) or temporal segments (for audio)—and selectively refine each patch across denoising steps. This enables fine-grained, complexityaware execution that fundamentally differs from, and is not achievable with existing model-parallel approaches. Realizing semantic patch parallelism requires overcoming three fundamental efficiency-quality challenges (§2.2): (1) Partitioning tax: object-level partitioning over-fragments the latent space, increasing synchronization overhead, while uniform partitioning ignores semantic boundaries, leading to incoherent patches and visual artifacts; (2) Error propagation: skipping steps reduces computation but risks local errors propagating to neighboring regions, especially as refinement needs evolve with the changing alignment between latent
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
features and the input prompt across steps; and (3) Execution bubbles: semantic partitioning may produce patches with heterogeneous sizes, creating stragglers and complicating concurrent request execution under dynamic serving loads. FlashDiff makes semantic patch parallelism practical by jointly addressing where to compute, when to compute, and how to schedule computation at scale, through three architectural components: the Patch Partitioner, Patch Gate, and Patch Scheduler (§3). The partitioner identifies where computation is needed by estimating the refinement demand of latent regions using cross-attention signals from early denoising steps. Semantically coherent latent regions with similar refinement velocities are grouped into patches to preserve structure while enabling parallel execution. Crucially, it dynamically adapts patch granularity based on runtime communication-to-computation ratio, serving load, and the distribution of region complexity (§4.1). The gate provides runtime control over when each patch should be refined. It captures spatiotemporal changes in self-attention to estimate both intra-patch activity and crosspatch influence. When a patch’s dynamics diminish, the current step is skipped; when dependencies re-emerge, the patch is reactivated. This design exploits temporal affinity while preserving global coherence (§4.2). The scheduler then translates this harvested semantic slack into system-level gains. It introduces lightweight, step-level patch switching to minimize overhead for accommodating high-need patches and to redistribute freed compute across requests to improve the global efficiency-quality frontier, such as maximizing global generation quality under low serving loads. To mitigate communication overhead and stragglers, it applies affinity-aware packing, co-locating dependent patches and balancing residual load across workers (§4.3). FlashDiff generalizes across model architectures and modalities, supporting image, video, and audio generation. We implement FlashDiff atop NVIDIA TensorRT [38], ensuring seamless integration with existing diffusion serving stacks (§5). Comparing to state-of-the-art engines including xDiT [12] and DistriFusion [25], across real-world T2I, T2V, and T2A workloads (e.g., SD3 [9], FLUX [23], WAN2 [50]), FlashDiff reduces end-to-end latency by 30–97% and improves throughput by 1.2–2.2×, by eliminating 24–66% of computation without compromising generation quality (§6). Overall, this paper makes the following contributions: • We propose semantic patch parallelism, a novel execution paradigm that treats the latent grid as a collection of heterogeneous, semantically-governed compute tasks, decoupling where and when refinement occurs to reduce generation execution (§2–§3). • We develop new mechanisms to perform efficiencyand quality-aware patch partitioning, patch-level execution gating, and affinity-aware scheduling (§4). • We show FlashDiff’s substantial efficiency gains on real-world T2I, T2V, and T2A workloads (§5–§6).
Text Prompt Portrait photo still of a spooky victorian witch holding a lantern, wearing a witches hat, in front of an old cemetery on fire at night
3 HKUST
4 NVIDIA
5 Texas A&M University
Iterative denoising steps
Text Encoder (e.g., CLIP, T5)
×T
xt Random Gaussian Noise
xt−1
（e.g., U-Net, DiT)
VAE Decoder Final Image
Text-to-Image Diffusion Models
Figure 1. Diffusion models generate images, videos, and audio through many iterative steps. Each step refines all latent elements, making inference compute-intensive.
2
Background and Motivation
2.1
Diffusion Model Serving
Diffusion models generate high-fidelity outputs via a reverse diffusion process, which gradually denoises an initial Gaussian latent tensor x𝑇 into a coherent latent representation x̂0 over many denoising steps (often 20–100 [2, 25, 55]). Figure 1 illustrates this process for text-to-image (T2I) generation; T2V and T2A generation follow a similar iterative workflow yet differ in their encoders and decoders. Modern diffusion models are increasingly based on the Diffusion Transformer (DiT) architecture due to its superior performance and compatibility with different modalities [41]. DiTs employ two complementary forms of attention: (1) Cross-attention, which conditions each latent token (spatial/temporal latent position) on the prompt text-token, and (2) Self-attention, which models interactions among latent tokens to capture longrange structure and global coherence. Practical deployments must satisfy two pressing needs: low latency for interactive user experiences, and high throughput to sustain large request volumes. Adobe’s Firefly integrates T2I generation into tools such as Photoshop and Express, where preview results must be produced in seconds to preserve creative workflow fluidity [1, 2]. Generative advertising platforms like Google Performance Max require rendering in seconds to avoid measurable revenue losses [36]. Suno reports over 10 million T2A generation users and highlights a similar need for interactive generation [49]. 2.2
Motivations for Semantic Patch Parallelism
Facing stringent latency and throughput requirements, scaling model execution across GPUs has historically delivered substantial success for LLMs. Existing multi-GPU techniques, such as tensor and sequence parallelism [28, 45], often benefit from higher compute-to-memory arithmetic intensity by partitioning model weights or contexts to accommodate ever-larger models and longer sequences. However, we identify a fundamental mismatch between these paradigms and the dynamic semantic nature of diffusion workloads. Existing parallelism falters due to the diffusion communication wall. Unlike LLMs, diffusion models typically contain only a few billion parameters, yet each denoising step is extremely compute-intensive. As shown in Figure 2,
Figure 2. Diffusion is Figure 3. Communication compute-bound, saturating overheads often overshadow GPUs at a small latent size. multi-GPU gains. generating a small 512×512 image with a state-of-the-art model such as Flux [23] drives an H100 GPU to 98.9% utilization. This per-step saturation directly translates into high end-to-end latency and severely caps serving throughput. Although multi-GPU execution could in principle mitigate this compute bottleneck, diffusion models possess a uniquely unfavorable activation-to-weight ratio: each denoising step requires exchanging large intermediate activations and 1D–3D latent feature maps. Worse, these transfers scale with resolution and GPU count. As shown in Figure 3, even with diffusion-optimized sequence parallelism (xDiT [12]), communication still dominates per-step latency at 8 GPUs (e.g., 76% in the SD3 model). Consequently, parallel efficiency collapses and even yields negative returns.
1.0 0.8 0.6 0.4 0.2 0.00
T2I (FLUX.1-dev) T2V (Wan-2.1) T2A (StableAudioOpen)
10
20
30
40
Generation Steps Needed
50
CDF across Patches
CDF across Patches
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving 1.0 0.8 0.6 0.4 0.2 0.0
FLUX.1-dev Wan2.1 SD3 Medium
102
103
104
105
Patch Size (Pixels)
106
Figure 4. Latent regions ex- Figure 5. Region sizes vary hibit heterogeneous refine- widely, producing imbalment needs. anced patch workloads. the amount of execution needed per request, but skipping refining regions or denoising steps can degrade generation quality. This raises system-architectural challenges: • Runtime dynamics: Refinement complexity is not static; it depends on the evolving non-linear alignment between the text prompt and the latent state. A system must identify skippable regions at runtime with minimal overhead, in a few milliseconds. • Compute-communication tradeoffs: Partitioning the latent into semantic patches complicates data dependencies. If patches are too small or semantically fragmented, the synchronization tax to maintain global coherence can eclipse the compute savings (Figure 3). • Execution bubbles: Going beyond single requests, practical serving must handle high request concurrency. As shown in Figure 5, semantic regions vary significantly in size and complexity. Region-level partitioning can lead to uneven patch sizes, thus imbalanced loads that complicate scheduling, making it difficult to sustain both high throughput and low latency.
Semantic heterogeneity within monolithic diffusion. Diffusion models operate over spatial or temporal latent fields (e.g., image regions, video segments, or audio windows) whose local refinement difficulty varies widely. This heterogeneity is intrinsic to the underlying content. For example, in the prompt “a white long-haired cat sitting before a black wall,” the detailed fur and contours of the cat require more iterative correction than the nearly uniform background. To quantify this generality, we study T2I generation using the FLUX model [23], T2V using the WAN-2.1 model [50], and T2A using the StableAudioOpen model [10], each evaluated on thousands of real user prompts (e.g., DiffusionDB for T2I [53]) with a standard 50-step denoising schedule. We perform post-hoc analysis to determine how many denoising steps can be skipped for each region without degrading output quality (detailed experiment setups in Section 6.1). As shown in Figure 4, regions exhibit sharply different refinement rates: approximately 23% converge within 30 steps, while others require nearly the full denoising steps. However, existing serving engines view denoising as a monolithic iterative process, mandating equal execution for every latent region regardless of its refinement needs. This rigid execution model fails to exploit semantic heterogeneity, where latent regions refine at heterogeneous rates.
We introduce FlashDiff, a serving engine that transforms the rigid, monolithic diffusion process into adaptive, semanticaware execution, while supporting T2I, T2V, and T2A generation. For a given request (e.g., from the upstream scheduler), it orchestrates execution across workers to jointly optimize generation latency and system throughput. The core of FlashDiff is the abstraction of semantic patch parallelism that partitions the latent grid into patches— semantically coherent “compute units” that represent distinct spatial or temporal regions. This abstraction allows the system to treat generative refinement as a scheduling problem, decoupling the refinement need from resource allocation.
Systemic challenges in harvesting semantic heterogeneity. Translating this semantic heterogeneity into systems efficiency introduces a fundamental efficiency-quality tension: selective refinement of latent regions could reduce
Workflow. As shown in Figure 6, FlashDiff operates in an online serving setting where requests arrive dynamically, integrating seamlessly with existing diffusion-model serving stacks through only a few-line API change. Upon receiving a
These challenges necessitate a new execution paradigm that treats latent regions as first-class, schedulable compute units rather than monolithic tensors.
3
FlashDiff Overview
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
User Request
Diffusion Pipeline one-line API change: pipe.set_flashdiff()
Saliency Task i Analysis Prompt
Importance
§4.1 Patch Partitioner
5 Texas A&M University
Binary Mask and Partitioned Regions
Final Generated Image
Latent Patches Encode Early Stage Partition Latent
Figure 6. FlashDiff overview with three key components. request from the upstream scheduler [2, 33], 1 Patch partitioning: FlashDiff performs a brief warm-up phase identical to standard diffusion for the first few denoising steps. During this phase, it extracts early semantic signals (e.g., crossattention maps) and invokes the Patch Partitioner to group latent elements into semantically coherent patches. The partitioner balances semantic coherence, refinement complexity, and system load when determining patch granularity. 2 Adaptive execution: During subsequent denoising steps, the Patch Gate determines whether each patch should be refined. It tracks spatiotemporal changes in attention to estimate refinement importance and selectively skips patches with low activity, while allowing them to resume when dependencies re-emerge. Active patches exchange updated latent representations at each denoising step, while skipped patches reuse cached states. 3 Patch scheduling: The Patch Scheduler assigns active patches to GPUs and reclaims resources freed by skipped patches. It adaptively redistributes computation across requests to load balance and improve overall system efficiency.
FlashDiff Design
We next describe how FlashDiff performs quality- and loadaware partitioning to generate semantically coherent patches (§4.1), selectively skips patch execution over denoising steps (§4.2), and schedules patches across workers to minimize serving latency and maximize throughput (§4.3). 4.1
Token Importance Selection
4 NVIDIA
keywords
§4.3 Patch Scheduler §4.2 Patch Gate Offline Output Calibrate Activated Monotask One-step Denoise Types Packing Patches (𝑡) (𝑤𝑖 , 𝜺) Decode Image Skipped Patch Patches Video at Step N Gate Update Audio
4
3 HKUST
Patch Partitioner: Enable Semantic Patch Parallelism
Unlike unstructured sparsity (e.g., skipping individual pixels), which current GPUs cannot efficiently exploit due to their reliance on contiguous tensor kernels, we target patch-level granularity to expose structured sparsity that aligns with modern accelerators and distributed execution primitives. An effective patch partitioning strategy must balance (i) semantic coherence to maintain the structural integrity of the latent grid, (ii) complexity affinity to group patches with similar refinement trajectories, allowing for collective execution
Figure 7. Semantic-aware latent partitioning workflow. gating, and (iii) systems efficiency, ensuring that the partitioning logic itself does not introduce a prohibitive runtime tax or execution bubbles. However, meeting these needs reveals multifold challenges. Fine-grained partitioning (numerous micro-patches) maximizes the theoretical parallelism and skipping potential but imposes a substantial cross-patch synchronization tax. Yet, rigid, coarse-grained partitioning (e.g., uniform grids) suffers from semantic entanglement, where a single highcomplexity object forces its low-complexity neighbors to remain in the compute-intensive path. Furthermore, off-theshelf segmentation primitives like Segment Anything [19] are computationally heavy, produce fragmented outputs, rely on high-fidelity features unavailable during noise-dominated early steps, and are often constrained to vision tasks. Capturing Patch Coherence and Complexity. FlashDiff leverages the model’s cross-attention maps to jointly capture semantic coherence and refinement need, enabling modelaware partitioning too. At each denoising step, diffusion models already compute the attention alignment between latent positions (e.g., spatial for images or temporal in audio) and prompt tokens. Existing ML theory [29, 31] shows that these cross-attention patterns stabilize early, while later steps primarily refine local details (e.g., texture) rather than introduce new regions. Therefore, FlashDiff extracts crossattention maps after a short warmup phase (e.g., the first five steps) to guide partitioning. We focus on salient tokens (e.g., nouns, verbs, and descriptive adjectives) that correspond to concrete semantic entities, while excluding function words (e.g., "the", "on"). These salient tokens can be identified via a part-of-speech tagger [17] applied directly to the prompt. This design generalizes well across modalities and remains lightweight (§6.2). As shown in Figure 7, for a set of salient token indices T , we aggregate their attention maps to yield Í a semantic saliency map 𝑆: 𝑆 𝑗 = | T1 | 𝑡 ∈ T 𝐴 𝑗,𝑖 , where 𝐴 𝑗,𝑖 is the attention weight from latent location 𝑗 to token 𝑖. This map serves as a compute density signal: focus regions with higher 𝑆 𝑗 represent high-entropy semantic units that require sustained refinement to avoid perceptual degradation. Recursive Load-Aware Patch Partitioning. Given the saliency map, FlashDiff partitions the latent into patches that expose parallelism while maintaining load balance. Standard partitioning methods (e.g., K-means clustering) are ill-suited,
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
LPIPS Reduction (%)
PSNR Gain
Number of Regions
40
15
LPIPS Reduction SSIM Gain (%)
1.25 11.5% 0.02 9.1% 8.4% 8.1% 1.00 4.1% 3.9% 3.6% 3.4% 0.75 0.01 0.50 0.25 0.00 n=4 n=8 n=12 n=16 0.00
30
10
20
5
10
0 36 37 38 39 40 41 42 43 0
Number of Equivalent Steps
Figure 8. Our partitioning Figure 9. Our skipping outoutperforms the uniform par- performs uniform step reductitioning method. tion under equivalent effective compute budgets. as they ignore spatial or temporal locality and produce fragmented, non-contiguous regions. Instead, we introduce a recursive saliency partitioning mechanism. Inspired by Otsu’s method [40], which thresholds grayscale images by maximizing the variance between foreground and background pixels, our partitioner bisects the given continuous saliency map 𝑆˜𝑗 by thresholding the saliency map: 𝑀 𝑗 = 1[ 𝑆˜𝑗 ≥ 𝜃 ]. Here, 𝜃 is automatically chosen to maximize inter-region variance. This produces two contiguous latent token categories: a focus region (high saliency) and a context region (low saliency). To further ensure load balance, given the target number of partitions (regions) 𝑅, we partition the current latent 𝑟 Í based on the relative token density. Let 𝑛 𝑓 = 𝑗 𝑀 𝑗 and 𝑛𝑐 = 𝐽 −𝑛 𝑓 denote the number of focus and context tokens, where 𝐽 = 𝐻 lat × 𝑊lat is the total number of tokens in the latent space. For example, 𝐻 lat and 𝑊lat correspond to the height and width of the image latent grid, respectively. We allocate 𝑛𝑓 𝑟 𝑓 focus regions by 𝑟 𝑓 = argmin𝑟 ∈ {1,...,𝑅−1} 𝑟 − 𝑅 𝑛 𝑓 +𝑛 to 𝑐 balance the token complexity. We then split the focus and context token index lists into 𝑟 𝑓 and 𝑟𝑐 = 𝑅 −𝑟 𝑓 . This process continues until 𝑅 patches are obtained. The result is a set of semantically coherent, contiguous patches with balanced workloads. Figure 8 shows that our partitioning consistently outperforms uniform partitioning in final generation quality across a range of partition counts (normalized to the latter; experiment setups in §6.1). Furthermore, Figure 9 demonstrates that, under equivalent execution budgets (e.g., FlashDiff skips 20 out of 50 steps versus directly running 30 steps without FlashDiff), FlashDiff achieves higher generation quality. This highlights that selectively allocating computation to semantically important regions is fundamentally more effective than uniformly reducing denoising steps. Adaptive Patch Granularity. A remaining challenge is determining the number of patches 𝑅. Increasing 𝑅 exposes more fine-grained skipping opportunities, but also amplifies cross-patch communication overhead and risks fragmenting semantically coherent regions. Determining the optimal 𝑅 a priori is impractical, as it depends on the input prompt, model dynamics, and future skip opportunities.
Instead, FlashDiff leverages a key structural property of this tradeoff. As 𝑅 increases, the opportunity for selective refinement grows monotonically, improving theoretical computational efficiency. However, system-level costs, particularly cross-patch communication, grow rapidly and eventually dominate execution time. At the same time, generation quality remains stable under moderate partitioning but degrades when patches become overly fragmented (§6.4). Together, these effects induce a unimodal performance profile with respect to 𝑅: performance improves with increasing parallelism up to a point, after which communication overhead and quality degradation outweigh the benefits. This structure enables efficient online optimization for achieving the efficiency-quality frontier. Specifically, FlashDiff performs a bounded binary search over 𝑅, guided by observed end-toend latency and quality signals from past requests. When increasing 𝑅 improves performance, the system continues exploring finer partitioning; when performance degrades, it reduces 𝑅 to avoid excessive communication and fragmentation. This self-adaptation can quickly and continuously converge to the near-optimal operating point under the current workload. Our online serving deployments confirm the effectiveness of this design (§6.2). 4.2
Patch Gate: Selectively Refine Patches
Given the partitioned patches, the Patch Gate determines when each patch should be refined along the denoising trajectory. This is non-trivial because refinement needs evolve dynamically: complex patches may require substantial updates early but stabilize later, while low-saliency patches may still need updates to preserve global coherence for other patches due to cross-patch dependencies. Adaptive Skipping under Spatiotemporal dynamics. We use self-attention as a proxy for refinement activity. Unlike cross-attention (used by the Patch Partitioner) that captures the alignment of the latent region with the prompt, self-attention reflects how the latent locations interact with each other and how updates propagate across regions [15, 29]. Moreover, due to the bidirectional attention in diffusion models, each latent position both attends to and is influenced by others. So patches with strong self-attention activity are important for their own refinement and guiding other patches. We quantify this using a refinement importance score (𝑡 ) 𝑅𝑖 for each patch 𝑖 at step 𝑡. Let A𝑖 denote the set of latent locations belonging to patch 𝑖, and let 𝑆𝐴 ∈ R 𝐽 ×𝐽 be the self-attention matrix with 𝐽 total tokens. We define: Í (𝑡 ) 𝑅𝑖(𝑡 ) = | A1𝑖 | 2 𝑢,𝑣 ∈ A𝑖 𝑆𝐴𝑢,𝑣 , which captures both internal and externally dependent attention strength of patch 𝑖. Intuitively, patches with high refinement importance 𝑅𝑖 should not be skipped. However, 𝑅𝑖 varies both across patches and over time, making static thresholding unreliable. Moreover, we would hope to incorporate the relative
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
efficiency gains from skipping different-sized patches. Fortunately, our recursive partitioning strategy (§4.1) has produced patches of approximately equal area for load balance, even though their shapes (e.g., width and height) may differ, thereby sidestepping reasoning about heterogeneous gains. Our key idea is to gate patches based on how much their refinement signal is changing relative to other patches, rather than on its absolute magnitude. For each patch 𝑖, we maintain (i) 𝑅¯𝑖 , the most recent importance when the patch was executed, and (ii) Δ𝑖 , a cached change magnitude. When executed at step 𝑡, we update Δ𝑖(𝑡 ) = 𝑅𝑖(𝑡 ) − 𝑅¯𝑖 and 𝑅¯𝑖 ← 𝑅𝑖(𝑡 ) . When the patch is skipped, we retain Δ𝑖(𝑡 ) = Δ𝑖(𝑡 −1) and then Δ
(𝑡 )
normalize these changes across patches: 𝑤𝑖(𝑡 ) = Í 𝑖 (𝑡 ) so 𝑗 Δ𝑗
that 𝑤𝑖(𝑡 ) measures how much patch 𝑖 contributes to the overall refinement activity of the latent. A patch is skipped if 𝑤𝑖(𝑡 ) < 𝜀. This decision is both step-aware and promptadaptive: patches are skipped only when their refinement dynamics are negligible relative to others. We show that FlashDiff achieves consistently better performance than existing advances across a wide span of 𝜀 (§6.4). Guaranteeing Generation Quality. When a patch is skipped, its noise prediction is reused from the most recent active step, incurring a local approximation error. Yet, we prove that our design introduces a bounded and well-behaved error across the denoising trajectory, ensuring good quality: Theorem 4.1 (Quality Bound). Let 𝑥ˆ𝑁 denote the final latent produced by FlashDiff after 𝑁 denoising steps, and let 𝑥 𝑁∗ denote the baseline latent without gating. Under the adaptive gating rule with threshold 𝜀, the deviation satisfies Í 𝑁 ∥𝑥ˆ𝑁 − 𝑥 𝑁∗ ∥ = 𝑂 𝜀 · 𝑛=1 𝐷𝑛 + 𝛾 , where 𝐷𝑛 is the total refinement activity at step 𝑛, and 𝛾 is a bounded residual induced by forced reactivation. In particular, the quality gap vanishes as 𝜀 → 0. Proof sketch. The result follows from three key observations. (1) Bounded local error: A patch is skipped at step 𝑛 only if its normalized refinement contribution satisfies 𝑤𝑖(𝑛) < 𝜀, implying its change magnitude Δ𝑖(𝑛) ≤ 𝜀·𝐷𝑛 . (2) Lipschitz continuity: The post-attention components (MLP, normalization, projection) are Lipschitz-continuous, so perturbations in attention induce proportionally bounded deviations in the predicted noise. Denoting the per-step prediction error by 𝛿𝑛(𝑖 ) , we obtain 𝛿𝑛(𝑖 ) = 𝑂 (𝜀 · 𝐷𝑛 ). (3) Bounded staleness: A forcedreactivation constraint ensures that no patch is skipped for more than a constant number of steps (§5), bounding the drift from reused predictions by a small additive term 𝛾. Aggregating these per-step errors across the denoising trajectory and applying a discrete Gronwall inequality [44] yields the stated bound. A tighter analysis exploiting contractivity of the reverse diffusion process is deferred to Appendix A. □
3 HKUST
4 NVIDIA
5 Texas A&M University
Algorithm 1: FlashDiff Serving Runtime Input: Request stream {𝑝}; warmup steps 𝑇𝑤 ; regions 𝑅; gate threshold 𝜀 Output: Generated sample 𝑦 1 Function PatchPartitioner(𝑝, 𝑇𝑤 , 𝑅) 2 latent ← InitNoise(); run 𝑇𝑤 warmup steps 3 Select salient tokens and build saliency map 𝑆 4 Allocate 𝑟 𝑓 , 𝑟𝑐 and split latent into 𝑅 patches 5 return (latent, patches P) Function PatchGate(P, state, 𝜀) Compute refinement importance 𝑅𝑖 from self-attention 8 Update Δ𝑖 and normalize to weights 𝑤𝑖 9 return active set U = {𝑖 : 𝑤𝑖 ≥ 𝜀}, updated state
6
7
Function PatchScheduler() 𝑄 ← ∅ // global monotask priority queue 12 while ¬AllDone() do 13 R ← PollArrivals() 14 foreach 𝑝 ∈ R do 15 𝑄 ← Enqueue(𝑄, ⟨Warmup, 𝑝,𝑇𝑤 ,
10
11
UpstreamPriority(p)⟩)
21
C ← CollectCandidates(𝑄) foreach 𝑤 ∈ GetIdleWorkers() do 𝑚 ← HighestPriorityCandidate(C, 𝑤) if Benefit(𝑚) > SwitchCost(𝑚) then 𝑔′ ← AffinityAwareAssign(𝑚); Dispatch(𝑔′, 𝑚)
22
RunStepAndUpdate (𝑄)
16 17 18 19 20
23
return latent
Empirically, our evaluations across diverse T2I, T2V, and T2A models and tens of thousands of real prompts confirm negligible quality degradation (§6.2, Table 2). 4.3
Patch Scheduler: Maximize Serving Goodput
Algorithm 1 summarizes FlashDiff’s serving runtime. Each request proceeds in three phases. (1) Warmup and Partitioning. FlashDiff first executes a few standard denoising steps to extract semantic saliency maps, which are used to recursively partition the latent into coherent, right-sized patches (Line 2– Line 5). (2) Patch-wise Denoising. an in-parallel patch denoising phase where each patch is either computed or skipped based on the adaptive gating mechanism (Line 7–Line 9; §4.2). (3) Dynamic Patch Scheduling. Fine-grained patch execution introduces potential execution bubbles, where FlashDiff needs to dynamically interleave skipped and active patches both within and across requests. As an execution engine, FlashDiff is designed to strictly adhere to the upstream request ordering (e.g., from the FIFO
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
Dispatch & execute GPU1 Idle
… Monotask Priority Queue
GPU2 Step m
GPU1 Gate off
Activated
GPU2 Step n
Figure 10. Patch monotask scheduler. Workers fetch tasks from a priority queue, where activated patches can preempt lower-priority ones for low latency and high throughput. scheduler [2]) while extracting maximal efficiency from available hardware. With the resource freed by patch skipping, we next introduce how the Patch Scheduler (1) enables patch switching efficiently, while (2) adapting to online serving dynamics (e.g., load burstiness) to jointly optimize latency, throughput, and generation quality for many requests. Efficient Patch Switching with Monotasks. FlashDiff introduces a priority-driven patch monotask abstraction to enable low-latency, preemptive execution. As illustrated in Figure 10, each executable unit, whether a full-latent warmup task or an individual patch in the parallel phase, is represented as a monotask and enqueued according to the request’s scheduling priority, assigned by the upstream scheduler [2, 33]. This allows patches from higher-priority requests to promptly preempt those from lower-priority ones. When a worker has leftover resources (e.g., because its current patch is gated off), FlashDiff dispatches the highestpriority monotask. When a previously skipped patch becomes active, its monotask is treated as a candidate to preempt a currently executing lower-priority monotask. Such preemption and switching occur only at step boundaries, ensuring that intra-step execution remains atomic and preventing partial progress within a step. Our monotask switching introduces negligible impact on per-request latency because: (i) switching occurs only at step boundaries (every a few hundred milliseconds), bounding deferral to at most one step; (ii) active patches of the same request continue executing during that step, preserving global progress; (iii) only a small amount of runtime state needs to be maintained per request (e.g., no LLM-style KV cache); and (4) our priority-based switching ensures that at most two times of requests’ states need to reside on the GPU at a time (§5)—one for the preempted request and one for the active request—keeping I/O minimal and predictable. Our evaluations validates that our priority-based switching design introduces negligible latency overhead while substantially improving throughput (§6.4). Maximizing Goodput with Monotask Packing. While FlashDiff respects upstream request priorities, monotasks from many concurrent requests continuously compete for GPU resources in online serving. Efficiently packing these monotasks is therefore critical to achieving both low latency and high throughput. Naïvely distributing patches from the
same request across GPUs increases cross-GPU communication and causes each denoising step to be bottlenecked by the slowest patch, due to cross-patch communication in a request. Conversely, aggressively co-locating patches of a request onto as few GPUs as possible can under-utilize available hardware, and may create a single-GPU bottleneck that increases per-step latency for the request. FlashDiff addresses this with a two-phase assignment policy. In the warm-up phase, before patch-level parallelism begins, the request’s single monotask is dispatched to the GPU with the lowest current load. Once patch-parallel denoising begins, FlashDiff assigns each patch monotask using a locality-aware least-maximum-load rule. Let 𝐿𝑔 denote the current load on GPU 𝑔, defined Í as the total compute cost of patches assigned to it: 𝐿𝑔 = 𝑖 ∈ P𝑔 𝑤𝑖 , where P𝑔 is the set of patches currently assigned to GPU 𝑔 and 𝑤𝑖 reflects patch 𝑖’s compute cost. Since each denoising step completes only when all patches finish their respective step, step latency is determined by the straggling patch on the most loaded GPU. Therefore, for an incoming patch 𝑗 with cost 𝑤 𝑗 , FlashDiff assigns it to the GPU that minimizes the maximum load: ( ! 𝐿𝑔′ + 𝑤 𝑗 if 𝑔′ = 𝑔 ∗ 𝑔 = arg min max (1) 𝑔 𝑔′ 𝐿𝑔′ otherwise Our design is lightweight with only O(|𝑔|) complexity, yet enabling both low per-request latency and high global throughput by fully utilizing available GPU capacity. Note that under low serving loads, FlashDiff will automatically prioritize quality (§4.1). Indeed, our evaluation confirms that this packing strategy achieves near-optimal serving latency and consistently superior quality (§6.2).
5
Implementation
We implement FlashDiff as a production-grade diffusion serving engine with approximately 5.5K lines of Python, C++, and CUDA, built on top of NVIDIA TensorRT [37]. FlashDiff Backend. FlashDiff is implemented as a multiprocess, multi-GPU runtime that uses NCCL for inter-GPU communication. During warm-up, workers construct the initial KV cache and synchronize it across GPUs. The lead warm-up worker then exports the initialized KV cache to the main serving workers via CUDA IPC handles, and forwards the request metadata (patch layout, step index, and priority) to the Patch Scheduler. Workers execute their assigned patches using the KV cache from the previous step, and exchange only the updated partial KV blocks corresponding to their patches. After each step, workers report per-patch refinement statistics to the scheduler, which invokes the Patch Gate to determine the active patch set for the next step. Fault Tolerance. FlashDiff maintains all control-plane state—including patch partitions, per-request step counters, and worker–patch assignments—in a replicated, lightweight
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
metadata store, enabling rapid recovery from worker or scheduler failures. Upon detecting a failure, the Patch Scheduler automatically reassigns affected patches to healthy workers and resumes execution. To guarantee correctness under retries, each patch execution is labeled with a (request_id, step_id, patch_id) tuple. Workers discard stale outputs, ensuring idempotent recovery.
6
Evaluation
We evaluate FlashDiff on tens of thousands of realistic requests spanning text-to-image (T2I), text-to-video (T2V), and text-to-audio (T2A) generation. Our key findings are: • FlashDiff reduces end-to-end serving latency by 30– 97% and improves throughput by 1.2–2.2× without compromising generation quality (§6.2); • FlashDiff optimizes statistical efficiency (skipping unnecessary refinement) and system efficiency, achieving near-optimal efficiency-quality tradeoffs (§6.3); • FlashDiff consistently outperforms existing advances across diverse serving loads and settings (§6.4). 6.1
Methodology
Cluster Setup and Workloads. We evaluate FlashDiff on a cluster of 8 NVIDIA H100 GPUs for Image and Video, and 4 NVIDIA A100 GPUs for Audio. Table 1 summarizes the four state-of-the-art diffusion models and real-world workloads used in our study. Following deployment practices [2], we use the FlowMatchEulerDiscreteScheduler for T2I and T2V with 50 steps, and the DPMSolverMultistepScheduler for T2A, with 100 steps in all experiments. We studied the impact of total denoising steps in ablation studies (§6.4). We use the realistic arrival patterns from DiffusionDB [53], scaled to match our cluster capacity such that the mean arrival rate is close to the per-request generation time, avoiding service failures. We also ablate the request arrival rate (§6.4). Baselines. We compare FlashDiff against three state-ofthe-art diffusion serving systems: • xDiT [12]: The state-of-the-art and production-scale multi-GPU inference engine for DiTs. It employs DiToptimized sequence parallelism [11]. • DistriFusion [25]: An advanced patch-parallel execution method, in NVIDIA TensorRT [37], that preserves cross-patch interaction via displaced patch reuse. It overlaps communication with computation. • NaivePatch [25]: A patch-parallel baseline that denoises patches independently and stitches them after each denoising step. Metrics. FlashDiff is designed to improve serving efficiency without degrading generation quality: (i) Efficiency: we report per-request end-to-end latency and serving throughput; and (ii) Quality: For text-to-image, we measure
3 HKUST
Task Model
4 NVIDIA
Workload
T2I
FLUX.1-dev [23] DiffusionDB
T2I
SD3 Medium [9] DiffusionDB
T2V Wan2.1-14B [50] VBench [16]
5 Texas A&M University
Quality Metrics PSNR, SSIM [52], LPIPS [58], HPSv3 [35] PSNR, SSIM [52], LPIPS [58], HPSv3 [35] Image Qual., Motion Smooth., Subject Consist. [16]
T2A StableAudioOpen AudioCaps [18] FD [8], KL [21], CLAP [54]
Table 1. Summary of evaluation workloads. PSNR, SSIM [52], LPIPS [58], and HPSv3 [35]. For text-tovideo, we report Imaging Quality, Motion Smoothness, and Subject Consistency [16]. For text-to-audio, we use FD [8], KL [21], and CLAP [54] score. All results are averaged over five independent runs. 6.2
End-to-End Performance
FlashDiff improves serving throughput. Figure 11 shows the serving throughput for text-to-image (T2I), text-toaudio (T2A), and text-to-video (T2V) workloads over a onehour online deployment. Compared to all baselines, FlashDiff consistently achieves 1.2–1.8× higher throughput across different workloads, by reducing the amount of generation execution needed per request. The magnitude of improvement varies across tasks, reflecting differences in patch-skipping opportunities and spatial or temporal heterogeneity in the generated content. Notably, during periods of request burstiness, FlashDiff sustains substantially higher throughput, while all baselines quickly saturate and plateau at a fixed bottleneck. This behavior demonstrates FlashDiff ’s ability to absorb sudden workload surges by reclaiming computation from skipped patches and dynamically redistributing GPU resources. We further analyze this load-adaptive behavior (e.g., performance under different system loads) in our ablation study (§6.4). FlashDiff reduces user-perceived serving latency. Figure 12 reports request completion time (finish time minus submission time) over the one-hour online deployment. Compared to baselines, FlashDiff consistently achieves 30–97% lower request completion times throughout the deployment. The largest latency reductions occur during periods of high concurrency, which are prevalent in online serving due to bursty request arrivals. Under such transient spikes, baselines quickly suffer from queue buildup and blocking among concurrent requests, leading to sharp latency escalation. In contrast, FlashDiff effectively bounds latency growth by (1) reducing the amount of computation and communication required per request and (2) redistributing resources from skipped patches to maintain a balanced GPU load. FlashDiff maintains generation quality. Table 2 reports quantitative quality metrics in our online deployment, spanning thousands of real requests. Across all modalities,
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving FlashDiff
NaivePatch
Throughput (req/min)
Throughput (req/min)
DistriFusion
4.0 2.0 0
10
20 30 40 Time (minutes)
50
60
(a) StableAudioOpen (T2A)
DistriFusion
2.5 2.0 1.5 0
FlashDiff
xDiT
3.0
Throughput (req/min)
FlashDiff
6.0
10
20 30 40 Time (minutes)
50
DistriFusion
3.5 3.0 2.5 0
(b) FLUX.1-dev (T2I)
10
20 30 40 Time (minutes)
50
60
(c) Wan2.1 (T2V)
0 0
15 30 45 Time(minutes)
(a) FLUX.1-dev (T2I)
50
0
15 30 45 Time(minutes)
(b) SD3 Medium (T2I)
200
FlashDiff DistriFusion NaivePatch
100
0 0
15 30 45 Time(minutes)
Request Completion Time (s)
200
100
FlashDiff DistriFusion xDiT
Request Completion Time (s)
400
FlashDiff DistriFusion xDiT
Request Completion Time (s)
Request Completion Time (s)
Figure 11. FlashDiff achieves higher serving throughput across a one-hour online deployment, generalizing across model architectures and modalities, whereas existing baselines are limited to specific tasks. FlashDiff DistriFusion
300 200 100
(c) StableAudioOpen (T2A)
0 0
15 30 45 Time(minutes)
(d) Wan2.1 (T2V)
Figure 12. FlashDiff reduces end-to-end per-request latency. By reducing effective computation, FlashDiff lowers both raw generation time and queueing delays, especially under high load. We add ablation studies on arrival rates in Section 6.4.
FlashDiff introduces negligible overhead. We evaluate the intrinsic overhead of FlashDiff by measuring singlerequest completion time in isolation (i.e., without concurrency). To isolate FlashDiff’s control-plane and scheduling overheads, we add FlashDiff support atop DistriFusion (i.e., using DistriFusion as the backend) and compare their runtimes under identical workloads. As shown in Figure 13, FlashDiff introduces very marginal overhead, 0.4–3.7%, across all models and modalities. This overhead comes primarily from patch partitioning, gating, and monotask scheduling, which involve lightweight control logic and irregular memory accesses (e.g., for loading different patches). By exploiting patch-level skipping and crossrequest backfilling, FlashDiff substantially improves generation latency and throughput, yielding net gains. 6.3
Performance Breakdown
Breakdown by Components. We evaluate the contribution of FlashDiff’s key design components by disabling them in an online deployment of the SD3-Medium image generation workload. Figure 14 shows the resulting distributions of request completion time. It reports that both the Patch
FlashDiff
DistriFusion
+0.6% FLUX.1-dev +3.7% SD3 Medium +0.4% Stable Audio Open +2.0% Wan2.1 0 10 20 30 Single Request Completion Time (s)
Figure 13. FlashDiff introduces negligible overhead.
CDF
FlashDiff incurs little to no degradation in generation fidelity and often achieves slightly higher quality. By preserving cross-patch interactions through bidirectional attention and adaptive gating, FlashDiff keeps the diffusion trajectory closely aligned with that of the unmodified model. In contrast, xDiT and DistriFusion rely on asynchronous execution or stale-state reuse for efficiency, which quickly accumulates approximation error across denoising steps.
1.00 0.75 0.50 0.25 0.00 1 10 102 103 Request Completion time(s)
FlashDiff FlashDiff w/o Patch Gate FlashDiff w/o Patch Scheduler DistriFusion
Figure 14. Performance breakdown Gate and the Patch Scheduler are critical to FlashDiff’s performance. Disabling the Patch Gate removes selective skipping, forcing all patches to be refined at every step; this substantially increases effective computation and directly inflates latency. Disabling the Patch Scheduler eliminates affinity-aware packing and dynamic backfilling, leading to GPU imbalance and straggler-dominated steps. Notably, each component alone already outperforms DistriFusion, demonstrating new contributions to each design. Breakdown by Statistical Efficiency. We quantify FlashDiff ’s statistical efficiency by measuring how much patch-level diffusion computation can be skipped without degrading output quality (i.e., skipped patch-steps). We use 50 denoising steps for FLUX.1-dev, SD3-Medium, and Wan2.1, and 100 steps for StableAudioOpen, with a 5-step warm-up,
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
Task
Model
4 NVIDIA
5 Texas A&M University
Quality Metrics
Method PSNR(↑)
SSIM(↑)
LPIPS(↓)
HPSv3(↑)
xDiT DistriFusion FlashDiff
15.4076 20.8541 20.9837
0.6218 0.7895 0.7849
0.4308 0.2162 0.2202
0.2210 0.2220 0.2214
SD3 Medium
xDiT DistriFusion FlashDiff
16.6547 17.2130 17.2302
0.7354 0.7479 0.7425
0.2708 0.2538 0.2609
0.2156 0.2161 0.2160
Stable Audio Open
T2V
3 HKUST
FLUX.1-dev T2I
T2A
2 Vanderbilt University
Wan2.1
NaivePatch DistriFusion FlashDiff
DistriFusion FlashDiff
FD𝑜𝑝𝑒𝑛𝑙 3 (↓)
KL𝑝𝑎𝑠𝑠𝑡 (↓)
CLAP𝐿𝐴𝐼𝑂𝑁 (↑)
155.5995 87.5675 87.0177
3.7025 2.2220 2.0705
0.1062 0.3273 0.3252
Img-Qual (↑)
Mot-Smooth (↑)
Subj-Cons (↑)
0.6526 0.6526
0.9929 0.9928
0.9471 0.9432
x1.05
x1.2
x1.5
Serving Load
(a) Serving latency.
6 4
x1.05
x1.2
5.2 5.5
7.5
8 5.2 5.5 6.2
568 488
277 197
x0.8
FlashDiff
5.2 5.4 5.4
Figure 15. FlashDiff performance breakdown by statistical efficiency, showing differences in skipped computation.
101
xDiT
10
4.1 4.1 4.1
100
20
80
102
14
20 30 60 Skipped patch-steps per request
103
75
10
DistriFusion
FlashDiff
20 14
0
xDiT
Average Throughput (req/min)
DistriFusion
FLUX.1-dev SD3 Medium Wan2.1 StableAudio Open
13 12 13
1.0 0.8 0.6 0.4 0.2 0.0
Request Completion Time (s)
CDF across Patches
Table 2. FlashDiff improves efficiency while preserving quality. Bold: best; underlined: second best.
2 0
x0.8
Serving Load
x1.5
(b) Serving throughput.
Figure 16. FlashDiff achieves lower latency and higher throughput across different request loads.
6.4
Impact of Number of GPUs. Figure 17 reports serving throughput as a function of the number of GPUs for both T2I and T2V workloads. Two key trends emerge. First, although additional GPUs provide more compute, the benefits are quickly diluted by the growing communication overhead. Second, across all GPU counts, FlashDiff consistently outperforms DistriFusion. By skipping low-impact patches and packing active patches efficiently, FlashDiff reduces interGPU communication and better utilizes available compute.
Sensitivity and Ablation Studies
Impact of Serving Load. We next investigate the impact of serving load using FLUX.1-dev with 2048×2048 image generation over a continuous 1-hour serving period. Here, we vary the system load by scaling the request arrival rate relative to the average generation time: a load below 1 indicates an under-utilized system, while values above 1 stress the serving pipeline. As shown in Figures 16b and 16a, under light load (e.g., ×0.8), all systems achieve comparable
Offline Throughput (req/min)
yielding 45 and 95 effective gated steps, respectively. We include ablation studies on different denosing steps in Section 6.4. Figure 15 shows large variation in skippable computation across models. At the median, StableAudioOpen skips 63 steps (66% of effective steps), followed by SD3-Medium with 20 (44%), Wan2.1 with 16 (36%), and FLUX.1-dev with only 11 (24%). These differences reflect fundamental properties of the generative trajectories: audio latents stabilize quickly over time, enabling aggressive skipping, while high-resolution image models like FLUX continue refining local structure deep into the denoising process. This diversity again highlights why a fixed skip policy is suboptimal. Even under the same quality constraints, different models and requests exhibit vastly different levels of patch-level redundancy, making an adaptive gating mechanism, such as our Patch Gate.
5.8
6 DistriFusion
5 4
0
3.3
2.7
2 1
3.6
3.4
3
FlashDiff
1.7
2.0
2.7
2.2
1.2
2.1
3.7
2.5
1.3
1GPU 2GPU 4GPU 8GPU
1GPU 2GPU 4GPU 8GPU
FLUX.1-dev
Wan2.1-T2V-14B
Figure 17. Impact of GPU numbers. latency and throughput, since requests can be served immediately. FlashDiff continues to increase throughput and keeps latency bounded even under heavy load. At ×1.5 load, FlashDiff achieves up to 25× faster request completion.
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
8.5
7.8
8 6
DistriFusion PSNR FlashDiff PSNR
5.3
7.1
4.8
6.9
4.5
20 PSNR
Offline Throughput (req/min)
DistriFusion Throughput FlashDiff Throughput
4.0
4
10
2
7
0
1
5 10 Warm-up Steps
0
20
Figure 18. Impact of warm-up steps. FlashDiff Throughput FlashDiff PSNR
9.8
10
21.0 20.5
6.8
6.1
6
20.0 4.6
4.4
4
PSNR (dB)
Offline Throughput (req/min)
DistriFusion Throughput DistriFusion PSNR
8
3.4
4.0 2.5
3.4
2
19.5 19.0
2.2
18.5
0
10
20
30
40
50
Total Inference Steps
Figure 19. Impact of total denoising steps.
Skip Rate
PSNR (dB)
24
1.0 PSNR (dB) Skip Rate 0.8 0.6 0.4 0.2 0.36 0.540.0
20 16 0.18
Epsilon ( )
Offline Throughput (req/min)
DistriFusion
28
12 0.0
Impact of Skipping Threshold (𝜖). Figure 20 varies the skipping threshold (§4.2), which controls the tradeoff between aggressive skipping and generation quality. FlashDiff maintains consistently high quality across a wide range of thresholds, while enabling substantial skip rates.
FlashDiff
11.2
10
8.1
9.4
8.1
6.5
5
0
3.7
5122
3.4 2.2
10242 15362 20482
Image Size
Figure 20. Quality under dif- Figure 21. Impact of image ferent 𝜖 thresholds. sizes (FLUX model). Impact of Warm-up Steps. Figure 18 reports both request completion time (left axis) and output quality (right axis) on SD3. Increasing the number of warm-up steps 𝑠 improves partition quality and hence generation quality, but reduces throughput because it delays the onset of patch-level skipping in the subsequent parallel phase. Nevertheless, even with conservative warm-up (e.g., 𝑠=10 or 20), FlashDiff still achieves up to 1.6× higher throughput than DistriFusion. Impact of Total Denoising Steps. Figure 19 shows that FlashDiff consistently achieves better efficiency across step settings (FLUX model). We note that FlashDiff preserves comparable output quality, with PSNR variations within 0.8% to +3% relative to DistriFusion. Impact of Image Size. We next vary the image resolution from 512 × 512 to 2048 × 2048. As shown in Figure 21, increasing resolution enlarges the latent grids and raises the per-step latency, causing throughput drops, but FlashDiff consistently achieves 1.4–2.2× higher throughput.
Related Work
Diffusion Model Optimizations. Existing diffusion model optimizations fall into two main categories: step reduction and per-step acceleration. Step reduction techniques aim to reduce the number of denoising steps, such as improved samplers such as DDIM [46], k-diffusion [51], and parallel sampling [61] to trade off compute for speed. Similarly, recent algorithm advances reduces per-step duration by reusing the latent (e.g., SANA-Video [5]) or reducing model size (e.g., model distillation [34, 47] or quantization methods like q-diffusion[26]). FlashDiff is complementary and introduces complexity-aware patch parallelism and an adaptive skipping mechanism, without altering models and samplers. Diffusion Serving Systems. Recent advances in diffusion serving have explored request scheduling to meet diverse SLOs (e.g., TetriServe [33]); reuse of intermediate latents across similar requests to accelerate warm-up (e.g., NIRVANA [2] and MoDM [55]); sequence-parallel inference across GPUs (e.g., DistriFusion [25], xDiT [12]); and large-scale model parallelism for video diffusion using fully sharded data parallelism [50, 60]. In contrast, FlashDiff reduces the amount of execution required per request. LLM Serving Optimizations. vLLM [22] improves memory efficiency by paging the KV cache, while SarathiServe [3] introduces chunked-prefill to increase throughput. TensorRT-LLM [38] focuses on GPU kernel optimizations, and DistServe [62] decouples the prefilling and decoding stages to ensure predictable latency. JITServe [59] orchestrates the scheduling of requests to meet SLO requirements. IC-Cache [57] repurposes prior requests as additional knowledge. FlashDiff tackles distinct challenges of diffusion-based generation, exploiting computational redundancy.
8
Conclusion
This paper presents FlashDiff, a novel diffusion serving engine that rethinks monolithic diffusion execution through semantic patch parallelism. FlashDiff combines semantic-aware partitioning, adaptive patch gating, and communication-aware scheduling to translate patch-level parallelism into system-level efficiency gains. Evaluations on real-world image, video, and audio generation workloads demonstrate 30–97% reductions in latency and 1.2–2.2× improvements in throughput, all with little quality drop.
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
References [1] Adobe Firefly Team. 2025. Adobe Firefly: The next evolution of creative AI is here. Adobe Blog. https://blog.adobe.com/en/publish/2025/04/ 24/adobe-firefly-next-evolution-creative-ai-is-here [2] Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. 2024. Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 1173–1189. https:// www.usenix.org/conference/nsdi24/presentation/agarwal-shubham [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/conference/osdi24/presentation/ agrawal [4] Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. 2023. Nearly 𝑑-linear convergence bounds for diffusion models via stochastic localization. arXiv preprint arXiv:2308.03686 (2023). [5] Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, Haozhe Liu, Hongwei Yi, Hao Zhang, Muyang Li, Yukang Chen, Han Cai, Sanja Fidler, Ping Luo, Song Han, and Enze Xie. 2025. SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. arXiv:2509.24695 [cs.CV] doi:10.48550/arXiv.2509.24695 arXiv v2 (last revised 13 Oct 2025). [6] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. 2022. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 (2022). [7] Zigeng Chen, Xinyin Ma, Gongfan Fang, Zhenxiong Tan, and Xinchao Wang. 2024. AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising. In Advances in Neural Information Processing Systems, Amir Globerson, Lester Mackey, Danielle Belgrave, Afra Fan, Ugo Paquet, Jakub Tomczak, and Cheng Zhang (Eds.), Vol. 37. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2024/ file/ad15848baa3932c0d2deabf0e11d1dcd-Paper-Conference.pdf [8] Jason Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. 2019. Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings. In ICASSP 2019 – 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Brighton, UK, 3852–3856. doi:10.1109/ICASSP.2019.8682475 [9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. arXiv:2403.03206 [cs.CV] doi:10.48550/arXiv.2403.03206 [10] Zach Evans, Julian D. Parker, C. J. Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. 2024. Stable Audio Open. arXiv:2407.14358 [cs.SD] doi:10.48550/arXiv.2407.14358 [11] Jiarui Fang, Jinzhe Pan, Aoyu Li, Xibo Sun, and Jiannan Wang. 2025. PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference. In Advances in Neural Information Processing Systems (NeurIPS 2025). to appear. https://neurips.cc/virtual/2025/loc/sandiego/poster/119821 NeurIPS 2025 poster. arXiv:2405.14430. [12] Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. 2024. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. arXiv preprint arXiv:2411.01738 (2024). https: //arxiv.org/abs/2411.01738 [13] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. 2023. Text-to-Audio Generation using Instruction Guided Latent Diffusion Model. In Proceedings of the 31st ACM International
3 HKUST
4 NVIDIA
5 Texas A&M University
Conference on Multimedia (MM ’23). ACM, 3590–3598. doi:10.1145/ 3581783.3612348 [14] Martin Gonzalez, Nelson Fernandez, Thuy Tran, Elies Gherbi, Hatem Hajri, and Nader Masmoudi. 2023. SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models. In Advances in Neural Information Processing Systems, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.), Vol. 36. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2023/ file/d6f764aae383d9ff28a0f89f71defbd9-Paper-Conference.pdf [15] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. 2023. Improving Sample Quality of Diffusion Models Using SelfAttention Guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 7462–7471. doi:10.1109/ICCV51070. 2023.00686 [16] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench: Comprehensive Benchmark Suite for Video Generative Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 21807–21818. https://openaccess.thecvf.com/content/CVPR2024/html/Huang_ VBench_Comprehensive_Benchmark_Suite_for_Video_Generative_ Models_CVPR_2024_paper.html [17] Vlado Keselj. 2009. Speech and Language Processing (second edition) Daniel Jurafsky and James H. Martin (Stanford University and University of Colorado at Boulder) Pearson Prentice Hall, 2009, xxxi+988 pp; hardbound, ISBN 978-0-13-187321-6, $115.00. Computational Linguistics 35, 3 (09 2009), 463–466. doi:10.1162/coli.B09-001 [18] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. AudioCaps: Generating Captions for Audios in The Wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 119–132. doi:10.18653/v1/N19-1011 [19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4015–4026. arXiv:2304.02643 doi:10.48550/arXiv.2304.02643 [20] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. 2024. HunyuanVideo: A Systematic Framework For Large Video Generative Models. arXiv:2412.03603 [cs.CV] doi:10.48550/arXiv.2412.03603 [21] Khaled Koutini, Jan Schlueter, Hamid Eghbal-zadeh, and Gerhard Widmer. 2022. Efficient Training of Audio Transformers with Patchout. In Proc. Interspeech 2022. 2753–2757. doi:10.21437/Interspeech.2022227 [22] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). ACM, Koblenz, Germany, 611–626. doi:10.1145/3600006.3613165
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
[23] Black Forest Labs. 2024. FLUX. https://github.com/black-forest-labs/ flux. [24] Junyoung Lee, Seohyun Kim, Shinhyoung Jang, Jongho Park, and Yeseong Kim. 2025. Diffusion-Based Generative System Surrogates for Scalable Learning-Driven Optimization in Virtual Playgrounds. ACM SIGMETRICS Performance Evaluation Review 53, 1 (2025), 43–45. doi:10.1145/3744970.3727282 [25] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. 2024. DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7183–7193. doi:10.1109/CVPR52733.2024.00686 [26] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. 2023. Q-Diffusion: Quantizing Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 17535–17545. doi:10.1109/ ICCV51070.2023.01608 [27] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. 2023. SnapFusion: Textto-Image Diffusion Model on Mobile Devices within Two Seconds. In Advances in Neural Information Processing Systems. arXiv:2306.00980 doi:10.48550/arXiv.2306.00980 [28] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 663–679. https://www.usenix.org/ conference/osdi23/presentation/li-zhouhan [29] Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. 2024. Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7817–7826. https://openaccess.thecvf.com/content/CVPR2024/ papers/Liu_Towards_Understanding_Cross_and_Self-Attention_in_ Stable_Diffusion_for_Text-Guided_CVPR_2024_paper.pdf [30] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. 2023. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 21450–21474. arXiv:2301.12503 doi:10.48550/arXiv.2301.12503 [31] Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Faccio, Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan-Manuel Perez-Rua, and Jürgen Schmidhuber. 2025. Faster Diffusion Through Temporal Attention Decomposition. Transactions on Machine Learning Research (Feb. 2025). https://openreview.net/forum?id=xXs2GKXPnH Published: 2025-02-26. [32] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In Advances in Neural Information Processing Systems, Sanmi Koyejo, Sidahmed Mohamed, Alekh Agarwal, Danielle Belgrave, Kyunghyun Cho, and Alice Oh (Eds.), Vol. 35. Curran Associates, Inc. arXiv:2206.00927 https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 260a14acce2a89dad36adc8eefe7c59e-Abstract-Conference.html [33] Runyu Lu, Shiqi He, Wenxuan Tan, Shenggui Li, Ruofan Wu, Jeff J. Ma, Ang Chen, and Mosharaf Chowdhury. 2025. TetriServe: Efficient DiT Serving for Heterogeneous Image Generation. arXiv:2510.01565 [cs.LG] doi:10.48550/arXiv.2510.01565 arXiv v2 (last revised 13 Oct 2025). [34] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. 2023. Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference. arXiv:2310.04378 [cs.CV]
[35] Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. 2025. HPSv3: Towards Wide-Spectrum Human Preference Score. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 15086–15095. https://openaccess. thecvf.com/content/ICCV2025/html/Ma_HPSv3_Towards_WideSpectrum_Human_Preference_Score_ICCV_2025_paper.html [36] Pallavi Naresh. 2023. Get creative with generative AI in Performance Max. Google Ads & Commerce Blog. https://blog.google/products/ads-commerce/get-creative-withgenerative-ai-in-performance-max/ [37] NVIDIA. 2025. NVIDIA TensorRT. https://github.com/NVIDIA/ TensorRT GitHub repository (tag v10.14, commit 3b4ddc1). Accessed 2026-01-13. [38] NVIDIA. 2026. TensorRT-LLM. https://github.com/NVIDIA/TensorRTLLM GitHub repository (tag v1.2.0rc6.post1, commit e4a6c99). Accessed 2026-01-13. [39] OpenAI. 2024. Sora System Card. Technical Report. OpenAI. https: //openai.com/index/sora-system-card/ System card for the Sora video generation model. [40] Nobuyuki Otsu. 1979. A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man, and Cybernetics 9, 1 (1979), 62–66. doi:10.1109/TSMC.1979.4310076 [41] William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4195–4205. arXiv:2212.09748 doi:10.48550/ arXiv.2212.09748 [42] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In The Twelfth International Conference on Learning Representations (ICLR). arXiv:2307.01952 doi:10.48550/arXiv.2307.01952 [43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695. arXiv:2112.10752 doi:10.1109/CVPR52688.2022.01042 [44] Silvestru Sever and Silvestru Dragomir. 2002. Some Gronwall type inequalities and applications. (12 2002). [45] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053 (2019). [46] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising Diffusion Implicit Models. In International Conference on Learning Representations (ICLR). OpenReview.net. arXiv:2010.02502 doi:10.48550/ arXiv.2010.02502 [47] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency Models. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 32211–32252. https://proceedings.mlr.press/v202/ song23a.html [48] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020). [49] Suno. 2024. Suno has raised $125 million to build a future where anyone can make music. Suno Blog. https://suno.com/blog/fundraisingannouncement-may-2024 [50] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. 2025. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv preprint arXiv:2503.20314 (2025). [51] Yuqing Wang, Ye He, and Molei Tao. 2024. Evaluating the Design Space of Diffusion-Based Generative Models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://proceedings.neurips.cc/paper_files/paper/2024/file/ 227404a13d20898dec2018ebe368b202-Paper-Conference.pdf [52] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (April 2004), 600–612. doi:10.1109/TIP.2003.819861 [53] Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. 2023. DiffusionDB: A Largescale Prompt Gallery Dataset for Text-to-Image Generative Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan BoydGraber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 893–911. doi:10.18653/v1/2023.acl-long.51 [54] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor BergKirkpatrick, and Shlomo Dubnov. 2023. Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-toCaption Augmentation. In ICASSP 2023 – 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. doi:10.1109/ICASSP49357.2023.10095969 [55] Yuchen Xia, Divyam Sharma, Yichao Yuan, Souvik Kundu, and Nishil Talati. 2026. MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’26). to appear. https://arxiv.org/abs/ 2503.11972 To appear in ASPLOS 2026. arXiv:2503.11972. [56] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park. 2024. One-step Diffusion with Distribution Matching Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6613– 6623. arXiv:2311.18828 doi:10.1109/CVPR52733.2024.00632 [57] Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Hank Levy, and David E. Culler. 2025. IC-Cache: Efficient Large Language Model Serving via In-context Caching. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP 2025). ACM, 375–398. doi:10.1145/3731569.3764829 [58] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 586–595. doi:10.1109/CVPR. 2018.00068 [59] Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2026. JITServe: SLO-aware LLM Serving with Imprecise Request Information. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI ’26). Renton, WA, USA, to appear. https://arxiv.org/abs/2504.20068 Accepted to NSDI 2026. arXiv:2504.20068. [60] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proceedings of the VLDB Endowment 16, 12 (Aug. 2023), 3848–3860.
3 HKUST
4 NVIDIA
5 Texas A&M University
doi:10.14778/3611540.3611569 [61] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. 2023. Fast Sampling of Diffusion Models via Operator Learning. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 42390–42402. https://proceedings.mlr.press/v202/zheng23d.html [62] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193– 210. https://www.usenix.org/conference/osdi24/presentation/zhongyinmin
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
A
Theoretical Analysis
We provide formal guarantees for FlashDiff’s two core decisions: when to skip patch refinement (§A.2) and where to partition the latent (§A.3). These results establish that FlashDiff’s gating and partitioning mechanisms are grounded in principled error control for the underlying probability flow ODE, rather than being purely heuristic. A.1
Setup and Notation
Consider the probability flow ODE governing the reverse diffusion process [48]: 𝑑𝑥 = 𝑓𝜃 (𝑥, 𝑡), 𝑥 (𝑇 ) ∼ N (0, 𝜎𝑇2 𝐼 ), (2) 𝑑𝑡 where 𝑓𝜃 : R𝑑 ×[0,𝑇 ] → R𝑑 encapsulates the learned velocity (or equivalently the score-based drift) and is evaluated at every denoising step to produce the final sample 𝑥 (0). Euler discretisation. We discretise (2) with the forward Euler method over 𝑁 steps at times 𝑇 = 𝑡 0 > 𝑡 1 > · · · > 𝑡 𝑁 = 0 with uniform step size ℎ = 𝑇 /𝑁 : ∗ 𝑥𝑛+1 = 𝑥𝑛∗ + ℎ 𝑓𝜃 (𝑥𝑛∗ , 𝑡𝑛 ).
(3)
Patch decomposition. The latent 𝑥 ∈ R𝑑 is partitioned 𝑅 satisfying 𝑃 ⊂ into 𝑅 non-overlapping patches {𝑃𝑖 }𝑖=1 𝑖 Ð𝑅 {1, . . . , 𝑑 } and 𝑖=1 𝑃𝑖 = {1, . . . , 𝑑 }. We write 𝑥 (𝑖 ) ∈ R |𝑃𝑖 | for the restriction of 𝑥 to patch 𝑖, and 𝑓𝜃(𝑖 ) for the corresponding components of 𝑓𝜃 . FlashDiff update rule. At step 𝑛, let A𝑛 ⊆ [𝑅] denote the active set determined by the Patch Gate, and S𝑛 = [𝑅] \ A𝑛 the skipped set. FlashDiff updates each patch as ( (𝑖 ) 𝑥ˆ𝑛 + ℎ 𝑓𝜃(𝑖 ) (𝑥ˆ𝑛 , 𝑡𝑛 ), 𝑖 ∈ A𝑛 , (𝑖 ) 𝑥ˆ𝑛+1 = (𝑖 ) (4) 𝑥ˆ𝑛 + ℎ 𝑐𝑛(𝑖 ) , 𝑖 ∈ S𝑛 , where 𝑐𝑛(𝑖 ) = 𝑓𝜃(𝑖 ) (𝑥ˆ𝑛𝑖 , 𝑡𝑛𝑖 ) is the cached drift from the most recent active step 𝑛𝑖 < 𝑛 of patch 𝑖. Forced reactivation. To prevent indefinite staleness, FlashDiff enforces a maximum consecutive skip bound 𝐾: if patch 𝑖 has been skipped for 𝐾 consecutive steps, it is forcibly reactivated at the next step regardless of its gating weight. Formally: 𝑛 − 𝑛𝑖 ≤ 𝐾
for every skipped patch 𝑖 at step 𝑛,
(5)
where 𝑛𝑖 is the most recent step at which patch 𝑖 was active. This guarantees that every cached drift value is at most 𝐾 steps old. Skip perturbation. Define 𝛿𝑛 ∈ R𝑑 component-wise by ( 0, 𝑖 ∈ A𝑛 , (𝑖 ) 𝛿𝑛 = (𝑖 ) (6) (𝑖 ) 𝑐𝑛 − 𝑓𝜃 (𝑥ˆ𝑛 , 𝑡𝑛 ), 𝑖 ∈ S𝑛 . This allows us to write the FlashDiff update compactly as 𝑥ˆ𝑛+1 = 𝑥ˆ𝑛 + ℎ 𝑓𝜃 (𝑥ˆ𝑛 , 𝑡𝑛 ) + 𝛿𝑛 . (7)
Patch Gate notation (recap from main text). At each active step of patch 𝑖, the gate computes the refinement importance 𝑅𝑖(𝑛) from self-attention and maintains: Δ𝑖(𝑛) = |𝑅𝑖(𝑛) − 𝑅¯𝑖 |,
𝑅¯𝑖 ← 𝑅𝑖(𝑛) ,
(8)
with Δ𝑖(𝑛) frozen at its last-active value when patch 𝑖 is skipped. We denote the frozen (possibly stale) value by e Δ𝑖(𝑛) and the hypothetical fresh value (if patch 𝑖 were recomputed) by Δ𝑖(𝑛),★. By (5), the staleness satisfies e Δ𝑖(𝑛) = Δ𝑖(𝑛𝑖 ) with 𝑛 − 𝑛𝑖 ≤ 𝐾. The normalised weight is 𝑤𝑖(𝑛) =
e Δ𝑖(𝑛) + 𝜂 , 𝐷𝑛
𝐷𝑛 =
𝑅 ∑︁
e Δ (𝑛) 𝑗 +𝜂 ,
(9)
𝑗=1
and patch 𝑖 is skipped when 𝑤𝑖(𝑛) < 𝜀. A.2
When to Skip: Accumulated Error Bound
We state three assumptions, then combine them into the main theorem. The first two are standard; the third formalises the relationship between self-attention dynamics and drift perturbation, accounting for the forced-reactivation mechanism. Assumption A.1 (Lipschitz Drift). The drift 𝑓𝜃 (·, 𝑡) is 𝐿Lipschitz in its first argument uniformly over 𝑡 ∈ [0,𝑇 ]: ∥ 𝑓𝜃 (𝑥, 𝑡) − 𝑓𝜃 (𝑦, 𝑡) ∥ ≤ 𝐿 ∥𝑥 −𝑦 ∥,
∀ 𝑥, 𝑦 ∈ R𝑑 , 𝑡 ∈ [0,𝑇 ].
This is standard in the score-based diffusion literature [6, 48] and holds for neural networks with bounded weights and Lipschitz activations (e.g. SiLU, LayerNorm with bounded inputs). Assumption A.2 (Temporal Smoothness of Drift). The drift 𝑓𝜃 (𝑥, ·) is 𝐿𝑡 -Lipschitz in time uniformly over 𝑥: ∥ 𝑓𝜃 (𝑥, 𝑡) − 𝑓𝜃 (𝑥, 𝑠) ∥ ≤ 𝐿𝑡 |𝑡 − 𝑠 |,
∀ 𝑥 ∈ R𝑑 , 𝑡, 𝑠 ∈ [0,𝑇 ].
This captures the smoothness of the denoising trajectory across steps and is likewise standard [4, 6]. Assumption A.3 (Fresh Attention–Drift Coherence). There exists a constant 𝐶 0 > 0 such that, for every patch 𝑖 and step 𝑛, if patch 𝑖 were freshly computed at step 𝑛, the drift perturbation relative to the cached value is bounded by the fresh attention change: ∥𝛿𝑛(𝑖 ) ∥ ≤ 𝐶 0 · Δ𝑖(𝑛),★ .
(10)
Justification. The drift 𝑓𝜃(𝑖 ) is obtained by passing selfattention outputs through post-attention layers (MLP, normalisation, linear projection). If these layers are collectively 𝐿out -Lipschitz, then changes in self-attention propagate proportionally to changes in drift: ∥ 𝑓𝜃(𝑖 ) (𝑥ˆ𝑛 , 𝑡𝑛 )−𝑓𝜃(𝑖 ) (𝑥ˆ𝑛𝑖 , 𝑡𝑛𝑖 ) ∥ ≤ 𝐿out ·∥SA (𝑖 ) (𝑥ˆ𝑛 , 𝑡𝑛 )−SA (𝑖 ) (𝑥ˆ𝑛𝑖 , 𝑡𝑛𝑖 ) ∥,
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
and Δ𝑖(𝑛),★ measures the right-hand side up to the per-patch normalisation factor |A𝑖 |, giving 𝐶 0 = 𝐿out /|A𝑖 |. The constant 𝐶 0 is model-specific and may additionally reflect contributions from cross-attention and residual pathways. Note that Assumption A.3 only requires coherence for freshly computed quantities, avoiding any circularity with stale values. We now show that forced reactivation lets us bridge from the stale gate signal e Δ𝑖(𝑛) to the fresh value Δ𝑖(𝑛),★, yielding an effective coherence bound that uses only observable (stale) quantities. Lemma A.4 (Staleness Gap). Under Assumptions A.1 and A.2, for any skipped patch 𝑖 at step 𝑛 with last-active step 𝑛𝑖 satisfying 𝑛 − 𝑛𝑖 ≤ 𝐾: |Δ𝑖(𝑛),★ − e Δ𝑖(𝑛) | ≤ 𝛽 · 𝐾ℎ,
(11)
where 𝛽 = 𝐿SA (𝐿 𝑀 𝑓 + 𝐿𝑡 ), 𝐿SA is the Lipschitz constant of the self-attention map with respect to (𝑥, 𝑡), and 𝑀 𝑓 = sup𝑥,𝑡 ∥ 𝑓𝜃 (𝑥, 𝑡) ∥ is the drift magnitude bound. Proof. The fresh attention-change magnitude measures how the self-attention output of patch 𝑖 has changed between step 𝑛𝑖 and step 𝑛: Δ𝑖(𝑛),★ ∝ ∥SA (𝑖 ) (𝑥ˆ𝑛 , 𝑡𝑛 ) − SA (𝑖 ) (𝑥ˆ𝑛𝑖 , 𝑡𝑛𝑖 ) ∥. Δ𝑖(𝑛) = Δ𝑖(𝑛𝑖 ) was computed at step 𝑛𝑖 and The stale value e reflects the change observed at that earlier time. Between steps 𝑛𝑖 and 𝑛, the latent state evolves by at most (𝑛 − 𝑛𝑖 ) Euler steps. At each step, the state changes by ℎ ∥ 𝑓𝜃 (𝑥ˆ𝑘 , 𝑡𝑘 ) ∥ ≤ ℎ 𝑀 𝑓 , so
3 HKUST
4 NVIDIA
5 Texas A&M University
Proof. By Assumption A.3: ∥𝛿𝑛(𝑖 ) ∥ ≤ 𝐶 0 Δ𝑖(𝑛),★. By the triangle inequality and Lemma A.4: Δ𝑖(𝑛),★ ≤ e Δ𝑖(𝑛) + 𝛽𝐾ℎ. Combining gives (12).
□
Lemma A.5 cleanly separates the “ideal” coherence (𝐶 0 ) from the staleness penalty (𝛾), and shows that the penalty is controlled by the maximum skip duration 𝐾 and the step size ℎ—both of which are small in practice (𝐾 ∼ 3–5, ℎ = 𝑇 /𝑁 ∼ 0.02 for 𝑁 = 50 steps). We now state and prove the main theorem. Theorem A.6 (Accumulated Skip Error Bound). Under Assumptions A.1–A.3 and the forced-reactivation constraint (5), 𝑁 be the standard Euler trajectory (3) and {𝑥ˆ } 𝑁 let {𝑥𝑛∗ }𝑛=0 𝑛 𝑛=0 the FlashDiff trajectory (4), both starting from the same initial noise 𝑥 0∗ = 𝑥ˆ0 . Then the terminal error satisfies ∥𝑥ˆ𝑁 − 𝑥 𝑁∗ ∥ ≤ 𝑒 𝐿𝑇
𝑁 −1 ∑︁ √ 𝑅 𝐶0 𝜀 ℎ 𝐷𝑛 + 𝛾 𝑇
(13)
𝑛=0
where 𝜀 is the gating threshold, 𝑅 is the number of patches, Í 𝐷𝑛 = 𝑅𝑗=1 (e Δ (𝑛) 𝑗 + 𝜂) is the per-step total refinement activity measured from (possibly stale) gate signals, 𝑇 = 𝑁ℎ, and 𝛾 = 𝐶 0 𝛽𝐾ℎ is the staleness penalty from Lemma A.5. Proof. Define the error 𝑒𝑛 = 𝑥ˆ𝑛 − 𝑥𝑛∗ . By construction 𝑒 0 = 0. Step 1 (Error recursion). Subtracting (3) from (7): 𝑒𝑛+1 = 𝑒𝑛 + ℎ 𝑓𝜃 (𝑥ˆ𝑛 , 𝑡𝑛 ) − 𝑓𝜃 (𝑥𝑛∗ , 𝑡𝑛 ) + ℎ 𝛿𝑛 . (14) Step 2 (Lipschitz term). By Assumption A.1:
∥𝑥ˆ𝑛 − 𝑥ˆ𝑛𝑖 ∥ ≤ (𝑛 − 𝑛𝑖 ) ℎ 𝑀 𝑓 ≤ 𝐾 ℎ 𝑀 𝑓 .
∥ 𝑓𝜃 (𝑥ˆ𝑛 , 𝑡𝑛 ) − 𝑓𝜃 (𝑥𝑛∗ , 𝑡𝑛 ) ∥ ≤ 𝐿 ∥𝑒𝑛 ∥.
The time gap is (𝑛 − 𝑛𝑖 ) ℎ ≤ 𝐾 ℎ. Since the self-attention map SA (𝑖 ) is 𝐿SA -Lipschitz with respect to (𝑥, 𝑡) jointly (this holds for softmax attention with bounded queries, keys, and values):
(15)
Step 3 (Skip perturbation bound). Since 𝛿𝑛(𝑖 ) = 0 for 𝑖 ∈ A𝑛 : ∑︁ ∥𝛿𝑛 ∥ 2 = ∥𝛿𝑛(𝑖 ) ∥ 2 . 𝑖 ∈ S𝑛
∥SA (𝑖 ) (𝑥ˆ𝑛 , 𝑡𝑛 ) − SA (𝑖 ) (𝑥ˆ𝑛𝑖 , 𝑡𝑛𝑖 ) ∥
By Lemma A.5, for each 𝑖 ∈ S𝑛 :
≤ 𝐿SA ∥𝑥ˆ𝑛 − 𝑥ˆ𝑛𝑖 ∥ + |𝑡𝑛 − 𝑡𝑛𝑖 | ≤ 𝐿SA 𝐾 ℎ 𝑀 𝑓 + 𝐾 ℎ = 𝐿SA (𝑀 𝑓 + 1) 𝐾ℎ.
∥𝛿𝑛(𝑖 ) ∥ ≤ 𝐶 0 e Δ𝑖(𝑛) + 𝛾 .
The difference between the fresh and stale Δ values is controlled by how much the self-attention landscape has shifted over at most 𝐾 steps, giving the stated bound with 𝛽 = 𝐿SA (𝐿 𝑀 𝑓 + 𝐿𝑡 ). (The exact form of 𝛽 absorbs the normalisation factor from the per-patch averaging.) □
(16)
The gating condition 𝑤𝑖(𝑛) < 𝜀 implies (from (9)): e Δ𝑖(𝑛) + 𝜂 < 𝜀 𝐷𝑛
=⇒
e Δ𝑖(𝑛) < 𝜀 𝐷𝑛 .
(17)
Substituting into (16): ∥𝛿𝑛(𝑖 ) ∥ ≤ 𝐶 0 𝜀 𝐷𝑛 + 𝛾 .
(18)
Lemma A.5 (Effective Coherence with Stale Signals). Under Assumptions A.1–A.3 and the forced-reactivation constraint (5), for every skipped patch 𝑖 at step 𝑛:
Aggregating over skipped patches (using ∥·∥ 2 across orthogonal patch dimensions): √︁ √ ∥𝛿𝑛 ∥ ≤ |S𝑛 | 𝐶 0 𝜀 𝐷𝑛 + 𝛾 ≤ 𝑅 𝐶 0 𝜀 𝐷𝑛 + 𝛾 . (19)
∥𝛿𝑛(𝑖 ) ∥ ≤ 𝐶 0 · e Δ𝑖(𝑛) + 𝐶 0 𝛽𝐾ℎ =: 𝐶 0 · e Δ𝑖(𝑛) + 𝛾,
Step 4 (Gronwall recursion). Taking norms in (14) and combining: √ ∥𝑒𝑛+1 ∥ ≤ (1 + ℎ𝐿) ∥𝑒𝑛 ∥ + ℎ 𝑅 𝐶 0 𝜀 𝐷𝑛 + 𝛾 . (20)
(12)
where 𝛾 = 𝐶 0 𝛽𝐾ℎ is a small constant that captures the worstcase staleness penalty.
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
This is a discrete Gronwall inequality 𝑎𝑛+1 ≤ (1 + 𝛼) 𝑎𝑛 + 𝑏𝑛 √ with 𝑎 0 = 0, 𝛼 = ℎ𝐿, and 𝑏𝑛 = ℎ 𝑅(𝐶 0𝜀𝐷𝑛 +𝛾). The standard solution gives: ∥𝑒 𝑁 ∥ ≤
𝑁 −1 ∑︁ 𝑛=0
𝑁 −1 Ö
𝑏𝑛
(1 + 𝛼)
𝑘=𝑛+1 𝑁 −1 ∑︁ 𝑁
≤ (1 + ℎ𝐿) √ ≤ 𝑒 𝐿𝑇 𝑅
𝑏𝑛
𝑛=0 𝑁 −1 ∑︁
ℎ 𝐶 0𝜀𝐷𝑛 + 𝛾
𝑛=0 𝑁 −1 𝑁 −1 ∑︁ ∑︁ √ = 𝑒 𝐿𝑇 𝑅 𝐶 0𝜀 ℎ𝐷𝑛 + 𝛾 ℎ , 𝑛=0
(21)
𝑛=0
|{z} =𝑇
where we used (1 + ℎ𝐿) 𝑁 ≤ 𝑒 𝑁 ℎ𝐿 = 𝑒 𝐿𝑇 .
□
Remark A.7 (Structure of the bound). The error bound (13) consists of two additive terms, each with a clear operational meaning: Í (a) Gating-controlled term 𝐶 0𝜀 𝑛 ℎ𝐷𝑛 : This is the error from patches that were skipped because the Patch Gate judged them unimportant. It is directly proportional to the threshold 𝜀 and vanishes as 𝜀 → 0 (i.e., never skip). (b) Staleness-penalty term 𝛾𝑇 : This is the residual error from using stale Δ values instead of fresh ones. It is proportional to 𝛾 = 𝐶 0 𝛽𝐾ℎ. With typical values (𝐾 = 5, ℎ = 0.02, 𝐶 0 𝛽 ∼ 𝑂 (1)), 𝛾 ∼ 0.1, and 𝛾𝑇 ∼ 0.1 is small. Crucially, 𝛾 shrinks as 𝐾 decreases (more frequent reactivation) or as ℎ decreases (more denoising steps), providing two independent control knobs. Remark A.8 (Recovering the ideal bound). In the hypothetical case where all Δ values are fresh (e.g., 𝐾 = 1, meaning every patch is recomputed at every step—no skipping), the staleness penalty vanishes (𝛾 → 0),√and the bound reduces to Í the simpler form ∥𝑥ˆ𝑁 − 𝑥 𝑁∗ ∥ ≤ 𝐶 0𝜀 𝑅 𝑒 𝐿𝑇 𝑛 ℎ𝐷𝑛 , matching the “ideal coherence” bound. A.3
Where to Partition: Optimality of Saliency-Based Splitting
We establish that FlashDiff’s Otsu-based partitioning minimises the error induced by the Patch Gate. 𝑅 and Definition A.9 (Skip Risk). For a partition P = {𝑃𝑖 }𝑖=1 gating threshold 𝜀, the skip risk is "𝑁 −1 # ∑︁ R (P, 𝜀) = E ∥𝛿𝑛 ∥ 2 , (22) 𝑛=0
where the expectation is over the initial noise 𝑥 0 N (0, 𝜎𝑇2 𝐼 ).
∼
The skip risk governs the error bound in Theorem A.6: a partition that reduces skip risk yields a tighter quality guarantee. Intuitively, mixing high-saliency tokens (which need frequent refinement) with low-saliency tokens (which can be safely skipped) forces the gate into a lose-lose choice: either skip the mixed patch and damage high-saliency content, or execute it entirely and waste computation on converged tokens. Assumption A.10 (Saliency–Perturbation Monotonicity). Higher-saliency tokens incur larger drift perturbation when skipped. Formally, for tokens 𝑗1, 𝑗2 with saliency values 𝑆 𝑗1 > 𝑆 𝑗2 , the expected per-step drift perturbation satisfies E[∥𝛿𝑛( 𝑗1 ) ∥] ≥ E[∥𝛿𝑛( 𝑗2 ) ∥]. This is natural: high saliency indicates strong cross-attention to prompt tokens, meaning the model is actively generating content in that region. Skipping such tokens disrupts ongoing refinement more than skipping already-converged regions. Proposition A.11 (Optimality of Saliency-Based Partitioning). Consider the binary partition of the latent into a focus set F𝜃 = { 𝑗 : 𝑆 𝑗 ≥ 𝜃 } and a context set C𝜃 = { 𝑗 : 𝑆 𝑗 < 𝜃 }, parameterised by the threshold 𝜃 . Under Assumption A.10, the Otsu threshold 2 𝜃 ∗ = arg max 𝜎𝐵2 (𝜃 ) = arg max 𝜔 𝑓 (𝜃 ) 𝜔𝑐 (𝜃 ) 𝜇 𝑓 (𝜃 )−𝜇𝑐 (𝜃 ) 𝜃
𝜃
(23) (where 𝜔 𝑓 , 𝜔𝑐 are class proportions and 𝜇 𝑓 , 𝜇𝑐 class means of the saliency values) simultaneously achieves: 2 (𝜃 ). (i) Minimum pooled within-class saliency variance 𝜎𝑊 (ii) Minimum worst-case intra-class skip-risk heterogeneity under Assumption A.10. Proof. Part (i). The total saliency variance decomposes as 2 2 𝜎total = 𝜎𝐵2 (𝜃 ) + 𝜎𝑊 (𝜃 ),
(24)
where 2 𝜎𝑊 (𝜃 ) = 𝜔 𝑓 (𝜃 ) Var(𝑆 | F𝜃 ) + 𝜔𝑐 (𝜃 ) Var(𝑆 | C𝜃 ). 2 Since 𝜎total is independent of 𝜃 , maximising 𝜎𝐵2 (𝜃 ) is equiva2 (𝜃 ). lent to minimising 𝜎𝑊 Part (ii). By Assumption A.10, the per-token skip perturbation is a monotone non-decreasing function of saliency 𝑆 𝑗 . Therefore, the within-class variance of skip perturbation magnitudes is bounded by the within-class variance of saliency values (via the variance-preserving property of monotone transformations applied to class-conditional distributions): Var ∥𝛿 ( 𝑗 ) ∥ | 𝑗 ∈ F𝜃 ≤ 𝜅 2 Var(𝑆 𝑗 | 𝑗 ∈ F𝜃 ), (25)
where 𝜅 is the Lipschitz constant of the saliency-toperturbation mapping (which exists by monotonicity and boundedness of both quantities). The analogous bound holds for C𝜃 .
1
Yaqi Qiao1 , Ping He12,♦ , Songrun Xie3 , Ayush Barik1 , Chensong Zhang4 , Zhengzhong Tu4 , Fan Lai1 1 University of Illinois Urbana-Champaign
2 Vanderbilt University
Combining (25) with Part (i), the Otsu threshold minimises the pooled within-class perturbation variance. For the minimax criterion, note that the load-balanced allocation in Eq. (1) of the main text ensures 𝜔 𝑓 ≈ 𝜔𝑐 ≈ 1/2 (up to the discrete approximation). Under balanced class sizes, 2 = 1 Var(𝑆 |F ) + 1 Var(𝑆 |C) also minimises minimising 𝜎𝑊 2 2 max{Var(𝑆 |F ), Var(𝑆 |C)} whenever the two within-class variances are comparable—which the balanced split encourages. □ Remark A.12 (Hierarchical extension). The binary Otsu split is applied recursively in FlashDiff’s Patch Partitioner. At each recursion level, the sub-partition is locally optimal in the sense of Proposition A.11. Since the monotonicity in Assumption A.10 is inherited by sub-ranges of the saliency distribution, each sub-split preserves intra-class homogeneity at every granularity level. A.4
Quality Metric Bound and Tradeoff Characterization
Theorem A.6 bounds the terminal latent error ∥𝑥ˆ𝑁 − 𝑥 𝑁∗ ∥. In practice, we care about perceptual quality metrics such as PSNR, SSIM [52], LPIPS [58], or human preference scores. We now show that the latent bound directly translates into a quality-metric bound under a mild continuity assumption. Assumption A.13 (Quality Continuity). Let 𝑄 : R𝑑 → R be a terminal quality functional that maps a denoised latent to a scalar quality score (e.g., PSNR, negative LPIPS). We assume 𝑄 is 𝐾𝑄 -Lipschitz: |𝑄 (𝑥) − 𝑄 (𝑦)| ≤ 𝐾𝑄 ∥𝑥 − 𝑦 ∥,
∀ 𝑥, 𝑦 ∈ R𝑑 .
This is satisfied by all commonly used quality metrics when restricted to the bounded range of valid latent representations: PSNR and SSIM are continuous functions of pixel values, LPIPS is computed by a neural network with bounded weights and Lipschitz activations, and HPSv2 is similarly Lipschitz. Corollary A.14 (Bounded Quality Gap). Under the assumptions of Theorem A.6 and Assumption A.13: |𝑄 (𝑥ˆ𝑁 ) − 𝑄 (𝑥 𝑁∗ )| ≤ 𝐾𝑄 𝑒 𝐿𝑇
𝑁 −1 ∑︁ √ 𝑅 𝐶 0𝜀 ℎ𝐷𝑛 + 𝛾𝑇 𝑛=0
(26)
3 HKUST
4 NVIDIA
5 Texas A&M University
The theorem does not claim that FlashDiff must produce lower quality than full execution. In practice, selective refinement can occasionally improve perceptual quality by avoiding over-refinement of already-converged regions (a form of implicit regularisation), as observed in Table 2 of the main text where FlashDiff sometimes achieves slightly higher quality scores. The bound captures the worst case; the typical case is often better. Corollary A.15 (Explicit Tradeoff Curve). Let 𝜌¯ = 1 Í𝑁 −1 ¯ = 𝑁 𝑅Í 𝑛=0 |S𝑛 | denote the average skip rate and 𝐷 𝑁 −1 1 𝑛=0 𝐷𝑛 the mean total refinement activity. Then: 𝑁 |𝑄 (𝑥ˆ𝑁 ) − 𝑄 (𝑥 𝑁∗ )| ≤ (i) Quality bound: √ 𝐿𝑇 𝑅 𝐶 0𝜀 𝑇 𝐷¯ + 𝛾𝑇 . 𝐾𝑄 𝑒 (ii) Computational savings: FLOPssaved ∝ 𝜌¯ 𝑁 𝑅. (iii) Quality-constrained gating: For a target quality tolerance 𝜏 > 0, setting √ 𝜏/(𝐾𝑄 𝑒 𝐿𝑇 𝑅) − 𝛾𝑇 (27) 𝜀 ≤ 𝐶 0 𝑇 𝐷¯ guarantees |𝑄 (𝑥ˆ𝑁 ) −𝑄 (𝑥 𝑁∗ )| ≤ 𝜏 while maximising the achievable skip rate. Í Proof. Part (i) follows from Corollary A.14 with 𝑛 ℎ 𝐷𝑛 = ¯ Part (ii) holds because each skipped patch-step avoids 𝑇 𝐷. one full patch denoising computation. Part (iii) is obtained by inverting the bound in (i). □ Remark A.16 (Operational insights). The bound reveals several design-relevant properties: (a) Linear control via 𝜀: The quality gap is linear in the gating threshold, theoretically confirming the smooth, monotonic quality–efficiency tradeoff observed in §6.4 of the main text. √ (b) Sub-linear patch-count dependence: The 𝑅 factor reflects that more patches create more independent perturbation sources, but the ℓ2 aggregation yields sub-linear growth. (c) Prompt-adaptive: The cumulative activity 𝑇 𝐷¯ is prompt-specific. Simpler prompts yield tighter bounds, explaining the higher skip rates observed for simpler content (cf. Figure 13 in the main text). (d) Staleness vanishes with frequent reactivation: As 𝐾 → 1 or ℎ → 0, the penalty 𝛾 → 0, and the bound converges to the ideal (zero-staleness) form.
Proof. Immediate from Theorem A.6 and Assumption A.13: |𝑄 (𝑥ˆ𝑁 ) − 𝑄 (𝑥 𝑁∗ )| ≤ 𝐾𝑄 ∥𝑥ˆ𝑁 − 𝑥 𝑁∗ ∥ ≤ 𝐾𝑄 · [RHS of (13)]. □
A.5
Tightening the Bound: Contractivity of Reverse Diffusion
Interpretation. Corollary A.14 states that FlashDiff is a bounded perturbation of full execution: its quality gap to the no-skip baseline cannot grow arbitrarily as long as the gating threshold 𝜀 and the staleness penalty 𝛾 remain controlled. The bound is agnostic to the particular choice of quality metric—any Lipschitz-continuous functional 𝑄 yields a valid guarantee.
The exponential factor 𝑒 𝐿𝑇 in Theorem A.6 arises from a worst-case Gronwall analysis. In practice, the reverse diffusion process is contractive at later steps as the trajectory approaches the data manifold. Assumption A.17 (Log-Sobolev Data Distribution). The target distribution 𝑝 data satisfies a log-Sobolev inequality with constant 𝛼 > 0.
FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving
Songrun Xie3, Ayush Barik1, Chensong Zhang4, Zhengzhong Tu4, Fan Lai1 1University of Illinois Urbana-Champaign 2Vanderbi Proof. For a skipped patch 𝑖, its state evolves via cached drift: Theorem A.18 (Improved Bound under Contractivity). Un(𝑖 ) (𝑖 ) der Assumptions A.1–A.17, if the effective drift Lipschitz con𝑥ˆ𝑚+1 = 𝑥ˆ𝑚(𝑖 ) + ℎ 𝑐𝑚 for 𝑚 = 𝑛𝑖 , . . . , 𝑛 − 1. Thus: stant at step 𝑛 satisfies 𝐿𝑛 ≤ 𝐿 − 𝛼 𝛾𝑐 (𝑡𝑛 ) for a monotonically (𝑖 ) ∥𝑥ˆ𝑛(𝑖 ) − 𝑥ˆ𝑛(𝑖𝑖 ) ∥ ≤ (𝑛 − 𝑛𝑖 ) ℎ max ∥𝑐𝑚 ∥ ≤ 𝐾 ℎ 𝑀𝑓 . increasing function 𝛾𝑐 : [0,𝑇 ] → [0, 1] with 𝛾𝑐 (0) = 0 and 𝑚 𝛾𝑐 (𝑇 ) = 1, then: Summing over at most 𝑅 skipped patches and weighting by ∥𝑥ˆ𝑁 − 𝑥 𝑁∗ ∥ ≤ 𝑒 (𝐿−𝛼𝛾¯𝑐 ) 𝑇
𝑁 −1 ∑︁ √ 𝑅 𝐶 0𝜀 ℎ𝐷𝑛 + 𝛾𝑇 ,
𝐿KV :
(28)
𝑛=0
∫ 1 𝑇
where 𝛾¯𝑐 = 𝑇
0
𝛾𝑐 (𝑡) 𝑑𝑡 ∈ (0, 1).
Proof. The proof follows Theorem A.6, replacing the uniform Lipschitz constant 𝐿 in (20) with the step-dependent 𝐿𝑛 ≤ 𝐿 − 𝛼𝛾𝑐 (𝑡𝑛 ): √ ∥𝑒𝑛+1 ∥ ≤ 1 + ℎ(𝐿 − 𝛼𝛾𝑐 (𝑡𝑛 )) ∥𝑒𝑛 ∥ + ℎ 𝑅(𝐶 0𝜀𝐷𝑛 + 𝛾). Unrolling: −1 𝑁 −1 Ö √ 𝑁∑︁ ℎ(𝐶 0𝜀𝐷𝑛 + 𝛾) 1 + ℎ(𝐿 − 𝛼𝛾𝑐 (𝑡𝑘 )) ∥𝑒 𝑁 ∥ ≤ 𝑅 𝑛=0
√ ≤ 𝑅
𝑁 −1 ∑︁
𝑘=𝑛+1 −1 𝑁∑︁
ℎ(𝐶 0𝜀𝐷𝑛 + 𝛾) exp
𝑛=0
=𝑒
(𝐿−𝛼𝛾¯𝑐 )𝑇
ℎ(𝐿 − 𝛼𝛾𝑐 (𝑡𝑘 ))
𝑘=0
∑︁ √ 𝑅 𝐶 0𝜀 ℎ𝐷𝑛 + 𝛾𝑇 .
□
𝑛
Remark A.19. For typical natural image distributions, 𝛼 > 0 and 𝛾¯𝑐 > 0, reducing the exponential factor from 𝑒 𝐿𝑇 to 𝑒 (𝐿−𝛼𝛾¯𝑐 )𝑇 , which can be several orders of magnitude smaller. This partially addresses the conservativeness of the Gronwall-based bound; the remaining gap between the theoretical bound and empirical observation reflects the inherent looseness of worst-case analysis over all possible prompts and initial noise realisations. A.6
Second-Order Analysis: Stale KV-Cache Effects
In practice, active patches compute attention using cached KV entrieso from skipped patches rather than fresh values. We show this introduces only a second-order correction. Proposition A.20 (Stale-KV Perturbation). Let 𝐿KV denote the Lipschitz constant of the transformer output with respect to its KV-cache entries. The additional per-step perturbation due to stale KV values satisfies ∑︁ ∥𝛿𝑛KV ∥ ≤ 𝐿KV ∥𝑥ˆ𝑛(𝑖 ) − 𝑥ˆ𝑛(𝑖𝑖 ) ∥. (29) 𝑖 ∈ S𝑛
Moreover, the cumulative state drift of each skipped patch satisfies ∥𝑥ˆ𝑛(𝑖 ) − 𝑥ˆ𝑛(𝑖𝑖 ) ∥ ≤ 𝐾ℎ𝑀 𝑓 , where 𝑀 𝑓 = sup𝑥,𝑡 ∥ 𝑓𝜃 (𝑥, 𝑡) ∥. Consequently, the stale-KV correction to the terminal error bound is 𝑂 (𝐾ℎ), which enters multiplicatively with the existing 𝑂 (𝜀) skip perturbation, yielding an 𝑂 (𝜀𝐾ℎ) cross-term dominated by the main bound when 𝐾ℎ ≪ 1.
∥𝛿𝑛KV ∥ ≤ 𝑅 𝐿KV 𝐾 ℎ 𝑀 𝑓 . Adding this to (20) yields a correction of ℎ · 𝑅 𝐿KV 𝐾ℎ𝑀 𝑓 per step. After Gronwall summation over 𝑁 steps, this contributes 𝑒 𝐿𝑇 · 𝑅 𝐿KV 𝐾ℎ𝑀 𝑓 𝑇 to the terminal error—a term proportional to 𝐾ℎ, which is small (e.g., 5 × 0.02 = 0.1) and adds linearly to the existing 𝑂 (𝜀) bound. □ Limitations of this analysis. This appendix compares two execution policies—FlashDiff and full (no-skip) execution—of the same pretrained diffusion model from the same initial noise. It does not bound either policy’s distance to the true data distribution, which would require additional assumptions about the model’s score approximation error [4, 6]. The Gronwall factor 𝑒 𝐿𝑇 is a worst-case amplification; Theorem A.18 partially mitigates this via contractivity, but the bound may remain conservative for specific prompts. Proposition A.11 proves optimality among binary threshold splits, not among all possible 𝑅-way partitions. The constants 𝐿, 𝐿𝑡 , 𝐶 0 , and 𝛽 are model-specific: while the form of the bound is universal, its numerical tightness varies across architectures and should be validated empirically on each target model. Remark A.21 (Possible practical advantage beyond the bound). The comparative bound above only upper-bounds the deviation of FlashDiff from full execution. It does not say that FlashDiff must be worse. Selective refinement may operate closer to an implicit early-stopping regime that avoids over-refinement of already-converged regions, which can occasionally improve perceptual quality relative to full execution. That effect is empirical and model-dependent, so we keep it separate from the formal guarantee.