Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

arXiv:2607.13770v1 [cs.AR] 15 Jul 2026

Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations Wenxuan Miao

Haosong Liu

Weiming Hu

Shanghai Jiao Tong University Shanghai, China [email protected]

Zihan Liu

Aiyue Chen

Jianlin Yu

Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China [email protected]

Huawei Technologies Shanghai, China [email protected]

Yiwu Yao

Yiming Gan

Jieru Zhao

Huawei Technologies Shanghai, China [email protected]

ICT, Chinese Academy of Sciences Beijing, China [email protected]

Shanghai Jiao Tong University Shanghai, China [email protected]

Jingwen Leng

Minyi Guo

Yu Feng∗

Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China [email protected]

and industry in the last two years. Their ability to generate highfidelity videos has led to rapid adoption in movie editing [17, 80], advertisement [1, 32], virtual world creation [8, 27], and more. Beyond their substantial economic potential, these models also offer a unique lens to understand the physical world by learning geometric and causal relationships in natural scenes. Thus, they are regarded as a foundation for general-purpose artificial intelligence [60]. Today’s mainstream large video generative models are predominantly based on video diffusion transformers (vDiTs). Similar to other diffusion models, a vDiT progressively reverses a forward noise-adding process, iteratively removing latent noises until a video is clean. However, this iterative denoising process is computeintensive. For example, generating even a short 5-second 720p video clip with a state-of-the-art vDiT, HunyuanVideo [35], can take over 30 minutes on a single Nvidia H100 GPU with 80 GB of memory. This inefficiency stems from two fundamental factors. First, the diffusion process consists of many denoising timesteps. Each requires a full forward inference, resulting in long inference latency. Second, each inference performs compute-intensive self-attention over all spatio-temporal tokens, making attention the dominant bottleneck as video resolutions or lengths grow. Recently, the community has made tremendous progress on reducing the number of timesteps via distillation [18, 38, 62, 91], caching [5, 43, 50, 75, 96, 100], etc. As the number of timesteps decreases, self-attention computation becomes the primary bottleneck (Fig. 4). In this paper, we address the compute-intensive self-attention in vDiTs via an algorithm–hardware co-design, Kaleido. Unlike prior work [21, 77, 83, 87, 88], which largely inherits sparse self-attention optimizations from large language models (LLMs), we introduce a

Abstract Video diffusion transformers (vDiTs) generate high-quality video but introduce extremely high compute cost due to the long diffusion timesteps and self-attention computation. As diffusion timesteps are reduced, the computation cost of self-attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatio-temporal correlation of video data. This paper presents Kaleido, an algorithm–hardware co-design that accelerates all operations in vDiTs by exploiting channel-wise spatio-temporal correlations in latent space. Based on this insight, we propose a lightweight channel-wise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic-array-like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9× speedup and 16.0× energy savings over state-of-the-art accelerators.

Keywords Video Diffusion Acceleration, Algorithm-Hardware Co-Design

Introduction

Large video generative models [16, 23, 36, 39, 57, 61, 63, 64, 69, 72, 73, 79, 90, 97] have drawn tremendous attention from both academia ∗ Corresponding Author.

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al. Reverse Markov Process

key insight that exploits the spatio-temporal similarity among latent tokens to dramatically reduce attention computation by up to 85%. The key is to dissect and understand the various patterns of selfattention scores that exist across all vDiT models [18, 35, 73, 84, 91, 97]. Our experiments show that the cause of these patterns is from the channel-wise position encoding used in individual token channels (Sec. 3.2). This encoding mechanism naturally induces spatial and temporal correlations, which accumulate across channels and lead to various patterns in attention scores. With our key insight, we introduce our channel-wise reuse algorithm in Sec. 4, which accelerates not only self-attention but also other operations in vDiTs. Our algorithm simply serves as a plug-in that identifies the correlations between tokens before each operation, e.g., self-attention, and skips unnecessary computations by reusing previous partial results. Specifically, our algorithm first evaluates the similarity of adjacent tokens at the token channel level to identify values that can safely share partial computations. During actual operation, channels with high similarity skip explicit computation and reuse previously computed partial results instead. Evaluations in Sec. 7 show that our algorithm saves significant computations while retaining much higher generative quality (>17dB) compared to prior algorithm-hardware co-designs [21, 34, 87]. While our algorithm significantly reduces overall computation, it introduces two key challenges. First, its reuse patterns are inherently incompatible with the dataflow of existing accelerators, e.g., systolic arrays. Second, reuse further induces irregular sparsity in the input data. To address these challenges, we co-design a reconfigurable processing element (PE) for systolic arrays, as described in Sec. 5.2. Our PE can be reconfigured into multiple execution modes and supports various dataflows, so that PEs can reuse the previously computed results without repeatedly loading them from on-chip memory. In addition, it incorporates mixed-precision computation to support computations with partially reused sparse inputs. To further mitigate the irregular data access patterns introduced by our reuse algorithm, we propose a data dispatcher in Sec. 5.3. The dispatcher applies a lightweight online clustering to group tokens with similar reuse patterns. Meanwhile, a runtime lookahead buffer is designed to merge compatible channels before feeding them into the PE array. Together, we show that our data dispatcher further improves PE utilization and overall accelerator efficiency (Sec. 5.3). We implement our algorithm-hardware co-design, Kaleido. The hardware builds on top of a classic systolic DNN accelerator [31] implemented in 16 nm technology. The Kaleido hardware augments the baseline accelerator with 7.9% area overhead. We evaluate Kaleido on four widely-adopted vDiTs: HunyuanVideo [35], Wan [73], CogVideoX [84] and TurboDiffusion [91]. In addition to the GPU baseline, we also include four recent diffusion accelerators [21, 33, 34, 87]. Kaleido achieves up to 5.9× speedup and 16.0× energy savings. The contributions of this paper are as follows:

Encode

(

0.1

Φ(x’T, T) Φ(x’1, 1) … 0.8 ) - z’T … - z’1 (

Tokens, x’T、 @Timestep T

Noisy Video, Vin

0.2

… 0.6 )

Decode

Tokens x’0 @Timestep 0

Denoised Video, Vout

Fig. 1: An example of a diffusion process. The random-noised video input, 𝑉𝑖𝑛 , is first encoded into the latent space and converted into a sequence of tokens, 𝑥𝑇′ . These tokens are then processed by the same diffusion model multiple times to predict the noises, 𝑧𝑡′ , at each timestep, 𝑡. After completing 𝑇 timesteps, the final denoised tokens, 𝑥 0′ , are decoded back into a final output video, 𝑉𝑜𝑢𝑡 , in RGB color space. Self-attn

Cross-attn

100

Normalized Exe. Time (%)

100 80 60 40 20 0

MLP

Others

80 60 40 20

n o X .2 t e .1 n a n2 yua sor Pla ide eo n2 gca rag Wa Hun Open nsora LTX-V ogVid Wa Lon Ave C e Op

0 n o X .2 t e .1 n a n2 yua sor Pla ide eo n2 gca rag Wa Hun Open nsora LTX-V ogVid Wa Lon Ave C e Op

(a) Nvidia H100 with HBM [56].

(b) Nvidia RTX PRO 6000 [54].

Fig. 2: The execution breakdown of eight mainstream vDiTs [18, 23, 35, 39, 70, 73, 97] on two recent Nvidia GPUs [54, 56]. Self-attention dominates the overall execution. • We introduce a reconfigurable accelerator architecture to support our reuse algorithm and a tailored data dispatcher to mitigate irregular sparsity and improve PE utilization. • Our architecture achieves up to 5.9× speedup and 16.0× energy savings over existing accelerator designs.

Background

Diffusion Models. The most widely adopted video generation paradigm is the diffusion model, which learns to generate data, i.e., images or videos, from random Gaussian noise through a reverse Markov process [9, 22, 81], as shown in Fig. 1. In this process, the initial input is a random Gaussian noise, 𝑉𝑖𝑛 , which is encoded into tokens, 𝑥𝑇′ , in latent space. Here, 𝑇 is the total number of denoising timesteps. Given noisy tokens 𝑥𝑇′ , the diffusion model restores the original data by gradually predicting and removing noise 𝑧𝑡′ from 𝑥𝑡′ at each timestep 𝑡. After a fixed number of denoising timesteps, the final result of the diffusion model, 𝑥 0′ , would be close enough to the original data 𝑥 0 in latent space. The mathematical expression of each denoising timestep is, 𝑥𝑡′ −1 = 𝛼𝑡 (𝑥𝑡′ − 𝛽𝑡 𝑧𝑡′ ) + 𝜎𝑡 𝑛𝑡′ , and 𝑧𝑡′ = Φ(𝑥𝑡′, 𝑡),

(1)

where Φ is the prediction function, i.e., the diffusion model, that predicts the noise 𝑧𝑡′ . Both 𝛼𝑡 and 𝛽𝑡 are hyper-parameters. 𝜎𝑡 𝑛𝑡′ is a renoising term to add randomness to the denoising process. Lastly, we decode 𝑥 0′ from the latent space to obtain the output video, 𝑉𝑜𝑢𝑡 . Current mainstream diffusion models in video generation often require 30 to 100 denoising steps to generate a short video [23, 35, 39, 69, 70, 73, 84, 97]. Recent studies also reduce to shorter timesteps, 4-10, via distillation [44, 91], finetuning [18, 41, 42], etc. Video Diffusion Transformer. A typical vDiT architecture first encodes input and prompt tokens into latent space with 𝑑 channels.

• We are the first to systematically characterize the large vDiT models and explain the spatio-temporal correlations in the latent space of large vDiTs. • We propose a lightweight channel-wise reuse algorithm that significantly reduces computation while achieving higher generative quality than prior methods. 2

102 101 0 1010 0

H100 Roofline Peak Performance (204.9 TFLOPs) Ridge Point (100.44 FLOPs/B) Wan2.1 HunyuanVideo Opensora

101

102

OpensoraPlan LTX-Video CogvideoX Wan2.2 Longcat-video

103

Arithmetic Intensity

104

(a) Nvidia H100 with HBM [56].

Performance (TFlops)

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

Conference’17, July 2017, Washington, DC, USA

HunyuanVideo CogVideoX

102 101 0 1010 0

RTX PRO 6000 Roofline Peak Performance (126.0 TFLOPs) Ridge Point (70.39 FLOPs/B) Wan2.1 HunyuanVideo Opensora

101

102

OpenSora v1.2 OpenSoraPlan v1.2

OpensoraPlan LTX-Video CogvideoX Wan2.2 Longcat-video

103

Arithmetic Intensity

104

Mochi

TurboDiﬀusion

Step-Video LTX-Video

Wan2.2

Wan2.1

Fig. 4: The ratio between token sequence length and the number of diffusion timesteps in vDiTs [18, 35, 39, 49, 69, 73, 84, 91, 97] has steadily increased over the years. It means that attention computation grows to be the only bottleneck.

(b) Nvidia RTX PRO 6000 [54].

Fig. 3: The roofline analyses of eight vDiT models [18, 23, 35, 39, 70, 73, 97] on two recent Nvidia GPUs [54, 56]. All vDiT models are primarily compute-bound.

intermediate results. As these techniques become widely adopted, the diffusion timestep is no longer a dominant bottleneck. In contrast, self-attention remains a key bottleneck, especially as recent vDiTs target longer video durations and higher resolutions [4, 25, 44, 70, 71, 91]. Both trends translate into longer token sequences, causing a quadratic increase in self-attention cost. As shown in Fig. 4, we show that the ratio between the token sequence and the number of timesteps across mainstream vDiTs increases over time. This means that accelerating self-attention will be the key to improving the performance of future vDiTs. While a substantial body of prior studies [10, 11, 20, 48, 74, 76, 85, 92] have studied sparse computation to accelerate self-attention in LLMs, these approaches primarily exploit attention sparsity to eliminate insignificant computation. Several recent works on image and video diffusion models [21, 77, 87, 88] have adopted similar ideas. However, as we show in Sec. 7.1, directly transferring LLMbased sparse attention techniques to vDiTs is both ineffective and fundamentally mismatched to the characteristics of vDiT models. Sparse attention is intuitive for LLMs because text sequences naturally exhibit discrete, hierarchical semantics. Thus, many longrange irrelevant dependencies can be safely pruned without any impact on generation quality. Unlike text tokens, video tokens are dense and carry continuous visual semantics. Every token contributes to spatial coherence and temporal consistency, especially in early denoising steps where global information must be retained to achieve consistent results [5, 77, 96]. Our experiment in Sec. 7.1 shows that naively removing attention correlations leads to motion discontinuities and flickering artifacts. Thus, rather than relying on sparsity patterns in LLMs, effective acceleration of self-attention in vDiTs should exploit their unique spatio-temporal correlations.

Then, for each timestep, those encoded tokens go through a stack of computational blocks. Each block is a combination of a selfattention, a cross-attention, and a MLP layer. Fig. 2 profiles the execution time of eight popular vDiT models [18, 23, 35, 39, 70, 73, 97]. Overall, self-attention accounts for 68% of total execution time. Self-Attention. The attention mechanism is the main reason that self-attention is the major computational bottleneck in vDiTs. Given input tokens 𝑋 in ∈ R𝑁 ×𝑑 , self-attention computes, 𝑄𝐾 𝑇 Attention(𝑄, 𝐾, 𝑉 ) = Softmax √ 𝑉, (2) 𝑑𝑘 where 𝑄, 𝐾, 𝑉 are query, key, and value, respectively. 𝑁 is the token sequence and 𝑑 is the channel dimension. These three token sequences have the same shape as the input tokens and are obtained by performing linear projections on 𝑋 in . 𝑃 = 𝑄𝐾 𝑇 ∈ R𝑁 ×𝑁 is the attention map. Overall, the computational complexity of self-attention is proportional to 𝑁 2 × 𝑑. Thus, it is compute-intensive. Fig. 3 shows the roofline analyses of vDiTs on Nvidia H100 [56] and Nvidia RTX PRO 6000 [54]. Under two different memory technologies, HBM2e and GDDR7, vDiT models are still compute-bound. Here, Opensora [97] is not compute-bound because it uses temporal-spatial separate attention, which is no longer used in recent vDiTs. Position Encoding. Next, we explain one of the key concepts in vDiT. The reason that vDiT models can capture spatial-temporal information is that all vDiTs apply rotary positional embedding (RoPE) [23, 39, 72, 73] to encode relative token positions, RoPE( [𝑥, 𝑦]) = (cos 𝜃 · 𝑥 − sin 𝜃 · 𝑦, sin 𝜃 · 𝑥 + cos 𝜃 · 𝑦), (3) where 𝜃 𝑝,𝑖 = 𝜑/100002𝑖/𝑑 . Importantly, the 𝑑 channels are often partitioned into three segments: initial channels encode temporal information, while the remaining channels capture spatial information along 𝑥 and 𝑦 axes. This structured encoding causes different channel groups to exhibit unique spatio-temporal patterns.

3.2

Spatio-Temporal Correlations in vDiTs

In this subsection, we first present the attention patterns that commonly exist in vDiTs, and then explain the underlying causes that induce these spatio-temporal correlations in their latent space. Spatio-Temporal Patterns. In Fig. 5, we show two representative self-attention patterns from HunyuanVideo [35], other vDiT models have similar patterns. Also, the patterns in Fig. 5 are consistent with patterns reported in prior studies [12, 77, 78, 89]. Overall, the patterns in self-attention maps of vDiTs can be categorized into two types: spatial patterns and temporal patterns. An example of a spatial pattern is shown on the left side of Fig. 5. Spatial patterns consist of small, seemingly “repetitive” tiles spanning across the entire attention map, 𝑃. Each tile captures the spatial correlations among tokens within a single frame, while

3 Motivation 3.1 Trends and Challenges of vDiT Models As mentioned in Sec. 2, the inference cost of a vDiT is primarily dominated by two factors: the number of denoising timesteps and the computation of self-attention. While both factors have historically posed major bottlenecks, recent studies [5, 7, 18, 41, 42, 91, 96, 100] exploit various techniques to leverage the similarity between timesteps and reduce the number of timesteps by approximating 3

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al.

One Frame

Key

Frame 0

y-dimension x-dimension

Channel

t-dimension

Token reuse t-dimension groups along t-dimension

Frame 1

reuse y-dimension groups along y-dimension

Query

One Row reuse x-dimension groups along x-dimension

Temporally Vary Spatially Uniform

Fig. 5: Example of various attention patterns, which can be classified into spatial patterns and temporal patterns. Spatial patterns capture the spatial correlations within a single frame by repetitive “tiles”, whereas temporal patterns capture the temporal correlations across frames via strong diagonal correlations. Both patterns arise from the RoPE encoding and the dominant channels in query 𝑄 and key 𝐾.

Fig. 6: An example of how different channel groups govern the final generation quality. The upper part illustrates how we reuse different channel groups shown by red arrows. Three channels represent t-, x-, and y-dimension channel groups. The lower part shows results after reusing.

MSE Loss

105

HunyuanVideo Wan2.1 TurboDiffusion CogVideoX LongCat OpenSoraPlan T-dimension X-dimension Y-dimension

16 56 56

44 42 42

32 48 48

44 42 42

32 32 32

Reuse along Temporal

104 103 15 20 25 30 35 40 45 50 Diffusion Timestep (a) T-channel group.

the patterns across different tiles remain largely unchanged. Thus, spatial patterns mainly capture spatial relationships within a frame. In contrast, temporal patterns, shown on the right side of Fig. 5, emphasize correlations across frames. In these patterns, the variations of attention scores within a single frame are less significant, whereas inter-frame correlations are more important. In particular, temporally adjacent frames tend to exhibit strong mutual attention, resulting in a diagonal correlation in the attention map. Underneath Mechanism. While prior studies [77, 83] also observed these patterns in vDiTs, they do not further investigate the underlying cause of these patterns. Existing approaches simply leverage these patterns to accelerate vDiTs via sparse computations of attention maps using the techniques from LLMs. However, without a fundamental understanding of the underlying mechanisms, these heuristics often lead to low generative quality. In this work, we take the first step to explain the causes of various attention patterns and propose a principled acceleration technique tailored for vDiTs. Our analysis shows that the dominant driver of those patterns is the channel-wise encoding of tokens, i.e., RoPE. As introduced in Sec. 2, RoPE is the primary positional encoding technique in vDiTs, where different channel groups encode positional information at different frequencies. Specifically, vDiT models partition their channel dimensions into three distinct groups: the temporal (t) dimension, the horizontal (x) dimension, and the vertical (y) dimension, as shown in Fig. 6. Each channel group captures information along a specific direction in latent video data. Tbl. 1 lists the channel partitioning of vDiT models. Meanwhile, we find that the dominant channels with higher values have a higher chance to contribute more to the attention scores in Fig. 5. Here, the bounding boxes on 𝑄 and 𝐾 highlight individual frames. Because RoPE causes the dominant channels to

105

Reuse along Horizontal

104 103 15 20 25 30 35 40 45 50 Diffusion Timestep (b) X-channel group.

105 MSE Loss

Table 1: The channel partitioning of different vDiT models [35, 39, 70, 73, 84, 91]. vDiT models apply RoPE encoding and partition the token channels into t-, x-, and y-dimension.

MSE Loss

Spatially Vary Temporally Uniform

Reuse along Vertical

104 103 15 20 25 30 35 40 45 50 Diffusion Timestep (c) Y-channel group.

Fig. 7: The MSE loss introduced by reusing different directions on different channel groups. For a given channel group, reuse along the other two dimensions leads to lower MSE loss than reuse along its own direction.

have different frequencies, leading to various patterns in 𝑄 and 𝐾. Different combinations of channels in 𝑄 and 𝐾 form various patterns in attention maps. As shown in Fig. 5, when spatial channels dominate, attention maps exhibit “spatial-like” patterns. Otherwise, attention maps exhibit “temporal-like” patterns. Theory Verification. To verify our theory, we design an experiment that purposely manipulates different channel groups and examines their impact on the final video quality. As shown in Fig. 6, for each channel group, we group pairs of adjacent tokens along the corresponding dimension and force the second token in each pair to reuse the value of the first token across all denoising steps. For example, for the temporal channel group, we allow the t-dimension channels of every second frame to reuse those of the first frame at the same coordinate, while leaving the x- and y-channel groups unchanged, and vice versa. Fig. 6 illustrates how we reuse different channel groups with a toy example of eight input tokens (two frames with 2 × 2 tokens per frame). The bottom part of Fig. 6 shows some results after applying our reuse method. As shown, reusing t-dimension channels introduces temporal distortions, whereas reusing x- or y-dimension channels produces stripe-like artifacts aligned with the corresponding direction. This shows that different channel groups govern different dimensions of information. Overall, all used frames are much blurrier than the baseline because the self-attention mechanism integrates global contributions across all tokens. 4

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

5.6 0.7 3.4 5.3 4.3 2.3 1.5 0.6 5.3 4.4 1.3 Qin 1.1 6.4 3.6 3.2 1.2 2.2 2.4

Row

Reshaped Q and K

0.2 3.2 5.4 0.1 2.1 1.3

Frame 1

y: reuse in t

1.3 5.4 2.4 3.2 1.3 5.3 2.3 3.4

x: reuse in y Frame 0

0 0.1 0.1 0.2

0.1 0.4 0.1 0.1

Partial Reuse

Reuse

0 5.2 0.4 0.3

2.4 2.4 Kin 1.1 5.4 3.2 3.2 4.3 0.2 2.6 3.2 3.2 3.4 3.3 0.1 2.5

Channels

Frame 1

1.2 1.3 3.3

y: reuse in t

x: reuse in y Frame 0

2.3 4.5 2.5

t: reuse in x

2.5 3.4 2.6 3.2 2.4 3.3 2.5 3.1

0.1 0.1 0.1 0.1

0.2 3.2 4.5 1.4

0.1 0 2.1 0.1

3.2 5.4 1.2 2.4

0.1 1.1 0.1 0.1

1 Calculate Similarity

Half-bit Computation

3.4

t-channel: 0.1 0.2 1.2 6.4 1.1 1.5 5.3 5.6 mask:

mask:

1.2 6.4 1.1

x-channel: 0.1 3.2 0.2 3.2 2.4 1.3 4.5 1.4

.5 5.3

2.1 3.2 .6

100% Compute

2.4 1.3 4.5

4.3 5.4 1.1

2.3

1.3 5.4 2.4 3.2

Partial Attention Scores

4.4 0.6 .4

Final Attention Scores

t-channel: 3.3 3.2 4.3 5.4 1.1 1.2 2.3 2.4 2 Mask Generation

100% Reuse

2.5 3.4 2.6 3.2 0.1 3.2

mask:

3.3 4.3 1.1 2.3

Skipped Computation y

y-channel: 2.5 3.4 2.6 3.2 2.4 3.3 2.5 3.1

0.1 3.2 2.4 1.3

Full-bit Computation

mask: x-channel: 2.1 3.2 2.2 3.6 4.4 0.6 4.3 0.7

0.1 2.4 1.4 3.1

Lower-bit Data

mask:

0.1 1.2 1.1 5.3 0.2 6.4 1.5 5.6

Compute

y-channel: 1.3 5.4 2.4 3.2 1.3 5.3 2.3 3.4

2.1 3.2 4.4 0.6 2.2 3.6 4.3 0.7

t: reuse in x

Similarity Scores

Conference’17, July 2017, Washington, DC, USA

3 Compute Partial Attention

4 Reuse & Aggregate

Fig. 8: An overview of our channel-wise reuse algorithm, which consists of four steps. We first compute the similarity between adjacent tokens along a selected axis. Based on this similarity, a subset of tokens could reuse previously computed partial attention scores, thus reducing the overall attention computation. 𝜃 th1 and 𝜃 th2 are set to be 0.1 and 1 for illustration purposes. Fig. 7 further quantifies the mean square error (MSE) loss between the baseline results and the results generated with different channel reuses. The x-axis shows the diffusion timesteps at which we apply the reusing strategy. For every channel group, we evaluate reuse along three different dimensions: t, x, and y. The result shows that, for a given channel group, reuse is less harmful when applied along the other two dimensions, rather than along the dimension that the channel group encodes. For instance, for the t-dimension channel group, applying reuse along the x- or y-dimension results in lower MSE than reusing along the t-dimension. This result further confirms that each channel group predominantly captures information specific to its corresponding dimension.

shown in Fig. 7, for a given channel group, it is better to reuse the channel values along the other two directions. For example, channels belonging to the t-channel group have lower MSE loss when reusing tokens along x- or y-directions. To better co-design with our architecture, we further restrict each channel group to reuse along one fixed direction. E.g., in Fig. 7, t-, x-, and y-channels are restricted to reuse x-, y-, and t-directions, respectively. At runtime, the token similarity is computed along the predefined direction. The similarity Δ is measured using the absolute difference, Δ(𝑥𝑎 , 𝑥𝑏 ) = |𝑥𝑎 − 𝑥𝑏 |,

where 𝑥𝑎 and 𝑥𝑏 are single-channel values of two adjacent tokens along a specific direction. For instance, along the t-direction, we compute the similarity Δ between every two adjacent frames. 2 We then check whether the token similarity Δ is below our predefined thresholds, 𝜃 th1 and 𝜃 th2 . Here, 𝜃 th1 < 𝜃 th2 . If Δ is below 𝜃 th1 , then the second token 𝑥𝑏 is marked as “reuse”, i.e., it can completely reuse the partial attention score of the first token 𝑥𝑎 . If Δ is below 𝜃 th2 , the second token 𝑥𝑏 is then marked as “partial reuse”, i.e., it can partially reuse the attention score of 𝑥𝑎 . 3 Once we identify reusable tokens in each channel group, we start to compute the partial attention scores. Here, 𝑄 and 𝐾 perform matrix multiplication channel by channel. If any tokens in 𝑄 and 𝐾 are marked as “reuse”, they will not perform computation and instead reuse previously computed results, as the white blocks shown in Fig. 8. If any tokens in 𝑄 and 𝐾 are marked as “partial reuse”, they would perform half multiplication. Consider a fixedpoint representation, we could reuse the higher ℎ bits multiplication from the previous token and only compute the subsequent 4 bits, as the half-colored blocks shown in Fig. 8. 4 Finally, all partial attention scores from all channels are aggregated to calculate the final attention scores 𝑃. Once the final attention scores are obtained, operations, e.g., Softmax, are performed the same as in the canonical self-attention. Lastly, 𝑃 × 𝑉 is performed similarly to 𝑄 × 𝐾 𝑇 . However, we only perform a reuse

Channel-Wise Reuse Algorithm

With the major finding of the spatio-temporal correlations in Sec. 3, we introduce our channel-wise reuse algorithm to accelerate all computation blocks in vDiTs. Here, we first provide an overall idea of our algorithm in Sec. 4.1 and then explain the rationale and design decisions behind our algorithm in Sec. 4.2.

4.1

(4)

Idea

Sec. 3 shows that the spatio-temporal patterns in attention maps are governed by dominant token channels, and each channel has a unique frequency governed by RoPE encoding (see Fig. 5). Here, we show how to leverage this unique insight to accelerate the operations in vDiTs via reusing intermediate results. Our algorithm assumes an 8-bit fixed-point numerical representation, which is widely adopted in diffusion accelerators [33, 34]. Here, we use selfattention to demonstrate our idea first. Other compute blocks, i.e., across-attention and MLP, can be accelerated in a similar fashion. Self-Attention. Our overall process to accelerate self-attention is shown in Fig. 8, which consists of four steps. 1 Given the mathematical expression of self-attention in Eqn. 2, we first compute the similarity of adjacent tokens in the query 𝑄 and key 𝐾. Later, these results will guide our reuse algorithm. As 5

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al.

𝑖=0

IQR 10

Ours

Sparsnee1 Sparsnee2 Baseli Baseli

20 30 40 Diffusion Timestep

Fig. 9: MSE comparison be- Fig. 10: The sensitivity of our tween our reusing method reuse technique accuracy to and two skipping techniques. different prompts. Here, we We pick 20 random prompts. show 20 random prompts. I/O

vDiT Accelerator

Global Global Buffer Buffer

Data Dispatcher 4x4 PE Array

…

4x4 PE Array

Data Dispatcher 4x4 PE Array

…

4x4 PE Array

Accumulation AccumulationBuffer Buffer

𝑖=0 Vector Unit

where 𝑎 is a window of activation elements along one channel. 𝑤 is the window size. Δ(𝑎) determines whether all elements within this window can be reused or partially reused. In Sec. 5.2, we will show that this window-based design can also map naturally onto our co-designed PE array.

4.2

MSE Loss

check on 𝑉 , not 𝑃. Because only 𝑉 has spatio-temporal correlations among its tokens. The reuse process of 𝑉 is similar to 𝑄 and 𝐾. Other Operations. In addition to self-attention, other operations in vDiTs can also benefit from a similar reuse strategy. However, unlike self-attention, for other operations such as MLPs, reuse can only be applied to activations rather than weights. This is because only activations inherit the spatio-temporal correlations, whereas weights do not. Thus, for these operations, we modify step 1 of our reuse algorithm; other steps remain largely unchanged. In step 1 , we first compute the similarity of activations on a per–channel-group basis. Similar to self-attention, all channels in each channel group are restricted to reuse along one direction. However, since only one input, e.g., activation, has spatial-temporal correlations, we propose a window-based comparison, rather than comparing two adjacent frames, to increase the reuse ratio. The similarity Δ is measured as, v u t𝑤−1 𝑤−1 ∑︁ ∑︁ (5) ¯ 2 /𝑤, 𝑎¯ = (𝑎𝑖 − 𝑎) 𝑎𝑖 /𝑤, Δ(𝑎) =

DMA

DRAM Input Tokens Linear Layer Weights Activa‐ tions

Fig. 11: The overall architecture design, which comprises a set of data dispatchers, multiple PE arrays, and a vector unit. the threshold impact on the final output is insensitive to the input prompts, as shown in Fig. 10. Thus, we can predetermine the threshold values offline and apply them to all inputs. To further simplify hardware implementation, we restrict all thresholds to powers of 2. In our design, we set 𝜃 th1 and 𝜃 th2 to preserve the most significant 5 bits and 3 bits, assuming an 8-bit fixed-point representation, respectively. Specifically, if the first 5 bits of 𝑥𝑎 and 𝑥𝑏 are the same (computed via a bitwise XOR), then 𝑥𝑏 fully reuses the partial results of 𝑥𝑎 . If the first 3 bits of 𝑥𝑎 and 𝑥𝑏 are the same, the most significant 3 bits of the partial result for 𝑥𝑏 are reused from 𝑥𝑎 , while only the subsequent 4 bits of 𝑥𝑏 are explicitly computed. Sec. 7.4 further the impact of different thresholds on model performance and generative quality.

Design Decisions

Next, we explain several key design rationales in our algorithm. Why Reuse? We first explain why our reuse technique is better than conventional sparse attention techniques [21, 77, 83]. Recall, sparse attention exploits the insignificance of 𝑄𝐾 𝑇 values by directly skipping their attention computations. In Fig. 9, we compare the MSE loss of our method against two sparse attention baselines under the same token-saving ratio, 𝜃 ∗ = 85%, on HunyuanVideo [72]. The first baseline skips 85% of tokens with the lowest values during the attention computation. The second baseline uses the same selection criteria as our reuse method to identify tokens, but instead of reusing previously computed results, it skips their attention computations. The results show the output MSE between the original model and different compute-saving techniques. Results show that our technique achieves an order of magnitude lower MSE loss compared to these two baselines. This shows that our technique is more effective than sparse computation methods. Why Channel? Next, we explain why our reuse strategy operates at the channel level rather than the token level, as proposed by prior works [2, 87]. Prior works target high-level tasks, like classification. For those tasks, aggregating token information is acceptable. However, video generation requires preserving dense and smooth content at the pixel level. Merging or reusing at the token level often leads to noticeable quality degradation. Thus, it is intuitive to reuse at the root cause, the channel level. Moreover, different channel groups encode distinct spatio-temporal information. Reusing them at the token level inevitably loses certain information. Sec. 7.1 shows that our channel-level reuse preserves much higher quality against other token-level approaches [2, 87]. Thresholds. The next question is how to determine the appropriate thresholds, 𝜃 th1 and 𝜃 th2 . Our key observation is that

Architectural Design

Although our channel-wise reuse algorithm significantly reduces the total number of operations across all computational blocks, its reuse-based data access patterns are inherently incompatible with off-the-shelf accelerators, e.g., a systolic array. To address this, we introduce a co-designed architecture that natively supports our algorithm. We first present the overall architectural design in Sec. 5.1, and then explain two key components: a PE array with a tailored dataflow in Sec. 5.2, and a dedicated data dispatcher in Sec. 5.3 to guarantee high PE utilization during the computation.

5.1

Overview

Motivation. As shown in Fig. 8, the proposed algorithm has various reuse patterns, which make it difficult to exploit efficiently using only GPU kernels due to its fine-grained, data-dependent execution. Directly implementing our idea on a GPU would lead to severe warp divergence, irregular memory accesses, and low 6

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

… … … …

Frame

8-bit W1 8 8-bit 8 W2

M 4-bit A1 4-bit A2

PE Design. Fig. 13 shows our augmentation of a canonical 8-bit fixed-point MAC unit, with the added logic highlighted in yellow. Specifically, we add two additional registers in each PE to store more intermediate results, along with extra multiplexers to select data from different sources. The control signal, denoted as M, configures the PE to operate in different execution modes. The following paragraphs explain how these configurations enable PEs to support various data reuse patterns. Reuse-Aware Dataflow. Based on our channel-wise reuse algorithm, reuse patterns vary across different frames within a 2 × 2 group. In Fig. 14, each frame has 3 × 3 tiles in the attention map. To minimize the data traffic between PEs and the accumulation buffer, each PE is configurable to support different reuse patterns. Our overall dataflow adopts an output-stationary pattern in a classic systolic array, with some changes to accommodate reuse. We showcase four execution modes using reuse along the t-dimension as an example; reuse along the other two dimensions works similarly. Additional modes are the variants of the four modes. Mode 0. This mode is used when computing tiles such as Tile 0, as annotated in Fig. 14. When processing Tile 0, the partial attention score could be reused by tiles in subsequent frames, i.e., Tile 1 and Tile 2. The first example in Fig. 14 shows this scenario, where the blue element in 𝑄 and the pink element in 𝐾 are reused. Three registers in PE are used to store the accumulated partial attention scores corresponding to the elements in Tile 0, Tile 1, and Tile 2. During computation, the input element of K is read from the column input port, while the input element of Q is read through the first row input port; the second row input port is disabled. After the partial attention score is computed, the result is accumulated into three corresponding registers. The multiplexers 𝑀2 and 𝑀3 control whether the partial score should be accumulated into registers 𝑅1 and 𝑅2 , respectively, based on the reuse mask described in Sec. 4.1. After computing all channels in one channel group, the current accumulated attention score of Tile 0 is temporarily read out from all 𝑅0 of the PE array to the accumulation buffer. Mode 1. The second mode is used when computing tiles such as Tile 1 or Tile 2. When reuse is applied along the t-dimension, elements in Tile 1 or Tile 2 can only be reused by Tile 3, as shown by the second case in Fig. 14. In this example, we show how elements in 𝑄 are reused, i.e., Tile 3 reuses the values in Tile 1. In this mode, the partially accumulated attention score of Tile 2 is temporarily stored in 𝑅2 to avoid reloading it from the accumulation buffer. Registers 𝑅0 and 𝑅1 hold the accumulated partial sums for Tile 1 and Tile 3, respectively. During computation, the input values of 𝑄 and 𝐾 are read from the row and column input ports, respectively. The resulting partial sum is accumulated into 𝑅0 for Tile 1. Meanwhile, it is also selectively accumulated into 𝑅1 based on the reuse criterion. Note that, if the resulting partial sum is considered “half-reuse”, i.e., the leading bits are reused, the multiplexer 𝑀2 is configured to accumulate only the leading ℎ bits into 𝑅1 . Mode 2. The third mode is also used when computing tiles such as Tile 1 or Tile 2. However, in this case, Tile 1 and Tile 2 have already partially reused the results from Tile 0, i.e., the leading ℎ-bit partial sums. Thus, Tile 1 and Tile 2 still need to calculate the remaining partial sum of the subsequent 4 bits after the ℎth bit. Specifically, in the third example in Fig. 14, the values in 𝑄 are reused, while the values in 𝐾 are partially reused. To use our PE,

\ \

… … … …

Conference’17, July 2017, Washington, DC, USA

<<4 0

… … … … Frame

Fig. 12: The computation order Fig. 13: The design of of self-attention computation one PE unit, which is when reusing along the t-axis. built upon a canonical 8Each PE array is responsible for bit MAC unit. The yellow a group of tiles, as highlighted in parts highlight our augcolors. The numbers denote the mented logics. computation order. tensor core utilization. These challenges motivate us to propose a dedicated architectural support for our algorithm. Overview. Fig. 11 illustrates the overall architecture of our vDiT accelerator. Our design comprises a set of PE arrays and a vector unit. A subset of PE arrays is also coupled with a data dispatcher. The PE arrays are responsible for compute-intensive matrix–matrix operations in vDiTs, e.g., self-attention, while the vector unit handles element-wise and vector operations, including Softmax. Each PE array is built upon a classic systolic array design with 4 × 4 PEs and is augmented with additional support to enable our reuseaware datapath, as shown in Fig. 13. The data dispatchers are used to feed input data into the PE arrays; their scheduling algorithm is explained in Sec. 5.3. Meanwhile, to support the pipelining between computation and data fetching, both the global buffer and the accumulation buffer are designed to be double-buffered.

5.2

PE Array

We first illustrate the overall computation order of our reuse algorithm. We then describe our augmentation to an 8-bit fixed-point multiply-and-accumulate (MAC) unit. Lastly, we show how the PE is configured to support various data computation patterns. Computational Order. We first describe our computation order, which minimizes on-chip data movement, and how the workload is distributed across multiple PE arrays. Here, we use self-attention as an example. As shown in Fig. 12, we organize every 2 × 2 frames in the attention map as a group and process them together. Specifically, we map the same 𝑁 × 𝑁 tile across the 2 × 2 frames to a single PE array. In Fig. 12, 𝑁 is 4. Here, we compute the partial accumulations on a channel group basis, since the reuse direction is the same within one channel group. For instance, when the reuse direction is along the t-dimension, the computational order is shown in Fig. 12. We first process the 4 × 4 attention elements in frame 0, and then we sequentially compute the same elements for frames 1, 2, and 3. Other reuse directions can be done similarly. Once we finish computing the partial attention scores within a channel group, we store the intermediate results in the accumulation buffer. 7

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al.

Lower Bits

Reuse

2 R1

M4 <<4

Lower Bits

Reuse

M4 <<4

Higher Bits

Mode 2: Computing Tile 1 after half-reusing on Tile 0 , and reusing on Tile 3 .

Lower Bits

M4 <<4

Mode 3: Computing Tile 3 , no reusing. Higher Bits

Lower Bits

Half Reuse

2 R0

Frame

Reuse

Higher Bits

Mode 1: Computing Tile 1 , reusing on Tile 3 .

Q 0

3 R0

M4 <<4

Q Reuse

Mode 0: Computing Tile 0 , reusing on Tile 1 and 2 .

Fig. 14: An overview of our reuse-aware dataflow for a single PE. In this example, 3 blocks in 𝑄 and 𝐾 stand for one frame, i.e., 3 × 3 tiles are a frame in the attention map. We show four representative cases to show the flexibility of our PE. The control logics are omitted in PEs. The gray color indicates the parts that are disabled during the computation.

5.3

Col. 2 Col. 3

Partial Reuse 5 6

Full Compute 7

Row: 0 Col. 0

(a) Low PE utilization schedule.

Col. 1 Col. 2 Col. 3

Complete Reuse 1 2 3

Partial Reuse 5 6

Full Compute 7

Row

Col. 1

PE PE PE PE

Col. 0

Complete Reuse 1 2 3

Row

Row: 0

PE PE PE PE

the lower bits of two 𝐾 elements are fed into two 4-bit column input ports, while two distinct 8-bit 𝑄 values are read from the two row input ports. The control logic configures the datapath such that the two 𝑄 values are processed by separate 4 × 8 MAC units. The resulting products are then directly added without performing the shift operation. The accumulated result is stored in registers 𝑅0 and 𝑅1 . Note that, if reuse is not applied to 𝑄, the corresponding multiplexer can simply be configured to not accumulate into 𝑅1 . Mode 3. The final mode is used when computing tiles in the last tile in a 2 × 2 group. In this mode, input values from 𝑄 and 𝐾 are fed into the PE through the row and column input ports, respectively. 𝑅1 and 𝑅2 are always disabled, as no need to reuse elements from Tile 3, as shown in the last example of Fig. 14. The computed partial sum is then accumulated into register 𝑅0 , which stores the accumulated results of Tile 3. Note that, the datapath of the last example shows accumulation for a single full 8-bit multiplication result. When values in K are partially reused, the PE can be configured, similar to Mode 2. Two lower-bit 𝐾 values are fed into two 4-bit column ports, so that we can compute two partial sums simultaneously. The only difference is that multiplexer 𝑀2 is configured to always disable the accumulation of the partial sum into register 𝑅1 . Other Operations. Self-attention has the most complex reuse patterns, whereas other operations are simpler because reuse applies to only one input. As a result, our PE can also accommodate reuse patterns in other vDiT operations, such as MLPs. Configuring the PE to support these operations is straightforward: register 𝑅0 is used to compute the current value, while the remaining two registers are used to accumulate partial results when reuse is applied. The overall configuration is similar to Mode 1. Meanwhile, we also support “half-reuse” in those operations. The configuration is similar to Mode 2 in Fig. 14.

(b) High PE utilization schedule.

Fig. 15: Examples of two scheduling cases with identical data sparsity. The left example shows low PE utilization because many PEs perform no effective computation. In contrast, the right example achieves high PE utilization because two adjacent inputs can be merged and processed together. The red blocks highlight the rows that are merged. queries 𝑄, is fully dense. Note that, 𝐾 is streamed from the column ports of the PE array. Each column port of the PE array accepts 8-bit data, whereas each row port can read at most 16-bit data, corresponding to two input elements. In principle, if two rows of 𝐾 are sufficiently sparse, they could be merged and processed at once to improve PE utilization. However, the sparsity pattern of 𝐾 is highly irregular, as shown in Fig. 15a. The adjacent two rows rarely exactly match and allow merging; thus, many PEs remain idle since fully reused elements skip computation entirely. In contrast, an ideal case is shown in Fig. 15b, where every pair of adjacent rows can be merged. For instance, the first and second rows can be combined and fed into the PE array within a single cycle. Note that, the only difference between Fig. 15a and Fig. 15b is the computation order. Thus, in this subsection, we design a data dispatcher to reorder computations and improve PE utilization. Design. Fig. 16 illustrates the overall design of our data dispatcher, which consists of a scheduler and a matcher. The scheduler groups tokens such that those with similar reuse patterns are dispatched to the PE array together. The matcher then further pairs channels within each token group to combine compatible input pairs, similar to the example in Fig. 15b, further improving PE utilization. The following paragraphs explain these two components. The left side of Fig. 16 illustrates our scheduler. The reuse pattern of each token is represented as a binary string, where “0” denotes full reuse and “1” denotes half-reuse or no reuse. These token strings

Data Dispatcher

Issues. With our reconfigurable PE array design, we are able to support various reuse data access patterns. However, given the fixed dataflow of the PE array, i.e., an output-stationary dataflow, sparse and irregular input patterns often lead to low PE utilization. Fig. 15a illustrates such a case. Here, we show only one side of the input data, i.e., keys 𝐾, while assuming that the other input, 8

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

0110010101…01 …

15:

Argmin XOR XOR XOR XOR

… 07 03 00 … 11 10 08 … 14 12 09 Clustered Queue

0110010101…01 16 channels

02:

… 06 02 01

0110010101…01

Row

01:

Pivot Strings 011011010…01 11100111001 011010011…01 011001010…01

0110010101…01

00:

Lookahead Buﬀer Matched Merge Unit

(Double-buffered)

Global Buﬀer

Clustering Unit

Feature Feature Buffer Buffer

String Queue

Conference’17, July 2017, Washington, DC, USA

Scheduler

Matcher

Fig. 16: An overview of our data dispatcher design, which consists of a scheduler and a matcher. The scheduler groups tokens with similar reuse patterns together to improve the PE utilization. The matcher then pairs channels within each token group to combine compatible input pairs. Table 2: Area breakdown of Kaleido architecture.

are stored in a string queue with 64 entries. Each entry contains a 16-bit string, i.e., 16 channels are processed as a batch. To cluster these 64 strings into four groups, the clustering unit first randomly picks four strings as pivot strings. It then computes the Hamming distance between each input string and each pivot string, and uses an adder tree to accumulate the total differences. Based on the results, each input string is assigned to the cluster corresponding to the pivot string with the minimum distance and is enqueued into the corresponding clustered queue. Note that, each clustered queue has a fixed maximum capacity, i.e., 16 strings in this case. If the preferred queue is already full, the input string is assigned to the queue with the next smallest distance instead. Through this process, each clustered queue ultimately contains exactly 16 input strings for subsequent computing. Once a batch of input strings is clustered, each cluster is mapped to one PE array. We design one matcher for one PE array. Each matcher reads the corresponding token features from the global buffer into the feature buffer. In each cycle, the merge unit examines one channel value from each of the four input tokens and checks whether these values can be merged with any other inputs. If a match is found, the merge unit combines two input values and feeds them into the PE array. Otherwise, the single input values are fed into the PE array. Note that, a scheduler is required for both sides of the data, i.e., 𝑄 and 𝐾. However, only one side requires a matcher. The other side follows the same merging policy implicitly.

Configuration Area (mm2 )

4 × 4 PE Array

Dispatcher

On-Chip Buffer

Total

128 × 128 25.1

64 × 32 1.3

64 MB+32 MB 68.4

– 94.8

Simulation Methodology. The RTL is synthesized using Synopsys tools and laid out using Cadence tools in TSMC 16nm FinFET technology, with SRAMs generated by an ARM compiler. Power is simulated using Synopsys PrimeTimePX, with full annotated switching activity. The off-chip DRAM is modeled as 16 DDR5-6400 channels based on Micron specifications [53], and DRAM energy is estimated using Micron’s System Power Calculator [26]. We build a cycle-level simulator of the architecture with the latency and energy of each component parameterized from post-synthesis results. Area. Tbl. 2 shows that the overall area of Kaleido is 94.8 mm2 at 16 nm. Compared to a baseline systolic array with 96 MB on-chip buffer, our augmentation on PE arrays and our data dispatcher introduce area overheads of 6.6% and 1.3%, respectively. Models and Metrics. We evaluate Kaleido on four widely adopted vDiT models: HunyuanVideo [35], Wan [73], CogVideoX [84] and TurboDiffusion [91]. Without further notice, we generate 5second videos with 480 × 540 resolution. Note that, TurboDiffusion is a distilled model with only 4 timesteps. To evaluate the generative quality, we use the VBench [24] as the video quality metric, which consists of 950 benchmark prompts. The generated videos are evaluated across 16 aspects from VBench. For image quality evaluation, we use PSNR, SSIM, and LPIPS, and compare against the videos generated by the baseline models. Hardware Baselines. We compare six hardware baselines:

Experimental Setup

Experimental Methodology. We develop validated RTL implementations for the Kaleido hardware. The hardware is based on a systolic array architecture, consisting of 128 × 128 PE arrays, each with 4 × 4 PEs clocked at 1 GHz. The PEs are designed for 8-bit fixed-point representation. The hardware also has a scalar unit, which consists of 512 parallel lanes, each capable of performing the Softmax and other activation functions. Both the global buffer and the accumulation buffer are implemented by SRAM. The global buffer is 64 MB to store input data and model weights. The Accumulation buffer is 32 MB in size to store output data. Both buffers are double-buffered. While we primarily evaluate Kaleido using this configuration, we will later show the sensitivity of Kaleido performance to different hardware resource configurations.

• A100: a Nvidia A100 GPU with 80 GB HBM [55]. • H100: a Nvidia H100 GPU with 80 GB HBM [56]. • Cambricon-D [34]: approximates the computation between adjacent timesteps to accelerate the diffusion process. • AdapTiV [87]: leverages the value similarity among tokens to reduce the effective computation for image classification. • Exion [21]: exploits inter- and intra-timstep sparsity among intermediate results to skip computation. • Ditto [33]: also leverages the value similarity between adjacent timesteps to reduce the effective computation. 9

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al.

PSNR (dB)

All accelerator baselines are implemented using 8-bit fixed-point precision and scaled to the same number of PEs, 512 × 512, at 1 GHz. The hardware simulation is modeled under 16 nm technology. The energy and performance of A100 and H100 are measured using the built-in power sensing circuitry on Nvidia A100 and H100. Software Baselines. In addition to six hardware baselines, we also compare Kaleido against four algorithmic baselines. Specifically, we compare one block caching technique, PAB [96]. We also compared three sparse attention acceleration methods, including MInference [30], SVG [77] and SVG2 [82]. The performance numbers of all software baselines are measured on a Nvidia A100 GPU. Variants. We evaluate three variants of Kaleido to separate the contributions in our paper:

40 30 20 10 0

SVG SVG2

Minference PAB

HunyuanVideo

Cambricon-D AdapTiV

Wan

TurboDiffusion

EXION Kaleido

CogVideoX

SSIM

(a) PSNR. Higher is better. 1.0 0.8 0.6 0.4 0.2

HunyuanVideo

Wan

TurboDiffusion

CogVideoX

(b) SSIM. Higher is better.

LPIPS

• Kaleido-pe: a variant which only consists of our modified PE arrays without our data dispatcher. • Kaleido-pe-sh: a variant which only consists of our modified PE arrays and the scheduler in our data dispatcher, but without the matcher. • Kaleido-pe-m: a variant which only consists of our modified PE arrays and the matcher in our data dispatcher, but without the scheduler. • Kaleido: our full-fledged design with all optimizations.

1.0 0.8 0.6 0.4 0.2 0.0

HunyuanVideo

Wan

TurboDiffusion

CogVideoX

VBench Score

HunyuanVideo

Wan

TurboDiffusion

CogVideoX

(d) VBench score. Higher is better.

For both variants, we set 𝜃 th1 and 𝜃 th2 to preserve the first 5 bits and first 3 bits of accuracy, respectively. We do not show a GPU implementation of our algorithm because its reuse-based data access patterns are fundamentally incompatible with existing attention acceleration frameworks [11, 37], e.g., FlashAttention. Thus, a direct comparison of GPU implementations is not meaningful.

Ground Truth

Cambricon-D

AdapTiV

EXION

Kaleido

(e) Qualitative comparison on HunyuanVideo. Please zoom in to check details.

Fig. 17: Video generation quality comparison.

7 Evaluation 7.1 Generative Quality

7.2

Performance Comparison

Performance. Fig. 18a compares the end-to-end performance comparison of different accelerators. Here, we compare Kaleido against four accelerators: Cambricon-D, AdapTiV, Exion and Ditto. Meanwhile, we also compare four GPU-based optimizations: SVG, SVG2, Minference, PAB, Across all four vDiT models, Kaleido achieves the highest speedup. Specifically, Kaleido delivers 6.6×, 5.6×, 5.2×, and 6.1× speedup over A100 on HunyuanVideo, Wan, TurboDiffusion, and CogVideoX, respectively. While methods like AdapTiV also achieve relatively high performance, over 5× speedup, AdapTiV achieves much lower generative quality compared to Kaleido. Because its acceleration technique, token merging, is designed for image classification and fundamentally ill-suited for generative tasks. Cambricon-D yields marginal speedups because its architecture is primarily designed for CNNlike models and cannot accelerate self-attention, which is the primary computation bottleneck in modern vDiT models. Finally, both Exion and Ditto struggle to achieve high speedups due to the irregularity of sparse computation in video generation, especially for long token sequences. Both designs have to deal with the workload imbalance between PEs. In contrast, Kaleido co-designs a data dispatcher to tame the workload imbalance and improve the PE utilization. Sec. 7.3 further dissects the impact of data dispatch. Energy. Fig. 18b shows the energy savings of all evaluated hardware baselines normalized to A100. Here, energy saving is defined as the ratio between the energy consumption of the baseline,

Here, we primarily compare against three approximation baselines: Cambricon-D, AdapTiV, and Exion. We also include four software-optimized baselines here. Since the algorithm proposed in Ditto is lossless, we do not include it in the quality comparison. Fig. 17 compares the video generation quality of Kaleido with the three approximation methods shown above. Across all four metrics, Kaleido consistently delivers the best quality. For instance, Kaleido improves reconstruction fidelity by a large margin in terms of PSNR. Kaleido achieves 29.9 dB, 24.5 dB, 24.4 dB, and 26.5 dB on HunyuanVideo, Wan, TurboDiffusion, and CogVideoX, respectively. In particular, Kaleido achieves 17.0 dB over Cambricon-D on HunyuanVideo. Across all models, Kaleido achieves over 6 dB higher than all prior accelerator methods. Even compared with the software-optimized baselines, Kaleido consistently achieves the highest results over the image-based metrics. This shows that the channel-wise reuse algorithm is the key to preserving high pixel-level precision. Similar trends can be seen in SSIM (Fig. 17b) and LPIPS (Fig. 17c). For instance, Kaleido achieves 0.87 to 0.90 in SSIM, while others only achieve 0.50 to 0.78 in SSIM. Finally, on the VBench metric (Fig. 17d), which captures highlevel video quality, Kaleido maintains the same scores as the baseline algorithms (around 0.81), exceeding the best prior methods. Fig. 17e further showcases the qualitative results of all methods. Visually, Kaleido preserves much higher quality compared to others. 10

Conference’17, July 2017, Washington, DC, USA

8 6 4 2 0

SVG2 Minference

HunyuanVideo

PAB H100

Wan

Ditto Cambricon-D

AdapTiV EXION

TurboDiffusion

Kaleido

K ALEIDO - PE - SH Speedup

Speedup

K ALEIDO - PE

A100 SVG

CogVideoX

K ALEIDO - PE - M

K ALEIDO

Normalized Energy

1.30 1.20 1.10 1.00 0.90 0.80

1.30 1.20 1.10 1.00 0.90 0.80 HunyuanVideo

Wan

TurboDiffusion

CogVideoX

Fig. 19: Ablation study of the different hardware components in Kaleido. Wan

TurboDiffusion

CogVideoX

Ops Count (×10 15 )

(b) Energy savings. Higher is better.

A100

H100

HunyuanVideo

Ditto

Cambricon-D

Wan

AdapTiV

EXION

TurboDiffusion

Kaleido

Oracle

6.8 6.6 6.4 6.2 6.0 5.8

Speedup

K ALEIDO -ATTN Attention Only

PSNR

K ALEIDO -F ULL Attention + Linear

30.3 30.2 30.1 30.0 29.9 29.8

HunyuanVideo

Speedup

HunyuanVideo

PSNR (dB)

25 20 15 10 5 0

Speedup

Energy Savings

(a) Speedup evaluation. Higher is better.

Normalized Energy

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

1.05 1.00 0.95 0.90 0.852x2

4x4

Wan

TurboDiffusion

8x8 16x16 32x32 64x64

# of PE

Fig. 20: Ablation study of per- Fig. 21: Sensitivity of perforformance and quality on Hun- mance to per-array PE size. yuanVideo, with and without The numbers are normalized applying Kaleido to linear to the default configuration projection layers. with 4×4 PEs per PE array.

CogVideoX

Fig. 18: Overall performance evaluation against prior work. i.e., A100, and that of the corresponding accelerator design. Overall, Kaleido achieves 14.8×, 15.6×, 15.2×, and 18.4× energy savings on HunyuanVideo, Wan, TurboDiffusion, and CogVideoX, respectively. The latency overhead of the data dispatcher is negligible. Note that, the scheduler in the data dispatcher can be pipelined and overlapped with the subsequent matrix computation. While AdapTiV exhibits higher energy savings than Kaleido, its energy savings primarily come from its aggressive merge ratio. By merging tokens, AdapTiV can directly reduce the overall token sequence length and reduce a large amount of off-chip DRAM accesses. However, any sparsity computation techniques, e.g., Kaleido or Exion, still have to maintain the full token sequence and cannot eliminate the off-chip data traffic. Nevertheless, as we mentioned earlier, the token merging technique in AdapTiV introduces unacceptable accuracy loss and is practically unacceptable. Remaining baselines all achieve lower energy savings compared to Kaleido. Note that, the data dispatcher contributes 0.1% of the total energy. Operation Reduction. Fig. 18c shows the total operation count for each hardware baseline. Here, we show two numbers of Kaleido: Kaleido and Oracle. Kaleido is the effective operation reduction achieved during actual execution under our hardware limitations. Oracle, on the other hand, is the ideal operation reduction without any hardware limitations. For other hardware baselines, we show their operation reduction based on their algorithms without considering their hardware constraints. We show that, by exploiting spatio-temporal correlations and reusing intermediate channel-wise results, Oracle can achieve up to 85% of operation reduction, eliminating a massive amount of computations. However, with hardware constraints, the operation reduction decreases to around 60%. There are two main reasons leading to the gap between Oracle and Kaleido. First is an algorithmic constraint. In Kaleido, each channel group is restricted to reuse along a single fixed direction to simplify scheduling and hardware mapping. In contrast, Oracle allows each channel to reuse in any direction. Second is hardware-aware mapping overhead. Due to PE-array granularity, some operations marked as reusable may

still be executed as part of a regular PE tile. Ditto achieves the second-best theoretical operation reduction because it aggressively skips 0-value computations. However, Ditto’s adder tree design cannot efficiently process these irregular operations, leading to severe pipeline stalling and poor PE utilization.

7.3

Ablation Study

Hardware Ablation. Fig. 19 shows an ablation study of the four variants in Sec. 6: Kaleido-pe, which includes only the reconfigurable PE array without the data dispatcher; Kaleido-pe-sh, which adds only the scheduler in the data dispatcher; Kaleido-pe-m, which adds only the matcher in the data dispatcher; and the full Kaleido design. Both speedup and energy are normalized toKaleido-pe. Across three vDiT models, we show that introducing our data dispatcher achieves 1.2× speedup and 9.0% of energy reduction. This is because, without the dispatcher, our channel-wise reuse algorithm would introduce irregular sparsity in computation and lead to PE under-utilization. The variant with the matcher achieves higher speedup than the variant with only the scheduler. With our data dispatcher, we improve the PE utilization from 69.4% to 85.2% by clustering tokens with similar reuse patterns and taming the irregularity during computation. The scheduler in the data dispatcher is pipelined and overlapped with the subsequent computation. The latency overhead of our data dispatcher is negligible. Algorithmic Ablation. Fig. 20 compares the performance and generative quality of applying our algorithm either to attention layers only or to all layers. We report the results on HunyuanVideo. Compared with applying Kaleido only to attention layers, the full design achieves higher speedup by exploiting additional channel-wise reuse opportunities in the linear projection layers. Meanwhile, the PSNR remains nearly unchanged. It shows that extending reuse to linear layers preserves generation quality while providing additional acceleration. 11

64x64 0.18 0.26 0.26 0.26 0.26 128x128 0.23 0.43 0.71 1.00 1.03 256x256 0.24 0.47 0.89 1.62 2.63 512x512 0.24 0.47 0.90 1.66 2.88 12 24 48 96 192 Buffer Size(MB)

64x64 4.29 2.44 1.53 1.06 0.83 128x128 4.25 2.40 1.47 1.00 0.77 256x256 4.25 2.39 1.46 0.99 0.75 512x512 4.24 2.39 1.46 0.99 0.75 12 24 48 96 192 Buffer Size(MB)

(a) Speedup.

4.0 3.5 3.0 2.5 2.0 1.5 1.0

Speedup

(b) Normalized energy.

(3, 1) (4, 2) (5, 3) (6, 4) Reuse Thresholds (θth1 , θth2 )

Speedup

8.0 6.0 4.0 2.0 0.0 General

PSNR

35 25

Fast-Motion 15

Fig. 23: Sensitivity of speedup Fig. 24: Comparison of perforand generative quality to mance and generative quality reuse thresholds, 𝜃 th1 and 𝜃 th2 . under fast-motion prompts.

7.5 7.4

480P (Empty) 720P (Solid)

30 75%EXION 25 50%Qth = 15 20 25% 15 4 5

Kaleido

Better 70%

AdapTiV

65%

6 Speedup

50%

inference. We find that setting 𝜃 th1 and 𝜃 th2 to 5 and 3 provides a good trade-off between performance and quality. Fast Motion. Fig. 24 compares the performance and generative quality of Kaleido under general prompts and fast-motion prompts on HunyuanVideo. We extract the prompts with fast motions from VBench. Compared with general prompts, fast-motion prompts achieve slightly lower speedup because rapid temporal changes reduce the amount of reusable spatio-temporal correlation. However, Kaleido still maintains substantial acceleration while preserving similar PSNR. It means that our channel-wise reuse remains effective even for videos with fast motions.

PSNR (dB)

32 30 28 26

PSNR

Speedup

Cambricon-D AdaptiV EXION Kaleido

Fig. 25: Scalability analysis Fig. 26: Comparison against on generating 720p high- AdapTiV and Exion under resolution videos. different configurations.

Fig. 22: Sensitivity of performance and energy efficiency to the number of PE arrays and buffer size. Data are normalized to the configuration with 128×128 PE arrays and 96 MB buffer. 7.20 6.40 5.60 4.80

35 30 Better 25 20 15 10 0 2 4 6 8 10

PSNR (dB)

2.5 2.0 1.5 1.0 0.5

32x32 4.49 2.68 1.77 1.30 1.07 #PE Arrays

#PE Arrays

32x32 0.06 0.06 0.06 0.06 0.06

Wenxuan Miao et al.

PSNR (dB)

Conference’17, July 2017, Washington, DC, USA

Scalability Analysis

Fig. 25 shows the scalability of different methods when generating higher-resolution (720p) videos. As the resolution increases from 480p to 720p, Kaleido consistently achieves both higher speedup and better visual quality compared to prior approaches. This is because our approach inherently exploits the spatio-temporal correlations across frames in the latent space. These correlations are stronger as the resolution increases. As a result, Kaleido not only scales efficiently with resolution but also further widens the gap in both performance and quality over prior approximation methods.

Sensitivity Study

PE Size. Fig. 21 shows the sensitivity of speedup to the per-array PE size, where we vary the number of PEs per array while retaining the total number of PEs constant. All numbers are normalized to the default 4×4 configuration. Overall, speedup degrades slightly as the per-array PE size increases. This is because larger PE arrays are less flexible in handling irregular reuse patterns, thus reducing PE utilization. Nevertheless, Kaleido remains robust even with larger PE arrays. Meanwhile, further reducing the array size to 2×2 achieves marginal benefit. Thus, we choose the 4×4 configuration. PE Array & Buffer Size. Fig. 22 shows the sensitivity of performance and energy efficiency to the number of PE arrays and buffer size. All results are normalized to our default configuration, i.e., 128×128 PE arrays and a 96 MB buffer. Each PE array consists of 4×4 PEs. As shown in Fig. 22a, increasing the number of PE arrays generally improves the overall performance. However, if the onchip buffer is not large enough, off-chip data communication would impact the performance. Larger buffers reduce off-chip memory traffic, often leading to higher performance. Fig. 22 further shows the normalized energy consumption under different PE and buffer configurations. Increasing the number of PE arrays slightly reduces overall energy because it reduces execution time. Meanwhile, a larger buffer size can better improve energy efficiency because it minimizes off-chip data traffic overhead. Overall, both Fig. 22a and Fig. 22 show that the number of PE arrays and the total size of on-chip buffer need to be balanced to achieve optimal performance and energy efficiency. Thresholds. In our algorithm design, our default threshold setting is to set 𝜃 th1 and 𝜃 th2 to preserve the most significant 5 bits and 3 bits of accuracy, respectively. In Fig. 23, we show the sensitivity of performance and accuracy to different threshold combinations. The results show that preserving fewer bits leads to noticeable degradation in visual quality, whereas preserving more bits slows

7.6

Comparison with AdapTiV and Exion.

Fig. 26 compares Kaleido with AdapTiV and Exion under different configurations. The configurations are annotated in Fig. 26. Although AdapTiV and Exion achieve higher speedup with more aggressive token merging or sparsity, their generative quality degrades substantially as the compression ratio increases. In contrast, Kaleido achieves higher speedup while maintaining much higher PSNR, demonstrating that channel-wise reuse better preserves video quality while still providing strong acceleration.

Related Work

Diffusion Acceleration Techniques. Algorithmically, early diffusion acceleration methods primarily focus on reducing the number of denoising timesteps, using techniques such as DDIM [65], DPMSolver [46, 47], and flow matching [40]. Subsequent work further exploits similarity across timesteps to skip less important denoising steps, e.g., PAB [96], Astraea [42], and ToCa [100]. In contrast, Kaleido operates at the channel level, allowing different channel groups to be reused differently according to the spatio-temporal dimensions they encode. This finer granularity can preserve better visual fidelity. Meanwhile, another line of research explores sparse patterns in attention scores [30, 52, 77, 83] to reduce self-attention 12

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

Conference’17, July 2017, Washington, DC, USA

overhead. Instead, Kaleido reuses previously computed partial attention results for correlated channel values. Thus, rather than discarding attention computations, Kaleido preserves them through reuse. Diffusion caching methods [5, 41] exploit redundancy across denoising timesteps by reusing intermediate activations. Kaleido targets a different source of redundancy. Therefore, Kaleido is complementary to diffusion caching methods. Diffusion Accelerators. More recently, the hardware community has also focused on diffusion acceleration [21, 33, 34, 87, 88, 101], proposing a range of accelerator designs that exploit computation sparsity and timestep similarity. For example, AdapTiV [87] merges similar tokens during execution to reduce inference computation, while EXION [21] exploits the similarity of attention sparsity across denoising steps to reduce masking overhead. Other works, such as Cambricon-D [34] and Ditto [33], leverage mixed precision to reduce computation overhead. However, none of these studies explores the unique channel-wise spatio-temporal correlations in the video latent space. In contrast, we are the first work that identifies the root cause of pattern diversity and proposes a principled reuse-based strategy to accelerate all vDiTs. Sparse Acceleration. A broad body of prior work has explored sparse computation techniques [6, 15, 19, 28, 29, 45, 48, 58, 59, 68, 74, 76, 93, 98]. Early efforts primarily focused on accelerating sparse matrix–matrix multiplication. E.g., OuterSPACE [58], SpArch [93], and MatRaptor [68] explore different dataflows to improve computational efficiency, while SIGMA [59] and STC [98] address irregular sparsity. As DNNs became the dominant workload, subsequent work focused on exploiting sparsity in DNNs. For instance, Eyeriss [6] improves flexibility for compact and sparse models, while SparTen [15] and S2TA [45] target different sparsity patterns. HighLight [76] further addresses workload imbalance in sparse computation. More recently, research has focused on accelerating attention mechanisms. For example, A3 [19], SpAtten [74], and Sanger [48] reduce attention overhead through techniques such as approximation, token pruning, and quantization. In contrast, our work targets a fundamentally different opportunity and exploits the channel-wise spatio-temporal correlations unique to vDiTs. Spatio-Temporal Similarity. There is a long history of exploiting spatio-temporal correlations to improve the efficiency of continuous vision [3, 14, 51, 67, 86, 94, 95, 99]. For example, both EVA2 [3] and Euphrates [99] observe the continuity in videos and propose motion-guided reuse techniques to reduce CNN inference cost. Subsequent work, e.g., Diffy [51] and ASV [14], extends this idea to other vision tasks. VR-DANN [67] and CMC [66] further exploit codec metadata to guide efficient video understanding. Meanwhile, Deja View [94] and Cicero [13] show that spatio-temporal correlations can also be used to improve the efficiency of VR applications.

leveraging this insight, we proposed a lightweight reuse algorithm with a co-designed accelerator to efficiently address the irregular sparsity. Our principled design achieves up to 5.9× speedup and 16.0× energy savings over state-of-the-art GPUs and accelerators.

Acknowledgments This work was supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM113), the National Natural Science Foundation of China (NSFC) Grants (62532006 and 62402312), Shanghai Pujiang Talent Program (24PJA044), and Shanghai Qi Zhi Institute Innovation Program (SQZ202316).

References [1] Akshay Kulkarni. 2026. Predis.ai Announces Launch of New State-of-the-Art (SOTA) AI Models for Ad Videos. https://www.manilatimes.net/2026/01/09/tmtnewswire/globenewswire/predisai-announces-launch-of-new-state-of-theart-sota-ai-models-for-ad-videos/2256039 [2] Daniel Bolya and Judy Hoffman. 2023. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4599–4603. [3] Mark Buckler, Philip Bedoukian, Suren Jayasuriya, and Adrian Sampson. 2018. EVA2 : Exploiting temporal redundancy in live computer vision. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 533–546. [4] Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, et al. 2025. SANAVideo: Efficient Video Generation with Block Linear Diffusion Transformer. (2025). arXiv:2509.24695 [cs.CV] https://arxiv.org/abs/2509.24695 [5] Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, ChristosSavvas Bouganis, Yiren Zhao, and Tao Chen. 2024. Delta-DiT: A TrainingFree Acceleration Method Tailored for Diffusion Transformers. arXiv preprint arXiv:2406.01125 (2024). [6] Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze. 2019. Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 2 (2019), 292–308. [7] Zheng Chen, Zichen Zou, Kewei Zhang, Xiongfei Su, Xin Yuan, Yong Guo, and Yulun Zhang. 2025. DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution. In NeurIPS. [8] Chris McKay. [n. d.]. World Labs’ AI System Can Generate a 3D World from an Image. https://www.maginative.com/article/world-labs-ai-system-cangenerate-a-3d-world-from-an-image/ [9] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 9 (2023), 10850–10869. [10] Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023). [11] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359. [12] Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. 2025. Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile. arXiv preprint arXiv:2502.06155 (2025). [13] Yu Feng, Zihan Liu, Jingwen Leng, Minyi Guo, and Yuhao Zhu. 2025. Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations. In Proceedings of the 51st Annual International Symposium on Computer Architecture (Buenos Aires, Argentina) (ISCA ’24). IEEE Press, 1293–1308. https://doi.org/10.1109/ISCA59077.2024. 00096 [14] Yu Feng, Paul Whatmough, and Yuhao Zhu. 2019. Asv: Accelerated stereo vision system. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture. 643–656. [15] Ashish Gondimalla, Noah Chesnut, Mithuna Thottethodi, and T. N. Vijaykumar. 2019. SparTen: A Sparse Tensor Accelerator for Convolutional Neural Networks. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (Columbus, OH, USA) (MICRO-52). Association for Computing Machinery, New York, NY, USA, 151–165. https://doi.org/10.1145/3352460.3358291 [16] Google. 2024. Veo 2: Our state-of-the-art video generation model. https: //deepmind.google/technologies/veo/veo-2/ [17] Google DeepMind. [n. d.]. Veo3: Our state-of-the-art video generation model. https://deepmind.google/models/veo/

Conclusion

As video generation continues to advance rapidly, the next frontier of generative AI will be vDiT models that can fundamentally understand the physical world. This paper introduced Kaleido, an algorithm–hardware co-design that rethinks how to accelerate video diffusion transformers. By exploiting channel-wise spatio-temporal correlations in the latent space, we reveal the fundamental relationships between the token channels and their attention scores. By 13

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al.

[37] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [38] Jianze Li, Jiezhang Cao, Zichen Zou, Xiongfei Su, Xin Yuan, Yulun Zhang, Yong Guo, and Xiaokang Yang. 2025. Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator. (2025). arXiv:2410.04224 [cs.CV] https://arxiv.org/abs/2410.04224 [39] Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. 2024. Open-Sora Plan: Open-Source Large Video Generation Model. arXiv preprint arXiv:2412.00131 (2024). [40] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022). [41] Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. 2024. Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model. (2024). arXiv:2411.19108 [cs.CV] https://arxiv.org/abs/2411.19108 [42] Haosong Liu, Yuge Cheng, Wenxuan Miao, Zihan Liu, Aiyue Chen, Jing Lin, Yiwu Yao, Chen Chen, Jingwen Leng, Yu Feng, and Minyi Guo. 2025. Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformers. (2025). arXiv:2506.05096 [cs.CV] https://arxiv.org/abs/2506.05096 [43] Joseph Liu, Joshua Geddes, Ziyu Guo, Haomiao Jiang, and Mahesh Kumar Nandwana. 2024. SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers. arXiv preprint arXiv:2411.10510 (2024). [44] Jinlai Liu, Jian Han, Bin Yan, Hui Wu, Fengda Zhu, Xing Wang, Yi Jiang, Bingyue Peng, and Zehuan Yuan. 2025. InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation. (2025). arXiv:2511.04675 [cs.CV] https://arxiv. org/abs/2511.04675 [45] Zhi-Gang Liu, Paul N Whatmough, Yuhao Zhu, and Matthew Mattina. 2022. S2ta: Exploiting structured sparsity for energy-efficient mobile cnn acceleration. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 573–586. [46] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems 35 (2022), 5775–5787. [47] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 (2022). [48] Liqiang Lu, Yicheng Jin, Hangrui Bi, Zizhang Luo, Peng Li, Tao Wang, and Yun Liang. 2021. Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture (Virtual Event, Greece) (MICRO ’21). Association for Computing Machinery, New York, NY, USA, 977–991. https://doi.org/10.1145/3466752.3480125 [49] Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang, Zecheng Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung-Yeung Shum, and Daxin Jiang. 2025. StepVideo-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model. arXiv:2502.10248 [cs.CV] https://arxiv.org/abs/2502.10248 [50] Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2024. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15762–15772. [51] Mostafa Mahmoud, Kevin Siu, and Andreas Moshovos. 2018. Diffy: a déjà vufree differential deep neural network accelerator. In Proceedings of the 51st Annual IEEE/ACM International Symposium on Microarchitecture (Fukuoka, Japan) (MICRO-51). IEEE Press, 134–147. https://doi.org/10.1109/MICRO.2018.00020 [52] Wenxuan Miao, Yulin Sun, Aiyue Chen, Jing Lin, Yiwu Yao, Yiming Gan, Jieru Zhao, Jingwen Leng, Minyi Guo, and Yu Feng. 2026. TimeRipples: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space. In

[18] Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. 2024. LTX-Video: Realtime Video Latent Diffusion. arXiv preprint arXiv:2501.00103 (2024). [19] Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W Lee, et al. 2020. Aˆ 3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 328–341. [20] Tae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae W. Lee. 2021. ELSA: Hardware-Software Co-design for Efficient, Lightweight Self-Attention Mechanism in Neural Networks. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). 692–705. https://doi.org/10.1109/ISCA52012.2021.00060 [21] Jaehoon Heo, Adiwena Putra, Jieon Yoon, Sungwoong Yune, Hangyeol Lee, Ji-Hoon Kim, and Joo-Young Kim. 2025. EXION: Exploiting Inter-and IntraIteration Output Sparsity for Diffusion Models. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 324–337. https: //doi.org/10.1109/HPCA61900.2025.00034 [22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840–6851. [23] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2022. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 (2022). [24] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. 2024. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21807–21818. [25] Team HunyuanWorld. 2025. HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels. arXiv preprint (2025). [26] Micron Technology Inc. 2025. Micron System Power Calculators. https://www. micron.com/support/tools-and-utilities/power-calc. [27] Jack Parker-Holder, Shlomi Fruchter. [n. d.]. Genie 3: A new frontier for world models. https://deepmind.google/blog/genie-3-a-new-frontier-for-worldmodels/ [28] Hanhwi Jang, Joonsung Kim, Jae-Eon Jo, Jaewon Lee, and Jangwoo Kim. 2019. MnnFast: a fast and scalable system architecture for memory-augmented neural networks. In Proceedings of the 46th International Symposium on Computer Architecture (Phoenix, Arizona) (ISCA ’19). Association for Computing Machinery, New York, NY, USA, 250–263. https://doi.org/10.1145/3307650.3322214 [29] Jun-Woo Jang, Sehwan Lee, Dongyoung Kim, Hyunsun Park, Ali Shafiee Ardestani, Yeongjae Choi, Channoh Kim, Yoojin Kim, Hyeongseok Yu, Hamzah Abdel-Aziz, Jun-Seok Park, Heonsoo Lee, Dongwoo Lee, Myeong Woo Kim, Hanwoong Jung, Heewoo Nam, Dongguen Lim, Seungwon Lee, Joon-Ho Song, Suknam Kwon, Joseph Hassoun, SukHwan Lim, and Changkyu Choi. 2021. Sparsity-Aware and Re-configurable NPU Architecture for Samsung Flagship Mobile SoC. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). 15–28. https://doi.org/10.1109/ISCA52012.2021.00011 [30] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems 37 (2024), 52481– 52515. [31] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. 2017. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual ACM/IEEE International Symposium on Computer Architecture. [32] Katie Deighton. [n. d.]. Coca-Cola Injects ‘Holidays Are Coming’ Ads With an Upgraded Dose of AI. https://www.wsj.com/articles/coca-cola-injectsholidays-are-coming-ads-with-an-upgraded-dose-of-ai-bc8921e2 [33] Sungbin Kim, Hyunwuk Lee, Wonho Cho, Mincheol Park, and Won Woo Ro. 2025. Ditto: Accelerating Diffusion Model via Temporal Value Similarity. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 338–352. [34] Weihao Kong, Yifan Hao, Qi Guo, Yongwei Zhao, Xinkai Song, Xiaqing Li, Mo Zou, Zidong Du, Rui Zhang, Chang Liu, et al. 2024. Cambricon-d: Full-network differential acceleration for diffusion models. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 903–914. [35] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. 2024. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024). [36] Kuaishou. 2024. Kuaishou Unveils Proprietary Video Generation Model ‘Kling’; Testing Now Available. https://ir.kuaishou.com/news-releases/news-releasedetails/kuaishou-unveils-proprietary-video-generation-model-kling 14

Kaleido : Algorithm-Hardware Co-Design for Video Diffusion Transformers

Conference’17, July 2017, Washington, DC, USA

Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. 2025. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv preprint arXiv:2503.20314 (2025). [74] Hanrui Wang, Zhekai Zhang, and Song Han. 2021. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 97–110. https://doi.org/10.1109/HPCA51647.2021.00018 [75] Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. 2024. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6211– 6220. [76] Yannan Nellie Wu, Po-An Tsai, Saurav Muralidharan, Angshuman Parashar, Vivienne Sze, and Joel Emer. 2023. HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’23). Association for Computing Machinery, New York, NY, USA, 1106–1120. https: //doi.org/10.1145/3613424.3623786 [77] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. 2025. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. arXiv preprint arXiv:2502.01776 (2025). [78] Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. 2025. Training-free and Adaptive Sparse Attention for Efficient Long Video Generation. arXiv preprint arXiv:2502.21079 (2025). [79] Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. 2024. Easyanimate: A high-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405.18991 (2024). [80] Lu Xu, Sijie Zhu, Chunyuan Li, Chia-Wen Kuo, Fan Chen, Xinyao Wang, Guang Chen, Dawei Du, Ye Yuan, and Longyin Wen. 2025. Beyond raw videos: Understanding edited videos with large multimodal model. In Proceedings of the Computer Vision and Pattern Recognition Conference. 503–512. [81] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. 2023. Diffusion models: A comprehensive survey of methods and applications. Comput. Surveys 56, 4 (2023), 1–39. [82] Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al. 2025. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. arXiv preprint arXiv:2505.18875 (2025). [83] Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, Jianfei Chen, Song Han, Kurt Keutzer, and Ion Stoica. 2025. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. (2025). arXiv:2505.18875 [cs.CV] https://arxiv.org/abs/2505.18875 [84] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. 2024. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024). [85] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005 (2025). [86] Ziyu Ying, Shulin Zhao, Haibo Zhang, Cyan Subhra Mishra, Sandeepa Bhuyan, Mahmut T Kandemir, Anand Sivasubramaniam, and Chita R Das. 2022. Exploiting Frame Similarity for Efficient Inference on Edge Devices. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS). IEEE, 1073–1084. [87] Seungjae Yoo, Hangyeol Kim, and Joo-Young Kim. 2024. AdapTiV: SignSimilarity Based Image-Adaptive Token Merging for Vision Transformer Acceleration. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 64–77. https://doi.org/10.1109/MICRO61859.2024.00015 [88] Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao, Yongan Zhang, Chaojian Li, Baopu Li, and Yingyan Lin. 2023. ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 273–286. https://doi.org/10.1109/HPCA56546.2023.10071027 [89] Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. 2024. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems 37 (2024), 1196–1219. [90] Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. 2024. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8850–8860.

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 25688–25698. [53] Micron Technology, Inc. 2023. Micron DDR5 128GB RDIMM. https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3A6ffd17ace709-469d-9473-a0a904681dd9/renditions/original/as/128gb-ddr5-rdimmproduct-brief.pdf Product Brief, Rev. A. [54] Nvidia. [n. d.]. NVIDIA RTX PRO 6000 Blackwell. https://www.nvidia.com/enus/products/workstations/professional-desktop-gpus/rtx-pro-6000/ [55] Nvidia. 2021. NVIDIA A100 PCIe 80 GB. https://www.nvidia.com/en-us/datacenter/a100/ [56] Nvidia. 2022. NVIDIA H100 PCIe 80 GB. https://www.nvidia.com/en-us/datacenter/h100/ [57] OpenAI. 2024. Sora: Bring your imagination to life with text, image, or video. https://openai.com/sora/ [58] Subhankar Pal, Jonathan Beaumont, Dong-Hyeon Park, Aporva Amarnath, Siying Feng, Chaitali Chakrabarti, Hun-Seok Kim, David Blaauw, Trevor Mudge, and Ronald Dreslinski. 2018. OuterSPACE: An Outer Product Based Sparse Matrix Multiplication Accelerator. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA). 724–736. https://doi.org/10. 1109/HPCA.2018.00067 [59] Eric Qin, Ananda Samajdar, Hyoukjun Kwon, Vineet Nadella, Sudarshan Srinivasan, Dipankar Das, Bharat Kaul, and Tushar Krishna. 2020. SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 58–70. https://doi.org/10.1109/HPCA47549.2020.00015 [60] Lenny Rachitsky. 2025. The Godmother of AI on jobs, robots and why world models are next. https://www.lennysnewsletter.com/p/the-godmother-of-ai [61] Runway. 2024. Runway Gen 2: The Next Step Forward for Generative AI. https://medium.com/@dbhatt245/runway-gen-2-the-next-step-forwardfor-generative-ai-an-introduction-b85bc90d3e45 [62] Tim Salimans and Jonathan Ho. 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 (2022). [63] Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. 2025. Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model. arXiv preprint arXiv:2504.08685 (2025). [64] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022. Makea-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022). [65] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020). [66] Zhuoran Song, Chunyu Qi, Fangxin Liu, Naifeng Jing, and Xiaoyao Liang. 2024. CMC: Video Transformer Acceleration via CODEC Assisted Matrix Condensing. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 201–215. https://doi.org/10.1145/3620665.3640393 [67] Zhuoran Song, Feiyang Wu, Xueyuan Liu, Jing Ke, Naifeng Jing, and Xiaoyao Liang. 2020. Vr-dann: Real-time video recognition via decoder-assisted neural network acceleration. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 698–710. [68] Nitish Srivastava, Hanchen Jin, Jie Liu, David Albonesi, and Zhiru Zhang. 2020. MatRaptor: A Sparse-Sparse Matrix Multiplication Accelerator Based on RowWise Product. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 766–780. https://doi.org/10.1109/MICRO50266.2020. 00068 [69] Genmo Team. 2024. Mochi 1. https://github.com/genmoai/models. [70] Meituan LongCat Team, Xunliang Cai, Qilong Huang, Zhuoliang Kang, Hongyu Li, Shijun Liang, Liya Ma, Siyu Ren, Xiaoming Wei, Rixu Xie, and Tong Zhang. 2025. LongCat-Video Technical Report. (2025). arXiv:2510.22200 [cs.CV] https://arxiv.org/abs/2510.22200 [71] Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, Yihang Chen, Jie Liu, Yansong Cheng, Yao Yao, Jiayi Zhu, Yihao Meng, Kecheng Zheng, Qingyan Bai, Jingye Chen, Zehong Shen, Yue Yu, Xing Zhu, Yujun Shen, and Hao Ouyang. 2026. Advancing Open-source World Models. arXiv preprint arXiv:2601.20540 (2026). [72] Tencent. 2024. Tencent launches and open-sources Hunyuan video-generation model. https://technode.com/2024/12/04/tencent-launches-and-open-sourceshunyuan-video-generation-model/ [73] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, 15

Conference’17, July 2017, Washington, DC, USA

Wenxuan Miao et al.

[91] Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. 2025. TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times. arXiv preprint arXiv:2512.16093 (2025). [92] Zhekai Zhang, Hanrui Wang, Song Han, and William J. Dally. 2020. SpArch: Efficient Architecture for Sparse Matrix Multiplication. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 261–274. https: //doi.org/10.1109/HPCA47549.2020.00030 [93] Zhekai Zhang, Hanrui Wang, Song Han, and William J. Dally. 2020. SpArch: Efficient Architecture for Sparse Matrix Multiplication. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 261–274. https: //doi.org/10.1109/HPCA47549.2020.00030 [94] Shulin Zhao, Haibo Zhang, Sandeepa Bhuyan, Cyan Subhra Mishra, Ziyu Ying, Mahmut T Kandemir, Anand Sivasubramaniam, and Chita R Das. 2020. Déja view: Spatio-temporal compute reuse for ‘energy-efficient 360 vr video streaming. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 241–253. [95] Shulin Zhao, Haibo Zhang, Cyan Subhra Mishra, Sandeepa Bhuyan, Ziyu Ying, Mahmut Taylan Kandemir, Anand Sivasubramaniam, and Chita Das. 2021. HoloAR: On-the-fly optimization of 3D holographic processing for augmented reality. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture. 494–506.

[96] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. 2024. Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:2408.12588 (2024). [97] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. 2024. Open-Sora: Democratizing Efficient Video Production for All. https://github.com/hpcaitech/Open-Sora [98] Maohua Zhu, Tao Zhang, Zhenyu Gu, and Yuan Xie. 2019. Sparse Tensor Core: Algorithm and Hardware Co-Design for Vector-wise Sparse Neural Networks on Modern GPUs. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (Columbus, OH, USA) (MICRO-52). Association for Computing Machinery, New York, NY, USA, 359–371. https://doi.org/10. 1145/3352460.3358269 [99] Yuhao Zhu, Anand Samajdar, Matthew Mattina, and Paul Whatmough. 2018. Euphrates: Algorithm-soc co-design for low-power mobile continuous vision. arXiv preprint arXiv:1803.11232 (2018). [100] Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, and Linfeng Zhang. 2024. Accelerating diffusion transformers with token-wise feature caching. arXiv preprint arXiv:2410.05317 (2024). [101] Zihan Zou, Xinming Yan, Shun Zhang, Peng Zheng, Guang Yang, Hao Cai, and Bo Liu. 2025. S-DMA: Sparse Diffusion Models Acceleration via SpatialityAware Prediction and Dimension-Adaptive Dataflow. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 432–444.

Record · ID 370370 · SHA-256 b8d37283d4b3d5e8
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.