arXiv:2604.18348v1 [cs.CV] 20 Apr 2026
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation Haoyue Tan1,2 *, Shengnan Wang3 *, Yulin Qiao4 , Juncheng Zhang1,5 , Youhui Bai1†, Ping Gong1 , Zewen Jin1 , Cheng Li1,2 1 University of Science and Technology of China 2 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center 3 Independent Researcher 4 University of Macau 5 The Chinese University of Hong Kong
Abstract Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity, or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity preserving clustering method to query vectors for higher compression, and designs a euclideansimilarity preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 via one A40 GPU demonstrate up to 1.67×-4.31× speedup with negligible quality degradation. Code: https://github.com/USTC-MLSys-Team/Adacluster
1. Introduction Diffusion transformer models (DiTs) have emerged as a dominant paradigm for video generation and physical scene simulation, demonstrating strong scalability and high-fidelity synthesis capabilities [3, 6, 26]. However, generation latency remains a major bottleneck due to inherently high computational complexity [4, 41, 43]. The primary source of this cost is the attention module, whose complexity scales quadratically with the input sequence length [38]. In video generation, the sequence length grows with both resolution and the number of frames, resulting in substan* Equal contribution. † Corresponding author, [email protected].
Full Attention | latency = 1639s
AdaCluster | latency = 973s | PSNR = 30.58dB
Figure 1. Performance comparison between Full Attention and AdaCluster methods on HunyuanVideo, with videos generated on a single A40. AdaCluster reduces inference latency from 1639s to 973s with a PSNR of 30.58dB, significantly improving inference speed while maintaining visual quality.
tially longer input sequences. For instance, in CogVideoX2B [52], we generated a video with 81 frames at a resolution of 720p on a single NVIDIA A40 GPU. The entire generation takes 1691 seconds, where the input sequence length is 70K tokens and attention accounted for 75% of the total generation time. We also observe similar phenomena in HunyuanVideo [16] and other DiT models. To address this, existing works, such as Top-K attention [27, 59], exploit the inherent sparsity of attention to accelerate computation, namely only a few critical tokens dominate output accuracy [40, 62]. One category of methods aggregates consecutive tokens into fixed-size blocks, and select a representative (usually the average of the tokens) from each block for significance estimation [56, 58]. However, consecutive tokens are not necessarily semantically close in the embedding space. Some tokens may be far away from the block representative, and hence leads to de-
graded attention accuracy. To address this issue, the token clustering methods are adopted in the later methods [51], through which higher intra-group token similarity is ensured, hence achieving better accuracy. In cluster-based sparse attention methods, queries and keys are typically clustered to achieve significant acceleration. However, existing approaches apply a uniform strategy to both queries and keys, using the same Euclidean distance-based clustering. This overlooks the distinct roles that queries and keys play in the attention mechanism, as well as the significantly different patterns they exhibit. In this paper, we propose AdaCluster, a training-free adaptive clustering framework that introduces a role-aware redesign of the entire “cluster-then-select” pipeline. We design customized clustering strategies for queries and keys separately. The core technical contributions are as follows: 1. Query Clustering. For the query tokens, we analyze that the relative magnitude of query–key scores is independent of the query length. Based on this, AdaCluster first normalizes queries and proposes an angle-based clustering method, which can compress the query tokens by a large ratio. 2. Key Clustering. For the key tokens, we show that the distribution of the key tokens varies greatly across different layers. Due to this, we propose an layerwise adaptive kmeans clustering method, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. 3. Evaluation. We build custom operators for AdaCluster on Triton [36] and FlashInfer [54], and test it on opensource DiT models including CogVideoX-2B [52], HunyuanVideo [16], and Wan-2.1 [39]. Experimental results on one A40 GPU demonstrate that AdaCluster achieves an end-to-end acceleration of 1.67×–4.31× for resolutions above 720p within our test range, while maintaining high visual fidelity with PSNR of 30.99.
2. Background and Motivation 2.1. DiT Models and Computation Bottleneck DiT models. Video generation has emerged as a transformative technology, enabling the creation of high-quality, realistic videos for applications in entertainment, advertising, and virtual reality [44]. Recently, Video Diffusion Transformers (DiTs) have demonstrated superior performance in generating temporally consistent and visually compelling videos by leveraging a Transformer-based architecture [5, 22, 42, 48, 60]. Popular DiT models share a similar architecture and inference workflow. Each layer typically includes an attention module to capture spatial-temporal dependencies across frames and a feed-forward network (FFN) for nonlinear feature enhancement [24, 48]. During inference, DiTs
generate videos by iteratively denoising random latent noise over dozens of steps, with each step requiring a full forward pass to ensure temporal coherence. Computation bottleneck of DiT. Despite their capabilities, generating high-quality videos with DiTs remains prohibitively time-consuming and resource-intensive. For instance, generating an 81-frame video at 1280×720 resolution takes more than 27 minutes on an A40 GPU with 48GB HBM. A higher-resolution version (1920×1120) of the same video takes drastically longer (130 minutes). Our analysis identifies the attention mechanism as the primary performance bottleneck, a finding consistent with prior research [25, 37, 61]. In the two tasks mentioned above, attention computation accounted for 67.1% and 83.8% of the total time, respectively. This bottleneck arises because the computational complexity of attention scales quadratically with the input sequence length [28]. In DiTs, this sequence length is the product of the frame count, height, and width, leading to extremely long sequences [11]. For our examples, the sequence lengths reached 70K and 180K tokens, respectively. We observed a similar phenomenon in CogVideoX-2B [52], another widely used DiT model. In summary, the performance bottleneck caused by excessively long sequences in DiT attention computation severely hinders the development of applications requiring high resolution or long video generation.
2.2. Sparse Attention To address the bottleneck of full attention, researchers have exploited the inherent sparsity of attention mechanisms [8, 50, 55]. Only a small fraction of key/value tokens contribute significantly to the final output, so selectively retaining these important tokens can substantially reduce computational overhead while preserving model quality [1, 13]. Existing approaches generally fall into two categories: static sparsity and dynamic sparsity. Static sparsity simplifies the sparsity identification during inference by using predefined and fixed attention patterns. Methods such as MInference [15] and SVG [46] perform offline analysis of attention behavior and assign fixed sparse patterns to each attention head based on empirical statistics or heuristic rules. These sparsity patterns typically capture spatial or temporal dependencies, reflecting the inherent characteristics of attention heads in DiT [23, 49]. However, static patterns lack sufficient generality and fail to capture all important attention regions, potentially omitting critical tokens. To address this, recent methods increasingly resort to dynamic sparsity schemes [19, 29, 31, 45, 47, 51, 56, 57]. The most representative family is the Top-K attention approach, which computes the attention score matrix S = QK T and selects the Top-K key–value pairs via these scores, hence preserving enhanced flexibility and model quality. However, computing the whole query-
Layer 15
3. Methodology This section introduces the overall design of AdaCluster. In the existing clustering based method SVG2 [51], both query and key vectors are treated equally, and they were clustered using the same manner. To maximize efficiency and mean-
7.5
5.0
5.0
2.5
2.5
0.0
0.0
-2.5
-2.5
-5.0
-5.0
-7.5
-7.5 -5.0
2.3. Limitations of Dynamic Sparsity Methods
0.0
5.0
10.0
-5.0
5.0
10.0
100.0
20.0
50.0 0.0
0.0
-50.0
-20.0 -40.0 -40.0
0.0
Layer 29
Layer 20
40.0
Dimension2
Here, we conduct a study to uncover the limitations of existing dynamic sparsity and motivate our new design. To do so, we generate videos with Hunyuan [16] and Wan-2.1 [39] models, and analyze the distribution of query-key vectors used for computing the attention scores. The main findings are summarized as follows. Distributions of query-key vectors are ignored. SpargeAttn [56] processes vectors in consecutive blocks without considering inherent properties such as numerical similarity. SVG2 [51] considers similarity but uses a fixed clustering scheme with 100 query clusters and 500 key clusters for all models. However, we observe that the distribution of query–key vectors differs substantially across models and layers. Here, we take the numerical distribution of key vectors as an example. As shown in Figure 2, some layers have highly dispersed token representations, requiring more clusters to preserve critical keys, or even being unsuitable for clustering. In contrast, layers with concentrated token distributions can be effectively compressed with fewer clusters. This diversity makes it difficult for existing static clustering methods to effectively capture critical tokens, potentially causing reduction in attention accuracy and output quality. Critical tokens may be missed after clustering. After clustering, we need to evaluate the significance of all clusters for each query vector and select tokens from the Top-K critical clusters for attention computation. Existing methods typically estimate cluster significance based on attention scores computed over cluster centers. However, this approach may overlook important tokens, since a cluster center cannot fully represent all critical tokens, especially when those tokens lie near the cluster boundaries rather than close to the center. Together, these limitations underscore the need for an adaptive query–key clustering mechanism that can dynamically determine the appropriate number of clusters, and efficiently identify critical tokens after clustering.
Layer 50
7.5
Dimension2
key scores for all the queries is very time-consuming. To address this, SpargeAttn [56] detects sparsity patterns at the block level by grouping consecutive tokens. In contrast, Sparse VideoGen2 (SVG2) [51] introduces token clustering for compression, yielding more flexible and efficient sparse attention scheduling. Sparse attention variations with dynamic input patterns are more widely adopted due to their generality and flexibility, which is the focus of this paper.
-100.0 -20.0
0.0
20.0
Dimension1
40.0
-100.0
-50.0
0.0
50.0
100.0
Dimension1
Figure 2. Token distribution visualization across different layers in Hunyuan(blue) and Wan-2.1(red) models. The token distributions are obtained by randomly sampling tokens across all steps of generating, and their 128-dimensional features are projected into a 2-dimensional space using PCA for visualization.
while ensuring accuracy, AdaCluster designs customized clustering methods for the queries and keys, respectively. We analyze that in the attention mechanism, the query and key have different patterns. In query clustering, we only need to ensure a relaxed angle similarity, so we normalize them before clustering, which can increase the compression ratio. Unlike this, the key vectors clustering is relatively complex, since it requires a critical euclidean similarity. To this end, we propose an effective method, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection.
3.1. Query Clustering We first introduce the method to compress the query vector. Since the query vectors are distributed in the highdimensional space, and the lengths of the vectors vary greatly, directly clustering on the original space is difficult, only limited compression ratio can be achieved. Note that for any query q, the relative magnitude of the querykey scores s is independent of the query vector length [2]. Hence, we can first normalize the query vectors, and use the normalized query to measure the significance of the key vectors. As shown in Figure 3 , the distribution of the queries is much more compact, so we can cluster the queries by a much high compression ratio. Overall, in our query clustering method, we actually group the queries that
After normalization
Before normalization 1 0.75 0.50
3.0 2.5 2.0 z
1.5 1.0 0.5 0.0
z
4
2
0 y -2 -4
-2
8 4 6 x 0 2
10
12
0.25 0 -0.25 -0.5 -0.75 1 1
0.5
0 y
-0.5
-1
1
-0.5
0.5 1
0 x
Figure 3. Query vector distributions before (left) and after (right) normalization. We normalize queries onto a unit sphere, making their distribution more compact and clustering more efficient. Compactness Score of Wan-2.1 model
Compactness Score of Hunyuan model Compactness Score
Compactness Score
N
1
1 0.8 0.6 0.4
MSEil =
0.8 0.6 0.4 0.2
0.2 0
keys. Instead, we only need to measure and compare the significance of clusters. According to above analysis, the compactness of the key vectors in the same cluster determines whether the critical keys can be accurately select, and hence affects the final model performance. SVG2 adopts uniform number of clusters for all the layers. However, we observe that the range of keys distribution varies greatly across different layers, as show in Figure 2. To measure the intra-cluster compactnessof different layers more explicitly, we randomly sample a request with length N , and cluster the key tokens of all the layers with a uniform number of cluster centers. Then we compute the mean squared reconstruction error (MSE) of each head i in layer l by
0
10
20 30 40 Layer Index
50
0
0 3 6 9 12 15 18 21 24 27 Layer Index
Figure 4. Layer-wise compactness scores of the Wan-2.1 and Hunyuan models. The x-axis represents the layer index, and the y-axis shows the normalized compactness score Cl . Note that the values are computed with respect to 400 cluster centers there.
are close at the angle level, which is much more efficient than the existing method using the Euclidean-level clustering.The corresponding theoretical analysis can be found in Appendix A.2.
3.2. Key Clustering 3.2.1. Observation and Analysis Though the complexity of attention is greatly reduced after query clustering, when the sequence length is long, identifying significant key tokens is still the most time-consuming step in Top-K attention. To further speed up the denoising process, we also consider to cluster the key vectors. Different from the queries, both the vector direction and the vector lengths of the keys affect the Top-K attention result, so the keys require Euclidean-level clustering. Considering an arbitrary k, assume c(k) is the cluster center of k after token clustering. Note that for any q, we have that q ⊤ c(k) − q ⊤ k ≤ ||q|| ∗ ||k − c(k)||, so if the tokens in the same cluster are compact enough, namely ||k − c(k)|| is very small, then we can efficiently approximate the qk score q ⊤ k by q ⊤ c(k), for all the keys in this cluster. In this situation, for any query q, we do not need to measure and compare the significance of all the
1 X i 2 k − c(kli ) 2 N i=1 l
where kli denotes the i-th key vector of head i from layer l. We define the compactness score of each layer l as Compl = 1/MSEl , where MSEl is the average of {MSEil }. Figure 4 measures the compactness scores of different layers for model Wan-2.1 and Hunyuan. It is shown that, the intra-cluster compactness differs significantly across different layers. Hence, we should assign larger cluster counts for the layers with relatively scattered data distribution, and less cluster counts for the layers with relatively concentrated data distribution. The compressity for different prompts within the same layer is similar (Appendix A.1). Therefore, we only need to adjust on a per-layer basis. 3.2.2. Multi-stage K-means Clustering As analyzed above, different number of clusters are required across layers to ensure the compactness of intracluster data distribution. However, it is difficult to determine the suitable number of clusters in advance. In this subsection, we adopt a multi-stage K-means clustering method to finish this task. Specifically, as shown in Figure 5, for each layer, at the initial stage, we cluster the tokens with a moderate number of clusters. After clustering, we select the outlier tokens which is relatively far from their cluster centers (beyond a predefined threshold). Then we re-cluster these outlier tokens with a few number of new clusters. We repeat such a strategy until each token is assigned to a compact cluster. In particular, if the number of clusters exceeds a preset upper limit, we will stop further clustering and treat this layer as a hard-to-compress layer. For such layers, we apply the vanilla full attention. The detailed process is shown in Algorithm 1. Efficient initialization. DiT involves multiple denoising steps during image/video generation. We observe that for each layer the token distributions change gradually across consecutive denoising steps, i.e., adjacent steps exhibit very similar distributions. Figure 6 visualizes token distributions
Iteration 1 (cluster_num=3→7)
Final (cluster_num=7 adaptive)
Iteration 2 (cluster_num=7)
Figure 5. Visualization of Multi-stage K-means clustering. Red crosses represent tokens, black crosses denote cluster centers, and dashed circles indicate cluster boundaries. In each iteration, samples are progressively clustered into finer groups. The number of clusters (cluster num) increases adaptively as unassigned samples are reclustered. Layer 0 (Red) & Layer 24 (Blue) - Steps 0, 20, 40
Algorithm 1 MultiStageClustering
across denoising steps (Layer 0 and Layer 24) using PCA. Therefore, the cluster centers computed at step t can approximately represent the token distribution of step t+1. Based on this observation, we can further simplify the whole process of DiT denoising. We only need to apply the above-mentioned multi-stage kmeans clustering at the first denoising step to determine the number of clusters for each layer. In the later denoising steps, the number of clusters is fixed, the specific cluster centers will be updated. In partic-
Dimension2
Layer 0 - Step 0
100
Layer 0 - Step 20
Layer 0 - Step 40
100
50
50
50
0
0
0
-50
-50
-50
-100 -100 -50 0 50 100 Layer 24 - Step 0 30
-100 -100 -50 0 50 100 Layer 24 - Step 20 30
-100 -100 -50 0 50 100 Layer 24 - Step 40 30
20
20
20
10
10
10
0
0
0
-10
-10
-10
-20
-20
-20
-30
-30
Dimension2
Require: Key vectors set K = {k1 , . . . , kn }, distance threshold τ , max cluster count Nmax Ensure: Cluster assignments A, Cluster centers C, use full attention flag F 1: C ← ∅ 2: U ← D 3: t ← 0, F ← False 4: while U ̸= ∅ do 5: if |C| ≥ Nmax then 6: F ← True 7: break 8: end if 9: Choose a moderate cluster number mt 10: Index, Ct ← KMeans(U, mt ) 11: for each ki ∈ K do 12: Compute the distance di ← ∥ki − c(ki )∥2 13: if di < τ then 14: Remove ki from U: U ← U \ {ki } 15: end if 16: end for 17: C ← C ∪ Ct 18: t←t+1 19: end while 20: Obtaining the cluster assignment set A by assigning each ki in K to the closest cluster center in C. 21: return A, C, F
100
-30 -20 -10 0 10 20 30 Dimension1
-30 -30 -20 -10 0 10 20 30 Dimension1
-30 -20 -10 0 10 20 30 Dimension1
Figure 6. Token distribution consistency of Wan-2.1 across timesteps for Layer 0 (top, red) and Layer 24 (bottom, blue). Each column corresponds to a different denoising step (0, 20, 40).
ular, we will use the cluster centers of previous denoising step as the initialization of the current step, which can accelerate the clustering procedure in practice. 3.2.3. Efficiently Identify Critical Clusters To efficiently identify critical tokens after clustering, we take inspiration from the existing method Quest [32], which is able to select the most potentially critical KV cache pages for the current query during LLM decoding. Specifically, given a query q, and a page of key vectors K := [k1 , k2 , ..., kn ], Quest identifies critical attention candidates by estimating the upper bound of attention weights across all key vectors, which is expressed as Quest(q, K) =
D X
max(q d ∗ max(K d ), q d ∗ min(K d )),
d=1
(1) where D is the total dimension of q and k, and K d denote the vector composed of the d-th dimension of K. One can refer to [32] for more details about Quest.
Algorithm 2 TensorQuest
3.3. Overall Process
Require: Query tensor Q ∈ RB×H×Lq ×D Key tensor K ∈ RB×H×Lk ×D Ensure: Quest score S 1: Q+ ← max(Q, 0) 2: Q− ← min(Q, 0) 3: K+ ← max(K, 0) 4: K− ← min(K, 0) 5: S ← matmul(Q+ , (K+ )⊤ ) + matmul(Q− , (K− )⊤ ) // shape: [B, H, Lq , Lk ] 6: return S
Finally, we describe the whole procedure of AdaCluster in the Algorithm 3. We cluster normalized queries using standard KMeans, while the keys are processed with our MultiStageClustering to determine whether a given layer should be compressed. For layers identified as compressible, we employ TensorQuest to rapidly select critical tokens for attention computation.
Inspired by Quest, we aim to adopt a similar method to measure the significance of a cluster. However, according to (1) we can see that the Quest score is purely computed in the GPU CUDA Core, which is inefficient. Compared with LLM decoding, the attention in diffusion denoising is more computationally intensive, so the existing Quest method cannot be directly applied, due to unacceptable time delay. In this section, we propose TensorQuest, which is equivalent to vanilla Quest method, but enables the main computation to be executed in the powerful tensor core, thereby significantly enhancing efficiency. To achieve this goal, we first extract the positive and negative parts of query and key separately, as follows, q+
=
max(q, 0),
q− = min(q, 0),
+
=
max(k, 0),
k− = min(k, 0).
k
Then the Quest score can be computed by Quest(q, K) = matmul(q + , k + ) + matmul(q − , k − ). (2) It can be verified that equation (2) is equivalent to (1). Except the light-weight positive and negative parts extraction, the main computation is finished in the CUDA Core. We illustrate the whole process in Algorithm 2. Algorithm 3 AdaCluster Require: Query Q, Key K, Value V , distance threshold τ , Top-K parameter topk, max cluster count Nmax Ensure: AttnOut 1: Aq , Cq ← KMeans(Normalize(Q)) 2: Ak , Ck , F ← MultiStageClustering(K, τ, Nmax ) 3: if F is True then 4: AttnOut ← Attention(Q, K, V ) 5: else 6: S ← TensorQuest(Q, K) 7: Index ← argtopk(S, topk) 8: K ∗ ← K[Index], V ∗ ← V [Index] 9: AttnOut ← Attention(Q, K ∗ , V ∗ ) 10: end if 11: return AttnOut
4. Experiment 4.1. Setup Models. We evaluate the performance of AdaCluster on several Diffusion Transformer (DiT) based video generation models, including HunyuanVideo [16, 33], Wan-2.1-T2V1.3B [30, 39], and CogVideoX-2B [35, 52]. We generate videos with multiple resolutions (e.g. 480p and 720p) via the Text-to-Video (T2V) method. Datasets and Metrics. For the generation task, we use the prompt dataset from PenguinVideoBenchmark [34]. To evaluate the performance of the AdaCluster, we measure the generated videos from two dimensions: similarity to original videos and overall video quality, with specific metrics selected as follows. For the similarity assessment between generated videos and reference videos, we choose three metrics: Peak Signal-to-Noise Ratio(PSNR), Learned Perceptual Image Patch Similarity(LPIPS) and Structural Similarity Index Measure(SSIM). For the overall video quality evaluation, we use VBench [14] to measure four aspects of generated videos:imaging quality, background consistency, subject consistency and aesthetic quality. Baselines. We compare the performance of AdaCluster with baseline models adopting FlashAttention [7]. Additionally, we select SpargeAttn [56] and SVG2 [51] as baselines, which represent the state-of-the-art sparse attention algorithms with dynamic input-driven patterns. We use the default hyperparameter settings reported in their original papers and code repositories. Platform and configurations. Main experiments are conducted on one NVIDIA A40 GPU with 48GB memory. Experiments on H100 are in Apendix A.6. Our implementation builds upon Transformer Diffuser [9] and integrates custom kernels from FlashInfer [54]. For skipped layers, we use FlashAttention [7] as the full attention, while for layers with clustering, we implement a Triton-based kernel to improve computational efficiency. To balance speed and accuracy, we set kmax such that the top 15% of layers use full attention—these layers are relatively small, making the additional latency acceptable. The threshold τ is defined as 1.5× of the average token-to-cluster-center distance computed in the first inference step. We fix the number of clusters for q to 65, while the number of clusters for k is dynamically adjusted according to our algorithm. We report
Table 1. Similarity, quality and efficiency benchmarking results of AdaCluster and baselines.
Wan-2.1-1.3B
HunyuanVideo
Config 720×480 SpargeAttn AdaCluster 832×480 SpargeAttn SVG2 AdaCluster 1280×720 SpargeAttn SVG2 AdaCluster
PSNR↑ SSIM↑ LPIPS↓ ImgeQual↑ BgConsis↑ SubConsis↑ TmpFlick↑ Speedup↑ 61.15% 96.06% 92.71% 97.31% 1× 28.189 0.517 0.618 55.77% 95.56% 94.35% 98.14% 1.23× 30.989 0.767 0.231 56.76% 93.95% 89.26% 95.18% 1.67× 67.53% 96.72% 95.08% 98.04% 1× 28.292 0.437 0.599 64.59% 97.24% 96.29% 98.36% 1.81× 28.230 0.358 0.679 66.43% 96.05% 94.36% 97.96% 1.61× 29.083 0.571 0.393 66.45% 96.35% 94.58% 97.84% 1.85× 62.64% 95.84% 93.05% 98.64% 1× 28.155 0.490 0.596 61.76% 95.89% 92.92% 98.69% 1.33× 29.319 0.794 0.308 65.42% 96.58% 92.14% 98.73% 1.57× 30.580 0.835 0.203 65.11% 95.58% 92.56% 98.85% 1.68×
the detailed number of clusters for k under different videos configuration in the Appendix. Hyperparameter sensitivity analysis is in Appendix A.5. Due to memory constraints of A40, we use 30 inference steps for Hunyuan, while setting 50 steps for the other two models. Based on this hardware setup, we ensure that all baselines use the exact same model. For all subsequent timesteps, the cluster centers from the previous step are reused.
4.2. Quality Evaluation We present all the accuracy evaluation results in Table 1 along two dimensions: similarity to the full-attention model and video quality scores measured by VBench. In terms of similarity, our results indicate that AdaCluster consistently outperforms the selected baselines across all three models. This demonstrates that AdaCluster effectively leverages diverse sparsity patterns across different model layers, thereby preserving video quality to a greater extent relative to the original full-attention approach. Regarding video quality evaluation, we observe that AdaCluster and the two baseline methods generally achieve performance comparable to that of the original model. However, two exceptions are noted: first, SpargeAttn occasionally achieves an unexpectedly high score—sometimes even surpassing the original model—which may be attributed to its tendency to generate videos that align more closely with VBench’s evaluation standards, thereby making such scores less meaningful in comparative analysis. Second, accuracy is stable on Wan-2.1 and Hunyuan with little variation across methods. On CogVideoX, however, both SpargeAttn and AdaCluster show a significant drop in image quality; SpargeAttn’s sub-consistency is much higher than the original, while ours is much lower. We attribute this to the model itself—CogVideoX exhibits the poorest video quality among the three (see Appendix). It is worth noting that video quality assessment remains an active area of research. Our objective here is primarily