arXiv CSopen access

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

FlashAttention improves efficiency through tiling, but its online softmax still relies on floatingpoint arithmetic for numerical stability, making full quantization difficult. We identify three main obstacles to integer-only FlashAttention: (1) scale explosion during tile-wise accumulation, (2) inefficient shift-based exponential operations on GPUs, and (3) quantization granularity constraints requiring uniform scales for integer comparison. To address these challenges, we propose QFlash, an endto-end integer FlashAttention design that performs softmax entirely in the integer domain and runs as a single Triton kernel. On seven attention workloads from ViT, DeiT, and Swin models, QFlash achieves up to 6.73× speedup over I-ViT and up to 8.69× speedup on Swin, while reducing energy consumption by 18.8% compared to FP16 FlashAttention, without sacrificing Top-1 accuracy on ViT/DeiT and remaining competitive on Swin under pertensor quantization. Our code is publicly available at https://github.com/EfficientCompLab/qflash.

Introduction

Transformer self-attention provides strong representational power, but its computation and memory cost grow as O(N 2 ) with sequence length. This growth incurs a memory bottleneck in Vision Transformers (ViTs), where intermediate tensors must be transferred between on-chip and off-chip memory. FlashAttention [Dao et al., 2022] was proposed to reduce this problem by splitting the sequence into tiles. Each tile is computed on-chip, and only the final outputs are written to off-chip memory. However, FlashAttention and later works [Dao et al., 2022; Dao, 2023; Shah et al., 2024] cannot compute the softmax in one step because of the tile-based design. They update the row-wise maximum for each tile, rescale the past results, and then add the new tile results. This process needs numerical stability and therefore depends on floating-point operations. Other works on quantizing attention include I-BERT [Kim et al., 2021], I-ViT [Li and Gu, 2023], QAttn [Kluska et al., ∗

Corresponding author

Q(INT8) K(INT8) V(INT8) MatMul(INT8) INT32 Dequantize FP32 Exp(FP32) FP32 Quantize INT8 MatMul(INT8) INT32 Partial Quantization

#t ile s

Abstract

#t ile s

arXiv:2604.25306v1 [cs.LG] 28 Apr 2026

Sehyeon Oh1,2 , Yongin Kwon3 , Jemin Lee4∗ 1 University of Science and Technology, Daejeon, Republic of Korea 2 Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea 3 Pusan National University, Busan, Republic of Korea 4 Jeonbuk National University, Jeonju-si, Republic of Korea [email protected], [email protected], [email protected]

Q(INT8) K(INT8) V(INT8) MatMul(INT8) INT32 Exp(INT32) INT8 MatMul(INT8) INT32 QFlash

Figure 1: Comparison of partial quantization and the proposed QFlash method.

2024], and INT-FlashAttention [Chen et al., 2024]. Although I-BERT and I-ViT quantize the Softmax operation, they do not consider a tile-based fused attention design like FlashAttention [Dao et al., 2022], leaving memory bottlenecks unresolved. QAttn and INT-FlashAttention quantize only the matrix multiplications, while the softmax still runs in floating point. As a result, there has been no work on a fully integeronly fused attention that reduces memory cost and removes floating-point operations. Through our analysis of integer-only FlashAttention, we identify three key challenges. (C1) Scale Explosion: In tilewise accumulation, approximation errors and scale changes from integer exponential operations propagate across tiles, causing numerical instability. (C2) GPU Inefficiency: The shift-based exponential approximation requires integer division to separate integer and fractional parts, which is significantly slower than multiplication or shift operations on GPUs. (C3) Quantization Granularity: Fused attention requires uniform scales across tiles for direct integer comparison; pertoken quantization introduces scale mismatches that necessitate costly dequantization. In this paper we propose QFlash, an integer-only fused attention on tile-based computation that addresses these challenges. As shown in Figure 1, QFlash keeps the tile-based design but quantizes all operators including the softmax, thus the full kernel runs in the integer domain. First, we imple-

ment online softmax [Milakov and Gimelshein, 2018] with shift-based exponential approximation and row-wise max updates, addressing C1 by carefully managing scale propagation across tiles. Second, we optimize the shift-based exponential computation to minimize GPU inefficiency (C2) through efficient integer arithmetic. Third, we adopt per-tensor quantization granularity to maintain uniform scales across the entire attention computation, enabling direct integer comparison without dequantization (C3). This full kernel fusion reduces off-chip memory use, improves scheduling, and raises GPU utilization. QFlash therefore keeps the benefits of fused attention while providing a fully integer path that improves both latency and energy efficiency. To validate the proposed method, we implement QFlash as an integer-only kernel in Triton and evaluate it on an NVIDIA GeForce RTX 5090 with TVM 0.8, PyTorch 2.7.1 with CUDA 12.8, and Triton 3.3.1. We extract seven representative attention workloads (A1–A7) from ViT, DeiT, and Swin models at 224 × 224 resolution to systematically evaluate latency performance. On DeiT and ViT workloads (A1– A3), QFlash achieves up to 4.54× speedup at batch 1 and 6.73× at batch 8 compared with I-ViT. On Swin workloads (A4–A7), the speedup reaches 7.69× and 8.69× respectively. We also observe that QFlash’s IMMA utilization is 8.0%, lower than FlashAttention-2’s HMMA utilization of 13.08%, but since IMMA provides twice the peak throughput on RTX 5090, QFlash delivers higher absolute performance. For the same workload, QFlash reduces energy consumption by 18.8%, from 929.6µJ with FP16 FlashAttention-2 to 754.6µJ. In terms of accuracy, QFlash improves SQNR by up to 6.7dB over I-ViT. It also maintains Top-1 accuracy close to FP32 on ViT and DeiT models evaluated on ImageNet-1K. These results show that an integer-only design can achieve high speed and efficiency without sacrificing accuracy in real inference.

Background & Related Work

2.1

FlashAttention

Standard self-attention requires computing the similarity between all query key pairs and storing the results in a matrix, leading to memory and computational complexity of O(N 2 ) with respect to the sequence length. As a result, both the computation cost and GPU memory usage increase rapidly when processing long sequences. FlashAttention was proposed to address this issue by leveraging the GPU memory hierarchy to avoid storing the intermediate similarity matrix in memory and instead performing computations in a tile-based manner. The key idea is to use online softmax [Milakov and Gimelshein, 2018], which incrementally updates the softmax normalization as each block is processed. In standard softmax, all similarity values must be collected at once to compute the denominator. In contrast, online softmax processes similarity values block by block, updating the running maximum and exponential sum as new blocks are computed. This approach maintains numerical stability while eliminating the need to store the entire similarity matrix in memory. Thanks to this design, FlashAttention performs computations directly in high-speed

GPU memory and writes results to global memory only when necessary. As a result, the number of memory accesses is significantly reduced, and the overall memory complexity is relaxed from O(N 2 ) to O(N ).

2.2

Vision Transformer Quantization

Prior work on Vision Transformer (ViT) compression has predominantly targeted linear operators, notably matrix multiplications [Li et al., 2023; Yuan et al., 2022; Zhong et al., 2024; Ramachandran et al., 2024]. Subsequent studies extended quantization to non-linear components (Softmax, GELU, and LayerNorm) using polynomial, log-domain, or bit-shift approximations [Kim et al., 2021; Lin et al., 2021; You et al., 2024; Hu et al., 2024; Kim et al., 2024]. These contributions, however, remain at the algorithmic level and do not translate the approximations into kernel-level speedups. I-ViT [Li and Gu, 2023] advances the state of the art by bitshift approximating Softmax and GELU and applying TVMbased intra-operator optimizations. Nevertheless, it forgoes inter-operator fusions akin to FlashAttention, leaving memory traffic largely unoptimized and thereby preserving considerable headroom for further latency reduction in quantized ViTs. There have been some efforts to optimize quantized attention kernels by fusing attention operators [Kluska et al., 2024; Chen et al., 2024]. These approaches reduce computation and memory usage by quantizing the key matrix multiplications in attention. However, the core operation of attention—Softmax—is still computed in the floating-point domain, which limits their applicability for integer-only inference. More recent works accelerate attention and nonlinear operators through hardware–software co-design. Fused Tensor Core [Jahadi et al., 2025] fuses matrix multiplication and Softmax on GPUs by offloading max and sum operations to Tensor Cores, but still relies on floating-point exponentiation. PICACHU [Qin et al., 2025] and QUARK [Zhao et al., 2025] accelerate Softmax, GELU, and LayerNorm using dedicated hardware or CGRA-based designs. While effective, these approaches either keep Softmax in the floating-point domain or depend on specialized hardware support, limiting their generality for software-only, integer-only attention on commodity GPUs. To address these limitations, this work proposes a fully integer-only FlashAttention kernel that quantizes all attention operations, including Softmax, using a software-only design on commodity GPUs.

2.3

Advantages of Integer-only Quantization

Peak performance of CUDA cores and Tensor Cores. For CUDA cores, the latency of integer and floating-point operations is almost the same, so full quantization is not very useful [Arafa et al., 2019]. On the RTX 5090, however, measurements show that Tensor Cores can run 174K operators per cycle with HMMA (FP16) and 348K with IMMA (INT8), which is two times higher1 . Applying quantization to all op1 Calculated based on 170 SMs configuration of RTX 5090 architecture.

erators therefore increases IMMA use and reduces data communication, giving a clear benefit. Performance per watt of Tensor Cores. Half-precision matrix multiply and accumulate (HMMA) uses less energy per operation, but the instruction overhead is still high, which reduces the overall performance per watt. Integer matrix multiply (IMMA) may consume a bit more energy per operation, but the instruction overhead is lower, so it reaches higher compute density in real workloads [Dally, 2023]. From this view, quantizing the model to use IMMA improves both energy efficiency and performance density. In this work, we show that using IMMA with quantization improves not only latency and throughput but also energy efficiency. Use of integer-only accelerators. In mixed-precision designs, matrix multiplications are quantized, but key operations such as softmax still depend on floating-point units. This causes extra communication cost between integer and floating-point hardware, which limits the acceleration and adds inefficiency from floating-point units. Low-cost integeronly accelerators cannot support such mixed-precision needs, so extra floating-point units must be added in a heterogeneous chip. This increases chip area and power budget, which raises the cost of model deployment.

Challenges of Fully Integer FlashAttention

3.1

C1: Scale Explosion in Tile-wise Integer Accumulation

The first challenge is how to manage scaling in a tile-wise accumulation setting to avoid overflow and severe accuracy degradation. FlashAttention computes attention outputs tile by tile and incrementally accumulates each partial result into the running output. Since the accumulated result from previous tiles is reused as input for subsequent tiles, numerical stability becomes critical once quantization is applied. In integer-based FlashAttention, the exponential function is replaced with an integer approximation. The resulting approximation errors and scale variations are repeatedly injected into the accumulation process across tiles, which can lead to rapid scale growth and instability if not carefully controlled. A detailed comparison of scale management strategies is provided in Appendix B.1.

3.2

C2: Inefficiency of Shift-based Exp/Softmax on GPU

The integer softmax proposed in I-ViT [Li and Gu, 2023] typically transforms the exponential function into an exp 2(·) form and implements it using shift operations combined with linear approximations. While this approach is theoretically efficient, it introduces practical inefficiencies on GPUs when applied to integer inputs. Specifically, computing exp 2(x) requires decomposing the input into integer and fractional parts. For quantized integer inputs, this decomposition involves integer division. On GPUs, integer division has significantly higher latency than multiplication or shift operations, making it a major performance bottleneck for softmax computation. Implementation details of our GPU-friendly solution are discussed in Appendix B.2.

3.3

C3: Quantization Granularity in Integer Fused Attention

Attention inputs have a multi-dimensional structure, and quantization granularity can be applied at the tensor, head, or token level. The choice of granularity directly affects both accuracy and performance. In FlashAttention, stable exponential computation requires subtracting the maximum value from QK ⊤ scores, with the maximum being updated at every tile. When this process is performed in an integer-only setting, updating the maximum requires either all tiles to be computed under the same scale or the values to be dequantized into floating point for comparison. However, dequantization introduces additional computation and memory accesses, which significantly undermines the performance benefits of fused attention kernels. For this reason, we aim to maintain a shared scale that enables direct comparison of integer values without dequantization. From this perspective, per-token quantization generates different scales for each token, making it difficult to apply in integer fused attention during max updates and accumulation. As a result, per-tensor and per-head quantization are the practical granularity choices for integer-based FlashAttention. A visual illustration of granularity constraints is presented in Appendix B.3.

Integer-Only Fused Attention Method

In this section, we describe in detail how the proposed QFlash algorithm operates. We first define the notation used in the description. Let b denote the bit-width, and let Iint(b) denote the set of signed b-bit integers as defined in Equation (1). Let sX be the scale factor, and let X̂ denote the integer quantization of X. Iint(b) := {z ∈ Z | −2b−1 ≤ z ≤ 2b−1 − 1}.

(1)

m×n

For a real matrix X ∈ R , the scale factor sX and the quantized matrix X̂ are defined as in Equation (2). sX =

∥X∥∞ ∈ R, 2b−1 − 1

X̂ =

X sX

∈ Im×n int(b) .

(2)

QFlash maintains the tile-based fused attention computation of FlashAttention [Dao et al., 2022] while quantizing all operators, including the softmax, to achieve both data size reduction and compute efficiency. The detailed procedure follows Figure 2 and Algorithm 1. One execution of the outer loop in Algorithm 1 computes a single output tensor Ôi . As shown in Figure 2, the process consists of the following eleven steps: 1 Compute Query–Key Score → 2 Compute Row-Max → 3 Update Max → 4 Compute Score Intensity → 5 Compute Max-Offset Factor → 6 Requantization → 7 Compute Row-Sum → 8 Compute Value MatMul → 9 Accumulate Denominator → 10 Accumulate Numerator → 11 Normalize.

4.1

MatMul

QFlash performs integer matrix multiplications for 1 Compute Query–Key MatMul and 8 Compute Value MatMul.

𝐵(

Inner Loop (𝟏 ≤ 𝒊 ≤ 𝑻𝒄 )

, 𝑙$

𝑚 "

1 (! Q

(#) S&!

Compute Query-Key MatMul

(! V

/ (&) 𝑙$!

Normalize

𝐵(

𝐵'

INT8

INT32

RowSum

(#) S"!

Compute Row-Max INT32

Off-Chip Mem On-Chip Mem Outer Loop (𝟏 ≤ 𝒊 ≤ 𝑻𝒓 )

max

(&!'()) P"!

𝑚 '!

Compute Score Intensity

INT32

(#*+)

𝑚 '!

ShiftExp2 (#)

(#)

𝑚 '!

α '!

Compute Max-Offset Factor

$ (&!'()) O !

$ (#) V

INT32

) ≫𝑛

(#)

(#*+) 𝑙,!

(&!'()) 𝑙,!

10 (

INT32

MatMul

Compute Value MatMul

(

ShiftExp2 (#)

(#) P"!

INT32

INT8

(#)

Update Max

Compute Row-Sum

𝑚 '!

(&!'()) 𝑙,!

INT32

(#*+)

(&!'())

𝑚 '!

(#) P"!

(&!'())

𝑚 '!

(#) S"!

(! O

Requantization

INT32

(#) P"!

(&!'()) P"!

(#) S"!

RowMax

( (&) O !

INT8

Requantize

$#% K

Inner Loop (𝟏 ≤ 𝒊 ≤ 𝑻𝒄 )

Outer Loop (𝟏 ≤ 𝒊 ≤ 𝑻𝒓 )

𝐵'

, 2 … 10

INT32

MatMul

$! Q

(#% K

INT32

INT8

(#) 𝑙,!

𝑀,

α '!

Accumulate Denominator

$ (&!'()) O !

+ $ (#*+) O !

(#)

α '!

) ≫ 𝑛

INT32

$ (#) O !

𝑀,

Accumulate Numerator

Figure 2: Proposed QFlash forward pass (Algorithm 1), showing the integer-based attention pipeline with quantization, shift-exp softmax approximation, and blockwise accumulation.

These operations can be expressed as the dot product of two r ×d c quantized matrices Â ∈ IB and B̂ ∈ Id×B int8 int8 . Here, Br denotes the size of the query block, Bc denotes the size of the key/value block, and d represents the inner dimension of the multiplication. The inputs are given in int8 format, and the results are accumulated in int32 format. Equation (3) defines the scaling relationship corresponding to this integer matrix multiplication. r ×Bc Ĉ = Â · B̂ ∈ IB , sC = sA · sB ∈ R. (3) int32 This operation can be highly optimized on GPUs and dedicated accelerators, replacing floating-point multiplications to significantly reduce both computational cost and memory access overhead.

4.2

Compute Row-Max and Update Max

Br ×Bc For a matrix X̂ ∈ Iint32 , the 2 Compute Row-Max step calculates the maximum value of each row to obtain a vecr tor m̂ ∈ IB int32 . Subsequently, in the 3 Update Max step, these values are compared with those from the previous tile to update the row-wise maxima progressively.

m̂i = max X̂[i, j], 0≤j<Bc

r m̂ = m̂1 , . . . , m̂Br ∈ IB int32 . (4)

Equation (4) defines the extraction of the maximum value from each row, which serves as the reference point for the subsequent softmax computation. By subtracting the rowwise maximum from all elements, potential overflow in the exponential calculation is prevented. In addition, Update Max incorporates the running maximum across tiles during block-wise operations, ensuring stable numerical computation throughout the entire sequence. This tile-wise max update requires consistent scaling for direct integer comparison, which motivates Challenge C3.

4.3

Compute Score Intensity and Max-Offset Factor

In QFlash, the exponential operation is used in step 4 Compute Score Intensity and step 5 Compute Max-Offset Factor. The Score Intensity step computes the exponential term of softmax in integer form to measure attention strength. The Max-Offset Factor step uses the Row-Max value to keep the exponential operation stable. We adapt the ShiftExp algorithm from I-ViT to approximate the exponential function of softmax with integer arithmetic. The key of softmax is the ex computation. By multiplying with the constant log2 (e), we can rewrite it as a power of two. The input matrix is X̂ ∈ Im×n int32 with scale factor sX ∈ R. After Row-Max is applied, X̂ always has negative values; therefore, the exponential is applied only to negative inputs. This process is given in Equation (5).

esX ·X̂ = 2(sX ·log2 e)·X̂ = 2s̃X ·X̂ ,

(5) Here, s̃X = sX · log2 e. Equation (6) rewrites the input X̂ as an integer part Q ∈ Z≤0 and a fractional part R. This decomposition typically requires integer division on GPUs, which motivates Challenge C2.

2s̃X ·X̂ = 2(s̃X ·R)+Q

(6) Since the fractional part R lies in (−1, 0], it can be replaced with a simple linear approximation as in Equation (7).

2(s̃X ·R) ≈ s̃X2·R + 1 = s̃X ·

R 2 +

1 s̃X

(7)

j m Note that s̃1X is a constant determined only by s̃X . Thus, it is precomputed outside the kernel, and the kernel performs

Algorithm 1 QFlash Algorithm ×d −1/2 1: Require: Q̂, K̂, V̂ ∈ IN · log2 e, block sizes Bc , Br , multiplier int8 , scale factors sQ , sK , sV ∈ R and s = sQ · sK · d

Mr , shift bits r.

2: Divide Q̂ into Tr = ⌈N/Br ⌉ blocks Q̂1 , . . . , Q̂Tr of size Br × d, and divide K̂, V̂ into Tc = ⌈N/Bc ⌉ blocks K̂1 , . . . , K̂Tc

and V̂1 , . . . , V̂Tc of size Bc × d each. ×d 3: Divide output Ô ∈ IN int8 into Tr blocks Ô1 , . . . , ÔTr of size Br × d each. 4: for i = 1 to Tr do 5: Load Q̂i from Q̂[(i − 1)Br : iBr , :] from off-chip to SRAM

6: 7: 8: 9: 10: 11: 12: 13: 14:

(0)

On chip, initialize Ôi = 0 ∈ ZBr ×d , ˆli = 0 ∈ ZBr , m̂i = −221 ∈ ZBr for j = 1 to Tc do ⊤ Load K̂⊤ j from K̂ [:, (j − 1)Bc : jBc ] and V̂j from V̂[(j − 1)Bc : jBc , :] from off-chip to SRAM (j)

← Q̂i K̂⊤ j (j) (j−1) (j) On chip, compute m̂i ← max(m̂i , rowmax(Ŝi )) (j) (j−1) (j) On chip, compute α̂i ← ShiftExp2(m̂i − m̂i , s) (tiled) (j) (j) On chip, compute P̂i ← ShiftExp2(Ŝi − m̂i , s) (j) (tiled) On chip, compute P̂i ← Requantization(P̂i , M, b) (j) (j−1) (j) (j) ˆ ˆ On chip, compute li ← ScaleRelease(li , α̂i , s) + rowsum(P̂i ) (j) (j−1) (j) (j) On chip, compute Ôi ← ScaleRelease(Ôi , α̂i , s) + P̂i V̂j

On chip, compute Ŝi

▷ 1 ▷ 2, 3 ▷ 4 ▷ 5 ▷ 6 ▷ 7, 9 ▷ 8 , 10

15: 16: end for 17: On chip, compute Ôi ← ⌊ÔTi c /ˆliTc ⌋ 18: Write Ôi to off-chip as the i-th block of Ô 19: end for ×d 20: Return the output Ô ∈ IN int8 and the output scale factor sO = sV .

only element-wise integer operations. Finally, Equation (8) shows the combination of the integer part and the approximated fractional part to compute the exponential.

esX ·X̂ ≈ s̃X ·

R 2 +

1 s̃X

· 2Q

(8)

This formulation enables an efficient integer-only approximation of the exponential operation using simple arithmetic and bit-shift operations.

4.4

▷ 11

j m r We precompute n, r, and ssX · 2 outside the kernel, Y since they depend only on the scale factors and bit-width and are independent of the input X̂.

4.5

Compute Row-Sum

For softmax normalization, the denominator requires the sum r ×Bc of each row. Thus, for the matrix X̂ ∈ IB , the 7 Comint8 pute Row-Sum step calculates the element-wise sum of each row.

Requantization

The result of the matrix multiplication is stored as X̂ ∈ r ×Bc IB . To be used in subsequent operations, this value must int32 r ×Bc be converted back into the range of Ŷ ∈ IB through the int8 6 Requantization step. This is achieved by applying a scaling transformation that reduces the integer range. First, we define a fixed-point multiplier as in Equation (9). Here, the scale ratio is converted into a binary logarithm to compute n, and the shift size r is set by considering the bit-width b. sX n = log2 , sY

r = b − n.

(9)

Based on this, the requantization is performed as shown in Equation (10).

sX r Mr = ·2 , sY

Ŷ = X̂ · Mr ≫ r.

(10)

ŝi =

Bc X

X̂[i, j],

r ŝ = ŝ1 , . . . , ŝBr ∈ IB int8 .

(11)

j=1

Equation (11) describes the computation of the row-wise sums, where ŝi represents the sum of the elements in the i-th row. These values serve as the key denominator terms in the softmax normalization.

4.6

Accumulate Numerator and Denominator

For softmax normalization, steps 9 Accumulate Denominator and 10 Accumulate Numerator must be performed. If these accumulations are carried out directly in the quantized domain, overflow may occur. A common approach dequantizes values before accumulation, but this introduces floating-point operations and weakens integer-only benefits. Related to Challenge C1, we approximate the inverse scale as an integer to perform division, enabling stable numerator/denominator accumulation and integer-only softmax normalization without overflow. We refer to this operation as

Source

#Win

Shape (W ×H×N ×D)

A1 A2 A3

ViT/DeiT-Tiny ViT/DeiT-Small ViT/DeiT-Base

✗ ✗ ✗

3 6 12

197 197 197

64 64 64

(3 × 197 × 64) (6 × 197 × 64) (12 × 197 × 64)

A4 A5 A6 A7

Swin-T/S Stage-1 Swin-T/S Stage-2 Swin-T/S Stage-3 Swin-T/S Stage-4

64 16 4 1

3 6 12 24

49 49 49 49

32 32 32 32

(64 × 3 × 49 × 32) (16 × 6 × 49 × 32) (4 × 12 × 49 × 32) (1 × 24 × 49 × 32)

ScaleRelease, whose definition and analysis are provided in Appendix B.1.

4.7

INT-Flash-Half [Mixed] I-ViT TVM [INT8]

Inference Latency

Latency comparison. Figure 3 compares the latency of different attention kernels across the seven attention workloads listed in Table 1. Since I-ViT is a representative integer-only attention kernel, we measure speedup primarily against I-ViT to assess the effectiveness of our fully integer QFlash design. In the batch-1 case, QFlash achieves up to 4.54× and 7.69× speedup over I-ViT on ViT/DeiT and Swin workloads, respectively. In the batch-8 case, QFlash is up to 6.73× and 8.69× faster than I-ViT on ViT/DeiT and Swin workloads, respectively. Despite the overhead of fully integer execution, QFlash consistently outperforms the integer-only baseline and remains generally faster or comparable to FP16 and mixed-precision kernels across workloads. Tensor Core utilization. To analyze the latency improvement of our kernel code, we used NVIDIA Nsight Compute [Leinhauser et al., 2021] to profile Tensor Core utilization. We report utilization as a ratio to the peak throughput,

QAttn [Mixed] Torch [FP32]

(a)

115 110 20 10 0

A4 (b)

Figure 3: Latency comparison of different attention kernels across the seven attention workloads (A1–A7) listed in Table 1. Results are reported for (a) batch size 1 and (b) batch size 8. Table 2: Tensor Core utilization (Peak %) for different attention implementations on workloads A2 and A7 with batch size 8. A2

Method

Evaluation

All experiments were run on an NVIDIA GeForce RTX 5090 GPU with TVM 0.8, PyTorch 2.7.1 (CUDA 12.8), and Triton 3.3.1. We evaluated latency, energy, and accuracy at the operator level and then tested Top-1 accuracy against IViT, I-BERT, FQ-ViT, FlashAttention-2, INT-FlashAttention (Full/Half), and QAttn. We extract seven unique attention workloads from different ViT/DeiT/Swin model variants, as summarized in Table 1. In addition, we evaluate both batch size 1 and batch size 8 to cover real-time inference and throughput-oriented inference settings, respectively.

5.1

15 10 5 0

QFlash [INT8] FlashAttention-2 [FP16] INT-Flash-Full [Mixed]

Normalization

After the inner loop, step 11 performs normalization. Both the numerator and denominator are accumulated as integers, and their ratio corresponds to softmax normalization without explicitly materializing the full score matrix. Since the scale factors cancel out during normalization, dequantization is implicitly handled. Importantly, we implement this division entirely in integer arithmetic, eliminating floating-point operations while producing int8 outputs directly usable for downstream computation.

Latency (ms)

Workload

Latency (us)

Table 1: Unique attention workload configurations used in ViT/DeiT and Swin inference at 224 × 224. #W in denotes the number of windows (only applicable to Swin), H the number of heads, N the context length, and D the head dimension.

Flash-2 QAttn INT-Flash-Half INT-Flash-Full QFlash (Ours)

IMMA

HMMA

IMMA

HMMA

– 3.75% 3.38% 7.18% 8.00%

13.08% 7.51% 6.75% – –

– 1.25% 1.12% 2.13% 2.55%

4.69% 2.51% 2.25% – –

using two workloads (A2 and A7) with batch size 8. The results are summarized in Table 2. The key observation is that on the RTX 5090, Tensor Cores achieve 174K operators per cycle with HMMA (FP16) and 348K with IMMA (INT8), meaning the peak throughput of IMMA is exactly twice that of HMMA. Therefore, even though the IMMA utilization of QFlash is only 8.0% compared to 13.08% HMMA utilization in FlashAttention-2, the higher peak throughput of IMMA means that QFlash still delivers greater overall performance. The same conclusion holds for A7.

5.2

Energy Consumption

We compared the energy consumption of five kernels on workload A2 with batch size 8. Power consumption was recorded using nvidia-smi [Yu et al., 2023; Bridges et al., 2016], and the number of operations was measured with Nsight Compute. All kernels executed the same total amount of computation. As listed in Table 3, QFlash shows the lowest energy consumption. There are two reasons for this result. First, QFlash achieves shorter execution time for the same workload, which reduces the integrated energy. Second, QFlash consistently uses the IMMA instruction for Tensor Cores, which provides higher energy efficiency per operation compared to HMMA [Dally, 2023; Horowitz, 2014]. The combination of these two effects makes

Table 3: Energy consumption (µJ) on workload A2 (Batch = 8). Workload

Flash

INT-Flash (Full)

INT-Flash (Half)

QAttn

QFlash (Ours)

A2 (B=8)

929.6

840.0

920.5

795.2

754.6

Table 5: Accuracy and efficiency comparison across quantization granularities. Symbols indicate granularity: △ per-token, ◦ perhead, ▽ per-tensor. For fairness, we use each prior work’s original algorithm and kernel implementation, which are tailored to their target granularity.

Table 4: Quantitative evaluation of attention implementations in terms of SQNR and MSE.

Model (FP32 Acc)

Method

BOPs (G)

Int-Only

Acc

ViT-S (FP32: 81.38)