BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

arXiv:2611.00547v1 [cs.AR] 31 Oct 2026

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs Hangyu Zheng

Zhihao Shu

Kunxiong Zhu

hyzheng@uga.edu University of Georgia Athens, GA, USA

Zhihao.Shu@uga.edu University of Georgia Athens, GA, USA

Kunxiong.Zhu@uga.edu University of Georgia Athens, GA, USA

Minghai Qin

Xiaolong Ma

Gagan Agrawal

qinminghai@gmail.com Western Digital Research San Jose, CA, USA

xiaolongma@arizona.edu University of Arizona Tucson, AZ, USA

gagrawal@uga.edu University of Georgia Athens, GA, USA

Wei Niu wniu@uga.edu University of Georgia Athens, GA, USA

Abstract The rapid scaling of large language models (LLMs) has unlocked remarkable capabilities across various domains. However, deploying LLMs on mobile devices remains challenging because of their high computational and memory requirements. Low-bit quantization (e.g., 4-bit weights, 8-bit activations) can reduce model size while preserving accuracy, but its practical efficacy on mobile GPUs has been limited so far. In this paper, we identify that the performance gap stems primarily from frequent data type conversions between quantized and floating-point values, and dynamic and inefficient memory access patterns caused by the growing Key-Value (KV) cache during autoregressive decoding. We introduce BitLLM, a mobile inference engine that addresses type conversion overhead and suboptimal memory accesses in quantized LLMs. Our approach features an adaptive layout and packing strategy for the KV cache that ensures efficient memory access throughout LLM decoding. This is coupled with an integer-native execution pipeline that defers and overlaps type conversion outside the compute hot path, keeping the critical MatMul stages in the integer domain. Implemented on off-the-shelf mobile GPUs, BitLLM outperforms leading engines, i.e., LLAMA.CPP, MNN-LLM, LiteRT-LM, and MLCLLM, achieving average speedups of 1.78×, 1.73×, 1.58×, and 1.47×, respectively, across various models and quantization levels. Under long-context generation, BitLLM reduces average energy consumption (per token) by up to 88.9%.

This work is licensed under a Creative Commons Attribution 4.0 International License. ASPLOS ’27, Heraklion, Greece. © 2027 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-3012-2/2027/04 https://doi.org/10.1145/3845814.3850028

CCS Concepts: • Human-centered computing → Ubiquitous and mobile devices; • Computing methodologies → Neural networks. Keywords: Large language models, Quantization, Mobile computing ACM Reference Format: Hangyu Zheng, Zhihao Shu, Kunxiong Zhu, Minghai Qin, Xiaolong Ma, Gagan Agrawal, and Wei Niu. 2027. BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs. In Proceedings of the 32nd ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS ’27), April 11–15, 2027, Heraklion, Greece. ACM, New York, NY, USA, 16 pages. https://doi.org/10.1145/3845814.3850028

1

Introduction

Large language models (LLMs) are now central to modern AI, having deeply impacted areas such as natural language processing, multimodal reasoning, and human-computer interaction [26, 36, 37, 66, 69]. This has driven growing demand to run LLMs on mobile devices [43, 55, 62], where on-device inference offers privacy, lower latency, resilience to connectivity issues [35, 46], and lower data-center energy costs by shifting inference away from centralized infrastructure. However, LLMs are inherently resource-intensive, requiring billions of parameters, multi-gigaflop computations per token, and substantial memory bandwidth for expanding Key-Value (KV) caches [79]. These demands strain the limited computational capacity and memory budget of mobile processors [59, 82]. Bridging this gap requires both algorithmic compression and hardware-aware system design. Quantization, converting high-precision floating-point parameters and activations into compact integer formats, has been shown to maintain model accuracy at 4–8-bit precision while significantly reducing memory footprint, bandwidth

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

requirements, and energy consumption per token [20, 21, 27, 38, 79, 81, 89]. On server platforms, quantization delivers substantial speedups by leveraging native low-bit execution paths such as Tensor Cores [39, 64, 71, 75, 86, 90]. In contrast, mobile GPUs often lack native support for low-bit arithmetic. As shown in Table 1, INT8 inference on a Snapdragon 8 Gen 3 GPU is up to 2× slower than FP16, while INT8 on the CPU (with native low-bit instructions) of the same SoC is 2× faster. Without dedicated integer execution instructions, every low-bit operation requires type promotion and conversion: unpacking eight INT4 values from a 32-bit word costs approximately 16 shift and 16 mask operations, exceeding the 8 multiply and 7 add operations of the dot-product itself (more details in Section 3). Beyond these general DNN considerations, LLMs face a second bottleneck on mobile: the dynamically growing KV cache. Mobile GPUs organize memory as 2D textures with RGBA vector lanes [48, 87], and the optimal data layout depends on the current decoding sequence length. Existing frameworks [70, 77, 78] apply a fixed layout throughout decoding, leading to misaligned memory accesses and periodic latency spikes as the KV cache grows.T-MAC [77] and BitNet.cpp [70] target CPU table-lookup computation, which produces uncoalesced accesses on GPUs and cannot exploit the 2D texture memory hierarchy. T-MAN [78] relies on NPU-specific table-lookup units and static data layouts that cannot adapt to the dynamically growing KV cache during decoding. Efficient quantized execution on mobile GPUs thus remains an open problem, particularly when the optimal layout shifts at runtime with changing sequence length. We propose BitLLM, a mobile GPU inference engine that realizes this formulation through runtime layout adaptation, hardware-aligned packing, and conversion-hiding integer execution. We target 4-/8-bit group-wise quantization, which prior work [20, 39] has shown to achieve best balance of efficiency and accuracy, rather than aggressive formats (e.g., 2-bit) [5, 6, 70] that often compromise accuracy. Although layout selection [28], low-bit packing [77], and kernel pipelining [14] have each been studied, prior approaches generally optimize them independently or assume fixed execution configurations [78]. In contrast, we formulate mobile-GPU low-bit decoding as a dynamic, quantization-aware layoutselection problem and use a lightweight Layout Efficiency Proxy (LEP) to guide runtime layout and packing choices as the KV cache grows. Our contributions are as follows: ● An empirical analysis showing that layout mismatch in the growing KV cache and data type conversion overhead are the key bottlenecks causing low-bit quantization to underperform on mobile GPUs. ● A novel LEP-guided framework that combines quantization with a layout-aware runtime, enabling efficient lowbit LLM execution on mobile GPUs. ● A set of novel mechanisms, including adaptive layouts for KV cache, hardware-aligned value packing, and an

Hangyu Zheng et al. Input

Normalization

MatMul

Split V

K

Append V-cache

Append K-cache

Q

MatMul

Softmax

MatMul

Reshape

MatMul

Add

Figure 1. Computational graph of an autoregressive transformer decoder. Each new query (𝑄) attends to all cached keys (K-cache) and values (V-cache), causing the KV cache to grow and dynamically alter tensor shapes. integer-native pipeline that defers type conversion outside the compute hot path. ● Extensive evaluation showing BitLLM outperforms leading frameworks across multiple models and mobile GPUs. BitLLM is evaluated on four LLMs with grouped-query attention (GQA) and traditional multi-head attention (MHA), namely Llama-3.2 [24], Qwen3 [84], Gemma-2 [67], and Phi2 [29], under multiple precisions (FP16, W8A8, W4A8)1 , context lengths, and mobile GPU platforms. BitLLM achieves average speedups of 1.78× over LLAMA.CPP [11], 1.73× over MNN-LLM [76], 1.58× over LiteRT-LM [12, 23], and 1.47× over MLC-LLM [9, 44], while eliminating the latency spikes observed in other frameworks during KV cache growth. Average per-token energy consumption is reduced by 56–76% relative to the state-of-the-art baselines. BitLLM enables stable execution of long-context models on resource-constrained devices where other frameworks exhibit severe degradation.

2

LLM and Quantization Background

2.1 Computation Flow in Transformer Decoder An autoregressive LLM generates tokens sequentially, with each step depending on all previously produced tokens. At step 𝑁 , the model processes the current input embedding while appending new projections to a growing KV cache. This cache helps avoid recomputation but its growing size introduces challenges to relate to total memory and memory access costs/efficiency. Figure 1 illustrates the computation graph of a Transformer decoder layer during autoregressive decoding. Let 𝑏 denote the batch size and 𝑠 the number of tokens processed at the current step (typically 𝑠=1). Let 𝑐 ′ be the updated cache length (𝑐 ′ = 𝑐 prev + 𝑠). For the model, 𝑛 is the number of attention heads, 𝑑 is the number of dimensions per head, and ℎ = 𝑛 ⋅ 𝑑 is the hidden size. The input embedding has shape (︀𝑏, 𝑠, ℎ⌋︀. After normalization and a linear projection, the representation becomes (︀𝑏, 𝑠, 3ℎ⌋︀, which is split into queries (𝑄), 1 FP16: 16-bit floating-point for weight and KV cache. W8A8: 8-bit weights

and 8-bit KV cache. W4A8: 4-bit weights and 8-bit KV cache.

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs -0.9 0

FP

Per-tensor

1.2

INT8 -128 -Z

Per-row

Per-col

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

Per-group

127

...

(a) Quantization scheme. (b) Quantization at different granularity. 16-bit

8-bit 112 9 101 12

112 Promote to 16 bits

1,008

9 101

32-bit Accumulate to 32 bits 2,220

Dequantize

Figure 3. Illustration of 2D texture memory layout.

?

1,212

12

Decoding Latency per Token

𝑄, 𝐾,𝑉 ∈ R(︀𝑏,𝑛,𝑠,𝑑⌋︀ . ′

𝑉cache ∈ R(︀𝑏,𝑛,𝑐 ,𝑑⌋︀,

(2)

The memory layouts for these caches are discussed later; broadly, they are chosen to avoid transpose operations in the subsequent attention computation. The attention mechanism then computes similarity scores between the current queries and the cached keys: ′

𝑆𝑐𝑜𝑟𝑒𝑠 = 𝑄𝐾 ⊺ ∈ R(︀𝑏,𝑛,𝑠,𝑐 ⌋︀ .

(3)

After applying a softmax along the 𝑐 ′ dimension, the result is multiplied by the value cache to obtain the context tensor 𝑆 = Softmax(𝑆𝑐𝑜𝑟𝑒𝑠),

𝑆𝑉 ∈ R(︀𝑏,𝑛,𝑠,𝑑⌋︀ .

(4)

This result is reshaped back to (︀𝑏, 𝑠, ℎ⌋︀, passed through an output projection, and combined with the residual connection to form the block output (︀𝑏, 𝑠, ℎ⌋︀. In the final decoder layer, the output is projected into the vocabulary space to produce logits for next-token prediction. While 𝑠 remains small and fixed in decoding, the cache length 𝑐 ′ grows linearly with the number of generated tokens. This growth causes the attention computations to increase linearly and introduces different shapes of matrix multiplication. Thus, although the KV cache enables efficient reuse of past computations, its expansion also inflates memory consumption and complicates GPU execution optimization strategies such as static tiling and memory coalescing. LLM Quantization

Quantization maps high-precision floating-point values into discrete and compact integer domains, reducing model size and lowering arithmetic cost. A real-valued tensor 𝑥 can be quantized using a uniform affine mapping with floatingpoint scale 𝑠 > 0 and integer zero-point 𝑧 ∈ Z: 𝑞 = clip (round(𝑥⇑𝑠) + 𝑧, 𝑞 min, 𝑞 max ) ,

2048

4096

6144

8192

10240

(1)

The new keys and values are appended to the caches, ′

2.5 2.0 1.5 1.0

Figure 4. Decoding latency and computational workload per token as functions of decode sequence index.

keys (𝐾), and values (𝑉 ), and reshaped into

𝐾cache ∈ R(︀𝑏,𝑛,𝑑,𝑐 ⌋︀,

900 600 300 0

GMACs per Token

GMACs

Figure 2. Overview of quantization techniques. (a) Quantization scheme. (b) Quantization at different granularities. (c) Type conversion pipeline for INT8 computation, involving promotion to INT16/INT32 and dequantization back.

Latency (ms)

(c) Data type promotion in 8-bit computation.

2.2

Logical Data 0 1 2 3 4 5 6 7 8 9 a b c d e f RGBA RGBA 3 7 c d Memory (4) (4) 2 6 8 9 1 a5be f 4 a5bb f 0 94 d 0 91 7 8 c 2 3 Layout-1 Layout-2

𝑥ˆ ≈ 𝑠 ⋅(𝑞 −𝑧), (5)

where 𝑞 is an integer within the target bit-width. As shown in Figure 2(a), this affine mapping projects floating-point values into a bounded integer range (i.e., -128 to 127). To balance accuracy and efficiency, various quantization schemes have been proposed. They also differ in the granularity at which the scale and zero-point are chosen. Figure 2(b) illustrates some of these different approaches. Pertensor quantization applies a single scale across the entire tensor, offering low overhead but suffering from sensitivity to outliers [20]. Per-row/column quantization assigns independent scales per row or column, improving accuracy for tensors with heterogeneous distributions. Per-group quantization generalizes this idea by partitioning tensors into groups of fixed size, balancing error reduction and implementation cost. Typical group sizes are 32 to 128 [38]. 2.3

Constraints of Mobile GPUs

As shown in Figure 3, the memory subsystem of mobile GPUs can be viewed as a 2-dimensional texture space [48]: a 2D grid with dimensions width 𝑊 and height 𝐻 , where each point stores a fixed-width vector – typically four elements, like an RGBA pixel. This structure, optimized for workloads with inherent 2D spatial locality, allows the dedicated cache to efficiently fetch and cache data in 2D blocks – a fundamental divergence from the linear memory space in CPUs. However, the ability to benefit from this feature depends on data layouts that align with the 2D memory space, as misalignment leads to mandatory padding, uncoalesced accesses, and bandwidth under-utilization. The optimal layout is generally dictated by the specific computation being performed. As an illustration of possibilities, Figure 3 also illustrates two distinct data layouts for a tensor of length 16, where Layout-1 uses row-major ordering, and Layout-2 leverages column-major ordering. On server-class platforms, quantization is effectively supported through optimized low-bit instructions such as those

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

Hangyu Zheng et al.

Table 1. Comparison of LLM decoding latency on mobile platforms with FP16 versus low-bit representation. Model

Data Type

Latency per Token CPU GPU

GPU Speedup

TinyLLama-1B [88]

FP16 INT8

61.9ms 31.2ms

56ms 67.8ms

1.11× 0.46×

Qwen3-1.7B [84]

FP16 INT8

78.8ms 45.4ms

73.6ms 130.2ms

1.07× 0.35×

Llama-3.2-3B [24]

FP16 INT8

140.5ms 80.1ms

137.4ms 159.2ms

1.02× 0.50×

seen in AVX-512, VNNI, and CUDA-INT4/INT8 extensions. However, mobile GPUs are primarily designed for FP16/FP32 pipelines. Figure 2(c) shows the sequence of data type conversion required for INT8 operands in the absence of specialized instructions: (i) unpacking, (ii) promotion to INT16 for multiplication, (iii) accumulation into INT32, and (iv) dequantization back to FP32 for subsequent operations. These steps add significant computational cost and register pressure.

interact on the critical path, optimizing one in isolation yields limited gains. More specifically, we first formalize the notion of the Layout Efficiency Proxy (LEP) (Section 4.1), which models the memory and data unpacking costs that dominate the runtime. Grounded in this formalization, we introduce three complementary mechanisms: i) adaptive layouts that evolve dynamically with sequence length (Section 4.2), ii) quantization-aware packing designed to minimize unpacking overhead (Section 4.3), and iii) a type conversion-hiding computation pipeline that preserves these gains through low-bit integer execution (Section 4.4). We use 𝐻 (attention heads), 𝐷 (hidden dimension per head), and 𝑇 (current sequence length) for the transformer dimensions from Section 2.1. 4.1

Optimization Overview

To understand the performance impact of the steps outlined above, we profile the decoding latency of various LLMs on a commercial mobile device (OnePlus 12) with MNN-LLM [76], comparing FP16 and INT8 on both CPU and GPU. The results, summarized in Table 1, first confirm the expected pattern for FP16 execution: GPUs consistently outperform CPUs, leveraging their superior floating-point throughput. However, INT8 quantization reverses this trend. INT8 reliably accelerates inference on the CPU (i.e., as compared to FP16), by leveraging reduced memory bandwidth and efficient integer arithmetic. However, on the mobile GPU, INT8 consistently results in degraded performance, with execution that is even slower than the CPU’s INT8 path. This inversion stems from the overhead of type promotion and bit-level manipulation that low-bit computation requires on mobile GPUs – operations that existing frameworks do not handle efficiently. The dynamically growing KV cache introduces a second bottleneck in the memory subsystem. Figure 4 highlights the impact of KV cache size on latency. The blue curve shows that the arithmetic workload (measured in GMACs per token) increases in a consistent linear fashion with sequence length. In contrast, the red curve reveals that decoding latency per token grows much faster and exhibits frequent stair-step spikes. These spikes occur when changes in KV tensor size and shape cause the existing layout to become suboptimal.

Two cost factors jointly determine runtime performance and guide our optimizations. The first is data access cost. Different attention operators favor different stride-1 axes: 𝑄𝐾 𝑇 benefits from 𝐷-contiguous layouts for stride-1 query access, while 𝑆𝑉 benefits from 𝑇 -contiguous layouts for efficient token traversal. A misaligned layout forces strided fetches and under-utilizes the GPU’s RGBA vector lanes. The data access cost, in summary, depends on the layout configuration and further changes with sequence length. The second factor is unpacking overhead. As shown in Figure 5, an FP32 8-element dot product uses 8 FMA instructions, but low-bit values must be unpacked into byte- or word-aligned operands before the mobile GPU can perform arithmetic on them. INT8 values are already byte-aligned, requiring only 8 mul and 7 add for accumulation. INT4 values, however, are sub-byte: eight 4-bit values share a single 32-bit word, and extracting them requires roughly 16 shift and 16 mask operations on top of the 8 mul and 7 add, which can rival or exceed the arithmetic cost itself. Because this overhead depends on the packing scheme, it must be modeled explicitly rather than folded into a generic compute cost. To guide our optimization decisions, we define the Layout Efficiency Proxy (LEP) for a layout ℒ of tensor 𝑡. Here, a layout configuration ℒ = (𝜋, 𝜌, 𝜙) specifies three choices: ● 𝜋 (contiguous-axis order) determines which logical tensor axis is stride-1 in memory, such as head 𝐻 , hidden dimension 𝐷, or token 𝑇 . ● 𝜌 (2D texture placement) determines how the ordered tensor axes are mapped to texture rows and columns on the mobile GPU. ● 𝜙 (low-bit value packing) determines how 4-/8-bit values are arranged within a machine word or RGBA lanes. The LEP is formulated as a weighted sum of two dominant cost factors:

4

LEP(ℒ, 𝑡) = 𝛼 1 ⋅

3

Empirical Study on LLM Inference

System Design

The key idea of BitLLM is to make low-bit LLM execution layout-adaptive on mobile GPUs. The motivation for this approach comes from the fact that layout and unpacking costs

Bytes(𝜋, 𝜌, 𝜙, 𝑡) + 𝛼 2 ⋅ UnpackOps(𝜙) (6) BWtex

The key is that both LEP components can be efficiently computed by combining analytical modeling with static analysis.

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs 4-bit

8-bit

16-bit

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

32-bit

H

Tensor view (3 dimension)

shift & mask mul

T D Dimensional Permutation

32-bit

INT4

T

Layout options

accumulate dequantize

D

T D

H

mul

[H,D,T]

.......

H [H,T,D]

T

....... D

[D,T,H] H

Texture Packing

32-bit

INT8 FP32

accumulate

Layout mapping

dequantize 32-bit

4D

H [H,T,D]

fused multiply-add

D

Quantization Packing Quantization packing

Figure 5. Execution pipelines for a length-8 dot product with INT4, INT8, and FP32 formats. FP32 uses direct fused multiply-add (fma); INT8 performs integer mul with 32-bit accumulation; INT4 additionally unpacks sub-byte weights with shift and mask instructions.

● Data access cost (Bytes(⋅)⇑BWtex ): this term is derived analytically by modeling the tensor (𝑡) access patterns against the memory mapping defined by 𝜋 and 𝜌. For an operator and 𝑡, we simulate the memory traversal pattern to count the bytes moved between DRAM and on-chip cache, considering cache line sizes and bank conflicts. ● Unpacking cost (UnpackOps(𝜙)): this term is quantified through static code analysis. For each quantization packing scheme 𝜙, we generate the corresponding kernel code and count the number of integer arithmetic and bitwise operations (shifts, masks, AND/OR) required to unpack a single packed word. This operation count is normalized to represent the overhead per arithmetic operation. ● Weight calibration (𝛼 1 , 𝛼 2 ): 𝛼 1 and 𝛼 2 are device-specific normalization weights calibrated once per GPU through offline microbenchmarks (<10 minutes) that separately stress the memory subsystem and integer ALU. The weights are model-independent and remain fixed across all models and sequence lengths. On our evaluated GPUs, 𝛼 1 = 0.5 and 𝛼 2 = 0.7 yield the best layout selection accuracy. To make LEP evaluation efficient at runtime, we first pre-compute Bytes(𝜋, 𝜌, 𝜙, 𝑡) and UnpackOps(𝜙) for all supported layout configurations during an offline profiling phase. During runtime selection, BitLLM queries these existing values and performs the weighted sum in Equation 6. This hybrid analytical-static approach allows BitLLM to efficiently evaluate thousands of (ℒ, 𝑡) combinations with minimal overhead. The LEP serves as a comparative metric to identify the most efficient layout configuration for the dominant operators in transformers (i.e., MatMul).

T

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 16 20 17 21 18 22 19 23 24 28 25 29 26 30 27 31

Figure 6. Layout optimization pipeline. Figure 6 shows an example of the layout optimization pipeline for tensor 𝐾 with shape (︀𝐻, 𝐷,𝑇 ⌋︀ under configuration ℒ. First, the dimension 𝐷 is chosen as the stride-1 axis since computation is performed along 𝐷, ensuring consecutive cache access. The tensor is then mapped into 2D texture space along the 𝐻 dimension, and finally, the low-bit quantized values are rearranged and packed according to 𝜙 for efficient dequantization. The subsequent sections describe our detailed mechanisms for minimizing the LEP. 4.2

Adaptive Layouts for Decoding

Having established the LEP as our efficiency metric, we now present the adaptive layout mechanism that minimizes this proxy. The challenge translates to dynamically selecting, for each KV cache operation, the layout configuration ℒ = (𝜋, 𝜌, 𝜙) that yields the lowest LEP given the current tensor 𝑡 and KV cache. Layout adaptation with sequence length. This section addresses the Bytes⇑BWtex term through runtime adaptation. Specifically, in layout adaptation with sequence length, the optimal stride-1 axis for the KV cache is a function of sequence length. Short contexts (e.g., < 256) are dominated by the 𝑄𝐾 ⊺ operation, favoring a 𝐷-contiguous layout. Long contexts (e.g., > 512) are dominated by 𝑆𝑉 , favoring a 𝑇 contiguous layout. Choosing a static layout leads to inefficient memory access and severe latency spikes (Figure 4). We adopt a greedy layout evolution strategy. Algorithm 1 formalizes this procedure, which periodically evaluates candidate layouts and selects the one minimizing LEP for the current and near-future sequence lengths. The algorithm generates a set of candidate layouts 𝒞 = {(𝜋, 𝜌, 𝜙)} by enumerating permissible stride-1 axis permutations (e.g., 𝐷-major vs. 𝑇 -major) along with corresponding texture packing schemes 𝜌 that comply with the mobile GPU’s memory constraints.

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

Hangyu Zheng et al.

Algorithm 1: Greedy Layout Evolution

H

ℒ𝑏𝑒𝑠𝑡 ← arg minℒ𝑐 ∈𝒞 LEP𝑐 if LEP𝑏𝑒𝑠𝑡 < LEP𝑐𝑢𝑟𝑟𝑒𝑛𝑡 then 6 ℒ𝑛𝑒𝑤 ← ℒ𝑏𝑒𝑠𝑡

4

8

It evaluates each candidate by computing its LEP score at the current sequence length 𝑡, estimating the efficiency of the predominant operator (e.g., 𝑄𝐾 ⊺ or 𝑆𝑉 ) under that layout. Because LEP uses precomputed values and one weighted sum per candidate, selection completes in microseconds with negligible decoding overhead. The lowest-LEP candidate is adopted only if it measurably improves upon the current configuration; otherwise, the current layout is retained. Folding along non-reduction axes. Mobile GPUs impose strict limits on the width and height of texture memory (e.g., ≤ 16K pixels on each dimension). As the KV cache grows in long context, tensor dimensions often exceed these limits, making direct mapping into 2D textures infeasible. A naive folding strategy partitions oversized dimensions arbitrarily. However, if folding occurs along a reduction axis – e.g., folding the hidden dimension 𝐷 in 𝑄𝐾 ⊺ or folding the token dimension 𝑇 in 𝑆𝑉 – the reduction spans across multiple tiles. This introduces strided, cross-cache-tile fetches, increasing memory traffic and degrading cache locality. We adopt an operator-aware folding policy. The key insight is that folding should preserve contiguous access along the dimension that is reduced during a computation-intensive operation (e.g., MatMul), as it is critical for achieving a stride-1 memory access pattern and high memory bandwidth utilization. Consequently, whenever folding is necessary, it is applied exclusively along non-reduction axes. This ensures the reduction dimension remains contiguous, preserving stride-1 access. As shown in Figure 7, for 𝑄𝐾 ⊺ (reduction over 𝐷), we fold along 𝑇 ; and for 𝑆𝑉 (reduction over 𝑇 ), we fold along 𝐷. This principle extends to other operators like Softmax, where reduction occurs along the last dimension, so we fold along preceding dimensions. This policy leverages operator semantics to make a locally optimal choice with zero search overhead. Append-only segments and phase-aware transitions. As the sequence length increases, the initially optimal KV cache layout can become suboptimal. Conventional systems typically resort to one of two costly approaches: 1) performing a re-layout of the entire cache – a process that copies hundreds

... ...

D Fold along T

H

D'

...

D... ... T

S

V

Fold T along D

Figure 7. Two folding examples.

else ℒ𝑛𝑒𝑤 ← ℒ𝑐𝑢𝑟𝑟𝑒𝑛𝑡

KT

T

5

7

D Q

T'

...

H

H

T ... ...

D

Input: : Current layout ℒ𝑐𝑢𝑟𝑟𝑒𝑛𝑡 , sequence length 𝑡, cache dimensions (𝐻, 𝐷,𝑇 ) Output: : New layout ℒ𝑛𝑒𝑤 1 𝒞 ← GenerateCandidates(𝐻, 𝐷,𝑇 ) 2 foreach candidate ℒ𝑐 ∈ 𝒞 do Bytes(ℒ ,𝑡 ) 3 LEP𝑐 ← 𝛼 1 ⋅ BWtex𝑐 + 𝛼 2 ⋅ UnpackOps(ℒ𝑐 )

of megabytes of data and imposes significant overhead, or 2) leaving the layout unchanged and accepting suboptimal performance. Both result in sharp latency spikes. A naive alternative is to switch layouts greedily at every small improvement in the Layout Efficiency Proxy (LEP). While this avoids re-layout, it produces many small fragments of the cache, significantly increasing iterator overhead and complicating memory management. We propose an append-only segmentation strategy coupled with phase-aware transitions. The key idea is Appendonly Segments, where KV cache is divided into immutable segments. Each segment records its layout (𝜋, 𝜌, 𝜙) and kernel schedule parameters (e.g., unroll factor, block size). When LEP indicates that a more efficient configuration is available, new tokens are appended into a new segment configured with that new layout. Previous segments remain intact, eliminating the need for copying or re-layout. To prevent excessive fragmentation, we design phase-aware transitions. Specifically, we apply a heuristic to LEP evaluation and trigger layout switches only when improvements persist over a sufficient sequence length. This compresses short-term fluctuations into a few stable computational phases, often aligned with operator-level shifts, such as transitions from 𝑄𝐾 ⊺ -heavy to 𝑆𝑉 -heavy workloads. This design avoids copying or relayout during transitions; switching only initializes metadata for a new segment. Across our 10K-token runs, it creates 2.25 segments on average and at most five. Because each segment is contiguous, the runtime traverses only a small number of segments, and total segmentation overhead remains below 1% of end-to-end decoding latency. 4.3

Quantization-Aware Packing

Low-bit quantization reduces memory footprint but introduces significant unpacking overhead when data formats are misaligned with hardware execution units. Modern mobile GPUs expose native integer dot-product instructions (e.g., dp4a and udot8), which multiply multiple INT8 pairs and accumulate the result into an INT32 register in a single cycle. To exploit these primitives effectively, operands must be packed into specific 32-bit vector layouts. Naively packing low-bit values (e.g., INT4) contiguously into a 32-bit word misaligns them with byte-addressable

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs Byte 0

1

uint32x4 (16 bytes) 2

3 ... 29 30 31

Unpacking 0

1

2

32 33 34 35 ... 61 62 63 64 65 66 67 ... 93 94 95 96 97 98 99 ... 125126 127

4

1

5 ... 30 27 31

Sequential byte extraction 0 1 2 3 .... ... 28 29 30 31 = W.x >> 28 = (W.w >> 4) & 0x000000F0

0

4

1

5 ... 30 27 31 W

32 36 33 37 ... 62 59 63 64 68 65 69 ... 94 91 95

0 1 2 3 ... ... 14 15 = (W&0xF0F0F0F0) >>4

96 100 97 101 ... 126123 127

16 17 18 19 ... ... 30 31 = W&0xF0F0F0F0

(b) Lane-interleaved packing

1 shift + 2 mask

16 0

4 ... 15 16 20 ... 31

0

4

1

5 ... 46 43 47 W

32 36 ... 47 48 52 ... 63 64 68 ... 79 80 84 ... 95

0 1 2 3 ... ... 14 15 = (W&0xF0F0F0F0) >>4

96 100 ... 111 112 116 ... 127

32 33 34 35 ... ... 46 47 = W&0xF0F0F0F0

(c) Block-swizzled packing

1 shift + 2 mask

Figure 8. Different quantization packing schemes. hardware. Eight linearly packed INT4 values cross byte boundaries, forcing the GPU to extract each value and expand it to INT8 before integer arithmetic. As quantified in Section 4.1, this expansion can cost more than the computation itself. To mitigate this inefficiency, we treat quantization packing as a schedulable dimension and redesign it to minimize the Unpacking term in LEP. Specifically, we introduce a set of structured packing families. Figure 8 illustrates the three schemes. The contiguous-in-byte scheme maximizes storage density but requires about 31 shift and 31 mask operations to extract values from a four-word vector, the highest unpack cost. Lane-interleaved packing rearranges nibbles so that each byte holds two aligned INT4 values, matching the four RGBA lanes of texture memory. This reduces unpacking to a fixed sequence of one shift and two masks per word, independent of tensor shape. Block-swizzled packing further organizes values into 2 × 2 texel tiles aligned with texture cache blocks, combining the constant-time unpacking of lane interleaving with improved memory coalescing. Formally, if packed operands are expressed as 3

𝐴 = ∑ 𝑎𝑖 ⋅ 28𝑖 , 𝑖=0

3

𝐵 = ∑ 𝑏𝑖 ⋅ 28𝑖 ,

𝑎𝑖 , 𝑏𝑖 ∈ Z8,

𝑖=0

the dp4a instruction computes 3

Table 2. 8-bit dot-product primitives on different platforms.

3 ... 29 30 31 W

(a) Contiguous-in-byte packing 31 shift + 31 mask 0

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

dp4a(𝐴, 𝐵, 𝑟 acc ) = 𝑟 acc + ∑(𝑎𝑖 × 𝑏𝑖 ),

𝑟 acc ∈ Z32 .

𝑖=0

Our packing strategies guarantee that unpacked INT4 values expand directly into the aligned INT8 operands required by such primitives, with only constant-time bit operations. Compared to naive packing, where each word requires up to 20 unpacking operations, our reordered packing reduces the requirement to approximately three instructions. Microbenchmarks show that this reduces unpacking overhead by about 80% and increases integer pipeline utilization

Platform

Instruction

Adreno/MALI GPU AMD GPU (GCN/RDNA) Intel GPU (Xe) NVIDIA GPU

udot8_acc / dot8_acc v_dot4_u32_u8 / v_dot4_i32_i8 dpas.s32.s8.s8 dp4a.s32.s8.s8

accordingly. Without this, the instruction cost of unpacking can negate the memory savings from quantization. Discussion: applicability to other GPUs. Our implementation targets Adreno and Mali GPUs (Section 5.4), but the packing design requires only 8-bit integer dot-product accumulate instructions. Table 2 shows that similar primitives exist on AMD, Intel, and NVIDIA GPUs, making the approach portable beyond mobile. Extension to CPUs and NPUs is less straightforward: low-bit systems on CPU/NPU [70, 77, 78] use table-lookup over SIMD lanes co-designed with linear memory hierarchies, whereas our packing and layout codesign targets warp-parallel execution. Comparison across these platforms is a direction for future work. 4.4

Type Conversion-Hiding Computation

Conventional low-bit computation on mobile GPUs incurs repeated data type conversions. INT8 multiplication requires upcasting to INT16 to avoid overflow, and many frameworks further convert quantized tensors to FP16 for MatMul. These conversions account for over 30% of arithmetic instructions in INT4 execution and leave the integer dot-product units underutilized, because conversion overhead sits directly on the critical path. We introduce an integer-native, tile-based pipeline that eliminates conversions from the core computation path. We reorganize the computation around tiles, which are small and contiguous blocks of the weight or activation matrix that serve as the fundamental unit of data transfer and parallel computation. Our design is built on three key principles: 1) maintaining all hot-path computations in the integer domain, 2) deferring dequantization and floating-point conversion to an epilogue stage, and 3) overlapping this epilogue asynchronously with the computation of subsequent tiles. This represents a paradigm shift from converting-then-computing to computing-then-converting. As shown in Figure 9, the pipeline is organized at the tile granularity and comprises three tightly overlapped stages. First (Load & Unpack), tiles are loaded in their compactlypacked form from texture memory. Leveraging the reordered packing scheme from the previous section, we unpack INT4 data into byte-aligned INT8 values using only ≈3 bitwise operations (1 shift and 2 mask) per 32-bit word – a 5× improvement over the 16 operations needed for naive contiguous packing. This unpacking occurs uniformly in registers, eliminating thread divergence and minimizing UnpackOps.

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

Stage 1

Stage 2

(1) Load & Unpack (2) Dot-product

Hangyu Zheng et al.

Stage 3 (3) Dequantize

Table 3. Model perplexity on the WikiText2 [42] dataset (lower is better) for different precisions.

9 load 2D unpack to INT8 memory

dp4a

INT32

FP16/32

Llama-3.2-1B [24] Qwen3-1.7B [84] Gemma-2-2B [67] Phi-2-2B [29]

Time Tile K-1 Tile K Tile K+1

Load & Unpack

Dot-product

Dequantize

Figure 9. Tile-based pipelining for type conversionhiding computation. The pipeline consists of three stages: (1) Load & Unpack, (2) Integer Compute, and (3) Deferred Conversion. Operations overlap across tiles, ensuring the hot path remains fully in the integer domain. In the second stage (Dot-product), the unpacked, bytealigned tiles are immediately fed into integer dot-product primitives (e.g., dp4a, udot8), which accumulate results directly into INT32 registers. This stage is entirely free of type conversions, allowing the integer units to operate at peak utilization. Tile sizes are co-designed with the adaptive layouts (Section 4.2) to balance register pressure and memory coalescing, keeping traffic near the LEP lower bound. In the third stage (Dequantization), the integer accumulations are converted to FP16/FP32 and fused with dequantization. This epilogue runs only at tile boundaries, asynchronously via warp specialization. As shown in Figure 9, warp group A computes on tile 𝑘 while group B loads tile 𝑘+1 and group C converts the results of tile 𝑘−1. Synchronization is needed only at tile boundaries, which effectively removes conversion latency from the critical path. Except when parallelism or compute slack is limited, as in very short contexts (sequence lengths below 12 in our kernels) or small matrices, the pipeline effectively overlaps conversion with computation. Such cases are uncommon in our evaluated workloads. Even under these conditions, conversion remains outside the inner dot-product loop, and only a portion of the epilogue latency is exposed on the critical path. The conventional pipeline requires two conversions per tile (unpacking before computation and dequantization after), which creates register traffic and leaves the integer ALUs idle. Our pipeline eliminates conversions from the hot path; the single deferred dequantization per tile is hidden behind the next tile’s computation. In practice, this reduces conversion instructions by over an order of magnitude and unpack overhead by ≈80% (Section 5.3).

5

Model

Evaluation

This section evaluates BitLLM’s effectiveness in terms of latency, energy consumption, memory/cache efficiency, optimization contributions, and portability across mobile GPUs.

5.1

#Params 1.24B 1.72B 2.61B 2.78B

FP16 13.75 9.84 18.09 17.41

Perplexity ↓ W8A8 W4A8 14.31 9.97 20.33 18.15

15.34 10.28 21.40 19.94

Experiment Setup

Workloads. We select four representative LLMs that are suitable for mobile GPU deployment: Llama-3.2-1B [24], Qwen31.7B [84], Gemma-2-2B [67], and Phi-2-2B [29]. They cover three attention mechanisms: grouped-query attention (GQA) as in Llama-3.2 and Qwen3, multi-head attention (MHA) as in Phi-2, and alternating local-global attention in Gemma2. These models are widely used in on-device studies and fit within the 8–16GB unified memory (shared with the OS and other apps) of current mobile GPUs; larger models (e.g., 7–8B) remain impractical at full precision. For each model, we evaluate two low-bit precision settings: W8A8, with 8bit weights, activations, and KV cache under group-wise scaling; and W4A8, with 4-bit weights and 8-bit activations and KV cache — a balance between compression and accuracy validated by prior work [39, 79]. The group size is 32, following [21, 38]. Although our main focus is low-bit quantization, we include FP16 to show our adaptive layouts benefit higher-precision computation as well. Before focusing on latency, energy, and memory behavior in the rest of this section, we demonstrate that the quantization levels chosen here result in high accuracy. Table 3 reports model-level accuracy on standard benchmarks. Across all four models, W8A8 incurs only a modest perplexity increase over FP16 (0.1–2.2 points), while W4A8 remains within 0.4–3.3 points of FP16, confirming that group-wise quantization at these precisions preserves model quality. BitLLM is independent of the quantization algorithm and focuses on execution efficiency. Therefore, for a given quantization scheme, BitLLM provides the same model accuracy as other frameworks. The remainder of our evaluation therefore focuses on system-level performance. Baselines. We compare BitLLM with 4 widely used LLM inference engines on mobile GPUs: LLAMA.CPP [11] (31914624), MNN-LLM [76] (7f1197e0), LiteRT-LM [23] (26f2178, powered by TensorFlow-Lite [12]), and MLC-LLM [44] (8f49ea6b, powered by TVM [9]). Several related frameworks, such as bitnet.cpp [70] and other works including TransformerLite [34], Ladder [71], PowerInfer-2 [83], and ExecuTorch [53], do not support the tested models on mobile GPUs. This is because they lack operator implementations or otherwise have hardware dependencies. In the reported results, we enable the auto-tuning capabilities available in all baselines to

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

180

MNN-LLM

MLC-LLM

LiteRT-LM

Prompt-Output: 768-512

120 60 0

Prompt-Output: 180 512-512

120 60 0 180

FP16 W8A8 W4A8 FP16 W8A8W4A8 FP16 W8A8W4A8 FP16 W8A8W4A8

Phi-2-2B

Figure 10. Average latency per token comparison across models, precisions, and Prompt–Output (𝑃–𝑂) window. achieve the best possible performance. The batch size is set to 1 in mobile environments. Energy consumption is collected through Android system interface. We run each experiment 50 times and report the average, as the variance is negligible. Testbed. Our primary platform is a OnePlus 12 with a Snapdragon 8 Gen 3 SoC, an Adreno 750 GPU, and 16 GB of unified memory. To assess portability (Section 5.4), we also evaluate a OnePlus 11 with a Snapdragon 8 Gen 2 SoC, an Adreno 740 GPU, and 16 GB of memory to test same-vendor generalization, and a Google Pixel 8 with a Tensor G3 SoC, a Mali-G715 GPU, and 8 GB of memory to validate crossvendor portability under tighter resource constraints. 5.2

MNN-LLM MNN-LLM

LiteRT-LM LiteRT-LM

MLC-LLM

Ours

W8A8

600 300 0 900 W4A8

Not support for longer decoding

600

0

60 LLaMa-3.2-1B Qwen-3.2-1.7B Gemma-2-2B

900

LLAMA.CPP LLAMA.CPP

300

Prompt-Output: 256-512

120 0

CPU: GPU:

Ours

Latency per Token (ms)

Average Decoding Latency per Token (ms)

LLAMA.CPP

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

End-to-End Performance Comparison

This section compares the end-to-end performance of BitLLM with the state-of-the-art baselines on mobile GPUs across various context lengths and model precisions. Average latency with fixed context length. We report average per-token decoding latency across four models and four baselines under three Prompt–Output (𝑃–𝑂) windows: 768–512, 512–512, and 256–512, where, e.g., 768–512 denotes a 768-token prompt followed by 512 autoregressively decoded tokens. As shown in Figure 10, BitLLM achieves lower per-token latency than all baselines across all configurations. Against LLAMA.CPP, the average speedups are 1.29× (FP16), 2.47× (W8A8), and 1.59× (W4A8); against MNN-LLM, 1.55×, 2.06×, and 1.60×; against LiteRT-LM, 1.37×, 1.84×, and 1.63×; against MLC-LLM, 1.38× (FP16) and 1.56× (W4A8) (this framework does not support W8A8). FP16 also benefits from our adaptive layouts, though the gains are more modest because conversion overhead is not a major factor here. Notably, BitLLM achieves 1.66× (W8A8) and 2.36× (W4A8) speedups over FP16, recovering 83% and 59% of the theoretical 2×/4× bounds. The gap stems from non-quantized

Figure 11. Per-token decoding latency as sequence length grows (up to 10𝐾 tokens). MLC-LLM does not support W8A8 or long contexts at W4A8. Our method (red) achieves the lowest latency and reduces the stair-step spikes observed in baselines. components (e.g., Softmax and Normalization) and fixed runtime overheads that are not reduced with bit-width. In contrast, baselines capture far less: LLAMA.CPP achieves only 0.75× at W8A8 (even slower than FP16) and 1.56× at W4A8; MNN-LLM reaches 1.25×/1.86×; LiteRT-LM reaches 1.14×/1.79×; MLC-LLM attains 1.76× at W4A8. This demonstrates that BitLLM translates quantization potential into end-to-end speedups more effectively than prior systems. Moreover, BitLLM’s advantage widens in longer prompts: against LLAMA.CPP, the average speedup grows from 1.62× (256–512) to 1.98× (768–512); similar trends hold for MNNLLM (1.61×→1.85×), LiteRT-LM (1.50×→1.68×), and MLCLLM (1.34×→1.62×). Longer prompts intensify prefill computations and expand the KV cache, increasing sensitivity to memory layout. Baselines apply a fixed layout regardless of shape, incurring growing penalties, whereas our cacheaware runtime adapts layouts as tensor shapes evolve and our conversion-hiding execution mitigates the overhead. Working with a fixed latency budget. Under a fixed latency budget, BitLLM can use its efficiency gains to run higher-precision configurations and improve model quality. Across the four models and three 𝑃–𝑂 windows in Figure 10, BitLLM at FP16 achieves geometric-mean speedups of 1.75×, 1.46×, and 1.24× over LLAMA.CPP, MNN-LLM, and LiteRT-LM at W8A8, respectively. MLC-LLM is omitted because it does not support W8A8. Similarly, BitLLM at W8A8 outperforms LLAMA.CPP, MNN-LLM, LiteRT-LM, and MLC-LLM at W4A8 by 1.08×, 1.07×, 1.04×, and 1.02×, respectively. Table 3 confirms the corresponding quality benefit: FP16 reduces perplexity by 0.13–2.24 points over W8A8, while W8A8 reduces it by 0.31–1.79 points over W4A8. Thus, BitLLM can either reduce latency at fixed precision or improve model quality under a fixed latency budget. Latency scaling with increasing context length. Next, we analyze how speedups evolve with decoding length. We fix 𝑃 and vary 𝑂 from 1 to 10𝐾. Due to space constraints, we

MLC-LLM

10 5 1

LiteRT-LM

Ours

6 4 2 1

2048 4096 6144 8192 10240

Average Energy

Energy per Token

10 5 1

2048 4096 6144 8192 10240

6 4 2 1 W4A8

(b) LLaMa-3.2-1B with W4A8 (normalized by ours)

Figure 12. Normalized (by ours) per-token and average energy consumption for low-bit precisions. report results for a representative model (Llama-3.2-1B), as other models exhibit similar trends. We present results for the low-bit precisions W8A8 and W4A8. Figure 11 shows that BitLLM consistently outperforms all baselines as 𝑂 increases. Our per-token latency remains flat, whereas all baselines exhibit periodic spikes from increasingly inefficient layouts as the KV cache grows. BitLLM achieves average speedups of 3.25×, 1.68×, and 1.50×, respectively, over LLAMA.CPP, MNN-LLM, and LiteRT-LM at W4A8. MLC-LLM degrades catastrophically at longer contexts (speedup over 43.81×) due to its static layout design; we therefore exclude it from the primary long-context comparison. At W8A8, the speedups are 3.96×, 2.14×, and 1.81× against LLAMA.CPP, MNN-LLM, and LiteRT-LM. Notably, for these baselines the CPU backend often outperforms their own GPU backend at low-bit precisions, owing to richer native support for low-precision integer arithmetic on CPUs. Even against their best results across platforms, which happen to be the CPU backends, BitLLM on GPU still achieves substantial speedups: 2.11×, 1.65×, and 1.56× over LLAMA.CPP, MNN-LLM, and LiteRT-LM at W8A8, and 1.74×, 1.42×, and 1.38× at W4A8. Speedups grow monotonically with 𝑂 as KV cache expansion amplifies layout sensitivity. From 𝑂=1 to 𝑂=10𝐾, the gain over LLAMA.CPP rises from 2.0× to 8.35× (W8A8) and 1.1× to 6.93× (W4A8); over MNN-LLM, from 1.49× to 2.66× (W8A8) and 1.38× to 2.48× (W4A8); over LiteRT-LM, from 1.24× to 2.03× (W8A8) and 1.21× to 1.98× (W4A8). The trend is pronounced against MLC-LLM at W4A8, where speedup escalates from 1.36× to 527×, as its static layout incurs increasingly severe penalties with KV cache growth. These results confirm that our adaptive layout and integer-native execution mitigate the overhead of growing context length. Energy consumption per token and average profile. Figure 12 reports normalized per-token energy (left) and average energy (right) for Llama-3.2-1B under W8A8 and W4A8, with 𝑂 from 1 to 10𝐾 and a fixed prompt of 512. BitLLM reduces

LT

LT+PK+TCH

LT+PK

2 1 0

W8A8

(a) LLaMa-3.2-1B with W8A8 (normalized by ours) 15

OurB

3

W8A8

W4A8

LLaMa-3.2-1B

W8A8

W4A8

Qwen3-1.7B

(a)

Speedup Gain (%)

MNN-LLM

Average Energy

Energy per Token

LLAMA.CPP

15

Hangyu Zheng et al. Speedup over Baseline

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

40

PK-HAP

PK-Unpack

TCH

W8A8

W4A8

30 20 10 0

W8A8

W4A8

LLaMa-3.2-1B

(b)

Qwen3-1.7B

Figure 13. Optimization breakdown on mobile GPU. (a) Cumulative speedup over OurB (baseline = 1) as each optimization is enabled. LT, PK, and TCH denote adaptive layout, quantization-aware packing, and type-conversion hiding, respectively. (b) Fine-grained breakdown of the incremental speedup gains from PK and TCH over OurB. The PK gain is separated into hardware-aligned packing (PK-HAP) and reduced INT4 unpacking (PK-Unpack).

average per-token energy by 75.7%/66.0% (W8A8/W4A8) over LLAMA.CPP, 66.7%/56.1% over MNN-LLM, 54.4%/44.2% over LiteRT-LM, and 88.9% over MLC-LLM at W4A8. Consistent with the latency, our energy consumption remains stable as 𝑂 grows, whereas baselines show rising energy overhead with longer sequences. 5.3

Performance Breakdown and Analysis

Optimization breakdown. Figure 13(a) reports the cumulative speedup of each optimization for two representative models (Llama-3.2-1B and Qwen3-1.7B) under a fixed 𝑃–𝑂 window of 512–512 at W8A8 and W4A8 precisions. Other models exhibit similar trends and are omitted for brevity. All bars are normalized to OurB, a variant of BitLLM with all key optimizations disabled; each subsequent bar adds one optimization cumulatively. Accordingly, the differences between adjacent bars quantify the incremental contribution of each component at each precision: OurB→LT for adaptive layout, LT→LT + PK for quantization-aware packing, and LT + PK→LT + PK + TCH for the type-conversion-hiding pipeline. At W8A8, LT (adaptive layout) delivers 1.71× and 2.23× speedups on Llama-3.2-1B and Qwen3-1.7B, respectively. Adding PK (quantization-aware packing) yields an additional 1.22× and 1.26×; TCH (type-conversion hiding) contributes a further 1.23× and 1.34×. At W4A8, the corresponding incremental gains are 1.37× and 1.34× (LT), 1.1× and 1.2× (PK), 1.2× and 1.29× (TCH). Among all components, LT contributes the largest improvement by preserving data locality through layout adaptation, enabling memory bandwidth to be utilized more effectively (further quantified in the following analysis). Quantization-aware packing breakdown. Figure 13(b) further shows the incremental gains over OurB that are the cumulative speedups reported in Figure 13(a). It decomposes PK gain into hardware-aligned packing (PK-HAP) and reduced sub-byte unpacking (PK-Unpack), and reports TCH separately as the contribution of the subsequent conversion-hiding kernel pipeline. PK-HAP captures the benefit of matching the

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

15 0

W8A8 W4A8

(a) Bandwidth

L1 cache

L2 cache

60 30 0

W8A8 W4A8 W8A8 W4A8

(b) Cache hit rate

LiteRT-LM

Ours

1.00

0.9 0.7

90

0.5 0.3

60

0.1

30 0

GPU

Oracle LEP Mean P95 search time search time slowdown ↓ slowdown ↓

0.95

a2

90

MLC-LLM

Normalized Speed

30

MNN-LLM

Utilization (%)

45

Cache hit rate (%)

Bandwidth (GB/S)

LLAMA.CPP

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

0.90

Adreno-750

1, 414s

45.5ms

1.86%

7.27%

Adreno-740

1,578s

52.9ms

2.36%

5.83%

Mali-G715

2,747s

95.2ms

3.89%

10.40%

0.1 0.3 0.5 0.7 0.9

a1

(a)

(b)

W8A8 W4A8

(c) Utilization

Figure 14. Memory, cache, and ALU utilization analysis.

stored operand order to the RGBA texture lanes and dotproduct instructions, whereas PK-Unpack isolates the reduction in shift-and-mask operations for packed INT4 values. At W8A8, PK-HAP contributes 21.9% and 25.7% on Llama-3.21B and Qwen3-1.7B, respectively, while PK-Unpack has no contribution because INT8 values require no sub-byte extraction. At W4A8, PK-HAP contributes 4.63% and 6.71%, and PK-Unpack contributes 8.95% and 13.2%, respectively. The subsequent TCH optimization contributes 22.6% and 33.7% at W8A8 and 22.6% and 28.7% at W4A8. The W8A8 packing gain therefore comes entirely from PK-HAP. Although the default W8A8 representation is bytealigned, its element order does not match the fixed lane order in which a 32-bit dot-product instruction consumes the four values returned by an RGBA texture fetch. This mismatch requires lane rearrangement. PK-HAP instead stores each group in consumption order, allowing the fetched values to feed the instruction directly. For the profiled Llama-3.21B workload(𝑃–𝑂=512–512, Adreno 750), Figure 14 shows that W8A8 exhibits 25.4%–60.2% higher measured bandwidth than W4A8 across baselines supporting both precisions, with mean L1/L2 cache-hit-rate decreases of 3.9%/6.3%. Given this higher memory pressure, BitLLM achieves a larger bandwidth gain at W8A8, reaching up to 1.84× the measured baseline bandwidth, compared with 1.65× at W4A8. This contrast explains the larger PK-HAP benefit in Figure 13(b), while W4A8 gains more from reduced INT4 unpacking. Memory, cache, and ALU analysis. The preceding breakdown quantifies each optimization’s contribution; we now profile the hardware-level metrics that explain these gains. Figure 14 reports memory bandwidth, cache behavior, and ALU utilization measured via Snapdragon Profiler [54]. Figure 14 (a) shows GPU memory bandwidth at W8A8 and W4A8 (MLC-LLM is excluded at W8A8 as it does not support this precision). BitLLM achieves the highest bandwidth across both precisions, reflecting the aligned access patterns produced by adaptive layouts and operator-aware folding (Section 4.2). Figure 14 (b) shows cache hit rates: BitLLM achieves 13.6%–18.6% higher L1 and 10.4%–37.3% higher L2 hit rates than all baselines. W4A8 yields higher hit rates than W8A8 due to the reduced memory footprint. This also explains why W4A8

Figure 15. LEP calibration and selection analysis. (a) Impact of different LEP weights (𝛼 1, 𝛼 2 ). (b) LEP search overhead and performance gap to oracle across mobile GPUs.

baselines can outperform W8A8 on mobile GPUs. In the profiled configuration, relative to W8A8, W4A8 reduces memory traffic by 39.8%, increases the L1 and L2 cache hit rates by 21.14% and 12.3%, respectively, and increases unpacking overhead by 9.04%. For memory- and cache-sensitive decoding, reduced data movement and improved cache locality can outweigh the additional INT4 unpacking cost. Conversely, the higher memory pressure of W8A8 increases the benefit of hardware-aligned packing, consistent with the larger PK-HAP contribution in Figure 13(b). The improved cache locality follows from stride-1 access preservation through non-reduction-axis folding and length-aware layout selection. Figure 14 (c) reports shader ALU utilization. BitLLM achieves the highest utilization at both precisions, with improvements of up to 18.6% at W8A8 and 7.9% at W4A8. This gain results from quantization-aware packing (reducing unpacking instructions by ≈80%) and conversion-hiding execution, which reduce unpacking work and remove type conversion from the compute critical path, respectively. Analysis of LEP weights. Figure 15(a) presents the sensitivity of LEP to different 𝛼 1 and 𝛼 2 settings on our primary testbed. The specific values are obtained from the offline calibration (in Section 4.1), which completes in less than 10 minutes. The parameters are model-independent, and the best-performing setting (𝛼 1 = 0.5, 𝛼 2 = 0.7) is selected and used across all models. Even drastically different parameter values result in only about a 10%–15% performance degradation. Incorporating automated tuning strategies (e.g., Bayesian optimization) to further streamline the calibration process is left as future work. Figure 15(b) compares LEP with exhaustive oracle search across representative mobile GPUs. Oracle search benchmarks every candidate layout, requiring 1413.6–2747.3 s across the three GPUs. LEP completes layout selection in only 45.5–95.2 ms, reducing search time by more than four orders of magnitude. The selected layouts remain close to the oracle, with mean slowdowns of 1.86–3.89% and P95 slowdowns of 5.83–10.40%. These results demonstrate stable selection quality across mobile-GPU architectures. 5.4

Portability Evaluation

We assess portability on two additional platforms: the OnePlus 11 (same vendor, earlier generation) and the Google

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

Average Latency (ms)

LLAMA.CPP

300

MNN-LLM 500.0

200

LiteRT-LM

Ours

180 120

100 0

MLC-LLM

Hangyu Zheng et al.

60 FP16 W8A8 W4A8 FP16 W8A8 W4A8

LLaMa-3.2-1B

Qwen3-1.7B

(a) Mali-G715 (Pixel 8)

0 FP16 W8A8 W4A8 FP16 W8A8 W4A8 LLaMa-3.2-1B Qwen3-1.7B

(b) Adreno 740 (Oneplus 11)

Figure 16. Portability evaluation. Empty bars indicate that the framework does not support the configuration due to missing operator support or insufficient device memory. Pixel 8 (cross-vendor, more constrained resources). Figure 16 reports per-token latency for Llama-3.2-1B and Qwen3-1.7B at FP16, W8A8, and W4A8 under 𝑃–𝑂=512–512. BitLLM runs all model–precision combinations on both devices. At low-bit precisions, BitLLM achieves 1.3×–2.9× speedups on OnePlus 11 and 1.4×–3.6× on Pixel 8 (average 1.7× vs. 1.9×, as tighter resources amplify the benefit of our optimizations). At FP16, speedups of 1.1×–2.1× and 1.1×–3.6× confirm that adaptive layouts alone provide significant benefits.

6

Related Work

Quantization for LLMs. Classical quantization methods for deep learning have evolved into LLM-specific schemes that preserve model accuracy at aggressively low-bit widths [60]. LLM.int8 [18] halves memory relative to FP16 with an outlieraware 8-bit scheme. SmoothQuant [79] migrates activation scaling into weights for W8A8 quantization. GPTQ [21] performs one-shot weight quantization for accurate 3–4 bit deployment. MobileQuant [65] develops mobile-friendly quantization algorithms for on-device applications, which is complementary to our setting. Quantization has also been applied to reduce KV cache costs: KVQuant [27] provides calibration-based 4-bit KV quantization, and Q-Hitter [89] incorporates token-oracle signals for better compression. SNIP [51] explores sub-byte mixed-precision for LLM training, while AWQ [38] rescales salient channels for 4-bit quantization. These works primarily target server-class platforms and their specialized low-bit instruction sets, such as CUDA INT4/INT8 and AVX-512 VNNI. Mainstream mobile GPUs (e.g., Adreno, Mali) lack such dedicated instruction support; our work is designed for these architectures. Low-bit computation optimizations. A line of work on precision-scalable hardware [2, 17, 31, 56, 57] scales execution width to operand precision, but largely targets custom accelerators. T-MAC [77] and BitNet.cpp [70] explore tablelookup-based low-bit computation on edge-device CPUs, but their access patterns map poorly to GPUs due to uncoalesced memory accesses and misalignment with GPU memory hierarchies. PMPD [7] focuses on mixed-precision scheduling, adapting precision across inference phases and over

the generated sequence to improve efficiency on resourceconstrained hardware. On the server side, Atom [90] and QServer [40] integrate mixed-precision quantization with CUDA tensor-core kernels unavailable on mobile GPUs. At format and language level, M2XFP [28] proposes a metadataaugmented microscaling format for low-bit quantization, and Tilus [19] provides a tile-level GPU programming language for low-precision computation. Other closely related work generates quantized kernels whose lane-interleaving or bit-packing layouts are dictated by specific SIMD instructions (e.g., ARM NEON [14], x86 VNNI [58], and Tensor Core MMA [22]), and are resolved statically at compile time or during offline preprocessing. LLM serving and compiler frameworks. Several frameworks target LLM inference on server GPUs [10, 13, 52, 80, 93]. DeepSpeed-Inference [4] combines kernel fusion, tensor parallelism, and quantization-aware execution. FlashAttention [15, 16] improves attention kernel utilization through optimized work partitioning. SGLang [92] and Vidur [1] explore scheduling and serving strategies. oFFN [63] accelerates inference with an outlier-aware structured FFN design. For KV cache management, vLLM [32] reduces fragmentation via paged attention. Paged KV-cache management and speculative decoding have been developed primarily for server-class LLM serving, where memory management in high-throughput request processing are central. In contrast, BitLLM targets efficient low-bit decoding on resourceconstrained mobile GPUs, but these techniques remain complementary. A paged KV cache could be integrated by treating pages or page groups as layout-managed segments. Speculative decoding would require additional KV-state management; we leave their implementation and evaluation to future work. Data layout also matters for efficient DNN execution [48]. TVM [9] optimizes graph-level fusion and layout selection; Ansor [91] and TMModel [25] search over loop transformations and tiling with learned cost models. LADDER [71] transforms computations into statically optimized low-precision pipelines with memory-layer-specific tensor transformations. Triton [68] enables tile-centric code generation with near-vendor-library performance. These compiler systems primarily optimize code generation and static schedules for predetermined layouts. BitLLM instead uses LEP to select quantization-aware layouts dynamically across decoding phases and couples the selected layouts with a type-conversion-hiding execution pipeline. On-device LLM inference. While earlier mobile inference systems focused on convolution-heavy workloads [30, 33, 47, 85, 87], the shift to LLMs has driven a new generation of on-device runtimes [3, 41, 49, 72]. LLAMA.CPP [11] and LiteRT-LM [23] provide cross-platform runtimes leveraging vector instructions and low-bit formats. MLC-LLM [44], built on TVM, enables low-bit LLM execution on mobile GPUs. However, as our evaluation reveals, these frameworks inherit static layout assumptions that do not adapt to the

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

dynamic memory access patterns of autoregressive decoding, leaving a performance gap on mobile GPUs. SoD2 [45] and Transformer-Lite [34] target dynamic-shape inference and memory planning for mobile deployment, achieving significant speedups through static shape propagation and operator-level optimizations. FlashMem [61] optimizes the mobile GPU memory hierarchy for LLM workloads, though it targets floating-point operations and does not handle the growing KV cache during decoding. Neuralink [74] reduces on-device LLM compute through neuron co-activation linking, which is orthogonal to the layout and type conversion overhead we focus on. Production frameworks including ExecuTorch [53] and ONNXRuntime [50] primarily target mobile CPUs, with limited low-bit GPU backend support. Other on-device efforts target NPUs and heterogeneous SoC execution, including PowerInfer [64, 83], Ripple [73], LLM.NPU [82], HeteroLLM [8], and T-MAN [78], but do not address the GPU-oriented, low-bit decoding challenges we focus on; extending our optimizations to such heterogeneous scenarios is a promising future direction. Overall, BitLLM’s distinction lies not in any individual layout, packing, or pipelining technique, but in their sequence-aware coordination for mobile-GPU decoding. The underlying framework, LEP, dynamically makes layout and packing decisions as the KV cache grows, while the conversion-hiding pipeline preserves these gains during execution.

7

Discussion and Future Work

Generality across execution environments. At the LEP level, supporting a new mobile GPU requires updating the supported layout candidates and recalibrating the devicespecific weights once offline, as detailed in Section 4.1 and evaluated across GPUs in Section 5.3. For a heterogeneous SoC (i.e., with CPU, GPU, and NPU), LEP can be extended by scoring backend–layout pairs and incorporating an interbackend transition term: Bytes𝑏 (𝜋, 𝜌, 𝜙, 𝑡) LEP𝑏 ′ →𝑏 (ℒ, 𝑡) = 𝛼 1,𝑏 BW𝑏 + 𝛼 2,𝑏 UnpackOps𝑏 (𝜙) + 𝑇trans (𝑏 ′ →𝑏, ℒ, 𝑡). Here, 𝑏 ′, 𝑏 ∈ {CPU, GPU, NPU} denote the source and target backends. The current GPU implementation assumes 2D texture and RGBA access, GPU-oriented packing, and low-bit dot-product instructions. Extending LEP to CPUs and NPUs would preserve the cost-model structure but require backend-specific layouts, memory models, and execution primitives (i.e., instruction sets). The transition term 𝑇trans captures synchronization, data movement, and other required layout conversion between backends; it is zero when 𝑏 ′ = 𝑏. The heterogeneous extension would retain LEP’s lightweight selection procedure. For a fixed source backend 𝑏 ′ , LEP evaluates ∑𝑏 ⋃︀𝒞𝑏 ⋃︀ backend–layout candidates. Because the backend-specific LEP parameters are precomputed, each candidate is evaluated with lightweight computation.

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

Concurrent workloads can change the effective backend and bandwidths, causing parameters obtained through offline calibration to become suboptimal. A contention-aware extension could update these parameters at decoding-phase boundaries based on resource-availability information provided by the runtime scheduler and then rescore the affected candidates. Extending to contention-aware heterogeneous execution is a promising direction. Extending to new quantization schemes. BitLLM currently evaluates W8A8 and W4A8 using a group size of 32; however, its layout-selection and packing formulation is not specific to these configurations. Supporting an additional quantization format requires incorporating compatible packing candidates and extending the LEP cost components to capture format-specific overheads. Specifically, the Bytes term could account for quantized data, whereas the UnpackOps term accounts for bit extraction, operand rearrangement, and type conversion. These costs may depend on both group size and bit width: group size determines dequantization granularity, while bit width determines packing density, unpacking cost, and the availability of compatible integer primitives. For each target device, a packing candidate is associated with a supported execution path; when the packed representation cannot be consumed directly by a native primitive, LEP can further incorporate the required unpacking and conversion costs. LEP then ranks the extended set of candidates using the resulting cost model. We leave additional quantization formats to future work.

8

Conclusion

We presented BitLLM, which addresses the unrealized potential of low-bit LLM inference on mobile GPUs through adaptive KV cache layouts, hardware-aligned packing, and an integer-native pipeline that defers type conversion outside the compute hot path. Extensive evaluation demonstrates that BitLLM delivers significant speedups (e.g., 1.78× over LLAMA.CPP on average with fixed context length) and substantial per-token energy reductions (up to 88.9%), while effectively eliminating latency spikes during KV cache growth to ensure predictable long-context decoding. Future work includes combining other low-bit quantization schemes with pruning co-optimization and support for concurrent workloads on heterogeneous mobile hardware (NPUs, TPUs).

Acknowledgments The authors would like to thank the anonymous reviewers and the shepherd for their valuable and constructive feedback. This work was supported in part by the National Science Foundation (NSF) under the awards of CCF-2428108, CCF-2333895, OAC-2403090, CSR-2341378, IIS-2540429, CNS2555319, and CCF-2553684. Any errors and opinions are not those of the NSF and are attributable solely to the author(s).

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece.

References [1] Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems 6 (2024), 351–366. [2] Jorge Albericio, Alberto Delmás, Patrick Judd, Sayeh Sharify, Gerard O’Leary, Roman Genov, and Andreas Moshovos. 2017. Bit-pragmatic deep neural network computing. In Proceedings of the 50th annual IEEE/ACM international symposium on microarchitecture. 382–394. [3] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 12562–12584. [4] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15. [5] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. 2023. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems 36 (2023), 4396–4429. [6] Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, et al. 2024. Db-llm: Accurate dual-binarization for efficient llms. arXiv preprint arXiv:2402.11960 (2024). [7] Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, and Stylianos I Venieris. 2024. Progressive mixed-precision decoding for efficient llm inference. arXiv preprint arXiv:2410.13461 (2024). [8] Le Chen, Dahu Feng, Erhu Feng, Rong Zhao, Yingrui Wang, Yubin Xia, Haibo Chen, and Pinjie Xu. 2025. HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs platform with Heterogeneous AI Accelerators. arXiv preprint arXiv:2501.14794 (2025). [9] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, and Luis Ceze. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI. 578–594. [10] FasterTransformer contributors. 2022. FasterTransformer. https:// github.com/NVIDIA/FasterTransformer [11] LLaMA.CPP contributors. 2023. LLaMA.CPP. https://github.com/ ggerganov/llama.cpp [12] TensorFlow contributors. 2017. LiteRT. https://ai.google.dev/edge/ litert [13] TensorRT-LLM contributors. 2022. TensorRT-LLM. https://github.com/ NVIDIA/TensorRT-LLM [14] Meghan Cowan, Thierry Moreau, Tianqi Chen, James Bornholt, and Luis Ceze. 2020. Automatic generation of high-performance quantized machine learning kernels. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization. 305–316. [15] Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023). [16] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359. [17] Alberto Delmas, Patrick Judd, Sayeh Sharify, and Andreas Moshovos. 2017. Dynamic stripes: Exploiting the dynamic precision requirements of activation values in neural networks. arXiv preprint arXiv:1706.00504 (2017). [18] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8 (): 8-bit matrix multiplication for transformers at scale.

Hangyu Zheng et al. Advances in neural information processing systems 35 (2022), 30318– 30332. [19] Yaoyao Ding, Bohan Hou, Xiao Zhang, Allan Lin, Tianqi Chen, Cody Hao Yu, Yida Wang, and Gennady Pekhimenko. 2026. Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 281–297. [20] Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. 2024. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643 (2024). [21] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323 (2022). [22] Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. 2025. Marlin: Mixed-precision auto-regressive parallel inference on large language models. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. 239–251. [23] Google team. 2026. LiteRT-LM. https://github.com/google-ai-edge/ LiteRT-LM [24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). [25] Jiexiong Guan, Zhenqing Hu, Christos D Antonopoulos, Nikolaos Bellas, Spyros Lalis, Evgenia Smirni, Gang Zhou, Gagan Agrawal, and Bin Ren. 2025. TMModel: Modeling Texture Memory and Mobile GPU Performance to Accelerate DNN Computations. In Proceedings of the 39th ACM International Conference on Supercomputing. 205–220. [26] Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, Longfei Li, Jinjie Gu, and Chenyi Zhuang. 2023. Intelligent virtual assistants with llm-based process automation. arXiv preprint arXiv:2312.06677 (2023). [27] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems 37 (2024), 1270–1303. [28] Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng, Tianchi Hu, Guanglin Li, Guipeng Hu, Junsong Wang, et al. 2026. M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1151–1167. [29] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog 1, 3 (2023), 3. [30] Joo Seong Jeong, Jingyu Lee, Donghyun Kim, Changmin Jeon, Changjin Jeong, Youngki Lee, and Byung-Gon Chun. 2022. Band: coordinated multi-dnn inference on heterogeneous mobile processors. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. 235–247. [31] Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor M Aamodt, and Andreas Moshovos. 2016. Stripes: Bit-serial deep neural network computing. In Microarchitecture (MICRO), 2016 49th Annual IEEE/ACM International Symposium on. IEEE, 1–12. [32] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626.

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs [33] Royson Lee, Stylianos I Venieris, Lukasz Dudziak, Sourav Bhattacharya, and Nicholas D Lane. 2019. Mobisr: Efficient on-device super-resolution through heterogeneous mobile processors. In The 25th Annual International Conference on Mobile Computing and Networking. 1–16. [34] Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, and Qin Xie. 2024. Transformer-lite: High-efficiency deployment of large language models on mobile phone gpus. arXiv preprint arXiv:2403.20041 (2024). [35] Xiangyu Li, Yuanchun Li, Yuanzhe Li, Ting Cao, and Yunxin Liu. 2024. FlexNN: Efficient and Adaptive DNN Inference on MemoryConstrained Edge Devices. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 709–723. [36] Ziming Li, Jiatan Huang, Xiaoguang Guo, Guilin Wang, and Chuxu Zhang. 2026. Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents. arXiv preprint arXiv:2605.06908 (2026). [37] Ziming Li, Xiaoming Wu, Zehong Wang, Jiazheng Li, Yijun Tian, Jinhe Bi, Yunpu Ma, Yanfang Ye, and Chuxu Zhang. 2026. Graph is a Substrate Across Data Modalities. arXiv preprint arXiv:2601.22384 (2026). [38] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, WeiChen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems 6 (2024), 87–100. [39] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532 (2024). [40] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. Proceedings of Machine Learning and Systems 7 (2025). [41] llamafile contributors. 2023. llamafile. https://github.com/MozillaOcho/llamafile [42] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016). [43] Guanhong Miao, A Adam Ding, and Samuel S Wu. 2022. Real-time privacy preserving disease diagnosis using ECG signal. arXiv preprint arXiv:2202.03652 (2022). [44] MLC team. 2023-2025. MLC-LLM. https://github.com/mlc-ai/mlc-llm [45] Wei Niu, Gagan Agrawal, and Bin Ren. 2024. Sod2: Statically optimizing dynamic deep neural network execution. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 386–400. [46] Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, and Bin Ren. 2021. DNNFusion: accelerating deep neural networks execution with advanced operator fusion. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 883–898. [47] Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang, Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. 2020. PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 907–922. [48] Wei Niu, Md Musfiqur Rahman Sanim, Zhihao Shu, Jiexiong Guan, Xipeng Shen, Miao Yin, Gagan Agrawal, and Bin Ren. 2024. SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol. 3. La Jolla, CA, USA. doi:10.1145/3620666.3651384 [49] ollama contributors. 2023. ollama. https://ollama.com/ [50] ONNX Runtime developers. 2018. ONNX Runtime. https://github. com/microsoft/onnxruntime.

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece. [51] Yunjie Pan, Yongyi Yang, Hanmei Yang, and Scott Mahlke. 2026. SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’26). [52] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of machine learning and systems 5 (2023), 606–624. [53] Pytorch. 2023. On-device AI across mobile, embedded and edge for PyTorch. pytorch.org/executorch [54] Qualcomm. 2016. Snapdragon Profiler. https://developer.qualcomm. com/software/snapdragon-profiler. https://developer.qualcomm.com/ software/snapdragon-profiler [55] Muhammad Kashif Saeed, Alanoud Al Mazroa, Bandar M Alghamdi, Fouad Shoie Alallah, Abdulrhman Alshareef, and Ahmed Mahmud. 2024. Predictive analytics of complex healthcare systems using deep learning based disease diagnosis model. Scientific Reports 14, 1 (2024), 27497. [56] Sayeh Sharify, Alberto Delmas Lascorz, Kevin Siu, Patrick Judd, and Andreas Moshovos. 2018. Loom: Exploiting weight and activation precisions to accelerate convolutional neural networks. In Proceedings of the 55th Annual Design Automation Conference. ACM, 20. [57] Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Joon Kyung Kim, Vikas Chandra, and Hadi Esmaeilzadeh. 2018. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 764–775. [58] Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, and Hengyu Meng. 2025. Efficient llm inference on cpus. In Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques. Springer, 33–46. [59] Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, and Yanzhi Wang. 2024. Agile-Quant: ActivationGuided Quantization for Faster Inference of LLMs on the Edge. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18944–18951. [60] Xuan Shen, Weize Ma, Jing Liu, Changdi Yang, Rui Ding, Quanyi Wang, Henghui Ding, Wei Niu, Yanzhi Wang, Pu Zhao, Jun Lin, and Jiuxiang Gu. 2025. QuartDepth: Post-Training Quantization for RealTime Depth Estimation on the Edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [61] Zhihao Shu, Md Musfiqur Rahman Sanim, Hangyu Zheng, Kunxiong Zhu, Miao Yin, Gagan Agrawal, and Wei Niu. 2026. FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). [62] Zhihao Shu, Xiaowei Yu, Zihao Wu, Wenqi Jia, Yinchen Shi, Miao Yin, Tianming Liu, Dajiang Zhu, and Wei Niu. 2024. Real-time coreperiphery guided ViT with smart data layout selection on mobile devices. Advances in Neural Information Processing Systems 37 (2024), 95744–95763. [63] Geunsoo Song, Hoeseok Yang, and Youngmin Yi. 2026. oFFN: Outlier and Neuron-aware Structured FFN for Fast yet Accurate LLM Inference. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1301–1315. [64] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 590–606. [65] Fuwen Tan, Royson Lee, Łukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, and Brais

ASPLOS ’27, April 11–15, 2027, Heraklion, Greece. Martinez. 2024. Mobilequant: Mobile-friendly quantization for ondevice language models. In Findings of the Association for Computational Linguistics: EMNLP 2024. 9761–9771. [66] Karan Taneja, Pratyusha Maiti, Sandeep Kakar, Pranav Guruprasad, Sanjeev Rao, and Ashok K Goel. 2024. Jill Watson: A Virtual Teaching Assistant powered by ChatGPT. In International Conference on Artificial Intelligence in Education. Springer, 324–337. [67] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024). [68] Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. 10–19. [69] Minh Duc Vu, Han Wang, Zhuang Li, Jieshan Chen, Shengdong Zhao, Zhenchang Xing, and Chunyang Chen. 2024. GPTVoiceTasker: LLM-powered virtual assistant for smartphone. arXiv preprint arXiv:2401.14268 (2024). [70] Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, and Furu Wei. 2025. Bitnet. cpp: Efficient edge inference for ternary llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 9305–9322. [71] Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, et al. 2024. Ladder: Enabling efficient {Low-Precision} deep learning computing through hardware-aware tensor transformation. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 307–323. [72] Rongxiang Wang, Zhiming Xu, and Felix Xiaozhu Lin. 2024. WhisperFlow: speech foundation models in real time. arXiv preprint arXiv:2412.11272 (2024). [73] Tuowei Wang, Ruwen Fan, Minxing Huang, Zixu Hao, Kun Li, Ting Cao, Youyou Lu, Yaoxue Zhang, and Ju Ren. 2024. Ripple: Accelerating LLM Inference on Smartphones with Correlation-Aware Neuron Management. arXiv preprint arXiv:2410.19274 (2024). [74] Tuowei Wang, Ruwen Fan, Minxing Huang, Zixu Hao, Kun Li, Ting Cao, Youyou Lu, Yaoxue Zhang, and Ju Ren. 2025. Neuralink: Fast on-Device LLM Inference with Neuron Co-Activation Linking. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 147–162. [75] Yuke Wang, Boyuan Feng, and Yufei Ding. 2022. QGTC: accelerating quantized graph neural networks via GPU tensor core. In Proceedings of the 27th ACM SIGPLAN symposium on principles and practice of parallel programming. 107–119. [76] Zhaode Wang, Jingbang Yang, Xinyu Qian, Shiwen Xing, Xiaotang Jiang, Chengfei Lv, and Shengyu Zhang. 2024. MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices. In Proceedings of the 6th ACM International Conference on Multimedia in Asia Workshops. 1–7. [77] Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2025. T-mac: Cpu renaissance via table lookup for low-bit llm deployment on edge. In Proceedings of the Twentieth European Conference on Computer Systems. 278–292. [78] Jianyu Wei, Qingtao Li, Shijie Cao, Lingxiao Ma, Zixu Hao, Yanyong Zhang, Xiaoyan Hu, and Ting Cao. 2025. T-MAN: Enabling End-toEnd Low-Bit LLM Inference on NPUs via Unified Table Lookup. arXiv preprint arXiv:2511.11248 (2025).

Hangyu Zheng et al. [79] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning. [80] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023). [81] Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2024. Empowering 1000 tokens/second ondevice llm prefilling with mllm-npu. arXiv preprint arXiv:2407.05858 (2024). [82] Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2025. Fast On-device LLM Inference with NPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 445–462. [83] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. Powerinfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282 (2024). [84] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [85] Hyunho Yeo, Chan Ju Chong, Youngmok Jung, Juncheol Ye, and Dongsu Han. 2020. Nemo: enabling neural-enhanced video streaming on commodity mobile devices. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1–14. [86] Chong Yu, Tao Chen, and Zhongxue Gan. 2023. Boost transformerbased language models with GPU-friendly sparsity and quantization. In Findings of the Association for Computational Linguistics: ACL 2023. 218–235. [87] Li Lyna Zhang, Shihao Han, Jianyu Wei, Ningxin Zheng, Ting Cao, Yuqing Yang, and Yunxin Liu. 2021. nn-Meter: towards accurate latency prediction of deep-learning model inference on diverse edge devices. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. 81–93. [88] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385 (2024). [89] Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Zhangyang Wang. 2024. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems 6 (2024), 381–394. [90] Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems 6 (2024), 196–209. [91] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, and Koushik Sen. 2020. Ansor: Generating {High-Performance} tensor programs for deep learning. In 14th USENIX symposium on operating systems design and implementation (OSDI 20). 863–879. [92] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583. [93] Kunxiong Zhu, Zhihao Shu, Hangyu Zheng, Minghai Qin, Miao Yin, Gagan Agrawal, and Wei Niu. 2026. EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models. arXiv preprint arXiv:2609.31551 (2026).

Related documents

• BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs#1536377
• Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method#984875
• SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability#1325641
• HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference#819897
• HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference#809901