SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions Feiyang Chen, Haibo Chen IPADS, Shanghai Jiao Tong University
arXiv:2606.16332v1 [cs.DC] 15 Jun 2026
Abstract
However, they do not study how to use SME systematically across the entire LLM inference process. In particular, it remains unclear how a runtime should schedule matrix extensions and CPU vector cores across inference phases such as prefill and decode, and across operators such as FFN, MoE and attention. This paper presents the first comprehensive characterization on CPUs with scalable matrix extensions for LLM inference. To reason about when SME units should be used, we build a unified roofline-based model that captures both the distinct computing throughput of SME and CPU cores and their shared memory bandwidth. For each LLM operator and shape, the model estimates whether execution is compute-bound, memory-bound, or in a backendasymmetric ridge-crossing region, and uses this classification to predict whether the operator should run on SME units, CPU cores, or both. We identify that treating SME as a drop-in replacement for CPU matrix kernels is insufficient. There are two primary reasons for this. First, for large matrix-heavy operators in prefill phases, SME and CPU cores can both contribute to throughput, and using only one backend leaves useful hardware idle. Second, different models and input requests can shift an operator between SME-friendly and CPU-friendly regimes, which causes a single fixed kernel replacement to underperform across the inference process. To address this problem, we introduce SMEPilot, an accelerated inference engine for CPUs with scalable matrix extensions. SMEPilot provides a unified framework that schedules SME and CPU cores across prefill and decode phases, and across operators with different shapes and hardware affinities. For each inference request, SMEPilot uses the roofline classification to generate an execution plan that schedules CPU-only, SME-only, or mixed SME+CPU execution for each operator. After the execution plan is generated, SMEPilot provides a performant runtime to execute the plan on SME units and CPU cores. However, attaining high performance requires more than choosing the right plan: the runtime must preserve parallelism, hardware utilization, and data locality while executing the plan. We identify three execution gaps that prevent the roofline-guided plan from realizing its predicted benefit.
Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide highthroughput matrix execution within the CPU. For LLM inference, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-toend inference performance by up to 3.94×.
1
Introduction
The rapid progress of large language models (LLMs) has made generative AI a common component of interactive applications, from chat assistants and AI agents to multimodal generation [25, 27, 41]. As these applications span platforms from edge devices to data centers, CPUs continue to play a critical role in executing diverse AI tasks owing to their widespread availability and cost efficiency [16, 35, 36]. At the same time, the CPU itself is evolving for LLMs. Modern CPUs are adding matrix extensions, such as Arm Scalable Matrix Extension (SME) [4], that provide highthroughput matrix instructions inside a conventional CPU package. They add specialized matrix registers and instructions for computing 2D tiled matrix operations, such as outer products, directly on the CPU. These extensions promise a middle ground between GPU-style accelerators and ordinary vector cores: they expose more matrix throughput without requiring the application to leave the CPU, reusing its easy and mature ecosystem. Prior work has studied how to use Scalable Matrix Extensions to accelerate specific kernels. For example, ARM SME has been characterized for dense matrix multiplication [12] and sparse matrix multiplication [23].
• Spatial utilization gap. A policy that assigns each operator entirely to a single execution target can leave useful hardware idle. Large GEMM operators expose 1
enough arithmetic intensity to benefit from both SME and CPU cores, but assigning the entire operator to only one side wastes the throughput of the other. • Temporal bubble gap. LLM inference operators, especially attention, interleave phases with different hardware affinity. Matrix multiplication such as 𝑄𝐾 ⊤ can benefit from SME, while softmax, masking, reductions, and other vector-heavy phases are better suited to CPU cores. If these phases are executed sequentially at coarse operator boundaries, one hardware unit waits while the other runs, creating bubbles that are invisible to a per-operator roofline model. • Layout compatibility gap. SME kernels require packed, tile-friendly matrix layouts to expose their throughput, whereas the rest of the LLM pipeline often consumes and produces conventional tensor layouts. Naively packing inputs before every SME invocation adds memory traffic and disrupts data locality, reducing the effective arithmetic intensity assumed by the roofline model.
CPU cores together for large matrix-heavy prefill operators. For decode phases, SMEPilot achieves up to 3.48× speedup over CPU-only inference. This demonstrates that SMEPilot can still improve hardware utilization in regimes with less computation and stronger memory bandwidth pressure.
2
Preliminaries
2.1
LLM Inference
Modern LLMs are commonly built from Transformer blocks, where each layer alternates between attention and feedforward network (FFN) computation [34]. From a system perspective, these blocks contain a mixture of operator types. Both of the attention and FFN blocks contains two kinds of operators: matrix operations such as matrix multiplication, and non-linear operations such as softmax, normalization, and activation functions. This mixture makes CPU matrix extensions promising for LLM inference, because they can accelerate the matrix-heavy parts, but also non-trivial to use, because not every phase benefits from matrix throughput. LLM inference also has two execution phases with different shapes: prefill and decode. During prefill, the model processes all prompt tokens at once and produces KVcache entries for them [2, 22, 42]. The token dimension is therefore large, which increases arithmetic intensity and exposes substantial matrix parallelism in attention and FFN operators. During decode, the model usually processes one new token per step, or a small number of tokens when using batching [39] or multi-token prediction [9, 17]. These operators are therefore more memory-bound: they read large weight and KV-cache matrices but perform little computation per request [28]. Therefore, during LLM inference, different computation characteristics occurs due to the different phases and different operators, which poses challenges for utilizing matrix extensions efficiently.
To bridge these gaps between the roofline-guided plan and runtime execution, SMEPilot adopts three techniques. • Tile-level work partitioning. To bridge the spatial utilization gap, SMEPilot partitions matrix work across SME and CPU cores at tile granularity. This allows both backends to contribute within the same operator while respecting the tile shapes required by their optimized kernels. • Phase-aware pipeline execution. To bridge the temporal bubble gap, SMEPilot pipelines SME-friendly matrix phases with CPU-friendly vector phases. This lets attention phases such as matrix multiplication and softmax/reduction overlap when their dependencies allow, reducing idle time on both sides. • Layout-aware runtime. To bridge the layout compatibility gap, SMEPilot tracks tensor layouts as runtime state and performs layout-aware graph conversion. Static weights are packed once off the request critical path, while online activations such as KV-cache entries are packed at the producer side when downstream SME consumers can reuse them. This avoids repeated packing and wasted memory bandwidth on SME critical paths.
2.2
Scalable Matrix Extension
The rising demands of AI workloads on both cloud and edge platforms have led to matrix extensions being added to modern CPU architectures. Arm Scalable Matrix Extension (SME) is an Armv9-A architecture extension that adds matrixoriented execution support to the CPU [4, 5]. SME introduces a specialized outer-product instruction and a new matrix register array storage, significantly increasing the matrix throughput of Arm CPUs [12, 29]. SME has been supported in a range of CPUs. Apple’s M4 was the first publicly available chip reported to support SME, and M4-family CPUs are now used across Apple mobile and laptop platforms [29]. Recent flagship mobile Arm platforms are also adopting SME. For example, MediaTek’s Dimensity 9500 introduces Armv9.3 support with SME2 instructions, and Qualcomm Snapdragon 8 Elite Gen 5 uses third-generation Oryon CPU cores that also support SME.
SMEPilot delivers state-of-the-art performance on Phone, PC, and server CPU platforms. In end-to-end evaluations, SMEPilot achieves up to 3.94× end-to-end speedup over llama.cpp [16] across the evaluated device-model-workload configurations, including both dense models and MoE models. For prefill phases, SMEPilot improves FFN GEMM by 1.42–1.67× over highly optimized Apple Accelerate [3](SME) and 6.06–7.43× over GGML [15](CPU), while improves attention by 1.56–3.50× over a FlashAttention CPU [11]. These gains come from SMEPilot’s ability to use SME and 2
Core 0
Core 2
Processing Grid SME Unit
ZA matrix register
Z vector register
Cluster Interface
not increase the memory roof. All three configurations sustain similar write bandwidth. Thus, adding SME raises the compute ceiling but does not raise the memory-bandwidth roof. These two properties make SME-enabled CPUs different from both homogeneous multicore CPUs and discrete accelerators. Compute resources can be added within one operator, but memory bandwidth remains coupled across them.
SME Unit
Core 1
Core 3 Z vector register
Fig. 1. The hardware design of CPU with SME. Left: an CPU cluster attaches ordinary CPU cores and an SME unit. Right: the SME unit executes outer products instruction from Z vector registers and output to ZA matrix registers.
3.2
The coupled structure of SME and CPU cores creates a placement problem that a conventional roofline model does not capture. A conventional roofline view can classify whether an operator is compute-bound or memory-bound on a single hardware target, but LLM inference on SMEenabled CPUs must choose among CPU-only, SME-only, and mixed SME+CPU execution. We therefore build a coupled roofline model with a shared memory-bandwidth roof and multiple compute ceilings for CPU cores, SME units, and mixed SME+CPU execution. For an operator 𝑜, let 𝑊𝑜 denote its floating-point work and 𝑄𝑜 denote its useful memory traffic. Its arithmetic intensity is 𝐼𝑜 = 𝑊𝑜 /𝑄𝑜 . For each execution target ℎ ∈ {CPU, SME, MIX}, let 𝑃ℎ denote the measured compute throughput. The mixed target represents concurrent SME+CPU execution and has a compute ceiling close to 𝑃SME +𝑃 CPU . Since all targets share the same memory system, they use a common memory bandwidth 𝐵. The attainable performance of target ℎ is modeled as
On the cloud side, Arm-based server CPUs such as newer Kunpeng 920 also support SME.
3
Roofline Characterization of SME-enabled CPUs
Scalable Matrix Extensions reshape the performance structure of CPU inference. In order to systematically analyze the performance of different LLM operators on the CPU with SME, we design a unified roofline model that can capture the performance characteristics of both SME and CPU cores. Based on this roofline model, we characterize the performance of different LLM operators on the CPU with SME, and derive the design principles for SMEPilot. 3.1
Multi-Ceiling Roofline Model
SME and CPU Cores: Additive Compute, Shared Bandwidth
Although SME instructions are exposed through the CPU programming model, the underlying execution resources are distinct from ordinary CPU vector cores, as shown in Figure 1. SME provides specialized matrix state and outer-product instructions for high-throughput tiled matrix computation, while CPU cores continue to provide general-purpose scalar and vector execution. Two properties determine how these resources should be used for LLM inference. Near-additive matrix throughput. For matrix-heavy operators with enough independent work, SME units and CPU cores can both contribute useful computation. As shown in Figure 3, two SME workers alone reach 2.92 TFLOP/s on FP16 matrix work and eight CPU-core workers reach 1.85 TFLOP/s. Running them together reaches 4.27 TFLOP/s, which is close to the sum of the individual throughputs. This indicates that SME and CPU cores should not be treated as mutually exclusive backends for large matrix operators. Shared memory bandwidth. The same additivity does not hold for memory traffic. SME units and CPU cores share the cache hierarchy and memory controller. As shown in Figure 3, CPU cores reach 247 GB/s read bandwidth, SME reaches 224 GB/s, and mixed SME+CPU execution does
𝑅ℎ (𝑜) = min(𝑃ℎ , 𝐵𝐼𝑜 ). The ridge point of target ℎ is 𝑟𝑖𝑑𝑔𝑒
𝐼ℎ
=
𝑃ℎ . 𝐵
Because 𝑃CPU < 𝑃SME < 𝑃MIX while 𝐵 is shared, the model contains one memory roof but multiple compute ceilings. As shown in Figure 2, unlike a traditional roofline model with only one ridge point, this multi-ceiling roofline model partitions the performance space into three regimes: memory-bound, compute-bound, and backend-asymmetric ridge-crossing. These three regimes correspond to three different placement plans for LLM operators on SME-enabled CPUs. Memory-bound regime. Operators below the CPU ridge point are memory-bound on all targets. Since SME does not increase memory bandwidth, moving these operators to SME cannot reduce the dominant cost. This regime includes single-token decode GEMV and decode attention, where the operator reads large weight or KV-cache matrices but performs little computation per request. 3
A unified roofline model
ridge-crossing
memory-bound
10k
Inference Plan
Compute-bound
Attainable performance (GFLOP/s)
Prefill FFN
SME+CPU cores Batch decode GEMM
3k
SME
CPU cores Batch decode Attention 1k
Prefill attention Decode attention
300 Decode FFN 100 1
3
10
30
100
4096 tokens
1 tokens
16 tokens
Attention
Attention
Attention
QK Linear
QK Linear
QK Linear
Softmax
Softmax
Softmax
PV linear
PV linear
PV linear
FFN
FFN
FFN
Up Linear
Up Linear
Up Linear
SwiGLU
SwiGLU
SwiGLU
Down Linear
Down Linear
Down Linear
out
out
out
Arithmetic intensity (FLOP/Byte)
CPU cores
SME
mixed
Fig. 2. Left: SME+CPU roofline view of representative LLM operators. CPU cores, SME, and mixed SME+CPU execution share the same memory bandwidth but expose different compute peaks, creating different ridge points: memory-bound, compute-bound and ridge-crossing. Right: The inference plan for different input shapes. Operators are placed into different backends based on their position estimated by the unified roofline model. 300
4 3 2 1 0
CPU
SME
CPU+SME
(a) Computing throughput
SME
Read bandwidth Write bandwidth
200
Latency (ms)
Bandwidth (GB/s)
Throughput (TFLOP/s)
5
100
0
CPU
SME
CPU+SME
150 100 50 20 15 10 5 0
M4Pro
CPU
MediaTek Dimensity 9500
Fig. 4. SME has lower throughput for non-linear vector operations. For a 4096 × 4096 fastexp kernel, CPU cores are faster than SME on both Apple M4 Pro and MediaTek Dimensity 9500.
(b) Memory bandwidth
Fig. 3. The computing and memory characteristics of SME and CPU cores. SME and CPU cores provide near-additive matrix throughput but share memory bandwidth.
SME-only execution is preferable for this region. These operators favor SME execution for higher computing throughput, but mixed SME+CPU execution provides limited additional benefit because memory bandwidth becomes the bottleneck. Compute-bound regime. Operators above the SME ridge point are compute-bound on both SME and CPU cores. In this region, the memory bandwidth is no longer the limiting resource, so the near-additive compute throughput of SME and CPU cores can be exploited. Large prefill FFN and attention matrix phases fall into this region. These operators benefit from mixed SME+CPU execution, provided that the runtime can utilize parallelism within the operator. Non-linear operations. Except for the linear operations, LLM inference also contains non-linear operations such as
CPU execution is preferable for this region because we observe that SME kernels typically has lower memory utilization than CPU cores due to higher instruction latency and lack of speculative execution. Backend-asymmetric ridge-crossing regime. Operators between the CPU and SME ridge points are computebound on CPU cores but memory-bound on SME. This regime is specific to a multi-ceiling roofline model. SME improves performance in this region not by reaching its peak compute throughput, but by lifting the operator above the CPU compute ceiling until it approaches the shared memory bandwidth. Batched decode GEMM and some grouped-query attention shapes fall into this region. 4
softmax, normalization and activation. We always place these non-linear operations on CPU cores, because SME has lower throughput for these operations than CPU cores. As shown in Figure 4, for a 4096 × 4096 fastexp kernel, SME is 4.10× slower than CPU cores on Apple M4 Pro and 3.26× slower on MediaTek Dimensity 9500 for this non-linear kernel. This is because SME has low throughput and higher latency for SIMD instructions, and non-linear operations often compose of a long sequence of SIMD instructions with strong data dependencies. To summarize, as shown in the right panel of Figure 2, this unified roofline model characterizes LLM operators across different performance regimes and guides the general placement strategy for SME-enabled CPUs.
SME
SME
CPU cores
CPU cores
tokens
𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛
tokens
SME
𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛
3.3
𝑤𝑒𝑖𝑔ℎ𝑡 𝑇 Activation partition
CPU cores
SME
CPU cores
𝑤𝑒𝑖𝑔ℎ𝑡 𝑇 Weight partition
Fig. 5. Tile partition closes the spatial utilization gap for compute-bound matrix operators. The planner uses measured SME and CPU rates, together with the worker budget and contention guard, to assign tiles so that both backends finish at similar times.
Scheduling Insights from the Roofline Analysis
The coupled roofline model not only provides placement plans for different operators, but also reveals two general insights for the runtime of SMEPilot. Insight 1: Mixed execution requires intra-operator scheduling. Operator-level placement is too coarse for SMEenabled CPUs. For operators in compute bound regime, the coupled model predicts that the best target is mixed SME+CPU execution. However, assigning the whole operator to either backend cannot realize this bound: CPU-only execution misses SME throughput, while SME-only execution leaves ordinary CPU cores idle. Reaching the mixed ceiling therefore requires the runtime to explore parallelism within one operator and achieve high utilization for both hardware sides. Insight 2: Shape-dependent placement requires dynamic scheduling. The same operator can occupy different regions of the coupled roofline as the request shape changes. For example, an FFN layer may be memory-bound during single-token decode, ridge-crossing under batched decode, and compute-bound during long-context prefill. Attention similarly shifts with sequence length, batch size, head configuration, and KV-cache length. Therefore, we cannot attach a fixed backend choice to each operator. Instead, the runtime must generate a shape-aware execution plan online, selecting CPU-only, SME-only, or mixed SME+CPU execution according to the operator’s current arithmetic intensity and available intra-operator parallelism. These two insights distinguish SMEPilot from a drop-in SME kernel replacement. SMEPilot does not simply decide whether an operator should use SME; it decides which tiles should use SME and how that decision changes with the current request shape.
4
SMEPilot Design
The characterization in Section 3 identifies the placement regimes for different operators. Based on this characterization, SMEPilot designs a unified runtime that turns rooflineguided placement into a concrete, performant execution plan. To realize intra-operator mixed SME+CPU execution with high hardware utilization, we design two techniques for FFN and attention operators: tile-level work partitioning (Section 4.1) and phase-aware pipeline execution (Section 4.2). To support dynamic scheduling while preserving data locality, we design a layout-aware runtime (Section 4.3) and an online planner (Section 4.4). 4.1
Tile-level Work Partitioning
Problem: spatial utilization gap. Large prefill GEMMs and FFN layers expose enough independent matrix work to use both SME and CPU cores. However, a whole-operator decision can choose only one backend, while a naive split can leave the faster backend waiting for the slower one. We need a partition plan that achieve high utilization for both SME and CPU cores. Technique: partition independent output tiles. SMEPilot partitions a GEMM operator into output tiles and assigns a subset of tiles to SME workers and the remaining tiles to CPU-core workers. For a matrix multiplication 𝐶𝑀 ×𝑁 = 𝐴𝑀 ×𝐾 𝐵𝐾 ×𝑁 , SMEPilot partitions only the output dimensions of 𝐶. This choice keeps different workers independent: each worker computes a disjoint set of output tiles, so there is no cross-worker reduction or synchronization. In contrast, partitioning the reduction dimension 𝐾 would require partial 5
sums from different workers to be merged, adding synchronization and memory traffic. There are two natural output-dimension partition strategies, as shown in Figure 5. A token-dimension partition splits the 𝑀 dimension, so different workers process different token rows. An output-channel partition splits the 𝑁 dimension, so different workers compute different output channels. SMEPilot chooses the partition dimension based on the operator shape. When the token dimension is larger, it partitions along 𝑀; when the output-channel dimension is larger, it partitions along 𝑁 . Splitting the larger output dimension avoids highly elongated regions, which improves cache locality, and makes it easier to keep each worker’s region aligned with the native tile shapes of the SME and CPU kernels. The partition ratio is determined by the online planner(describe in Section 4.4). MoE expert partitioning. Mixture-of-Experts (MoE) layers introduce another form of uneven matrix work. After the router assigns tokens to experts, the number of tokens received by each expert is often skewed: some experts receive only a few tokens, while others receive many. SMEPilot therefore treats each expert FFN as a separate matrix operator whose token dimension is determined by the routed token count. Experts with only a small number of tokens are kept on CPU cores, because their thin matrix shapes are memory-bound and cannot amortize SME setup, packing, and tile-alignment overheads. Experts with many tokens are classified by the roofline model as larger GEMM operators and can use SME, or mixed SME+CPU execution, when their arithmetic intensity and tile count are sufficient. This expertlevel decision avoids forcing the same backend choice on both cold and hot experts in the same MoE layer. Tile-level work partitioning is used for compute-bound operators such as prefill FFN layers and large attention matrix phases. For memory-bound decode GEMV-like operators, SMEPilot avoids mixed partitioning unless the effective token width becomes large enough to expose useful parallel matrix work. 4.2
SME QK^T
CPU cores softmax
SME PV
t0
t1
t2
t3
QK 0
QK 1
QK 2
Softmax 0
Softmax 1
Softmax 2
PV 0
PV 1
t4
PV 2
Fig. 6. Pipeline execution closes the temporal bubble gap in attention. Each tile moves through SME-friendly 𝑄𝐾 ⊤ , cores-friendly softmax, and SME-friendly 𝑃𝑉 stages. Tilelevel dependencies allow different tiles to occupy different hardware-specialized stages instead of forcing an operatorwide barrier after each phase. Existing fused attention algorithms, such as FlashAttention [11], reduce memory traffic by computing softmax online and avoiding materialized attention matrices. However, their primary goal is memory efficiency; they do not by themselves address the hardware-affinity mismatch between SME-friendly matrix phases and CPU-friendly vector phases on SME-enabled CPUs. Technique: inter-phase pipelining. SMEPilot addresses this gap with inter-phase pipelining. Instead of executing attention as three operator-wide phases, SMEPilot allows different attention tiles to occupy different phases at the same time. Each tile still follows the dependency order 𝑄𝐾 ⊤ → softmax → 𝑃𝑉 , but independent tiles can advance through the pipeline concurrently. As shown in Figure 6, the two matrix phases are assigned to SME workers, while the softmax phase is assigned to CPU-core workers. Once the 𝑄𝐾 ⊤ result for a tile is available, its softmax stage can start without waiting for the entire operator to finish the 𝑄𝐾 ⊤ phase. This allows SME to compute the matrix phase of later tiles while CPU cores process the vector phase of earlier tiles. Tile sizing. The tile size controls the effectiveness of the pipeline. Very small tiles expose more overlap opportunities, but they increase scheduling overhead and reduce kernel efficiency. Very large tiles improve per-kernel efficiency, but they increase pipeline prologue and epilogue time and make the stage times harder to balance. SMEPilot therefore chooses tile sizes using the offline profile and online planning: it estimates the SME time for the matrix stages and the CPUcore time for the softmax stage based on the current request shape and the profiled SME and CPU throughputs, then selects a tile shape whose stage times are small enough to reduce long stalls while still preserving kernel-aligned matrix tiles. Mask-aware matrix phases. Attention masks introduce another source of wasted work in the matrix stages of the pipeline. Causal and sliding-window masks can make entire score blocks irrelevant, but a conventional GEMM
Phase-aware Pipeline Execution
Problem: temporal bubble gap. Attention is not a single homogeneous matrix operator. It contains matrix-heavy phases, such as 𝑄𝐾 ⊤ and 𝑃𝑉 , and vector-heavy phases, such as masking, softmax, and reductions. These phases have different hardware affinity: the matrix phases can benefit from SME throughput, whereas the vector-heavy phases are better executed by CPU cores. A coarse phase-level schedule therefore creates temporal bubbles. If the runtime waits for 𝑄𝐾 ⊤ to finish before starting softmax, CPU cores sit idle during the matrix phase; if it waits for softmax work before starting 𝑃𝑉 , SME sits idle during the vector phase. These idle periods move real execution away from the performance predicted by the roofline model. 6
SME K packed
still computes these masked elements before discarding them. SMEPilot uses mask-aware matrix kernels inside the pipeline to skip tiles that are fully masked. For 𝑄𝐾 ⊤ , the mask determines which score blocks are produced; for 𝑃𝑉 , the same mask determines which score blocks participate in the reduction. This optimization reduces the amount of matrix work entering the pipeline without changing the tile-level dependency structure.
token 0
token 0
token 1
token 1
token 2
token 2
token 3
token 3
SME token 1 Vector token 2 length
token 4
SME K packed
token 3
token 4
token 4
token 5
token 5
token 5
token 6
token 6
token 6
token 7
token 7
token 7
SME K Cache
CPU KV Cache
4.3
SME Vector length
token 0
SME V Cache
Layout-aware Runtime Fig. 7. Producer-side packing for SME-friendly KV cache layout. A standard CPU KV cache stores each token contiguously in row-major order. SMEPilot stores K and V entries in SME-specific packed layouts at production time, grouping elements according to the SME vector length and the Kpacked tile shape required by downstream attention kernels.
Problem: layout compatibility gap. SME kernels prefer packed, tile-friendly layouts, but LLM runtimes often store tensors in standard row-major or framework-native layouts. A naive runtime therefore packs inputs immediately before each SME kernel. This conversion adds memory traffic around the useful matrix computation and places layout conversion on the critical path. For decode FFN and attention, layout conversion can take a similar amount of time as the SME kernel itself, substantially reducing the net benefit of SME execution. Technique: layout-aware graph conversion. To reduce this overhead, SMEPilot tracks tensor layout as part of the runtime state. Each task declares which layouts it can consume and which layout it produces. The runtime then inserts conversions only when a consumer requires a layout that the producer does not already provide. SMEPilot uses two strategies to keep repeated packing off SME critical paths. Off-critical-path packing for model weights. Model weights are static during inference, so their layout can be decided before any request executes. If the runtime determines that a weight tensor will be consumed by SME kernels, SMEPilot packs it once at model load time and stores the packed copy for reuse. This avoids repacking the same weights for every request and removes weight conversion from the request critical path. Producer-side packing for activation tensors. Activation tensors are generated online and cannot be packed at model load time. For these tensors, SMEPilot moves conversion to the producer side when the downstream consumer is SME. This removes an extra memory copy from the consumer path and, for reused activations such as KV-cache entries, avoids repeatedly unpacking the same data across attention invocations. For example, the key and value tensors produced by the K-projection and V-projection are written directly into a packed KV-cache layout. As shown in Figure 7, a standard CPU KV cache stores each token as a contiguous row, while SME attention prefers layouts that group tokens and feature elements according to the SME vector length and kernel tile shape. The runtime therefore stores K and V cache entries in separate SME-friendly layouts. Subsequent attention calls can then consume the packed KV cache without repacking all previous entries on every invocation.
Offline Profiler
Online Planner
Measured performance
Operators SME
CPU cores
Profiling Result
Planning Model&Req uest analysis
z
Profiling Result
Execution Plan
Operator Placement
CPU
Thread allocation
SME
Partition Shape
mixed
Execution plan Inference Runtime for SME-enabled CPU QKVproj
Attention
FFN/MOE Up
FFN/MOE Down
partition pipeline
…
Packed KVCache
Weight
Packed Weight
Fig. 8. The overall architecture of SMEPilot. The offline profiler runs representative operators on SME and CPU cores to collect hardware-specific performance metrics. The online planner combines the profiling result with model and request analysis to choose operator placement, thread allocation, and partition shape. The resulting plan drives the inference runtime, including the partition, pipeline, and packed-layout mechanisms used by different LLM operators. 4.4
Execution Plan Generation
Putting everything together, Figure 8 shows that SMEPilot generates an execution plan in two stages. First, an offline profiler measures the hardware capability of the target SMEenabled CPU. Second, an online planner combines these measurements with the current request shape and the given model to produce a tile-level execution plan. The planner is invoked at operator granularity, but the generated plan is executed at tile granularity so that one operator can use SME, CPU cores, or both. At runtime, the plan selects the partitioned, pipelined, and packed-layout mechanisms used by QKV projection, attention, and FFN/MoE operators. 7
eff and 𝑃 eff be the effective throughputs CPU cores. Let 𝑃 SME CPU measured by the profiler for the selected kernel family and worker counts. The ideal fraction of the output dimension assigned to SME is
Performance Profiler The profiler executes a set of operator kernels on CPU cores and SME to measure the performance metrics needed by the roofline-model analysis in Section 3. The performance metrics include: (1) SME metrics: the SME unit count, matrix throughput, vector throughput (2) CPU metrics: the CPU core count, per-core throughput for matrix and vector work. (3) Memory metrics: the memory bandwidth and the cache size. These metrics are only related to the hardware and are independent of the model and the request, so they can be measured offline once and reused for different requests and models. Online Planner The online planner determines the optimal execution plan for each operator in the computation graph, given the current request and the model. It contains two processes. The first is operator placement: for each operator, the planner decides whether it should run on SME only, CPU cores only, or a mixed SME+CPU configuration. The second is tile-plan construction: the planner decides the thread counts and the partition size. Roofline-based operator placement. For an operator 𝑜, the planner first obtains its work 𝑊𝑜 , useful memory traffic 𝑄𝑜 , and shape parameters from the operator graph and request state. It then evaluates the roofline lower bound in Section 3 for three execution targets: SME-only, CPU-only, and mixed SME+CPU execution. The SME-only target is preferred when the operator has enough arithmetic intensity to amortize SME setup and layout conversion, but does not expose enough independent output tiles to keep CPU workers useful. The CPU-only target is preferred for memorybound or vector-heavy operators, such as narrow decode GEMV and softmax, where SME’s matrix throughput cannot reduce the critical path. The mixed target is preferred when the operator is compute-bound and has enough independent output tiles for both SME and CPU cores. This decision is shape-dependent: the same logical operator may be CPUonly during single-token decode, SME-only for a moderatewidth decode shape, and mixed SME+CPU during longcontext prefill. Thread allocation. After operator placement, the planner chooses the number of SME and CPU threads for the selected target. SMEPilot uses a heuristic rather than an exhaustive search because the choice is hardware-dependent, and the same request must be planned quickly. The heuristic first tries to cover the hardware units selected by the placement decision. It then checks whether the operator exposes enough kernel-aligned tile regions for the candidate threads. A candidate thread is dropped if its assigned region would be smaller than the native kernel tile or would force a nonaligned boundary. This rule avoids a common failure mode of over-partitioning: adding threads increases scheduling overhead and may break the tile shapes that the SME and CPU kernels were optimized for. Partition size. After choosing worker counts, the planner computes the amount of output work assigned to SME and
𝜌=
eff 𝑃SME eff + 𝑃 eff 𝑃SME CPU
.
(1)
The planner first computes the ideal split point 𝜌𝐷, where 𝐷 is the selected output extent. It then rounds this split point to the nearest boundary aligned with both the SME and CPU kernel tile sizes. If several aligned candidates are close to the ideal ratio, the planner chooses the one that minimizes the predicted maximum of the SME-side and CPUside completion times. As a result, the final split is not a fixed FLOP ratio; it is the closest tile-aligned shape ratio supported by the current kernel and thread allocation.
5
Evaluation
5.1 Experimental Setup We implement SMEPilot, an optimized LLM inference engine for CPUs with scalable matrix extensions. SMEPilot is written in C and C++ and includes SME-aware runtime components and SME kernels. The SME kernels are developed using the optimized assembly microkernels from KleidiAI [6] and Arm C Language Extensions (ACLE) SME intrinsics [5]. We evaluate the performance of SMEPilot on three platforms equipped with CPUs supporting Scalable Matrix Extension (SME): Apple M4 Pro CPU, MediaTek Dimensity 9500, and KunPeng 920 72F8 Server CPU with SME support. These are recently released CPU chips that support SME or SME2, and they represent the current state of the art for SME performance. Table 1 summarizes their CPU resources, SMEto-CPU resource ratio, and supported instruction sets. In our evaluation, we compare SMEPilot against llama.cpp [16], one of the most widely used CPU LLM inference framework. We use the default CPU backend for all platforms. We select both dense and MoE models for evaluation: Llama-3.2-3B [26], Qwen3-4B [38], and Qwen330B-A3B [38]. The Qwen3-30B-A3B model uses a 4-bit weight quantization to fit into device memory. We evaluate three representative tasks: long-text processing [18], simple question answering [10], and long-text generation [7], as shown in Table 2. The Ruler row sweeps sequence lengths of 4K and 8K, while GSM8K and LongWriter-6K sweep batch size. We use 10 threads and 8 threads for Apple M4 Pro and MediaTek Dimensity 9500, respectively, which equals the number of performance cores on the two platforms. For the KunPeng 920 Server CPU, we use 10 threads for a fair comparison. 5.2
End-to-End Inference
Figure 9 shows the end-to-end inference latency of SMEPilot compared to the llama.cpp baseline on the CPU platforms. 8
Table 1. CPU platforms used in the evaluation. Platform
CPU cores
SME:CPU ratio
Supported instruction sets
Apple M4 Pro CPU
14 cores (10 performance + 4 efficiency); 10 workers used
SME, SME2
MediaTek Dimensity 9500 KunPeng 920 72F8 Server CPU
8 cores (1 C1-Ultra + 3 C1-Premium + 4 C1-Pro) server-class multi-core CPU; 10 workers used
2:10 (1 SME unit per 5 performance cores) 1:8 1:1
llama.cpp prefill llama.cpp decode M4 Pro Llama3.2-3B
M4 Pro Qwen3-4B
SMEPilot prefill SMEPilot decode
Dimensity 9500 Llama3.2-3B
M4 Pro Qwen3-30B-A3B
Dimensity 9500 Qwen3-4B
400 50
30
60
Ruler
40 20
40
30 20
10
20
10 0
4K
0
8K
4K
0
8K
4K
800
300
600
200
400
100
200
0
8K
SME, SME2 SME
4K
4K
KunPeng Qwen3-30B-A3B 600
300
0
8K
KunPeng Llama3.2-3B
200
400
100
200
0
8K
4K
8K
0
4K
8K
Sequence length M4 Pro Llama3.2-3B
20
Latency (s)
150
40
20
100
10
10 10
5
1 2 3 4 6 8 16
0
5
1 2 3 4 6 8 16
M4 Pro Llama3.2-3B
LongWriter-6K
60
3000
0
1 2 3 4 6 8 16
M4 Pro Qwen3-4B
20
50
0
0
1 2 3 4 6 8 16
Batch size
3000
1000
2000 1000
500
1 2 3 4 6 8 10 12 14 16
0
1 2 3 4 6 8 16
0
100
60
75
40
50
20
25
0
1 2 3 4 6 8 16
1000
1 2 3 4 6 8 16
0
1 2 3 4 6
1 2 4 8 16
KunPeng Qwen3-30B-A3B
6000
4000
3000
1000
2000
500
1000
0
0
4000
1500
2000
1000
125
80
2000
3000
KunPeng Qwen3-30B-A3B
100
Dimensity 9500 Dimensity 9500 KunPeng Llama3.2-3B Qwen3-4B Llama3.2-3B
M4 Pro Qwen3-30B-A3B
2000
KunPeng Llama3.2-3B
1 2 3 4 6 8 16
4000
1500
0
Dimensity 9500 Qwen3-4B
15
15
0
Dimensity 9500 Llama3.2-3B
M4 Pro Qwen3-30B-A3B
30
20
GSM8K
M4 Pro Qwen3-4B
1
2
0
2000
1 2 4 8
0
1 2 4 8
Batch size
Fig. 9. End-to-end latency across devices and real-world datasets. Rows correspond to Ruler, GSM8K, and LongWriter-6K. Within each row, panels compare the measured device-model pairs: Apple M4 Pro with Llama3.2-3B, Qwen3-4B, and Qwen330B-A3B; MediaTek Dimensity 9500 with Llama3.2-3B and Qwen3-4B; and KunPeng Server CPU with Llama3.2-3B and Qwen3-30B-A3B. We skip some batch or model-pair because of the memory constraints of device. SMEPilot produces up to 3.94 × speedup across different datasets and CPU platforms. More specifically, in the prefill stage, SMEPilot achieves 1.13× to 5.42× speedup. In the decode stage, SMEPilot produces a up to 3.48× speedup.
In the long text processing task, SMEPilot produces a 1.13× to 3.56× speedup on three different CPUs. The speedup arises from both utilizing CPU cores and SME units together for compute-bound prefill operators, including the attention, FFN and MoE. 9
Table 2. End-to-end benchmark configurations. Dataset
Mean prefill tokens
Mean decoding tokens
Long-context QA Math QA Long-form generation
Ruler-4k/8k GSM8K LongWriter-6K
3324 / 7338 92 339
4 100 5424
5
TFLOP/s
Task type
SMEPilot Apple Accelerate GGML CPU
ARM Kleidiai SME FlashAttention CPU Vanilla Attention w/ SME
(a) FFN
(b) Attention 2.5
0
0.0 512 1024 2048 4096 8192
512 1024 2048 4096 8192
Prefill length
In the simple question answering task, SMEPilot produces a 1.0× to 3.94× speedup on three different CPUs. The speedup arises from our roofline-based operator placement for prefill, decode phases and different batch sizes. For batch sizes of 1 or 2, SMEPilot produces a comparable performance with the baseline because the operators are more memory-bound and the runtime selects CPU execution. When the batch size increases, SMEPilot produces a more significant speedup. This is because larger batch sizes reach the ridge-crossing regime where SME can be more effective, which demonstrates the effectiveness of our online planner. The speedup is smaller for the MoE models. This is because tokens are routed to different experts, which reduces the effective batch size for each expert and makes the operators more memorybound, thus reducing the benefit of using SME. In the long-text generation task, SMEPilot produces a 1× to 2.95× speedup. This long generation task has a much longer decoding phase, and SMEPilot accelerates the decode phase by using SME for operators in the ridge-crossing regime. Notably, the speedup is more significant for the MoE model in this task. This is because Qwen3-30B-A3B uses group-query-attention with larger group size, which increases the arithmetic intensity of attention and help gain more speedup from our SME-aware scheduling. 5.3
Prefill length
Fig. 10. Prefill operator throughput on the Apple M4 Pro CPU. (a) FFN GEMM for the 𝐵 × 3072 × 8192 shape, compared with Apple Accelerate, GGML CPU, and ARM Kleidiai SME baselines. (b) Attention for Llama-3.2-3B with 24 query heads, 8 KV heads, and head dimension 128, compared with FlashAttention CPU and vanilla attention with SME.
Latency (ms)
SMEPilot Kleidiai CPU
GGML CPU FlashAttention CPU
(a) FFN
(b) Attention
1.0
2
0.5 0.0
0 1
4
8
12
Batch size
16
1
2
3
4
6
Decode tokens
8
Fig. 11. Decode operator latency on the Apple M4 Pro CPU. (a) FFN GEMV for the 2560 × 𝑇 × 9728 shape, compared against Kleidiai CPU and GGML CPU baselines. (b) Attention with KV length 8192, 32 query heads, 8 KV heads, and head dimension 128, compared with the FlashAttention CPU baseline.
Operator Performance Breakdown
To better analyze the end-to-end speedup and demonstrate the effectiveness of our design, we conduct a detailed tests for operators in the prefill and decode phases on the Apple M4 Pro CPU.
5.3.2 Prefill Attention. Figure 10(b) shows the prefill attention performance of SMEPilot compared to the CPU FlashAttention and vanilla attention with SME across different sequence lengths. For attention with 24 query heads and 8 KV heads, SMEPilot sustains 1.50–4.38 TFLOP/s as the prefill length increases from 512 to 8192, and achieves a 1.56–3.50× speedup over the FlashAttention CPU baseline. The gain demonstrates the effectiveness of our phase-aware pipelining design in Section 4.
5.3.1 FFN. Figure 10(a) shows the prefill FFN GEMM performance of SMEPilot compared with Apple Accelerate [3], GGML [15] CPU, and the ARM Kleidiai SME [6], across different prefill lengths. For prefill length from 512 to 8192, SMEPilot sustains 4.10–4.44 TFLOP/s. Compared with the GGML CPU, SMEPilot achieves a 6.06–7.43× speedup across different prefill lengths, demonstrating the effectiveness of our roofline analysis. Compared with the Apple Accelerate and ARM Kleidiai SME, which uses the SME unit but does not use SMEPilot’s mixed execution plan, SMEPilot still achieves a 1.30–1.67× speedup. The performance improvement arises from using the SME unit and CPU cores together (Section 3) and from the partition technique that balances the two sides (Section 4).
5.3.3 Decode FFN. Figure 11(a) reports the decode FFN result. For different decode batch sizes, SMEPilot achieves a comparable performance with the CPU baselines when the batch size is small, and achieves 1.45× and 3.95× speedup compared to the Kleidiai CPU and GGML CPU respectively when the batch size is large. This is because the runtime selects CPU execution for smaller batches, but selects SME for larger batches that reach the ridge-crossing regime. 10
50
Attention
70.6
200
48.25
100
0
(b)
GEMV 2
202.7
wo part.
1.71
156.3 98.1
1
0
Full
(c)
Latency (s)
Latency (ms)
GEMM (a)
pack
0.52
0
Full
wo w 2-stage pipe pipe
Full
Naive pack
avg W
E2E energy (J)
5.6
3
Batch size
8
16
GPU Inference Comparison
We also compare SMEPilot with GPU inference on the Apple M4 Pro platform because Macbook Pro is equipped with a 20-core GPU with 8 TFlops. As shown in Figure 13, SMEPilot achieves 0.72–0.96× performance compared to the GPU inference. This result positions SMEPilot as a practical CPU inference engine for systems where the GPU is unavailable or reserved for other workloads, or to coordinate with GPU execution.
5.3.4 Decode Attention. Figure 11(b) shows the latency of decode attention of SMEPilot and the FlashAttention CPU baseline. SMEPilot improve decode attention latency from 1.32× to 4.75× depending on the decode token width. The benefit arises from the runtime’s ability to use SME when the decode token width is large enough and to pipeline matrix and vector phases when dependencies permit. The speedup grows with the decode token width because wider decode attention has larger matrix shapes and has higher arithmetic intensity, making it more profitable to use SME and to pipeline matrix and vector phases. This result validates the design choice of treating decode attention as a shapedependent case instead of a purely memory-bound operator that should always run on cpu cores.
6
Related Work
6.1
CPU LLM Inference
CPU inference has become an important deployment path for LLMs because CPUs are widely available across edge devices, personal computers, and servers. Practical runtimes such as llama.cpp [16] show that careful low-level kernels, weight quantization, and CPU-friendly memory layouts can make local LLM inference usable on commodity machines. Intel Extension for Transformers studies CPU LLM inference on Xeon processors, combining weight-only quantization, operator fusion, memory management, and runtime optimizations for popular LLMs [31]. ArcLight further studies lightweight LLM inference on many-core CPUs [36]. Other CPU-oriented systems target low-bit or sparse execution: TMAC replaces mixed-precision GEMM with table lookup for edge CPUs [35], while sparse Transformer inference systems reduce CPU-side compute and memory traffic [13]. These systems demonstrate that CPU LLM inference is practical, but their main target is model compression, quantized arithmetic, or CPU-only kernel optimization. Several works further analyze CPU inference as a phasedependent systems problem. Sandwich observes that prefill and decode have different execution characteristics and searches different CPU configurations for the two phases [40]. Platform studies show that LLM inference stresses memory bandwidth, scheduling, and
Ablation Study
Our ablation study focuses on the three key design mechanisms in Section 4: tile-level work partitioning, phase-aware pipeline execution, and layout-aware execution. As shown in Figure 12, removing tile partitioning increases GEMM latency by 1.46× because the operator can no longer use SME and CPU cores together. Removing the attention pipeline increases prefill attention latency by 2.07×. Finally, naive onpath layout packing increases decode GEMV latency from 0.52 ms to 1.71 ms, confirming that layout conversion must be kept off repeated SME critical paths. 5.5
2
reduces the energy to 0.485× of the llama.cpp baseline. The lower energy mainly comes from finishing inference faster and better utilize SME.
llama.cpp baseline 3 188.812 ± 7.787 26.050 ± 2.193 482.813 ± 23.213 SMEPilot 3 338.012 ± 0.680 22.611 ± 0.296 233.931 ± 2.751
5.4
6
2.5
Fig. 13. End-to-end Llama3.2-3B GSM8K latency on Apple M4 Pro, comparing SMEPilot on the CPU with GPU inference across batch sizes.
Table 3. End-to-end energy on Apple M4 Pro for Qwen3-4B running the Ruler-4K workload. Throughput (tok/s)
4
5.0
1
𝑛
GPU
0.0
Fig. 12. Ablation study on the Apple M4 Pro CPU. Tile partition, phase-aware pipelining, and layout-aware execution reduce GEMM(4096x3072x8192), prefill attention(seq 4096), and decode GEMV(16x3072x8192) latency, respectively.
Mode
SMEPilot 7.5
Power Consumption
We further run a power measurement on the Mac platform using Qwen3-4B and the Ruler-4K workload. We choose this platform because macOS provides the powermetrics tool for sampling system power during inference. Table 3 reports the end-to-end energy from this measurement. SMEPilot 11
7
distributed inference resources differently across model sizes and request regimes [8]. On Arm CPUs, prior work characterizes Transformer inference on many-core processors [19] and optimizes attention by exploiting data reuse across Arm multi-core CPUs [14]. Other systems use CPUs as partners in heterogeneous LLM inference. FlexGen offloads tensors across GPU, CPU, and storage [32]; PowerInfer splits work between GPU and CPU on commodity PCs [33]; PowerInfer2 coordinates NPU, CPU, and storage on smartphones [37]; and Toppings uses CPU computation to hide LoRA adapter loading during LLM serving [24]. SMEPilot is complementary to these CPU inference and LLM serving systems [30]. Rather than scheduling requests across devices, or optimizing a conventional vector-core runtime, SMEPilot targets CPUs that expose both matrixextension units and ordinary CPU cores. It therefore focuses on when to use SME, when to keep work on CPU cores, and when to coordinate both within one operator. 6.2
Discussion and Future Work
Design implications for future CPU matrix extensions. Based on our characterization in using SME for LLM inference, we suggest that current SME can be improved along following dimensions. First, stronger support for vector-heavy and GEMV-like phases is important. Decode, softmax, and normalization do not naturally exploit SME outer-product throughput, and direct SME execution can be slower than using CPU cores. Increasing matrix throughput alone therefore will not fully accelerate LLM inference unless the architecture or runtime also improves these non-matrix phases or makes them easier to overlap with matrix work. Second, matrix extensions need more convenient layout management. SME kernels require packed, tile-friendly layouts, while the surrounding LLM pipeline often uses conventional tensor layouts. Without architectural or runtime support for cheap conversion, layout traffic can erase much of the gain from a faster matrix kernel. Third, future systems would benefit from clearer isolation and between SME units and CPU cores. Currently, SME and CPU cores can provide near-additive compute throughput, but they still contend for cache capacity, and the CPU cores that issue SME instructions. A more decoupled SME from cpu pipeline would reduce contention. Cooperation with GPU and NPU. While this paper focuses on CPU with scalable matrix extensions, future LLM inference systems will likely need to coordinate across CPU, GPU, and NPU execution. CPU matrix extensions are attractive because they are integrated into general-purpose CPUs, which are widely deployed and cost-effective across both edge and server platforms, and can execute CPUside work without an additional device transfer. GPUs and NPUs, in contrast, provide much higher throughput for large regular matrix workloads. A future inference runtime could therefore use GPUs or NPUs together with CPU matrix extensions, exposing more overlap opportunities. Realizing this cooperation requires a broader planner that accounts for factors, such as device transfer cost, memory placement and etc. We leave such cooperation with SME to future work.
CPU Matrix Extension and SME
Modern CPUs increasingly include matrix extensions that provide accelerator-like matrix throughput inside the CPU package. Intel AMX adds tile registers and tile matrix instructions, and prior work has used AMX to accelerate LLM inference on CPUs [21]. SparAMX combines AMX with unstructured sparsity to accelerate compressed LLM token generation on Intel CPUs [1]. LIA uses AMX-enabled CPU computation as part of a CPU-GPU-CXL cooperative inference system [20]. These works show that CPU matrix extensions can be useful for LLM workloads, especially when the software stack exposes enough matrix work to the extension. However, they primarily study Intel AMX, compressed models, or CPU-GPU offload. SMEPilot instead studies Arm SME and the scheduling problem inside an SMEenabled CPU cluster, where SME and CPU cores share cache and memory bandwidth. Arm SME introduces matrix-oriented architectural state and outer-product instructions on top of the Arm vector execution model [4]. Recent SME studies characterize the extension and build optimized kernels for dense matrix multiplication [12, 29] and sparse matrix multiplication [23]. These kernel-level studies are an important foundation for understanding SME’s peak throughput and data layout requirements. SMEPilot builds on this line of work but asks a different systems question: how should an LLM runtime use SME across prefill, decode, attention, FFN, vector operations, and layout-changing operator boundaries? The answer is not to replace every kernel with an SME kernel. SMEPilot uses a roofline-guided planner to select SME-only, CPUonly, or mixed SME+CPU execution; partitions computebound operators across heterogeneous units; pipelines SMEfriendly matrix phases with CPU-friendly vector phases; and tracks layouts to keep packing out of repeated SME critical paths.
8
Conclusion
This paper studies how Arm SME should be used for LLM inference. Our characterization shows that SME is not a universal substitute for CPU vector cores. SMEPilot uses this observation to select SME-only, CPU-only, or mixed execution, partition work at tile granularity, overlap matrix and vector phases, and avoid repeated layout conversion. The resulting speedups show that CPU matrix extensions are most effective when their throughput is applied selectively and amortized across suitable inference regimes.
12
References
[17] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David LopezPaz, and Gabriel Synnaeve. 2024. Better & Faster Large Language Models via Multi-token Prediction. arXiv:2404.19737 [cs.CL] https: //arxiv.org/abs/2404.19737 [18] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654 [19] Jiazhi Jiang, Jiangsu Du, Dan Huang, Dongsheng Li, Jiang Zheng, and Yutong Lu. 2022. Characterizing and Optimizing Transformer Inference on ARM Many-core Processor. In Proceedings of the 51st International Conference on Parallel Processing (ICPP ’22). Association for Computing Machinery, 20:1–20:11. doi:10.1145/3545008.3545022 [20] Hyungyo Kim, Nachuan Wang, Qirong Xia, Jinghan Huang, Amir Yazdanbakhsh, and Nam Sung Kim. 2025. LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, 544–558. doi:10.1145/3695053.3731092 [21] Hyungyo Kim, Gaohan Ye, Nachuan Wang, Amir Yazdanbakhsh, and Nam Sung Kim. 2024. Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference. IEEE Computer Architecture Letters 23, 1 (2024), 117–120. doi:10.1109/LCA.2024.3397747 [22] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626. [23] Kelun Lei, Hailong Yang, Kaige Zhang, Kejie Ma, Yiqing Wang, Xin You, Yufan Xu, Enrique S. Quintana-Orti, Zhongzhi Luan, Yi Liu, and Depei Qian. 2025. LOw-cOst yet High-Performant Sparse Matrix-Matrix Multiplication on Arm SME Architectures. arXiv:2511.08158 [cs.DC] doi:10.48550/arXiv.2511.08158 [24] Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. 2025. Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, 613–629. https://www.usenix.org/conference/ atc25/presentation/li-suyi-toppings [25] Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S. Yu, and Ming Zhang. 2025. Large Language Model Agent: A Survey on Methodology, Applications and Challenges. arXiv:2503.21460 [cs.CL] https://arxiv.org/abs/2503.21460 [26] Meta AI. 2024. Llama 3.2: Revolutionizing Edge AI and Vision with Open, Customizable Models. Meta AI Blog. https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/ [27] Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2025. Large Language Models: A Survey. arXiv:2402.06196 [cs.CL] https://arxiv. org/abs/2402.06196 [28] Onur Mutlu, Saugata Ghose, and Rachata Ausavarungnirun. 2018. Recent Advances in Overcoming Bottlenecks in Memory Systems and Managing Memory Resources in GPU Systems. arXiv:1805.06407 [cs.AR] doi:10.48550/arXiv.1805.06407 [29] Stefan Remke and Alexander Breuer. 2024. Hello SME! Generating Fast Matrix Multiplication Kernels Using the Scalable Matrix Extension. arXiv:2409.18779 [cs.DC] doi:10.48550/arXiv.2409.18779 [30] Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX
[1] Ahmed F. AbouElhamayed, Jordan Dotzel, Yash Akhauri, Chi-Chih Chang, Sameh Gobriel, J. Pablo Munoz, Vui Seng Chua, Nilesh Jain, and Mohamed S. Abdelfattah. 2025. SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs. arXiv:2502.12444 [cs.LG] doi:10.48550/arXiv.2502.12444 [2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawal [3] Apple. 2026. Accelerate Framework. Apple Developer Documentation. https://developer.apple.com/documentation/accelerate Accessed: 2026-06-08. [4] Arm. 2024. Part 1: Arm Scalable Matrix Extension (SME) Introduction. Arm Community Blog. https://community.arm.com/arm-communityblogs/b/architectures-and-processors-blog/posts/arm-scalablematrix-extension-introduction [5] Arm. 2026. Arm C Language Extensions. Arm Developer Documentation. https://arm-software.github.io/acle/main/acle.html Accessed: 2026-06-08. [6] Arm. 2026. KleidiAI: Optimized Micro-kernels for AI Workloads on Arm CPUs. GitHub repository. https://github.com/ARM-software/ kleidiai Accessed: 2026-06-08. [7] Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. arXiv:2408.07055 [cs.CL] https://arxiv.org/abs/2408.07055 [8] Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Suvinay Subramanian, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. 2024. Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models. arXiv:2406.01698 [cs.AR] doi:10.48550/arXiv.2406.01698 [9] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774 [cs.LG] https://arxiv.org/abs/2401.10774 [10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] https://arxiv.org/abs/2110.14168 [11] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359. [12] Chencheng Deng, Weiling Yang, Jianbin Fang, and Dezun Dong. 2025. Demystifying ARM SME to Optimize General Matrix Multiplications. arXiv:2512.21473 [cs.DC] doi:10.48550/arXiv.2512.21473 [13] Ditto PS, Jithin VG, and Adarsh MS. 2024. Inference Acceleration for Large Language Models on CPUs. arXiv:2406.07553 [cs.DC] doi:10. 48550/arXiv.2406.07553 [14] Xiao Fu, Weiling Yang, Dezun Dong, and Xing Su. 2024. Optimizing Attention by Exploiting Data Reuse on ARM Multi-core CPUs. In Proceedings of the 38th ACM International Conference on Supercomputing (ICS ’24). Association for Computing Machinery, 137–149. doi:10.1145/3650200.3656620 [15] Georgi Gerganov and contributors. 2023. GGML: Tensor Library for Machine Learning. GitHub repository. https://github.com/ggmlorg/ggml [16] Georgi Gerganov and contributors. 2023. llama.cpp: LLM inference in C/C++. GitHub repository. https://github.com/ggml-org/llama.cpp 13
Feiyang Chen, Haibo Chen
Association, 397–411. https://www.usenix.org/conference/atc21/ presentation/romero [31] Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, and Hengyu Meng. 2023. Efficient LLM Inference on CPUs. arXiv:2311.00502 [cs.LG] doi:10.48550/arXiv.2311.00502 [32] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 31094–31116. https: //proceedings.mlr.press/v202/sheng23a.html [33] Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 590–606. doi:10.1145/3694715.3695964 [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. CoRR abs/1706.03762 (2017). arXiv:1706.03762 http://arxiv.org/abs/1706.03762 [35] Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. 2025. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). Association for Computing Machinery. doi:10.1145/3689031.3696099 [36] Yuzhuang Xu, Xu Han, Yuxuan Li, and Wanxiang Che. 2026. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs. arXiv:2603.07770 [cs.DC] doi:10.48550/arXiv.2603.07770 ACL 2026 Demo.
[37] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv:2406.06282 [cs.AR] https://arxiv.org/abs/2406. 06282 [38] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). https://arxiv.org/abs/2505.09388 [39] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 521–538. https://www.usenix.org/conference/osdi22/ presentation/yu [40] Juntao Zhao, Jiuru Li, and Chuan Wu. 2025. Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving. arXiv:2507.18454 [cs.AR] doi:10.48550/arXiv.2507.18454 DAC ’26. [41] Shanshan Zhao, Xinjie Zhang, Jintao Guo, Jiakui Hu, Lunhao Duan, Minghao Fu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. 2026. Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities. arXiv:2505.02567 [cs.CV] https://arxiv.org/abs/2505.02567 [42] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 193–210. https: //www.usenix.org/conference/osdi24/presentation/zhong-yinmin