Waypoint: Analytically Modeling GEMM for CPUs

Waypoint: Analytically Modeling GEMM for CPUs Srinidhi N

Advanced Micro Devices Inc India Srinidhi.N@amd.com

arXiv:2610.34914v1 [cs.DC] 26 Oct 2026

Eashan Dash

Advanced Micro Devices Inc India Eashan.Dash@amd.com

Sunanda Somwase

Advanced Micro Devices Inc India SomwaseSunanda.Vishnu@amd.com

Varun Agrawal

Advanced Micro Devices Inc India Varun.Agrawal@amd.com

Abstract High-performance General Matrix Multiplication (GEMM) implementations rely on breaking down GEMM into carefully selected tiles to achieve peak performance. We present Waypoint, a new analytical model to automatically identify the optimal tile size for a given hardware and input matrix shape. Waypoint estimates the achievable performance for a given tile choice by capturing all the architectural effects including interactions with memory hierarchy, instructionlevel parallelism, and multi-threading. Using the estimates as a proxy, we search through the space of possible tile sizes to identify the one that maximizes performance. Additionally, Waypoint helps hardware designers estimate realistic performance on experimental hardware optimizations. We build a new GEMM library libzenmm for CPUs that uses Waypoint to automatically break down GEMM and achieve performance that matches state-of-the-art GEMM libraries and helps accelerate LLM inference on CPUs achieving 1.36 − 1.98× speedups.

1

Introduction

Matrix Multiplication operations make up for a large part of the core computation in several modern machine learning based applications such as image processing, recommendation systems, and more recently, Large Language Model (LLM) inference. To ensure that these applications run efficiently, it is crucial to optimize the performance of General Matrix Multiplication (GEMM) operations on the underlying hardware. While the industry has moved towards developing specialized hardware accelerators such as GPUs, CPUs remain important for cost-sensitive use cases and small-scale inference workloads. Hardware vendors continue to enhance CPU architectures with new features aimed at improving GEMM performance. © 2026 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, EPYC, AOC-BLAS, AOC-DLP, PACE, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.

Hemant Kumar Chandrakant Chandavar Advanced Micro Devices Inc India hkcc375@gmail.com

Prakash Raghavendra

Advanced Micro Devices Inc India Prakash.Raghavendra@amd.com

On CPUs, improving GEMM performance involves careful coordination between the hardware and the software that implements GEMM operations. Hardware designers continue to innovate and introduce features that push the peak performance achievable. At the same time, software implementations optimize the use of these hardware features to achieve high performance. A naive 3-loop implementation of the matrix multiplication algorithm is far from achieving peak performance. Instead, the libraries use specialized hardware features (e.g. vector/matrix instructions), and carefully organize the input matrices in memory so as to reduce the cost of accessing them for the computation. They do so by breaking down the GEMM operation into smaller units called tiles. For each tile, the libraries use vector/matrix instructions to compute the output, and between successive tiles, they ensure that memory accesses are efficient. The tile sizes are critical to achieving high performance and need to be chosen carefully. Selecting suboptimal tile sizes can lead to inefficient memory accesses and underutilization of the hardware’s computational capabilities. The right tile depends both on the hardware and the specific input matrix size that the application invokes [31]. Therefore, an ideal solution would adapt the tile sizes not only to the hardware but also to specific input sizes. Existing state-of-the-art GEMM libraries [16, 30] identify the tile sizes that are either optimal for a given hardware, or the specific class of shapes (e.g., tall and skinny [23], irregular [31]) but do not adapt them to the application’s needs. Many techniques are used to identify the tile sizes for a given hardware such as - manual expert-driven tuning resulting in handwritten assembly, automated empirical analysis [7, 10], and analytical modeling [24, 26, 31]. In this paper we propose a new analytical model to derive optimal tile sizes that are tailored not just for the hardware but also for the specific input matrix sizes that the application uses. Our model addresses another crucial challenge that the hardware designers face when innovating to accelerate performance on future hardware. They need to be able

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

to estimate the realistic performance achievable on experimental hardware designs. They routinely propose new ISA extensions (e.g. AMX [21], ACE [33], APX [20]) aimed at optimizing GEMM performance. Existing analytical models address the challenge partially. They provide a Roofline [32] estimate of the maximum achievable performance. But depending on the matrix sizes, the realistic achievable performance is far from the Roofline. Building a realistic performance model requires accounting for different architectural effects that are accompanied with a GEMM operation such as cache effects, pipelining, multi-threaded execution, and memory bandwidth limitations. The traditional Roofline model makes simplistic assumptions to over-approximate the peak performance achievable. It assumes that either the cost of computation dominates the cost of accessing data (i.e. compute-bound) or vice versa (i.e. memory-bound), which rarely holds true in practice. Therefore, a more nuanced model that captures both computation and memory access costs accurately is necessary to predict realistic GEMM performance. In what follows, we describe our contribution - a new analytical model Waypoint that is parametric both in the hardware architecture, and the application’s needs (input matrix sizes). Waypoint provides accurate estimates of the realistic performance achievable, aiding hardware designers, and identifies the optimal tile sizes, enabling automatic and efficient GEMM implementations. We build our model in two stages. First, we develop Waypoint-Strict, to provide realistic estimates of GEMM performance on a given hardware independent of the software implementation. We prove theoretically that Waypoint-Strict is tighter than the Roofline model and show empirically that state-of-the-art GEMM libraries realize these estimates. In compute-bound scenario, Waypoint-Strict accounts for the cost of additional instructions that are always issued with the main compute instruction. Similarly for the memory-bound scenario, Waypoint-Strict uses more accurate bandwidth values to calculate the time it takes to fetch data. As a result, in both scenarios, Waypoint provides estimates that are tighter than the Roofline. Second, we refine Waypoint-Strict for a tile based GEMM implementation resulting in Waypoint-Tile that estimates GEMM performance for a specific tile size. Using WaypointTile’s estimates, we search through possible tile sizes to select the optimal one. Building Waypoint-Tile is challenging because it requires accurately modeling the interaction between the tile size and the memory hierarchy, including cache levels, cache line sizes, and memory bandwidth limitations. Waypoint-Tile accurately estimates the number of cache misses that are imposed by a tile size and adds the associated miss costs. In building Waypoint-Tile, we identify several insights that allow us to rule out certain tile sizes and narrow down the search for an optimal one.

Empirically, we evaluate the accuracy of both WaypointStrict and Waypoint-Tile against state-of-the-art GEMM libraries. We find that Waypoint-Strict estimates are consistently closer to the observed performance (a geomean of 1.096×). Similarly, we compare the estimate of the best tile size identified by Waypoint-Tile and find that the model is accurate in predicting realistic performance (a geomean of 1.012×). These accuracy results allow us to confidently use the models for identifying the optimal tile size and estimating the performance on future hardware. We develop a GEMM library libzenmm that uses Waypoint-Tile to select the best tile size for each input matrix size and hardware configuration. libzenmm performs comparable to state-of-the-art GEMM libraries across different hardware configurations and input sizes. Further on new hardware configurations, libzenmm is able to quickly adapt and achieve near-peak performance without any manual tuning. Finally, we use libzenmm to accelerate LLM inference on CPUs achieving 1.36 − 1.98× speedups on new hardware. To summarize, our contributions are, • An accurate implementation agnostic analytical model Waypoint-Strict, to estimate GEMM performance that is parametric in both the hardware architecture and the application’s needs. A more refined model, Waypoint-Tile that is specific to a tile-based GEMM implementation and used in practice to identify the optimal tile-size. • A GEMM library libzenmm that uses Waypoint-Tile to automatically select optimal tiling parameters for a given GEMM shape and hardware configuration, achieving near-peak performance without extensive manual or automated tuning. • Using libzenmm, we accelerate LLM inference on current and future CPUs.

2

Analytical Model

The Roofline model [32] distinguishes a computation between a compute-bound scenario, where performance is limited by the processing capacity of the CPU, and a memorybound scenario, where performance is limited by the memory bandwidth. It provides upper bounds in either case based on the peak computational throughput and memory bandwidth of the hardware. Building on the Roofline model specifically for GEMM computations, we first develop an implementation-agnostic Waypoint-Strict that refines the cost in both computebound and memory-bound scenarios. Then, we further refine the compute cost assuming a tiled GEMM implementation to obtain a Waypoint-Tile that captures accurately the number of cache misses and the associated costs. We prove theoretically that the Waypoint-Strict is tighter than the Roofline model and empirically show (in Section 4) that the Waypoint-Tile is accurate.

Waypoint: Analytically Modeling GEMM for CPUs

2.1

Waypoint-Strict

The Waypoint-Strict is parametric in the hardware configuration, including cache sizes, memory bandwidth, and compute instructions and provides an estimate of the number of cycles a GEMM operation will take on the given hardware. Formally a hardware configuration 𝐻 consists of the following constants: • The instruction reciprocal throughput for the core compute instruction 𝜏 𝑓 𝑚𝑎 and the dimensions of the instruction 𝛿 𝑓 𝑚𝑎 . The instruction is defined for each data-type and instruction set architecture (ISA) of the CPU. For example, in an x86-ISA based CPU using AVX512 family of SIMD instructions, the instruction reciprocal throughput for a single precision FMA instruction in AMD ‘Zen 5’ architecture is 0.5 cycles/instruction and contains 16 × 1 elements. • Register file information such as the number of registers available to issue FMA instructions 𝑁𝑟𝑒𝑔 and width of the registers in bits 𝑊𝑟𝑒𝑔 . • Cache sizes (𝑆𝐿1, 𝑆𝐿2, 𝑆𝐿3 ) and access reciprocal throughput (𝑇𝐿1,𝑇𝐿2,𝑇𝐿3 ) for the different levels of the memory hierarchy. For example, on AMD ‘Zen 5’ cores, the L1 data cache is 48 KB with an access reciprocal throughput of 0.5 cycles. • Memory bandwidth information, including the peak bandwidth per memory channel 𝐷𝑏𝑤 and the number of channels 𝐷𝑛 . • Maximum number of cores 𝐶𝑚𝑎𝑥 and the frequency 𝑓 of the CPU. The frequency is used to convert the number of cycles to time. We define a GEMM operation 𝑜𝑝 as one that computes an output matrix 𝐶 = 𝑎(𝐴 · 𝐵) + 𝑏𝐶 of size 𝑀 · 𝑁 where the input matrices are of size 𝑀 · 𝐾 and 𝐾 · 𝑁 using 𝑇 threads on the data-type 𝑑 of width 𝑊𝑑 bits, and scalars 𝑎 and 𝑏. We assume for simplicity that 𝑎 = 1 and 𝑏 = 0. Given the hardware and GEMM operation, the WaypointStrict computes the latency 𝐿𝑜𝑝 of the GEMM operation as follows:  𝐿𝑐 (𝑜𝑝) , 𝐿𝑚 (𝑜𝑝) + 𝐿𝑚𝑡 (𝑜𝑝) 𝐿(𝑜𝑝) = max 𝑜𝑝.𝑇 

where 𝐿𝑐 is the compute latency, 𝐿𝑚 is the memory latency, and 𝐿𝑚𝑡 is the threading overhead for the GEMM operation 𝑜𝑝. In what follows, we describe each of the components in detail. 2.1.1 Compute latency. The Waypoint-Strict assumes a deterministic compute instruction to be used for a given data-type and hardware architecture e.g. AVX512 VNNI is used for Int8 GEMM on x86-ISA based CPUs. The exact mapping can be provided as input. We first compute the number of compute instructions 𝑁 𝑓 𝑚𝑎 required to perform the GEMM:

𝑁 𝑓 𝑚𝑎 = 𝑀 · 𝑁 · 𝐾/𝛿 𝑓 𝑚𝑎 Where 𝛿 𝑓 𝑚𝑎 is used to denote the number of elements in an instruction e.g. In an AVX512 VNNI instruction that computes a column-wise dot product of two 4 × 16 matrices of 8-bit integers, 𝛿 𝑓 𝑚𝑎 = 4 × 16 = 64. Instruction Modularity. By allowing the user to define the dimensions of the core compute instruction, we accommodate new advances in the x86-ISA architecture that enable matrix instructions such as AMX, ACE, Arm SME. A matrix instruction can compute a matrix product of two matrices of size 𝑚 𝑓 𝑚𝑎 · 𝑘 𝑓 𝑚𝑎 and 𝑘 𝑓 𝑚𝑎 · 𝑛 𝑓 𝑚𝑎 in a single instruction. Therefore, the number of compute instructions is 𝑁𝑐 = 𝑀 · 𝑁 · 𝐾/(𝑚 𝑓 𝑚𝑎 · 𝑘 𝑓 𝑚𝑎 · 𝑛 𝑓 𝑚𝑎 ). The latency of the compute instructions is computed as 𝑁 𝑓 𝑚𝑎 ∗𝜏 𝑓 𝑚𝑎 where 𝜏 𝑓 𝑚𝑎 is the instruction reciprocal throughput for the core computation. In addition to the latency of the compute instruction, the total compute latency 𝐿𝑐 includes the latency of support instructions for zeroing the registers 𝐿𝑧 , loading the input matrices 𝐿𝑑𝑖𝑛 into the registers (assuming perfect cache behavior), and storing the output matrix back to memory 𝐿𝑑𝑜𝑢𝑡 . We assume that stores hit the nearest cache and thus incur the lowest latency. 2.1.2 Memory latency. The memory latency 𝐿𝑚 is computed using the peak memory bandwidth applicable for the GEMM operation. 𝐿𝑚 (𝑜𝑝) =

(𝑀 · 𝐾 + 𝐾 · 𝑁 + 𝑀 · 𝑁 ) · 𝑊𝑑 · 𝑓 𝐷𝑏𝑤 · 𝑑𝑛

where 𝑑𝑛 ≤ 𝐷𝑛 is the number of memory channels accessible to the GEMM operation. When running multithreaded GEMM programs, the bandwidth available to each thread may be lower than the peak bandwidth because of contention among threads accessing memory. We capture the effect of this contention using a model that distributes work across different memory channels, allowing us to predict data-access latency more accurately. For example, on AMD ‘Zen 5’ processors, if a GEMM operation uses fewer than eight threads (the size of a single Core Complex Die, or CCD), it can utilize the memory bandwidth of a single CCD. If more threads are used, the operation may need to access multiple CCDs, potentially increasing effective memory latency because of contention across memory channels. 2.1.3 Threading overhead. To compute the threading overhead, we assume that there is no data contention between threads. This assumption is grounded in practice, as it reflects the behavior of well-optimized GEMM implementations. When the assumption does not hold, the threads must

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

synchronize to read and write data, incurring additional overhead. It follows from our assumption that each thread consumes the entire 𝐾 dimension to compute the result and the input matrices are split only in the 𝑀 and 𝑁 dimensions. Let 𝑀𝑡 and 𝑁𝑡 be the splits across the 𝑀 and 𝑁 dimensions of the output matrix for each thread. The data needed by each thread is 𝑆𝐴𝑡 = 𝑀𝑡 · 𝐾 · 𝑊𝑑 and 𝑆 𝐵𝑡 = 𝐾 · 𝑁𝑡 · 𝑊𝑑 . The data that does not fit in the caches, Δ = 𝑆𝐴𝑡 + 𝑆 𝐵𝑡 − (𝑆𝐿3 + 𝑆𝐿2 + 𝑆𝐿1 ), is repeatedly accessed from memory by multiple threads, incurring significant overhead. We define the threading overhead as follows: 𝐿𝑚𝑡 = min(Δ · 𝑀𝑡 , Δ · 𝑁𝑡 ) ·

1

𝐷𝑏𝑤 · 𝑑𝑛 To capture the scenario in which the 𝐾 dimension is also split, we account for the cost of the reduction operation that follows the partial computations across different threads. This reduction typically involves summing the partial results along the 𝐾 dimension, which introduces additional memory accesses and synchronization overhead. 2.1.4 Theoretical results. The Roofline model estimates peak performance for a given GEMM operation as, 2·𝑀 ·𝑁 ·𝐾 · 𝑓 𝐿𝑟 (𝑜𝑝) where 𝐶𝑝𝑒𝑎𝑘 is the peak compute performance of the CPUs, 𝐵𝑝𝑒𝑎𝑘 is the peak memory bandwidth of the CPU, 𝐼 (𝑜𝑝) is the operational intensity of the GEMM operation, and 𝐿𝑟 (𝑜𝑝) is the Roofline latency. Formally, 𝑃 = min(𝐶𝑝𝑒𝑎𝑘 , 𝐵𝑝𝑒𝑎𝑘 · 𝐼 (𝑜𝑝)) =

𝐶𝑝𝑒𝑎𝑘 =

2 · 𝛿 𝑓 𝑚𝑎 · 𝑜𝑝.𝑇 · 𝑓 𝜏 𝑓 𝑚𝑎

𝐵𝑝𝑒𝑎𝑘 = max (𝐷𝑏𝑤 · 𝐷𝑛 ) 𝐷

2·𝑀 ·𝑁 ·𝐾 (𝑀 · 𝐾 + 𝐾 · 𝑁 + 𝑀 · 𝑁 ) · 𝑊𝑑 Substituting the definitions, the resulting latency estimate 𝐿𝑟 (𝑜𝑝) of the Roofline model is: 𝐼 (𝑜𝑝) =

𝐿𝑟 (𝑜𝑝) =

 𝑀 · 𝑁 · 𝐾 · 𝜏 𝑓 𝑚𝑎       𝛿 ·𝑇 𝑓 𝑚𝑎

if 𝐼 >

𝐶𝑝𝑒𝑎𝑘 𝐵𝑝𝑒𝑎𝑘

 (𝑀 · 𝑁 + 𝑀 · 𝐾 + 𝐾 · 𝑁 ) · 𝑊𝑑 · 𝑓   otherwise  𝑚𝑎𝑥 · 𝐷 𝑚𝑎𝑥  𝐷𝑏𝑤 𝑛  To compare, the Waypoint-Strict is designed to provide more realistic estimates of latency in comparison to the Roofline model. We capture the result in the following theorem, Theorem 2.1. For any GEMM operation 𝑜𝑝 on a given hardware configuration 𝐻 , 𝐿𝑟 (𝑜𝑝) ≤ 𝐿(𝑜𝑝). The theorem follows trivially from the definitions of 𝐿(𝑜𝑝) and 𝐿𝑟 (𝑜𝑝). In the compute bound case, the Waypoint-Strict accounts for additional cost in the compute latency 𝐿𝑐 such

'Zen 5'

'Zen 6'

INT8

BF16

FP32 0.01%

0.1%

1%

3%

0.01%

0.1%

1%

3%

Extra latency: Waypoint-Strict vs. Roofline

Figure 1. Additional latency captured by the WaypointStrict relative to Roofline across GEMM shapes, separated by hardware and data type. Boxes show the interquartile range and median; points show individual shapes.

as the cost of zeroing registers, loading, and storing data. Similarly, in the memory bound case, the effective bandwidth is at most 𝐷𝑏𝑤 · 𝐷𝑛 . In both cases, the additional threading overhead 𝐿𝑚𝑡 might be non-zero. Therefore, the WaypointStrict is tighter than Roofline. 2.1.5 Soundness of the assumptions. Figure 1 reinforces the theoretical result empirically. We compare the latency estimates of the Waypoint-Strict and the Roofline model on two different hardware configurations for many different shapes. The modest improvement over Roofline (upto 3%) is significant since performance of optimized GEMM libraries match the estimates (with 0.03% gap) for specific input matrix sizes. 2.2

Waypoint-Tile

While the Waypoint-Strict provides a tighter-than-roofline estimate, it does not provide a mechanism to realize the estimates. Therefore, we refine the Waypoint-Strict to account for a tiling-based GEMM implementation. The WaypointTile helps identify the best tiling parameters for a given GEMM operation on a given hardware configuration. Using the tiling parameters, we construct a GEMM implementation (Section 3) that demonstrates the realizability of the estimates. To identify the best tiling parameters, we augment the GEMM operation with three levels of tiling parameters: the register tiles (𝑀𝑟 , 𝑁𝑟 ) that form the two inner loops of Algorithm 1, the cache-tile parameters (𝑀𝑐 , 𝑁𝑐 , 𝐾𝑐 ), and the multithreading-tile parameters (𝑀𝑡 , 𝑁𝑡 ). The Waypoint-Tile computes the latency of the GEMM operation for a given tiling option. Note that Algorithm 1 is one possible instantiation of the tiled implementation. In our implementation, we select the loop ordering that works best for a given tile parameter set.

Waypoint: Analytically Modeling GEMM for CPUs

Given the Waypoint-Tile, we then search the space of possible tiling options to identify the best. While theoretically, the search space suffers from combinatorial explosion, we describe heuristics to prune the search. In Section 4, we also show that the best tiling parameters identified using the Waypoint-Tile improve over the state-of-the-art GEMM libraries across many different matrix shapes and hardware configurations. Waypoint-Tile makes certain assumptions of the behavior of the underlying hardware and implementation such as strided data accesses, fixed frequency workload, and using the best compute instruction available (e.g. AVX512 on x86 Systems). For the sake of brevity, we defer the details to Appendix C Algorithm 1 Tiled GEMM: 𝐶 += 𝐴 · 𝐵 Require: 𝐴 ∈ R𝑀 ×𝐾 , 𝐵 ∈ R𝐾 ×𝑁 , 𝐶 ∈ R𝑀 ×𝑁 Require: Thread tile (𝑀𝑡 , 𝑁𝑡 ), Macro tile (𝑀𝑐 , 𝑁𝑐 , 𝐾𝑐 ), Micro tile (𝑀𝑟 , 𝑁𝑟 , 𝐾𝑟 ) 1: for 𝑗𝑑 ← 0 to 𝑁 − 1 step 𝑁𝑡 2: for 𝑖𝑑 ← 0 to 𝑀 − 1 step 𝑀𝑡 3: for 𝑗𝑐 ← 𝑗𝑑 to 𝑗𝑑 + 𝑁𝑡 − 1 step 𝑁𝑐 4: for 𝑖𝑐 ← 𝑖𝑑 to 𝑖𝑑 + 𝑀𝑡 − 1 step 𝑀𝑐 5: for 𝑘𝑐 ← 0 to 𝐾 − 1 step 𝐾𝑐 6: for 𝑗𝑟 ← 𝑗𝑐 to 𝑗𝑐 + 𝑁𝑐 − 1 step 𝑁𝑟 7: for 𝑖𝑟 ← 𝑖𝑐 to 𝑖𝑐 + 𝑀𝑐 − 1 step 𝑀𝑟 8: 𝐶 [𝑖𝑟 , 𝑗𝑟 ] += 𝐴 [𝑖𝑟 , 𝑘𝑐 ] · 𝐵 [𝑘𝑐 , 𝑗𝑟 ] 9: end for 10: end for 11: end for 12: end for 13: end for 14: end for 15: end for Apart from providing accurate estimates of GEMM operation for a given tiling option, the Waypoint-Tile also ranks the different tiling options accurately, based on their expected performance. We show empirically that the estimates of the best-tiling options are accurate within 5% for existing hardware configurations. The Waypoint-Tile reformulates the compute cost 𝐿𝑐 taking the tiling parameters into account. Specifically, we first break up the computation into micro-kernels – the inner most loop that iterates over the (𝑀𝑟 , 𝑁𝑟 , 𝐾𝑐 ) dimensions. Then, for each micro-kernel, we account for additional costs such as cache access overheads when the tiling parameters are not optimal. Further, we add overheads for data movement costs at the macro-kernel level – the outer (𝑀𝑐 , 𝑁𝑐 ) loops. We redefine the compute cost 𝐿𝑐 as follows, 𝐿𝑐 = 𝐿𝜇 ∗ 𝑁 𝜇 + 𝐿𝜔 ∗ 𝑁𝜔 where 𝐿𝜇 and 𝑁 𝜇 are the latency and number of microkernels, and 𝐿𝜔 and 𝑁𝜔 are the latency and number of macrokernels.

We identify several analytical insights that capture the architectural behavior of the CPU and help prune the search space for the best tiling parameters. We describe the Waypoint-Tile in detail below. 2.2.1 Micro-kernel cost. The micro-kernel loop computes an output matrix of size 𝑀𝑟 × 𝑁𝑟 from input matrices of sizes 𝑀𝑟 × 𝐾𝑐 and 𝐾𝑐 × 𝑁𝑟 . The micro-kernel is the innermost loop of the GEMM operation, where the bulk of the computation occurs, and is typically implemented in handwritten assembly code optimized for the specific hardware. The data needed for the micro-kernel is fetched from the L1 cache (closest to the CPU). The handwritten assembly code for the micro-kernel has the following segments: (1) zero/load instructions to initialize the registers used to store the result; (2) prefetch instructions to fetch future data into the cache; (3) load instructions to load the input matrices into registers; (4) FMA instructions to perform the core computation; and (5) store instructions to store the result back to memory. We associate a cost with each of the above segments and compute their sum as the total cost of the micro-kernel. All components depend on the number of registers used in the hand-written assembly in relation to the number of registers available in the hardware. Reuse Overhead. A key component of the micro-kernel latency cost is accurately modeling the cost of loading data. Optimal kernels under a perfect cache incur no cost to load data because of out-of-order execution. However, depending on the size of the data loaded (in terms of cache lines), a fraction of the load instructions will miss the cache and incur additional costs. We refer to this cost as reuse overhead. The reuse overhead is computed as the number of cache lines that incur a cache miss multiplied by the effective access latency from the L2 cache. Let 𝐴𝑟𝑒𝑔 and 𝐵𝑟𝑒𝑔 to be the number of registers used to store data from the input matrices 𝐴 and 𝐵 (derived using 𝑀𝑟 and 𝑁𝑟 respectively). Then, then number of registers used to store the result, 𝐶𝑟𝑒𝑔 is 𝐶𝑟𝑒𝑔 = 𝐴𝑟𝑒𝑔 · 𝐵𝑟𝑒𝑔 . Together, the total number of registers used in one loop iteration of the micro-kernel is 𝑅𝑢𝑠𝑒𝑑 = 𝐴𝑟𝑒𝑔 + 𝐵𝑟𝑒𝑔 + 𝐶𝑟𝑒𝑔 . Two equations govern the computation of reuse-overhead and, in effect, determine the optimal values of 𝑀𝑟 and 𝑁𝑟 . First, a lower bound on the register pressure, defined as the ratio of input and output registers. Second, an upper bound on the load-to-fma latency ratio, defined as the ratio of the load instruction latency to the FMA instruction latency. Formally, Lemma 2.2. Register pressure equation: The ratio of input and output registers is lower bounded by an architecturedependent constant 𝛼. 𝐴𝑟𝑒𝑔 + 𝐵𝑟𝑒𝑔 ≥𝛼 𝐶𝑟𝑒𝑔

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

Register-pressure bound

Algorithm 2 Macro-kernel cost 𝐿𝜔

Feasible tiles

Require: Working-set size 𝑆, L2 size 𝑆𝐿2 , L3 size 𝑆𝐿3 , thread count 𝑇 for a given hardware configuration 𝐻 1: procedure CapacityMissCost(𝐻,𝑇 , 𝑆) 2: if 𝑆 ≤ 𝑆𝐿2 then 3: return 0 4: end if 5: 𝑆𝐿3-𝑟𝑒𝑠𝑖𝑑𝑒𝑛𝑡 ← min(𝑆, 𝑆𝐿3 ) − 𝑆𝐿2 6: 𝐿 ← MoveFromL3Latency(𝐻, 𝑆𝐿3-𝑟𝑒𝑠𝑖𝑑𝑒𝑛𝑡 ) 7: if 𝑆 > 𝑆𝐿3 then 8: 𝐿 ← 𝐿 +FetchFromMemLatency(𝐻, 𝑆 −𝑆𝐿3,𝑇 ) 9: end if 10: return 𝐿 11: end procedure

Load/FMA bound

10

8

Breg

6

4

2

loads fewer bytes is preferred since it minimizes the number of total memory accesses.

0 0

2

4

6

8

10

Areg

Figure 2. Candidate register tile sizes (𝑀𝑟 , 𝑁𝑟 ) for AMD ‘Zen 5’ architecture. The candidates are generated by enumerating all possible values of 𝑀𝑟 and 𝑁𝑟 that satisfy the register pressure and load-to-fma latency ratio bounds. For a given hardware architecture 𝐻 , we calculate the bound 𝛼 based on integer solutions to the equation, 𝐴𝑟𝑒𝑔 + 𝐵𝑟𝑒𝑔 + 𝐶𝑟𝑒𝑔 ≤ 𝑁𝑟𝑒𝑔 Lemma 2.3. Load-to-fma latency ratio: The ratio of the load instruction latency to the FMA instruction latency is upper bounded by 1. (𝐴𝑟𝑒𝑔 + 𝐵𝑟𝑒𝑔 ) · 𝜏𝑙𝑜𝑎𝑑 ≤1 𝐶𝑟𝑒𝑔 · 𝜏 𝑓 𝑚𝑎 When either of the conditions are violated, the microkernel stalls to load the data, incurring a non-zero reuse cost. When register pressure bound is violated, the micro-kernel is using more registers than available and hence leads to register spilling. Similarly, when load-to-fma latency ratio bound is violated, some of the loads are not hidden due to pipelining. The additional loads incur a cost of loading from the higher level cache. The equations, along with the assumption that 𝑁𝑟𝑒𝑔 is small (typically ≤ 32), effectively reduce the search space for optimal 𝑀𝑟 and 𝑁𝑟 (Figure 2 highlights the possible candidates for ‘Zen 5’). Therefore, we are able to enumerate all possibilities and find the optimal one. To further narrow the list, we order them based on the number of bytes loaded from memory in one iteration. The candidate that

2.2.2 Macro-kernel cost. The micro-kernel cost described above accounts for all the compute cost of the GEMM operation. In contrast, the macro-kernel cost we describe below accounts for the cost of moving data between the primary memory and the L2 cache. The macro-kernel computes an output matrix of size 𝑀𝑐 × 𝑁𝑐 using inputs of size 𝑀𝑐 × 𝐾𝑐 and 𝐾𝑐 × 𝑁𝑐 , and is where the bulk of data movement happens. When computing the output, the macro-kernel loop streams through either the 𝐴 or 𝐵 matrix. The number of cache misses depends on which matrix is streamed. Without loss of generality, we describe the macro-kernel cost assuming 𝐵 is streamed. We define the macro-kernel cost 𝐿𝜔 as, 𝐿𝜔 = CapacityMissCost(𝐻,𝑇 , 𝑆𝐴𝑐 + 𝑆𝐶𝑐 ) + 𝛽 · CapacityMissCost(𝐻,𝑇 , 𝑆 𝐵𝑐 )

(1)

where 𝑆𝐴𝑐 = 𝑀𝑐 ∗ 𝐾𝑐 ∗ 𝑊𝑑 , 𝑆 𝐵𝑐 = 𝐾𝑐 ∗ 𝑁𝑐 ∗ 𝑊𝑑 , and 𝑆𝐶𝑐 = 𝑀𝑐 ∗ 𝑁𝑐 ∗ 𝑊𝑑 are the sizes of the input matrices A and B and output matrix C respectively. CapacityMissCost(𝐻,𝑇 , 𝑆) denotes the cost of a capacity miss for a working set of size 𝑆 and is defined in Algorithm 2. Essentially, if the data set overflows L2, then we compute the cost of moving data from L3 to L2 and similarly if the data overflows L3, we compute the cost of fetching the overflowed data from memory to L3. We compute the capacity cost independently for the B matrix, and the A and C matrices together due to their differences in access patterns. Additionally, when 𝐾𝑐 < 𝐾, we account for the cost of accumulating the partial results by including the cost of loading the result matrix. The parameter 𝛽 ∈ [0, 1] in (1) is an architecture-dependent cost determined by the aggressiveness of the prefetcher. The prefetcher performs well when the data access pattern follows a fixed stride length, as it does when we stream through the B matrix in the macro-tile. Therefore, only a fraction of the B matrix incurs a capacity-miss cost. However, the A

Waypoint: Analytically Modeling GEMM for CPUs

and C matrices are not streamed, so we do not expect the prefetcher to perform well. 2.3

Seed programs candidate C++ heuristics

LLM + config prompt, models, MAP-Elites

OpenEvolve search mutate, select, archive

Packing

GEMM libraries expect that the input matrices are pre-packed into memory a specific layout that depends on the tiling parameters. They expose routines to pack the matrices into the required format before performing the matrix multiplication. Without packing the inputs, the performance of the resulting GEMM operation degrades. The instruction used to load data in the innermost kernel differs from the optimal one, leading to inefficient memory accesses and suboptimal utilization of the cache hierarchy. Since the packing format depends on the tiling parameters, adopting a shape-dependent tiling strategy incurs additional costs. However, with Waypoint-Tile, we identify a common packing strategy that works across different tiling parameters. Specifically, the packed layout should ensure two invariants: each access loads an entire cache line, and successive access locations follow a fixed stride length. We find that additional packing optimizations do not yield significant performance improvements and only lead to higher packing costs. Therefore, in libzenmm, we pack the input matrices into blocks of (𝑀𝑝 × 𝐾𝑝 ), where 𝑀𝑝 and 𝐾𝑝 are the least common multiples of all possible values of 𝑀𝑟 and 𝐾𝑟 , respectively.

3

Initial seeds handwritten heuristic families

Model based GEMM library

To demonstrate that the parameters obtained from Waypoint are realizable and achieve the expected performance, we implement a GEMM library libzenmm that uses the parameters derived from Waypoint-Tile for a given GEMM operation and hardware. libzenmm is implemented in C++. In the library, we first use the tiling parameters generated by Waypoint to break down the large GEMM operation and then dispatch the micro-kernel to a JIT based small GEMM library included in libzenmm. To break down the large GEMM, we use Parlooper[15], a library that dynamically generates loops with support for OMP based parallelization. Then, in the inner most loop, the JIT based small GEMM library generates the assembly instructions to compute the micro-kernel and caches it for subsequent uses. As an alternative to the JIT based GEMM library, we evaluated Libxsmm [17] and found it lacking the flexibility needed to generate kernels based on analytical-model-derived parameters. Therefore, we use our own JIT based small GEMM library. In Section 4 we report the GEMM performance comparison with state-of-the art GEMM libraries and show that libzenmm is competitive. libzenmm supports two methods for finding the best tiling parameters: exhaustive search and fast heuristic-driven search. During exhaustive search, we evaluate all possible tiling parameters after pruning the search space based on insights

Materialize candidate

Compile + run Pinned reference set 396 GEMM shapes

Score candidate

Best heuristic single-answer C++ decision function

Inputs Search Evaluation Outcome

Figure 3. High-level workflow for evolving a Waypoint CPU split-factor heuristic. Seed heuristics and LLM-generated mutations are evaluated against pinned exhaustive-search optima; the resulting metrics drive selection of the best heuristic.

from Waypoint-Tile. Specifically, we search only optimal 𝑀𝑟 and 𝑁𝑟 values and bound the 𝑀𝑐 , 𝑁𝑐 , and 𝐾𝑐 values based on the cache sizes. However, for latency-sensitive applications such as LLM inference, the fast heuristic search offers a quick alternative. The fast-heuristic search is developed using recent advances in algorithmic evolution [8, 9, 27]. We use OpenEvolve [28], an open-source implementation of AlphaEvolve [27], to discover the heuristics. Our evolution process is focused on obtaining fast heuristics for a curated set of GEMM operations (or shapes) that modern LLM inference workloads invoke. However, the methodology can be used to discover heuristics that are tailored for any set of GEMM operations that the workload demands. The fast-heuristic generation is a one-time cost and the resulting heuristics can be used for all future GEMM operations. Figure 3 demonstrates the flow to obtain the heuristic search algorithm. OpenEvolve starts with a set of seed programs as candidates for heuristics search. Our seeds are derived using insights from the tiling parameters generated by Waypoint-Tile. Then, OpenEvolve runs for many iterations, generating candidate heuristics in a genetic-mutation based algorithm. In each iteration we check the correctness of the tiling parameters and use Waypoint-Tile is used to

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

Table 1. Evaluated GEMM shapes Category

Count

Small GEMM (<512) Medium Square (512-4096) Large GEMM (>4096) Tall skinny (M»N) Fat Short (M«N) Odd Shapes LLM Shapes

4 9 8 4 2 4 20

score the candidates. A candidate is good if the estimate of the selected tile parameter is not far from the best known. After running for 100 iterations, OpenEvolve generates a fast-heuristic search algorithm that can be used to very quickly select near-optimal tiling parameters for a given set of GEMM operations and hardware. We defer the details of the OpenEvolve configuration to Appendix A. The resulting fast-heuristic search is 18851× faster than the exhaustive search and discovers tiling parameters whose estimated latency is within 1.94% (maximum) of the best tiling parameters derived by Waypoint1 . In our evaluation we show that the fast-heuristic search accelerates LLM performance on both current and future hardware. Our results include the cost of the fast heuristic search.

4

Evaluation

We evaluate the accuracy of our analytical model and the performance of libzenmm relative to three existing state-ofthe-art libraries, AOCLTM -BLAS [3], AOCLTM -DLP [2], and SFC [13]. AOCL-BLAS and AOCL-DLP are state-of-the-art tiling-based GEMM libraries that are optimized for AMD CPUs and support low-precision GEMM kernels used in machine learning workloads. While AOCL-BLAS uses handwritten micro-kernels, AOCL-DLP contains a JIT engine that dynamically generates the micro-kernel. In contrast, SFC is applicable to all x86 processors and implements a novel traversal mechanism (using space-filling curves) to minimize data access costs. Our evaluation is focused on the AMD Zen architecture. Specifically, we evaluate on ‘Zen 5’ (Turin) [11] and ‘Zen 6’ (Venice). Across both architectures, all our evaluations are performed for 3 different data types, FP32, BF16, and INT8. We evaluate the performance of the libzenmm for a set of 51 unique GEMM shapes (summarized in Table 5) (306 comparisons in total) that are commonly used in modern LLM inference workloads. A detailed list of the shapes used to evaluate can be found in Appendix B. With our evaluation we aim to answer the following research questions: RQ1 Is the model accurate for different data types and architectures?

RQ2 How well does libzenmm compare to state-of-the-art implementations? RQ3 Does using the analytical model based libzenmm improve the performance of end-to-end LLM inference? The Waypoint-Strict and Waypoint-Tile models are evaluated against the Roofline model and the actual performance of GEMM operations implemented in the three libraries. Waypoint-Strict is tighter than Roofline and predicts actual performance to within 9.6%. Waypoint-Tile is accurate both in ranking different tiling options and in estimating the performance of the best tiling option compared with the libraries (geometric mean of 1.2%). libzenmm achieves performance comparable to that of the three libraries across all configurations. When including packing costs, libzenmm beats AOCL-BLAS by 16.7%, AOCLDLP by 13.6%, and SFC by 21.0%. Further, when comparing pure GEMM computation, libzenmm is 10.8% faster than AOCL-BLAS and achieves performance similar to AOCLDLP (0.85% slower) and SFC (0.8% faster). Finally, we evaluate end-to-end LLM inference performance using PACETM [4], a vLLM-compatible framework with optimized inference kernels for AMD CPUs. We embed the fast-heuristic search in PACE and evaluate it against a baseline implementation that does not use Waypoint. We find that the shape-adaptive tiling strategy of Waypoint yields average time-to-first-token (TTFT) speedups of 1.36× to 1.98× across different batch sizes, input lengths, models, and hardware architectures. For the remainder of the section, we first describe the experimental setup and then elaborate on the results for each of the research questions. 4.1

Experimental Setup

The models are compared against two hardware architectures. One, AMD ‘Zen 5’ (Turin) EPYC 9755 128-Core Processor with 512MB of L3 cache running at a fixed frequency of 2.7GHz. The other, AMD ‘Zen 6’ (Venice) EPYC 9996 256Core Processor with 1GB of L3 cache running at a fixed frequency of 2.5GHz. We were provided advanced access to ‘Zen 6’ processor by AMD. While production systems are run with variable frequency, we fix the frequency to ensure consistent performance measurements and to isolate the performance improvements of our approach. We present comparison numbers to AOCL-BLAS and AOCL-DLP for all 3 data types (Int8, BFloat16, and FP32), and SFC for only 2 data types (BFloat16 and FP32) since SFC does not support Int8. The libzenmm is implemented in C++ and compiled using GCC 13.3.0. All comparisons are done by measuring cycles using the rdtsc instruction. To mitigate the effects of randomness when measuring the performance of GEMM libraries, for each shape, we run 10 warm-up trials and 100 measurement trials and report the best latency numbers. All numbers are reported using geometric mean.

Waypoint: Analytically Modeling GEMM for CPUs

Table 2. Roofline vs Waypoint-Strict comparison for different data types across architectures. We report the median, mean, and max ratios of Waypoint-Strict to Roofline estimates across different shapes. A value of ‘1.000x‘ means Waypoint-Strict and Roofline estimate match.

Figure 4. Waypoint-Tile vs Waypoint-Strict vs Implementations (AOCL-BLAS, SFC, and AOCL-DLP). We report the ratios and distribution for all the shapes. 0.94×

Waypoint-Strict

Datatype

Threads

n

Median

Mean

Max

AOCL-BLAS

Int8 Int8 Int8 BFloat16 BFloat16 BFloat16 Float32 Float32 Float32

1 2-8 >8 1 2-8 >8 1 2-8 >8

58 24 20 58 24 20 58 24 20

1.001x 1.000x 1.000x 1.001x 1.000x 1.001x 1.001x 1.001x 1.002x

1.005x 1.001x 1.001x 1.004x 1.002x 1.001x 1.004x 1.003x 1.002x

1.031x 1.011x 1.002x 1.031x 1.013x 1.005x 1.031x 1.014x 1.006x

SFC

n/a 1.02×

AOCL-DLP 0.94×

Waypoint-Strict

1.00×

SFC

0.96×

AOCL-DLP 0.94×

FP32

1.00×

AOCL-BLAS

4.2

BF16

0.98×

AOCL-BLAS

Waypoint-Strict

For measuring end-to-end LLM inference times, we integrate libzenmm into AMD-PACE[4]. Our measurement compares against a baseline version that uses TPP [14] to dispatch GEMM calls. Our inference runs on 128 threads.

INT8

1.05×

1.01×

SFC

0.97×

AOCL-DLP 0.5

0.6

0.8

1

2

Cycles / Waypoint-Tile model

RQ1: Model accuracy

Table 2 shows the comparison of Waypoint-Strict against the Roofline model across different data types and architectures. As shown, Waypoint-Strict is tighter than the Roofline model. Specifically when the data needed for computation is large, Waypoint-Strict deviates from the Roofline (up to 1.031x) accounting for the storage, and multi-threaded execution costs. Waypoint-Tile predicts the performance of a given tiling solution more accurately than Waypoint-Strict but does not guarantee a tight upper bound. Figure 4 compares the predictions of Waypoint-Tile and Waypoint-Strict with the measured performance of the three implementations. As shown, Waypoint-Strict serves as a tight upper bound on GEMM performance. The depicted AOCL-BLAS performance measurements form a bimodal distribution, with the two modes corresponding to the different architectures (‘Zen 5’ and ‘Zen 6’). Further, the Waypoint-Tile is able to accurately rank different tiling options for a given GEMM shape. To ensure the correctness of the ranking, we evaluate the 1st, 5th, and 10th tiling options for each GEMM shape and compare the predicted performance against the measured performance. We find that the Waypoint-Tile is able to accurately rank the top tiling option for 88.7% (within 5% of the peak performance) of the evaluated GEMM shapes and strictly the best in 62.7% of the cases. Future hardware predictions. A key contribution of Waypoint is its ability to predict the performance of future

Table 3. Predicted geometric-mean GEMM speedup from the proposed AI Compute Extensions (ACE) across 44 shapes per datatype vs ‘Zen 5’ hardware. ACE throughput is effective number of Tile operations that can be issued per cycle. ACE throughput BFloat16 1 2 4

7.21x 12.84x 21.47x

Int8 7.45x 13.48x 23.00x

proposed hardware. To demonstrate this, we evaluate the performance of the GEMM operation on a hypothetical future hardware that implements the proposed AI compute instructions [33] where one Tile operation computes a 16 × 16 × 4 (for Int8) and 16 × 16 × 2 (for BFloat16). Table 3 shows the predicted performance gap of the GEMM averaged across all shapes for different throughput configurations of the AI compute instructions relative to ‘Zen 5’ hardware. In ‘Zen 5’, the VNNI instruction can be issued at the rate of 2/cycle and the tile op does the work equivalent to 16 VNNI instruction. As shown in the table, the model tempers the expected performance improvements of 8× (1 tile op/cycle),16×, and 32×, accounting for the data movement costs when compute is accelerated. The result shows that adding many compute units will not automatically scale the performance linearly because the kernels will become memory bound.

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

4.3

RQ2: libzenmm performance comparison

Figure 5 compares the performance of the libzenmm relative to three state-of-the-art implementations and the WaypointTile estimates for the same set of GEMM shapes. We observe that the libzenmm closely follows the performance predicted by the Waypoint-Tile, demonstrating the accuracy and practical utility of our approach. On ‘Zen 5’, libzenmm meets the performance of existing, highly optimized GEMM libraries. However on ‘Zen 6’ the libraries are yet to be tuned (especially for the Int 8 datatype) as the hardware is not readily available, but libzenmm demonstrates a significant performance advantage over the existing implementations. Packing cost differences The different implementations also differ in their packing strategies. As mentioned in 2.3, the implementations extract peak performance by assuming that the data is pre-packed into contiguous memory regions in a specific format. For example, SFC requires that the data is laid out in the sequence of the space filling curve. This additional transformation increases the packing cost compared to more straightforward packing strategies used in AOCL-BLAS, AOCL-DLP, and libzenmm. When comparing GEMM performance, SFC often achieves higher throughput due to its optimized data layout and tiling strategies. However, this comes at the cost of increased packing overhead. Comparing the packing costs, we find that libzenmm achieves a 1.40× speedup over AOCL-DLP, 1.29× over AOCL-BLAS, and 2.44× over SFC. We include a detailed table in Appendix B. The higher packing cost can offset the performance gains in scenarios where the packing overhead is significant relative to the time taken to compute the result. Discussion Including both the packing cost and GEMM latency, we observe that libzenmm is overall 1.168× faster than AOCL-BLAS, 1.210× faster than SFC, and 1.136× faster than AOCL-DLP. Conceptually, the gains of libzenmm over AOCL-BLAS and AOCL-DLP are primarily due to its shapeadaptive tiling strategies and comparable packing cost. In contrast, the gains over SFC are primarily driven by significantly lower packing cost and comparable GEMM performance matching the data reuse achieved by the optimized SFC algorithm. 4.4

RQ3: End-to-end LLM inference performance

We evaluate the impact of Waypoint on end-to-end inference performance of Large Language models using, AMDPACE [4]. Figure 6 shows the average end-to-end inference speedup on prefill heavy inference workloads that primarily rely on the performance of GEMM calls. Other workloads that involve more GEMV operations retain existing performance levels. As shown, Waypoint consistently improves inference performance across different hardware and model configurations. The graph depicts speedups across batch sizes of 8, 16, 32, 64, and 128 for input lengths of 1024 and 2048. With higher batch

sizes, the cost of inference heavily skews towards GEMM operation latency. Therefore, by integrating Waypoint, we see the highest gains in higher batch sizes (e.g. 3.6× for batch size 128 on ‘Zen 5’ with OPT 6 7B and 9.15× for batch size 128 on ‘Zen 6’ with Qwen 3 8B). The gains are attributed primarily to improvements in GEMM computation times of the odd shapes encountered in LLM inference, driven by the shape-adaptive tiling strategy of Waypoint. The graph also shows the ability of Waypoint to accelerate inference on new hardware. The range of speed up is larger in ‘Zen 6’ compared to ‘Zen 5’. 4.5

Extending to other architectures

While our focus in the evaluation is on AMD x86 processor family, the principles used in both Waypoint-Strict and Waypoint-Tile are general and can be easily extended to other architectures. The estimates of performance depend primarily on a layered memory hierarchy and a pipelined vector/matrix processing unit. The model immediately translates to Intel x86 processors given their similarity in ISA and hardware architecture to AMD x86 processors. Further, the model can be adopted to ARM processors. ARM processors also feature a layered memory hierarchy and pipelined vector/matrix processing units, although the specific details of the cache sizes, memory bandwidth, and vector instruction sets differ. Specifically, the variable vector length in ARM’s Scalable Vector Instruction and Scalable Matrix Extension fit well with the parametric nature of Waypoint-Strict and Waypoint-Tile. The length and number of lanes of the registers are parametric in the model and can be configured based on the hardware architecture.

5

Related work

There has been extensive work on modeling and optimizing the performance of General Matrix Multiplication (GEMM) operations on various hardware architectures. GotoBLAS [16] introduced a layered approach to implement GEMM operations in the BLAS standard library [1]. They identified the optimization that most libraries [2, 3, 30] today adopt. Namely, that breaking down the large GEMM operation into smaller blocks (tiles) significantly improves cache utilization and overall performance. Subsequent works [6, 7, 10] developed empirical methods to identify the optimal tiling parameters for a given hardware architecture and study the effects of different tiling choices on performance [5]. However, to overcome the time-consuming and resource intensive process of empirically identifying the best tiling parameters, analytical model approaches [24, 34, 35] have been proposed. These models [26] adapt to different hardware and to specific classes of shapes [31]. Waypoint extends prior works to identify optimal tiling parameters by being

Waypoint: Analytically Modeling GEMM for CPUs

1.03×

Waypoint-Tile

'Zen 5'

'Zen 6'

0.87×

1.02×

INT8

1.48×

AOCL-BLAS SFC

n/a 1.00×

n/a

1.01×

0.99×

1.17×

AOCL-DLP Waypoint-Tile

0.96×

AOCL-BLAS SFC

BF16

1.22×

0.98×

1.00×

0.94×

0.98×

AOCL-DLP 1.01×

Waypoint-Tile

0.98×

0.99×

AOCL-BLAS SFC

FP32

1.22×

1.00×

0.99×

0.96×

0.97×

AOCL-DLP 0.6

0.8

1

2

Cycles / OPAM

3

0.6

0.8

1

2

3

Cycles / OPAM

Figure 5. Relative performance of AOCL-BLAS, SFC, AOCL-DLP, and the Waypoint-Tile vs libzenmm, separated by hardware architecture.

'Zen 5'

'Zen 6'

1.36×

Llama 3.1 8B

1.37×

1.83×

OPT 6 7B

1.98×

1.62×

Qwen 2.5 7B

1.65×

Qwen 3 8B 0

5

0

5

Geometric-mean TTFT speedup

Figure 6. End-to-end inference speedup of Waypoint on prefill heavy LLM workloads across different hardware and models. Bars show the geometric mean; whiskers show the full range across batch sizes and input lengths.

parametric and general not just in the hardware architecture but also in the specific input matrix shape. The performance of tile-based GEMM implementations are sensitive to the data-layout of the input matrices in memory. To alleviate this, recent works have borrowed ideas from communication-avoiding algorithms [12, 13] to reduce data movement and improve cache utilization. They propose and show that by carefully organizing the input matrices

in the order of a Space Filling Curve reduces cache misses and helps extract peak performance. Waypoint models are general enough to accurately predict the performance of such alternative implementations while at the same time, matching their performance with libzenmm. While the traditional Roofline model [32] has been widely adopted to predict peak performance on proposed hardware designs, it fails to provide realistic estimates. As a result, the software never ceases to be optimized on existing hardware and hardware designers are unable to predict the realistic achievable performance on proposed hardware. ExecutionCache-Memory (ECM) [18, 19], instruction-level models [22], and GPU specific models [25] have extended the Roofline model to capture different architectural effects and provide more realistic estimates. However, these models are either heavily tailored to a specific architecture, or require execution traces to be collected from the target hardware. Waypoint furthers the research by being both accurate and parametric in the hardware architecture, allowing hardware designers to get reliable estimates of the realistic performance achievable. Our work is closely motivated by a similar effort focused on GPUs [29] which predicts performance of GEMM operations on GPUs by accounting for cache-miss rates, memory access overheads and other architectural effects. Our work Waypoint, generalizes the approach while demonstrating its applicability to CPUs.

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

6

Discussion and Conclusion

Under multi-threaded GEMM operations, Waypoint makes simplistic assumptions on the strategy to split work among different threads such as equal distribution of tiles or static scheduling, which may not always reflect the optimal parallel execution strategy. We observe that for certain input matrix sizes, using lower than the prescribed number of threads can yield better performance, highlighting the need for more sophisticated thread management strategies in future work. We present Waypoint, a two part analytical model which first provides a tighter than Roofline estimate for GEMM performance and second, helps select the optimal tiling parameters to realize the estimates. The model derives several insights into estimating, and narrowing the search for optimal tiling parameters, ultimately enabling high-performance GEMM across diverse hardware architectures. We implement a GEMM library libzenmm that uses the analytical model of Waypoint to guide its tiling decisions, and show that it achieves performance close to the theoretical estimates. libzenmm matches the performance of state-of-theart GEMM libraries on existing hardware while showing considerable gains on future hardware architectures. Finally, we show that using libzenmm and Waypoint speeds up endto-end CPU inference tasks significantly, demonstrating the practical impact of our approach.

References [1] 2002. An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Softw. 28, 2 (June 2002), 135–151. https://doi.org/ 10.1145/567806.567807 [2] 2026. amd/aocl-dlp. https://github.com/amd/aocl-dlp original-date: 2025-09-02T10:58:34Z. [3] Advanced Micro Devices, Inc. 2025. AMD AOCL-BLAS: BLAS-like Library Instantiation Software Framework. https://github.com/amd/ blis. Accessed: 2026-07-06. Micro Devices, Inc. 2026, 1.2.0. [4] Advanced AMD PACE: AMD Platform Aware Compute Engine. https: //github.com/amd/amd-pace [5] Guillermo Alaejos, Adrián Castelló, Pedro Alonso-Jordá, Francisco D. Igual, Héctor Martínez, and Enrique S. Quintana-Ortí. 2024. Algorithm 1039: Automatic Generators for a Family of Matrix Multiplication Routines with Apache TVM. ACM Trans. Math. Softw. 50, 1, Article 6 (March 2024), 34 pages. https://doi.org/10.1145/3638532 [6] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan RaganKelley, Jeffrey Bosboom, Una-May O’Reilly, and Saman Amarasinghe. 2014. OpenTuner: An Extensible Framework for Program Autotuning. In Proceedings of the 23rd International Conference on Parallel Architectures and Compilation (PACT ’14). Association for Computing Machinery, 303–316. https://doi.org/10.1145/2628071.2628092 [7] Jeff Bilmes, Krste Asanovic, Chee-Whye Chin, and Jim Demmel. 1997. Optimizing matrix multiply using PHiPAC: a portable, highperformance, ANSI C coding methodology. In Proceedings of the 11th International Conference on Supercomputing (Vienna, Austria) (ICS ’97). Association for Computing Machinery, New York, NY, USA, 340–347. https://doi.org/10.1145/263580.263662 [8] Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Shubham Agarwal, Mert Cemri, Bowen Wang, Alexander Krentsel, Tian Xia, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Ashwin Naren, Shulu Li, Ruiying Ma, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. 2025. Let the Barbarians In: How AI Can Accelerate Systems Performance Research. arXiv:2512.14806 [cs.SE] https: //arxiv.org/abs/2512.14806 [9] Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. 2025. Barbarians at the Gate: How AI is Upending Systems Research. arXiv:2510.06189 [cs.AI] https://arxiv. org/abs/2510.06189 [10] R Clint Whaley, Antoine Petitet, and Jack J Dongarra. 2001. Automated empirical optimizations of software and the ATLAS project. Parallel Comput. 27, 1-2 (Jan. 2001), 3–35. [11] Brad Cohen, Mahesh Subramony, and Mike Clark. 2024. Next Generation "Zen 5" Core. In 36th IEEE Hot Chips Symposium, HCS 2024, Stanford, CA, USA, August 25-27, 2024. IEEE, 1–27. https: //doi.org/10.1109/HCS61935.2024.10665102 [12] James Demmel, David Eliahu, Armando Fox, Shoaib Kamil, Benjamin Lipshitz, Oded Schwartz, and Omer Spillinger. 2013. CommunicationOptimal Parallel Recursive Rectangular Matrix Multiplication. In 2013 IEEE 27th International Symposium on Parallel and Distributed Processing. IEEE, 261–272. https://doi.org/10.1109/IPDPS.2013.80 [13] Evangelos Georganas, Alexander Heinecke, and Pradeep Dubey. 2026. Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple. arXiv:2601.16294 [cs.DC] https://arxiv. org/abs/2601.16294 [14] Evangelos Georganas, Dhiraj Kalamkar, Sasikanth Avancha, Menachem Adelman, Cristina Anderson, Alexander Breuer, Jeremy Bruestle, Narendra Chaudhary, Abhisek Kundu, Denise Kutnick, Frank Laub, Vasimuddin Md, Sanchit Misra, Ramanarayan Mohanty, Hans Pabst, Barukh Ziv, and Alexander Heinecke. 2021. Tensor processing

Waypoint: Analytically Modeling GEMM for CPUs primitives: a programming abstraction for efficiency and portability in deep learning workloads. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 14, 14 pages. https: //doi.org/10.1145/3458817.3476206 [15] Evangelos Georganas, Dhiraj Kalamkar, Kirill Voronin, Abhisek Kundu, Antonio Noack, Hans Pabst, Alexander Breuer, and Alexander Heinecke. 2024. Harnessing Deep Learning and HPC Kernels via HighLevel Loop and Tensor Abstractions on CPU Architectures. In 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 950–963. https://doi.org/10.1109/IPDPS57955.2024. 00089 [16] Kazushige Goto and Robert A. van de Geijn. 2008. Anatomy of HighPerformance Matrix Multiplication. ACM Trans. Math. Software 34, 3 (May 2008), 1–25. https://doi.org/10.1145/1356052.1356053 [17] Alexander Heinecke, Greg Henry, Maxwell Hutchinson, and Hans Pabst. 2016. LIBXSMM: accelerating small matrix multiplications by runtime code generation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Salt Lake City, Utah) (SC ’16). IEEE Press, Article 84, 11 pages. https://doi.org/10.1109/SC.2016.83 [18] Johannes Hofmann, Dietmar Fey, Jan Eitzinger, Georg Hager, and Gerhard Wellein. 2016. Analysis of Intel’s Haswell Microarchitecture Using the ECM Model and Microbenchmarks. In Architecture of Computing Systems – ARCS 2016, Frank Hannig, João M. P. Cardoso, Thilo Pionteck, Dietmar Fey, Wolfgang Schröder-Preikschat, and Jürgen Teich (Eds.). Springer International Publishing, Cham, 210–222. [19] Aleksandar Ilic, Frederico Pratas, and Leonel Sousa. 2014. Cache-aware Roofline model: Upgrading the loft. IEEE Computer Architecture Letters 13, 1 (2014), 21–24. https://doi.org/10.1109/L-CA.2013.6 [20] Intel. 2026. Introducing Intel® Advanced Performance Extensions (Intel® APX) — intel.com. https://www.intel.com/content/www/us/ en/developer/articles/technical/advanced-performance-extensionsapx.html [Accessed 10-09-2026]. [21] Intel Corporation. 2022. Accelerate Artificial Intelligence (AI) Workloads with Intel Advanced Matrix Extensions (Intel AMX). Solution Brief. Intel Corporation. https://www.intel.com/content/ dam/www/central-libraries/us/en/documents/2022-12/accelerateai-with-amx-sb.pdf [22] Jan Laukemann, Julian Hammer, Johannes Hofmann, Georg Hager, and Gerhard Wellein. 2018. Automated Instruction Stream Throughput Prediction for Intel and AMD Microarchitectures. In 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). 121–131. https: //doi.org/10.1109/PMBS.2018.8641578 [23] Chendi Li, Haipeng Jia, Hang Cao, Jianyu Yao, Boqian Shi, Chunyang Xiang, Jinbo Sun, Pengqi Lu, and Yunquan Zhang. 2021. AutoTSMM: An Auto-tuning Framework for Building HighPerformance Tall-and-Skinny Matrix-Matrix Multiplication on CPUs. In 2021 IEEE Intl Conf on Parallel and Distributed Processing with Applications, Big Data and Cloud Computing, Sustainable Computing and Communications, Social Computing and Networking (ISPA/BDCloud/SocialCom/SustainCom). 159–166. https://doi.org/10. 1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00034 [24] Tze Meng Low, Francisco D. Igual, Tyler M. Smith, and Enrique S. Quintana-Orti. 2016. Analytical Modeling Is Enough for HighPerformance BLIS. ACM Trans. Math. Softw. 43, 2, Article 12 (Aug. 2016), 18 pages. https://doi.org/10.1145/2925987 [25] Sangkug Lym, Donghyuk Lee, Mike O’Connor, Niladrish Chatterjee, and Mattan Erez. 2019. DeLTA: GPU Performance Model for Deep Learning Applications with In-Depth Memory System Traffic Analysis. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 293–303. https://doi.org/10.1109/

ISPASS.2019.00041 [26] Héctor Martínez, Sandra Catalán, Francisco D Igual, José R Herrero, Rafael Rodríguez-Sánchez, and Enrique S Quintana-Ortí. 2025. Cacheaware optimization of matrix multiplication and matrix factorizations on multicore processors. Cluster Comput. 28, 12 (Nov. 2025). [27] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv:2506.13131 [cs.AI] https://arxiv.org/abs/2506.13131 [28] Asankhaya Sharma. 2025. OpenEvolve: an open-source evolutionary coding agent. https://github.com/algorithmicsuperintelligence/ openevolve [29] Ryan Swann, Muhammad Osama, Xiaohu Guo, Bryant Nelson, Lixun Zhang, Alex Brown, Yen Ong, Ali Yazdani, Sean Siddens, Ganesh Dasika, and Alex Underwood. 2025. tritonBLAS: Tritonbased Analytical Approach for GEMM Kernel Parameter Selection. arXiv:2512.04226 [cs.DC] https://arxiv.org/abs/2512.04226 [30] Field G. Van Zee and Robert A. van de Geijn. 2015. BLIS: A Framework for Rapidly Instantiating BLAS Functionality. ACM Trans. Math. Software 41, 3 (June 2015), 14:1–14:33. https://doi.org/10.1145/2764454 [31] Cunyang Wei, Haipeng Jia, Yunquan Zhang, Jianyu Yao, Chendi Li, and Wenxuan Cao. 2024. IrGEMM: An Input-Aware Tuning Framework for Irregular GEMM on ARM and X86 CPUs. IEEE Transactions on Parallel and Distributed Systems 35, 9 (2024), 1672–1689. https://doi. org/10.1109/TPDS.2024.3432579 [32] Samuel Williams, Andrew Waterman, and David A. Patterson. 2009. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52, 4 (2009), 65–76. https://doi.org/10.1145/ 1498765.1498785 [33] x86 Ecosystem Advisory Group. 2026, 1.16.2. AI Compute Extensions (ACE) Specification. Technical Report. x86 Ecosystem Advisory Group. https://x86ecosystem.org/resource/aicompute-extensions-ace-specification Specification Release. [34] Kamen Yotov, Xiaoming Li, Gang Ren, Michael Cibulskis, Gerald DeJong, Maria Garzaran, David Padua, Keshav Pingali, Paul Stodghill, and Peng Wu. 2003. A comparison of empirical and model-driven optimization. SIGPLAN Not. 38, 5 (May 2003), 63–76. https://doi.org/ 10.1145/780822.781140 [35] K. Yotov, Xiaoming Li, Gang Ren, M.J.S. Garzaran, D. Padua, K. Pingali, and P. Stodghill. 2005. Is Search Really Necessary to Generate HighPerformance BLAS? Proc. IEEE 93, 2 (2005), 358–386. https://doi.org/ 10.1109/JPROC.2004.840444

A

Appendix A: Heuristic Search Details

We run OpenEvolve to discover fast-search heuristics that offer a fast alternative to the exhaustive search of Waypoint. To discover the heuristics, we first curate a set of GEMM operations (or shapes) that modern LLM inference workloads invoke. Then, we build a baseline of best tiling parameters for each shape using Waypoint. Starting with a predefined set of seed programs that contain initial heuristics, OpenEvolve runs many iterations, generating candidate heuristics that are evaluated against the baseline. Below we elaborate on the details of the seed programs and the evaluation process.

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

A.1

Seed programs

We generate 6 seed programs each containing independent set of heuristics that target a specific class of the shapes. The goal with the seeds is to provide a diverse set of possible heuristics that could be used to arrive at the final program and allow OpenEvolve to mutate, combine, and evolve them over many iterations to discover effective fast-search heuristics. Each seed program corresponds to a single island of evolution in OpenEvolve. The seeds are, 1. Tiling based primarily on data type and ratio of matrix dimensions. 2. Uses L1/L2 cache sizes to bound the tiling options. 3. Targets optimal multi-threading splits based on the shape of the GEMM operation. 4. Use arithmetic intensity to guide the inner most tiling decisions. 5. Target specifically the AI Compute Instructions / AMX instructions. 6. Initially baseline or the previous best program. A.2

Fast heuristic search results

We plot below the average difference in expected latency between the fast heuristic search algorithm and the exhaustive search algorithm.

B

Appendix B: GEMM evaluation shapes

B.1

Evaluation shapes

Table 5 lists the 51 unique GEMM shapes used in our evaluation. Each shape is evaluated on both hardware configurations and for all three data types described in Section 4. B.2

Packing cost

Table 6 contains the packing cost speedup of libzenmm. Values are geometric-mean ratios of implementation packing cycles to libzenmm packing cycles for all the shapes.

C

Model Assumptions

The estimates derived using Waypoint (both the Simple and Waypoint-Tiles) rely on certain assumptions about the behavior of the underlying hardware. We list below the assumptions along with the reasoning. Fixed frequency workloads: Waypoint estimates the performance of a GEMM operation assuming that the CPU frequency is fixed. We calculate the number of cycles required to access data from memory using the frequency of the machine. Under a variable frequency scenario, one can extend the model by incorporating the dynamic changes in frequency to adjust the cycle estimates accordingly. Best instruction for compute: Due to the highly optimized nature of GEMM kernels, Waypoint assumes that the best instruction i.e. the one with the highest throughput and lowest latency, is used for the compute operations. For example, in x86-ISA based CPU, Waypoint assumes that the

AVX512 FMA/VNNI instruction is used compute operations. We do not explore the possibility of a mixed instruction use for a specific workload. Strided Data access: Waypoint assumes that the data needed for compute is loaded into the registers in strides of contiguous memory locations. Waypoint accounts for the strided access by assuming the lowest cost for loading data into registers. For GEMM workloads, the assumption is valid since data is pre-packed to align accesses with cache lines and stride lengths. Cache memory model: Waypoint assumes that the different level of caches provide different access latencies and that lower levels of caches have faster access times. We do not make any assumptions on the inclusivity, write-back policies, or replacement strategies of the caches. The model only considers the access latency differences between cache levels. A more complex model of the caches will help in determining the access latencies better. Partial bandwidth when loading data: For multi-core (thread) GEMM operations, Waypoint assumes that the memory bandwidth is shared across all threads. Hence, each thread only receives a fraction of the total bandwidth available. The fractional cost is accounted for when computing the data accesses from memory. Peak store throughput: We assume that stores occur at the peak throughput of the memory subsystem. Since the result-matrix locations targeted by store instructions typically reside in caches rather than main memory, the store operations are not expected to be bottlenecked by main memory and can achieve peak throughput.

Waypoint: Analytically Modeling GEMM for CPUs

Table 4. Average difference in expected latency between the fast heuristic search algorithm and the exhaustive search algorithm across different shape categories and architectures. Shape category

Z5 Float

Z5 BF16

Z5 Int8

Z6 Float

Z6 BF16

Z6 Int8

Small Square Medium Square Large Square Tall Skinny Fat short Odd Shapes LLM Shapes Overall

0.002% 0.144% 0.089% 0.692% 0.110% 0.698% 0.250% 0.247%

0.000% 0.060% 0.059% 0.145% 0.121% 0.651% 0.200% 0.157%

0.443% 0.059% 0.055% 0.473% 0.127% 0.838% 0.235% 0.253%

0.010% 0.242% 0.234% 0.111% 0.106% 0.710% 0.081% 0.200%

0.018% 0.251% 0.295% 0.093% 0.114% 0.698% 0.074% 0.211%

0.000% 0.426% 0.514% 0.006% 0.209% 1.590% 0.292% 0.427%

Table 5. GEMM evaluation shapes. 𝑀, 𝑁 , and 𝐾 are the matrix dimensions, and 𝑇 is the number of threads. 𝑀

𝑁

𝐾

𝑇

𝑀

𝑁

𝐾

𝑇

64 64 96 128 128 128 128 128 128 128 128 128 128 128 128 128 128 256 288 512 768 992 1024 2048 2048 2048

64 2048 96 32 128 2880 2880 2880 4096 4096 4096 5120 5760 6144 14336 28672 128256 256 192 512 512 768 1024 64 128 2048

64 64 96 2880 128 2880 4096 5760 128 4096 14336 2880 2880 4096 4096 4096 4096 256 384 512 256 1024 1024 2048 2048 2048

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

4096 4096 4096 128 128 128 2048 4096 8192 128 128 128 2048 4096 8192 128 128 2048 4096 8192 16384 8192 16384 16384 16384

64 128 4096 14336 28672 128256 2048 4096 8192 14336 28672 128256 2048 4096 8192 14336 128256 2048 4096 8192 16384 8192 16384 16384 16384

4096 4096 4096 4096 4096 4096 2048 4096 8192 4096 4096 4096 2048 4096 8192 4096 4096 2048 4096 8192 16384 8192 16384 16384 16384

1 1 1 4 4 4 4 4 4 8 8 8 8 8 8 16 16 16 16 16 16 32 32 64 128

Srinidhi N, Sunanda Somwase, Hemant Kumar Chandrakant Chandavar, Eashan Dash, Varun Agrawal, and Prakash Raghavendra

Table 6. Packing cost relative to libzenmm. Values are geometric-mean ratios of implementation packing cycles to libzenmm packing cycles for different thread counts. Hardware Impl

/

Type

1

2–8

>8

‘Zen 5’ AOCL-BLAS

INT8 BFloat16 Float32

2.02× 3.01× 10.34× 0.40× 1.37× 4.40× 1.11× 1.78× 5.05×

AOCL-DLP

INT8 BFloat16 Float32

2.11× 3.75× 10.23× 0.44× 1.42× 4.48× 1.17× 1.89× 5.05×

SFC

BFloat16 Float32

2.05× 1.64× 3.92× 2.26×

1.73× 3.66×

‘Zen 6’ AOCL-BLAS

INT8 BFloat16 Float32

1.62× 1.51× 0.50× 0.65× 1.01× 0.79×

1.94× 0.97× 1.52×

AOCL-DLP

INT8 BFloat16 Float32

1.60× 2.93× 0.55× 0.61× 1.13× 0.86×

2.35× 0.98× 1.57×

SFC

BFloat16 Float32

2.45× 1.15× 3.49× 1.89×

1.36× 3.55×

Related documents

• Waypoint: Analytically Modeling GEMM for CPUs#1458068
• Stochastic Lanczos Quadrature for Computational Uncertainty in Linear Algebra#996996
• POLAR-PIC: A Holistic Framework for Matrixized PIC with Co-Designed Compute, Layout, and Communication#124024
• Volumetric path following algorithms for linear programming#429974
• The 2M Multiplication Algorithm for Complex Matrices#972486