CUTEv2: Unified and Configurable Matrix Extension for Diverse CPU Architectures with Minimal Design Overhead Jinpeng Ye1,2 , Chongxi Wang1,2,∗ , Wenqing Li1,2 , Bin Yuan1,2 , Shiyi Wang1,2 , Fenglu Zhang1,2 , Junyu Yue1,2 , Jianan Xie1,2 , Yunhao Ye1,2 , Haoyu Deng1,2 , Yingkun Zhou1,2 , Xin Cheng1,2 , Fuxin Zhang1,2 , Jian Wang1,2 1 State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China 2 University of Chinese Academy of Sciences, Beijing, China
arXiv:2604.11615v1 [cs.AR] 13 Apr 2026
Abstract Matrix extensions have emerged as an essential feature in modern CPUs to address the surging demands of AI workloads. However, existing designs often incur substantial hardware and software design overhead. Tight coupling with the CPU pipeline complicates integration across diverse CPUs, while fine-grained synchronous instructions hinder the development of high-performance kernels. This paper proposes a unified and configurable CPU matrix extension architecture. By decoupling matrix units from the CPU pipeline, the design enables low-overhead integration while maintaining close coordination with existing compute and memory resources. The configurable matrix unit supports mixed-precision operations and adapts to diverse compute demands and memory bandwidth constraints. An asynchronous matrix multiplication abstraction with flexible granularity conceals hardware details, simplifies matrix-vector overlap, and supports a unified software stack. The architecture is integrated into four open-source CPU RTL platforms and evaluated on representative AI models. Matrix unit utilization under GEMM workloads exceeds 90% across all platforms. When configured with compute throughput and memory bandwidth comparable to Intel AMX, our design achieves speedups of 1.57×, 1.57×, and 2.31× on ResNet, BERT, and Llama3, with over 30% of the gains attributed to overlapped matrix-vector execution. A 4 TOPS@2GHz matrix unit occupies only 0.53 mm2 in 14nm CMOS. These results demonstrate strong cross-platform adaptability and effective hardware–software co-optimization, offering a practical matrix extension for the open-source community.1
1
Introduction
The rapid advancement of AI technologies has driven widespread deployment across domains such as natural language processing, computer vision, multimodal generation, and embodied intelligence, spanning platforms from edge devices to data centers[21, 26]. As a foundational component of general-purpose computing systems, CPUs continue to play a critical role in executing diverse AI tasks[5]. To address the surging demand for matrix-intensive computation, CPU vendors have integrated matrix extensions including Intel AMX[18], Arm SME[27], and IBM MMA[23] into products. The RISC-V community has also proposed several matrix extension standards such as IME[2], VME[12], and AME[29]. These extensions effectively leverage existing compute and memory resources, particularly vector units and cache subsystems, to significantly enhance AI performance with modest area and power overheads. ∗ Corresponding author is Chongxi Wang: [email protected]. 1 Code available at https://github.com/OpenCUTE/CUTE.
However, existing CPU matrix extensions pose challenges for hardware integration and software programmability[31]. Architecturally, matrix units are often tightly coupled with the CPU pipeline, entailing close interaction with vector register files or load-store units. Fine-grained synchronous matrix instructions introduce substantial structural and data hazards, increasing integration complexity and verification effort. This coupling limits portability across microarchitectures. On the software side, AI workloads typically combine matrix multiplication with a large number of element-wise operations, requiring close coordination between matrix and vector units. Expressing such fine-grained interleaving within a single instruction stream places significant burden on both programmers and compilers, complicating kernel design and scheduling. To address these challenges, this paper proposes a unified and configurable CPU matrix extension architecture designed for agile integration and efficient execution across platforms. The matrix unit is carefully decoupled from the CPU pipeline to avoid intrusive modifications to register files and memory paths, reducing co-design and verification complexity. To accommodate diverse compute and bandwidth constraints, it supports flexible microarchitectural configurations guided by a compute-bandwidth constraint model, and can be scaled from 0.5 to 32 TOPS to guarantee resource utilization and platform adaptability. The matrix unit supports FP8/INT8/FP16/BF16/TF32 mixed-precision computing to meet varying accuracy and performance requirements. As for ISA and programming model, only an asynchronous matrix multiplication and a synchronization primitive are defined, forming a minimal and unified interface that supports flexible granularity. This abstraction hides hardware-specific details and simplifies the programming of overlapped matrix-vector execution, improving programmability and enabling a portable software stack. We integrate and validate the proposed architecture on four opensource CPU RTL platforms: Rocket[4], Shuttle[38], BOOM[37], and XiangShan-Kunminghu[33]. We further evaluate GEMM and AI inference performance against Intel AMX, Arm SME, and IBM MMA. The matrix unit achieves over 90% utilization on GEMM workloads across all integrated platforms. On the Shuttle CPU with a 512-bit Saturn [36] vector unit, a 4 TOPS@8-bit Matrix Unit and 48 GB/s memory bandwidth, the design delivers 1.57×, 1.57×, and 2.31× speedups on ResNet[16], BERT[11], and Llama3[15] inference compared to Xeon 8580. Overlapped matrix–vector execution contributes 66.7%, 50.9%, and 33.6% of the performance gain on three workloads.The design also outperforms IBM S1022 MMA (8.87×, 3.33×, 3.08×) and Apple M4 SME (5.04×, 2.11×, 3.16×). A 4 TOPS@2GHz matrix unit occupies only 0.53 mm2 in 14nm CMOS. The key contributions of this paper are as follows:
rmsnorm & quant
• Proposing a unified and configurable CPU matrix extension architecture that enables agile cross-platform integration and efficient execution. • Presenting a co-designed hardware–software solution that delivers substantial performance gains on representative AI models over commercial CPU matrix extensions. • Integrating the proposed matrix extension into four opensource CPU RTL platforms, with all RTL implementations and high-performance kernels fully open-sourced.
linear linear linear quant quant quant
QKV
conv
linear
MHA
relu & quant
softmax & quant linear
conv relu & quant
conv
AI workloads have been widely deployed across diverse computing platforms, from edge devices to data centers. Typical models consist of layers with heterogeneous characteristics. Compute-intensive layers (e.g., linear, attention, convolution) are dominated by matrix multiplications, while element-wise operations (e.g., activation, (de)quantization, normalization) are generally memory intensive. Figure 1 illustrates the architecture and kernel fusion strategies of three representative models - ResNet, BERT, and Llama3. Kernel fusion[7, 8, 20] enhances overall performance by exploiting data locality and fusing operators into tiled pipelines, thereby reducing memory traffic and improving resource utilization.
relu & quant
Related Work
2.3
Motivation
Although existing CPU matrix extensions have achieved significant throughput improvements, key challenges in hardware integration
linear S* softmax linear A*
GQA
quant
linear
conv
conv
Up
linear conv
gelu & quant linear
Down
layernorm resadd
resadd relu & quant
ResNet
O
linear quant
conv relu & quant
In recent years, CPU vendors - including Intel, Arm, and IBM - as well as the RISC-V community have introduced matrix extensions to enhance AI capabilities on general-purpose processors. These extensions typically employ fine-grained synchronous matrix instructions and differ in register architectures, execution models, and integration strategies, as illustrated in Figure 2. IBM MMA and RISC-V IME reuse the data path of the vector unit, repurposing part of the vector register file as accumulators and restricting matrix operations to vector registers. As a result, register size and bandwidth limit the granularity and throughput of matrix operations. IBM Power10 S1022 delivers a per-core INT8 peak of 2 TOPS at 4 GHz. Arm SME and RISC-V VME introduce dedicated accumulator registers, enabling larger matrix units decoupled from vector register organization. Apple M4 implements SME on both performance and efficiency cores, with the performance core achieving a per-core INT8 peak of 4 TOPS. Intel AMX and RISC-V AME decouple the matrix unit from the vector pipeline by introducing tile registers and dedicated load–store paths, enabling larger matrix operations. Sapphire Rapids and Emerald Rapids deliver a per-core INT8 peak of 2 TOPS/GHz, with the Xeon 8580 reaching 4.6 TOPS at 2.3 GHz under TDP limits. The academic community has also explored CPU matrix units in both configurable forms[14, 34] and fixed-size implementations[6, 13, 17, 25]. However, these designs typically lack close cooperation with existing CPU vector units and offer limited scalability and precision support. In comparison, this work introduces an adaptable and configurable CPU matrix extension, providing the open-source community with a practical implementation.
QKV
layernorm resadd & quant
2 Background 2.1 AI Workloads
2.2
O
quant
resadd relu & quant
linear linear linear roperope
BERT
rmsnorm & quant Gate
Up
linear
linear
silu mul & quant linear resadd
Down
Llama3
Figure 1: AI Model Architectures and Kernel Fusion Patterns. and software programmability still restrict their broad applicability and efficient execution across diverse CPU and workloads. For hardware integration, most existing designs tightly couple matrix units with the CPU pipeline, necessitating invasive modifications across instruction decode, dispatch, and execution stages. Matrix units often interact with vector register files or load-store units, creating complex control paths and redundant state management, increasing integration and verification cost. In Arm SME, IBM MMA, and RISC-V IME/VME, matrix and vector instructions contend for vector registers, resulting in structural and data hazards. Although Intel AMX and RISC-V AME introduce independent tile registers and enable direct access to L1D or L2 cache, they remain constrained by synchronous semantics, which require modifications to ensure correctness between matrix and scalar/vector memory operations either by stalling potentially conflicting instructions or by resolving address conflicts within the load-store unit. For software programmability, most CPU matrix extensions adopt fine-grained synchronous matrix instructions, forcing programmers to manually orchestrate scheduling between matrix and memory operations. Moreover, in AI workloads where matrix and element-wise kernels often require overlapped execution, programming complexity increases significantly. Developers must express memory, matrix, and vector tasks within a single instruction stream and keep all functional units busy within a limited instruction window. Furthermore, disparities among matrix-extension ISAs and software interfaces hinder unified abstractions, requiring developers to perform microarchitecture-specific optimizations and exacerbating software fragmentation. This paper proposes a unified and configurable CPU matrixextension architecture, designed to enable agile cross-platform integration and efficient execution with minimal design overhead.
3
Architecture Overview
This paper proposes a unified and configurable CPU matrix extension, based on three key design principles: (1) Structural decoupling between the matrix unit and the CPU pipeline, without intrusive modifications to the decoder, instruction issue logic, register file, or load–store units; (2) Configurable microarchitectural parameters
Limited Vector-Matrix Overlap Window VLEN=128, 4×4×4 8bit-MAC/Inst
Limited Vector-Matrix Overlap Window VLEN=512, 16×16×4 8bit-MAC/Inst
Limited Vector-Matrix Overlap Window 16×16×64 8bit-MAC/Inst
Vector Unit
Vector Unit
ZA.reg (Accumulator)
IBM MMA - Power10
L1D Cache
L2 Cache
ARM SME - Apple M4
Vector
Check
DDR
8 KiB Tile Registers
L1D Cache
L2 Cache
Matrix Unit
L2 Cache / TCM DDR
Unit
Vector RegFile Hazard LSU Free
Kunminghu CSR
Optional
Vector RegFile
RoCC
High Performance
Matrix
tio na
Hazard Load-Store Unit
LSU L2 Cache
Vector
Vector
Arbitrary Granularity / Inst
l
Matrix Unit
Shuttle/BOOM
Vector Unit
Matrix Unit
Op
LSU L1D Cache
Z.reg (Vector RegFile)
Edge Computing
Matrix Unit
Efficient Vector-Matrix Overlap Execution
Hazard Matrix Unit
Rocket RoCC
Sync
Matrix Unit Execution
Hazard Hazard Vector RegFile
Vector
Vector
Vector
Vector
Issue
Vector
Async Matrix
Vector
Vector
Matrix
Vector
Vector
Vector
Matrix
Vector
Vector
Vector
Matrix
Vector
...
Embedded
Instructions
Instructions Vector
Vector
Matrix
...
...
Vector Unit
Vector
Vector
Matrix
Vector
Vector
Vector
Matrix
Vector
Vector
Vector
Matrix
Vector
Vector
Vector
Matrix
Vector
Vector
Vector
Matrix
Vector
Vector
Vector
Instructions
Instructions
Matrix Unit
L2 Cache
L2 Cache/ TCM
L1D Cache
Kunminghu CSR
Matrix Unit
L2 Cache LLC / NoC / DDR
CUTEv2
Intel AMX - Emerald Rapids
Figure 2: Existing CPU Matrix Extensions.
Figure 3: Architecture Overview.
within the matrix unit to accommodate diverse computational requirements and memory system constraints across platforms; (3) An abstraction for asynchronous matrix-multiplication instructions that enables fusion scheduling with flexible granularity, which simplifies the programming model and improves execution efficiency. Table 1: Interface Registers.
computation proceeds, the checkMatmul instruction ensures the matrix multiplication of the corresponding tile is complete, thereby handling data dependencies correctly.
asyncMatMul ( TILE_M , TILE_N , K , ...) ; // tile 0 for ( i =1; i <( M / TILE_M ) *( N / TILE_N ) ; i ++) { asyncMatMul ( TILE_M , TILE_N , K , ...) ; // tile i checkMatmul () ; // wait tile i -1 ... // tile i -1 epilogue } checkMatmul () ; // wait last tile ... // last tile epilogue
Listing 1: Programming Example. The asynchronous matrix-multiplication abstraction substantially simplifies programming complexity for matrix extensions and enables efficient fusion of matrix–vector operations. Listing 1 shows a fused kernel example performing matrix multiplication followed by element-wise epilogue computation. The asyncMatMul macro dispatches a task per tile, with tile size determined by shared storage capacity between CPU and matrix unit. During asynchronous execution, the CPU issue window can be fully utilized by the vector unit to compute epilogue operations. Before vector
Micro Instruction BaseAddr
Stride
LoadMode
Size
Data Controller Micro Instruction
Scratchpad
Memory
Size
Loop
Request Generator
k
Request Generator Data Reorder
Broadcast
D E C O D E
E. A D D
E. M A X M. M U L
E. M A X
E. G E T
M. A L N
PE Pipeline
A D D
A D D
N O R M
PE
Broadcast
Figure 3 illustrates the hardware architecture. The matrix unit is decoupled from the CPU pipeline and driven by asynchronous matrix multiplication instructions. Depending on ISA and microarchitectural support, the CPU dispatches these instructions via a RoCC-like or CSR-based interface, with registers defined in Table 1. The matrix unit connects to cache or memory independently of the CPU’s load–store unit through a platform-adaptable interconnect. This design reduces integration complexity, supports rapid deployment across CPUs, and allows flexible microarchitectural configuration for embedded, edge, and high-performance platforms.
Core
Memory Loader
an
{M,N,K} uint32 Matrix Size Base {A,B,Bias,C} uint64 Memory Base Address Stride {A,B,Bias,C} uint32 Memory Stride DataType enum Data Precision BiasType enum Bias Type (Zero, Row-Repeat, Full) Transpose bool Result Transpose Flag Status uint32 Async Operation Status
As illustrated in Figure 4, the matrix unit consists primarily of the Memory Loader, Scratchpad, Data Controller, and PE Array.
tiB
Type Description
M ul
Field
4 Design 4.1 Matrix Unit Microarchitecture
PE Array
Figure 4: Matrix Unit Microarchitecture. Memory Loader generates memory access requests and handles all data reads and writes for matrix computations. Within it, the Request Generator translates the tensor described by matrix multiplication instructions into memory and Scratchpad addresses, generating and issuing the corresponding requests. The Data Reorder module receives the returned data and reorder it as required, and writes it to the target storage. Scratchpad temporarily stores matrix partitions to improve data reuse in the matrix unit. Accumulation results can remain resident in the Scratchpad, thereby reducing the need to write back high-precision results. A multi-bank Scratchpad further enables overlapping of data loading and computation tasks. Data Controller supplies data to the computation array. Three Data Controllers are instantiated: two dedicated to source matrices A and B in matrix multiplication, and one dedicated to bias and accumulation matrix C. PE Array combines outer-product and vector dot-product operations, where operands A and B are broadcast row-wise and column-wise across the array, respectively. Each PE performs innerproduct operations and supports mixed-precision computing for
tiny programming cost
TF32, BF16, FP16, INT8, and FP8 formats. Within each PE, multiplication results are aligned to a common exponent, truncated, and accumulated. The PE is organized into a six-stage pipeline to achieve a 2 GHz operating frequency with 14nm process node.
4.2
Check Tile 0
Vector
Vector
Vector
Vector
Matrix Unit
MatMul Tile 0
MatMul Tile 1
MatMul Tile 2
MatMul Tile 3
MatMul Tile 4
Epilogue Tile 0
Epilogue Tile 1
Epilogue Tile 2
Epilogue Tile 3
Vector Unit
Configurable Matrix Extension
high overlap efficiency
Table 2: Configurable Architectural Parameters. Parameters
Meaning
Case Study
Freq 𝑀𝑝𝑒 𝑁𝑝𝑒 𝐾𝑝𝑒 𝑀𝑠𝑐𝑝 𝑁𝑠𝑐𝑝 𝐾𝑠𝑐𝑝
Clock Frequency Row of PE Array Column of PE Array PE Reduce Width Max Resident M in Scratchpad Max Resident N in Scratchpad Max Resident K in Scratchpad
2.0 GHz 4 4 512 Bits 64 64 64 Bytes
Data Bandwidth Throughput (8-bit)
asyncMatmul
4.4
In order to support the generation of implementations with different compute capabilities, we provide a set of configurable microarchitectural parameters to tune the matrix unit for the compute requirements of diverse SoCs. Table 2 lists the configurable microarchitectural parameters of the matrix unit, with a case-study configuration aligned with the compute throughput and data bandwidth of Intel Xeon 8580 AMX. The throughput of the PE array is determined by 𝑀𝑝𝑒 , 𝑁𝑝𝑒 , and 𝐾𝑝𝑒 . For an n-bit data format, the theoretical throughput is:
...
Time
Vector Instructions
Here, 𝐷𝑎𝑡𝑎𝐵𝑎𝑛𝑑𝑤𝑖𝑑𝑡ℎ denotes the data-supply bandwidth of the lower-level memory hierarchy, determined by the cache structure, on-chip network and QoS, memory bandwidth, and other related system factors.
Vector-Matrix Overlap
The asynchronous matrix multiplication abstraction allows matrix tasks to be issued from the CPU pipeline without occupying it until completion, creating more opportunities for matrix–vector overlap. For the operators shown in Figure 1, we adopt the programming method illustrated in Listing 1 to implement matrix–vector fused kernels, with the vector unit executing the prologue and epilogue while the matrix unit handling linear and convolution. Matrix multiplication and vector operations are orchestrated in a software pipeline at the granularity of matrix tiling, producing the fused execution behavior depicted in Figure 5.
Low Design Overhead Integration
The proposed matrix extension was integrated into the four opensource CPU RTL platforms listed in Table 3, covering architectures from in-order single-issue to out-of-order six-issue designs. For Rocket, Shuttle, and BOOM, the integration reused the RoCC[4] interface and was completed in a few days; for the XiangShanKunminghu processor, a new CSR interface was added, taking several weeks. Across all four processors, integrating the matrix extension required only 200–500 additional lines of RTL code, demonstrating the ease of integration of the proposed design. Table 3: Development Cost of Integration. CPU
Micro Architecture Interface Code* Time
Rocket[4] In-order, 1-issue Shuttle[38] In-order, 3-issue BOOM[37] Out-of-order, 4-issue Xiangshan[33] Out-of-order, 6-issue
RoCC RoCC RoCC CSR
254 3 days 512 5 days 301 3 days 361 3 weeks
*Code indicates the lines of integration-related RTL modifications.
(1)
The parameters 𝑀𝑠𝑐𝑝 , 𝑁𝑠𝑐𝑝 , and 𝐾𝑠𝑐𝑝 define the scratchpad size. By adjusting the scratchpad capacity, the data-reuse level of the matrix unit can be adjusted to match different memory-bandwidth constraints. To avoid wasted compute, it is necessary to ensure that, under the output-stationary scheduling strategy, the compute time in the matrix-multiplication loop does not exceed the memoryaccess time: 𝑀𝑠𝑐𝑝 × 𝑁𝑠𝑐𝑝 × 𝐾𝑠𝑐𝑝 (𝑀𝑠𝑐𝑝 + 𝑁𝑠𝑐𝑝 ) × 𝐾𝑠𝑐𝑝 (2) ≤ 𝐹𝑟𝑒𝑞 × 𝑀𝑝𝑒 × 𝑁𝑝𝑒 × 𝐾𝑝𝑒 𝐷𝑎𝑡𝑎𝐵𝑎𝑛𝑑𝑤𝑖𝑑𝑡ℎ
4.3
checkMatmul
...
Figure 5: Vector and Matrix overlap.
48 GB/s 4 TOPS
Throughput (n-bit) = 𝐹𝑟𝑒𝑞 × 𝑀𝑝𝑒 × 𝑁𝑝𝑒 × (𝐾𝑝𝑒 /𝑛) × 2
...
Issue
5 Evaluation 5.1 Methodology The experimental setup is shown in Table 4, with Rocket, Shuttle, and BOOM evaluated on the Chipyard platform, and XiangShanKunminghu on the XiangShan platform. Table 4 also lists parameter configurations used to validate the scalability of the matrix extension. For reference, matrix extension configurations of three commercial CPUs are shown in the case-study column of Table 2. This work is compared against three commercial matrix extensions representative of prior work, as listed in Section 2.2. All baselines are evaluated with 8-bit inference on ResNet-50 v1.5, BERT-base, and Llama3.2-1B, with Llama3.2-1B quantized using SmoothQuant-O1[32] to maintain accuracy. Peak performance and per-core memory bandwidth are reported in Table 5, with memory bandwidth measured using MLC[19] and STREAM[22]. IBM S1022 is evaluated with two threads to fully utilize the MMA unit[30]. Intel AMX is tested with OneDNN v3.9[24] and OpenVINO 2025.2.0[9]; Arm SME with KleidiAI 1.14.0[3] and ONNX Runtime (ORT) 1.21[10]; IBM MMA with OpenBLAS 0.3.27[35] and ORT 1.16.3.
5.2
Integration Across Various CPU Platforms
The matrix extensions on four CPU platforms including Rocket, Shuttle, BOOM, and Kunminghu are configured for a peak throughput of 2 TOPS, and GEMM workloads are evaluated with 𝑀 = 512, 𝑁 = 512, and 𝐾 ranging from 256 to 8192. As shown in Figure