Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE Jie Ren

Ivy Peng

William & Mary USA jren03@wm.edu

KTH Sweden ivybopeng@kth.se

arXiv:2610.32711v1 [cs.DC] 23 Oct 2026

Abstract Memory demands from data analytics and LLM workloads grow rapidly, driving up memory costs in cloud and data centers. Multitier memory has emerged as a promising solution to balance performance and cost. However, effectively leveraging it requires a deep understanding of workload memory behaviors. This work introduces a high-resolution memory characterization methodology for analyzing workload-specific opportunities for multi-tier memory usage. In response to the increasing deployment of ARMbased processors in data centers, we implement our methodology using the recent Statistical Profiling Extension (SPE) for ARM. By characterizing eight memory-intensive workloads from CloudSuite and LLM applications, we identify region-specific access patterns, evaluate the effectiveness of cloud-oriented cache hierarchies, and propose new metrics based on memory idle periods and latency profiles to determine suitable data placement for tiered memory.

Keywords ARM SPE, tiered memory, LLM, cloud

1

Introduction

Memory-intensive workloads are becoming increasingly important to cloud and data centers (DCs). The rapid proliferation of large language models (LLMs), alongside traditional workloads such as real-time analytics, in-memory databases, and graph processing, continues to drive up the demand for high-capacity, highperformance memory. At the same time, the rising procurement and operational costs of memory have made multi-tier memory system an attractive solution for meeting these demands cost-effectively. Multi-tier memory systems pair a fast but expensive memory tier with a larger, more affordable, but slower tier, striking a balance between cost and performance [1, 11, 20, 21]. A notable example is the integration of high-bandwidth memory (HBM) with DDR memory in commercial systems such as Intel’s Sapphire Rapids (Xeon Max Series), which combines HBM with DDR5. Another emerging approach is disaggregated memory, enabled by Compute Express Link (CXL), which offers high-bandwidth, low-latency cache-coherent interconnects to implement memory pooling at rack-scale [13, 15]. Effectively managing workloads in multi-tier memory systems requires a deep understanding of their memory access patterns and the interplay between hardware and system software. OS kernel and hardware performance counters are two main mechanisms for building tools for memory profiling and characterization. For instance, Linux’s DAMON [17, 18] (Data Access MONitor) is a kernel feature that can monitor memory access patterns within a given overhead budget by trading off accuracy for performance. Many

profiling tools are based on linear scanning of PTEs (Page Table Entries) to reveal page-level access and residency information. When Performance Monitoring Units (PMUs) are augmented with information, such as virtual addresses, latency, and program counters, as presented by Processor Event-Based Sampling (PEBS) on Intel processors or Instruction-Based Sampling (IBS) on AMD processors, they can provide detailed low-level information. Recently, data centers (DCs) and cloud providers have increasingly adopted high-end ARM processors as an alternative to x86 platforms [12]. This trend includes ARMv8.x processors targeting cloud and compute-intensive workloads. For instance, Fugaku supercomputer is built on ARM A64FX processors. NVIDIA and Amazon have also used ARM for their server-class processors, including NVIDIA Grace (used in the Jupiter supercomputer) and AWS Graviton. Despite this growing momentum, most existing memory profiling and analysis tools are specific to x86 architectures. As a result, ARM-based platforms have remained relatively underexplored in terms of fine-grained memory characterization, partly due to the limited support of precise sampling features on earlier ARM implementations. With the introduction of the Statistical Profiling Extension (SPE) in ARMv8.2, detailed memory analysis on ARM platforms becomes feasible. This work fills the gap by providing a memory characterization method on ARM-based systems that incorporate SPE. First, we propose a methodology for understanding memory-intensive workloads from two complementary perspectives: workload characteristics and memory hierarchy utilization. Second, we then characterize eight memory-intensive workloads using a set of metrics tied to efficient utilization of multi-tier memory. Finally, we showcase integrating the SPE profiling analysis into a data migration optimization runtime. Figure 1 presents an example of detailed memory access analysis in LLM training, showing L2 cache hits associated to virtual memory address over time, distinguishing three groups of memory regions (weights, activations/gradients, and optimizer states) and their distinct access patterns across training and forward-only phases. In summary, we made the following contributions in this work. • We provide a high-resolution memory characterization method comprising a profiling tool and comprehensive analysis framework targeting ARM processors. • We provide a high-performance implementation of the method based on ARM SPE, and quantify its accuracy and overhead in comparison with the state-of-the-art DAMON. • We characterize a set of eight memory-intensive workloads from CloudSuite and LLMs on a data center-grade ARMbased platform.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Jie Ren and Ivy Peng

Validation (compute loss)

model weights

activations & gradients

Optimizer states

Training (10 steps)

Training (10 steps)

Validation (inference only)

Figure 1: L2 cache hits by virtual address over time during GPT2 training. Forward-only intervals include validation-loss computation and text generation. • We propose a set of workload-focused metrics for assessing the feasibility of multi-tier memory and evaluate their usefulness through experiments with an SPE-guided data migration runtime.

2

Characterization Methodology

Our memory characterization serves two objectives. The first aims to provide insights for deriving effective prefetching strategies to optimize workloads on multi-tier memory. For this, access patterns inherent in workload characteristics need to be identified comprehensively. The second aims to provide insights for optimizing and tailoring cache and memory configurations in future system designs for specific workloads. For this, accurately monitoring utilization and identifying hotspots at each level of the cache and memory hierarchy is desirable. To support these goals on real-world workloads, we define the following design principles. (1) Comprehensive Characteristics. Capture all workload access patterns critical for deriving prefetching and data placement strategies, including execution phases, temporal locality, spatial locality, and hot and cold regions. (2) Memory Hierarchy Awareness. Quantify the utilization of the complex cache hierarchy, coherence interconnects, and main memory, and correlate this utilization with specific memory regions.

(3) Support High Concurrency. Support isolating per-thread memory access information in highly concurrent workloads using tens and hundreds of threads in data centers and cloud environments. (4) High Resolution. Support high resolution in time to detect fast phase changes in real applications, and high resolution in memory space to reduce fragmentation. (5) Low Overhead. Incur low resource and runtime overhead during workload execution. (6) Programming Language Agnostic. Support diverse cloud workloads in different programming languages and frameworks transparently.

2.1

An ARM SPE-Based Design and Implementation

Guided by these design principles, we compare various memory profiling mechanisms, including binary instrumentation, compiler analysis, OS kernel, and Performance Monitoring Unit (PMU). Binary instrumentation and compiler analysis offer detailed information at the memory-instruction level, but lack visibility into architectural information of the cache and memory hierarchy, thus violating principle (2). OS kernel-based methods, such as DAMON [18], leverage page-table entry (PTE) scans for region-based profiling, identifying only page-level access patterns of frequently accessed regions.

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM Conference SPE acronym ’XX, June 03–05, 2018, Woodstock, NY

This approach results in coarse-grained profiling and misses detailed insights into less active or quickly changing regions, violating principles (1)–(4). Given these limitations, we adopt a PMU-based mechanism leveraging the ARM SPE extension to develop our memory characterization method. ARM SPE is a recent extension to the PMU that augments hardwareassisted CPU profiling with detailed information, such as virtual memory addresses, latency, and events throughout the execution pipeline. It was introduced in the Armv8-A architecture and first became available with the Neoverse N1 CPU. As an instruction sampling-based mechanism, ARM SPE selects sampled instructions at a user-defined interval, i.e. the sample period 𝑁 . The sampling counter selects an operation approximately every 𝑁 eligible operations, with randomization to reduce bias, and tracks it through the execution pipeline. If it is a memory instruction, ARM SPE can capture detailed information such as supported cache and TLB events, a data-source identifier, and latency counts for the sampled operation. A sample record is created and selectively filtered before saving it into a buffer in memory as a sequence of packets. Filtering options relevant to memory characterization include events such as L1 Data (L1D) cache misses and Translation Lookaside Buffer (TLB) misses. The main tradeoff between accuracy and overhead is determined by the sampling period 𝑁 , which has a machine-specific minimum value. We designed and implemented the SPE-based memory profiling tool to support transparent profiling by preloading the library using the dynamic linker. Environment variables are used to adapt the profiling configuration, e.g. the sample period. A key difference between Intel’s PEBS and ARM SPE is that PEBS is event-based while SPE is instruction-based. PEBS triggers records from selected performance events, while SPE selects operations for profiling. An SPE sample record can combine address, event, data-source, and latency packets for the same operation [3]. Our tool separates samples on different cores, supporting perthread analysis when worker threads are pinned to cores. With this capability, we can detect workload imbalance among threads and optionally trace only a subset of the cores to reduce overhead in symmetric workloads. Internally, we implement the tool with separate staging buffers to avoid synchronization among threads for high scalability. The characterization supports high resolution in time, with profiling windows as fine-grained as 200 ms. The resolution of memory address space is at the byte-addressable level. The tool reduces performance overhead by carefully selecting event filters, sample frequency, and buffer size. For memory characterization in this work, we set the SPE filter to sample L1D load misses with a sampling period of 1,024 and use a 1 MiB buffer size. We leverage the sampled data source information to correlate the utilization of cache and memory hierarchy to specific memory regions. In particular, the supported hierarchical information includes L1D, L2, peer CPU, local cluster, system cache, peer cluster, and DRAM. We further study DRAM accesses as they incur the longest latency, by capturing three latency counts: issue latency, address translation latency, and total latency. Issue latency measures the number of cycles waiting for the instruction to be issued. Total latency measures the number of cycles to execute the instruction. For samples attributed to DRAM, the reported latency is an SPE

operation-latency measurement and can include delays beyond DRAM service time. With this detailed latency information, we can identify problematic memory regions that suffer from high latency.

2.2

Validation

In this section, we thoroughly evaluate our proposed memory characterization method, providing a validation of ARM SPE’s suitability for optimizing multi-tier memory on ARM-based platforms. To validate correctness, we use three benchmarks with wellknown access patterns as ground truth. STREAM is a widely-used benchmark that employs sequential access patterns for measuring peak memory bandwidth. We use its multi-threaded version to validate our tool’s ability to accurately capture per-thread streaming accesses. RandomAccess performs random access to a large heapallocated array following a normal distribution, which is used to validate our tool’s efficiency in capturing random access patterns with varying spatial localities. Additionally, MASIM [16] provides a phase-based benchmark that emulates real-world workload behavior through configurable hot regions and access phase transitions, enabling validation of dynamic memory access pattern tracking. We quantify our tool’s accuracy using precision and recall for hot page identification, and measure runtime overhead across various sampling rates. We compare both accuracy and overhead against DAMON [18], a state-of-the-art OS kernel-based profiling tool. Access patterns. We assess whether the tool can effectively capture workload access patterns in terms of spatial and temporal locality and execution phases. Figure 2 presents the memory access patterns captured in STREAM. The left plot shows the distribution of accessed virtual addresses (y-axis) over execution time (x-axis), with each color representing a different thread. The visualization reveals how each thread accesses a contiguous memory region in a streaming pattern, with execution phases repeating periodically. Notably, such fine-grained per-thread access patterns cannot be detected by OS kernel-level mechanisms such as DAMON. The right plot displays access density across the virtual address space over time, showing that array c (higher addresses) receives more accesses than arrays a and b (lower addresses), reflecting different sampled access counts among the arrays in this run. Figure 3 compares the access distributions captured by our SPEbased tool (top left) and DAMON (top right) for the RandomAccess benchmark. The left plot shows that the ARM SPE tool accurately reconstructs the normal distribution of memory accesses, with the captured samples (blue bars) closely matching the theoretical distribution (red line). This demonstrates that ARM SPE can reliably capture different hotness levels across the memory space, which is critical for page placement decisions in multi-tier memory systems. In contrast, DAMON’s captured distribution (top right plot) shows significant information loss, particularly for pages with moderate to low access frequencies. This limitation is due to DAMON’s PTEscanning approach, which has lower sampling coverage and easily misses infrequent accesses during its periodic scans. Figure 3 (bottom) shows the memory access heatmap captured by our SPE-based tool for MASIM. The heatmap clearly reveals distinct access patterns across three execution phases: the first phase exclusively accesses array a, the second phase only accesses array b, and the third phase accesses both arrays equally. These

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Jie Ren and Ivy Peng

STREAM

×1014

RandomAccess

Virtual Address

Precision 2.8147

1.0

2.81465

0.8 0.6

2.8146

0.4

2.81455

0.2

2.8145

0.0 0

2.81445

0

10

Time

20

30

40

80

0

50

100

RandomAccess: SPE

RandomAccess: DAMON

Expected RandomAccess: normal dist.SPE 1e-9 Expected normal dist.

RandomAccess: Observed DAMON 1e-8 Observed 1e 8

1.4

6 1e-9

1.2 0.8

4 2

0.8 0.4

2 0

0.4 0.0 6.8

7.6

8.0

Virtual7.6 Address 7.2 8.0 9 + 2.8146×1014 ×10 Virtual Address

8.4

×109 + 2.8146×1014

MASIM 262135 262132 262135 262127 262132 174767 262127 0 174767 0

100

0.0

0.2

0.6

0.8

1.0

1.2

1.4

1.6 ×109

10

100

1000

1

10

100

1000

Time (s)

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

DAMON Precision DAMON Recall

0 0

20

40

60

Percentage of Execution Time (%)

50

0.2

DAMON Precision DAMON Recall

0 80

100

0

100

0

20

DAMON Precision DAMON Recall

0 40

60

Percentage of Execution Time (%)

50

80

100

0

100

0

20

40

60

Percentage of Execution Time (%)

50

80

100

100

Figure 4: Precision and Recall measured over time in three benchmarks. Top row: SPE-based tool; bottom row: DAMON. SPE-based tool achieves perfect precision with stable recall.

9

1

10 10

0.4

1

(b) DAMON

Address 0.4 Virtual0.8 1.2 1.6 1.4 0.4 0.6 0.8 1.0 1.2 1.6 ×10 Virtual Address (×109) 0.0 Virtual 0.4 Address 0.8 1.2 1.6 9 Virtual Address (×10 ) 0.2

Recall

1

Execution time (%)

Expected Normal Dist. DAMON Observed

0.0

0.00.0 0.0

8.4

MASIM Address Address (GB)(GB)

1.0 1.2 0.8 1.0 0.6 0.8 0.4 0.6 0.2 0.4 0.0 0.2

6 4

1.0 0.8 0.6 0.4 0.2 0.0

0

Expected Normal Dist. DAMON Observed

1e-8 1.21.2 1.4 1e 8

7.2

50

Time (s)

Figure 2: The scatter plot of L1D misses in STREAM, exhibiting temporal and spatial locality. Each color represents a different thread.

6.8

0

(a) SPE-based tool

40

Precision

0

MASIM

Recall

20

30

20

30

Time (s)

Figure 3: Top two: memory access histograms from a SPEbased tool and DAMON on RandomAccess. Bottom: memory access heatmap from the SPE-based tool on MASIM, showing three access phases across different address ranges.

phase transitions are captured with clear boundaries, demonstrating the ability to track dynamic changes in memory access patterns over time with our tool. Under the tested configuration, DAMON does not clearly resolve these phase changes. Accuracy. We use the Precision and Recall metrics to quantify the accuracy of identifying hot regions in workloads. Precision is calculated in a profiling window as the fraction of identified hot memory regions that are truly hot, while recall measures the fraction of actual hot regions that are correctly identified. We establish ground truth based on application semantics derived from source code analysis. In STREAM, the memory regions corresponding to the three heap-allocated arrays are considered hot as they are continuously accessed throughout execution. In RandomAccess, we define memory regions within one standard deviation (1𝜎) of the mean in the normal distribution as hot, capturing 68.3% of all accesses. MASIM’s ground truth varies in the three phases. Figure 4 presents the precision and recall measurements over time for both our SPE-based tool (top row) and DAMON (bottom

row) across the three benchmarks. For our SPE-based tool, precision consistently maintains 1.0 throughout execution in all three benchmarks, indicating that all identified hot memory regions are truly hot. This ensures that page migration decisions would not mistakenly move cold regions into the fast tier. The recall achieves 0.5 and 0.6 for RandomAccess and MASIM respectively, but only 0.02 for STREAM. This variation occurs because we classify the top 50 sampled regions as hot at each time step, which limits the number of selected hot regions and causes low recall in benchmarks with many uniformly accessed memory regions like STREAM. DAMON achieves comparable recall for most benchmarks due to its region-based profiling that naturally covers larger memory areas. However, it exhibits low precision and high variability in both metrics within the same workload. This instability, particularly shown in the scattered point distributions, demonstrates DAMON’s difficulty in consistently identifying hot regions across different access patterns. Overhead. We evaluate performance overhead using MASIM configured with four problem sizes ranging from 64 KB to 100 GB regions. We use three different sampling periods for our SPE-based profiling tool and DAMON. The results in Figure 5 show that our SPE-based approach incurs less than 0.6% overhead in the 64 KB, 1 GB, and 10 GB configurations, while the 100 GB configuration reaches 3.6–4.5%. Overhead remains consistent across different SPE sampling intervals (1024, 2048, 4096) in 64 KB, 1 GB and 10 GB MASIM, while overhead decreases with larger sampling intervals in 100 GB MASIM with SPE4096 showing the lowest overhead and SPE1024 the highest. Under the tested configurations, DAMON incurs about 1.5–6.1% runtime overhead.

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM Conference SPE acronym ’XX, June 03–05, 2018, Woodstock, NY

Figure 5: The performance overhead of ARM SPE using three sample periods and DAMON in four MASIM configurations.

3

Experimental Methodology

We use a testbed based on ARM Ampere® Altra® Max 64-Bit [2] processor with 128 Neoverse N1 (Armv8.2+) cores running at 3.0 GHz. Every two cores are co-located within a DSU (DynamiQ Shared Unit). The cache hierarchy consists of 64 KB L1I and 64 KB L1D per core, 1 MB L2 per core, and a shared 16 MB System Level Cache (SLC) as the last-level cache (LLC). The main memory has a total of 512 GB DDR4 and 200 GB/s peak bandwidth. This cache hierarchy configuration, in contrast to shared L2 and large LLC often used in HPC systems, is tailored for multi-tenant environments in data centers. The testbed runs Ubuntu 22.04 with Linux kernel 5.15. Applications were built using GCC 11.4, OpenMPI v4.1.2, and OpenBLAS 0.3.20. We also evaluated on an AmpereOne® processors to ensure portability and traces are similar to those obtained on Ampere® Altra® testbed and hence omitted.We evaluate eight workloads commonly deployed in data center environments using our SPE-based profiling tool, covering LLM training and inference, graph analytics, and data processing applications, as summarized in Table 1. Table 1: Memory-intensive applications for characterization. Application

Language

Domain

Llama 8B GPT2 PageRank (PR) Connected Components (CC) Triangle Counting (TC) In-Memory Analytics Memcached Cassandra

C++ C Scala, Java Scala, Java Scala, Java Java C Java

LLM Inference LLM Pretraining Graph Analytics Graph Analytics Graph Analytics Data Processing Data Caching Data Serving

LLM Workloads. For inference, we use Llama 8B model (denoted as Llama) with the llama.cpp [9] CPU implementation. We use quantized model weights while keeping the KV cache in float16 to ensure KV cache access patterns remain visible in the memory footprint. We evaluate with batch size 8, processing 128 prompt tokens to generate 100 tokens on 16 threads, measuring performance in tokens per second. For training, we evaluate GPT-2 124M pretraining using the train_gpt2 in llm.c [10] CPU implementation, configured with maximum sequence length of 1024 and 1192 batches in the training dataset, reporting execution time. To further examine KV-cache access patterns during inference (Figure 9), we use Qwen3-30B-A3B-Instruct-2507 [4] with the llama.cpp

CPU implementation on the same testbed and with the same SPE sampling period of 1,024. This mixture-of-experts model uses UDIQ2_XXS GGUF weights (2.71 effective bits per weight) and FP16 K and V caches. We process 2,000 prompt tokens and generate 512 tokens with one sequence on 128 threads. The context capacity is 4,096 tokens, with a token batch limit of 2,048 and a microbatch limit of 512. Graph Analytics. We evaluate three fundamental graph algorithms: PageRank for computing node importance in web ranking applications, Triangle Counting for community detection and network analysis, and Connected Components for identifying clusters within graphs. All three workloads process a Twitter dataset using 32 threads and 128 GB memory. Data Processing. We evaluate three widely used data processing applications. Memcached runs as an in-memory key-value store with a Twitter dataset scaled 112×, using approximately 40 GB memory. After warmup, it operates with a 0.8 get-set ratio, reporting 99th percentile latency. Cassandra, evaluated using YCSB, processes 100K records with 8 threads after warmup, measuring throughput in operations per second. For in-memory analytics, we run Apache Spark MLlib’s movie recommendation system using the MovieLens dataset, configured with 32 threads and 256 GB memory allocation, measuring execution time for generating top-50 recommendations.

4

Memory-intensive Workloads

We apply the characterization methodology to eight memory-intensive workloads. Our characterization study aims to provide insights and metrics for workloads on multi-tier memory systems, such as HBMDRAM and CXL-based disaggregated memory systems. Therefore, we focus on three aspects of the characterization: workload-level characteristics, memory hierarchy utilization, and memory region profiles.

4.1

Workload Insights

We examine memory access patterns based on temporal and spatial locality characteristics, including strided, sparse, and block patterns. Figure 6 presents memory access sample-count heatmaps filtered for L1D load misses. Pattern I: Strided Pattern represents memory accesses with consistent address offsets. In the heatmaps, these appear as horizontal lines (i.e., stride-0, indicating repeated access to the same address over time) or diagonal lines (i.e., stride-𝑛, indicating traversal with fixed address increments). As an example of strided patterns, we examine high-resolution profiling results from our SPE-based profiling tool. Figure 7 zooms into specific memory regions during GPT-2 pretraining. The upper region exhibits V-shaped diagonal patterns from model parameter accesses: ascending diagonals during forward passes and descending diagonals during backward passes, reflecting the opposite layer traversal order of forward and backward passes. The lower region shows optimizer state accesses after the backward pass. After the backward pass computes gradients layer by layer (from last to first), the optimizer uses these gradients to update its internal states. The cooler colors in the lower region indicate fewer sampled memory accesses per bin compared to the upper region.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Jie Ren and Ivy Peng 262140

262134

104 104

102

Virtual address (GB)

Virtual address (GB)

103 103

102

101

101

100

262128

20

40

60

Time (s)

80

100

120

100

262136

5

10

15

Time (s)

20

25

(a) Cassandra

(b) Memcached

(c) Llama 8B Inference

(d) GPT2 Pretraining

(e) In-Memory Analytics

(f) Triangle Counting

(g) Page Rank

(h) Connected Components

384

0 384

5

6

7

8

9

10

0

Offset within selected region (MiB)

Figure 6: Sampled L1D load misses by virtual address (sampling period: 1024; bins: 200 ms by 8 MiB).

Time (s)

Figure 7: GPT-2 pretraining accesses at 5–10 s. Upper: model parameters; lower: optimizer states. Each panel spans 384 MiB with its own address origin. Similar strided patterns appear across diverse workloads. Cassandra (Figure 6a) exhibits horizontal strided patterns in upper address ranges from repeated metadata structure accesses, while diagonal patterns in lower addresses indicate systematic memory traversal during compaction operations. Memcached (Figure 6b) shows prominent horizontal strided patterns with highly concentrated hot bands, indicating specific hot keys and critical metadata structures being repeatedly accessed. In-Memory Analytics (Figure 6e) displays numerous horizontal strided patterns distributed across the address space with varying intensities, reflecting access to multiple data structures with different usage frequencies. Graph

analytics workloads (Figures 6f, 6g, and 6h) also display horizontal strided patterns in their lower address spaces. Pattern II: Sparse Pattern represents memory accesses to small, irregular subsets of addresses within a region over time. In the heatmaps, these appear as scattered dots or isolated points with no clear continuity in either temporal (horizontal) or spatial (vertical) dimensions, indicating infrequent and non-contiguous memory usage. To illustrate sparse patterns, Figure 8 zooms into the activation/gradient memory region during GPT-2 pretraining. During forward passes, activations (visible as scattered blue dots) exhibit sparse sampled access patterns in these intervals. Similar sparse patterns emerge during validation phases as shown in Figure 6d, which consist of forward computation only, and in other scenarios where data structures are accessed infrequently or irregularly. Pattern III: Block Pattern represents memory accesses concentrated within specific address regions. In the heatmaps, time-local bursts of memory accesses create rectangular blocks, where contiguous address ranges experience high-frequency access over short periods. Figure 8 also demonstrates block patterns during backward passes. In contrast to the sparse forward pass, backward propagation creates dense rectangular blocks of memory accesses in the activation/gradient region. This is consistent with tensor-wise operations on gradient buffers during backward computation. Block patterns appear in multiple workloads, particularly in LLM and graph processing. Llama inference (Figure 6c) exhibits block patterns over broad address ranges, with repeated vertical bands in the upper region after about 40 s. Graph analytics workloads shown

768

5

6

7

8

9

10

0

Offset within selected region (MiB)

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM Conference SPE acronym ’XX, June 03–05, 2018, Woodstock, NY

Time (s)

Figure 8: GPT-2 activation/gradient accesses at 5–10 s over 768 MiB: sparse patterns during forward passes and block patterns during backward passes. in Figures 6f, 6g, and 6h also exhibit block patterns corresponding to frequently accessed data structures such as vertex properties or adjacency lists that become hot during iterative computations. KV Cache Accesses in LLM Inference. To relate strided access patterns to inference computation, we examine KV-cache reuse in Qwen3. Figure 9 shows its memory access overview and a detailed view of the KV cache. The repeated ascending bands reflect traversal and reuse of cached key and value tensors.

4.2

Utilization of Cache and Memory Hierarchy

For the eight workloads in Table 1, we further investigate how the memory hierarchy interacts with different memory regions. These insights are not only useful for optimizing memory-intensive workloads on a multi-tier memory system, but can also guide system design in configuring suitable cache hierarchies for targeted workloads. We focus on three levels: the private L2 cache, shared LLC, and coherence interconnect. We capture hits in each level and correlate them to specific memory regions in each workload. The private L2 cache is the most effectively utilized cache level. Figure 10 shows the distribution of L2 hits and DRAM access in the same memory region in the LLM GPT2 pretraining workload. By contrasting the two, it is clear that L2 absorbs a vast amount of accesses coming from the middle memory regions, which exhibit an interleaving pattern of dense block access and sparse access as described in Section 4.1. This memory region has very few recorded DRAM-access samples as shown in the bottom panel of Figure 10, leaving the space almost blank over the execution time. The other workloads also exhibit high utilization of L2 cache. This high utilization of L2 supports the design choice of large (1MB) private L2 for systems targeting such cloud workloads, which is consistent with the initial CloudSuite paper [7]. Most workloads can also leverage the shared LLC, except Llama. Llama shows nearly no utilization of LLC, even though it has good utilization of L2, indicating limited use of the shared LLC in this Llama experiment. We further investigate memory regions that hit in LLC in each workload and find that mostly only the stride pattern (i.e., horizontal lines in Figure 6) hits in LLC, while the other patterns, such as sparse and block patterns, have relatively lower utilization of it. Note that the LLC on the system is small, i.e., 16

MB compared to 128 MB of aggregate private L2 capacity. Such a configuration is consistent with the finding that large LLCs are wasted for CloudSuite workloads [7]. We analyze data-sharing in these workloads by identifying data fetched from peer CPUs (i.e., cores in the same DSU) and clusters (other DSUs), which indicates the utilization of the coherence interconnect. The right panel of Figure 10 presents the same memory regions in Memcached that fetch data from peer DSUs (top) and peer CPU (bottom). Data sharing with other DSUs (peer Cluster) occurs in both phases, while data sharing with peer CPUs only occurs after the initial phase. The initial phase warms the dataset, and as such only consists of set operations that fill up the records, where the second phase consists of a mixture of get and set operations for the data cache. Llama shows very few samples attributed to Peer CPU or Peer Cluster, indicating limited observed use of these data sources in this experiment.

4.3

Memory Region-based Characterization

We next analyze memory regions to identify candidates for placement and migration on multi-tier memory. A memory region is defined as a contiguous memory space and they are often used as the basis for placement and migration decisions on multi-tier memory. Our first analysis aims to identify the opportunity of running a workload onto multi-tier memory. For this, hot memory regions need to be identified so that they can either be placed into the fasttier memory statically or be migrated from the slower memory into the fast-tier memory at runtime. A commonly used metric is the count of DRAM accesses to a memory region. One such example is illustrated in the heatmap in Figure 11, where hot regions are colored red. While access counts provide a good indication of the relative importance among memory regions, our further analysis of regionspecific DRAM access latency distribution in two hot regions reveals that they may have very different latency characteristics, as shown in the histogram in Figure 11. Region 1, which spans a larger memory size, experiences a short period of hotness, where DRAM latency mostly occurs between 300-500 cycles. In contrast, the latency distribution in Region 2 has a long tail of up to 2500 cycles. This difference suggests that access pattern and concurrency level can affect the latency observed in different regions. Since Region 2’s size is only one-third of Region 1, it should have higher priority to be considered for the fast memory tier. We analyze the latency distribution in all workloads and find that Memcached, Triangle Counting, Page Rank, and Connected Components have longer tails in DRAM latency distribution compared to Llama and GPT2. However, compared to other workloads, Llama shows more samples near 4096 cycles in the recorded latency distribution. 4.3.1 Region-Latency Distribution. We leverage the DRAM latency distribution in memory regions to identify workloads that are promising for multi-tier memory systems. For this, we build the histogram of DRAM accesses in memory regions weighted by their DRAM latency, which better reflects memory access bottlenecks in workloads, compared to the access-only distribution. Figure 12 presents the cumulative share of sampled DRAM-access latency as memory regions are ranked by their summed latency, from highest

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Jie Ren and Ivy Peng

Virtual address (GiB)

104

103

192

102

101

100

262121 0

40

80

120

160

Time (s)

40

42

44

46

48

Relative virtual address (MiB)

384

262133

0

Time (s)

Figure 9: Qwen3 inference: sampled L1D load-miss overview (left) and KV-cache detail at 40–48 s (right). The detail uses offsets within the 384 MiB KV cache. Both panels use 200 ms by 8 MiB bins and the same logarithmic color scale.

Figure 10: Left: L2 hits and DRAM access in LLM GPT2. Right: Peer CPU and Peer Cluster access in Memcached.

Figure 11: DRAM-access heatmap and region-specific latency distributions for two hot regions in PageRank. to lowest. The sum provides a relative ranking of regions; overlapping memory operations mean that it does not measure elapsed stall time. We identify five workloads showing a highly skewed distribution of DRAM access, including Cassandra, GPT2 training, In-Memory Analytics, page ranking, and connected components. Among them, In-Memory Analytics shows the highest skewed distribution, where 50% latency attributed by merely 0.7% memory regions, as shown in Figure 12e. This workload has nearly all

memory regions with the stride access pattern (horizontal lines in Figure 6) concentrated in small memory regions. With a similar access pattern, Cassandra also has 50% latency attributed to only 3% memory regions. Connected Components and Page Ranking both have their highest latency access associated with memory regions exhibiting stride and block patterns. Finally, the GPT2 training workload has 50% latency attributed to about 5.2% memory regions, which have the stride pattern (streaming). This highly skewed distribution in memory regions is generally favorable on multi-tier memory systems because only a small fraction of memory pages need to be placed in the fast memory tier. Llama, triangle counting, and Memcached, show a relatively even distribution of DRAM access. For Llama, about 27.7% of memory regions account for 50% of the sampled latency, as shown in Figure 12c. Its latency is therefore less concentrated in a small set of regions than in the highly skewed workloads. Memcached has most DRAM accesses uniformly distributed to four memory regions associated with the stride pattern. Triangle Counting also has approximately half of its memory regions associated with the block pattern. Therefore, workloads with most memory regions dominated by the block access pattern are less feasible for multi-tier memory than those with a highly skewed distribution. 4.3.2 Idle Period Distribution. We propose a new metric, the longest idle period fraction, for assessing the feasibility of data migration on multi-tier memory systems. The longest idle period fraction is the longest time window without any recorded DRAM-access samples, normalized by the total execution time. We compute the metric for each memory region. This is an observed gap in sampled activity; its duration depends on the sampling period and does not establish that the region received no accesses. Note that the longest idle period is different from the fraction of idle periods, which sums up all idle periods and cannot distinguish a memory region that frequently interleaves between a busy period (i.e. frequent DRAM accesses) and a short idle period (i.e. no recorded DRAM-access

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM Conference SPE acronym ’XX, June 03–05, 2018, Woodstock, NY

Llama 8B

(a) Cassandra

(b) Memcached

(c) Llama 8B

(d) GPT2 Training

(e) In-Memory Analytics

(f) Triangle Counting

(g) PageRank

(h) Connected Components

Figure 12: Cumulative share of sampled DRAM-access latency, with memory regions ranked by summed latency. Llama 8B

Figure 13: The longest observed idle period distribution in memory regions in the eight workloads. samples) and thus, is infeasible for data migration on multi-tier memory systems. The Longest Idle Period metric can be useful in multiple scenarios. For CXL-enabled multi-tier systems, it can be used to determine whether there are sufficient inactive memory regions that can be offloaded to a slower but less costly memory tier, such as CXL memory pools. It could also be used for assessing whether migrating a memory region is feasible on a system, i.e., if the idle period is long, data migration may overlap with the idle period to mitigate the performance impact on the workload. We present the distribution of the longest idle period and its CDF in Figure 13. We use a longest observed idle period above 30% of execution time as a heuristic for identifying migration candidates, although this threshold can be adapted on different target systems. From the CDF in Figure 13, Cassandra, In-Memory Analytics,

Triangle Counting, Page Rank, and Connected Components have more memory regions with long observed idle periods than Llama and Memcached, although the proportions differ across workloads. Llama, instead, has very few memory regions above this threshold, indicating limited opportunities for overlapping migration with long observed idle periods. GPT2 pretraining has longer observed idle periods in some memory regions, consistent with periodically occurring forward-only intervals, which do not need accesses to optimizer states. Finally, Memcached has less than 5% of its memory regions above the idle-period threshold. Together with the regionlatency distributions, these results suggest fewer opportunities for runtime migration in Llama and Memcached, while GPT2 has more concentrated latency and some long observed idle periods. 4.3.3 SPE-guided Optimization on Memory Tiers. To demonstrate the effectiveness of the proposed memory metrics in optimizing

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Jie Ren and Ivy Peng

160%

5

Relative Performance

140% 120% 100% 80% 60% 40% 20% 0%

Cassandra

Memcached

Llama 8B

GPT2

In-memory analytics

Triangle Counting

Page Ranking

Connected Components

Figure 14: The relative performance of the eight workloads with online data migration on multi-tier memory on ARM processor.

memory usage on ARM processors, we integrated the SPE-based memory profiling feature into a data migration runtime for multitier memory. The runtime makes data migration decisions periodically based on memory access samples collected in a time interval. The runtime identifies hot memory regions by selecting the top N memory regions with the most samples from last-level cache misses. Leveraging the physical address of the memory samples, the runtime determines whether a top memory region is already on the fast tier or needs to be migrated from the slow tier. To mitigate the performance interference of memory region migration to the running program, the runtime limits the maximum data size that can be migrated in a time interval. In this experiment, we use move_pages system call to migrate all selected memory regions in one call and set the migration frequency to be every second. For this migration experiment, we used a two-NUMA ARMv8 processor based testbed, on which we emulated a two-tier memory system by injecting memory traffic to one NUMA node and limiting the free memory space on the other NUMA node. Effectively, the emulated multi-tier memory has a memory bandwidth ratio of 1.8× between the fast and the slower tier, and a memory capacity ratio of 2×. The results of SPE-guided optimization on multi-tier memory are presented in Figure 14. The memory analysis of region latency distribution in Section 4.3.1 identifies workloads suitable for multitier memory systems as those with skewed distribution. Consistent with this memory analysis, performance improvement is observed in the suitable workloads such as page ranking and connected components. In contrast, Llama and Memcached are less suitable for the tested migration policy, and they achieve no performance benefit. The longest observed idle period metric identifies workloads with few long gaps in sampled activity, for which runtime migration may cause interference. For instance, Memcached is identified as one of the least promising workloads for runtime migration because it has very few regions with long observed idle periods. Consequently, in Figure 14, Memcached suffers from a huge performance loss, as is quantified with its built-in 99th percentile latency, from runtime migration. Moreover, Cassandra has a substantial proportion of memory regions with long observed idle periods, making it a candidate for runtime migration. The results confirm that Cassandra achieves the highest performance improvement. In summary, the results show the usefulness of the memory region based metrics as introduced in Sections 4.3.1 and 4.3.2 in guiding optimization on multi-tier memory.

Related Works

Existing work explores memory access profiling with OS-based and PMU-based method. OS-based methods observe page-level access information, providing limited visibility into individual memory operations. Page poisoning techniques [1, 8, 19] track specific pages through customized fault handlers but incur substantial overhead from fault handling. Other work explores PTE-scan based profiling methods [6, 15, 16, 18, 21], which clear the PTE access bit and periodically scan the page table to determine if any page has been accessed between two scans. However, such method cannot detect repeated page accesses within profiling intervals, missing true hot regions. Methods using OS structures like active/inactive queues [11, 25] are lightweight but provide limited information. In contrast, our ARM SPE-based approach provides accurate, fine-grained memory access profiling with supported memory hierarchy information (L1, L2, and TLB level accesses). As demonstrated in our evaluation, we achieve higher hot-region precision and lower overhead than DAMON under the tested configurations, while recall depends on the workload. PMU based techniques rely on the performance monitoring units (PMUs) in modern processors, including Intel PEBS, AMD IBS, and Arm SPE. Among these, PEBS-based profiling methods have been widely used in NUMA bottleneck analysis [14], page hotness identification [15], and memory traces [5]. Studies show PEBS offers accurate event sampling while IBS provides richer per-sample data [22], though sampling bias and overhead remain challenges [23, 24, 26]. ARM SPE combines supported cache and TLB events, data-source information, and operation latency in a sample record, enabling memory characterization on ARM platforms.

6

Conclusions

In this work, we present a high-resolution memory characterization methodology based on the ARM SPE feature, addressing the increasing deployment of ARM-based processors in data centers. Motivated by new capabilities enabled by memory-centric sample tracing, we propose a set of new metrics, such as latency profiles and longest idle periods, for assessing the opportunity and feasibility of a workload on multi-tier memory systems. We provide a validated high-performance, low-overhead implementation of the characterization method. By evaluating eight memory-intensive workloads, including CloudSuite and LLM applications, on an ARMv8-based platform, our results highlight the usefulness of the proposed approach in studying a cloud-oriented cache hierarchy and in guiding the memory optimizations.

References [1] Neha Agarwal and Thomas F. Wenisch. 2017. Thermostat: Applicationtransparent Page Management for Two-tiered Main Memory. SIGPLAN Not. 52, 4 (April 2017), 631–644. doi:10.1145/3093336.3037706 [2] AmpereComputing. [n. d.]. Ampere Altra Max 64-Bit Multi-Core Processor Features. https://amperecomputing.com/assets/Altra_Max_Rev_A1_DS_v1_25_ 20240130_73cfcc518a_4705c00046.pdf. [3] Brian Jeff. 2023. Arm SPE: SoC Telemetry and Performance Analysis Using Statistical Profiling Extension. https://developer.arm.com/community/arm-communityblogs/b/architectures-and-processors-blog/posts/statistical-profile-extension. [4] Qwen Team. 2025. Qwen3-30B-A3B-Instruct-2507 Model Card. https:// huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507. [5] David Boehme, Todd Gamblin, David Beckingsale, Peer-Timo Bremer, Alfredo Gimenez, Matthew LeGendre, Olga Pearce, and Martin Schulz. 2016. Caliper: performance introspection for HPC software stacks. In SC’16: Proceedings of the

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM Conference SPE acronym ’XX, June 03–05, 2018, Woodstock, NY

International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 550–560. [6] Jinyoung Choi, Sergey Blagodurov, and Hung-Wei Tseng. 2021. Dancing in the Dark: Profiling for Tiered Memory. In 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). doi:10.1109/IPDPS49936.2021.00011 [7] Michael Ferdman, Almutaz Adileh, Onur Kocberber, Stavros Volos, Mohammad Alisafaee, Djordje Jevdjic, Cansu Kaynak, Adrian Daniel Popescu, Anastasia Ailamaki, and Babak Falsafi. 2012. Clearing the clouds: a study of emerging scale-out workloads on modern hardware. Acm sigplan notices 47, 4 (2012), 37–48. [8] Jayneel Gandhi, Arkaprava Basu, Mark D. Hill, and Michael M. Swift. 2014. BadgerTrap: A Tool to Instrument x86-64 TLB Misses. SIGARCH Comput. Archit. News 42, 2 (Sept. 2014), 20–23. doi:10.1145/2669594.2669599 [9] Georgi Gerganov and contributors. 2023. llama.cpp: Inference of LLaMA models in pure C/C++. https://github.com/ggerganov/llama.cpp. [10] Andrej Karpathy. 2024. llm.c: LLM training in simple, raw C/CUDA. https: //github.com/karpathy/llm.c. [11] Jonghyeon Kim, Wonkyo Choe, and Jeongseob Ahn. 2021. Exploring the Design Space of Page Management for Multi-Tiered Memory Systems. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 715–728. https://www.usenix.org/conference/atc21/presentation/kim-jonghyeon [12] Chigusa Kobayashi, Kazuto Ando, Tsuyoshi Yamaura, Hikaru Inoue, and Hitoshi Murai. 2026. Performance analysis of Arm-based processors across multiple compilers for HPC workloads.. In Proceedings of the Supercomputing Asia and International Conference on High Performance Computing in Asia Pacific Region. 236–246. [13] Huaicheng Li, Daniel S. Berger, Lisa Hsu, Daniel Ernst, Pantea Zardoshti, Stanko Novakovic, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, and Ricardo Bianchini. 2023. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms. In ASPLOS. [14] Xu Liu and Bo Wu. 2015. Scaanalyzer: A tool to identify memory scalability bottlenecks in parallel programs. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–12. [15] Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit Kanaujia, and Prakash Chauhan. 2023. Tpp: Transparent page placement for cxl-enabled tiered-memory. In ASPLOS. 742–755.

[16] Alan Nair, Sandeep Kumar, Aravinda Prasad, Ying Huang, Andy Rudoff, and Sreenivas Subramoney. 2024. Telescope: telemetry for gargantuan memory footprint applications. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). 409–424. [17] Tapti Palit, Yongming Shen, and Michael Ferdman. 2016. Demystifying Cloud Benchmarking. In 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 122–132. [18] SeongJae Park, Yunjae Lee, and Heon Y Yeom. 2019. Profiling dynamic data access patterns with controlled overhead and quality. In Proceedings of the 20th International Middleware Conference Industrial Track. [19] Zhenlin Qi, Shengan Zheng, Ying Huang, Yifeng Hui, Bowen Zhang, Linpeng Huang, and Hong Mei. 2025. Chrono: Meticulous Hotness Measurement and Flexible Page Migration for Memory Tiering. In Proceedings of the Twentieth European Conference on Computer Systems. 835–853. [20] Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Erez, and Simon Peter. 2021. HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles. [21] Jie Ren, Dong Xu, Junhee Ryu, Kwangsik Shin, Daewoo Kim, and Dong Li. 2024. MTM: Rethinking Memory Profiling and Migration for Multi-Tiered Large Memory. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece) (EuroSys ’24). ACM, 803–817. doi:10.1145/3627703.3650075 [22] Muhammad Aditya Sasongko, Milind Chabbi, Paul HJ Kelly, and Didem Unat. 2023. Precise event sampling on amd versus intel: Quantitative and qualitative comparison. IEEE Transactions on Parallel and Distributed Systems 34, 5 (2023), 1594–1608. [23] Akiyama Soramichi and Takahiro Hirofuchi. 2017. Quantitative evaluation of intel pebs overhead for online system-noise analysis. ROSS (2017). [24] Hao Xu, Qingsen Wang, Shuang Song, Lizy Kurian John, and Xu Liu. 2019. Can we trust profiling results? Understanding and fixing the inaccuracy in modern profilers. In Proceedings of the ACM International Conference on Supercomputing. 284–295. [25] Zi Yan, Daniel Lustig, David Nellans, and Abhishek Bhattacharjee. 2019. Nimble Page Management for Tiered Memory Systems. In International Conference on Architectural Support for Programming Languages and Operating Systems. [26] Jifei Yi, Benchao Dong, Mingkai Dong, and Haibo Chen. 2020. On the precision of precise event based sampling. In Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems. 98–105.

Related documents

• Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE#1448227
• Impact of Security Modeling Fidelity on Hyperscale Data Center Hosting Capacity Assessment#1343916
• Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers#10316
• Query Density-Driven Partitioning for Spatiotemporal Load Balancing on Processing-in-Memory Systems#422345
• SKYE: Write-Optimized Key-Value Store with Fine-Grained Control over Persistent Memory Accesses#1034944