arXiv:2604.12300v1 [cs.OS] 14 Apr 2026
TierBPF: Page Migration Admission Control for Tiered Memory via eBPF Xi Wang
Tal Zussman
Yuang Xu
University of California, Merced USA
Columbia University USA
University of California, Merced USA
Bin Ma
Asaf Cidon
Dong Li
University of California, Merced USA
Columbia University USA
University of California, Merced USA
Abstract Existing software-based memory tiering systems decide which pages to place on the slower or faster tier. However, they do not take into account two important factors that greatly influence application performance: the size of the migrated pages, and the underlying hardware device and tiering topology. We introduce TierBPF, a software mechanism that can be plugged into existing memory tiering systems to take these factors into account, by making simple binary page admission decisions. TierBPF is implemented as a set of eBPF hooks, which allow users to define their own custom policies. In order to make its decisions, TierBPF utilizes a lightweight tracking mechanism for page profiling which is not dependent on the application’s working set size. TierBPF, integrated into three memory tiering systems and evaluated with 17 workloads, achieves geomean throughput gains of up to 17.7% with improvements of up to 75% for individual workloads.
1
Introduction
As the number of cores available in modern processors grows faster than local DRAM server capacity [21, 57], datacenter operators are deploying tiered-memory architectures that pair fast, capacity-limited local DRAM with a slower, highercapacity memory tier. Memory tiering systems are responsible for migrating pages across these tiers. Memory tiering systems are typically designed around a high-level abstraction: a heterogeneous memory architecture composed of slow memory and fast memory. This abstraction simplifies the page migration decision based on page access frequency and recency [21, 34, 39, 43, 48, 50]. That is, migrating hot (frequently or recently accessed) pages to the fast tier, and keeping cold (infrequently used) pages in the slow tier. Albeit simple to reason about, this high-level abstraction causes two problems. Problem 1: The abstraction ignores the interaction between page size and migration. Many production systems enable Transparent Huge Pages (THP), allocating memory in 2 MB pages to reduce the TLB miss overhead. However, page migration in memory tiering systems is intertwined with the page size. When the kernel migrates a 2 MB THP,
it must copy the entire page, even if only a small subset of its constituent base pages are frequently accessed. This behavior wastes migration bandwidth and may squander scarce fast-tier capacity with cold data. Splitting the THP into 512 base pages and migrating only the hot ones avoids this waste, but at the price of increased TLB overhead. The problem of granular hot memory regions in THP is not specific to migration. Indeed, Linux v6.8 attempted to address this by introducing multi-size THP (mTHP) [36], which supports intermediate page sizes from 16 KB to 1 MB, but the page size is determined only at allocation time. As such, no existing tiering system exploits splitting pages into intermediate sizes during page migration. In existing tiered memory systems, a page originally allocated as a 1 MB page can only be migrated in one big chunk. Problem 2: The fast-slow tier abstraction hides deployment specific properties. Tiering deployments are heterogeneous, both in terms of the types of devices and in terms of the deployment topology. CXL-attached memory expansion connects over a PCIe link, Intel Optane PMEM is integrated via the DDR bus, while emerging substrates, such as CXL-attached flash [15, 52] and disaggregated memory pools [14, 44, 54], further broaden the design space. These deployment configurations differ not only in latency and bandwidth, but also in architectural properties such as memory duplex mode (i.e., read-write interference), persistence, and cache coherence. As a result, there is no one-size-fits-all policy that works for any deployment configuration. For example, a policy that improves performance on a CXL-based memory expansion may have little benefit—or even incur overhead—on persistent memory-based platforms (§3.3). These two problems share a common root cause: there is no existing mechanism for tiered memory systems to take page size or the tiered memory’s deployment configuration into account. To this end, we introduce TierBPF, a system that interfaces with existing tiered memory systems, making them adaptable to factors like page size granularity and tiering architecture. TierBPF sits between the memory tiering system’s migration decision-making and the page migration mechanism. Our key insight is TierBPF makes existing tiered memory systems aware of their broader context by applying page migration admission control. When the tiering system
decides that a page should be migrated, TierBPF applies two filters before the migration proceeds: a granularity filter that determines the right page size for migration, and an architecture filter that determines whether the migration actually benefits the deployed hardware. These two filters transform a coarse migrate-or-not decision into a fine-grained one that accounts for subpage access patterns, runtime conditions (i.e., memory bandwidth contention), and hardware characteristics (i.e., memory duplex mode). The granularity filter addresses Problem 1. Instead of migrating the full 2 MB THP or splitting it all the way to 4 KB, this filter selects an intermediate mTHP size that is large enough to preserve TLB benefits yet small enough to avoid migrating cold data. The choice is contention-aware: under low slow-tier bandwidth contention, the filter keeps pages at 2 MB for maximum TLB coverage; under high contention— common in multi-tenant cloud environments—it selects a smaller mTHP size (e.g., 64 KB or 128 KB), so each migration moves less data, wastes less bandwidth, and occupies less fast-tier capacity (§3.2). The architecture filter addresses Problem 2. The specific device hardware affects the effectiveness of page migration. For example, CXL memory uses full-duplex links where reads and writes proceed on independent channels [55], while Intel Optane PMEM uses half-duplex DDR where reads and writes share the same bus. Hence, on a CXL system with readdominant memory traffic, selectively holding back writeheavy pages from page promotion can improve bandwidth utilization by keeping the full-duplex channels balanced. The same policy on PMEM would merely prevent hot pages from reaching fast memory, with no bandwidth benefit. The page migration admission control allows memory tiering to customize migration decisions based on the duplex mode. The critical obstacle in supporting these filters is accurate and low-overhead memory profiling. Building and maintaining per-page profiling (as in MEMTIS [12]) is not scalable in terms of runtime and memory overhead. Instead, TierBPF introduces lightweight eBPF-based profiling. In particular, TierBPF processes hardware memory-access samples entirely in-kernel using eBPF, avoiding the context switches and data copying suffered by the user-space profiling. In addition, TierBPF uses a lightweight, compact global subpage histogram (using only 4 KB per process) instead of per-page tracking structures, enabling lightweight profiling with overhead independent of the workload’s working set size. To make page-migration admission control deployable across different tiering systems and adaptable to evolving memory architectures, we implement all policy logic as usercustomizable eBPF programs. We expose three hooks along the kernel’s NUMA migration path: one for split-size selection, one for subpage hot/cold classification, and one for migration admission. This design cleanly separates policy (i.e., which page to migrate and at what granularity) from
mechanism (i.e., how to migrate), allowing the admission control to easily plug into existing memory tiering systems. Our paper’s major contributions are: • We identify two problems in existing memory tiering systems caused by the fast-vs-slow memory abstraction: the lack of both migration-time page-granularity awareness and architecture-aware page migration filtering. • We present TierBPF, the first system to support dynamic, contention-aware mTHP-size selection and memory duplex mode-aware page migration. TierBPF implements all migration policies as eBPF programs, allowing operators to adapt migration policies to hardware topologies. • We evaluate TierBPF on two memory platforms: (a) CXL memory expansion, and (b) Intel PMEM. We integrate TierBPF with three memory tiering systems: AutoNUMA [7], TPP [21], and Colloid [43]. Our results demonstrate that TierBPF significantly improves both adaptability and performance across architectures, consistently outperforming prior approaches under diverse workloads: TierBPF achieves geomean throughput gains of 7.4-17.7%, with improvements of up to 75% for individual workloads, and outperforms MEMTIS by up to 26%.
2
Background
Multi-size transparent huge pages. Virtual-to-physical address translation depends heavily on the TLB. Using 4 KB pages significantly limits TLB reach (e.g., a few hundred kilobytes per TLB), increasing TLB miss rates for modern memory-intensive workloads. Since each TLB miss can invoke a multi-level page-table walk that may cost tens to hundreds of cycles depending on cache residency, addresstranslation overheads alone have been shown to reduce application performance by up to 30% [18, 22, 46]. THPs alleviate this by using 2 MB pages, reducing translation frequency and improving TLB hit rates. However, 2 MB THPs introduce their own challenges. Allocating physically contiguous 2 MB regions becomes costly when memory is fragmented. To address this, Linux 6.8 introduced mTHP, which supports anonymous page allocation at intermediate sizes (16 KB, 32 KB, 64 KB, 128 KB, 256 KB, 512 KB, and 1 MB) alongside the traditional 4 KB and 2 MB options. Each size can be enabled or disabled independently via sysfs. mTHP provides a practical middle ground: smaller huge-page sizes are easier to allocate, reduce memory waste, and still improve TLB efficiency compared to 4 KB pages. Internally, the Linux kernel represents each (m)THP as a folio; we use folio and page interchangeably in this paper. However, current mTHP support is limited to allocation. The kernel does not consider page sizes when migrating pages across memory tiers. MEMTIS [12] improves on this by making a binary choice: migrating the entire 2 MB page or splitting it into 512 individual 4 KB base pages for selective 2
3
Motivation
Access Count
6 4 2 0
2
0
100
200 300 400 Subpage (0 511)
500
bc-web
0 4
0
100
200 300 400 Subpage (0 511)
500
200 300 400 Subpage (0 511)
500
silo_ycsb
1e5
2
0.5 0.0
NPB-CG.D
1e3
1
1.0
0
100
200 300 400 Subpage (0 511)
500
0
0
100
Figure 1. Hot subpage distribution within 2 MB THPs. We use workloads summarized in Table 3.
3.1
THP Migration Dilemma
When THP is enabled, the kernel manages memory in 2 MB pages by default. Consider a 2 MB THP where only a small cluster of subpages is frequently accessed. Existing memory tiering systems face two suboptimal choices. Option 1 (Linux default): migrate the full 2 MB THP. This wastes migration bandwidth—the system copies 2 MB of data even if only 64 KB is hot—and consumes scarce fast-tier capacity by allocating 2 MB of DRAM for cold data. Moreover, migrating a 2 MB page requires a contiguous 2 MB free region on the destination node, which is often unavailable under memory fragmentation, causing migrations to fail (§6.3). Option 2 (e.g., MEMTIS [12]): split the THP into 4 KB base pages and migrate the hot ones. While this preserves bandwidth and fast-tier space, it destroys TLB coverage. After splitting, the application experiences more TLB misses, which can offset the performance gains of page migration. Figure 1 shows the distribution of hot 4 KB subpages within 2 MB THPs across representative workloads. Access patterns are rarely uniform: most THPs contain a concentrated hot region spanning tens to hundreds of kilobytes— far larger than a single 4 KB page but far smaller than the full 2 MB. This suggests that intermediate mTHP sizes (e.g., 64 KB or 256 KB) could more precisely capture the hot region, migrating only what is necessary while preserving TLB coverage for the migrated portion. Observation 1: Only a small subset of subpages within each THP is hot. Migrating at 2 MB wastes bandwidth and fast-tier capacity; splitting to 4 KB discards TLB benefits. Intermediate mTHP sizes can capture the hot region without either penalty.
3.2
Despite recent advances in tiered memory architecture, THP support, and eBPF programmability, today’s memory tiering stack suffers from three fundamental limitations.
NPB-SP.D
1e4
1e3
Access Count
migration, with no consideration of other mTHP sizes. However, splitting a 2 MB page all the way down to 4 KB pages loses the TLB performance benefits entirely, even when a 64 KB or 128 KB split would have been sufficient. Furthermore, MEMTIS’s design cannot scale to support intermediate mTHP sizes, for three reasons. First, MEMTIS relies on per-page metadata tracking, which is not scalable: MEMTIS maintains per-subpage access metadata for every 4 KB subpage within each huge page, which grows linearly with the working set size of the workload (e.g., tens of GB on a TB-scale server). The metadata is typically placed in fast memory because of its high access frequency, taking most of the fast memory capacity (tens of GBs [14]) in a data-center server. This leaves little space in the fast tier for workload pages that could improve performance. Second, MEMTIS can lead to large runtime overhead. MEMTIS decides whether to split a huge page by comparing the measured fast-tier hit ratio against an estimated base-page hit ratio. That means that supporting 𝑁 candidate mTHP sizes in MEMTIS would need to build and maintain 𝑁 separate emulated histograms and hit-ratio estimates, multiplying both memory footprint and splitting-decision computation. Evaluating with memory-intensive benchmarks (LU.D and SP.D shown in Table 3), we observe that MEMTIS leads to up to 15% runtime overhead because of the above computation and metadata management. Third, MEMTIS is architecture-dependent. In order to calculate fast-tier hit ratio to decide page size, MEMTIS relies on specific hardware counters to determine from which tier a memory access comes. This limits the generality of MEMTIS on diverse memory architectures. This motivates supporting multi-size THP during page migration in a scalable and architecture-independent manner. eBPF for kernel customization. Extended Berkeley Packet Filter (eBPF) is a Linux kernel technology that enables user-defined programs to run safely inside the kernel in a sandboxed environment, without requiring kernel source modifications or custom kernel modules. The eBPF verifier enforces strict safety guarantees: programs cannot crash the kernel, access arbitrary memory, or execute unbounded loops. Originally designed for network packet filtering, eBPF has evolved into a general-purpose mechanism for extending kernel functionality. Recent work highlights its versatility in customizing core kernel subsystems [24, 25, 59]. Collectively, these systems illustrate a broader trend: eBPF is making kernel memory management programmable.
There is No One Ideal Page Size
Deciding the mTHP size is challenging. With different levels of memory contention, the same application prefers different page sizes to improve performance. 3
Read Only
Read:Write = 1:1
(a) DDR5
0
25
50
75
Bandwidth (GB/s) (c) CXL (PCIe4)
100
Latency (ns)
1250
Latency (ns)
2500 2000 1500 1000 500
Figure 2. The optimal mTHP size shifts under memory contention, as per §6.1. Stars indicate the best performance.
1000 800 600 400 200
(b) Intel Optane PMem
Latency (ns)
better
Latency (ns)
300 250 200 150 100
0
10
20
10
20
30
Bandwidth (GB/s) (d) CXL (PCIe5)
1000
5
10
15
Bandwidth (GB/s)
20
750 500 250
Bandwidth (GB/s)
30
Figure 3. Memory bandwidth differs on different memory tiering architectures.
Observation 2: The optimal mTHP size depends on workload and runtime memory contention. Under low contention, larger pages are preferred for TLB coverage; under high contention, smaller mTHP sizes cause less overhead, waste less fast-tier capacity, and yield better overall performance. A static size policy cannot adapt to these changes.
When an application runs alone and the slow tier is not under bandwidth pressure, larger THP sizes are generally preferable. In this setting, the slow tier can offer relatively low memory-access latency [43], so the cost of migrating huge pages is small, and the bandwidth consumed by such migrations does not meaningfully affect performance. As a result, the TLB benefits of 2 MB pages dominate. However, in cloud environments where multiple workloads share the same machine and memory tiers, the trade-offs shift. Under contention in the slow tier, migrating 2 MB pages becomes more expensive: misidentified hot pages waste migration bandwidth, and truly hot pages left in the slow tier suffer greater performance penalties due to increased access latency. In these scenarios, smaller mTHP sizes reduce both slow-tier bandwidth consumption and fast-tier capacity usage per migration, allowing more distinct hot regions to reside in fast memory simultaneously. Figure 2 shows the optimal mTHP size under memory contention. The results show that there is no one-size-fitsall size: the optimal page size is different when varying the workloads and memory contention level. This implies that any static page-size policy—whether configured system-wide via sysfs or defined per application—will be suboptimal in a dynamic, multi-tenant environment. The system must instead adapt its mTHP size selection at runtime, guided by the memory architecture and prevailing memory pressure. Linux mTHP [36] allows administrators to enable or disable specific mTHP sizes via sysfs, but this is a static, manual configuration. It does not adapt to runtime conditions. eBPFmm [25] uses eBPF to select page sizes at allocation time, but it relies on pre-defined application profiles and does not consider page migration. Neither system provides dynamic, contention-aware mTHP size selection for page migration.
3.3
Architecture-Agnostic Migration
Existing tiered memory systems apply the same migration policy without considering the underlying memory architecture. To explain why this is problematic, we use a case study of CXL memory and Optane persistent memory. CXL memory provides full-duplex links where reads and writes use independent channels [55]. The migration policy can take advantage of this property by actively rebalancing the read/write traffic across the full-duplex channels. In contrast, an PMEM system has half-duplex DDR interfaces where read and write traffics compete for the same link. As a result, rebalancing read/write traffic does not improve bandwidth utilization but merely shifts contention from one direction to the other. Worse, selectively holding back read- or writeheavy pages from promotion prevents genuinely hot pages from reaching the fast tier, degrading overall performance. As a result, a traffic-rebalancing policy that performs well on a CXL platform can lose effectiveness on a PMEM platform. Memory-tiering systems must therefore be tailored to the memory architecture. Figure 3 illustrates this discrepancy: CXL reaches 149% and 125% of the read-only peak bandwidth on PCIe4 and PCIe5, respectively, with the mixed read/write pattern by leveraging its full-duplex link, whereas 4
PMEM, constrained by its half-duplex DDR interface, experiences substantial bandwidth degradation (only 17% of the read-only peak bandwidth) under the same traffic pattern. Despite this, most memory tiering systems (e.g., AutoNUMA [7], TPP [21], Nimble [51], MEMTIS [12], Colloid [43], and Nomad [48]) do not fully consider architectural differences and cannot flexibly customize the policy for the hardware deployment.
Workload User Space
THP Access Heat Analyzer
Memory Traffic Monitor
Kernel Space
eBPF-based profiling Numa Migration Path Migration Admission
Observation 3: Migration policies produce different outcomes on different hardware configurations. We must customize the policy to specific tiered-memory hardware setups to maximize performance.
Tiered Memory
Fast Tier Memory
mTHP Splitting Size
Promote Demote
Hot/Cold subfolio Classify
Slow Tier Memory
Figure 4. TierBPF architecture overview.
4
Key Ideas in TierBPF
TierBPF addresses the three gaps identified in §3. First, it splits 2 MB THPs into optimal mTHP sizes based on hot subpage distribution, avoiding both bandwidth waste and unnecessary TLB misses (§3.1). Second, it activates the THP splitting policy based on runtime memory contention (§3.2). Third, it customizes migration decisions based on memory architecture characteristics (§3.3). The three policies are implemented as eBPF programs, making them safe, low-overhead, and deployable without further kernel modifications. Lightweight memory profiling. Each policy decision in TierBPF—which mTHP size to split into, which subpages are hot, and whether to admit a migration—relies on knowing page access patterns. TierBPF maintains a global subpage histogram—a fixed 512-entry array (4 KB) per application— that maps hardware-sampled accesses from all THPs to their subpage offset. This constant-size structure does not grow with the working set of the workload. The profiling logic runs as an eBPF program attached to hardware performance events, aggregating samples directly in interrupt context and avoiding context switches and data copies between kernel and user space (§5.1). This lightweight profiling builds a foundation to guide three policy decisions, described below. Contention-aware mTHP splitting. TierBPF dynamically selects the largest mTHP size that captures most of the hot subpages within a dense memory region. The mTHP size is not a static configuration. Because the optimal size changes as memory contention varies (Observation 2), TierBPF activates THP splitting only when the slow-tier bandwidth is under pressure. Beyond reducing bandwidth waste, splitting also reduces migration failures: a smaller subfolio requires far less contiguous free space than a 2 MB THP, so migrations succeed more often under memory fragmentation. Architecture-aware migration policy. Even when a page is hot enough to merit promotion, migrating it does not always help. On a CXL system with full-duplex channels,
reads and writes proceed independently. If CXL read bandwidth is the bottleneck, promoting read-heavy pages alleviates pressure, but promoting write-heavy pages offers little benefit, because the write channel is typically uncongested (Observation 3). On a half-duplex PMEM system, however, the same selective policy would be counterproductive: reads and writes share a single channel, so all hot pages benefit equally from promotion. TierBPF adds an architecture-aware migration-admission hook. Before a page is promoted, an eBPF program evaluates its read/write ratio in the context of real-time CXL traffic. Write-heavy pages are deferred unless the CXL write channel is genuinely saturated. This hook is enabled only on CXL platforms and becomes a no-op on PMEM (or other halfduplex memory architectures), or single-tier systems. eBPF as the policy layer. TierBPF uses eBPF to decouple migration policy from migration mechanism. It exposes three hooks along the kernel’s NUMA migration path as eBPF attachment points: one for split-size selection, one for subpage hot/cold classification, and one for migration admission. When no eBPF program is loaded, the kernel falls back to safe defaults (no splitting and unconditional migration). Figure 4 illustrates the architecture of TierBPF, which spans three layers. In the kernel, an eBPF program attached to hardware performance events profiles memory accesses and aggregates them into BPF maps. User-space daemons derive policy decisions from the profiling data and publish them to the pinned BPF maps. On the kernel’s NUMA migration path, three eBPF hooks serving as page migration admission control enforce these decisions at migration time: (1) mTHP Splitting Size selects the target mTHP granularity, (2) Hot/Cold Subfolio Classify determines which subfolios to promote or not, and (3) Migration Admission accepts or defers a migration based on the current CXL traffic pattern. 5
Table 1. Three eBPF hooks in TierBPF. Hook
Location
Default
bpf_thp_pick_ split_order
mm/migrate.c
2MB (no split)
bpf_subpage_ is_cold
mm/migrate.c
0 (all hot)
bpf_thp_numa_ migrate_admission
kernel/sched/ 0 (allow) fair.c
5
Why can a global histogram work? Folding all THPs into one histogram loses access distribution patterns for individual THPs. However, the key insight is that with samplingbased profiling, the hardware sampler naturally focuses on frequently accessed pages. The global histogram is therefore dominated by the subpage distribution of hot THPs— precisely the ones that are candidates for migration. Cold THPs contribute few samples and have little influence on the histogram. Since cold THPs should remain in slow memory and are not to be migrated, their subpage distribution is irrelevant to splitting decisions. In other words, the global histogram is not a low-quality approximation to access distribution patterns. Rather, it is a direct representation of the THPs that matter for migration.
TierBPF Design
We describe the three core mechanisms in TierBPF. First, we describe the lightweight memory profiling subsystem (§5.1), which combines a fixed-size global subpage histogram, a dual blocked counting Bloom filter (bCBF), and eBPF-based in-kernel sampling to collect access patterns with minimal overhead. Second, we present the mTHP splitting algorithm (§5.2), which uses normalized Shannon entropy to detect skewed access distributions and selects the largest mTHP size whose heat density is sufficient to capture hot regions, ensuring that only hot subfolios are migrated. Third, we describe the duplex-aware migration admission control (§5.3), which classifies each page’s read/write ratio and admits or defers migrations based on real-time memory traffic. TierBPF realizes its policies through three eBPF hooks inserted into the kernel’s NUMA-migration path (Table 1). All three follow the same design pattern: a noinline kernel function annotated with ALLOW_ERROR_INJECTION that returns a safe default when no eBPF program is attached, but whose return value can be overridden by an fmod_ret eBPF program. These hooks enable the three core mechanisms. We discuss the design in detail in this section. 5.1
5.1.2 eBPF-Based Kernel Profiling. Prior tiering systems such as MEMTIS [12] and HeMem [32] process hardware event samples in user-space. As shown in Figure 5, the user-space approach follows a multi-step data path: the kernel serializes each sample into a perf ring buffer, wakes up a user-space thread, and the thread reads and parses each sample before updating its own metadata. This path incurs substantial overhead from multiple sources. First, each raw sample gets copied multiple times: the kernel serializes a perf_record_sample into the perf ring buffer, the user-space thread reads it from the ring buffer, deserializes each field, and writes the extracted result into its own hash table. Second, delivering these samples requires context switches: the kernel must wake up the user-space thread, and the thread returns to sleep after processing each batch. Third, processing 𝑁 individual samples in user-space also forces the CPU to alternate between ring buffer pages and hash table pages, causing CPU cache pollution as profiling metadata competes with application data for cache capacity. At high sampling rates (e.g., 105 –106 samples/sec), these costs compound and become significant. TierBPF avoids all these overheads by running the profiling logic inside the kernel as an eBPF program attached to perf events (as shown in Figure 5). When a hardware event sampling interrupt fires, the eBPF program executes directly in the interrupt context. It reads the sampled address, computes the update, and writes to a BPF map—all without entering user-space. The ring buffer is bypassed entirely: the raw sample never leaves the kernel, requiring no serialization, no parsing, and no cross-boundary data transfer. User-space only reads the BPF maps when it needs to make a policy decision, and at that point it reads aggregated results (hundreds of entries), not raw samples (potentially millions).
Lightweight Memory Profiling
Every policy decision in TierBPF—which mTHP size to split to, which subfolios to promote, whether to admit a migration— relies on access profiles collected from hardware event sampling. The profiling must be both effective and cheap. 5.1.1 Global Subpage Histogram in mTHP Splitting. For mTHP splitting, what matters is not the exact access count of each individual page, but the access distribution pattern within a 2 MB THP region (i.e., which subpages are hot and which are cold). Instead of creating a subpage histogram of 512 entries per huge page, TierBPF captures hot subpage distribution with a global subpage histogram: an array of 512 entries (representing 512 subpage offsets) per application, where each hardware event sample is mapped to a slot by: page_addr % 511. This single modulo operation maps accesses from all 2 MB THPs of an application into one 512slot array, requiring only 512 × 8 B = 4 KB per application regardless of working set size.
5.1.3 R/W Classification in Migration Admission. For duplex-aware migration admission (§5.3), TierBPF must classify each page’s read/write ratio. This is inherently a perpage query that the global histogram cannot serve. Explicitly maintaining per-page counters is not scalable (§2). To address this problem, TierBPF introduces a blocked counting 6
reaches a target coverage ratio 𝑃 (default 80%): the Path of eBPF-based Monitor
the Path of User-Space Monitor
PMU Sampling (IBS/PEBS)
PMU Sampling (IBS/PEBS)
Hardware interrupt
Hardware interrupt
𝐾∗ ∑︁
𝑐˜ 𝑗 ≥ 𝑃 · 𝐶 total
(1)
𝑗=0 Perf Subsystem
Perf Subsystem
Routes samples to BPF handler
Routes samples to ring buffer
eBPF Program
Perf Ring Buffer
The hot threshold is then 𝑇 = 𝑐˜𝐾 ∗ . This coverage-based threshold adapts automatically to workload skew and varying access distributions, ensuring that the most frequently accessed subpages are consistently classified as hot. Stage 2: hot subfolio classification. Each subpage is classified using the threshold 𝑇 : ( 1 (hot), if 𝑐𝑖 ≥ 𝑇 ℎ𝑖 = (2) 0 (cold), if 𝑐𝑖 < 𝑇
Updates BPF map poll/epoll wakeup
BPF Map M aggregated entries
Kernel / User-Space Boundary Reads M entries on demand
M << N
User Thread Parse N raw samples Insert/Update hash table
User Daemon
User Hash Table
Figure 5. Comparison of profiling data paths.
producing a binary hot/cold map for 512 subpages in a THP. Stage 3: mTHP size selection via heat density. The goal of this stage is to select the largest mTHP size that still tightly captures a hot region. For each candidate size (2 MB, 1 MB, 512 KB, 256 KB, 128 KB, 64 KB), the THP is partitioned into aligned subfolios of 𝐿 contiguous subpages, where 𝐿 equals the candidate size divided by 4 KB (e.g., 𝐿 = 128 for a 512 KB subfolio). The heat density of a subfolio is defined as the fraction of its subpages classified as hot:
Bloom filter (bCBF) inspired by HybridTier [39]. A counting Bloom filter (CBF) provides space-efficient approximate count estimation; a blocked CBF further improves cache efficiency by partitioning counters into cache-line-sized blocks, so each lookup touches exactly one cache line. Concretely, a bCBF maps each page address via hashing to 𝑘 counters within a single block. A Get returns the minimum of the 𝑘 counters; an Update increments only the counters at the current minimum [39]. TierBPF extends this mechanism with a dual-bCBF design, one for reads and one for writes. The write ratio of a page is computed as 𝑤 min /(𝑤 min + 𝑟 min ), where 𝑤 min and 𝑟 min are the Get results from the write and read bCBFs respectively. TierBPF allocates the dual bCBF only when CXL memory is present and duplex-aware admission is active. On PMEM or single-tier systems, TierBPF does not need this counting mechanism, because the half-duplex interface does not benefit from read/write traffic rebalancing (§3.3). 5.2
heat_density( [𝑖, 𝑖+𝐿)) =
𝑖+𝐿−1 ∑︁
ℎ𝑗 / 𝐿
(3)
𝑗=𝑖
TierBPF selects the largest mTHP size for which at least one aligned subfolio achieves a heat density of at least 𝜏ℎ . A lower 𝜏ℎ would admit subfolios with excessive cold data, wasting fast-tier capacity; a higher 𝜏ℎ would force unnecessarily small splits, sacrificing TLB coverage. By default 𝜏ℎ = 75%, keeping selected subfolios predominantly hot while still favoring larger mTHP sizes for less TLB pressure. Stage 4: migration target selection. After splitting, TierBPF does not migrate all subfolios. A subfolio is added to the migration list if it satisfies either of two conditions: (1) it is classified as hot based on heat density, or (2) it contains the NUMA-faulted address that triggered the migration. The NUMA fault reflects a recent locality mismatch that the hardware event sampler may not yet have observed. By combining these two signals, TierBPF selects subfolios using both frequency-based information (from the subpage histogram) and recency-based information (from the NUMA fault). Subfolios that are cold and do not contain the faulting address remain in slow memory. Flat distribution detection. Before executing the stages above, TierBPF first checks whether the access distribution is nearly uniform using normalized Shannon entropy (𝐻 norm ) [29], an information-theoretic measure that quantifies the spread of a probability distribution as a single scalar in [0, 1]. Applied here, it captures how evenly accesses are distributed across the 512 subpages. 0 means all accesses hit
mTHP Splitting
Given the subpage histogram from §5.1, TierBPF must decide what mTHP size to split into, which subfolios are hot, and which subfolios to migrate. We do not change the Linux kernel’s inherent page merging mechanism. 5.2.1 THP Splitting Decision. When slow-tier bandwidth is contended and THP splitting is activated, a background user-space analyzer reads the global subpage histogram for each application and computes the target mTHP size. Let 𝑐𝑖 denote the sampled access count of the 𝑖-th 4 KB subpage within a 2 MB THP region (𝑖 = 0, 1, . . . , 511), and let Í511 𝐶 total = 𝑖=0 𝑐𝑖 . The algorithm to split THP has four stages. Stage 1: hot threshold determination. This stage separates hot subpages from cold ones. TierBPF sorts subpages by their access counts (𝑐˜0 ≥ 𝑐˜1 ≥ · · · ≥ 𝑐˜511 ) and finds the smallest 𝐾 ∗ so that the cumulative sum of top-𝐾 ∗ subpages 7
Table 2. Memory duplex mode-aware migration admission policy on the CXL memory. R=read and W=write. R-dominant CXL traffic
Balanced CXL traffic
W-dominant CXL traffic
Allow Prevent
Allow Allow
Prevent Allow
R-heavy page W-heavy page
one subpage, 1 means perfectly uniform: ∑︁ 𝐻 norm = − 𝑝𝑖 log2 𝑝𝑖 / log2 512, 𝑝𝑖 = 𝑐𝑖 /𝐶 total
• A server with CXL memory expansion with dual-socket AMD EPYC 9555s (64 cores per socket). Each socket has 8×32 GB DDR5 DRAM as local memory. Socket 0 connects to a CXL Type-3 Memory Expansion device (128 GB DDR4) over PCIe 5.0, providing a full-duplex link. • A server with Intel Optane PMEM with dual-socket Intel Xeon Gold 6252 (24 cores per socket). Each socket has 6×16 GB DDR4 DRAM as the fast tier and 6×128 GB Intel Optane PMEM as the slow tier. By default, we use the CXL server. To isolate the impact of page migration from cross-socket NUMA effects, we use Socket 0 only, following prior efforts in [12, 21, 50]. For hardware event sampling, the sampling rate is 4,000 samples/sec per CPU. System integration. To evaluate the portability of TierBPF, we integrate TierBPF into three kernel-level memory tiering systems: AutoNUMA, TPP, and Colloid. For AutoNUMA, we use the vanilla Linux kernel v6.12. For TPP, we change AutoNUMA’s hotness detection from NUMA hint fault latency to an LRU active-list check and add TPP’s asynchronous background demotion mechanism [21]. For Colloid, we use its TPP-based implementation [43] and replace Intel CHA-based memory tier latency monitoring with AMD Data Fabric performance counters. For each system, we set /proc/sys/kernel/numa_balan cing=2 to enable promotion and /sys/kernel/mm/numa/ demotion_enabled=1 to enable demotion. For THP, under /sys/kernel/mm, we set (i) /transparent_hugepage/ena bled=always, (ii) /transparent_hugepage/defrag=always, and (iii) /transparent_hugepage/hugepages-2048kB/ena bled=inherit. Baselines. On the CXL server, we integrate TierBPF into AutoNUMA, TPP, and Colloid, and compare against each system’s unmodified baseline. On the PMEM-enabled server, we additionally compare TierBPF-enhanced AutoNUMA against MEMTIS [12], the state-of-the-art system for huge-pageaware page migration. We cannot evaluate MEMTIS on the CXL platform because MEMTIS relies on Intel PEBS events to distinguish memory access sources (i.e., local DRAM or remote DRAM) at the hardware level. AMD IBS, used on our CXL server, does not provide equivalent hardware performance counters. Workloads. We evaluate TierBPF on 17 memory-intensive benchmarks, summarized in Table 3. The benchmarks span three categories: (1) 12 graph-processing workloads from the GAP Benchmark Suite (GAPBS) [5], consisting of four algorithms (betweenness centrality, breadth-first search, connected components, and PageRank) run on three input graphs (Twitter, uniform random, and Web); (2) an in-memory database engine (Silo-YCSB [42]); and (3) four high-performance computing benchmarks (LU, SP, CG, and MG) from the NAS Parallel Benchmarks (NPB) [4]. These workloads have been used extensively in related work [12, 17, 21, 31, 34, 43, 44].
(4)
𝑖
If 𝐻 norm ≥ 0.95, the distribution is essentially uniform—there is no hot region to isolate, so splitting provides no benefit. In this case, TierBPF falls back to the default NUMA fault behavior: the entire THP is migrated without splitting, following the recency-based signal from the fault. 5.2.2 Kernel-Side Enforcement. When a NUMA fault triggers migration of a THP and splitting is enabled, the kernel calls the first eBPF hook (bpf_thp_pick_split_order) to obtain the target mTHP splitting size. It then splits the THP into subfolios of that order and calls the second hook (bpf_subpage_is_cold) for each subfolio to detect hot subfolios. Hot subfolios and the faulting subfolio are added to the migration list; cold subfolios are left in place. 5.3
Duplex-Aware Migration Admission Control
The splitting pipeline above determines how to migrate (sub)pages. We also determine whether to migrate or not. A user-space daemon monitors the CXL read/write traffic by hardware event counters and stores a sliding window of measurements in a BPF map. On each NUMA fault, the admission eBPF hook (bpf_thp_numa_ migrate_admission) classifies the faulting page as read-heavy or write-heavy using the dual bCBF (§5.1.3), and checks the current CXL traffic pattern. The admission control policy is summarized in Table 2: TierBPF migrates pages whose type matches the dominant traffic, and holds back the rest. The admission control runs before the splitting stage: the eBPF hook decides whether to admit the page for migration and only admitted pages proceed to the splitting pipeline.
6
Evaluation
TierBPF comprises ∼6,800 lines of user-space code (eBPF programs, loaders, daemons, and analyzers) and 540 lines of kernel modifications to Linux v6.12 for eBPF hooks and page splitting. Since neither TPP nor Colloid support mTHP, we re-implemented both on Linux v6.12, requiring ∼270 and 1,160 lines of kernel changes, respectively. 6.1
Evaluation Methodology
Evaluation platforms. To evaluate TierBPF, we use two servers with different memory architectures. 8
Table 3. Benchmarks used for evaluation.
6.2
Suite
Benchmark
Input / Configuration
Footprint
GAP [5]
BC BFS CC PR
Twitter / Uniform Random / Web
12–24 GB
LU.D SP.D CG.D MG.D
Lower-Upper Gauss-Seidel solver Scalar Penta-diagonal solver Conjugate Gradient Multigrid
58 GB 76 GB 16 GB 28 GB
YCSB
YCSB, scale factor 200K
29 GB
Evaluation of THP Splitting
Figure 6 shows the end-to-end performance of TierBPF integrated into AutoNUMA, TPP, and Colloid on the CXL-based server. All results are normalized to each system’s unmodified baseline (i.e., 2 MB THP without splitting). Across all 17 workloads, TierBPF improves the geomean throughput of AutoNUMA by 12.3%, TPP by 17.7%, and Colloid by 7.4%.
Figure 6. Performance of TierBPF (mTHP splitting) integrated into three tiering systems on the CXL platform, normalized to the performance of vanilla tiering systems.
6.2.1 Analysis by Workload. Graph workloads benefit the most. For example, on AutoNUMA, bc-twitter improves by 30%, cc-web by 24%, cc-twitter by 23%, and bfs-web by 21%. These workloads exhibit highly skewed intra-THP access: only a fraction of subpages within each 2 MB THP are frequently accessed. By splitting THPs and migrating only the hot subfolios, TierBPF avoids wasting scarce fast-tier capacity on cold data. The effect is strongest on Twitter and Web graphs, whose power-law degree distributions concentrate accesses on a small set of high-degree vertices. The uniform random graphs (bc-urand, bfs-urand, etc.) show smaller but consistent gains, demonstrating that even they develop access hotspots when using iterative algorithms. Silo consistently improves across all three systems (13% on AutoNUMA, 19% on TPP, and 20% on Colloid). This improvement stems from YCSB’s Zipfian access distribution, which induces well-defined hot and cold regions within THPs. With this highly concurrent transaction workload, frequent NUMA faults arise and trigger THP splitting. NPB workloads show consistent improvements on AutoNUMA (up to 9%), and larger gains on TPP for CG (29%) and SP (18%). These array-based scientific kernels exhibit intra-THP access skew due to indirect indexing (CG), strided multi-dimensional array traversals (SP), and non-uniform access frequency across data grids (MG).
All GAPBS and NPB benchmarks run with 12 threads; SiloYCSB runs with 8 threads since it crashes with 12 threads. All experiments use cgroup-based resource isolation; each benchmark runs within a dedicated cgroup with controlled CPU and memory limits to ensure reproducibility. Memory contention. To emulate the bandwidth pressure commonly observed in virtualized environments, we run a memory stressor on the slow tier alongside benchmarks. The stressor spawns antagonist threads that issue sequential, non-temporal memory accesses using movntdqa/movntdq instructions over a buffer allocated and pinned in the slow tier. By bypassing the CPU cache hierarchy, these accesses maximize memory bus utilization. The antagonist threads are pinned to dedicated CPU cores separate from the benchmark to avoid direct CPU contention. We vary the number of antagonist threads to generate sustained background traffic of approximately 0, 5, 10, and 20 GB/s.
6.2.2 Analysis by Memory Tiering Software. TPP benefits the most among the three memory tiering systems, with a 17.7% geometric-mean improvement. Unlike AutoNUMA, which infers page hotness from hint fault latency (i.e., the time between unmapping a page and the subsequent access fault), TPP relies on an LRU-based heuristic: a page is deemed hot if it resides on the kernel’s active LRU list [21]. However, for a 2 MB THP comprising 512 subpages, a single access to any subpage causes the entire folio to be referenced and promoted to the active list. Consequently, TPP’s LRU check is rendered ineffective at THP granularity—nearly all THPs are set to be promoted regardless of how little of the folio is actually accessed. This behavior leads TPP to over-promote cold data into the fast tier, populating it with folios that are only sparsely utilized. Splitting restores the discriminative power of the LRU filter. A 64 KB subfolio, for example, contains only 16 subpages, allowing cold subfolios to age into the inactive list and be correctly classified as cold. As a result, TPP experiences the largest performance gains.
NPB [4]
better
better
better
Silo [42]
9
Workload
64KB
Split target (count) 128KB 256KB 512KB
1MB
Hot Subfolios (%)
cc-twitter cc-web bc-twitter bfs-web
4,101 0 1,998 0
21 6,013 186 0
0 0 0 881
9.1% 21.9% 22.5% 53.2%
0 147 1,907 422
0 1,470 1,103 1,170
better
Table 4. Page splitting for four workloads with the largest improvement on AutoNUMA. Split target columns show the number of split events for each mTHP size; Hot Subfolios is the fraction of resulting subfolios classified as hot.
Figure 7. Effects of CXL memory bandwidth contention on splitting benefit (AutoNUMA). The performance is normalized to that of the baselines without THP splitting.
6.3 Why does Splitting Help? Further Analysis from the Perspective of Page Migration
Table 5. Page migration statistics: NoSplit vs. THP Split.
Workload
Page Demotion NoSplit THP Split
Page Promotion Fail NoSplit THP Split
Page Promotion Success NoSplit THP Split
bc-twitter bc-urand bc-web bfs-twitter bfs-urand bfs-web cc-twitter cc-urand cc-web pr-twitter pr-urand pr-web
1.10 M 1.61 M 1.44 M 1.48 M 1.81 M 1.54 M 1.07 M 1.81 M 1.56 M 1.07 M 1.71 M 1.49 M
18.01 M 8.46 M 6.58 M 4.53 M 7.03 M 2.66 M 7.66 M 1.28 M 6.00 M 5.69 M 0.53 M 1.58 M
1.26 M 2.32 M 1.94 M 1.63 M 2.19 M 2.09 M 1.24 M 2.31 M 1.94 M 1.37 M 2.08 M 2.02 M
1.34 M 1.96 M 2.20 M 1.07 M 2.02 M 1.82 M 2.20 M 2.19 M 2.54 M 1.12 M 1.71 M 1.49 M
0.08 M 0.01 M 1.18 M 2.18 M 1.60 M 0.08 M 0.69 M 0.004 M 0.44 M 2.95 M 0.19 M 1.23 M
To understand why mTHP splitting improves performance, we analyze page migration statistics collected from kernel vmstat counters on AutoNUMA. Table 5 compares migration with and without splitting across the 12 GAP workloads. The most pronounced change is the reduction in migration failures. Without splitting, the kernel attempts to migrate entire 2 MB THPs; when the destination node cannot provide a contiguous 2 MB free region, the migration fails. For bc-twitter, 18 million migration attempts fail without splitting, whereas with splitting only 80 thousand fail—a 224× reduction. Similar trends hold across all workloads: bc-urand drops from 8.5 M to 11 K failures (777×), bfs-web from 2.7 M to 83 K (32×), and cc-twitter from 7.7 M to 694 K (11×). The underlying reason is clear. Splitting a 2 MB THP into, for example, 64 KB mTHP subfolios reduces migration’s allocation requirement to just 64 KB of contiguous free space on the destination node—an order of magnitude easier constraint to satisfy. As a result, far fewer migrations fail, allowing AutoNUMA to effectively relocate hot data. This sharp reduction in failures translates directly into more successful migrations. For bc-twitter, the number of successful migrations nearly doubles, increasing from 1.26 M to 2.44 M (1.9×). For cc-web, successful migrations grow from 1.94 M to 4.28 M (2.2×), and for cc-twitter from 1.24 M to 3.80 M (3.1×). As a result, a larger fraction of hot data is successfully placed in the fast tier, while cold subpages remain in the slow tier. Across all 12 workloads, splitting increases the number of successful migrations by 1.0×–3.1× while reducing migration failures by up to 777×. In summary, splitting improves performance through two reinforcing effects: it lowers the bar for migration success by reducing allocation size requirements, and it improves migration precision by ensuring that only hot subfolios are moved. Together, these effects enable AutoNUMA to migrate hot data more reliably and more selectively.
2.44 M 3.34 M 3.79 M 1.71 M 3.25 M 2.77 M 3.80 M 2.91 M 4.28 M 1.80 M 2.17 M 1.92 M
Colloid shows the smallest improvement (7.4% geometric mean) because its access-latency–balanced promotion policy already selects pages more judiciously. Nevertheless, TierBPF still yields meaningful gains: silo_ycsb improves by 20%, bcweb by 20%, and cc-web by 18%. These results highlight that even a well-tuned page selection policy cannot fully avoid bandwidth waste when migration is constrained to coarse-grained 2 MB pages—moving an entire page incurs unnecessary data movement when only a fraction is hot. 6.2.3 How Does TierBPF Split Pages? Table 4 reports the split-size distribution and the fraction of hot subfolios after splitting for four workloads that benefit most from AutoNUMA. Two key observations emerge. First, no single mTHP size fits all workloads. For instance, bc-twitter distributes its splits across four mTHP sizes (64 KB – 512 KB), reflecting that the THP’s hot region extent varies across execution phases. This underscores that a static, onesize-fits-all splitting granularity would leave substantial performance untapped; the dynamic size selection is essential. Second, hot subfolios are typically a minority. Across the workloads, the fraction of hot subfolios ranges from just 9.1% (cc-twitter) to 53.2% (bfs-web). Without splitting, the kernel would promote entire 2 MB THPs, wasting up to 91% of migration bandwidth and fast-tier capacity on cold data.
6.4 Evaluation with Memory Bandwidth Contention TierBPF activates THP splitting only when slow-tier bandwidth becomes contended (§5.2). To validate this design choice and motivate the contention level used in our main 10
6.5
better
evaluation (§6.2), we run the 12 GAP workloads on AutoNUMA under four levels of CXL memory bandwidth pressure: 0, 5, 10, and 20 GB/s of sustained background traffic, generated by a memory-streaming process on the CXL device. Figure 7 presents the results. Under no contention (0 GB/s), splitting provides little benefit: the geometric-mean improvement is only 3.4%, and just 7 of the 12 workloads see speedups. When bandwidth is abundant, migrating cold subpages alongside hot ones within a 2 MB THP incurs negligible cost—the additional data does not displace useful traffic. In this regime, the increased TLB pressure introduced by splitting often offsets any savings in migration bandwidth. At 5 GB/s of memory contention, splitting still fails to consistently pay off. The geometric-mean improvement is effectively zero (-0.1%), and only 6 of the 12 workloads improve. At this moderate contention level, the bandwidth savings from less cold-data migration are not yet sufficient to reliably outweigh TLB overhead introduced by smaller page sizes. At 10 GB/s of memory contention, splitting begins to provide measurable benefits, but the gains remain uneven. The geometric-mean improvement rises to 7.0%; however, only 9 of the 12 workloads improve, and most see only single-digit speedups. The benefits are concentrated among workloads with highly skewed access patterns—for example, bc-twitter (18%), bc-web (20%), and cc-web (31%)—while workloads with more uniform access patterns remain flat or even regress. At this level of contention, the bandwidth savings enabled by splitting are still insufficient to make it broadly effective. At 20 GB/s of memory contention—the highest contention level—splitting delivers clear and consistent improvements. The geometric mean increases to 14.1%, and all 12 workloads improve, including bc-twitter (30%), cc-web (24%), cc-twitter (23%), and bfs-web (21%). Under this level of pressure, migrating a 2 MB page that is only 20% hot wastes 1.6 MB of scarce bandwidth; splitting eliminates this waste, and the resulting bandwidth savings decisively outweigh the TLB penalty. We therefore use this contention level in our end-to-end evaluation (§6.2), as it reflects a realistic scenario in which multiple memory-intensive workloads share a CXL device and bandwidth becomes the primary performance bottleneck. Together, these results validate TierBPF ’s contention-aware activation policy. Under low contention, TierBPF preserves 2 MB THPs to maximize TLB coverage. As contention increases and bandwidth efficiency becomes more critical, TierBPF selectively enables splitting, activating it only when its bandwidth savings outweigh the associated TLB costs.
Figure 8. Performance of TierBPF and MEMTIS on the PMEM platform, normalized to that of AutoNUMA. Table 6. L2 DTLB misses per 1K instructions on the PMEM platform. MEMTIS’s 4 KB splitting incurs the highest TLB miss rates; TierBPF’s mTHP splitting reduces TLB overhead. Workload MG.D LU.D
AutoNUMA
TierBPF
MEMTIS
0.12 2.22
0.18 2.90
0.21 3.26
Figure 8 presents the results. FlexTier-enhanced AutoNUMA outperforms the AutoNUMA baseline by 7.1% in geometric mean across 15 workloads. The largest gains occur for the graph workloads with highly skewed access patterns: bc-twitter improves by 36%, pr-twitter by 26%, cc-twitter by 14%, bfs-twitter by 11%, and cc-web by 10%. The performance trends on PMEM closely mirror those observed with CXL, confirming that THP splitting addresses a fundamental limitation of THP-based migration—namely, the mismatch between the coarse 2 MB migration granularity and fine-grained, sub-page access skew. Comparison with MEMTIS. Figure 8 also compares against MEMTIS, a state-of-the-art system for huge page-aware tiered memory. MEMTIS relies on a background thread to split THPs, but supports only two page sizes: the original 2 MB THP or 4 KB base pages. It cannot generate intermediate mTHP sizes (e.g., 64 KB or 256 KB). This all-or-nothing splitting granularity incurs excessive loss of TLB coverage once splitting is applied. The impact is most evident on NPB workloads, which traverse large arrays and are particularly sensitive to TLB coverage. Relative to the AutoNUMA baseline, MEMTIS degrades MG.D by 7% and LU.D by 8%. In contrast, TierBPF improves MG.D by 11% while maintaining LU.D’s performance, as mTHP splitting preserves sufficient TLB coverage to avoid the penalties incurred by aggressive 4 KB splitting. TLB misses. Table 6 confirms the above observations with the measurement of L2 DTLB misses. MEMTIS consistently causes the highest TLB miss rate across the workloads. For LU.D, MEMTIS’s L2 DTLB misses reach 3.26 per 1K instructions—12% higher than TierBPF (2.90) and 47% higher than AutoNUMA (2.22). For MG.D, the same trend holds: MEMTIS’s L2 DTLB misses (0.21) exceed TierBPF ’s (0.18) by 17%. For workloads with large working sets, this TLB penalty outweighs the migration-precision gains.
Evaluation on PMEM
To demonstrate that TierBPF’s design generalizes beyond CXL, we evaluate mTHP splitting on the PMEM platform. We only enable the THP splitting policy, because maintaining mixed read and write traffic does not benefit performance given PMEM’s half-duplex DDR interface. 11
better
better
better
better
Figure 9. Performance under 20 GB/s CXL contention, normalized to (a) AutoNUMA and (b) TPP baselines.
Figure 10. Performance under 10 GB/s CXL contention, normalized to (a) AutoNUMA and (b) TPP baselines.
Overall, TierBPF outperforms MEMTIS on 11 of 15 workloads. These results underscore that splitting granularity matters: while splitting to 4 KB base pages—as MEMTIS does—recovers migration precision, it does so at the cost of TLB efficiency. In contrast, the mTHP splitting in TierBPF strikes a better balance, preserving TLB coverage while still enabling fine-grained migration. 6.6
When applied alone (without Split), BalanceTraffic yields a modest improvement. On TPP in particular, only 7 workloads see improvements while 10 workloads degrade, and the overall geomean gain is only 3.2%. Moderate contention (10 GB/s). The results improve: BalanceTraffic achieves a geomean improvement of 5.2% on AutoNUMA and 4.7% on TPP, with 13 of 17 workloads improved (Figure 10). With more bandwidth headroom, the fullduplex rebalancing has room to take effect without starving hot pages of fast-tier placement. As such, selectively deferring promotions that would add traffic to the non-bottleneck CXL channel shows benefits.
Evaluation with Traffic Balancing
This section evaluates the duplex-aware migration admission policy (§5.3), which exploits CXL’s full-duplex channels by selectively admitting or deferring page migrations based on each page’s read/write classification and the instantaneous CXL traffic pattern (Table 2). We refer to this policy as BalanceTraffic. We evaluate BalanceTraffic under two memory-contention regimes: high contention at 20 GB/s and moderate contention at 10 GB/s. High contention (20 GB/s). As shown in Figure 9, BalanceTraffic does not complement splitting under high contention. On AutoNUMA, enabling Split alone improves geomean performance by 12.3%, while Split+BalanceTraffic reduces the gain to 6.6%. The effect is more pronounced on TPP: Split alone achieves 17.7% geomean throughput improvement, whereas Split+BalanceTraffic yields only 1.8%. This is due to the following: Under 20 GB/s contention, aggregate bandwidth is already the dominant bottleneck, and performance is primarily determined by whether hot pages are placed in the fast tier. In this regime, rebalancing read and write traffic across CXL’s full-duplex channels provides little additional benefit—the limiting factor is total bandwidth, not directional imbalance. Moreover, BalanceTraffic’s selective admission policy can be counterproductive: by deferring migrations of hot pages whose R/W classification does not match the dominant traffic direction (Table 2), it prevents them from reaching the fast tier, where they are most needed.
7
Related Work
Memory tiering systems. AutoNUMA [7] is the default system in Linux to manage tiered memory based on access recency. TPP [21] extends AutoNUMA with proactive page demotion and LRU active-list-based hot page promotion. HeMem [31], MEMTIS [12], MTM [34], and FlexMem [50] migrate pages based on access frequency. Memstrata [57] utilizes Intel Flat Memory mode to build cache-level data migration. NOMAD [48] builds non-exclusive memory tiering, aiming to mitigate memory thrashing when fast memory is under pressure. Colloid [43] decides page migration by balancing access latencies across memory tiers. HybridTier [39] tracks long-term access frequency and short-term access momentum simultaneously to adapt to shifting hotness distributions. AOL [17] considers the impact of memory-level parallelism to avoid unnecessary page migration. TierBPF is different from those efforts in terms of page size selection and memory traffic distribution. There are application semantics-guided tiering systems, including Unimem [47], Xmem [8], memkind [1], HM-ANN [35], and WarpX-HM [33]. Some efforts [2, 23, 30] enable memory 12
tiering using specialized hardware. In contrast, TierBPF is application-transparent and requires no hardware changes. CXL memory can increase memory scalability and simplify software stacks [37]. From the perspective of CXL applications, Wang et al. [44] and Xu et al. [49] leverage CXL memory sharing for point-to-point and collective communications; Wang et al. [45], Liu et al. [16], Tang et al. [41], Mao et al. [20] and Ji et al. [40] characterize the performance of real CXL hardware; CXL memory is also used for tensor offloading for AI workloads [10, 45, 53] and database [54]. From the perspective of CXL system software, TPP [21] refines system software for page migration to accommodate CXL. Pond [14] introduces a CXL memory pooling system to hold cold data. Apta [28] is designed for function-as-aservice over CXL. HydraRPC [19] utilizes CXL HDM for data transmission. ReScure [38] focuses on reliable and secure CXL memory. There are also recent efforts to support CXL memory pooling/sharing across hosts through innovation in memory allocation [27], formal method-based checking [9], programming model [3], and CXL bridges [13]. Lupin [58] is designed to tolerate partial failures for distributed applications using a shared CXL pod for replication. TierBPF works on CXL memory and provides new optimizations considering page size and memory duplex mode. eBPF for memory optimization. eBPF is used to customize memory management policies, such as huge page placement, page fault handling, and page table designs [26, 60]. P2Cache [11], cache_ext [59], PageFlex [56], and FetchBPF [6] use eBPF to customize page cache, prefetching, and swapping policies. TierBPF leverages eBPF to provide policy flexibility for new scenarios of memory management.
8
United States). IEEE Computer Society Press, Los Alamitos, CA, USA, 386–393. [5] Scott Beamer, Krste Asanović, and David Patterson. 2015. The GAP Benchmark Suite. In arXiv preprint arXiv:1508.03619. [6] Xuechun Cao, Shaurya Patel, Soo Yee Lim, Xueyuan Han, and Thomas Pasquier. 2024. {FetchBPF}: Customizable prefetching policies in linux with {eBPF}. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). 369–378. [7] J. Corbet. [n. d.]. AutoNUMA: the Other Approach to NUMA Scheduling. http://lwn.net/Articles/488709. [8] Subramanya R Dulloor, Amitabha Roy, Zheguang Zhao, Narayanan Sundaram, Nadathur Satish, Rajesh Sankaran, Jeff Jackson, and Karsten Schwan. 2016. Data tiering in heterogeneous memory systems. In Proceedings of the Eleventh European Conference on Computer Systems. ACM, 15. [9] Simon Guo, Conan Truong, and Brian Demsky. 2026. CXLMC: Model Checking CXL Shared Memory Programs. In International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. [10] Hyungyo Kim, Qirong Xia, Jinghan Huang, Nachuan Wang, Younjoo Lee, Jung Ho Ahn, Wajdi K Feghali, Ren Wang, and Nam Sung Kim. 2026. LiLo: Harnessing the on-Chip Accelerators in Intel CPUs for Compressed LLM Inference Acceleration. In IEEE International Symposium on High Performance Computer Architecture (HPCA). [11] Dusol Lee, Inhyuk Choi, Chanyoung Lee, Hyungsoo Jung, and Jihong Kim. 2026. P2Cache: Enhancing data-centric applications via application-guided management of OS page caches. ACM Transactions on Storage 22, 1 (2026), 1–33. [12] Taehyung Lee, Sumit Kumar Monga, Changwoo Min, and Young Ik Eom. 2023. MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size Determination. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP). 17–34. [13] Anatole Lefort, Julian Pritzi, Nicolò Carpentieri, David Schall, Simon Dittrich, Soham Chakraborty, Nicolai Oswald, and Pramod Bhatotia. 2026. vCXLGen: Automated Synthesis and Verification of CXL Bridges for Heterogeneous Architectures. In International Conference on Architectural Support for Programming Languages and Operating Systems. [14] Huaicheng Li, Daniel S. Berger, Lisa Hsu, Daniel Ernst, Pantea Zardoshti, Stanko Novakovic, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, and Ricardo Bianchini. 2023. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). [15] Shaobo Li, Yirui (Eric) Zhou, Hao Ren, and Jian Huang. 2025. ByteFS: System Support for (CXL-based) Memory-Semantic Solid-State Drives. In International Conference on Architectural Support for Programming Languages and Operating Systems. [16] Jinshu Liu, Hamid Hadian, Yuyue Wang, Daniel S. Berger, Marie Nguyen, Xun Jian, Sam H. Noh, and Huaicheng Li. 2025. Systematic CXL Memory Characterization and Performance Analysis at Scale. In International Conference on Architectural Support for Programming Languages and Operating Systems. [17] Jinshu Liu, Hamid Hadian, Hanchen Xu, and Huaicheng Li. 2025. Tiered Memory Management Beyond Hotness. In Proceedings of USENIX Conference on Operating Systems Design and Implementation (OSDI). [18] Daniel Lustig, Abhishek Bhattacharjee, and Margaret Martonosi. 2013. TLB Improvements for Chip Multiprocessors. ACM Transactions on Architecture and Code Optimization 10, 2 (2013), 1–31. [19] Teng Ma, Zheng Liu, Chengkun Wei, Jialiang Huang, Youwei Zhuo, Haoyu Li, Ning Zhang, Yijin Guan, Dimin Niu, Mingxing Zhang, et al. 2024. {HydraRPC}:{RPC} in the {CXL} Era. In USENIX Annual Technical Conference.
Conclusion
Memory tiering architectures are rapidly diversifying, which calls for consideration of page-migration interactions with other system components and memory architecture details. This paper discusses how to improve the memory tiering system to accommodate emerging changes in the memory architecture, especially from the perspectives of page size and memory access distribution. TierBPF enables sufficient flexibility to develop novel policies in diverse environments, such as more than two memory tiers or CXL-SSD, which could be implemented in the future.
References [1] [n. d.]. pmem.io. memkind. https://pmem.io/memkind/. [2] Julian T. Angeles, Mark Hildebrand, Venkatesh Akella, and Jason Lowe-Power. 2021. Investigating Hardware Caches for Terabyte-scale NVDIMMs. In Annual Non-Volatile Memories Workshop. [3] Gal Assa, Moritz Lumme, Lucas Bürgi, Michal Friedman, and Ori Lahav. 2026. A Programming Model for Disaggregated Memory over CXL. In International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. [4] D. H. Bailey, L. Dagum, E. Barszcz, and H. D. Simon. 1992. NAS parallel benchmark results. In Supercomputing ’92: Proceedings of the 1992 ACM/IEEE conference on Supercomputing (Minneapolis, Minnesota, 13
[20] Shunyu Mao, Jiajun Luo, Yixin Li, Jiapeng Zhou, Weidong Zhang, Zheng Liu, Teng Ma, and Shuwen Deng. 2024. CXL-Interference: Analysis and Characterization in Modern Computer Systems. arXiv:2411.18308 [cs.AR] https://arxiv.org/abs/2411.18308 [21] Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petrov, Prakash Chalapathi, Mosharaf Chaudhry, and Russ Cranney. 2023. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 742–755. [22] John D. McCalpin. 2012. Measuring TLB Miss Handling Cost in x86-64. Technical report and Intel performance counter analysis. [23] Mitesh R Meswani, Sergey Blagodurov, David Roberts, John Slice, Mike Ignatowski, and Gabriel H Loh. 2015. Heterogeneous memory architectures: A HW/SW approach for mixing die-stacked and offpackage memories. In International Symposium on High Performance Computer Architecture (HPCA). [24] Meta and Google. 2024. sched_ext: Extensible Scheduler Class with BPF. https://docs.kernel.org/scheduler/sched-ext.html. Linux kernel feature, merged in Linux 6.12. [25] Konstantinos Mores et al. 2024. eBPF-mm: Userspace-guided Memory Management in Linux with eBPF. arXiv preprint arXiv:2404.xxxxx. [26] Konstantinos Mores, Stratos Psomadakis, and Georgios Goumas. 2024. eBPF-mm: Userspace-guided memory management in Linux with eBPF. arXiv preprint arXiv:2409.11220 (2024). [27] Newton Ni, Yan Sun, Zhiting Zhu, and Emmett Witchel. 2026. Cxlalloc: Safe and Efficient Memory Allocation for a CXL Pod. In International Conference on Architectural Support for Programming Languages and Operating Systems. [28] Adarsh Patil, Vijay Nagarajan, Nikos Nikoleris, and Nicolai Oswald. ¯ 2023. Apta: Fault-tolerant object-granular CXL disaggregated memory for accelerating FaaS. In 2023 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). [29] E. C. Pielou. 1966. The Measurement of Diversity in Different Types of Biological Collections. Journal of Theoretical Biology 13 (1966), 131–144. [30] Luiz Ramos, Eugene Gorbatov, and Ricardo Bianchini. 2011. Page Placement in Hybrid Memory Systems. In International Conference on Supercomputing. [31] Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Erez, and Simon Peter. 2021. HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles. [32] Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Zhong, Tapan Palit, Justine Sherry, Greg Epping, Aasheesh Phan, and Hakim Weatherspoon. 2021. HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP). 392–407. [33] Jie Ren, Jiaolin Luo, Ivy Peng, Kai Wu, and Dong Li. 2021. Optimizing Large-Scale Plasma Simulations on Persistent Memory-based Heterogeneous Memory with Effective Data Placement Across Memory Hierarchy. In International Conference on Supercomputing (ICS). [34] Jie Ren, Dong Xu, Junhee Ryu, Kwangsik Shin, Daewoo Kim, and Dong Li. 2024. MTM: Rethinking Memory Profiling and Migration for MultiTiered Large Memory Systems. In European Conference on Computer Systems. [35] Jie Ren, Minjia Zhang, and Dong Li. 2020. HM-ANN: Efficient BillionPoint Nearest Neighbor Search on Heterogeneous Memory. In Conference on Neural Information Processing Systems (NeurIPS). [36] Ryan Roberts and Linux Kernel Community. 2024. Multi-size THP (mTHP). https://docs.kernel.org/admin-guide/mm/transhuge.html. Linux kernel feature, merged in Linux 6.8. [37] Debendra Das Sharma, Robert Blankenship, and Daniel S. Berger. 2023. An Introduction to the Compute Express Link (CXL) Interconnect.
arXiv:2306.11227 [cs.AR] [38] Chihun Song, Austin Antony Cruz, Michael Jaemin Kim, Minbok Wi, Gaohan Ye, Kyungsan Kim, Sangyeol Lee, Jung Ho Ahn, and Nam Sung Kim. 2026. ReScue: Reliable and Secure CXL Memory. In IEEE International Symposium on High Performance Computer Architecture (HPCA). [39] Kevin Song, Jiacheng Yang, Zixuan Wang, Jishen Zhao, Sihang Liu, and Gennady Pekhimenko. 2025. HybridTier: an Adaptive and Lightweight CXL-Memory Tiering System. In International Conference on Architectural Support for Programming Languages and Operating Systems. [40] Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim. 2023. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. In IEEE/ACM International Symposium on Microarchitecture. [41] Yupeng Tang, Ping Zhou, Wenhui Zhang, Henry Hu, Qirui Yang, Hao Xiang, Tongping Liu, Jiaxin Shan, Ruoyun Huang, Cheng Zhao, Cheng Chen, Hui Zhang, Fei Liu, Shuai Zhang, Xiaoning Ding, and Jianjun Chen. 2024. Exploring Performance and Cost Optimization with ASICBased CXL Memory. In Proceedings of the European Conference on Computer Systems. [42] Stephen Tu, Wenting Zheng, Eddie Kohler, Barbara Liskov, and Samuel Madden. 2013. Speedy Transactions in Multicore In-Memory Databases. In Proceedings of the 24th ACM Symposium on Operating Systems Principles (SOSP). 18–32. [43] Midhul Vuppalapati, Rodrigo Fonseca, and Hakim Weatherspoon. 2024. Colloid: A Latency-Aware Tiered Memory Management System. In Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP). [44] Xi Wang, Bin Ma, Jongryool Kim, Byungil Koh, Hoshik Kim, and Dong Li. 2025. cMPI: Using CXL Memory Sharing for MPI One-Sided and Two-Sided Inter-Node Communications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. [45] Xi (Sherry) Wang, Jie Liu, Jianbo Wu, Shuangyan Yang, Jie Ren, Bhanu Shankar, and Dong Li. 2025. Performance Characterization of CXL Memory and Its Use Cases. In International Parallel and Distributed Processing Symposium. [46] Peter Weisberg and Yitzhak Wiseman. 2009. Using 4KB Page Size for Virtual Memory Is Obsolete. In Proceedings of the 8th IEEE International Symposium on Network Computing and Applications (NCA). 262–265. [47] K. Wu, Y. Huang, and D. Li. 2017. Unimem: Runtime Data Management on Non-Volatile Memory-based Heterogeneous Main Memory. In International Conference for High Performance Computing, Networking, Storage and Analysis. [48] Lingfeng Xiang, Zhen Lin, Arkaprava Basu, and Rong Lv. 2024. Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). [49] Dong Xu, Han Meng, Xinyu Chen, Dengcheng Zhu, Wei Tang, Fei Liu, Liguang Xie, Wu Xiang, Rui Shi, Yue Li, Henry Hu, Hui Zhang, Jianping Jiang, and Dong Li. 2026. CCCL: Node-Spanning GPU Collectives with CXL Memory Pooling. arXiv:2602.22457 [cs.DC] https://arxiv.org/abs/ 2602.22457 [50] Dong Xu, Junhee Ryu, Jinho Baek, Kwangsik Shin, Pengfei Su, and Dong Li. 2024. FlexMem: Adaptive Page Profiling and Migration for Tiered Memory. In 30th USENIX Annual Technical Conference (ATC). [51] Zi Yan, Daniel Lustig, David Nellans, and Abhishek Bhattacharjee. 2019. Nimble Page Management for Tiered Memory Systems. In Proceedings of the 24th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 331–345. [52] Shao-Peng Yang, Minjae Kim, Sanghyun Nam, Juhyung Park, Jin yong Choi, Eyee Hyun Nam, Eunji Lee, Sungjin Lee, and Bryan S. Kim. 2023. Overcoming the Memory Wall with CXL-Enabled SSDs. In USENIX Annual Technical Conference. 14
Conference (USENIX ATC 25). 291–306. [57] Yuhong Zhong, Daniel S. Berger, Carl Waldspurger, Ryan Wee, Ishwar Agarwal, Rajat Agarwal, Frank Hady, Karthik Kumar, Mark D. Hill, Mosharaf Chowdhury, and Asaf Cidon. 2024. Managing memory tiers with CXL in virtualized environments. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (OSDI). [58] Zhiting Zhu, Newton Ni, Yibo Huang, Yan Sun, Zhipeng Jia, Nam Sung Kim, and Emmett Witchel. 2024. Lupin: Tolerating partial failures in a cxl pod. In Proceedings of the 2nd Workshop on Disruptive Memory Systems. [59] Tal Zussman et al. 2025. cache_ext: Customizing the Page Cache with eBPF. In Proceedings of the 30th ACM Symposium on Operating Systems Principles (SOSP). [60] Tal Zussman, Teng Jiang, and Asaf Cidon. 2024. Custom page fault handling with ebpf. In Proceedings of the ACM SIGCOMM 2024 Workshop on eBPF and Kernel Extensions.
[53] Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yicong Zhu, Yuqi Zhou, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, et al. 2025. Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management. arXiv preprint arXiv:2511.20172 (2025). [54] Xinjun Yang, Yingqiang Zhang, Hao Chen, Feifei Li, Gerry Fan, Yang Kong, Bo Wang, Jing Fang, Yuhui Wang, Tao Huang, Wenpu Hu, Jim Kao, and Jianping Jiang. 2025. Unlocking the Potential of CXL for Disaggregated Memory in Cloud-Native Databases. In Companion of the 2025 International Conference on Management of Data. [55] Yiwei Yang, Yusheng Zheng, Yiqi Chen, Zheng Liang, Kexin Chu, Zhe Zhou, Andi Quinn, and Wei Zhang. 2025. CXLAimPod: CXL Memory is all you need in AI era. arXiv:2508.15980 [cs.OS] https: //arxiv.org/abs/2508.15980 [56] Anil Yelam, Kan Wu, Zhiyuan Guo, Suli Yang, Rajath Shashidhara, Wei Xu, Stanko Novaković, Alex C Snoeren, and Kimberly Keeton. 2025. {PageFlex}: Flexible and Efficient User-space Delegation of Linux Paging Policies with {eBPF}. In 2025 USENIX Annual Technical
15