Virtual-Memory Assisted Buffer Management In Tiered Memory Yeasir Rayhan and Walid G. Aref Purdue University, West Lafayette, IN, USA {yrayhan,aref}@purdue.edu
arXiv:2603.03271v1 [cs.DB] 3 Mar 2026
Abstract Tiered memory architectures have gained significant traction in the database community in recent years. In these architectures, the on-chip DRAM of the host processor is typically referred to as local memory, and forms the primary tier. Additional byte-addressable, cache-coherent memory resources, collectively referred to as remote memory (RMem, for short), form one or more secondary tiers. RMem is slower than local DRAM but faster than disk, e.g., NUMA memory located on a remote socket, chiplet-attached memory, and memory attached via high-performance interconnect protocols, e.g., RDMA and CXL. In this paper, we discuss how traditional two-tier (DRAM-Disk) virtual-memory assisted Buffer Management techniques generalize to an 𝑛-tier setting (DRAM-RMem-Disk). We present vmcache𝑛 , an 𝑛-tier virtual-memory-assisted buffer pool that leverages the virtual memory subsystem and operating system calls to migrate pages across memory tiers. In this setup, page migration can become a bottleneck. To address this limitation, we introduce the move_pages2 system call that provides vmcache𝑛 with fine-grained control over the page migration process. Experiments show that vmcache𝑛 can achieve up to 4× higher query throughput over vmcache for TPC-C workloads.
1
Introduction
Most DBMSs implement a two-Tier (DRAM-Disk) Buffer Pool, where database pages1 are cached into DRAM for efficient access. Traditionally, DBMSs implement a Hash Table to index cached pages by their page identifier, i.e., PID. A PID of a cached page maps to a virtual memory address of the corresponding page in memory. This Hash Table-based design introduces an additional indirection as each page access requires a Hash Table Lookup. This can incur significant overhead even when the workload is fully DRAM-resident and large, as the workload becomes CPU-bound due to hash table bloat, pointer chasing, latch contention, and reduced cache locality [15, 39]. To mitigate the overhead of Hash Table-based indirection, several alternatives have been proposed, e.g., pointer swizzling [12, 24, 26], and virtual-memory assisted buffer management [22, 29], LIPAH [27, 28]. In this paper, we focus on virtual-memory assisted Buffer Pools, i.e., vmcache. We examine how this design generalizes beyond the traditional two-Tier setting and extends to an 𝑛-tier storage architecture, where the first n-1 tiers are memory-based, e.g., DRAM, Persistent Memory, CXL Memory, and the final tier is disk-based, e.g., NVMe SSD, HDD. Despite the steady decline in DRAM prices until 2020 [9, 18], recent market dynamics have led to a renewed spike in DRAM costs by up to 90% [6]. Hence, memory stranding, i.e., underutilized memory across servers has become an increasingly critical issue. 1 Throughout the rest of this paper, the term page refers to a database page unless
stated otherwise.
Recent research, e.g., [1, 3, 5, 13, 14, 16, 21, 25, 31, 33, 37], advocates for memory disaggregation and multi-tiered memory architectures to improve memory utilization and to reduce cost per effective gigabyte (GB) of memory. While early work demonstrates the feasibility of tiered memory architectures in buffer pool designs [14, 29, 32, 38], the design space of virtual-memory assisted buffer management in a general 𝑛-tier setting remains largely unexplored. Given its success in two-tier designs, a natural next step is to understand their behavior and design trade-offs in an 𝑛-tier setting. In this paper, we present vmcache𝑛 , an 𝑛-tier virtual-memory assisted Buffer Pool built on top of vmcache. vmcache𝑛 inherits the core design principle of vmcache, and enforces the following invariant: The virtual address associated with a page remains fixed throughout its lifetime. Similar to vmcache, vmcache𝑛 delegates PID translation to the OS Page Table, and retains control over page promotion and page eviction from disk to memory tiers through libaio interface [8]. The inclusion of additional memory tiers imposes an additional constraint on vmcache𝑛 , thereby extending the prior invariant: the physical frame backing a database page in memory must support dynamic mapping across different memory-resident tiers over time while ensuring that the corresponding virtual address of the page remains fixed throughout its lifetime. In the remainder of this paper, we focus our discussion to 3-tier virtual-memory assisted buffer pools, where Tier-0, Tier-1, and Tier-2 correspond to a DRAM, a remote memory, and an NVMe disk, respectively. The techniques presented naturally generalize to an 𝑛-tier setting, where the intermediate layers may consist of multiple remote memory tiers. vmcache𝑛 retains control over page promotion and page demotion across memory tiers, and leverages page migration system calls, e.g., mbind and move_pages to migrate pages between memory tiers. These system calls preserve a page’s virtual address while updating the physical frame that backs it. The syscalls further install the virtual-to-physical mapping in the OS’s Page Table, and thus making the physical location of the page frame transparent to the buffer manager without requiring any additional Hash Table Lookup. On modern hardware with fast memory tiers and NVMe SSDs, the overheads of the OS’s memory management unit (MMU) and page migration system calls can become a performance bottleneck. Hence, we implement batching to amortize the migration cost of multiple pages. To further improve performance, we build on top of the native kernel system call move_pages and propose move_pages2. The move_pages2 custom system call enables vmcache𝑛 to retain finegrained control over the page migration process with the following two knobs: migration_mode and nr_max_batched_migration. The migration_mode knob lets vmcache𝑛 choose the strictness of the migration policy. The nr_max_batched_migration knob lets vmcache𝑛 choose the maximum number of pages that can be batched together for migration.
Conference’17, July 2017, Washington, DC, USA
Table 1: Conceptual comparison of buffer management techniques in tiered memory. mmap
tradi. Ptr swiz. [14, 38] [32] PID-transl. page tbl. hash tbl. invasive tier-trk. page tbl. hash tbl. hash tbl. control OS DBMS DBMS mig-unit. OS page cacheline cacheline mig-gran. 𝑛 n n data-mov. autonuma memcpy memcpy different different vir addr. same page dup. no yes yes easy hard hard var. size graphs yes yes no implem. med easy hard ∗ vmcache𝑛 also supports mbind and move_pages
2
Hyrise [29] page tbl. page tbl. DBMS OS page 1 mbind same no easy yes easy
vmcache𝑛 Sec. 2 page tbl. page tbl. DBMS OS page n move_pages2∗ same no easy yes easy
Virtual-Memory Assisted Buffer Management in Tiered Memory
By design, in vmcache [22], the virtual address associated with a page remains fixed throughout its lifetime. A page may be evicted from DRAM and later re-cached at the same virtual memory address. This stability allows vmcache to safely delegate PID translation to the OS Page Table. If the page is DRAM-resident, the OS Page Table resolves the access to the corresponding physical frame. Otherwise, the OS raises a page fault, and vmcache loads the page from disk. This design eliminates the additional Hash Table lookup cost, as page accesses are resolved through direct virtual-memory translation. While this is conceptually straightforward in a two-tier setting, the situation becomes more nuanced in an 𝑛-tier setting, where multiple memory tiers are involved. vmcache𝑛 : An 𝑛-tier Virtual-Memory Assisted Buffer Pool. vmcache𝑛 extends vmcache’s two-tier design to an 𝑛-tier setting. Unlike vmcache that maintains a single DRAM cache, vmcache𝑛 maintains separate caches for each memory tier. The key distinction is that vmcache𝑛 ’s 𝑛-tier design requires the physical frame backing a page to be dynamically remapped across multiple memoryresident tiers without changing its virtual address. Next, we outline the design principles of vmcache𝑛 . 1. The fundamental invariant of an 𝑛-tier virtual-memory assisted Buffer Pool is a stable virtual addressing along with dynamically changeable physical frame mappings across the memory tiers. 2. Page migration can only update the physical frame backing a PID while preserving the virtual memory address of the PID. 3. At any point in time, a page can reside in exactly one memory tier. Pages are not replicated across tiers due to the use of a single, fixed virtual address per PID. 4. Due to the invariant of stable virtual addressing, only memory tiers configured in system-RAM mode are compatible with virtual-memory assisted buffer management [3, 33]. vmcache𝑛 is incompatible with alternative memory-tier configurations, e.g., the Device Direct Access (DAX) mode [17]. In DAX mode, the memory tier is exposed as a character device, and the virtual memory mappings created from the memory tier are permanently backed by physical frames from the corresponding device. This prevents remapping the virtual address to a different physical frame, i.e., the fundamental invariant of vmcache𝑛 .
vmcache𝑛 vs. Other Buffer Management Techniques. Table 1 presents a conceptual comparison of alternative buffer pool designs with vmcache𝑛 . vmcache𝑛 is built on top of vmcache. It inherits the core advantages of vmcache including support for variablesized pages, graph workloads, and robust performance for both in-memory and out-of-memory workloads. In an 𝑛-tier setting, several new design dimensions arise including tracking the physical location of pages (tier-trk), defining the migration unit (mig-unit), determining migration granularity (mig-gran), specifying the data movement interface (data-mov), and handling page duplication (page dup). In traditional designs [14, 38] and in pointer-swizzling approaches [32], the hash table explicitly tracks the physical location of each page across memory tiers. These designs support data movement at cache-line granularity using memcpy leading to page duplication across tiers. These designs may assign different virtual addresses to the same logical page over its lifetime. Hyrise adopts a virtual-memory assisted buffer pool design that relies on mbind to migrate pages individually, which does not scale in modern hardware. vmcache𝑛 performs batched migration of 𝑛 pages and leverages move_pages2 to enable efficient page migration. 2.1. Page Table Manipulation. Following vmcache, at startup, vmcache𝑛 reserves a virtual memory address space equaling the size of the 𝑛-th Tier, i.e., disk. This reservation establishes a stable virtual address for every database page throughout its lifetime. To cache a page from disk into a memory-resident tier, vmcache𝑛 uses the pread system call. However, in an 𝑛-tier setting, vmcache𝑛 must support more flexible page placement than the traditional two-tier case, where pages are cached exclusively into DRAM. Precisely, vmcache𝑛 must be able to cache pages from disk into any memoryresident tier, not only DRAM. In addition, vmcache𝑛 must support transparent migration of pages across memory-resident tiers while preserving the page’s virtual address. Adding pages from disk to a target memory-tier cache. Following [29], vmcache𝑛 uses the mbind system call to add a page to a target memory-tier cache. For example, we want to cache Page P3 in Tier-1. mbind sets the physical memory allocation policy for Page P3 to MPOL_BIND. MPOL_BIND ensures that the physical frame backing P3 is allocated from the memory tier specified by the tgt_tier bitmask, i.e., 0b0010 (Tier-1) in the example. In addition, vmcache𝑛 sets the MPOL_MF_MOVE flag to ensure that, if the page already resides in system memory, it is migrated to the memory tier indicated by the target-tier bitmask. When mbind succeeds, vmcache𝑛 reads Page P3 from disk via pread to the corresponding physical frame mapped by the virtual memory address. Then, the OS installs the virtual-to-physical mapping for P3 in the OS Page Table. Any subsequent access to P3 is served from Tier-1, until P3 is evicted or is moved to a different memory tier. u64 offset = 3* pageSize ; u64 tgt_tier = 0 b0010 ; int mode = MPOL_BIND ; int flags = MPOL_MF_MOVE ; mbind ( virtMem + offset , pageSize , mode , tgt_tier , 8* sizeof ( tgt_tier ) , flags ) ; pread ( fd , virtMem + offset , pageSize , offset ) ;
Adding and removing pages across memory-tier caches. In an 𝑛-tier hierarchy, pages may migrate between two memory tiers. For example, when the working set exceeds the capacity of the DRAMtier, pages can be evicted from DRAM and can be offloaded to a
Virtual-Memory Assisted Buffer Management In Tiered Memory
RMem tier rather than to disk. Similarly, pages can be prefetched from a RMem tier back to the DRAM tier. Migrations between intermediate RMem tiers are also possible to reduce a page’s access latency. vmcache𝑛 maintains the following invariant during migration, i.e., although the physical frame backing a page may change, its virtual memory address, i.e., PID, must remain stable. This invariant precludes software-level copy-based page migration mechanisms, e.g., memcpy, as it changes the virtual address of a page. vmcache𝑛 supports two system calls, namely, mbind and move_pages to migrate pages between memory tiers. Both system calls preserve vmcache𝑛 ’s invariant. Once the migration completes, the syscalls update the corresponding page-table entry of the Page Table to reflect the updated physical frame. Although both mbind and move_pages can migrate pages, they differ in their migration granularity. While move_pages can migrate multiple pages, mbind can only migrate one page at a time. For example, assume that we want to migrate Page P3 and Page P6 to Tier-1 memory. vmcache𝑛 needs to invoke mbind twice (i.e., separately) to migrate those pages. u64 int u64 for
offset1 =3* pageSize ; u64 offset2 =6* pageSize ; mode = MPOL_BIND ; int flags = MPOL_MF_MOVE ; tgt_tier = 0 b0010 ; offsets in [ offset1 , offset2 ]: mbind ( virtMem + offset , pageSize , mode , tgt _tier , 8* sizeof ( tgt_tier ) , flags ) ;
In contrast, vmcache𝑛 invokes the move_pages system call once to migrate multiple pages (See the example below). The first argument (=0) refers to the calling process, i.e., vmcache𝑛 . The second argument (=2) sets the number of pages to migrate, and defines the size of the subsequent three array arguments. The pages array contains the virtual addresses of the pages to be migrated, i.e., {P3, P6}. The tgt_tier array specifies the destination memory tier for each page. The status array reports the migration status of each page upon migration completion. Finally, the MPOL_MF_MOVE flag requests the migration of the specified pages’ physical frames to the target memory tiers. u64 [] pages = { virtMem +3* pageSize , virtMem +6* pageSize }; int flag = MPOL_MF_MOVE ; int [] tgt_tiers ={1 , 1}; int [] status ={ -1 , -1}; move_pages (0 , 2, offset , tgt_tier , status , flag ) ;
2.2. Page States & Synchronization. Following vmcache, vmcache𝑛 uses 64 bits to represent the state of a page. The most significant 8 bits represent 4 page states in vmcache, i.e., Unlocked (0), LockedShared (1-252), Locked (253), Marked (254), and Evicted (255). To accommodate additional memory tiers in an 𝑛-tier setting, vmcache𝑛 encodes the location, i.e., memory tier of the physical frame backing the page using log2 (𝑛 − 1) bits. For example, in a 3-tier setting (DRAM-RMem-Disk), 1 bit encodes the memory tier, thereby requiring a total of 9 bits to encode the state of a page. This yields tier-specific page states, e.g., UnlockedDRAM, UnlockedRMem, etc. Compared to vmcache, vmcache𝑛 reserves fewer bits, i.e., 56-log2 (𝑛 − 1) to encode the version counters. State transition. Figure 1 shows the state transition diagram of a vmcache𝑛 page in a 3-tier setting. Unlike vmcache, vmcache𝑛 maintains tier-specific page states to allow for flexible page movements across memory tiers. Following [38], vmcache𝑛 maintains four migration flags to probabilistically migrate pages between memory
Conference’17, July 2017, Washington, DC, USA
tiers, i.e., Dr, Dw, Rr, Rw that we discuss below. Assume that vmcache𝑛 needs to access Page P1 that is in Evicted state. vmcache𝑛 calls fix(P1). In a 3-tier setting, fix(P1) has two possible target tiers: DRAM or RMem, chosen probabilistically by Rr. Assume that vmcache𝑛 chooses DRAM as the target tier. P1 transitions to the Locked State using a Compare-and-Swap operation (CAS, for short). vmcache𝑛 reads the page from disk and copies it to DRAM using the mbind system call. mbind implicitly updates the OS Page Table. After access, vmcache𝑛 calls unfix(P1), and P1 transitions to the Unlocked State. When the DRAM cache utilization reaches a predefined threshold, e.g., 95%, vmcache𝑛 pro-actively marks unlocked pages, e.g., P1 with the clock replacement algorithm. A Marked page in DRAM has two target tiers, i.e., RMem or disk, controlled by Rw. If Rw selects RMem, vmcache𝑛 uses the move_pages system call to demote P1 from DRAM to the RMem. The syscall implicitly updates the OS Page Table with P1’s updated resident memory tier. Page P1 starts fresh in State Unlocked in RMem. Otherwise, P1 is evicted to disk. In another scenario, assume that vmcache𝑛 needs to access Page P2 that resides in RMem. vmcache𝑛 invokes fix(P2). Owing to the byte-addressability of RMem, vmcache𝑛 may either access P2 directly in place or first migrate it to DRAM before accessing it. This decision is made by Rr. Suppose that Rr chooses to promote P2 to DRAM. In this case, vmcache𝑛 prefetches a set of Unlocked pages from the remote tier and bundles them with P2, promoting all selected pages together in a batched migration. 2.3. Page Replacement. vmcache𝑛 maintains a separate cache for each memory tier. When all the caches get full, vmcache𝑛 needs to evict pages from them. Following vmcache, vmcache𝑛 uses the clock replacement algorithm for this purpose. Unlike vmcache, in a 3-tier setting, there are two distinct eviction paths, i.e., eviction between memory tiers and eviction from a memory tier to disk. In addition, the introduction of additional memory tiers introduces additional promotion paths between two memory tiers. vmcache𝑛 uses a simple heuristic to move pages upwards in the 3-tier hierarchy. Batch eviction between memory tiers. Unlike vmcache, each cache in vmcache𝑛 maintains a separate resident set, i.e., an openaddressing hash table that indexes the resident page PIDs in the corresponding memory tier. For example, in a 3-Tier setting, when DRAM cache size reaches a threshold and the target tier is RMem, vmcache𝑛 starts the following eviction routine: (1) Scan the DRAM resident set, collect the set of Marked pages selected for eviction, and lock these pages. (2) Update the location of the pages in the OS Page Table using move_pages. Since the pages remain memoryresident, no additional handling is required for dirty pages. (3) Remove the locked pages from the DRAM resident set and insert them into the RMem resident set. (4) Update the physical tier location metadata for each page and release the locks. Local Memory fix
Locked
unfix
Unlocked
fix
Evicted
promote fix
Locked
evict (disk)
cand.
promote unfix
Unlocked
fix
Remote Memory
Marked demote
cand.
Marked
evict (disk)
Figure 1: Page states in a 3-tier vmcache𝑛 .
Conference’17, July 2017, Washington, DC, USA
Batch promotion between memory tiers. In vmcache𝑛 , a page may reside in memory, but in a RMem tier. When a page lookup requires accessing a page from the RMem tier, vmcache𝑛 probabilistically decides whether the page should be promoted. If promotion is triggered, vmcache𝑛 locks the target page, scans the corresponding resident set, and collects a set of Unlocked pages selected for promotion. vmcache𝑛 locks these pages, and follows Steps 2–4 of the batch eviction routine.
3
move_pages2: Efficient Page Migration Across Memory Tiers
vmcache𝑛 relies on the virtual memory subsystem and OS-driven system calls, e.g., mbind [20] and move_pages [19], to dynamically alter the physical location of a page while preserving its virtual address. In any virtual-memory assisted buffer pool operating in an 𝑛-tier setting, page migration lies on the critical path of query execution. Hence, the performance of these system calls is paramount to achieving an efficient virtual-memory-assisted buffer pool. Although the native page migration system calls suffice in general, they may encounter performance and scalability issues under certain workloads. To address this limitation, we propose move_pages2. 3.1. Motivation. Linux handles page migration through two system calls, mbind [20] and move_pages [19]. We focus on move_pages as it enables batched page migration. In contrast, mbind migrate pages individually, and does not scale. 1. TLB Shootdowns. move_pages takes a list of pages as input and migrates them in multiple rounds. In each round, the kernel groups up to NR_MAX_BATCHED_MIGRATION consecutive pages targeting the same memory tier, batches them, and enqueues them into a pagelist queue. The batch size directly impacts performance. Larger batches reduce TLB invalidation overhead and decrease the number of invocations of the store_status helper function that updates per-page status. TLB shootdowns are expensive because they require inter-processor interrupts (IPIs) to invalidate TLB entries across CPUs. By default, in move_pages, NR_MAX_BATCHED_MIGRATION is set to 512. The default migration mode is MIGRATE_SYNC, which may trigger up to NR_MAX_BATCHED_MIGRATION TLB shootdowns per round, significantly degrading migration performance. 2. Abort-on-failure strategy. During migration, move_pages invokes the kernel function do_pages_move, whose error-handling strategy has significant performance implications. When the kernel encounters a user-space access failure, an invalid target node, or a permission error for any page, it immediately aborts the operation. Before aborting, it migrates only the pages accumulated in the current batch, skipping all remaining pages in the input pages list, even if they are eligible for migration. This “abort-on-failure" strategy is particularly detrimental in multi-threaded environments, where concurrent activity increases the likelihood of pages being locked, under writeback, or temporarily unavailable. 3.2. Design Principle & Implementation Details. move_pages2. move_pages2 excludes the pid and flags parameters from the original move_pages interface and instead introduces two new parameters, migrate_mode and
nr_max_batched_migration, to provide vmcache𝑛 with finegrained control over page migration. By default, move_pages2 sets pid = 0 and flags = MPOL_MF_MOVE, thereby granting vmcache𝑛 exclusive control over the migration process. The signature of the proposed custom move_pages2 implementation is as follows. long move_pages2 (unsigned long count, void *pages[.count], const int nodes[.count], int status[.count], enum migrate_mode mode, int nr_max_batched_migration); • count: The number of pages to process. • pages: An array of page-pointers that need to be processed. • nodes: An array of target NUMA node IDs. • status: An array to store the migration status of each page. • mode: The mode of migration. • nr_max_batched_migration: The maximum number of pages that can be accumulated before TLB shootdown is invoked.
1. Reducing TLB Shootdown. move_pages2 introduces nr_max_batched_migration as a parameter to regulate the number of pages migrated in a single batch. This parameter directly impacts migration performance by amortizing the cost of TLB shootdowns up to nr_max_batched_migration pages. In addition, move_pages2 exposes three page migration modes through the migration_mode parameter. • MIGRATE_ASYNC performs asynchronous migration, i.e., it proceeds with the next page even if the current page migration fails, making it a non-blocking approach. • MIGRATE_SYNC performs synchronous migration, i.e., the kernel blocks until the current page migration succeeds. • MIGRATE_SYNC_LIGHT does not block on page writebacks to reduce the stall time. 2. Optimistic Failure Handling. move_pages2 adopts an optimistic error-handling strategy by allowing partial migration. Invoking a system call incurs the overhead of crossing the OS-vmcache𝑛 boundary. move_pages2 amortizes this overhead by migrating as many pages as possible within each invocation rather than terminating prematurely due to an error. 3.3. Implementation Details. We introduce approximately 150 lines of code changes across the following four kernel functions: do_pages_move, move_pages_and_store_status, do_move_pages_to_node, and migrate_pages to implement move_pages2. We implement move_pages2 in Linux Kernel 6.8.0 on Ubuntu 24.04 LTS. The kernel disk image is available on CloudLab [10] and can be accessed here2 . 1. Migration Modes and Bulk Batching. Our custom implementation of migrate_pages (cf. Listing 1) enables vmcache𝑛 to override the default Linux migration mode, i.e., MIGRATE_SYNC. For MIGRATE_SYNC, the kernel follows the default blocking path (Line 13). Otherwise, the kernel follows the non-blocking path (Line 15). move_pages2 further allows vmcache𝑛 to dynamically select the maximum batch size during migration (Lines 6, 9). 1 int migrate_pages2 ( . . . , enum migrate_mode mode , . . . , int nr_max_batched_migration ) { 2 again : 3 nr_pages = 0; // Iterate over the pages in " from " 4 list_for_each_entry_safe ( folio , folio2 , from , lru ) { 5 nr_pages += folio_nr_pages ( folio ) ; 6 if ( nr_pages >= nr_max_batched_migration ) break ; 2 urn:publicid:IDN+utah.cloudlab.us+image+pmoss-PG0:LINUX6.8.12_MIGRATE
Virtual-Memory Assisted Buffer Management In Tiered Memory 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 }
} // Batch the pages in " from " into folios and migrate if ( nr_pages >= nr_max_batched_migration ) list_cut_before (& folios , from , & folio2 - > lru ) ; else list_splice_init ( from , & folios ) ; if ( migration_mode == MIGRATE_SYNC ) rc = migrate_pages_sync ( . . . , MIGRATE_SYNC , . . . ) ; else rc = migrate_pages_batch ( . . . , mode , . . . , nr_max_batched_migration ) ; // Store non - migrated pages in " ret_folios " for retry list_splice_tail_init (& folios , & ret_folios ) ; if ( rc < 0) { rc_gather = rc ; list_splice_tail (& split_folios , & ret_folios ); goto out ; } rc_gather += rc ; if (! list_empty ( from ) ) goto again ; // Prep next folios out : // Move back the non - migrated pages in the current // migration round to " folios " for potential // retry in the next migration round list_splice (& ret_folios , from ) ; if ( list_empty ( from ) ) rc_gather = 0; return rc_gather ;
Listing 1: Pseudocode of migrate_pages2. 2. Optimistic Error Handling. Our custom implementation of Kernel Function do_pages_move (cf. Listing 2) ensures that if the kernel encounters a user-space access failure, an invalid target node, or a permission issue for a particular page, it does not abort immediately. Instead, it records the error in the status array and invokes the custom Kernel Function move_pages_store_status2 to migrate the pages accumulated in the current round. After completing the batch, the kernel initiates a new migration round and continues processing the remaining pages in the pages list (Lines 26–31). move_pages2 follows the same routine when it encounters the errors during migration (Lines 19–25). 1 static int do_pages_move2 ( . . . , enum migrate_mode mode , int nr_max_batched_migration ) { 2 for (i= start =0; i < nr_pages ; i ++) { 3 if ( error in copying pages [ i ] or nodes [ i ]) 4 goto handle_error ; 5 if ( error in handling target tier ) 6 goto handle_error ; 7 if ( current_node == NUMA_NO_NODE ) { // Start mig . 8 current_node = node ; start = i ; 9 } else if ( node != current_node ) { 10 // End the current migration round and migrate 11 // the pages collected in the current round 12 err = move_pages_and_store_status2 ( . . . , mode , nr_max_batched_migration ) ; 13 if ( err ) goto migrate_error ; 14 start = i; current_node = node ; 15 } 16 // Queue page p for the current migration round 17 err = add_page_for_migration () ; if ( err > 0) continue ; 18 err = store_status () ; 19 migrate_error : 20 if (i + 1 == nr_pages || err < 0) { 21 err1 = move_pages_and_store_status2 () ; // Migrate 22 if ( err >= 0) err = err1 ; 23 current_node = NUMA_NO_NODE ; // New mig . round 24 } 25 continue ; // Continue migration 26 handle_error : 27 err1 = store_status ( status , i , err , 1) ; 28 if ( err1 ) err = err1 ; 29 err1 = move_pages_and_store_status2 () ; // Migrate 30 if ( err >= 0) err = err1 ; 31 current_node = NUMA_NO_NODE ; // New round@
Conference’17, July 2017, Washington, DC, USA 32 } 33 out_flush : 34 if ( current_node != NUMA_NO_NODE ) { 35 err1 = move_pages_and_store_status2 ( . . . , mode , nr_max_batched_migration ) ; 36 if ( err >= 0) 37 err = err1 ; 38 } 39 out : 40 return err ; 41 }
Listing 2: Pseudocode of do_pages_move2.
4
Performance Evaluation
All experiments run on a CloudLab [10] sm220u node, i.e., an Intel(R) Xeon(R) Silver 4314 CPU with 2 NUMA sockets, each with 32 logical threads. We treat DRAM of NUMA Socket 0 as local memory (Tier1), DRAM of NUMA Socket 1 as remote memory (Tier-2) and a 960 GB Samsung PCIe4 NVMe as disk (Tier-3). All experiments use 32 threads of NUMA Socket 0. Following [22], we use the same TPC-C and a key-value workload that consists of random point lookups. The key–value workload uses 8-byte uniformly distributed keys and 120-byte values. The TPC-C and random-read workloads span approximately 190 GB and 130 GB, respectively. vmcache𝑛 vs. vmcache. Figures 2a and 2d compare the performance of vmcache𝑛 and vmcache over time for the TPC-C and random-read workloads. The local memory capacity is fixed at 32 GB, while the remote memory capacity varies from 8 GB to 128 GB. Initially, at time 𝑇0 , the remote memory tier is empty. As transactions execute over time, it gradually becomes populated. For TPC-C, vmcache𝑛 achieves up to 1.67× and 3.82× higher throughput than vmcache when the remote memory capacity is 2× and 4× the size of local memory, respectively. Although vmcache𝑛 reduces disk I/O, when the remote memory is small, traffic between memory tiers becomes the performance bottleneck. For the random-read workload, with no writes, the performance gain is not significant, e.g., vmcache𝑛 achieves 1.36× better transactions throughput when remote memory is 4× the size of the local memory. Here, page transfers between memory tiers dominate the execution cost as the working set exhibits random access patterns. Where does the time go? Figure 3 gives the execution time breakdown of vmcache𝑛 for the TPC-C workload (Fig. 3a) and the randomread workload (Fig. 3b) using alternate page migration syscalls. Disk I/O dominates the TPC-C workload, whereas page transfers between memory tiers dominate the random-read workload. Using mbind incurs substantial overhead for the random-read workload, accounting for 64.8% of the total execution time, as mbind does not support bulk page movement. Even move_pages and move_pages2 incur significant overheads of 48.5% and 48.4%, respectively, for the random-read workload. These results highlight the critical importance of an efficient page-movement interface for virtual-memory assisted buffer pools in an 𝑛-tier memory hierarchy. For these experiments, the local and remote memory capacity is set to 32 and 64 GB, respectively. The migration ratios and the batch eviction size between memory tiers are set to 1 and 512, respectively. Performance evaluation of move_pages2. Figures 2b and 2e give the performance of move_pages2 in vmcache𝑛 against baseline page migration system calls, i.e., mbind and move_pages. mbind(1) sets the batch eviction size between memory tiers to 1. For the rest, the
Conference’17, July 2017, Washington, DC, USA
Transactions / sec
Total I/O (MB/s) 0
0
500 Time [seconds]
0
(a) vmcache𝑛 vs. vmcache (TPC-C)
106
106
Transactions / sec
Total I/O (MB/s)
Transactions / sec
0
102
500 Time [seconds]
0
0
500 Time [seconds]
26 27 28 29 210 211 212 NR_MAX_BATCHED_MIGRATION
500 Time [seconds]
(d) vmcache𝑛 vs. vmcache (Random read)
(c) Max migrations. 105
105 104
105
103
0
500 Time [seconds]
SYNC (Read) SYNC_LIGHT (TPC-C) SYNC_LIGHT (Read)
105
(b) move_pages2 performance (TPC-C).
103
105
500 Time [seconds]
ASYNC (TPC-C) ASYNC (Read) SYNC (TPC-C)
move_pages2
104
104
102
500 Time [seconds]
move_pages
Transactions / sec
Transactions / sec
104
mbind
105
103
105
mbind (1)
Average Transactions / sec
vmcachen(32) vmcachen(128)
Page Transfers / sec
vmcachen(8) vmcachen(16)
Page Transfers / sec
vmcachen(64) vmcache
0
500 Time [seconds]
(e) move_pages2 performance (Random read)
64 128 256
512 1024
2048 4096
TPC-C Read
104 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Dr , Dw , Rr , Rw
(f) Migration ratio.
Figure 2: vmcache𝑛 vs. vmcache. batch eviction size is set to 512. move_pages2 exhibits similar performance to move_pages for the TPC-C workload (cf. 2b). However, for the random-read workload, move_pages2 achieves 1.42×, 1.32× better query and page migration throughputs over move_pages, respectively (cf. 2e), due to the maximum number of memory pages move_pages2 can batch together in a single round compared to the baselines, thus amortizing the cost of TLB invalidation. Ablation analysis of move_pages2. Figure 2c gives the performance of move_pages2 under varying |NR_MAX_BATCHED_MIGRATION| values and migration modes. |NR_MAX_BATCHED_MIGRATION| decides the maximum number of memory pages that can be migrated in a single round. In vmcache𝑛 , this is set to 2× the batch eviction size between memory tiers. For the random-read workload, increasing this value improves performance, as larger batches amortize migration overhead. For TPC-C, which is write-dominated, increasing the batch size does not provide similar benefits. Migration modes ASYNC, SYNC and SYNC_NO_COPY yield similar performance benefit in vmcache𝑛 , as the pages are locked before migration, regardless. Hence, allowing partial page migration does not translate into measurable performance improvement in vmcache𝑛 . Figure 2f gives the performance of vmcache𝑛 under different migration ratios. While a larger |NR_MAX_BATCHED_MIGRATION| improves performance when page migrations are infrequent, it does not provide the same benefit as the migration frequency increases.
80% 60% 40% 20% 0%
IO
44.8% 17.9% 37.4% mbind
Migration
Compute
46.5% 1.9%
45.8% 1.8%
51.6%
52.4%
move_pages move_pages2
(a) TPC-C workload.
100%
Time breakdown (%)
Time breakdown (%)
100%
80% 60% 40% 20% 0%
IO
Migration
20.3%
30.9%
31.0%
64.8%
48.5%
48.4%
20.6%
20.6%
14.9% mbind
Compute
move_pages move_pages2
(b) Random read workload.
Figure 3: Execution time breakdown of vmcache𝑛 .
5
Concluding Remarks
We present vmcache𝑛 , an 𝑛-tier virtual-memory assisted buffer pool that delivers up to 4× higher TPC-C throughput than vmcache. The key findings can be summarized as follows. 1. For virtual-memory assisted buffer pools, remote memory investment has a clear cost break-even point between, when the remote memory capacity is between 1× and 2× DRAM capacity. Below this threshold, the net gain in QPS/$ compared to a 2-tier buffer pool is negative, as page migration overheads between memory tiers dominate any capacity benefit. Beyond the break-even point, the net gain in QPS/$ becomes strongly positive and increases with scale, reaching +41 QPS/$ and +87 QPS/$ 3 , when remote memory capacity is 2×, 4× DRAM capacity, respectively. 2. Page migration between memory tiers is the bottleneck in an 𝑛-tier virtual-memory assisted buffer pool, particularly when the working set mostly fits within the memory tiers. move_pages2 mitigates this overhead by amortizing the migration cost through large batched transfers per each migration round. However, page migration overheads still remain. In our experiments, less than 0.005% of execution time is attributed to kernel–user mode transitions and migration preparation overheads [35]. Thus, re-architecting the system around microkernels to manage pages and page tables in user space is unlikely to address the dominant bottleneck [23]. Rather, more efficient page migration implementations are likely to yield greater benefit, be it in user space [30] or kernel space [2, 4, 7, 11, 34, 36]. 3. Due to the invariant of stable virtual addresses with alternating physical memory tiers, an 𝑛-tier virtual-memory assisted buffer pool can include remote memory tiers only if they are exposed to the host server in System-DRAM mode. Memory allocated via DAX or App Direct mode cannot be used in an 𝑛-tier virtual-memory assisted buffer pool, which limits the applicability of virtual-memory assisted buffer pools in this setting. 3 Assuming, a Micron DDR4 3200MHz ECC RDIMM Memory is priced at 175$
Virtual-Memory Assisted Buffer Management In Tiered Memory
References [1] Minseon Ahn, Andrew Chang, Donghun Lee, Jongmin Gim, Jungmin Kim, Jaemin Jung, Oliver Rebholz, Vincent Pham, Krishna T. Malladi, and Yang-Seok Ki. 2022. Enabling CXL Memory Expansion for In-Memory Database Management Systems. In DaMoN. ACM, 8:1–8:5. [2] Nadav Amit. 2017. Optimizing the TLB Shootdown Algorithm with Page Access Tracking. In USENIX. 27–39. [3] Ramesh Aravind and Groves John. 2024. Study of CXL Memory Sharing with FamFS and its Use cases. In HiPC Workshops. IEEE, 73–77. doi:10.1109/ HIPCW63042.2024.00022 [4] Amro Awad, Arkaprava Basu, Sergey Blagodurov, Yan Solihin, and Gabriel H. Loh. 2017. Avoiding TLB Shootdowns Through Self-Invalidating TLB Entries. In PACT. 273–287. [5] Yannis Chronis, Anastasia Ailamaki, Lawrence Benson, Helena Caminal, Jana Giceva, Dave Patterson, Eric Sedlar, and Lisa Wu Wills. 2025. Databases in the Era of Memory-Centric Computing. In CIDR. [6] Counterpoint Research. 2026. Memory Prices Surge Up to 90% From Q4 2025. https://counterpointresearch.com/en/insights/Memory-Prices-Surge-Upto-90-From-Q4-2025. Accessed February 11, 2026. [7] Guilherme Cox and Abhishek Bhattacharjee. 2017. Efficient Address Translation for Architectures with Multiple Page Sizes. In ASPLOS. 435–448. [8] Daniel Ehrenberg. 2026. The Asynchronous Input/Output (AIO) interface. https: https://github.com/littledan/linux-aio. Accessed February 11, 2026. Trends in DRAM price per gigabyte. [9] Daniel Ehrenberg. 2026. https://wiki.aiimpacts.org/ai_timelines/hardware_and_ai_timelines/trends_in_ dram_price_per_gigabyte. Accessed February 11, 2026. [10] Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuang-Ching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. 2019. The Design and Operation of CloudLab. In USENIX. USENIX Association, 1–14. [11] Jayneel Gandhi, Vasileios Karakostas, Furkan Ayar, Adrián Cristal, Mark D. Hill, Kathryn S. McKinley, Mario Nemirovsky, Michael M. Swift, and Osman S. Unsal. 2016. Range Translations for Fast Virtual Memory. IEEE Micro 36, 3 (2016), 118–126. [12] Goetz Graefe, Haris Volos, Hideaki Kimura, Harumi A. Kuno, Joseph A. Tucek, Mark Lillibridge, and Alistair C. Veitch. 2014. In-Memory Performance for Big Data. Proc. VLDB Endow. 8, 1 (2014), 37–48. doi:10.14778/2735461.2735465 [13] Yunyan Guo and Guoliang Li. 2024. A CXL- Powered Database System: Opportunities and Challenges. In ICDE. IEEE, 5593–5604. [14] Xiangpeng Hao, Xinjing Zhou, Xiangyao Yu, and Michael Stonebraker. 2024. Towards Buffer Management with Tiered Main Memory. SIGMOD 2, 1 (2024), 31:1–31:26. [15] Stavros Harizopoulos, Daniel J. Abadi, Samuel Madden, and Michael Stonebraker. 2008. OLTP through the looking glass, and what we found there. In SIGMOD. 981–992. doi:10.1145/1376616.1376713 [16] Yibo Huang, Newton Ni, Vijay Chidambaram, Emmett Witchel, and Dixin Tang. 2025. Pasha: An Efficient, Scalable Database Architecture for CXL Pods. In CIDR. [17] Intel Corporation. 2024. CXL* Type 3 Memory Device Software Guide. Revision 1.1. https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url= https://cdrdv2-public.intel.com/643805/643805_CXL_Memory_Device_SW_ Guide_Rev1_1.pdf. Accessed February 9, 2026. [18] John C. McCallum. 2026. Memory Prices 1957+. https://docs.google.com/ spreadsheets/d/1qF5vJ2-jkRygMQQIk3fj6G-WEW0ScDiKjyGseovFGqc/. Accessed February 11, 2026. [19] Michael Kerrisk. 2025. move_pages(2) — Linux manual page. https://man7.org/ linux/man-pages/man2/move_pages.2.html. Accessed: February 11, 2026. [20] Andi Kleen. 2026. mbind — Linux manual page. https://man7.org/linux/manpages/man2/mbind.2.html. Accessed: February 11, 2026. [21] Sangjin Lee, Alberto Lerner, Philippe Bonnet, and Philippe Cudré-Mauroux. 2024. Database Kernels: Seamless Integration of Database Systems and Fast Storage via CXL. In CIDR. [22] Viktor Leis, Adnan Alhomssi, Tobias Ziegler, Yannick Loeck, and Christian Dietrich. 2023. Virtual-Memory Assisted Buffer Management. Proc. ACM Manag. Data 1, 1 (2023), 7:1–7:25. doi:10.1145/3588687 [23] Viktor Leis and Christian Dietrich. 2024. Cloud-Native Database Systems and Unikernels: Reimagining OS Abstractions for Modern Hardware. VLDB 17, 8 (2024), 2115–2122. doi:10.14778/3659437.3659462 [24] Viktor Leis, Michael Haubenschild, Alfons Kemper, and Thomas Neumann. 2018. LeanStore: In-Memory Data Management beyond Main Memory. In ICDE. 185– 196. doi:10.1109/ICDE.2018.00026 [25] Jinshu Liu, Hamid Hadian, Yuyue Wang, Daniel S. Berger, Marie Nguyen, Xun Jian, Sam H. Noh, and Huaicheng Li. 2025. Systematic CXL Memory Characterization and Performance Analysis at Scale. In ASPLOS. 1203–1217. [26] Thomas Neumann and Michael J. Freitag. 2020. Umbra: A Disk-Based System with In-Memory Performance. In CIDR. https://vldb.org/cidrdb/2020/umbra-adisk-based-system-with-in-memory-performance.html
Conference’17, July 2017, Washington, DC, USA [27] Riki Otaki, Charles Benello, Aaron J. Elmore, and Goetz Graefe. 2025. Resource-Adaptive Query Execution with Paged Memory Management. In CIDR. https://vldb.org/cidrdb/2025/resource-adaptive-query-execution-withpaged-memory-management.html [28] Riki Otaki, Jun Hyuk Chang, Aaron J. Elmore, and Goetz Graefe. 2025. Enhancing Transaction Processing through Indirection Skipping. Proc. VLDB Endow. 18, 11 (2025), 4104–4116. doi:10.14778/3749646.3749680 [29] Niklas Riekenbrauck, Marcel Weisgut, Daniel Lindner, and Tilmann Rabl. 2024. A Three-Tier Buffer Manager Integrating CXL Device Memory for Database Systems. In HardBD & Active. 395–401. [30] Felix Schuhknecht and Nick Rassau. 2026. Taking the Leap: Efficient and Reliable Fine-Grained NUMA Migration in User-space. CoRR abs/2602.05540 (2026). arXiv:2602.05540 http://arxiv.org/abs/2602.05540 [31] Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim. 2023. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. In MICRO. 105–121. [32] Alexander van Renen, Viktor Leis, Alfons Kemper, Thomas Neumann, Takushi Hashida, Kazuichi Oe, Yoshiyasu Doi, Lilian Harada, and Mitsuru Sato. 2018. Managing Non-Volatile Memory in Database Systems. In SIGMOD. 1541–1555. doi:10.1145/3183713.3196897 [33] Marcel Weisgut, Daniel Ritter, Florian Schmeller, Pınar Tözün, and Tilmann Rabl. 2025. CXL-Bench: Benchmarking Shared CXL Memory Access. In ADMS. https://pure.itu.dk/en/publications/cxl-bench-benchmarking-sharedcxl-memory-access/ [34] Lingfeng Xiang, Zhen Lin, Weishu Deng, Hui Lu, Jia Rao, Yifan Yuan, and Ren Wang. 2024. Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration. In USENIX. 19–35. [35] Tong Xing and Antonio Barbalace. 2025. Rethinking Applications’ Address Space with CXL Shared Memory Pools. In HCDS. 52–59. doi:10.1145/3723851.3723858 [36] Zi Yan, Daniel Lustig, David W. Nellans, and Abhishek Bhattacharjee. 2019. Nimble Page Management for Tiered Memory Systems. In ASPLOS. 331–345. [37] Xinjun Yang, Yingqiang Zhang, Hao Chen, Feifei Li, Gerry Fan, Yang Kong, Bo Wang, Jing Fang, Yuhui Wang, Tao Huang, Wenpu Hu, Jim Kao, and Jianping Jiang. 2025. Unlocking the Potential of CXL for Disaggregated Memory in Cloud-Native Databases. In SIGMOD. 689–702. [38] Xinjing Zhou, Joy Arulraj, Andrew Pavlo, and David E. Cohen. 2021. Spitfire: A Three-Tier Buffer Manager for Volatile and Non-Volatile Memory. In SIGMOD. 2195–2207. doi:10.1145/3448016.3452819 [39] Xinjing Zhou, Viktor Leis, Xiangyao Yu, and Michael Stonebraker. 2025. OLTP Through the Looking Glass 16 Years Later: Communication is the New Bottleneck. In CIDR. https://vldb.org/cidrdb/2025/oltp-through-the-looking-glass-16-yearslater-communication-is-the-new-bottleneck.html