arXiv:2607.15569v1 [cs.OS] 17 Jul 2026
Scaling Unmodified Multithreaded Applications with Elastic CXL-based Distributed Shared Memory Guowei Liu
Kang Chen
Laiping Zhao
Tianjin University Tianjin, China [email protected]
Tsinghua University Beijing, China [email protected]
Tianjin University Tianjin, China [email protected]
Yiming Li
Hanwen Liu
Chen Peng
Tianjin University Tianjin, China [email protected]
Tianjin University Tianjin, China [email protected]
Tianjin University Tianjin, China [email protected]
Yichi Chen
Sheng Chen
Zhiyuan Su
Tianjin University Tianjin, China [email protected]
Tianjin University Tianjin, China [email protected]
Inspur Electronic Information Industry Co., Ltd. Jinan, China [email protected]
Wenyu Qu Tianjin University Tianjin, China [email protected]
Abstract
1
While CXL presents a promising hardware substrate for Distributed Shared Memory (DSM), seamlessly scaling multithreaded applications across multiple nodes remains a formidable challenge. Existing CXL-based DSMs fall short: they require manual code modifications to share non-heap data, employ rigid data placement policies that fail under diverse and dynamic workloads, and suffer from severe page-fault processing overheads in sub-microsecond (𝜇s) environments. We present xDSM, a full-space, elastic DSM system built over CXL that transparently scales unmodified multithreaded applications. To eliminate the burden of manual code rewrites, xDSM employs an OS-runtime co-design that establishes a globally coordinated address space, seamlessly sharing all memory segments. To mask CXL access penalties, xDSM abandons static placement rules in favor of a dynamic, latencydriven policy that actively balances data between local DRAM and CXL memory. Finally, to resolve the fundamental tension between high base-page fault overheads and severe hugepage false sharing, xDSM introduces spatial locality-aware elasticity, dynamically coalescing and splitting pages on the fly to amortize processing costs. Evaluated across diverse workloads using 15 system configurations, xDSM outperforms CXL-only baselines by 1.5× to 2.2× and state-of-the-art hybrid DSMs by 1.1× to 2.2×, while achieving near-linear scalability.
Data-intensive applications, such as graph analytics [33, 45], data mining [55], and machine learning [1, 28], place unprecedented demands on memory capacity and computational throughput. When applications outgrow the resources of a single machine, they must scale out across multiple nodes. However, manual porting of multi-threaded programs [18] to distributed models, such as message passing or PGAS [10, 52], is labor-intensive and error-prone. Distributed shared memory (DSM) systems [23, 27, 37] were designed to bridge this gap. They provide the abstraction of a single shared address space across multiple nodes with software-maintained memory consistency. Yet, their reliance on Ethernet or RDMA networks introduces substantial protocol overhead, typically slowing applications by an order of magnitude [34, 39]. Compute Express Link (CXL) 3.0 [44] provides a new opportunity to build efficient DSM systems by replacing slow software coherence with hardware-managed, multi-node cache coherence via Back-Invalidation Snoop (BISnp). Despite this powerful hardware capability, constructing an efficient full-space DSM across multiple nodes remains challenging. The CXL-only approach places all data on the shared CXL device. While avoiding software protocol overheads, it incurs 2.0× to 4.0× higher latency than local DRAM [32, 46], degrading performance by up to 2.2× (§8.3). Conversely, the local-only approach treats CXL merely as a communication channel and pulls all data into local memory [22, 23]. This wastes CXL capacity and triggers frequent 1
Introduction
Table 1. Comparison of DSM systems. “Addr.” specifies whether the system relies on heap-only sharing or achieves full-space shared execution environment. “Place.” denotes the use of static or dynamic page placement strategies. “Gran.” refers to fixed or elastic page management granularity.
cross-node invalidations, causing up to an 8.0× performance degradation (§8.3). Firework [32] introduces a hybrid design utilizing both tiers via a static placement rule where shared pages map to CXL and private pages remain local. However, this static policy fails dynamic workloads. For instance, readshared pages are forcibly placed on the CXL tier, inflicting severe latency penalties on every access. This paper presents xDSM, a full-space, CXL-based hybrid DSM system. Through an OS-runtime co-design, xDSM bridges the low latency of local DRAM and the massive capacity of CXL memory. This architecture allows it to seamlessly execute unmodified multithreaded (pthread) programs across CXL-connected nodes. To realize this execution model, xDSM introduces three architectural innovations that directly overcome the fundamental limitations of existing systems. First, existing DSM systems are fundamentally restricted by the heap-only sharing limitation. They rely on customized memory allocators that leave global and static variables unshared, forcing developers to manually rewrite code to achieve distributed execution. xDSM overcomes this by natively establishing a full-space shared execution environment. Through globally coordinated addressing and lightweight VMA(Virtual Memory Area)-anchored page transfers, the system shares all memory segments to eliminate manual code modifications entirely. Second, current hybrid systems suffer from rigid page placement limitations. They typically enforce static rules, such as mapping all shared pages to CXL memory and private pages to local DRAM, which severely penalizes performance under dynamic or phase-changing workloads. xDSM abandons these static rules in favor of an adaptive data placement policy. By utilizing hardware-sampled, exponentially binned latency histograms, the system actively restores latency equilibrium between local DRAM and CXL memory to dynamically mask access penalties. Finally, existing approaches are constrained by fixedgranularity limitations, which create a dilemma in submicrosecond CXL environments. Operating at a 4 KB base granularity triggers overwhelming software page fault processing overheads, whereas adopting 2 MB huge pages exacerbates false sharing and wastes bandwidth. xDSM resolves this tension by introducing spatial locality-aware elasticity. The system dynamically coalesces contiguous pages in the background to amortize fault processing costs and performs on-demand splitting to isolate write-induced invalidations. The novelty of xDSM lies in a comprehensive solution that leverages CXL shared memory to seamlessly scale multithreaded programs across nodes, while fundamentally eliminating the performance bottlenecks associated with rigid, fixed-granularity memory placement. We extensively evaluate the system using five diverse benchmarks. xDSM achieves up to a 2.2× speedup over CXL-only architecture, outperforms traditional local-only DSMs by up to 8.0×, and surpasses static-hybrid DSMs by up to 2.2×. Furthermore, xDSM
Category
System
Addr.
Place.
Gran.
Local-only Local-only Local-only Local-only
IVY [27] TreadMarks [23] DSM-Threads [37] DRust [34]
Heap Heap Heap Heap
N/A N/A N/A N/A
Fixed Fixed Fixed Object
CXL-only
Baseline
Full
N/A
N/A
Hybrid
Firework [32]
Heap
Static rules
Fixed
Hybrid
xDSM
Full
Dynamic
Elastic
seamlessly adapts to dynamic workloads, achieving nearlinear scalability across multiple nodes. In summary, this paper makes the following contributions: • We comprehensively analyze existing approaches, identifying three fundermental limitations: the lack fullspace sharing, inflexible static data placement, and severe page-fault overheads. • Our system, xDSM, is a full-space CXL-based hybrid DSM featuring three core innovations: Coordinated Addressing with VMA-anchored page transfer, GapDriven Asymmetric Migration for latency equilibrium, and Spatial Locality-Aware Elasticity. • We thoroughly evaluate xDSM. The results demonstrate that xDSM significantly outperforms CXL-only, local-only, and static-hybrid DSMs.
2
Background and Motivation
2.1
CXL 3.0 and Multi-Node Shared Memory
Compute Express Link (CXL) [44] extends node capacity by allowing direct load/store accesses to remote memory. In CXL 3.0, a single memory device can simultaneously connect 8–16 independent compute nodes [17, 44] and enforce hardware cache coherence via back-invalidation (BISnp) [21, 44]. This hardware-managed architecture fundamentally eliminates the software consistency overheads inherent in traditional network-based DSMs (e.g., 2 𝜇s for RDMA network [53]). CXL reduces remote access latency to merely 200 ns–400 ns, approaching the ∼100 ns of local DRAM [46]. However, CXL 3.0 lacks a native global virtual address space, as compute nodes run independent OSes with isolated page tables. Constructing a multi-node DSM over CXL requires establishing a full-space shared execution environment to guarantee identical mappings across nodes, alongside distributed thread management to execute unmodified applications. Leveraging this hardware substrate, existing 2
CXL-based DSM systems fall into three categories. The CXLonly approach allocates all data exclusively on the shared CXL device, eliminating software consistency overheads but penalizing every access with higher interconnect latency. Conversely, the local-only approach provisions all data within local memory, utilizing CXL strictly as a high-speed communication medium similar to classical network-based DSM architectures [23, 27]. Finally, the hybrid approach integrates both memory tiers, attempting to optimize performance by placing data based on specific access patterns [32]. 2.2
and private pages locally. This static rule fails to accommodate dynamic workloads. For instance, forcing read-shared pages to CXL during a 16-thread BFS on a 6.8 GB read-only graph degraded performance by 34% (2.41 × 108 vs. 3.25 × 108 traversed edges per second) compared to local replication. To overcome this rigid page placement limitation, the system must replace static rules with an adaptive policy that distributes data across heterogeneous tiers on the fly. This is challenging because optimal placement depends on both the sharing pattern (e.g., private, shared-read, or shared-write) and the real-time load on each tier, both of which fluctuate significantly during runtime.
Limitations and Challenges
Table 1 compares representative systems. Drawing from this comparison, we identify three fundamental limitations in existing approaches and outline the system-level challenges required to overcome them.
2.2.3 Challenge 3: Mitigating Costly Page-Fault Processing Overheads. Existing DSM systems typically rely on page faults to transparently detect memory accesses and trigger data migration [23, 27, 32, 37]. Inherently, these systems adopt the fixed page granularity of the underlying hardware, typically 4 KB (Table 1). Traditional network-based DSMs tolerate page fault processing overheads as network delays dominate the critical path. Because CXL data accesses complete in under 400ns, the 24𝜇s to 30𝜇s software pipeline of a page fault (kernel trap, lock acquisition, and permission updates) instantly becomes a severe bottleneck [9]. This creates a strict fixed-granularity dilemma. Operating at a 4KB base page granularity triggers overwhelming trap overheads, whereas adopting 2MB huge pages reduces fault counts but exacerbates false sharing and wastes interconnect bandwidth [9]. To overcome this fixed-granularity limitation, the system must redesign traditional page fault processing. The core challenge is breaking this rigid granularity barrier to resolve the inherent tension between overwhelming fault processing costs at 4 KB and severe false sharing at 2 MB.
2.2.1 Challenge 1: Establishing a Full-Space Shared Execution Environment. Existing DSM systems, including local-only [23, 27, 34, 37] and hybrid [32] designs, rely on customized memory allocators to redirect heap allocations into shared memory. Consequently, global variables (.data), uninitialized globals (.bss), and function-scope statics remain unshared. Sharing these regions imposes a severe manual burden, as developers must explicitly rewrite static declarations into dynamic heap allocations and modify all corresponding access sites. However, non-heap shared data is pervasive in real-world applications. Among the 13 PARSEC workloads [8], 12 heavily rely on global or static variables for thread-shared states. For example, fluidanimate references 33 global variables (e.g., particle arrays and simulation parameters) across 401 lines in 9 files. Similarly, vips accesses 46 global variables at 1,843 reference sites scattered across 377 files. Manually converting such programs to a heap-only DSM is a prohibitively laborious and error-prone process. To overcome this heap-only sharing limitation, a DSM must natively share all memory segments. The fundamental challenge is that existing OSes (e.g., Linux) lack native mechanisms to establish a globally shared address space across distributed nodes. Each node runs an independent kernel with its own VMAs and page tables, and runtime allocations independently assign virtual addresses that may easily conflict across nodes.
2.2.4 Motivation: The Need for a New Comprehensive Solution. To fully unleash the potential of CXL-based DSM and execute unmodified multithreaded programs efficiently, a new comprehensive solution is required. Specifically, the system must (1) natively establish a full-space shared execution environment without imposing manual code-rewrite burdens, (2) replace rigid rules with an adaptive policy to optimally distribute data across heterogeneous tiers, and (3) break the fixed-granularity barrier to amortize 𝜇s-scale page fault processing overheads.
2.2.2 Challenge 2: Adapting Data Placement to Dynamic Workloads. Page placement directly dictates application performance because local DRAM and CXL memory (§8.1) differ significantly in latency (109 ns vs. 285 to 376 ns) and per-node bandwidth (36.5 GB/s vs. 29 to 37 GB/s). Unfortunately, existing approaches adopt rigid page placement policies. The CXL-only approach forces all accesses through higher-latency links, whereas the local-only approach wastes CXL capacity and bandwidth. Firework [32] attempts a hybrid design but maps shared pages to CXL
3
xDSM Design
3.1
Architecture Overview
xDSM’s overall architecture (Figure 1) follows an OS-runtime co-design. To efficiently manage sub-𝜇s CXL memory, xDSM introduces several foundational abstractions. Spatially, the virtual address space is statically divided into fixed-size Chunks (e.g., 2 MB) as the top-level allocation boundaries, 3
Compute Node 0 Application
T0
T1
T2
Compute Node 1 T3
Management Threads
xDSM Runtime
Sampling
xDSM Kernel Module
Decision
mmap
a Migration
promote/demote
redirect
T4
Sampling Migration Fault Handler
Write Fault
xDSM Kernel Module
Replicas
Local DRAM
RO → RW
1
W/R Fault
2
3
4
Unmapped
5
1
xdsm_database epte_statistics
epte_status
epte_migration
block_map
Counts/Access
Copy/Dirty bitmap
Migration Tasks
VA-to-CXL offset
xdsm_mapped stack
Virtual Memory Layout
Count
.text .data .bss heap
(2) Latency-Driven Placement (§5) P90
P90
Local CXL
Latency Histogram (per node)
4
5
CXL Shared Memory (3) Elastic Page Management (§6) merge
(1) Full-Space DSM (§4) memory-mapped
T5
xDSM Runtime
Read Fault
Fault Handler
Local DRAM CXL Link
Application
User Threads
(scan)
8P
split
4P (write fault)
4P 2P
2P
P0 P1 P2 P3 P4 P5 P6 P7
...
Elastic Page: 4KB - 2MB
Figure 1. Overview of xDSM architecture. The system employs an OS-runtime co-design. In user space, three types of management threads (Sampling, Decision, and Migration) as well as page fault handlers coordinate memory migration and consistency. In the shared CXL memory, a global xdsm_database maintains three core tables: epte_statistics tracks memory access statistics (e.g., access counts), epte_status manages consistency states (e.g., owner and dirty bitmaps), and epte_migration queues tiering tasks. Additionally, a global block_map records the deterministic mapping from virtual addresses to CXL offsets. while the OS natively manages 4 KB Base Pages at the bottom. Elastic Page is xDSM’s core management unit. It is an entity dynamically aggregated from contiguous base pages with same accessors, varying in size from 4 KB to 2 MB. All metadata tracking and data migration are executed at this elastic granularity. Temporally, xDSM operates in fixed-duration windows called Epochs (or Ticks, 1 s by default). During each tick, Sampling Threads profile memory accesses via hardware performance counters (one sample per 20,000 operations, where each operation is a macro-op dispatched after instruction decode). Each sampled L3-miss access is recorded in epte_statistics. For load misses, the cache-miss latency is additionally recorded into a per-node histogram (§5.1). At the tick boundary, a singleton Decision Thread reads these statistics alongside the consistency states from epte_status to generate batch page migration plans. These plans instruct the system to either Promote hot pages from the shared CXL device to local DRAM for latency reduction, or Demote local pages back to CXL to relieve memory pressure. The tasks are queued in epte_migration, where per-node Migration Threads dequeue and execute them via lightweight kernel module interfaces. Together, these coordinated abstractions and components resolve the three fundamental challenges, which we distill into the following architectural insights.
3.1.1 Insight 1: Coordinated Global Addressing and Lightweight VMA-Anchored Page Transfers (§4). To resolve heap-only sharing limitations (C1) and execute unmodified applications, xDSM establishes a globally coordinated virtual address layout. At initialization, it maps the entire application memory directly to the shared CXL device to guarantee a unified global view. To prevent collisions during runtime, all local memory allocations are assigned from strictly disjoint virtual address ranges across nodes. Meanwhile, any memory mapped to the CXL device retains the exact same virtual address globally. Consequently, the system guarantees a single unified address space at all times. xDSM further introduces a lightweight VMA-anchored page transfers mechanism to accelerate memory tiering. In Linux, a VMA, represented by vm_area_struct, describes a contiguous range of virtual addresses that have the same permission attributes and are backed by the same object (e.g., a file or anonymous mapping) [30]. When moving data between tiers, xDSM updates only the underlying Page Table Entries (PTEs) to point to local memory. It deliberately leaves the VMA unmodified. The VMA acts as an anchor pointing to the CXL backing store. This decoupling makes promote and demote operations extremely lightweight on the fast path. It fundamentally avoids expensive cross-node coordination, severe mmap_lock contention, and the overhead of repeatedly re-allocating shared CXL memory. Heavyweight VMA 4
Table 2. Tables in xdsm_database.
modifications are deferred to a slow path, invoked solely to unmap the CXL region and reclaim remote capacity when a page achieves long-term local residency.
Table / Field
Description
epte_status
3.1.2 Insight 2: Restoring Latency Equilibrium via Gap-Driven Asymmetric Migration (§5). To overcome the limitations of static placement rules and dynamically adapt to shifting workloads (C2), xDSM abandons rigid policies in favor of an adaptive placement policy. Existing tiered memory systems often rely on average access latencies [31, 51], a single-value metric highly vulnerable to long-tail outliers. xDSM addresses this limitation by introducing Expbinned Latency Histograms. By recording hardware-sampled latencies into exponentially growing intervals, xDSM naturally absorbs long-tail anomalies and establishes a stable metric. The placement policy’s core objective is to restore and maintain latency equilibrium, which is defined as P90-Bin Alignment—the state where the 90th-percentile latency bins of local DRAM and CXL memory intersect. To swiftly achieve this equilibrium, xDSM pipelines the execution through two mechanisms. First, it employs GapProportional Volume Scaling, dynamically computing the number of pages to migrate proportional to the measured P90-bin gap (Δ). Second, it enforces Asymmetric Migration Pacing during execution. xDSM aggressively promotes active pages to mask CXL latency penalties, while conservatively throttling demotion volumes to prevent thrashing.
private owner dirty copyset lock epte_statistics
ld_cnt/st_cnt Per-tick load/store counts. accessor Per-tick accessor node bitmask. hist_{ld,st}_accessor Cumulative reader/writer node bitmask (periodically reset). epte_migration vaddr/nr_pages type target
Full-space Shared Environment
4.1
Coordinated Global Addressing
Target address range. Transfer type (promote-move/ promote-copy /demote). Nodes to execute the transfer.
During runtime, driven by the adaptive placement policy (§5), actively accessed pages are dynamically promoted to local memory, creating a hybrid backing store. To preserve global VA uniformity under this dynamic model, xDSM enforces a strictly coordinated allocation scheme. First, all local non-shared memory allocations are assigned from strictly disjoint VA ranges across nodes to prevent collisions. Second, any memory mapped to the CXL device is mapped to the exact same VAs globally. Consequently, a global unified address space is guaranteed, allowing native execution of unmodified applications. CXL Memory Management. Within the memory-mapped segment, xDSM reserves an xdsm_mapped region backed by CXL physical memory. xDSM implements a two-level memory allocator here. A global block allocator manages 64 MB blocks via a global bitmap and block_map, while a pernode allocator handles fine-grained allocations. Crucially, this region also hosts the globally shared xdsm_database (Table 2), which underpins the latency-driven placement policy (§5) and elastic page management (§6).
3.1.3 Insight 3: Spatial Locality-Aware Elasticity (§6). To resolve the fundamental trade-off between the high trap overhead of 4 KB base pages and the false sharing exacerbated by 2 MB huge pages (C3), xDSM overcomes this fixedgranularity limitation via spatial locality-aware elasticity. Instead of enforcing a static page size, xDSM dynamically merges contiguous pages with identical consistency states and the same accessors into variable-sized elastic pages (up to 2,MB) This background coalescing significantly amortizes fault resolution and migration overheads. Conversely, since a write fault on a shared large elastic page would invalidate the entire region and penalize unrelated sub-regions, xDSM performs on-demand splitting upon write faults, iteratively reducing the page size down to 4 KB. This approach isolates write-induced invalidations, eliminating false sharing penalties while retaining the efficiency of bulk migration.
4
Exclusive ownership flag. Owner node ID. Bitmask of nodes with modified copies. Bitmask of nodes holding local copies. Per-entry mutex.
4.2
Lightweight VMA-Anchored Page Transfers
Decoupled VMA and Page Tables. Each node maintains its own local Virtual Memory Area (VMA) structures and page tables. To ensure high performance when transferring data between tiers, xDSM introduces VMA-anchored fast paths. When promoting or demoting a page, xDSM updates only the underlying Page Table Entries (PTEs) to point to local memory, leaving the VMA unmodified. The VMA acts as an anchor pointing to the CXL backing store. This decoupling makes operations lightweight on the fast path because it fundamentally avoids expensive cross-node coordination. Specifically, a node does not need to wait for other nodes to unmap their regions before it can update its own mapping
Preserving a Unified Address Space. The fundamental goal of xDSM is to project a globally identical virtual address (VA) space across all distributed threads. At initialization, xDSM maps the entire application memory layout (including .text, .data, .bss, heap, memory-mapped, and stack) directly to the shared CXL device. This guarantees a uniform VA projection across all nodes at startup (Figure 1). 5
4.3
Bisect
Count
BinRange (Cycles) to local memory. Furthermore, this approach eliminates the Local Memory CXL Memory P90 P90 2 [32, 47] overhead of repeatedly freeing and re-allocating shared CXL CXL Local =2 100 4 [64, 95] (a) memory during dynamic tiering. Heavyweight VMA modifi6 [128, 191] (c) 0 cations are deferred to a slow path, invoked solely to detach 8 [256, 383] P90 P90 10 [512, 767] the CXL connection and reclaim remote capacity when a Local CXL =1 ×2 50 11 [768, 1023] page achieves long-term local residency. (b) 12 [1024, 1535] Fast-Path Page Transfers. At the end of each tick, the de0 13 [1536, 2047] 0 2 4 6 8 10 12 14 16 18 cision thread reads access statistics from epte_statistics 14 [2048, 3071] Bin index and consistency states from epte_status (② in Figure 1), then categorizes each elastic page into one of three fast-path Figure 2. Per-node latency histograms for local (blue) memtransfer actions, enqueued in epte_migration (③): ory and CXL (red) from Jacobi (§8.2). Dashed lines mark • Promote-Move: For pages with only one accessor (sinP90 bins. (a) CXL P90 exceeds local P90, triggering progle bit in accessor and hist_{ld,st}_accessor), xDSM motion. (b) After promotion saturates local DRAM, lomoves the pages from CXL to that node’s local memory cal P90 exceeds CXL P90, triggering demotion. (c) Bin-tofor exclusive access (private and owner will be set). latency mapping. Intervals [2𝑘 , 2𝑘+1 ) grow by powers of • Promote-Copy: For pages read by multiple nodes without two and each is bisected into two bins: e.g., bin 10 and writes (multiple bits in accessor with zero hist_st_accessor), bin 11 split the interval [512, 1024) into two equal-width xDSM replicates the pages to local memory of all readers bins [512, 767] and [768, 1023], while bin 12 begins the next as read-only copies (copyset will be set). interval [1024, 2048). • Demote: Local replicas on nodes under memory pressure are evicted back to CXL memory (private and copyset will be cleared). Thread Lifecycle. The runtime intercepts pthread_create Per-node migration threads dequeue these tasks (④) and to dispatch execution to remote nodes via a round-robin execute the data transfers via the kernel module (⑤). Crupolicy. Because the VA space is globally uniform, function cially, pages written by multiple nodes (multiple bits in pointers and arguments remain natively valid cluster-wide. hist_st_accessor) remain pinned in CXL memory to avoid Each thread is assigned a globally unique identifier (GUID), costly consistency traffic. which the runtime uses to transparently track and manage Transparent Access Resolution. Following page transfers, remote threads during operations such as pthread_join. xDSM relies on page faults to transparently resolve memory Synchronization. Standard synchronization primitives (e.g., accesses and enforce both Sequential Consistency (SC) [27] mutexes, spinlocks, barriers) are natively supported by aland Release Consistency (RC) [23]. The fault handler uses the locating their underlying objects directly within the shared block_map and epte_status to resolve accesses at elastic CXL memory. The user-space runtime intercepts these primia in Figure 1): granularity on the critical path (○ tives to seamlessly enforce the configured consistency model • Unmapped Fault: Triggered when accessing an unmapped without requiring source-code modifications. page. The handler resolves the mapping via block_map and establishes the VMA mapping. 5 Latency-Driven Page Placement • Read Fault: Triggered when reading an invalidated page, Effective page placement is critical to tiered-memory DSM e.g., the page is exclusively owned by another node. The performance. Existing hybrid systems often rely on rigid handler forces the owner to flush dirty data to CXL, then static rules [32] or a single value such as average access lagrants read-only (RO) accessor to intercept future writes. tency [31, 51] to guide data movement. Rigid static rules fail • Write Fault: Triggered when writing to a read-only page. to accommodate diverse access patterns. Average latency Under SC, the handler invalidates all remote copies (identimetrics are vulnerable to outliers in sub-𝜇s CXL environfied by copyset) before granting read-write (RW) accessor. ments, causing placement instability. To overcome these limUnder RC, it upgrades to RW locally and defers invalidaitations, xDSM employs an adaptive, latency-driven placetions until the next synchronization point. ment policy. This policy maintains a latency equilibrium To prevent race conditions when background page transbetween local memory and CXL memory. fers interleave with concurrent page fault handling, xDSM strictly coordinates these events using the fine-grained lock 5.1 Exp-Binned Latency Histograms embedded within each epte_status entry. To capture dynamic tier latencies and filter out long-tail anomalies, xDSM introduces Exp-binned Latency Histograms. Sampling threads (① in Figure 1) continuously record L3 load miss latencies into two per-node histograms (one for local memory and one for CXL). Because latencies within a
Application Thread Management
xDSM intercepts POSIX threading APIs to transparently support unmodified distributed execution. 6
workload exhibit high variance, the histograms categorize sampled latencies into exponentially growing intervals that compress outliers into fewer bins. Each interval is bisected into two equal-width bins to provide sufficient resolution for distinguishing local memory from CXL latency. For a latency sample ℓ (in cycles), its bin index is calculated by Equation 1: 0, 1, bin(ℓ) = ℓ − 2 ⌊log2 ℓ ⌋ , 2 ⌊log ℓ⌋ − 4 + 2 2 ⌊log2 ℓ ⌋ −1
the latency gap is large, effectively preventing thrashing and stabilizing the local working set.
1 ≤ ℓ ≤ 15, 16 ≤ ℓ ≤ 31, ℓ ≥ 32.
Spatial Locality-Aware Elasticity
Gap-Proportional Volume Scaling
Once the migration direction is determined, xDSM dynamically scales the migration volume in proportion to the measured P90-bin gap (Δ𝑖 = |cxl_bin𝑖 − local_bin𝑖 |). This gap reflects the severity of the performance imbalance. At each tick, the decision thread computes the migration volume for node 𝑖 by Equation 2. volume𝑖 =
cnt𝑖 2𝛼 −min(𝛼,Δ𝑖 )
6.2
Hierarchical Coalescing and Splitting
To track these variable-sized pages with low metadata overhead, xDSM maintains a hierarchical bitmap within each 2 MB chunk (Figure 1 (3)). Level 𝑒 manages aligned groups of 2𝑒 base pages, and a set bit indicates that the corresponding 2𝑒 pages form a single elastic page. For instance, if an entire 512-page chunk exhibits uniform access behavior, it coalesces to level 9, represented by a single bit and enabling 2 MB batch operations. Background Coalescing. At the end of each tick, immediately prior to making placement decisions (§5), the decision thread sweeps the bitmap to identify eligible sibling groups. If two adjacent level-𝑒 siblings meet the coalescing criteria above, they are coalesced into a level-(𝑒+1) group. The bitmap is atomically updated by clearing the two child bits and setting the parent bit. This proactively enlarges the management granularity for read-heavy or stable memory regions, optimizing them for bulk transfers. On-Demand Splitting. On-demand splitting is triggered upon a write access to a read-only replica. When a write fault occurs on a shared elastic page, the fault handler triggers a split. It iteratively splits the covering level-𝑒 group down to the 4 KB base granularity. At each division step, the parent consistency state is propagated to both children. However, the bitmap traversal only continues splitting the specific half encompassing the faulting address. Consequently, writeinduced invalidations are isolated to the targeted 4 KB page. This leaves the remainder of the original elastic page intact and valid for remote nodes, eliminating false sharing penalties while retaining the efficiency of bulk transfers.
(2)
Here, cnt𝑖 is the number of L3-miss samples observed on the source tier of node 𝑖 during the current tick, and 𝛼 is a scaling parameter that controls migration aggressiveness (§5.3). The gap Δ𝑖 shrinks the denominator, increasing migration volume until it saturates at cnt𝑖 when Δ𝑖 ≥ 𝛼. The decision thread then sorts elastic pages by per-tick access count (ld_cnt+st_cnt) in descending order, selecting pages for migration and decrementing volume𝑖 by their access counts until volume𝑖 reaches zero. 5.3
Elastic Page Management
6.1
To resolve the fundamental trade-off between high base-page fault processing overheads and severe huge-page false sharing, xDSM introduces spatial locality-aware elasticity. This mechanism dynamically coalesces contiguous base pages into variable-sized elastic pages ranging from 4 KB to 2 MB. By treating these coalesced regions as a single unified management entity, xDSM significantly amortizes the costs of fault resolution, permission updates, and page transfers. The runtime governs this elasticity by exploiting spatial locality. Two adjacent elastic pages are eligible for coalescing only if they share the same consistency state in epte_status (i.e., identical copyset, dirty bitmask, and owner, with no pending locks or in-flight transfers) and have the same accessors in epte_statistics: neither is written by multiple nodes, single-writer pages share same writer, and read-only pages share the same set of readers. Pages without any access can be merged with adjacent elastic pages.
(1) The placement policy uses P90-Bin Alignment as its criterion for latency equilibrium. At each tick, the decision thread computes the 90th-percentile (P90) bin index for local memory (local_bin𝑖 ) and CXL (cxl_bin𝑖 ) on each node 𝑖 (Figure 2). The P90 bin is the lowest index encompassing at least 90% of samples. If local_bin𝑖 < cxl_bin𝑖 , the policy promotes active pages to local memory. If local_bin𝑖 > cxl_bin𝑖 , it demotes pages to CXL. If the two align, latency equilibrium is achieved and no migration is triggered. 5.2
6
Asymmetric Migration Pacing
To address asymmetric CXL latencies and prevent placement oscillation, xDSM enforces Asymmetric Migration Pacing by configuring 𝛼 differently for promotion and demotion. For page promotion, xDSM acts aggressively by setting 𝛼 = 2 and using CXL sample count for cnt𝑖 in Equation 2. It rapidly moves hot pages to local memory to reduce CXL latency penalties. Conversely, for page demotion, xDSM acts conservatively by setting 𝛼 = 4 and using local memory sample count for cnt𝑖 . This higher 𝛼 intentionally throttles the eviction rate. It bounds the demotion volume even when 7
7
Implementation
systems. We use four NUMA nodes to emulate four independent compute nodes in a CXL-connected cluster, this setting aligned with existing studies [3, 17, 32]. Comparison systems. We compare 15 system configurations organized into three categories: Type 1: CXL-ONLY (baseline). All application data resides on the shared CXL device. Type 2: Local-only DSM. Application data is allocated on local DRAM and consistency is maintained through software protocols, using CXL only as the communication channel (replacing Ethernet/RDMA). This category includes four systems adapted to our CXL platform: SC (Sequential Consistency, IVY [27]), RC (Release Consistency, TreadMarks [23]), SWAP (a page-swapping DSM that migrates full pages on access, similar to Fastswap [4]), and DRust-C (a C reimplementation of DRust’s [34] ownership-based protocol). Type 3: Hybrid (local + CXL). These systems simultaneously use local DRAM and CXL memory with different placement strategies and page granularities. We use the naming convention H[consistency]-[strategy]-[granularity], where H stands for Hybrid, consistency is RC or SC, strategy is S1/S2/D (explained below), and granularity is F (fixed 4 KB page) or E (elastic 4 KB–2 MB). • Strategy 1 (S1): A static rule that classifies pages as shared or private. Shared pages remain on CXL, while private pages are promoted to local DRAM. This yields HSC-S1-F and HSC-S1-E. • Strategy 2 (S2): A finer static rule that further distinguishes read vs. write. Private read-write pages and shared read-only pages are promoted to local DRAM, while shared read-write pages remain on CXL. This yields HSC-S2-F, HSC-S2-E, HRC-S2-F, and HRC-S2-E. • Strategy 3 (D): The latency-driven dynamic placement (§5). This yields HSC-D-F, HRC-D-F, and the two xDSM configurations: xDSM-SC (= HSC-D-E) and xDSM-RC (= HRC-D-E). Type 3 systems are compared against Type 1 and Type 2 to demonstrate the benefit of hybrid memory utilization. The comparison of S1, S2, and D shows the effect of different placement strategies, while the comparison of F and E shows the effect of elastic page management.
We implemented xDSM on Linux 6.3 (x86-64) with ∼23,800 lines of C code: ∼21,400 in the user-space runtime (libxdsm. so) and ∼2,400 in the kernel module (/dev/xdsm). An additional ∼8,300 LOC implement other systems for comparison. All source code will be open-sourced upon publication. User-Space Runtime. Injected via LD_PRELOAD, the runtime intercepts __libc_start_main, POSIX threading APIs, and memory allocators. At startup, it parses /proc/self/ maps, reads existing pages via process_vm_readv and writes them to shared CXL memory, and logs VA-to-CXL mappings so remote nodes can reconstruct the same address space. Cross-node coordination leverages per-thread CXL ring buffers. The runtime also implements both RC and SC. Hardware Profiling. Each node’s sampling thread profiles memory accesses via AMD IBS [12], opening one perf_event descriptor per user thread (l3missonly mode) and polling ring buffers to harvest latency samples with low overhead. Kernel-Level Page Management. The module exposes an ioctl interface for fast-path memory operations that modify only Page Table Entries (PTEs), leaving VMAs intact. The user-space runtime resolves all data races before these calls. • Promotion (xdsm_promote_sharedpages): It has three stages: snapshotting file-backed PTEs, allocating local anonymous pages to copy from CXL (copy_mc_user_highpage), and validating PTEs are unchanged since the snapshot before batch-updating mappings. Both the snapshot and update stages process PTEs in batches under Page Table Locks (PTLs), with the final stage issuing a single flush_tlb_range for the entire batch. • Demotion (xdsm_demote_sharedpages): It discards clean local pages. Otherwise, it writes back to the CXL pages via the VMA’s file mapping (filemap_grab_folio) and frees memory using MADV_DONTNEED.
8
Evaluation
We evaluate xDSM along four dimensions: (1) end-to-end application performance across 15 configurations, (2) periteration breakdown to evaluate the impact of placement strategy and page granularity separately, (3) dynamic workload adaptation, and (4) scalability from 1 to 4 nodes. 8.1
Experimental Setup
8.2
Hardware. All experiments run on a dual-socket AMD EPYC server with Linux 6.3. We enable NPS2 (NUMA-Per-Socket = 2) in BIOS, which partitions each socket into two NUMA domains, yielding four NUMA nodes. Each node has 8 physical cores and 4 GB of local DDR5 DRAM (16 GB aggregated). A 64 GB CXL 1.1 Type-3 memory expander is attached and is accessible from all four NUMA nodes as a shared DAX device. Although CXL 3.0 multi-headed devices are not commercially available, CXL 1.1 supports hardware cache coherence within a single machine, satisfying the requirements of our
Benchmarks and Datasets
We select five applications with diverse access patterns: • PageRank [5]: Iterative graph ranking over an adjacency matrix. Each thread computes ranks for a partition of vertices, reading the shared matrix and writing private rank vectors. Access pattern: dominated by private read-write to rank arrays with read-only sharing of the matrix. • Jacobi [40]: Iterative 2D stencil relaxation. Threads operate on horizontal strips and synchronize via barriers each iteration. Access pattern: predominantly private read-write with boundary sharing between adjacent strips. 8
200
0
0
(b)
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
0
Black-Scholes (15 GB)
1198 973 492.1k 1163
Streamcluster (12.9 GB)
200 0
400 200 0
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
2500
Graph500 (15 GB)
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
400
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
0
600
5000
121.1k
2830
0
(a)
200
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
0
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
0
7500 100
1803 1028 227.1k 1334
8902
0
Jacobi (13 GB, 40000 iters)
xDSM-SC xDSM-RC Black-Scholes (6.0 GB)
400 200
Jacobi (13 GB, 200 iters)
50 Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
Time (s)
100
HRC-S2-F HSC-D-F HRC-S2-E HRC-D-F Streamcluster (6.0 GB)
200
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
PageRank (~15.5 GB, 200 iters)
HSC-S2-F HSC-S2-E Graph500 (7.5 GB)
2500 Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
0
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
Typ e1 Typ e2 Typ e3 Typ S1 e3 -S2 Typ e3 -D
0
400
5000
50
20
Jacobi (6.0 GB, 40000 iters)
11.0k
1495
Time (s)
40
HSC-S1-F HSC-S1-E 7310
SC SWAP RC DRust-C Jacobi (6.0 GB, 200 iters)
CXL-ONLY PageRank (6.7 GB, 200 iters)
Figure 3. End-to-end execution time.(a) 16 threads (2 nodes), small datasets. (b) 32 threads (4 nodes), medium datasets. • Graph500 [38]: Breadth-first search on a large random graph stored in compressed sparse row (CSR) format. Multiple BFS traversals run in parallel from independent roots, with each thread reading the shared graph structure and maintaining a private visited set and queue. Access pattern: dominated by shared read-only traversal of the CSR arrays with irregular, pointer-chasing access. We use the BFS kernel of Graph500 with our own pthread-based implementation. • Streamcluster [8]: An online clustering algorithm from the PARSEC benchmark suite. Threads cooperatively compute facility-location costs over a shared point set, with a mix of shared reads (point coordinates) and private writes (membership updates, cost accumulators). Access pattern: mixed private read-write and shared read. • Blackscholes [8]: European option pricing from the PARSEC benchmark suite. Threads independently compute prices for disjoint partitions of the option array with no inter-thread sharing. Access pattern: compute-intensive with small memory pressure. Dataset Scales. We consider three dataset scales: Small datasets (4–8 GB) for 2-node configurations. Medium datasets (8–16 GB) for 4-node configurations. Large datasets (>16 GB), exceeding four nodes’ aggregated local DRAM, thus requiring CXL memory. The data capacity of traditional DSM systems is strictly bounded by the aggregate local memory across all nodes. Consequently, Type 2 systems cannot execute workloads with large datasets.
8.3
Hybrid (Type 3) vs. CXL-only (Type 1). Across all memoryintensive benchmarks, the best hybrid configuration (xDSMRC or xDSM-SC) outperforms CXL-ONLY by 1.5 × –2.2×. At 2 nodes, xDSM reduces execution time by 49% on PageRank, 36% on Jacobi, 38% on Streamcluster, and 32% on Graph500. Results of 4 nodes show even more advantage: 55% on PageRank, 38% on Jacobi, 41% on Streamcluster, and 46% on Graph500. Hybrid systems place hot pages on local DRAM with lower latency, while CXL-ONLY forces all accesses through higherlatency CXL links. Hybrid (Type 3) vs. local-only DSM (Type 2). xDSM-RC outperforms the best Type 2 system by 1.8× on PageRank and by 8× on Streamcluster at 4 nodes. SWAP suffers from page thrashing on shared pages, as each shared page bounces between the two accessing nodes, making it orders of magnitude slower on Streamcluster. The hybrid approach avoids this overhead: shared-read pages are replicated locally, and only write-contended pages incur consistency costs. Strategy comparison (S1, S2, D). Within Type 3, more sophisticated placement strategies yield consistent gains. Comparing at fixed-page granularity (-F) on 2 nodes: S1 places shared pages on CXL indiscriminately, so HSC-S1-F performs close to CXL-ONLY. S2 improves by promoting shared-read pages to local DRAM, reducing execution time by 6%–29% over S1. The dynamic strategy D further improves over S2 for long-running workloads: on Jacobi with 40 000 iterations, D is 12.1% faster than S2, because the latencydriven policy adapts to phase changes that static rules miss. Granularity comparison (F vs. E) at fixed strategy. At every strategy level, elastic groups (-E) outperform fixed pages (-F). The benefit is most pronounced for PageRank: at S2 on 2 nodes, -E is 37% faster than -F. At D on 4 nodes, xDSM-RC is 50% faster than HRC-D-F. Jacobi shows a similar pattern: 31% faster at both 2 and 4 nodes. The advantage comes from coalescing: PageRank and Jacobi access large
Application Performance
Figure 3 compares the end-to-end execution time of all 15 system configurations on 2 nodes (16 threads, small datasets) and 4 nodes (32 threads, medium datasets), respectively. We analyze the results along four dimensions. 9
BFS: shared read-only. Multiple threads perform BFS on a shared CSR graph. Each round launches all threads in parallel, and each thread traverses from a distinct root. The graph arrays are read-only and accessed with irregular, pointerchasing patterns. Figure 5 reports TEPS (traversed edges per second) per round. Type 2 systems behave alike, so we show only SC. HSC-S2 performs the same as HSC-D in this workload, so we show only HSC-D. Type 1 (CXL-ONLY) is flat at ∼2.4 × 108 TEPS. All graph data sits on CXL, so every pointer chase pays CXL latency. Type 2 (SC) reaches ∼3.3 × 108 TEPS at steady state, 37% above CXL-ONLY. The graph is read-only, so SC replicates it into each node’s local DRAM without write invalidations. Type 3, strategy comparison (S1 vs. S2/D). We fix elastic granularity and compare HSC-S1-E against xDSM-SC. HSCS1-E stays flat at ∼2.6 × 108 TEPS, nearly identical to CXLONLY, because S1 classifies the graph as “shared” and keeps it on CXL, unable to recognize that shared-read pages would benefit from local replication. xDSM-SC reaches ∼3.7 × 108 TEPS by round 100 and stays stable, 47% above HSC-S1-E. Both S2 and D detect that the graph pages are read-only and replicate them to local DRAM, reducing access latency without incurring write invalidations. This also explains why HSC-S2 performs the same as HSC-D in this workload. Type 3, granularity comparison (fixed vs. elastic). We fix strategy D and compare HSC-D-F against xDSM-SC. HSCD-F climbs to ∼3.3 × 108 TEPS by round 1 000, but after round 1 500 its throughput oscillates between 2.7 × 108 and 3.7 × 108 TEPS, because at 4 KB granularity the profiler is more likely to misclassify shared pages as private, causing pages to migrate back and forth between nodes. xDSM-SC reaches ∼3.7 × 108 TEPS by round 100 and stays stable, 16% above Type 2. Elastic pages coalesce the shared graph into large management units, providing more accurate page classification and eliminating the oscillation seen in HSC-D-F.
contiguous arrays, so elastic groups batch up to 512 pages per fault, cutting fault count by two orders of magnitude. Blackscholes is compute-bound and insensitive to memory placement: Type 1 and Type 3 configurations perform within 5% of each other, while Type 2 systems incur ∼12% overhead from software consistency. 8.4
Performance Breakdown
To understand the effects of different configurations, we examine per-iteration throughput on two microbenchmarks with complementary access patterns: an iterative program (private read-write, results in Figure 4) and BFS (shared readonly, results in Figure 5). Together, these two patterns cover the dominant access behaviors across all five applications. Both run 3 000 iterations (rounds) on 2 nodes with 16 threads. Iterative program: private read-write. Each thread reads and writes a disjoint partition of a large array, separated by barriers. We select six configurations for analysis (Figure 4). Type 2 systems behave alike (DRust-C differs only in a faster first iteration), so we show only SC. For Type 3, the two consistency protocols (sequential and release consistency) show no difference, so we use sequential consistency. HSCS1 and HSC-S2 perform the same because all data is private, so both strategies promote all pages to local DRAM. We show only HSC-S1. Type 1 (CXL-ONLY) delivers a flat ∼50 GB/s. All data resides on CXL, so there is no warm-up, no access to faster local DRAM. Type 2 (SC) starts at <1 GB/s because the array is initialized on the main thread’s node, so worker threads on the other node trigger page faults on every first access. After ∼50 iterations, pages settle into local DRAM and the throughput stabilizes at ∼62 GB/s, faster than CXL-ONLY but capped by local DRAM bandwidth. Type 3, strategy comparison (S1/S2 vs. D). We fix elastic granularity and compare HSC-S1-E (S1) against xDSM-SC (D). HSC-S1-E peaks at ∼77 GB/s around iteration 50 when pages are split between CXL and local DRAM, then drops to ∼62 GB/s as S1 pulls all pages to local memory, losing the additional CXL bandwidth. xDSM-SC reaches ∼79 GB/s by iteration 50 and sustains ∼78 GB/s throughout all 3,000 iterations, 27% above HSC-S1-E at steady state. This advantage stems from strategy D maintaining a balanced split between CXL and local DRAM, exploiting bandwidth from both tiers. Type 3, granularity comparison (fixed vs. elastic). We fix strategy D and compare HSC-D-F against xDSM-SC. HSCD-F starts at ∼51 GB/s and slowly climbs to ∼79 GB/s by iteration 1 000, because each page is tracked and migrated individually at 4 KB granularity. xDSM-SC reaches the same ∼79 GB/s by iteration 50, converging 20× faster because elastic pages coalesce contiguous regions (including unaccessed pages) into single management units, migrating up to 512 pages at once. Both converge to 27% above Type 2 and 58% above CXL-ONLY, confirming that elastic granularity accelerates warm-up without affecting placement quality.
Summary. Strategy D adapts placement to access patterns and maintains stable performance via latency-driven migration, outperforming static rules in both workloads. Elastic granularity enables faster migration and more accurate page classification than fixed pages. xDSM combines both for the best throughput and fastest convergence. 8.5
Dynamic Workload
We run LLM inference using Qwen3-30B-A3B [48], a Mixtureof-Experts (MoE) model with 128 experts per layer. Each token activates only a small subset of experts, and the activated subset changes across tokens and requests, making the memory access pattern inherently dynamic. The int8-quantized model is approximately 37 GB, exceeding the 16 GB aggregate local DRAM of four nodes and requiring CXL memory. We replay a sequence of requests from the Azure LLM Inference Trace [50] serially on 4 nodes. Figure 6 shows the causal chain from top to bottom. The top panel shows which experts 10
Throughput (GB/s)
80 60 40 20 0
Type1 CXL-Only
0
Type2 SC
1500
3000
Iteration
0
Type3-S1 HSC-S1-F
1500
3000
Iteration
0
Type3-S1 HSC-S1-E
1500
3000
Iteration
0
1500
Iteration
Type3-D HSC-D-F
3000
0
Type3-D xDSM-SC
1500
3000
Iteration
0
1500
3000
Iteration
Figure 4. Per-iteration throughput for the iterative program (private read-write, 4.1 GB, 16 threads / 2 nodes).
TEPS
1e8
Type 1 CXL-Only
Type 2 SC
Type 3 Strategy 1 (S1) HSC-S1-F
Type 3 Strategy 1 (S1) HSC-S1-E
0
0
Type 3 Strategy 3 (D) HSC-D-F
Type 3 Strategy 3 (D) xDSM-SC
2 0
0
1500
Round
3000
0
1500
Round
3000
1500
3000
Round
1500
Round
3000
0
1500
3000
Round
0
1500
3000
Round
Figure 5. Per-round TEPS (traversed edges per second) for BFS (shared read-only, 7.46 GB, 16 threads / 2 nodes).
E0 1e5 1.0 0.5
Colloid+
(a) Overall Performance 1.0 0.5 0.0
rank 200 40k ster oles 500 PageJacobi-JacSotrbei-amBclaluckschGraph
0.0
Alto+Colloid+
xDSM-SC
(b) Iterative Program Breakdown
Throughput (GB/s)
Migrations (pages) TPOT (ms)
Memtis+
8 4 2 1 0
0.97 0.52 0.49 0.97 0.67 0.68 0.91 0.90 0.90 0.61 0.61 0.64 1.00 1.00 1.00 0.84 0.84 0.84
xDSM-SC
Expert
HSC-D-F
Normalized Performance (xDSM-SC = 1)
CXL-ONLY
E6143
80 70 60
50 Memtis+ Colloid+ Alto+Colloid+ 0
3k 0
3k 0
Iteration
3k
Figure 7. Placement policy comparison on 4 nodes (32 threads). (a) Normalized performance relative to xDSM-SC (higher is better). (b) Per-iteration throughput breakdown for the iterative program (3,000 iterations).
100 0
50
100
150
200
Wall-clock time (s)
250
300
Figure 6. LLM inference (Qwen3-30B-A3B) driven by Azure traces. Top: expert activation heatmap, where darker color indicates higher activation rate. Middle: page migration per second. Bottom: TPOT over wall-clock time.
with lower per-page overhead than fixed granularity. Thus, xDSM-SC migrates 3.6 M pages in total versus 1.9 M for HSCD-F (Figure 6, middle), adapting to activation shifts faster. Goodput. Following [2, 58], we set a TPOT SLA of 100 ms and define goodput as the number of output tokens that meet this SLA per second. CXL-ONLY meets the SLA for only 58.7% of tokens, yielding a goodput of 5.77 tok/s. HSC-D-F reaches 90.5% SLA attainment and a goodput of 9.77 tok/s. xDSM-SC reaches 91.1% and a goodput of 11.00 tok/s, 1.13× higher than HSC-D-F and 1.91× higher than CXL-ONLY.
are activated over time (each row is one expert, darker means more activations). Tracing any single row horizontally, an expert is not activated continuously but intermittently, meaning the working set of weight matrices shifts over time. The middle panel shows how each system responds with page migration, and the bottom panel shows the resulting Time Per Output Token (TPOT): a system that migrates fast enough to track the shifting activations achieves lower TPOT. TPOT. CXL-ONLY keeps all weights on CXL, yielding a flat TPOT of 101.7 ms. Both HSC-D-F and xDSM-SC batchmigrate hot expert pages to local memory, but differ in granularity. HSC-D-F migrates at fixed granularity, reaching 92.6 ms (1.10× faster). xDSM-SC migrates at elastic granularity, reaching 82.8 ms (1.23× faster than CXL-ONLY, 1.12× faster than HSC-D-F). Expert weight matrices are large contiguous regions. Elastic pages continuously migrate pages
8.6
Placement Policy Comparison
We compare xDSM-SC against three state-of-the-art placement policies adapted to our batch migration runtime. Memtis+ [25] leverages a page access histogram to promote the hottest 2 MB huge pages locally. Colloid+ [51] utilizes a binary search algorithm to balance average loaded access latency between memory tiers. Finally, Alto+Colloid+ [31] builds upon Colloid+ by integrating Amortized Offcore Latency to adjust migration intensity according to memorylevel parallelism. 11
2
Node
3
4
200 100 1
2
Node
3
xDSM-RC
4
Blackscholes (18 GB)
Time (s)
100
xDSM-SC
Streamcluster (~16 GB)
Time (s)
Time (s)
Time (s)
200 1
HRC-D-F
Jacobi (18.26 GB)
400 200 1
2
Node
3
4
Graph500 (20.9 GB)
1500
Time (s)
HSC-D-F
PageRank (25.6 GB)
1000 500
1
2
Node
3
4
600 400 200
1
2
Node
3
4
Figure 8. Scalability from 1 to 4 nodes (8 to 32 threads) across five benchmarks with large datasets. Figure 7 shows the results. Memtis+ is competitive on short, private-RW-dominated benchmarks (97% on PageRank and Jacobi-200), because its 2 MB huge pages adapt quickly. On longer workloads, Memtis+ greedily fills local DRAM, losing CXL bandwidth (Figure 7b), 10% slower than xDSMSC on Jacobi-40K. Memtis+ also lacks shared page management, 1.63× slower on Streamcluster and 1.19× on Graph500. Colloid+ and Alto+Colloid+ rely on the average loaded access latency to guide placement, reaching only 49–52% on PageRank and 67–68% on Jacobi-200. Their binary-search algorithm migrates few pages per step when the latency gap is large, converging slowly. The average latency also fluctuates between ticks, causing pages to migrate back and forth. xDSM addresses these with an exponential budget that scales migration intensity with the latency gap, and per-node P90 latency histograms that are less sensitive to latency fluctuations. Like Memtis+, Colloid+ and Alto+Colloid+ also lack shared page management. Blackscholes is compute-bound and all configurations perform within 1% variance. 8.7
16% for Graph500, and 12% for Streamcluster. Blackscholes performs within 0.3% across all configurations.
9
Discussion
The performance gains from dynamic page placement are fundamentally tied to the latency disparity between local and remote memory. Our NUMA-based emulation exhibits a 2×– 4× latency gap, under which our policy already demonstrates substantial efficacy. In actual CXL 3.0 deployments, remote accesses traverse CXL switches [44], introducing network latencies absent in our emulation. Because a wider latency gap penalizes remote accesses more severely, the benefits of promoting hot pages to local memory will increase on physical CXL hardware. Currently, the eptp_database resides in globally shared CXL memory, operating under a trusted-environment assumption. Consequently, a compromised node could maliciously read or tamper with the mapping metadata of other nodes. Hardening this shared state against adversarial access is left as future work, which could be addressed by integrating hardware-enforced memory protection mechanisms, such as Memory Protection Keys (MPK) [19].
Scalability
We run all five benchmarks with large datasets (16–25.6 GB) that exceed the aggregate local DRAM of all four nodes (16 GB). At this scale, Type 2 local-only DSM systems cannot run: the working set does not fit in local DRAM even when all nodes participate. The hybrid configurations (Type 3), which leverage the 64 GB CXL shared memory pool, can execute these workloads. Figure 8 shows performance from 1 node (8 threads) to 4 nodes (32 threads). Near-linear scaling. The memory-intensive benchmarks scale well: PageRank reaches 5.1×, while Jacobi reaches 4.0×, Graph500 3.8×, and Streamcluster 3.2×. Blackscholes, being compute-bound, also achieves 3.5× speedup at 4 nodes. PageRank achieves superlinear scaling (5.1× on 4 nodes) by leveraging increased aggregate local DRAM. With a 25.6 GB dataset, a single node’s 4 GB memory forces most data onto the CXL tier. Expanding to 4 nodes provides 16 GB of local capacity, shifting data locally and fully utilizing local bandwidth. Consequently, steady-state throughput jumps massively from ∼21 GB/s on 1 node to ∼117 GB/s on 4 nodes. Additionally, the elastic variants (xDSM-SC/RC) outperform their fixed-page counterparts (HSC/HRC-D-F), and the advantage grows with node count: at 4 nodes, elastic groups reduce execution time by 39% for PageRank, 30% for Jacobi,
10
Related Work
Transparent Execution Environment. GiantVM [56] and vSMP Foundation [49] aggregate multiple physical machines into a single virtual machine to enable the execution of unmodified applications across multiple nodes. Fundamentally, these systems still rely on traditional software consistency protocols [27]. Consequently, they share the same performance limitations as the Type 2 systems evaluated in our study (§8.3). Furthermore, our analysis reveals that in sub-𝜇s CXL environments, the specific choice of software-based consistency protocol [6, 7, 23, 24, 27, 34, 37, 39, 42, 60] has a limited impact on overall performance. MIND [43] also provides a global shared address space, but builds it on a different hardware substrate, an in-network programmable switch, rather than the CXL-attached shared memory that xDSM targets. Page Placement Policies. Tiered memory systems (Type 3) allow direct access to both local and remote (CXL) memory. They migrate pages either based on frequency [13, 25, 36, 41, 54, 57] or latency [31, 51]. Frequency-based systems identify hot pages and try to promote them to local memory but 12
may lead to local contention. Latency-based systems avoid such contention. xDSM improves these policies by using the exp-binned histograms and maintaining a latency equilibrium between local and remote memory(§8.6). Far memory systems (Type 2) [11, 14, 15, 29, 47, 59] use remote memory solely as a backing tier, forcing all accessed pages into local memory before execution. When applied to CXL, this approach squanders the ability to access local and remote memory simultaneously. xDSM operates as a Type 3 system to avoid this limitation. Adaptive Sharing Granularity. While AdaptableView [20] also recognizes the performance impact of sharing granularity, it relies heavily on manual programmer annotations to define sharing views for specific variables and execution phases. In contrast, xDSM demonstrates a highly feasible automated solution. By transparently tracking runtime access patterns, xDSM dynamically adapts the management granularity on the fly, completely eliminating the need for source code modifications. MIND [43] also adopts variablesized regions, but they define the granularity of its coherence directory, while data is transferred at a fixed 4 KB page. In xDSM, the elastic page is itself the unit of transfer, coalesced and split by access pattern and consistency state. Other CXL-based Systems. Pond [26] pools CXL memory to avoid stranded capacity, focusing on allocation strategy and performance prediction. Telepathic [35] shares native pointers across nodes to bypass serialization, focusing on memory access isolation. TrEnv [16] and CXLfork [3] share process snapshots across nodes, focusing on page table manipulation and copy-on-write. Tigon [17] synchronizes transactions on CXL shared memory, focusing on locking protocols co-designed with coherence boundaries. xDSM targets general-purpose distributed shared memory over CXL, focusing on page placement and elastic granularity.
11
Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with sarathi-serve. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 7, 18 pages. [3] Chloe Alverti, Stratos Psomadakis, Burak Ocalan, Shashwat Jaiswal, Tianyin Xu, and Josep Torrellas. 2025. Cxlfork: Fast remote fork over cxl fabrics. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 210–226. [4] Emmanuel Amaro, Christopher Branner-Augmon, Zhihong Luo, Amy Ousterhout, Marcos K. Aguilera, Aurojit Panda, Sylvia Ratnasamy, and Scott Shenker. 2020. Can Far Memory Improve Job Throughput?. In Proceedings of the Fifteenth European Conference on Computer Systems (EuroSys ’20). ACM, 1–16. [5] Scott Beamer, Krste Asanović, and David Patterson. 2015. The GAP Benchmark Suite. arXiv preprint arXiv:1508.03619 (2015). [6] J. K. Bennett, J. B. Carter, and W. Zwaenepoel. 1990. Munin: Distributed Shared Memory Based on Type-Specific Memory Coherence. In Proceedings of the 2nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). ACM, 168–176. [7] Brian N. Bershad, Matthew J. Zekauskas, and Wayne A. Sawdon. 1993. The Midway Distributed Shared Memory System. In Proceedings of the IEEE CompCon. IEEE, 528–537. [8] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC Benchmark Suite: Characterization and Architectural Implications. In Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques (PACT). ACM, 72– 81. [9] Irina Calciu, M. Talha Imran, Ivan Puddu, Sanidhya Kashyap, Hasan Al Maruf, Onur Mutlu, and Aasheesh Kolli. 2021. Rethinking software runtimes for disaggregated memory. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Virtual, USA) (ASPLOS ’21). Association for Computing Machinery, New York, NY, USA, 79–92. [10] Barbara Chapman, Tony Curtis, Swaroop Pophale, Stephen Poole, Jeff Kuehn, Chuck Koelbel, and Lauren Smith. 2010. Introducing OpenSHMEM: SHMEM for the PGAS Community. In Proceedings of the 4th Conference on Partitioned Global Address Space Programming Model (PGAS’10). ACM, Article 2, 3 pages. [11] Lei Chen, Shi Liu, Chenxi Wang, Haoran Ma, Yifan Qiao, Zhe Wang, Chenggang Wu, Youyou Lu, Xiaobing Feng, Huimin Cui, Shan Lu, and Harry Xu. 2024. A Tale of Two Paths: Toward a Hybrid Data Plane for Efficient Far-Memory Applications. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 77–95. [12] Paul J. Drongowski. 2007. Instruction-Based Sampling: A New Performance Analysis Technique for AMD Family 10h Processors. Technical Report. Advanced Micro Devices, Inc. AMD Technical Report. [13] Padmapriya Duraisamy, Wei Xu, Scott Hare, Ravi Rajwar, David Culler, Zhiyi Xu, Jianing Fan, Christopher Kennelly, Bill McCloskey, Danijela Mijailovic, Brian Morris, Chiranjit Mukherjee, Jingliang Ren, Greg Thelen, Paul Turner, Carlos Villavieja, Parthasarathy Ranganathan, and Amin Vahdat. 2023. Towards an Adaptable Systems Architecture for Memory Tiering at Warehouse-Scale. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (Vancouver, BC, Canada) (ASPLOS 2023). Association for Computing Machinery, New York, NY, USA, 727–741. [14] Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin. 2017. Efficient Memory Disaggregation with Infiniswap. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI). 649–667. [15] Zhiyuan Guo, Zijian He, and Yiying Zhang. 2023. Mira: A ProgramBehavior-Guided Far Memory System. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23).
Conclusion
xDSM seamlessly scales multi-threaded applications across CXL-connected nodes. By co-designing a kernel module and user-space runtime, it delivers a full-space shared environment, latency-driven dynamic page placement, and elastic page management. Evaluations show that xDSM outperforms existing systems, adapts to dynamic workloads, and achieves near-linear scalability.
References [1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 265–283. [2] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran 13
Association for Computing Machinery, New York, NY, USA, 692–708. [16] Jialiang Huang, MingXing Zhang, Teng Ma, Zheng Liu, Sixing Lin, Kang Chen, Jinlei Jiang, Xia Liao, Yingdi Shan, Ning Zhang, et al. 2024. Trenv: Transparently share serverless execution environments across different functions and nodes. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 421–437. [17] Yibo Huang, Haowei Chen, Newton Ni, Yan Sun, Vijay Chidambaram, Dixin Tang, and Emmett Witchel. 2025. Tigon: A Distributed Database for a CXL Pod. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA. [18] IEEE. 2017. IEEE Standard for Information Technology—Portable Operating System Interface (POSIX). Technical Report IEEE Std 1003.1-2017. IEEE. [19] Intel Corporation. 2024. Intel 64 and IA-32 Architectures Software Developer’s Manual. Intel. Vol. 3A, Protection Keys for User Pages. [20] Ayal Itzkovitz, Nitzan Niv, and Assaf Schuster. 2000. Dynamic Adaptation of Sharing Granularity in DSM Systems. The Journal of Systems and Software 55 (2000), 19–32. [21] Sunita Jain, Nagaradhesh Yeleswarapu, Hasan Al Maruf, and Rita Gupta. 2024. Memory Sharing with CXL: Hardware and Software Design Approaches. arXiv preprint arXiv:2404.03245 (2024). [22] Pete Keleher, Alan L. Cox, and Willy Zwaenepoel. 1992. Lazy Release Consistency for Software Distributed Shared Memory. In Proceedings of the 19th Annual International Symposium on Computer Architecture (ISCA). ACM, Gold Coast, Australia, 13–21. [23] Pete Keleher, Sandhya Dwarkadas, Alan L. Cox, and Willy Zwaenepoel. 1994. TreadMarks: Distributed Shared Memory on Standard Workstations and Operating Systems. In Proceedings of the USENIX Winter Technical Conference. USENIX Association, San Francisco, CA, 115– 131. [24] Sang-Hoon Kim, Ho-Ren Chuang, Robert Lyerly, Pierre Olivier, Changwoo Min, and Binoy Ravindran. 2020. DeX: Scaling Applications Beyond Machine Boundaries. In Proceedings of the 40th IEEE International Conference on Distributed Computing Systems (ICDCS). IEEE, 864–876. [25] Taehyung Lee, Sumit Kumar Monga, Changwoo Min, and Young Ik Eom. 2023. MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size Determination. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 17–34. [26] Huaicheng Li, Daniel S Berger, Lisa Hsu, Daniel Ernst, Pantea Zardoshti, Stanko Novakovic, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, et al. 2023. Pond: Cxl-based memory pooling systems for cloud platforms. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 574–587. [27] Kai Li. 1988. IVY: A Shared Virtual Memory System for Parallel Computing. In Proceedings of the International Conference on Parallel Processing. 94–101. [28] Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su. 2014. Scaling Distributed Machine Learning with the Parameter Server. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 583–598. [29] Quanxi Li, Hong Huang, Ying Liu, Yanwen Xia, Jie Zhang, Mosong Zhou, Xiaobing Feng, Huimin Cui, Quan Chen, Yizhou Shan, and Chenxi Wang. 2025. Beehive: A Scalable Disaggregated Memory Runtime Exploiting Asynchrony of Multithreaded Programs. In Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI). 167–187. [30] Linux Kernel Developers. 2024. Concepts overview — The Linux Kernel documentation. https://docs.kernel.org/admin-guide/mm/concepts. html.
[31] Jinshu Liu, Hamid Hadian, Hanchen Xu, and Huaicheng Li. 2025. Tiered memory management beyond hotness. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 40, 17 pages. [32] Shi Liu. 2025. Adaptive Memory Management for MemoryDisaggregated Data Centers. Ph. D. Dissertation. University of California, Los Angeles. Chapter 4: Firework: Efficient and Transparent Compute Scaling with CXL Shared Memory. [33] Yucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin, and Joseph Hellerstein. 2012. Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud. arXiv preprint arXiv:1204.6078 5, 8 (2012), 716–727. [34] Haoran Ma, Yifan Qiao, Shi Liu, Shan Yu, Yuanjiang Ni, Qingda Lu, Jiesheng Wu, Yiying Zhang, Miryung Kim, and Harry Xu. 2024. DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’24). USENIX Association, 97–115. [35] Suyash Mahar et al. 2024. Telepathic Datacenters: Fast RPCs using Shared CXL Memory. arXiv preprint arXiv:2408.11325 (2024). [36] Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit Kanaujia, and Prakash Chauhan. 2023. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, Vancouver, BC, Canada, 742–755. [37] Frank Mueller. 1997. Distributed shared-memory threads: DSMThreads. In Workshop on Run-Time Systems for Parallel Programming. 31–40. [38] Richard C. Murphy, Kyle B. Wheeler, Brian W. Barrett, and James A. Ang. 2010. Introducing the Graph 500. In Cray User’s Group (CUG). [39] Jacob Nelson, Brandon Holt, Brandon Myers, Preston Briggs, Luis Ceze, Simon Kahan, and Mark Oskin. 2015. Latency-Tolerant Software Distributed Shared Memory. In Proceedings of the 2015 USENIX Annual Technical Conference (ATC). USENIX Association, 291–305. [40] Louis-Noël Pouchet and Tomofumi Yuki. 2016. PolyBench/C 3.2: The Polyhedral Benchmark Suite. https://www.cs.colostate.edu/~pouchet/ software/polybench/. [41] Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Erez, and Simon Peter. 2021. HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (Virtual Event, Germany) (SOSP ’21). Association for Computing Machinery, New York, NY, USA, 392–407. [42] Daniel J. Scales and Kourosh Gharachorloo. 1997. Towards Transparent and Efficient Software Distributed Shared Memory. In Proceedings of the 16th ACM Symposium on Operating Systems Principles (SOSP). ACM, 157–169. [43] Seung seob Lee, Yanpeng Yu, Yupeng Tang, Anurag Khandelwal, Lin Zhong, and Abhishek Bhattacharjee. 2021. MIND: In-Network Memory Management for Disaggregated Data Centers. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP). ACM, 488–504. [44] Debendra Das Sharma and Ishwar Agarwal. 2022. Compute Express Link (CXL) 3.0 Specification. Technical Report. CXL Consortium. https://computeexpresslink.org/wp-content/uploads/2023/12/ CXL_3.0_white-paper_FINAL.pdf [45] Julian Shun and Guy Blelloch. 2013. Ligra: A Lightweight Graph Processing Framework for Shared Memory. In Proceedings of the 18th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP). ACM, 135–146.
14
[46] Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim. 2023. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. In Proceedings of the 56th IEEE/ACM International Symposium on Microarchitecture (MICRO). ACM, Toronto, ON, Canada, 105–121. [47] Brian R. Tauro, Brian Suchy, Simone Campanoni, Peter Dinda, and Kyle C. Hale. 2024. TrackFM: Far-out Compiler Support for a Far Memory World. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 401–419. [48] Qwen Team. 2025. Qwen3 Technical Report. https://qwenlm.github. io/blog/qwen3/. [49] Auburn University. [n. d.]. vSMP (ScaleMP). https://www.eng.auburn. edu/ens/hpcc/vsmp-scale-mp.html. [50] Prabhu Vellaisamy, Thomas Labonte, Sourav Chakraborty, Matt Turner, Samantika Sury, and John Paul Shen. 2025. Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures . In 2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE Computer Society, Los Alamitos, CA, USA, 49–61. [51] Midhul Vuppalapati and Rachit Agarwal. 2024. Tiered Memory Management: Access Latency is the Key!. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP). ACM, Austin, TX, USA, 79–94. [52] Mattias De Wael, Stefan Marr, Bruno De Fraine, Tom Van Cutsem, and Wolfgang De Meuter. 2015. Partitioned Global Address Space Languages. Comput. Surveys 47, 4, Article 62 (2015), 27 pages. doi:10. 1145/2716320 [53] Xingda Wei, Fangming Lu, Tianxia Wang, Jinyu Gu, Yuhan Yang, Rong Chen, and Haibo Chen. 2023. No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computing. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 497–517. https://www. usenix.org/conference/osdi23/presentation/wei-rdma [54] Lingfeng Xiang, Zhen Lin, Weishu Deng, Hui Lu, Jia Rao, Yifan Yuan, and Ren Wang. 2024. Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Santa Clara, CA, 19–35. [55] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI). 15–28. [56] Jin Zhang, Zhuocheng Ding, Yubin Chen, Xingguo Jia, Boshi Yu, Zhengwei Qi, and Haibing Guan. 2020. GiantVM: A Type-II Hypervisor Implementing Many-to-One Virtualization. In Proceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 30–44. [57] Yuhong Zhong, Daniel S. Berger, Carl Waldspurger, Ryan Wee, Ishwar Agarwal, Rajat Agarwal, Frank Hady, Karthik Kumar, Mark D. Hill, Mosharaf Chowdhury, and Asaf Cidon. 2024. Managing memory tiers with CXL in virtualized environments. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (Santa Clara, CA, USA) (OSDI’24). USENIX Association, USA, Article 3, 20 pages. [58] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024. USENIX Association, 193–210.
[59] Yijie Zhong, Minqiang Zhou, Zhirong Shen, and Jiwu Shu. 2024. UniMem: redesigning disaggregated memory within a unified localremote memory hierarchy. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC’24). USENIX Association, USA, Article 29, 15 pages. [60] Yuanyuan Zhou, Liviu Iftode, and Kai Li. 1996. Performance Evaluation of Two Home-Based Lazy Release Consistency Protocols for Shared Virtual Memory Systems. In Proceedings of the 2nd USENIX Symposium on Operating Systems Design and Implementation (OSDI). 75–88.
15