arXiv CSopen access

Two-sided RDMA Striking Back for Disaggregated Memory Databaases

Two-sided RDMA Striking Back for Disaggregated Memory Databases Hokeun Cha

Aditya Akella

Xiangyao Yu

[email protected] University of Wisconsin-Madison Madison, WI, USA

[email protected] University of Texas at Austin Austin, TX, USA

[email protected] University of Wisconsin-Madison Madison, WI, USA

arXiv:2607.26227v1 [cs.DB] 28 Jul 2026

Abstract RDMA has enabled high-speed data access and low-latency communication in disaggregated memory databases. While various optimization techniques have been proposed to accelerate transactions with RDMA in this setting, two-sided RDMA has been largely underexplored in favor of one-sided RDMA due to its remote CPU involvement. However, the heavy use of one-sided RDMA introduces fundamental limitations. Its limited APIs cannot express complex system functions such as starvation prevention, priority-based scheduling, and preemption, which are all critical functions in concurrency control protocols. Moreover, indexing requires multiple network round-trips, causing network amplification. In this work, we revisit the long-standing debate between onesided RDMA and two-sided RDMA in the context of disaggregated memory databases. We present Lotus, which addresses the conventional limitation of two-sided RDMA—CPU bottlenecks in memory servers—by leveraging the rich functionality of two-sided RDMA with two key optimization techniques: (1) lightweight caching and (2) efficient batching. Lotus demonstrates that limited CPU resources in memory servers, when intelligently utilized, can transform a perceived weakness into a significant advantage. Our experimental study shows that Lotus achieves up to 8.2× higher throughput and 42.9× lower p999 tail latency than state-of-the-art one-sided RDMA-based approaches in YCSB benchmark.

1

Introduction

Remote Direct Memory Access (RDMA) has emerged as a cornerstone technology for high-performance distributed database systems over the past decade, offering unprecedented network performance with low latency and high bandwidth. The systems research community has extensively studied RDMA for its ability to dramatically improve the performance of distributed OLTP databases, leading to significant advances in indexing [25, 43, 57, 58], concurrency control [13, 14, 30, 45, 46], and data replication [36, 38, 53]. The RDMA research has been marked by a fundamental debate: whether to use one-sided RDMA [13, 14, 30, 46] or two-sided RDMA [10, 20–22] in network communication. This debate has shaped the evolution of RDMA-based database systems, with each approach offering distinct trade-offs that have influenced design decisions across the research community. The salient feature of one-sided RDMA, i.e., direct access to remote memory without remote CPU involvement, has opened up a new paradigm in system designs. In contrast, two-sided RDMA has enabled smooth transition for existing systems via its socket-like APIs. However, two-sided RDMA has been largely underexplored in disaggregated memory, where compute and memory resources are decoupled. In such settings, where memory servers typically

have limited computing resources compared to compute servers, two-sided RDMA systems were considered suboptimal for handling a high volume of requests from compute servers. In this work, we revisit the legacy one-sided versus two-sided RDMA debate in the context of memory disaggregation and argue that the common wisdom favoring one-sided RDMA needs careful review. Our analysis reveals that two-sided RDMA is a compelling alternative that can overcome its limitations and outperform one-sided RDMA in B+-tree indexing and two-phase locking concurrency control. In particular, two-sided RDMA in memorydisaggregated OLTP databases can leverage three key features: (1) Rich Functionality. Two-sided RDMA requires remote CPU involvement, which has been considered a major drawback due to limited CPU resources in memory servers. However, this involvement enables rich functional features that can express complex system functions that are not easily achievable with one-sided RDMA. For example, one-sided RDMA makes it extremely difficult to implement starvation prevention, priority-based scheduling, and preemption—all of which are critical functions in concurrency control protocols. Moreover, one-sided RDMA often requires multiple network round-trips for index operations. Index traversal requires reading the entire node data over the network, and each node access requires at least three RDMA READs to ensure correctness in optimistic synchronization, i.e., checking a version, reading node data, and validating the version [56]. In contrast, these operations require much fewer network round-trips with two-sided RDMA. (2) Caching. To mitigate the network amplification problem in one-sided RDMA indexing, prior work has employed caching in compute servers to reduce remote memory accesses [26, 29, 41, 43]. While caching has shown effectiveness in one-sided RDMA indexing, it has not been explored in the context of two-sided RDMA indexing. We make a key observation that these caching techniques can also be applied to two-sided RDMA to avoid unnecessary network round-trips, which can significantly improve performance. (3) Batching. While two-sided RDMA has been compared with one-sided RDMA in prior work [10, 20–22], these studies primarily target key-value stores or remote procedure call (RPC) systems that process one request at a time. However, a database transaction usually consists of multiple data access requests. Those requests within a transaction can be batched into a network message, which can substantially reduce the number of network round-trips. Regardless, some transactions may only have a few requests or have data dependency, which cannot exhibit sufficient batching effects. In such cases, requests from different transactions can be combined into a single network message to further reduce network overhead. Motivated by these observations, we design and implement Lotus, a two-sided RDMA-based system for disaggregated memory OLTP databases that leverages rich functionality, caching, and batching

Compute Servers

Hokeun Cha, Aditya Akella, and Xiangyao Yu

DIMM

2.1

…

CPUs DB

Partition K

Partition 2

Memory Servers

Partition 1

…

Compute Servers

(a) Non-partitioned memory. Partition 1

Partition 2 DIMM

Partition K

…

CPUs

Memory Servers

DB

… (b) Co-partitioned memory.

Figure 1: Memory disaggregation models.

techniques. Following prior research on RDMA-accelerated OLTP systems [13, 14, 26, 30, 41, 43, 45, 46, 51, 58], we evaluate the performance of Lotus against one-sided RDMA schemes in indexing and concurrency control. In particular, we focus on B+-tree indexing and two-phase locking (2PL), the most widely-used classes of indexing and concurrency control schemes in DBMSs. Our performance study shows that Lotus is starvation free and achieves up to 8.2× higher throughput and 42.9× lower p999 tail latency compared to state-of-the-art one-sided RDMA schemes in YCSB workloads. In summary, this paper makes the following contributions: • We revisit the legacy one-sided RDMA versus two-sided RDMA debate in the context of disaggregated memory databases and analyze the advantages and disadvantages of both modes. • We develop Lotus, a scalable two-sided RDMA-based mechanism that efficiently utilizes limited memory server CPU resources by minimizing network overhead through caching and batching. • We comprehensively evaluate Lotus against one-sided RDMAbased indexing and concurrency control schemes and show that Lotus outperforms them in both throughput and tail latency. The rest of this paper is organized as follows. Section 2 presents the background and motivation of the research. Section 3 provides a design overview of Lotus. Section 4 describes concurrency control in Lotus. We discuss its optimization techniques, caching and batching, in Sections 5 and 6, respectively. Section 7 evaluates Lotus against the state-of-the-art one-sided RDMA schemes. Section 8 reviews related work, and Section 9 concludes the paper.

2

Background and Motivation

RDMA has been a crucial component for optimizing distributed database systems due to its high performance and ability to directly access remote memory, particularly in disaggregated memory architectures. In this section, we review the background of disaggregated memory architectures and RDMA, and discuss the designs and limitations of RDMA-based indexing and concurrency control.

Disaggregated Memory Architecture

The emergence of memory disaggregation has led to a redesign of traditional distributed database systems to support important features such as scalability, elasticity, and cost efficiency [3, 19, 39, 42, 44, 51]. Unlike legacy monolithic server architectures, where database components are tightly coupled together, a memory-disaggregated database is decoupled into two layers of servers, memory servers and compute servers. A memory server is equipped with a large amount of memory resources and small computing resources. The large memory space stores data structures and tables for a DBMS. A compute server has powerful computational resources, while its memory resources are limited. The two layers of servers can scale independently to strike the best balance of the resources. Figure 1 illustrates two representative memory disaggregation models, non-partitioned memory and co-partitioned memory. In nonpartitioned memory, a DBMS is partitioned across memory servers, and each compute server can access any partition in memory servers, providing a globally shared view. In co-partitioned memory, the DBMS is additionally partitioned in compute servers so that each compute server accesses only its own partition, providing a local view. Note that the partitioning functions in the two server layers can be different from each other to efficiently balance the load across servers. While transactions, in both memory models, are processed in compute servers by accessing data structures and tables in memory servers, the difference in the models leads to different behaviors in some database components, e.g., cache coherence and concurrency control. We discuss the details in Section 2.3.

2.2

RDMA

RDMA network provides higher bandwidth and lower latency than conventional TPC/IP networks. Moreover, it enables direct access to remote memory without remote CPU involvement. These features have opened up new opportunities for optimizing distributed database systems. Research on RDMA can be divided into two directions, enhancing traditional RPC-based systems with two-sided RDMA and redesigning the systems to leverage one-sided RDMA. Two-sided RDMA provides a lightweight message-passing abstraction with a SEND/RECV interface, commonly used to implement RPCs [10, 20–22]. A message is exchanged when a SEND matches a RECV. On the receiver side, the host must poll for incoming requests and manager the receive buffer. Therefore, two-sided RDMA involves the CPUs on both the sending and receiving ends, not providing the CPU-bypassing property as in one-sided RDMA. One-sided RDMA registers the memory region of a remote host in its local NIC and provides direct access to remote memory through READ/WRITE/ATOMIC interfaces. RDMA ATOMIC supports Fetch-And-Add (FAA) and Compare-And-Swap (CAS) at 8byte granularity, which are commonly used to implement synchronization protocols. Since one-sided RDMA does not involve remote CPUs, it has been considered as a well-fitting communication method in disaggregated systems [25, 43, 51, 57].

2.3

RDMA-based Transaction Processing in Disaggregated Architecture

Transaction processing in existing disaggregated systems mainly works with one-sided RDMA. This preference is largely because

Two-sided RDMA Striking Back for Disaggregated Memory Databases

Compute Server RDMA

N2 N3

Cache

Memory Server

N1

READs

2 4 6

20 40

6 12

…

N4 8 12

…

Compute Server RDMA

Nk-1

N2

50 Nk 45 50

Nk+1 55 60

(a) One-sided RDMA indexing.

Cache

Memory Server

N1

SEND/REC V

RDMA ECV SEND/R

N3 2 4 6

20 40

6 12

…

N4 8 12

…

Nk-1 50 Nk 45 50

Nk+1 55 60

(b) Two-sided RDMA indexing.

Figure 2: Indexing scenarios of read operations (𝑘𝑒𝑦 = 4) in disaggregated memory databases. one-sided RDMA can leverage strong computational power in compute servers to directly access remote memory in memory servers, while two-sided RDMA requires involvement of limited CPU resources in memory servers [25, 43, 44, 51]. We break down the transaction processing into indexing and concurrency control and describe how they work with RDMA in the two memory disaggregation models, non-partitioned memory and co-partitioned memory. 2.3.1 Indexing. While a lot of prior work has studied RDMA-based indexing, most designs are hash-based [13, 30, 40, 53, 58]. This is because one-sided RDMA indexing incurs multiple network transfers, and hashing bounds the number of remote memory accesses by limiting bucket probes [32]. However, hash indexes have a critical limitation that they do not support range scans and thus cannot be used in many practical workloads. B+-tree-based RDMA indexes, on the other hand, support various workload requirements but suffer from network amplification, as index traversal requires multiple RDMA READs to locate each node from the root to the leaf. Figure 2(a) illustrates an indexing scenario in one-sided RDMA B+-tree. The compute server retrieves the value for key 4 by traversing the tree in the memory server. Given an index height of 3, the compute server performs at least 3 RDMA READs, i.e., 𝑁 1 , 𝑁 2, and 𝑁 3 , to reach the target leaf. For each access, it reads the entire node data, typically 1 KB in size [26, 43, 57], to locate the child. Although omitted from the figure for clarity, each node access requires at least three dedicated RDMA READs to ensure correctness in optimistic synchronization [56], i.e., reading the node version, reading the node data, and validating the version. Overall, these RDMA READs for traversal and synchronization cause significant network amplification, which gets worse as the index grows. To mitigate the network amplification problem, prior work has employed caching in compute servers to reduce remote accesses during index traversal [26, 41, 43]. In non-partitioned memory, multiple compute servers may cache the same data since they have shared access to memory servers. To keep cache coherence simple, recent studies cache shortcut hints to data tuples, e.g., last-level nodes in a B+-tree [41, 43]. Then, the system uses a cached hint to locate remote data and validates staleness by checking its metadata. In co-partitioned memory, each compute server owns a logical data partition in memory servers and caches only the index nodes in its partition, avoiding coherence management across compute servers. While these techniques have shown effectiveness in one-sided RDMA indexing, they have not been explored in the context of twosided RDMA indexing. We make a key observation that the caching approaches can also be applied to two-sided RDMA systems. In non-partitioned memory, shortcut hints can be cached in compute servers and sent to memory servers via network messages to reduce indexing overhead. In co-partitioned memory, each compute server

can cache data tuples that belong to its own partition, completely eliminating network round-trips for cached data retrieval.

2.3.2 Concurrency Control. In co-partitioned memory, each compute server accesses its own partition in memory servers, allowing concurrency control to be handled locally in compute servers. However, this requires an additional distributed transaction layer for cross-partition transactions on top of compute servers, e.g., proxy servers or client servers. In non-partitioned memory, compute servers have shared access to memory servers, requiring a scalable concurrency control protocol to manage concurrent accesses. Optimistic concurrency control (OCC) [23] and two-phase locking (2PL) [4, 5] are the two most widely-used protocols in RDMAbased systems [8, 13, 45, 46, 52, 53]. OCC defers conflict detection to the end of execution by validating versions of accessed records, while 2PL synchronizes via explicit lock acquisition in a shared or exclusive mode. Although both are simple and efficient, OCC lacks native support for priority-aware conflict handling, causing starvation prevention problems. Recent work has sought to reduce high tail latency in OCC by incorporating pessimistic mechanisms from 2PL [35, 49], highlighting the importance of lock-based control under contention. Therefore, we focus on 2PL in this work. To implement 2PL with one-sided RDMA, an 8-byte value is sliced into two smaller values to indicate the number of lock owners for exclusive and shared access, respectively. Acquiring a lock requires at least two RDMA round-trips. First, the lock value is read with RDMA READ to check for conflicts. If no conflict exists, the value is then atomically updated with RDMA CAS. A critical limitation of 2PL with one-sided RDMA is that it cannot guarantee starvation prevention. In conventional 2PL, WAIT_ DIE and WOUND_WAIT prevent starvation by scheduling transactions based on priorities and allowing higher-priority transactions to wait for or preempt lower-priority ones. However, the limited APIs of one-sided RDMA make it extremely difficult to implement such functionalities. Priority-based scheduling requires transferring lock ownership to the next waiting transaction in priority order, which in turn demands a notification mechanism among compute servers. Similarly, preemption requires notifying transactions of their preempted status. Neither of the protocols can be efficiently implemented with one-sided RDMA since all these operations need to be executed in an atomic way. One way to reduce latency is to place an additional 8-byte value alongside a lock to track the priority of a current lock owner [40]. After lock acquisition, the owner updates its priority value via RDMA CAS, allowing other transactions to decide whether to wait or abort upon a conflict. However, this still does not guarantee starvation prevention. Since a lock should be acquired via RDMA

Hokeun Cha, Aditya Akella, and Xiangyao Yu

CAS, whichever transaction succeeds in swapping the lock value becomes the next owner, regardless of priority. While these limitations of one-sided RDMA remain unsolved, two-sided RDMA can easily support those functionalities with the help of remote CPUs. As transaction requests are transferred to memory server CPUs over network messages, the system can leverage optimized legacy data structures and algorithms to implement starvation prevention, priority-based scheduling, and preemption.

3

Lotus Overview

We introduce Lotus, a scalable transaction processing system for disaggregated DBMSs that supports both non-partitioned and copartitioned memory. It leverages the rich semantics with two-sided RDMA for indexing and concurrency control and further optimizes the processing pipeline through caching and batching. In this section, we describe the design of indexing and concurrency control in Lotus and provide an overview of our optimization techniques. Indexing. Unlike in many one-sided RDMA indexes [25, 26, 29, 41, 43, 57], which maintain a global index in memory servers, Lotus maintains a local index in each memory server. An indexing operation is executed by sending a request from a compute server to the corresponding memory server over two-sided RDMA. The limitation of one-sided RDMA indexing, i.e., network amplification, can be easily addressed with two-sided RDMA, as it offloads index traversal to memory server CPUs. Processing an index request needs at most one network round-trip with minimal traffic. Figure 2(b) shows an indexing scenario using two-sided RDMA. The memory server maintains a B+-tree index, and the compute server is trying to retrieve the value for a key. The compute server first sends the index request to the memory server via RDMA SEND, which is only a few bytes in size. The memory server then traverses the index by directly accessing its local memory and returns only the result to the compute server. The cache in the compute server is used to reduce the overhead of network and data processing. Concurrency control. Similarly to indexing, the limitation of onesided RDMA concurrency control, i.e., functional deficiency, can be effectively addressed with two-sided RDMA by offloading lock management to memory server CPUs. Lotus maintains lock tables in each memory server and implements critical system functions in concurrency control protocols, including starvation prevention, priority-based scheduling, and preemption. We discuss the technical details in Section 4. Optimization techniques. While the adoption of two-sided RDMA provides rich system functionality for indexing and concurrency control, the involvement of memory server CPUs may introduce performance overhead. To address this challenge, we present two key optimization techniques that minimize the overhead of network and data processing. These optimizations can significantly reduce the number of remote data accesses and network round-trips. • Caching. Lotus adopts the state-of-the-art caching techniques discussed in Section 2.3.1 for non-partitioned memory and copartitioned memory to accelerate indexing. We enhance these techniques by converting them to be compatible with two-sided RDMA. We describe the details in Section 5.

• Batching. Lotus applies two batching techniques, intra-transaction batching and inter-transaction batching, to minimize the number of network round-trips in transaction execution. We discuss these techniques in Section 6.

4

Concurrency Control

Concurrency control with two-sided RDMA enables rich functionalities that are not supported with one-sided RDMA, as discussed in Section 2.3.2. By offloading lock management to memory server CPUs, the system can use optimized data structures and algorithms to implement starvation prevention, priority-based scheduling, and preemption. Compute servers send lock requests to memory servers over two-sided RDMA, and memory servers process these requests by directly accessing their local lock tables. We first describe how Lotus implements these functionalities and then walk through an example to provide a comprehensive understanding of the details.

4.1

Priority-based Scheduling

To achieve low latency and ensure fairness among concurrent transactions, Lotus implements priority-based scheduling in memory servers. Each memory server maintains a wait queue per lock to manage conflicting requests in priority order. Upon a conflict, the lock manager compares the priority of the incoming request with that of the lock owners. Based on the concurrency control protocol, the manager may add the request to the wait queue, preempt lowerpriority owners, or reject the request. When a lock is released, its ownership is granted to the waiting transaction according to the priority order, ensuring starvation prevention.

4.2

Preemption

Preemption is another critical functionality in concurrency control protocols that forces lower-priority transactions to abort. We implement preemption by maintaining an additional structure that tracks transaction status. Upon a lock conflict, the lock manager checks the lock table and attempts to preempt lower-priority owners by removing them from the table and atomically updating their status. This process is protected by a mutex to ensure atomicity. If all conflicting transactions have been preempted, the lock is granted to the incoming request. Some transactions may not be preempted if they have higher priorities or have already entered a commit phase. Then, the request is added to the wait queue.

4.3

Lazy Notification

Transactions may have to wait for lock acquisition or be preempted due to conflicts, as discussed above. These functions may increase the complexity of communication between compute servers and memory servers. In two-sided RDMA, message delivery requires a matching pair of SEND and RECV. Since a transaction may need extra waiting time to acquire locks and be preempted at arbitrary points, it may lead to a mismatch of the two. For example, after receiving a lock grant, a transaction may post another SEND to a memory server to request additional locks. However, if the previously granted lock has been preempted, the memory server must notify the preempted compute server transaction to abort by posting a SEND, leading to a SEND-RECV mismatch.

Two-sided RDMA Striking Back for Disaggregated Memory Databases

Memory Server

Compute Server Priority (8) Priority (5) Priority (3)

T1

T2

WLock(A) WAIT

T3

RLock(A)

RDMA CAS

Compute Server

T1

WAIT WAIT

Unlock(A)

WLock(A)

T2

T3

RDMA SEND/RECV

RLock(A)

WLock(A)

WLock(A)

WLock(A)

Memory Server

Priority (8) Priority (5) Priority (3)

T1

Unlock(A)

Unlock(A)

Unlock(A)

T2

T3

RLock(A)

WLock(A)

WLock(A)

Memory Server RDMA SEND/RECV

WLock(A) WAIT Unock(A) Unlock(A)

WLock(A)

RLock(A)

WAIT

(a) One-sided RDMA WAIT_DIE.

Compute Server Priority (8) Priority (5) Priority (3)

(b) Two-sided RDMA WAIT_DIE.

(c) Two-sided RDMA WOUND_WAIT.

Figure 3: Execution scenario of two write transactions (𝑇1 and 𝑇2 with priorities 8 and 5) and one read transaction (𝑇3 with priority 3) — higher values represent higher priorities. Blue arrows denote lock requests, orange arrows represent successful lock acquisitions, and red arrows indicate lock acquisition failures. To handle the mismatch problem in a simple and efficient way, Lotus notifies lock status to compute servers in a lazy manner. For a waiting transaction, a memory server does not immediately notify the compute server about the waiting status. Instead, the memory server keeps the lock request in the wait queue, periodically checking for lock availability, and lets the compute server wait. Once the lock becomes available, it processes the request and then notifies the compute server. For a preempted transaction, the memory server simply marks its status as preempted without immediate notification. When the next request from the preempted transaction arrives, the memory server informs that the transaction has been preempted, forcing to abort. This lazy notification further eliminates unnecessary network communication between servers.

When 𝑇3 completes, the memory server hands over the lock to 𝑇1 . After 𝑇1 completes, 𝑇2 retries and acquires the lock. Figure 3(c) illustrates the two-sided RDMA WOUND_WAIT case. 𝑇3 first acquires the lock. However, after 𝑇1 sends its request, the memory server preempts 𝑇3 and grants the lock to 𝑇1 since 𝑇1 has higher priority. Note that 𝑇3 is not notified yet about its preempted status, as discussed in Section 4.3. After receiving the request from 𝑇2 , the memory server adds it to the wait queue. Then, when 𝑇3 tries to commit, the memory server notifies 𝑇3 to abort as it keeps track of 𝑇3 ’s preempted status. After 𝑇1 completes, the lock is granted to the next waiter, 𝑇2 . When 𝑇3 retries, its request is added to the wait queue since the current lock owner, 𝑇2 , has higher priority.

5 4.4

Example

Now, we illustrate how Lotus guarantees starvation freedom through priority-based scheduling, preemption, and lazy notification. For a clear comparison, we first review one-sided RDMA concurrency control. Figure 3 shows execution scenarios of two write transactions, 𝑇1 and 𝑇2 , and one read transaction, 𝑇3 , all accessing the same tuple 𝐴. 𝑇1 has the highest priority, followed by 𝑇2 and 𝑇3 . We assume the following execution order: 𝑇3 → 𝑇1 → 𝑇2 . For clarity, we omit metadata and data accesses and focus on lock operations. Figure 3(a) shows the execution scenario in one-sided RDMA WAIT_DIE protocol, described in Section 2.3.2. 𝑇3 first acquires a read lock via RDMA CAS.𝑇1 tries to acquire a write lock but fails due to a conflict. Then, 𝑇1 compares its priority with that of the current owner, 𝑇3 . Since 𝑇1 has higher priority, it waits until 𝑇3 completes. Meanwhile, 𝑇2 also fails to acquire the lock and waits. The waiting in one-sided RDMA involves periodic RDMA READs to poll the lock availability. Once 𝑇3 releases the lock, both 𝑇1 and 𝑇2 try to acquire it. However, 𝑇2 succeeds before 𝑇1 because lock acquisition via RDMA CAS does not enforce ordering. Therefore, 𝑇1 has to keep waiting until 𝑇2 completes, which may suffer indefinitely. Figure 3(b) describes the same scenario in two-sided RDMA WAIT_DIE. 𝑇3 , 𝑇1 , and 𝑇2 send their lock requests to the memory server over RDMA SEND. Since 𝑇3 ’s request has arrived first, it acquires the lock. Then, the memory server schedules the high priority transaction, 𝑇1 , in the wait queue and informs 𝑇2 to abort since 𝑇2 has lower priority than the waiting transaction, 𝑇1 . Note that 𝑇1 is notified when the lock is granted, as discussed in Section 4.3.

Caching

Caching can significantly reduce both remote data processing and network processing overhead. Based on the memory disaggregation models that handle cache coherence in different ways, as discussed in Section 2, we adopt two caching techniques for both (1) nonpartitioned memory and (2) co-partitioned memory. Our caching techniques operate at a finer granularity than conventional nodelevel caching, i.e., record-level caching. In the following, we describe how each technique is applied to our indexing design.

5.1

Caching at Record Granularity

Unlike prior one-sided RDMA indexing schemes that cache index nodes, Lotus caches individual records. Caching index nodes naturally complements one-sided RDMA, where compute servers directly traverse index structures in memory servers. In two-sided RDMA, compute servers send requests and receive responses, while memory servers handle index traversal internally on their behalf. This execution model not only preserves abstraction boundaries but also enables effective record-level caching along the requesthandling path, which is difficult to achieve with one-sided RDMA. Record-level caching also improves cache utilization by providing finer-grained control over data access. Recent studies have shown that caching index nodes may lead to inefficient cache utilization since a single node contains many records and only a few of them may be frequently accessed [12, 16, 55]. Cache replacement in Lotus is also performed at the record level. Lotus tracks the access frequency of each cache record and evicts the least frequently accessed one when the cache becomes full.

Hokeun Cha, Aditya Akella, and Xiangyao Yu

5.2

Caching for Non-Partitioned Memory

As discussed in Section 2.3.1, multiple compute servers may cache the same data in non-partitioned memory. To keep cache coherence management simple, Lotus follows the approach from prior work [41, 43] that caches pointers to data tuples. In non-partitioned memory caching, read and write requests follow the same handling path. A compute server sends a request to a memory server, and the memory server traverses its index with the request key and retrieves the pointer to the corresponding data. Along with the data, the memory server returns this pointer, which is then cached in the compute server. On subsequent accesses to the matching key, the compute server provides the pointer to the memory server as a shortcut hint. This allows the memory server to directly access the data, bypassing index traversal. Some cache entries may become stale if the original data in a memory server has been updated by requests from other compute servers. Validation of such stale cache entries is offloaded to memory servers. When a shortcut hint is provided, the memory server checks tuple metadata to determine its validity. If stale, the memory server retrieves the up-to-date data via index traversal and returns it to the compute server. The compute server then invalidates the stale cache entry and updates it with the new one. Cache replacement may occur upon admission if the cache becomes full. Since cached data serves only as a hint to accelerate remote data access, eviction does not require coherence management. The compute server simply removes the entry from the cache.

5.3

Caching for Co-Partitioned Memory

Unlike caching for non-partitioned memory, caching for co-partitioned memory stores actual tuple data, since cache coherence does not occur across compute servers. When a compute server processes a request, it first checks its local cache. On a cache miss, the compute server requests the data from the memory server and caches it locally upon return. On a cache hit, the compute server accesses the data directly from the cache without involving the memory server. For a write request, the compute server directly updates the cached data, marks it as dirty, and defers write-back to the memory server until eviction. During eviction of a dirty entry, the data is latched to ensure correctness and prevent concurrent modification during write-back. Once the updated data is flushed to the memory server, the entry is removed from the cache. Clean entries are evicted immediately without additional operations.

6

Batching

In this section, we present our batching techniques, a key optimization that fundamentally reshapes the performance trade-offs of two-sided RDMA by substantially reducing network round-trips between compute servers and memory servers. While prior studies have evaluated two-sided RDMA against one-sided RDMA in various distributed systems [30, 46, 51], they mainly focus on workloads for key-value stores or RPC systems that handle a single request at a time. Figure 4(a) illustrates such cases, where each request in a transaction incurs a separate network round-trip. However, this per-request communication can be inefficient, particularly in disaggregated memory systems, where many requests require remote memory accesses.

We make a key observation that database transactions usually consist of multiple requests that can be issued and processed concurrently. We exploit this property by aggregating multiple requests targeting the same memory server into a single two-sided RDMA message. This optimization is inherently specific to twosided RDMA, as its simple request-response communication abstraction naturally supports aggregation of requests and responses, whereas one-sided RDMA decomposes each request into multiple complex network operations for indexing and concurrency control. Leveraging this insight, we introduce two batching techniques: (1) intra-transaction batching and (2) inter-transaction batching.

6.1

Intra-Transaction Batching

Intra-transaction batching combines multiple requests within a transaction into one network message. Figure 4(b) shows an example of intra-transaction batching. When a compute server executes a transaction, it first collects multiple requests that can be processed concurrently. Then, it sends those requests to a memory server via an RDMA SEND operation. Upon receiving the batched requests, the memory server processes them sequentially and returns their results to the compute server with another RDMA SEND. This approach minimizes the number of network round-trips per transaction, significantly reducing network processing overhead. While batching opportunities vary across transactions, some may have few requests and may include dependencies, still requiring multiple round-trips. To further improve batching effectiveness in such cases, we additionally apply inter-transaction batching.

6.2

Inter-Transaction Batching

Inter-transaction batching aggregates requests from multiple transactions into a single network message to further reduce network round-trips. Figure 4(c) illustrates an example of inter-transaction batching. In each compute server, Lotus organizes threads that execute transactions concurrently into several batch groups (e.g., 8 threads per group). Instead of sending requests individually as in intra-transaction batching, group members submit their requests to a group leader. The leader then collects and sends these requests to a memory server with a single RDMA SEND. The memory server processes the batch of requests and returns the results to the leader with another RDMA SEND. Finally, the leader dispatches the results to the members. Next, we discuss its design details. 6.2.1 Leader Selection. Selecting an appropriate group leader is critical for efficient batching. A static leader may delay batching progress because if the leader’s transaction aborts and enters backoff, other members must wait until the leader comes back. Lotus addresses this issue by employing dynamic leader selection. Each group maintains a leader ID. When a member finishes submitting its requests, it checks whether a leader exists. If no leader is present, the member atomically updates the leader ID to its own ID and becomes the new leader. Then, the leader begins collecting requests from other group members. The leader serves for multiple batches to reduce the coordination overhead of leader selection. Its term ends when the batch window expires or its transaction aborts. The leader then resigns by resetting the value of the leader ID, and the next member that updates it becomes the new leader.

Two-sided RDMA Striking Back for Disaggregated Memory Databases

Read Write

T1

Write

(a) Non-batching.

Compute Server

Tk

Write

Read

Read Write

(b) Intra-transaction batching.

Read Read

T2

Read

Tk

Write

Read

… …

Tk

Read

…

Read Write

Read

T2

Write

Read

Memory Server

Read

Read

Memory Server

Read

Memory Server

Read

T2

…

Compute Server

Read

Read

T1 Compute Server

T1

Read Write

Write

(c) Inter-transaction batching.

Figure 4: Batching techniques in Lotus. 6.2.2 Batch Formation. Timely batch formation is essential to balance batching efficiency and request latency. Lotus employs a timeout-based fallback mechanism with dynamic waiting times. A batch manager maintains lightweight statistical metrics for each member such as batch success rate, timeout rate, and conflict rate. For each batching decision, it computes an appropriate timeout value based on these metrics to avoid excessive delays. If a member’s request is not claimed by the leader within the waiting time, it falls back to individual request transmission. Note that these statistics do not require coordination among group members. 6.2.3 Contention-awareness. Workload contention may affect batching effectiveness. Under high contention, aborted transactions may temporarily leave the group and rejoin after backoff, reducing the group utilization. As a result, the leader may collect fewer requests, and members may time out and fall back to individual transmission. Lotus addresses this issue in two ways. First, each group maintains the number of active members and adjusts batching decisions accordingly. When the count falls below a threshold, members send requests individually instead of waiting for batch formation. Second, Lotus uses the statistical metrics to guide batching decisions dynamically. Rather than batching all requests indiscriminately, each group selectively applies batching based on the observed patterns. 6.2.4 Adaptivity. In order to improve effectiveness under varying workloads, Lotus maintains batching parameters in an adaptive manner. For each network request, the system records statistics that capture recent batching behavior, and for each batch window, these parameters are updated using an exponentially weighted moving average (EWMA) [28], enabling the batching mechanism to smoothly respond to workload changes over time.

6.3

Discussion

Prior work has proposed doorbell batching to accelerate two-sided RDMA operations [21, 22]. This technique reduces the overhead of memory-mapped I/Os by combining multiple RDMA work requests into a single doorbell at the network level. However, it does not reduce the number of network round-trips and is typically applicable to systems where multiple threads share a single queue pair, a design commonly used with unreliable datagram (UD) transport, which does not guarantee reliable message delivery.

In contrast, Lotus presents lightweight batching techniques at the software level. Unlike doorbell batching, our approaches combine multiple transaction requests into a single RDMA operation, thereby reducing the number of network round-trips. These techniques are generally applicable across distributed systems regardless of the underlying network transport, including both RDMA and traditional socket-based networks. Lotus supports both UD and reliable connection (RC) and uses RC by default to ensure reliable message delivery while supporting varying message sizes. Although reliability can be provided over UD, it comes with limitations. For example, prior work uses a timeout-based mechanism to detect and handle packet loss in UD [22]. However, this requires restarting the entire system upon a loss event. Moreover, UD imposes a strict packet size limit, since it does not support packets larger than the hardware’s maximum transfer unit (MTU), typically only a few KB [31]. This requires application-level mechanisms for packet fragmentation at the sender and reassembly at the receiver, which adds complexity and incurs additional network round-trips. We investigate the performance of RC and UD in Section 7.8.

7

Evaluation

We divide our performance study into two parts based on the memory disaggregation models, non-partitioned memory and copartitioned memory. We first describe our experimental setup and benchmarking workload in Sections 7.1 and 7.2, respectively. We outline the details of our benchmarking system, indexing, and concurrency control baselines in Section 7.3. Then, we present and analyze the performance of Lotus from Sections 7.4 to 7.10. In particular, we aim to answer the following questions in our evaluation: • How sophisticated concurrency control in Lotus achieves starvation freedom and sustains high performance under contention. • How caching in Lotus improves performance by reducing remote memory accesses. • How batching techniques in Lotus reduce network round-trips and improve throughput.

7.1

Experimental Setup

We conduct all experiments on four CloudLab [15] machines (c6525100g instances). Each machine contains an AMD EPYC 7402P CPU (24 cores/48 hyper-threads), 128 GB of DRAM, and a 100 Gbps Mellanox ConnectX-5 NIC. Following prior work on RDMA-based disaggregated memory systems [26, 41, 43], we configure each machine

Hokeun Cha, Aditya Akella, and Xiangyao Yu

Lotus WAIT_DIE Sherman/Deft WAIT_DIE

Lotus WOUND_WAIT

Throughput (Ktxns/sec)

Lotus NO_WAIT Sherman/Deft NO_WAIT

to act as a compute server and a memory server. On each machine, we allocate 40 threads for the compute server and 8 threads for the memory server, following configurations in prior work [51, 57]. We also evaluate sensitivity to the number of memory server threads in Section 7.10. Similar to a recent study [26], each compute server is provisioned with a 128 MB cache (i.e., 8% of the index data size). After loading the data, we warm up the DBMS and then measure the performance by running each workload for 10 seconds.

7.2

Figure 5: Throughput with a varying number of compute threads in YCSB workloads.

Workload

We use Yahoo! Cloud Serving Benchmark (YCSB) [11], a widely used benchmark for key-value store evaluation. We use a 100 GB database scale consisting of a single table with 100 million records. Each record is 1 KB in size, containing a single primary key and 10 additional columns of randomly generated string data. Each transaction accesses 16 records by primary key. Record accesses follow a Zipfian distribution (𝜃 = 0.9), and each access is either a read or write. We use three workload configurations to capture different levels of read-write conflicts: • Workload A is write-intensive, with 50% reads and 50% writes. • Workload B is read-mostly, with 95% reads and 5% writes. • Workload C is read-only.

7.3

Implementation

7.3.1 Testbed. We implement on top of DBx1000 [50], an in-memory DBMS research prototype. We extend it to a disaggregated system with RDMA by separating the computing layer and the memory layer of the DBMS. 7.3.2 Index. The DBMS maintains a primary index for each table, where each index stores pointers to data records. For Lotus, we use a B+-tree that employs optimistic latch coupling for its synchronization [9]. We compare Lotus with two state-of-the-art RDMA-based B+-tree indexes, i.e., Sherman/Deft [41, 43] and DEX [26], which all use one-sided RDMA for remote memory access. • Sherman [43] is a write-optimized B+-tree designed for nonpartitioned memory that leverages shortcut hint caching. • Deft [41] extends Sherman by segmenting tree nodes to reduce network amplification and uses shared-exclusive latching. • DEX [26] targets co-partitioned memory to eliminate cache coherence across compute servers and employs opportunistic operation pushdown to memory servers. We use the open-source implementations of Sherman and DEX from the authors. We use the same parameter settings as in prior work, e.g., RPC pushdown rate in DEX. We modify Sherman’s optimistic synchronization since its original version may lead to incorrect behavior [56]. We do not directly use Deft because it heavily relies on experimental one-sided RDMA atomic operations, i.e., extended CAS and extended masked FAA, which were deprecated in 20201 . Instead, we adjust the node size in Sherman to match the segmented node size in Deft to demonstrate its reduced network I/O. 7.3.3 Concurrency Control. We evaluate three 2PL protocols: NO _WAIT, WAIT_DIE, and WOUND_WAIT [4, 5]. For Lotus, we implement all three protocols using two-sided RDMA, as discussed 1 https://docs.nvidia.com/networking/display/ofedv512580/release+notes

(a) Workload A

100 0

400

(b) Workload B

500

200 1 4 8 16

32 40

Number of Threads per Compute Server

0

(c) Workload C

250 1 4 8 16

32 40

Number of Threads per Compute Server

0

1 4 8 16

32 40

Number of Threads per Compute Server

in Section 4, together with two-phase commit (2PC) [18]. For Sherman/Deft, we implement NO_WAIT and WAIT_DIE using onesided RDMA, following prior work [40]. WOUND_WAIT is not included because implementing preemption is extremely difficult due to limited primitives of one-sided RDMA, as discussed in Section 2.3. Note that WAIT_DIE in Sherman/Deft does not guarantee starvation prevention for the same reason. For co-partitioned memory systems, i.e., Lotus and DEX, we focus on indexing performance. As discussed in Section 2.3.2, concurrency control in co-partitioned memory can be handled locally in compute servers, thus the system performance depends solely on indexing efficiency. 7.3.4 Fault Tolerance. While fault tolerance is a critical component in distributed systems, disaggregated architectures still present many open design choices for implementing logging and recovery, particularly with respect to network transports. Some systems adopt a conservative approach by employing centralized log servers [3, 8, 34, 39], while others decentralize logging by decomposing conventional fault tolerance protocols to independently handle failures in compute servers and memory servers [37, 48, 54]. Moreover, these design choices fuel an ongoing debate between one-sided RDMA and two-sided RDMA for logging and recovery [7, 36, 44, 53]. Therefore, we leave fault tolerance as future work and focus on indexing and concurrency control in this paper.

7.4

Scalability Analysis

7.4.1 Throughput. We first evaluate the scalability of Lotus against Sherman/Deft. Figure 5 shows the throughput for workloads A, B, and C with a varying number of compute threads. For workloads A and B, all schemes do not scale due to conflicts. However, Lotus shows much higher throughput than Sherman/Deft. Sherman/Deft lacks functional capabilities to implement starvation prevention and priority-based scheduling due to limited APIs of one-sided RDMA. This significantly degrades performance when frequent read-write conflicts occur. The impact of high contention on Lotus is much smaller, because of its rich functionalities in concurrency control that handle conflicts based on transaction priorities. Specifically, Lotus WOUND_WAIT achieves the highest throughput and sustains good performance with an increasing number of compute threads as it avoids excessive lock thrashing. For workload C, all schemes scale well as the number of compute threads increases, since there are no read-write conflicts. Lotus generally outperforms Sherman/Deft. In Sherman/Deft, network amplification limits system performance since each transaction request requires multiple RDMA READs for index traversal. In contrast, Lotus minimizes the number of network round-trips by

Two-sided RDMA Striking Back for Disaggregated Memory Databases

99.9th 2 1

0

0

(a) Workload A

Commit

600

Abort

(b) Workload C

0

Lotus WOUND_WAIT

(b) Workload B

(c) Workload C

20

10

2

10 1

2

4

8

0

16 32

Number of Requests per Transaction

1 1

2

4

8

16 32

1

Number of Requests per Transaction

2

4

8

16 32

Number of Requests per Transaction

Figure 8: Relative performance with varying transaction lengths in YCSB workloads, 40 threads per compute server. Table 1: Average number of requests per message in Lotus. Req/Txn Workload

1

2

4

8

16

32

A B C

1.14 1.17 1.18

1.21 1.48 1.53

1.53 1.91 2.25

2.28 2.33 3.82

4.09 4.11 7.16

8.07 8.12 12.77

20

400 10

Average

0

99.9th

99.99th

(b) 32 requests per transaction

Sherman/Deft WAIT_DIE

Sherman/Deft NO_WAIT

Lotus WOUND_WAIT

Lotus WAIT_DIE

0

Lotus NO_WAIT

0

Sherman/Deft WAIT_DIE

250 Sherman/Deft NO_WAIT

batching multiple requests in a network message in two ways, i.e., intra-transaction and inter-transaction.

99th 500

50

Lotus WOUND_WAIT

Figure 7: Network traffic in YCSB-A and C.

95th

(a) 4 requests per transaction Latency (ms)

Lotus NO_WAIT Lotus WAIT_DIE Lotus WOUND_WAIT Sherman/Deft NO_WAIT Sherman/Deft WAIT_DIE

0

Lotus WAIT_DIE

200

Lotus NO_WAIT Lotus WAIT_DIE Lotus WOUND_WAIT Sherman/Deft NO_WAIT Sherman/Deft WAIT_DIE

Network Traffic per Transaction (KB)

Figure 6: Latency percentile breakdown in YCSB workloads, 40 threads per compute server.

Lotus WAIT_DIE Sherman/Deft WAIT_DIE

(a) Workload A

Lotus NO_WAIT

0

200 Lotus NO_WAIT Lotus WAIT_DIE Lotus WOUND_WAIT Sherman/Deft NO_WAIT Sherman/Deft WAIT_DIE

250

Lotus NO_WAIT Sherman/Deft NO_WAIT

99.99th

(c) Workload C

Speedup Rate over Sherman/Deft NO_WAIT

99th

(b) Workload B

Lotus NO_WAIT Lotus WAIT_DIE Lotus WOUND_WAIT Sherman/Deft NO_WAIT Sherman/Deft WAIT_DIE

95th

500

Lotus NO_WAIT Lotus WAIT_DIE Lotus WOUND_WAIT Sherman/Deft NO_WAIT Sherman/Deft WAIT_DIE

Latency (ms)

Average

(a) Workload A

7.4.2 Latency. Next, we analyze the latency of Lotus and Sherman/Deft. Figure 6 shows the latency percentile breakdown. For workloads A and B, NO_WAIT schemes show high tail latency as they do not consider transaction priorities upon conflicts. While WAIT_DIE schemes report lower latency, the latency of Sherman/Deft WAIT_DIE is much higher than Lotus WAIT_DIE since Sherman/Deft WAIT_DIE does not guarantee starvation prevention, as discussed in Section 2.3.2. Lotus WOUND_WAIT shows the lowest latency as it allows preemption in addition to priority-based scheduling. For workload C, all schemes report much lower latency than workloads A and B as it is read-only. Lotus shows lower average latency than Sherman/Deft because Lotus minimizes the number of network round-trips with batching, while Sherman/Deft suffers from network amplification. However, Lotus shows higher tail latency due to coordination overhead in inter-transaction batching.

Figure 9: Latency percentile breakdown in YCSB-A, 40 threads per compute server.

7.5

7.6

Network Traffic Analysis

To understand the impact of network amplification, we measure the network traffic of each scheme during the execution of YCSB workloads. For Lotus, we measure the traffic of two-sided RDMA operations, i.e., SEND and RECV. For Sherman/Deft, we measure the traffic of one-sided RDMA operations, i.e., READ, WRITE, CAS, and FAA. We do not include the traffic of data transfers for committed transactions, as all schemes transfer the same amount. Figure 7 shows the average network traffic per transaction for workloads A and C. All schemes generate more traffic in workload A than in C, due to read-write conflicts. However, Lotus consistently incurs much less traffic than Sherman/Deft. This is because Lotus only transfers request and response information in network messages. Although Sherman/Deft employs caching, it still needs to traverse the index from the root node to a leaf node upon a

cache miss and further requires multiple RDMA READs for correct synchronization during traversal. In workload A, Lotus WAIT_DIE and Lotus WOUND_WAIT generate less traffic compared to Lotus NO_WAIT since priority-based scheduling and preemption effectively handle conflicts, reducing the amount of transaction aborts. However, Sherman/Deft WAIT_ DIE incurs more traffic than Sherman/Deft NO_WAIT due to additional priority management. Sherman/Deft WAIT_DIE requires extra RDMA operations to check and update transaction priorities in lock acquisition. In workload C, Lotus schemes show significantly reduced network traffic compared to Sherman/Deft schemes by minimizing the number of network round-trips via batching.

Varying Transaction Lengths

Next, we examine how transaction length affects performance. We vary the number of requests per transaction from 1 to 32 to provide a comprehensive performance analysis from short transactions to long transactions. We report the relative performance of each scheme against Sherman/Deft NO_WAIT since throughput varies substantially across transaction lengths. Figure 8 shows the relative throughput for YCSB workloads. Sherman/Deft shows slightly better performance than Lotus for short transactions, but as the transaction length increases, Lotus outperforms Sherman/Deft. Lotus effectively leverages intra-transaction batching and inter-transaction batching as it can aggregate multiple requests into a network message, which substantially reduces the number of network round-trips. In contrast, Sherman/Deft handles each request individually using one-sided RDMA, which leads to

Hokeun Cha, Aditya Akella, and Xiangyao Yu

WAIT_DIE

75

600

50

400

25

200

0

illa ing XN XN Vanotus Cach ntra-Thing nter-Thing L + +I Batc +I Batc

0

Network

WOUND_WAIT

(b) Workload C

Sherman/Deft (w/ caching) Sherman/Deft (w/o caching) illa ing XN XN Vanotus Cach ntra-Thing nter-Thing L + +I Batc +I Batc

Figure 10: Throughput analysis of Lotus in YCSB-A and C.

illa ing XN XN Vanotus +Cach ntra-Thing nter-Thing L +I Batc +I Batc

99.99th

(b) Workload C

NW WD WW

0

99.9th

NW WD WW

1

0

NW WD WW

200 NW WD WW

2

NW WD WW

400

NW WD WW

Latency (ms)

99th

NW WD WW

95th

(a) Workload A

NW WD WW

Average

illa ing XN XN Vanotus +Cach ntra-Thing nter-Thing L +I Batc +I Batc

Figure 11: Latency analysis of Lotus in YCSB-A and C (NW: NO_WAIT, WD: WAIT_DIE, WW: WOUND_WAIT). an increased number of round-trips in proportion to the number of requests in a transaction. Table 1 reports the average number of requests per message in Lotus, showing improved batching effects as transactions grow longer. Note that the averages are lower than the total number of requests per transaction as each transaction spans multiple partitions across memory server, requiring requests to be split into separate messages targeting different servers. We further analyze the latency of each scheme in workload A under different transaction lengths. Figure 9 shows latency percentile breakdown for short transactions (4 requests per transaction) and long transactions (32 requests per transaction). While all schemes report similar average latency for short transactions, Lotus WAIT _DIE and WOUND_WAIT, as well as Sherman/Deft WAIT_DIE, show lower tail latency due to their consideration of transaction priorities. For long transactions, the latency gap between protocols that guarantee starvation prevention (i.e., Lotus WAIT_DIE and Lotus WOUND_WAIT) and those do not (i.e., Lotus NO_WAIT, Sherman/Deft NO_WAIT, and Sherman/Deft WAIT_DIE) becomes significantly larger. This highlights the importance of starvation prevention in concurrency control as more frequently transactions abort, they suffer increasingly from lock thrashing.

7.7

Analysis of Design Factors

In this section, we analyze the performance impact of design factors in Lotus. We first break down each optimization, i.e., caching, intratransaction batching, and inter-transaction batching, and apply them incrementally to vanilla Lotus. 7.7.1 Throughput and Latency. Figure 10 and 11 show the throughput and latency of each optimization across concurrency control protocols for YCSB workloads, respectively. Caching reduces data processing overhead in memory servers as cache hints help quickly locate data while bypassing index traversal. This reduces tail latency and improves throughput by up to 1.3× compared to the baseline, vanilla Lotus. Intra-transaction batching minimizes the number of network round-trips per transaction by combining multiple requests within

Runtime Overhead (%)

Throughput (Ktxn/sec)

NO_WAIT

(a) Workload A

100 75 50 25 0

Lock

(a) Workload A

illa ing XN XN Vanotus +Cach ntra-Thing nter-Thing L +I Batc +I Batc

Index 100 75 50 25 0

Data

Auxiliary

(b) Workload C

illa ing XN XN Vanotus +Cach ntra-Thing nter-Thing L +I Batc +I Batc

Figure 12: Runtime breakdown of Lotus in YCSB-A and C. a transaction into a single network message. The throughput improves by up to 1.8× and 1.7× for workloads A and C, respectively, compared to the previous increment. The performance gain is higher in workload A because batching also shortens the duration of locks in proportion to the reduced number of network roundtrips, mitigating lock contention. The reduced number of network round-trips also contributes to significant reduction in tail latency. Inter-transaction batching further reduces the number of network messages by combining requests from multiple transactions in a group into a single message. While this significantly improves the performance in workload C, it may cause performance degradation in workload A. Read-write conflicts lead to transaction aborts, making batch groups difficult to make optimal batching decisions due to backoffs, as discussed in Section 6.2. The results demonstrate the effectiveness of our adaptive batching strategy, improving the throughput by up to 1.2× and 1.5× in workloads A and C, respectively. This also reduces the average latency but increases the tail latency in both workloads due to coordination overhead in batching. 7.7.2 CPU Utilization in Memory Servers. We further analyze the impact of optimization techniques in Lotus on CPU utilization at memory servers. For deeper insight, we break down the execution time into five categories: • Network includes time spent on RDMA SEND/RECV operations. • Lock indicates time spent on concurrency control. • Index measures time spent on index traversal. • Data includes time spent on data reads and writes. • Auxiliary covers system overhead, including message parsing, metadata management, and memory management. Figure 12 presents the runtime breakdown of Lotus WOUND_ WAIT for workloads A and C. We do not include results for Sherman/Deft since it uses memory server CPUs only for auxiliary operations. Caching reduces indexing overhead by allowing memory server CPUs to directly locate data using hints from compute server caches, bypassing index traversal. However, its overall impact is limited since network remains the primary bottleneck, as each transaction request requires a network round-trip. Intra-transaction batching and inter-transaction batching effectively mitigate this bottleneck by significantly reducing the number of network roundtrips. While these optimizations enable more balanced utilization across system components, their impact is comparatively limited in workload A due to frequent read-write conflicts.

7.8

Analysis on Unreliable Datagram Transport

Next, we evaluate Lotus across different RDMA transports, including RC and UD, as prior studies on two-sided RDMA have primarily focused on UD [20–22]. We decompose the batching factors of Lotus,

RC + Intra-TXN Batching UD + Intra-TXN Batching

(a) Workload A

100

(b) Workload C

500

50 0

RC + Inter-TXN Batching UD + Inter-TXN Batching

64B

128B 256B 512B

1KB

0

64B

128B 256B 512B

1KB

Record Size Record Size Figure 13: Throughput of Lotus on different RDMA transports under varying record sizes, in YCSB-A and C.

i.e., intra-transaction batching and inter-transaction batching, and incrementally apply them to vanilla Lotus since UD transport has limited support for large message sizes, as discussed in Section 6.3. Figure 13 shows the throughput of Lotus WOUND_WAIT on RC and UD transports in workloads A and C under varying record sizes. Overall, RC-based schemes generally achieve lower throughput than UD-based ones, due to additional overhead in reliability mechanisms such as acknowledgments and in-order delivery guarantees. However, UD-based schemes fail to support large record sizes in both workloads, resulting in missing data points in the figure. This limitation arises as network message sizes exceed the hardware MTU, as discussed in Section 6.3. Those large messages cannot be transmitted using UD without explicit application-level handling, which incurs additional network round-trips for packet fragmentation and reassembly of the out-of-order fragmented packets. In contrast, RC supports those features at the transport layer, making it more flexible for workloads with varying record sizes. While the results show several advantages of RC over UD, the choice of transports is orthogonal to Lotus ’s design. Lotus is compatible with both RC and UD, as well as other transport protocols, and can benefit from improvements in the underlying transport.

7.9

Comparison with DEX

Next, we compare Lotus with DEX [26], the state-of-the-art onesided RDMA-based indexing scheme that employs compute-side partitioning to eliminate cache coherence across compute servers. In this setting, each compute server logically owns a specific data partition so that cache coherence across multiple compute servers can be completely eliminated. We focus on indexing performance in this experiment, since both Lotus and DEX can perform concurrency control locally in compute servers without remote memory accesses, as discussed in Section 2.3.2. 7.9.1 Scalability Analysis. Figure 14 shows the throughput of Lotus and DEX in YCSB workloads with a varying number of compute threads. Lotus consistently outperforms DEX as it leverages a finer granularity of caching. Lotus caches individual records that are frequently accessed, while DEX caches index nodes that may contain many infrequently accessed ones. This allows Lotus to exploit better cache locality, especially in skewed workloads [12, 16, 55]. The performance gap between Lotus and DEX is larger in workload A than in B and C, due to the higher frequency of write operations that modify cache entries. Upon cache eviction of dirty entries, both Lotus and DEX must write back updated data to memory servers, incurring remote accesses. Lotus reduces this overhead by aggregating multiple write-back requests into a single network message, thereby minimizing the number of network round-trips.

Throughput (Mtxns/sec)

RC (vanilla) UD (vanilla)

Lotus

(a) Workload A

DEX

(b) Workload B

(c) Workload C

4

4

4

2

2

2

0

14 8

16

32 40

Number of Threads per Compute Server

0

14 8

16

32 40

Number of Threads per Compute Server

0

14 8

16

32 40

Number of Threads per Compute Server

Figure 14: Throughput with a varying number of compute threads in YCSB workloads.

Network Traffic per Transaction (KB)

Throughput (Ktxns/sec)

Two-sided RDMA Striking Back for Disaggregated Memory Databases

(a) Workload A

(b) Workload B

(c) Workload C

10

5.0

5.0

5

2.5

2.5

0

0.0

Lotus

DEX

Lotus

DEX

0.0

Lotus

DEX

Figure 15: Network traffic in YCSB workloads. However, DEX handles each write-back request in an either way: direct data update via RDMA WRITE or operation pushdown via RDMA SEND/RECV, incurring one network round-trip per request. 7.9.2 Network Traffic Analysis. Now, we compare the network traffic of Lotus and DEX. As in Section 7.5, for Lotus, we measure the traffic of two-sided RDMA operations. For DEX, we measure both one-sided RDMA operations and two-sided RDMA operations for pushdown requests as it employs opportunistic offloading to leverage the benefits of both RDMA types. Figure 15 shows the average network traffic per transaction for each index under YCSB workloads. Overall, the co-partitioned memory configuration incurs substantially less traffic than the non-partitioned memory setup analyzed in Section 7.5. This reduction, particularly in workload A, stems from differences in memory models. Caching in co-partitioned memory avoids remote accesses, while non-partitioned memory systems still require remote accesses for cache coherence across compute servers, as discussed in Section 2.1. Moreover, in the non-partitioned memory setting, most traffic is due to aborted transactions. In this experiment, we focus on indexing, which does not introduce transaction aborts. Lotus generates significantly less network traffic than DEX across all workloads. Upon a cache miss, Lotus only exchanges request information and the response, which are only few bytes in size, and further reduces the number of exchanges through batching. In contrast, DEX needs multiple RDMA READs for index traversal upon a cache miss, and each node access incurs at least three RDMA READs to ensure correct synchronization [56]. In particular, DEX generates much higher traffic than Lotus in workload A, a writeintensive workload. Upon eviction of a modified cache entry, Lotus writes back only the updated record, whereas DEX writes back the entire index node containing that record. Although DEX employs operation pushdown that can reduce data movement, it is an opportunistic optimization and may still trigger multiple network round-trips when child nodes reside on different memory servers. 7.9.3 Cache Sensitivity Analysis. Next, we study the impact of cache size on index performance. Figure 16 shows the throughput of Lotus and DEX under varying cache sizes for workloads A and C. In workload A, Lotus outperforms DEX due to its lightweight

Throughput (Mtxn/sec)

Hokeun Cha, Aditya Akella, and Xiangyao Yu

(a) Workload A

Lotus

DEX (b) Workload C

10

10

5

5

0

2

4 8 16 32 64 Cache Size / Dataset Size (%)

0

2

4 8 16 32 64 Cache Size / Dataset Size (%)

Lotus NO_WAIT Lotus WAIT_DIE Lotus WOUND_WAIT

200 0

12 4

8

16

Sherman/Deft NO_WAIT Sherman/Deft WAIT_DIE

Throughput (Mtxns/sec)

Throughput (Ktxns/sec)

Figure 16: Throughput under different cache sizes in YCSB-A and C. (a) Non-partitioned memory (b) Co-partitioned memory 5 0

12 4

Lotus

DEX

8

16

Number of Threads Number of Threads per Memory Server per Memory Server Figure 17: Throughput under a varying number of memory server threads in YCSB-B.

write-back mechanism. In workload C, Lotus shows better performance with small cache sizes as its fine-grained record-level caching improves cache utilization. As cache size grows, their performance eventually becomes comparable as both indexes can effectively serve most requests from local cache.

7.10

Sensitivity on Memory Server CPUs

Finally, we examine the performance impact of computing resources on memory servers under both non-partitioned memory and copartitioned memory. We vary the number of memory server threads and adjust the number of compute threads accordingly. Figure 17(a) shows the performance under non-partitioned memory. Lotus shows increasing throughput as the number of threads increases, effectively leveraging parallelism in both network and data processing. We report the performance of Sherman/Deft with its best configuration since memory server threads are only used for auxiliary tasks such as remote memory allocation and registration. Figure 17(b) shows the performance under co-partitioned memory. Lotus continues to scale up to 8 threads per memory server, but its performance decreases at 16 threads. This is because memory servers become underutilized when fewer compute threads issue remote requests. In particular, caching in co-partitioned memory fundamentally reduces the number of remote accesses. DEX also utilizes memory server threads for operation pushdown. However, reducing the number of compute threads has a more negative impact on performance than increasing memory server threads. Overall, these results show that Lotus outperforms Sherman/Deft in non-partitioned memory and achieves comparable performance to DEX in co-partitioned memory, even under constrained resources. They also indicate that memory server CPU allocation can serve as an effective performance tuning knob.

8

Related work

RDMA indexes. Various index designs have been proposed to efficiently leverage RDMA. NAM-tree [57] explores three alternative B+-tree designs that can be implemented with one-sided RDMA, two-sided RDMA, and a hybrid of the two. Sherman [43] focuses on one-sided RDMA and uses shortcut hint caching to reduce

RDMA READs during index traversal. Deft [41] extends Sherman by segmenting index nodes to reduce the size of RDMA READs. ROLEX [25] replaces B+-tree nodes with learned models that are much smaller in size, reducing network traffic during index traversal. DEX adopts a co-partitioned memory model to avoid remote access for cached data retrieval. While these systems primarily focus on one-sided RDMA to mitigate network amplification, Lotus leverages two-sided RDMA to reduce network traffic at the source. Shared memory databases with RDMA. Early studies on RDMA databases have explored various design alternatives for distributed transactions. Pilaf [30] adopts a hybrid approach, performing onesided RDMA for read requests while handling write requests with two-sided RDMA. FaRM [13, 14, 33] exploits one-sided RDMA to minimize remote CPU involvement. HERD [21] and FaSST [22] discuss the drawbacks of one-sided RDMA, i.e., making system software inherently complex, by revisiting data processing with two-sided RDMA. DrTMs [45, 46] leverage hardware transactional memory and RDMA to transform distributed transactions into local transactions. While prior work targets monolithic server architectures, we consider disaggregated architectures, which lead to different design focuses. Our optimization techniques focus on reducing network round-trips to mitigate the overhead of data processing and network processing in memory servers. Disaggregated memory databases with RDMA. NAM-DB [51] proposes a disaggregated memory database by leveraging one-sided RDMA. RCC [40] evaluates various concurrency control protocols and provides insights to transform those legacy protocols to support one-sided RDMA. In this work, we tackle the fundamental limitations of the one-sided RDMA approaches in indexing and concurrency control, i.e., network amplification and lack of functionality, and provide practical solutions using two-sided RDMA that support starvation prevention, priority-based scheduling, and preemption. We also propose caching and batching techniques to further reduce network I/O while improving system performance. CXL memory systems. Compute Express Link (CXL) memory is an emerging technology that enables memory sharing via PCI Express interconnects. It is being actively explored for various system designs, including as remote caches [17, 24], key-value stores [6, 27], and in-memory databases [1, 2, 47]. While CXL memory has the potential to become a new standard for memory disaggregation, it is still in its early stages and limited to rack-scale deployments. In this work, we focus on RDMA-based memory disaggregation, which is widely deployed in modern cloud environments.

9

Conclusion

This paper revisits the long-standing debate between one-sided RDMA and two-sided RDMA in the context of disaggregated memory databases. We present Lotus, a two-sided RDMA-based disaggregated memory OLTP system, and challenge the prevailing view that one-sided RDMA is inherently superior due to limited computing resources in memory servers. Lotus enables rich functionality for indexing and concurrency control and turns this limitation into a strength by leveraging caching and batching to minimize network round-trips and efficiently utilize remote CPU resources. Our performance study demonstrates that Lotus outperforms one-sided RDMA-based schemes across various workload scenarios.

Two-sided RDMA Striking Back for Disaggregated Memory Databases

References [1] Minseon Ahn, Andrew Chang, Donghun Lee, Jongmin Gim, Jungmin Kim, Jaemin Jung, Oliver Rebholz, Vincent Pham, Krishna Malladi, and Yang Seok Ki. 2022. Enabling CXL memory expansion for in-memory database management systems. In Proceedings of the 18th International Workshop on Data Management on New Hardware (Philadelphia, PA, USA) (DaMoN ’22). Association for Computing Machinery, New York, NY, USA, Article 8, 5 pages. [2] Minseon Ahn, Thomas Willhalm, Norman May, Donghun Lee, Suprasad Mutalik Desai, Daniel Booss, Jungmin Kim, Navneet Singh, Daniel Ritter, and Oliver Rebholz. 2024. An examination of CXL memory use cases for in-memory database management systems using SAP HANA. Proc. VLDB Endow. 17, 12 (Aug. 2024), 3827–3840. [3] Panagiotis Antonopoulos, Alex Budovski, Cristian Diaconu, Alejandro Hernandez Saenz, Jack Hu, Hanuma Kodavalla, Donald Kossmann, Sandeep Lingam, Umar Farooq Minhas, Naveen Prakash, Vijendra Purohit, Hugh Qu, Chaitanya Sreenivas Ravella, Krystyna Reisteter, Sheetal Shrotri, Dixin Tang, and Vikram Wakade. 2019. Socrates: the new SQL server in the cloud. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD ’19). Association for Computing Machinery, New York, NY, USA, 1743–1756. [4] P.A. Bernstein, D.W. Shipman, and W.S. Wong. 1979. Formal aspects of serializability in database concurrency control. IEEE Transactions on Software Engineering SE-5, 3 (1979), 203–216. [5] Philip A. Bernstein and Nathan Goodman. 1981. Concurrency control in distributed database systems. ACM Comput. Surv. 13, 2 (jun 1981), 185–221. [6] Miao Cai, Junru Shen, Yifan Yuan, Zhihao Qu, and Baoliu Ye. 2023. BonsaiKV: towards fast, scalable, and persistent key-value stores with tiered, heterogeneous memory system. Proc. VLDB Endow. 17, 4 (Dec. 2023), 726–739. [7] Qingchao Cai, Wentian Guo, Hao Zhang, Divyakant Agrawal, Gang Chen, Beng Chin Ooi, Kian-Lee Tan, Yong Meng Teo, and Sheng Wang. 2018. Efficient distributed memory management with rdma and caching. Proc. VLDB Endow. 11, 11 (jul 2018), 1604–1617. [8] Wei Cao, Yingqiang Zhang, Xinjun Yang, Feifei Li, Sheng Wang, Qingda Hu, Xuntao Cheng, Zongzhi Chen, Zhenjun Liu, Jing Fang, Bo Wang, Yuhui Wang, Haiqing Sun, Ze Yang, Zhushi Cheng, Sen Chen, Jian Wu, Wei Hu, Jianwei Zhao, Yusong Gao, Songlu Cai, Yunyang Zhang, and Jiawang Tong. 2021. Polardb serverless: a cloud native database for disaggregated data centers. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD ’21). Association for Computing Machinery, New York, NY, USA, 2477–2489. [9] Sang K. Cha, Sangyong Hwang, Kihong Kim, and Keunjoo Kwon. 2001. Cacheconscious concurrency control of main-memory indexes on shared-memory multiprocessor systems. In Proceedings of the 27th International Conference on Very Large Data Bases (VLDB ’01). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 181––190. [10] Youmin Chen, Youyou Lu, and Jiwu Shu. 2019. Scalable RDMA RPC on reliable connection with efficient resource sharing. In Proceedings of the Fourteenth EuroSys Conference 2019 (Dresden, Germany) (EuroSys ’19). Association for Computing Machinery, New York, NY, USA. [11] Brian Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, 143–154. [12] Justin DeBrabant, Andrew Pavlo, Stephen Tu, Michael Stonebraker, and Stan Zdonik. 2013. Anti-caching: a new approach to database management system architecture. Proc. VLDB Endow. 6, 14 (Sept. 2013), 1942–1953. [13] Aleksandar Dragojević, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: fast remote memory. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). USENIX Association, Seattle, WA, 401–414. [14] Aleksandar Dragojević, Dushyanth Narayanan, Edmund B. Nightingale, Matthew Renzelmann, Alex Shamis, Anirudh Badam, and Miguel Castro. 2015. No compromises: distributed transactions with consistency, availability, and performance. In Proceedings of the 25th Symposium on Operating Systems Principles (Monterey, California) (SOSP ’15). Association for Computing Machinery, New York, NY, USA, 54–70. [15] Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. 2019. The design and operation of CloudLab. In Proceedings of the USENIX Annual Technical Conference (ATC). 1–14. [16] Ahmed Eldawy, Justin Levandoski, and Per-Åke Larson. 2014. Trekking through Siberia: managing cold data in a memory-optimized database. Proc. VLDB Endow. 7, 11 (July 2014), 931–942. [17] Donghyun Gouk, Sangwon Lee, Miryeong Kwon, and Myoungsoo Jung. 2022. Direct access, High-Performance memory disaggregation with DirectCXL. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX Association,

Carlsbad, CA, 287–294. [18] Jim Gray. 1978. Notes on data base operating systems. In Operating Systems, An Advanced Course. Springer-Verlag, Berlin, Heidelberg, 393–481. [19] Daegyu Han, Jaeyoon Nam, Hokeun Cha, Changdae Kim, Kwangwon Koh, Taehoon Kim, Sang-Hoon Kim, and Beomseok Nam. 2025. Disaggregated memory for file-backed pages. ACM Trans. Storage 21, 4, Article 41 (Nov. 2025), 27 pages. [20] Anuj Kalia, Michael Kaminsky, and David Andersen. 2019. Datacenter RPCs can be general and fast. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, Boston, MA, 1–16. [21] Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA efficiently for key-value services. In Proceedings of the 2014 ACM Conference on SIGCOMM (Chicago, Illinois, USA) (SIGCOMM ’14). Association for Computing Machinery, New York, NY, USA, 295–306. [22] Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. FaSST: fast, scalable and simple distributed transactions with two-Sided RDMA) datagram RPCs. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 185–201. [23] H. T. Kung and John T. Robinson. 1981. On optimistic methods for concurrency control. ACM Trans. Database Syst. 6, 2 (jun 1981), 213–226. [24] Huaicheng Li, Daniel S. Berger, Lisa Hsu, Daniel Ernst, Pantea Zardoshti, Stanko Novakovic, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, and Ricardo Bianchini. 2023. Pond: CXL-based memory pooling systems for cloud platforms. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Vancouver, BC, Canada) (ASPLOS 2023). Association for Computing Machinery, New York, NY, USA, 574–587. [25] Pengfei Li, Yu Hua, Pengfei Zuo, Zhangyu Chen, and Jiajie Sheng. 2023. ROLEX: A scalable RDMA-oriented learned key-value store for disaggregated memory systems. In 21st USENIX Conference on File and Storage Technologies (FAST 23). USENIX Association, Santa Clara, CA, 99–114. [26] Baotong Lu, Kaisong Huang, Chieh-Jan Mike Liang, Tianzheng Wang, and Eric Lo. 2024. DEX: scalable range indexing on disaggregated memory. Proc. VLDB Endow. 17, 10 (June 2024), 2603–2616. [27] Mengting Lu, Gaocong Liu, Kun Wang, Feng Zhu, and Shu Li. 2025. CHash: a high cost-performance hash design for CXL-based disaggregated memory system. In Abstracts of the 2025 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (Stony Brook, NY, USA) (SIGMETRICS ’25). Association for Computing Machinery, New York, NY, USA, 124–126. [28] James M. Lucas and Michael S. Saccucci. 1990. Exponentially weighted moving average control schemes: properties and enhancements. Technometrics 32, 1 (1990), 1–12. [29] Xuchuan Luo, Pengfei Zuo, Jiacheng Shen, Jiazhen Gu, Xin Wang, Michael R. Lyu, and Yangfan Zhou. 2023. SMART: a high-performance adaptive radix tree for disaggregated memory. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 553–571. [30] Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using one-sided RDMA reads to build a fast, CPU-efficient key-value store. In 2013 USENIX Annual Technical Conference (USENIX ATC 13). USENIX Association, San Jose, CA, 103– 114. [31] NVIDIA Corporation. 2023. RDMA aware networks programming user manual. https://docs.nvidia.com/networking/display/rdmaawareprogrammingv17. [32] R. Pagh and G. A. Gibson. 2004. Cuckoo hashing. Journal of Algorithms 51, 2 (2004), 122–144. [33] Alex Shamis, Matthew Renzelmann, Stanko Novakovic, Georgios Chatzopoulos, Aleksandar Dragojević, Dushyanth Narayanan, and Miguel Castro. 2019. Fast general distributed transactions with opacity. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD ’19). Association for Computing Machinery, New York, NY, USA, 433–448. [34] Jiacheng Shen, Pengfei Zuo, Xuchuan Luo, Tianyi Yang, Yuxin Su, Yangfan Zhou, and Michael R. Lyu. 2023. FUSEE: A fully memory-disaggregated key-value store. In 21st USENIX Conference on File and Storage Technologies (FAST 23). USENIX Association, Santa Clara, CA, 81–98. [35] Jeff Shute, Radek Vingralek, Bart Samwel, Ben Handy, Chad Whipkey, Eric Rollins, Mircea Oancea, Kyle Littlefield, David Menestrina, Stephan Ellner, John Cieslewicz, Ian Rae, Traian Stancescu, and Himani Apte. 2013. F1: a distributed SQL database that scales. Proc. VLDB Endow. 6, 11 (Aug. 2013), 1068–1079. [36] Yacine Taleb, Ryan Stutsman, Gabriel Antoniu, and Toni Cortes. 2018. Tailwind: fast and atomic RDMA-based replication. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 851–863. [37] Shin-Yeh Tsai, Yizhou Shan, and Yiying Zhang. 2020. Disaggregating persistent memory and controlling them remotely: an exploration of passive disaggregated key-value stores. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 33–48. [38] Michalis Vardoulakis, Giorgos Saloustros, Pilar González-Férez, and Angelos Bilas. 2022. Tebis: index shipping for efficient replication in LSM key-value stores. In Proceedings of the Seventeenth European Conference on Computer Systems (Rennes, France) (EuroSys ’22). Association for Computing Machinery, New York, NY, USA, 85—-98.

Hokeun Cha, Aditya Akella, and Xiangyao Yu

[39] Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, and Xiaofeng Bao. 2017. Amazon Aurora: design considerations for high throughput cloud-native relational databases. In Proceedings of the 2017 ACM International Conference on Management of Data (Chicago, Illinois, USA) (SIGMOD ’17). Association for Computing Machinery, New York, NY, USA, 1041–1052. [40] Chao Wang and Xuehai Qian. 2023. RDMA-enabled concurrency control protocols for transactions in the cloud era. IEEE Transactions on Cloud Computing 11, 1 (2023), 798–810. [41] Jing Wang, Qing Wang, Yuhao Zhang, and Jiwu Shu. 2025. Deft: a scalable tree index for disaggregated memory. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 886–901. [42] Jianguo Wang and Qizhen Zhang. 2023. Disaggregated database systems. In Companion of the 2023 International Conference on Management of Data (Seattle, WA, USA) (SIGMOD ’23). Association for Computing Machinery, New York, NY, USA, 37–44. [43] Qing Wang, Youyou Lu, and Jiwu Shu. 2022. Sherman: a write-optimized distributed B+tree index on disaggregated memory. In Proceedings of the 2022 International Conference on Management of Data (Philadelphia, PA, USA) (SIGMOD ’22). Association for Computing Machinery, New York, NY, USA, 1033–1048. [44] Ruihong Wang, Jianguo Wang, Stratos Idreos, M. Tamer Özsu, and Walid G. Aref. 2022. The case for distributed shared-memory databases with RDMA-enabled memory disaggregation. Proc. VLDB Endow. 16, 1 (sep 2022), 15–22. [45] Xingda Wei, Zhiyuan Dong, Rong Chen, and Haibo Chen. 2018. Deconstructing RDMA-enabled distributed transactions: hybrid is better!. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 233–251. [46] Xingda Wei, Jiaxin Shi, Yanzhe Chen, Rong Chen, and Haibo Chen. 2015. Fast in-memory transaction processing using RDMA and HTM. In Proceedings of the 25th Symposium on Operating Systems Principles (Monterey, California) (SOSP ’15). Association for Computing Machinery, New York, NY, USA, 87–104. [47] Xinjun Yang, Yingqiang Zhang, Hao Chen, Feifei Li, Gerry Fan, Yang Kong, Bo Wang, Jing Fang, Yuhui Wang, Tao Huang, Wenpu Hu, Jim Kao, and Jianping Jiang. 2025. Unlocking the potential of CXL for disaggregated memory in cloud-native databases. In Companion of the 2025 International Conference on Management of Data (Berlin, Germany) (SIGMOD/PODS ’25). Association for Computing Machinery, New York, NY, USA, 689–702. [48] Xinjun Yang, Yingqiang Zhang, Hao Chen, Feifei Li, Bo Wang, Jing Fang, Chuan Sun, and Yuhui Wang. 2024. PolarDB-MP: a multi-primary cloud-native database via disaggregated shared memory. In Companion of the 2024 International Conference on Management of Data (Santiago AA, Chile) (SIGMOD ’24). Association for Computing Machinery, New York, NY, USA, 295–308. [49] Chenhao Ye, Wuh-Chwen Hwang, Keren Chen, and Xiangyao Yu. 2023. Polaris: enabling transaction priority in optimistic concurrency control. Proc. ACM Manag. Data 1, 1, Article 44 (may 2023), 24 pages. [50] Xiangyao Yu, George Bezerra, Andrew Pavlo, Srinivas Devadas, and Michael Stonebraker. 2014. Staring into the abyss: an evaluation of concurrency control with one thousand cores. Proc. VLDB Endow. 8, 3 (nov 2014), 209–220. [51] Erfan Zamanian, Carsten Binnig, Tim Harris, and Tim Kraska. 2017. The end of a myth: distributed transactions can scale. 10, 6 (feb 2017), 685–696. [52] Erfan Zamanian, Julian Shun, Carsten Binnig, and Tim Kraska. 2020. Chiller: contention-centric transaction execution and data partitioning for modern networks. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (Portland, OR, USA) (SIGMOD ’20). Association for Computing Machinery, New York, NY, USA, 511–526. [53] Erfan Zamanian, Xiangyao Yu, Michael Stonebraker, and Tim Kraska. 2019. Rethinking database high availability with rdma networks. Proc. VLDB Endow. 12, 11 (jul 2019), 1637—-1650. [54] Yingqiang Zhang, Chaoyi Ruan, Cheng Li, Xinjun Yang, Wei Cao, Feifei Li, Bo Wang, Jing Fang, Yuhui Wang, Jingze Huo, and Chao Bi. 2021. Towards costeffective and elastic cloud database deployment via memory disaggregation. Proc. VLDB Endow. 14, 10 (jun 2021), 1900–1912. [55] Xinjing Zhou, Xiangyao Yu, Goetz Graefe, and Michael Stonebraker. 2023. Two is better than one: the case for 2-tree for skewed data sets. In Proceedings of the 2023 Conference on Innovative Data Systems Research (CIDR 23). Amsterdam, The Netherlands. [56] Tobias Ziegler, Jacob Nelson-Slivon, Viktor Leis, and Carsten Binnig. 2023. Design guidelines for correct, efficient, and scalable synchronization using one-sided RDMA. Proc. ACM Manag. Data 1, 2, Article 131 (jun 2023), 26 pages. [57] Tobias Ziegler, Sumukha Tumkur Vani, Carsten Binnig, Rodrigo Fonseca, and Tim Kraska. 2019. Designing distributed tree-based index structures for fast RDMA-capable networks. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD ’19). Association for Computing Machinery, New York, NY, USA, 741–758. [58] Pengfei Zuo, Jiazhao Sun, Liu Yang, Shuangwu Zhang, and Yu Hua. 2021. Onesided RDMA-conscious extendible hashing for disaggregated memory. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association,

15–29.