RAMC: Remote Access Memory Channels over HPE Slingshot Whit Schonbein Matthew G.F. Dosanjh Scott Levy [email protected] [email protected] [email protected] Sandia National Laboratories Albuquerque, New Mexico, USA
arXiv:2606.05094v1 [cs.NI] 3 Jun 2026
Abstract In this paper, we present Remote Access Memory Channels (RAMC), an explicit one-sided communication library designed to leverage the capabilities of HPE Cray Slingshot network hardware. Existing one-sided communication frameworks, such as MPI RMA and OpenSHMEM, rely on monolithic shared memory models that introduce scalability and usability challenges. These frameworks often assume symmetric memory regions or require blocking collective operations for window creation, which can mismatch user communication needs and hinder performance. Implicit models, such as PGAS and UPC, aim to simplify programming by treating local and remote memory as a unified region but ultimately rely on explicit mechanisms to implement data movement. MPI’s recently-introduced partitioned communication API offers a persistent point-to-point interface but sacrifices the dynamic flexibility of RDMA. RAMC is designed to address these limitations. Based on the core concept of a persistent uni-directional communication channel, RAMC leverages Slingshot’s unique memory region counters to enable efficient completion notification. Experiments with a RAMC-based heat diffusion code demonstrate RAMC has no difficulty scaling to 19.6 thousand processes across 250 nodes, and microbenchmark studies across multiple libfabric versions show RAMC can outperform Cray’s proprietary MPI implementation (e.g., increases in bandwidth ranging from ∼100%-130% for 1B-4KiB messages under libfabric 1.15.2, and from ∼30%-45% under libfabric 2.3.1) while identifying additional areas for improvement, such as small message latencies.
CCS Concepts • Networks → Network architectures; Programming interfaces;
Keywords Networks, Network APIs, Message Passing Interface, Slingshot, Remote Memory Access
1
Introduction
Traditional point-to-point communication models face growing challenges from increasing intra-node parallelism incurred by higher CPU core counts and the use of accelerators such as GPUs. These models, exemplified by MPI’s message-passing paradigm, rely on complex branching code paths for message processing and destination negotiation. Such complexity can lead to performance bottlenecks, particularly in highly multi-threaded environments and on
processors less suited to branching, such as GPUs. To address these limitations, networks like HPE Cray’s Slingshot offer advanced support for one-sided remote direct memory access (RDMA) communication, enabling new possibilities for efficient and scalable communication. Existing one-sided communication frameworks, such as MPI RMA and OpenSHMEM, rely on monolithic shared memory models that introduce scalability and usability challenges. These frameworks often assume symmetric memory regions or require blocking collective operations for window creation or transitions between epochs, which can mismatch user communication needs and hinder performance. Implicit models, such as PGAS and UPC, aim to simplify programming by treating local and remote memory as a unified region but ultimately rely on explicit mechanisms at some level. MPI’s recently-introduced Partitioned Communication API offers a persistent point-to-point interface but sacrifices the dynamic flexibility of RDMA. These limitations motivate the need for a new approach that fully exploits modern network hardware capabilities. In this paper, we introduce Remote Access Memory Channels (RAMC), a novel explicit one-sided communication library designed to leverage the flexibility of HPE Cray’s Slingshot hardware. Based on a core building block of a persistent, uni-directional channel between initiator and target, RAMC enables dynamic and flexible communication patterns between processes. RAMC leverages Slingshot’s unique memory region counters to enable lightweight RMA completion for efficient synchronization, addressing key limitations of existing one-sided communication frameworks. In general, RAMC provides users with a lightweight, adaptable interface that simplifies the transition from traditional MPI point-to-point communication while unlocking the full potential of RDMA. To evaluate RAMC, we conduct correctness tests, scalability experiments using a heat diffusion code, and performance benchmarks under two different versions of libfabric. The scalability experiments show that RAMC has no difficulty scaling to 19.6 thousand processes across 250 nodes. Microbenchmark studies across multiple libfabric versions show RAMC can outperform Cray’s proprietary MPI implementation (e.g., increases in bandwidth ranging from ∼100%-130% for 1B-4KiB messages under libfabric 1.15.2, and from ∼30%-45% under libfabric 2.3.1) while identifying additional areas for improvement, such as small message latencies. The contributions of this paper are as follows: • Development of RAMC, a novel one-sided communication library leveraging advanced Slingshot features.
Schonbein et al.
delay
time
(a) Traditional RMA
delay
time
(b) Earlybird
Figure 1: A simple ‘three-point’ stencil illustrating how relaxed synchronization may accommodate process delay and create opportunities for communication/computation overlap in comparison to traditional approaches. Dashed lines indicate fences.
• Comprehensive exploration of RAMC’s design and implementation, including key architectural decisions. • Evaluation of RAMC’s functionality, scalability, and performance.
2
Motivation
Interprocess communication based on RDMA is an attractive prospect for a variety of reasons, including but not limited to avoiding target OS involvement in data movement operations, allowing buffers to be repeatedly reused once registered, and flexibility of data placement via offsets and ranges. For example, many modeling and simulation applications exhibit highly regular communication patterns (e.g., halo exchanges and collectives), repeatedly interacting with the same peers and exchanging similar data volumes into consistent regions of application memory. The setup and coordination of these patterns can be expensive, such as in alltoall and allreduce collectives. This highlights the need for persistence: A reusable communication interface where the overhead of remote access setup is incurred once, and the resulting data movement can be efficiently invoked many times. A natural fit for this is onesided communication leveraging the performance of modern RDMA networks. However, contemporary user-level communication libraries (such as the Message Passing Interface (MPI)) can impose constraints on how RMA is expected to be used, making it difficult to deploy in desired ways, and potentially impacting performance. For example, Levy et al. [18] propose a SmartNIC-based data movement service: During setup, a process on a SmartNIC is given access to certain regions of host memory, and when subsequently signaled by the host application process, pulls data from those regions for purposes of checkpointing or inserting into a distributed database. This seems like a prime candidate for RDMA communication. However, setting up such a communication channel is not straightforward. For instance, MPI_Win_create is a collective operation within an MPI communicator, where each process makes a buffer available to every other process in the communicator for subsequent RMA operations [21]. On one hand, because exposing buffers for RDMA
operations is monolithic (in the sense every process in a communicator participates), creating a simple point-to-point relation requires down-selecting to a group of size two and creating a new communicator object. On the other, even when this is done, the presupposed mutual involvement of subsequent RMA operations results in a superfluous buffer (on the SmartNIC) that may not be used. This example motivates an approach to RMA communication that is conceptually closer to traditional point-to-point communication than those that treat windows as group-level abstractions (such as MPI’s RMA windows or OpenSHMEM’s symmetric heap). For this alternative approach, the fundamental building block of RMA communication should simply be a relation between an initiator (who issues RMA operations) and a target (the specified buffer RMA operations act on). This alternative facilitates potential performance optimizations in group communication (such as halo exchanges and collectives) that may be difficult (but not impossible) to achieve under current APIs. Figure 1 shows an example of a three-point stencil, where each process (dot) exchanges data with its neighbors to the east and west. A traditional MPI RMA approach (Figure 1a) fences to guarantee all prior communication operations have completed for all processes (a collective operation) and then engages in data exchange before fencing again. In this contrived example, the first and fourth processes are delayed, and all data exchanges are likewise delayed until the first fence completes (dashed line). In contrast, if synchronization is not monolithic, and involves only checking that the intended target of an RMA operation is ready to handle that request, data exchange can occur in an ‘earlybird’ fashion: While some processes may be delayed, others can proceed with their data exchanges as they become ready, as illustrated in Figure 1b. The result is twofold: First, because data exchanges can occur while other processes are delayed, there is less communication to perform when the last process is ready, so the overall group operation may complete earlier than otherwise [20]. Second, because processes may receive their data while others are still in a communication phase, there is the possibility of doing work with that data, overlapping computation and communication. By reducing the need for global synchronization, this approach reduces the sensitivity of the composed application to performance perturbation and variability, see Ferreira et al. [8]. Although inter-process synchronization may still exist in the form of chains of point-topoint communication operations, the resulting dependencies from these operations impose less strict synchronization, see Levy et al. [17], and allow for communication delays to be absorbed rather than being compounded by global synchronization operations. Finally, we note that modern interconnects such as Cray HPE’s Slingshot provide increasingly capable one-sided communication support, including independent progress (once data is received by the NIC, it will eventually appear in application memory without host involvement) and hardware mechanisms for tracking the completion of RDMA operations despite the host being oblivious (e.g., Slingshot’s completion counters). In contrast, because they are largely organized around monolithic, collectively created windows and group-centric setup and synchronization semantics, current one-sided programming interfaces (including MPI RMA and OpenSHMEM) make it difficult for users to access the full one-sided
RAMC: Remote Access Memory Channels over HPE Slingshot
communication flexibility and performance that the hardware supports.
3
In this section we describe the overarching design goals guiding the development of RAMC, as well as its core concepts and functionality.
3.1
Design Goals
The first goal of RAMC is to avoid monolithic RMA window creation: Consistent with a point-to-point communication model, target windows should be able to be exposed to individual initiating processes, and without requiring initiating processes reciprocate with an RMA window of their own. Moreover, the window creation process should be non-blocking to avoid the possibility of deadlock when creating multiple target windows. This in turn implies window creation should utilize creation criteria other than call (program) order, such as MPI point-to-point’s tag matching semantics. Second, relations between initiators and targets should be genuinely persistent so as to underwrite the repeated use of target buffers by fixed or dynamic communication patterns. This is in contrast to, for example, MPI’s ‘persistent communication’ API which caches communication parameters but still requires a new tag match and destination negotiation on each activation and does not establish a persistent remote-access relationship. A genuinely persistent building block can lead to more efficient implementations of higher-level API abstractions, such as MPI’s partitioned communication [7]. Third, RAMC should leverage hardware capabilities to provide efficient RMA with notification [4]. Because RDMA bypasses the host OS when accessing target memory, host applications are oblivious to the fact an RMA operation has occurred. The traditional way to address this is to build a notification mechanism by following the write of data with a second write to a notification buffer, relying on underlying ordering guarantees. Contemporary networks may provide mechanisms for avoiding this additional step, e.g., by allowing RMA operations to generate events on a target’s completion queue or through NIC-based hardware counters. Finally, RAMC should embrace relaxed synchronization, i.e., targets and initiators can be synchronized pair-wise (initiator/target), even in cases where many processes are participating in a global operation such as a collective. If needed, larger scale synchronization can be constructed using the more fundamental building block of a point-to-point RMA-based channel.
3.2
status value
Design
Core Concepts
3.2.1 Counters. Consistent with the design principle of leveraging Slingshot hardware capabilities, RAMC makes use of hardware counters to track the completion of local and remote communication operations. Specifically, RAMC leverages two types of counters, which we refer to as endpoint and memory region counters. The former are associated with libfabric endpoints (cf. the libfabric man pages for fi_cntr), and are configured to count local FI_WRITE and FI_READ operations. When incremented, these counters indicate the associated operation has completed. In the case of a write, the
op cntr data
Figure 2: Target-side channel data structure.
status value target status value address target buffer address
Figure 3: Initiator-side channel data structure.
counter incrementing indicates the local buffer is available for reuse – that is, an ACK has been received from the target NIC confirming the data has been received – and in the case of a read, the counter indicates the retrieved data is visible to the application. Building on the Portals network programming API [2], Slingshot also provides counters that can be associated with specific buffers (cf. the libfabric man pages for fi_mr). These memory region (MR) counters count either the number of remote operations that have occurred on the associated buffer (e.g., the number of remote writes), or the number of bytes manipulated by such operations (e.g., the number of bytes written). RAMC currently uses memory region counters to count operations. Using counters as a lightweight notification mechanism is important because it avoids additional followup communication typically required to ensure data visibility to a target. 3.2.2 Channels. Inter-process communication in RAMC is built around the concept of a channel, which is a relationship between an initiator process and a target window. The target side of a channel is anchored by a structure with two buffers, as shown in Figure 2. The data buffer is the location to which data is written or from which it is read by initiators. Associated with the data buffer is a memory region counter (𝑜𝑝_𝑐𝑛𝑡𝑟 ) used for testing remote operation completion (FI_REMOTE_WRITE and FI_REMOTE_READ). The status value is an unsigned integer made available to passively coordinate data transfers across the channel by providing information regarding the state of the target buffer. At channel creation, the status is initialized to a user-specified value (≥ 2), and the user can manipulate this value using ‘increment’ and ‘set’ commands provided by the RAMC API. The semantic for this value is determined by the user. For instance, examples in this document initialize the status to two, and interpret even status values as indicating the target is in an OK_TO_READ state, while odd values indicate OK_TO_WRITE. Updating (i.e., switching) between states is accomplished by incrementing the status value by one, which also indicates the target
Schonbein et al.
Update init status
1
tgt status
tgt status < init status
bb status
tag (uint64 t)
Update tgt status Test/wait on op cntr
2 3
tgt status
tgt status == init status
target buffer addr info
read cntr
target status addr info
4 Data Test/wait on write counter; Update init status
Test/wait succeeds Update tgt status
5
Initiator
Target
Figure 4: A simple write example.
buffer has moved forward a phase in the overall communication trajectory of the channels it anchors. In other words, the status value is the means by which a user can secure pair-wise initiator/target synchronization. The initiator side of a channel is anchored by a data structure comprising three pieces of information (Figure 3): (i) Addressing information for a target window’s status value, (ii) addressing information for that target’s data buffer, and (iii) a local status value. The first allows an initiator to query (fi_read) the target’s status value, and the second allows an initiator to perform puts/writes and gets/reads to and from the target’s data buffer. Since Slingshot does not use virtual addresses (i.e., it is offset-based), this addressing information comprises the memory keys returned by the CXI provider at the time these buffers are registered. The mechanism by which an initiator acquires these keys is described below. Mirroring that of the target, the examples in this article use the status value held by the initiator to represent (i) the state of the initiator relative to that target (‘ready-to-send’ or ‘ready-to-receive’), and (ii) its communication phase relative to the start of the application run. An example of how these components come together to coordinate a write from an initiator to a target is illustrated in Figure 4. In step (1), the initiator is ready to perform a write to the target buffer, so it first increments its local status value (from ‘ready-to-read’ to ‘ready-to-write’), reads the target’s status, and compares it with its own. The logic guiding the outcome of this comparison is straightforward: (i) If 𝑡𝑔𝑡_𝑠𝑡𝑎𝑡𝑢𝑠 < 𝑖𝑛𝑖𝑡_𝑠𝑡𝑎𝑡𝑢𝑠, the the target is behind the initiator, and the initiator should refrain from performing a write. (ii) If 𝑡𝑔𝑡_𝑠𝑡𝑎𝑡𝑢𝑠 > 𝑖𝑛𝑖𝑡_𝑠𝑡𝑎𝑡𝑢𝑠, then the target is ahead of the initiator, which may be a catastrophic error (because the target moved on in computation without relevant data). Finally, passing these checks guarantees 𝑡𝑔𝑡_𝑠𝑡𝑎𝑡𝑢𝑠 == 𝑖𝑛𝑖𝑡_𝑠𝑡𝑎𝑡𝑢𝑠, in which case the write can proceed. This is just one way to make use of the target and initiator status values; RAMC is flexible and it allows the user to decide how these values are used. Returning to the example (Figure 4), step (1) concludes with the initiator determining the target is behind in its communication phase, so the initiator is free to do other work. At step (2), the target increments its status value to an OK_TO_WRITE state. Consequently,
Figure 5: The Bulletin Board.
when (at step (3)) the initiator returns to check the target’s status again, it is able to perform the write (step (4)). After the write, the initiator may test/wait on the local endpoint counter (configured to count write operations) to determine the source buffer may be reused, and updates its status value to indicate it has moved past the write phase (step (5)). Similarly, the target’s test or wait on the operations counter associated with the window will succeed, and the target updates the buffer status value again, returning it to an OK_TO_READ state. 3.2.3 The Bulletin Board. Creating a channel requires that a target register status and data buffers (using fi_mr_[reg,bind,enable]) to obtain memory keys for those buffers, and share that addressing information with initiators who will participate in the channel. In RAMC, the exchange of addressing information is accomplished through a bulletin board (BB) (Figure 5). At initialization, a RAMC process allocates space for the BB and shares the addressing information for its BB with all other processes (using PMI [6]). The BB comprises two distinct buffers, each of which can be read by any other process. The first contains a BB status and tag, and the second the addressing information for a target buffer’s status and data buffers. To share addressing information via the BB, a target first fills in the status and data buffer addressing information, the tag, and then switches the BB status from inactive to active. The target then can test or wait on a memory region counter associated with the BB addressing information and configured to count FI_REMOTE_READs until all expected initiators have retrieved that information. When this condition is met, the target deactivates the BB posting. Similarly, to retrieve addressing information, an initiator reads the target’s status and tag from its BB. If the status is active, the initiator checks to see if the tag matches, and if so, reads the target buffer addressing information. If the BB is inactive or the tag does not match, the initiator can try again later. The BB mechanism satisfies the design goal of realizing nonblocking window creation, and ensures tag matching occurs only once, during the initial acquisition of target addressing information. In its current implementation, targets are limited to posting a single BB entry at a time; however, it is trivial to extend the BB to accommodate multiple postings or an arbitrary number of postings (via a linked list). 3.2.4 Discussion. It should be clear from this broad summary that RAMC affords significant flexibility in how a user can affect interprocess communication. For example, the procedure illustrated
RAMC: Remote Access Memory Channels over HPE Slingshot
in Figure 4 is straightforwardly extended to reads by adjusting when the target and initiator status values are updated (so that data transfer occurs when the target is in an OK_TO_READ state). Second, because the semantic for status values is user-defined, RAMC is not limited to the two states used by the examples presented here; for example, it is simple to define a third state (e.g., NOT_OK) prohibiting an initiator doing any remote operation on a target window. Third, during a write or read phase, multiple operations may occur, either from the same initiator using the same source buffer, the same initiator using multiple source buffers, or from multiple initiators; the target simply adjusts the expected value of the operation counter associated with the data buffer. Fourth, there are few restrictions on the roles various buffers can play: A buffer used as a source for a write or destination for a read can also be a target buffer in a RAMC channel, or an initiator can use different buffers for different communication operations. Finally, a single initiator can be in different states relative to different targets. It may be readyto-write relative to one target window, and ready-to-read relative to another. We also explicitly note the current design is passive target in the sense the initiator reads status information from the target to determine whether the target buffer can be read from or written to. In contrast, under an active target paradigm, an initiator tests or waits for a clear to send message from a target (e.g., via a write from target to initiator), issuing communication operations only once the signal is received. Both approaches have their merits, and active targets are intended for future RAMC development.
4
API
In this section we provide an overview of the current RAMC API. Functions are divided broadly into three categories: Common functions, target-side functions, and initiator-side functions.
4.1
Purpose
create_window post_window activate_bb deactivate_bb await_bb_reads
Create target window Post window addressing info to BB Set BB status to active Set BB status to inactive Wait (blocking) on expected number of BB reads Test (non-blocking) for expected number of BB reads Wait (blocking) on expected number of data buffer communication operations Test (non-blocking) for expected number of data buffer communication operations Increment target buffer status Set target buffer status Destroy target window
test_bb_reads await_win_ops
test_win_ops
increment_win_status set_win_status destroy_window
Table 1: RAMC Target API. All functions have the prefix ramc_tgt_ (not shown).
API
Purpose
check_bb_status
Query BB status of target window. Returns RAMC_SUCCESS if BB active and tag matches Retrieve the BB status of target window Get addressing information from target BB Returns RAMC_SUCCESS if target status == initiator status; returns RAMC_AHEAD if target status > initiator status; returns RAMC_BEHIND if target status < initiator status Get target window status Increment initiator status Set initiator status
get_bb_status get_bb_posting check_win_status
Common Functions
Common functions are those shared by both targets and initiators. These are primarily initialization and finalize routines. At startup, each RAMC process calls ramc_init() to select the CXI provider for libfabric, create endpoints, allocate NIC resources, share addressing information, set up the bulletin board, and so forth. At the time of writing, RAMC attempts to select a NIC that is close to each process, although generalizing this mechanism is left for future work. The last RAMC call made by each process is ramc_finalize(), which frees any resources not released by other calls.
4.2
API
Target Functions
Target functions are intended to be called by the target of a RAMC channel (Table 1). Creating a target window (ramc_tgt_create_window) registers the user-provided status and data buffers with the NIC, initializes the target status value to a user-defined value, and returns a structure that can be used by a target for subsequent operations, such as testing or waiting on data buffer communication operations (ramc_tgt_await_win_ops or ramc_tgt_test_win_ops), updating the target status (ramc_tgt_increment_win_status, ramc_tgt_set_win_status), or tearing down a channel (ramc_tgt_destroy_window). Bulletin board manipulation
get_win_status increment_status set_status
Table 2: RAMC Initiator API. All functions have the prefix ramc_init_ (not shown).
functionality – such as posting target addressing information and testing or waiting on an expected number of remote reads of that information – is also provided through the target API. Destroying a window de-registers the data and status buffers.
4.3
Initiator Functions
Initiator functions are called by an initiator in a RAMC channel, and include functions for interacting with a target BB and window (Table 2) and communication operations (Table 3). ramc_init_check_bb_status returns RAMC_SUCCESS when the status of the target BB is active and the tag matches; the ‘get’ version simply returns the status and tag. When the BB status is active and
Schonbein et al.
API
Purpose
ramc_put ramc_put_nb ramc_await_all_puts
Put to target (blocking) Put to target (non-blocking) Wait until expected number of PUTs complete (blocking) Get from target (blocking) Get from target (non-blocking) Wait until expected number of GETs complete (blocking)
ramc_get ramc_get_nb ramc_await_all_gets
Table 3: RAMC Communication Operations (called only by initiator).
19
21 22 23 24 25 26 27 28 29 30 31
33 34
5
Example
In this section we walk through a simple example illustrating how RAMC decouples initiator and target. To simplify exposition, error checking and some setup code has been removed. 1 2 3 4
unsigned char init_buf [ BUFFER_SIZE ]; unsigned char tgt_buf [ BUFFER_SIZE ]; struct ramc_target_win_info_s my_target_info ; struct ramc_init_win_info_s peer_info ;
5 6
ramc_init () ;
7 8 9
10 11 12 13 14 15 16 17
if (1 == my_rank ) { ramc_tgt_create_window ( tgt_buf , BUFFER_SIZE , TAG , & my_target_info , INIT_STATUS_VAL ) ; ramc_tgt_post_window (& my_target_info ) ; ramc_tgt_activate_bb () ; ramc_tgt_await_bb_reads (1) ; ramc_tgt_deactivate_bb () ; } else { do { ret = ramc_init_check_bb_status ( peer_rank , tag ) ; } while ( ret != RAMC_SUCCESS ) ;
}
20
32
the tag matches, ramc_init_get_bb_posting is used to retrieve the target status and buffer addressing information from the target’s BB; this call also initializes the initiator’s status value to a userspecified value. At this point the initiator holds a structure with the relevant addressing and synchronization information (cf. Figure 3), and the channel is ready to use. Data movement operations are listed in Table 3. These come in blocking and non-blocking flavors. For example, ramc_put blocks on a single increment of the local FI_WRITE endpoint counter. That is, the call only returns when the initiator receives an acknowledgment from the target NIC that the data has been received, so the source buffer can be reused. RAMC uses the CXI default FI_TRANSMIT_COMPLETE completion semantics. In contrast, ramc_put_nb is non-blocking: A call to this function issues a fi_write operation and increments a local expected_write_counter_value to record that a write was issued. Later, a call to ramc_await_all_puts blocks on the local FI_WRITE counter reaching this expected value. Both versions of put utilize fi_inject_write for messages that satisfy the maximum inject size threshold (192B on Slingshot), and standard fi_write otherwise.
ramc_init_get_bb_posting ( peer_rank , & peer_info , INIT_STATUS_VAL ) ;
18
if (0 == my_rank ) { ramc_init_increment_status (& peer_info , 1) ; do { ret = ramc_init_check_win_status (& peer_info ); } while ( ret != RAMC_SUCCESS ) ; ramc_put ( buffer , BUFFER_SIZE , 0) ; ramc_init_increment_status (& peer_info , 1) ; } else { < WORK > ramc_tgt_increment_win_status (& , y_target_info , 1) ; < WORK > ramc_tgt_await_win_ops (& my_target_info , 1) ; ramc_tgt_increment_win_status (& my_target_info , 1) ; }
35 36 37 38 39 40
ramc_barrier_binary () ; if (1 == my_rank ) { ramc_target_destroy_window (& my_target_info ); } ramc_finalize () ;
Listing 1: RAMC put example Listing 1 Illustrates an initiator (rank 0) putting to a target (rank 1). Lines 1-4 are the primary data structures required for this example. The init_buf contains the source data transmitted by the initiator, and the tgt_buf is the target for this data. The instance of ramc_target_win_info_s declared on line 3 is used by the target to store information regarding tgt_buf, such as its size, the tag used for channel creation, libfabric addressing information for the target buffer and its status, etc. Likewise, the ramc_init_win_info_s structure (line 4) holds addressing information for accessing the target buffer and tracking its status. RAMC is initialized on line 6; called by all processes, ramc_init does basic libfabric setup (select the CXI provider, exchange endpoint addressing information, etc.), creates the BB for each process, and exchanges BB data and status addressing information so that every process can access the BB of every other process. This is a blocking operation. On lines 9-13, the target creates and makes available information regarding a target window. On line 9, the target creates the window using tgt_buf, specifying its size (in bytes), the tag to be used during channel creation, the structure in which to store target window information, and the initial target status value. This function caches the address of the buffer, its size, the tag, and creates a status buffer (initialized to read only). The function also registers and obtains addressing information for both the status and the data buffer via libfabric, and creates a memory region counter associated with tgt_buf. The target then posts the addressing information for tgt_buf and its status to the BB on line 10, changes the BB status to active (line 11), waits (blocking) for a single read of the BB information (line 12), and deactivates the BB after the read has occurred (line 13). On lines 15-18, the initiator polls the target’s BB status until the target activates it (at which point the test returns RAMC_SUCESS), and reads the posted addressing information into peer_info (line 18). Checking the BB status/matching the tag must wait for data to be returned from the target, but is otherwise a
RAMC: Remote Access Memory Channels over HPE Slingshot
non-blocking operation, offering an opportunity for the initiator to do other work if available. The data transfer occurs over lines 21-34. On the initiator side (rank 0), the initiator first increments its local status value by one to indicate it expecting the target buffer to enter a write-enabled state (line 22; see Section 3.2.2 for a description of how the examples in this article utilize the target and initiator status values). On lines 23-25, the initiator queries the status of the target buffer until it enters an OK_TO_WRITE phase as indicated by the status value of the target being equal to that of the initiator. Note that while the initiator must wait to receive data regarding the target’s status, the target status check is otherwise non-blocking; at this point an application thus has an opportunity to perform other work, or to iterate through multiple target buffers looking for any that are ready to receive. When the status of the target is equal to that of the initiator (so the target is in a OK_TO_WRITE state), the initiator performs the put (line 26) and increments its local status value (line 27), indicating the initiator now expects the target buffer to enter a read only state. Note that the initiator is not limited to performing one communication operation; multiple operations can be issued before changing state. On the target side, we note (line 29) that the target may be engaged in work when the initiator initially checks the target status (Figure 4, steps 1 and 2). When this hypothetical work is complete, the target increments the status value of the target window (line 30). Since the write operation performed by the initiator requires no target involvement, this offers another opportunity to do additional work, as noted on line 31. When work is complete, the initiator waits for the single write to complete (line 32), and then updates the status of the target window to indicate it is OK_TO_READ (and hence will accept no more writes). Lines 36-40 tear down what was set up for the example. The barrier (line 36) is a helper function implemented using a binary tree, and is not optimized. The target destroys the target window on line 38, which de-registers memory registered with libfabric and sets the status to a special ’destroyed’ mode; if an initiator subsequently checks the status of this target buffer, the read status thereby informs the initiator the buffer no longer exists. The call to ramc_finalize (line 40) cleans up any remaining RAMC and libfabric resources.
6
Evaluation
Fundamental RAMC capabilities and performance were assessed on Sandia National Laboratories’ Eldorado supercomputer. Eldorado is an HPE Cray EX4000 system with 384 AMD MI-300A nodes. Each node has two sockets and four Slingshot 200 Gb/s NICs. To reduce system noise, a subset of cores on each node is reserved for the OS and other background processes [19], yielding 84 cores (out of 96) available for user processes. For tests involving MPI, we used Cray MPICH 9.0.1, the default on Eldorado. The system offers two versions of libfabric: a vendor-specific branch of libfabric 1.15.2 (known as Cray libfabric 2.1.3), and the open-source libfabric 2.3.1. The former has been superseded by the latter, but both are in use on Eldorado. Both libfabric versions have their own versions of the CXI libfabric provider.
Figure 6: Iteration 1000 from a heat diffusion code using RAMC for 5-point stencil communication. Data is for 19600 processes accross 250 nodes.
As noted above, RAMC is initially implemented with passive targets, which can generate significant network traffic while initiators poll targets for permission to write. As a test of RAMC’s ability to scale, we implemented a standard heat diffusion code using RAMC for 5-point stencil communication. After a process has updated its internal temperature, it increments its local status value (thereby telling neighbors they can write their temperatures to the process’ target window), and queries its neighbors – north, east, south, west – to determine whether a direction is ready to receive the updated value (through calls to ramc_init_check_win_status). When a neighbor is OK_TO_WRITE, the process writes its data to that process. When all neighbors have been serviced, the process waits on its own target counter (expecting four writes, one from each neighbor). When all expected writes have occurred, the process updates its temperature, and the cycle repeats. RAMC experienced no issues scaling this heat diffusion code to 19600 processes across 250 nodes (79 processes per node maximum; Figure 6). HPE’s Cray MPICH is highly optimized for Slingshot, and Portalsbased NICs ultimately use RMA operations for data movement [2]. With this in mind, we wrote unidirectional bandwidth and pingpong latency tests for RAMC that mirror those of the well-known OMB micro-benchmarks, for purposes of comparison with Cray MPICH MPI two-sided results [1]. We opted to compare with OMB two-sided bandwidth and latency rather than OSU one-sided benchmarks because the latter target performance characteristics of less immediate interest. For example, the OMB put latency benchmark measures the ‘local’ latency of an MPI_PUT call, i.e., the time lapsed before the source buffer can be reused, rather than the latency of message passing. All results are averages of ten runs with error bars indicating standard deviation.
Schonbein et al.
Unidirectional Bandwidth (libfabric 1.15.2) 25000
Latency (libfabric 1.15.2)
MPI RAMC
102 Latency (µs)
BW (MB/s)
20000 15000 10000
MPI RAMC (counter) RAMC (explicit)
101
5000 0 4194304 2097152 1048576 524288 262144 131072 65536 32768 16384 8192 4096 2048 1024 512 256 128 64 32 16 8 4 2 1
4194304 2097152 1048576 524288 262144 131072 65536 32768 16384 8192 4096 2048 1024 512 256 128 64 32 16 8 4 2 1
Message Size (B)
Message Size (B)
Figure 7: Unidirectional bandwidth under libfabric 1.15.2 for OMB MPI P2P (Cray MPICH) and RAMC using counters. Dashed red line is theoretical maximum.
Figure 9: Ping-pong latencies under libfabric 1.15.2 for OMB MPI P2P (Cray MPICH), RAMC using counters, and RAMC using explicit notification.
Unidirectional Bandwidth (libfabric 2.3.1) 25000
Latency (libfabric 2.3.1)
MPI RAMC
102 Latency (µs)
BW (MB/s)
20000 15000 10000
MPI RAMC (counter) RAMC (explicit)
101
5000 0 4194304 2097152 1048576 524288 262144 131072 65536 32768 16384 8192 4096 2048 1024 512 256 128 64 32 16 8 4 2 1
4194304 2097152 1048576 524288 262144 131072 65536 32768 16384 8192 4096 2048 1024 512 256 128 64 32 16 8 4 2 1
Message Size (B)
Message Size (B)
Figure 8: Unidirectional bandwidth under libfabric 2.3.1 for OMB MPI P2P (Cray MPICH) and RAMC using counters. Dashed red line is theoretical maximum.
Figure 10: Ping-pong latencies under libfabric 2.3.1 for OMB MPI P2P (Cray MPICH), RAMC using counters, and RAMC using explicit notification.
Figure 7 compares RAMC unidirectional bandwidth with OMB results, under libfabric 1.15.2. RAMC consistently outperforms MPI, showing increases in mean bandwidth ranging from approximately 100-130% for messages in the range of 1 to 4KiB, at which point the gains begin to recede to parity by 32KiB. The gap is less prominent under libfabric 2.3.1, as shown in Figure 8. In this case, RAMC bandwith is approximately 30-45% greater than MPI for messages in the range of 1-8KiB, and MPI holds a slight lead (2-4%) for 64KiB and 128KiB messages. This change is due to both an apparent increase in mean bandwidth for MPI and a slight decrease in the same for RAMC with the switch from libfabric 1.15.2 to 2.3.1. For example, MPI shows increases in mean bandwidth in the range of 32-66% for 1B-8KiB messages, while RAMC bandwidths drop by up to 11% over the same range. A benefit of Slingshot is lightweight target-side RMA completion notification via memory region counters. A more traditional approach uses explicit completion notification in the form of a
second RMA operation following the first; this followup operation ensures completion of the first through ordering constraints, and lets the target know the preceding operation has occurred. To better understand the impact of Slingshot MR counters on latencies, we implemented an explicit notification version of RAMC by adding a notification buffer to each target window; after issuing a write to the data buffer, the initiator performs an atomic increment (through libfabric atomics) on this notification buffer. Instead of testing on a MR counter, the target checks this notification buffer for the expected number of increments. Note this explicit notification mechanism is simple to build by creating an additional RAMC channel for notifications. Figure 9 shows ping-pong latency results using libfabric 1.15.2 for three cases for MPI, standard RAMC (i.e., using MR counters for completion), and RAMC with explicit notification. Explicit notification latencies are nearly indiscernible from those of standard RAMC for 1-128B messages, at which point the latencies of the
RAMC: Remote Access Memory Channels over HPE Slingshot
former jump, showing an 86% increase at 256B. The gap between explicit and standard decreases as message sizes grow (e.g., 52% at 32KiB), until there is little difference by 1MiB. Note the location of the increase in latencies for explicit notification corresponds to the maximum inject size (192B) for Slingshot. For small message sizes (1-64B), Cray’s MPI shows a distinct advantage in mean latencies, with RAMC lagging by 430-540 ns. By the inject threshold (192B), this gap is eliminated. RAMC subsequently shows reductions in latencies (relative to MPI) of up to 32% for 16-512KiB messages, where 16KiB is the eager-rendezvous threshold. Figure 10 shows latency results under libfabric 2.3.1. In this case, explicit notification exhibits an approximate latency penalty of 1324% compared to standard RAMC for small messages (1-64KiB), and the jump at the inject limit surges to a 128% increase. Cray’s MPI continues to exhibit lower latencies than standard RAMC for small messages; these are of the same magnitude as for 1.15.2: 480-500 ns. In contrast, under 2.3.1 this trend continues into medium-sized messages, with Cray’s MPI averaging approximately 339 ns faster than standard RAMC. As with 1.15.2, crossing the eager-rendezvous threshold incurs penalties for MPI, and RAMC shows decreases in latencies of up to 32% for 16-512KiB messages. Comparing MPI and RAMC latencies across libfabric 1.15.2 and 2.3.1, we observe Cray’s MPI shows decreases in latencies for 116KiB messages of 13-26%, while RAMC shows decreases of 10-13%. These decreases reflect additional performance tuning within the CXI provider, but also reveal these optimizations have a greater impact on Cray’s MPI than RAMC. For this work, we considered several opportunities for addressing this disparity, including (i) carefully limiting requested provider capabilities to the minimum required for RAMC; (ii) confirming that Cray MPI CXI defaults correspond to those used by RAMC; (iii) explicitly disabling receiver-side message processing ordering constraints that appear to be enabled by default (as indicated by the structure returned by fi_getinfo), and (iv) using an aliased endpoint configured to use a low-latency traffic class (FI_TC_LOW_LATENCY) for transmitting small messages: None of these strategies showed any improvement to RAMC latencies, and in some cases increased them (case (iv)). Finally, we also investigated whether RAMC latencies were the result of a reliance on MR counters. Specifically, Slingshot provides two types of memory regions: ‘standard’ and ‘optimized’. The latter (identified by user-specified memory keys in the range of 0-99) allow initiators to use smaller headers, reducing latencies for small writes, and are intended for applications that use a small number of large MRs. However, they are not intended for use with MR counters (i.e., FI_RMA_EVENT should not be an enabled provider capability). To investigate the potential impact of optimized MRs on latencies, we gutted all use of MR counters from RAMC (for both BB and channels), replacing all counter completion with explicit notification and disabling the FI_RMA_EVENT capability. Unfortunately, using optimized MRs with explicit notification incurred increases in latencies of up to 60% for small messages (1-128B), and of up to 160% for medium messages (256B-16KiB), compared to standard RAMC. Consequently, based on this initial exploration, optimized MRs do not appear to be responsible for Cray MPI’s reduced latencies.
7
Related Work
RAMC provides an explicit one-sided communication model, and there has been significant prior work in this area. A prominent example is the MPI RMA API, which provides explicit one-sided communication, and has been under development for decades. Introduced in MPI 2.0 with an initial set of functionality, the RMA API was expanded in MPI 3.0 to include passive target synchronization [9, 21], and MPI RMA has seen a nuber of efforts in improving performance, such as in multithreaded use cases [13]. In contrast with RAMC, however, MPI RMA treats windows as monolithic (i.e., all processes participate in window creation and synchronization), and lacks a standardized RMA-with-notification semantic as provided by Slingshot’s MR counters. Note, however, there is currently an RMA-with-notification proposal under consideration for inclusion in an upcoming revision of the MPI specification [4]. From this perspective, RAMC can be viewed as one approach towards implementing MPI RMA-with-notification, although exploring this possibility is left for future work. OpenSHMEM is another major explicit one-sided communication model targeting high performance computing [24]. It has also seen major interest including for GPU-based communication with the NVSHMEM implementation [14]. Like RAMC (and unlike MPI) OpenSHMEM and its variants (NVSHMEM, CraySHMEM) include signaled RMA operations designed to avoid an explicit follow-up notification message, although how this is implemented will differ across network hardware (e.g., an alternative to counters is allowing incoming RMA operations to force events to appear on a completion queue) [16]. In contrast to RAMC, OpenSHMEM still treats windows as monolithic constructs, shared across all processes in the group (e.g., as a symmetric heap). A core aspect of RAMC is that it provides genuinely persistent communication channels. MPI also includes API calls that invoke persistence, although there are important differences in comparison to RAMC. Despite the name, MPI persistent point-to-point communication (MPI_Send_init, MPI_Recv_init, MPI_Start) involves a distinct tag matching step on each transmission (e.g., one can match a call to MPI_Start with an target-side MPI_Irecv); the persistence is really a form of information caching rather than a fixed target buffer as with RAMC. In contrast, MPI Partitioned Communication, introduced in MPI 4.0 [22], aims to leverage the benefits of true persistence by creating a permanent match across a channel that can be reused multiple times [10, 11, 25]. In contrast to RAMC, partitioned communication is more restricted in that it utilizes datatypes and counts rather than bytes, and offsets are constrained to partition boundaries. Moreover, under the current MPI specification, an implementation of partitioned communicaton over RMA will not be specification compliant [7]. Finally, we note RAMC is closely related to research on triggered communication operations (i.e., deferred work queues). Specifically, besides providing lightweight RMA completion notification on a target, Portals-style MR counters provide a mechanism for offloading communication progress to the NIC. Communication operations can be associated with an MR counter and given a threshold; when the counter is incremented, all operations whose threshold is lessthan-or-equal-to the current counter value are automatically executed by the NIC, independent of the host CPU. Originally proposed
Schonbein et al.
to facilitate the offloading of collectives and rendezvous messaging, this mechanism has recently been adopted to allow GPU-triggered communication as well [3, 5, 12, 23]. While triggered operations are not part of the current RAMC API, they will be considered in future revisions.
8
Challenges and Future Work
In this work we’ve presented RAMC, a communication library that leverages the hardware capabilities of Cray HPE’s Slingshot interface to provide efficient and easy-to-use unidirectional RMAbased channels. In contrast to existing user-level RMA APIs, RAMC does not require monolithic window creation or synchronization, enabling point-to-point style persistent communication semantics as well as opportunities for earlybird communication. An obvious limitation of RAMC is that it relies on hardware capabilities of the HPE Cray Slingshot network. On the one hand, this is deliberate. RAMC is not proposed as a general-purpose userlevel communication API, but rather as an exploration of capabilities provided by the Slingshot network. If these capabilities turn out to be especially useful or performant, there is reason to advocate for wider adoption. On the other, there is nothing in RAMC that precludes implementation using other underlying mechanisms, such as substituting explicit notifications for Slingshot MR counters. A second potential point of concern is that the current implementation is passive target insofar as an initiator queries a target’s status until that status indicates the initiator may perform its operations. As noted in Section 6, this situation has the potential to generate significant traffic (cf. [15]). An alternative is an active target strategy, where an initiator instead checks for a ‘ready-to-operate’ signal from a target. This approach would reduce synchronization network traffic, and remove a round trip communication before an initiator issues a communication operation. A third challenge for RAMC is imposed by libfabric (and Slingshot). Because target-side MR counters are associated with specific memory regions, a target can distinguish between operations on different MRs. In contrast, the endpoint completion counters used by initiators do not distinguish between targets, because they count all operations of a given type issued on that endpoint. Consequently, for example, if an initiator issues writes to two buffers on the same target process, it cannot test for completion of one or the other; it must wait for both to complete to conclude that either has. Bringing the granularity of initiator-side completion counters into alignment with those of the target would enable more fine-grained coordination between initiators and targets. Finally, MR counters in Slingshot are not limited to signaling completion but are also part of a strategy for offloading progress to the NIC. Specifically, communication operations can be registered with a Slingshot NIC but deferred until an associated counter reaches a specified threshold. When this threshold is reached, the operation is triggered, with no host involvement. Originally intended for the offloading of collective and rendezvous operations [3, 12, 26], this capability has more recently been leveraged to enable GPUs to initiate communication without host involvement [5, 23]. Future work on RAMC will explore further leveraging the completion counters intrinsic to RAMC’s operation to offload communication progress.
Acknowledgments Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC (NTESS), a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (DOE/NNSA) under contract DENA0003525. This written work is authored by an employee of NTESS. The employee, not NTESS, owns the right, title and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government. The publisher acknowledges that the U.S. Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOE Public Access Plan. SAND2026-20283C
References [1] 2024. OSU Micro-Benchmarks (OMB) 7.5. Online. https://mvapich.cse.ohiostate.edu/benchmarks/ Accessed: 2025. [2] Brian W. Barrett, Ron Brightwell, Ryan E. Grant, Whit Schonbein, Scott Hemmert, Kevin Pedretti, Keith Underwood, Rolf Riesen, Mathieu Barbe, Luiz H. Suraty Filho, Alexandre Ratchov, and Arthur B. Maccabe. 2022. The Portals 4.3 Network Programming Interface. Technical Report SAND2022-8810. Sandia National Laboratories, Albuquerque, New Mexico. https://www.sandia.gov/portals/ [3] Brian W. Barrett, Ron Brightwell, K. Scott Hemmert, Kyle B. Wheeler, and Keith D. Underwood. 2011. Using Triggered Operations to Offload Rendezvous Messages. In Recent Advances in the Message Passing Interface (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg, 120–129. https://doi.org/10.1007/978-3642-24449-0_15 [4] Roberto Belli and Torsten Hoefler. 2015. Notified Access: Extending Remote Memory Access Programming Models for Producer-Consumer Synchronization. In 2015 IEEE International Parallel and Distributed Processing Symposium. 871–881. https://doi.org/10.1109/IPDPS.2015.30 [5] Patrick G. Bridges, Derek Schafer, Jack Lange, James B. White III, Anthony Skjellum, Evan Suggs, Thomas Hines, Purushotham Bangalore, Matthew G. F. Dosanjh, and Whit Schonbein. 2026. Co-Design and Evaluation of a CPU-Free MPI GPU Communication Abstraction and Implementation. arXiv:2602.15356 [cs.DC] https://arxiv.org/abs/2602.15356 [6] Ralph H. Castain, Joshua Hursey, Aurelien Bouteiller, and David Solt. 2018. PMIx: Process management for exascale environments. Parallel Comput. 79 (Nov. 2018), 9–29. https://doi.org/10.1016/j.parco.2018.08.002 [7] Matthew GF Dosanjh, Andrew Worley, Derek Schafer, Prema Soundararajan, Sheikh Ghafoor, Anthony Skjellum, Purushotham V Bangalore, and Ryan E Grant. 2021. Implementation and evaluation of MPI 4.0 partitioned communication libraries. Parallel Comput. 108 (2021), 102827. [8] Kurt B. Ferreira, Patrick Bridges, and Ron Brightwell. 2008. Characterizing application sensitivity to OS interference using kernel-level noise injection. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (Austin, Texas) (SC ’08). IEEE Press, Article 19, 12 pages. [9] Message Passing Interface Forum. 1997. MPI: A Message-Passing Interface Standard Version 2.0. https://www.mpi-forum.org/docs/mpi-2.0/mpi20-report.pdf [10] Thomas Gillis, Ken Raffenetti, Hui Zhou, Yanfei Guo, and Rajeev Thakur. 2023. Quantifying the performance benefits of partitioned communication in mpi. In Proceedings of the 52nd International Conference on Parallel Processing. 285–294. [11] Ryan E Grant, Matthew GF Dosanjh, Michael J Levenhagen, Ron Brightwell, and Anthony Skjellum. 2019. Finepoints: Partitioned multithreaded MPI communication. In International Conference on High Performance Computing. Springer, 330–350. [12] K. Scott Hemmert, Brian Barrett, and Keith D. Underwood. 2010. Using Triggered Operations to Offload Collective Communication Operations. In Recent Advances in the Message Passing Interface (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg, 249–256. https://doi.org/10.1007/978-3-642-15646-5_26 [13] Nathan Hjelm, Matthew GF Dosanjh, Ryan E Grant, Taylor Groves, Patrick Bridges, and Dorian Arnold. 2018. Improving MPI multi-threaded RMA communication performance. In Proceedings of the 47th International Conference on Parallel Processing. 1–11.
RAMC: Remote Access Memory Channels over HPE Slingshot
[14] Chung-Hsing Hsu, Neena Imam, Akhil Langer, Sreeram Potluri, and Chris J Newburn. 2020. An initial assessment of nvshmem for high performance computing. In 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 1–10. [15] Weihang Jiang, Jiuxing Liu, Hyun-Wook Jin, Dhabaleswar K. Panda, Darius Buntinas, Rajeev Thakur, and William D. Gropp. 2004. Efficient Implementation of MPI-2 Passive One-Sided Communication on InfiniBand Clusters. In Recent Advances in Parallel Virtual Machine and Message Passing Interface, Dieter Kranzlmüller, Péter Kacsuk, and Jack Dongarra (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 68–76. [16] Jithin Jose, Sreeram Potluri, Hari Subramoni, Xiaoyi Lu, Khaled Hamidouche, Karl Schulz, Hari Sundar, and Dhabaleswar K Panda. 2014. Designing scalable outof-core sorting with hybrid MPI+ PGAS programming models. In Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models. 1–9. [17] Scott Levy, Kurt B. Ferreira, Patrick Widener, Patrick G. Bridges, and Oscar H. Mondragon. 2016. How I Learned to Stop Worrying and Love In Situ Analytics: Leveraging Latent Synchronization in MPI Collective Algorithms. In Proceedings of the 23rd European MPI Users’ Group Meeting (Edinburgh, United Kingdom) (EuroMPI ’16). Association for Computing Machinery, New York, NY, USA, 140–153. https://doi.org/10.1145/2966884.2966920 [18] Scott Levy, Whit Schonbein, and Craig Ulmer. 2024. Leveraging HighPerformance Data Transfer to Offload Data Management Tasks to SmartNICs. In 2024 IEEE International Conference on Cluster Computing (CLUSTER). 346–356. https://doi.org/10.1109/CLUSTER59578.2024.00037 [19] Edgar A. León, Joseph Glenski, Mark Stock, Kim McMahon, William Loewe, Clark Snyder, Larry Kaplan, Srinath Vadlamani, Timothy I. Mattox, Trent D’Hooge, Brian Behlendorf, Nathan Hanford, Ramesh Pankajakshan, and Matthew L.
Leininger. 2025. Breaking the System Noise Barrier at Exascale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). Association for Computing Machinery, New York, NY, USA, 411–436. https://doi.org/10.1145/3712285.3759793 [20] W. Pepper Marts, Donald A. Kruse, Matthew G. F. Dosanjh, Whit Schonbein, Scott Levy, and Patrick G. Bridges. 2024. CMB: A Configurable Messaging Benchmark to Explore Fine-Grained Communication. In 2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid). 28–38. https://doi.org/10.1109/CCGrid59990.2024.00013 [21] Message Passing Interface Forum. 2012. MPI: A Message-Passing Interface Standard Version 3.0. https://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf [22] Message Passing Interface Forum. 2021. MPI: A Message-Passing Interface Standard Version 4.0. https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf [23] Naveen Namashivayam, Krishna Kandalla, Trey White, Nick Radcliffe, Larry Kaplan, and Mark Pagel. 2022. Exploring GPU Stream-Aware Message Passing using Triggered Operations. https://arxiv.org/abs/2208.04817 _eprint: 2208.04817. [24] Stephen W. Poole, Oscar Hernandez, Jeffery A. Kuehn, Galen M. Shipman, Anthony Curtis, and Karl Feind. 2011. OpenSHMEM - Toward a Unified RMA Model. Springer US, Boston, MA, 1379–1391. https://doi.org/10.1007/978-0-387-097664_490 [25] Yıltan Hassan Temuçin, Whit Schonbein, Scott Levy, Amirhossein Sojoodi, Ryan E Grant, and Ahmad Afsahi. 2024. Design and Implementation of MPINative GPU-Initiated MPI Partitioned Communication. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 436–447. [26] K. D. Underwood, J. Coffman, R. Larsen, K. S. Hemmert, B. W. Barrett, R. Brightwell, and M. Levenhagen. 2011. Enabling Flexible Collective Communication Offload with Triggered Operations. In 2011 IEEE 19th Annual Symposium on High Performance Interconnects. 35–42. https://doi.org/10.1109/HOTI.2011.15