arXiv:2604.22228v1 [cs.DC] 24 Apr 2026
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs Amirhossein Sojoodi
Yıltan Hassan Temuçin
Amirreza Barati Sedeh
Queen’s University Kingston, Ontario, Canada [email protected]
Queen’s University Kingston, Ontario, Canada [email protected]
Queen’s University Kingston, Ontario, Canada [email protected]
Hamed Sharifian
Ahmad Afsahi
Queen’s University Kingston, Ontario, Canada [email protected]
Queen’s University Kingston, Ontario, Canada [email protected]
Abstract Effective intra-node GPU communication is essential for optimizing performance in MPI-based HPC applications, especially when leveraging multiple communication paths. In this study, we propose a novel approach that integrates CUDA Graphs into the UCX framework to enhance intra-node multi-path point-to-point GPU communication. By concurrently leveraging multiple paths, including NVLink and PCIe through the host, and optimizing communication workflows using CUDA Graph, we achieve significant reductions in communication overhead and improve execution efficiency. To the best of our knowledge, our proposed approach is the first to seamlessly integrate CUDA Graphs into UCX. Through extensive experiments on a four-GPU node, our proposed CUDA Graph-based multi-path communication approach achieves up to a 2.95× bandwidth improvement, compared to the single-path UCX (UCT::CUDA-IPC), in GPU-to-GPU OMB bandwidth test when utilizing the host path and two other GPU paths, at message sizes up to 512MB.
CCS Concepts • Software and its engineering → Message passing.
Keywords MPI, UCX, GPU, CUDA Graph, Multi-Path Communication, NVLink, PCIe
1
Introduction
The integration of Graphics Processing Unit (GPU) into HighPerformance Computing (HPC) infrastructures has attracted much attention over the past decade, coupled with the advent of many applications across various domains [6, 29]. In order to efficiently utilize the computational power of GPUs, distributed applications need to allow for efficient GPU communication. In particular, data exchange between GPUs on the same node with the help of Message Passing Interface (MPI) communication protocols, the de facto standard for distributed systems [14], plays a critical role in reaching optimal performance.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Traditionally, Point-to-Point (P2P) communication in GPU-accelerated systems relies on direct data transfers through a single NVLink or Peripheral Component Interconnect Express (PCIe) communication path, which often becomes a bottleneck and limits overall performance [27]. While several studies have attempted to improve P2P communication by message striping across multiple paths, such as in the Unified Communication X (UCX) library, these approaches still lack the ability to provide concurrent data staging through both host and device [16, 27]. Our earlier work demonstrated that leveraging host-staged transfer alongside other GPU-staged transfers could significantly enhance P2P performance [25, 27, 28]. Building upon this foundation, we propose a unified multi-path communication framework with two sub-designs: a static runtime-based pipelining engine and a dynamic CUDA Graphbased communication execution. CUDA Graph enables capturing and reusing the workflow of execution that can be represented as a directed graph. Utilizing CUDA Graph leads to reducing the launch overhead, and has gained popularity in recent studies [10, 34]. In our proposed approach, we dynamically construct CUDA Graphs to encapsulate P2P communication workflows, allowing efficient execution of data transfers with minimal launch overhead. Additionally, by caching CUDA Graphs in the UCX library and reusing them for repetitive communication patterns, we improve efficiency and reduce unnecessary synchronization costs. With this approach, the CPU is relieved from managing and orchestrating the flow, enabling more efficient data transfers between GPUs. While our earlier work demonstrated that exploiting multiple intra-node communication paths can significantly increase GPUto-GPU bandwidth [25], the focus of this paper is on integrating CUDA Graphs directly into the UCX transport to reduce launch and synchronization overheads. In particular, this work studies when CUDA Graphs provide net benefit within a multi-path communication framework and quantifies their overheads in transport-level executions. In summary, the main contributions of this paper are as follows: (1) We extend our multi-path communication framework with a CUDA Graph-based design that dynamically constructs and reuses communication workflows to minimize launch and intranode communication overhead. This integration is, to the best of our knowledge, the first of its kind within the UCX framework.
Sojoodi et al.
(2) We show the benefits of our proposed framework in both microbenchmarks and a real-world application. Our experimental evaluation on a four-GPU node demonstrates that this approach achieves up to a 2.95× increase in GPU-to-GPU bandwidth test from OSU Micro-Benchmarks (OMB) compared to the singlepath UCX transport (UCT::CUDA-IPC). We also evaluate our framework using a distributed Jacobi application implemented with MPI, where we achieve up to 1.28× improvement in overall execution time compared to the traditional single-path communication approach. (3) Furthermore, we provide a comprehensive analysis of the performance trade-offs and overheads associated with CUDA Graphs in the context of multi-path communication, offering insights and guidelines into their practical implications for HPC applications. The remainder of this paper is organized as follows. Section 2 provides the necessary background for this work, and Section 3 provides a review of the related work. Section 4 describes the design and implementation of our multi-path communication framework, including both static and dynamic models. Section 5 presents the experimental setup, performance evaluation, and analysis of the results. Finally, Section 6 concludes the paper, summarizing the contributions and discussing future directions.
2 Background and Motivation 2.1 GPU-enabled Systems, MPI, and UCX Efficient utilization of modern GPU-enabled systems is a common challenge in HPC frameworks and applications. These systems typically feature multiple accelerators per node, connected via highbandwidth interconnects such as NVLink, and to the host CPUs through PCIe. One of the most widely adopted communication standards in HPC is MPI, which provides an extensive suite of communication primitives suitable for parallel and distributed applications. Among the various implementations, Open MPI and MPICH are notable for their flexibility and widespread use in both traditional HPC and emerging AI workloads [15, 19]. To address the evolving communication needs in applications that leverage GPU acceleration, MPI implementations are often reliant on high-performance communication libraries, for example, the UCX library [21]. The UCX library is an open-source, modular communication framework intended to enable low-latency, highbandwidth, and efficient communication through multiple hardware interconnects, including shared memory, InfiniBand, RoCE, PCIe, NVLink, etc. [4, 17, 31]. Figure 1 displays the UCX hierarchy with a focus on important pieces, including the Unified Communication Transports (UCT) layer. The UCT layer enables a uniform interface for communication primitives while at the same time providing hardware-specific implementations through the separate transport modules. The UCT layer comprises several core abstractions [31]: (1) Memory Domain handles memory registration and memory access management specific to each hardware transport; (2) Interface represents a communication resource tied to a specific transport on a given device, encapsulating the capabilities and constraints of the transport; and (3) Endpoint represents a logical communication link to a remote peer, managing connection state and message flow.
MPI libraries
UCX
ML/DL Middleware
Applications
UCP (Protocols) - High Level API
UCT (Transports) - Low Level API GPGPU IB
TCP
NVIDIA CUDA
AMD ROCm
UCS (Services)
UCM (Memory)
Figure 1: UCX architecture and some of its components
2.2
CUDA, UCX-CUDA Modules, and Direct Communication
Compute Unified Device Architecture (CUDA) is a widely used platform and programming model developed by NVIDIA to harness the massive parallelism offered by GPUs [1]. Over the past decade, several middleware solutions have emerged to integrate GPUs into distributed HPC applications [8, 9, 24, 32]. While this study primarily targets NVIDIA GPUs and CUDA, many of the underlying concepts are extensible to other accelerator platforms such as those from AMD and Intel. The UCX library offers built-in support for CUDA through its uct_cuda transport module [31]. This module consists of a shared base and three distinct submodules: • cuda_copy: Optimized for intra-process communication, supporting device-to-device memory transfers within the same process. • gdr_copy: Utilizes GPUDirect_Remote Direct Memory Access (RDMA) to allow direct memory access for small messages. • cuda_ipc: Enables zero-copy inter-process communication via CUDA Inter-Process Communication (IPC) handles, making it well-suited for large messages and intra-node MPI workloads. In our framework, we primarily target the cuda_ipc submodule, which is critical for enabling efficient large message exchanges across GPUs residing on the same node. Typically, when message sizes exceed 64KB, communication in UCX defaults to a rendezvous protocol. Depending on which process, sender or receiver, initiates the transfer and on internal configuration options, UCX chooses between a put or a get operation. In both scenarios, successful communication requires that the source and destination memory buffers reside within a shared CUDA context, accomplished through the exchange of CUDA IPC memory handles. A CUDA context encapsulates a process’s view of the GPU, including memory allocations, execution resources, and synchronization primitives such as CUDA streams and CUDA events. Each CUDA context is associated with a single device but can be shared across multiple threads or processes. The maintenance of a context is essential to coordinate memory accesses and manage ownership. Historically, coordinating computation and communication between GPUs required explicit staging through host memory, increasing latency and memory overhead. With the advent of GPUDirect technologies, direct communication became feasible, including GPUDirect_P2P, which enables direct peer-to-peer data movement between intra-node GPUs, bypassing the host.
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs
2.4
PCIe 3.0
UPI
NVLinks
Figure 2: (a) A typical four-GPU node with NVLink (two sub-links) per GPU pair, (b) A communication between GPU-0 and GPU-1 is split and transferred through multiple paths
Figure 2 illustrates both the hardware topology and a common communication scenario. In Figure 2(a), we show a standard fourGPU node with bidirectional NVLink interconnects, and in Figure 2(b), we demonstrate how a data transfer between GPU-0 and GPU-1 can be split across multiple paths, utilizing idle bandwidth on alternative links.
2.3
CUDA Graphs and Advanced Runtime Integration
Despite the potential for high bandwidth, managing concurrent communication paths introduces overhead in terms of coordination, memory management, and synchronization. As message sizes grow and communication becomes more complex, these overheads can limit scalability and performance. To address this, we integrate CUDA Graph into our multi-path communication framework. CUDA Graph is a CUDA feature that allows users to capture a sequence of GPU operations and replay them as a single executable graph [2, 10, 34]. This reduces launch overhead and enables better scheduling and optimization by the CUDA runtime. A CUDA Graph consists of two main components: • Nodes: Represent individual GPU operations such as memory copies or kernel launches. • Dependencies: Specify ordering constraints between nodes, forming a directed graph. CUDA Graphs can be created either via explicit Application Programming Interface (API) calls or by capturing a stream of GPU operations. By encapsulating multi-path communication as a CUDA Graph, we enable efficient execution of complex workflows that involve multiple memory transfers, staging buffers, and synchronization points. Furthermore, CUDA Graphs can be cached and reused for repetitive patterns, reducing overhead for recurring communication tasks. This is particularly advantageous in HPC applications with regular communication patterns, such as stencil computations or iterative solvers.
Motivation
As illustrated in Figure 2(b), traditional communication via a single path (e.g., between GPU-0 and GPU-1) can saturate available bandwidth, leading to performance bottlenecks. Splitting large data transfers into smaller chunks and routing them across multiple available paths can help mitigate this issue and fully exploit the system’s communication capabilities. While earlier approaches such as host-staging or manual pipelining can improve throughput, they often require careful tuning and can introduce additional CPU overhead. Additionally, ensuring data integrity, minimizing synchronization costs, and maximizing overlap between transfers are non-trivial challenges. Incorporating CUDA Graph into our design enables an automated, low-overhead solution to manage and schedule these complex workflows efficiently. It provides an elegant mechanism to orchestrate fine-grained transfers across multiple paths, thereby improving scalability and reducing CPU involvement. Together, these technologies form the foundation of our proposed framework: a multi-path communication model that supports both static runtime scheduling and dynamic, CUDA Graph-enabled execution for optimized intra-node GPU communication in HPC environments.
3 Related Work 3.1 Multi-Path Communication Tatsugi and Nukada [26] propose a method to enhance the performance of a data transfer from a GPU to host, by utilizing the idle GPUs. Their framework targets single-GPU applications running on a multi-GPU node, while our approach is designed for multi-GPU applications. As part of our prior studies [27, 28], we enhance P2P communication within the UCX library by utilizing host-staging multi-path communication to devise more performant collective communication for Deep Learning applications. In [22], we extend this work with a performance model to predict the optimal communication pattern for a given message size and hardware configuration, and in [23], we present a heterogeneous multi-path communication framework to optimize Allreduce operations. In [16], Nukada proposes a method that utilizes the PCIe path to accelerate Allreduce on a multi-GPU system. Although their method involves collectives, they follow a similar approach to our previous work to enhance each P2P communication. Again, our approach utilizes both NVLinks and PCIe paths, while their method is limited to PCIe. In this work, we utilize both NVLinks and PCIe paths to further enhance the performance of P2P communication between intranode GPUs, and we utilize CUDA Graphs inside the UCX library to optimize the communication workflow.
3.2
CUDA Graphs
Numerous studies have demonstrated that CUDA Graphs effectively reduce CPU overhead by consolidating multiple kernel launches into a single execution unit, thereby enhancing GPU utilization and overall application performance [11, 13, 20, 33, 34]. For example, frameworks have reported substantial speedups, such as up to 3.47×
Sojoodi et al.
in kernel execution [33] and over 1.4× through kernel batching in iterative HPC applications [11]. Beyond reducing launch overhead, CUDA Graphs also provide the opportunity for more efficient resource management, including notable reductions in GPU memory consumption via metadata optimizations [34]. These improvements underscore the role of CUDA Graphs in achieving high performance in HPC workloads. To facilitate the adoption of this feature, several frameworks and programming models have emerged that simplify CUDA Graph development [12, 13, 20]. Lin et al.[13] proposed a lightweight programming framework that abstracts the complexity of constructing CUDA Graphs for GPU computation. Similarly, Huang et al.[12] introduced a task-graph programming model designed to aid the development of Task Graph Computing System (TGCS), where function calls and their inter-dependencies are represented as task graphs; these are then submitted using CUDA Graphs on supported GPUs. Qiao et al. [20] leveraged CUDA Graphs within a source-tosource compiler for image processing Domain-Specific Languages (DSLs), achieving improved workflow optimization through graphbased execution. Libraries like NVIDIA Collective Communications Library (NCCL) and Unified Communication Collectives (UCC) have also integrated CUDA Graph support to optimize collective communication operations [18, 30]. NCCL utilizes CUDA Graphs to capture and replay collective operations, reducing overhead, and UCC uses CUDA Graphs to group multiple data transfers between intra-node GPUs. At the time of writing this paper, UCC does not put computations on the CUDA Graphs, and it only uses them for communication operations. Our work is different from both of these libraries, as we focus on optimizing P2P communication between intra-node GPUs by leveraging multiple paths (with staging) and utilizing CUDA Graphs to encapsulate the entire communication workflow. Despite these advances, challenges remain in fully exploiting CUDA Graphs, particularly for dynamic workloads and complex control flows [3]. Many implementations struggle with applications featuring varying input sizes or data-dependent execution paths, often requiring intricate compiler or runtime support and incurring higher memory overhead [34]. Additionally, optimizing dynamic and irregular communication/computation patterns using CUDA Graphs paradigm continues to be challenging. Addressing these limitations is essential for extending the applicability of CUDA Graphs to a broader spectrum of modern HPC and scientific computing workloads.
3.3
Comparison with our Previous Work
In [25], we introduced a multi-path communication framework within the UCX library to enable multi-path P2P communication between intra-node GPUs through both NVLink and PCIe interconnects. In the current work, we extend this framework further by integrating CUDA Graphs to optimize communication workflows and reduce synchronization costs. Our approach allows for dynamic construction and caching of CUDA Graphs to save execution time and reduce overheads associated with launching multiple GPU operations. We build this CUDA Graph engine inside the UCX library, to provide a seamless and unified interface for users to leverage CUDA Graphs in their applications. Furthermore, we provide a
comprehensive performance evaluation of CUDA Graphs lifecycle in the context of multi-path communication, analyzing their overheads and benefits when applied to middleware libraries like UCX. Our work is distinct from collective communication libraries such as NCCL and UCC. While these libraries have incorporated CUDA Graphs to optimize the collective operations, our work integrates CUDA Graphs directly into the UCX transport layer to accelerate P2P communication. This enables graph-based execution of fine-grained, staged, multi-path P2P transfers that are transparent to upper-layer libraries and applications, including collectives built on top of UCX.
4
Design and Implementation
To enable efficient and scalable communication between GPUs in the same node, we have designed a system that builds upon the existing functionality of uct_cuda transport module in UCX. This is built by adding both a multi-path communication engine, which supports the use of multiple communication paths simultaneously, and a runtime acceleration mechanism based on CUDA Graph, which minimizes the synchronization costs. Our overall design philosophy is to prioritize maintaining compatibility with the internal structure of UCX, introducing innovative features on top of it, and utilizing the advantages of concurrent paths, with minimal overhead additions. In this section, we will present a detailed description of our architecture, describe the pipelining mechanism that we have used, discuss several strategies for runtime tuning, and ensure that we discuss the guarantees pertaining to data integrity in detail.
4.1
Design Objectives
Our design addresses the following core goals to support highperformance communication: (1) Multi-GPU awareness: Automatically detect and utilize all available GPUs and their interconnects for efficient communication path construction within UCX instances. (2) Path selection: Dynamically select the most suitable and available paths between source and destination GPUs, including direct and indirect routes via staging GPUs or host. (3) Communication scheduling: Efficiently distribute data along the selected paths using fine-grained pipelining. (4) Path optimization: Maximize throughput and minimize idle time by overlapping transfers across paths. (5) Data integrity: Maintain correctness by ensuring ordered, synchronized, and contention-free transfers. (6) Low overhead: Keep the framework’s runtime and memory overhead negligible compared to the overall communication cost. (7) Seamless integration: Ensure compatibility with existing UCX applications, requiring minimal changes to leverage the new capabilities. (8) Dynamic adaptability: Support dynamic graph construction and caching to optimize repetitive communication patterns.
4.2
Framework Architecture
Figure 3 presents a simplified overview of our enhanced framework. The major components are described below:
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs
CUDA Graph Based Multi-Path UCT CUDA module
2
MD = Memory Domain, EP = Endpoint, IFACE = Interface
Base MD IFACE 3
1
MD IFACE
CUDA IPC
IPC Cache
EP
Figure 4: A simplified view of 2-D pipelined communication from GPU-0 to GPU-1 using the available NVLinks and PCIe. Staging GPUs’ timelines are not shown for simplicity.
Figure 3: Interaction between the Multi-Path Communication Handler, the 2-D Pipelining Engine, and the CUDA Graph Engine. The handler selects the communication paths, and the pipelining engine determines chunking and ordering. Then these configurations are passed to the CUDA Graph Engine, which constructs the graph that executes the staged multi-path workflow. The graph is cached for future reuse.
Start
C-0
C-7
B-0
B-5
A-0
A-3
0 -> 2
0 -> 2
0 -> 3
0 -> 3
0 -> H
0 -> H
D 0 ->1
C-0
C-7
B-0
B-5
A-0
A-3
2 -> 1
2 -> 1
3 -> 1
3 -> 1
H -> 1
H -> 1
B-5 0 -> 3
• Base Module: This module is responsible for probing the system’s hardware topology and initializing internal structures for the framework. It detects the available GPUs and their interconnects, and creates the necessary runtime resources for each GPU, including CUDA streams, CUDA events, device buffers, path metadata, and the communication proxy objects that bridge CUDA IPC and the graph engine. Some of these functionalities are inherited from the original uct_cuda module, and we use the already available functionalities to detect the GPUs and their interconnects. However, the rest of the explained functionalities are newly implemented as part of our framework. • CUDA IPC Module: This module handles incoming communication requests, selecting the suitable paths, handling configurations from environment variables, and scheduling the communication along the selected paths with the 2-D pipelining engine. The connection between the base and cuda_ipc modules is established through proxy entities. • CUDA Graph Engine: This engine constructs reusable CUDA Graphs in a lazy fashion using the explicit CUDA Driver API. It abstracts the pipelined multi-path workflow into a single CUDA Graph, reducing launch overhead and synchronizations. Each graph represents a unique communication configuration between two GPUs, with varying numbers of paths and chunks. • CUDA Graph Cache: The cache stores instantiated CUDA Graphs based on the communication configuration, including source/destination buffer addresses and message size. It uses a configurable least-recently-used (LRU) eviction policy and is implemented as a fixed-size hash table, tunable via environment variables.
4.3
2-D Pipelining Engine and CUDA Graph Execution
Our 2-D pipelining engine is the core of the data distribution mechanism. As shown in Figure 4, a large message is first partitioned according to the number of selected paths (horizontal split). Then,
End
Figure 5: A CUDA Graph-based multi-path communication from GPU-0 to GPU-1 using the available NVLinks and PCIe. The graph in this figure is generated by the CUDA Graph Engine shown in Figure 3. Each node denotes a memory-copy operation, and edges represent the control-flow dependencies ensuring staged transfers execute in the required order.
each path is further subdivided into smaller chunks (vertical split) that are transferred in a pipelined fashion. The number of chunks per path is a tunable parameter depending on message size and the capabilities of each link. Our experiments indicate that a chunk size of 1MB generally provides a good balance between overhead and concurrency. However, to provide maximum overlap between paths, we use the tuned chunk count as described in Section 4.4. Each data chunk is processed using two separate CUDA streams: one for the source-to-staging transfer and another for the stagingto-destination transfer. This parallelization allows communication along multiple routes to proceed concurrently, while also enabling fine-grained overlap within each route. When CUDA Graphs are enabled, the entire communication workflow, including chunk partitioning, staging transfers, synchronization, and final commit, is encapsulated into a Directed Acyclic Graph (DAG), as shown in Figure 5. Each node in the DAG represents a distinct memory copy or kernel operation, and edges represent control-flow or execution dependencies. This abstraction reduces per-transfer launch overhead, improves reuse for repetitive patterns, and allows for better optimization by the CUDA runtime scheduler. In the presence of CUDA Graphs, the multi-path workflow is orchestrated as follows: the Multi-Path Communication Handler selects the available paths and delegates to the 2-D Pipelining Engine, which determines chunk distribution and ordering. This engine
Sojoodi et al.
Algorithm 1: CUDA Graph Creation and Launch for MultiPath UCX Transfers
Algorithm 2: CUDA Graph Nodes Creation and Configuration
Function CreateGraph(𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑠𝑖𝑧𝑒, 𝑐𝑡𝑥𝑠𝑟𝑐 , 𝑐𝑡𝑥𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚): 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 ← get device IDs from 𝑐𝑡𝑥𝑠𝑟𝑐 , 𝑐𝑡𝑥𝑑𝑠𝑡 3 𝑔𝑟𝑎𝑝ℎ_𝑖𝑑 ← get unique graph ID based on 𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑠𝑖𝑧𝑒 4 𝑐𝑜𝑛𝑓 𝑖𝑔 ← GetPathConfig(𝑠𝑖𝑧𝑒) 5 UpdateTempIDs(𝑐𝑜𝑛𝑓 𝑖𝑔, 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 ) 6 𝑠𝑒𝑛𝑡 _𝑑𝑎𝑡𝑎 ← 0 7 foreach path 𝑝 in 𝑐𝑜𝑛𝑓 𝑖𝑔 do 8 𝑐ℎ𝑢𝑛𝑘_𝑠𝑖𝑧𝑒 [𝑝 ] ← ⌈ (𝑠𝑖𝑧𝑒 × 𝑠ℎ𝑎𝑟𝑒 [𝑝 ] )/𝑚𝑎𝑥_𝑐ℎ𝑢𝑛𝑘𝑠 ⌉ 9 𝑠𝑟𝑐 𝑝 ← OffsetPtr(𝑠𝑟𝑐, 𝑠𝑒𝑛𝑡 _𝑑𝑎𝑡𝑎) 10 𝑑𝑠𝑡𝑝 ← OffsetPtr(𝑑𝑠𝑡, 𝑠𝑒𝑛𝑡 _𝑑𝑎𝑡𝑎) 11 𝑠𝑒𝑛𝑡 _𝑑𝑎𝑡𝑎 += 𝑐ℎ𝑢𝑛𝑘_𝑠𝑖𝑧𝑒 [𝑝 ] 12 if 𝑐𝑜𝑛𝑓 𝑖𝑔.𝑡𝑚𝑝_𝑑𝑒𝑣 [𝑝 ] is direct then 13 PeerToPeerCopy(𝑠𝑟𝑐 𝑝 , 𝑑𝑠𝑡𝑝 , 𝑐ℎ𝑢𝑛𝑘_𝑠𝑖𝑧𝑒 [𝑝 ], 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚𝑠 [𝑝 ], 𝑔𝑟𝑎𝑝ℎ) 14 end 15 else if 𝑐𝑜𝑛𝑓 𝑖𝑔.𝑡𝑚𝑝_𝑑𝑒𝑣 [𝑝 ] is host then 16 StageHostCopy(𝑠𝑟𝑐 𝑝 , 𝑑𝑠𝑡𝑝 , 𝑐ℎ𝑢𝑛𝑘_𝑠𝑖𝑧𝑒 [𝑝 ], 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚𝑠 [𝑝 ], 𝑔𝑟𝑎𝑝ℎ) 17 end 18 else if 𝑐𝑜𝑛𝑓 𝑖𝑔.𝑡𝑚𝑝_𝑑𝑒𝑣 [𝑝 ] is GPU then 19 StageGPUCopy(𝑠𝑟𝑐 𝑝 , 𝑑𝑠𝑡𝑝 , 𝑐ℎ𝑢𝑛𝑘_𝑠𝑖𝑧𝑒 [𝑝 ], 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 , 𝑐𝑜𝑛𝑓 𝑖𝑔.𝑡𝑚𝑝_𝑑𝑒𝑣 [𝑝 ], 𝑠𝑡𝑟𝑒𝑎𝑚𝑠 [𝑝 ], 𝑔𝑟𝑎𝑝ℎ)
Function PeerToPeerCopy(𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑠𝑖𝑧𝑒, 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚𝑠, 𝑔𝑟𝑎𝑝ℎ): 2 for 𝑠 ← 0 to 𝑠𝑡𝑟𝑒𝑎𝑚𝑠 − 1 do 3 create memcpy node: device → device 4 config node with 𝑠𝑖𝑧𝑒 and device ids (𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 ) 5 add memcpy node from 𝑠𝑟𝑐 [𝑠 ] to 𝑑𝑠𝑡 [𝑠 ] on 𝑔𝑟𝑎𝑝ℎ 6 end 7 end 8 Function StageHostCopy(𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑠𝑖𝑧𝑒, 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚𝑠, 𝑔𝑟𝑎𝑝ℎ): 9 for 𝑠 ← 0 to 𝑠𝑡𝑟𝑒𝑎𝑚𝑠 − 1 do 10 create memcpy node: device (𝑖𝑑𝑠𝑟𝑐 ) → host (staging) 11 create memcpy node: host → device (𝑖𝑑𝑑𝑠𝑡 ) 12 add dependency between staging and completion nodes 13 end 14 end 15 Function StageGPUCopy(𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑠𝑖𝑧𝑒, 𝑖𝑑𝑠𝑟𝑐 , 𝑖𝑑𝑑𝑠𝑡 , 𝑖𝑑𝑠𝑡𝑎𝑔𝑒 , 𝑠𝑡𝑟𝑒𝑎𝑚𝑠, 𝑔𝑟𝑎𝑝ℎ): 16 for 𝑠 ← 0 to 𝑠𝑡𝑟𝑒𝑎𝑚𝑠 − 1 do 17 create memcpy node: device (𝑖𝑑𝑠𝑟𝑐 ) → staging GPU (𝑖𝑑𝑠𝑡𝑎𝑔𝑒 ) 18 create memcpy node: staging GPU (𝑖𝑑𝑠𝑡𝑎𝑔𝑒 ) → device (𝑖𝑑𝑑𝑠𝑡 ) 19 add dependency between staging and completion nodes 20 end 21 end
1
2
end end 22 store constructed graph instance with 𝑔𝑟𝑎𝑝ℎ_𝑖𝑑 in cache 23 end 24 Function LaunchGraph(𝑐𝑡𝑥𝑠𝑟𝑐 , 𝑐𝑡𝑥𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚, 𝑔𝑟𝑎𝑝ℎ_𝑖𝑑): 25 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 ← get graph instance with 𝑔𝑟𝑎𝑝ℎ_𝑖𝑑 from cache 26 if 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 is NULL then 27 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 ← CreateGraph(𝑠𝑟𝑐, 𝑑𝑠𝑡, 𝑠𝑖𝑧𝑒, 𝑐𝑡𝑥𝑠𝑟𝑐 , 𝑐𝑡𝑥𝑑𝑠𝑡 , 𝑠𝑡𝑟𝑒𝑎𝑚) 28 end 29 launch 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒 on appropriate stream 30 record event and synchronize with input 𝑠𝑡𝑟𝑒𝑎𝑚 31 end 20
1
21
then issues a structured workflow to the CUDA Graph Engine, which builds or retrieves the required CUDA Graph for execution. Algorithm 1 and Algorithm 2 outline the algorithms for creating and launching these CUDA Graphs. As shown in Algorithm 1, the CreateGraph function is responsible for constructing a new CUDA Graph instance based on the communication parameters, while the LaunchGraph function retrieves and launches a CUDA Graph instance. In more detail, Algorithm 1 starts by determining the source and destination device IDs from their respective CUDA contexts (Line 4). It then generates a unique graph ID based on the source/destination addresses and message size (Line 5). The path configuration is obtained using the GetPathConfig function, which considers factors such as message size and available paths (Line 6). Temporary device IDs for staging are updated according to the source and destination devices and the configuration (Line 7). The message is then partitioned into chunks for each path, and the appropriate
copy functions are invoked based on whether the path is direct or staged (Lines 9-21). In our original implementation, we used a round-robin approach to distribute chunks across paths one-by-one, but we provide the simplified version here for clarity. Finally, the constructed graph instance is stored for future use (Line 22). In the LaunchGraph function, the graph instance is retrieved using the unique graph ID (Line 28) from the cache. If the instance does not exist, it is created using the CreateGraph function (Lines 29-31). The graph is then launched on the appropriate stream (Line 32), and a CUDA event is recorded on the CUDA stream and passed to UCX for further processing (Line 33). In Algorithm 2, we detail the helper functions used to create specific types of memory copy nodes, including direct peer-to-peer copies and staged transfers via host or intermediate GPUs. In each function, we create the necessary CUDA Graph nodes and configure them with the appropriate parameters, such as source/destination addresses, sizes, and device IDs. These nodes are then added to the graph instance passed by reference. In Lines 18 and 29, we add the necessary dependencies to ensure the correct ordering of operations for staged transfers of chunks.
4.4
Runtime Tuning and Configuration
To accommodate diverse hardware topologies and workloads, our framework supports both user-defined and auto-tuned configurations. • Environment Configuration: Users can configure the framework via environment variables to enable or disable specific paths (e.g., host-staging), specify the number of concurrent paths, and define the number of chunks per path. These configurations
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs
allow flexibility for both expert users and general-purpose deployment. • CUDA Graphs based Topology Tuning: We extend the offline tuning mechanism to also support CUDA Graph-based workflows. This tuning stage exhaustively searches for optimal communication configurations based on the number of GPUs, available links with various bandwidths and latencies, and transfer sizes. The tuned parameters are used as defaults unless overridden by the user. Note that with the CUDA Graph integration, this tuning should be performed separately for CUDA Graph-enabled and non-CUDA Graph modes, as the optimal configurations may differ due to the reduced overheads and improved scheduling capabilities of CUDA Graphs. Therefore, the number of paths and the number of chunks per path are fixed values determined by the offline topology tuning mechanism. These configurations are used in all evaluation experiments to ensure optimal performance.
4.5
Ensuring Data Integrity and Correctness
To preserve correctness, our design ensures strict data consistency and avoids race conditions by leveraging CUDA’s synchronization semantics. The following guarantees are enforced with both the non-CUDA Graph and CUDA Graph-based implementations: • Contention Avoidance: For each directional path (e.g., GPU-2 to GPU-3), only one transfer is allowed at a time. This is achieved by mapping all transfers for a specific direction to the same CUDA stream and synchronizing using CUDA events. • Dependency Handling: For paths involving staging buffers (e.g., via host or intermediate GPU), the second transfer (staging to destination) is issued only after the first completes. • Ordering Guarantee: Transfers for each chunk within a message are scheduled in a well-defined order and completed before subsequent messages are handled. Reordering is not an issue in our design because each chunk writes into a pre-allocated, non-overlapping region of the destination buffer. The sender determines the destination offset for every chunk before initiating the transfer. Final completion is signaled only after all chunkspecific synchronization points indicate successful arrival. • Final Synchronization: All CUDA streams are synchronized at the end of a message to ensure that the transfer has completed and that the data in the destination buffer is consistent and ready for use. As explained earlier, this is done by recording a CUDA event on each stream within each path and instructing the main CUDA stream to wait for all events to complete before returning control to the caller. This is done in a non-blocking manner, allowing UCX to continue processing other requests while waiting for the communication to complete. These mechanisms ensure that pipelined and concurrent communications do not result in corrupted or out-of-order data, making the framework suitable for latency-sensitive and bandwidth-bound applications.
5
Evaluation
To assess the performance benefits and practical effectiveness of our proposed framework, we evaluated both the pipelined multipath communication engine and the CUDA Graph-based runtime
extensions under a diverse set of benchmarks. These include synthetic micro-benchmarks as well as an application-level use case. The micro-benchmarks consist of UCX Put Bandwidth, OSU MicroBenchmarks (OMB) MPI Bandwidth (OMB_BW), and OMB MPI Bidirectional Bandwidth (OMB_BIBW) [7]. For application-level evaluation, we used the Jacobi iterative solver [5], a representative stencil computation pattern. Moreover, to analyze the behavior and impact of various configurations of CUDA Graphs in our multi-path framework, we performed a series of experiments using OMB MPI Latency tests. These tests help us understand how different graph structures and sizes affect performance, particularly in terms of latency reduction and overhead minimization in various stages of a CUDA Graph lifecycle, including creation, construction, instantiation, and launch.
5.1
Experimental Setup
All experiments were conducted on two multi-GPU compute nodes from the Digital Research Alliance of Canada clusters: Beluga and Narval. The Beluga node is equipped with four NVIDIA V100 GPUs, each connected via two bidirectional NVLink links in a full-mesh topology. The Narval node includes four NVIDIA A100 GPUs with a denser interconnect, where each pair of GPUs is connected by four NVLinks. The full topological layout of Beluga is depicted in Figure 2. Both systems ran UCX version 1.14.0. For MPI support, we used Open MPI version 5.0.4 for both the non-CUDA Graph and CUDA Graph-based evaluations. In all tests, unless otherwise noted, we used pinned host memory and CUDA IPC for all device memory allocations and transfers. We also performed these experiments 1000 times and report the average results. All performance comparisons in this section use the traditional single-path UCX (UCT::CUDAIPC) transport as the baseline.
5.2
UCX Put Bandwidth Evaluation
Figure 6 shows the performance of UCX Put operations measured with two ranks, one on each GPU, on both Beluga (top) and Narval (bottom) clusters. We compare the baseline performance of default UCX (UCT::CUDA-IPC) against our pipelined multi-path and CUDA Graph-based framework. The key observations include: (1) When using three GPU paths and optionally including a hoststaging path, our approach achieves up to 2.85× and 2.75× bandwidth improvements over the baseline on Beluga and Narval, respectively, for message sizes larger than 32MB, which is important for bandwidth-bound applications such as Large Language Model (LLM) and Deep Learning (DL) workloads. (2) The integration of CUDA Graphs further enhances performance, especially for larger data sizes, due to the increased number of nodes/operations in the CUDA Graph comparing to the small data sizes. The results show up to 2.95× and 2.85× speedup on Beluga and Narval, respectively, for the same size range. The reason for more improvement for larger message sizes is due to the fact that the number of operations in the CUDA Graph is larger. For instance, for a 128MB message size, where we use three GPU paths and four chunks per path (on average) the created CUDA Graph has 16 memory copy nodes, with four chunks per path on average. Packing more operations into a single CUDA Graph
Sojoodi et al.
100 80 60
2.5 2.0 1.5
2 GPU Paths, wo/ host 2 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, wo/ host 3 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, w/ host 3 GPU Paths, w/ host, w/ CUDA Graph
1.0 0.5
(a) OMB_BW, Window = 1, on Beluga
3.0 2.5
40 20 512KB 1MB
3.0
Speedup
Bandwidth (GB/s)
120
Default UCX MultiPath, 2 GPU Paths, without host path MultiPath with CUDA Graph, 2 GPU Paths, without host path MultiPath, 3 GPU Paths, without host path MultiPath with CUDA Graph, 3 GPU Paths, without host path MultiPath, 3 GPU Paths, with host path MultiPath with CUDA Graph, 3 GPU Paths, with host path
Speedup
140
2MB
4MB
8MB
16MB 32MB 64MB 128MB 256MB 512MB
2.0 1.5 1.0
Message Size
0.5
(b) OMB_BW, Window = 4, on Beluga
3.0
230
2.5
Speedup
200
Bandwidth (GB/s)
170
2.0 1.5 1.0
140
0.5
110
KB
512
B
1M
B
2M
80 50 20 512KB 1MB
2MB
4MB
8MB
16MB 32MB 64MB 128MB 256MB 512MB
B
4M
B
8M
B
16M
B
32M
B
64M
MB
128
(c) OMB_BW, Window = 16, on Beluga
MB 12MB 256 5
Figure 7: Multi-Path OMB Unidirectional MPI Bandwidth (BW) comparison with and without CUDA Graph, on Beluga
Message Size 3.0 2.5
Speedup
Figure 6: UCX Put Bandwidth comparison of our multi-path framework with and without CUDA Graph against default UCX (UCT::CUDA-IPC), on Beluga (top) and Narval (bottom)
2.0 1.5
2 GPU Paths, wo/ host 2 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, wo/ host 3 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, w/ host 3 GPU Paths, w/ host, w/ CUDA Graph
1.0 0.5
(a) OMB_BW, Window = 1, on Narval
3.0
Speedup
2.5 2.0 1.5 1.0 0.5
(b) OMB_BW, Window = 4, on Narval
3.0 2.5
Speedup
reduces the launch overhead and synchronization costs on the CPU side, leading to better performance. (3) Including the host path contributes marginally to the bandwidth improvement (up to 15%), which is expected given the lower performance of PCIe relative to NVLink. In many scenarios, this contribution is negligible compared to enabling additional direct GPU paths. (4) To avoid clutter in the figure, intermediate configurations (e.g., one or two GPU paths with host path enabled) are omitted, since they show similar patterns to the configurations with three GPU paths.
2.0 1.5 1.0
5.3
MPI Micro-benchmark Results
Figure 7 and Figure 8 present OMB MPI unidirectional bandwidth (BW) measurements on Beluga and Narval, respectively. Also, Figure 9 and Figure 10 depict the bidirectional bandwidth (BIBW) tests for both clusters. These tests were conducted with two MPI ranks, each assigned to one GPU, and various message sizes ranging from 1MB to 64MB. All tests were conducted across different message sizes and window sizes (1, 4, and 16). The window size defines how many messages can be posted without waiting for prior ones to complete. We observe the following key results: (1) On Beluga, results are consistent with those observed in the UCX Put test. On Narval, however, we see less improvement for window size one, which suggests that its four NVLinks do
0.5
KB
512
B
1M
B
2M
B
4M
B
8M
B
16M
B
32M
B
64M
(c) OMB_BW, Window = 16, on Narval
MB
128
MB 12MB 256 5
Figure 8: Multi-Path OMB Unidirectional MPI Bandwidth (BW) comparison with and without CUDA Graph, on Narval
not saturate as quickly as Beluga’s two NVLinks. This pattern is observable in both classic and CUDA Graph-enhanced versions. (2) For both platforms, bandwidth utilization increases with higher window sizes, particularly for message sizes between 8MB and 64MB. This trend is visible in both unidirectional and bidirectional bandwidth plots.
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs 3.0
2 GPU Paths, wo/ host 2 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, wo/ host 3 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, w/ host 3 GPU Paths, w/ host, w/ CUDA Graph
Speedup
2.5 2.0 1.5 1.0 0.5
(a) OMB_BIBW, Window = 1, on Beluga
3.0
Speedup
2.5 2.0
Figure 11: Jacobi communication pattern (a) without multi-path, and (b) with multi-path (two paths per communication) on a four-GPU node
1.5 1.0 0.5
(b) OMB_BIBW, Window = 4, on Beluga
3.0
Speedup
2.5 2.0 1.5 1.0 0.5
KB
512
B
1M
B
2M
B
4M
B
8M
B
16M
B
32M
B
64M
128
MB
(c) OMB_BIBW, Window = 16, on Beluga
256
MB
MB
512
Figure 9: Multi-Path OMB Bidirectional MPI Bandwidth (BIBW) comparison with and without CUDA Graph, on Beluga 3.0
2 GPU Paths, wo/ host 2 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, wo/ host 3 GPU Paths, wo/ host, w/ CUDA Graph 3 GPU Paths, w/ host 3 GPU Paths, w/ host, w/ CUDA Graph
Speedup
2.5 2.0 1.5 1.0 0.5
(a) OMB_BIBW, Window = 1, on Narval
3.0
Speedup
2.5 2.0 1.5 1.0 0.5
(b) OMB_BIBW, Window = 4, on Narval
3.0
We also observed performance drops for message sizes of 1MB and 2MB, which we suspect are caused by internal algorithm or protocol switches within UCX or MPI. This phenomenon is under further investigation.
2.5
Speedup
outstanding/non-blocking requests (i.e., larger window sizes) can benefit more from CUDA Graph integration. This means that multiple CUDA Graphs can be launched concurrently, allowing the CUDA runtime to better schedule and overlap operations. (4) In contrast, for messages smaller than 8MB, the number of operations in the CUDA Graph is small, and the graph launch overhead becomes significant, negating its benefits. This can be seen in the performance drop for message sizes 512MB and 4MB in Figure 7(a), Figure 7(b), and Figure 7(c), where we have only one or two copy nodes in the CUDA Graph. This pattern is also observable in Narval results and in bidirectional tests. (5) Enabling host paths generally does not improve performance significantly, especially on Narval with window size one (see Figure 8(d) and Figure 10(c)). On Narval, the difference in bandwidth between NVLink and the host-staging path is more prominent than Beluga, and the host path contribution to overall bandwidth is minimal. (6) In bidirectional tests, enabling the host path consistently degrades performance, especially in CUDA Graph versions. This is because both directions share the same PCIe link to the host, creating contention and bottlenecks. This pattern is more pronounced with higher window sizes, as more messages are inflight. See Figure 9(c) and Figure 10(c) for instance.
2.0 1.5 1.0
5.4
0.5
5
B 12K
1
MB
2
MB
4
MB
8
MB
16
MB
32
MB
64
MB
1
B 28M
(c) OMB_BIBW, Window = 16, on Narval
256
MB
MB
512
Figure 10: Multi-Path OMB Bidirectional MPI Bandwidth (BIBW) comparison with and without CUDA Graph, on Narval
(3) For messages larger than 8MB, enabling CUDA Graph enhances performance even further, especially when the window size is increased to 4 or 16. CUDA Graph allows for overlapping transfers and reduced launch overhead, leading to better performance. This observation suggests that applications that issue multiple
Jacobi Iterative Solver
To validate the framework in an application scenario, we evaluated the performance of the Jacobi iterative solver using MPI with one rank per GPU (a total of four ranks per node). Each rank computes a subregion of the domain and exchanges boundary data with its immediate neighbors using a 2-D halo exchange pattern, as shown in Figure 11(a), which is basically a ring. Considering the unutilized diagonal links, and the NVLink’s bidirectional feature, 2/3 of the total available bandwidth is unused in this pattern. Therefore, we can enable multi-path communication for all of these data exchanges, and select the staging GPUs in a way that there is no contention on the NVLinks. Figure 11(b) demonstrates how each communication is split into two paths, and the data is transferred concurrently through the NVLinks.
Application Runtime Speedup on Beluga
Sojoodi et al.
1.2 1.1
MultiPath MPI+UCX with 2 GPU Paths MultiPath MPI+UCX with 2 GPU Paths, with CUDA Graphs
1.0 0.9
B
MB
64M
128
MB
256
MB
512
1GB
2GB
4GB
8GB
Application Data Size
Application Runtime Speedup on Beluga
(a) Beluga with two GPU paths 1.2 1.1
MultiPath MPI+UCX with 3 GPU Paths MultiPath MPI+UCX with 3 GPU Paths, with CUDA Graphs
1.0 0.9
B
MB
64M
128
MB
256
MB
512
1GB
2GB
4GB
8GB
Application Data Size
Application Runtime Speedup on Narval
(b) Beluga with three GPU paths 1.15 1.10 1.05 1.00 0.95 0.90
MultiPath MPI+UCX with 2 GPU Paths MultiPath MPI+UCX with 2 GPU Paths, with CUDA Graphs
B
64M
MB
128
MB
256
MB
512
1GB
2GB
4GB
8GB
4GB
8GB
Application Data Size
Application Runtime Speedup on Narval
(c) Narval with two GPU paths 1.15 1.10 1.05 1.00 0.95 0.90
MultiPath MPI+UCX with 3 GPU Paths MultiPath MPI+UCX with 3 GPU Paths, with CUDA Graphs
B
64M
MB
128
MB
256
MB
512
1GB
2GB
Application Data Size (d) Narval with three GPU paths
Figure 12: Jacobi runtime speedup over default UCX (UCT::CUDA-IPC) using four MPI ranks on Beluga and Narval clusters
We varied the problem size by fixing the vertical dimension to 8 and increasing the horizontal dimension from 223 to 230 . This means that for the total application data size of 8GB on four GPUs, each rank exchanges 256MB of boundary data with its two neighbors in each iteration. We ran the solver for 1000 iterations and measured the total execution time. We also disabled host staging for this evaluation, as it consistently degraded performance in previous tests. The results, shown in Figure 12, are compared against the default UCX (UCT::CUDA-IPC) configuration. The key findings indicate:
(1) Using two concurrent paths per communication improves application runtime by up to 1.26× and 1.15× on Beluga and Narval, respectively. (2) Integration of CUDA Graphs results in further performance gains, achieving up to 1.28× and 1.16× speedup on Beluga and Narval, respectively, for larger problem sizes. (3) As discussed earlier, the benefits of CUDA Graphs are more pronounced for larger messages due to the increased number of operations in the graph, which helps amortize the launch overhead. For smaller problem sizes, the overhead of CUDA Graph creation and launch can outweigh its benefits, leading to marginal or no improvement: for application sizes smaller than 1GB (message size = 32MB), in which the number of operations in the CUDA Graph is insufficient to amortize the overhead. This is also consistent with our observations in the bidirectional micro-benchmarks (Section 5.3). (4) Even with three GPU paths per transfer, potentially incurring additional contention, our framework still improves the runtime. Although the gains are smaller than with two paths, we still observe improvements of up to 1.2× and 1.08× on Beluga and Narval, respectively. This suggests that the benefits of increased concurrency do not outweigh the contention costs in this scenario, and two paths may be the optimal choice. (5) It is important to note that in this application we are only using the CUDA Graphs for enabling multi-path transfers, and none of the other operations (e.g., computation kernels) are included in the CUDA Graph. The communication patterns created and launched in this test map to a graph which has a very small number of vertices along each path (only two nodes per chunk per path). This limits the potential for optimization from CUDA Graph utilization. These results show that in scenarios with more complex communication patterns, the advantages of CUDA Graphs could be more significant, especially when kernel executions are involved. (6) Importantly, Jacobi’s numerical convergence remains unaffected, demonstrating the correctness of our pipelined and concurrent transfer mechanisms.
5.5
CUDA Graph Overhead Analysis
In this section, we characterize the overheads associated with various CUDA Graph operations in our multi-path framework, including creation, construction, instantiation, and launch. This analysis helps us understand the trade-offs involved in using CUDA Graphs, particularly in terms of latency reduction and overhead minimization for different message sizes and number of nodes in the graph. These insights can guide users in deciding when and how to effectively utilize CUDA Graphs in their applications. We collected timing data during the OMBLatency micro-benchmark on Narval for messages between 2MB and 512MB (note that multi-pathing starts at 2MB with only two nodes in the graph). In more detail, these are the various phases: (1) Creation involves allocation and initialization of the CUDA Graph object (cudaGraph_t) during the first time execution. For subsequent executions, this step is skipped, and the existing CUDA Graph object is fetched from a hash table and reused.
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs
(2) Construction includes defining and creating the graph nodes, which denotes the data movements, as well as their inter-dependencies. We used the CUDA Graph explicit API to construct the CUDA Graph design and update the existing CUDA Graph object. Similar to creation, this step is only performed during the first execution, and the constructed graph is reused in subsequent executions. (3) Instantiation is a one-time expensive operation that allocates resources for the CUDA Graph execution and optimizing it. After this step, the object cudaGraphExec_t is created, which can be launched multiple times. This step is synchronous, and its overhead can vary based on the complexity of the graph. (4) Launch refers to the initiation of the actual execution of the CUDA Graph on the GPU. Similar to a CUDA kernel launch, this process is asynchronous and expensive, and its overhead might vary depending on the complexity of the graph. Figure 13 and Figure 14 demonstrate how the overhead of various CUDA Graph operations changes according to the number of nodes within the graph when they are called for the first/subsequent times. Analyzing the results, we observe the following: • Observation 1: In all the graph phases during the first iteration, the overhead increases with the number of nodes. See Figure 13(a), Figure 13(b), Figure 13(c), and Figure 13(d) for creation, construction, instantiation, and launch, respectively. This is expected, as more nodes imply more operations and dependencies to manage. Notably, instantiation depicts the highest overhead, reaching up to 3ms for 34 nodes (512MB message size with dual-path communication). • Observation 2: During the subsequent calls, the overhead stays constant and low for all the cases, except for the launch phase. Figure 14(a) shows how the launch overhead increases with the number of nodes (figures of other phases are not provided here for brevity). • Observation 3: Contribution of each graph phase to the total latency varies significantly between the first and subsequent iterations. While instantiation is the most dominant operation during the first iteration, the launch phase becomes more significant in subsequent iterations. Compare Figure 14(b) and Figure 14(c). • Observation 4: The overall latency benefits from the multipath optimization become more pronounced as the message size increases. Although the overhead of CUDA Graph operations increases with node count (and message size), the relative impact on latency decreases (See Figure 14(d)). These results suggest that CUDA Graphs provide scalable performance improvements as the complexity of the communication pattern increases. Overall, these results suggest that while CUDA Graph operations introduce some overhead, especially during the first iteration, the benefits of multi-path communication outweigh these costs for larger messages. As we saw, the overhead of creation, construction, and instantiation is amortized over multiple launches, making them negligible for long-running applications. This is why libraries such as UCX can benefit from CUDA Graphs, as they can create and instantiate the graphs once and reuse them multiple times for recurring communication patterns.
6
Conclusions and Future Work
We proposed a CUDA Graph-based multi-path communication framework implemented within the UCX library that reuses communication workflows to minimize launch overhead and improve intra-node communication efficiency. By leveraging available communication channels, our 2-D pipelining engine scatters P2P communication across both NVLink and PCIe channels to maximize communication bandwidth between intra-node GPUs. We also provide end users with tuning capabilities for scheduling parameters, ensuring adaptability to diverse communication patterns and hardware configurations. Our experimental evaluation on a four-GPU node demonstrates that this approach achieves up to a 2.95× increase in GPU-to-GPU OMB MPI bandwidth test compared to UCX single-path method for very large messages (≥ 32𝑀𝐵). We also observed that besides utilizing unused interconnects, harnessing NVLink’s bidirectional features will also improve communication performance. However, the lack of the same feature in PCIe communication channels may lead to contention, and consequently, performance degradation. Finally, we showed that our multi-path communication framework can improve the performance of the Jacobi iterative solver by up to 1.28×. We showed that the benefits of CUDA Graphs were marginal relative to the non-CUDA Graph multi-path framework, due to the limited number of operations in the graph. Our results suggest that in scenarios with more complex communication patterns, the advantages of CUDA Graphs could be more significant, especially when kernel launches are involved, and when the overhead of kernel/communication launches are considerable compared to the actual execution time of these operations. Furthermore, we discussed the overheads associated with various CUDA Graph operations in the context of multi-path communication, including creation, construction, instantiation, and launch. Our analysis indicates that while these operations introduce some overhead, particularly during the first iteration, the benefits of multi-path communication outweigh these costs for larger messages, and the overheads are amortized over multiple launches, making them negligible for long-running applications. While our approach can improve the communication bandwidth, the performance gain is dependent on the concurrent communication pattern. This is why we believe that a more adaptive approach could yield better results. A possible future direction is to dynamically adapt the communication pattern based on both the application’s communication pattern and the hardware configuration. Moreover, we plan to extend our framework to optimize our CUDA Graphs engine to minimize their creation and instantiation time, applying smarter caching mechanisms, and dynamically adjusting the CUDA Graphs structure and parameters through their explicit API. Another interesting future direction is to explore the integration of our multi-path communication framework with collective communication operations, such as Allreduce, and Alltoall. By leveraging the benefits of multi-path communication in designing collective operations, we can potentially achieve significant performance improvements for applications that rely heavily on these operations.
Sojoodi et al.
graph_creation_latency
graph_construction_latency Latency (µs)
Latency (µs)
101 6 × 100 4 × 100 3 × 100
101
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
Message Size (B) with number of graph nodes in brackets below
Message Size (B) with number of graph nodes in brackets below
(a) Overhead of the first CUDA Graph creation.
(b) Overhead of the first CUDA Graph construction.
graph_instantiation_latency
graph_launch_latency
102 Latency (µs)
Latency (µs)
103
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
102
101 101
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
Message Size (B) with number of graph nodes in brackets below
Message Size (B) with number of graph nodes in brackets below
(c) Overhead of the first CUDA Graph instantiation.
(d) Overhead of the first CUDA Graph launch.
Figure 13: Measurement of various CUDA Graph operations during the first iteration of OMB Latency benchmark on Narval for dual-path communication from message sizes 2MB to 512MB. Note that multi-path is enabled from the 2MB. Fraction of Total Latency
graph_launch_latency Latency (µs)
102
101
1.0 0.8 0.6 0.4 0.2 0.0
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
graph_creation_latency graph_construction_latency graph_instantiation_latency graph_launch_latency
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
Message Size (B) with number of graph nodes in brackets below
(a) Overhead of the subsequent CUDA Graph launches for various node counts.
(b) Contribution of each graph phase to the total latency (first iteration).
1.0 0.8 0.6
graph_creation_latency (avg) graph_construction_latency (avg) graph_instantiation_latency (avg) graph_launch_latency (avg)
0.4
102 101
0.2 0.0
103
Latency (µs)
Fraction of Total Latency
Message Size (B) with number of graph nodes in brackets below
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
data_transfer_latency (avg) graph_creation_latency (avg) graph_construction_latency (avg) graph_instantiation_latency (avg) graph_launch_latency (avg) 47%
36%
27%
29%
22%
16%
11%
8%
5%
3%
2%
219 220 221 222 223 224 225 226 227 228 229 [1] [1] [2] [2] [4] [6] [8] [12] [16] [24] [34]
Message Size (B) with number of graph nodes in brackets below
Message Size (B) with number of graph nodes in brackets below
(c) Contribution of each graph phase to the total latency (subsequent iterations).
(d) Comparison of subsequent CUDA Graph phases with the CUDA Graph enabled multi-path transfer.
Figure 14: Measurement of various CUDA Graph operations during OMB Latency benchmark on Narval for dual-path communication from message sizes 2MB to 512MB. Note that multi-path is enabled from 2MB.
Accelerating Intra-Node GPU-to-GPU Communication Through Multi-Path Transfers with CUDA Graphs
Finally, we plan to empirically validate the scalability of our framework on 8+ GPU nodes and multi-node architectures, including NVSwitch topologies and DGX systems. We believe that our approach can provide substantial benefits in these topologies, where multiple high-bandwidth links are available between GPUs.
7
Acknowledgments
This research was supported in part by the Natural Sciences and Engineering Research Council of Canada and Digital Research Alliance of Canada. Computations were performed on Beluga and Narval with support from Calcul Québec (calculquebec.ca).
References [1] 2025. CUDA. https://docs.nvidia.com/cuda/index.html [Accessed: 2025-04-01]. [2] 2025. CUDA Graphs. https://developer.nvidia.com/blog/cuda-graphs/ https: //developer.nvidia.com/blog/cuda-graphs/ [Accessed: 2025-04-01]. [3] 2025. CUDA Graphs in Dynamic Environments. https://developer. nvidia.com/blog/employing-cuda-graphs-in-a-dynamic-environment/ https://developer.nvidia.com/blog/employing-cuda-graphs-in-a-dynamicenvironment/ [Accessed: 2025-04-01]. [4] 2025. InfiniBand Trade Association. https://www.infinibandta.org/ https: //www.infinibandta.org/ [Accessed: 2025-04-01]. [5] 2025. Multi-GPU Jacobi Solver. https://github.com/NVIDIA/multi-gpuprogramming-models https://github.com/NVIDIA/multi-gpu-programmingmodels [Accessed: 2025-04-01]. [6] David E. Bernholdt, Swen Boehm, George Bosilca, Manjunath Gorentla Venkata, Ryan E. Grant, Thomas Naughton, Howard P. Pritchard, Martin Schulz, and Geoffroy R. Vallee. 2020. A survey of MPI usage in the US exascale computing project. Concurrency and Computation: Practice and Experience (CCPE) 3 (2020), 1–16. doi:10.1002/cpe.4851 [7] Devendar Bureddy, H. Wang, A. Venkatesh, S. Potluri, and D. K. Panda. 2012. OMB-GPU: A micro-benchmark suite for evaluating MPI libraries on GPU clusters. In Proceedings of the European MPI Users’ Group Meeting (EuroMPI). 110–120. doi:10.1007/978-3-642-33518-1_16 [8] Chen Chun Chen, Kawthar Shafie Khorassani, Pouya Kousha, Qinghua Zhou, Jinghan Yao, Hari Subramoni, and Dhabaleswar K. Panda. 2023. MPI-xCCL: A Portable MPI Library over Collective Communication Libraries for Various Accelerators. In Proceedings of the SC Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W). 847–854. doi:10.1145/3624062.3624153 [9] Yuxin Chen, Benjamin Brock, Katherine Yelick, and John D Owens. 2022. Scalable Irregular Parallelism with GPUs : Getting CPUs Out of the Way. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). 1–16. doi:10.1109/SC41404.2022.00055 [10] Jaemin Choi, David F. Richards, and Laxmikant V. Kale. 2022. Improving Scalability with GPU-Aware Asynchronous Tasks. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 1–10. arXiv:2202.11819 doi:10.1109/IPDPSW55747.2022.00097 [11] Jonah Ekelund, Stefano Markidis, and Ivy Peng. 2025. Boosting Performance of Iterative Applications on GPUs: Kernel Batching with CUDA Graphs. arXiv (2025), 1–8. arXiv:2501.09398 http://arxiv.org/abs/2501.09398 [12] Tsung Wei Huang, Dian Lun Lin, Chun Xun Lin, and Yibo Lin. 2021. Taskflow: A Lightweight Parallel and Heterogeneous Task Graph Computing System. IEEE Transactions on Parallel and Distributed Systems (2021), 1303–1320. doi:10.1109/ TPDS.2021.3104255 [13] Dian Lun Lin and Tsung Wei Huang. 2021. Efficient GPU Computation Using Task Graph Parallelism. In Proceedings of the European Conference on Parallel Processing (Euro-Par). Springer International Publishing, 435–450. doi:10.1007/978-3-03085665-6_27 [14] MPI Forum. 2025. https://www.mpi-forum.org/ [Accessed: 2025-04-01]. [15] MPICH. 2025. https://www.mpich.org/ [Accessed: 2025-04-01]. [16] Akira Nukada. 2022. Performance Optimization of Allreduce Operation for Multi-GPU Systems. In Proceedings of the International Conference on Big Data (Big Data). IEEE, 1–6. doi:10.1109/bigdata52589.2021.9672073 [17] NVIDIA. 2025. https://www.nvidia.com/ [Accessed: 2025-04-01]. [18] NVIDIA. 2025. NVIDIA Collective Communications Library. https://github. com/NVIDIA/nccl [Accessed: 2025-04-01]. [19] Open MPI. 2025. https://www.open-mpi.org/ [Accessed: 2025-04-01]. [20] Bo Qiao, M. Akif Ozkan, Jurgen Teich, and Frank Hannig. 2020. The best of both worlds: Combining CUDA graph with an image processing DSL. In Proceedings of the ACM/IEEE Design Automation Conference (DAC). 1–6. doi:10.1109/DAC18072. 2020.9218531
[21] Pavel Shamis, Manjunath Gorentla Venkata, M Graham Lopez, Matthew B Baker, Oscar Hernandez, Yossi Itigin, Mike Dubman, Gilad Shainer, Richard L Graham, Liran Liss, and Yiftah Shahar. 2015. UCX : An Open Source Framework for HPC Network APIs and Beyond. In Proceedings of the IEEE Symposium on HighPerformance Interconnects (HOTI). IEEE, 40–43. doi:10.1109/HOTI.2015.13 [22] Amirhossein Sojoodi, Mohammad Akbari, Hamed Sharifian, Ali Farazdaghi, Ryan E. Grant, and Ahmad Afsahi. 2025. Accelerating Intra-Node GPU Communication: A Performance Model for Multi-Path Transfers. In Proceedings of the Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W). Association for Computing Machinery, 449–460. doi:10.1145/3731599.3767392 [23] Amirhossein Sojoodi, Ali Farazdaghi, Hamed Sharifian, Ryan E Grant, and Ahmad Afsahi. 2025. Collaborative Bandwidth-Efficient Intra-Node Allreduce. In Proceedings of the International Workshop on Accelerators and Hybrid Emerging Systems (AsHES). 1–5. doi:10.1109/IPDPSW66978.2025.00016 [24] Amirhossein Sojoodi, Majid Salimi Beni, and Farshad Khunjush. 2020. IgniteGPU: a GPU-enabled in-memory computing architecture on clusters. Journal of Supercomputing (2020), 1–28. doi:10.1007/s11227-020-03390-z [25] Amirhossein Sojoodi, Yıltan Hassan Temucin, and Ahmad Afsahi. 2024. Enhancing Intra-Node GPU-to-GPU Performance in MPI + UCX through Multi-Path Communication. In Proceedings of the International Workshop on Extreme Heterogeneity Solutions (ExHET). 1–6. doi:10.1145/3642961.3643800 [26] Yuya Tatsugi and Akira Nukada. 2022. Accelerating data transfer between host and device using idle GPU. In Proceedings of the Workshop on General Purpose Processing using GPUs (GPGPU). 1–6. doi:10.1145/3530390.3532732 [27] Yıltan Hassan Temucin, Amirhossein Sojoodi, Pedram Alizadeh, and Ahmad Afsahi. 2021. Efficient Multi-Path NVLink / PCIe-Aware UCX based Collective Communication for Deep Learning. In Proceedings of the IEEE Symposium on High-Performance Interconnects (HOTI). 1–10. doi:10.1109/HOTI52880.2021.00018 [28] Yıltan Hassan Temucin, Amirhossein Sojoodi, Pedram Alizadeh, Benjamin W Kitor, and Ahmad Afsahi. 2021. Accelerating Deep Learning using InterconnectAware UCX Communication for MPI Collectives. IEEE Micro (2021), 1–9. doi:10. 1109/MM.2022.3148670 [29] Top500. 2025. https://top500.org/ [Accessed: 2025-04-01]. [30] Unified Communication Framework Consortium. 2025. Unified Collective Communication (UCC). https://github.com/openucx/ucc [Accessed: 2025-04-01]. [31] Unified Communication Framework Consortium. 2025. Unified Communication X (UCX). https://openucx.org/ [Accessed: 2025-04-01]. [32] Manjunath Gorentla Venkata, Valentine Petrov, Sergey Lebedev, Devendar Bureddy, Ferrol Aderholdt, Joshua Ladd, Gil Bloch, Mike Dubman, and Gilad Shainer. 2024. Unified Collective Communication ( UCC ): An Unified Library for CPU , GPU , and DPU Collectives. In Proceedings of the IEEE Symposium on HighPerformance Interconnects (HOTI). IEEE, 37–46. doi:10.1109/HOTI63208.2024. 00018 [33] Yuxuan Zhao, Qi Sun, Zhuolun He, Yang Bai, and Bei Yu. 2023. AutoGraph: Optimizing DNN Computation Graph for Parallel GPU Kernel Execution. Proceedings of the AAAI Conference on Artificial Intelligence 37 (2023), 1–9. doi:10.1609/aaai.v37i9.26343 [34] Bojian Zheng, Cody Hao Yu, Jie Wang, Yaoyao Ding, Yizhi Liu, Yida Wang, and Gennady Pekhimenko. 2023. Grape: Practical and Efficient Graphed Execution for Dynamic Deep Neural Networks on GPUs. In Proceedings of the International Symposium on Microarchitecture (MICRO). 1364–1380. doi:10.1145/3613424.3614248