arXiv CSopen access

HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters

HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters Peking University

Beijing Academy of Artificial Intelligence

Tao Chang

Yuanyuan Zhao Peking University

Beijing Academy of Artificial Intelligence

Zhiyu Li

Yanmin Jia

Yan Zhang

Mingjun Zhang

Yuejie Wang

Beijing Academy of Artificial Intelligence

Infrawaves

arXiv:2605.31000v1 [cs.NI] 29 May 2026

Yongzhe He

Institute of Computing Technology, Chinese Academy of Sciences

Infrawaves

Yonghua Lin

Beijing Academy of Artificial Intelligence

Infrawaves

Zeyu Gu

Beijing Academy of Artificial Intelligence

He Liu

Infrawaves

Guyue Liu

Peking University

to meet their immense computational and memory requirements. Second, enterprises are increasingly adopting diverse GPUs from multiple vendors (e.g., NVIDIA [29], AMD [2], Huawei [20]). Relying solely on a single vendor to build homogeneous clusters (e.g., NVIDIA’s A100 accelerators [29]) is becoming increasingly impractical due to factors such as cost efficiency, incremental upgrades, and supply constraints (details in §2.1). The shift towards multi-vendor heterogeneous clusters presents a promising opportunity to enhance resource utilization and cost-effectiveness for LLM training. Realizing this potential hinges on the efficiency of collective communication libraries (CCLs) [21, 28, 30], which orchestrate essential operations such as AllReduce and AllGather. These operations are fundamental to parallelization strategies [36, 40] like data parallelism (DP) and tensor parallelism (TP) etc., ensuring efficient synchronization and data exchange across GPUs. Unfortunately, existing communication libraries are designed for homogeneous environments and struggle to efficiently support heterogeneous clusters. Existing collective communication libraries broadly fall into two categories: • Device-centric libraries, such as NCCL from Nvidia [30] and RCCL from AMD [3], are kernel-based collective algorithm implementations deeply optimized for their respective hardware, utilizing vendor-specific techniques such as NVSHMEM [12] and GPU-direct RDMA [11] for fast intra- and inter-node communication. CCLs of this category achieve high efficiency in homogeneous clusters, but are inherently vendor-locked, preventing interoperability with GPUs from other manufacturers.

Abstract Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse network and computational characteristics. Existing collective communication frameworks (e.g., NCCL, RCCL) designed for homogeneous environments fail to address mixed-hardware setups, while communication libraries with heterogeneous support (e.g., Gloo, OpenMPI) incur heavy overhead in the data path. This paper presents HetCCL, a framework that enables heterogeneous collective communication by efficient P2P transport across heterogeneous devices (e.g., GPUs), eliminating the host-device memory copy overhead while offloading the control to the CPUs. For combining collectives (e.g., AllReduce, ReduceScatter), HetCCL introduces a bordercommunicator mechanism that achieves vendor independence by using the intrinsic reduction in the combining collectives in vendor collective communication libraries. With efficient heterogeneous P2P transport and portable reduction mechanism, HetCCL proposes a hierarchical topology abstraction for heterogeneous clusters, dissecting collective communication into cluster-level primitives that guarantee optimal cross-cluster data transfer volume and optimal bandwidth utilization. We implement HetCCL with 4 different vendor support and evaluate it in 4 heterogeneous settings with benchmarks and end-to-end LLM tasks. Our evaluation shows that HetCCL achieves 17 ∼ 19× higher bandwidth than Gloo in heterogeneous communications, and speeds up end-to-end training by up to 16.9% in the per-step-time.

1

Yulong Ao

• Host-centric libraries, such as OpenMPI [1] and Gloo [14], are designed for traditional HPC workloads and rely on host memory as an intermediate buffer for data transfers. While CCLs of this category enable broader hardware compatibility, the frequent host-device memory copies introduce substantial overhead. This inefficiency becomes particularly problematic in large-scale LLM training, where high communication costs can significantly degrade overall performance.

Introduction

The rapid advancement of Large Language Models (LLMs) has driven two key trends in large-scale training. First, the increasing size of LLMs demands ever-larger GPU clusters. State-of-the-art models have grown from billions to trillions of parameters [15, 37], requiring thousands of GPUs 1

Hardware

Vendor Software Stack

APP

arXiv preprint, 2026

Wang et al. ML Frameworks (eg., PyTorch) & Applications (eg., LLM, NLP, Multi-modal)

Framework for V1 Clusters

Framework for V2 Clusters

Collective Communication Algorithm (V1CCL)

Collective Communication Algorithm (V2CCL)

Framework for Mixed-vendor Clusters

Heterogeneous Collective Communication Algorithm - HetCCL Collective Primitive

Device Library V1 Programming Model V1

Device Library V2

RDMA Transport

Vendor 1 Hardware Computation Node

Programming Model V2

X Accelerator Cards (eg. GPU)

RDMA Transport

V1CCL Collective Communication API

Vendor 2 Hardware Intra-node Interconnects

V1 Device API

RDMA Transport

RDMA Library (eg. verbs API)

Vendor 1 Hardware Intra-node Switch

Device Primitive

V2CCL Collective Communication API

V2 Device API

Vendor 2 Hardware

NIC

Figure 1. Existing and our proposed architecture for clusters with hardware from multiple vendors, dealing with the hardware heterogeneity, differences in the programming models, and varying visibility and control over the underlying hardware.

Neither approach provides an efficient solution for heterogeneous LLM training. Device-centric libraries are restricted to single-vendor environments, while host-centric libraries suffer from excessive communication overhead. As a result, existing solutions force a trade-off between implementation efficiency and cross-vendor compatibility, limiting their practicality in heterogeneous clusters. This reality raises a timely and important question: How to design a collective communication library that efficiently supports heterogeneous LLM training without compromising performance or efficiency? Answering this question requires overcoming challenges at multiple levels, including the data path efficiency for device-to-device communication, implementation portability of data reduction across platforms, and the abstraction for heterogeneous cluster topologies and collective algorithms. In this paper, we propose HetCCL, a novel framework that enables the modeling and optimization of collective communications in mixed-vendor accelerator card clusters. HetCCL is built on the following key ideas: • Cross-vendor Device Data Transport: Existing data exchange in collective communications either copies data to a CPU bounce buffer (CPU-forwarding approach [1, 14]) or utilizes vendor-specific transports on device memory buffers, such as NVLink and Infinity Fabric for intra-server interconnect, and GPU-Direct RDMA (GDR [17, 33]) for inter-server RDMA transport. The CPU-forwarding approach suffers from significant host-device memory copy overhead, and the vendor-specific transports embedded in collective communication kernels are not portable across platforms. We decouple the control logic and data path of the device buffer RDMA, using a host-centric, kernelfree control logic to ensure vendor compatibility, while keeping the data path completely on-device, thereby eliminating the host-memory copying overhead associated with CPU-based approaches.

the programmability discrepancies across different vendor hardware in heterogeneous clusters, it is challenging to implement data reduction compatible with a wide range of vendors. Although existing work has proposed offloading reduction to the CPU [5], this approach is against the design principle of keeping the data path on-device. Our design utilizes the device computation resource for faster data reduction and lower data copy overhead. We propose a cross-vendor data exchange pattern that stores local and received data on separate ranks and uniformly implements reduction utilizing the reduce interface provided by existing homogeneous collective libraries. • Hierarchical Algorithm Design: In mixed-vendor clusters, the key challenge lies in the trade-off between algorithm flexibility and implementation efficiency. Using peer-topeer (P2P) data transfer as the core primitive for collective communications offers high algorithm flexibility (e.g., existing works on automatic collective algorithm optimization [25, 26, 38, 45]). On the other hand, homogeneous device-centric collective implementations provide higher implementation efficiency. We propose a fused approach based on our hierarchical topology abstraction that divides the heterogeneous cluster into multiple homogeneous clusters. At the homogeneous cluster level, we propose an algorithm primitive abstraction including both intra- and inter-cluster operations, leveraging the flexibility of cluster-level P2P primitives and the efficiency of device-centric homogeneous collective primitives. • Pipelined Collective Algorithm Execution: Sequentially executing the algorithm primitives would lead to bandwidth underutilization, as the inter-cluster links remain idle while waiting for intra-cluster operations to complete. On top of the primitive abstraction and collective algorithm design, we build a pipelined execution workflow to overlap intra- and inter-cluster primitives, maximizing bandwidth utilization across the heterogeneous cluster. We implement HetCCL with 30k LOC, supporting 8 different vendors, and incorporate HetCCL into PyTorch via

• Vendor-independent Reduction: Apart from data movements, combining collectives (i.e., AllReduce, ReduceScatter, and Reduce) also performs data computation. Given 2

HetCCL

arXiv preprint, 2026

the customized backend. We evaluated it on a heterogeneous cluster containing hardware from 4 vendors with P2P SendRecv, collective communication benchmarks, as well as end-to-end performance, including training with Llama33B/8B models and serving with Qwen2-7B model. To the best of our knowledge, this is the first heterogeneous collective communication library that supports full device-buffer data transfer and MPI-style collective operations. HetCCL is publicly available. We omit the link to comply with anonymity requirements. Our evaluation results show that HetCCL achieves up to 91.4% hardware bandwidth of the slowest vendor in heterogeneous SendRecv and 97% the bandwidth of homogeneous collectives in heterogeneous collective communications. In end-to-end training, HetCCL accelerates the perstep-time by 9.1% and 16.9% for Llama3-3B and Llama38B models, where the computation dominates communication. We expect even greater benefits from HetCCL in more communication-intensive scenarios, such as large-scale model training [13, 27]. Ethics: This work does not raise any ethical issues.

2

overall resource utilization in cloud environments. Furthermore, enterprises leverage heterogeneous cluster setups to avoid being limited by a single vendor’s hardware production capacity, ensuring procurement flexibility while maximizing the utility of legacy hardware. This blend of scalability, adaptability, and optimization underscores the increasing reliance on heterogeneous clusters in modern LLM workloads. Hardware NVIDIA A100 AMD MI300x Intel Gaudi 3 Vendor 1 Vendor 2 Vendor 3

TFLOPS (FP32) 156 TFLOPS 163.4 TFLOPS 1835 TFLOPS (FP16) 32 TFLOPS 512 TOPS (INT8) 200TFLOPS

Scale-up Network 4.8 TB/s NVLink Infinity Fabric 4.2 TB/s RoCE 192GB/s 8x100GB/s 240GB/s

Scale-out Network 8x200G IB configurable IB 3x300G RoCE 100G IB 2x200G IB 2x400G IB

Table 1. Summary of various accelerator card vendors and servers. Today’s major hardware vendors include NVIDIA, AMD, and Intel.

Heterogeneous collective communications: Collective communication libraries (CCLs) play a pivotal role in maximizing the efficiency of heterogeneous hardware by coordinating essential operations such as AllReduce and AllGather. Some collective operations only involve data movements, which are commonly referred to as non-combining collectives [6]. On the other hand, AllReduce, ReduceScatter, and Reduce operations collect multiple versions of data and produce a reduced output, which are referred to as combining collectives. As listed in Table 2, these operations serve as the backbone of widely adopted parallel training strategies, e.g., data parallelism (DP), pipeline parallelism (PP), and tensor parallelism (TP) [36, 40], facilitating efficient synchronization and data exchange across multiple GPUs.

Background and Motivation

In §2.1, we introduce the necessity of heterogeneous collective communication and the requirements for an ideal solution. In §2.2∼§2.4, we analyze the limitations of existing approaches and the challenges in meeting the requirements. 2.1

#dev 8 8 8 16 8 8

Heterogeneous Collective Communication

Type Non-combining Non-combining Combining Combining

LLM training in heterogeneous accelerator clusters: Enterprises are increasingly adopting heterogeneous hardware [8, 18, 19, 23] for large language model (LLM) training for practical reasons. First, the diversity of hardware vendors and architectures has expanded significantly over the years. Beyond major vendors like NVIDIA and AMD, other vendors (e.g., Huawei [20], Graphcore [16]) offer enterprises broader options considering availability and task-specific optimizations. Table 1 presents representative hardware settings, with vendor names anonymized due to confidentiality considerations. Second, the growing computational demands of stateof-the-art LLMs necessitate scalable infrastructure, which is rarely deployed in a uniform, single-generation manner. Instead, clusters often evolve incrementally, reflecting the natural life cycle of data centers where multi-generational hardware must coexist and interoperate effectively. Moreover, in commercial server rental and cloud provisioning scenarios, it is often difficult for users to acquire a large number of homogeneous server instances concurrently due to dynamic availability and fragmented resource supply, which can constrain the scale of LLM workloads or incur substantial waiting time. As a result, integrating heterogeneous and fragmented available compute resources for LLM training and inference becomes a practical approach to improving

Collective Operation SendRecv (P2P) AllGather AllReduce ReduceScatter

Parallel Strategy Pipeline Parallelism (PP) Tensor Parallelism (TP) Data and Tensor Parallelism (DP, TP) Tensor Parallelism (TP)

Table 2. Summary of collective communications used in common parallel strategies (DP, TP and PP).

Unfortunately, existing communication libraries are designed for homogeneous environments, as shown in Figure 1 (left). Applications such as ML frameworks adapt to each vendor CCL as separate backends to achieve portability across homogeneous clusters of different vendors, but cannot run across a heterogeneous cluster due to incompatible CCL backends. More specifically, vendor-specific CCLs implement collective algorithms (the scheduling of data movements, reductions, and synchronizations) with tailored device code, using specialized low-level device libraries and programming models (e.g., CUDA and ROCm) for different types of hardware. This lack of interoperability makes it difficult for upper-layer frameworks to adapt to new hardware vendors or apply optimizations across mixed-vendor clusters. An ideal solution for heterogeneous collective communication should meet the following requirements: 1) portability across heterogeneous hardware, 2) seamless integration with 3

arXiv preprint, 2026 data path GPU 0

Wang et al.

control path Incompatible device implementation

app data

kernel

net buffer

d2d memcpy

NIC

RDMA (GDR)

proxy CPU 0

memory buffers

device kernel

GPU 1

PCIe switch & link

GPU 0

kernel

app data

d2d memcpy

net buffer

GPU 1

app data

D2H / H2D copy overhead PCIe bottleneck

NIC

NIC

TCP/ RDMA

net proxy buffer d2h memcpy

proxy

GPU 0 app data

CPU 1

(a) Device-centric control + device-buffer data

d2d memcpy

NIC proxy

net buffer

(b) Host-centric control + host-forwarding data

NIC proxy

CPU 1

GPU 1 app data

On-device data path

net buffer

h2d memcpy

CPU 0

app data

CPU 0

Dev-buffer RDMA

net buffer

NIC

Compatible host implementation

d2d memcpy

proxy CPU 1

(c) Host-centric control + device-buffer data

Figure 2. Inter-node device data transfer mechanism comparison. downstream applications, 3) minimum software adaptation required from each vendor, 4) high performance in collective communication. Fulfilling these requirements calls for an alternative architecture, as shown in Figure 1 (right), with a collective communication library that wraps the underlying heterogeneity across hardware vendors and exposes high-performance collective interfaces to applications. We observe that the hardware heterogeneity across vendors includes their accelerator card model and intra-server highbandwidth interconnects, but they share the same scale-out network architecture, namely the inter-server RDMA (Remote Direct Memory Access) transport. Therefore, we choose the common RDMA APIs (i.e., verbs) as the bridge for crossvendor device data transport, with a minimal set of unified device APIs universally provided by various vendors. Building a collective communication library with our proposed architecture faces several challenges, spanning from the data transfer primitive to the design of the collective algorithms. 2.2

libraries leverage in-depth knowledge of their respective hardware architectures and programming models to achieve high performance. Their inter-node data transfer typically follows Figure 2 (a), where the data movements are devicedriven and directly in device memory buffers via GPUDirect RDMA [17, 33], ROCm [4], etc.. This mechanism avoids costinefficient operations such as host-device memory copies. The limitation is the strong dependency on vendor-specific technologies, which limits their cross-vendor compatibility. Their only support of hardware heterogeneity is across different hardware generations from a single vendor. Host-centric approaches: Another line of collective communication implementations, such as OpenMPI [7] and Gloo [14], originally focuses on distributed CPU tasks, such as traditional HPC applications. While some have been extended to support GPU transport via third-party integrations, their core architecture remains host-centric. These libraries can support cross-platform device data exchange, as shown by Figure 2 (b), where the device buffer is copied to a host bounce buffer and forwarded to the remote node’s host buffer via TCP or CPU RDMA, then copied to the peer device buffer. This mechanism falls short in its data-path efficiency, as data must frequently move between the device and the host, incurring heavy memory-copy overhead and PCIe bottleneck. For modern workloads such as LLM training that demand high-speed collective operations, this can become a significant bottleneck. Data path efficiency comparison: Figure 3 shows the memory copying time in different data path implementations, profiled during 2GB SendRecv operations of NVIDIA (nv) and Vendor 1 (v1) hardware. For mechanism (b) in Figure 2, a device-to-host memory copy (d2h) on the sender and a hostto-device memory copy (h2d) on the receiver takes up more than 3.8× time on average than two device-to-device (d2d) memory copies. On the other hand, a fully device-driven method would require a unified and efficient programming model with device RDMA support (e.g., CUDA and GDR for NVIDIA hardware) that is compatible with every vendor, which lacks established standards for varying hardware architectures and abilities. Therefore, keeping an on-device data path for heterogeneous data transfer is challenging, as it

Heterogeneous Data Transfer Challenge

Challenge 1: How to perform efficient and kernel-free device data transfer between heterogeneous peers. We group common collective communication libraries into device-centric and host-centric approaches based on their different inter-node data transfer mechanisms 1 . Table 3 summarizes the characteristics of the two approaches. CCL NCCL RCCL OpenMPI Gloo HetCCL

Performance High High Low Low Relatively High

Compatibility Single vendor Single vendor Mixed vendor Mixed vendor Mixed vendor

Data Path GPU-buffer RDMA GPU-buffer RDMA CPU forwarding CPU forwarding GPU-buffer RDMA

Control Path GPU kernel GPU kernel CPU control CPU control CPU control

Table 3. Summary of existing frameworks on their vendor compatibility, performance, and the data and control path implementation for inter-node data transfer.

Device-centic approaches: Vendor-developed CCLs, such as NCCL [30] for NVIDIA and RCCL [3] for AMD, are meticulously optimized for homogeneous environments. These 1 Accelerator cards are provided and purchased as integrated servers, so

hardware within the same node is homogeneous, and heterogeneous peers only exist in inter-node data transfers

4

Mem Copy Time (ms)

HetCCL

arXiv preprint, 2026

80 70 60 50

nv_h2d nv_d2h nv_d2d

For heterogeneous clusters, completely relying on kernelfree P2P data transfer introduces higher overhead across homogeneous peers than existing kernel implementations. In a word, vendor-provided CCL implementations fall short in cross-vendor portability and algorithm flexibility. Modeling-based collective optimization approaches provide a more flexible algorithm design space, but when executed with heterogeneous P2P transport primitives, yield higher implementation overhead. We need to balance the trade-off between these two approaches in our heterogeneous collective algorithm design, achieving both the high implementation efficiency of device kernels and the flexibility of P2P data transfers.

v1_h2d v1_d2h v1_d2d

40 30 20 10 0

nv

v1

Figure 3. Data path overhead of different mechanisms. requires balancing compatibility and performance between the two existing inter-node data transfer mechanisms. 2.3

Portable Reduction Challenge

Challenge 2: Implement vendor-agnostic data reduction. In §2.2, we focus on the efficiency of data transfers across heterogeneous peers, which is sufficient to construct noncombining collective operations (Table 2), but combining collectives such as AllReduce and ReduceScatter are also important components in downstream tasks, which additionally demand the ability to reduce data (e.g., compute sum, min, and max). This introduces new challenges. Existing reduction implementations rely on the portability of their underlying programming model to scale to more hardware types, which lacks a universal solution. Writing separate device code for each distinct device model requires vigorous and continuous effort, and does not apply to vendors that do not provide an open programming platform. A recent work, HFReduce [5], offloads data reduction to the CPU, potentially making it portable across vendors. But this is incompatible with the goal of maintaining an on-device data path for heterogeneous collective communication. The challenge lies in reducing device data in a uniform way across various vendors in heterogeneous combining collectives. 2.4

3

Design Overview

In this section, we present the key insights to address the above challenges for heterogeneous collective communication, followed by an overview of the HetCCL architecture.

3.1

Collective Algorithm Efficiency Challenge

Challenge 3: Efficiency trade-off between collective algorithm flexibility and implementation efficiency. The collective algorithm determines the theoretical communication performance. Industrial solutions such as NCCL [30] implement limited pre-defined algorithms, such as the ring or tree algorithm. Downstream applications such as LLM training frameworks use these libraries via the exposed APIs in a black-box manner. However, pre-defined algorithms do not guarantee optimal latency or bandwidth for arbitrary topologies, especially in heterogeneous settings. Recent works [6, 26, 38] search for optimal customized algorithms under their communication cost models, using P2P data transfer and device data reduction as algorithm primitives. Collective operations, namely scheduling of these primitives, run in a high-performance execution backend [6, 28] implemented for homogeneous NVIDIA or AMD hardware only. 5

Key Ideas

Idea 1. Decouple data and control paths for P2P transfer of device data: We separate the RDMA data and control paths to eliminate the host-device data copy overhead for heterogeneous peers. HetCCL adopts a novel host-driven device-buffer RDMA mechanism, where the control logic (e.g., memory region management, RDMA operations, connection management, event handling, etc.) is scheduled on the host side for maximum hardware and programming compatibility, and the data path remains on the device to avoid host-device data movement and the potential PCIe bottleneck (Figure 2 (c)). Idea 2. Vendor-independent reduction using native combining collectives: Vendor-provided CCLs provide built-in kernel implementations of data reduction for combining collectives. By leveraging the multiple ranks involved in a collective operation, we insert an intermediate receiving rank and align the offsets for P2P data transfers to perform the data computation using built-in reduction implementations in existing CCLs. Idea 3. Hierarchical collective algorithm design based on vendor hardware groups: Existing vendor CCLs are more efficient for homogeneous device subgroups, whereas the P2P-level data transfer primitive offers greater flexibility and better algorithmic optimality. We propose a hierarchical topology abstraction and a corresponding collective algorithm design, applying high-efficiency device kernels within homogeneous device subgroups in the heterogeneous cluster, while leveraging the flexibility of P2P data transfers to meet the remaining communication requirements.

arXiv preprint, 2026

Wang et al. Grouped Data Transfers

APP

Application Frameworks (eg. PyTorch) PyTorch Custom Backend: Heterogeneous Collective API

Send Steps

Task Process

preconnect

CPU Proxy

register buffer

0

1

2

CPU

launch

§4.3 Pipelined Heterogeneous Collective Algorithm Cluster-level Collective Operation Breakdown

Pipelined Algorithm Execution Workflow

Recv Steps 3 complete

hostFunc

enqueue

Comm Stream

hostFunc

HetCCL

GPU Copy Stream

§4.2 Cluster-level Collective Primitives Hierarchical Topology Abstraction

Multi-channel C2C Data Transfer

Vendor-agnostic Reduction

memcpy 0 memcpy 1

memcpy 2

RDMA send queue

send 0

send 1

RDMA recv queue

recv 2

recv 3

memcpy 3

NIC Vendor CCL Wrapper Module NCCL

RCCL

MSCCL

§4.1 Device P2P Transport ...

Mem Ops

Stream Ops

Host Func

ROCm

...

NVLink

Infinity Fabric

IB Verbs

Vendor

Figure 5. P2P Transport for Device Data. Programming Models & Vendor Device Libraries Hardware

NV GPU

CUDA

AMD GPU

NPU

...

and explains how HetCCL addresses the compatibility issue encountered by combining collectives. §4.3 introduces the hierarchical collective algorithm and its pipelined execution.

libverbs ...

RNIC

Figure 4. Design Overview 3.2

4.1

HetCCL Architecture Design

Figure 4 shows the design overview of HetCCL, including three key components and a wrapper module: 1) a devicelevel primitive abstraction for efficient heterogeneous P2P transport, 2) a hierarchical topology abstraction and corresponding cluster-level primitives, 3) the heterogeneous collective algorithm with a pipelined execution workflow, and 4) a light-weight vendor API wrapper module for leveraging vendor CCL optimizations. ① P2P transport for device data (§4.1): We summarize a list of node-level primitives (Table 5) to enable hostdriven device-buffer heterogeneous P2P transport. The primitives include device memory operations, control operations (e.g., stream management), host-device coordination, and RDMA resource management. ② Cluster-to-cluster primitives (§4.2): Our hierarchical topology abstraction breaks the heterogeneous cluster into homogeneous subgroups, enabling homogeneous collective semantics and group data transfer semantics via the cluster-level primitives (Table 7). We ensure the efficiency of cross-vendor communication primitives via multichannel load-balanced P2P data transfer. For combining collectives, we design a data transfer-reduce primitive that uses the native homogeneous combining collective implementation to produce the reduction result for the global collective operation. ③ Collective algorithm and pipelined execution (§4.3): We propose a collective operation breakdown (Algorithm 1) aligned with the hierarchical topology, where cluster-level primitives collectively implement the semantics of heterogeneous collective operations. The collective algorithm efficiency is guaranteed by optimal cross-vendor data transfer volume and pipelined execution workflow.

4

Device Buffer P2P Transport

The basic building block of heterogeneous collective communication is the underlying heterogeneous P2P transport. HetCCL enables device buffer data transfer across heterogeneous peers via a novel host-device-coordinated RDMA mechanism. As summarized in §2.2, the data transfer across heterogeneous devices in existing frameworks copies device buffers to host buffers and uses host transport (RDMA or TCP) to transfer data across heterogeneous nodes, which suffers from significant memory copying overhead and PCIe bottleneck. HetCCL proposes mechanism (c) of Figure 2 to follow the data path of mechanism (a) while keeping the cross-vendor portability of mechanism (b). More specifically, mechanism (c) is built with the following components: Host-driven control logic: Based on the common host functionalities for managing device memory and RDMA operations across various vendors, instead of the device-centric data transfer logic (e.g., NCCL [30], RCCL [3]), we offload the RDMA control logic to a host proxy, guaranteeing high vendor compatibility. The host proxy functionalities include initializing connections, resource management (e.g., managing memory region (MR), completion queue (CQ), protection domain (PD), RDMA buffers, etc.), memory operations (e.g., malloc and memcpy), and RDMA operations (e.g., posting Work Requests (WR) and polling from the CQ). When the send host proxy handles a data transfer job, it first calls d2dMemcpy to move the target data to an available RDMA send buffer on the device. The proxy queries the stream state for the completion of d2dMemcpy, after which the proxy enqueues the send WR to the send queue of the QP to the target receiver. On the receiver side, the host proxy receives data by polling the RDMA receive queue, after which the receiver buffer is copied to the target device buffer. On-device data path: This mechanism keeps the RDMA data path fully on-device: 1) the sender copies data from the device memory to the RDMA send buffer, 2) the RNIC transfers the RDMA buffer to the remote peer through the RNIC, and 3) the receiver copies the receive buffer to the

HetCCL Design

In §4.1, we introduce our heterogeneous device-buffer P2P transport design. §4.2 presents the cluster-level primitives 6

HetCCL

arXiv preprint, 2026 Cluster Hardware a) Hom. Coll.

Border Comm

Cluster 0 Vendor 1

Topology Abstraction

Heterogeneous Cluster a) Hom. Coll. b) Het.

[Het] Global CommH [Hom] Vendor Device Groups

C2C c) Hom. Coll.

Cluster CommC c) Hom. Coll. Cluster 0

Border CommB Cluster 1

Border Rank

Internal Rank

Vendor 2

Primitive

Type

Communicator

homColl

HOM

within a 𝐶𝑜𝑚𝑚𝐶 or 𝐶𝑜𝑚𝑚𝐵

c2cCpy

HET

𝐶𝑜𝑚𝑚𝐵 to 𝐶𝑜𝑚𝑚𝐵

c2cRed

HET

𝐶𝑜𝑚𝑚𝐵 to 𝐶𝑜𝑚𝑚𝐵

Custom Params send/recv buffer offsets, data length send/recv buffer offsets, #border_ranks available bounce buffers, #border_ranks

Table 4. Heterogeneous collective algorithm primitives. Figure 6. Heterogeneous Cluster Topology Abstraction device memory. Compared with the CPU-forwarding mechanism, host-device memory copies are replaced with deviceto-device memory copies (similar to existing device-centric solutions), eliminating the most significant data-path overhead. HetCCL further pipelines the above control logic to overlap the memory-copying and RDMA transfer time and to reuse a pre-allocated RDMA buffer pool. A data transfer job is sliced into fixed-sized data chunks, and the memcpy and RDMA operations are launched asynchronously (e.g., in Figure 5, Data-1 does not require the completion of Data-0 to start its memcpy). Host-device coordination: In addition to satisfying the data movement semantics, the P2P primitive also guarantees operational integrity and maintains the same device-side dependency as a device operation. For instance, jobs submitted to a device stream should execute sequentially. However, since HetCCL offloads the control logic to the CPU, there is no control instance on the device execution pipeline (e.g., CUDA stream) to ensure that future jobs wait for previous RDMA operations to complete. Figure 5 shows the interaction between the host and the device side to maintain the correct dependency while pipelining the memory copy and RDMA transfer in a P2P operation. The proxy simultaneously calls the launchHostFunc API to serve as a placeholder on the device communication stream, marking events for synchronization and maintaining dependencies. The proxy polls the CQ for completion of the RDMA WR, after which it can mark the send task as complete and safely release the related RDMA resource for future operations. Takeaway: By decoupling the control and data path in RDMA transport into separate implementations on the host and device, HetCCL removes the host-device memory-copying overhead and the PCIe bottleneck without device-dependent logic. HetCCL ensures the correct dependency for non-blocking execution, thereby achieving high performance and compatibility across various hardware vendors.

4.2

4.2.1

Heterogeneous topology abstraction

HetCCL’s hierarchical topology abstraction groups the devices into homogeneous sub-clusters, connected by crosscluster RDMA channels. Figure 6 shows the hierarchical abstraction for heterogeneous clusters and the corresponding software structures. During the global communicator (denoted by 𝐶𝑜𝑚𝑚𝐻 ) initialization, the cluster uses a CPUbased bootstrapping network to gather the rank information globally (usually, each accelerator card is initialized as a rank). Then, HetCCL groups ranks (devices) by vendors, which are also the maximal device subsets that can execute kernel-based collective operations. For instance, Vendor 2 includes 4 nodes, each with 4 accelerator cards and 2 NICs. A vendor device group can further be divided into disjoint sub-clusters, each initialized with a homogeneous communicator. For simplicity, in this paper, we use the term Cluster to refer to the sub-clusters, in which vendor-provided libraries can perform homogeneous collectives. As the intra-node topologies of different vendors are not identical, for instance, the number of devices per node and NICs per node may vary, we define the ranks that have the minimum NUMA distance to an RDMA NIC as border ranks and others as internal ranks. Additionally, we create an internal border communicator (𝐶𝑜𝑚𝑚𝐵 ) for the border ranks of each Cluster. For the example in Figure 6, Vendor 2 is further divided into two symmetric Clusters, each with a border communicator consisting of 4 ranks. Every collective operation can be viewed as a communication requirement of data chunks [6], decomposing a collective communication into intra- and inter-cluster data transfer requirements. As demonstrated by the red arrows in Figure 6, each global collective operation can break down into 3 steps, namely a) start intra-cluster (homogeneous) operations, b) cluster-to-cluster (C2C) data transfers, and c) end intra-cluster (homogeneous) operations. We define three cluster-level primitives to perform at each step in the next section.

Hierarchical Topology Abstraction and Cluster-to-cluster Primitives

4.2.2

HetCCL comprises a hierarchical topology abstraction (§4.2.1) and cluster-level primitives (§4.2.2), combining heterogeneous P2P RDMA with existing homogeneous communication to achieve a balance between algorithm flexibility and implementation efficiency.

Cluster-level Primitives

Combining the vendor-provided homogeneous collectives and the flexibility of heterogeneous P2P data transfer in §4.1, HetCCL can cover the data movements required by global collective communications in a heterogeneous cluster. Table 4 defines our cluster-level primitives: 7

arXiv preprint, 2026

Wang et al.

Cluster 0

Cluster 0

rank 0 (border) 0

1

2

rank 1 (border) 3

0

1

2

Cluster 0

rank 0 (border) 3

0 4

1 5

2

rank 1 (border) 3

0

1

2 2 6

3 3 7

6 6

rank 4 (border) 5

6

rank 5 (border) 7

0 4

1 5

2 6

0 4

3 7

Cluster 1

6 6

0 0

rank 1 (border) 0 0

0 0

0 0

6 6

rank 0 (border) 6 6

6 = sum(0..3) 22 = sum(4..7)

Send/Recv

4

Cluster 0

rank 0 (border)

rank 4 (border) 1 5 6 7

0 4

rank 5 (border) 2 3 1 5 2 6 3 7

6 0

22 0

0 0

rank 1 (border)

22 0

22 0

22 0

6 0

6 0

Reduce sum(0..3)+sum(4..7)

Send/Recv

rank 4 (border) 22 22 0 0 22 22 0 0

Cluster 1

6 0

Cluster 0

rank 5 (border) 0 22 22 0 22 22

rank 0 (border) 28 28 0 0 0 0 0 0

Cluster 1

0 0

rank 1 (border) 0 28 28 0 0 0

c2cCpy

c2cRed

Figure 7. Inter-cluster data transfer primitive: c2cCpy

Figure 8. Vendor-agnostic reduction primitive: c2cRed

• homColl performs a homogeneous collective communication with customized send-buffer and receive-buffer offsets and buffer lengths (§5).

leaving other offsets currently unoccupied. The data transfer is similar to c2cCpy, except that the data received from Cluster 1 are routed to ranks where the corresponding data offset is available, then reduced to the target rank and offset by performing a Reduce in the border communicator2 .

• c2cCpy transfers required data from the border ranks of the source cluster to the border ranks of the destination cluster. The data is divided proportionally to the NIC bandwidth and scattered among the receiving border ranks, guaranteeing load balance and optimal transfer volume.

Algorithm 1: C2C Collective Breakdown Params :𝑐𝑜𝑙𝑙: collective operation type, 𝐶𝑜𝑚𝑚𝐻 : global heterogeneous communicator, 𝐶𝑜𝑚𝑚𝐶 : homogeneous cluster communicator, 𝑟𝑎𝑛𝑘: local rank in the communicator, data[𝑖]: the 𝑖th data chunk. 1 Func C2C_Collective_Breakdown: 2 𝐶𝑜𝑚𝑚𝐶 ← 𝐶𝑜𝑚𝑚𝐻 .get_hom_comm(rank) 3 for 𝑖 ← 1 to 𝑛_𝑠𝑡𝑎𝑟𝑡 _𝑙𝑜𝑜𝑝𝑠 do 4 homColl(𝑐𝑜𝑙𝑙.startColl, 𝐶𝑜𝑚𝑚𝐶 , data[𝑖])

• c2cRed additionally performs a combining collective among the border ranks of the destination cluster, achieving a vendor-agnostic implementation of reduction. The key challenge to decomposing a collective operation lies in 1) ensuring optimal cross-cluster data transfer volume during step b) data exchange and 2) implementing vendoragnostic reduce operations for combining collectives. c2cCpy primitive: HetCCL ensures optimal cross-cluster data transfer volume and maximizes the bandwidth utilization of multiple cross-cluster channels and border ranks. HetCCL adopts a cluster-level ring algorithm (only exchanging data with the previous and next cluster) to minimize the total cross-cluster data transfer volume. During each c2cCpy primitive call, only one data copy is transferred across clusters (i.e., the set of received data is distributed across the border ranks of the receiving cluster), while dividing data transfer workload proportionally to the cross-cluster channel bandwidth (e.g., RNIC bandwidth of border ranks). Figure 7 shows an example of c2cCpy primitive, where data 0 ∼ 3 from Cluster 0 and 4 ∼ 7 from Cluster 1 are exchanged across their border ranks. c2cRed primitive: For combining collectives, a key challenge to reducing data on the receiver side is cross-platform compatibility. Especially for emerging vendors, providing a collective communication library implementation on their hardware is significantly easier—and more common in practice—than developing a comprehensive programming platform (e.g., CUDA). This means that it is not always convenient to handcraft a reduction kernel for each type of device in the cluster. Although moving the reduction to the CPU would be a feasible solution, this violates our key insight of keeping the data path on-device. Figure 8 illustrates a running example of c2cRed. Suppose that some partially reduced data are distributed on multiple ranks in Cluster 0,

5 6 7 8 9 10 11

4.3 4.3.1

for 𝑖 ← 1 to 𝑛_𝑐2𝑐_𝑙𝑜𝑜𝑝𝑠 do if 𝑐𝑜𝑙𝑙 is non-combining then c2cCpy(𝐶𝑜𝑚𝑚𝐶 , 𝐶𝑜𝑚𝑚𝐶 .peer_cls) else c2cRed(𝐶𝑜𝑚𝑚𝐶 , 𝐶𝑜𝑚𝑚𝐶 .peer_cls) for 𝑖 ← 1 to 𝑛_𝑒𝑛𝑑_𝑙𝑜𝑜𝑝𝑠 do homColl(𝑐𝑜𝑙𝑙.endColl, 𝐶𝑜𝑚𝑚𝐶 , data[𝑖])

Hierarchical Collective Breakdown Algorithm and Pipelined Execution Cluster-level collective algorithm and communication primitives

In this section, we elaborate on the C2C collective algorithm design of HetCCL, which performs a global collective operation with cluster-level primitives. Algorithm 1 shows the 3-step hierarchical breakdown of collective operations using these primitives. Line 3 ∼ 4 perform a group of intra-cluster homogeneous collective operations (𝑠𝑡𝑎𝑟𝑡𝐶𝑜𝑙𝑙) by calling collective APIs provided in vendor CCLs. Line 5 ∼ 12 performs C2C data transfers, such that each cluster’s border ranks will possess all necessary data from other clusters to complete the global collective 2 This approach cannot work with single-rank vendor clusters, which we

have never encountered in production clusters. In case this happens, we also implement a fallback solution that offloads data reduction to the CPU.

8

HetCCL

arXiv preprint, 2026

Execution dependency: Sequential AllGather:

start homColl in src cluster

c2cCpy/c2cRed

as CPUs are often underutilized during model training and the control logic is lightweight and implemented in a separate proxy thread. The vendor-agnostic reduction design sometimes requires additional memory to reduce different data versions, but scratch buffers are inevitable for combining collectives such as ReduceScatter, and HetCCL does not require larger scratch buffers than existing approaches.

end homColl in dst cluster

comm stream

Pipelined AllGather: intra-cluster stream 0 inter-cluster stream 1

Figure 9. Pipelined Collective Algorithm Execution operation. Line 13 ∼ 14 wraps up the collective communication with another group of intra-cluster collective operations (𝑒𝑛𝑑𝐶𝑜𝑙𝑙), generating the final output value to the internal ranks in the cluster. According to the global communication pattern and the cross-cluster data transfer requirement for each type of collective operation, we summarize the intracluster start and end collectives in Table 7, together with two concrete examples of our algorithm (Figure 20 and Figure 21), in Appendix A. 4.3.2

5

We implement HetCCL with > 30k LOC in C and Python, including the heterogeneous device-buffer RDMA mechanism, hierarchical topology representation, buffer management for implementing efficient heterogeneous collective communications, several lightweight wrappers of vendor-specific libraries, and PyTorch backend plugin for seamless integration with downstream tasks. Device APIs: Using the on-device data path for heterogeneous RDMA requires specific device operations from all vendor hardware. Table 5 lists the necessary vendor functions and their definitions, summarized into four types of device APIs: ib_interface, dev_mem_opr, stream_opr, and host_func. The functionality of these APIs is generally supported by all hardware vendors, so we automatically detect the local vendor and redirect these function calls accordingly. The ib_interface and device_mem_opr ensure hardware access and interaction with device memory and RNICs, and stream_opr and host_func provide device-host coordination, including synchronization and device state-checking. RDMA Transport: At the bootstrap phase (using a TCP bootstrap network), each rank uses these primitives to get access to available RNICs and register RDMA resources for each heterogeneous connection. We statically allocate a fixedsize RDMA buffer pool per connection (64M by default, divided into 4M size chunks), and use the host-side asynchronous device-to-device (d2d) memory copy API for moving

Pipelined Execution Workflow

At a high level, Algorithm 1 represents the construction logic of collective operation with our primitives, but executing the loops sequentially results in low hardware utilization. Take AllGather as an example, we may observe in a cluster that receiving data from 𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝑖 and the intra-cluster broadcasting of data already received from 𝐶𝑙𝑢𝑠𝑡𝑒𝑟𝑖 −1 has no data dependency or bandwidth contention. Furthermore, it is rather common in modern data centers to have every rank in a cluster be a border rank, in which case all data from that cluster is already available for cross-cluster transfers from the beginning, eliminating the need to wait for intra-cluster AllGather. Figure 9 compares the pipelined and sequential execution of AllGahter in this case. Thus, HetCCL adopts a pipelined execution workflow to overlap independent cross-cluster data transfers and intra-cluster collective steps. 4.4

Implementation

Optimality and Tradeoff Analysis

Optimal cross-cluster data transfer: Note that the heterogeneous SendRecv makes step b) synchronous across clusters, with the bottleneck determined by the minimum total bandwidth among vendor device groups. To mitigate this, we optimize both the data transfer volume and bandwidth utilization. For each collective operation, we estimate the minimum cross-cluster data volume, which is constant for most operations, except for Scatter and Gather for nonroot clusters, which scale with cluster size. Our hierarchical algorithm (Algorithm 1) ensures this volume is minimized to reduce cross-cluster P2P overhead. Since total bandwidth increases with cluster size, we further balance bandwidth by computing each vendor group’s capacity during initialization and dividing larger groups into subgroups with roughly equal total bandwidth. System Overhead: One concern for HetCCL’s system overhead is the CPU control path, but we consider it acceptable,

Function

Type

ibv_get_device_list

ib_interface

ibv_open_device

ib_interface

ibv_alloc_pd ibv_reg_mr ibv_create_cq ibv_create_qp ibv_modify_qp ibv_post_send ibv_post_recv ibv_poll_cq devMalloc/Free

ib_interface ib_interface ib_interface ib_interface ib_interface

d2d/d2h/h2dMemcpy

dev_mem_opr

streamCreate/Destroy streamSync streamQuery

stream_opr stream_opr stream_opr

launchHostFunc

host_func

ib_interface ib_interface dev_mem_opr

Description Get the list of available IB devices on the system Open IB device, returning a context for future interaction Create Protection Domain (PD). Register Memory Region (MR). Create Completion Queue (CQ). Create Queue Pair (QP). Modify QP state. Post Work Requests (WRs) to the send/recv queue of a QP. Poll CQ for completed WRs Alloc/free device memory. Device-to-device/device-tohost/host-to-device memory copy. Create/destroy a stream Synchronize streams Query stream state Launch a host function on a device stream for maintaining dependency

Table 5. Device APIs 9

arXiv preprint, 2026

Wang et al.

device memory to RDMA send buffers (or receive buffers to device memory), pipelining d2d memory copying and RDMA transfer for large messages. Vendor CCL light-weight wrapper: HetCCL integrates each supported vendor’s specialized CCL with a lightweight wrapper, and inherits the respective homogeneous communicator (constructed within the heterogeneous communicator for each homogeneous cluster) such that within a homogeneous cluster, HetCCL can utilize vendor-optimized collective operation to implement the desired semantics for sync primitives with better efficiency, without requiring hardware knowledge that may not be publicly available, such as the intra-node interconnect information. System Integration: PyTorch supports integrating various communication infrastructures with its custom backend feature. We integrate HetCCL’s collective communication interfaces into the PyTorch backend plugin feature, such that downstream applications and tasks are unaware of the underlying communication implementation. Specifically, we extend the Work and Backend classes with a set of APIs as custom C++ extensions [34, 35]. With the HetCCL PyTorch backend, heterogeneous LLM training can utilize HetCCL without changing any application code.

6

CCL Baselines: We compare HetCCL with vendor-provided native CCLs (NCCL, V1CCL, V2CCL, and V3CCL) and a hostforwarding approach implemented with Gloo. Specifically, the CPU bounce buffers in Gloo are transferred via RDMA transport in our baseline. Vendor CCL Wrapper Efficiency: Figure 10 compares the algorithm bandwidth of vendor CCLs and the HetCCL wrapper with NCCL-style perf-test of SendRecv, AllGather, and AllReduce operations. The left and right sub-figures represent NCCL and V1CCL, and the solid and dashed lines represent the wrapper and the native CCLs. HetCCL achieves 98% ∼ 99% bandwidth of the native CCLs for arbitrary message sizes, which validates that HetCCL wrapper overhead is negligible. For the rest of the evaluation, we only show the performance of HetCCL wrappers as the vendor CCL performance baselines. 6.1 6.1.1

Benchmark Evaluation Heterogeneous P2P Performance

Figure 11 shows the device-buffer RDMA transport performance of HetCCL. We compare the SendRecv bandwidth of HetCCL with vendor CCLs (for homogeneous peers) and with Gloo (for CPU-forwarding between heterogeneous peers, NVIDIA and Vendor 3). Figure 11 shows that HetCCL achieves > 6× higher bandwidth than the existing heterogeneous communication framework Gloo. We use linear regression to synthesize the latency and bandwidth according to the 𝛼-𝛽 cost model3 for P2P transport. We label the synthesized latency and bandwidth on the horizontal dotted lines in Figure 11. Under asymmetric hardware bandwidth settings, HetCCL achieves 13 ∼ 45% higher bandwidth than the lower homogeneous P2P bandwidth between the two hardware vendors. For symmetric hardware bandwidth settings (using HetCCL’s device-buffer RDMA transport between two NVIDIA GPUs), HetCCL achieves 10.4% higher bandwidth than NCCL SendRecv operation. The synthesized latency cost of HetCCL is 1.2 ∼ 2.4× vendorspecialized libraries (0.05 ∼ 0.18𝑚𝑠 VS. 0.10 ∼ 0.40𝑚𝑠), but still significantly lower than Gloo (1.73𝑚𝑠).

Evaluation

We evaluate HetCCL from the following perspectives: 1. The efficiency of the heterogeneous device-buffer RDMA transport (§6.1.1). HetCCL achieves > 6× bandwidth of Gloo in heterogeneous P2P SendRecv operations. 2. The performance of heterogeneous collective communications (§6.1.2). HetCCL achieves > 97% the performance of homogeneous AllGather and > 70% the performance of homogeneous AllReduce for heterogeneous collective communications. 3. The efficiency of C2C heterogeneous collective algorithm (Algorithm 1) design (§6.1.3). Compared with NCCL, the 2-cluster C2C algorithm achieves 97.4% AllGather bandwidth in a 4-node, 32-GPU cluster. 4. The end-to-end performance gain of HetCCL in heterogeneous training and serving (§6.2). HetCCL accelerates the per-step-time for Llama3-3B and Llama3-8B training by 9.1% and 16.9% (§6.2.1), and reduces TTFT of Qwen-7B serving by > 65% (§6.2.2). Testbed Settings: The evaluation of HetCCL involves 4 out of the 8 supported hardware vendors, including a major hardware vendor, NVIDIA A800 (NV in the figures), and three other minor vendors’ accelerator cards, which we anonymously refer to as Vendor 1 (V1), Vendor 2 (V2) and Vendor 3 (V3), as listed in Table 6. The evaluated hardware architectures include both GPGPUs (General-Purpose computing on Graphics Processing Units) and ASIC (Application-Specific Integrated Circuits).

6.1.2

Collective Communication Benchmarks

Figure 12 and Figure 13 show the collective communication performance of HetCCL for AllGather and AllReduce, which are the most commonly used non-combining and combining collectives in LLM training tasks. Different heterogeneous hardware combinations using HetCCL (NVIDIA 3 The 𝑅 2 (R-squared) value indicates precisions higher than 0.999.

Vendor NVIDIA Vendor 1 Vendor 2 Vendor 3

Hardware A800 (GPGPU) HW-1 (GPGPU) HW-2 (ASIC) HW-3 (GPGPU)

#Node 4 2 2 4

#dev/node 8 16 8 8

RNIC BW 8x200Gbps 1x100Gbps 8x400Gbps 8x400Gbps

Code Y Y N N

Table 6. HetCCL evaluation testbed settings 10

Topo Y N N N

HetCCL

arXiv preprint, 2026 nccl nv hom v1ccl v1 hom v2ccl v2 hom v3ccl v3 hom gloo nv+v3 het hetccl nv+nv hetccl nv+v1 het hetccl nv+v2 het hetccl nv+v3 het

Bandwidth (GB/s)

40 30 20

lat=0.05ms, bw=17.42GB/s lat=0.18ms, bw=8.43GB/s lat=0.17ms, bw=23.88GB/s lat=0.16ms, bw=43.59GB/s lat=1.73ms, bw=3.08GB/s lat=0.10ms, bw=19.23GB/s lat=0.25ms, bw=12.25GB/s lat=0.40ms, bw=20.19GB/s lat=0.19ms, bw=19.28GB/s

10

12 8K 25 6K 51 2K 1M 2M 4M 8M 16 M 32 M 64 M 12 8M 25 6M 51 2M 1G 2G 4G 8G

0

Message Size

50

150 100 50

12 8K 25 6K 51 2K 1M 2M 4M 8M 16 M 32 M 64 M 12 8M 25 6M 51 2M 1G 2G 4G

12 8K 25 6K 51 2K 1M 2M 4M 8M 16 M 32 M 64 M 12 8M 25 6M 51 2M 1G 2G 4G

12 8 25 K 6 51 K 2K 1M 2M 4M 8M 16 M 32 M 64 12 M 8 25 M 6 51 M 2M 1G 2G 4G

0

0

Message Size

160 140 120 100 80 60 40 20 0

4-node native AllGather 2+2 C2C AllGather 4-node native AllReduce 2+2 C2C AllReduce

Bandwidth (GB/s)

100

Bandwidth (GBps)

Bandwidth (GBps)

150

160 140 120 100 80 60 40 20 0

nccl v1ccl v2ccl v3ccl ours nv+v1 ours nv+v2 ours nv+v3 ours v2+v3

200

Bandwidth (GB/s)

nccl v1ccl v2ccl v3ccl ours nv+v1 ours nv+v2 ours nv+v3 ours v2+v3

200

Figure 11. P2P Data Transfer Performance

Message Size

Message Size

#NIC=1 AllGather #NIC=2 AllGather #NIC=4 AllGather #NIC=8 AllGather #NIC=1 AllReduce #NIC=2 AllReduce #NIC=4 AllReduce #NIC=8 AllReduce

12 8 25 K 6 51 K 2K 1M 2M 4M 8M 16 M 32 M 64 12 M 8M 25 6 51 M 2M 1G 2G 4G

Figure 10. HetCCL vendor CCL wrapper performance

Message Size

Figure 12. Heterogeneous Figure 13. Heterogeneous Figure 14. Efficiency of C2C col- Figure 15. Compatibility with AllGather Performance AllReduce Performance lective breakdown multi-NIC hardware

30000

10000 5000

12000

Llama-3B

Model

Llama-8B

0

1500

6000

1000

2000 v3 16

het 8+8

Setup

0

het 16+16 het 32+32

3000 2000

8000 4000

nv 16

Output Total

3500 2500

10000

10000

2500

Mean Median 99P

14000

20000

7500

0

16000

tp2-dp8-pp1 tp1-dp8-pp2 tp2-dp16-pp1 tp1-dp16-pp2 tp2-dp8-pp2 tp1-dp32-pp2 tp2-dp16-pp2

40000

Token Throughput (token/s)

12500

ours nv:8,v1:16 gloo nv:8,v1:16 ours nv:8+8,v1:16+16 gloo nv:8+8,v1:16+16

TTFT (ms)

Per-step Time (ms)

15000

Per-step Time (ms)

17500

nv+nv nccl

nv+nv host

nv+nv ours

nv+v3 host

500 0

nv+v3 ours

nv+nv nccl

nv+nv host

nv+nv ours

nv+v3 host

nv+v3 ours

Figure 16. End-to-end speed up Figure 17. End-to-end training Figure 18. End-to-end serving Figure 19. End-to-end serving of HetCCL over Gloo

scalability of HetCCL

latency performance

A800 with hardware from each of the three minor vendors, plus a combination of two minor vendors, Vendor 2 and Vendor 3) are compared with vendor CCLs in their respective homogeneous environments, all using a 2-node setup. The heterogeneous AllGather in HetCCL achieves 85.7 ∼ 97.8% of the bandwidth of the slower homogeneous AllGather implementation from the two vendors, as the performance of the slower hardware inevitably becomes the bottleneck. For AllReduce, as we trade more intra-cluster communication for compatibility (§4.2.2), the bandwidth can reach up to 70.8% of that of the slower vendor implementation.

59.1% bandwidth of NCCL for AllGather and AllReduce operations, respectively. For the last benchmark, we test the compatibility and scalability of HetCCL to multi-NIC hardware. Figure 15 shows the AllGather and AllReduce performance of HetCCL in our 4-node NVIDIA A800 environment, each node equipped with 8 GPUs per node and using 1/2/4/8 RNICs per node. The collective bandwidth grows proportionally to the number of NICs in use, validating the compatibility and high hardware utilization of HetCCL. 6.2

6.1.3

throughput performance

Algorithm Design Microbenchmarks

6.2.1

To assess the HetCCL collective algorithm design regardless of hardware capability discrepancies, we evaluate the C2C algorithm against NCCL native implementation using 4 NVIDIA server nodes, each containing 8 A800 GPUs. In Figure 14, the 2+2 C2C lines denote the performance of running Algorithm 1 on a hierarchical topology of two 2-node clusters, while the 4-node native lines denote the performance of native NCCL in the same cluster. HetCCL achieves 97.4% and

End-to-end Evaluation LLM Training Performance

Communication Speedup: First, we evaluate the end-toend training speedup of HetCCL compared to the hostforwarding approach. Detailed setups are listed in Appendix B Table 8. Given hardware differences, we adopt asymmetric parallel strategies [32] to match computational capabilities, and reduce 1 PP layer at the start and end nodes for the extra embedding and loss computation. This is the best heterogeneous training strategy we know of for this testbed. 11

arXiv preprint, 2026

Wang et al.

HetCCL’s communication speedup should be effective regardless of parallel strategies, and optimizing heterogeneous parallel training strategies is beyond the scope of this research. Figure 16 shows that HetCCL can improve Llama33B and Llama3-8B training throughput by 9.98% and 20.38%. The performance gain in end-to-end training evaluation is less significant than the benchmark because computation dominates the training time in smaller models and clusters. As communication becomes more dominant in larger models and clusters, we expect HetCCL to yield greater speedup as the training scale grows. Hardware Scalability: Figure 17 demonstrates the training performance comparison of HetCCL and native vendor CCLs in another testbed consisting of NVIDIA and Vendor 3 hardware. HetCCL effectively utilizes heterogeneous computational resources, which were previously unavailable due to the lack of a heterogeneous collective communication library. We train the Llama3-8B model using various parallel strategies in 5 different setups: two homogeneous (two NVIDIA A800 servers, two HW-3 servers) and three heterogeneous (1 A800 + 1 HW-1, 2 A800 + 2 HW-1, 4 A800 + 4 HW-1). Despite hardware heterogeneity, the per-step time using two heterogeneous servers increases by only 7.6% compared to using two homogeneous NVIDIA servers. The training throughput improves by 56% and 64% when using a 4-node heterogeneous cluster compared to 2-node homogeneous NVIDIA or Vendor 3 clusters, respectively. The 8-node heterogeneous cluster additionally improves training throughput by up to 51% compared to the 4-node setup. 6.2.2

throughput by 33%, which is bottlenecked by the hardware processing capability rather than communication. This indicates that coordinating heterogeneous serving strategies with HetCCL communication hides a large portion of hardware differences across vendors, leading to more efficient hardware utilization.

7

Related Works

Heterogeneous Communication Frameworks: Gloo [14] and OpenMPI [7] use the CPU-forwarding mechanism (Figure 2), incurring high data-path overhead compared to our on-device approach. The Unified Collective Communication Library (UCC [43]), built on Unified Communication X (UCX [9, 10, 39]), employs a component-based design with team layers (TLs) (e.g., UCX, CUDA, NCCL) for transport abstraction. However, UCC also relies on CPU-forwarding for cross-device communication, facing the same CPU and PCIe bottlenecks as Gloo and OpenMPI. Collective Algorithm Optimizations: Several works leverage MSCCL [6] to decompose operations into P2P send/recv primitives. TACCL [38] enhances algorithm search with userinput communication sketches, while TCCL [25] focuses on the path-finding and congestion issues in PCIe GPU clusters. TE-CCL [26] models communication as a multi-commodity flow problem with MILP encoding. ForestColl [45] employs a spanning tree-based approach with polynomial solving time but is limited to tree-based algorithms and optimizes only for bandwidth. Note that the heterogeneous support claimed by their mutual execution backend, MSCCL [6], refers to supporting deployment of its algorithms on NVIDIA or AMD GPUs, but each deployment itself has to be homogeneous, i.e., not applicable to a cluster containing both vendors’ hardware, which keeps all these approaches from generalizing to heterogeneous clusters. Portable Programming Models: Programming models such as Triton [41, 46], SYCL [24] and OpenACC [31] are portable across GPUs from multiple vendors. However, they are designed to write portable kernels for computation tasks such as matrix multiplication or convolution, which lack the necessary interfaces such as transport resource management and IB operations, to implement kernel-based communication (e.g., device RDMA without CPU intervention). Heterogeneous Training: Existing works such as Whale [22], Pipette [44], and Metis [42] explore parallel strategies such as tensor parallelism, data parallelism, and pipeline parallelism, relying on collective communication operations like AllReduce, Broadcast, and AllGather to synchronize data across devices. This line of research is orthogonal to our communication optimizations. Regardless of the parallel strategy, heterogeneous training requires high communication efficiency to prevent bottlenecks. HetCCL tackles this challenge, enabling scalable and efficient training across multi-vendor GPU environments.

LLM Serving Performance

Lastly, we evaluate the LLM serving performance of HetCCL, demonstrating another promising use case of mixed-vendor cluster deployment. We adopt the recent prefill-decode disaggregation approach [47] to accommodate the different capabilities of heterogeneous hardware, as the prefill phase is more computation-intensive and the decode phase is more memory-intensive. Detailed evaluation setup is as listed in Appendix B.2 Table 10. We compare the mean, median, and 99-percentile (99P) of time-to-first-token (TTFT) (Figure 18) and end-to-end output and total token throughput of 100 requests (Figure 19) of the Qwen2-7B model serving, where the data transfer across prefill and decode phases is carried out by NCCL (for the NVIDIA homogeneous setup), host-forwarding (for both setups), or HetCCL (treating both setups as heterogeneous). HetCCL reduces 65% TTFT and improves 19% output token throughput compared with the hostforwarding data transfer approach. Compared with NCCL, using HetCCL on two NVIDIA A800 GPUs yields slightly longer TTFT but outperforms NCCL in output and total token throughput. In addition, in the heterogeneous hardware settings, compared with using NCCL for two NVIDIA A800 GPUs, using a combination of NVIDIA and Vendor 3 hardware induces less than 30% longer TTFT and decreases 12

HetCCL

8

arXiv preprint, 2026

Discussion

Device-centric RDMA: HetCCL’s device RDMA depends on a CPU-centric control logic. Implementing a completely device-logic collective faces the following problems, which we seek to solve in future works: 1) Lack in programmability: Device programming libraries (CUDA, ROCm) lack universal abstractions/interfaces to use kernel threads for RDMA control paths. 2) Performance degradation: Extending existing frameworks with RDMA primitives to support entirely kernel-based heterogeneous send/recv usually comes with performance degradation and limitations in the algorithm search space. Intra- and inter-cluster coordinated algorithm design: HetCCL currently relies on vendors to provide high performance homogeneous collective communications. For future improvements, we seek to extend the algorithm encoding and search engine to consider intra-cluster as well as inter-cluster hardware features and topologies to reduce vendor-provided functionality requirements and optimize heterogeneous collective algorithms with a more comprehensive approach. Failure handling: HetCCL currently does not provide additional failure handling mechanisms. The vendor-CCL wrapper component in HetCCL inherits the native failure-handling capability provided in these third-party libraries, and HetCCL further provides NCCL-style logs for the heterogeneous device data P2P transport for failure root cause analysis.

9

Conclusion

We propose HetCCL, a novel heterogeneous collective communication framework, which implements heterogeneous device-buffer RDMA and utilizes homogeneous high performance collective operations to construct efficient heterogeneous collective algorithms in multi-vendor clusters. We implement and evaluate HetCCL with 4 hardware vendors, which achieves over 97% and 70% of the performance of homogeneous AllGather and AllReduce collectives. HetCCL is integrated into PyTorch using the customized backend feature, which accelerates LLM training jobs by up to 16.9%.

Acknowledgments This work was supported by National Natural Science Fund for the Excellent Young Scientists Fund Program (Overseas), and Peking University startup fund. This work was supported in part by the Beijing Major Science and Technology Project under Contract no. Z251100008125042. This work was supported by Beijing Academy of Artificial Intelligence (BAAI). Guyue Liu is the corresponding author.

13

arXiv preprint, 2026

Wang et al.

References

Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. 2024. DeepSeek-V3 Technical Report. (2024). arXiv:cs.CL/2412.19437 https://arxiv.org/abs/2412.19437 [14] Facebook. 2024. Gloo. (2024). https://github.com/facebookincubator/ gloo/. [15] William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39. [16] Graphcore. 2024. Graphcore. (2024). https://www.graphcore.ai/. [17] Khaled Hamidouche, Akshay Venkatesh, Ammar Ahmad Awan, Hari Subramoni, Ching-Hsiang Chu, and Dhabaleswar K Panda. 2015. Exploiting GPUDirect RDMA in designing high performance OpenSHMEM for NVIDIA GPU clusters. In 2015 IEEE International Conference on Cluster Computing. IEEE, 78–87. [18] Seongmin Hong, Seungjae Moon, Junsoo Kim, Sungjae Lee, Minsub Kim, Dongsoo Lee, and Joo-Young Kim. 2022. Dfx: A low-latency multifpga appliance for accelerating transformer-based text generation. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 616–630. [19] Yingbing Huang, Lily Jiaxin Wan, Hanchen Ye, Manvi Jha, Jinghua Wang, Yuhong Li, Xiaofan Zhang, and Deming Chen. 2024. New solutions on LLM acceleration, optimization, and application. In Proceedings of the 61st ACM/IEEE Design Automation Conference. 1–4. [20] Huawei. 2024. Ascend Computing. (2024). https://e.huawei.com/en/ products/computing/ascend. [21] Intel. 2024. OneCCL. (2024). https://www.intel.com/content/www/us/ en/developer/tools/oneapi/oneccl.html. [22] Xianyan Jia, Le Jiang, Ang Wang, Wencong Xiao, Ziji Shi, Jie Zhang, Xinyuan Li, Langshi Chen, Yong Li, Zhen Zheng, et al. 2022. Whale: Efficient giant model training over heterogeneous {GPUs}. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 673–688. [23] Christoforos Kachris. 2025. A survey on hardware accelerators for large language models. Applied Sciences 15, 2 (2025), 586. [24] Khronos. 2024. SYCL. (2024). https://www.khronos.org/sycl/. [25] Heehoon Kim, Junyeol Ryu, and Jaejin Lee. 2024. TCCL: Discovering Better Communication Paths for PCIe GPU Clusters. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 999–1015. [26] Xuting Liu, Behnaz Arzani, Siva Kesava Reddy Kakarla, Liangyu Zhao, Vincent Liu, Miguel Castro, Srikanth Kandula, and Luke Marshall. 2024. Rethinking machine learning collective communication as a multi-commodity flow problem. In Proceedings of the ACM SIGCOMM 2024 Conference. 16–37. [27] Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. (2024). arXiv:cs.AI/2407.21783 https://arxiv.org/abs/2407.21783 [28] Microsoft. 2024. MSCCL. (2024). https://github.com/microsoft/msccl. [29] NVIDIA. 2024. A100. (2024). https://www.nvidia.com/en-us/ data-center/a100/. [30] NVIDIA. 2024. NCCL. (2024). https://developer.nvidia.com/nccl. [31] OpenACC Organization. 2024. OpenACC. (2024). https://www. openacc.org/. [32] Jay H Park, Gyeongchan Yun, M Yi Chang, Nguyen T Nguyen, Seungmin Lee, Jaesik Choi, Sam H Noh, and Young-ri Choi. 2020. {HetPipe}: Enabling large {DNN} training on (whimpy) heterogeneous {GPU} clusters through integration of pipelined model parallelism and data parallelism. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). 307–321.

[1] 2024. OpenMPI. (2024). https://www.open-mpi.org/. [2] AMD. 2024. AMD Instinct Accelerators. (2024). https://www.amd. com/en/products/accelerators/instinct.html. [3] AMD. 2024. RCCL. (2024). https://github.com/ROCm/rccl. [4] AMD. 2024. ROCm. (2024). https://www.amd.com/en/products/ software/rocm.html. [5] Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, et al. 2024. Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware CoDesign for Deep Learning. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–23. [6] Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. 2021. Synthesizing optimal collective algorithms. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 62–75. [7] Chen-Chun Chen, Kawthar Shafie Khorassani, Pouya Kousha, Qinghua Zhou, Jinghan Yao, Hari Subramoni, and Dhabaleswar K Panda. 2023. MPI-xCCL: A Portable MPI Library over Collective Communication Libraries for Various Accelerators. In Proceedings of the SC’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis. 847–854. [8] Hongzheng Chen, Jiahao Zhang, Yixiao Du, Shaojie Xiang, Zichao Yue, Niansong Zhang, Yaohui Cai, and Zhiru Zhang. 2024. Understanding the potential of fpga-based spatial acceleration for large language model inference. ACM Transactions on Reconfigurable Technology and Systems (2024). [9] UCF Consortium. 2024. Unified Communication X Library Source Code https://github.com/openucx/ucx. (2024). [10] UCF Consortium. 2024. Unified Communication X https://openucx. org/. (2024). [11] NVIDIA Corporation. 2026. GPU-Direct RDMA (GDR). https:// developer.nvidia.com/gpudirect. (2026). Accessed: 2026-02-07. [12] NVIDIA Corporation. 2026. NVSHMEM: NVIDIA SHMEM Library. https://developer.nvidia.com/nvshmem. (2026). Accessed: 2026-02-07. [13] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, 14

HetCCL

arXiv preprint, 2026

[33] Sreeram Potluri, Khaled Hamidouche, Akshay Venkatesh, Devendar Bureddy, and Dhabaleswar K Panda. 2013. Efficient inter-node MPI communication using GPUDirect RDMA for InfiniBand clusters with NVIDIA GPUs. In 2013 42nd International Conference on Parallel Processing. IEEE, 80–89. [34] PyTorch. 2024. Custom C++ and CUDA Extensions. (2024). https: //pytorch.org/tutorials/advanced/cpp_extension.html. [35] PyTorch. 2024. Third-party backends. (2024). https://pytorch.org/ docs/stable/distributed.html#third-party-backends. [36] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. (2020). arXiv:cs.LG/1910.02054 https://arxiv.org/abs/1910. 02054 [37] Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, Andrey Bout, Irina Piontkovskaya, Jiansheng Wei, Xin Jiang, Teng Su, Qun Liu, and Jun Yao. 2023. PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing. (2023). arXiv:cs.CL/2303.10845 https://arxiv.org/abs/2303.10845 [38] Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. {TACCL}: Guiding Collective Algorithm Synthesis using Communication Sketches. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 593–612. [39] Pavel Shamis, Manjunath Gorentla Venkata, M. Graham Lopez, Matthew B. Baker, Oscar Hernandez, Yossi Itigin, Mike Dubman, Gilad Shainer, Richard L. Graham, Liran Liss, Yiftah Shahar, Sreeram Potluri, Davide Rossetti, Donald Becker, Duncan Poole, Christopher Lamb, Sameer Kumar, Craig Stunkel, George Bosilca, and Aurelien Bouteiller. 2015. UCX: An Open Source Framework for HPC Network APIs and Beyond. In 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects. [40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. (2020). arXiv:cs.CL/1909.08053 https://arxiv.org/abs/1909.08053 [41] Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. 10–19. [42] Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeongjae Jeon. 2024. Metis: Fast Automatic Distributed Training on Heterogeneous GPUs. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). USENIX Association, Santa Clara, CA, 563–578. https: //www.usenix.org/conference/atc24/presentation/um [43] Manjunath Gorentla Venkata, Valentine Petrov, Sergey Lebedev, Devendar Bureddy, Ferrol Aderholdt, Joshua Ladd, Gil Bloch, Mike Dubman, and Gilad Shainer. 2024. Unified Collective Communication (UCC): An Unified Library for CPU, GPU, and DPU Collectives. In IEEE Symposium on High-Performance Interconnects, HOTI 2024, Albuquerque, NM, USA, August 21-23, 2024. IEEE, 37–46. https: //doi.org/10.1109/HOTI63208.2024.00018 [44] Jinkyu Yim, Jaeyong Song, Yerim Choi, Jaebeen Lee, Jaewon Jung, Hongsun Jang, and Jinho Lee. 2024. Pipette: Automatic Fine-Grained Large Language Model Training Configurator for Real-World Clusters. In 2024 Design, Automation and Test in Europe Conference and Exhibition, DATE 2024 - Proceedings (Proceedings -Design, Automation and Test in Europe, DATE). Institute of Electrical and Electronics Engineers Inc., United States. Publisher Copyright: © 2024 EDAA.; 2024 Design, Automation and Test in Europe Conference and Exhibition, DATE 2024 ; Conference date: 25-03-2024 Through 27-03-2024. [45] Liangyu Zhao, Saeed Maleki, Aashaka Shah, Ziyue Yang, Hossein Pourreza, and Arvind Krishnamurthy. 2024. Forestcoll: Efficient collective

communications on heterogeneous network fabrics. arXiv preprint arXiv:2402.06787 (2024). [46] Size Zheng, Wenlei Bao, Qi Hou, Xuegui Zheng, Jin Fang, Chenhui Huang, Tianqi Li, Haojie Duanmu, Renze Chen, Ruifan Xu, et al. 2025. Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler. arXiv preprint arXiv:2504.19442 (2025). [47] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210.

15

arXiv preprint, 2026

Wang et al. Border rank

Internal rank

Input

Empty buffer

Cluster 0

Cluster 0

rank 2 (internal) 2

rank 3 (internal) 3

rank 0 (border)

rank 1 (border)

0

rank 2 (internal) 0 1 2 3 AllGather

1

0

rank 3 (internal) 0 1 2 3

rank 0 (border) 1 2 3

0

rank 1 (border) 1 2 3

a) start coll Cluster 0 rank 0 (border) 0 1 2 3

Cluster 0 rank 1 (border) 0 1 2 3

Send/Recv

Cluster 1 rank 4 (border) 4

5

6

rank 0 (border) 0 1 2 3 4 5

rank 5 (border) 7

0 4

1 5

2 6

0

Cluster 1 rank 4 (border) 0 1 4 5 6 7

3 7

rank 1 (border) 1 2 3 2 6 3 7

rank 5 (border) 2 3 0 4 1 5 2 6 3 7

b) c2cCpy Cluster 0

Cluster 0

rank 2 (internal) 0 1 2 3

rank 3 (internal) 0 1 2 3

rank 0 (border) 0 1 2 3 4 5

rank 1 (border) 0 1 2 3 2 6 3 7

rank 2 (internal) 0 1 2 3 4 5 6 7 Broadcast 0 4

rank 0 (border) 1 2 3 5 6 7

rank 3 (internal) 0 1 2 3 4 5 6 7

0 4

rank 1 (border) 1 2 3 5 6 7

c) end coll

Figure 20. Collective breakdown of AllGather

A

Hierarchical Algorithm for Heterogeneous Collectives

In this section, we present the detailed collective breakdown logic of AllGather and AllReduce in Figure 20 and 21. We demonstrate the operations in a heterogeneous environment with two types of hardware (divided into Cluster 0 and Cluster 1, respectively), each with 4 devices (4 ranks) and 2 RNICs (2 border ranks). The precise semantics of AllGather and AllReduce Heterogeneous Collective AllReduceH AllGatherH ReduceScatterH BcastH ReduceH GatherH ScatterH AllToAllH

start ℎ𝑜𝑚𝐶𝑜𝑙𝑙

C2C primitive

Reduce / ReduceScatter AllGather Reduce Bcast Reduce Gather Scatter AllToAll / sendrecv

b1) c2cRed: ReduceScatter b2) c2cCpy c2cCpy c2cRed: ReduceScatter c2cCpy c2cRed: ReduceScatter c2cCpy c2cCpy c2cCpy

C2C total send volume

C2C total recv volume

end ℎ𝑜𝑚𝐶𝑜𝑙𝑙

≈ 2 ∗ 𝑛 ∗ (𝐶 − 1)/𝐶

≈ 2 ∗ 𝑛 ∗ (𝐶 − 1)/𝐶

Bcast / No-op

≈𝐺 ∗𝑛 (𝐺 − 𝑁 ) ∗ 𝑛 𝑛 (root) 𝑛 (non-root) 𝑁 ∗ 𝑛 (non-root) (𝐺 − 𝑁 ) ∗ 𝑛 (root) (𝐺 − 𝑁 ) ∗ 𝑛

≈𝐺 ∗𝑛 (𝐶 − 1) ∗ 𝑁 ∗ 𝑛 𝑛 (non-root) 𝑛 (root) (𝐺 − 𝑁 ) ∗ 𝑛 (root) 𝑁 ∗ 𝑛 (non-root) (𝐺 − 𝑁 ) ∗ 𝑛

Bcast ReduceScatter Bcast Reduce Gather Scatter SendRecv/ Scatter

We use 𝐶 for the number of clusters and 𝐺 for the total number of ranks in the heterogeneous cluster topology, 𝑁 for the number of ranks in the current cluster, 𝐵 for the number of border ranks, and 𝑛 for the data count of the send buffer of the collective operation. Table 7. Cluster-level Primitive Breakdown of Heterogeneous Collectives 16

HetCCL

arXiv preprint, 2026

operations are constructed with our cluster-level primitives (§4.2.2). The start, c2c, and end operations of Algorithm 1 for other collective operations are as listed in Table 4. Cluster 0 rank 2 (internal) 2 2 2 2 2 2 2 2

0 0

Cluster 0 rank 3 (internal) 3 3 3 3 3 3 3 3

rank 0 (border) 0 0 0 0 0 0

1 1

rank 1 (border) 1 1 1 1 1 1

rank 2 (internal) 0 0 0 0 0 0 0 0 Reduce sum(0..3)

6 6

rank 3 (internal) 0 0 0 0 0 0 0 0

rank 0 (border) 6 0 0 6 0 0

0 0

rank 1 (border) 0 6 6 0 6 6

a) start coll

Cluster 0 rank 0 (border) 6 6 0 0 6 6 0 0

Cluster 0 rank 1 (border) 0 0 6 6 0 0 6 6

rank 0 (border) 6 6 22 22 0 0 0 0 Send/Recv

6=sum(0..3) 22=sum(4..7)

Reduce sum(0..3)+sum(4..7)

Cluster 1 rank 4 (border) 22 22 0 0 22 22 0 0

rank 1 (border) 22 22 6 6 0 0 0 0

Cluster 0 rank 5 (border) 0 0 22 22 0 0 22 22

rank 0 (border) 28 28 0 0 0 0 0 0

0 0

rank 1 (border) 0 28 28 0 0 0

0 0

rank 1 (border) 0 28 28 0 28 28

0 0

rank 5 (border) 0 28 28 0 28 28

b.1) c2cRed

Cluster 0 rank 0 (border) 28 28 0 0 0 0 0 0

Cluster 0 rank 1 (border) 0 0 28 28 0 0 0 0

Send/Recv

Cluster 1 rank 4 (border) 0 0 0 0 28 28 0 0

rank 0 (border) 28 28 0 0 28 28 0 0

0 0

rank 5 (border) 0 0 0 0 28 28

Cluster 1 rank 4 (border) 28 28 0 0 28 28 0 0

b.2) c2cCpy Cluster 0

Cluster 0

rank 2 (internal) 0 0 0 0 0 0 0 0

rank 3 (internal) 0 0 0 0 0 0 0 0

rank 0 (border) 28 28 0 0 28 28 0 0

0 0

rank 1 (border) 0 28 28 0 28 28

Broadcast

rank 2 (internal) 28 28 28 28 28 28 28 28

rank 3 (internal) 28 28 28 28 28 28 28 28

rank 0 (border) 28 28 28 28 28 28 28 28

rank 1 (border) 28 28 28 28 28 28 28 28

c) end coll

Figure 21. Collective breakdown of AllReduce

17

arXiv preprint, 2026

Wang et al.

LLM #Layers Global Batchsize Server Model Accelerators Parallel Strategy CCL

Setup 1 Setup 2 Llama3-3B Llama3-8B 36 32 128 256 1 × A800 + 1 × HW-1 2 × A800 + 2 × HW-1 NV:8,V1:16 NV:8 × 2,V1:16 × 2 PP: (NV:26,V1:10), DP: (NV:8,V1:16) PP: (NV:10, 11,V1:6, 5), DP: (NV:8,V1:16) Gloo / HetCCL Table 8. End-to-end training performance evaluation setup

Setup 3 LLM #Layers Global Batchsize Server Model Accelerators Parallel Strategy CCL

Setup 4

Setup 5 Llama3-8B 32 512 1 × A800 + 1 × HW-3 NV:8,V3:8 TP:2,DP:8,PP:1 TP:1,DP:8,PP:2 -

2 × A800 NV:8 × 2 TP:2,DP:8,PP:1 TP:1,DP:8,PP:2 NCCL

Setup 6

2 × HW-3 2 × A800 + 2 × HW-3 V3:8 × 2 NV:8 × 2,V3:8 × 2 TP:2,DP:8,PP:1 TP:2,DP:16,PP:1 TP:1,DP:8,PP:2 TP:1,DP:16,PP:2 TP:2,DP:8,PP:2 V3CCL HetCCL Table 9. End-to-end training scalability evaluation setup

B

End-to-end Evaluation Setups

B.1

Training Evaluation Setups

Setup 7

4 × A800 + 4 × HW-3 NV:8 × 4,V3:8 × 4 TP:1,DP:32,PP:2 TP:2,DP:16,PP:2

Communication Speedup: Table 8 lists the evaluation setups for communication comparison between HetCCL and Gloo. In Setup 1, we train a Llama3-3B model in a mixed-vendor environment with 1 NVIDIA 8-GPU server and 1 Vendor 1 16-GPU server. Setup 2 trains a Llama3-8B model in a mixed-vendor environment with 2 NVIDIA 8-GPU servers and 2 Vendor 1 16-GPU servers. We adopt pipeline parallelism across the two vendor hardware groups, with the pipeline layers divided according to each hardware’s computation capability. Then we adopt data parallelism within the group (DP equals the number of GPUs per PP group). The same model and parallel strategies are executed with HetCCL and Gloo as communication backends, respectively, to evaluate the end-to-end impact of communication efficiency. Hardware Scalability: Table 9 lists the setups for hardware scalability evaluation of HetCCL. We train the same Llama3-8B model in both homogeneous and heterogeneous clusters of various scales. For NVIDIA and Vendor 3 homogeneous clusters, we train with their native CCLs, respectively, and for heterogeneous setups, we train with HetCCL. For 2-node homogeneous and heterogeneous setups (Setup 3, 4, and 5), we test two different parallel strategies, namely [𝑇 𝑃, 𝐷𝑃, 𝑃𝑃] = [2, 8, 1] and [𝑇 𝑃, 𝐷𝑃, 𝑃𝑃] = [1, 8, 2]. For 4-node and 8-node heterogeneous setups (Setup 6 and 7), we extend the previous parallel strategies with 𝐷𝑃 groups or 𝑃𝑃 groups to evaluate HetCCL’s ability to utilize heterogeneous computation resources. B.2

Serving Evaluation Setups Setup 1 Setup 2 Qwen2-7B Qwen2-7B 2 × A800 1 × A800 + 1 × HW-3 NV:2 NV:1,V3:1 NV:1 NV:1 NV:1 V3:1 NCCL CCL Host Host HetCCL HetCCL Table 10. LLM serving evaluation setup.

LLM Server Accelerators Prefill Decode

Table 10 lists the end-to-end evaluation setups for LLM serving scenario. In both setups, we put the prefill phase on NVIDIA hardware and the decode phase on the other hardware. The Qwen2-7B serving is implemented with vllm, adapted to transfer the KV cache between prefill and decode phase with various communication backends (NCCL, host-forwarding and HetCCL). 18