arXiv:2605.12396v1 [cs.DC] 12 May 2026
NCCLZ: Compression-Enabled GPU Collectives with Decoupled Quantization and Entropy Coding 1st Jiamin Wang
2nd Zhijing Ye
3rd Xiaodong Yu
Department of Computer Science Stevens Institute of Technology Hoboken, USA [email protected]
Department of Computer Science Stevens Institute of Technology Hoboken, USA [email protected]
Department of Computer Science Stevens Institute of Technology Hoboken, USA [email protected]
Abstract—Collective communication is a major bottleneck for multi-node GPU workloads in scientific computing and distributed deep learning, especially when inter-node bandwidth is limited. Although NCCL provides optimized GPU-centric collectives, large messages can still dominate end-to-end performance. Existing compression-enabled collective libraries either rely on MPI-based stacks that cannot fully exploit NCCL, omit entropy coding, or tightly couple full compressors with communication primitives, limiting compression ratio, flexibility, and communication-computation overlap. This paper presents NCCLZ, a compression-enabled GPU collectives that decouples quantization and entropy coding and integrates them at different layers of the stack. NCCLZ places quantization at the interface, embeds entropy coding into NCCL primitives, uses a lightweight device-side selector to choose coding strategies, and overlaps compression with communication to reduce exposed overhead. Experiments on scientific datasets, training gradients, and synthetic workloads show up to 9.65× speedup over NCCL and up to 3.34× improvement over prior compression-assisted collective libraries. Index Terms—NCCL, collective communication, compression
I. I NTRODUCTION Modern HPC clusters are increasingly dominated by NVIDIA GPU-based systems, where both scientific applications and distributed deep learning rely on multi-GPU execution across nodes. In such environments, performance is often limited by inter-GPU communication, especially in internode settings with constrained bandwidth. For example, dataparallel training for AI workloads requires gradient synchronization via collectives such as AllReduce [1] for each iteration. As GPUs process more batches, synchronization becomes more frequent and message sizes grow, making communication a dominant bottleneck at scale [2], [3]. NVIDIA provides NCCL, a vendor-optimized collective library with GPU-native abstractions and topology-aware optimizations for NVLink. Compared to GPU-aware MPI, NCCL better exploits NVIDIA hardware and has therefore become the de facto backend for multi-GPU systems. However, despite near-device-level intranode performance, NCCL remains bandwidth-limited for large messages in inter-node settings. Lossy compression is widely used in both scientific and AI workloads to reduce communication message size. State-ofthe-art scientific compressors combine lossy quantization with lossless entropy coding to maximize compression ratios [4],
[5]. In contrast, modern AI workloads, especially LLMs, rely on quantization alone, omitting entropy coding due to its high computational cost [6]–[8]. In principle, entropy coding can further improve compression ratios for LLMs, as in scientific compressors. However, its computational overhead often outweighs these gains, leading to negative overall performance. If this cost can be effectively hidden by communication, entropy coding can regain its benefits for AI workloads. Several recent works have explored integrating lossy compression into collective communication libraries (CCLs) to alleviate network bottlenecks while hiding compression overhead. MVAPICH2 [9] extends MPI collectives with on-thefly compression, demonstrating early benefits. gZCCL [10] and ghZCCL [11] further optimize GPU-aware MPI collectives with GPU-based lossy and homomorphic compressors, respectively, using overlap of compression and communication to improve scalability. More recently, COCCL [12] targets NVIDIA GPUs by integrating lightweight quantization into NCCL APIs, providing compression-accelerated collectives with vendor-optimized communications. Although these works demonstrate performance gains for specific workloads, they do not effectively address entropycoding challenges, especially for AI workloads. They either omit entropy coding (e.g., COCCL integrates only LLM quantization) or embed it within full scientific compressors (e.g., MVAPICH2 and gZCCL). Moreover, their integration strategies limit extensibility. COCCL makes it difficult to incorporate entropy coding or switch to scientific compressors, thus benefiting AI workloads only from quantization while missing potential gains from entropy coding. In contrast, MPIbased approaches tightly couple MPI with specific scientific compressors, leading to suboptimal performance for AI workloads and potential error propagation during communication. These limitations are further discussed in Sec. III and motivate rethinking how compression should be integrated into modern GPU collective communication stacks. To address these challenges, we propose a new integration strategy for compression-enabled collectives in NCCL. The key idea is to decouple compression into quantization and entropy coding and integrate them at different layers: quantization at the interface layer and entropy coding within NCCL communication primitives. This design enables flexible
composition of compression techniques across communication workloads, maximizes compression–communication overlap by isolating entropy-coding overhead, and avoids error propagation in reduction-heavy collectives by keeping only lossless components in the communication path. It also preserves NCCL’s native execution workflow and programming semantics without modifying transport layers. Building on our integration strategy, we develop NCCLZ , a compression-enabled NCCL-based framework for bandwidthconstrained NVIDIA GPU clusters. NCCLZ decouples compression into external quantization at the interface layer and entropy coding within NCCL primitives, preserving NCCL’s native workflow without modifying APIs. This design supports flexible quantization for both scientific and AI workloads, integrates multiple GPU-optimized entropy coders with a novel lightweight device-side arbitrator to adaptively select the most cost-effective encoding strategy (including disabling entropy coding) at runtime. NCCLZ also introduces an overlap-aware mechanism that leverages NCCL protocols to hide entropycoding overhead. We evaluate NCCLZ on real scientific datasets, distributed training gradients, and synthetic workloads across multiple node scales, demonstrating consistently higher compression ratios and significantly improved end-toend throughput over NCCL and prior compression-assisted CCLs. The contributions are summarized as follows: • We propose a new integration strategy for compressionenabled NCCL that decouples quantization and entropy coding across stack layers. This design improves flexibility, enables efficient overlap, avoids error propagation, and preserves NCCL semantics. • We develop NCCLZ following our integration strategy, which integrates quantization at the interface layer and entropy coding with communication primitives, enabling efficient compression-accelerated GPU collectives. • We design a novel lightweight device-side arbitrator for adaptive entropy coding and an efficient compressioncommunication overlap mechanism that exploits NCCL design patterns to optimize end-to-end performance. • We conduct extensive evaluations on scientific datasets, distributed deep learning workloads, and synthetic benchmarks across multiple node configurations, achieving up to 9.65× speedup over NCCL and 3.34× over existing compression-assisted CCLs. II. BACKGROUND This section provides background on NCCL and data compression. We first discuss NCCL, and then we briefly summarize data compression.
Fig. 1. System architecture of NCCLZ, with the interaction between application workloads, the NCCL runtime, GPU-resident entropy coding, and the underlying transport and network layers.
and AllGather; a collective operates on a group of GPU ranks, and AllReduce aggregates values across ranks and returns the result to every rank [19]. Internally, NCCL decomposes collectives into low-level communication primitives (e.g., send, recv, recvReduceSend) that serve as building blocks for ring/tree algorithms over different transports and protocols [19]. A NCCL program creates a communicator for participating GPU ranks and enqueues collective operations onto CUDA streams. To amortize launch overhead, NCCL can group multiple calls and submit them as a batch; runtime performance is then shaped by channel-level parallelism together with protocol and topology efficiency [19]. For large tensors, NCCL partitions data across multiple communication channels so disjoint chunks can be processed concurrently. Each channel uses a fixed-size buffer divided into NCCL_STEPS slots for pipelined production, transmission, and consumption [19]. During communicator initialization, NCCL builds logical topologies such as ring and tree and selects a protocol to balance latency and bandwidth, including LL, Simple, and, when supported, LL128 [19]. NCCL uses different data paths for intra-node and internode communication [19]. Within a node, it prioritizes GPUdirect peer-to-peer transfers over NVLink/PCIe; on NVLink, the high scale-up bandwidth often reduces the benefit of compression [20]. Across nodes, NCCL uses network transports such as sockets or RDMA, typically with a host-side proxy thread to drive NIC operations for GPU transfers [19]. In the common baseline path, data is staged through pinned host memory, making performance sensitive to PCIe, CPU progress, and network bandwidth.
A. NCCL NCCL (NVIDIA Collective Communications Library) is a widely used GPU communication library for distributed deep learning and other multi-GPU workloads. It is integrated into mainstream frameworks and systems to execute high-throughput collectives [1], [13]–[18]. NCCL provides collectives such as AllReduce, Broadcast, AlltoAll,
B. Lossy Compression Lossy compression reduces communication volume by encoding numerical tensors into fewer bits at the cost of fidelity loss and extra computation. In distributed training, error-bounded lossy compression is widely used to improve throughput while preserving convergence through strict dis-
tortion guarantees [21]. Representative compressors such as SZ [22] and ZFP [23] follow a four-stage pipeline: (1) prediction/transform, (2) quantization, (3) entropy coding, and (4) lossless backend. Among them, quantization and entropy coding largely determine the compression–accuracy trade-off and communication efficiency. Quantization. Quantization maps values to a finite set of levels to reduce representation cost. In error-bounded compressors, it is typically deterministic, assigning values to uniformly spaced bins based on user-specified error bounds and thereby ensuring strict error control. By contrast, QSGD [24] uses stochastic quantization, applying randomized rounding with probabilities proportional to distance, yielding unbiased estimates while helping preserve convergence. Entropy coding. Entropy coding encodes quantized symbols more compactly according to their statistical distribution, at the cost of extra computation. Its coding efficiency can vary significantly across different data distributions. Fixed-length coding, as used in cuSZp [25], assigns the same number of bits to each symbol, minimizing coding complexity and favoring parallel execution, especially on GPUs. It is effective when the symbol distribution is close to uniform. By contrast, Huffman coding, used in compressors such as SZ [22], assigns shorter codes to more frequent symbols, achieving better compression for skewed distributions at the cost of codebook overhead and lower parallel efficiency. III. P ROBLEM A NALYSIS AND D ESIGN M OTIVATION Integration strategies for incorporating compressors into CCLs are central to compression-enabled communication frameworks and largely determine their efficiency. COCCL [12] integrates only LLM-oriented quantizers into NCCL by by re-wrapping NCCL calls at the API layer and overlapping lightweight quantization with communication using multiple CUDA streams [26]. In contrast, MPI-based frameworks, such as MVAPICH2 [27], gZCCL [10], and ghZCCL [11], tightly integrate whole scientific compressors into the MPI communication path and overlap their computation with MPI communication primitives. However, these integration strategies exhibit key limitations that constrain performance, particularly when supporting both scientific and AI workloads and enabling entropy coding on demand, as discussed below. Lack of flexibility. Existing integration designs lack flexibility in switching between optimal compressors for scientific and AI workloads, as well as in selecting entropy coding schemes. COCCL supports multiple quantization methods for LLMs (e.g., SDP4Bit and minmaxUint8) [26], providing some flexibility for AI workloads. However, it does not support the flexible integration of entropy coding, preventing it from leveraging its benefits. Moreover, due to this limitation, COCCL cannot easily replace quantization with scientific compressors, resulting in suboptimal performance on scientific workloads. In contrast, MPI-based approaches tightly integrate full scientific compressors into MPI, making it difficult to substitute
AI-oriented quantization or flexibly enable or disable entropy coding, leading to suboptimal performance for AI workloads. Inefficient overlapping. Entropy coding is the most computationally expensive component in lossy compression and thus the most critical to overlap with communication. However, existing strategies overlap quantization (or even the entire compression pipeline) with communication, leaving insufficient opportunity to hide entropy coding cost. This issue is more pronounced in NCCL-based frameworks, where communication is faster than MPI due to vendor-optimized GPU collectives, further shrinking the available overlap window. Consequently, overlapping schemes effective in MPI do not necessarily translate to NCCL. Since NCCL is the de facto backend for modern multi-GPU and multi-node training [19], such designs are inherently less effective for AI workloads. Error propagation in reduction collectives. Existing integration strategies suffer from error propagation in reductionheavy collectives (e.g., AllReduce) [10], [28]. During these operations, intermediate reduction results are repeatedly decompressed and recompressed. Because quantization is lossy and tightly coupled with communication paths, repeated dequantization and requantization accumulate distortion, degrading workload utility and accuracy. Although ghZCCL avoids this issue via homomorphic compression, this benefit stems from the compressor itself rather than the integration strategy, and such compressors are not efficient for all workloads. In contrast, entropy coding is lossless and introduces no error propagation even when coupled with communication paths. These limitations motivate us to design a new integration strategy that enables seamless switching between scientific and AI compressors with on-demand entropy coding, maximizes compression–communication overlap, and eliminates error propagation, as introduced in Sec. IV-A. IV. NCCLZ D ESIGN AND I MPLEMENTATION In this section, we present the design of NCCLZ , an opt-in compression-aware extension to NCCL that targets bandwidthdominated inter-node communication. A. Integration Strategy Design We design a new integration strategy to address the limitations of existing approaches (Sec. III). The key idea is to decouple compression into two stages and integrate them at different layers. Quantization is performed once at the interface boundary before NCCL communication primitives are invoked. The lossless entropy-coding stage is then integrated into NCCL communication primitives to further compress the symbol stream during transmission when necessary. This separation makes the approximation boundary explicit, keeps numerical semantics outside the collective runtime, and aligns in-runtime processing with NCCL’s native data path. This design directly addresses the limitations in Sec. III. First, decoupling quantization and entropy coding avoids the rigidity of both MPI-based monolithic compressors and quantization-only NCCL-based designs. The interface layer can flexibly select quantizers for scientific and AI workloads,
while the runtime applies entropy coding when quantization alone is insufficient. Second, with only the lossless stage inside NCCL’s native communication path, in-path overhead is reduced and easier to overlap with communication, while preserving vendor-optimized execution and native grouping semantics. Third, keeping quantization outside NCCL communication primitives prevents repeated lossy recompression of intermediate reduction results, avoiding error propagation. B. NCCLZ Overview We leverage our integration strategy to design and implement NCCLZ , a new NCCL-based framework with decoupled interface-level quantization and in-NCCL entropy coding. At a high level, NCCLZ is organized around three components that directly reflect our integration design: an outside quantizer that performs the one-time approximation before data enter NCCL, a device-side Runtime Entropy Arbitration (REA) module that selects among R AW, fixed-length, and GPU Huffman paths under runtime conditions (Sec. IV-C), and an overlap-aware in-NCCL execution mechanism that pipelines encode/decode with NCCL communication and fused reduction progress to hide entropy-coding overhead (Sec. IV-D4). Figure 2 summarizes how these three components are arranged along the NCCL invocation path. As shown in Figure 2, NCCLZ first intercepts input, and if the input is non-quantized, the external quantizer converts it into an integer symbol stream and only introduce one time approximation. This is to make sure approximation is explicitly controlled before communication and does not compound inside the communication runtime across iterations. For the external quantizer, we instantiate two quantization modes: (1) for gradients, we use QSGD-style stochastic quantization [24], (2) for non-gradient input, we use a deterministic error-bounded quantizer with error bound (e.g., SZ/ZFP-style bounded quantization) [22], [23]. The quantizer outputs: (i) the integer symbol stream, and (ii) minimal side metadata (e.g., scale parameters) required for dequantization. This metadata is carried either in the batch header or in a small companion buffer that is transmitted alongside the payload, ensuring receivers can reconstruct the quantized values correctly. By design, the entropy coding stage is lossless, thus the only approximation comes from quantization. Next, the device-side Runtime Entropy Arbitration runs on a small sample window, up to 64 KiB, and uses constant-time gating (e.g., size threshold and Huffman context validity) to decide whether to evaluate fixed-length packing and/or GPU Huffman. It estimates the expected compressed payload size, applies a minimum-gain threshold, and performs strict capacity checks. On success, it materializes a self-describing frame in a staging buffer, with Frame = Header + Payload, where the header records codecId, rawBytes, payloadBytes, and codec-specific parameters. When compression is not beneficial, or when staging space is insufficient, NCCLZ falls back safely to R AW semantics. In the last step, the in-NCCL integration attaches encodeon-send and decode-on-recv at NCCL primitive boundaries,
so compressed bytes still follow NCCL’s native ordering and progress semantics. To make entropy coding effective in this bandwidth-bound regime, NCCLZ then fixes the framing granularity to an 8-slot batch. Each frame spans exactly eight slots (raw upper bound ≈ NCCL_BUFFSIZE, 4 MiB by default), and its compressed payload is laid out contiguously across these slots to form a larger and more stable coding window. Moreover, to avoid stalling the pipeline, NCCLZ uses two ping-pong staging banks to overlap work across consecutive batches. While batch i is in flight on the NET path, batch i+1 can be encoded into the alternate bank. On the receive side, the incoming frame is validated and decoded on a per-frame basis, with a robust R AW fallback when validation or decoding fails. Finally, for AllReduce, NCCLZ further leverages the fused recvReduceSend primitive to stream recv→decode→reduce→encode→send within each step, so neighboring chunks overlap in time and entropy coding overhead is largely hidden behind the reduction pipeline. C. Runtime Entropy Arbitration NCCLZ uses a device-side runtime entropy arbitration (REA) module to choose whether the quantized symbol stream is sent as R AW, encoded with F IXED L EN, or encoded with GPU H UFFMAN. The choice is not fixed; it depends on message size, sample compressibility, and the effective transport regime. This is important because NCCLZ targets the bandwidth-dominated inter-node regime, where reducing transmitted bytes is often the main performance lever. When communication remains within a node and NCCL can exploit high-bandwidth GPU-resident paths such as NVLink or other direct P2P interconnects, however, the marginal gain from entropy coding often does not amortize its encode/decode cost. In that regime, REA naturally selects R AW, which is the optimal outcome rather than a conservative fallback. Arbitration mechanism. REA formulates codec selection as a runtime break-even decision rather than a fixed rule chain. REA determines the most beneficial transmission route under the current message size, sample compressibility, and transport regime. Concretely, it decides whether the current input should be emitted directly as R AW, or encoded with an entropy codec to reduce transmitted bytes at the cost of additional GPU work. The role of arbitration is therefore not merely to rank codecs by predicted compressed size, but to identify which side of the communication–compression boundary the current input lies on. When R AW is preferred. The R AW path is preferred when the communication path already provides sufficiently high effective bandwidth, so shrinking the payload no longer amortizes the exposed encode/decode overhead. This commonly occurs in intra-node GPU communication, where NCCL can exploit high-bandwidth paths such as NVLink or other direct GPU-resident P2P transports. In such regimes, transmission contributes less to end-to-end time, while codec execution remains on the critical path. As a result, even a nontrivial byte reduction may not improve runtime, and REA naturally selects R AW. In this sense, R AW is not a fallback caused by failed
Device-side Selector
Pre - Quantizer Module QSGD
In-NCCL Entropy Coding
LL/ LL128: small payload, low-latency, compression less effective
constant-time gating
Symbols (int)
SZ/ZFP
Sample <= 64 KiB
Frame = Header + Payload Selector Core
aligned
encode on-send
Codec Dispatcher
Netwrok Path via Simple
Metadata tag GPU Huffman
NCCL Validator
GPU Huffman
Fixed length
Input tensors (floating point / non-quantized)
Validate & Decode
Fixed-length
Frame Builder codec id
raw bytes
Decode on Recv
payload params bytes
Dequantization
Fig. 2. Overview of NCCLZ’s layered design.
compression, but the optimal decision in the high-bandwidth region of the break-even surface. When entropy coding is preferred. Entropy coding is preferred in the bandwidth-dominated regime targeted by NCCLZ , especially for inter-node transfers where reducing injected bytes is often the dominant lever for improving endto-end time. REA first profiles a bounded sample window and derives lightweight compressibility statistics to estimate the payload reduction achievable by each codec. For F IXED L EN, the estimate is driven by the effective symbol width implied by the sample. For GPU H UFFMAN, it is driven by the skewness of the sampled symbol distribution under the cached codebook. Intuitively, F IXED L EN is favored when the symbol range is narrow and low-overhead packing is sufficient, whereas H UFFMAN is favored only when additional distribution skew indicates that its higher compression ratio can amortize its extra codec cost. Huffman is also gated by implementation constraints, including a minimum message-size threshold and the availability of a valid Huffman context. To make this trade-off explicit, REA evaluates the exposed critical-path time of each admissible choice. Let B denote the raw payload size, Pbc the predicted transmitted size under codec c, and βbeff (π) the calibrated effective bandwidth associated with transport hint π. REA uses the following decision model: Pbc b b Tbc = αc + λenc + λdec (1) c Ec + c Dc , b βeff (π) bc and D b c denote the estimated encode and decode where E enc costs, and λc , λdec ∈ [0, 1] represent the fractions of these c costs that remain exposed after overlap. For R AW, PbR = B and codec costs are negligible; for entropy codecs, Pbc is predicted from the bounded sample together with codecspecific metadata overheads. REA admits only entropy candidates that satisfy three conditions: the predicted payload fits within the staging capacity, the predicted compression gain exceeds the minimum-gain threshold, and the codec-specific enable conditions hold. It then selects c⋆ = arg min Tbc , (2) c∈{R}∪Cadm
where Cadm is the set of admissible entropy codecs. If no
entropy codec survives these checks, or if the realized output after materialization fails capacity or gain validation, REA commits a safe R AW frame. This makes the arbitration boundary explicit while keeping the mathematical layer subordinate to the runtime decision logic. Arbitration procedure. Algorithm 1 instantiates the mechanism above in four stages. It takes as input a raw byte stream raw[0..rawBytes), a staging destination outHdrBase with total capacity stageCapBytes, and codec-side controls including a pre-initialized Huffman context huffCtx, the Huffman enable threshold Thuff , and the minimum-gain requirement minGainPermil. Lines 1–4 first reject degenerate inputs and derive the available payload budget, after which Lines 5–7 provide a fast path that directly commits a R AW frame when arbitration is unnecessary. If the input does not take this path, Lines 8–11 enter the planning stage, where REA derives a transport hint, profiles a bounded sample, and invokes A RBITRATE P LAN to select the most beneficial transmission route under the current message size, sample compressibility, transport regime, and codec constraints. Lines 12–17 then materialize the selected non-R AW plan and accept it only if the realized output passes the post-checks in ACCEPT R EALIZED. Finally, Line 18 provides a unified fallback path through C OMMIT R AW O R FAIL, which commits a safe R AW frame when the unencoded payload still fits in the staging region and otherwise returns failure. Overall, the procedure keeps control flow explicit through early guards, planning, selective materialization, and safe fallback. Arbitration overhead. REA preserves the low-overhead property of the current implementation. Control gating is O(1) and consists of a small number of integer checks, including the small-batch fast path, the Huffman enable threshold, and context-validity gating. Sampling touches at most 64 KiB per 8-slot batch, estimation is executed once per batch by a single control thread, and the planning phase never touches the full payload. The computational work of planning is limited to one max-|x| scan for F IXED L EN and one 256-bin histogram plus expected-code-length accumulation for H UFFMAN. Fullframe encoding is attempted only after a single codec plan has already been selected by the arbitration model. Therefore, REA keeps the decision path lightweight while making the
Input: raw[0..rawBytes), outHdrBase, stageCapBytes, huffCtx, Thuff , minGainPermil Output: return (codecId, payloadBytes, totalBytes) and write a frame at outHdrBase on success 1: if rawBytes ≤ 0 or stageCapBytes ≤ HdrBytes then 2: return (0,0,0) 3: end if 4: payloadCap ← stageCapBytes − HdrBytes 5: if S MALL BATCH FAST PATH(rawBytes) then 6: return C OMMIT R AW O R FAIL(raw, rawBytes, outHdrBase, payloadCap) 7: end if 8: hint ← T RANSPORT H INT 9: sampleBytes ← S AMPLE W INDOW(rawBytes) 10: stats ← P ROFILE S AMPLE(raw[0:sampleBytes], huffCtx) 11: plan ← A RBITRATE P LAN(rawBytes, payloadCap, stats, hint, huffCtx, Thuff , minGainPermil) 12: if plan.codecId ̸= R AW then 13: P ← E NCODE P LAN(plan, raw, rawBytes, outHdrBase, payloadCap, huffCtx) 14: if ACCEPT R EALIZED(plan.codecId, P , rawBytes, payloadCap, minGainPermil) then 15: return (plan.codecId, P , HdrBytes+P ) 16: end if 17: end if 18: return C OMMIT R AW O R FAIL(raw, rawBytes, outHdrBase, payloadCap)
codec boundary explicit, estimable, and automatically tunable at runtime. D. Overlapping Entropy Coding with Communication This subsection focuses on how NCCLZ realizes overlap for the remaining in-runtime entropy-coding stage. Existing overlap mechanisms do not cleanly preserve NCCL-native execution. MPI-based designs typically overlap either the full compressor or a collective-level compression pipeline with MPI progress [9]–[11], but these schemes are not aligned with NCCL’s native primitive and thus cannot directly preserve its device-side execution semantics. COCCL instead overlaps lightweight quantization through re-wrapped NCCL APIs, but its repository notes that compression-supported APIs cannot be used with ncclGroupStart/ncclGroupEnd, breaking NCCL’s native grouping interface [26]. In contrast, NCCLZ overlaps only the lossless entropy-coding stage within NCCL’s per-channel slot pipeline, allowing encode/decode progress to proceed alongside NET transfers while preserving native communication ordering and grouping semantics. The rest of this subsection explains how NCCLZ realizes this overlap strategy through NCCL primitive boundaries, batch framing, and pipelined execution. 1) Primitive coverage and full-duplex collectives: Rather than specializing for a specific collective algorithm, NCCLZ integrates framing and codec selection at NCCL’s communication boundaries in the device-side primitives [19]. Concretely, NCCLZ applies encode-on-send to any primitive that produces an outgoing transfer—right before the payload is enqueued into the connection FIFO / submitted to the network path. Symmetrically, it applies decode-on-recv to any primitive that consumes an incoming transfer—immediately after a frame is dequeued and before its bytes are handed to subsequent copy/reduce logic. This boundary-based design naturally covers all five collective primitives in NCCL where Send and Recv are both active in the same step (e.g., recvReduceSend in AllReduce), because the receive-side decode and the send-side encode
are simply triggered by their respective boundaries within the same primitive. As a result, any collective expressed as an iterative sequence of these primitives automatically goes through the same framing and encode_best/decode path. 2) Batch framing (header + payload): Instead of assuming a fixed payload size per slot, NCCLZ encapsulates each entropy-coded unit as a frame consisting of a compact header followed by a variable-length payload. The header records codecId, rawBytes, payloadBytes, and codec-specific parameters (e.g., fixed-length bitwidth or Huffman context information). This framing allows the receiver to validate and decode each frame independently, and to safely fall back to raw-copy semantics if validation or decoding fails. 3) 8-slot batching: NCCL Simple uses a fixed perchannel buffer of NCCL_BUFFSIZE=4 MiB, partitioned into NCCL_STEPS=8 slots, giving 512 KiB per slot [19]. NCCLZ therefore adopts fixed 8 slot batching as its framing and entropy coding granularity, so each frame spans one full channel buffer. This choice is motivated by a core NCCL constraint: entropy coding produces variable-length outputs, whereas NCCL Simple NET progress relies on fixed-slot FIFO bookkeeping between the GPU kernel and the proxy thread [19]. A naive per-slot design would either pad each compressed chunk back to slot size and waste bandwidth, or violate proxy expectations on how many bytes are ready to send. NCCLZ resolves this mismatch by treating each 8 slot batch as a self-describing frame with byte-accurate sizes on the NET path, allowing the proxy to transfer only valid bytes while preserving slot rotation and FIFO ordering. The concrete sender/receiver realization of this framing is described in Sec. IV-E. This design also gives entropy coding a larger context window up to NCCL_BUFFSIZE raw bytes, which amortizes selector and header overhead. Figure 3 shows that although small messages exhibit fill/drain effects, the fixed and non-fixed curves converge once the link is saturated. Therefore, fixed 8-slot batching does not reduce throughput in the target bandwidth-bound regime, while providing a larger framing window that improves entropy coding effectiveness. Therefore, fixing the granularity 240000
Fixed (8 slots) Non-fixed
180000
Latency (time, us)
Algorithm 1 Device-side runtime entropy arbitration
120000 2000 1750 1500 1250 600001000 750 500 250 0 0.5 0 0.5
1 1
2
4
8 16 Message size
32
64
Fig. 3. Fixed 8-slot batching vs. NCCL baseline.
128
to 8 slot batches does not sacrifice performance in the target regime, yet it provides a larger framing window that improves entropy coding effectiveness by increasing compressible context and amortizing per frame overhead. 4) Two level overlap: Entropy coding adds GPU side compute on the critical path unless it can be hidden behind an existing bottleneck. NCCLZ targets the bandwidth dominated Simple protocol on the inter node NET path, where network progress is primarily driven by the NIC and NCCL CPU proxy thread, while the GPU kernel frequently stalls on FIFO and step availability. In this regime, scheduled encode and decode work can occupy these stall periods without extending end to end time. a) Batch level overlap via ping pong staging: NCCLZ allocates two staging banks per channel and alternates them across 8 slot batches. When batch i has been published to the connection FIFO and is being progressed by the NET proxy and NIC, the GPU can encode batch i+1 into the alternate bank. This avoids overwriting in flight outputs and preserves NCCL FIFO ordering semantics because publication still respects the original slot availability protocol. On the receive side, NCCLZ decodes a completed batch immediately after it is dequeued from the FIFO, while the next batch is concurrently being transferred in the background by the proxy and NIC. The overlap is across consecutive batches and relies only on existing NCCL progress mechanisms. b) In primitive overlap for ring AllReduce: For ring AllReduce, NCCL uses fused primitives such as recvReduceSend, which already form a streaming dependency chain. NCCLZ inserts decode right after bytes are dequeued and encode right before bytes are enqueued, creating a per chunk pipeline recv to decode to reduce to encode to send. Within a step, different chunks are processed in a staggered manner, while chunk k is in reduction, chunk k+1 can be decoded and chunk k−1 can be encoded, reducing bubbles in the fused primitive without changing NCCL collective schedule. We enable this overlap only when the message is sufficiently large to amortize codec overhead, otherwise NCCLZ conservatively falls back to R AW. E. NCCLZ Implementation All components are implemented and integrated into NCCL as an opt-in extension. Both codecs as well as the selector are wrapped as header-only CUDA modules (.cuh) and invoked via a small codec dispatcher inside NCCL device code. This keeps the integration lightweight and minimizes intrusion into NCCL build/link logic. The integration point is the NET-only
Fig. 4. Batch level overlap within encode/decode stage.
Algorithm 2 NCCLZ Entropy coding pipeline for NET Path Input: connection state conn (FIFO, 8 slots), staging banks staging[0..1], sender buffer raw[0..rawBytes), receiver buffer dst[0..rawBytes), Huffman context huffCtx, Thuff , minGainPermil Output: sender enqueues a self-describing frame; receiver reconstructs dst (lossless) with robust fallback Sender-side (per 8-slot batch): 1: bank ← batchId mod 2 2: WAIT S LOTS F REE(conn, 8) 3: outHdrBase ← staging[bank] 4: stageCapBytes ← S TAGE C AP B YTES(staging[bank]) 5: (codecId, payloadBytes, totalBytes) ← ENCODE BEST(raw, rawBytes, . . . , minGainPermil) 6: if totalBytes > 0 then 7: E NQUEUE F RAME T O F IFO 8(conn, outHdrBase, totalBytes) 8: end if Receiver-side (per 8-slot batch): 9: frameBase ← D EQUEUE F RAME F ROM F IFO 8(conn) 10: hdr ← PARSE H EADER(frameBase) 11: if not VALIDATE H EADER(hdr) then 12: M EMCPY(dst, frameBase + HdrBytes, rawBytes) 13: else 14: codecId ← hdr.codecId; payloadBytes ← hdr.payloadBytes 15: D ECODE O R M EMCPY(codecId, frameBase HdrBytes, payloadBytes, dst, hdr.rawBytes, huffCtx, hdr.params) 16: end if
+
path of the Simple protocol for relevant primitives: NCCLZ intercepts the per-batch (8-slot) data segment right before it is enqueued into the connection FIFO, and similarly intercepts it right after it is dequeued, enabling encode/decode to overlap with NET progress. A complete workflow is summarized in Algorithm 2. The inputs are connection state conn with 8-slot FIFO batching, ping-pong staging banks staging[0..1], user buffers raw/dst, and codec controls huffCtx, Thuff , minGainPermil. The sender produces a self-describing frame (header + payload), and the receiver reconstructs dst losslessly, with a safe RAW fallback when header validation fails. a) Sender-side (Lines 1–8): Lines 1–2 first establish the pipeline context for this batch: the sender picks the ping-pong staging bank using batchId mod 2 and then waits until 8 FIFO slots become available, so the subsequent enqueue respects NCCL’s slot-based ordering and progress semantics. With slots guaranteed, Lines 3–4 configure the staging destination by setting the frame base pointer (outHdrBase) and its usable capacity (stageCapBytes). On this prepared staging region, Line 5 invokes encode_best (Algorithm 1) to (i) choose the most cost-effective codec among FixedLen/Huffman/RAW and (ii) materialize a complete self-describing frame (header + payload) in-place, returning the final byte size totalBytes. Finally, Lines 6–8 publish the batch by enqueuing the frame into the 8-slot FIFO only when totalBytes> 0, preventing partial-frame exposure and ensuring clean failure. b) Receiver-side (Lines 9–16): Symmetrically, Line 9 retrieves the next frame for this 8-slot batch from the FIFO, and Line 10 parses its header to learn how the payload should be interpreted. Lines 11–12 then gate the fast/slow paths via header validation: if validation fails, the receiver immediately falls back to RAW semantics and copies the expected rawBytes from frameBase+HdrBytes into dst, preserving correctness even under corruption or incompatibility. Otherwise, the receiver proceeds with the codec-directed path in Lines 13– 15: it extracts codecId and payloadBytes from the header
(Line 14) and dispatches DecodeOrMemcpy (Line 15) to either decode FixedLen/Huffman or perform a direct memcpy for RAW. Line 16 closes the conditional, completing lossless reconstruction for the batch. V. E VALUATION In this section we provide details about evaluation method and analyze the results. A. Experimental setup a) Platform and software: We conduct all experiments on the Polaris supercomputer at Argonne National Laboratory [29]. Each Polaris compute node is equipped with an AMD EPYC Milan CPU, 512 GB of system memory, and four NVIDIA A100 GPUs. In our deployment, nodes are interconnected via a high-speed HPE Slingshot network. Across all experiments, we use unmodified NCCL 2.28.3 as the baseline and keep the software stack and runtime configuration consistent, including the same NCCL_PROTO, NCCL_ALGO, and other relevant environment variables. b) Dataset: To comprehensively evaluate the effectiveness of NCCLZ , we select a diverse set of datasets that cover heterogeneous workload characteristics and communication patterns. Specifically, to assess the applicability and practicality of our design across representative HPC workloads, we use two scientific datasets from SDRBench, QMCPack and CESM-ATM. And we additionally collect per-epoch training gradients from ResNet-18 and ResNet-34 to reflect realistic distributed learning traffic. Finally, to isolate scalability effects and obtain controlled measurements across a wide range of node counts and message sizes, we also benchmark NCCLZ using the synthetic FP32 payloads provided by nccl tests, enabling a systematic evaluation of performance trends under standardized communication inputs. B. Compression Ratio Study We first study the compression ratio achieved by NCCLZ on representative scientific datasets and training gradients. For scientific fields, including QMCPack and CESM-ATM, we use the deterministic error-bounded quantization with relative error bound REL= 1E-4. For training data, we take one epoch of gradients and quantize them with QSGD. We then run AllReduce under 2, 4, and 8 nodes. To isolate the compression behavior of different entropy coders and to support the codec arbitration analysis in later sections, we pin NCCLZ to use F IXED L EN or GPU H UFFMAN and report the resulting compression ratios in Table 1. We compute compression ratio using the raw pre quantiraw zation byte size as the reference, CR = B Bout . Quantization is a mandatory pre processing step for entropy coding in our pipeline, so we report two related ratios. The quantization only raw ratio is CRquant = BBquant where Bquant is the byte size after quantization without entropy coding. The end to end ratio is raw CRfinal = BBentropy where Bentropy is the compressed payload size produced after applying entropy coding on quantized symbols.
Table I reports (i) quantization-only CR, (ii) CR when forcing F IXED L EN or GPU H UFFMAN after quantization. Across all datasets and node counts, GPU H UFFMAN consistently achieves higher CR than F IXED L EN. This suggests that the quantized symbol streams still contain substantial distribution skew that GPU H UFFMAN can exploit, whereas F IXED L EN mainly removes representational slack through bitwidth trimming. For example, averaged across node counts, QMCPack improves from 3.28× under quantization only to 4.63× with F IXED L EN and 8.25× with GPU H UFFMAN, while gradients improve from 3.27× to 5.95× and 8.07×, respectively. Overall, entropy coding provides a second-stage reduction on top of quantization, producing smaller communicated payloads and greater potential gains in bandwidth-bound settings. C. End-to-end performance on Real Datasets In this subsection, we evaluate end-to-end AllReduce throughput on three real-world tensors, two scientific datasets both with REL=1E-4 and one training-gradient quantized by QSGD.We compare NCCLZ F IXED L EN and NCCLZ GPU H UFFMAN against the unmodified NCCL baseline, CoCCL, and ghZCCL. As shown in Figures 5, NCCLZ consistently achieves the highest end-to-end AllReduce throughput among all compared methods across all three datasets and all node counts. On QMCPack, NCCLZ raises throughput from roughly 10 GB/s to about 26–33 GB/s, reaching up to 2.89× with F IXED L EN and 3.13× with GPU H UFFMAN. On QSGD gradients, NCCLZ still improves throughput to about 24–25 GB/s, achieving up to 2.37× with F IXED L EN and 2.46× with GPU H UFFMAN. In all cases, NCCLZ outperforms CoCCL and the ghZCCL results, and the advantage of GPU H UFFMAN over F IXED L EN is most pronounced on CESM-ATM, where the symbol stream exhibits lower entropy and thus higher compressibility. The performance gains come from two complementary effects. First, external quantization combined with entropy coding increases the effective compression ratio, thereby reducing the bytes injected into NCCL’s network path and improving throughput in the bandwidth-bound regime. Second, NCCLZ follows NCCL’s native execution workflow and preserves the pipelined AllReduce schedule, so encoding and decoding can overlap with communication instead of incurring a separate staging path. As a result, NCCLZ consistently outperforms prior compression-assisted libraries. The results further reveal a practical trade-off between the two codecs: F IXED L EN provides robust performance due to its low and predictable overhead, while GPU H UFFMAN is beneficial only when its extra compression gain is sufficient to amortize its higher codec cost, as in CESM-ATM. D. NCCLZ Scalability Study We next evaluate scalability by sweeping message sizes and comparing the achieved Bus Bandwidth (BusBW) under Alltoall and AllReduceacross 8, 16, and 32 nodes. Figure 6 and 7 shows that NCCLZ consistently improves
TABLE I C OMPRESSION RATIO (CR) ACROSS NODE COUNTS FOR SCIENTIFIC DATASETS (REL=1E-4) AND TRAINING GRADIENTS AS WELL AS CR OF QUANTIZATION - ONLY ( NO ENTROPY CODING ) FOR A COMPARISON . Quant-only (CR)
Dataset
FixedLen (CR)
Avg
4-node
8-node
(2/4/8)
2-node
4-node
8-node
(2/4/8)
2-node
4-node
8-node
(2/4/8)
3.05× 3.20× 2.55×
3.10× 3.25× 2.60×
3.00× 3.18× 2.52×
3.05× 3.21× 2.56×
4.61× 6.74× 5.26×
4.59× 7.24× 6.14×
4.68× 6.35× 6.45×
4.63× 6.78× 5.95×
8.15× 9.45× 8.44×
8.25× 9.65× 7.31×
8.35× 9.85× 8.47×
8.25× 9.65× 8.07×
CoCCL
ghZCCL
NCCLZ-FixedLen
NCCLZ-GPU Huffman
60
40 30
Throughput (GB/s)
Throughput (GB/s)
50 40
3.13×
2.89× 2.57×
0
2
30
4.79× 4.16× 3.58×
20
1.15×
10
Avg
2-node
Baseline NCCL
20
GPU Huffman (CR)
1.54×
10
4
(a) QMCPack (1.2 GiB, REL=1E-4).
0
8
2
4
(b) CESM-ATM (2.0 GiB, REL=1E-4).
8
35 30 25 20 15 10 5 0
Throughput (GB/s)
QMCPack (1.2 GiB) CESM-ATM (2.0 GiB) Grad (QSGD, 150 MiB)
Avg
2.46× 2.37×
1.99×
1.40×
2
4
(c) QSGD gradients (150 MiB).
8
Fig. 5. End-to-end AllReduce throughput on real-world datasets with NCCL baseline, CoCCL, ghZCCL, and NCCLZ (F IXED L EN/GPU H UFFMAN).
BusBW (GB/s)
60 50 40
20
30
15
20 10
communication becomes bandwidth bound. For messages up to 64 KiB, the gap is small because fixed overheads dominate. From 512 KiB onward, reducing the transmitted bytes translates into substantially higher BusBW, and NCCLZ consistently outperforms baseline NCCL in both Alltoall and AllReduce. This advantage persists from 8 to 32 nodes, indicating that the benefit comes from shrinking the network payload rather than from a scale-specific artifact.
8-node Baseline 8-node NCCLZ 16-node Baseline 16-node NCCLZ 32-node Baseline 32-node NCCLZ 6.93x
70
10 5 0
0
1K
1K
8K
8K
64K
64K
E. NCCLZ Ablation Study
512K
512K
4M
Message Size
32M
256M
1G
Fig. 6. Alltoall BusBW versus message size on 8, 16, and 32 nodes. 70 60
40 30 20 10 0
6.56x
BusBW (GB/s)
50
8-node Baseline 8-node NCCLZ 16-node Baseline 16-node NCCLZ 32-node Baseline 32-node NCCLZ
12.5 10.0 7.5 5.0 2.5 0.0
1K
1K
8K
8K
64K
64K
512K
512K
4M
Message Size
32M
256M
1G
Fig. 7. AllReduce BusBW versus message size on 8, 16, and 32 nodes.
AllReduce BusBW across 8, 16, and 32 nodes. The gap is small for messages up to 64 KiB, where all configurations remain latency dominated. From 512 KiB onward, the benefit becomes pronounced: baseline NCCL stays around 10 GB/s or below, while NCCLZ reaches tens of GB/s and peaks at about 68 GB/s. The largest improvement is 6.61×. Overall, the same scaling trend is preserved from 8 to 32 nodes, indicating that the gain of NCCLZ remains effective under scale-out. The key takeaway is that NCCLZ is most effective once
To understand where NCCLZ’s gains come from, we conduct the ablation study and decompose the design into the compression path and the overlap mechanism. Entropy Coding Benefit. Figure 8 compares the average compression ratio across node counts for three representative workloads, where the x-axis shows the workload category and the y-axis reports the compression ratio relative to the raw pre-quantization size. For each workload, we compare three configurations: quantization only, quantization followed by FixedLen, and quantization followed by GPU Huffman. The result shows that entropy coding provides a clear second-stage reduction beyond quantization alone. Across all three workloads, FixedLen consistently improves compression over quant-only, while GPU Huffman achieves the largest gains, though the magnitude varies with the post-quantization symbol distribution. This workload-dependent gap also motivates REA: since no single entropy coder is uniformly best, NCCLZ selects at runtime the codec whose extra compression is most likely to justify its encode/decode overhead. Overall, the figure shows that entropy coding is important, and that adaptive arbitration is necessary to translate its benefit into robust end-to-end gains. Overlap Efficiency. To quantify how effectively NCCLZ overlaps compression with communication, we compare its measured end-to-end time against a synthetic no-overlap baseline in which compression is fully serialized. For each mes-
VI. R ELATED W ORK
Compression ratio (avg across node counts)
Compression and Quantization on GPU. GPU-resident data reduction is increasingly used to mitigate bandwidth bottlenecks without CPU staging, especially in communicationand I/O-intensive workloads. In scientific computing, prior work has focused on error-bounded lossy compression for floating-point data. SZ-style compressors combine prediction, error-controlled quantization, and entropy coding to achieve high compression on smooth fields [4], while ZFP uses a block-based transform with fixed-rate and fixed-accuracy modes for high-throughput numerical workloads [30]. Recent systems adapt these designs to GPUs: cuSZ reworks SZ for GPU execution with GPU-oriented entropy coding while preserving error guarantees [5], and ZFP also supports CUDA-based whole-array compression and decompression [31]. In distributed deep learning, related work instead emphasizes communication compression, such as gradient Quant-only
10
4
+ GPU Huffman
+170% 8.25x
8 6
+ FixedLen +201% 9.65x
+215% 8.07x +111% 6.78x
+132% 5.95x
+52% 4.63x
25000
Encode Comm
Decode With overlap
26246 1.46×
20000
17977
15000
13727 9402
10000 5000 0
21491472
64MiB
4160 2849
128MiB
6458 4423
256MiB
Message Size
512MiB
1GiB
Fig. 9. NCCLZ overlap time versus no-overlap time.
quantization with error feedback, rather than strict numerical error bounds [32]. CPU-centric compression communication libraries. A large body of work improves communication efficiency by integrating compression into CPU-centric message-passing stacks, where data are compressed and decompressed on the host and injected transparently into MPI. Early systems such as AdOC and cMPI explored runtime compression without requiring application changes [33], [34]. Later work emphasized adaptivity: CoMPI and PRAcTICaL-MPI select codecs at runtime based on message characteristics and communication conditions [35], [36]. More recent efforts revisit this design for modern GPU clusters. The Panda line co-designs collective-level online compression with MPI algorithms to improve overlap in GPU-aware MPI libraries [37], [38], while C-Coll integrates error-bounded lossy compression into MPI collectives with explicit control over numerical distortion [39]. State-of-the-Art Collective Communication Libraries. MPI remains the dominant general-purpose collective substrate in HPC, with mature implementations providing portability and highly tuned collectives. MPICH is a widely used reference implementation and the basis of many derived distributions [40], [41]. Modern MPI stacks increasingly support CUDA/ROCm-aware communication and GPU-oriented optimizations, but they still follow MPI’s general abstraction and progress model. MVAPICH2 is a high-performance MPICHderived design for InfiniBand/RDMA, and GPU-oriented variants such as MVAPICH2-GDR incorporate GPUDirect RDMA and optimized device-buffer collectives [28], [42]. Vendor CCLs instead expose GPU-centric APIs and kernel-level implementations that bypass much of the MPI stack. RCCL targets AMD GPUs across intra- and inter-node settings [43], oneCCL targets Intel and heterogeneous deep learning environments and now provides a NCCL-like C API [44], and Gloo offers a practical multi-backend collective layer widely used in PyTorch distributed workloads [45]. VII. C ONCLUSION AND F UTURE W ORK
3.05x
3.21x
QMCPack
CESM-ATM
2.56x
2 0
30000
Time (us)
sage size, the no-overlap time sums encode, communication, and decode, with communication anchored by the measured NCCLZ end-to-end time. Figure 9 shows that overlap consistently reduces end-to-end runtime across 64,MiB–1,GiB. Without overlap, encode and decode are added directly on top of communication, whereas in NCCLZ a large fraction of codec work is hidden behind NET transfer. The gap between the no-overlap and NCCLZ bars therefore directly reflects the effective overlap benefit from pipelining. The breakdown also provides insight into why overlap becomes more challenging at very large messages. As message size grows, the encode segment in the no-overlap baseline becomes more prominent, reflecting the increasing Huffman encoding overhead at scale (e.g., higher bitstream construction cost and global-memory traffic). This growth reduces the amount of communication slack available to hide codec work, so the remaining exposed portion of encoding overhead contributes more to the gap between the ideal and realized performance. Nevertheless, even at 512 MiB and 1 GiB, NCCLZ preserves a clear advantage over the serialized baseline, demonstrating that our integration strategy and slot-level pipelining are effective at overlapping most compression work with communication in the bandwidth-bound regime.
Gradients
Fig. 8. Average CR across node counts for three workloads under quantization only, quantization + FixedLen, and quantization + GPU Huffman.
In this Paper, we presented NCCLZ, a compression-enabled NCCL-based framework targeting optimized communication over bandwidth-constrained NVIDIA GPU clusters. NCCLZ decomposes lossy compression into the external quantization stage in the interface layer and the entropy coding stage in NCCL primitives. Through evaluations on large GPU cluster,
NCCLZ consistently improves compression ratios and endto-end collective throughput, delivering up to a 9.65× gain over unmodified NCCL and up to a 3.34× improvement over existing compression-assisted CCLs. Looking ahead, we plan to tune NCCLZ for scenarios such as sharded data parallelism and collective-heavy LLM parallelism. ACKNOWLEDGMENT ChatGPT was used only for language polishing and for checking grammar, spelling, and formatting errors in this manuscript. All technical content, results, claims, and references were reviewed and verified by the authors. R EFERENCES [1] A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” arXiv preprint arXiv:1802.05799, vol. abs/1802.05799, pp. 1–13, 2018. [Online]. Available: https: //doi.org/10.48550/arXiv.1802.05799 [2] Z. Zhang, C. Chang, H. Lin, Y. Wang, R. Arora, and X. Jin, “Is network the bottleneck of distributed training?” in Workshop on Network Meets AI & ML, ser. NetAI ’20. Virtual Event, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3405671.3405810 [3] H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xie, and E. P. Xing, “Poseidon: An efficient communication architecture for distributed deep learning on gpu clusters,” in 2017 USENIX Annual Technical Conference (USENIX ATC 17). Santa Clara, CA, USA: USENIX Association, 2017, pp. 181–193. [Online]. Available: https://www.usenix.org/conference/atc17/technical-sessions/ presentation/zhang [4] S. Di and F. Cappello, “Fast error-bounded lossy HPC data compression with SZ,” in 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Chicago, IL, USA: IEEE, 2016, pp. 730–739. [Online]. Available: https://szcompressor.org/tabs/publication/ [5] J. Tian, S. Di, K. Zhao, C. Rivera, M. Hickman Fulp, R. Underwood, S. Jin, X. Liang, J. Calhoun, D. Tao, and F. Cappello, “cusz: An efficient GPU-based error-bounded lossy compression framework for scientific data,” in Proceedings of the 29th International Conference on Parallel Architectures and Compilation Techniques (PACT). New York, NY, USA: Association for Computing Machinery, 2020, pp. 1–12. [Online]. Available: https://doi.org/10.1145/3410463.3414624 [6] C. Chen, Y. He, P. Li, W. Jia, and K. Yuan, “Greedy low-rank gradient compression for distributed learning with convergence guarantees,” arXiv preprint arXiv:2507.08784, 2025. [Online]. Available: https: //doi.org/10.48550/arXiv.2507.08784 [7] H. Feng, B. Zhang, F. Ye, M. Si, C.-H. Chu, J. Tian, C. Yin, S. Deng, Y. Hao, P. Balaji, T. Geng, and D. Tao, “Accelerating communication in deep learning recommendation model training with dual-level adaptive lossy compression,” in SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, 2024, pp. 1–16. [Online]. Available: https://doi.org/10.1109/SC41406.2024.00095 [8] G. He, Y. Cao, Y. He, T. Bai, K. Yuan, and B. Yuan, “Tah-quant: Effective activation quantization in pipeline parallelism over slow network,” arXiv preprint arXiv:2506.01352, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.01352 [9] Q. Zhou, C.-H. Chu, N. S. Kumar, P. Kousha, S. M. Ghazimirsaeed, H. Subramoni, and D. K. Panda, “Designing high-performance MPI libraries with on-the-fly compression for modern GPU clusters,” in 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Portland, OR, USA: IEEE, 2021, pp. 444–453. [Online]. Available: https://doi.org/10.1109/IPDPS49936.2021.00053 [10] J. Huang, S. Di, X. Yu, Y. Zhai, J. Liu, Y. Huang, K. Raffenetti, H. Zhou, K. Zhao, X. Lu, Z. Chen, F. Cappello, Y. Guo, and R. Thakur, “gZCCL: Compression-accelerated collective communication framework for GPU clusters,” in Proceedings of the 38th ACM International Conference on Supercomputing (ICS ’24). New York, NY, USA: Association for Computing Machinery, 2024, pp. 437–448. [Online]. Available: https://doi.org/10.1145/3650200.3656636
[11] J. Huang, S. Di, Y. Huang, Z. Chen, F. Cappello, Y. Guo, and R. Thakur, “ghzccl: Advancing GPU-aware collective communications with homomorphic compression,” in Proceedings of the 2025 International Conference on Supercomputing, ser. ICS ’25. New York, NY, USA: Association for Computing Machinery, 2025. [Online]. Available: https://doi.org/10.1145/3721145.3733642 [12] X. Liu, H. Kong, H. Zhao, S. Lyu, Z. Wei, M. Liu, X. Tian, L. Zhao, Z. Chen, F. Wang, Z. Chen, Z. Wang, G. Tan, and D. Tao, “Coccl: A collective communication library supporting easy integration and configuration of customized compression for scalable llm training,” in Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, ser. PPoPP ’26, 2026, pp. 384–397. [Online]. Available: https: //doi.org/10.1145/3774934.3786432 [13] NVIDIA, “NVIDIA collective communications library (NCCL),” 2025, accessed: 2026-01-16. [Online]. Available: https://github.com/NVIDIA/ nccl [14] ——, “NVIDIA collective communications library (NCCL),” 2026, accessed: 2026-01-27. [Online]. Available: https://developer.nvidia.com/ nccl [15] PyTorch, “Distributeddataparallel — PyTorch documentation,” 2026, accessed: 2026-01-27. [Online]. Available: https://docs.pytorch.org/ docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html [16] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, vol. abs/1909.08053, pp. 1–12, 2019. [Online]. Available: https: //doi.org/10.48550/arXiv.1909.08053 [17] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” arXiv preprint arXiv:2309.06180, vol. abs/2309.06180, pp. 1–17, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.06180 [18] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient execution of structured language model programs,” arXiv preprint arXiv:2312.07104, vol. abs/2312.07104, pp. 1–16, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2312.07104 [19] Z. Hu, S. Shen, T. Bonato, S. Jeaugey, C. Alexander, E. Spada, J. Dinan, J. Hammond, and T. Hoefler, “Demystifying nccl: An in-depth analysis of GPU communication protocols and algorithms,” arXiv preprint arXiv:2507.04786, vol. abs/2507.04786, pp. 1–24, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2507.04786 [20] NVIDIA, “NVIDIA A100 80GB PCIe GPU (product brief),” NVIDIA, Tech. Rep., 2022, reports up to 600 GB/s NVLink bandwidth with NVLink bridges. [Online]. Available: https://www.nvidia.com/content/ dam/en-zz/Solutions/Data-Center/a100/pdf/PB-10577-001 v02.pdf [21] F. Liang, Z. Zhang, H. Lu, V. C. M. Leung, Y. Guo, and X. Hu, “Communication-efficient large-scale distributed deep learning: A comprehensive survey,” 2024. [Online]. Available: https://doi.org/10. 48550/arXiv.2404.06114 [22] S. Di, D. Tao, X. Liang, and F. Cappello, “Efficient lossy compression for scientific data based on pointwise relative error bound,” IEEE Transactions on Parallel and Distributed Systems, vol. 30, no. 2, pp. 331–345, 2018. [Online]. Available: https: //doi.org/10.1109/TPDS.2018.2859932 [23] P. Lindstrom, “Fixed-rate compressed floating-point arrays,” IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 12, pp. 2674–2683, 2014. [Online]. Available: https://doi.org/10.1109/ TVCG.2014.2346458 [24] D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-efficient SGD via gradient quantization and encoding,” arXiv preprint arXiv:1610.02132, vol. abs/1610.02132, pp. 1–14, 2017. [Online]. Available: https://doi.org/10.48550/arXiv.1610.02132 [25] Y. Huang, S. Di, X. Yu, G. Li, and F. Cappello, “CuSZp: An ultra-fast GPU error-bounded lossy compression framework with optimized endto-end performance,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’23. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3581784.3607048 [26] X. Liu, H. Kong, Z. Wei, L. Zhao, Y. Wang, and J. Yang, “COCCL: Compression and precision co-aware collective communication library,” 2025, accessed: 2026-01-26. [Online]. Available: https://github.com/ hpdps-group/COCCL
[27] Q. Zhou, C. Chu, N. S. Kumar, S. M. G. Pouya Kousha and, H. Subramoni, and D. K. Panda, “Designing high-performance MPI libraries with on-the-fly compression for modern GPU clusters,” in 35th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2021, Portland, OR, USA, May 17–21, 2021. Portland, OR, USA: IEEE, 2021, pp. 444–453. [Online]. Available: https://doi.org/10.1109/IPDPS49936.2021.00053 [28] MVAPICH Project, “Mvapich2-gdr user guide,” 2026, accessed: 202602-06. [Online]. Available: https://mvapich.cse.ohio-state.edu/userguide/ gdr/ [29] A. L. C. Facility, “Polaris,” Argonne Leadership Computing Facility (ALCF), 2026, accessed: 2026-02-03. [Online]. Available: https: //www.alcf.anl.gov/polaris [30] P. Lindstrom, “Fixed-rate compressed floating-point arrays,” IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 12, pp. 2674–2683, 2014. [Online]. Available: https://doi.org/10.1109/ TVCG.2014.2346458 [31] LLNL, “zfp: Compressed floating-point and integer arrays (cuda support),” GitHub repository, 2026, accessed 2026-02-06. [Online]. Available: https://github.com/LLNL/zfp [32] F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” in INTERSPEECH. Singapore: ISCA, 2014, pp. 1058–1062. [Online]. Available: https://www.microsoft.com/en-us/ research/wp-content/uploads/2016/02/IS140694.pdf [33] E. Jeannot and P. Strazdins, “Improving middleware performance with AdOC: An adaptive online compression library for data transfer,” in Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS). Denver, CO, USA: IEEE, 2005, pp. 1–8. [Online]. Available: https://doi.org/10.1109/IPDPS.2005.254 [34] J. Ke, M. Burtscher, and E. Speight, “Runtime compression of MPI messages to improve the performance and scalability of parallel applications,” in Proceedings of the ACM/IEEE Conference on Supercomputing (SC ’04). Pittsburgh, PA, USA: IEEE Computer Society, 2004, p. 59. [Online]. Available: https://doi.org/10.1109/SC. 2004.52 [35] R. Filgueira, D. E. Singh, A. Calderón, and J. Carretero, “Compi: Enhancing MPI based applications performance and scalability using run-time compression,” in Recent Advances in Parallel Virtual Machine and Message Passing Interface (EuroPVM/MPI 2009), ser. Lecture Notes in Computer Science, vol. 5759. Espoo, Finland: Springer, 2009, pp. 207–218. [Online]. Available: https: //doi.org/10.1007/978-3-642-03770-2 27 [36] R. Filgueira, M. Atkinson, A. Nuñez, and J. Fernández, “An adaptive, scalable, and portable technique for speeding up MPIbased applications,” in Euro-Par 2012 Parallel Processing, ser. Lecture Notes in Computer Science, vol. 7484. Rhodes Island, Greece: Springer, 2012, pp. 729–740. [Online]. Available: https: //doi.org/10.1007/978-3-642-32820-6 72 [37] Q. Zhou, P. Kousha, Q. Anthony, K. S. Khorassani, A. Shafi, H. Subramoni, and D. K. Panda, “Accelerating MPI all-to-all communication with online compression on modern GPU clusters,” in High Performance Computing – 37th International Conference, ISC High Performance 2022, Hamburg, Germany, May 29–June 2, 2022, Proceedings, ser. Lecture Notes in Computer Science, A. L. Varbanescu, A. Bhatele, P. Luszczek, and M. Baboulin, Eds., vol. 13289. Hamburg, Germany: Springer, 2022, pp. 3–25. [Online]. Available: https://doi.org/10.1007/978-3-031-07312-0 1 [38] B. R. Qinghua Zhou and, A. Shafi, M. Abduljabbar, H. Subramoni, and D. K. Panda, “Accelerating MPI allreduce communication with efficient gpu-based, compression schemes on modern GPU clusters,” in ISC High Performance 2024 Research Paper Proceedings (39th International, Conference), Hamburg, Germany, May 12-16, 2024. Hamburg, Germany: Prometeus GmbH / IEEE, 2024, pp. 1–12. [Online]. Available: https://doi.org/10.23919/ISC.2024.10528931 [39] J. Huang, S. Di, X. Yu, Y. Zhai, J. Liu, K. Raffenetti, H. Zhou, K. Zhao, Z. Chen, F. Cappello, Y. Guo, and R. Thakur, “C-coll: Introducing error-bounded lossy compression into mpi collectives,” arXiv preprint arXiv:2304.03890, vol. abs/2304.03890, pp. 1–19, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.03890 [40] MPICH Project, “Mpich overview,” 2026, accessed: 2026-02-06. [Online]. Available: https://www.mpich.org/about/overview/ [41] W. Gropp, E. Lusk, N. Doss, and A. Skjellum, “A high-performance, portable implementation of the mpi message passing interface standard,”
Parallel Computing, vol. 22, no. 6, pp. 789–828, 1996. [Online]. Available: https://doi.org/10.1016/0167-8191(96)00024-5 [42] W. Huang, G. Santhanaraman, H.-W. Jin, Q. Gao, and D. K. Panda, “Design of high performance mvapich2: Mpi2 over infiniband,” in Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID). Singapore: IEEE, 2006, pp. 43–48. [Online]. Available: https://doi.org/10.1109/CCGRID.2006.32 [43] AMD, “Rccl documentation (rocm communication collectives library),” 2026, accessed: 2026-02-06. [Online]. Available: https://rocmdocs.amd. com/projects/rccl/en/latest/index.html [44] UXL Foundation, “oneapi collective communications library (oneccl) documentation,” 2026, accessed: 2026-02-06. [Online]. Available: https://uxlfoundation.github.io/oneCCL/index.html [45] PyTorch, “Gloo: Collective communications library,” 2026, accessed: 2026-02-06. [Online]. Available: https://github.com/pytorch/gloo