UCCL-Zip: Lossless Compression Supercharged GPU Communication Shuang Ma† ChonLam Lao‡ Zhiying Xu§∗ Zhuang Wang§∗ Ziming Mao ¶ Delong Meng§∗ Zhen Jia§∗ Jun Wu§∗ Ion Stoica ¶ Yida Wang§∗ Yang Zhou† † UC Davis ‡ Harvard University § Amazon Web Services ¶ UC Berkeley Abstract
Kernel Launch Overhead Kernel Launch Kernel LaunchKernel Communication Overhead Overhead (NCCL)
✕ No OverLap Compression Latency
✕ No OverLap ✕ No OverLap Compression Latency
arXiv:2604.17172v2 [cs.DC] 21 Apr 2026
Network The rapid growth of large language models (LLMs) has made Data Block Compression Kernel Compression Latency Send Communication Kernel Network Communication Kernel Network Data Block Compression Kernel Time GPU communication a critical bottleneck. While prior work Data Block Compression Kernel (NCCL) Send Fraction (7 bits) (NCCL) Send Time 1 0 1 0 0 1 0reduces 0 0 BFloat16 Time communication volume via quantization or lossy (a) Baseline. GPU Persistent Kernel Chunk 1 Chunk 2 Chunk 3 w floating point tensor compression, these approaches introduce numerical errors Local Frequency Time S1 S2 S3 S1 S2 S3 S1 S2 S3 Time GPU Persistent Kernel Table Chunk 1 Chunk 2 Chunk 3 GPU Persistent Kernel Chunk 1 Chunk 2 Chunk 3 Breakdown Chunk Compression Steps... s Local Frequency Time S1 S2 S3 S1 Network S2 S3 S1 S2Steps S3 in Local Frequency Time S1 S2 that can degrade convergence, accuracy, and stability. We Fused S3 S1 S2 S3 S1 S2 S3 Time compressed part Step3 Step1 Step2 t Table Data Table Send 14% S1 Reduce ... Early-transmittable Compression compression ... Send Better Overlap Steps in present UCCL-Zip, a unified design that integrates lossless Early-transmittable Fused Data (from S1) Network Fused Network Data Send Data block Compressed data block 62% Reduce Data Send Reduce Data (from Step 1) Early-transmittable Compression S2 Send Compression Send Send compression Send compression directly into GPU communication primitives. Data (from S1) a blocks UCCL-Zip supports both point-to-point (P2P) and collective 24% S3 (b) Uzip-P2P. (c) Uzip-NCCL. communication without modifying user-facing APIsChunk or2 comChunk 3 Send small fraction of S1 S2 S3 S1 S2 S3 Figure 1. Overview of UCCL-Zip.the(a)restNaive design suffers from lack promising numerical correctness. For P2P communication, ... Symbols 136 and142 1 Symbols 1 136 142 Uzip-P2P of overlap additional kernel overhead. (b) enables Uzip-P2P employs a split-send pipeline that exposes transBitstream 100010001000100000000001 Bitstream 100010001000100000000001 NIC Idle GPU Compute NIC Send early transmission and overlaps compression with communication; missible data early and overlaps compression with commuTensor Tensor detailed steps are shown in Figure 4. (c) Uzip-NCCL integrates comSend over half amount of data nication, while preserving high GPU efficiency by operating ... ... CTA CTA CTA CTA from compression step 1. CTA CTA CTA Send compressed data pression into the persistent kernel, CTA eliminating redundant memory on large data blocks. For collective communication, Uzipve approach: compress entire tensor then send Synchronization Time traffic andSymbol extra Count kernel launches. across thread 1 3 NCCL integrates compression into NCCL’s persistent kernel Localized Frequency Tables blocks (CTAs) 142 1 Sample -> Build Table -> Encode is required. model via fused execution, eliminating redundant memory ... communication ... To reduce overhead, prior work has ex(Same Kernel) Global Frequency Table al conventional pipeline: traffic split tensor into blocks and kernel launches. In real workloads, UCCL-Zip Time tensively explored quantization and other lossy compresBlock 2 Block 3 Block accelerates RL4 weight synchronization by up to 47.5% and sion techniques to reduce data volume [1, 22, 30, 44, 48, 50]. lock 1 Send Block 2 Block 3 Send Block 4 Receive in & Decompression reducesSend vLLM end-to-end inference latency by up to 10%, all While effective improving bandwidth utilization, these alistic chunk pipeline: overhead from small blocks Time without application changes. approaches inevitably introduce numerical errors. These GPU0 GPU1 GPU2 send compressed part Stage 1 errors can slow convergence during training or degrade ssed part GPU1 GPU2 GPU0 model accuracy at inference time, and may also cause ip split-send pipeline: transmit early Time 1 dataIntroduction GPU2 GPU0 GPU1 compressed many times as small blocks training–inference mismatch in RL [18]. More recently, (b) Ring reduce_scatter with compression. The rapid growth of model size and context length in large lossless compression techniques on GPUs have begun to language models (LLMs) [9, 11, 15, 40] has made multi-node, GPU119, 34, 37, 52], GPU0demonstrating the potential emerge [3, 17, GPU0 multi-GPU execution indispensable for both training and Reduce Send GPU1 GPU1 to reduce data sacrificing numerical fidelity. GPU2 size without inference. In such settings, large volumes of data, includThis observation opens up a promising opportunity: applycompressed few times as s Reduce Decompress Send GPU0 large blocks ing gradients, model parameters, activations, and intermeing lossless compression to GPU communication. However, GPU2 GPU1 diate states, are exchanged across GPUs within and across (c) Two-Shot reduce_scatter with compression. despite its potential, this direction remains largely undernodes. Collective communication [2, 27, 43] and point-toexplored, particularly in the context of tightly integrated Compressed Data Reduce operations (e.g., sum) Invalid Data point (P2P) communication are two widely used commucommunication Receive pipelines. Send Original nication primitives GPU systems. Collective ssed Data Reduce operations (e.g., sum) Invalid Datain distributed Receive & Decompression Effectively NCCL hiding the overhead of lossless compression Send compressed many times as small blocks Decompress Compress communication is commonly used in regular and strucReceive within the pipeline remains a key Send challenge. NCCLcommunication with Decompress Send back Compression tured Compress communication patterns, such as tensor parallelismLoop On modern GPUs, compression latency does not scale linto GPU0 (TP), pipeline parallelism (PP), and distributed GPU0 otherGPU1 GPU2training early with input size due to their highly parallel execution Ring reduce_scatter with compression. ... paradigms [31,... 36, 42]. P2P communication provides greater ... compressed once as a large batch model, where ...throughput is maximized only when suffi... ... ... flexibility and is commonly used in scenarios such as KV PU 0 GPU 1 GPU 2 cient parallelism is to GPU fully utilize hardware reGPUexposed 0 1 GPU 2 cache transfer [33, 35, 39, 46] and asynchronous weight upsources [38]. As a result, compressing smaller chunks often GPU0 GPU2 GPU1 dates in reinforcement learning workloads. Ascompression. a result, GPU Two-Shot reduce_scatter with incurs similar latency to larger ones, undermining convencommunication performance has become a critical bottletional fine-grained pipelining strategies that partition data neck [5, 6, 41], directly impacting the efficiency and scalabilinto small blocks and overlap compression with transmission. ity of LLM training and inference. In such designs, excessive fragmentation can even amplify overheads, leading to degraded end-to-end performance. ...
...
...
Symbols
∗ This work does not relate to the position at Amazon.
1
Count
Symbols
Count
Symbols
Count
Symbols
Count
As shown in Figure 1a, this challenge is further exacerbated in collective communication. Libraries such as NCCL launch a single kernel per message to orchestrate computation and communication. Data is processed at a fine granularity, typically on the order of a few kilobytes per chunk, and staged through internal buffers to enable overlap. Directly inserting a standalone compression stage into this workflow introduces additional kernel launches and redundant memory traffic, which can negate the benefits of compression. In this work, we present UCCL-Zip, a unified design that integrates lossless compression directly into GPU communication primitives, thereby addressing the fundamental tension between compression overhead and communication efficiency. UCCL-Zip seamlessly supports both collective and P2P communication, enabling bandwidth reduction without modifying user-facing APIs or compromising numerical correctness. Our approach consistently improves effective bandwidth and, in many cases, approaches or exceeds the hardware-imposed network limit on modern GPU clusters. For P2P communication, Uzip-P2P employs a compressionaware split-send pipeline that aligns with the structure of floating-point compression (Section 3.2). As shown in Figure 1b, Uzip-P2P processes data in sufficiently large chunks to maintain high GPU utilization, and decouples compression stages to expose data that can be transmitted early. This allows a significant portion of the payload to be sent over the network before full compression is complete, effectively overlapping communication with compression. For collective communication, such as AllReduce, UzipNCCL integrates compression into NCCL’s persistent kernel execution model (Section 3.4). As shown in Figure 1c, we redesign the compression workflow by introducing localized frequency tables that enable block-wise, on-thefly entropy modeling, eliminating the need for global preprocessing. We then fully fuse compression into the compute–communication pipeline, avoiding extra kernel launches and reducing redundant HBM accesses. By aligning compression with the execution granularity of the collective loop, compression and communication proceed in a tightly coupled streaming fashion without introducing additional synchronization or kernel launches. UCCL-Zip is designed to be portable across heterogeneous hardware platforms. It supports multiple GPU architectures, including AMD and NVIDIA devices, and operates over diverse RDMA-capable interconnects such as NVIDIA CX7, Broadcom Thor-2, and AWS EFA NICs. In addition, UCCL-Zip extends DietGPU [34] to support float8 compression, enabling compatibility with a wide range of floating-point formats commonly used in modern machine learning workloads, including bfloat16, float16, float32, float8_e4m3fn, and float8_e5m2. This broad compatibility enables UCCL-Zip to be readily deployed in existing training and inference systems without requiring changes to data representation or communication backends.
Tensor Type Size (MB) Data Type Compression Ratio Gradient Activation Weight
214 16 107
FP32 BF16 BF16
0.848 0.679 0.675
Table 1. Compression ratios of representative tensors collected during GLM4-9B training [47]. We define the compression ratio as the ratio between the compressed size and the original (uncompressed) size; thus, lower values indicate better compression.
We evaluate UCCL-Zip across a diverse set of GPU clusters, including multi-node RDMA environments with EFA and RoCEv2 (Section 5). In reinforcement learning training, UzipP2P improves weight synchronization throughput by up to 47.5% on large tensors. In distributed LLM inference with vLLM, Uzip-NCCL reduces end-to-end inference latency by up to 10%. These benefits are achieved transparently, without modifying application code. We also reveal the architecture incompatibility of the widely-used NCCL with lossless compression, and identify concrete future directions in designing a better collective library (e.g., two-shot collectives and increasing chunk sizes). UCCL-Zip has been open-sourced at https://github.com/uccl-project/uccl/tree/main/p2p.
2
Background and Motivation
2.1
Basic GPU Lossless Compression Design
2.1.1 Statistical Structure of Floating-Point Values. Many existing works [17, 19, 34, 52] exploit the statistical properties of floating-point tensors to enable efficient lossless compression. Floating-point values consist of three fields: sign, exponent, and fraction (mantissa). Among these components, the fraction and sign bits typically exhibit nearuniform distributions and thus provide limited compression opportunities. In contrast, the exponent field often follows a narrow and highly skewed distribution in many machine learning workloads. This skew becomes more pronounced when tensors are normalized to a bounded range. Consider the bf16 format, which contains 1 sign bit, 8 exponent bits, and 7 fraction bits. When a bf16 tensor is normalized, the exponent field frequently collapses to a small number of repeated patterns. Such highly skewed symbol distributions are well suited for entropy coding techniques such as Asymmetric Numeral Systems (ANS) [3, 34, 37], enabling high compression ratios. Table 1 reports compression ratios of representative tensors from GLM4-9B [47] training. The ratio remains stable across tensors of the same type: float32 gradients achieve 0.85, while bfloat16 tensors consistently achieve around 0.68 across layers and parameter types. 2.1.2 GPU Compression Algorithm for Floating-Point Tensors. Prior work, such as DietGPU [34], leverages GPUs to accelerate compression of floating-point tensors. Exploiting the statistical structure described above, such approaches decompose each value into two components: the exponent 2
exponent (8bits)
fraction (7bits)
400 200
B M
64
B M
B M
32
M B
8
16
M B
1
0
KB
Throughput (GB/s)
Compress Decompress
✕ No OverLap
Kernel Launch Overhead
Compression Latency
Data Size
Figure 2. Overview and time breakdown of a typical GPU-based floating-point ANS compression pipeline (S1–S3 denote Steps 1–3).
600
25 6
B M
B 64
24% S3
B
0
M
Coalescing compressed data blocks
Compressed data block 62% S2
M
Data block
32
Step 3
Block-wise compression
14% S1
M B
Step 2
compressed part
20
16
Step 1
Time Breakdown
UCCL-P2P NCCL
40
M B
Raw floating point tensor Split the tensor in two parts uncompressed part
BFloat16
8
0
1
Fraction (7 bits) 0 0 1 0 0
KB
1
25 6
Exponent (8 bits) Sign 0 1 0 0 0 0 0 1 0
Throughput (GB/s)
0
Tensor Size
Data Block
(a) P2P throughput.
Time
Communication Kernel (NCCL)
Compression Kernel
(b) Compression throughput.
Figure 3. Throughput of P2P communication and DietGPU comGPU Persistent Ker Chunk 1(two Chunk 2 Chunk 3 pression on AWS p5en.48xlarge nodes, EFA).
NIC Idle NIC Compute NIC Sendand comfield and GPU the remaining bitsGPU (sign and fraction), Local Frequency Time S1 S2 S3 S1 S2 S3 S1 S2 S3 press Split onlyEncode the exponent field. This design improves comCoalesce Table ... compressed data design avoids global synchronization and enables Fused Steps in high hardpression efficiency compared Send to directly compressing raw Network Data Reduce Early-transmittable Compress compression Send ware utilization, but also limits coordination across CTAs (a) Naive approach: compress entire tensor then send floating-point representations. Figure 2 illustrates the overall Time Data (from S1) within a kernel. In particular, CTAs do not cooperatively Block1 of a typical GPU-based ANS compression pipeline, structure Send Blockof 1 three steps, each performing a single pass share intermediate statistics or synchronize frequently durwhich consists (b) Ideal conventional pipeline: split tensor into blocks ing execution, as such coordination would break the pipeline Time over global memory. Symbols 136 1 142 Symbols 136 142 parallelism and incur substantial overhead. Block 1 Block 2 Block 3 Block 4 Step 1: Float Splitting and Frequency Table ConstrucBitstream 100010001000100000000001 Bitstream 10001000100010000000 Send Block 1 Send Block 2 Send Block 3 Send Block 4 As a result, operations requiring global state across the tion. During the first pass, each value is decomposed into its Tensor Tensor (c) Realistic chunk from small blocks exponent and remaining bits pipeline: (sign overhead and fraction) (Figure 2), Time entire tensor (e.g., constructing a global frequency table for ... ... CTA CTA CTA CTA CTA CTA CTA send compressed part ANS encoding) are fundamentally incompatible with NCCL’s Coalesce whichSplit are Encode written to two contiguous buffers in global memSend uncompressed part execution model. Supporting such operations would require Synchronization Symbol Count ory: the compressed part (exponents) and the uncompressed across thread 1 3 kernel launch or explicit cross-CTA barLocalized Frequency Tab data each exposure8(d)bit LZipsegment split-send pipeline: data early part.Early Then in transmit the compressed part is Time either an additional blocks (CTAs) 142 1 Sample -> Build Table -> Enco required. pipeline ... continuity ... riers, both of whichisbreak and preclude interpreted as a symbol. During the split step, the kernel si(Same Kernel) Global Frequency Table tight integration with NCCL’s collective execution. multaneously collects symbol frequencies, constructing the global frequency table required by the ANS encoder. Recv Reduce Send Step 2: Independent Block-wise Compression. Once the 2.3 Challenges of Applying Lossless Compression to frequencyRecv table Compress is constructed, multiple GPU thread blocks Reduce Decompress Send GPU Communication perform compression independently using the shared global frequency table (Figure 2). Each thread block reads a chunk Challenge 1: Compression latency must not offset comof exponent symbols from global memory and applies ANS munication savings. The latency of compression can outencoding locally. Entropy coding produces variable-length weigh the savings from reduced data transfer. As we show, Compressed Data Receive outputs. Therefore, each Data block Reduce writes compressed a Originalofconventional approaches to hiding compression latency Compressed operations (e.g., sum) data Invalidto Data NCCL Send Original temporary globalReceive memory buffer instead of the final output & Decompression ten yieldReceive negligible gains or even degrade performance. Decom Receive NCCL NCCL with buffer. NCCL A naive design compresses all data prior to transmission, Receive Decompress Compress Send Compression with Step 3:Compression Coalescing Compressed Blocks. In the final step, assuming reduced data volume will improve throughput. (b) Ringareduce_scatter with compression. the variable-length outputs generated by different blocks are Consider representative example in RL training pipelines, ... ... merged into a single contiguous output buffer (Figure 2). The ... where model weights are transferred between training and ... ... GPU 0 GPU 1 GPU write 2 coalescing process introduces a third global memory rollout nodes via P2P send/recv. A typical tensor size is 16 MB GPU 0 Two-Shot reduce_scatter with compression. to assemble the final compressed stream. (e.g.,(c) bf16 tensors of shape [4096, 2048] in MoE models such as Qwen3-35B-A22B [4]). As shown in Figure 3, compressing a 16 MB tensor incurs ∼90 𝜇s of latency, while the reduced 2.2 NCCL Collectives Implementation data volume saves only ∼108 𝜇s in communication time, reModern GPU collective libraries such as NCCL [27] adopt a sulting in a marginal throughput improvement of ∼4%. channel-based execution model, where the input tensor is A natural mitigation is to partition tensors into chunks and partitioned into multiple chunks and processed by indepenpipeline compression with communication. However, this dent channels in parallel. Each channel is typically mapped to strategy is ineffective for GPU-based compression. Splitting a Cooperative Thread Array (CTA), which is responsible for a 16 MB tensor into four 4 MB chunks does not alleviate the moving and reducing a disjoint portion of the data through bottleneck. Each 4 MB chunk still incurs ∼70 𝜇s of compresthe communication topology (e.g., ring or tree) [20, 43]. sion latency, while the per-chunk communication savings To maximize overlap between communication and compushrink to only ∼20 𝜇s, resulting in a net performance degratation, NCCL decomposes communication into fine-grained dation. This asymmetry stems from GPU inner parallelism: slices that are pipelined across CTAs, with each CTA operatcompression latency does not scale linearly with data size, ing largely independently on its assigned data range. This as the GPU remains highly unutilized for small inputs. ...
Symbols
3
Count
Symbols
Count
Symbols
Count
S
Step 1 Step 2 Step 3
Challenge 2: Compression must be co-designed with GPU communication frameworks. Modern frameworks such as NCCL rely on persistent kernels and chunk-based pipelining, which impose fundamental barriers to integrating compression. First, existing compression approaches introduce global synchronization, conflicting with the persistent kernel design. Second, chunk-based pipelines require minimal data movement to sustain throughput; naive on-GPU compression introduces additional global memory copies, significantly degrading communication efficiency. Taken together, existing lossless compression methods are impractical when naively integrated into GPU communication stacks.
3
Design
3.1
Overview
Data block
NIC
Chunk 2
C
Time S1 S2 S3 S1 S2 S3 S1
14% S1
Network Send
Compressed data block 62% S2
Coalescing compressed data blocks
GPU
24% S3
GPU Compute
NIC Idle
NIC Send
Encode Coalesce Send compressed data (a) Naive approach: compress entire tensor then send
Time
(b) Ideal conventional pipeline: split tensor into blocks
Time
Block1 Send Block 1
Block 1
Block 2 Send Block 1
Block 3 Send Block 2
Block 4 Send Block 3
Send Block 4
(c) Realistic chunk pipeline: overhead from small blocks Split
Time
send compressed part Encode Coalesce Send uncompressed part
Early data exposure (d) Uzip-P2P split-send pipeline: transmit data early
Time
Figure 4. Comparison of pipelining designs for overlapping compression and communication. (b)(c) Chunk-based pipelining assumes latency size but is ineffective for GPU compresRecv scales with Reduce Send sion. (d) Split-send exposes transmissible data early and overlaps Recv Compress Reduce Decompress Send communication with remaining compression.
UCCL-Zip aims to address the above challenges when applying lossless compression to GPU communication. For P2P scenarios (Section 3.2), UCCL-Zip introduces a split-send pipeline that overlaps compression with network transmission. Unlike naive pipelining schemes, Uzip-P2P aligns pipeline stages with the internal steps of the compression algorithm. This stage-aligned design operates on large data blocks rather than fine-grained chunks, preserving GPU efficiency while exposing data that can be transmitted early in the compression process. Consequently, network transmission begins earlier and overlaps with subsequent compression steps, enabling faster bandwidth ramp-up and improved network utilization. For NCCL scenarios (Section 3.4), UCCL-Zip integrates compression into the NCCL data path via a fused kernel (Section 3.3). The kernel combines Steps 1 and 2 in Figure 2 using a redesigned frequency table aligned with NCCL’s execution model, reducing global memory traffic. Step 3 is eliminated via warp-level execution, directly writing compressed outputs to a GPU-resident FIFO. 3.2
compressed part
Block-wise compression
Split
Chunk 1
Breakdown
Split the tensor in two parts uncompressed part
early and overlapping communication with compute-heavy compression stages, split-send reduces communication stall time and improves end-to-end throughput. Compressed Data Reduce operations (e.g., sum) Invalid Data Receive & Decompress Original NCCL
Receive
NCCL with Compression
Receive
Send Decompress
Compress
compressed many times as sm
Send
GPU1 3.2.1 Key Properties of GPU Compression. Our designRingGPU0 reduce_scatter with com ... compressed once as a large b ... ... floatingis motivated by two key properties of GPU-based ... GPU 0 GPU 1 GPU 2 point compression. GPU0 GPU1 Property 1: Compression latency scales sub-linearlyTwo-Shot reduce_scatter with with data size. Due to massive GPU parallelism and kernel launch overheads, the latency of GPU compression increases much more slowly than the input size. Consequently, compressing small chunks may take nearly as long as compressing substantially larger ones. For example, on an NVIDIA H200 GPU, compressing a 16 MB floating-point tensor takes approximately 90 𝜇s, while compressing a 4 MB tensor takes approximately 70 𝜇s. Although the data size is reduced to 1/4, the latency decreases modestly by only 22%. This behavior indicates that splitting tensors into many small chunks can significantly reduce GPU compression efficiency. Property 2: Early availability of transmissible data. The first step of the compression pipeline accounts for 14% of the total compression time but already exposes a large portion of the final representation (Figure 2). The first stage of the compression pipeline performs a structural split of the input tensor into two components: an uncompressed part and a compressed part. The ratio between the two portions depends on the floating-point format. For example, for bf16 tensors, the uncompressed and compressed portions each account for approximately half of the tensor. For float32, the uncompressed portion occupies roughly three quarters of the data. Since the compressed portion will be further reduced by the subsequent compression stages, the uncompressed portion constitutes an even larger fraction of the final transmitted data. Importantly, once the split stage completes, the uncompressed portion requires no further computation and can therefore be transmitted immediately.
Split-Send Communication Pipeline
To hide compression overhead in the send/recv path, we design a split-send communication pipeline that overlaps compression with network transmission. Unlike conventional chunk-based pipelining that partitions tensors into independent blocks, our design aligns pipeline boundaries with the internal steps of the compression algorithm. Split-send is motivated by the observation that the initial stage of the compression pipeline (Step 1 in Figure 2) incurs relatively low latency, while subsequent stages are more compute-intensive. Once Step 1 completes, UCCL-Zip immediately initiates transmission of the corresponding uncompressed data. This early transmission overlaps with the execution of later compression stages (Step 2 and Step 3), which dominate the overall compression cost. After compression completes, only a small volume of compressed data remains to be transmitted. By exposing transmissible data 4
Symbols 136 136 142 142 1 Symbols Symbols136 136142 142 1 1 Symbols 1 1000100010001000 00000001Bitstream Bitstream 1000100010001000 00000001 Bitstream 1000100010001000 00000001 1000100010001000 00000001 Bitstream Tensor Tensor
... CTA CTA CTA CTA
... CTA CTA CTA CTA
Synchronization Synchronization Symbol CountCount Symbol across threadthread 1 across 1 3 3 blocks (CTAs) 142 blocks (CTAs) 142 1 1 is required. ... ... ... ... is required. Global Frequency Table Global Frequency Table
Symbols
SymbolsCountCount Symbols
Symbols
Count
... CTA CTA CTA CTA Symbols
Count
Symbols Count Count Symbols
Symbols
Count
(b) Localized frequency tables.
1.00 0.75 0.50
M B
M B
64
B
B
M B
16
25
6K B
0.00
Global Frequency Table Localized Frequency Tables
8M
0.25
1M
Compression Ratio
Count
Localized Frequency TablesTables Localized Frequency Sample -> Build Encode Sample -> Table Build -> Table -> Encode (Same(Same Kernel)Kernel)
(a) Global frequency table construction.
3.2.3 Split-Send Pipeline Design. Based on these observations, we design a split-send communication pipeline that overlaps compression with communication while preserving large-block GPU efficiency. After the first compression stage completes, we divide the tensor into two components:
Tensor Size
(c) Compression ratio. Figure 5. Localized frequency tables eliminate global coordination and enable a fully fused compression pipeline. (a) A global frequency table requires cross-CTA synchronization, preventing kernel fusion and introducing additional memory passes. (b) With localized tables, each CTA independently samples and constructs its own table, enabling a fused pipeline within a single kernel without synchronization. (c) This design preserves compression effectiveness.
• Uncompressed portion. This portion is finalized after the first stage and can be transmitted immediately. • Compressed portion. The remaining data continues through the subsequent compression stages before transmission.
a single kernel and incurs an additional round of memory access. We design localized frequency tables (Figure 5b) to enable kernel fusion with a comparable compression ratio and lower synchronization overhead. Instead of maintaining a shared global frequency table, each thread block constructs its own local table by sampling a small portion of its assigned data range (e.g., the first 256 KB). This localized sampling captures the exponent distribution without a full traversal, enabling fusion into a single kernel and reducing global memory accesses to one pass. Local frequency estimation preserves compression effectiveness while eliminating synchronization overhead. Because exponent distributions in neural network tensors exhibit stable statistical structure and are concentrated [14], tables constructed from sampled subsets of each block approximate the global distribution well, achieving compression ratios close to the global-table baseline while removing an entire memory pass. This introduces only a minor approximation: it may not fully capture rare exponent values, but still preserves near-optimal coding efficiency. As shown in Figure 5c, on real tensor data, this design incurs only about a 4.5% reduction in compression ratio, and this small cost remains consistent across different tensor sizes. Each block then uses its local table for both compression and decompression, eliminating the need for global frequency table lookups or cross-block synchronization.
As shown in Figure 4 (d), while the GPU processes the remaining compression stages, the network interface concurrently transmits the uncompressed portion. Once compression finishes, the smaller compressed payload is transmitted. This pipeline exposes data early and o‘verlaps compression with network communication without sacrificing GPU throughput. 3.3
TensorTensor
... CTA CTA CTA CTA
32
3.2.2 Limitations of Chunk-Based Pipelines. Conventional pipelining splits tensors into independent chunks to reduce the latency of producing the first transmittable byte and to overlap computation with communication. As shown in Figure 4 (b), this approach assumes that compression latency decreases proportionally with data size. However, this assumption does not hold for GPU compression. Compression involves multiple GPU kernels and memory transformations whose overheads do not scale with input size. Consequently, compressing a small chunk may incur nearly the same latency as compressing a large tensor while significantly reducing overall GPU throughput. As shown in Figure 4 (c), naïve chunking can therefore degrade performance in compression–communication pipelines.
Fused Compression Kernel
To integrate compression into NCCL efficiently, we fuse the multi-step compression pipeline (in Figure 2) into a single fused kernel execution. The first two steps are merged through localized frequency tables, while the third coalescing step is eliminated by directly streaming compressed outputs into the communication pipeline. This design simplifies integration with NCCL primitives and improves performance by reducing global memory traffic and avoiding extra kernel launch overhead. 3.3.1 Localized Frequency Tables. The original design (Section 2.1), such as in DietGPU, builds a global frequency table by scanning the entire tensor (Figure 5a), which is incompatible with NCCL’s channel-based execution model. This is because NCCL CTAs process disjoint data partitions independently without synchronization, preventing efficient global coordination. As a result, global frequency table construction prevents fusing the first two stages in Figure 2 into 5
compress
Send Send
Compressed Data
3.3.2 Eliminating the Third Coalescing Step. We furoptimize the pipeline by having each thread block write perations (e.g., sum) ther Invalid Data Receive & Decompression Send compressed outputs directly into NCCL’s FIFO buffer, elimCompress inating Send the third coalescing step in the original workflow (Figure 2). NCCL internally maintains staging buffer, where (b) Two-Shot AllReduce withacompression. user data is first copied before being transmitted over the ... network. ... In our design, we directly write the compressed out... PU 1 GPU 2produced in Step puts 2 (Figure this NCCL-managed (c) Ring AllReduce2) withinto compression. buffer. This enables a fully fused single-kernel implementation, reducing global memory accesses from three passes to one and avoiding multiple kernel launches. 3.4
Reduce operations (e.g., sum)
Receive
Original NCCL
Receive
NCCL with Compression
Invalid Data Send
Decompress
Compress
Send
... ...
... GPU 0
GPU 1
...
GPU 2
Figure 6. Overview of NCCL all_reduce with compression. The original NCCL pipeline performs receive–reduce–send operations per slice. We extend it by integrating decompression and compression into the dataflow.
Compression-Integrated NCCL Pipeline
process data segments and issue communication operations without global synchronization. The warp-level structure also preserves architectural portability: because NCCL collectives are already implemented as warp-cooperative kernels, our approach requires no architecture-specific tuning across GPU generations. In practice, compression executes within the same kernel launch as reduction and communication, enabling effective overlap between computation and data transfer while utilizing otherwise idle SM cycles. Amortizing compression metadata overhead. To amortize metadata overhead, we transmit the ANS table only once per collective invocation. Float tensors typically exhibit stable value distributions across communication steps, rendering per-block table recomputation unnecessary. We therefore track the initialization state of compression tables and transmit the table only during the first step of the collective, reusing it thereafter. This design renders metadata overhead negligible for large-scale collectives. Intra- vs. inter-node communication. For intra-node communication, compressed data is transmitted directly between GPUs via NVLink. For inter-node communication over RDMA, as well as intra-node transfers over PCIe, compressed outputs are written to a GPU-resident FIFO buffer, which is consumed by the host networking stack and forwarded through the network interface. This design eliminates additional data copies between compressed buffers and NCCL buffers, while preserving compatibility with NCCL’s existing transport abstractions. Selective compression across collective stages. Not all data in collective operations benefit equally from compression. For point-to-point primitives such as send and recv, we always apply compression before transmission and decompression upon reception. However, multi-step collectives such as all_reduce require finer control because data passes through several intermediate aggregation stages, each potentially involving multiple sources and destinations. We therefore apply compression selectively: on the sender side, compression is performed only when a GPU transmits data to another node or writes to a remote FIFO buffer. During intermediate reduction steps that combine locally available data with received partial results, only the remote data
We integrate the fused compression kernel directly into NCCL’s collective communication pipeline, eliminating extra launches and operating directly on data in registers or shared memory. This design allows compression, communication, and decompression to execute in a tightly overlapped pipeline without requiring any changes from user applications. Figure 6 illustrates the workflow of NCCL all_reduce with compression. In the original NCCL pipeline, each GPU performs a sequence of receive–reduce–send steps for every data slice. Our design extends this pipeline by inserting compression and decompression stages while preserving the streaming execution model. Specifically, data is compressed at the sender, transmitted in compressed form, and decompressed at the receiver before reduction. The reduced result is then recompressed and forwarded to the next GPU in the ring. For example, GPU 1 receives compressed slice 𝑎 0 from GPU 0, decompresses it, reduces it with local slice 𝑏 0 to produce 𝑎 0 + 𝑏 0 , and then recompresses and forwards the result to the next GPU 2. Integration into the NCCL data path. We integrate compression directly into NCCL’s data processing layer, where collective primitives such as all_reduce, all_to_all, and send/recv are implemented. The key modification occurs in the stage previously handled by CopyReducePacks, the core routine responsible for processing fixed-size data chunks during collective execution. This routine loads elements from source buffers, applies the required reduction operator (e.g., sum, min, or max), and writes the results back to global memory. We fuse compression and decompression into this stage, allowing data to be compressed immediately before transmission and decompressed upon arrival. By eliminating intermediate memory transfers between computation and communication, compressed data can be transmitted directly over NVLink or RDMA, reducing memory traffic and improving pipeline efficiency. Warp-level execution and compatibility. Compression and communication are executed at warp granularity, with each warp responsible for processing one compression or decompression block. This design matches NCCL’s warpcooperative execution model, allowing independent warps to 6
are decompressed and merged, while local GPU data remain uncompressed. This policy minimizes unnecessary computation while ensuring that all cross-node traffic is transmitted in compressed form. Similarly, on the receiver side, decompression is triggered only for data that originates from remote peers, while final output paths bypass decompression entirely. Handling data alignment. NCCL processes tensors in fixed-size chunks. To simplify integration and avoid partialblock handling overhead, compression is applied only to chunk-aligned regions (e.g., 32 KB). Any remaining tail elements are transmitted in uncompressed form. This selective strategy preserves correctness across heterogeneous message sizes while avoiding additional synchronization or control logic.
4
UCCL-P2P’s support for diverse RDMA-capable NICs (e.g., EFA and InfiniBand) to enable seamless deployment across heterogeneous GPU–NIC combinations without requiring application changes. We replace PTX-dependent components in DietGPU with portable, HIP-compatible implementations to support AMD GPUs while preserving performance-critical optimizations. We extend DietGPU [34] to support multiple floatingpoint formats. Uzip-P2P supports the primary data types used in modern LLM, including float16, float32, bfloat16, float8_e4m3fn, and float8_e5m2. In particular, we extend support to both FP8 formats. We pack two FP8 values into a single 16-bit unit and jointly extract their exponent fields, producing an 8-bit exponent stream for compression. This design enables byte-granular writes in the split stage and avoids memory misalignment overhead. We employ a singleton Compressor to orchestrate the compression pipeline. The compressor maintains GPU-resident buffers for both compression and decompression. It exposes a set of modular primitives corresponding to distinct stages of the compression workflow. Given that UCCL-P2P uses a single send thread and a single receive thread per GPU, we adopt a single compressor instance per GPU to serve all operations, thereby avoiding redundant buffer allocation and reducing memory overhead. Combined with chunk-based transmission for large messages, this design bounds the total memory footprint to ∼164 MB per GPU, which accounts for only about 0.1% of HBM capacity on an H200 GPU.
Implementation
We integrate Uzip-P2P into two GPU communication systems, UCCL-P2P [46] and NCCL [27], referred to as UzipP2P and Uzip-NCCL, respectively. Uzip-P2P supports both NVIDIA and AMD GPU platforms, enabling cross-vendor applicability. Specifically, we extend NCCL v2.23.4 with approximately 1.8 K lines of CUDA code to incorporate compression, and integrate compression into UCCL-P2P with around 4 K lines of C++ code. Notably, our modifications are lightweight and modular, and can be readily ported to RCCL (the AMD counterpart of NCCL) with minimal effort. In both systems, the integration is fully transparent to user applications. 4.1
4.2
NCCL Compression Implementation
Uzip-NCCL integrates UCCL-Zip into NCCL v2.23.4 [27] through lightweight modifications to the collective communication pipeline. The implementation adds 1.8K lines of CUDA to NCCL, embedding compression directly into collective kernels to enable transparent support for Ring all_reduce, all_to_all, and send/recv without API changes. Uzip-NCCL performs compression entirely within the collective kernels using a shared-memory workspace. Each kernel allocates a contiguous scratch region to store intermediate state required by the compression pipeline, including histograms, probability tables, symbol buffers, and encoded outputs. The workspace is logically partitioned into sender-side compression buffers and receiver-side decompression buffers. During transmission, the sender constructs histograms, generates ANS encoding tables, and produces compressed blocks, while the receiver reuses the same region to reconstruct decoding tables and decompress incoming symbols. This design enables fully in-kernel compression and decompression, eliminating additional global memory staging and auxiliary kernel launches. Uzip-NCCL integrates compression into the collective data path via a warp-parallel pipeline that operates on fixed-size blocks in each iteration of the collective loop. Each value is decomposed into a non-compressible component (sign and
Peer-to-Peer Communication Implementation
We build Uzip-P2P on UCCL-P2P, providing synchronous and asynchronous send/recv with transparent compression. UCCL-P2P offers easy-to-use P2P APIs and achieves performance comparable to or exceeding NCCL/RCCL [2, 27], NVIDIA NIXL [39], and Mooncake TE [35]. Our design requires no modifications to the user-facing API. Compression changes the effective size of transmitted data, which requires modifications to the metadata management in UCCL-P2P. UCCL-P2P is built on top of the RDMA write_with_imm primitive provided by RDMAcapable NICs, which enables one-sided remote memory writes without CPU involvement on the receiver side. This operation requires the sender to obtain the remote memory address and size in advance. UCCL-P2P allocates a fixed-size metadata buffer at initialization to coordinate data addresses and sizes for write_with_imm during send and receive. To support compression, Uzip-P2P extends the metadata to include additional information such as the data type and the sizes before and after compression, enabling correct reconstruction and efficient transmission of compressed data. We extend the DietGPU [34] compression implementation to support both NVIDIA and AMD GPUs, and leverage 7
B
B
1G
8M 12
B
B
8M
1M
Throughput (GB/s)
B
B
B
1G
8M
(b) Uzip-NCCL P2P throughput.
Tensor Size
(a) all_to_all throughput.
2
0
Uzip-NCCL NCCL
B
1
2G
1G
8M B
0
B
Uzip-NCCL NCCL
3
8M B 32 M B 12 8M B 51 2M B
2
Throughput (GB/s)
4
4
2M B
6
12 8M B
Testbeds. We evaluate UCCL-Zip across diverse GPU–NIC platforms, including two p5en.48xlarge instances connected via Elastic Fabric Adapter (EFA), a two-node AMD cluster with AMD Instinct MI355X GPUs connected via RoCEv2, and a single-node AWS g6e.48xlarge instance with 8 NVIDIA L40S GPUs. By default, Uzip-P2P is evaluated on the p5en.48xlarge cluster, while Uzip-NCCL is evaluated on the g6e.48xlarge instance. Hardware specifications are summarized in Table 2. Baselines. We compare against multiple baselines. For NCCL with compression, we use NCCL v2.23.4 with its default configuration as the baseline. For fair comparison, NCCL is restricted to 4 SMs, matching Uzip-NCCL. For P2P communication, we use UCCL-P2P [46] with its default synchronous send/recv interface as the baseline, as Uzip-P2P is built directly on top of it. We compare Uzip-P2P against two designs: (1) a naive encode-send scheme that transmits only after full compression, and (2) a conventional native pipeline that partitions data into 8 MB chunks and overlaps compression with communication. We adopt an 8 MB chunk size, as it empirically provides a reasonable balance between compression effectiveness and pipeline overhead. Compression is enabled only for messages larger than 1 MB, as overhead outweighs benefits for smaller messages. We use bfloat16 as the default data format. 5.2
12
6K 25
Experiment Setup
0
Figure 7. Throughput comparison for bfloat16 peer-to-peer communication across tensor sizes. Throughput (GB/s)
5.1
Uzip-NCCL NCCL
Tensor Size
(a) Uzip-P2P P2P throughput.
12 8K B
Evaluation
2
Tensor Size
exponent), which is transmitted directly, and a compressible mantissa symbol used for encoding. Threads cooperatively build histograms over mantissa symbols to construct ANS probability tables, after which each warp independently encodes its local symbols into a variable-length compressed stream. The resulting compressed data and minimal metadata are written directly into the communication buffers, replacing the original data blocks in the collective pipeline.
5
B
0
4
8K
25
6
12
50
B
Table 2. Hardware specifications of our experimental platforms.
UCCL-P2P Uzip-P2P
8M
AWS EFA 8×H200 Xeon 8488C (192) 16×EFA (200 Gbps) AMD 8×MI355X EPYC 9575F (128) 8×RDMA NICs (400 Gbps) AWS g6e 8×L40S 192 vCPUs Ethernet (400 Gbps)
75
B
Network
1M
CPU (#cores)
1M B
GPU
Throughput (GB/s)
Cluster
Tensor Size
(b) Ring all_reduce throughput.
Figure 8. Throughput of Uzip-NCCL collective communication primitives across varying sizes.
approaching the theoretical upper bound of 73.8 GB/s derived from Amdahl’s Law under a 64% compression ratio. For smaller tensors, the benefits are more modest (e.g., 8% at 16 MB and 24% at 32 MB), as compression overhead partially offsets bandwidth savings. We generate bf16 tensors with values uniformly distributed in [−1, 1], achieving a stable compression ratio of approximately 64% across all tensor sizes. For small messages (e.g., 4 MB), compression overhead cannot be fully amortized. However, in practical workloads such as RL weight updates and KV cache transfers, tensor sizes typically exceed this threshold, making Uzip-P2P effective in practice. 5.2.2 Uzip-NCCL Performance. We evaluate the performance of Uzip-NCCL on two representative collective operations widely used in NCCL: all_to_all and all_reduce. We also evaluate the point-to-point send/recv primitive. The Uzip-NCCL consistently outperforms NCCL for send/recv across all message sizes (Figure 7b), achieving over 20% higher throughput for messages larger than 32 MB. A similar trend is observed for the all_to_all collective (Figure 8a), which effectively consists of multiple send/recv operations. Due to additional GPU scheduling and synchronization overheads, the performance gain is slightly reduced, reaching about 18% for large messages (>32 MB). Uzip-NCCL performs worse than NCCL for the all_reduce (Figure 8b). This is because NCCL adopts a ring-based all_reduce algorithm, which is inherently unfriendly to compression integration. all_reduce consists
End-to-End throughput
5.2.1 Uzip-P2P Performance. Figure 7a reports the throughput of our Uzip-P2P compression pipeline across tensor sizes from 256 KB to 1 GB. As tensor size increases, Uzip-P2P delivers progressively higher performance gains over UCCL-P2P. For medium-to-large tensors (>8 MB), Uzip-P2P consistently outperforms the UCCL-P2P. At 1 GB, it improves throughput by up to 52.9% (72.2 GB/s vs. 47.2 GB/s), closely 8
GPU1
GPU2 Reduce
Decompress
Send
compressed few times as large blocks
GPU0 GPU2
GPU1 (c) Two-Shot reduce_scatter with compression. Compressed Data
Reduce operations (e.g., sum)
Invalid Data
Tensor Size
(a) Two-Shot throughput.
all_reduce
compressed once as a large batch
vs.
compression_ratio Decompress
Compress
0.5
25 ...
0
roj roj roj o0_p B) p_p B) q_pGPU GPU B) 1 ttn. ttn. e_u gat (214 M self_a (32 M self_a (32 M
0.0
Send
... ... GPU 2
Weights Tensor
(a) Weight update throughput for GLM4-9B.
Ring
Figure 9. Throughput of two-shot all_reduce implemented with asynchronous isend/irecv on two p5en.48xlarge instances (16 GPUs total) with NVLink disabled.
of two phases: reduce-scatter and all-gather. The all-gather phase involves only data movement without computation, and thus exhibits similar compression behavior across different implementations (e.g., ring-based and two-shot). In contrast, the reduce-scatter interleaves communication with reduction operations (e.g., sum or average), which imposes stricter constraints on compression. Each received data chunk must be decompressed before the reduction. As shown in Figure 9b, the ring-based implementation partitions data into small chunks and propagates them along a logical ring topology. Each chunk undergoes multiple rounds of compression, transmission, decompression, and reduction as it traverses the ring. This repeated compression–decompression cycle incurs significant overhead, reducing the effectiveness of compression in NCCL all_reduce. Instead, the two-shot all_reduce design significantly reduces this overhead. Each GPU compresses its data only once as a large chunk or batch before transmission, and the receiving GPU performs a single decompression prior to reduction (Figure 9b). We implement a two-shot all_reduce using asynchronous isend/irecv primitives and integrate UCCL-Zip into the communication path (Figure 9a). Unlike NCCL’s ring-based all_reduce, which partitions data into fine-grained per-CTA slices, our implementation operates on large contiguous chunks. This coarse-grained design, combined with the reduced number of compression invocations inherent to the two-shot approach, allows compression to be applied more efficiently. UCCL-Zip Two-Shot begins to deliver performance benefits once the data size reaches 32 MB, achieving a 13.3% throughput improvement over the baseline two-shot implementation. The gain continues to increase with larger message sizes, reaching up to 35.7% at 1 GB.
5.3
50
j pro ) wn_ .do 07 MB p l m (1
GPU0 GPU2 GPU1 Two-Shot reduce_scatter with compression.
(b) Two-Shot all_reduce.
75
Send
1.0
75
UCCL-P2P
Uzip-P2P
compression_ratio
50
1.0
0.5
25 0
z v oj oj roj roj_ ) _pr ) _pr ) j_qk ) t_p B) n_p pro B B B B .ou tn.o tn.q .in_ (32 M _attn.i (16 M r_attn (16 M elf_at (16 M elf_at (32 M r s s a attn a _ e e r lin lin nea
li
0.0
Compression Ratio
2M B 1G B
...
51
B M
8M 12
GPU 1 0
B
... ... GPU 2
GPU0 GPU2 GPU1 Ring reduce_scatter with compression.
Original NCCL UCCL-P2P Uzip-P2P Receive NCCL with Compression
Compression Ratio
Loop back to GPU0
20 10
Receive & Decompression compressed many times as small blocks
Throughput (GB/s)
30
Invalid Data Send
NCCL (Ring) Uzip Two-Shot Compress Send Two-Shot
32
Weights Tensor
(b) Weight update throughput for Qwen3.5-35B-A3B. Figure 10. Application-level evaluation of Uzip-P2P on bf16 weight tensors during RL training. (a) GLM4-9B and (b) Qwen3.5-35B-A3B. Latency (ms)
U0
40
B
Decompress
Throughput (GB/s)
Reduce operations (e.g., sum)
8M
d Data
Throughput (GB/s)
Receive
NCCL Uzip-NCCL
2000 1000 0
80
76
0 24
10
0 36
15
0 0 0 00 60 20 80 24 25 51 76 10 Input Token
Figure 11. KV cache transfer latency under prefill–decode disaggregation (P1D3) in vLLM for the Qwen-7B-Chat model.
two representative models: the dense GLM4-9B (9B parameters) and the mixture-of-experts (MoE) model Qwen3.535B-A3B (35B parameters). The training pipeline runs on 8 GPUs, where 4 GPUs perform policy optimization and the remaining 4 GPUs generate rollouts. Figure 10 presents the communication throughput across representative weight tensors from different layers and training stages. The x-axis lists tensor names and sizes. For the dense GLM4-9B model (Figure 10a), weight tensors are relatively large, enabling substantial communication reduction. In particular, Uzip-P2P achieves up to 47.5% higher throughput for the gate_up_proj tensor (214 MB). For the MoE model Qwen3.5-35B-A3B (Figure 10b), tensors are smaller due to sparse expert activation. Despite this, Uzip-P2P still delivers up to 28.8% improvement for the self_attn.q_proj.weight tensor (32 MB). For smaller tensors (e.g., 16 MB), Uzip-P2P maintains around 10% improvement. Finally, the compression ratio remains stable across models, layers, and training stages. As it directly determines communication reduction, this stability indicates that our lossless design generalizes well to diverse LLM training workloads.
Application-Level Performance
5.3.1 Weight Updates in Reinforcement Learning. We evaluate Uzip-P2P on weight tensors generated during the update phase of LLM reinforcement learning. We consider 9
...
1G B
B 8M 12
B 8M
1M
B
0
6K B
Throughput (GB/s)
25
25
1G B
B 8M
Tensor Size
(a) Impact of buffer memory footprint.
25
B 1G
12
8M
B
0
B
KB
Tensor Size
50
UCCL-P2P 100% SM 75% SM 50% SM
8M
0
Throughput (GB/s)
UCCL-P2P Chunked Uzip-P2P Uzip-P2P
25
75
6K B 1M B
50
6
Tensor Size
(b) Compression ratios across floating-point types.
Figure 13. Performance of Uzip-P2P across different floating-point data types.
Tensor Size
(b) Impact of SM utilization constraints.
Figure 16. Throughput of Uzip-P2P under constrained GPU resources.
5.3.2 KV Cache Transfer in Prefill–Decode Disaggregation. We integrate Uzip-NCCL with the Prefill–Decode disaggregation inference pipeline of vLLM [28] to evaluate its performance in realistic distributed LLM serving workloads without application changes. Experiments follow the default Prefill–Decode disaggregation configuration (P1D3) in vLLM, where one GPU performs the prefill stage and three GPUs execute decoding. We measure the KV cache transfer latency. Figure 11 shows that Uzip-NCCL consistently reduces KV cache transfer latency compared to NCCL, achieving up to 30.1% improvement. Since KV cache transmission constitutes a significant portion of the inference latency, these improvements translate directly into end-to-end performance gains. For example, when the input length is 7,680 tokens, KV cache transfer accounts for approximately 23% of the total execution time, implying an overall application speedup of about 10% over the NCCL.
5.4
50
25
Throughput (GB/s)
B 1G
6M B
float8_e4m3fn float8_e5m2
25
M B
0.00
64
0.25
M B
Compression Ratio
0.50
75
25
Tensor Size
(a) Uzip-P2P throughput across floating-point types.
0.75
bfloat16 float16 float32
UCCL-P2P encode-send Uzip-P2P naive pipeline
Figure 14. Uzip-P2P through- Figure 15. Throughput of put comparison with NCCL compression–communication send/recv. integration strategies.
1.00
16
B 1G
8M 12
8M
25
B
0
B
float8_e4m3fn float8_e5m2 UCCL-P2P
25
6K B 1M B
Throughput (GB/s)
50
bfloat16 float16 float32
75
Tensor Size
Figure 12. Communication throughput for different versions of the gate_up_proj weight tensor (214 MB) during GLM4-9B RL training. 75
0
12
0.0
100
1M
60 80 Weights Version
8 M 32 B M 64 B M 12 B 8 M 51 B 2 M B 1 G B
40
6K B
20
25
25
0
50
B
0.5
NCCL UCCL-P2P Uzip-P2P
B
50
75
8M
1.0
compression_ratio
Throughput (GB/s)
Uzip-P2P
Compression Ratio
Throughput (GB/s)
UCCL-P2P
5.4.2 Performance Across Different Floating-Point Data Types. We evaluate Uzip-P2P across different floatingpoint formats, including float16, float32, float8_e4m3fn, and float8_e5m2 (Figure 13). The achievable compression ratio is primarily determined by the floating-point exponent field, as our scheme encodes only exponent values. In IEEE formats, float16 uses 5 exponent bits out of 16 bits, while float32 uses 8 exponent bits out of 32 bits. Consequently, both formats offer limited compression opportunities compared to bfloat16, which retains an 8-bit exponent within a 16-bit representation. This trend is reflected in Figure 13b, where the compression ratios for float16, float32, bfloat16, float8_e4m3fn, and float8_e5m2 are approximately 83%, 82%, 64%, 77%, and 70%, respectively. These differences translate into performance gains in Figure 13a. Uzip-P2P achieves up to 41.9% throughput improvement for float8_e5m2, compared to 30.2% for float8_e4m3fn over UCCL-P2P.
Ablation Studies
5.4.1 Transmission Throughput and Compression Ratio Across RL Steps. We examine compression behavior during reinforcement learning training by measuring throughput and compression ratio for different versions of the gate_up_proj weight tensor (214 MB) from GLM4-9B. As shown in Figure 12, the compression ratio remains stable across training checkpoints, close to that of randomly generated tensors. Correspondingly, Uzip-P2P consistently outperforms the UCCL-P2P, achieving stable throughput gains throughout training. 10
5.4.3 Comparison with NCCL send/recv. We further compare baseline UCCL-P2P implementation and Uzip-P2P against the NCCL send/recv interface. The UCCL-P2P implementation achieves performance comparable to NCCL for tensors larger than 8 MB and slightly outperforms NCCL for smaller messages (Figure 14). By incorporating compression, Uzip-P2P further improves throughput and consistently outperforms NCCL across the entire range of tensor sizes.
B M
6 51 MB 2M B 1G B
25
B
0
M
B 0M
100
64
B 0M
Tensor Size
40
B
10
M 10
1M
25
B
0
by the architecture incompatibility of NCCL with lossless compression: fine-grained chunking causes compression/decompression inefficiency. Addressing this issue would require faster compression algorithm designs and larger compression granularity to better utilize GPU hardware. A promising direction is to develop GPU-friendly lossless compression algorithms with more efficient kernel implementation, potentially leveraging Tensor Cores or emerging hardware support for compression primitives [49]. Building a Compression-Friendly Collective Library. Results in Figure 9 indicate that reducing the number of compression invocations is critical to amortizing compression overhead. Our two-shot all_reduce demonstrates the feasibility of compression-aware collective design by minimizing redundant compression and decompression. However, our current implementation is built on NCCL isend/irecv, where reduction and compression remain loosely coupled, incurring additional kernel launches and memory copy overheads. As future work, we plan to build a compressionfriendly collective communication library that tightly integrates compression into collective primitives.
4 SMs 16 SMs 64 SMs
200
B
25
Uzip-NCCL NCCL
16
Throughput (GB/s)
50
300
4M
UCCL-P2P Uzip-P2P
6K B
Throughput (GB/s)
75
Tensor Size
Figure 17. Uzip-P2P through- Figure 18. Uzip-NCCL P2P put on AMD MI355X over Ro- throughput over NVLink on a CEv2. single p5en.48xlarge node.
5.4.4 Comparison with Naive Methods. We compare Uzip-P2P against two alternative designs for integrating compression with communication. Figure 15 shows that UzipP2P consistently outperforms both alternatives across tensor sizes. For small tensors (e.g., 8 MB), compression overhead dominates, and encode-send degrades throughput by 18% relative to the UCCL-P2P, while Uzip-P2P reduces this to 6% via partial overlap. For larger tensors, Uzip-P2P outperforms encode-send by effectively overlapping compression and communication. The naive pipeline slightly underperforms the UCCL-P2P, as fine-grained chunking incurs excess compression overhead that outweighs pipelining benefits. 5.4.5 Performance under Resource Constraints. We evaluate the robustness of Uzip-P2P under constrained GPU memory and limited SM availability. Memory Footprint. Uzip-P2P incurs additional GPU memory overhead for staging compression buffers, scaling with block granularity. To reduce memory usage, large tensors can be partitioned into smaller chunks at the cost of extra kernel launches. Despite this trade-off, Uzip-P2P remains effective under tight memory constraints: with a 164 MB buffer per GPU (approximately 0.1% of the 141 GB HBM on H200), chunked Uzip-P2P still achieves up to 41.2% throughput improvement for 128 MB tensors (Figure 16a). SM Utilization. We evaluate Uzip-P2P under restricted SM availability using CUDA Green Context [10]. As shown in Figure 16b, Uzip-P2P achieves up to 20.4% improvement even with 50% of SMs. Uzip-P2P leverages otherwise idle SMs during communication, as dependencies often prevent LLM communication from fully overlapping with computation.
7
GPU Communication Library. Prior work has improved GPU communication efficiency for both P2P and collective operations. Systems such as NIXL [39], Mooncake [35], pplxgarden [33], and UCCL-P2P [46] focus on high-performance P2P, while NVSHMEM [29], NCCL [27], and RCCL [2] provide both collective primitives and P2P abstractions. More recent efforts, including UCCL [54] and MSCCL++ [43], optimize collective communication at the network and scheduling layers. However, none of these systems natively support compression within the communication pipeline. Lossless Compression Algorithms on GPUs. Existing work explores a variety of lossless GPU compression techniques. ZipNN [19] employs Huffman-based entropy coding, while NeuZip [17], DFloat11 [52], TDT [26], and DietGPU [34] adopt ANS-based designs optimized for GPU execution. MPC [51] targets massively parallel architectures, and ZipServ [12] proposes hardware-aware compression with fixed-length encoding. Earlier efforts such as Deep Compression [16] apply Huffman coding to quantized models, while nvCOMP [37] provides general-purpose GPU compression. However, existing approaches largely optimize compression in isolation, without integrating it into GPU communication. GPU Communication Library with Compression. Existing work reduces GPU communication volume via quantization and other lossy compression techniques. Systems such as C-Coll [23], ZCCL [24], gZCCL [22], OmniReduce [13], and ghZCCL [21] apply error-bounded compression to accelerate collectives. Zhou et al. [53] extend MVAPICH2 with both lossless and lossy compression for HPC workloads, while numerous frameworks [1, 7, 8, 25, 30, 32, 44, 45, 48, 50]
5.4.6 Uzip-P2P Throughput on AMD GPUs and NICs. We evaluate Uzip-P2P on an AMD cluster with MI355X GPUs and Pensando DSC SmartNICs. Figure 17 shows that UzipP2P consistently improves end-to-end P2P throughput across tensor sizes, achieving up to 45.3% higher throughput at large sizes and generalizing across GPU and NIC vendors.
6
Related Work
Discussion
Applying Compression to NVLink Communication. We acknowledge that Uzip-NCCL over NVLink gives negative gains, shown in Figure 18. This is essentially caused 11
adopt lossy compression for training and inference. While effective, these approaches introduce approximation error.
8
2025. Optimizing SLO-oriented LLM Serving with PD-Multiplexing. arXiv:2504.14489v1 [cs.OS] [11] DeepSeek-AI, Aixin Liu, Bei Feng, et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] [12] Ruibo Fan, Xiangrui Yu, Xinglin Pan, Zeyu Li, Weile Luo, Qiang Wang, Wei Wang, and Xiaowen Chu. 2026. ZipServ: Fast and MemoryEfficient LLM Inference with Hardware-Aware Lossless Compression. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 2264–2280. doi:10.1145/3779212.3790250 [13] Jiawei Fei, Chen-Yu Ho, Atal N. Sahu, et al. 2021. Efficient Sparse Collective Communication and Its Application to Accelerate Distributed Deep Learning. In Proceedings of the 2021 ACM SIGCOMM Conference (SIGCOMM ’21). 676–691. doi:10.1145/3452296.3472904 [14] Tianxiang Gao, Xiaokai Huo, Hailiang Liu, and Hongyang Gao. 2023. Wide neural networks as Gaussian processes: lessons from deep equilibrium models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2397, 34 pages. [15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] [16] Song Han, Huizi Mao, and William J. Dally. 2015. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. arXiv:1510.00149 [cs.CV] [17] Yongchang Hao, Yanshuai Cao, and Lili Mou. 2024. NeuZip: MemoryEfficient Training and Inference with Dynamic Compression of Neural Networks. arXiv:2410.20650 [cs.LG] [18] Horace He and Thinking Machines Lab. 2025. Defeating Nondeterminism in LLM Inference. https://thinkingmachines.ai/blog/defeatingnondeterminism-in-llm-inference/. Thinking Machines Lab: Connectionism. [19] Moshik Hershcovitch, Andrew Wood, Leshem Choshen, Guy Girmonsky, Roy Leibovitz, Ilias Ennmouri, Michal Malka, Peter Chin, Swaminathan Sundararaman, and Danny Harnik. 2025. ZipNN: Lossless Compression for AI Models. In 2025 IEEE 18th International Conference on Cloud Computing (CLOUD). 186–198. doi:10.1109/CLOUD67622.2 025.00028 [20] Zhiyi Hu, Siyuan Shen, Tommaso Bonato, et al. 2025. Demystifying NCCL: An In-Depth Analysis of GPU Communication Protocols and Algorithms. In 2025 IEEE Symposium on High-Performance Interconnects (HOTI). 48–59. doi:10.1109/HOTI66940.2025.00024 [21] Jiajun Huang, Sheng Di, Yafan Huang, et al. 2025. GhZCCL: Advancing GPU-aware Collective Communications with Homomorphic Compression. In Proceedings of the 39th ACM International Conference on Supercomputing (ICS ’25). 43–56. doi:10.1145/3721145.3733642 [22] Jiajun Huang, Sheng Di, Xiaodong Yu, et al. 2024. gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusters. In Proceedings of the 38th ACM International Conference on Supercomputing (ICS ’24). 437–448. doi:10.1145/3650200.3656636 [23] Jiajun Huang, Sheng Di, Xiaodong Yu, et al. 2024. An Optimized Error-Controlled MPI Collective Framework Integrated with Lossy Compression. In 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS). 752–764. [24] Jiajun Huang, Sheng Di, Xiaodong Yu, et al. 2025. ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression. arXiv:2502.18554 [cs.DC] [25] Siyuan Huang, Brian D. Hoskins, Matthew W. Daniels, et al. 2023. Low-Rank Gradient Descent for Memory-Efficient Training of Deep In-Memory Arrays. Journal of Emerging Technologies in Computing Systems 19, 2, Article 16 (2023), 24 pages. doi:10.1145/3577214 [26] Samirasadat Jamalidinan and Kazem Cheshmi. 2025. FloatingPoint Data Transformation for Lossless Compression.
Conclusion
The rapid scaling of large language models has made GPU communication a key performance bottleneck. In this work, we introduced UCCL-Zip, a unified approach that embeds lossless compression into GPU communication primitives. UCCL-Zip supports both point-to-point and collective communication without sacrificing numerical fidelity. UCCL-Zip accelerates RL weight synchronization by up to 47.5% and reduces vLLM end-to-end inference latency by up to 10%.
References QSGD: [1] Dan Alistarh, Demjan Grubic, Jerry Li, et al. 2017. Communication-Efficient SGD via Gradient Quantization and Encoding. In Advances in Neural Information Processing Systems (NeurIPS ’17). [2] AMD. 2024. RCCL: AMD ROCm Collective Communication Library. https://github.com/ROCmSoftwarePlatform/rccl. Accessed: 2026. [3] Noushin Azami, Alex Fallin, and Martin Burtscher. 2025. Efficient Lossless Compression of Scientific Floating-Point Data on CPUs and GPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS ’25). doi:10.1145/3669940.3707280 [4] Jinze Bai, Shuai Bai, Yunfei Chu, et al. 2023. Qwen Technical Report. arXiv:2309.16609 [cs.CL] [5] Jiamin Cao, Yu Guan, Kun Qian, et al. 2024. Crux: GPU-Efficient Communication Scheduling for Deep Learning Training. In Proceedings of the ACM SIGCOMM 2024 Conference (ACM SIGCOMM ’24). Association for Computing Machinery, 1–15. doi:10.1145/3651890.3672239 [6] Li-Wen Chang, Wenlei Bao, Qi Hou, et al. 2024. FLUX: Fast Softwarebased Communication Overlap On GPUs Through Kernel Fusion. arXiv:2406.06858 [cs.LG] [7] Chuyan Chen, Yutong He, Pengrui Li, et al. 2025. Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees. arXiv:2507.08784 [cs.LG] [8] Chen-Chun Chen, Yu-Min Chou, and Jerry Chou. 2023. PHY: A Performance-Driven Hybrid Communication Compression Method for Distributed Training. J. Parallel and Distrib. Comput. 180 (2023), 104719. doi:10.1016/j.jpdc.2023.104719 [9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. Res. 24, 1, Article 240 (2023), 113 pages. [10] Weihao Cui, Yukang Chen, Han Zhao, Ziyi Xu, Quan Chen, Xusheng Chen, Zhou Yangjie, Shixuan Sun, and Minyi Guo. 12
[44] Baixi Sun, Weijin Liu, J. Gregory Pauloski, et al. 2025. COMPSO: Optimizing Gradient Compression for Distributed Training with SecondOrder Optimizers. In Proceedings of the 30th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’25). 212–224. doi:10.1145/3710848.3710852 [45] Yuki Takezawa, Kenta Niwa, and Makoto Yamada. 2023. Communication Compression for Decentralized Learning With Operator Splitting Methods. IEEE Transactions on Signal and Information Processing over Networks 9 (2023), 581–595. doi:10.1109/TSIPN.2023.3307894 [46] UCCL Project. 2024. KV Transfer Engine: High-Performance GPU Communication in UCCL. https://uccl-project.github.io/posts/kvtransfer-engine/. Accessed: 2026. [47] V Team, Wenyi Hong, et al. 2025. GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning. arXiv:2507.01006 [cs.CV] [48] Guanhua Wang, Heyang Qin, Sam Ade Jacobs, et al. 2023. ZeRO++: Extremely Efficient Collective Communication for Giant Model Training. arXiv:2306.10209 [cs.DC] [49] Ceyu Xu, Yongji Wu, Xinyu Yang, Beidi Chen, Matthew Lentz, Danyang Zhuo, and Lisa Wu Wills. 2025. LLM.265: Video Codecs are Secretly Tensor Codecs. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25). Association for Computing Machinery, New York, NY, USA, 445–460. doi:10.1145/3725843.3756078 [50] Hang Xu, Chen-Yu Ho, Ahmed M. Abdelmoniem, et al. 2021. GRACE: A Compressed Communication Framework for Distributed Machine Learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) (ICDCS ’21). 561–572. doi:10.1109/ICDC S51616.2021.00060 [51] Annie Yang, Hari Mukka, Farbod Hesaaraki, and Martin Burtscher. 2015. MPC: A Massively Parallel Compression Algorithm for Scientific Data. In 2015 IEEE International Conference on Cluster Computing. 381– 389. doi:10.1109/CLUSTER.2015.59 [52] Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, et al. 2025. 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11). In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [53] Q. Zhou, C. Chu, N. S. Kumar, et al. 2021. Designing High-Performance MPI Libraries with On-the-fly Compression for Modern GPU Clusters. In 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS) (IPDPS ’21). 444–453. doi:10.1109/IPDPS49936.2021.00053 [54] Yang Zhou, Zhongjie Chen, Ziming Mao, et al. 2025. An Extensible Software Transport Layer for GPU Networking. arXiv:2504.17307 [cs.NI]
arXiv:2506.18062 [cs.DB] [27] Sylvain Jeaugey. 2017. NCCL: Optimized Primitives for Collective Multi-GPU Communication. https://developer.nvidia.com/nccl. [28] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. doi:10.1145/3600006.3613165 [29] Alexander Langer, Samuel Howell, Sreeram Potluri, et al. 2021. Dynamic Symmetric Heap Allocation in NVSHMEM. In OpenSHMEM and Related Technologies (Lecture Notes in Computer Science). Springer, 187–198. doi:10.1007/978-3-031-04888-3_12 [30] Minghao Li, Ran Ben Basat, Shay Vargaftik, et al. 2024. THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression. arXiv:2302.08545 [cs.LG] [31] Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. 2023. Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training. In Proceedings of the 52nd International Conference on Parallel Processing (ICPP ’23). Association for Computing Machinery, 766–775. doi:10.114 5/3605573.3605613 [32] Xue Li, Cheng Guo, Kun Qian, et al. 2024. Near-Lossless Gradient Compression for Data-Parallel Distributed DNN Training. In Proceedings of the ACM Symposium on Cloud Computing (SoCC ’24). 977–994. doi:10.1145/3698038.3698541 [33] Nandor Licker, Kevin Hu, Vladimir Zaytsev, and Lequn Chen. 2025. RDMA Point-to-Point Communication for LLM Systems. arXiv:2510.27656 [cs.DC] [34] Meta AI Research. 2026. DietGPU. https://github.com/facebookresea rch/dietgpu. GitHub repository, accessed 2026-03-07. [35] Mooncake Project. 2024. Mooncake Transfer Engine. https://github.c om/kvcache-ai/Mooncake. Accessed: 2026. [36] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’21). Association for Computing Machinery, Article 58, 15 pages. doi:10.1145/3458817.3476209 [37] NVIDIA. 2023. nvCOMP: NVIDIA GPU Data Compression Library. https://github.com/NVIDIA/nvcomp. Accessed: July 31, 2023. [38] NVIDIA. 2023. NVIDIA CUDA C Programming Guide. https://docs.n vidia.com/cuda/cuda-c-programming-guide/. [39] NVIDIA. 2025. NIXL: NVIDIA Inference Xfer Library. https://github .com/ai-dynamo/nixl. [40] Qwen, An Yang, Baosong Yang, et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL] [41] Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. 2024. CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI ’24). USENIX Association, Article 78, 18 pages. [42] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’20). Association for Computing Machinery, 3505–3506. doi:10.1145/3394486.3406703 [43] Aashaka Shah, Abhinav Jangda, Binyang Li, Caio Rocha, Changho Hwang, Jithin Jose, Madan Musuvathi, Olli Saarikivi, Peng Cheng, Qinghua Zhou, Roshan Dathathri, Saeed Maleki, and Ziyue Yang. 2025. MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications. arXiv:2504.09014 [cs.DC]
13