CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism

CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism Xinhang Zhang∗

arXiv:2611.05906v1 [cs.AI] 4 Nov 2026

Lei Sun

Boning Zhang∗

Chengchun Liu

Weifeng Zhang

Chunpu Li Limin Xiao†

Lenovo Research

Abstract As sequence lengths continue to grow, distributed attention based on sequence parallelism (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation–communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3× and 1.7× on a dual-NUMA PCIe architecture, and 2.8× and 3.9× on a dual-node NVLink architecture, respectively.

1

Introduction

As large language models (LLMs) scale up [3], longer context lengths are required to fully exploit their performance [29, 24]. Notably, LLM training requires processing long documents to enhance model capabilities [5]. Meanwhile, the inference of LLMs involves cross-document understanding [1], long program comprehension [18], and multi-turn interactive dialogues [15]. To handle this, self-attention is the core mechanism for capturing long-range dependencies using queries (q), keys (k), and values (v) [9, 25]. However, its computation and memory costs scale with sequence length, making long-context training and inference expensive [13, 27]. Therefore, sequence parallelism (SP) and distributed attention have become mainstream approaches. They partition the sequence across accelerators to reduce per-device overhead, enabling longer context windows [23, 12]. Nevertheless, existing SP methods (e.g., Ring-Attention) exhibit two limitations. Firstly, due to the communication dependency, each communication relies on the completion of the preceding communication [28]. Consequently, it limits the communication-computation overlap and reduces GPU utilization. Secondly, in the cross-hardware (Non-Uniform Memory Access, NUMA or node), imbalanced bandwidth leads to extra synchronization overhead [21]. Therefore, communication dependencies cause GPUs to idle, degrading overall performance. ∗ Equal contribution. † Corresponding author.

40th Conference on Neural Information Processing Systems (NeurIPS 2026).

communication CPU0

NUMA0

idle

contention CPU1

GPU0

NUMA1

PCIe Switch

PCIe Switch

GPU0 GPU1 GPU2 GPU3

GPU4 GPU5 GPU6 GPU7

GPU1

NVSwitch0

(a) Blocking comm due to link contention in PCIe

GPU2

GPU3

NVSwitch1

GPU4

GPU5

NVSwitch2

GPU6

GPU7

NVSwitch3

(b) Non-blocking comm on dedicated links in NVLink

Figure 1: Communication under PCIe and NVLink. The left of (a) shows single communication at full bandwidth without contention, while the right shows multiple communications at limited bandwidth with contention. (b) shows multiple communications at full bandwidth without contention. Based on these observations, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We design a merge-based communication mechanism to reorganize the sequential communications into several levels and achieve intra-level independence. Secondly, we leverage multiple communication streams to better overlap computation and communication. Furthermore, by decoupling intra- and inter-hardware communication, we avoid performance degradation due to synchronization overhead. As shown in Figure 2, GPU idle time has been significantly reduced, leading to substantial performance improvements. Our contributions are summarized as follows: • We propose a novel communication and parallel mechanism. To optimize the ring pattern, we reorganize communications into logarithmic levels and design a multistream parallelism for kv and q to achieve intra-level parallelism. • We develop a Cross-Hardware Query Parallelism to decouple intra- and inter-hardware communication. By isolating cross-hardware q and kv communications, the parallelism avoids synchronization induced by bandwidth imbalance and improves performance. • We implement both forward and backward of our method and conduct extensive experiments across different hardware architectures. The results demonstrate that our forward and backward consistently outperform Ring-Attention, achieving maximum speedups of 1.4× and 1.9× on PCIe, and 3.2× and 4.3× on NVLink.

2

Related Work

2.1

Interconnection Architecture

GPU Idle Time (msec)

Most large-scale training and inference are deployed on multi-GPU systems connected via either PCIe or NVLink [19, 6]. Figure 1 illustrates the interconnects of these two architectures. PCIe-based systems often suffer from link contention due to concurrent communication requests, Ring-Attention TASP CHARM+ while NVLink provides high-bandwidth, 500 2.3× dedicated links that enable contention-free 250 communication [11]. Furthermore, both 150 architectures have imbalanced communi100 2.9× cation bandwidth between cross-hardware 50 (i.e., cross-NUMA or cross-node) and intra40 hardware [17]. Consequently, it incurs substantial synchronization overhead, severely 2.7× 20 degrading communication performance. 0

Small (4K)

Medium (13K) Sequence Size

2.2

Large (52K)

Sequence Parallelism

As long-context requirements in LLMs continue to grow, the sequence parallelism and the distributed attention mechanism have become critical techniques for scaling attention by distributing sequence computation across multiple devices [12, 23]. Existing methods (e.g., Ring-Attention) rely on distributed kv exchange, where kv blocks are transmitted across devices in a ring topology [14] and processed Figure 2: GPU Idle Time. By setting various sequence lengths, we compare GPU idle time with Ring-Attention and TASP. Our method achieves higher GPU utilization.

2

𝑄3,7 𝐾3 𝑉3

𝑄7 𝐾7 𝑉7

𝑁𝑈𝑀𝐴 0 2

6

1

0

𝑁𝑈𝑀𝐴 1 Comp 𝑄7 𝐾𝑉3 𝐾1 , 𝑉1 KV 𝑄0 , 𝑄4 Q 4 5 Out

(a) Cross-Hardware Query Parallelism [step 0] 𝑄3,7 𝐾3 𝑉3

𝑄7 𝐾7 𝑉7

𝑁𝑈𝑀𝐴 0 𝑄3,7 , 𝑄0,4 𝐾3 , 𝐾1 𝑉3 , 𝑉1

2

𝑄3,7 𝐾𝑉0

6

1

0

6

5

4

0

𝑁𝑈𝑀𝐴 1 Comp 𝑄3 𝐾𝑉3 [a] 𝑄7 𝐾𝑉3 [b] 𝐾1 , 𝑉1 [b] KV 𝑄7 [a] 𝑄0 , 𝑄4 [b] Q 4 5 Out

0

𝑄7 𝐾𝑉2 𝑄3 𝐾𝑉2

𝑁𝑈𝑀𝐴 0 7

1

𝐾2 , 𝑉2

𝑁𝑈𝑀𝐴 0 2

𝑄2,6 𝐾2 𝑉2

(c) Re-Merge Multistream Mechanism [step 2]

(b) Re-Merge Multistream Mechanism [step 1] 3

1

Comp KV Q Out

𝑄4 𝐾𝑉3 𝑄0 𝐾𝑉3 𝑄3 𝐾𝑉1 𝑄7 𝐾𝑉1

𝑁𝑈𝑀𝐴 0 2

𝑁𝑈𝑀𝐴 0 𝑄3,7 𝐾3 , 𝐾2 𝑉3 , 𝑉2

𝑄0,4 𝐾0 𝑉0

𝑄1,5 𝐾1 𝑉1

𝑄 comm 𝐾𝑉 comm 𝑂𝑢𝑡 comm Sync

𝑄4 𝐾𝑉3 [b] 𝑄0 𝐾𝑉3 [b] 𝑄3 𝐾𝑉1 [b] 𝑄7 𝐾𝑉1 [b] 𝑄7 𝐾𝑉2 [c] 𝑄3 𝐾𝑉2 [c] 𝐾2 , 𝑉2 [c] 𝑄3,7 𝐾𝑉0 [b]

𝑄3 𝐾𝑉4~7 [𝐝]

(e) Overall Comp-Comm Pipeline

(d) Cross-Hardware Query Parallelism [step 3]

Figure 3: Overall Framework of CHARM+. (a) - (d) illustrate the complete process of the algorithm using an 8-GPU system with a dual-NUMA configuration. (a) displays parallel communication of q is first performed across NUMA. As a result, GPU 3 receives the q from GPU 7, eliminating the need for further cross-NUMA communication. (b) shows the parallel communication and computation of q and kv. In this step, GPU 3 communicates q with GPU 0 and kv with GPU 1. After completing the computation of Q0,4 , GPU3 communicates back to GPU 0 and receives the output. (c) performs the last step of RM+, where GPU 3 only communicates kv with GPU 2. (d) shows that once all kv results are computed, the output is communicated back to GPU 7. (e) illustrates the overall cross-NUMA communication and the full overlap of computation and communication of GPU3 happened in (a)-(d).

incrementally [16]. However, there exists a dependency in ring-based communication: each communication cannot start until the previous one completes. Moreover, each communication can only overlap with a single computation [22]. Consequently, as communication time grows, this dependency prolongs GPU idle time and reduces effective compute utilization. Several approaches have been proposed to improve the efficiency and scalability of Ring-Attention. TASP [26] aims to better utilize available communication links by adopting the all-to-all communication. Attention-2D [10] splits the self-attention computation into two dimensions along q and kv. Mesh-Attention [4] replaces one-dimensional Ring-Attention with a two-dimensional tiled assignment. Striped-Attention [2] reorders the input sequence into uniformly distributed blocks. However, these approaches do not optimize the ring-based pattern. Reducing the ring length or enabling multiple rings to communicate in parallel does not eliminate the communication dependencies, nor does it improve the communication–computation overlap. Therefore, efficient computation–communication overlap and high GPU utilization remain challenges in distributed attention.

3

Method

This section presents our method illustrated in Figure 33 . We first explain our motivation and introduce a merge-based communication mechanism. Secondly, we propose a multistream parallelism to improve the communication-computation pattern in the communication mechanism. Furthermore, we leverage a cross-hardware query parallelism to mitigate the imbalanced bandwidth between crosshardware (NUMA or node) communication and intra-hardware communication and use intra-pipeline to hide cross-hardware communication. All notations utilized in this section are described in Table 1.

3 For clarity, the computation of the FlashAttention in the illustration will ignore causal masks.

3

Table 1: Notation Description. Description Notation

Notation Q, q V, v O Tcomm (·) 3.1

K, k n ℓ(·) Tcomp (·)

Query matrix Value matrix Unhidden communication time Communication time of a tensor

Description

Key matrix Degree of sequence parallelism Function that clips value to zero Computation time of Attention

Motivation

As introduced in Section 2, the current ring-based attention involves gradually passing kv through a ring path for attention computation and update. This mechanism enables partial overlap between communication and computation. However, dependencies between communications prevent a communication from starting before the previous one is complete. To maximize GPU utilization, we seek to maximize communication–computation overlap, defined as follows: min O =

N X

i i ℓ(Tcomm (k, v) − Tcomp (q, k, v))

(1)

i=1 i i Let N denote the length of critical path that is non-overlapping, Tcomm (·) and Tcomp (·) are the ith communication time, the ith computation time of FlashAttention [8]. We define O as the unhidden communication time, where the GPU is idle. Therefore, minimizing O maximizes GPU utilization.

For Ring-Attention [16], due to the ring structure, this objective can be expressed as: Oring =

n−1 X

i i ℓ(Tcomm (k, v) − Tcomp (q, k, v))

(2)

i=1

This O(n) critical path dependency becomes the primary bottleneck limiting the ability to overlap communication and computation. In particular, the bandwidth imbalance between cross-hardware (NUMA or node) and intra-hardware communication causes extra communication synchronization to significantly increase the latency of each step along the critical path. Therefore, this effect further exacerbates the overall bottleneck, we further corroborate these observations with Nsight profiling results in Figure 9a of Section C.1. Based on these observations, our motivation becomes: “How can we efficiently overlap computation and communication while simultaneously minimizing the overall critical path of the execution? " As inspired by the traditional merge algorithm [20, 7], we present a communication mechanism. Compared to the ring-based kv exchange, the mechanism redesigns this process as kv aggregation. The aggregation is structured as a logarithmic-level merge (in Figure 4a, a sequence parallelism of 8 corresponds to a three-level merge). At each level i, kv are aggregated in parallel between device pairs with a stride of 2i−1 . As communication dependencies persist between merge levels, the critical path of the merge mechanism becomes O(log n), which is significantly shorter than Ring-Attention. For the merge communication mechanism, we model the communication time at each merge level as: (1) Ci =

i−1 2X

j Tcomm (k, v)

(3)

j=1

At the ith merge level, the communication mechanism performs 2i−1 communications. Meanwhile, the computation at level i processes the kv received at the communication of level i − 1. Thus, KV communication at level i overlaps with computation at level i−1, whose cost is 2ℓ(i−2) . Consequently, the unhidden communication time is given by: log2 (n)

Omerge =

X

(1) ℓ(Ci −

i=1

2ℓ(i−2) X j=1

4

j Tcomp (q, k, v))

(4)

KVComm Comm KV

Comm QQComm

00

11

22

33

44

55

66

77

퐾퐾 00

퐾퐾 11

퐾퐾 22

퐾퐾 3 3

퐾퐾 4 4

퐾퐾 5 5

퐾퐾 6 6

퐾퐾7 7

00

11

22

퐾퐾 0~1 0~1 00

11

33

55

66 7 7

퐾퐾 2~3 2~3

퐾퐾 4~5 4~5

퐾퐾 6~7 6~7

22

44

33

44

66

77

퐾퐾 4~7 4~7

퐾퐾 0~3 0~3

Comp ��77퐾퐾 77 Comp KV 퐾퐾66, , 66 KV

55

��7 퐾 ��77퐾퐾 66 7퐾 5 5 7 퐾 4 4 �� 7퐾 퐾퐾4~5 , , 4~5 퐾퐾0~3 , , 0~3 4~5 4~5 0~3 0~3

00 11

22 33

44 55

66 77

퐾퐾0~1 0~1

퐾퐾2~3 2~3

퐾 퐾4~5 4~5

퐾 퐾6~76~7

00 11 22 3 3

44 55 66 77

퐾퐾0~3 0~3

퐾 퐾4~74~7

Idle Idle

Comp �7�퐾7 퐾7 7 �4�퐾4 퐾7 7�7�퐾7 퐾6 6�2�퐾2 퐾7 7�7�퐾7 퐾5 5�0�퐾0 퐾7 7�7�퐾7 퐾3 � Comp 퐾7 퐾1 1 3 7� 퐾3퐾, 3 ,3 3 퐾1퐾, 1 ,1 1 KV 퐾6퐾, 6 , 6 6 퐾5퐾, 5 ,5 5 KV QQ �4�4 �0�0 �2�2 Out �7�퐾7 퐾2 2 Out �7�퐾7 퐾4 4 �7� 퐾7 퐾0 0

Idle Idle

�� 7퐾0 0 7퐾

KV Comm KV Comm

�� 7 퐾3 3 7퐾

(a) Merge Communication Mechanism.

(b) Re-Merge Multistream Mechanism.

Figure 4: The Computation and Communication Overlap on GPU 7. For 8 GPUs, (a) contains three levels. Each GPU communicates at groups of 1, 2, and 4 at different levels, aggregating the communicated kv while overlapping computation. (b) maintains consistency with (a) in the first levels, but further enables multistream communication and computation of q and kv at other each merge level. Each GPU concurrently communicates q and kv with different GPUs, computes the q of other GPU and communicates the output back. The red region indicates GPU idle time. As can be observed, compared to the Ring-Attention, the merge communication mechanism restructures the linear communication path into a merge path with logarithmic level, significantly shortening the communication critical path. Each merge level accommodates more computation, increasing opportunities to overlap communication with computation and improving efficiency. 3.2

Re-Merge Multistream Mechanism

The merge communication mechanism reduces dependencies between ring communications, but communication at each merge level still follows the ring pattern. To further increase communication–computation overlap, we design the Re-Merge Multistream Mechanism, named RM+. It still follows the structure of the merge communication mechanism, but optimizes the communication at each merge level to enable more communication-computation overlap. As described in Section 3.1, Ring-Attention involves communication only for kv, and each round of kv communication can start only after the previous round completes. Notably, the communication of q is independent of the kv communication. By exploiting this independence, we parallelize q communication with kv communication, improving communication-computation overlap, as illustrated in Figure 4. Moreover, we aim to overlap the computation after the final merge level(e.g., Q7 KV0 · · · Q7 KV3 in Figure 4a. During each merge level, kv is fetched sequentially from peer GPUs for computation. In addition, parallel communication of q and kv further reduces the communication time of each merge level. Consequently, as illustrated in Figure 4b, we obtain: (2) Ci =

2ℓ(i−2) X

j j max Tcomm (q), Tcomm (k, v)

(5)

j=1

Note that, Eq. (5) follows Eq. (3). Due to the parallel communication between q and kv, the communication time is bounded by the maximum. Meanwhile, advancing the computation after the final merge level increases computations at earlier merge levels, thereby reducing the unhidden communication time. Specifically, all communication except the final one at each level can be overlapped, with the last communication deferred to the next merge level. Therefore, for the ith merge level, there are 2i−1 computations. Consequently, the unhidden communication time is: ORM+ =

log Xn

(2) ℓ(Ci −

i=1

i−1 2X

j Tcomp (q, k, v))

(6)

j=1

Under the assumption that all parallel communications sustain identical bandwidth, Eq. (4) outperforms Eq. (2) due to its shorter critical path and greater communication overlap. However, parallel communication of multistream can lead to PCIe link contention, increasing latency. Consequently, on 5

the PCIe architecture, the single communication of RM+ can be longer than that in Ring-Attention. Therefore, performance on a GPU system using NVLink is significantly better than on a PCIe architecture, as further demonstrated in Section 4.4. 3.3

Cross-Hardware Query Parallelism

In multi-hardware architectures (NUMA or Node), each KV communication in Ring-Attention simultaneously involves both intra-hardware and cross-hardware communication. Since cross-hardware communication typically provides lower bandwidth, the faster intra-hardware comSync Q Comm munication is forced to be idle while waiting for the communication synchronizaIntra Pipeline 0 Intra Pipeline 1 tion, leading to substantial performance inefficiency. As illustrated in Figure 5, we propose a novel parallelism where crossIntra Pipeline 2 Intra Pipeline 3 hardware communication of the q is interleaved with the intra-node pipeline. The �0 �0 �0 Comp �0 pipelines process the current queries (Q) Q ��1 ��2 ��3 from different pipelines with local KV usOut 푂� �2 푂� �1 푂� �3 ing the same computation and communication flow, as detailed in Section 3.2. NoFigure 5: Cross-Hardware Query Parallelism. The com- tably, since all kv and q communications plete process is illustrated with pipeline 0 as an exam- occur in the pipeline come from the same ple. Each intra-pipeline processes queries (Q) from NUMA or node, the bottleneck caused KV Comm Q Comm other pipelines and uses the same workflow with its by synchronized communication is largely local KV, returning the final out back. Effective overlap eliminated. With effective overlap illusallows inter-pipeline 1 3 communication 5 to be 7 by trated in Figure 5, after all computations 0 4 2 6 hidden 1 2 3 0 intra-pipeline, further improving performance. are completed, only a final out communi퐾 퐾 0 퐾 퐾 6 퐾as5the number 퐾 4 퐾 3 Therefore, 1 cation is퐾not2 hidden. of 7nodes increases, effective overlap ensures that only a single communication remains exposed, making the advantage퐾of our method퐾increasingly 2~3 0~1 pronounced, which is further validated in the ablation study in Section 4.4 and Section 4.5. We 1 2 3 5 6 7 0 4 provide Nsight profiling results of CHARM+ in Figure 9b of Section C.1, demonstrating that our overlap strategy Ring-Attention. 1 0 2 3 퐾 0~1 퐾 6~7 퐾 2~3significantly퐾 outperforms 4~5 04

1Experiments 2 3

4

5

6

7

퐾 0~3

KV Comm 4

6

5

퐾 6~7

퐾 4~5 4

7

5

6

7

퐾 4~7

Comp �7 퐾 7 �4 퐾 7 �7 퐾 6 �2 퐾 7 �7 퐾 5 �0 퐾 7 �7 퐾 3 �7 퐾3 , 3 퐾1 , 1 퐾 , 퐾5 , 5 KV 6 6 Q �0 �7 퐾 0 �4 �2 �7 퐾 4 �7 퐾 5 �7 퐾 6 �7 퐾 3 Comp �7 퐾 7 conduct experiments on, GPU Out and detailed specifications �7 퐾 4 are �7 퐾 2 �7 퐾 0 퐾6 , We 퐾4~5 , 4~5 퐾0~3 KV 6 0~3 systems with different configurations provided in Table 2. We use official NCCL and RDMA test benchmarks to measure the intra-hardware and cross-hardware unidirectional P2P Table 2: The GPU Configuration and Bandwidth. bandwidth. On L20, we evaluate the performance in both cross-NUMA and intraBandwidth (GB/s) GPU NVLink Num TFLOPS NUMA settings. Furthermore, we explore Cross Intra the impact of link contention in PCIe-based systems. On H20 and H100, we evaluate L20 ✗ 8 119.5 14.7 24.0 cross-node and intra-node performance and H20 ✓ 16 148.0 42.6 297.2 H100 ✓ 32 989.0 42.5 365.1 further compare results under NVLink architecture, where link contention is absent. 퐾 0~3 4.1 Experimental Setup

퐾 4~7

Idle

We evaluate our method under diverse configurations by varying batch size (B), sequence length (S), and hidden dimensions (H). Specifically, sequence lengths range from 1K to 100K (up to 200K in some scenarios), with four hidden-dimension configurations formed by combining dimensions of 12 or 20 with sizes of 128 or 192. We conduct a grid search over B, S, and H, ensuring that the product BS does not exceed 500K due to memory constraints. Each configuration is evaluated over 100 iterations following 20 warm-up iterations, with results averaged across the measured runs. To enable a unified characterization of different configurations, we use the computation-to-communication ratio 6

Speedup

Speedup

Backward 2.1 CHARM+ 1.9 1.7 1.5 1.3 1.1 0.1 0.2 0.3 0.4 0.5 0.6 CCR (a) 8×L20 GPU

2.0 CHARM+ 1.7 1.4 1.1 0.8 0.5 0.5 1.0 1.5 2.0 2.5 3.0 CCR

Speedup

2.0 CHARM+ 1.7 TASP 1.4 1.1 0.8 0.5 0.5 0.8 1.1 1.4 1.7 2.0 CCR

Speedup

Speedup

Speedup

Forward 2.1 CHARM+ 1.8 TASP 1.5 1.2 0.9 0.6 0.1 0.2 0.3 0.4 0.5 0.6 CCR

Forward + Backward 2.1 CHARM+ 1.9 1.7 1.5 1.3 1.1 0.1 0.2 0.3 0.4 0.5 0.6 CCR

2.0 CHARM+ 1.7 1.4 1.1 0.8 0.5 0.5 1.5 2.5 3.5 4.5 5.5 CCR

(b) 8×H20 GPU

Speedup Speedup

Speedup

Speedup

Speedup

Speedup

Figure 6: Single-node Experiments on 8×L20 (PCIe) and 8×H20 (NVLink). Speedup is normalized to Ring-Attention, comparing forward, backward, and end-to-end time. In different architectures, our method consistently outperforms the baseline and TASP. 2 × 8 H20 Zigzag 2 × 8 H20 2 × 4 H20 5 2.1 5 CHARM+ CHARM+ CHARM+ 4 1.8 4 TASP TASP TASP-Zigzag 3 1.5 3 1.2 2 2 0.9 1 1 0.6 0.1 0.4 0.7 1.0 1.3 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 CCR CCR CCR 5 5 2.5 CHARM+ CHARM+ CHARM+ 4 4 2.0 3 3 1.5 1.0 2 2 0.5 0.2 0.5 0.8 1.1 1.4 1 0.1 0.2 0.3 0.4 0.5 1 0.05 0.15 0.25 0.35 0.45 CCR CCR CCR Figure 7: Multi-node Experiments on 8×H20 and 16×H20 (NVLink). Speedup is normalized to Ring-Attention and Ring-Zigzag. In the forward (top) and the backward (bottom) comparison, our method achieves consistently superior performance. Particularly, the 2×4 and 2×8 experiments highlight that our method’s performance grows as the number of GPUs increases.

(CCR) as a single metric. Following TASP [26], we define the CCR as: CCR =

N X

i Tcomp(q,k,v) /

i=1

N X

i Tcomm(k,v)

(7)

i=1

Here, q, k, and v are tensors that depend on the sequence length, the batch size, the hidden dimensions, and the number of attention heads. We accumulate the computation and communication time of Ring-Attention under sequential execution. The final metric is defined as the ratio of the computation time to the communication time (detailed analysis of CCR is provided in Section B.1). We compare our method against Ring-Attention [16], TASP, and their corresponding zigzag variants [2]4 . 4.2

Single-node Evaluation

As illustrated in Figure 6, we perform single-node experiments on L20 with 8 GPUs under a dualNUMA PCIe architecture and on H20 with 8 GPUs using NVLink. Our method outperforms TASP 4 As TASP does not provide the implementation of the backward algorithm, we compare forward performance only.

7

Table 3: Memory Overhead. By setting various parameter scales, we compare the memory overhead with Ring-Attention. Our method shows comparable overhead to Ring-Attention. Batch Size

Seq Len

Heads

Ring (MB)

CHARM+ (MB)

Batch Size

Seq Len

Heads

Ring (MB)

CHARM+ (MB)

16 16 64 128

1024 1024 8192 8192

32 64 32 16

2,179 3,701 45,079 44,720

2,215 3,881 45,423 45,432

2 4 2 4

53,760 53,760 26,880 26,880

12 12 12 12

13,123 19,301 10,745 14,029

13,467 20,565 10,803 14,425

and Ring-Attention, even in the presence of link contention (Figure 6a). The poor performance of TASP on L20 is mainly attributed to its all-to-all collective communication, which fully utilizes all links and induces severe contention. Furthermore, in the H20 experiments, the observed speedup is slightly lower. This is due to the very high intra-node bandwidth (as reported in Table 2), which increases the CCR relative to other experiments. In other words, computation outweighs communication, limiting the benefits of the communication-computation overlap. Memory Overhead Analysis. We evaluate memory overhead on L20 with 8 GPUs across varying sequence lengths for both forward and backward passes, comparing Ring-Attention with our method in Table 3. The results indicate that our method incurs only a modest increase in memory consumption. Furthermore, as the number of additional buffers introduced by our approach is constant (i.e., O(1)), memory overhead does not scale with the number of GPUs. See Section B.2 for detail analysis. Numerical correctness verification. We evaluate the numerical correctness of CHARM+ by comparing its outputs and gradients against FlashAttention. The results show that CHARM+ exhibits numerical differences of the same order of magnitude as Ring-Attention across the output, log-sumexp (LSE), and Q/K/V gradients, indicating that CHARM+ introduces no substantial additional numerical error. Detailed experimental configurations and results are provided in Section C.3. 4.3

Multi-node Evaluation

We conducted multi-node experiments on the H20 system with 8 or 16 GPUs using NVLink. On the 16-GPU setup, we compare our method against Ring-Attention, TASP, and their load-balanced variants, Ring-Zigzag and TASP-Zigzag. As shown in Figure 7, our method shows a clear advantage in cross-node configurations. This is because cross-node bandwidth is significantly lower than Table 4: Ablation Study. We compare the forintra-node bandwidth, as listed in Table 2. Con- ward performance of Ring-Attention, RM+, and sequently, Ring-Attention and TASP incur GPU CHARM+ across various GPU configurations and idle time due to frequent synchronized commu- sequence lengths. Notably, CHARM+ will be evalnication, resulting in a performance bottleneck uated only in cross-hardware configurations. that degrades efficiency. GPUs

Impact of Scaling on Performance. To evaluate scalability, we conducted experiments on the H20 system with 2 nodes using 8 GPUs and 2 nodes using 16 GPUs. As shown in Figure 7, our method achieves substantial speedups at larger scales by leveraging additional GPUs for better parallelism and effectively overlapping the communication with the computation. 4.4

Ablation Study

Forward Time (ms)

Cross

Seq

Ring

RM+

Ours

4 × L20

✗

1792 7168

9.35 36.86

10.29 39.66

– –

8 × L20

✓

1792 7168

17.61 69.85

25.91 98.49

14.81 52.40

4 × H20

✗

1792 7168

5.38 38.62

4.76 33.20

– –

8 × H20

✗

1792 7168

5.51 26.79

3.65 13.84

– –

1792

32.17

39.04

12.16

16 × H20 ✓ The effectiveness of the Re-Merge Multi7168 84.58 98.98 28.91 stream Mechanism (RM+). We evaluate RM+ across sequence lengths on a single 8-GPU H20 node to isolate its effectiveness from link contention and cross-node communication. As shown in Table 4, RM+ consistently outperforms Ring-Attention. This improvement stems from the parallel communication enabled by multiple independent q and kv streams. As a result, RM+ achieves a maximum speedup of 1.9× over Ring-Attention.

The effectiveness of Cross-Hardware Query Parallelism. We compare CHARM+ against RingAttention and RM+ in both cross-NUMA and cross-node configurations. Specifically, for cross8

Table 5: Performance on H100. Note that TASP requires the sequence length to be divisible by n(n − 1). Therefore, the sequence lengths for the four-node and two-node settings differ slightly. B

Forward

Seq Ring

TASP

Backward

CHARM+ Speedup

Ring

TASP CHARM+ Speedup

GPUs = 2 × 8 98.01 49.27 97.65 97.15

63.93 31.84 61.80 61.54

16 31,744 185.42 183.22 16 15,872 89.27 93.30 64 7,936 179.13 182.93 128 3,968 178.51 183.09

68.17 34.64 66.48 66.43

16 26,880 149.97 16 13,440 72.65 64 6,720 141.75 128 3,360 140.55

2.35× 2.28× 2.29× 2.28×

449.58 226.46 448.84 450.01

– – – –

141.52 65.35 123.32 121.99

3.18× 3.47× 3.64× 3.69×

559.34 279.85 556.25 560.63

– – – –

146.01 70.62 134.48 133.05

3.83× 3.96× 4.14× 4.21×

GPUs = 4 × 8 2.72× 2.58× 2.69× 2.69×

NUMA experiments, we use 8 GPUs on the L20 system, while for cross-node experiments, we use 16 GPUs on the H20 system. As shown in Table 4, RM+ suffers significant performance degradation in cross-hardware configurations, whereas CHARM+ achieves substantial improvements, with up to 1.9× and 3.5× speedup in cross-NUMA and cross-node configurations, respectively. The performance gain is smaller on L20 than on H20 due to link contention on L20, as discussed in the next subsection. The performance drop of RM+ is primarily due to a bandwidth imbalance between intra- and crosshardware communication, as well as increased cross-node communication resulting from multistream parallelism. In contrast, CHARM+ decouples inter-node and intra-node communication to reduce synchronization overhead and improve overall performance. The impact of link contention in PCIe architectures. To avoid communication synchronization issues caused by cross-NUMA communication on the L20 system, we conducted single-NUMA experiments on an L20 node with 4 GPUs, and performed comparison experiments on an H20 node with 4 GPUs connected via NVLink, which eliminates link contention. 4.5

Evaluation on High-end GPU

Speedup

As shown in Table 4, on L20, RM+ suffers significant performance degradation due to link contention, performing worse than Ring-Attention. In contrast, RM+ exhibits much better performance on H20. We further conduct a comprehensive evalubs=2 bs=4 bs=8 ation with various batch sizes and sequence lengths, summarized in Figure 8. The re1.2 sults indicate that link contention degrades performance in almost all configurations. 1.0 The exception occurs when both the batch size and sequence length are small.

0.8

10k

20k 30k 40k Sequence Length

50k

60k

Figure 8: Performance of RM+ under PCIe and NVLink. We evaluate the forward time of RM+ on 4×H20 GPUs (dashed) and 4×L20 GPUs (solid) across different batch sizes (bs). Compared to Ring-Attention, performance on L20 is lower than on H20 due to the link contention. TASP, as shown in Figure 6a.

In this case, the bandwidth is not fully utilized, so the link contention is negligible. Meanwhile, it is worth noting that in the 8× L20 experiments, link contention leads to smaller performance gains than on H20. Nevertheless, enabled by Cross-Hardware Query Parallelism, CHARM+ still significantly outperforms Ring-Attention and

To completely validate our method on the high-end GPUs, we rented 32 H100 GPUs equipped with high-bandwidth NVLink. We conducted full forward and backward experiments under both 2×8 and 4×8 configurations, and the results are reported in Table 5 (detailed results can be found in Section C.2). When TASP begins to leverage high-performance GPUs, our method still maintains a leading performance, achieving an average speedup of 2× in the forward and 3× in the backward compared to ring-attention. Additionally, as described in Section 3.3, our method consistently makes 9

Table 6: End-to-end Performance Comparison of Ring-Attention and CHARM+ with Model Scaling Batch Size

Seq Len

Ring

Model

CHARM+

Speedup

Latency

Memory

Latency

Memory

16 8 4 2

2,048 4,096 8,192 16,384

GPT-350M GPT-350M GPT-350M GPT-350M

10082.6 10032.6 9914.5 10089.5

11,372 11,348 11,436 12,112

4847.0 4871.5 5173.6 5637.3

11,518 11,544 11,552 12,170

2.08× 2.06× 1.92× 1.79×

16 8 4 2

2,048 4,096 8,192 16,384

GPT-1.3B GPT-1.3B GPT-1.3B GPT-1.3B

21505.5 22073.7 21857.8 22064.5

30,816 31,190 31,414 32,270

11738.3 11768.1 12097.5 12796.6

31,602 31,546 32,030 32,312

1.83× 1.88× 1.81× 1.72×

intermediate cross-node communication effectively overlapped by the intra-node pipeline. In contrast, other approaches suffer from synchronization constraints, making them susceptible to the slower cross-node communication. Consequently, our method exhibits greater performance as the scale increases (speedups of 2.28×/3.69× on 16 H100 GPUs and 2.69×/4.21× on 32 H100 GPUs at maximum performance), highlighting the advantage of our method. 4.6

End-to-end Training Performance

High-end GPU. We integrate our operator into Megatron-LM and compare it with Megatron’s ring implementation. The experimental results are shown in Table 7. We use context parallelism (CP) for training on 16 high-end H20/H100 GPUs and report the average per-iteration time in milliseconds. Table 7: End-to-end Performance Comparison of Ring-Attention and CHARM+ with High-end GPU Batch Size 16 32 64 128

Seq Len 16,384 8,192 4,096 2,048

Ring

Model GPT3-350M GPT3-350M GPT3-350M GPT3-350M

CHARM+

Speedup

H20

H100

H20

H100

H20

H100

7304.6 7193.7 7169.6 7143.6

3385.1 3291.5 3234.6 3210.8

5601.3 4669.7 4452.5 4242.5

1925.7 1679.2 1538.6 1482.0

1.30× 1.54× 1.61× 1.68×

1.76× 1.96× 2.10× 2.17×

Notably, Megatron-LM includes extensive optimizations for ring attention that our method does not incorporate. Moreover, non-attention linear layers dominate end-to-end training, limiting the impact of distributed attention optimization. Despite this, our method still significantly outperforms ringattention on H20 and H100, achieving max 1.68× and 2.17× speedup, respectively. Furthermore, our method performs better at larger batch sizes, making it more suitable for real-world workloads. Model scaling. To evaluate the scalability of CHARM+ across model sizes, we conduct additional end-to-end experiments on 16 L20 GPUs with model sizes of 350M and 1.3B, measuring iteration latency (ms) and GPU memory consumption (MB). Although model configurations affect the relative computational costs of attention and MLP layers, they do not fundamentally change the communication patterns targeted by CHARM+. As shown in Table 6, CHARM+ achieves consistent speedups of 1.72×–2.08× across both model sizes and all tested configurations, with nearly identical memory overhead. These results demonstrate the effectiveness of CHARM+ across different model scales.

5

Conclusion

In this work, we present CHARM+, a novel approach that reorganizes the communication process of Ring-Attention using a merge-based strategy. Furthermore, it leverages a multistream parallelism of q and kv to reduce communication dependencies and decouples cross-hardware and intra-hardware communication. We implement both the forward and backward algorithms of the mechanism and conduct extensive experiments and ablation studies across various settings. Experiment results across diverse hardware platforms and GPUs with varying computational throughput demonstrate that CHARM+ consistently achieves significant performance improvements over state-of-the-art methods. 10

References [1] Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3639–3664, 2025. [2] W. Brandon, A. Nrusimha, K. Qian, Z. Ankner, T. Jin, Z. Song, and J. Ragan-Kelley. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431, 2023. [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [4] S. Chen, J. Chen, S. Zhu, Z. Jiang, Y. Peng, and X. Qian. Mesh-attention: A new communicationefficient distributed attention with improved data locality. arXiv preprint arXiv:2512.20968, 2025. [5] Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023. [6] J. Choquette, E. Lee, R. Krashinsky, V. Balan, and B. Khailany. 3.2 the a100 datacenter gpu and ampere architecture. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), volume 64, pages 48–50. IEEE, 2021. [7] B. Citko. On the nature of merge: External merge, internal merge, and parallel merge. Linguistic Inquiry, 36(4):475–496, 2005. [8] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344– 16359, 2022. [9] P. Doukhan, G. Oppenheim, and M. Taqqu. Theory and applications of long-range dependence. Springer Science & Business Media, 2002. [10] V. Elango. Attention2d: Communication efficient distributed self-attention mechanism. arXiv preprint arXiv:2503.15758, 2025. [11] A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. R. Tallent, and K. J. Barker. Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect. IEEE Transactions on Parallel and Distributed Systems, 31(1):94–110, 2019. [12] S. Li, F. Xue, C. Baranwal, Y. Li, and Y. You. Sequence parallelism: Long sequence training from system perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2391–2404, 2023. [13] T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024. [14] X. Li. Niching without niching parameters: particle swarm optimization using a ring topology. IEEE Transactions on Evolutionary Computation, 14(1):150–169, 2009. [15] Y. Li, X. Shen, X. Yao, X. Ding, Y. Miao, R. Krishnan, and R. Padman. Beyond single-turn: A survey on multi-turn interactions with large language models. arXiv preprint arXiv:2504.04717, 2025. [16] H. Liu, M. Zaharia, and P. Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023. [17] P.-J. Lu, M.-C. Lai, and J.-S. Chang. A survey of high-performance interconnection networks in high-performance computer systems. Electronics, 11(9):1369, 2022. [18] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. 11

[19] R. Neugebauer, G. Antichi, J. F. Zazo, Y. Audzevich, S. López-Buedo, and A. W. Moore. Understanding pcie performance for end host networking. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, pages 327–341, 2018. [20] P. O’Neil, E. Cheng, D. Gawlick, and E. O’Neil. The log-structured merge-tree (lsm-tree). Acta informatica, 33(4):351–385, 1996. [21] D. A. Patterson. Latency lags bandwith. Communications of the ACM, 47(10):71–75, 2004. [22] K. Punniyamurthy, K. Hamidouche, and B. M. Beckmann. Optimizing distributed ml communication with fused computation-collective operations. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–17. IEEE, 2024. [23] A. Sun, W. Zhao, X. Han, C. Yang, Z. Liu, C. Shi, and M. Sun. Burstattention: An efficient distributed attention framework for extremely long sequences. arXiv preprint arXiv:2403.09347, 2024. [24] A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [26] Y. Wang, K. Hong, X. Li, Y. Xu, W. Wang, G. Dai, and Y. Wang. Tasp: Topology-aware sequence parallelism. arXiv preprint arXiv:2509.26541, 2025. [27] G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024. [28] L. Zhao, S. Pal, T. Chugh, W. Wang, J. Fantl, P. Basu, J. Khoury, and A. Krishnamurthy. Efficient {Direct-Connect} topologies for collective communications. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 705–737, 2025. [29] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023.

12

Appendix A

Implementation

We implement both the forward and backward of CHARM+. Algorithm 1 presents the logic for constructing the communication groups used in our method. Specifically, the communication peers are partitioned into two disjoint subgroups corresponding to kv communication and q communication at each merge level. To avoid imbalanced communication overhead where multiple GPUs simultaneously initiate communication to the same GPU, we determine the order of kv and q communication based on each GPU’s relative position within the global communication group. For example, when GPU 0 and 1 communicate with GPU 2 and 3, GPU 0 sends kv to GPU 2 and q to GPU 3, whereas GPU 1 sends q to GPU 2 and kv to GPU 3. This complementary scheduling ensures that both the kv and q processing streams on each GPU remain active, effectively preventing stream idling and improving overall communication efficiency. Algorithm 1 GenerateCommList Input: processGroup pg Output: commListq , commListkv Initialize tmp ← [], n ← len(pg), x ← getRank(pg) l←1 while l < n do g ← {x ⊕ i | i ∈ [l, 2 · l)} Append g to tmp l ←2·l end while Initialize g1 , g2 ← [], append tmp[0] to g1 g3 ← Sort(tmp[0] ∪ {x}) for each g in tmp[1 :] do first, second ← split(g), index ← getPosition(x, g3 ) half ← |g3 |/2 if index < half then Append first to g1 , append second to g2 else index ← index − half Append second to g1 , append first to g2 end if g[ 3] ← Sort(g3 ∪ g) end for commListkv , commListq ← concatenate(g1 , g2 ) Return commListkv , commListq Algorithm 2 provides the core logic chain of the forward algorithm 5 , which is consistent with the overall pipeline in Figure 3(f). Notably, for the q communication, the attention output needs to be communicated back to the original GPU after the attention computation finishes (i.e., the Out communication in Figure 3). Therefore, we prioritize the q communication stream during execution, enabling earlier output communications and effective overlap with the subsequent computation of the kv communication stream.

B

Analysis

We conduct a detailed analysis of the factors affecting CCR, as well as the memory usage of RingAttention and CHARM+. 5 Since the forward and backward share the same overall logic, we omit the detailed description of the backward pass for

brevity.

13

Algorithm 2 CHARM+ Forward Input: tensor q, k, v, processGroup pg Output: out commListq , commListkv ← GenerateCommList(pg) qcross ← CrossComm(q) Initialize out, outcross ← N one, n ← getsize(pg) Commwait() for i ← 1 to log(n) do kother , vother ← KVComm(k, v) qother , qotherC ← QComm(q, qcross ) if i == 1 then out ← CompAttention(q, k, v, out) outcross ← CompAttention(qcross , k, v, out) else outcross ← CompAttentionComm(qotherC , k, v, out) out ← CompAttentionComm(qother , k, v, out) outcross ← CompAttention(qcross , kother , vother , out) out ← CompAttention(q, kother , vother , out) end if Commwait() end for out ← CommAndUpdate(out, outcross )

B.1

Analysis of CCR

As mentioned in Eq. (7), CCR is defined by us as: CCR =

N X

i Tcomp(q,k,v) /

i=1

N X

i Tcomm(k,v)

(8)

i=1

The communication time depends only on the amount of data being transferred and the available bandwidth. Different on-device and off-device connections have different bandwidths. Therefore, the relation between communication time and factors can be defined as: BSH Tcomm ∝ (9) Bandwidth And the computation time is essentially the time complexity of FlashAttention, which is BS 2 H + BSH 2 (10) F LOP S Note that, while the theoretical time remains the relation above, the actual runtime is dominated by memory IO in standard attention and both the actual FLOPS and bandwidth exhibit some variability. It can be seen that once the input size and computation pattern are fixed, the time per step is constant. Thus, the total time equals the sum of non-overlappable times along the critical path, which means reducing the number of steps (minimizing the overall critical path) directly reduces the total time. In addition, we provide some quantitative results under different hardware and input specifications, as shown in Section C.2. Due to fluctuations in actual time, we collected the time data in real-time through the interface provided by CUDA events. Tcomp ∝

B.2

Analysis of Memory Overhead

Since we do not change the computation pattern but instead improve the overlap between computation and communication, we focus our analysis on the major memory-consuming buffers and ignore the memory overhead of the attention computation. We assume that the input query, key, and value are all [b, s, h] (i.e., batch size, sequence length and hidden dims). For Ring-Attention, the memory overhead of the buffers is: MRing ≈ 3bsh + 2bsh + 1.5bsh + 5bsh + 3bsh + 1.5bsh + 2bsh + 2bsh ≈ 20bsh 14

(11)

Specifically, under FP32 precision, each buffer occupies BSH memory. In Ring-Attention, one buffer stores the local query block, two buffers hold the key and value blocks being processed, and two more buffers receive the next KV blocks to overlap computation and communication. One buffer stores the accumulated output, and half a buffer holds intermediate FlashAttention results (usually in BF16/FP16). During backward, five buffers store q, k, v, the output, and its gradient; three buffers store total gradients of q, k, v; 1.5 buffers store FlashAttention gradients; and two buffers handle the next KV block computation. Finally, because KV s propagate across GPUs in a ring, the accumulated gradients must be sent back to the original GPU, requiring two additional buffers. We further analyze the memory overhead of CHARM+, which can be formulated as: MCHARMS+ ≈ 3bsh + 2bsh + 1.5bsh + 5bsh + 3bsh + 1.5bsh + 2bsh + 6bsh − 2bsh ≈ 24bsh

(12)

During forward and backward, each GPU must retain its own KV and receive Q of cross-hardware, requiring three extra buffers. Moreover, FlashAttention results can be updated immediately, so the buffers used in ring-attention for gradient sending can be omitted. Overall, our improved method logically uses four more buffers than ring-attention. However, GPU memory is temporally reused across forward and backward and buffer lifetimes do not fully overlap, the peak buffer usage remains roughly the same, resulting in only a 3%~5% increase in observed memory.

C

Experimental Details

C.1

Nsight Result

(a) Ring-Attention

(b) CHARM+

Figure 9: Nsight Analysis of Ring-Attention and CHARM+ To better understand the performance improvement, we profile the forward pass of Ring-Attention and CHARM+ on 8 L20 GPUs using Nsight Systems, as shown in Figure 9a and Figure 9b, respectively. The L20 platform has relatively limited communication bandwidth, making communication kernels substantially longer than computation kernels. This imbalance makes communication–computation overlap particularly important, as the overall execution time is largely determined by how effectively communication latency can be hidden behind computation. In Figure 9a, Ring-Attention exhibits substantial GPU idle time due to its sequential communication dependencies. Each communication step depends on the completion of the preceding step, limiting scheduling flexibility and leaving computation unable to fully hide communication latency. In contrast, as shown in Figure 9b, CHARM+ reorganizes ring-based communication into a merge-based structure, shortening the dependency chain, and leverages multiple streams to execute independent communication and computation tasks concurrently. This design substantially reduces idle gaps and improves communication–computation overlap. Notably, both traces are plotted on the same time scale, clearly showing that CHARM+ completes execution in less time than Ring-Attention, demonstrating the effectiveness of CHARM+ in improving overall execution performance. C.2

Evaluation on High-end GPU

We validate our method on the high-performance H100 GPUs, with detailed configurations provided in Table 2. We conduct both forward and backward experiments under 2×8 and 4×8 setups (with a 15

hidden dimensions of 12 × 192). We additionally conducted experiments on ultra-long sequences, specifically those exceeding 100k tokens. Table 9 reports the forward results for the 2×8 configuration, Table 10 reports the corresponding backward results, Table 11 presents the forward results for the 4×8 configuration, and Table 12 presents the corresponding backward results. We report the CCR as well as the detailed computation time (Tcomp ) and communication time (Tcomm ). Our method shows better performance on 32 GPUs than on 16 GPUs. This is because the cross-node communication can be effectively overlapped with the intra-node pipeline. After decoupling via cross-hardware query parallelism, the only dependency is that cross-node query communication must finish before the intra-node pipeline begins. Accordingly, the query required for the next step can be communicated concurrently during the intra-node pipeline. Based on this analysis and results, our method consistently confines cross-node communication to the start and end, with intermediate cross-node communication effectively overlapped by the intra-node pipeline. In contrast, other approaches suffer from synchronization constraints, making them susceptible to the slower cross-node communication. Consequently, our method exhibits greater performance as the scale increases (speedups of 2.28×/3.69× on 16 H100 GPUs and 2.69×/4.21× on 32 H100 GPUs at maximum performance), demonstrating our advantage. In the scenario of ultra-long sequence lengths (over 200k tokens) on 16 H100 GPUs in Table 9 and Table 10, the CCR exceeds 1, indicating that the computation time is greater than the communication time. At this point, computation becomes the main bottleneck, diminishing the benefits of overlap of communication and computation. Nevertheless, our method still significantly outperforms all other methods. C.3

Numerical correctness verification

In Ring-Attention, updates on each GPU occur in different orders, and the only precision loss comes from BF16 → FP32 → BF16 conversion. This type of error exists in all distributed attention methods, and our method does not introduce any additional precision errors. We verified numerical correctness of our method. The results were obtained on an 8-GPU L20 setup. Specifically, we set the sequence length to 8K and generated 100 groups of randomly initialized Q, K, V , and output gradients, all sampled from a normal distribution. We executed FlashAttention, Ring-Attention, and CHARM+, respectively. By performing element-wise subtraction, we computed the difference matrices between RingAttention and FlashAttention, as well as between CHARM+ and FlashAttention. We further evaluated these differences by calculating the mean and maximum, and finally reported the averages over the 100 runs, as summarized in Table 8. Table 8: Comparison of numerical differences between CHARM+ and Ring-Attention. Diff

CHARM+ (Max)

CHARM+ (Mean)

Ring (Max)

Ring (Mean)

out lse dq dk dv

8.34147e-04 1.65642e-06 1.26290e-03 2.87713e-03 4.47177e-03

4.51229e-05 2.94155e-07 3.29077e-05 3.53419e-05 3.42940e-05

8.07648e-04 1.75893e-06 1.25977e-03 2.89413e-03 4.48120e-03

4.24147e-05 3.04284e-07 3.27659e-05 3.53193e-05 3.42894e-05

These results demonstrate that the numerical differences between CHARM+ and FlashAttention remain at the same order of magnitude as those between Ring-Attention and FlashAttention, indicating that CHARM+ does not introduce additional numerical errors.

16

Table 9: Forward performance comparison on 16-H100. B

Seq

Tcomm

Tcomp

CCR

Ring

TASP

CHARM+

Speedup

86.93 171.91

66.74 125.43

1.75× 1.92×

26.36 51.46 101.74

21.43 42.28 81.88

2.06× 2.11× 2.22×

25.81 50.29 99.39

18.25 34.36 66.91

2.15× 2.30× 2.34×

13.56 25.64 49.75 98.01

9.43 17.03 32.60 63.93

1.95× 2.22× 2.30× 2.35×

8.18 13.56 25.48 49.27 97.30

5.89 9.07 16.57 31.84 62.34

1.55× 1.97× 2.16× 2.28× 2.29×

8.19 13.47 25.27 49.23 97.65

5.63 8.91 16.31 31.53 61.80

1.62× 1.98× 2.19× 2.25× 2.29×

8.10 13.37 25.13 49.06 97.15

5.62 8.82 16.23 31.21 61.54

1.63× 1.98× 2.13× 2.26× 2.28×

Seq = 215,040 1 2

215,040 215,040

81.49 164.31

86.01 171.51

1.06 1.04

116.76 241.03

Seq = 107,520 1 2 4

107,520 107,520 107,520

38.34 74.25 146.17

23.16 45.20 88.94

0.60 0.61 0.61

44.08 89.20 182.15

Seq = 53,760 2 4 8

53,760 53,760 53,760

38.27 74.39 146.24

14.39 26.50 51.13

0.36 0.36 0.35

39.18 78.95 156.57

Seq = 26,880 2 4 8 16

26,880 26,880 26,880 26,880

20.13 38.24 74.44 146.53

5.63 9.87 17.22 32.05

0.28 0.26 0.23 0.22

18.35 37.90 75.12 149.97

Seq = 13,440 2 4 8 16 32

13,440 13,440 13,440 13,440 13,440

10.89 20.31 38.31 74.55 145.95

2.92 4.10 6.87 11.83 21.79

0.27 0.20 0.18 0.16 0.15

9.15 17.89 35.72 72.65 143.05

Seq = 6,720 4 8 16 32 64

6,720 6,720 6,720 6,720 6,720

10.92 20.12 38.26 74.53 146.14

2.53 3.27 4.61 7.90 17.05

0.23 0.16 0.14 0.12 0.11

9.12 17.59 35.76 70.84 141.75

Seq = 3,360 8 16 32 64 128

3,360 3,360 3,360 3,360 3,360

10.98 20.17 38.23 74.53 146.20

2.55 3.05 4.61 7.90 14.36

0.25 0.15 0.12 0.11 0.10

9.17 17.49 34.50 70.54 140.55

17

Table 10: Backward performance comparison on 16-H100. B

Seq

Tcomm

Tcomp

CCR

Ring

TASP

CHARM+

Speedup

– –

207.07 405.90

1.51× 1.55×

– – –

64.09 125.01 242.15

1.79× 1.83× 1.86×

– – –

45.91 88.55 173.18

2.47× 2.55× 2.60×

– – – –

20.42 37.51 72.59 141.52

2.79× 2.99× 3.12× 3.18×

– – – – –

10.81 18.48 33.84 65.35 128.53

2.67× 3.09× 3.34× 3.47× 3.49×

– – – – –

10.22 17.34 32.38 62.46 123.32

2.85× 3.27× 3.48× 3.59× 3.64×

– – – – –

10.04 17.06 32.14 62.20 121.99

2.80× 3.33× 3.51× 3.61× 3.69×

Seq = 215,040 1 2

215,040 215,040

218.48 437.82

257.76 503.06

1.18 1.15

312.45 628.21

Seq = 107,520 1 2 4

107,520 107,520 107,520

109.45 218.37 436.10

68.03 135.44 264.15

0.62 0.62 0.61

114.64 228.70 451.53

Seq = 53,760 2 4 8

53,760 53,760 53,760

110.04 219.95 439.16

37.64 74.53 145.33

0.34 0.34 0.33

113.30 226.22 450.33

Seq = 26,880 2 4 8 16

26,880 26,880 26,880 26,880

56.73 112.12 221.73 442.86

11.47 22.35 43.86 86.72

0.20 0.20 0.20 0.20

57.03 112.24 226.13 449.58

Seq = 13,440 2 4 8 16 32

13,440 13,440 13,440 13,440 13,440

28.73 56.86 112.90 222.95 444.64

4.50 8.42 15.44 29.43 57.34

0.16 0.15 0.14 0.13 0.13

28.93 57.08 113.19 226.46 448.54

Seq = 6,720 4 8 16 32 64

6,720 6,720 6,720 6,720 6,720

28.88 56.98 112.98 224.56 445.33

3.23 5.94 10.90 20.72 40.32

0.11 0.10 0.10 0.09 0.09

29.11 56.67 112.72 224.23 448.84

Seq = 3,360 8 16 32 64 128

3,360 3,360 3,360 3,360 3,360

28.72 56.94 112.99 224.63 445.33

2.74 4.98 9.41 18.13 35.34

0.10 0.09 0.08 0.08 0.08

18

28.11 56.74 112.68 224.41 450.01

Table 11: Forward performance comparison on 32-H100. B

Seq

Tcomm

Tcomp

CCR

Ring

TASP

CHARM+

Speedup

94.17 184.59

39.95 77.02

2.56× 2.70×

48.54 93.60 184.12

19.80 36.62 70.62

2.34× 2.62× 2.73×

25.38 48.38 93.35 183.22

11.47 18.91 35.27 68.17

1.97× 2.39× 2.62× 2.72×

14.09 25.24 48.23 93.30 183.29

9.19 10.93 18.47 34.64 66.85

1.31× 2.04× 2.42× 2.58× 2.70×

14.09 25.31 48.34 93.28 182.93

8.97 10.83 18.34 34.44 66.48

1.34× 2.06× 2.40× 2.60× 2.69×

14.08 25.23 48.17 93.11 183.09

8.69 10.90 18.39 34.44 66.43

1.37× 2.04× 2.43× 2.60× 2.69×

Seq = 126,976 2 4

126,976 126,976

95.28 186.17

36.30 70.03

0.38 0.38

102.34 208.12

Seq = 63,488 2 4 8

63,488 63,488 63,488

49.95 96.13 187.15

12.50 22.28 41.96

0.25 0.23 0.22

46.32 95.79 193.04

Seq = 31,744 2 4 8 16

31,744 31,744 31,744 31,744

26.70 50.19 96.07 187.54

6.28 8.99 15.58 28.46

0.24 0.18 0.16 0.15

22.59 45.25 92.39 185.42

Seq = 15,872 2 4 8 16 32

15,872 15,872 15,872 15,872 15,872

15.54 26.73 50.16 96.20 186.74

4.98 5.09 7.13 11.93 21.26

0.32 0.19 0.14 0.12 0.11

12.06 22.29 44.77 89.27 180.66

Seq = 7,936 4 8 16 32 64

7,936 7,936 7,936 7,936 7,936

15.63 27.24 50.01 96.10 186.69

4.96 5.01 6.33 10.17 17.81

0.32 0.18 0.13 0.11 0.10

12.02 22.31 43.95 89.66 179.13

Seq = 3,968 8 16 32 64 128

3,968 3,968 3,968 3,968 3,968

15.55 27.24 50.60 96.26 189.89

4.71 5.12 6.31 9.56 16.43

0.30 0.19 0.12 0.10 0.09

11.91 22.29 44.73 89.71 178.51

19

Table 12: Backward performance comparison on 32-H100. B

Seq

Tcomm

Tcomp

CCR

Ring

TASP

CHARM+

Speedup

– –

111.60 213.85

2.51× 2.62×

– – –

46.30 86.70 166.44

3.05× 3.24× 3.37×

– – – –

24.09 40.55 75.16 146.01

2.93× 3.49× 3.74× 3.83×

– – – – –

14.92 22.11 37.75 70.62 137.84

2.40× 3.23× 3.75× 3.96× 4.05×

– – – – –

14.51 21.27 36.61 69.02 134.48

2.42× 3.34× 3.86× 4.03× 4.14×

– – – – –

13.97 20.95 36.35 68.40 133.05

2.57× 3.41× 3.90× 4.09× 4.21×

Seq = 126,976 2 4

126,976 126,976

259.62 521.26

102.57 196.56

0.40 0.38

280.48 559.74

Seq = 63,488 2 4 8

63,488 63,488 63,488

138.65 269.38 529.05

29.95 58.46 111.96

0.22 0.22 0.21

141.18 280.96 560.30

Seq = 31,744 2 4 8 16

31,744 31,744 31,744 31,744

70.47 140.16 275.78 536.42

10.16 19.39 37.57 73.45

0.14 0.14 0.14 0.14

70.57 141.71 280.92 559.34

Seq = 15,872 2 4 8 16 32

15,872 15,872 15,872 15,872 15,872

35.78 70.70 140.84 277.48 548.62

4.67 7.28 13.69 26.20 50.62

0.13 0.10 0.10 0.09 0.09

35.77 71.35 141.53 279.85 558.53

Seq = 7,936 4 8 16 32 64

7,936 7,936 7,936 7,936 7,936

35.82 70.72 140.74 278.53 550.51

3.71 5.82 10.90 20.68 39.76

0.10 0.08 0.08 0.07 0.07

35.18 71.06 141.24 278.38 556.25

Seq = 3,968 8 16 32 64 128

3,968 3,968 3,968 3,968 3,968

35.79 70.73 140.73 278.31 551.62

3.24 5.16 9.70 18.40 35.58

0.09 0.07 0.07 0.07 0.06

20

35.86 71.33 141.84 279.62 560.63

Related documents

• ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding#1456542
• ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding#1450757
• Decoupling Logical Masks from GPU Execution for Dynamic Block-Sparse Attention#1061497
• A Hybrid GPU and Processing-Near-Memory Architecture with Shared DRAM for Efficient Prefill-Decode Disaggregation#1401310
• A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer#1216676