VarioPath: Workload-Aware All-to-All Communication for PCIe GPU Clusters Yao Fei1 Jin Fang1 Size Zheng2 Gongming Zhao1 Hongli Xu1 Jiacheng Zhu1 Zhijing Xin1 1 University of Science and Technology of China
2 Tsinghua University
arXiv:2609.34340v1 [cs.DC] 28 Sep 2026
{yao_fei,fangjin98,zhu_jc,zhijingxin}@mail.ustc.edu.cn {gmzhao,xuhongli}@ustc.edu.cn
Abstract
ferent volumes of data to different peers. For example, inputdependent expert routing can produce a skewed traffic distribution [15, 16, 20], with a few destinations receiving substantially more data than others. AlltoAllv’s many-to-many exchange pattern complicates the coordination of concurrent transfers and limits efficient utilization of interconnect bandwidth [9, 16, 17, 22]. Consequently, AlltoAllv communication often becomes a major bottleneck in distributed large-model inference. This communication bottleneck is even more pronounced on PCIe GPU systems. In tightly integrated GPU servers (e.g., NVIDIA HGX H100 [27]), a dedicated NVLink/NVSwitch scale-up fabric [28] provides high-bandwidth, non-blocking intra-node GPU connectivity [34]. By contrast, PCIe systems connect GPUs and NICs through shared PCIe switches and links. These interconnects vary across server designs and do not guarantee non-blocking communication [17,22,32]. Their bandwidth is also substantially lower than that of dedicated scale-up networks. Despite these limitations, PCIe systems are widely deployed for inference because their availability and customizability help control infrastructure cost and meet different resource requirements [9, 17]. For example, Alibaba and ByteDance deploy PCIe-connected NVIDIA T4 and L40S GPUs in production clusters comprising thousands of GPUs [39, 45]. Cloudflare also deploys multiple types of PCIe GPUs for its Workers AI service [18]. On these widely deployed PCIe GPU systems, AlltoAllv presents three challenges: link contention, traffic skew, and dynamic demand. First, the typically non-Clos interconnect can concentrate traffic on the same bottleneck links [22, 32], while the absence of a separate scale-up network makes intranode GPU communication and NIC-based inter-node traffic compete for the same PCIe resources. Traffic therefore concentrates on some links, creating hotspots while leaving others underutilized and reducing aggregate throughput. Second, AlltoAllv demand can be highly skewed, requiring some GPUs to transfer much more data than others. This imbalance keeps certain GPUs and PCIe links busy after others have finished, creating a communication tail that delays the
AlltoAllv communication is a critical primitive in distributed large-model inference, particularly for mixture-of-experts (MoE) models. The growing adoption of PCIe GPU systems for cost-efficient inference makes AlltoAllv performance on these systems increasingly important. Without a dedicated scale-up interconnect (e.g., NVLink or Infinity Fabric), PCIe GPU systems carry both intra-node and inter-node traffic through the PCIe hierarchy, where concurrent transfers can contend for PCIe link bandwidth. This link contention, compounded by skewed traffic distributions and dynamic traffic demand, makes efficient AlltoAllv scheduling challenging. Existing approaches are either poorly suited to PCIe GPU systems or incur substantial schedule synthesis overhead that reduces their practicality in real-world deployments. We present VarioPath, an efficient AlltoAllv scheduling framework for PCIe GPU systems. It combines an offline topology-aware analyzer with an online demand-aware scheduler. The analyzer records contention-free transfer patterns as AlltoAllv channels and exploits topology symmetry to build a compact catalog for efficient search. The online scheduler decomposes each AlltoAllv invocation’s demand across a sequence of channels, adapting to rapidly changing and skewed traffic while incurring low planning overhead. Evaluation on four platforms (up to 256 GPUs) shows average AlltoAllv speedups of 5.88× over FAST and 1.72× over DeepEP. Endto-end experiments show that VarioPath reduces Qwen3 inference latency by up to 27.2% and Wan2.1 generation latency by 6.1%.
1
Introduction
AlltoAllv is a critical communication primitive for distributed model inference. In expert parallelism (EP) [8, 11, 20, 21, 31, 33], it dispatches activations to experts and returns their outputs. In sequence parallelism (SP) [10, 14], it redistributes attention tensors across the sequence and head dimensions. An AlltoAllv invocation may require each GPU to send dif1
entire invocation even without link contention. Finally, dynamic AlltoAllv demand makes static schedules impractical and requires schedulers to adapt in real time. Prior work [3, 8, 20, 23, 32, 38] primarily targets tightly integrated servers with dedicated scale-up networks. For example, FAST [20] and DeepEP [8] rebalance AlltoAllv traffic by introducing additional intra-node transfers over dedicated, high-bandwidth scale-up networks. On PCIe GPU systems, however, these approaches exacerbate link contention because the additional intra-node transfers compete with NIC traffic for the same shared PCIe links. General-purpose libraries such as NCCL [29] and MSCCL [25] incur little runtime scheduling overhead, but their static scheduling strategies cannot effectively adapt to dynamic AlltoAllv demands in realworld workloads. Only a few systems, including Blink [38], BlueConnect [4], and TCCL [17], explicitly optimize collective communication for PCIe GPU systems. However, they focus primarily on regular collectives such as AllReduce [30] and AllGather [1], often using ring-based algorithms. Their optimizations therefore do not directly apply to AlltoAllv. MILP-based schedulers [3, 23, 32, 40] jointly model network topology and communication demand but can take seconds to hours to synthesize a schedule. Because AlltoAllv demand changes across invocations, repeatedly running this synthesis is impractical for real-world workloads. Addressing the above challenges while maintaining low scheduling overhead is difficult on PCIe GPU systems. This is because avoiding link contention requires time-consuming topology analysis, while dynamic AlltoAllv demand requires schedules to adapt to every invocation. Our key insight is that the PCIe topology remains stable and determines which concurrent transfer patterns avoid link contention. These patterns can therefore be identified in advance and recorded as reusable AlltoAllv channels, each comprising concurrent GPU pairs and their communication paths. These channels distill the topology’s communication capabilities into reusable scheduling primitives. Channels offer different distributions of transfer capacity, making them suitable for different communication demands. On the demand side, skewed AlltoAllv demand can be divided into chunks and mapped to a sequence of channels in real time. This decomposition balances traffic across different paths to mitigate communication tails. Overall, this separation makes effective, low-overhead AlltoAllv scheduling possible. Based on this insight, we present VarioPath, combining an offline topology-aware analyzer with an online demand-aware scheduler. To avoid enumerating and storing an exponentially growing channel space, the offline analyzer exploits topology symmetry to represent channels compactly as channel families. It then organizes these families in a tree-structured catalog, enabling successive filtering of candidates. At runtime, the scheduler maps demand chunks to these families, jointly selecting channels and allocating transfer volumes to their paths. The scheduler prioritizes communication bottlenecks
when matching channels to the current demand. Reusing the offline catalog allows it to achieve near-optimal schedules with low scheduling overhead. We make the following contributions: • We present VarioPath, an AlltoAllv scheduling framework for PCIe GPU systems. By separating topologydependent offline analysis from demand-dependent online scheduling, VarioPath enables topology-aware and demand-adaptive AlltoAllv scheduling with low runtime overhead. • We distill topology information into the AlltoAllv channel abstraction, with each channel representing a contention-free concurrent transfer pattern. The offline analyzer exploits topology symmetry to organize channels into a compact tree-structured catalog for efficient search, while the bottleneck-aware online scheduler decomposes each invocation’s demand and allocates it across a sequence of channels. • We validate VarioPath on four PCIe GPU platforms, from single-node servers to 256 GPUs, using collective benchmarks and end-to-end Qwen3 and Wan2.1 inference. VarioPath consistently outperforms state-of-the-art systems, achieving up to a 5.88× AlltoAllv speedup and reducing inference latency by up to 27.2%.
2
Background and Motivation
2.1 AlltoAllv Scheduling Challenges on PCIe Systems An AlltoAllv invocation is described by a matrix of pairwise data volumes, where each entry specifies the data sent from one source rank to one destination rank. Support for independently sized rank-pair transfers makes AlltoAllv well suited to key data-redistribution operations in distributed large-model inference, including token dispatch and output return in expert parallelism (EP) [8, 12, 20] and attention-tensor redistribution in sequence parallelism (SP) [10, 14]. In MoE inference, token-to-expert assignments are recomputed at every layer, producing a new AlltoAllv demand matrix. With per-layer execution times often below one millisecond [24, 42, 43], the scheduler may need to adapt to each new demand matrix on a similarly short timescale. Otherwise, scheduling can delay the critical path and erase the latency gains of a better communication plan. Following prior pairwise-exchange schedules [1,20,26,36], all-to-all exchanges are commonly divided into endpointdisjoint stages. Within each stage, every rank sends to at most one peer and receives from at most one peer. Because each rank receives from at most one peer within a stage, this scheduling structure avoids receiver-side incast, where multiple senders compete for a receiver’s ingress bandwidth and 2
A fA fA0 fAlltoAllv Stage Stage Stage A AA 0 AlltoAllv 0 AlltoAllv
G0 G0G0
A A A21.3GB/s 21.3GB/s 21.3GB/s G1 G1G1 f 1 f 1f 1 21.3GB/s 21.3GB/s 21.3GB/s 21.3GB/s 21.3GB/s fA2 fA2fA21.3GB/s 2
G2 G2G2
G3 G3G3 G0 G0G0
B fB0 fBAlltoAllv Stage Stage Stage B B B 0f AlltoAllv 0 AlltoAllv
G3 G3G3 G0 G0G0
CAlltoAllv AlltoAllv Stage Stage Stage C C C fC0 fCAlltoAllv 0f 0 64GB/s 64GB/s 64GB/s
G3 G3G3
G4 G4G4 G1 G1G1
32GB/s 32GB/s fB1 fB1fB1 32GB/s
G4 G4G4 G1 G1G1
fC1 fC1fC1
G4 G4G4
G5 G5G5 G2 G2G2
fB2 fB2fB2
32GB/s 32GB/s 32GB/s G5 G5G5 G2 G2G2
50GB/s 50GB/s 50GB/s N0 N0N0
Inter-node Inter-node Inter-node Network Network Network
N1 N1N1 N0 N0N0
src src0src 0 1 0 1 2 1 23 2 34 3 45 4 5total_bw. 5 total_bw. dst dst3dst 3 4 3 4 5 4 50 5 01 0 12 1 2 2 total_bw. pathpath path 128GB/s B BBB BBB BBB BBB BBB B 128GB/s B 128GB/s A AA A AA A A A fA0 fAfA0f1Af0Af1Af2Af1AfA 2f3 f2 f 3f4 f3 f 4f5 f4 5f 5
fC2 fC2fC2
64GB/s 64GB/s 64GB/s G5 G5G5
50GB/s 50GB/s 50GB/s N1 N1N1 N0 N0N0
Inter-node Inter-node Inter-node Network Network Network
Inter-node Inter-node Inter-node Network Network Network
N1 N1N1
src src0src 0 1 0 1 2 1 23 2 34 3 45 4 5 5 total_bw. dst dst3dst 3 4 3 4 5 4 50 5 01 0 12 1 2total_bw. 2 total_bw. pathpath path B BBB BN0BN0BN0BBB BN1BN1 N228GB/s 228GB/s 1 228GB/s
src src0src 0 1 0 1 2 1 23 2 34 3 45 4 5 5 total_bw. dst dst1dst 1 4 1 4 5 4 50 5 03 0 32 3 2total_bw. 2 total_bw. pathpath path I I B I BN0BN0BN0BI B IN1I N1 N356GB/s 356GB/s 1 356GB/s
(b) Path-only adaptation
(c) Joint peer–path selection
fB0 fBf0Bf1Bf0Bf1Bf2B1fBf2Bf3Bf2Bf3Bf4Bf3BfB4f5Bf4B5fB5
(a) Fastest-path assignment
fC0 fCf0Cf1C0fCf1Cf2Cf1Cf2Cf3Cf2Cf3Cf4Cf3Cf4Cf5Cf4C5fC5
Figure 1: Joint peer–path selection avoids PCIe contention. Panels (a)–(c) compare fastest-path assignment, path-only adaptation, and joint peer–path selection, whose modeled aggregate bandwidths are 128, 228, and 356 GB/s, respectively. Each panel depicts an endpoint-disjoint stage from the same AlltoAllv invocation. Each flow f represents a logical transfer together with its selected physical path. Transfers not selected in the current stage remain for later stages. For clarity, each topology draws only f0X – f2X , while its table and aggregate bandwidth include all six logical transfers selected in the stage. The table below each topology lists every source’s destination and selected path. I, B, and N0 /N1 denote a same-switch PCIe path (PIX), an inter-switch PCIe path (PXB), and NIC paths through NIC0/NIC1, respectively. For simplicity, we use nominal PCIe and NIC bandwidths of 64 and 50 GB/s, respectively. (a) Three AlltoAllv flows G0
G1
G2
N0
NIC 50 GB/s
NIC path PXB path
G3
G4 flow A 114MB
G5 flow B 64MB
rics are typically non-Clos [5, 17, 22, 32], so logical transfers that are endpoint-disjoint may still share bandwidth-limited PCIe links. Figure 1(a) shows that endpoint-disjoint logical transfers can still contend when their paths converge on the same PCIe link. The resulting contention reduces the effective rates of all paths using that bottleneck and increases the stage completion time while alternative paths remain underutilized. Accumulated across stages, this loss directly lowers end-to-end AlltoAllv throughput. Skewed demand creates long communication tails. AlltoAllv demand may be skewed, with some logical transfers carrying substantially more data than others. Some paths may consequently carry disproportionately heavy loads relative to their available rates. These paths remain active after more lightly loaded paths have finished and become idle, as illustrated in Figure 2(b). AlltoAllv completes only after its last logical transfer, so these stragglers create a long communication tail that delays the entire invocation.
(b) Whole-flow schedule PXB 64 GB/s
A · 114 MB B · 64 MB
idle C · 50 MB Time
Network
N1 flow C 50MB
(c) Chunk-level schedule Stage 1
Stage 2
PXB 64 GB/s
A1 · 64 MB
B · 64 MB
NIC 50 GB/s
C · 50 MB
A2 · 50 MB
Saved Time
Figure 2: Chunk-level path remapping mitigates communication tails. (a) Three cross-switch transfers can use either a 64-GB/s PXB or a 50-GB/s NIC path; block width in (b)–(c) denotes modeled transfer time. (b) Whole-flow binding creates imbalanced path loads: B and C serialize on the NIC, while the PXB path finishes early and becomes idle. (c) Chunking A enables path switching in Stage 2, balancing load and eliminating the tail. cause severe congestion [13,19]. However, even with pairwiseexchange scheduling, PCIe GPU systems still present two additional AlltoAllv scheduling challenges: Shared-link contention limits AlltoAllv throughput. A non-blocking NVLink/NVSwitch fabric [22] supports such endpoint-disjoint logical transfers without internal-link oversubscription. PCIe-based systems lack an independent scaleup fabric and therefore cannot guarantee contention-free execution of endpoint-disjoint transfers. Specifically, this limitation stems from two properties of PCIe systems. First, intranode P2P and NIC-based inter-node logical transfers traverse the same PCIe hierarchy and may contend for GPU-facing links and other shared PCIe links [32]. Second, PCIe fab-
2.2
Why Existing Designs Fall Short
General-purpose communication libraries. NCCL [29] incurs little runtime scheduling overhead but uses a static AlltoAllv schedule that does not adapt to changing communication demand. MSCCLang [6] schedules logical transfers by controlling their ordering and concurrency, while leaving their physical paths fixed. This is insufficient for PCIe systems, where logically independent transfers may still contend on shared physical links. Consequently, neither system can adapt its schedule to rapidly changing AlltoAllv demand. Demand-aware AlltoAllv schedulers for hierarchical systems. Some systems optimize AlltoAllv for servers with hier3
archical architectures that provide independent scale-up and scale-out networks. FAST [20] uses additional intra-node forwarding to balance scale-out traffic, whereas DeepEP [8] uses it to reduce inter-node traffic. On PCIe GPU systems without an independent scale-up fabric, however, these schemes may fail to deliver their intended gains and instead exacerbate PCIe contention: their additional intra-node transfers and NIC-based inter-node traffic traverse the same switched PCIe hierarchy, increasing load on shared PCIe links. The resulting contention reduces the effective bandwidth of concurrent transfers and ultimately lowers end-to-end AlltoAllv throughput. Topology-aware synthesis. TACCL [32], TE-CCL [23], and SCCL [2] synthesize a schedule by jointly considering a specified communication demand and a detailed model of physical paths and shared resources. Prior systems report synthesis times ranging from minutes to hours [2, 23, 32]. Regenerating such a schedule for every dynamic AlltoAllv invocation is therefore impractical, particularly at scale.
idle while slower ones determine completion. Figure 2(b) illustrates this mismatch: the PXB path becomes idle before the NIC path finishes. In (c), splitting transfer A and remapping its second chunk to NIC balances the two paths over two rounds. This example shows why path assignments must be revisited at chunk granularity: between rounds, outstanding traffic can be remapped to match load to path rates. Key insight. Together, these observations reveal a natural separation between topology and demand. The topology changes infrequently and determines which peer–path configurations can coexist without modeled contention; VarioPath precomputes these configurations as reusable channels. At runtime, VarioPath represents each invocation’s demand as a linear combination of channels and assigns invocationspecific data volumes to the selected paths. This separation replaces repeated combinatorial topology search with lightweight online demand decomposition.
2.3
Model setup. We model an AlltoAllv invocation on a fixed topology comprising a set V of N GPU ranks and a demand matrix D ∈ ZN×N ≥0 , where Di j denotes the data volume from rank i to rank j. Self-traffic is handled locally and omitted, so Dii = 0. For each ordered pair i ̸= j, let Pi j denote the nonempty set of available concrete paths. Each path p ∈ Pi j has an estimated rate bw(p) > 0 and uses a set of modeled resources L (p) ⊆ U . The resource set U contains the directed shared PCIe and NIC-facing resources considered by our model. We exclude endpoint-local GPU-to-switch links from U because the one-to-one peer matching defined below ensures that no two lanes in a channel share the same endpoint link in the same direction. Table 1 summarizes the notation used in Sections 3 and 4.
3
Design Observations and Key Insight
Observation 1: Avoiding PCIe link contention requires joint peer–path selection. Within an AlltoAllv stage, peer matching determines which logical transfers execute concurrently, while path assignment determines which physical resources they use. These decisions jointly determine contention: optimizing either one in isolation can still map multiple transfers onto the same bottleneck link while leaving alternative paths underutilized. Figure 1 compares three possible stages in an AlltoAllv schedule: fastest-path assignment (Stage A), path-only adaptation (Stage B), and joint peer–path selection (Stage C). In Stages A and B, the cyclic matching [20, 26, 36] uses k = 3, so rank i sends to (i + k) mod 6. Under this peer matching, path-only adaptation raises modeled aggregate bandwidth from 128 to 228 GB/s but leaves PXB transfers contending. Stage C jointly changes matching and paths, combining local PIX, noncontending PXB, and NIC transfers to eliminate modeled PCIe contention and reach 356 GB/s. Inspired by this observation, we use a channel to represent the static peer–path configuration of a stage: concurrent GPU pairs and one physical path per pair. Intuitively, a feasible channel contains peer–path choices whose transfers can execute concurrently without causing PCIe link contention, as illustrated by Figure 1(c). Because feasible channels combine different peer pairs and path rates, they provide different transfer capabilities. Their feasibility depends only on the topology, allowing them to be precomputed offline. Observation 2: Skewed demand requires adaptive chunkto-path mapping. AlltoAllv demand may be skewed, and each logical transfer may have multiple candidate paths with different rates. Whole-flow binding can therefore imbalance path loads even without link contention, leaving faster paths
Scheduling Model
Table 1: Key notation used in Sections 3 and 4. Symbol
Definition
V Va
Set of all GPU ranks. Switch-local group of topology-equivalent GPU ranks, indexed by a. Set of group-level path instances from source group Va to destination group Vd . Nonself permutation mapping each source rank i to a distinct destination π(i). Channel specifying permutation-defined peer pairs and one concrete communication path per pair. Channel activation obtained by assigning a nonzero lane-volume vector to a channel. Rank-level demand-contribution matrix served by activation E. Channel-shape matrix; Mad is the number of lanes from Va to Vd . Channel family B is a set of group-level path instances whose transfers can execute concurrently without sharing a modeled resource; catalog bucket B [M] contains all families that realize shape M. Unserved rank-level demand and its aggregation by ordered rank-group pair.
Rad π C E X(E) M B, B [M]
b Drem , D
4
OFFLINE
Channel definition. A channel C = (π, p) consists of a permutation π over V and a path vector p = (pi )i∈V . The mapping π assigns each source rank i to a distinct destination π(i) ̸= i, and pi ∈ Pi,π(i) selects the corresponding concrete path. Because π is a permutation, each channel contains exactly one outgoing and one incoming lane per rank, preventing same-direction fan-out and fan-in at an endpoint. A channel specifies only this peer–path pattern. A channel activation E = (C, x) represents an AlltoAllv stage, assigning a nonnegative integer volume xi to each lane i ∈ V . All nonzero-volume lanes execute concurrently in that stage. Specifically, X(E)i,π(i) = xi for i ∈ V , and all other entries are zero. Thus, X(E) records the rank-level demand served in that stage. Channel feasibility. A channel is feasible under our model when no two selected paths share a modeled directed resource:
∑ 1[ℓ ∈ L (pi )] ≤ 1,
∀ℓ ∈ U .
Catalog tree
Topology SW0
64GB/s
SW1
NIC0 0
1
2
3
NIC1 (Sec. 4.1)
64GB/s 64GB/s
channel analysis
ONLINE Traffic demand
(Sec. 4.2) 1 decompose demand
0 10MB
30MB
2
64GB/s
3
Channel family B1 Channel C1 Channel C2 Channel C3
Activation E1 10MB 10MB Activation E4 20MB
Figure 3: VarioPath’s workflow. Offline analysis stores compatible channel families in a reusable catalog tree; online channel-guided decomposition instantiates concrete channels and allocates each invocation’s demand across the resulting activations. avoiding repeated topology analysis for each demand. Online, each iteration instantiates candidate channels from the catalog, assigns lane volumes, and selects the highest-utility activation, progressively decomposing the demand.
(1)
i∈V
Opposite directions and parallel physical links are distinct resources. We conservatively make each shared PCIe link exclusive within a channel because concurrent use would make path rates traffic-dependent; transfers sharing it run in separate activations. Schedule validity. A schedule is an ordered sequence of activations S = ⟨E1 , . . . , EK ⟩, where Ek = (Ck , xk ). The schedule is valid if (i) every Ck is feasible and (ii) the activations exactly reconstruct the requested demand, ∑Kk=1 X(Ek ) = D. This equality formalizes the linear combination view from Section 2: each activation contributes a channel-shaped component X(Ek ), and these components sum to D. Optimization objective. We estimate schedule completion time under an execution model in which activations run sequentially and the lanes within each activation run concurrently. Let λ ≥ 0 denote a fixed per-activation overhead. Under this model, an activation’s modeled duration is its fixed overhead plus the completion time of its slowest lane. For Ek = (Ck , xk ) with permutation πk and paths pk , summing these durations gives K xk,i b T (S ) = ∑ λ + max , S ⋆ ∈ arg min Tb(S ). i∈V bw(pk,i ) S valid k=1 (2) The objective captures the trade-off between shortening communication tails and limiting the overhead incurred by additional activations.
4.1
Topology-Aware Channel Analysis
Topology abstraction with rank groups and path instances. As discussed in Section 2, concurrent transfers can contend on endpoint-local GPU-to-switch links and shared PCIe links, especially inter-switch links. Because every channel contains exactly one outgoing and one incoming lane per rank, no endpoint-local link is reused in the same direction. We therefore model only the remaining conflicts on shared PCIe and NIC-facing resources in U . We partition the N = Gg ranks into G switch-local rank groups Va , each containing g topology-equivalent ranks on the same PCIe switch. A group-level path instance r ∈ Rad records one possible sharedresource route from Va to Vd while omitting endpoint-local GPU-to-switch attachments. Endpoint binding adds these attachments later; in Figure 4, the same instance r1 can thus connect GPU 0 to GPU 3 or GPU 1 to GPU 4. Representing group-pair transfers as channel shapes. Because ranks within each group are topology-equivalent, their identities do not affect the channel’s group-level structure. Aggregating a channel’s lanes by ordered rank-group pair therefore preserves its concurrency pattern. We encode these lane counts in a matrix M, called a channel shape. Formally, M ∈ ZG×G ≥0 , where Mad is the number of lanes from Va to Vd . An admissible shape has balanced rows and columns:
4 VarioPath Design
∑ Mad = g ∀a, d
Overview. VarioPath constructs schedules through an offline– online workflow, as shown in Figure 3. Offline, it represents sets of feasible channels as channel families and organizes the channel families in a reusable catalog tree for the fixed topology. This catalog tree records feasible choices once,
∑ Mad = g ∀d.
(3)
a
These constraints allocate exactly g outgoing and g incoming positions to each group. Assigning the positions one-to-one to its g ranks gives every rank one outgoing and one incoming lane, as required by the permutation in the channel definition. 5
Algorithm 1 Offline catalog tree construction
A larger Mad assigns more concurrent lanes to Va -to-Vd traffic, allowing the scheduler to favor group pairs with greater demand. Grouping channels into channel families. A shape fixes the number of lanes between each ordered group pair but leaves their routes unspecified. A channel family B refines a shape M by selecting group-level path instances while leaving their rank-level endpoints unbound. Let Bad (r) ∈ {0, 1} indicate whether instance r ∈ Rad is selected. A family realizes M only if it satisfies:
Require: Path-instance sets {Rad }; group size g Ensure: Pruned channel-family catalog tree B 1: order the instances as R = (r1 , . . . , rm ); B ← ∅ 2: E XPLORE(1, ∅) 3: B [M] ← P RUNE FAMILIES(B [M]) for all M ∈ dom(B ) 4: return B 5: procedure E XPLORE(t, B) 6: if C OMPLETE(B) then 7: Mad ← |B ∩ Rad | for all (a, d) 8: insert B into B [M]; return 9: end if 10: if t > m or not S UFFIX OK(t, B) then return 11: E XPLORE(t + 1, B) 12: let rt ∈ Rad 13: if C ANA DD(rt , B) then 14: E XPLORE(t + 1, B ∪ {rt }) 15: end if
∑ Bad (r) = Mad , ∀(a, d),
r∈Rad
∑ ∑ Bad (r)1[ℓ ∈ L (r)] ≤ 1,
∀ℓ ∈ U .
(4)
a,d r∈Rad
The first constraint enforces shape consistency by selecting exactly Mad path instances for every ordered group pair (a, d). The second enforces resource compatibility by allowing each modeled directed resource ℓ ∈ U to appear in at most one selected instance. Each compatible family is stored in the bucket B [M]; hence, M is realizable exactly when B [M] ̸= ∅. Figure 4 shows that satisfying the balance constraints alone does not guarantee realizability: M1 has no compatible family, whereas B1 ∈ B [M2 ] realizes M2 . Instantiating a channel from a family. A family fixes compatible path instances but represents multiple rank-level channels because their endpoints remain unbound. To instantiate one channel, endpoint binding assigns each selected instance r ∈ Rad a pair (i, j) ∈ Γad , where Γad = {(i, j) ∈ Va × Vd : i ̸= j} excludes self-transfers. These choices are subject to one-to-one source and destination assignments within each group. Together with the balance constraints, a valid binding uses every rank exactly once as a source and once as a destination, yielding the permutation required by Section 3. Assigning (i, j) turns r into a concrete path p ∈ Pi j by adding its GPU-to-switch attachments. Under the group-equivalence assumption, these attachments are excluded from U and preserve the modeled footprint and configured rate: L (p) = L (r) and bw(p) = bw(r). Depth-first construction of the catalog tree. Algorithm 1 enumerates candidate path instances in a fixed order using a pruned binary DFS. At state (t, B), B is the partial family and rt the next candidate; the fixed order gives each subset a unique decision sequence. The exclude branch skips rt , whereas the include branch is explored only if C ANA DD preserves resource disjointness and the group-degree limits. S UFFIX OK stops when the remaining instances cannot fill every group’s g outgoing and g incoming slots. Because these tests reject only incompatible or uncompletable states, all valid families remain reachable. C OMPLETE holds once every group has g selected outgoing and incoming instances. Then B is valid; its group-pair counts determine shape M, and B is inserted into B [M]. Pruning and search cost. Dominance pruning operates
Catalog Tree Topo
M1 unrealizable
B2 ...
C1 C2 C3 ...
Channel family B
Channel C
No compatible family B[M1] = { }
V0
V1
0
3
1
4
B1
M2
M3 ... Group-level channel Group-level topology shape M 3 R00 V0 0
1
R01 r1 r2
V0
r3 r4 R
10
R11 3
4 V1
V1 3
2
5
1 V0
2 2
1 V1
r0
V0
r5 r1,r2 r3,r4
V1 2
5 C1
Figure 4: Topology-aware construction of the channelfamily catalog tree. Balance constraints define admissible shapes, while the group-level topology determines which shapes have compatible families. Here M1 is unrealizable because B [M1 ] = ∅, whereas B1 ∈ B [M2 ] realizes M2 . A oneto-one endpoint binding then instantiates B1 as channel C1 . at two levels. Before the DFS, P RUNE PATHS removes a path instance r when an alternative r′ for the same group pair satisfies L (r′ ) ⊆ L (r) and bw(r′ ) ≥ bw(r), with at least one strict inequality. Such an alternative uses no additional modeled resource and offers no lower rate. After the DFS, P RUNE FAMILIES removes exact duplicates and families dominated within the same shape. One family dominates another when their instances can be paired within every group pair with identical footprints and no lower rates, with at least one paired instance having a higher rate. Families with different footprints remain as alternatives because they expose different resource choices to endpoint binding. These two filters therefore target different costs: path-level pruning reduces the DFS branching factor, whereas family-level pruning limits the catalog alternatives exposed to endpoint binding and online scheduling. The DFS may visit O(2|R | ) candidate subsets in the worst case, although resource checks, suffix pruning, and the two dominance filters reduce the explored and stored choices in 6
Algorithm 2 Online channel-guided demand decomposition
practice. The catalog is built once per machine configuration and reused across invocations, keeping this topology search off the per-invocation critical path.
4.2
Require: Feasible demand D; complete catalog tree B ; τ > 0, λ ≥ 0; Alternating Algorithm (AA) hyperparameters Nstart , IAA > 0 Ensure: Schedule S with ∑E∈S X(E) = D 1: Drem ← D; S ← ⟨⟩ 2: while ∥Drem ∥1 > 0 do b ad ← ∑i∈V ∑ j∈V Drem for all (a, d) 3: D ij a d b ad Mad 4: M ⋆ ← arg maxM∈dom(B ) ∑a,d D 5: for B ∈ B [M ⋆ ] do 6: CB ← M ULTI S TARTAA(B, Drem , τ, Nstart , IAA ) 7: (xB ,UB ) ← E VALUATE(CB , Drem , τ, λ) 8: end for 9: B⋆ ← arg maxB∈B [M ⋆ ] UB 10: E ⋆ ← (CB⋆ , xB⋆ ) 11: S ← S ∥ ⟨E ⋆ ⟩ 12: Drem ← Drem − X(E ⋆ ) 13: end while 14: return S
Channel-Guided Demand Decomposition
Online scheduling procedure. Algorithm 2 repeatedly decomposes the remaining demand into channel activations; Figure 5 illustrates one iteration. The scheduler initializes Drem = D, where Drem is the demand not yet served. Each iteration traverses the catalog from coarse to fine. It first aggregates Drem by rank-group pair, scores all channel shapes, and selects the highest-scoring shape M ⋆ . It then uses the rank-level demand to bind every family B ∈ B [M ⋆ ] to endpoints, yielding a candidate channel CB . For each candidate, it assigns lane volumes and evaluates the full modeled utility UB . The scheduler appends the highest-utility activation E ⋆ , subtracts its served demand from Drem , and repeats until Drem = 0. This coarse-to-fine search narrows the topology choices at the group level while retaining demand-aware endpoint and volume assignment. Step 1: Bottleneck-aware channel-shape selection. A channel shape specifies group-pair lane counts, so the scheduler aggregates the remaining demand at the same granularity: b ad = ∑ ∑ Drem D ij .
Remaining demand
(5)
a,d
max
M∈dom(B )
s(M).
V2
V3
V0 0 1 10
81 27
10 02
01 00
V1 1 0 00
01 10
72 17
V2 1 0 02
01 00
01 10
V3 7 1
10 02
10 01
Group-level demand view
Channel shape
2 18 3 1 1. 1 2 17 3 2. Select shape 1 0 Aggregate 3 1 2 19 02 16 3 2 2 81 28
High-demand group pairs
01 10
5. Update remaining demand
Activation
b ad has more residual bytes to drain A group pair with larger D and is more likely to determine the completion tail if assigned b ad as its current insufficient concurrency. We therefore use D bottleneck pressure. Because Equation 3 gives every admissible shape the same total lane budget, allocating more lanes to one group pair necessarily reduces the concurrency available to others. We score each shape by M ⋆ = arg
V1
17
i∈Va j∈Vd
b ad Mad , s(M) = ∑ D
V0
0200 0020 0002 2000
3. Bind endpoints
4. Assign volumes & select Candidate
Figure 5: One iteration of channel-guided demand decomposition. The group-level view summarizes the remaining demand Drem by rank-group pair and guides the selection of channel shape M ⋆ . For each family B ∈ B [M ⋆ ], rank-level endpoint binding instantiates candidate channel CB . Candidate volumes and utilities determine activation E ⋆ , which updates Drem . Channels are shown schematically.
(6) their minimum measures the useful assignment rate. We score a complete binding by
The objective directs the fixed concurrency budget toward high-pressure pairs, while restricting the search to dom(B ) guarantees topology feasibility. Recomputing the pressures and M ⋆ after each update to Drem lets the scheduler follow the evolving communication bottleneck. Step 2: Binding paths to form candidate channels. For each family B under M ⋆ , the scheduler binds its path instances one-to-one to rank-level GPU pairs, producing a candidate channel CB . Let τ be a fixed invocation-wide time quantum, and let sB (r) and dB (r) denote r’s assigned endpoints. For r ∈ Rad and a valid pair (i, j) ∈ Γad , define the useful-rate credit rem Di j hi jr = min , bw(r) , (7) τ
FB (sB , dB ) = ∑ hsB (r),dB (r),r .
(8)
r∈B
Thus, τFB is its provisional useful byte volume. Because each credit jointly depends on both endpoints, optimizing both one-to-one assignments yields a bilinear assignment problem (BAP) [7]. We search for a high-scoring binding with the Alternating Algorithm (AA) [35]. Fixing either endpoint assignment reduces the remaining optimization to a standard assignment problem, so AA alternately optimizes the source and destination assignments. Each update solves one g × g assignment problem per rank group without decreasing FB . A run ends when neither update improves the score or the sweep limit is reached. Figure 6 illustrates this process, where the displayed binding credit rises from 5 to 15.
with zero credit outside Γad . The two terms are the demand rate needed within one quantum and r’s supported rate, so 7
0
3
1
4
2
5
5 Family Equal rate estimates
0
3
1
4
2
5
VarioPath comprises about 3,000 lines of C++ for the offline
and online schedulers and 5,000 lines of C++/CUDA for its runtime communication kernels. VarioPath implements all inter-node communication with NCCL GPU-Initiated Networking (GIN) [29] over the GDAKI backend. Overall, the implementation combines topology-aware path profiling with device-resident data movement and synchronization, specializes transfers for sparse expert traffic and small messages, and adapts execution granularity to each demand. Topology, path, and rate model. At setup, we reconstruct the topology from the GPU/HCA PCIe hierarchy and socket affinity, enumerate supported direct-P2P and NIC paths, and record their directed shared-resource footprints and bandwidths. Using this information, we construct the group-level path instances described in Section 4.1. The runtime loads the resulting profile once and reuses it across calls. Device-resident buffer layout. AlltoAllv may scatter each destination’s data across noncontiguous user-buffer regions. Sending fragments separately creates small transfers and underutilizes links. A customized packing kernel groups elements by destination rank and packs each peer’s metadata and payload contiguously into the symmetric send buffer; the receive kernel unpacks them directly into the output layout. This fused layout conversion eliminates a separate memoryreordering pass and avoids fine-grained transfers. To avoid the high cost of a global barrier, measured at approximately 20 µs on our 32-GPU RTX 5090 testbed, we use double buffering to eliminate the global barriers otherwise required before and after resetting signals. We implement this scheme with two symmetric-memory slots, each containing independent payloads, signal counts, indices, and other invocation-related state variables. In practice, CUDA Graphs capture and replay these communication operations. To avoid host–device state inconsistencies across replays, the counters, slot indices, and related state reside in device memory rather than host memory. Efficient sparse data movement for expert parallelism. In sparse AlltoAllv workloads, source–destination rank pairs with zero demand require neither data transfer nor completion waits. For a destination rank hosting L local experts, each active source–destination pair uses a 32-bit control word: the lower L bits indicate which local experts receive data, while the remaining usable high-order bits encode the exact token count for a designated local expert. Each local expert is mapped to a queue pair (QP), using a dedicated QP when resources permit and sharing QPs round-robin otherwise. The sender posts the designated expert’s payload followed by the control word on the same QP, so NIC-level in-order execution guarantees data-before-control ordering without an additional fence. The receiver uses the bitmap to skip inactive experts and obtains active-expert counts from GIN signals, avoiding separate count transfers and output-count atomics for empty
(a) Inputs to Endpoint Binding
0
3 4
4 1
5 3
0
3 4
4 1
5 3
1
5
8
2
1
5
8
2
7
3
5
2
Alternating 3 5 Algorithm 2 7 After optimization Initial binding
1
4
2
3
Candidate
(b) Binding-credit matrices and the Alternating Algorithm
Figure 6: Endpoint binding with the Alternating Algorithm. (a) Family B provides two unbound, equal-rate path instances from V1 to V2 . (b) Colored entries in the credit matrices hi jr mark the current endpoint assignments. Alternating source and destination optimization raises FB from 5 to 15 and produces candidate channel CB . Other family slots are omitted for clarity. To reduce initialization sensitivity, M ULTI S TARTAA runs AA from Nstart feasible initial endpoint bindings. Each run stops after IAA sweeps or when a complete sweep does not improve FB . The procedure returns the highest-scoring channel CB = (πB , pB ) for evaluation in Step 3. Step 3: Allocating volumes and selecting an activation. For candidate CB , E VALUATE assigns each lane the largest volume allowed by both its path capacity within τ and its remaining demand: n o xB,i = min τbw(pB,i ), Drem (9) i,πB (i) . We compare the resulting activations by utility UB : useful bytes divided by modeled duration, including the slowestlane time and per-activation overhead from Section 3. We define UB = 0 for a candidate with ∑i xB,i = 0; otherwise, UB =
∑i∈V xB,i xB,i . λ + maxi∈V bw(pB,i )
Implementation
(10)
The candidate with the largest UB supplies the next activation E ⋆ . For feasible nonzero demand and the complete catalog produced by Algorithm 1, a positive-scoring M ⋆ admits at least one positive-volume candidate. Thus, every iteration reduces Drem . Planning complexity. Let J = | dom(B )| and H ⋆ = |B [M ⋆ ]|. Aggregating demand and scoring all shapes cost O(N 2 + JG2 ) per iteration. Each AA sweep solves 2G assignments of size g × g; with Nstart starts and at most IAA sweeps, binding and evaluating all families cost O H ⋆ (Nstart IAA Gg3 + N) . Thus, K activations cost O K[N 2 + JG2 + H ⋆ (Nstart IAA Gg3 + N)] ; Section 6.4 measures the resulting CPU overhead. 8
Name R5080-CX8 R5080-BF3 R5090-BF3 L20-CX7
GPU
#GPUs per node
PCIe generation
PCIe BW (GB/s)
NIC
# NIC per node
NIC BW/node (Gb/s)
PCIe topology
RTX 5080 RTX 5080 RTX 5090 L20
8 8 16 16
Gen5 Gen5 Gen5 Gen4
64 64 64 32
ConnectX-8 BF3-mini BF3-mini ConnectX-7
4 4 8 8
3200 1600 3200 3200
Tree Tree Ring Tree
Table 2: Evaluation platforms. PCIe generation and bandwidth describe each GPU’s PCIe version and nominal one-way capacity. GPU/NIC counts and aggregate NIC bandwidth are per node. groups. Low-latency protocol. GIN signal mode appends an RDMA atomic notification after the payload, incurring additional notification overhead. NCCL LL avoids the separate notification by embedding 8 bytes of flags alongside every 8 bytes of payload in each 16-byte FIFO line, reducing payload utilization to 50%. To achieve ultra-low latency without this overhead, we design a specialized protocol. Receivers initialize idle buffers to a sentinel value (e.g., negative zero), and senders issue unsignaled puts. Receivers poll 16-byte vectors until no sentinel remains. In a 1 KiB protocol microbenchmark on two RTX 5090 servers, P50 latency is 7.872 µs; GIN signal and prepacked NCCL LL take 12.928 and 9.726 µs, respectively. To avoid split PCIe TLPs, we pad the receive-slot stride so that each token’s receive address is 128-byte aligned. Global execution granularity. Each invocation begins with a target round count Ktarget , the approximate number of scheduling rounds over which the current demand should be drained. Given Ktarget , the current demand, and estimated path rates, the runtime derives a global time quantum τ so that the transfer is expected to complete in roughly Ktarget rounds. A smaller Ktarget therefore yields a larger τ and fewer rounds, reducing scheduling and synchronization overhead, whereas a larger Ktarget yields a smaller τ and enables finer-grained adaptation of peers and paths as demand drains. In practice, for an N-rank invocation, we set Ktarget between N and 3N.
6
testbed comprises 32 R5080-CX8 nodes, 8 R5080-BF3 nodes, 4 R5090-BF3 nodes, and 2 L20-CX7 nodes. The nodes are interconnected through a two-tier leaf–spine network. Workloads. We evaluate VarioPath with random-demand AlltoAllv, skew AlltoAllv, AlltoAll, and end-to-end inference. In AlltoAll, every rank sends the same volume to every other rank. Random-demand AlltoAllv uses demand matrices derived from randomized top-k MoE routing over 256 experts, with k = 8. Following FAST [20], we generate skew AlltoAllv by drawing pairwise volumes from a Zipfian distribution. Our end-to-end workloads cover expert-parallel Qwen3 prefill and sequence-parallel Wan2.1 generation. A separate 256-GPU experiment evaluates VarioPath’s large-scale scalability. Metrics. Our primary communication metric is algorithmic bandwidth, computed as the total transferred payload divided by the product of the number of ranks and completion time [20]. Higher is better. Because skew AlltoAllv can make ranks finish at different times, invocation completion time is the maximum communication time across ranks. Aggregate comparisons use geometric-mean speedup over matched points. End-to-end experiments report latency. Planning experiments report online CPU time per emitted activation and the ratio of total online-planning time to communication time; lower is better for both. Baselines. Across the communication and end-to-end experiments, we compare VarioPath with NCCL [29], MSCCL [6], FAST [20], and DeepEP [8]1 MSCCL executes user-defined schedules that control transfer ordering and concurrency. FAST is built on NVSHMEM and rebalances skewed traffic within each node before balanced one-to-one inter-node transfers. DeepEP provides specialized MoE dispatch and combine kernels over NCCL GIN.
Evaluation
We evaluate VarioPath to answer four key questions: • How does VarioPath compare with state-of-the-art systems across collective workloads, transfer sizes, and deployment scales (§6.1)?
6.1
• What end-to-end latency reductions does VarioPath provide for MoE and sequence-parallel inference workloads (§6.2)?
Collective Communication Performance
We first evaluate random-demand AlltoAllv across transfer sizes and scales, then AlltoAll and skew AlltoAllv, and finally large-scale random-demand AlltoAllv on 256 GPUs. These experiments test whether VarioPath’s benefits persist across payload sizes, deployment scales, and traffic distributions. Random-demand AlltoAllv. We vary the per-rank payload on 32- and 64-GPU deployments with different local PCIe organizations. R5080-BF3 uses four and eight 8-GPU nodes,
• How closely does VarioPath’s channel-level peer–path co-design approach the full MILP optimum (§6.3)? • How scalable are VarioPath’s offline catalog and online planning (§6.4)? Testbeds. Table 2 lists the GPUs, PCIe generations and bandwidths, NICs, and topology of our four platforms. Our
1We evaluate DeepEP V2 from its official GitHub repository.
9
Bandwidth (GB/s)
VarioPath
NCCL
FAST
DeepEP
24 16 8 0
32
256 2048 32768 32 Payload/rank (KiB) (a) R5080-BF3 4 nodes (32 GPUs)
256 2048 32768 32 Payload/rank (KiB) (b) R5080-BF3 8 nodes (64 GPUs)
256 2048 16384 32 Payload/rank (KiB) (c) R5090-BF3 2 nodes (32 GPUs)
256 2048 16384 Payload/rank (KiB) (d) R5090-BF3 4 nodes (64 GPUs)
Figure 7: Random-demand AlltoAllv bandwidth on 32- and 64-GPU deployments. Bandwidth is per-rank payload divided by p50 completion time.
0
1
4 16 64 512 Message size (MiB)
10 0
1
4 16 64 512 Message size (MiB)
16 8 0
1
4 16 64 512 Message size (MiB)
(a) R5080-CX8, 1 node (8 GPUs)
(b) R5080-BF3, 1 node (8 GPUs)
(c) R5090-BF3, 1 node (16 GPUs)
(d) L20-CX7, 1 node (16 GPUs) Bandwidth (GB/s)
4 16 64 512 Message size (MiB)
20
24
Bandwidth (GB/s)
1
15
30
Bandwidth (GB/s)
0
30
MSCCL
Bandwidth (GB/s)
8
Bandwidth (GB/s)
16
Bandwidth (GB/s)
24
NCCL
Bandwidth (GB/s)
Bandwidth (GB/s)
VarioPath
24 16 8 0
1
4 16 64 512 Message size (MiB)
(e) R5080-CX8, 2 nodes (16 GPUs)
30 15 0
1
4 16 64 512 Message size (MiB)
(f) R5080-BF3, 2 nodes (16 GPUs)
30 20 10 0
1
4 16 64 512 Message size (MiB)
(g) R5090-BF3, 2 nodes (32 GPUs)
24 16 8 0
1
4 16 64 512 Message size (MiB)
(h) L20-CX7, 2 nodes (32 GPUs)
Figure 8: AlltoAll bandwidth on one- and two-node deployments. Bandwidth is per-rank payload divided by completion time. AlltoAll. VarioPath’s advantage persists under uniform traffic distributions. Across the 80 configurations in Figure 8, it achieves geometric-mean speedups of 1.82× over NCCL and 1.99× over MSCCL. It outperforms NCCL in every configuration and MSCCL in all but the 1-MiB, single-node R5080BF3 case. NCCL fixes transport paths at communicator setup, and MSCCL inherits those paths. Neither baseline can adapt an active pair’s physical path to the current stage. VarioPath instead adapts paths by stage, distributing uniform traffic across usable PCIe and NIC paths and avoiding concentration on shared links. The result confirms that this benefit is not limited to skewed traffic.
whereas R5090-BF3 uses two and four 16-GPU nodes. Figure 7 shows that VarioPath achieves the highest bandwidth on all 42 matched test points. Its geometric-mean speedup over DeepEP, the closest baseline, ranges from 1.55× to 2.00×. Over FAST, it ranges from 3.49× to 9.41×, and over NCCL from 3.73× to 10.61×. By jointly selecting each channel’s peer matching and physical paths, VarioPath avoids concentrating concurrent flows on shared PCIe links and maps chunks onto concurrently usable PCIe and NIC paths. To examine scaling, we fix the per-rank payload while increasing the number of GPUs. At 32 MiB on R5080-BF3, scaling from 32 to 64 GPUs reduces VarioPath bandwidth by 8.6% and FAST bandwidth by 5.7%, while VarioPath remains 1.72× faster. At 16 MiB per rank on R5090-BF3, the corresponding reductions are 3.6% and 12.6%. Thus, VarioPath retains its bandwidth advantage at 64 GPUs on both platforms by re-selecting channels across rounds as individual transfers complete, instead of leaving the additional flows pinned to a small set of shared links.
Skew AlltoAllv. We increase the Zipf skewness factor from 0.4 to 0.8 on all four two-node deployments. Larger factors concentrate more traffic in fewer rank pairs, making the demand harder to distribute across concurrent paths. Figure 9 shows that VarioPath remains fastest in all 12 deployment– skew configurations, with a geometric-mean speedup of 2.16× over FAST, ranging from 1.99× to 2.39× across plat10
Bandwidth (GB/s)
VarioPath
NCCL
FAST
(a) R5080-CX8, 2 nodes (16 GPUs)
(b) R5080-BF3, 2 nodes (16 GPUs)
(c) R5090-BF3, 2 nodes (32 GPUs)
(d) L20-CX7, 2 nodes (32 GPUs)
0.4 0.6 0.8 Skewness factor s
0.4 0.6 0.8 Skewness factor s
0.4 0.6 0.8 Skewness factor s
0.4 0.6 0.8 Skewness factor s
30 20 10 0
Figure 9: Skew AlltoAllv bandwidth on four two-node deployments. Each bar averages five Zipfian workload seeds. NCCL
AlgoBW (GB/s)
VarioPath
DeepEP
the highest algorithmic bandwidth at all eight values of M and reaches 30.91 GB/s at M = 128, compared with 28.11 GB/s for DeepEP and 18.85 GB/s for NCCL. This result shows that demand-aware channel scheduling continues to expose usable PCIe and NIC capacity when the deployment grows to hundreds of GPUs.
30 20 10 0
1
2
4
8
M
16
32
64
128
6.2
Figure 10: Large-scale random-demand AlltoAllv on 256 R5080-CX8 GPUs. M is the number of input tokens per rank; hidden dimension is 7168 and top-k is 8. VarioPath
23.7% 19.1%
1000 0
1200
Latency (s)
Latency (ms)
2000
(b) Wan2.1 generation
27.2%
3000
EP16
EP24
EP32
We replace NCCL-based AlltoAllv communication with VarioPath in uncached Qwen3–30B–A3B prefill [41, 44] and Wan2.1 video generation [37]. Qwen runs on two to four R5080-BF3 nodes with EP16, EP24, and EP32; each trial prefills 8,192 input tokens and generates one token, and each point is the client median of 15 requests. Wan2.1 runs on 8 and 16 L20 GPUs and generates a five-second BF16 1280×720 video with sequence length 75,600, 40 attention heads, and head dimension 128. In Figure 11, H and C denote the head- and context-parallel factors within the stated sequence-parallel degree. For Qwen3–30B–A3B, replacing the exchange with VarioPath reduces prefill latency by 19.1–27.2% across EP16, EP24, and EP32. The reduction increases with the expertparallel degree because expert dispatch and combine lie on the prefill critical path, and larger degrees expose more crossGPU and cross-node traffic to demand-aware channel scheduling. For Wan2.1, the speedup ranges from 1.011× to 1.066× across the four sequence-parallel layouts, with the largest gains at H8C1 and H8C2. At a fixed sequence-parallel degree, comparing H8C1 with H4C2 and H8C2 with H4C4 shows how the head/context decomposition reshapes rank-pair demand. Thus, the end-to-end benefit depends on workload and layout, not the sequence-parallel degree alone.
NCCL
(a) Qwen3 prefill
6.1%
1.2%
800
1.1%
5.5%
SP16 H4C4
SP16 H8C2
400 0
SP8 H8C1
SP8 H4C2
End-to-End Inference Performance
Figure 11: End-to-end latency for (a) Qwen3 prefill and (b) Wan2.1 generation. Labels report latency reduction from NCCL to VarioPath. forms. From s = 0.4 to 0.8, its bandwidth drops 9.4–12.3%, versus NCCL’s 15.2–27.0%. FAST declines less (2.0–6.0%), but starts from a substantially lower bandwidth and remains slower throughout the sweep. FAST’s node-local rebalancing is less effective on PCIe systems because its additional intranode forwarding competes with GPU-to-NIC transfers for shared PCIe bandwidth. Balancing NIC traffic can therefore leave the PCIe bottleneck unresolved or intensify it. By rescoring families and rebinding peers, paths, and chunks after each activation, VarioPath can redirect outstanding traffic to newly available path capacity. Large-scale experiment. We further evaluate VarioPath on 32 R5080-CX8 nodes with 256 GPUs using a random-demand AlltoAllv workload generated by top-k MoE routing with k = 8 and hidden dimension 7168. Figure 10 sweeps the perrank input-token count M from 1 to 128. VarioPath achieves
6.3
Design Validation
We first use an ablation study to quantify the contribution of joint peer–path selection, and then compare VarioPath with a full MILP optimum to measure the optimality gap of its online scheduler. 11
MILP solver
BW (GB/s)
(a) R5080-BF3, 8 GPUs
(b) R5080-BF3, 16 GPUs 24
30
100
20
16
99
10
8
98
0
8
16
32
64 128
Table 4: Catalog tree scale and online planning overhead. Catalog tree size is the estimated compact serialized payload. Online is planning time per emitted activation; O/C is total online-planning time divided by communication time.
Mean ratio
0
8
16
32
64 128
Ratio to MILP (%)
VarioPath
Deployment R5080-BF3–8 GPUs R5080-BF3–16 GPUs R5080-CX8–256 GPUs R5090-BF3–16 GPUs R5090-BF3–32 GPUs L20-CX7–16 GPUs L20-CX7–32 GPUs
97
Input demand per rank (MiB)
Figure 12: Scheduler quality against a full MILP optimum. Bars show mean per-rank bandwidth, and the line shows the mean VarioPath/MILP ratio over five demand matrices per size.
Fixed nearest path
Dynamic path
2.84× 2.03×
2.00× 1.00×
3 41 1616 115 308 92 352
Catalog tree (KiB)
Offline Online (s) (µs/act.)
6 1.7 0.800 438 132.0 4.950 2140 172682.6 5673.603 248 123.0 15.520 389 806.7 99.250 270 121.3 11.720 462 905.0 86.940
O/C (%)
3.2 0.0715 20.6 0.7554 78.12 4.1241 8.3 0.3249 21.1 1.1051 6.8 0.1476 20.8 0.7504
50 instances, VarioPath attains 99.18% of the MILP-optimal bandwidth on average, corresponding to an average optimality gap of 0.82%. After averaging within each topology–inputsize configuration, the ratio ranges from 98.34% to 99.41%, with a minimum of 91.92% among individual instances.
Table 3: Peer–path co-design ablation on two-node R5080BF3. Values are completion time normalized to demandaware peers with dynamic paths. Fixed peer rotation Demand-aware peers
# # Shapes Families
6.4 Catalog Tree Scale and Planning Overhead Catalog tree size. Table 4 reports offline catalog construction across deployments from 8 to 256 GPUs. The retained grouplevel trees contain 3–1,616 shapes and 6–2,140 families, occupy between 1.7 KiB and 168.6 MiB, and take between 0.8 s and 94.6 min to construct. Construction applies compatibility filtering, structural deduplication, and rate-dominance pruning without expanding concrete rank bindings, so one family represents many concrete channels. This one-time cost is amortized across demands on the same topology. Equalsize R5090-BF3 and L20-CX7 deployments retain different numbers of shapes and families, showing that catalog scale depends on topology as well as GPU count. Planning cost. We average online decomposition over 100 1-GiB-per-source demands. Planning takes 3.2–78.12 µs per activation; total planning time accounts for 0.0715–4.1241% of communication time (O/C in Table 4). Even at 256 GPUs, planning takes 78.12 µs per activation, while total planning time accounts for 4.1241% of communication time. Encoding topology feasibility offline limits online work to family search, endpoint binding, and transfer-size assignment, enabling perinvocation adaptation at substantially lower cost than reported MILP-based synthesis times [23, 32].
Peer/path ablation. This ablation tests the offline channel hypothesis that high-throughput execution requires jointly retaining peer matchings and path choices that avoid modeled shared-resource contention. To isolate this effect, we keep all other system components unchanged and vary only the eligible channel set. We replay the same five skew AlltoAllv demand matrices with s = 0.6 on two-node R5080-BF3. Fixed peers restrict each stage to a cyclic peer rotation, while fixed paths use each pair’s preassigned nearest path; the other dimension remains adaptive. Table 3 shows that fixing either dimension approximately doubles completion time, while fixing both increases it to 2.84× that of the unrestricted design. Thus, peer–path co-design is necessary to retain the unrestricted design’s performance on this workload. With only dynamic paths, a cyclic peer rotation can still select flows that compete for the same shared links; with only demand-aware peers, preassigned paths cannot redirect those flows to idle capacity. The similar penalties of the two one-sided variants show that neither decision substitutes for the other. Scheduler quality. To evaluate scheduling quality, we compare VarioPath with a full MILP optimum on one- and twonode R5080-BF3 deployments. We use per-rank input demands of 8, 16, 32, 64, and 128 MiB and evaluate five independently generated dense Zipf demand matrices for each topology and input size, giving 50 complete skew AlltoAllv calls. Following the modeling approach of TE-CCL [23], the MILP jointly optimizes concrete peer matchings, physical paths, transfer sizes, and their execution order over the entire call. It searches the complete modeled path space, and every instance reaches a solver-certified optimum. Figure 12 reports the algorithmic bandwidth of both methods. Across all
7
Conclusion
In this paper, we present VarioPath, an AlltoAllv framework for PCIe GPU systems that separates topology-dependent channel analysis from demand-dependent scheduling. Its offline analyzer compresses feasible channels into reusable families, while its lightweight online scheduler decomposes each invocation’s demand into channel activations without repeating topology search. Across four PCIe platforms and deploy12
ments of up to 256 GPUs, VarioPath outperforms existing systems and reduces Qwen3 prefill and Wan2.1 generation latency by up to 27.2% and 6.1%.
In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pages 1787–1802. USENIX Association, 2026. [10] Jiarui Fang and Shangchun Zhao. USP: A unified sequence parallelism approach for long context generative AI. arXiv preprint arXiv:2405.07719, 2024.
References [1] Jehoshua Bruck, Ching-Tien Ho, Shlomo Kipnis, Eli Upfal, and Derrick Weathersby. Efficient algorithms for all-to-all communications in multiport message-passing systems. IEEE Transactions on Parallel and Distributed Systems, 8(11):1143–1156, 1997.
[11] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
[2] Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. Synthesizing optimal collective algorithms. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 62–75, 2021.
[12] Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Efficient sparse training with mixture-of-experts. In Proceedings of Machine Learning and Systems, volume 5, pages 288–304, 2023. [13] Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. RDMA over ethernet for distributed training at Meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 57–70, 2024.
[3] Jiamin Cao, Shangfeng Shi, Jiaqi Gao, Weisen Liu, Yifan Yang, Yichi Xu, Zhilong Zheng, Yu Guan, Kun Qian, Ying Liu, et al. Syccl: Exploiting symmetry for efficient collective communication scheduling. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 645–662, 2025.
[14] Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang, Xun Chen, Yingtong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Zhao, Jiarui Fang, et al. LoongTrain: Efficient training of long-sequence LLMs with head-context parallelism. arXiv preprint arXiv:2406.18485, 2024.
[4] Minsik Cho, Ulrich Finkler, David Kung, and Hillery Hunter. BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy. In Proceedings of Machine Learning and Systems, volume 1, pages 241–251, 2019.
[15] Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. FasterMoE: Modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 120–134, 2022.
[5] Charles Clos. A study of non-blocking switching networks. Bell System Technical Journal, 32(2):406–424, 1953. [6] Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, and Yifan Xiong. MSCCLang: Microsoft collective communication language. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 502–514, 2023.
[16] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, HoYuen Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale. In Proceedings of Machine Learning and Systems, volume 5, pages 269–287, 2023.
[7] Ante Ćustić, Vladyslav Sokol, Abraham P. Punnen, and Binay Bhattacharya. The bilinear assignment problem: Complexity and polynomially solvable special cases. Mathematical Programming, 166(1–2):185–205, 2017.
[17] Heehoon Kim, Junyeol Ryu, and Jaejin Lee. TCCL: Discovering better communication paths for PCIe GPU clusters. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 999– 1015, 2024.
[8] DeepSeek. DeepEP: Expert parallel communication library. https://github.com/deepseek-ai/DeepEP, 2025. [9] Jiangsu Du, Hongbin Zhang, Taosheng Wei, Zhenyi Zheng, Jiazhi Jiang, Kaiyi Wu, Zhiguang Chen, and Yutong Lu. Efficient LLM serving on commodity GPU clusters with data-reduced cross-instance orchestration.
[18] JQ Lau, Ma Xiong, and Syona Sarma. Cloudflare’s 12th generation servers—145% more performant and 63% more efficient. Cloudflare Blog, 2024. 13
[19] Yiran Lei, Dongjoo Lee, Liangyu Zhao, Daniar Kurniawan, Chanmyeong Kim, Heetaek Jeong, Changsu Kim, Hyeonseong Choi, Liangcheng Yu, Arvind Krishnamurthy, et al. FLASH: Fast all-to-all communication in GPU clusters. arXiv preprint arXiv:2505.09764, 2025.
[29] NVIDIA. NCCL user guide: Point-to-point communication. https://docs.nvidia.com/deeplearning/ nccl/user-guide/docs/usage/p2p.html, 2026. [30] Pitch Patarasuk and Xin Yuan. Bandwidth optimal allreduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing, 69(2):117–124, 2009.
[20] Yiran Lei, Dongjoo Lee, Liangyu Zhao, Daniar Kurniawan, Chanmyeong Kim, Heetaek Jeong, Changsu Kim, Hyeonseong Choi, Liangcheng Yu, Arvind Krishnamurthy, Justine Sherry, and Eriko Nurvitadhi. FAST: An efficient scheduler for all-to-all GPU communication. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pages 2515–2531. USENIX Association, 2026.
[31] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 18332–18346. PMLR, 2022.
[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.
[32] Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. TACCL: Guiding collective algorithm synthesis using communication sketches. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 593–612, 2023.
[22] Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R Tallent, and Kevin J Barker. Evaluating modern GPU interconnect: PCIe, NVLink, NV-SLI, NVSwitch, and GPUDirect. IEEE Transactions on Parallel and Distributed Systems, 31(1):94–110, 2019.
[33] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparselygated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
[23] Xuting Liu, Behnaz Arzani, Siva Kesava Reddy Kakarla, Liangyu Zhao, Vincent Liu, Miguel Castro, Srikanth Kandula, and Luke Marshall. Rethinking machine learning collective communication as a multi-commodity flow problem. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 16–37, 2024.
[34] Brian Slechta, Nick Comly, Ashraf Eassa, Joe DeLaere, and Shivam Raj. NVIDIA NVLink and NVIDIA NVSwitch supercharge large language model inference. NVIDIA Technical Blog, August 2024.
[24] Meituan LongCat Team. LongCat-Flash technical report. arXiv preprint arXiv:2509.01322, 2025.
[35] Vladyslav Sokol, Ante Ćustić, Abraham P. Punnen, and Binay Bhattacharya. Bilinear assignment problem: Large neighborhoods and experimental analysis of algorithms. INFORMS Journal on Computing, 32(3):730– 746, 2020.
[25] Microsoft. Microsoft collective communication library (MSCCL). https://github.com/microsoft/msccl. Accessed September 10, 2026.
[36] Rajeev Thakur, Rolf Rabenseifner, and William Gropp. Optimization of collective communication operations in mpich. The International Journal of High Performance Computing Applications, 19(1):49–66, 2005.
[26] Naeris Netterville, Ke Fan, Sidharth Kumar, and Thomas Gilray. A visual guide to mpi all-to-all. In 2022 IEEE 29th International Conference on High Performance Computing, Data and Analytics Workshop (HiPCW), pages 20–27. IEEE, 2022.
[37] Team Wan, Ang Wang, Baole Ai, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
NVIDIA fabric manager user guide. [27] NVIDIA. https://docs.nvidia.com/hgx-platforms/ fabric-manager-user-guide/index.html, 2025. Accessed September 11, 2026.
[38] Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee, Nikhil Devanur, Jorgen Thelin, and Ion Stoica. Blink: Fast and generic collectives for distributed ML. In Proceedings of Machine Learning and Systems, volume 2, pages 172–186, 2020.
[28] NVIDIA. NVIDIA NVLink and NVLink Switch. https://www.nvidia.com/en-us/data-center/ nvlink/, 2025. 14
[39] Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. MLaaS in the wild: Workload analysis and scheduling in Large-Scale heterogeneous GPU clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 945–960. USENIX Association, 2022. [40] Changbo Wu, Zhuolong Yu, Gongming Zhao, and Hongli Xu. Enabling reconfiguration-communication overlap for collective communication in optical networks. arXiv preprint arXiv:2510.19322, 2025. [41] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [42] Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, et al. Comet: Fine-grained computation-communication overlapping for mixtureof-experts. arXiv preprint arXiv:2502.19811, 2025. [43] Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, et al. Insights into DeepSeek-V3: Scaling challenges and reflections on hardware for AI architectures. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, 2025. [44] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2024. [45] Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. MegaScale-Infer: Serving mixture-of-experts at scale with disaggregated expert parallelism. arXiv preprint arXiv:2504.02263, 2025.
15