Multi-Plane HyperX: A Low-Latency and Cost-Effective Network for Large-Scale AI and HPC Systems Ziyu Wang
Fei Lei
Dezun Dong∗
College of Computer Science and Technology, National University of Defense Technology Changsha, Hunan, China [email protected]
College of Computer Science and Technology, National University of Defense Technology Changsha, Hunan, China [email protected]
College of Computer Science and Technology, National University of Defense Technology Changsha, Hunan, China [email protected] NIC port
Multi-plane architectures have become increasingly prevalent in the Fat-Tree networks of AI data centers. By leveraging multiple ports on a single network interface card (NIC) or multiple NICs within a scale-up domain, each port or NIC is allocated to an independent network plane, thereby provisioning the overall system with multiple network planes. However, no prior literature has explored the application of multi-plane technologies to direct networks such A as HyperX. This paper investigates the multi-plane HyperX network and demonstrates that, compared to state-of-the-art network topologies like multi-plane Fat-Tree, Dragonfly, and Dragonfly+, the B C multi-plane HyperX architecture achieves a significantly smaller network diameter and superior cost-effectiveness.
Keywords multi-plane, HyperX, direct topology A1
1
B1
INTRODUCTION
∗ Corresponding author
…
A2 Recently, various multi-plane Fat-Tree networks haveB2been proposed in both academia and industry. For instance, Weiyang et al. introduced the Rail-only [7] network, InterA3 where each Network B3 face Card (NIC) within a high-bandwidth domain belongs to an independent Fat-Tree network plane. Alibaba proposed the HPN 7.0 [5] network, which significantly reduces the risk of hashBkcollisions Ak in Equal-Cost Multi-Path (ECMP) routing by leveraging dual-plane network characteristics. DeepSeek proposed an ideal multi-plane network [8] architecture where a NIC is equipped with multiple physical ports, each belonging to an independent Fat-Tree plane. Furthermore, the multi-plane network proposed by DeepSeek requires NICs to uniformly spray traffic across all physical ports; this necessitates that the NICs possess switching functionalities and provide native support for packet-spraying technologies and out-of-order packet reception. The benefits of multi-plane Fat-Tree networks are substantial, as they eliminate the core layer of traditional three-tier Fat-Tree networks, thereby significantly reducing both network diameter and costs. However, current research on multi-plane technology remains predominantly limited to indirect topologies like Fat-Tree. Direct topologies, such as Dragonfly [4], Flattened Butterfly [3], and HyperX [1], are widely deployed in leadership-class large-scale HPC systems and supercomputers. In this paper, we pioneer the application of multi-plane technology to the HyperX direct topology.
…
arXiv:2604.23519v1 [cs.NI] 26 Apr 2026
Abstract
Switch
4 Ports of a Single NIC
Figure 1: 4-plane 1D HyperX (MPHX(4,4,4)) network. Each NIC is equipped with four ports, with each port belonging to an independent 1D HyperX network plane.
HyperX is a superset of the Flattened Butterfly network and comprises multiple dimensions, where switches within each dimension are interconnected in a full-mesh configuration. We demonstrate that the multi-plane HyperX network achieves smaller network diameter and superior cost-effectiveness compared to several stateof-the-art topologies, including multi-plane Fat-Tree, Dragonfly, and Dragonfly+.
2
MOTIVATION
Driven by considerations such as fault tolerance, Network Interface Cards (NICs) in many contemporary large-scale AI and HPC systems are equipped with multiple ports, establishing the foundation for constructing multi-plane networks. A multi-port NIC evenly distributes its total outbound bandwidth across all its physical ports. Concurrently, switches typically support various modes of port breakout; for example, the 51.2 Tbps switch deployed in Alibaba’s HPN 7.0 supports breakout configurations of 128 × 400 Gbps and 256 × 200 Gbps. Consequently, it is a natural approach to simultaneously apply port breakouts to both the NICs and the switches, ensuring their bandwidths match. Performing port breakouts on a switch significantly increases its radix, which enables the reduction of the network diameter when constructing a network of the same scale, thereby achieving lower network latency and superior cost-effectiveness. Guided by this concept, we can construct the Multi-Plane HyperX (MPHX) network, which will be detailed in the following section.
3
DESIGN
The HyperX network is a superset of the Flattened Butterfly network, featuring multiple dimensions where switches within each dimension are fully interconnected. The primary distinction between the HyperX and Flattened Butterfly topologies is that HyperX allows for a more flexible distribution of the number of switches
Anonymous.
Table 1: Symbols used in this paper Symbols
Explain
𝑁 𝐵 𝑛 𝐵 ×𝑘 𝐷 𝑝 𝑑 𝑁𝑠 𝑁𝑜
The number of NICs of the network The total outbound bandwidth of a NIC The number of ports of each NIC The bandwidth of each switch The number of dimensions of the HyperX network The number of connected NIC ports of each switch The diameter of the network The number of switches in the network The number of optical modules in the network
across each dimension. For further information regarding the HyperX network, please refer to [1]. Next, we detail the construction of the multi-plane HyperX network. Table 1 summarizes the notation used in this paper. We define the total outbound bandwidth of a NIC as 𝐵 and the total bandwidth of a switch as 𝐵 × 𝑘, indicating that it can be configured with a maximum of 𝑘 ports, each with a bandwidth of 𝐵. The number of NIC ports (which corresponds to the number of independent network planes) is denoted by 𝑛, yielding a bandwidth of 𝐵𝑛 for each NIC port. Consequently, a switch can also be configured in a 𝑛𝐵′ × 𝑛 ′𝑘 port breakout format (in default, we assume 𝑛 ′ = 𝑛). Each switch connects to 𝑝 NIC ports. Assuming the HyperX network has 𝐷 dimensions, and letting 𝐷 1, 𝐷 2, . . . , 𝐷 𝐷 represent the number of switches in each respective dimension, a multi-plane HyperX network can be denoted as MPHX(𝑛, 𝑝, 𝐷 1, 𝐷 1, . . . , 𝐷 𝐷 ). Equation 1 defines the total number of Network Interface Cards (NICs) that a HyperX network can accommodate: 𝑁 =𝑝 ×
𝐷 Ö
𝐷𝑖
(1)
𝑖=1
In a balanced, maximum-scale multi-plane HyperX network, the 𝑛𝑘 configuration should satisfy 𝑝 = 𝐷 1 = 𝐷 2 = · · · = 𝐷 𝐷 = 𝐷+1 . Consequently, Equation 2 presents the total number of NICs that can be connected within this maximum-scale architecture: 𝐷+1 𝑛𝑘 𝑁𝑚𝑎𝑥 = (2) 𝐷 +1 In practice, the value of 𝑛 is bounded. In this paper, we assume the maximum value of 𝑛 to be 8, meaning that a single NIC can be broken out into at most 8 ports, resulting in a maximum of 8 network planes.
4
COST
In this section, we compare the cost-effectiveness of the three-tier Fat-Tree, multi-plane Fat-Tree, and MPHX networks. We assume the use of a switch with a total switching bandwidth of 102.4 Tbps, which can be configured as 64 × 1.6 Tbps, 128 × 800 Gbps, 256 × 400 Gbps, and 512 × 200 Gbps. Furthermore, we assume this switch is utilized to construct a medium-to-large-scale system comprising 65K NICs. The cost analysis for each topology is detailed in Table 2.
As observed, constructing a three-tier Fat-Tree network with non-breakout switches incurs exorbitant costs. In contrast, as the number of network planes increases, the MPHX topology progressively demonstrates superior cost-effectiveness. Compared to the multi-plane Fat-Tree network, the average cost per NIC is reduced by 28.0%. Furthermore, when factoring in the use of copper cables for connections between NICs and access switches, the costeffectiveness of the MPHX network is further amplified.
5 DISCUSSION 5.1 Other multi-plane topologies Theoretically, multi-plane technology can be applied to any topology. Let us first consider the multi-plane Dragonfly [4] network. By employing multi-plane technology, switches can be broken out into a larger number of finer-grained ports, thereby reducing the network diameter. In the Dragonfly topology, each time the switch radix doubles, the number of global ports a router connects to also doubles, while the number of NICs per group quadruples, and the total number of network groups is reduced to a quarter. Consequently, as switch ports are further broken out, it becomes highly probable that the global ports of a single router will suffice to connect to all groups within the network. At this stage, the multi-plane Dragonfly network flattens into a multi-plane 2D HyperX network. Take the Frontier supercomputer as an example: its current switch radix is 64, each switch features 16 global ports, each group accommodates 512 NICs, and the network comprises 80 groups in total. If the switch ports are broken out into 128 finer-grained ports, each group will house 2,048 NICs, leaving the network with only 20 groups. Concurrently, each switch will possess 32 global ports, signifying that every switch can directly connect to all the remaining groups, thereby effectively forming a 2D HyperX network. For the Dragonfly+ [6] network, the conclusion is analogous to that of the Dragonfly topology. When increasing the switch radix, a single switch becomes highly likely to directly connect to all groups, thereby forming an architecture similar to a two-tier Fat-Tree combined with HyperX. As the switch radix is increased further, the network will likely be reduced to a single group and flatten into a multi-plane Fat-Tree network. Regarding the recently proposed Zettafly-3 and Zettafly-4 networks [2], the findings parallel those of the Dragonfly and Dragonfly+ architectures. Increasing the switch radix eliminates the necessity for global switches, ultimately causing them to flatten into multi-plane HyperX and multi-plane Fat-Tree networks as well.
5.2
Challenges in Implementing Multi-Plane Technology
Despite its numerous advantages, deploying multi-plane technology is not trivial and imposes significant challenges on Network Interface Card (NIC) design. First, the NIC must be capable of uniformly distributing communication traffic across all planes, which essentially requires it to possess certain switching functionalities and the capability for out-of-order packet reception. Furthermore, for non-Fat-Tree multi-plane architectures, such as the multi-plane HyperX network proposed in this paper, it is additionally required that switches support adaptive routing. This is because the number
Multi-Plane HyperX: A Low-Latency and Cost-Effective Network for Large-Scale AI and HPC Systems
Table 2: Cost-effectiveness comparison of various topologies when constructing a system with approximately 65K NICs. We assume an outbound bandwidth of 1.6 Tbps per NIC and that the entire network exclusively utilizes optical cables, without the use of any copper cables. The bare-metal price of a single 102.4 Tbps switch is assumed to be $40,000, and the unit prices for 200 Gbps, 400 Gbps, 800 Gbps, and 1.6 Tbps optical transceivers are $100, $200, $450, and $1,200, respectively. Although the specific pricing information may deviate slightly from actual market conditions, the overall pricing trends and the reflected results should remain consistent. Notably, to maintain a reasonable bisection bandwidth, each switch in the second dimension of the MPHX(4, 86, 86, 9) network retains the same number of links as in the first dimension, i.e., 85. Consequently, within this second dimension, switches are interconnected by multiple links rather than a single link. Topologies
𝑑
Switch configuration
𝑁
𝑁𝑠
𝑁𝑜
Cost per NIC [$]
3-layer Fat-Tree 8-Plane 2-layer Fat-Tree Dragonfly Dragonfly+ 1-Plane 3D HyperX (MPHX(1,16,16,16,16)) 2-Plane 2D HyperX (MPHX(2,41,41,41)) 4-Plane 2D HyperX (MPHX(4,86,86,9)) 8-Plane 1D HyperX (MPHX(8,256,256))
4 2 3 3 3 2 2 1
64x1.6T 512x200G 64x1.6T 64x1.6T 64x1.6T 128x800G 256x400G 512x200G
65,536 65,536 65,536 65,536 65,536 68,921 66,564 65,536
5,120 3,072 4,096 4,096 4,096 3,362 3,096 2,048
393,126 (1.6T) 2,097,152 (200G) 323,584 (1.6T) 327,680 (1.6T) 315,392 (1.6T) 544,644 (800G) 1,058,832 (400G) 1,570,816 (200G)
10,323 5,075 8,425 8,500 8,275 5,507 5,041 3,647
of links between adjacent switches within a single plane is limited; consequently, the bandwidth of minimal paths is relatively low during cross-switch communication, necessitating the use of non-minimal paths to achieve effective load balancing.
6
FUTURE WORK
This paper provides only a preliminary overview of the multi-plane HyperX network. We plan to update this manuscript shortly to refine the technical details, propose a routing design for the multiplane HyperX architecture, and present a comprehensive performance evaluation comparing it against topologies such as Dragonfly, Dragonfly+, multi-plane Fat-Tree, and Zettafly under synthetic traffic, as well as HPC and AI application workloads. We anticipate demonstrating the low-latency advantages of MPHX stemming from its reduced network diameter, and further highlighting its cost-effectiveness.
References [1] Jung Ho Ahn, Nathan Binkert, Al Davis, Moray McLaren, and Robert S. Schreiber. 2009. HyperX: topology, routing, and packaging of efficient large-scale networks. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (Portland, Oregon) (SC ’09). Association for Computing Machinery, New York, NY, USA, Article 41, 11 pages. doi:10.1145/1654059.1654101 [2] Dezun Dong, Ziyu Wang, and Fei Lei. 2025. Zettafly: A Network Topology with Flexible Non-blocking Regions for Large-scale AI and HPC Systems. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 835–848. doi:10.1145/ 3695053.3731098 [3] John Kim, William J. Dally, and Dennis Abts. 2007. Flattened Butterfly: A CostEfficient Topology for High-Radix Networks. In Proceedings of the 34th Annual International Symposium on Computer Architecture (ISCA ’07). Association for Computing Machinery, 126–137. [4] John Kim, Wiliam J. Dally, Steve Scott, and Dennis Abts. 2008. TechnologyDriven, Highly-Scalable Dragonfly Topology. In Proceedings of the 35th Annual International Symposium on Computer Architecture (ISCA ’08). IEEE Computer Society, USA, 77–88. doi:10.1109/ISCA.2008.19 [5] Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. 2024. Alibaba hpn: A data center network for large language model training. In Proceedings of the ACM SIGCOMM 2024 Conference. 691–706. [6] Alexander Shpiner, Zachy Haramaty, Saar Eliad, Vladimir Zdornov, Barak Gafni, and Eitan Zahavi. 2017. Dragonfly+: Low Cost Topology for Scaling Datacenters. In 2017 IEEE 3rd International Workshop on High-Performance Interconnection
Networks in the Exascale and Big-Data Era (HiPINEB). 1–8. doi:10.1109/HiPINEB. 2017.11 [7] Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. 2024. Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters. In 2024 IEEE Symposium on High-Performance Interconnects (HOTI). 1–10. doi:10.1109/HOTI63208.2024.00013 [8] Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, et al. 2025. Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 1731–1745.