arXiv CSopen access

EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

arXiv:2605.18683v1 [cs.DC] 18 May 2026

EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet Yitao Yuan1 , Jianglong Nie1 , Tianyu Bai1 , Ruizhe Zhou1 , Siyuan Cao1 , Xujie Fan1 , Yuchen Xu1 , Junkai Chen1 , Chenqi Zhao1 , Nengyuan Zhang1 , Shaoke Fang1 , Jiangyuan Chen2 , Yuanfeng Chen3 , Jiaqi Sun1 , Zhan Wang4 , Xiaohua Xu2 , Yuchao Zhang5 , Yang Liu5 , Xiangrui Yang3 , Jing Lin6 , Xiaohe Hu6 , Yang Li7 , Chao Jiang7 , Limin Xiao7 , Weifeng Zhang7 , Junjie Wang8 , Wei Cheng8 , Yazhu Lan9 , Jianbo Dong9 , Binzhang Fu9 , Wenfei Wu1 ∗ 1 PKU, 2 USTC, 3 NUDT, 4 ICT, CAS, 5 BUPT, 6 Infrawaves, 7 Lenovo Research, 8 Centec, 9 Alibaba Cloud ETH+ Consortium

ABSTRACT

mission time, and lowering overhead [25–27]. This efficacy has been proven by products such as NVIDIA SHARP [20, 21] and various prototypes [5, 40, 50, 64]. The open Ethernet ecosystem, one of the most mature and widely deployed networks [2, 17, 18, 51, 60, 61], is evolving into networking AI clusters and integrating INC capabilities. Enabling INC within the open Ethernet ecosystem faces a core challenge: single participants struggle to implement cross-layer INC systems independently. Building INC requires the joint design of communication libraries, stacks, NICs, switches, and controllers, which are developed by different vendors and operators. The system’s integrity relies heavily on the interoperability of these components. This fragmentation poses two major risks for single participants, particularly switch vendors. First, isolated efforts within each scope fail to yield system-level benefits without corresponding host support, resulting in wasted investment. Second, the complexity of INC necessitates excessive upfront investment and prolonged coordination, potentially causing features to be shelved due to their impact on existing chip functions. As a protocol standardization working group, we propose EPIC (Ethernet Polymorphic In-network Collectives), an INC protocol leveraging “unified abstraction, polymorphic realization” as a viable path forward on Ethernet. (i) It establishes a unified abstraction defining INC behaviors, participant functional scopes, and rigorous interoperability interfaces, while retaining implementation flexibility within each scope (§3). (ii) EPIC’s data plane aligns components with ecosystem roles via strict semantics, and we also propose three-mode polymorphic realizations (§4). (iii) EPIC’s control plane operates via software-defined networking (SDN), where a central controller manages global resources, policies, and rule dissemination (§6). EPIC addresses three critical challenges: the prohibitive development overhead for vendors to support various polymorphic realizations, the

In-Network Collective (INC) acceleration holds immense potential for optimizing AI training and inference; however, its cross-layer nature has historically hindered investment and adoption within the open Ethernet ecosystem. To bridge this gap, we propose EPIC (Ethernet Polymorphic In-network Collective), an INC protocol specification and reference system built on the principle of “Unified Abstraction, Polymorphic Realization.” EPIC introduces an abstraction compatible with standard Ethernet that aligns functional boundaries with participant roles, while offering polymorphic realizations tailored to varying hardware capabilities. We address three fundamental challenges: first, we employ a modular design that enables an evolutionary path from simple to complex implementations, allowing vendors to iterate their hardware incrementally; second, we apply formal verification methodologies to prove the correctness of all proposed polymorphic modes; and third, we develop a unified resource management model versatile enough for diverse INC scenarios. Extensive validation—spanning model checking, packet/flow simulations, VM emulation, Tofino Testbed, and FPGA/RTL verification—confirms EPIC ’s correctness, performance gain, and feasibility. This paper does not raise any ethical issues.

INTRODUCTION

As processing power for large-scale ML model training surges, communication overhead has become a critical performance impactor [6, 10, 24, 41, 41, 64, 73]. In-Network Collective (INC) communication, a new network communication paradigm, addresses this by offloading collective operations like AllReduce to switches, thereby compressing traffic, reducing trans∗ Wenfei

Wu is the corresponding author; contact him by [email protected]. Authors are also members of ETH+ Consortium. 1

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. Table 1: Implementations and Key Results Implementation Target Property Model Checking Correctness Tofino Testbed Performance Acceleration Emulation Interoperability, Evolvability Flow-level Gain in Resource Simulation Management Packet-level Loss Tolerance Simulation FPGA Latency, Resource Cost RTL Chip Feasibility

numerous dispatch/combine traffic patterns also challenge the forward table volume if implemented using IP multicast and EPIC’s lookup table. We have another design for MoE AlltoAllv, which will be released separately. EPIC benefits communication and end-to-end model training/inference. INC enhances communication by reducing latency (fewer hops), minimizing traffic congestion, and lowering end-host overhead to better saturate bandwidth. For end-to-end tasks, INC provides direct acceleration in communication-bound clusters. In optimized environments, it still yields substantial gains for expensive hardware by freeing GPU resources for computation, raising the performance floor, and simplifying system tuning.

Key Results All 3 modes are correct. Up to 1.59x for collectives and 1.35x for training. All 3 modes work with RoCE. Mode-III reuses 61% of Mode-II. Reduce GPT-3 JCT by up 45.8%, multi-job 99%-tile JCT by 30.9% Mode-III outperforms Mode-II. 50 ns processing latency O(1) MB SRAM 3.2 Tbps; 4.89 mm2 @ 28 nm

complexity of verifying protocol correctness across these modes, and the lack of a unified resource model to support various management policies (§2.3). EPIC protocol specification was distributed to multiple organizations for comprehensive validation. The protocol working group members make various implementations and validate EPIC’s properties as in Table 1. In this paper, our contributions are as follows. • A unified abstraction of six-primitive collectives to enable INC systems in Ethernet ecosystem, • A polymorphic data plane enabling the switch manufacture for vendors with diverse capabilities, • Modularized data plane enabling switch vendors to evolve INC switch between polymorphic modes, • Analysis of properties of polymorphic modes, including model-checking verified correctness, transmission efficiency, space complexity, logic complexity, and fault tolerance, • Resource model derived from the abstraction that enables various resource management policies.

2 2.1

2.2

Requirements of INC on Ethernet

EPIC should satisfy requirements from ecosystem organization, hardware-offloading trend, and AI collective patterns. (1) EPIC must strictly adhere to Ethernet functional scopes and interoperability. It partitions components according to the industry division of labor, where functionality results from collaboration between CCL developers, NIC/switch vendors, and operators via standard protocols like IP and RoCE. Standardizing these interaction primitives is essential to avoid the risks of closed, vertically integrated solutions. By defining clear interoperability specifications, EPIC enables cross-layer optimization without sacrificing vendor neutrality. This alignment is a prerequisite for breaking ecosystem deadlock and enabling deployable INC. (2) EPIC must strictly align with RoCE, one of the de facto standards for AI interconnects [17]. As bandwidth scales toward 1.6 Tbps, hardware transport offload via RoCE is essential to bypass the CPU “performance wall” and maintain zero-copy, low-latency transfers. Unlike previous softwarebased INC (e.g., SwitchML 1 or ATP), EPIC leverages mature hardware-offloaded engines to sustain line-rate speeds. This approach treats INC as an enhancement of existing infrastructure rather than a replacement. By inheriting RoCE’s congestion control and memory semantics, EPIC enables vendors to scale bandwidth without the cost of proprietary protocols, ensuring a production-ready and evolvable ecosystem. (3) EPIC must support a comprehensive suite of collective primitives beyond AllReduce. To keep pace with evolving AI workloads—such as FSDP (ZeRO) [62, 82], which relies on ReduceScatter and AllGather—a modern INC framework must offer versatile support for diverse patterns, including Broadcast and Reduce. Exhaustive primitive support is critical for deployment viability. Broad functional coverage eliminates the need for hybrid deployments, where missing hardware-offloaded primitives force a mix of INC

BACKGROUND EPIC’s Application Scope and Benefits

EPIC focuses on scale-out networks. AI clusters use dualinterconnects: scale-out (NIC/switch-based) and scale-up (e.g., NVLink [55], UALink [69]). EPIC targets the Ethernet/RoCE ecosystem [30, 42] for scale-out expansion, embedding INC semantics in RoCE header fields, which is the de facto standard on Ethernet [2, 18]. This avoids the fragmentation of scale-up protocols and the prohibitive hardware barrier of redesigning accelerator IO dies for new protocol layers, ensuring multi-vendor interoperability. EPIC focuses on regular collectives. We distinguish primitives by data patterns: (1) Regular: AllReduce, Reduce, and Broadcast. These maintain strict alignment between element indices, packet sequences, and memory addresses. (2) Irregular: MoE token dispatch/combine (AlltoAllv)[49]. These involve asymmetric traffic and lack positional correspondence. Designing for irregular patterns requires complex index-to-memory mapping and congestion management. Its

1 SwitchML is developed on DPDK or RDMA UC.

EPIC and conventional CCLs. By avoiding such fragmentation and its associated orchestration overhead, EPIC serves as a general-purpose accelerator. This ensures a consistent, highperformance communication fabric that is both future-proof and production-ready for AI training and inference.

2.3

send_state recv_state send_state recv_state

ep1 leaf

Challenges

flow1

endpoint

flow2

ep5 ep2

ep6 root ep12

ep4 ep3 leaf

edge

ep11 ep10

ep8

ep7 leaf

ep9 leaf

(a) An INC tree (IncTree)

We overcome the challenges of ensuring evolvability and correctness in a polymorphic data plane while maintaining control plane generality. (1) Diverse performance-complexity tradeoffs (§5) and high development overhead hinder universal polymorphic INC support. In the Ethernet ecosystem, switch capabilities vary: fixed-function ASICs prioritize throughput over flexibility, while programmable targets (e.g., Tofino[31]) are limited by SRAM and ALU resources. This disparity forces vendors to choose between costly full-spectrum implementation or restrictive single-mode solutions. To resolve this, EPIC modularizes switch INC functionality, enabling a low-cost transition from simple to complex modes. By decoupling functionalities into reusable components, vendors achieve scaling through module composition. This adaptive approach allows implementations to align with specific hardware constraints—from basic packet replication to full RoCE endpoints—facilitating an incremental and sustainable deployment path for the industry (§4). (2) Polymorphic realizations complicate the verification of computational integrity. Ensuring protocol termination and server-equivalent results is challenging, as each mode introduces unique intermediate states and cross-layer interactions. Analyzing these diverse designs individually incurs prohibitive analytical overhead. To address this, we performed a formal analysis of the EPIC protocol suite. Using automated model checking, we systematically explored the state space of the polymorphic data plane and network conditions (e.g., packet loss). This rigorous approach automatically verifies protocol termination and computational accuracy across all modes, providing provable guarantees of system integrity while eliminating manual, error-prone proofs (§5). (3) Distributed switch resources lack a unified model to support diverse, scenario-specific management policies. The increasing heterogeneity of AI workloads—ranging from multi-tenant environments to varied cluster scales— demands unique allocation strategies. Manually coordinating customized INC management for every infrastructure combination is infeasible due to extreme complexity. To bridge this gap, we propose a unified INC resource model based on the SDN paradigm. By virtualizing switch resources and decoupling the control from the data plane, this model provides generic interfaces for centralized orchestration. This architecture supports diverse policies—including

switch 3 IP6, QP6, eth2 IP12, QP12, eth3

switch 4

IP5, QP5, eth1 switch 1 IP2, QP2, eth2 IP4, QP4, eth3

IP11, QP11, eth1 switch 2 IP8, QP8, eth2 IP10, QP10, eth3

IP1, QP1, eth0 IP3, QP3, eth0 host 1 host 2

IP7, QP7, eth0 host 3

IP9, QP9, eth0 host 4

(b) IncTree on Ethernet cluster

Figure 1: EPIC’s abstraction isolation and contention-based sharing—transforming fragmented resources into a flexible, programmable fabric. This approach minimizes administrative overhead while maximizing resource utilization across dynamic AI scenarios (§6).

EPIC OVERVIEW

3.1

Abstraction

We define EPIC’s abstraction and map it to Ethernet clusters. EPIC’s realization can support 6 collective primitives. INC Tree. Collective communication is performed on a group of parallel application processes, called ranks. EPIC employs an INC Tree (IncTree) to describe the logical topology of a communication group (Figure 1). Each rank maps to a leaf node, while non-leaf nodes act as intermediate aggregation or replication points within the network fabric. The tree is assigned a root. An edge connects two nodes on the tree, and is undirected. Data unit (packets) on an edge with the same direction form a flow. We define an endpoint to describe flow routing and transmission. An edge has two endpoints, each on one of the edge’s two nodes. Note that a node can connect to multiple edges, but those edges’ endpoints on that node are distinct. On an edge, a flow’s transmission states consist of sending states on its source endpoint and receiving states on its sink endpoint (Figure 1a). In a node, a flow’s routing from one edge to another can be described as from one endpoint to another. Collectives on IncTree. EPIC performs various collective primitives on the aggregation tree. It supports six primitives: AllReduce, Reduce, Broadcast, Barrier, ReduceScatter, and AllGather. The latter three can be derived from the former three (Figure 2a to Figure 2c): Barrier is equivalent to an AllReduce with an empty data payload, ReduceScatter is the sequential execution of multiple Reduces, and AllGather is similarly the execution of multiple Broadcasts (Appen3

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

S3 S1

result

data

Legends

S3 S2

h1 h2 h3 (a) AllReduce ep6

S1 h4

data IO via collectives

S3 S2

message IO via verbs

Control Plane

Host GPU GPU

APPLICATION

CommLib

COL. COM. Lib.

NIC RoCE

RoCE NIC

TRANSPORT

packet IO

h1 h2 h3 h4 h1 h2 h3 h4 (b) Reduce (to rank 0) (c) Broadcast (from rank 0)

IncManager

Switch IncAgent IncEngine

IDX In Port Invocation Out Ports 1 ep2 | ep4 AllReduce, * {ep5} ep5 S1 2 ep5 AllReduce, * {ep2, ep4} ep2 ep4 3 ep4 | ep5 Reduce, rank0 {ep2} 4 ep2 Broadcast, rank0 {ep4, ep5} ep1 ep3 (d) Switch S1’s endpoints (e) Switch S1’s routing table (partial)

NETWORK LAYER Network

Figure 3: EPIC Architecture

3.3

Workflow

3.3.1 System and Group Lifecycle Bootup. The lifecycle begins with component initialization: CommLib daemons launch on hosts, and IncEngine is enabled on switches. All instances report local states to the IncManager, which constructs the global topology and manages resources (§6.1). When an application calls InitGroup(), the IncManager computes and maps a logical IncTree onto the physical fabric. It then disseminates configurations: host CommLibs receive flow information, and switch IncAgents install local routing states into their respective IncEngines. Connection and Routing. Following the setup, CommLib establishes RoCE connections to neighbor switch nodes. Switches support polymorphic connection handling, ranging from full stack functionality to state-less translation (§4.2, §4.3, §4.4). During initialization, IncManager pre-computes rules for all 2𝑁 +1 traffic patterns (AllReduce, Reduce, Broadcast) and configures forwarding for both RoCE data and ACK packets. Teardown. Upon completion, DestroyGroup() triggers the IncManager to instruct IncAgents to delete local states and forwarding rules. The IncManager then releases group-specific resource reservations for future allocation. 3.3.2 Runtime Collective Invocation Control Signaling. EPIC supports multiple runtime primitives using in-band signaling, avoiding the custom INC headers like SHARP or the CPU-intensive header prepending like NetReduce. Before data transmission, the CommLib sends a standalone RDMA Send with Immediate message2 to notify IncTree nodes of the collective type, root, and data size. This signaling is pipelined with subsequent data to hide latency. If lost, the switch refuses data processing until retransmission, ensuring protocol safety through validated PSN range (derived from the data size). Data Processing. The execution follows a five-step pipeline: (1) Chunking & Flow Control: CommLib partitions tensors into messages and applies application-level flow control to prevent switch buffer overflow. (2) Standard Transport: The commodity NIC encapsulates messages into RoCE packets for delivery to the IncEngine. (3) Switch Processing: The In-

Figure 2: Data Flow and Routing in IncTree dix §A). Thus, in the following text, we focus on the design of AllReduce, Reduce, and Broadcast. IncTree on Ethernet Cluster. In Ethernet-based Clos topologies, hosts reside at the edge while switches form the core. An IncTree maps leaf nodes to hosts and intermediate nodes to switches. Each endpoint is defined as an <IP, QP> tuple associated with a specific host NIC or switch port (Figure 1b). Consequently, an IncTree edge may span multiple physical links defined by the path between its two endpoints. The IncTree abstraction also applies to accelerator-centric topologies [33], with non-leaf nodes on accelerators. During collectives, switches execute local actions via matchaction tables. These tables recognize and forward packets to ports that align with the IncTree’s data flow. For instance, in an AllReduce operation (Figure 2e), switch 𝑆2 aggregates packets from child endpoints (𝑒𝑝8, 𝑒𝑝10) toward the parent (𝑒𝑝11) and distributes results back down those same paths.

3.2

Host GPU GPU

Architecture

Architecture. EPIC decouples control and data planes, aligning with industry functional boundaries. The data plane comprises host-side CommLib, standard NIC RoCE, and switchbased IncEngine. The control plane features switch-resident IncAgents and a centralized IncManager for resource discovery, policy formulation, and rule dissemination (Figure 3). Interfaces. Components interact through standardized interfaces across two paths: (1) Control Path: The IncManager coordinates with CommLib and IncAgent via RPC. The IncAgent then configures the IncEngine using internal register or table updates; (2) Data Path: A layered collaboration ensures interoperability. The data path consists of three components. (1) CommLib: Provides collective primitives (e.g., AllReduce) to applications and exchanges data with the NIC via standard RDMA Verbs. (2) RoCE: Executes standard RDMA operations without hardware changes, interfacing with the IncEngine via RoCE packets over physical links. (3) IncEngine: Performs packet-level parsing, processing (aggregation/replication), and forwarding between NICs or peer engines.

2 Send With Immediate has a distinct RoCE OP Code from other verbs.

EPIC cEngine performs aggregation, replication, or forwarding. It identifies the collective setting via the preceding control message and uses pre-configured rules to route packets through the IncTree. (4) Reliable Reception: Receiver NICs reassemble packets and signal CommLib via Work Completions (WC). RoCE ACK/NAK packets are processed by the IncEngine on the reverse path to manage state release (§4). (5) Completion: CommLib assembles the results and returns control to the application.

3.4

Send Buffer

msg

send WR recv WR

msg

advance window recv WC RDMA QP

write buffer

Figure 4: Flow Control in CommLib Table 2: Symbols and meaning 𝑈 : Maximum Transmission Unit (MTU); unit is byte 𝑀 : message size; unit is MTU; a message is 𝑈 𝑀 bytes 𝑊 : window size; unit is message; a window is 𝑈 𝑀𝑊 bytes 𝑁 : array size in IncTree state 𝐻 : aggregation tree depth 𝐷 : node degree 𝐵 : link bandwidth 𝐿 : link latency

System Fault Tolerance.

transmission: ReceiveAck, SendAck, and Retransmission. There are modules for data (packet payload) operation: CheckDuplicate, AggregateData, RecycleBuffer, and ReplicateData. Module internal logic is elaborated in Algorithm 1 and 2 in Appendix B.

4.2

Mode-I: Connection Terminated

Mode-I provides a “heavyweight” realization of the EPIC abstraction for high-end switches capable of supporting a full RoCE stack. In this mode, the switch functions as a Parameter Server (PS), terminating host connections and maintaining complete transport states to ensure reliable transmission. The architecture utilizes a layered protocol stack where the switch handles deduplication, ACK processing, and message reassembly. AllReduce workflow. The host-side CommLib initiates communication with the control signal. To manage traffic, the system employs message-granularity flow control using paired Send-Recv verbs. This choice is critical as one-sided Write verbs that do not provide work completion (WC) signals that confirm “result” readiness. Flow control is achieved by setting the RoCE outstanding WR to be 𝑊 messages, where 𝑊 is configured as a system parameter window size (Figure 4). Intermediate switches identify EPIC packets via lookup tables, directing data to internal transport layers for hop-byhop reliable delivery, and to the internal application layer for data aggregation (performed by the AggregateData module). Once the IncTree root completes data reduction, the ReplicateData module broadcasts results back to child nodes via the transport layer again; the child nodes relay the results towards host receivers (Figure 5a). Reduce and Broadcast workflows. They serve as simplified variations of this process. In a Reduce operation, data flows unidirectionally from multiple senders to a single receiver through switch-based aggregation, with host CommLibs executing only Send or Recv verbs, respectively. Conversely, Broadcast transmits data from one sender to multiple receivers using the replication module. Both primitives lever-

POLYMORPHIC DATA PLANE

EPIC modulizes IncEngine; composing modules in different ways forms different modes, and reusing modules evolves modes from simple to complex.

4.1

msg

Window

Running distributed training with EPIC needs to handle runtime failures from switches, links, and/or ranks. EPIC can run as a network slice (separate CommLib and IncEngine instances from traditional CCL and switch forwarding), and set up NCCL as failover; the failover process can be enabled via services like MCCS [75]. EPIC itself handles failures by reinitializing groups [70]. System-level failover requires sandboxing ranks and runtime group member change; EPIC can apply Continuun [39].

Receive Buffer

AllReduce

Modulize IncEngine

We decouple the runtime communication group states from the processing logic, and decompose all functionalities in IncEngine into independent modules. At runtime, these modules retrieve the corresponding states using the header of the current packet, and execute specific logic. States and Context. The collection of all states involved in an INC group at runtime is referred to as the group’s Context. While an INC group logically possesses a global context across the entire cluster, physically, each switch is only required to install the Local Context relevant to its operation; where there is no ambiguity, we use “context” to refer to “switch local context”. On an IncTree switch node, the states include: (1) routing states, in the format of rules to look up packet headers and decide the output ports, (2) transmission states, where each endpoint involves sending states for its outgoing flow and receiving states for its incoming flow, and (3) computation states, which are buffers to temporally store intermediate states. At runtime, modules utilize the header of the current packet to retrieve its context. These three kinds of states are described and retrieved by the endpoint in the IncTree abstraction. Function Modules. We prepare the following modules to construct IncEngine. There are modules for state retrieval and routing based on the packet header: LookupTable, TranslateHeader, and Forward. There are modules for flow 5

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. LookupTable packets RoCE (De)packetization

LookupTable

msg

HandleAck

LookupTable

AggregateData

UP_DATA

ConnectionMgmt Retransmission

IncEngine

CheckDuplicate

AggregateData msg

DOWN_DATA

ReplicateData

ACK

DATA

CheckDuplicate

ReceiveAck

AggregateData

RecycleBuffer

ReplicateData

SendAck

CheckDuplicate AggregateData

ReplicateData ACK

TranslateHeader

Forward

(a) Mode-I

ReceiveAck

Retransmission

Forward

(b) Mode-II

Figure 5: Modules and data flow in three modes

SendAck

ReplicateData

TranslateHeader

timer

Forward

ACK

DATA

RecycleBuffer

TranslateHeader

packets

LookupTable

UP_DATA DOWN_DATA

Retransmission

timer Forward

Figure 6: A pitfall in ModeIII design

age the same pre-configured routing rules and polymorphic RoCE functions, ensuring high-performance communication without the bidirectional result distribution required by AllReduce. Interaction with RoCE Flow Control and Congestion Control. EPIC’s flow control means keeping inflight data volume from overwhelming the switch buffers. The three mechanisms — EPIC flow control, RoCE congestion control, and RoCE flow control (e.g., PFC or CBFC) — are layered in the order above, and they work independently. The actual runtime traffic rate, progress, and volume are the minimum of the three mechanisms. Such independence of EPIC flow control from RoCE congestion control and flow control holds for all three modes.

ther determines the next hops, and TranslateHeader clones the packet and modifies header fields (e.g., Dest IP, Dest QP). Finally, Forward sends the packet to corresponding switch port.5 RecycleBuffer: The payload/degree size is set to twice the window (2𝑀𝑊 MTUs). To support streaming with minimum switch SRAM space, payload is used circularly: upon completion of aggregation (degree[idx]==FAN-IN), the slot at (𝑝𝑠𝑛 + 𝑀𝑊 )%(2𝑀𝑊 ) is cleared for subsequent packets; this is a safe operation as that slot is out of all senders’ windows [50, 80]. (3) The IncTree root sends result packets downward. Switches identify downstream packets (Figure 7b, Figure 7f) and process them via ReplicateData, TranslateHeader, and Forward modules (Figure 5). 4.3 Mode-II: Connection Translated (4) The result packet finally reaches the RoCE NIC, and Mode-II offers a minimalist design for switch vendors by triggers an ACK (Figure 7c). This ACK is reflected by the substituting full transport termination with a “connection first-hop switch back to the RoCE NIC to acknowledge the translation” approach. Instead of maintaining complete RoCE upstream data packet (Figure 7d). RoCE ACKs are cumulative, states, the switch modifies and forwards packets while reand this mechanism ensures correctness even if ACKs are lying on end-hosts for reliability, ensuring transport transcoalesced. Negative ACKs (NAKs) generated due to packet parency. This mode requires all NIC endpoints to be initialloss are handled in the same way as ACKs: reflected to trigger ized with identical Packet Sequence Numbers (PSNs). 3 standard RoCE retransmission at the sender. In the case of AllReduce Workflow. (1) CommLib works in the same way packet transmission, each step of the workflow is idempotent as mode-I. The switch maintains a payload (array of MTU) (repeatable without affecting the result). buffer and a degree buffer (array of counters), both of size N. Reduce and Broadcast Workflows. (1) Reduce: Control (2) The switch identifies upward data packets by header and data packets are handled the same as AllReduce; the fields (Figure 7f), and processes them by modules. CheckDuplicate receiver’s RoCE NIC replies ACKs, which are broadcasted identifies retransmissions. For the first arrival of a packet, along the tree to the senders. NAKs are handled similarly. AggregateData module sums up its payload into payload (2) Broadcast: Control and data packets are sent to receivers buffer (at idx=pkt.psn%N)4 and increments the degree at along the IncTree, replicated at each node. Receivers reply idx; for retransmission, aggregation is skipped. If aggrewith ACKs towards the sender, which are aggregated by gation is incomplete (degree[idx]<FAN-IN), the packet is IncTree nodes. ACKs are cumulative; intermediate nodes dropped; otherwise, the result (payload[idx]) is copied back maintain an 𝑎𝑐𝑘𝑃𝑠𝑛 for each endpoint to record the progress to the packet, forming a result packet. ReplicateData furof cumulative ACK. A node also records a 𝑛𝑜𝑑𝑒𝐴𝑐𝑘𝑃𝑠𝑛 as the 5 The endpoint states also include RDMA memory_region (MR) address

3 PSN can be set through ibv_modify_qp().

4 If reproducibility is needed (adding numbers in a deterministic order) [7, 72],

and rkey to enable the write admission to the RDMA registered memory region. address and rkey are distributed (from host CommLib to switch IncEngine via IncManager) during group initialization.

IncEngine needs extra buffers to store data first, and add data up when the degree is full.

EPIC Legends upward data

downward data

acknowledge

ep6 ep5 ep2

Legends upward data

acknowledge

(b) Broadcast

Idx Header Fields Invocation SendWithImm ep1→ep2 ep3→ep4 ep6→ep5 ep1→ep2, ACK ep3→ep4, ACK ……

Class

States

FanIn Out Ports

CTRL AllReduce,* UP_DATA ctx1, ep2 AllReduce,* UP_DATA ctx1, ep4 AllReduce,* DOWN_DATA ctx1, ep5 AllReduce,* ACK ctx1, ep2 AllReduce,* ACK ctx1, ep4 …… …… ……

{ep5} {ep5} {ep2, ep4} {ep2} {ep4} ……

2 2 -

ep4

ep1 ep3 (a) Aggregation 1 (b) Aggregation 2 (c) Broadcast 1

(d) Broadcast 2

Idx Header Fields

Invocation

Class

States

FanIn

Out Ports

1 2 3 4 5 6 7

AllReduce,* AllReduce,* AllReduce,* AllReduce,* AllReduce,* AllReduce,*

CTRL DATA DATA DATA ACK ACK ACK

ctx1, ep2 ctx1, ep4 ctx1, ep5 ctx1, ep2 ctx1, ep4 ctx1, ep5

2 2 1 -

{ep5} {ep5} {ep2, ep4} -

SendWithImm ep1→ep2 ep3→ep4 ep6→ep5 ep1→ep2, ACK ep3→ep4, ACK ep6→ep5, ACK

(e) The highlighted switch’s lookup tables (AllReduce entries)

(e) The highlighted switch’s lookup tables (changes with Mode-II highlighted)

Figure 7: Lookup tables in Mode-II

Figure 8: Lookup Tables in Mode-III

minimum of all child 𝑎𝑐𝑘𝑃𝑠𝑛 values. An ACK is forwarded upstream only if it updates the 𝑛𝑜𝑑𝑒𝐴𝑐𝑘𝑃𝑠𝑛, effectively preventing ACK amplification. NAKs caused by packet loss are not aggregated but are forwarded to the sender to trigger retransmission [28, 43]. Group PSN Synchronization. A primary challenge in multicollective support is PSN divergence, where asymmetric primitives like Reduce or Broadcast cause sequence numbers to drift between ranks, breaking subsequent AllReduce alignment. A “work-around” method for this issue is to mandate single-primitive groups and use multiple subgroups for complex operations like ReduceScatter. This method increases the lookup table entries: an 𝑁 -member group needs to configure 2𝑁 +1 EPIC groups, 𝑁 for reduce, 𝑁 for broadcast, and 1 for AllReduce. Considering a switch only holds a few local groups and 𝑁 ’s scale, the table entry cost is acceptable. The switch memory is temporally multiplexed among collective invocations. We also suggest a “RoCE-Refined” approach: allowing a northbound NIC interface to manually synchronize sequence numbers after asymmetric operations. This architectural flexibility ensures that Mode-II functions correctly across diverse traffic patterns while minimizing switch hardware modification.

4.4

Endpoint ep

ACK packet

ep5

ep2

(a) Aggregation

downward data

ep6

ep4

ep1 ep3 1 2 3 4 5 6 …

Endpoint ep

ACK packet

outgoing endpoints

incoming endpoints data pkt ACK

Endpoint 1 (Recv States) arrived[N] epsn

Pipe payload buffer

Endpoint A (Send States) lastAcked

degree buffer

ACK

iff pkt.psn ∈ [psnStart, psnStart+N) psnStart ≡ min{o.lastAcked | o∈outgoing}+1

Figure 9: The Pipe Abstraction. When an incoming endpoint receives a data packet, it performs a readiness check: if the PSN falls within the valid range, the AggregateData module processes the payload and immediately generates an ACK to the upstream node; otherwise, the packet is dropped, and a NAK is sent to trigger retransmission or congestion control. If the packet is processed and the degree is full, the switch sends the packet further to downstream endpoints (via ReplicateData and TranslateHeader). When an outgoing endpoint receives an ACK, the endpoint maintains and updates a lastAcked state, which is the maximum ACK PSN it observed. This value is used to dynamically update the pipe’s psnStart (defined as the minimum of all outgoing endpoints’ lastAcked + 1), thereby advancing the sliding window and freeing buffer space for subsequent packets (Algorithm 2 in Appendix §B). AllReduce Workflow. (1) CommLib and RoCE NIC remain consistent with Mode-I/II. (2) Upward data packets are handled by an aggregation pipe, with multiple incoming endpoints and one outgoing endpoint. (3) Downward packets are handled similarly by a broadcast pipe, with one incoming endpoint and multiple outgoing endpoints. (4) RoCE NIC sends NAKs when observing packet loss, and switch handles NAK by retransmitting packets starting from 𝑙𝑎𝑠𝑡𝐴𝑐𝑘𝑒𝑑 + 1. Whether a switch actively sends NAK when it observes packet loss (e.g., out-of-order) is optional, both choices work correctly, but sending NAK is recommended for performance issue (Appendix H.4). Reduce and Broadcast work with one pipe on each switch, enabling one-direction data flow. Group PSN Synchronization. IncTree edges allow bidirec-

Mode-III: Connection Augmented

Mode-III serves as a middle ground between the full-stack complexity of Mode-I and the minimalist nature of ModeII, providing hop-by-hop retransmission (Link-Level Retry, LLR) without the overhead of complete message packetization (Figure 8). A Pipe Abstraction. This mode introduces a pipe abstraction to manage buffer readiness and transport reliability at each switch hop (Figure 9). Each pipe consists of a payload buffer and a degree buffer—both sized N—and maintains a psnStart value representing the valid PSN range [psnStart, psnStart+N). A pipe is associated with one or several incoming endpoints (whose count is the FAN-IN degree) and one or several outgoing endpoints. 7

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. tional transmission; unidirectional operations (Reduce/Broadcast) reception and transmission. desynchronize the sequence numbers of the reverse flow on The global protocol state is represented by the composithe same edge. To resolve this in Mode-III, when an ACK tion of these individual NFAs into a unified, complex sysfor a flow in one direction arrives at an endpoint, the system. To verify the protocol, the model checker exhaustively tem must simultaneously update the PSN progress of the simulates all execution branches from every state, accountreverse flow (i.e., ePsn) on that endpoint to match the group ing for network uncertainties such as packet loss and outprogress. We leave the discussion of other possible modes of-order delivery. For each transmitted packet, the checker and switch microarchitectures in the appendices §D and §E. explores every possible outcome—success, loss, or reorderCongestion Control for Rate Synchronization. In Modeing—creating separate execution paths for each scenario. III, rank windows progress independently. If ranks experiThe protocol is deemed correct only if every explored ence different congestion conditions, the congested ranks branch satisfies two invariant properties: computational acprogress slowly while the uncongested ones progress quickly. curacy (the final result matches server-side reduction) and It is a better choice to have the switch slow down the faster liveness (the protocol eventually reaches a terminal state). ranks to avoid excessive sending and dropping (due to beThis rigorous exploration of the state space ensures that the ing out of the pipe PSN range). On the first-hop switch, if polymorphic data plane remains robust under any possible a packet is dropped due to exceeding the PSN range, the network condition. switch replies to the sender with a congestion signal (e.g., The EPIC model checker provides a specification language CNP for DCQCN) to slow down that rank. to describe an INC system, including topology and node logic, and a compiler to compile a user program to a TLA+ program; 5 ANALYSIS OF MODES it runs TLA+ model checker to verify the system correctness. 5.1 Correctness The EPIC model checker overcomes two challenges. First, it enables describing network nondeterminisms including Analysis. The correctness of the EPIC protocol is established reliable, unreliable, and out-of-order packet delivery. Second, through two primary pillars: computational equivalence to it handles state explosion by input space reduction, nondesingle-node processing and guaranteed protocol termination. terminism constraints, and symmetric state elimination. In Mode-I, correctness is inherited from the underlying Fixing a Pitfall. Model checking revealed a critical pitfall RoCE transport layer, which ensures that IncEngine only when evolving from Mode-II to Mode-III regarding buffer processes ordered, unique, and valid packets. This reliabilmanagement. In Mode-II, buffer slots out of the current winity allows the engine to index data structures directly by dow are recycled when an aggregation event occurs inside sequence number for inherently accurate aggregation. the current window, as the “aggregate-then-forward” mechFor Mode-II, end-host retransmission ensures data comanism naturally synchronizes window advancement across pleteness, while the CheckDuplicate module prevents reall ranks (RecycleBuffer in §4.3). dundant computation. Addressing conflicts are mitigated by However, direct application of this logic to Mode-III is the “aggregate-then-forward” mechanism, which constrains incorrect (Figure 6). In Mode-III, window progression is indethe skew between the fastest and slowest ranks. This enpendently governed by ACKs, meaning that the completion sures the PSN range of in-flight packets never exceeds twice of an aggregation at a specific 𝑃𝑆𝑁 does not guarantee that the window size (2𝑊 ). By sizing switch buffers to 2𝑊 , we the 𝑃𝑆𝑁 + 𝑀𝑊 position is outside the active windows of all prevent packet collisions and incorrect buffer mapping. participants. Clearing the buffer at 𝑃𝑆𝑁 + 𝑀𝑊 prematurely Mode-III similarly utilizes hop-by-hop retransmissions risks erasing data from faster ranks, resulting in computaand redundancy detection. Here, each computation buffer tional corruption. To resolve this, we introduced the pipe maintains a strict permissible PSN range exactly matching its abstraction (pipe in §4.4), which mandates a unified writable capacity (𝑁 ). This prevents addressing conflicts where packrange to forcibly synchronize sender progress. ets with a PSN difference greater than 𝑁 might otherwise In addition, the EPIC model checker also finds design risks alias to the same memory location. in ATP, SwitchML, and NetReduce. And we fix them in EPIC Across all modes, a robust timeout-retransmission failsafe technical report. We put other insights in Appendix§H. guarantees eventual protocol termination under any network condition. 5.2 Other Properties Model Checking. The correctness of EPIC is verified via We compare other properties across three modes, and reach formal model checking, where each network node is modthe following conclusion. Formal analysis is placed in the eled as a Non-deterministic Finite Automaton (NFA). State Appendix §F, and Figure 10 illustrates their comparison. transitions are triggered by packet processing, and nonTransmission Efficiency. Switches operate under a storedeterminism arises from the concurrent nature of multi-port and-forward model but at different granularities. Mode-I 8

EPIC Transmission Efficiency Correctness Space Simplicity

Mode-I Mode-II Mode-III

ctx1 ctx1 ctx2 ctx2

Loss Tolerance traffic

requires full message reception prior to processing, whereas Mode-II/III utilize packet-level (MTU) pipelining. Consequently, Mode-II/III reduces end-to-end latency by approximately (2𝐻 − 1) (𝑀 − 1)𝑈 /𝐵, offering advantages unless the message size 𝑆 is exceptionally large or the propagation delay 𝐿 dominates. Logic Complexity. Considering the module organization, Mode-II is the simplest to implement, followed by Mode-III and Mode-I. Space Complexity. Mode-II carries the highest space overhead (4(𝐻 − 1)𝐵𝐿) because its end-to-end flow control must buffer for the entire path bandwidth-delay product (BDP). In contrast, Mode-I and Mode-III utilize hop-by-hop mechanisms, requiring only single-hop BDP buffers (non-reproducible 4𝐵𝐿 to reproducible (𝐷 + 1)2𝐵𝐿). Mode-III achieves the best balance, offering lower space requirements than Mode-II while supporting non-reproducible options to further minimize overhead. Loss Tolerance. Throughput in Mode-I and Mode-III scales with the slowest independent hop due to local retransmissions. Mode-II, however, suffers from multiplicative degradation because packet loss at any rank forces global synchronization and retransmission across the entire IncTree. While all modes perform similarly in lossless environments, Mode-I and Mode-III are significantly more robust to network noise.

6.1

tables

Header States

Figure 10: Comparison of Modes (outer is better).

IncAgent

control plane data plane

Group 1

…

context ctx1

pipe buffer

ctx2

pipe2

pipe1

SRAM

Logic Simplicity

IncEngine Modules

Group 2

Figure 11: Resource model maximizing efficiency across the cluster. Indirection Layer for Resource Allocation. In the EPIC architecture, switches manage group-specific rules in matchaction tables and associated contexts in high-speed SRAM. Direct context reallocation via table modification is computationally expensive due to the high volume of rules. Analysis reveals that most SRAM is consumed by transient states—such as payload and degree buffers (𝑂 (BDP))—which remain idle between collective invocations, while persistent endpoint states (𝑂 (𝐷)) remain minimal. To optimize utilization, EPIC introduces an indirection layer that decouples these large transient states from the fixed context. By utilizing pointers within the context to reference dynamic SRAM regions, the IncManager can calculate and assign memory offsets in real-time. This pointer-based approach facilitates agile, fine-grained resource reallocation and significantly enhances SRAM utilization without the overhead of reconfiguring global forwarding tables.

6.2

Supporting Various Policies

Cluster-wide INC resource management essentially needs to place an IncTree on the topology and decide the resource allocation on each of its switch nodes. EPIC resource model can support various existing and new policies, where we list three classes below. Edge-Disjoint Tree. Legacy INC solutions often utilize an Edge-Disjoint Tree (EDT) approach [12, 63] necessitated by fixed-function hardware. These systems reuse static ingress buffers that cannot be virtualized, meaning a physical port can typically serve only one communication group. This creates a strict topological constraint: multiple aggregation trees must not share edges to avoid resource contention. The EPIC resource model maintains compatibility with EDT by decoupling IncTree placement from resource allocation. EPIC first determines an EDT for a group and then independently allocates SRAM resources to bind the tree. We provide an algorithm to calculate candidate EDTs within Clos networks. The process removes edges occupied by active EDTs, then iteratively scans switches from lower to upper tiers. It identifies the lowest level containing switches reachable by all group members without traversing higher

INC RESOURCE MANAGEMENT Resource Model

Architecture. The EPIC control plane enables intelligent resource management through a centralized IncManager and distributed IncAgents. During initialization, the IncManager constructs a global view by aggregating reported device capabilities, port statuses, and available on-chip SRAM from each IncAgent, while simultaneously performing global topology discovery via link-state reporting. At runtime, the IncManager serves as the decision-making hub, instantiating communication groups by issuing instructions to IncAgents. These instructions configure the switchlocal context, including lookup tables and computation states such as aggregation buffer offsets. By decoupling policy formulation from hardware execution, the framework supports dynamic resource reallocation and real-time state updates based on network conditions. This architecture ensures finegrained, centralized control over distributed INC resources, 9

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. levels. The algorithm returns these switches as potential IncTree roots or indicates if no such candidate exists. Spatial Multiplexing. Legacy solutions like SwitchML [64] and NetReduce [50] achieve isolation by statically partitioning switch SRAM, typically for single-tier scenarios. EPIC extends this to cluster-wide, multi-layer topologies, enabling non-interfering concurrent groups and robust multi-tenancy. To manage this, EPIC jointly optimizes IncTree placement and resource allocation by evaluating both logical connectivity and “path width”—a metric combining available bandwidth and SRAM capacity. Notably, the system may prioritize longer paths if their larger aggregation buffers support superior congestion control and higher overall throughput. For Clos networks, we employ a greedy iterative algorithm to identify the Pareto frontier of candidate IncTrees. By scanning from lower to upper levels, the algorithm identifies potential roots that balance tree depth against resource capacity. This provides administrators with a selection of optimal solutions, ranging from low-latency trees for synchronization to high-bandwidth trees for data-intensive training. Temporal Multiplexing. To maximize efficiency, EPIC supports temporal resource sharing like ATP [40, 45, 46]. It models switch capacity as a combination of unallocated space and oversubscribed blocks weighted by duty cycle. This allows the IncManager to perform joint placement and allocation even in congested environments, effectively multiplexing physical SRAM across multiple communication groups. Mutual exclusivity can be maintained primarily through scheduled coordination, which staggers group invocations. This is particularly effective for 3D parallel training, where Tensor Parallel (TP) and Data Parallel (DP) groups communicate in interleaved phases, enabling them to share physical hardware without contention. In decentralized or multi-tenant scenarios where scheduling is infeasible, EPIC employs a contention-and-fallback mechanism. Groups attempt to “lock” resources via a firstcome-first-served (FCFS) policy at switch-resident recorders. If a group fails to secure the complete IncTree, it performs an all-or-nothing release and falls back to host-based communication (e.g., NCCL), preventing partial occupancy and ensuring system-wide progress.

7 7.1

connected VMs, flow simulation in OMNET++, packet simulation in NS3. Environments. Our testbed has four servers and one switch in a star topology. Each server has two NVIDIA GeForce RTX 3090 GPUs, two Xeon(R) Silver 4316 CPUs with 40 physical cores, 192 GB memory, 1 TB SSD, two 100 GbE CX-5 NICs (each NIC bonded with a GPU); the switch has 100 Gbps*32 ports. Due to limitations on programmability, we cannot implement Mode-I/III in Tofino. We integrate EPIC with Pytorch, where the CommLib needs to pipeline the memory copy between GPU and RDMA registered buffer and EPIC’s packet I/O. (2) Emulation runs on VMs connected by OVS [59], where host VMs run SoftRoCE, and Mode-I and Mode-II/III switch VMs have IncEngine built atop libibverbs and libpcap. (3) Simulations are based on NS3 or OMNET++, which run on a PC. Topology. We use the following notion to describe the topology “Tree-𝑥-𝑦”, where 𝑥 means tree depth and 𝑦 means tree branches at non-leaf nodes. For example, a star topology with four servers is denoted as Tree-2-4 (servers counted as one tier, and the switch as another). Workloads. We run two kinds of workloads: (1) Collective communication of the 6 primitives to study communication acceleration, (2) single training job to study EPIC’s acceleration, and (3) multi-tenant training to study resource management. In multi-tenant training, we use job traces from production clusters [6]. Baselines. In testbed experiments, we compare EPIC with NCCL (Ring and Tree); for AllReduce specifically, we compare EPIC with ATP and SwitchML6 . In packet/flow-level simulation, we compare EPIC with ring-based algorithm. In emulation, we compare EPIC with MPI (uses CPU memory instead of GPU; more suitable for VMs). In the experiment description, we use EPIC-I/II/III to denote EPIC’s three modes. Metrics. We measure the following metrics to compare solutions: (1) collective’s algorithm throughput, which is the application data size of the ranks divided by the overall collective completion time, and (2) job completion time (JCT).

7.2

Feasibility

Correctness. We run model checking with the Tree-3-2 topology, and verify Mode-II/III’s AllReduce, Reduce, and Broadcast. The network environment is set reliable, lossy, and out-of-order. In all environments, EPIC is correct: the computation result is equivalent to a single server’s result, and the protocol eventually terminates. Interoperability. Table 3 shows AllReduce throughput in emulation. All three modes in EPIC interoperate with hostside SoftRoCE correctly. And they all give correct results and

EVALUATION Experiment Settings

Implementation. The EPIC standard specification is provided to the organizations in the working group, and the group contributes several different implementations, each of which can validate the specific attributes of EPIC as described in Table 1. The following implementations are anonymously open-sourced in the Appendix. Model checking is built in TLA+ [38], Tofino testbed prototype, OVS emulation [59]

6We compare with SwitchML DPDK version. SwitchML provides an RDMA

UC version, where the RDMA is configured without reliability guarantee, and the host still needs to spare CPU to handle transmission failures. 10

EPIC Throughput (Gbps)

Table 3: [Emulation, Tree-3-2] AllReduce Algorithm Throughput (Gbps) Msg. Size (B) 4K 16K 64K 256K 1M 4M 16M 64M 256M 1G EPIC-I 2.47 10.24 35.9 79.0 182 248 382 418 426 427 EPIC-II 14.7 48.1 67.3 85.1 89.0 93.7 86.6 89.3 94.1 97.9 EPIC-III 15.7 64.1 88.7 86.8 72.5 75.5 74.9 75.5 75.1 75.6 1.71 3.72 14.9 21.1 73.9 181 316 249 434 502 MPI EPIC-II ATP 7-to-1

SwitchML ATP 8-to-1

NCCL-Ring-1KB NCCL-Tree-1KB

Tput (Gbps)

102

16K 64K 256K 1M Message size (B)

0 1M

4M 16M 64M 256M 1G Message size (B)

Figure 12: [Testbed, Tree-2-8] AllReduce Algorithm Throughput finish normally. EPIC-III has more complex switch workflow, so its throughput is lower than EPIC-II. It is hard to make a fair performance comparison (and not the goal of emulation) among EPIC-I vs EPIC-II/III vs MPI, because their software stack differs (details in Appendix §J). Evolvability. The development of EPIC-II and III follows the modular design. EPIC-II and III consist of 7604 and 7400 lines of code, respectively, with 61% reuse. Switch vendors can iterate INC feature from EPIC-II to EPIC-III, with correctness guarantee while keeping improving performance. Resource Affordability. Across various hardware, EPIC requires only 1–16 MB of on-chip SRAM, with platforms such as Intel Tofino, Agilex 7 FPGA, and Xilinx VU13P supporting 8 MB, 16 MB, and 1 MB, respectively (§M, §I). For Mode-II, the most memory-intensive realization, a 100 Gbps network with 10 µs RTT necessitates only 250 KB per job (twice the path BDP). Simulations confirm that even a 1 MB allocation—supporting four concurrent groups—significantly boosts performance in 3D training and multi-tenant environments, thereby reducing JCT despite memory constraints (§L). So the resource cost of EPIC is affordable in production scenarios. Chip Area. Evaluated by a chip vendor, when synthesized using a 28 nm process technology, EPIC’s IncEngine instance, configured with 512 FP32 ALUs, 512 UINT ALUs and a 1 MB payload buffer, occupies an area of 4.89 mm2 . The total area overhead for the eight integrated engines (25.6 Tbps) amounts to 39.12 mm2 , demonstrating a scalable and efficient footprint for high-bandwidth INC.

7.3

10−1 10−2

Reduce RS Bcast AG 4KB Message

Reduce RS Bcast AG 1GB Message

vides a significant lower end-to-end latency due to reduced hop counts and a thinner software stack. For large messages, the performance gain stems from the substantial reduction in network traffic volume. When the link capacity is saturated for all solutions, INC solutions—EPIC, ATP, and SwitchML— outperforms NCCL. But EPIC spends less CPU resources to saturate links due to its RoCE-compatibility and hardware transmission offloading. SwitchML achieves high throughput using a DPDKbased end-host implementation; however, because DPDK lacks transport-layer reliability, SwitchML must handle packetization and retransmissions, leading to higher CPU overhead compared to EPIC (1 core vs. 6 cores). While ATP performs well with a dedicated Parameter Server (PS) on additional links, its small-message latency remains high, and throughput drops when the PS competes for bandwidth with workers (ATP 8-to-1). Tofino constrains EPIC’s packet payload size as 256B. Comparing NCCL with 256B payloads, EPIC achieves a 1.59× speedup. NCCL with 1KB payloads improves efficiency, which suggests that future EPIC support for larger payloads would yield greater gains. Finally, the NCCL Tree algorithm suffers from the overhead of multiple binomial rounds, proving less effective than the pipelined Ring approach. When performing Barrier, EPIC takes ~5 µs for one operation, while NCCL takes ~2 ms. EPIC outperforms NCCL by 400×. Figure 13 shows results across four primitives. EPIC provides a significant advantage for small messages, due to reduced hop counts and minimal software overhead. For large-scale data transfers, NCCL’s Reduce/Broadcast shows a similar throughput due to their ring-based algorithm; EPIC outperforms NCCL in Broadcast due to less traffic to switch, but equals NCCL in Reduce due to the computation overhead in switch. In terms of ReduceScatter and AllGather, NCCL performs concurrent rings so its throughput surges, EPIC keeps RS/AG similar to Reduce/Broadcast because its sequential execution, but EPIC causes less traffic to the network. Traffic Volume Reduction. We evaluate overlapping ReduceScatter (RS) and AllGather (AG) operations across two communication groups in NCCL and EPIC. The result is shown in Figure 14. In NCCL, aggregate throughput is capped by line rate because nodes must simultaneously transmit and

40 20

10−2

100

NCCL-Ring 100

Figure 13: [Testbed, Tree-2-8] Collective Algorithm Throughput

NCCL-Ring-256B NCCL-Tree-256B

100

EPIC-II

Performance Acceleration from INC

Collective Communication. Figure 12 shows AllReduce performance on testbed. All INC solutions consistently outperform the baseline NCCL. For small messages, INC pro11

Network Tput (Gbps)

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

150

NCCL-RS EPIC-RS

NCCL-AG EPIC-AG

NCCL-RS+AG EPIC-RS+AG

Table 4: [Testbed, Tree-2-8] Model Training Iteration Time (s), context_length=2K, batch_size=256.

128MB 256MB

Model EPIC-II SwitchML ATP NCCL GPT-2 Large 5.46 5.48 5.51 5.53 DP=8 Qwen2.5-0.5B 3.59 3.61 3.62 3.64 Llama-3.2-1B 7.18 7.22 7.24 7.28 GPT-2 Large 21.79 24.33 26.62 29.35 TP=8 Qwen2.5-0.5B 13.10 14.58 15.92 17.51 Llama-3.2-1B 21.63 23.88 25.93 28.35

100 50 0

16MB

32MB

64MB

Table 5: [Testbed, Tree-2-4] Inference Performance

Figure 14: [Testbed, Tree-2-8] Bandwidth Complements between ReduceScatter and AllGather

Performance (ms) EPIC-II NCCL

receive for both ring-based collectives, causing bandwidth contention. Conversely, EPIC breaks this limit by exploiting directional link independence. By orchestrating nodes to transmit for RS while receiving for AG (and vice versa), EPIC ensures non-overlapping directional flows. Consequently, EPIC achieves a theoretical throughput of 2× the link bandwidth, as evidenced by the additive throughput observed in our experiments. Training Applications. We train three models (GPT-2 Large, Qwen2.5, Llama-3.2) and report their iteration times in Table 4. EPIC accelerates DP and TP by 1.4% and 31.1% (Llama3.2). Meanwhile, packet-level simulation based on SimAI [74] and NS3 reports a 1.3%–34.7% speedups for Llama-7B to GPT3-175B (Appendix §K). This experiment assesses EPIC ’s acceleration across various parallel strategies rather than comparing the inherent efficiency of DP versus TP. While EPIC improves both, the speedup is more pronounced in TP due to its higher communication intensity, whereas DP yields more modest gains from its relatively lower volume. Considering cluster architecture, EPIC excels when TP is deployed on scaleouts, but it does not when TP and INC are deployed on “high-bandwidth” scaleups, because the TP AllReduce time is reduced due to high bandwidth, which also reduces INC’s improvement space (Table 6’s last column is improved by 0-0.79% when applying INC on scaleups). Note that this experiment does not aim to discover whether the experimented models favor DP or TP in training, but rather to show the acceleration to TP/DP. In large-scale training where model shards exceed single-GPU memory, TP is indispensable. For environments utilizing consumergrade GPUs lacking high-end interconnects (e.g., NVLink), TP traffic frequently becomes the primary bottleneck. In these common scenarios—including fine-tuning and posttraining—EPIC delivers significant performance dividends. Inference Applications. We infer GPT-2 small/large with TP=4 and sequence length=1024 (Table 5). EPIC reduces GPT2 large’s TTFT/TPOT by 29.1%/30.8%, respectively. For inference’s small-message patterns, EPIC’s reduced hops and latency translate directly into end-to-end performance gains.

7.4

GPT-2 Small TTFT TPOT 12.03 8.809 13.03 10.05

GPT-2 Large TTFT TPOT 26.83 25.09 37.85 36.27

Loss Tolerance

To study loss tolerance, simulations were conducted using an idealized model where congestion control is disabled, forcing the sender to saturate link bandwidth while prioritizing retransmissions for lost packets. Throughput vs Loss Rate. This setup allows for a rigorous comparison of EPIC-II and EPIC-III by tuning the packet loss rate on a single link. While both modes see throughput declines as loss rates rise, EPIC-II experiences a sharp collapse after a 10% loss rate. This occurs because a single lost packet in EPIC-II triggers redundant retransmissions across all ranks for that sequence number, whereas EPIC-III allows lossless ranks to receive ACKs and continue, preventing excessive retransmission. Throughput vs Lossy Links. Further experiments maintaining a constant 5% loss rate while increasing the number of lossy links reveal that EPIC-II suffers more severe degradation than EPIC-III. In EPIC-II, the throughput decline is the cumulative result of losses across all affected links, creating a stacked decrease effect. Conversely, in EPIC-III, the overall performance is determined only by the link with the highest loss rate, as the retransmission times on other links are effectively masked by the most bottlenecked one. Rate Synchronization for EPIC-III. We run AllReduce on Tree-2-8 with 200 Gbps link capacity. One link is congested by background traffic to 100 Gbps. We apply DCQCN on ranks. Full results are in Table 35 of Appendix §K. Without rate synchronization (switch replying CNP to faster ranks), Throughput (Gbps)

EPIC-II 80

40 20

EPIC-III

0 0.1 0.5 1 2 5 8 10 12 15 Loss Rate (%) @ 1 link

0 1 2 3 4 5 6 7 8 Lossy Links (5% loss)

Figure 15: [Packet Simulation, Tree-2-8] AllReduce Algorithm Throughput 12

EPIC Table 6: [Flow Simulation] JCT (three iterations) of GPT3-175B on 128-GPU Fat-tree Ring EDT Spatial Mux Temporal Mux

w/o scaleup 4 8 253 253 179 179 179 158 158 137

CDF

SRAM (Unit)

w/ scaleup 4 8 106 106 84.6 84.6 84.6 63.6 84.6 63.6

5,000

6,000

7,000

JCT (s)

Figure 16: [Flow Simulation] Tail 15% JCT of Alibaba Trace on 2048-GPU Fat-tree and hybrid invocations through specialized control signals and group PSN synchronization, and fixes their bugs by model checking. There are product switches with six collectives supported [5, 20, 21], which belong to EPIC Mode-I; EPIC proposes a modular design and supports evolvability from simple to complex. The Ultra Ethernet Consortium is finalizing the INC standard [70], which is a clean-slate, ModeI-like design with an INC header and UET transport. There are in-network computing solutions for irregular traffic patterns [22, 29] and generic computing [71, 77, 80], which we would consider in the future. INC Resource Management. INC management in a cluster should consider function placement [3, 66], routing [12, 14, 63, 65, 76, 83], job placement [81], and resource allocation [79]; EPIC provides a unified resource model to enable these policies. INC Switch Micro Architecture. EPIC focus on protocol design, and is compatible with existing switch micro architecture designs [7, 72]. Modular Network Design. There were modular design of software switch/router [9, 37, 58], network functions [16, 52], and network layers [13]. EPIC differs from them in the application scenario: it implements the same functionality with different algorithms and applies modularity to enable algorithm evolution. ClickINC [77] implements the same algorithms on different platforms. Click [37] uses modules to flexibly compose different functionalities. P4 [4] provides a platform-independent language to program different functionalities. Collective Communication Libraries. There are various collective communication libraries and their optimizations [1, 11, 23, 32, 35, 53, 54, 67]; EPIC’s CommLib works at the same layer and should be integrated within mainstream CCLs.

Performance Gain from Policy

Single 3D Parallel Training Job. Table 6 shows a single training job. We tune switch SRAM sizes based on BDPrelative units (§5). Groups select candidate aggregation trees greedily following our algorithms (§6.2). Ring-based NCCL serves as the baseline, yielding the longest JCT due to lack of INC acceleration. The EDT policy suffers from placement conflicts, leaving several communication groups without INC resources. In contrast, Spatial Mux grants more groups INC access when resources permit. Temporal Mux further maximizes benefits by sharing limited resources across groups, benefiting the entire workload (up to 1.85×). Integrating scaleup networks yields similar conclusions, though overall INC acceleration gains diminish as high-bandwidth scaleup links absorb significant traffic, reducing relative network transmission time (1.67×). Multi-tenant Training Jobs. Figure 16 shows the simulation results of multi-tenant training on a 2,048-GPU cluster with production trace [6], varying INC management policies. Compared to the baseline, all INC-enabled policies significantly reduce the average Job Completion Time (JCT). Among these strategies, Temporal Mux achieves the highest overall acceleration. JCT distribution analysis reveals that while Temporal Mux is less efficient than EDT for jobs in the 88th to 98th percentiles, it drastically reduces tail latency. At the 99th percentile, JCTs for Temporal Mux, EDT, and Ring are 4917 s, 5420 s, and 7115 s, respectively. This improvement stems from Temporal Mux allowing all jobs to contend for INC resources; previously blocked jobs gain acceleration, while formerly exclusive jobs face only minor interference.

0.9 0.85

the algorithm throughput is 43.3 Gbps (16 MB message); with rate synchronization, the throughput becomes 95.9 Gbps. Applying congestion control-based rate synchronization can avoid excessive sending and dropping, thereby improving the overall collective throughput.

7.5

Ring EDT Spatial Mux Temporal Mux

0.95

CONCLUSION

To address Ethernet fragmentation in INC, we propose EPIC protocol featuring unified abstraction and polymorphic realization. Its modular design allows seamless, incremental hardware evolvement while maintaining protocol integrity. Validated via model checking, EPIC ensures robust performance across diverse conditions. EPIC unifies resource management and enhances control-plane agility. EPIC signifi-

RELATED WORK

INC Primitive Design. Many existing programmable switchbased INC solutions focus on certain primitives (e.g., AllReduce [8, 19, 36, 40, 44–46, 50, 64, 78], Broadcast [28, 34, 43]), which belong to EPIC Mode-II; EPIC supports six primitives 13

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. cantly accelerates collective communication, and reduces single- and multi-tenant training JCT.

EPIC

REFERENCES

Dongarra, Jeffrey M. Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, Ralph H. Castain, David J. Daniel, Richard L. Graham, and Timothy S. Woodall. 2004. Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation. In Proceedings of the 11th European PVM/MPI Users’ Group Meeting. Budapest, Hungary. [16] Massimo Gallo and Rafael Laufer. 2018. ClickNF: a Modular Stack for Custom Network Functions. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 745–757. [17] Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. 2024. Rdma over ethernet for distributed training at meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference. 57–70. [18] Yixiao Gao, Qiang Li, Lingbo Tang, Yongqing Xi, Pengcheng Zhang, Wenwen Peng, Bo Li, Yaohui Wu, Shaozong Liu, Lei Yan, et al. 2021. When cloud storage meets RDMA. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). 519–533. [19] Nadeen Gebara, Manya Ghobadi, and Paolo Costa. 2021. In-network aggregation for shared machine learning clusters. Proceedings of Machine Learning and Systems 3 (2021), 829–844. [20] Richard L Graham, Devendar Bureddy, Pak Lui, Hal Rosenstock, Gilad Shainer, Gil Bloch, Dror Goldenerg, Mike Dubman, Sasha Kotchubievsky, Vladimir Koushnir, et al. 2016. Scalable hierarchical aggregation protocol (SHArP): A hardware architecture for efficient data reduction. In 2016 First International Workshop on Communication Optimizations in HPC (COMHPC). IEEE, 1–10. [21] Richard L Graham, Lion Levi, Devendar Burredy, Gil Bloch, Gilad Shainer, David Cho, George Elias, Daniel Klein, Joshua Ladd, Ophir Maor, et al. 2020. Scalable hierarchical aggregation and reduction protocol (sharp) tm streaming-aggregation hardware design and evaluation. In International Conference on High Performance Computing. Springer, 41–59. [22] Yongchao He, Wenfei Wu, Yanfang Le, Ming Liu, and ChonLam Lao. 2023. A generic service to provide in-network aggregation for keyvalue streams. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 33–47. [23] Mert Hidayetoglu, Simon Garcia de Gonzalo, Elliott Slaughter, Pinku Surana, Wen-mei Hwu, William Gropp, and Alex Aiken. 2024. HiCCL: A Hierarchical Collective Communication Library. arXiv preprint arXiv:2408.05962 (2024). [24] Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, Salvatore Di Girolamo, Shigang Li, Marco Heddes, Deepak Goel, Miguel Castro, and Steve Scott. 2024. Hammingmesh: A network topology for large-scale deep learning. Commun. ACM 67, 12 (2024), 97–105. [25] Torsten Hoefler, Mikhail Khalilov, Josiah Clark, Surendra Anubolu, Mohan Kalkunte, Karen Schramm, Eric Spada, Duncan Roweth, Keith Underwood, Adrian Caulfield, et al. 2025. In-Network Collective Operations: Game Changer or Challenge for AI Workloads? Computer 59, 1 (2025), 24–33. [26] Torsten Hoefler, Andrew Lumsdaine, and Wolfgang Rehm. 2007. Implementation and performance analysis of non-blocking collective operations for MPI. In Proceedings of the 2007 ACM/IEEE conference on Supercomputing. 1–10. [27] Torsten Hoefler and Dmitry Moor. 2014. Energy, memory, and runtime tradeoffs for implementing collective communication operations. Supercomputing frontiers and innovations 1, 2 (2014), 58–75. [28] Chengyuan Huang, Yixiao Gao, Wei Chen, Duoxing Li, Yibo Xiao, Ruyi Zhang, Chen Tian, Xiaoliang Wang, Wanchun Dou, Guihai Chen, et al. 2023. MC-RDMA: Improving Replication Performance of RDMA-based Distributed Systems with Reliable Multicast Support. In 2023 IEEE 31st

[1] AMD. 2024. RCCL. (2024). https://github.com/ROCm/rccl. [2] Wei Bai, Shanim Sainul Abdeen, Ankit Agrawal, Krishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, et al. 2023. Empowering azure storage with RDMA. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 49–67. [3] Marcel Blöcher, Lin Wang, Patrick Eugster, and Max Schmidt. 2021. Switches for HIRE: Resource scheduling for data center in-network computing. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 268–285. [4] Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, et al. 2014. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 87–95. [5] Broadcom Inc. 2026. StrataXGS Tomahawk 5 Series: 51.2 Tb/s Ethernet Switch ASIC Family. https://www.broadcom.com/products/ethernet -connectivity/switching/strataxgs/bcm78920-series. (2026). Accessed: 2026-02-06. [6] Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao, Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. 2024. Crux: Gpu-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference. 1–15. [7] Daniele De Sensi, Salvatore Di Girolamo, Saleh Ashkboos, Shigang Li, and Torsten Hoefler. 2021. Flare: Flexible in-network allreduce. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–16. [8] Salvatore Di Girolamo, Andreas Kurth, Alexandru Calotoiu, Thomas Benz, Timo Schneider, Jakub Beránek, Luca Benini, and Torsten Hoefler. 2021. A RISC-V in-network accelerator for flexible high-performance low-power packet processing. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). IEEE, 958–971. [9] Mihai Dobrescu, Norbert Egi, Katerina Argyraki, Byung-Gon Chun, Kevin Fall, Gianluca Iannaccone, Allan Knies, Maziar Manesh, and Sylvia Ratnasamy. 2009. RouteBricks: Exploiting parallelism to scale software routers. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. 15–28. [10] Jianbo Dong, Bin Luo, Jun Zhang, Pengcheng Zhang, Fei Feng, Yikai Zhu, Ang Liu, Zian Chen, Yi Shi, Hairong Jiao, et al. 2024. Boosting large-scale parallel training efficiency with c4: A communicationdriven approach. arXiv preprint arXiv:2406.04594 (2024). [11] Jianbo Dong, Shaochuang Wang, Fei Feng, Zheng Cao, Heng Pan, Lingbo Tang, Pengcheng Li, Hao Li, Qianyuan Ran, Yiqun Guo, et al. 2021. ACCL: Architecting Highly Scalable Distributed Training Systems with Highly Efficient Collective Communication Library. IEEE Micro 41, 5 (2021), 85–92. [12] Shichen Dong, Zhixiong Niu, Mingchao Zhang, Zhiying Xu, Chuntao Hu, Pengzhi Zhu, Qingchun Song, Lei Qu, Peng Cheng, Cam-Tu Nguyen, et al. 2025. Mina: Fine-Grained In-network Aggregation Resource Scheduling for Machine Learning Service. In IEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 1–10. [13] Cheng Tien Ee, Rodrigo Fonseca, Sukun Kim, Daekyeong Moon, Arsalan Tavakoli, David E Culler, Scott Shenker, and Ion Stoica. 2006. A Modular Network Layer for Sensornets.. In OSDI, Vol. 6. 249–262. [14] Jin Fang, Gongming Zhao, Hongli Xu, Changbo Wu, and Zhuolong Yu. 2023. GRID: Gradient routing with in-network aggregation for distributed training. IEEE/ACM Transactions on Networking 31, 5 (2023), 2267–2280. [15] Edgar Gabriel, Graham E. Fagg, George Bosilca, Thara Angskun, Jack J. 15

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. International Conference on Network Protocols (ICNP). IEEE, 1–11. [29] Guyue Huang, Hao Li, Le Qin, Jiayi Huang, Yangwook Kang, Yufei Ding, and Yuan Xie. 2025. TRACI: Network Acceleration of InputDynamic Communication for Large-Scale Deep Learning Recommendation Model. In Proceedings of the 52nd Annual International Symposium on Computer Architecture. 1880–1893. [30] InfiniBand Trade Association. [n. d.]. Supplement to InfiniBand™ Architecture Specification Volume 1 Release 1.2.1: Annex A17: RoCEv2. Technical Specification Supplement Release 1.2.1, Annex A17. InfiniBand Trade Association. Proprietary document; available via InfiniBand Trade Association membership. [31] Intel. 2024. Intel Tofino 2. (2024). https://www.intel.com/content/ww w/us/en/products/details/network-io/intelligent-fabric-processors/ tofino-2.html [32] Intel. 2024. oneAPI Collective Communications Library (oneCCL). (2024). https://github.com/oneapi-src/oneCCL. [33] Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. 2023. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th annual international symposium on computer architecture. 1–14. [34] Mikhail Khalilov, Salvatore Di Girolamo, Marcin Chrapek, Rami Nudelman, Gil Bloch, and Torsten Hoefler. 2024. Network-offloaded bandwidth-optimal broadcast and Allgather for distributed AI. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–17. [35] Heehoon Kim, Junyeol Ryu, and Jaejin Lee. 2024. TCCL: Discovering Better Communication Paths for PCIe GPU Clusters. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 999–1015. [36] Benjamin Klenk, Nan Jiang, Greg Thorson, and Larry Dennison. 2020. An in-network architecture for accelerating shared-memory multiprocessor collectives. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 996–1009. [37] Eddie Kohler, Robert Morris, Benjie Chen, John Jannotti, and M Frans Kaashoek. 2000. The Click modular router. ACM Transactions on Computer Systems (TOCS) 18, 3 (2000), 263–297. [38] Leslie Lamport. 2002. Specifying systems: The TLA+ language and tools for hardware and software engineers. Addison-Wesley. [39] ChonLam Lao, Jiaqi Gao, Jiamin Cao, Zhipeng Zhang, Pengcheng Zhang, Jiangfei Duan, Minlan Yu, Aditya Akella, Zhilong Zheng, Yu Guan, Yichi Xu, Yong Li, Ennan Zhai, Dennis Cai, Zhengping Qian, and Jingren Zhou. 2026. Continuum: An Interruption-Resilient Runtime for ML Training. In OSDI. [40] ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift. 2021. ATP: In-network aggregation for multi-tenant learning. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). 741–761. [41] Wenxue Li, Xiangzhou Liu, Yuxuan Li, Yilun Jin, Han Tian, Zhizhen Zhong, Guyue Liu, Ying Zhang, and Kai Chen. 2024. Understanding communication characteristics of distributed training. In Proceedings of the 8th Asia-Pacific Workshop on Networking. 1–8. [42] Wenxue Li, Xiangzhou Liu, Yunxuan Zhang, Zihao Wang, Wei Gu, Tao Qian, Gaoxiong Zeng, Shoushou Ren, Xinyang Huang, Zhenghang Ren, et al. 2025. Revisiting RDMA Reliability for Lossy Fabrics. In Proceedings of the ACM SIGCOMM 2025 Conference. 85–98. [43] Wenxue Li, Junyi Zhang, Yufei Liu, Gaoxiong Zeng, Zilong Wang, Chaoliang Zeng, Pengpeng Zhou, Qiaoling Wang, and Kai Chen. 2024. Cepheus: accelerating datacenter applications with high-performance roce-capable multicast. In 2024 IEEE International Symposium on HighPerformance Computer Architecture (HPCA). IEEE, 908–921.

[44] Youjie Li, Iou-Jen Liu, Yifan Yuan, Deming Chen, Alexander Schwing, and Jian Huang. 2019. Accelerating distributed reinforcement learning with in-switch computing. In Proceedings of the 46th International Symposium on Computer Architecture. 279–291. [45] Zhaoyi Li, Jiawei Huang, Yijun Li, Aikun Xu, Shengwen Zhou, Jingling Liu, and Jianxin Wang. 2023. A2TP: Aggregator-aware in-network aggregation for multi-tenant learning. In Proceedings of the Eighteenth European Conference on Computer Systems. 639–653. [46] Zhaoyi Li, Jiawei Huang, Tao Zhang, Shengwen Zhou, Qile Wang, Yijun Li, Jingling Liu, Wanchun Jiang, and Jianxin Wang. 2023. PA-ATP: Progress-Aware Transmission Protocol for In-Network Aggregation. In 2023 IEEE 31st International Conference on Network Protocols (ICNP). IEEE, 1–11. [47] Linux man-pages project. [n. d.]. rxe(7): Software RDMA over Ethernet (RoCE) driver. ([n. d.]). https://man7.org/linux/man-pages/man7/rxe. 7.html Documents Linux kernel module rdma_rxe (Soft-RoCE/RXE). [48] Linux RDMA Community. 2024. libibverbs: Userspace InfiniBand Verbs Library. https://github.com/linux-rdma/rdma-core/tree/maste r/libibverbs. (2024). Part of rdma-core; provides the ibv_* API for RDMA device management, QP/CQ/MR operations, and data transfer. [49] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [50] Shuo Liu, Qiaoling Wang, Junyi Zhang, Wenfei Wu, Qinliang Lin, Yao Liu, Meng Xu, Marco Canini, Ray CC Cheung, and Jianfei He. 2023. In-network aggregation with transport transparency for distributed training. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 376–391. [51] Qingkai Meng, Hao Zheng, Zhenhui Zhang, ChonLam Lao, Chengyuan Huang, Baojia Li, Ziyuan Zhu, Hao Lu, Weizhen Dang, Zitong Lin, et al. 2025. Astral: A datacenter infrastructure for large language model training at scale. In Proceedings of the ACM SIGCOMM 2025 Conference. 609–625. [52] Zili Meng, Jun Bi, Haiping Wang, Chen Sun, and Hongxin Hu. 2019. MicroNF: An efficient framework for enabling modularized service chains in NFV. IEEE Journal on Selected Areas in Communications 37, 8 (2019), 1851–1865. [53] Microsoft. 2023. MSCCL. (2023). https://github.com/microsoft/msccl. [54] NVIDIA. 2024. NCCL. (2024). https://github.com/NVIDIA/nccl. [55] NVIDIA Corporation. [n. d.]. NVIDIA NVLink High-Speed Interconnect: Application Performance. Whitepaper. NVIDIA Corporation. [56] OMNeT++ Community. 2024. OMNeT++ Discrete Event Simulator. https://omnetpp.org. (2024). Version 6.2. [57] OpenInfra Foundation. [n. d.]. OpenStack: Open source cloud computing infrastructure. ([n. d.]). https://www.openstack.org/ [58] Aurojit Panda, Sangjin Han, Keon Jang, Melvin Walls, Sylvia Ratnasamy, and Scott Shenker. 2016. NetBricks: Taking the V out of NFV. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 203–216. [59] Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan J. Jackson, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Jonathan Stringer, Pravin Shelar, Keith Amidon, and Martin Casado. 2015. The Design and Implementation of Open vSwitch. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15). USENIX Association, 117–130. https://www.usenix.org/system/files/conference/nsdi15/ns di15-paper-pfaff.pdf [60] Chenchen Qi, Wenfei Wu, Yongcan Wang, Keqiang He, Yu-Hsiang Kao, Zongying He, Chen-Yu Yen, Zhuo Jiang, Feng Luo, Surendra Anubolu, et al. 2025. SGLB: Scalable and Robust Global Load Balancing in Commodity AI Clusters. In Proceedings of the ACM SIGCOMM 2025 16

EPIC Conference. 626–644. [61] Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. 2024. Alibaba hpn: A data center network for large language model training. In Proceedings of the ACM SIGCOMM 2024 Conference. 691–706. [62] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16. [63] Ori Rottenstreich and Jose Yallouz. 2024. Edge-disjoint tree allocation for multi-tenant cloud security in datacenter topologies. IEEE/ACM Transactions on Networking 32, 4 (2024), 2858–2874. [64] Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtárik. 2021. Scaling distributed machine learning with In-Network aggregation. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). 785–808. [65] Raz Segal, Chen Avin, and Gabriel Scalosub. 2021. SOAR: Minimizing network utilization with bounded in-network computing. In Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies. 16–29. [66] Raz Segal, Chen Avin, and Gabriel Scalosub. 2022. Constrained innetwork computing with low congestion in datacenter networks. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications. IEEE, 1639–1648. [67] Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 593–612. [68] The Tcpdump Group. [n. d.]. libpcap: Portable packet capture library. ([n. d.]). https://www.tcpdump.org/ [69] UAlink Consortium. 2024. UAlink Consortium. Online Consortium Website. (2024). https://ualinkconsortium.org/ [70] Ultra Ethernet Consortium. 2024. Ultra Ethernet Specification Update. Ultra Ethernet Consortium Blog. (29 August 2024). https://ultraether net.org/ultra-ethernet-specification-update/ Accessed: 2026-02-06. [71] Xinchen Wan, Luyang Li, Han Tian, Xudong Liao, Xinyang Huang, Chaoliang Zeng, Zilong Wang, Xinyu Yang, Ke Cheng, Qingsong Ning, et al. 2025. A Generic and Efficient Communication Framework for Message-level In-Network Computing. In IEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 1–10. [72] Ruiqi Wang, Dezun Dong, Fei Lei, Junchao Ma, Ke Wu, and Kai Lu. 2023. Roar: A router microarchitecture for in-network allreduce. In Proceedings of the 37th International Conference on Supercomputing. 423–436. [73] Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2023. TopoOpt: Co-optimizing network topology and parallelization strategy for distributed training jobs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 739–767. [74] Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, et al. 2025. SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). 541–558. [75] Yongji Wu, Yechen Xu, Jingrong Chen, Zhaodong Wang, Ying Zhang, Matthew Lentz, and Danyang Zhuo. 2024. Mccs: A service-based approach to collective communication for multi-tenant cloud. In Proceedings of the ACM SIGCOMM 2024 Conference. 679–690. [76] Junxu Xia, Wenfei Wu, Lailong Luo, Geyao Cheng, Deke Guo, and

Qifeng Nian. 2024. Accelerating and securing federated learning with stateless in-network aggregation at the edge. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). IEEE, 692–702. [77] Wenquan Xu, Zijian Zhang, Yong Feng, Haoyu Song, Zhikang Chen, Wenfei Wu, Guyue Liu, Yinchao Zhang, Shuxin Liu, Zerui Tian, et al. 2023. Clickinc: In-network computing as a service in heterogeneous programmable data-center networks. In Proceedings of the ACM SIGCOMM 2023 Conference. 798–815. [78] Mingran Yang, Alex Baban, Valery Kugel, Jeff Libby, Scott Mackie, Swamy Sadashivaiah Renu Kananda, Chang-Hong Wu, and Manya Ghobadi. 2022. Using trio: juniper networks’ programmable chipsetfor emerging in-network applications. In Proceedings of the ACM SIGCOMM 2022 Conference. 633–648. [79] Bohan Zhao, Chang Liu, Jianbo Dong, Zheng Cao, Wei Nie, and Wenfei Wu. 2023. Enabling switch memory management for distributed training with in-network aggregation. In IEEE INFOCOM 2023-IEEE conference on computer communications. IEEE, 1–10. [80] Bohan Zhao, Wenfei Wu, and Wei Xu. 2023. NetRPC: Enabling InNetwork computation in remote procedure calls. In 20th USENIX symposium on networked systems design and implementation (NSDI 23). 199–217. [81] Bohan Zhao, Wei Xu, Shuo Liu, Yang Tian, Qiaoling Wang, and Wenfei Wu. 2024. Training job placement in clusters with statistical in-network aggregation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 420–434. [82] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 (2023). [83] Haowen Zhu, Zehua Guo, and Minghao Ye. 2025. DINA: Toward Determined In-Network Aggregation for Distributed Machine Learning. IEEE Transactions on Networking (2025).

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

SEMANTICS OF COLLECTIVES

Algorithm 1: Algorithm and control flow of Mode-II

AllReduce: Its semantic is to sum up (or perform other reduction operations on) data vectors from all ranks and distribute the result to all ranks. In EPIC IncTree, data streams from leaves to the root, being aggregated at each intermediate node and sent to the parent until reaching the root. The root streams the final result back, and intermediate nodes multicast the result to their children until it reaches the leaves (Figure 2a). Reduce: Its semantic is to sum up all ranks’ data and deliver the result to one specific rank (the root). EPIC configures all non-receivers to stream data toward the receiver along the aggregation tree. Intermediate nodes aggregate incoming data streams into one and forward it toward the receiver. The receiver obtains the intermediate results and adds its own data to produce the final result (Figure 2b). Broadcast: Its semantic is to send one rank’s data to all other ranks. In EPIC, the sender streams data into the tree, and intermediate nodes replicate the data, forwarding it to all neighbors except the incoming interface, ensuring all ranks eventually receive the data copy (Figure 2c). Barrier: Its semantic is to synchronize the state among members. It is performed as an AllReduce operation with an empty data payload, leveraging the synchronization property of the reduction tree. ReduceScatter: Its semantic is to reduce data from all members, split the result, and scatter fragments to members. EPIC performs it as multiple Reduce operations. AllGather: Its semantic is to gather data from all members, concatenate it, and return the concatenated data to all members. EPIC performs this as multiple Broadcast operations.

struct Pipe 𝑝𝑎𝑦𝑙𝑜𝑎𝑑: Address of MTU array; /* in SRAM, §6 */ 3 𝑑𝑒𝑔𝑟𝑒𝑒: Address of Int array; /* in SRAM, §6 */

struct EndPoint // configuration 5 𝑟𝑒𝑚𝑜𝑡𝑒: Endpoint; 6 𝑖𝑝: IP address; 𝑞𝑝: QP Number; 𝑝𝑜𝑟𝑡: switch port; // receive states 7 𝑎𝑟𝑟𝑖𝑣𝑒𝑑: Array of Bit;

struct Context // persistent states 9 𝑁 : Int of array size; 𝑒𝑝𝑠: Array of EndPoint; 𝑝𝑖𝑝𝑒: Pipe; // invocation states 10 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑖𝑣𝑒: Enum; 𝑟𝑜𝑜𝑡: Int; 𝑓 𝑖𝑟𝑠𝑡𝑃𝑠𝑛: Int; 𝑙𝑎𝑠𝑡𝑃𝑠𝑛: Int; 8

while receive a packet pkt do (ctx, ep, class, fanin, outs) = LookupTable(pkt); 13 pkt.idx = pkt.psn%ctx.N; 14 pkt.idx2 = (pkt.psn+ctx.W)%ctx.N; 15 if class = UP_DATA then 16 isDup = CheckDuplicate(pkt, ep); 17 if not isDup then AggregateData(ctx.pipe, pkt) ; 18 if ctx.pipe.degree[pkt.idx] < fanin then continue ; 19 pkt.payload = ctx.pipe.payload[pkt.idx]; 20 RecycleBuffer(ctx.pipe, pkt.idx2, pkt.idx2 + 1); 12

21 22 23

Function CheckDuplicate(pkt, ep): v=ep.arrived[pkt.idx]; ep.arrived[pkt.idx]=1; 26 return v==1;

INCENGINE ALGORITHMS

Function AggregateData(pipe, pkt): pipe.payload[pkt.idx] += pkt.payload; 29 pipe.degree[pkt.idx] += 1; 28

Algorithm 1 shows the IncEngine algorithm for Mode-II, and Algorithm 2 shows that for Mode-III. Algorithm 2 can evolve from Algorithm 1 with changes highlighted.

for (pkt’, out) ∈ DuplicateData(pkt, outs) do pkt’ = TranslateHeader(pkt’, out); Forward(pkt’, out.port);

Function RecycleBuffer(pipe, start, end): for i ∈ [start, end) do 32 pipe.payload[i]=pipe.degree[i]=0;

IMPACT FROM ROCE’S RELIABILITY

Mode-II is compatible with both versions of RoCE reliability mechanisms. (1) Go-Back-N (GBN): Timeouts and NAKs trigger the sender to retransmit from the first missing PSN. EPIC keeps idempotent when processing retransmissions. An optimization is to maintain an 𝑒𝑝𝑠𝑛 (expected PSN) in the switch endpoint state, dropping out-of-order packets immediately. This matches the receiver’s logic (which would NAK out-oforder packets anyway) and reduces switch buffer access cost. (2) Selective Retransmission (SR): SACKs describe ranges of out-of-order packets. EPIC switches and receiver buffers can handle out-of-order arrival. The complexity arises in Broadcast ACK aggregation, where merging SACK ranges requires computing the intersection of multiple intervals.

Given the low packet loss rate in data centers, and effective congestion control and flow control (e.g., credit-based, or priority-based), these corner cases are infrequent. Mode-III does not confine the selection of reliability mechanism in Mode-III. Both GBN and SR can work correctly in switch nodes. The module Retransmission need to adapt the corresponding packet sending to the selection.

OTHER POSSIBLE INCENGINE MODES

There could be other modes, representing a tradeoff between complexity, cost, and performance. For example, in Mode-II AllReduce, the switch node can buffer broadcasted result 18

EPIC Algorithm 2: Algorithm and control flow of ModeIII (changes hightlighted)

Algorithm 3: Algorithm and control flow of ModeIII (continued, changes hightlighted)

struct Pipe 𝑝𝑎𝑦𝑙𝑜𝑎𝑑: Address of MTU array; 3 𝑑𝑒𝑔𝑟𝑒𝑒: Address of Int array; 4 𝑓 𝑟𝑜𝑚𝐸𝑝𝑠: Array of EndPoint; 5 𝑡𝑜𝐸𝑝𝑠: Array of EndPoint; 6 𝑝𝑠𝑛𝑆𝑡𝑎𝑟𝑡 : Int;

Function ReceiveAck(pkt, ep): ep.lastAcked = Max(ep.lastAcked, pkt.psn); ep.resend.set( ep.maxPsnSent > ep.lastAcked ? TIMEOUT : 0 ); 3 Function SendAck(pkt, ep): pkt’=MakePacket(ep.ip, ep.remote.ip, ep.remote.qp, ep.epsn-1, ep.epsn-1==pkt.psn?ACK:NAK); 6 return (pkt’, ep.port)

struct EndPoint // configuration 8 𝑟𝑒𝑚𝑜𝑡𝑒: Endpoint; 9 𝑖𝑝: IP address; 𝑞𝑝: QP Number; 𝑝𝑜𝑟𝑡 : switch port; // receive states 10 𝑎𝑟𝑟𝑖𝑣𝑒𝑑: Array of Bit; 𝑒𝑝𝑠𝑛: Int; 𝑓 𝑟𝑜𝑚𝑃𝑖𝑝𝑒: Pipe; // send states 11 𝑟𝑒𝑠𝑒𝑛𝑑: Timer; 𝑙𝑎𝑠𝑡𝐴𝑐𝑘𝑒𝑑: Int; 𝑚𝑎𝑥𝑃𝑠𝑛𝑆𝑒𝑛𝑡 : Int; 𝑡𝑜𝑃𝑖𝑝𝑒: Pipe; 12 7

Function Retransmission(timer, ep, ctx): for idx ∈ [ep.lastAcked + 1, ep.maxPsnSent] do 9 if ep.fromPipe.degree[idx%ctx.N] != fanin then 10 continue; 7

struct Context 𝑁 : Int of array size; 𝑒𝑝𝑠: Array of EndPoint; 15 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑖𝑣𝑒: Enum; 𝑟𝑜𝑜𝑡 : Int; 𝑓 𝑖𝑟𝑠𝑡 _𝑝𝑠𝑛: Int; 𝑙𝑎𝑠𝑡 _𝑝𝑠𝑛: Int; 𝑎𝑔𝑔𝑃𝑖𝑝𝑒: Pipe; 𝑏𝑐𝑎𝑠𝑡𝑃𝑖𝑝𝑒: Pipe; 16

For another example, in Mode-II, if RoCE applies Go-backN (GBN) in retransmission, the switch can also maintains an epsn state (expected PSN) to optimize traffic; it filters all packets out-of-order to reduce network traffic (because out-of-order packets are dropped and retransmitted by NIC). The design space is large, and there could also be pitfalls (§5.1).

while receive a packet pkt do (ctx, ep, class, fanin, outs) = LookupTable(pkt, table); 19 pkt.idx = pkt.PSN % ctx.N; 20 if pkt is ACK then 21 ReceiveAck(pkt, ep); 22 pipe = ep.fromPipe; eps = pipe.toEps; 23 psnStart0 = pipe.psnStart; 24 pipe.psnStart = Min({e.lastAcked | e ∈ eps}) + 1; RecycleBuffer(pipe, psnStart0, pipe.psnStart); 25 26 continue;

27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

pkt = MakePacket(ep.ip, ep.remote.ip, ep.remote.qp, idx, ep.fromPipe.payload[idx % ctx.N]); Forward(pkt); timer.set(TIMEOUT);

SWITCH MICROARCHITECTURE ② ingress ①

pipe = ep.toPipe; toSend = ∅; if pkt.psn ∉ [pipe.psnStart, pipe.psnStart + N) then continue;

Traffic ③ Manager

egress

Figure 17: EPIC in switch micro architecture Modern high-performance switches typically utilize a pipelined architecture comprising an Ingress Pipeline, a Traffic Manager (TM), and an Egress Pipeline. The Ingress Pipeline performs packet parsing and forwarding decisions, while the TM serves as the data plane core, managing on-chip shared memory for buffering and scheduling. The Egress Pipeline completes the process with packet rewriting and encapsulation. Within this framework, the IncEngine can be integrated through three distinct architectural patterns: Ingress, Loopback, or Bypass. 1 embeds the IncEngine logic diIngress Integration (○) rectly into the ingress stages, allowing computation and modification to occur before packets reach the TM. This approach, exemplified by programmable targets like Tofino, enables line-rate processing with minimal latency. However, it is fundamentally constrained by pipeline resources, such as limited ALU stages, and frequently requires packet recirculation to handle large payloads, which can significantly consume internal switching bandwidth. 2 treats the IncEngine Alternatively, Loopback Integration (○) as a "look-aside" accelerator attached to a dedicated loopback

isDup ← CheckDuplicate(pkt, ep); while ep.arrived[ep.epsn] == 1 do ep.epsn++ ; toSend.add( SendAck(pkt, ep) ); if isDup then goto FORWARD ; AggregateData(pkt, pipe); if pipe.degree[pkt.idx] < fanin then goto FORWARD ; pkt.payload = pipe.payload[pkt.idx]; for out ∈ outs do pkt1 = TranslateHeader(pkt, out); out.maxPsnSent = Max(out.maxPsnSent, pkt1.psn); toSend.add(pkt1, out.port); out.resend.set(TIMEOUT) FORWARD: for (pkt’, port) ∈ toSend do Forward(pkt’, port) ;

packets, and return the result directly when data packet retransmission happens; but this change consequently requires the upward data packet workflow to check whether the broadcast buffer has the result ready to send, which is workable but with a bit more complexity. 19

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. port. Packets identified by the Ingress Pipeline are routed to this port for processing before being re-submitted to the ingress stage [72]. While this mode offers high compatibility and supports complex operations without modifying the core TM logic, it inevitably increases latency due to multiple pipeline passes and places additional strain on internal switching capacity. 3 positions the IncEngine Finally, Bypass Integration (○) as an auxiliary component connected directly to the on-chip shared memory via a dedicated interface. In this configuration, the Ingress Pipeline tags INC packets, triggering the TM to notify the engine to aggregate data directly from memory and write results back. This design balances latency and complexity by avoiding repetitive pipeline traversals and leveraging shared memory bandwidth for high-throughput aggregation, aligning with advanced designs like Roar.

Transmission Efficiency

packet with U bytes

message with M packets, MU bytes time

overall data is S bytes

2H hops (2H-1) ×(L+U/B)

L+S/B

(2H-1) ×(L+MU/B)

L+S/B

Switches operate in Store-and-Forward model but at different granularities across modes. Following the definitions in Table 2, Mode-I switches operate with a message granularity (𝑀 MTUs), requiring the reception of a full message before processing. In contrast, Mode-II/III operate with a packet (MTU) granularity (Figure 18). Assuming an AllReduce operation traverses a tree of depth 𝐻 where the path length is 2𝐻 − 1 hops. The total time to transmit application data size 𝑆 is composed of propagation delay and transmission delay: In Mode-I, the total time includes (2𝐻 − 2)-hop propagation delay plus (2𝐻 − 2)-hop of store-and-forward delay (message length 𝑀 × 𝑈 ) plus the final transmission time:

Space Complexity

F.4

Loss Tolerance

  𝐵,    𝑇𝑖 = 𝑇𝑖 −1 · min 𝑗 ∈𝑖.children 𝐹 (𝑟 𝑗 ),   𝑇𝑖 −1 · min 𝑗 ∈𝑖.siblings 𝐹 (𝑟 𝑗 ), 

(1)

In Mode-II/III, due to packet-level pipelining, the delay is dominated by the single packet serialization time: 𝑇𝑀𝑜𝑑𝑒 −𝐼 𝐼 /𝐼 𝐼 𝐼 = (2𝐻 − 2)𝐿 + (2𝐻 − 1)𝑈 /𝐵 + 𝑆/𝐵

F.3

Let 1−𝑟 denote the packet loss rate, i.e., 𝑟 is the packet success rate. Endpoint throughput achieves bandwidth saturation with 𝑟 = 1, and the throughput decreases by a factor of 𝐹 (𝑟 ) with 𝑟 < 1. We consider the AllReduce on a depth-𝐻 IncTree, there are 2𝐻 − 1 hops. We first model the impact of packet loss on throughput in AllReduce. Let 𝑟 be the transmission success rate (1 − 𝑟 is loss rate) and 𝐹 (𝑟 ) be the throughput decay factor. With an aggregation of depth 𝐻 , AllReduce data packets traverses 2𝐻 − 2 hops; hop 𝑖 to 𝑖 + 1 is upward when 1 ≤ 𝑖 ≤ 𝐻 − 1, and downward when 𝐻 ≤ 𝑖 ≤ 2𝐻 − 2. In Mode-I/III, in aggregation and broadcast, transmissions between sibling nodes are independent due to independent loss retransmission. The parent’s throughput is synchronized by the slowest child. Let 𝑇𝑖 be the receive throughput at hop 𝑖. Thus,

Figure 18: Transmission Efficiency

𝑇𝑀𝑜𝑑𝑒 −𝐼 = (2𝐻 − 2)𝐿 + (2𝐻 − 1)𝑀𝑈 /𝐵 + 𝑆/𝐵

Logic Complexity

Mode-I uses hop-by-hop flow control, requiring buffers for one-hop bandwidth-delay product (BDP) (2𝐵𝐿). For aggregation with degree 𝐷, it needs 𝐷 − 1 receive buffers, 1 aggregation buffer, and 1 broadcast buffer. Total ≈ (𝐷 + 1) × 2𝐵𝐿. Mode-II relies on end-to-end flow control, requiring buffers to cover the path BDP (2 × (2𝐻 − 2)𝐵𝐿). Total usage is 4(𝐻 − 1)𝐵𝐿. Supporting reproducibility needs separate buffer for data packets, increasing the space to 4(𝐻 − 1) (𝐷 + 1)𝐵𝐿. Mode-III introduces hop-by-hop reliability, buffering for single-hop BDP (2𝐵𝐿). Total usage is 4𝐵𝐿 for aggregate and broadcast. With reproducibility, it becomes (𝐷 + 1) × 2𝐵𝐿. The order of space overhead from small to large is ModeIII = Mode-I < Mode-II. Mode-I forces reproducibility, but Mode-III provides non-reproducibility options which achieve a smaller space.

ANALYSIS OF MODES

F.1

F.2

Comparing the number of modules and lines of code in implementation, the order of modes from simple to complex is Mode-II, Mode-III, and Mode-I.

if 𝑖 = 1 if 2 ≤ 𝑥 < 𝐻 if 𝑥 ≥ 𝐻 .

In Mode-II, in aggregation, the success of aggregating all packets with the same PSN is forced to synchronize with the slowest one; in broadcast, the ranks affected by packet loss would spare extra bandwidth to retransmit, affecting sending new packets and also slowing down other lossless ranks (due to synchronization). We mode the overall throughput loss by a multiplicative model:

(2)

Mode-II/III always outperforms Mode-I by (2𝐻 − 1) (𝑀 − 1)𝑈 /𝐵 less time. In practice, if 𝑆 ≫ 𝑀𝑈 (very large message) or 𝐿 ≫ 𝑆/𝐵 (very small message), this advantage is not significant. 20

EPIC

  𝐵, if 𝑖 = 1    Î 𝑇𝑖 = 𝑇𝑖 −1 · 𝑗 ∈𝑖.children 𝐹 (𝑟 𝑗 ), if 2 ≤ 𝑥 < 𝐻   𝑇𝑖 −1 · Î 𝑗 ∈𝑖.siblings 𝐹 (𝑟 𝑗 ), if 𝑥 ≥ 𝐻 .  We simplify the model by assuming 𝑟 𝑗 = 𝑟 and 𝐹 (𝑟 ) = 𝑟 for further analysis 7 . Mode-I & Mode-III achieve: 𝑇 ≈ 𝐵 ·𝑟 2(𝐻 −1) ; Mode-II achieves: 𝑇 ≈ 𝐵 · (𝑟 𝐷 −1 ) 2(𝐻 −1) = 𝐵 · 𝑟 2(𝐻 −1) (𝐷 −1) . Mode-I and Mode-III outperform Mode-II in loss tolerance. In practice, if lossless layer-2 mechanisms (PFC/CBFC) is applied where 𝑟 ≈ 1, the three modes perform similarly.

INCMANAGER ORGANIZATION

job placement, tree placement, and resource allocation Edge-Disjoint Tree

Spatial Mux

Resource Model

Topology

Temporal Mux

Switches

Groups

Southbound APIs to/from CommLibs & IncAgents

Figure 19: IncManager and policies Figure 19 shows the organization of IncManager.

7 In practical system, 𝐹 (𝑟 ) could be worse than linear, because packet loss

triggers congestion control, which decreases sending rate or congestion window.

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

MODEL CHECKING

H.1

Implementation

antees correctness under all possible concurrent executions, even in the presence of packet loss, out-of-order, and duplication. Since ReduceScatter and AllGather can be regarded as multiple independent Reduce and Broadcast operations, their correctness is guaranteed by this independence together with the verified correctness of Reduce and Broadcast. The topology is small but suffices to capture bugs. Due to node symmetry and state locality, a larger topology does not provide new protocol behaviors.

Table 7: [Model Checking] LOC and Verification Time Mode/Primitive Mode-II/AllReduce Mode-II/Reduce Mode-II/Broadcast Mode-III/AllReduce Mode-III/Reduce Mode-III/Broadcast

TLA+ LOC Time 1222 20s 1139 3s 1108 10s 1741 146min 1394 9s 1386 113s

H.4

In this section, we present some representative pitfalls of our early design, discovered through model checking. Window Synchronization. In Mode-III, hop-by-hop acknowledgments decouple the windows across different flows. Therefore, an additional synchronization mechanism must be used, as discussed in §5.1 Root-Specific Treatment. During the recursive aggregation process, the root node of the aggregation tree is special and requires dedicated handling. For example, in Mode-III, when performing AllReduce, aggregation and broadcast are decoupled in all switches but the root, which acts as a translation point of UP_DATA to DOWN_DATA. It cannot directly copy the data in AggregateData module to ReplicateData, which has the risk of overwriting an ongoing broadcast. Instead, the root should generate an local DOWN_DATA packet as if receiving from its parent, and regenerates the packet on timeout. In later design, we summarize the pipe abstraction and unify the workflow §4.4. NAK Necessity and Rate Limiting. In Mode-III, the NAK mechanism is strongly suggested to be enabled; otherwise, a single packet loss can stall the entire aggregation tree and trigger timeout-based retransmission in every node, degrading system performance. However, one packet loss could cause all subsequent packets to trigger NAKs, leading to redundant retransmissions. Given the limited processing capability of switches, we adopt the simplest implementation: maintaining a nak_sent flag per flow. When an in-order packet is received, the flag is set to false. When an out-of-order packet is received, the switch checks this flag—if it is false, the switch replies with a NAK and sets the flag to true; otherwise, the packet is silently dropped.

We model the IncEngine logic of Modes-II and III based TLA+ specification [38] and verify them with the TLA+ model checker (TLC). Our implementation covers three primitives—AllReduce, Reduce, and Broadcast. Table 7 summarizes the line of code (LOC) and the verification time. The code is open-sourced at GitHub8 .

H.2

Experiment Settings Table 8: [Model Checking] State Space Size

Mode/Primitive Mode-II/AllReduce Mode-II/Reduce Mode-II/Broadcast Mode-III/AllReduce Mode-III/Reduce Mode-III/Broadcast

Diameter Total Distinct 77 3981257 901703 57 52105 14136 60 1763880 320128 115 2242781644 439476508 65 2291727 447972 67 67902618 9911424

We evaluate model checking efficiency on a workstation with 75 cores of 2.3GHz Intel(R) Xeon(R) Silver 4316 CPU and 128 GB DRAM. The network topology is set to be a three-level full binary tree topology (i.e., one spine switch, two leaf switch, and four clients). Each client sends three request packets. At most one packet loss may occur at any arbitrary moment. The entire state space (i.e., all possible concurrent executions) is explored.

H.3

Design Pitfalls

Experiment Results

Efficiency. Table 7 and Table 8 shows the verification time and state space size of model checking. The diameter refers to the maximum distance from the initial state to all reachable states. The total number of states represents the total number of explored states, while the number of distinct states indicates the de-duplicated count. The most time-consuming setting completes in 146 min. We regards it as acceptable, because it is a one-time cost during design, instead of runtime overhead. All other settings complete in 3 s-113 s. Correctness. We verified our final design under different system configurations. Results show that the design guar8 https://anonymous.4open.science/r/epic-verification-37D3

EPIC

TESTBED EVALUATION

cause its RDMA latency is lower than the DPDK latency used by SwitchML and ATP. ATP’s Parameter Server (PS) introI.1 Implementation duces additional network hops, resulting in slightly higher We implemented the EPIC server-side library in C++ and its latency compared to SwitchML. NCCL exhibits the highest switch logic in P4, managed by a lightweight Python-based latency and lowest bandwidth due to significant server-side control plane. The P4 program was compiled and deployed control overhead. on a programmable Intel Tofino switch for hardware-level For large messages, all three in-network computing schemes execution. demonstrate higher bandwidth than NCCL by compressWhen integrating EPIC CCL with PyTorch, the message ing network traffic. Furthermore, EPIC ’s RDMA achieves buffer in EPIC are registered RDMA buffers, and the applislightly higher network bandwidth utilization than DPDK. cation data are in the application buffer, not in the regisIn the ATP 8-to-1 configuration, a bottleneck occurs because tered buffers. Copying data between buffers is costly. We the PS must simultaneously transmit data to be aggregated also pipeline the memory copy with the RDMA message and aggregated results to the switch; consequently, the bandsend/receive. width is halved compared to ATP 7-to-1, falling even below As the Tofino switch only handles integer numbers, when that of NCCL. performing collectives on floating-point numbers, we (de)quantize For other collectives (Tables 10 to 14), NCCL continues to the number with a fixed scaling factor; if value summation underperform compared to EPIC for small messages due to overflows, the switch rounds the value to maximum integer its control latency. value; by end-to-end test, such (de)quantization does not For large messages, EPIC’s performance in ReduceScatter affect training accuracy. We use the same method to handle is inferior to NCCL because it does not provide traffic comdata (de)quantization as ATP [40]. pression for this specific collective. However, in asymmetric Our code is publicly available.9 communication scenarios like Reduce, EPIC outperforms NCCL by reducing the number of network hops. Regarding I.2 Experiment Settings Broadcast and AllGather, since there is no requirement for Workloads. We evaluate the collective and application perin-network payload aggregation, formance of EPIC. The environment settings are as §7.1. For collective evaluation, we run all six collectives with message sizes ranging from 4KB to 1GB with a scaling factor of 4. For application evaluation, we run an LLM training job with tensor and data parallelism as shown in Table 15 and Table 16. Baselines. In the collective evaluation, we compare EPIC against SwitchML, ATP, and NCCL. We evaluate two ATP configurations: in ATP 7-to-1, one GPU serves as the Parameter Server (PS) to receive in-network aggregated data and perform local aggregation before broadcasting the results; ATP 8-to-1 follows a similar logic, but the PS also transmits data to the switch, eliminating the need for local aggregation. For NCCL, we vary the MTU (1KB and 256B) and routing algorithms (Ring and Tree) to assess their impact. Note that the Tree algorithm is only supported by AllReduce; furthermore, we omit MTU and algorithm tuning for the Barrier collective as they do not affect its performance. Importantly, SwitchML and ATP only support the AllReduce collective. For application evaluation, we utilize the same baselines, specifically using ATP 7-to-1 to represent ATP and NCCLRing-1KB to represent NCCL.

I.3

I.4

Application Performance

I.5

Overhead

The application-level evaluation yields results consistent with those observed in the collective communication analysis. In-network computing schemes outperform NCCL, with EPIC specifically demonstrating superiority over SwitchML and ATP. For Data Parallelism (DP), the performance gap is relatively small because communication accounts for a smaller fraction of the total execution time. Conversely, for Tensor Parallelism (TP), the performance differences are more pronounced due to the higher proportion of communication overhead. Table 17 illustrates EPIC ’s switch resource utilization. As the total size of the aggregators increases, the SRAM consumption grows significantly.

Collective Performance

Table 9 illustrates the performance of the all-reduce operation. For small messages, EPIC outperforms the others be9 https://anonymous.4open.science/r/EPIC-D242.

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. Table 9: [Testbed] AllReduce Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II SwitchML ATP 7-to-1 ATP 8-to-1 NCCL-Ring-1KB NCCL-Ring-256B NCCL-Tree-1KB NCCL-Tree-256B

4K 5.42 3.85 1.98 1.92 0.015 0.015 0.015 0.015

16K 16.84 13.10 7.21 6.47 0.058 0.060 0.060 0.060

64K 41.72 32.73 21.24 15.93 0.243 0.251 0.244 0.223

256K 59.46 52.34 41.34 25.07 0.91 0.82 0.89 0.86

1M 60.79 61.56 54.16 29.27 3.47 3.71 3.17 3.12

4M 62.65 64.39 58.71 30.54 13.34 12.10 11.05 10.12

16M 64.35 65.14 59.97 30.88 29.06 26.55 24.85 21.39

64M 61.70 65.37 60.29 30.96 40.99 36.01 31.89 27.37

256M 66.64 65.37 60.37 30.96 46.33 39.73 33.51 29.01

1G 65.29 65.38 60.37 30.94 47.97 41.07 34.10 29.29

Table 10: [Testbed] Reduce Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II NCCL-1KB NCCL-256B

4K 4.19 0.016 0.015

16K 16.58 0.065 0.060

64K 37.17 0.263 0.240

256K 55.56 1.018 0.997

1M 71.63 3.867 3.586

4M 74.39 14.25 12.73

16M 67.70 40.72 12.73

64M 66.28 64.97 57.70

256M 68.38 67.09 59.82

1G 66.68 67.26 59.56

Table 11: [Testbed] ReduceScatter Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II NCCL-1KB NCCL-256B

4K 3.97 0.014 0.014

16K 15.64 0.056 0.056

64K 39.59 0.218 0.240

256K 62.47 0.926 0.927

1M 61.36 3.34 3.65

4M 74.33 12.92 12.70

16M 66.61 41.07 35.97

64M 71.55 72.29 62.24

256M 67.21 89.58 77.74

1G 66.30 95.58 81.32

Table 12: [Testbed] Broadcast Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II NCCL-1KB NCCL-256B

4K 4.45 0.015 0.015

16K 16.08 0.061 0.063

64K 41.22 0.24 0.24

256K 72.92 0.913 0.99

1M 86.18 3.52 3.61

4M 89.76 12.92 12.92

16M 90.86 39.99 35.63

64M 92.13 65.61 55.54

256M 88.73 66.60 60.02

1G 88.96 66.60 62.86

Table 13: [Testbed] AllGather Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II NCCL-1KB NCCL-256B

4K 4.37 0.015 0.014

16K 16.66 0.059 0.058

64K 39.11 0.232 0.235

256K 72.56 0.883 0.885

1M 86.26 3.76 3.70

4M 89.97 13.58 12.34

16M 91.69 39.93 37.26

64M 92.08 70.26 62.01

256M 90.73 89.37 74.22

1G 90.55 95.58 80.74

EPIC Table 14: [Testbed] Barrier Throughput Solution EPIC-II NCCL

Throughput (K requests/second) 165 0.458

Table 15: [Testbed] Training job iteration time (s), context_length=2k, batch_size=256, DP=8 Model EPIC-II SwitchML ATP NCCL

GPT-2 Large 5.46 5.48 5.51 5.53

Qwen2.5-0.5B 3.59 3.61 3.62 3.64

Llama-3.2-1B 7.18 7.22 7.24 7.28

Table 16: [Testbed] Training job iteration time (s), context_length=2k, batch_size=256, TP=8 Model EPIC-II SwitchML ATP NCCL

GPT-2 Large 21.79 24.33 26.62 29.35

Qwen2.5-0.5B 13.10 14.58 15.92 17.51

Llama-3.2-1B 21.63 23.88 25.93 28.35

Table 17: [Testbed] Tofino Resource Usage SRAM Hash Bit Gateway SRAM TCAM VLIW Instr Map RAM Meter ALU PHV

128KB 5.65% 22.92% 7.92% 1.39% 8.59% 12.33% 72.92% 34.80%

512KB 6.05% 22.92% 7.92% 1.39% 8.59% 12.33% 72.92% 34.80%

2MB 6.45% 22.92% 11.25% 1.39% 8.59% 17.88% 72.92% 34.80%

8MB 6.85% 22.92% 31.67% 1.39% 8.59% 51.91% 72.92% 34.80%

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

J J.1

EMULATION Implementation

We implement six collective communication primitives (AllReduce, Reduce, Broadcast, Barrier, ReduceScatter, and AllGather) across three EPIC modes on a virtualmachine testbed using libibverbs [48] and libpcap [68], demonstrating the interoperability between EPIC and RDMA verbs. EPIC-I. In Mode 1, the switch establishes native RDMA Queue Pair (QP) connections via the libibverbs API and performs aggregation at the message level upon receiving complete RDMA messages. The RDMA transport layer handles all reliability semantics natively, eliminating the need for custom PSN tracking or ACK generation. The switch application layer maintains a set of aggregation slots, which are reused through modular indexing. Control information is conveyed via the 32-bit Immediate Data field encoded as [slot_id:20][primitive:2][op:2][sender:4][root:4]. The switch routes messages according to its role (Leaf or Root) and the primitive type: a Leaf switch aggregates data from its local hosts before forwarding to the parent switch, while the Root switch aggregates contributions from all child switches and broadcasts the result back down the tree. EPIC-II. In Mode 2, the switch operates as a transparent middlebox, intercepting RoCEv2 packets via libpcap, performing payload aggregation and header rewriting before forwarding, without terminating the RDMA connection. The host RDMA stack retains full responsibility for reliability. The switch maintains a simplified global context (without retransmission timers). The core state array is indexed by PSN mod SWITCH_ARRAY_LENGTH, and a 32-bit arrival_state bitmap tracks the arrival status of each connection. The I/O framework in Mode 2 is built upon a single-threaded event loop using Linux epoll. Each physical network interface is assigned a pcap handle (configured in promiscuous mode, non-blocking, with a 256 MB kernel buffer) and registered with epoll. The main loop processes up to 256 packets per device per iteration, mapping source IPs to connection IDs before dispatching to a P4-style three-stage pipeline: parser (extracting ETH/IP/UDP/BTH headers) → match (determining direction and PSN window membership) → action (primitive-specific logic). EPIC-III. In Mode 3, the switch terminates RDMA connections—from the host’s perspective, the switch acts as a full RoCEv2 endpoint. It intercepts raw Ethernet frames via libpcap, parses the complete RoCEv2 protocol stack (ETH → IPv4 → UDP → BTH → Payload → ICRC), performs aggregation at per-PSN granularity, and constructs new packets for downstream broadcast. The switch manages its own PSN space, generates ACK/NAK responses, and implements timeout-based retransmission. 26

Baseline. In addition to the three modes described above, we deploy OpenMPI 4.1.5 [15] in the same environment as two baselines. MPI routes traffic through a simple L3 forwarding switch (soft_switch) that relays IP packets between subnets via libpcap without any payload inspection or modification. MPI-ICRC uses the same L3 switch but additionally recomputes the RoCEv2 Invariant CRC (ICRC) on every forwarded packet, matching the per-packet ICRC overhead incurred by Mode 2 and Mode 3 when they rewrite packet headers. Comparing MPI and MPI-ICRC isolates the cost of software ICRC recomputation from the cost of in-network aggregation. Code. Our emulation source code is publicly available.10

J.2

Experiment Settings

J.3

Development Workload

J.4

Performance

All three modes are evaluated on the same 1-2-4 fat-tree topology, deployed across eight OpenStack [57] virtual machines interconnected by an OVS virtual network. RDMA transport uses Soft-RoCE [47] (the rdma_rxe kernel module). Message sizes increase by a factor of four, covering both small and large transfers; the payload per packet is 4KB. The sliding window, aggregation slot size, and CQ size are all configured large enough to avoid performance bottlenecks. Each experiment is run 10 times, and we report the average throughput. The startup order is: Spine (ROOT) → Leaf 1 → Leaf 2 → four hosts in parallel. Table 18 summarizes the code volume reported in the three mode documents. Mode 2 and Mode 3 share the same host API and controller codebase; therefore, their totals are not additive. Specifically, Mode 3 reuses 4,630 lines from Mode 2— including the INCCL host library, protocol utilities, and the controller communication module—accounting for approximately 61% of Mode 2’s total codebase. AllReduce (Table 19). Mode 1 achieves the highest throughput among the three modes, reaching 427 Mbps at 1 GB— roughly 4× that of Mode 2 and Mode 3. This advantage stems from message-level aggregation (64 KB per RDMA message), which reduces the number of switch-side operations by up to 64× compared with the per-packet modes. Mode 2 and Mode 3 saturate near 75 Mbps and 95 Mbps respectively once the message size exceeds 256 KB, reflecting the per-packet processing overhead of libpcap-based interception. Mode 3 slightly outperforms Mode 2 because its connection-termination design enables independent PSN management and avoids head-of-line blocking from the host retransmission timer. All three modes surpass MPI for small messages (≤64 KB), where the fixed cost of MPI’s multi-hop 10 https://anonymous.4open.science/r/EPIC_Emulation-0138/

EPIC Table 18: Development Workload Mode EPIC-I EPIC-II EPIC-III

Switch code 3,244 3,660 2,898

Host/shared 510 – 3,944

Tests 458 – 583

Scripts 1,909 – –

Total / Notes 6,142 (includes build files) 3,660 (switch only) 7,400 (includes shared host API)

Table 19: [Emulation] AllReduce Algorithm Throughput (Mbps) Msg. Size (B) EPIC-I EPIC-II EPIC-III MPI MPI-ICRC

4K 2.47 14.74 15.69 1.71 1.71

16K 10.24 48.10 64.14 3.72 4.97

64K 35.89 67.32 88.69 14.89 19.86

256K 78.97 85.11 86.77 21.12 18.95

1M 182.44 88.96 72.54 73.90 75.73

ring or recursive-doubling algorithm dominates; MPI catches up at large sizes owing to its optimized pipelining, ultimately reaching 502 Mbps at 1 GB. Reduce (Table 20). Mode 1 again leads, peaking at 512 Mbps and sustaining 506 Mbps at 1 GB. The unidirectional nature of Reduce (no broadcast phase) allows Mode 1 to approach the raw RDMA link rate. Mode 2 and Mode 3 plateau around 110 Mbps and 95 Mbps respectively, consistent with the perpacket processing ceiling observed in AllReduce. MPI Reduce throughput grows steadily with message size, reaching 455 Mbps at 1 GB, but remains below Mode 1 across all sizes. ReduceScatter (Table 21). Mode 2 and Mode 3 achieve comparable peak throughput (∼105 Mbps and ∼120 Mbps), while Mode 1 plateaus at ∼72 Mbps. Mode 1’s lower throughput is attributable to its simulation strategy: ReduceScatter is decomposed into 𝑁 sequential Reduce calls (one per rank), incurring 𝑁 × the connection setup and synchronization overhead. MPI ReduceScatter scales to 601 Mbps at 1 GB, benefiting from its native implementation that overlaps reduction and data redistribution in a single pass. Broadcast (Table 22). Broadcast exhibits a different ranking: Mode 3 achieves the highest peak at 381 Mbps (256 KB) and sustains ∼200 Mbps for large messages. This is because Broadcast involves no aggregation—the switch simply replicates packets—and Mode 3’s connection termination allows it to drive downstream links at full speed without waiting for end-to-end ACKs. Mode 2 sustains ∼140 Mbps, while Mode 1 plateaus near 80 Mbps, limited by the overhead of posting individual RDMA Send operations for each 4 KB chunk across multiple QP connections. MPI Broadcast scales well beyond 1 Gbps at 64 MB due to its tree-based pipelining, significantly outperforming all three INC modes for large transfers. Mode 1 achieves the highest INC throughput at 313 Mbps (1 GB), followed by Mode 2 at 263 Mbps. Mode 3 exhibits anomalously low throughput (169 Mbps at 1 GB, and below 1 Mbps for messages ≤16 KB), suggesting a performance bottleneck in its broadcast-based AllGather path—likely caused by serialized per-rank broadcast rounds combined with perpacket ACK generation overhead. MPI AllGather reaches

4M 247.87 93.74 75.54 181.22 163.95

16M 381.91 86.64 74.86 316.46 235.53

64M 417.60 89.33 75.46 249.17 264.85

256M 425.77 94.11 75.13 434.01 314.91

1G 426.94 97.86 75.64 501.59 389.61

827 Mbps at 1 GB, again benefiting from its native ring-based algorithm that pipelines data transfers across all ranks simultaneously. Barrier (Table 30). Mode 3 delivers the highest barrier throughput at 2,502 requests/second, followed by Mode 1 at 1,331 requests/second and Mode 2 at 590 requests/second. All three modes outperform MPI (38 requests/second) by 15–66×, demonstrating the latency advantage of in-network synchronization: a single switch-hop round trip replaces the 𝑂 (log 𝑁 ) steps required by MPI’s tree-based barrier. Mode 3 leads because its packet-level operation avoids the QP setup and message-framing overhead of Mode 1, while its connection termination eliminates the retransmission-timer delays that slow Mode 2. Mode-I and MPI outperform Mode-II/III at large message sizes. The throughput ceiling of Mode 2 and Mode 3 (typically 75–140 Mbps for aggregation-based primitives) is fundamentally limited by the libpcap user-space packet processing path. Every RoCEv2 packet traverses the kernel capture buffer, crosses the kernel–user boundary, and is parsed, aggregated, header-rewritten, ICRC-recomputed, and re-injected—all in software. With a per-packet payload of only 1,024 bytes (Soft-RoCE active MTU), a 1 GB transfer generates over one million packets per connection, and the switch must process each one individually. This per-packet overhead establishes a hard throughput ceiling that cannot be overcome by increasing the message size. Mode 1 bypasses this bottleneck entirely by operating at the message level. The switch establishes native RDMA QP connections and receives complete 64 KB messages via the kernel RDMA stack, reducing the number of switch-side operations by 64× compared with Mode 2/3. Aggregation is performed on whole messages rather than individual packets, and the RDMA transport layer handles segmentation, reassembly, and reliability transparently. As a result, Mode 1 scales to 427 Mbps (AllReduce) and 512 Mbps (Reduce) at 1 GB, approaching the effective Soft-RoCE link capacity. MPI achieves high throughput for large messages through a different mechanism: algorithmic pipelining. Rather than re27

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. Table 20: [Emulation] Reduce Algorithm Throughput (Mbps) Msg. Size (B) EPIC-I EPIC-II EPIC-III MPI MPI-ICRC

4K 4.49 19.02 18.65 9.70 9.49

16K 25.29 66.77 49.05 37.94 34.39

64K 88.78 103.23 82.01 14.90 19.74

256K 108.50 112.01 105.59 30.29 28.65

1M 210.33 98.91 105.75 51.77 80.84

4M 449.22 98.82 107.95 184.31 148.66

16M 511.03 96.11 110.28 328.64 253.76

64M 504.83 97.17 112.99 311.63 282.72

256M 511.96 95.25 107.59 343.51 306.78

1G 506.10 93.96 110.28 454.53 326.42

Table 21: [Emulation] ReduceScatter Algorithm Throughput (Mbps) Msg. Size (B) EPIC-I EPIC-II EPIC-III MPI MPI-ICRC

4K 3.27 6.55 6.48 1.71 1.70

16K 9.69 21.84 24.63 6.83 8.69

64K 19.64 65.53 57.72 5.28 5.28

256K 62.09 104.85 98.65 17.37 19.83

1M 71.67 104.85 95.06 73.54 78.24

4M 72.54 101.68 116.64 181.02 127.33

16M 70.88 96.55 120.05 345.97 163.14

64M 70.66 95.69 121.37 436.25 206.49

256M 70.10 95.35 118.08 464.15 243.26

1G 68.47 70.20 120.32 601.30 262.97

Table 22: [Emulation] Broadcast Algorithm Throughput (Mbps) Msg. Size (B) EPIC-I EPIC-II EPIC-III MPI MPI-ICRC

4K 35.23 25.23 48.30 9.55 9.71

16K 14.17 62.27 165.46 36.47 38.63

64K 50.57 117.95 324.95 14.89 19.80

256K 75.10 115.80 381.13 58.17 44.65

1M 78.75 125.39 230.07 222.20 239.01

4M 81.84 139.29 210.80 599.20 425.39

16M 81.92 145.74 209.80 902.72 573.81

64M 90.75 132.86 188.76 1212.81 583.87

256M 78.53 142.72 188.09 832.36 661.86

1G 76.33 138.55 223.44 984.33 800.26

Table 23: [Emulation] AllGather Algorithm Throughput (Mbps) Msg. Size (B) EPIC-I EPIC-II EPIC-III MPI MPI-ICRC

4K 32.66 6.35 0.01 6.83 6.83

16K 119.70 24.40 0.04 27.32 28.15

64K 92.34 75.88 0.18 33.57 34.35

256K 181.59 147.62 0.71 107.00 122.71

1M 281.61 200.50 2.84 194.20 174.96

Table 24: [Emulation] Barrier Throughput Solution EPIC-I EPIC-II EPIC-III MPI MPI-ICRC

Throughput (requests/second) 1331.25 590.00 2502.15 37.81 37.88

lying on in-network aggregation, MPI decomposes collective operations into multiple point-to-point transfers arranged in optimized topologies (ring, recursive doubling, or binomial tree). These algorithms overlap communication and computation across pipeline stages, allowing multiple segments to be in flight simultaneously. For Broadcast, MPI’s binomial-tree pipeline achieves over 1 Gbps by saturating all links in parallel, far exceeding the single-switch fan-out capacity of the INC modes. However, this pipelining advantage diminishes for small messages (≤64 KB), where the 𝑂 (log 𝑁 ) startup latency of multi-hop algorithms dominates—precisely the regime where single-hop in-network aggregation excels.

4M 302.07 216.69 10.87 384.41 263.41

16M 316.87 265.47 37.18 429.84 271.78

64M 322.67 276.48 95.66 585.25 351.22

256M 308.44 272.88 146.18 566.77 391.54

1G 313.16 262.54 169.30 826.64 419.66

EPIC

PACKET-LEVEL SIMULATION

K.1

Implementation

K.2

Experiment Setting

K.3

Collective Performance

mechanism’s impact, we disable congestion control (i.e., the sending window does not shrink upon packet loss), focusing purely on how each design handles retransmissions. As shown in Table 31, EPIC-III consistently outperforms EPIC-II across all loss rates. At 5% loss rate, EPIC-III achieves 84.9 Gbps versus EPIC-II’s 79.5 Gbps. The gap widens significantly at higher loss rates: at 15% loss, EPIC-III maintains 77.7 Gbps while EPIC-II drops to only 39.0 Gbps. Table 32 shows similar trends when varying the number of lossy links: with all 8 links experiencing 5% loss, EPIC-III achieves 72.3 Gbps versus EPIC-II’s 52.5 Gbps. The performance gap stems from different retransmission mechanisms. In EPIC-II, retransmissions are initiated by ranks upon timeout, requiring a full round-trip delay before recovery. In EPIC-III, the switch actively detects missing packets and sends NAKs to trigger immediate retransmission. Furthermore, EPIC-III’s decoupled aggregation-broadcast design prevents completed aggregations from being blocked by ongoing retransmissions, reducing head-of-line blocking effects.

We implement a packet-level simulator based on ns-3 (version 3.43) to evaluate the performance of EPIC-II, EPIC-III, and Ring across six collective communication primitives: AllReduce, Reduce, ReduceScatter, Broadcast, AllGather, and Barrier. The code is publicly available.11 For EPIC-II and EPIC-III, we use a star topology with 8 ranks connected to a single switch. For Ring, 8 ranks form a ring topology. Each link has 100 Gbps bandwidth and 1 µs latency. We evaluate message sizes ranging from 4KB to 1GB. We vary packet loss rate from 0% to 15% on a single link, and the number of lossy links from 0 to 8 with fixed 5% loss rate per link. The performance difference between EPIC and Ring stems from two factors: theoretical bandwidth efficiency at large messages and hop count at small messages. Let 𝑁 denote message size, 𝐵 link bandwidth, and 𝐾 the number of ranks. Since throughput is inversely proportional to completion time, we analyze performance through theoretical completion time. Large messages: theoretical completion time matters. At large message sizes where bandwidth dominates, performance is determined by the theoretical completion time. For AllReduce, EPIC completes in 𝑁 /𝐵 while Ring requires 2(𝐾 − 1)𝑁 /(𝐾𝐵), resulting in EPIC achieving 1.8× higher throughput (89.9 vs 48.8 Gbps at 1GB). For other collectives (Reduce, Broadcast, ReduceScatter, AllGather), the theoretical gap is smaller or absent—Ring’s (𝐾 − 1)𝑁 /(𝐾𝐵) versus EPIC’s 𝑁 /𝐵 for scatter/gather operations, and identical 𝑁 /𝐵 for Reduce and Broadcast. Consequently, all approaches converge to near line-rate ( 91 Gbps) at 1GB for these collectives. Small messages: hop count matters. At small message sizes where latency dominates, EPIC’s single-hop communication through the switch consistently outperforms Ring’s 𝑂 (𝐾) hops. This advantage is most pronounced at 4KB messages, where EPIC achieves 6–10× higher throughput than Ring across all collectives. The gap narrows at medium sizes as pipelining effects begin to benefit Ring. EPIC-III vs EPIC-II. EPIC-III achieves marginally higher throughput than EPIC-II in AllReduce due to its decoupled aggregation-broadcast design, which enables better pipelining. For Barrier, EPIC-III reaches 3.33×105 req/s versus EPICII’s 2×105 req/s.

K.4

K.5

Large Model Training Performance

We run SimAI to simulation large model training, with EPICns3 as the backend. Model settings are shown in Table 33. Table 34 shows the model training iteration time. EPIC-II accelerates GPT-3-175B, GPT-3-13B, Llama-65B, and Llama-7B by 2.5%, 17.7%, 12.4%, and 10.2%, respectively. In most cases, the performance gain is significant (up to 17.7%); even in the case where the gain is 2.5%, the absolute reduction of JCT is still considerable considering the scale.

Loss Tolerance

We evaluate EPIC-II and EPIC-III under packet loss by measuring AllReduce throughput. To isolate the retransmission 11 https://anonymous.4open.science/r/EPIC-PktSim-8532

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. Table 25: [Packet Simulation] AllReduce Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II EPIC-III Ring

4K 6.10 10.17 1.02

16K 20.35 30.52 3.81

64K 48.83 61.04 12.21

256K 75.12 81.38 27.90

1M 84.92 88.78 41.12

4M 87.54 89.03 46.64

64M 88.25 89.90 48.66

256M 88.28 89.92 48.76

1G 88.30 89.92 48.79

Table 26: [Packet Simulation] Reduce Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II EPIC-III Ring

4K 10.17 10.17 1.80

16K 30.52 30.52 5.09

64K 61.04 61.04 9.39

256K 81.38 81.38 19.53

1M 88.78 88.78 47.64

4M 90.84 90.84 74.40

64M 91.46 91.46 90.19

256M 91.49 91.49 91.16

1G 91.50 91.50 91.41

Table 27: [Packet Simulation] ReduceScatter Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II EPIC-III Ring

4K 10.17 10.17 1.91

16K 30.52 30.52 7.12

64K 61.04 61.04 23.74

256K 81.38 81.38 53.41

1M 88.78 88.78 77.68

4M 90.84 90.84 87.64

64M 91.46 91.46 91.24

256M 91.49 91.49 91.43

1G 91.50 91.50 91.48

Table 28: [Pakcet Simulation] Broadcast Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II EPIC-III Ring

4K 10.17 10.17 2.77

16K 30.52 30.52 6.78

64K 61.04 61.04 10.61

256K 81.38 81.38 20.78

1M 88.78 88.78 49.45

4M 90.84 90.84 75.48

64M 91.46 91.46 90.29

256M 91.49 91.49 91.19

1G 91.50 91.50 91.41

Table 29: [Pakcet Simulation] AllGather Algorithm Throughput (Gbps) Msg. Size (B) EPIC-II EPIC-III Ring

4K 6.98 7.18 13.35

16K 22.71 23.25 37.15

64K 52.08 52.79 67.02

256K 76.59 76.97 83.88

1M 87.29 87.41 89.50

4M 90.42 90.45 90.99

64M 91.43 91.43 91.47

256M 91.48 91.48 91.49

1G 91.49 91.49 91.50

Table 30: [Pakcet Simulation] Barrier Throughput Solution EPIC-II EPIC-III ToRank0

Throughput (requests/second) 2E5 3.33E5 1.42E5

Table 31: [Packet Simulation] Throughput vs Loss Rate Loss(%) EPIC-II EPIC-III

0 87.54 89.03

0.1 86.33 88.53

0.5 86.81 88.53

1 86.09 88.28

2 84.46 87.54

5 79.52 84.92

8 75.48 82.67

10 73.01 81.17

12 47.17 79.72

15 39.01 77.74

Table 32: [Packet Simulation] Throughput (Gbps) vs Lossy Links (5% loss) Links EPIC-II EPIC-III

0 87.54 89.03

1 79.52 84.92

2 74.05 82.02

3 70.07 79.31

4 64.97 77.54

5 63.78 76.78

6 59.07 75.67

7 55.31 74.94

8 52.52 72.34

Table 33: [SimAI/NS3 Simulation] Large Model Settings Settings

GPU Flops

num layers

hidden size

num parameters

GPT-3 175B GPT-3 13B Llama-65B Llama-7B

125e12 312T 312T 312T

96 40 80 32

12288 5120 8192 4096

175e9 13B 65B 6.7B 30

num tokens per_seq 2048 2048 4096 4096

batch size

dtype size

TP, DP, PP

1536 128 128 128

16 2bytes 2bytes 2bytes

4,32,8 8,16,1 8,16,1 8,16,1

EPIC Table 34: [SimAI/NS3 Simulation] Model Training Iteration Time Model EPIC-II EPIC-III Ring

GPT-3-175B 32.35s 32.37s 33.19s

GPT-3-13B 1.154s 1.159s 1.403s

Llama3-65B 8.831s 8.856s 10.08s

Llama3-7B 1.132s 1.134s 1.261s

Table 35: [Packet Simulation, Mode-III] AllReduce Algorithm Goodput (Gbps), with/without switch replying CNP Msg. Size (B) Switch Not Replying CNP Switch Replying CNP

1K 13.96 13.96

16K 38.66 38.66

16K 70.37 70.37

256K 88.09 88.09

1M 94.38 94.38

4M 95.99 44.60

16M 95.87 43.32

64M 90.45 20.25

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

L L.1

FLOW-LEVEL SIMULATION

current resource availability. We model a three-tier fat-tree topology consisting of leaf, spine, and core switches. Each switch has 32 ports, and each link has a bandwidth capacity of 100 Gbps. Communication groups are mapped onto aggregation trees spanning the fattree topology. Switches are associated with a limited amount of INC-related resources, abstracted differently by each INC policy. We consider two server-side configurations depending on the network fabric. In the pure fat-tree setting, each GPU is modeled as an independent endpoint connected to the network, corresponding to a server equipped with a single GPU and a dedicated network interface (or equivalently, a multi-GPU server with one network interface per GPU). In the fat-tree with ScaleUp setting, multiple GPUs are colocated within the same server. Intra-server communication among these GPUs is handled by the ScaleUp fabric and does not consume fat-tree network bandwidth, while inter-server communication is carried over the fat-tree network. We evaluate the simulator under four experimental settings, formed by the Cartesian product of workload type and network fabric. Specifically, we consider single-tenant training and multi-tenant training, and for each workload type we test on both a pure fat-tree fabric and a fat-tree with ScaleUp fabric. The single-tenant setting models a training job using 128 GPUs, while the multi-tenant setting models concurrent jobs on a 2048-GPU cluster.

Implementation

Simulator. The implementation of the flow-level simulator used in this section is publicly available in an anonymized repository. 12 The simulator is implemented in C++ based on OMNeT++ [56], and consists of approximately 3.9K lines of code. We implement a custom flow-level simulator based on OMNeT++, designed to model single-tenant and multi-tenant large-scale distributed training workloads over a fat-tree interconnect, with explicit support for different in-network compute (INC) resource allocation policies. The simulator operates at the granularity of communication groups (i.e., collective operations), rather than packets, which enables scalable evaluation of job-level performance metrics such as job completion time (JCT) under realistic cluster scales. The simulator explicitly models both rank behaviors and network behaviors. In each simulation step, ranks alternate between computation and communication phases, while the network allocates bandwidth and INC resources subject to link capacity and switch-level constraints. During communication, the network applies a waterfilling-based algorithm to estimate per-group throughput and advances data transmission accordingly. Each job consists of multiple iterations, and each iteration triggers a fixed set of communication groups. INC resource allocation decisions are made independently for each communication group based on the selected execution mode and

L.2

Experimental Settings

Baselines. We implement four execution policies and settings corresponding to different INC strategies. In Setting 0 (Ring), all communication groups execute using conventional collective communication without innetwork aggregation. Communication volumes are not reduced, and all traffic traverses the network without consuming INC resources. In Setting 1 (EDT), we implement an edge-disjoint tree based INC policy. A communication group is granted INC if and only if none of the links on its aggregation tree are currently used by another INC-enabled group. Link usage is tracked explicitly. If any link conflict exists, the group falls back to non-INC execution. INC allocation is therefore enforced at link granularity, ensuring strict edge disjointness among concurrent INC trees. In Setting 2 (Spatial Mux), each switch maintains a fixed number of INC resource slots. A communication group is granted INC if all switches along its aggregation tree have at least one available slot. Once allocated, INC resources are reserved for the entire lifetime of the job, and are not released until the job completes. This models static partitioning of switch INC resources across concurrent jobs. In Setting 3 (Temporal Mux), INC allocation follows the

12 https://anonymous.4open.science/r/inc-flow-sim-77D5/

EPIC same admission condition as spatial multiplexing, but resources are allocated and released dynamically at the granularity of individual communication groups. After a communication group completes, its occupied INC resources are immediately released and may be reused by other groups. This enables time-sharing of INC resources across jobs and improves overall utilization under multi-tenant contention. For each communication group, the simulator determines whether INC is enabled according to the selected mode and current resource availability. If INC is granted, the effective communication volume is reduced accordingly; otherwise, the group executes in non-INC mode. The simulator advances until all communication groups and computation phases of a job complete, at which point the job completion time is recorded. Topology. We adopt a three-tier fat-tree topology consisting of leaf, spine, and core switches to model communication in large-scale distributed training systems. In the single-tenant setting, the topology is configured to support a single 128-GPU training job. Each leaf switch connects to 8 servers. A pod contains 4 leaf switches and 4 spine switches with full leaf–spine connectivity, and each spine switch connects to 4 core switches. Under this configuration, each pod interconnects 4 × 8 = 32 GPUs, and the system consists of 4 pods in total, providing exactly 128 GPUs. This setting exposes both intra-pod and inter-pod communication paths for a large single-tenant workload. In the multi-tenant setting, we scale up the topology to support concurrent jobs. Each leaf switch connects to 16 servers, and each pod consists of 16 leaf switches and 16 spine switches with full leaf–spine connectivity, while each spine switch connects to 8 core switches. As a result, each pod interconnects 16×16 = 256 GPUs, and the overall system comprises 8 pods, forming a shared network fabric for multitenant training with increased contention at the spine and core layers. For scale-up modeling, we bind 8 GPUs to a single server and assume that GPUs within the same server communicate via high-bandwidth, low-latency links such as NVLink. This intra-server communication does not consume fat-tree network bandwidth, whereas inter-server communication traverses the fat-tree fabric, allowing us to distinguish scaleup and scale-out communication in our simulations. Workloads. We evaluate our design using three workload traces that differ in how job compositions and network pressures are constructed. Trace1 is a manually configured synthetic workload that serves as a controlled baseline. In this trace, we explicitly define the job size distribution rather than deriving it from production data. We consider five representative job sizes, namely 8, 16, 32, 64, and 128 GPUs, and assign them fixed proportions of 30%, 30%, 25%, 10%, and 5% of the total jobs,

respectively. By construction, Trace1 provides a stable and interpretable workload composition, allowing us to isolate the impact of network configurations and INC strategies without interference from trace-specific artifacts. Trace2 is derived from the Alibaba Lingjun dataset [6], which provides CSV summaries of distributed training jobs collected from a large-scale production GPU cluster. We implement a custom script to extract job sizes and their relative proportions from these summaries, and use the resulting distribution to generate synthetic workloads for simulation. Trace2 therefore reflects the job mix observed under the default production network configuration. Trace3 is constructed by keeping the same workload composition as Trace2, while modifying the network topology to increase communication pressure. Specifically, we reduce the number of core-layer switches by half, which increases the likelihood of cross-pod communication and contention in the upper layers of the fat-tree. This trace enables us to study the behavior of different INC strategies under heightened network bottlenecks while holding the workload mix constant.

L.3

Performance

We first compare the three INC-based schemes with the conventional ring-based collective communication. Across all evaluated settings, all INC schemes consistently achieve lower job completion time (JCT) than the ring baseline, demonstrating the benefit of in-network aggregation. We then analyze the impact of limited switch memory on JCT. When switch memory is scarce, the JCT performance ordering follows EDT, spatial multiplexing, and temporal multiplexing, with temporal multiplexing achieving the lowest JCT due to its more flexible allocation of switch resources. As switch memory becomes sufficient, the JCT gap between spatial and temporal multiplexing diminishes, and both schemes achieve comparable performance, while EDT remains inferior due to its stricter allocation constraints. In the multi-tenant setting, temporal multiplexing may lead to slightly higher JCT for some individual jobs, as INC resources can be released during computation phases, whereas spatial multiplexing holds resources for the entire job. However, this dynamic allocation significantly improves tail JCT and results in the lowest average JCT overall by utilizing switch memory more efficiently.

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. Table 36: [Flow Simulation] JCT of GPT-3-175B on 128GPU Fat-tree

Table 40: [Flow Simulation] JCT of GPT-3-175B on 128GPU Fat-tree with Scaleup

Switch SRAM (Unit) 4 8 16 32 Ring 253 253 253 253 EDT 179 179 179 179 Spatial Mux 179 158 137 137 Temporal Mux 158 137 137 137 Table 37: [Flow Simulation] JCT of GPT-3-13B on 128GPU Fat-tree

Switch SRAM (Unit) 4 8 16 32 Ring 106 106 106 106 EDT 84.6 84.6 84.6 84.6 Spatial Mux 84.6 63.6 63.6 63.6 Temporal Mux 84.6 63.6 63.6 63.6 Table 41: [Flow Simulation] JCT of GPT-3-13B on 128GPU Fat-tree with Scaleup

Switch SRAM (Unit) 4 8 16 32 Ring 33.55 33.55 33.55 33.55 EDT 20.66 20.66 20.66 20.66 Spatial Mux 20.66 19.10 17.54 17.54 Temporal Mux 19.10 17.54 17.54 17.54 Table 38: [Flow Simulation] JCT of Llama-65B on 128GPU Fat-tree

Switch SRAM (Unit) 4 8 16 32 Ring 7.94 7.94 7.94 7.94 EDT 6.38 6.38 6.38 6.38 Spatial Mux 6.38 4.82 4.82 4.82 Temporal Mux 6.38 4.82 4.82 4.82 Table 42: [Flow Simulation] JCT of Llama-65B on 128GPU Fat-tree with Scaleup

Switch SRAM (Unit) 4 8 16 32 Ring 211 211 211 211 EDT 129 129 129 129 Spatial Mux 129 121 113 113 Temporal Mux 121 113 113 113 Table 39: [Flow Simulation] JCT of Llama-7B on 128GPU Fat-tree

Switch SRAM (Unit) 4 8 16 32 Ring 47.6 47.6 47.6 47.6 EDT 39.8 39.8 39.8 39.8 Spatial Mux 110.0 39.8 32.0 32.0 Temporal Mux 110.0 39.8 32.0 32.0 Table 43: [Flow Simulation] JCT of Llama-7B on 128GPU Fat-tree with Scaleup

Switch SRAM (Unit) Ring EDT Spatial Mux Temporal Mux

4 37.78 21.29 21.29 20.49

8 37.78 21.29 20.49 19.68

16 37.78 21.29 19.68 19.68

Switch SRAM (Unit) Ring EDT Spatial Mux Temporal Mux

32 37.78 21.29 19.68 19.68

4 8 16 5.01 5.01 5.01 4.20 4.20 4.20 4.20 3.40 3.40 4.20 3.40 3.40

32 5.01 4.20 3.40 3.40

Table 44: [Flow Simulation] Average JCT of Multitenant Jobs on 2048-GPU Fat-tree Workloads Trace1 Ring 6190 EDT 4108 Spatial Mux 3920 Temporal Mux 3750

Trace2 6245 4635 4722 4621

Trace3 6267 4666 4791 4618

Table 45: [Flow Simulation] 8-GPU Jobs’ JCT in Trace2 on 2048-GPU Fat-tree with and without scaleup Workload without scaleup Ring 72.0 EDT 56.0 Spatial Mux 56.2 Temporal Mux 56.0

with scaleup 41.9 41.1 40.9 40.9

EPIC

1 0.8 0.6 0.4 0.2 0

EDT

Spatial Mux

Ring

Temporal Mux

(a) All Jobs

CDF

Ring

3,000

4,000

5,000

6,000

7,000

8,000

9,000

1 0.8 0.6 0.4 0.2 0

EDT

4,500

5,000

Job Completion Time (s)

CDF

CDF 3,000

4,000

5,000

6,000

7,000

8,000

Ring

4,500

5,000

EDT

Spatial Mux

5,000

5,500

6,000

Ring

Temporal Mux

6,500

7,000

1 0.8 0.6 0.4 0.2 0

EDT

CDF

0.95 0.9 5,000

5,500

6,000

6,500

7,000

Spatial Mux

Temporal Mux

Job Completion Time (s)

(b) Tail 15% Jobs

4,500

6,500

(a) All Jobs

Job Completion Time (s) 1

6,000

Figure 22: [Flow Simulation] JCT CDF of Trace 3 on 2048-GPU Fat-tree.

(a) All Jobs

4,500

5,500

Job Completion Time (s)

CDF

7,000

0.9 0.8

9,000

Figure 20: [Flow Simulation] JCT CDF of Trace 1 on 2048-GPU Fat-tree.

CDF

6,500

0.85

Job Completion Time (s)

0.85

6,000

0.95

0.6

1 0.8 0.6 0.4 0.2 0

5,500

(b) Tail 20% Jobs

0.8

0.4

Temporal Mux

Job Completion Time (s)

(b) Tail 60% JCT

Spatial Mux

(a) All Jobs

7,000

1 0.98 0.96 0.94 0.92 0.9

(b) Tail 10% Jobs

Job Completion Time (s)

Figure 23: [Flow Simulation] 8-GPU’s JCT CDF of Trace 2 on 2048-GPU with scaleup.

Figure 21: [Flow Simulation] JCT CDF of Trace 2 on 2048-GPU Fat-tree.

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al.

CDF

Ring

1 0.8 0.6 0.4 0.2 0

EDT

Spatial Mux

Temporal Mux

(a) All Jobs

CDF

Job Completion Time (s) 1 0.98 0.96 0.94 0.92 0.9

(b) Tail 10% Jobs

Job Completion Time (s)

Figure 24: [Flow Simulation] 8-GPU’s JCT CDF of Trace 2 on 2048-GPU with INC-improved scaleup.

EPIC

FPGA IMPLEMENTATION

M.1

ards arising from varying data path lengths between initial and retransmitted packets without stalling the pipeline. Experimental evaluations conducted at a 250MHz clock frequency demonstrate that the system achieves line-rate 100Gbps throughput across various packet types, with endto-end latencies for 1 KB payloads ranging from 96 ns for direct forwarding to 244 ns for FP32 upstream aggregation. Detailed resource utilization analysis shows that for a 1 MB buffer configuration, the Int32 implementation consumes 867,479 LUTs (50.20%) and 235 BRAMs (8.74%), while the FP32 configuration increases the demand to 987,308 LUTs (57.14%) and 343,724 Flip-Flops (9.95%), reflecting the increased complexity of floating-point operations. Table 47 and Table 48 shows the overall and IncEngine’s resource consumption.

Implementation from a Cluster Vendor

As EPIC Mode-II presents the lowest barrier to implementation, we collaborated with a cluster vendor to facilitate its FPGA-based realization and evaluation. The implementation was developed using Verilog on the Intel Agilex™ 7 FPGA M-Series 039 (R47A) platform. To assess the performance of the proposed IncEngine, we conducted cycle-accurate RTL simulations focused on throughput, per-packet latency, and resource utilization. Experimental results indicate that for an 8-rank configuration, the total latency from the arrival of the final packet to the commencement of aggregated result output is approximately 230 ns. This total duration encompasses the entire processing pipeline of the parser and deparser, with the core functional latency of the aggregation engine specifically accounting for 53 ns. The hardware resource overhead is primarily influenced by the buffer capacity and supported data types. The current design supports the FP32 data type with a default buffer specification of 8 MB, which can be configured for either 16 ranks × 2 groups or 32 ranks × 1 group. Under the FP32 configuration, as the buffer capacity scales from 2 MB to 16 MB, the Adaptive Logic Module (ALM) consumption ranges from 265,969 to 306,642, while memory bit utilization increases from approximately 54.5 Mb to 322.3 Mb. Across all buffer sizes, the Digital Signal Processor (DSP) utilization remains constant at 256 units (Table 46).

M.2

Implementation from an Institute

The EPIC specification was delivered to a research institute for comprehensive FPGA implementation and performance evaluation. The system is implemented on the Xilinx Virtex UltraScale+ VU13P platform, utilizing approximately 6,500 lines of Verilog code for the entire logic, with the IncEngine itself comprising 5,500 lines. To ensure high-throughput processing, two critical FPGAspecific optimizations are integrated: first, a block-based SIMD parallel computing method is employed to divide large bit-width payloads into independent sub-vectors, thereby overcoming the bottlenecks of element-wise serial processing; second, a speculation-based consistency control mechanism is implemented to resolve read-after-write (RAW) hazTable 46: [FPGA, Cluster Vendor] Resource Consumption with FP32 Buffer Size (MB) 2 4 8 16

ALM 265969 271160 287376 306642

Memory Bits 54484032 92753792 169571264 322347776

DSP 256 256 256 256 37

EVALUATION BY CHIP VENDOR

The EPIC specification was delivered to a prominent chip vendor for a comprehensive hardware implementation assessment. The core logic of the engine was developed using Verilog, comprising 66,377 lines of code to define the Register-Transfer Level (RTL) hardware description, which was subsequently verified through cycle-accurate SystemVerilog simulations. Architecturally, the design prioritizes high-performance collective communications, supporting critical operations such as Broadcast, AllReduce, and ReduceScatter with internal FP32 precision to ensure numerical consistency. The design also support other data types such as FP16, BF16, INT32, but internally, all data types except INT32 are converted to FP32 for computation and converted back after computation. The design chooses the reproducible computation (buffering all data then computing). Experimental results from the RTL simulations indicate that each packet incurs a processing latency of 50 ns, with a single engine providing a throughput of 3.2 Tbps. When integrated into the switching chip, a cluster of eight IncEngine units achieves an aggregate processing capacity of 25.6 Tbps. Each IncEngine supports 64 communication groups, each group with 16 members. Regarding hardware overhead, the design was synthesized using a 28nm process technology. A single IncEngine instance, configured with 512 FP32 ALUs, 512 UINT ALUs and a 1 MB payload buffer, occupies an area of 4.89 mm2 . Consequently, the total area overhead for the eight integrated engines amounts to 39.12 mm2 , demonstrating a scalable and efficient footprint for high-bandwidth in-network computing. The chip specification include: • Collective Types: Broadcast, AllGather, Reduce, ReduceScatter, AllReduce. • Operation Types: SUM, MIN, MAX. • Data Types: FP32, FP16, BF16, INT32.

Y. Yuan, J. Nie, T. Bai, R. Zhou, S. Cao, X. Fan, et al. Table 47: [FPGA, Institute A] Overall Resource Consumption Data Type Int32 Int32 Int32 Int32 FP32 FP32 FP32 FP32

Buffer Size 1MB 512KB 256KB 128KB 1MB 512KB 256KB 128KB

LUTs (1,728,000) 867479(50.20%) 863426(49.97%) 863433(49.97%) 468170(27.09%) 987308(57.14%) 983245(56.90%) 983250(56.90%) 588010(34.03%)

LUTRAM (791,040) 637900(80.64%) 637900(80.64%) 637900(80.64%) 319116(40.34%) 638924(80.77%) 638924(80.77%) 638924(80.77%) 320140(40.47%)

FF (3,456,000) 169382(4.90%) 165024(4.78%) 165026(4.78%) 158769(4.69%) 343724(9.95%) 339366(9.82%) 339369(9.82%) 333111(9.64%)

BRAM (2,688) 235(8.74%) 122(4.54%) 122(4.54%) 122(4.54%) 235(8.74%) 122(4.54%) 122(4.54%) 122(4.54%)

Table 48: [FPGA, Institute A] IncEngine Resource Consumption Data Type Int32 Int32 Int32 Int32 FP32 FP32 FP32 FP32

Buffer Size 1MB 512KB 256KB 128KB 1MB 512KB 256KB 128KB

LUTs (1,728,000) 866023(49.48%) 850970(49.25%) 850973(49.25%) 462529(26.77%) 974852(56.42%) 970789(56.18%) 970804(56.18%) 582370(33.70%)

LUTRAM (791,040) 637568(80.6%) 637568(80.6%) 637568(80.6%) 318784(40.3%) 638592(80.73%) 638592(80.73%) 638592(80.73%) 319808(40.43%)

• Capacity: 25.6 Tbps

FF (3,456,000) 144989(4.20%) 140631(4.07%) 140629(4.07%) 140583(4.07%) 319331(9.24%) 314973(9.11%) 314971(9.11%) 314971(9.11%)

BRAM (2,688) 235(8.74%) 122(4.54%) 122(4.54%) 122(4.54%) 235(8.74%) 122(4.54%) 122(4.54%) 122(4.54%)

Record · ID 200444 · SHA-256 bedf17b904dc2d87
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.