Rondo: Controlling Reordering in High-Throughput Packet Spraying

arXiv:2611.04594v1 [cs.NI] 3 Nov 2026

Rondo: Controlling Reordering in High-Throughput Packet Spraying Xizhi Zhang1 Zhongjie Chen2 1 Tsinghua University

Yang Zhang1 Fengyuan Ren1 2 Microsoft Research

Abstract

In recent years, packet-level multipath transport, or packet spraying, has gained increasing attention [9, 13, 18, 35, 42, 53, 61, 63, 65]. These transports select a path for each packet, allowing even a small number of flows to utilize capacity across multiple paths. Unfortunately, packet spraying can introduce packet reordering, posing particular challenges for hardware transports. Gaps in the received packet sequence may indicate either packet loss or out-of-order arrival. RoCEv2, a single-path protocol that enforces strict in-order delivery, interprets any out-of-order arrival as packet loss. Even a small out-of-order degree can cause severe performance degradation [38, 42]. Recent hardware transports are designed to tolerate limited packet reordering [9, 61, 63, 65], but still face two challenges. First, excessive packet reordering makes packet loss hard to infer. There is an inherent tradeoff between loss detection and out-of-order tolerance: a higher reordering tolerance delays loss recovery, whereas a lower one makes multipath transport more susceptible to spurious retransmissions. Second, excessive packet reordering imposes substantial pressure on NIC memory. Tracking out-of-order packets requires metadata, while strict in-order delivery also requires reorder buffers for payload. Across many flows, both metadata and reorder buffers may exceed NICs’ MB-scale on-chip memory budgets [42, 46, 67]. Once the NIC exhausts its on-chip resources, it may have to drop incoming packets or apply backpressure to the sender. The resulting retransmissions and sender stalls reduce throughput and increase tail latency. However, existing multipath transports struggle to keep packet reordering low. First, these transports have to make a tradeoff between bandwidth utilization and packet reordering. For example, Falcon [63] and UET [65] use receiver backpressure to reduce the sending window when receiver resource occupancy is high. MP-RDMA [42] prunes slower paths that contribute to excessive reordering. Both strategies can leave available path capacity unused, sacrificing throughput to reduce reordering. Second, these transports control packet reordering reactively, responding only after it has occurred. Under bursty traffic, feedback delays cause reordering

Modern datacenter applications demand high network bandwidth, motivating packet-level multipath transport to exploit the aggregate capacity of multiple paths. However, packet reordering under packet spraying poses challenges for hardware transports. Excessive packet reordering makes it difficult to infer packet loss, imposes substantial pressure on NIC resources, and ultimately degrades performance. Existing transports either prioritize throughput without controlling packet reordering, or mitigate reordering by restricting transmissions, which leaves available path capacity unused. In this paper, we identify two fundamental sources of packet reordering: transient inter-path queueing differences caused by control dynamics, and persistent path-delay differences due to path heterogeneity. These observations motivate a joint design of a multipath controller and a packet scheduler to address transient and persistent delay differences, respectively. We present Rondo, a packet-level multipath transport that achieves both high throughput and low packet reordering. Its multipath controller uses ACK-based per-path capacity estimation and probing to maintain stable path loads, while its packet scheduler adjusts packet transmission order to compensate for persistent delay differences. Testbed experiments and large-scale simulations show that Rondo reduces the maximum out-of-order degree by up to 97.5% while maintaining high throughput. With constrained receiver buffers, Rondo achieves up to 8.51× higher goodput.

1

Introduction

Hardware transport, such as Remote Direct Memory Access (RDMA), has become a key enabler of high-throughput, lowlatency communication for datacenter applications. Datacenter networks usually employ multiple parallel paths [3,34]. Exploiting this path diversity for hardware transports [3,9,34,54] can benefit throughput-oriented applications, ranging from distributed storage and data analytics to large language model (LLM) training and inference [54, 57, 70]. 1

to accumulate before these mechanisms can respond. We revisit the root causes of packet reordering to identify how multipath transports can proactively limit it. We make two observations. First, existing multipath transports introduce substantial inter-path queueing differences. During synchronized flow startup, these transports often inject large bursts before assessing path conditions, creating queue imbalances when traffic is unevenly distributed. Even with a fixed set of active flows, continual per-path load adjustments cause queues to build and drain asynchronously, leading to fluctuating queueing delay differences across paths. Thus, an ideal multipath transport should both maintain shallow queues and balance load across paths.

2

Background and Motivation

2.1

Multipath in Datacenters

Modern datacenter topologies are typically provisioned with multiple paths between racks. Exploiting the path diversity can aggregate their available bandwidth, avoid congested paths, and tolerate path failures. Since RDMA is a single-path transport by design with a strict in-order delivery requirement, the most common scheme is flow-level load balancing, i.e., equal-cost multipath (ECMP). However, it has been widely studied to suffer from hash collisions and fail to balance traffic effectively [42, 54]. Flowlet-level schemes improve adaptability by allowing a flow to switch paths after an inter-packet gap [5, 55, 66]. With a sufficiently large gap, a flowlet can change paths while largely preserving packet order. However, high-throughput transfers rarely expose gaps for flowlet formation, and this scheme essentially prevents the opportunity to aggregate multiple paths’ bandwidth [24]. Recent multipath transports use a much more fine-grained approach, i.e., packet spraying, to distribute packets of the same flow across multiple paths [9, 61, 63, 65]. A key component of a packet spraying transport is the multipath controller, which consists of congestion control and path selection. The congestion control regulates sending rates at either the flow or path granularity, while the path selection distributes packets across paths based on different rules. Although packet spraying enables fine-grained load balancing and efficient utilization of available paths, it can easily cause substantial packet reordering. Unfortunately, it’s challenging for hardware transports to handle packet reordering effectively [42].

Second, even when queues are stable, persistent delay differences across heterogeneous paths can still cause packet reordering. Excluding slower paths can help but at the cost of bandwidth utilization [42]. Our key insight is that once queueing delays stabilize, packet scheduling can compensate for persistent delay differences across paths. For example, the multipath transport can send packets with higher packet sequence numbers (PSNs) earlier on higher-delay paths, followed by lower-PSN packets on lower-delay paths. Based on these observations, we present Rondo, a packetlevel multipath transport that achieves both high throughput and low packet reordering. Rondo coordinates the multipath controller and a packet scheduler to jointly achieve these goals. The multipath controller combines ACK-rate-based congestion control with deficit-based path selection to stabilize per-path loads and sustain high utilization. It also uses a low-priority capacity probing mechanism to accelerate flow startup with minimal impact on queueing. With queueing delays stabilized by the multipath controller, the packet scheduler adjusts transmission order to compensate for persistent delay differences across paths, eliminating packet reordering.

2.2

Reordering Issues

In this section, we examine the costs of packet reordering. We introduce two metrics to quantify the impact: Out-of-Order Degree (OOD), the difference between the largest PSN received and the next expected PSN; Reorder Buffer Occupancy (RBO), the number of out-of-order packets buffered while the receiver waits for missing earlier packets. OOD is always no smaller than RBO, and the two metrics usually follow the same trend in practice. Ambiguity between packet loss and reordering. When the received packet sequence contains a gap, the transport cannot immediately determine whether the missing packet was lost or merely reordered. Even in lossless networks with PFC enabled, packet loss remains unavoidable in production due to link failures and faulty devices [20, 54]. RoCEv2 treats all gaps as packet losses and triggers inefficient Go-Back-N retransmissions [38, 42]. Selective Repeat (SR), adopted by the latest ConnectX NICs, still generates a NAK for each out-of-order packet, degrading performance significantly under heavy packet reordering [63]. Falcon and UET instead introduce RACK-TLP and OOD-based Early Recovery (ER)

In summary, we make the following contributions. • Analyzing the impact of reordering on multipath transport, and identifying two sources of packet reordering: inter-path queueing differences and persistent path-delay differences that require different remedies. • Designing Rondo, a packet-level multipath transport that achieves high throughput with low and controllable packet reordering. • Implementing Rondo in DPDK and evaluating it through extensive experiments. In typical datacenter workloads, Rondo eliminates out-of-order arrivals under incast and reduces maximum out-of-order degree by 28.6%–97.5% under the other traffic patterns, while achieving JCT comparable to or lower than that of state-of-the-art multipath transports. With a reorder buffer of approximately one BDP, Rondo achieves 1.15×–3.60× the goodput of other multipath transports, increasing to 1.38×–8.51× as the buffer becomes more scarce. 2

80

40

20 0

0 8

32

128

Retransmission threshold (packets)

(a)

Throughput (Gbps)

120

Retransmission delay (µs)

Spurious retransmit (%)

512 KB

60 40

ery semantics [42, 63]. Even emerging AI/ML applications such as DeepEP [17] require ordering guarantees across consecutive operations and therefore cannot run over transports that do not support in-order delivery [39,44]. In addition, DDP does not fully eliminate the costs of packet reordering: packet loss remains difficult to distinguish from reordering, and tracking out-of-order packets still consumes NIC resources even without reorder buffers. For example, IRN [46], a single-path, DDP-based transport, reserves five BDP-sized bitmaps for each QP. This per-QP metadata scales linearly with the number of QPs and can therefore consume substantial NIC memory [67]. We confirmed this limitation on ConnectX-7 NICs, which also support DDP; Appendix A provides details. Summary. Excessive packet reordering degrades transport performance by obscuring packet loss and increasing receiverside resource pressure. Controlling reordering can therefore improve transport efficiency, reduce buffer occupancy, and even benefit DDP-based transports that do not require a reordering buffer.

Reorder buffer size

Spurious retransmit Retransmission delay

1 MB

2 MB

40

30

20 0

128

256

512

Out-of-order degree (packets)

(b)

Figure 1: (a) There is an inherent tradeoff between loss detection and out-of-order tolerance. (b) Throughput drops sharply as OOD approaches the reorder-buffer limit. to distinguish packet loss from reordering [63, 65]. However, they still face a fundamental tradeoff: recovering losses promptly risks spurious retransmissions when packet reordering is severe. To demonstrate this trade-off, we implement ER in a DPDK-based multipath transport and configure the switch to induce an OOD of 64 packets and a random packet drop rate of 0.1%. ER retransmits an unacknowledged packet when the gap between its PSN and the highest acknowledged PSN exceeds a fixed threshold. Figure 1a shows that suppressing spurious retransmissions under this reordering requires higher thresholds, at the cost of delayed loss recovery. Reordering poses memory pressure. Many datacenter applications rely on the strict in-order delivery semantics provided by the transport layer. To absorb out-of-order packets, the receiver must buffer them until the missing earlier packets arrive. However, commodity NICs typically provide only several megabytes of on-chip memory [67], making it impractical to provision a dedicated reordering buffer for every flow. Falcon, for example, chooses to use a shared one-BDP reorder buffer across all flows [63]. When RBO exhausts this capacity, the receiver must drop packets, leading to performance degradation, head-of-line blocking or even deadlock [29]. Falcon mitigates such buffer pressure by adjusting sending windows based on receiver buffer occupancy. We implement a similar mechanism in DPDK and measure it on our ConnectX-7 NICs. The switch is configured to create a delay difference across paths, corresponding to a target OOD under line-rate transmission. As shown in Figure 1b, throughput begins to degrade once OOD becomes comparable to the reorder-buffer capacity, and decreases further as OOD grows. A large OOD increases receiver buffer pressure, causing the sender to reduce its window and preventing it from sustaining line rate. Direct Data Placement (DDP) is not a panacea. Some transports [9, 61, 65, 67] use DDP to write packets directly to their final memory locations regardless of arrival order. However, this technology is only applicable to some AI/ML workloads, whereas many datacenter applications rely on in-order deliv-

3

Understanding the Sources of Packet Reordering

In this section, we first analyze how inter-path delay differences translate into packet reordering, and then examine how these delay differences arise in multipath transport.

3.1

What Determines Out-of-order Degree?

We denote the forward delay of path p at time t as D p (t). Consider packet i transmitted at time ti on path pi . It will arrive at the receiver at time ti + D pi (ti ). We say a packet j overtakes packet i if it is transmitted later but arrives earlier: t j > ti

and t j + D p j (t j ) < ti + D pi (ti ),

(1)

The PSN gap ( j − i) is the number of packets transmitted during the overtaking window (t j − ti ), which can be approximated by the integral of the aggregate sending rate R(u) over that window: j−i ≈

Z tj

R(u) du

(2)

ti

Suppose packet i is the next expected packet at the receiver, and j is the highest-PSN packet that overtakes it. Immediately before packet i arrives, j − i is exactly the OOD. Equation (1) shows that the overtaking window is bounded by the inter-path delay difference. In high-throughput packet spraying, where inter-packet transmission gaps are small, the maximum overtaking window is approximately the maximum delay difference among all active paths. Let R̄ denote the average aggregate sending rate over this window. Equation (2) gives the following estimate of the maximum OOD:   OODmax ≈ R̄ max D p (t) − min D p (t) (3) p∈P

3

p∈P

Path 1

150

750

MP-RDMA

75

500 250

2.0

2.5

1

2

3

250

0

ࡿ૚

ࡿ૛

P4

ࡿࡺ

ToR switches

Startup

P3

ࡾ૚

ࡾ૛

ࡾ૜

ࡾ૝

Queue (KB)

P2

2.5

1

2

3

0 0

P1

2.0

750

500

500

500

250

250

250

0.1

0.2

Time (ms)

0.3

0 1.5

2.0

2.5

1

2

3

0

750

0 0.0

10

250

750

0 0.0

0.1

0.2

Time (ms)

0.3

Ideal

OOD

0 0

Falcon

Ideal

20

500

50 1.5

MP-RDMA

Path 4

750

125

500

0 1.5

Path 3

Falcon

200

750

Min queue across paths Path 2

0 0.0

200 100 0

0

1

2

3

0 0.0

0.1

0.2

0.3

20 10 0 0.0

0.1

0.1

0.2

0.2

Time (ms)

0.3

0.3

OOD (pkts)

Overall

Spine switches

Queue (KB)

Insets:

OOD (pkts)

Max queue across paths

200 100

Time (ms)

Figure 2: Multipath topology with four paral- Figure 3: Queue dynamics and NIC-level OOD under synchronized flows. NIClel paths. level OOD denotes the sum of the OOD of all flows at the NIC. path’s available capacity, overloading different paths to different degrees and thereby creating transient inter-path queue imbalance. The same problem arises when a transport rapidly increases its sending rate to utilize spare bandwidth. Reactive load adjustment across paths causes sustained oscillations. After 1.5 ms, MP-RDMA and Falcon enter a steady phase, yet their queues differ by ∼100 KB, resulting in an OOD of approximately 100 packets. The congestion signals used by these transports, ECN and delay, reflect the congestion level on each path, but do not directly indicate the load that each flow should maintain on that path. Therefore, multipath controllers continuously adjust each path’s load in response to congestion feedback. These reactive adjustments occur asynchronously across paths, causing queues to build and drain at different times and creating sustained queue oscillations, as shown in the insets of Figure 3. Ideal queue balancing effectively tames such reordering. The ideal controller perfectly balances load across paths and fully utilizes available bandwidth, exhibiting almost zero OOD. It shows that excessive packet reordering is not an inherent consequence of packet spray. However, the foreseen and global knowledge is impractical because feedback is delayed and incomplete, and control is distributed across flows. Taming reordering while keeping high throughput remains still challenging.

Appendix B provides the detailed derivation. Equation (3) shows that the maximum OOD is determined mainly by two factors: the aggregate sending rate and the inter-path delay difference. To maximize R̄ near the available network capacity while keeping the OOD under control, the inter-path delay difference should be minimized. However, existing multipath transports do not perform ideally in this regard, which, as we will show in the following two subsections, is due to two different sources of inter-path delay difference: (1) imbalanced queuing delay and (2) persistent path-delay heterogeneity.

3.2

Inter-path Queuing Delay Imbalance

We decompose the delay of path p into a persistent pathspecific delay D̄ p and a queuing delay D̃ p (t): D p (t) = D̄ p + D̃ p (t)

(4)

On homogeneous paths, D̄ p is identical across paths, so interpath delay differences arise primarily from D̃ p (t). Existing multipath transports keep each queue shallow, but do not necessarily keep queuing delays balanced across paths. We demonstrate this problem through ns-3 [48] using the topology in Figure 2. Sixteen senders simultaneously transmit to four receivers over four parallel paths bottlenecked at ToR ports P1–P4. We compare two representative multipath transports, MP-RDMA [42] and Falcon [63], with a transport using an ideal controller. The ideal controller has perfect prior knowledge of all flows and path states, and evenly divides the bottleneck bandwidth among the competing flows to determine each flow’s ideal capacity share. It then controls each flow’s per-path inflight according to this capacity share, such that the aggregate inflight on each path remains at BDP plus 5 KB. Figure 3 shows the result. Capacity-agnostic rate ramp-up causes severe transient imbalance. Within the first 0.1 ms after flow startup, MPRDMA and Falcon exhibit inter-path queue-length differences of ∼200 KB, causing OOD to peak at approximately 200 packets. At startup, paths may differ in both residual bandwidth and the number of newly arriving flows. Existing multipath controllers inject traffic aggressively before learning each

3.3

Persistent Path-delay Heterogeneity

Equal-cost physical paths may not have identical D̄ p in practice. Paths may span different physical distances or traverse heterogeneous switching equipment, while link failures can further exacerbate path asymmetry. In production datacenters, these effects can introduce persistent inter-path delay differences ranging from several to more than ten microseconds [12, 20, 40]. Unlike queuing-delay imbalance, balancing load across paths cannot eliminate reordering caused by persistent path-delay heterogeneity. We demonstrate this problem using the topology in Figure 2, with four senders, each transmitting to a different receiver. We vary the propagation delay of a link on path 4 from 2 µs to 27 µs in 5 µs increments, creating a persistent delay 4

Out-of-order degree (packets)

280

MP-RDMA Falcon Ideal Controller Theory Rondo

240 200

Adjust per-flow window for stable path state (§4.2.1)

160 120 80

Multipath Controller

Rondo in Sender

ACK-rate-based CC

Sending probe packet to ramp up safely (§4.2.2)

Capacity Probing

Select the path to transmit packet (§4.2.3)

Deficit-based Path Selection

TX Engine

Receiver

Path ‫݌‬ Packet ݊ Path ‫݌‬

Packet Scheduler

Select the packet to be transmitted (§4.3)

40 0

0

5

10

15

20

Figure 5: The architecture of Rondo.

25

Persistent Path delay difference (μs)

controller, so it preserves the per-path load established by the controller. Figure 5 shows the architecture of Rondo. The multipath controller determines when the sender can transmit and which path to use. Given a transmission selected by the controller, the packet scheduler selects the packet to transmit on that path. The receiver returns per-packet ACKs carrying the corresponding PSN and path information, and buffers out-of-order packets until they can be delivered in order.

Figure 4: The OOD under persistent delay difference. difference of up to 25 µs between path 4 and the other paths. Persistent path-delay heterogeneity causes reordering that cannot be eliminated by controllers. Figure 4 shows that OOD increases with the persistent inter-path delay difference under MP-RDMA, Falcon, and even the ideal controller. A persistent path-delay difference of 15 µs can cause a NIC-level OOD of 124–153 packets. What’s more, as the persistent path-delay difference grows and becomes the dominant source of inter-path delay difference, it enlarges the overtaking window and thus increases OOD. This trend is consistent with Equation (3), which provides a linear estimation of OOD based on the inter-path delay difference and the aggregate sending rate, as shown in the theoretical line in Figure 4. A naive way to reduce such OOD is assigning less traffic to a slower path or avoiding it entirely. However, doing so leaves the path bandwidth underutilized. The desired transport behavior is to utilize the available transmission opportunities on slower paths while mitigating the resulting OOD.

4

Rondo Design

4.1

Overview

4.2

Multipath Controller

The multipath controller in Rondo should achieve the precise and stable load control demonstrated by the ideal controller. It consists of three components. ACK-rate-based congestion control (ARCC) adjusts the per-path windows, capacity probing rapidly and safely discovers newly available bandwidth, and deficit-based path selection smoothly distributes transmissions according to the per-path windows. 4.2.1

ACK-rate-based Congestion Control

Maintaining stable per-path load requires a quantitative target for each flow’s load on that path. Existing congestion signals indicate congestion but do not provide such a target, causing queue oscillations. The ideal controller uses global knowledge to calculate the appropriate allocation of bottleneck capacity among flows, but such knowledge is unavailable in practice. ARCC instead uses per-path ACK rate to estimate the capacity share currently obtained by each flow. ACK rate as a capacity indicator. At a fully utilized bottleneck with bandwidth C, the capacity shares currently obtained by all flows traversing it sum to C. Each sender can observe its own share through its ACK rate on this path. ARCC uses this observation to construct a target per-path window

The observations in §3 lead to two design insights for controlling OOD without sacrificing throughput. First, to suppress reordering caused by inter-path queueing imbalance, the multipath controller should approach the behavior of the ideal controller in §3.2, which matches per-path load to available capacity precisely and avoids large queue oscillations. Second, persistent path-delay heterogeneity requires mechanisms beyond load control. That motivates introducing a packet scheduler to the transport, which adjusts the packet transmission order to compensate for persistent inter-path delay heterogeneity. Based on the above analysis, we propose Rondo, a multipath transport that co-designs the multipath controller and the packet scheduler. The multipath controller aims for reducing queue fluctuations, and also maintains stable path-delay conditions for the packet scheduler. By transmitting packets assigned to slower paths earlier, the packet scheduler can better align their arrival order with packets sent on faster paths. It fully utilizes the transmission opportunities provided by the

Wp∗ = r̂ p T + q p ,

(5)

where r̂ p is the measured ACK rate of this flow on path p, T is the base RTT, and q p is a small per-flow margin. Figure 6 illustrates the rationale of this target window. The first term r̂ p T constitutes most of the window according to the capacity share observed from the ACK rate. r̂ p T of all flows collectively provides just enough inflight to sustain the bottleneck capacity. The margin q p provides a small amount of additional inflight, allowing flows to adjust their capacity 5

∗(௙) =

ܹ௣

‫ݎ‬ෝ௣ (௙) ܶ (௙)

+

ECN and delay cannot quantitatively reveal the spare capacity in an underutilized link, so conservative ramp-up takes O (BDP) or O (log(BDP)) rounds [73]. Aggressive packettrain-based approaches [7] converge faster but can build large transient queues [2], while in-network telemetry requires additional switch support.

(௙)

‫ݍ‬௣

෍ ‫ݎ‬ෝ௣ (௙) = ‫ܥ‬

෍ ‫ݍ‬௣ → ߬௤ ‫ܥ‬

௙∈ி

௙∈ி

(௙)

‫ݐ‬ℎ‫݃ݑ݋ݎ‬ℎ‫߬ → ݁ݑ݁ݑݍ & ܥ = ݐݑ݌‬௤ ‫ܥ‬ Figure 6: Rationale of ARCC.

Low-priority probing. The key idea of capacity probing is to translate the spare capacity into ACK streams without filling it with data packets. Rondo uses dedicated, low-priority MTUsized probe packets that carry no application payload. The probe packets use a reserved PSN distinct from normal data packets. During probing, the sender transmits probe packets whenever data packet transmission is blocked by the per-path window. The probes fill the remaining transmission opportunities up to line rate.

shares while controlling the queue. ARCC adjusts q p using AIMD such that the sum of q p across flows approaches τqC, where τq is the target queueing delay. Because throughput is primarily sustained by the ACK-rate term, only the small q p term requires reactive adjustment and can therefore be updated conservatively. This allows ARCC to maintain high utilization with a shallow and stable queue. Algorithm. ARCC updates per-path windows every Tu . At each update on path p, ARCC computes the average ACK rate over the last interval and then applies an Exponentially Weighted Moving Average (EWMA) to obtain an ACK rate estimate r̂ p . This progress filters out short-term fluctuations in ACK arrivals. ARCC uses a method similar to Swift [33] to obtain the base RTT T , which can be directly derived from the datacenter topology. It then estimates the pipe size as C p ← r̂ p T and the flow’s excess inflight as Q p ← pktIn f l − C p , which serves as an estimate of its queued data. ARCC next performs AIMD on q p to drive the queueing delay toward the target τq . When the estimated queue is below the target, it additively increases q p up to q0 ; otherwise, it multiplicatively decreases q p according to the ratio between the target and estimated queue. Finally, ARCC computes Wp∗ according to Equation (5). If the flow’s estimated queue is large, ARCC directly sets Wp to Wp∗ to quickly drain the queue. Otherwise, it applies an EWMA to smooth the window update. Per-path window with flow-level pacing. Pacing smooths packet transmission and helps maintain a stable path load. However, maintaining an independent pacer for every path of every flow increases the state overhead in NIC hardware [56, 60]. ARCC therefore uses a flow-level pacer. After each window update, ARCC sums the per-path windows and computes the aggregate pacing rate as ∑ p Wp /T . The pacing rate bounds the aggregate sending rate of the flow, while per-path windows independently control the per-path inflight. 4.2.2

As shown in Figure 7, switches place probe packets in a low-priority probe queue, while normal data packets use the high-priority queue. The switch applies strict-priority scheduling, so it serves data packets first and transmits probe packets only with bandwidth left unused by data packets. Rondo configures a small drop threshold for the probe queue, so excess probe packets are dropped to save the buffer. Since probe packets carry no application payload, their loss does not require retransmission. Both strict-priority queues and configurable per-queue drop thresholds are standard QoS mechanisms supported by commodity datacenter switches [16]. These mechanisms allow Rondo to aggressively probe for spare capacity, while introducing little interference with data packet transmission and switch buffer space. Converting probe ACKs into data capacity. The receiver acknowledges each arriving probe packet. The sender distinguishes probe ACKs from normal data ACKs by their PSNs and measures the probing ACK rates separately. Because probe packets receive service only from capacity left unused by data packets, the probe ACK rate reveals the additional capacity that the path can provide to the flow. Rondo incorporates this probe ACK rate into ARCC’s per-path capacity estimate and increases the data window accordingly. To use the discovered capacity promptly, each probe ACK allows the sender to transmit one additional data packet before the first window update that incorporates probe ACKs. Triggering probing. Rondo activates probing at flow startup and when a path appears to have spare capacity. At flow startup, each path starts with an initial window W0 , and probes for additional capacity. During transmission, if all data packets sent at least one base RTT earlier have been acknowledged for five consecutive updates, Rondo starts a probing phase lasting three base RTTs. Small increases in spare capacity can be captured by ARCC’s margin term q p , and capacity probing is therefore primarily used for rapid ramp-up when substantial spare capacity becomes available.

Capacity Probing

ARCC works well when the bottleneck is fully utilized, but the ACK rate cannot reveal unused bandwidth when a path is underutilized. A new mechanism is needed for O (1)1 rate increase. However, existing rate-increase mechanisms face a trade-off between fast bandwidth discovery and path stability. 1We use O notation to characterize how the time required for CC to converge to the efficient state scales with the deviation of the current state from the target state [73].

6

Data Packet

Probe Packet

S M F

i

Slow/Medium/Fast Path

Packet i

Switch

S

HP

S

M

1

HP

2

3

4

F 1

4

Sender

3

2

1

2

3

4

Receiver Arrival order

(b) Packet Scheduling in Rondo

       
′ W p  ′ D − D , 0 max skip p =  p p  ∑ RTT p′ ′

(7)

p ∈P p′ ̸= p

where D p′ is the measured forward delay of path p′ . For a packet sent on path p, packets transmitted on a faster path p′ within D p − D p′ can arrive earlier. Multiplying this interval by the transmission rate on p′ estimates their number. The sum therefore estimates the current transmission opportunity’s relative position in the expected arrival order across all paths. Rondo transmits the packet with the (skip p + 1)-th smallest unsent PSN, leaving the preceding lower-PSN packets for transmissions expected to arrive earlier. Delay measurement. Packet scheduling requires the forwarddelay difference across paths. For each used path, Rondo periodically timestamps a packet to measure the forward delay and RTT. The scheduler does not require fine-grained clock synchronization because the sender-receiver clock offset cancels when computing inter-path delay differences. The skip calculation assumes stable inter-path delay differences and sufficiently smooth short-term transmission rates. The former is provided by the controller’s stable queue control, while flow-level pacing and deficit-based path selection suppress bursty packet assignments. Moreover, Rondo enables packet scheduling only after the forward-delay estimate remains stable for several consecutive samples.

(6)

j∈P

It then selects the path with the largest deficit and decreases its deficit by one. The deficit tracks the difference between how many transmissions a path should have received according to its open-window share and how many it has actually received. A path that has received less traffic accumulates deficit and is more likely to be selected subsequently.

4.3

1

and skips the corresponding number of unsent packets, leaving the skipped lower-PSN packets for those earlier-arriving transmissions. For a transmission on path p,

Per-path windows determine how much load each path can carry, but the sender still needs to decide how to distribute individual transmissions across multiple available paths. Roundrobin or random selection [18, 38, 41] does not account for differences in path conditions, while selecting paths with larger open windows [63] can create bursty packet assignments. Rondo therefore uses deficit-based path selection to distribute transmissions according to the per-path windows while avoiding bursts. For each path p, Rondo computes its open window as O p = max(0,Wp − pktIn f l p ). When ARCC permits a transmission, Rondo increases the deficit Z p of every available path by its normalized open-window share, Op . ∑ Oj

2

Figure 8: PSN-order transmission can cause out-of-order arrivals, while Rondo adjusts the transmission order to mitigate it.

Deficit-based Path selection

Zp ← Zp +

3

Transmission order

Arrival order

(a) Transmission in PSN order

Figure 7: Probes use low-priority queues and consume only spare bandwidth.

4

Sender

Receiver

Transmission order

LP

4.2.3

M

F

LP

Packet Scheduler

Even with stable path control, persistent inter-path delay differences can cause packets to arrive out of order. Since the multipath controller has already determined the appropriate per-path load, the scheduler should mitigate such reordering without changing the transmission time or path selected by the controller. As shown in Figure 8, existing transports always send the smallest unsent PSN for each transmission opportunity, and the packet on a slower path may be overtaken by packets transmitted later on faster paths. The packet scheduler in Rondo instead selects higher-PSN packets for transmissions on slower paths, better aligning the resulting arrival order with the PSN order. Computing the skip. Recall from §3.3 that, for a transmission opportunity on path p, we can estimate how many packets transmitted later on the other paths are expected to arrive before the packet sent on path p. Rondo sums these estimates

4.4

Theoretical Analysis

We analyze Rondo with a fluid model, summarizing the conclusion below and providing proofs in Appendix C. Proposition 1 (Fairness). For N flows’ logical paths2 passing a single bottleneck of capacity C, ARCC converges geometrically to equal throughput C/N. 2 A logical path is a transport-level path identifier that the network maps

to a physical forwarding path.

7

Proposition 2 (Out-of-order degree bound). Rondo provides the following worst-case bound on the OOD caused by interpath queue differences:   Nl q0 Rmax Tmax Cτq + + NSW0 (8) OOD ≤ C Tmin 4

• How does Rondo compare with existing transports in throughput and OOD across diverse workloads and network conditions? (§6.2.2–§6.2.4) • How does reducing OOD improve performance? (§6.2.5) • How does each design component contribute to Rondo’s performance? (§6.3) Baselines. We compare Rondo with ECMP and five representative multipath transports. • ECMP [23] + DCQCN [75]: This configuration is widely deployed in today’s RDMA datacenters. Since ECMP pins each flow to a single path, it does not introduce packet reordering in the absence of packet loss. It uses selective repeat for recovery. • MP-RDMA [42]: MP-RDMA uses a per-flow, ECN-driven congestion control. Its ACK-clocked path selection transmits packets over responding paths and temporarily prunes paths whose delayed ACKs indicate excessive OOD. • Falcon [63]: Falcon uses a per-path Swift-like [33] congestion control, adjusting its window according to the path’s RTT. It transmits packets on the path with the maximum available window. It uses PLB [55] to remap persistently congested paths, a NIC congestion window to avoid buffer exhaustion and per-path RACK-TLP for loss recovery. • UET [65]: UET uses NSCC, a per-flow congestion control that adjusts its window using both ECN and RTT. Its REPS path selection [13] learns path conditions from ACKs and caches paths for packet spraying. UET uses ER and RTO with dynamic thresholds for loss recovery. • RoCE Adaptive Routing (AR) [31]: RoCE AR delegates path selection to switches, which choose output ports based on local queue occupancy. • Round-robin Packet Spraying (RRPS) [38, 41]: RRPS performs congestion-oblivious packet spraying by assigning packets to paths in a round-robin manner.

where Nl and NS are the upper bounds on active and starting logical paths in the bottleneck, respectively; Rmax is the maximum aggregate sending rate among these flows; Tmin and Tmax are the minimum and maximum base RTTs. This is a conservative worst-case bound that requires extreme queue imbalance across paths. Proposition 3 (Packet scheduling correctness). Suppose each path p of a flow has an arbitrary but fixed forward delay D p and a constant sending rate r p . If the delays and rates are accurately measured, the packet scheduler eliminates the OOD caused by persistent path-delay heterogeneity.

5

Implementation

We implement a Rondo prototype in DPDK [19] with ∼3000 LoC. Using DPDK’s mlx5 PMD, the prototype follows a datapath similar to that of an RNIC. All Rondo control logic runs on the sender’s DPDK lcore. We implement UDP-sourceport-based path steering, strict-priority scheduling, and tail drop in ∼400 lines of P4 code. Overhead. The DPDK implementation saturates the NIC line rate on our testbed. We further estimate Rondo’s arithmetic overhead based on [59]. Path selection and packet scheduling are performed per packet, while ARCC and skip calculations run periodically off the critical path. With four paths per flow, the estimated overhead is 25 cycles per packet on the critical path and 170–210 cycles per flow every Tu for periodic control. For comparison, single-path HPCC takes 180 cycles per ACK under the same model [28]. Appendix D provides implementation details, optimizations, and the overhead breakdown. Compatibility. Rondo is compatible with the transport semantics and upper-layer interfaces of existing datacenter transports, and relies on widely supported ACK feedback and delay measurement [11, 27, 33]. Packet scheduling may require additional support in hardware transports, but Tonic [8] and SR in modern RNICs suggest that flexible packet selection is feasible in high-speed NICs. The switch-side mechanisms require only standard strict-priority scheduling and configurable queue drop thresholds available on commodity switches.

6

6.1

Testbed Evaluation

Testbed setup. Our testbed uses two servers with ConnectX-7 NICs connected through an Intel Tofino programmable switch over 100 Gbps links. We configure the switch to forward packets from the ingress port through one of four internal ports, creating four equal-cost paths, as shown in Figure 9a. In the homogeneous-delay test, all paths have the same base RTT of ~20 µs, and we limit their bandwidths to 25, 20, 15, and 10 Gbps to emulate different degrees of congestion. In the heterogeneous-delay test, we add an additional 10 µs delay to one path by chaining additional loopback stages. We compare Rondo with RRPS, MP-RDMA and Falcon in the testbed. Result. As shown in Figure 9b, Rondo achieves 68.18 and 68.14 Gbps in the homogeneous- and heterogeneous-delay tests, respectively, matching Falcon’s throughput while outperforming RRPS and MP-RDMA. Its per-path windows and deficit-based path selection adapt the inflight traffic to each

Evaluation

We evaluate Rondo using our DPDK prototype on a hardware testbed and large-scale ns-3 [48] simulations. Our evaluation answers the following questions: • Can Rondo achieve high throughput while maintaining low OOD on a hardware testbed? (§6.1) 8

OOD (packets)

20

(a) Testbed with four paths.

P99

Maximum

80 60 40

18 12 6

20 A M D

M

PR

PS

co n Fa l

do

RR

Ro n

A M D PR

PS

M

M

Fa l

RR

Ro n

n

A M PRD

co

PS

do

Homogeneous-delay test

Fa l

do

RR

Ro n

n

A M

M

M

Heterogeneous-delay test

(b) Rondo fully utilizes the available path capacity.

PRD

PS

co Fa l

do

RR

M A D PR

PS

co n Fa l

do

RR

Ro n

M A D

M

PR

PS

co n

Homogeneous-delay test

co n

0 0

0

R

Average

100

40

Ro n

S

Path 3

Ro n

100

Path 2

60

do

Switch BW=100Gbps

10Gbps

Fa l

15

RR

20

Throughput (Gbps)

25

Path 1

queueing time (µs)

24 Path 0

80

Homogeneous-delay test

Heterogeneous-delay test

(c) Rondo limits the out-of-order degree.

Heterogeneous-delay test

(d) Rondo reduces queueing-delay oscillations.

Figure 9: Testbed topology and results. path’s capacity. In contrast, RRPS distributes packets evenly across paths, so the 10-Gbps bottleneck path limits its aggregate throughput. As shown in Figure 9c, Rondo reduces average OOD by 82–92% relative to RRPS, 64–88% relative to Falcon, and 72–81% relative to MP-RDMA. In the homogeneous-delay test, this benefit mainly comes from better queue control, as shown in Figure 9d. In the heterogeneousdelay test, packet scheduling further compensates for the persistent delay difference.

6.2

Large-Scale Simulation

6.2.1

Simulation Setup

and MoE-like all-to-all traffic. Incast induces last-hop congestion, while permutation traffic has each host send to one host and receive from exactly one host, stressing the network fabric. In MoE-like all-to-all traffic, each host simultaneously sends data to multiple randomly selected receivers. (ii) Tracedriven workloads include both realistic datacenter traffic and distributed training. The datacenter workloads are derived from production Web Search [6] and Cache Follower [58] traces. For distributed training, we use STAGE [43] and AstraSim [69] to simulate the communication trace of training a 7B MoE Transformer model. 6.2.2

We compare Rondo with ECMP, MP-RDMA, Falcon, UET and AR in the simulation. We exclude comparison with RRPS here as our testbed experiments show that it substantially underutilizes the available path capacity. Except in §6.2.5, all schemes use a sufficiently large reorder buffer to independently characterize each scheme’s bandwidth utilization and OOD behavior, without conflating them with OOD-induced performance degradation. Specifically, each NIC has a 1.2 MB shared reorder buffer (approximately 8× the BDP), which accommodates the entire observed OOD distribution. When the buffer is full, the NIC accepts only the next expected packet, drops other packets, and triggers NAKs. Parameters. We configure Rondo with Tu = 0.5 base_RT T , q0 = 3 MTU, τq = 0.04 max_base_RT T , the ACK-rate EWMA smoothing parameter α = 0.8, and γ = 0.5. For the other baselines, we use the recommended parameter settings when available and otherwise use the best-performing configuration obtained through parameter tuning. Simulation topology. We use a three-tier Clos topology [34] comprising 320 servers across 5 pods. Each pod contains 4 ToR and 4 aggregation switches, and 16 core switches interconnect the pods. Server-to-ToR links operate at 100 Gbps, while inter-switch links operate at 400 Gbps. The default topology is non-oversubscribed (1:1) and provides 4 equalcost paths between ToRs in the same pod and 16 between ToRs in different pods. We also evaluate a 4:1 oversubscribed configuration with only 4 active core switches. Workloads. We evaluate Rondo using the following workloads: (i) Typical traffic patterns comprise incast, permutation,

Typical Traffic Patterns

Figure 10 shows the performance of different algorithms under typical datacenter traffic patterns with 1:1 and 4:1 oversubscription. The NIC-level OOD is the sum of the OOD across all flows at the NIC. In the all-to-all and permutation scenarios, Rondo reduces the maximum NIC-level OOD by 28.6%–91.5% compared to MP-RDMA, 77.3%–92.7% compared to Falcon, 76.3%–97.0% compared to UET, and 43.8%– 97.5% compared to AR. In the incast scenario, Rondo does not cause any out-of-order arrivals, whereas other multipath schemes exhibit maximum OOD values of 338–1348 packets. Meanwhile, Rondo achieves speedups of 1.12×–3.26× and provides the lowest JCT in four of the six scenarios, with Falcon and UET showing comparable performance. AR performs poorly under 4:1 oversubscription because it cannot capture end-to-end congestion. Rondo’s OOD reduction mostly comes from the controller’s ability to keep queues stable, while the throughput benefit comes from its rapid convergence to the ideal operating point. In summary, Rondo substantially reduces OOD while preserving high throughput under all traffic patterns. 6.2.3

Trace-Driven Workloads

Distributed-training workloads. We use Astra-sim to simulate the communication traffic of distributed ML workloads, including Allreduce, All-to-All, Allgather, Reduce-Scatter, and point-to-point communication. Figure 11 shows that Rondo accelerates training communication by 1.48×, matching Falcon and outperforming the other schemes. Moreover, 9

Rondo All-to-All OS1 Speedup

MP-RDMA

All-to-All OS4

1.5

Falcon

UET

Incast OS1

AR

p99

1.0

0.8

Incast OS4

0.8

0.8 0.4

1.5

0.0

0.0

0.4

0.4

0.0

0.0

0.0 317

300

Permutation OS1

1.2

0.5

638 768

max

1.2

338 879 553 1336

100

Permutation OS4 1.8

1.2

300

NIC-level OOD

ECMP

3.0 1.2

476 857 766 1348

100

100

0.6 0.0 131 114

100

225

225

75

75

75

75

150

150

50

50

50

50

75

75

25

25

25

25

0

0

0

0

0

0

297

Figure 10: Performance under typical datacenter traffic patterns. JCT speedup is measured relative to ECMP. Rondo

ECMP

MP-RDMA

Falcon

UET

AR

p99

max

the delays of randomly selected links. We also add single-path background flows to the network for mixed traffic scenarios. Across all heterogeneous scenarios, Rondo achieves the best speedup and maintains the lowest OOD. This is because, for heterogeneity caused by fabric bandwidth changes or background traffic, its ACK-rate measurements detect capacity changes and quickly adapt to the new capacities. For linkdelay heterogeneity, its packet scheduler compensates for the delay differences to keep OOD low.

696

320

NIC-level OOD

Speedup

1.5

1.0

0.5

240 160 80

0.0

0

Figure 11: Communication-time speedup and NIC-level OOD during MoE Transformer training. MP-RDMA Falcon

UET AR

84

Rondo ECMP

p99 OOD max OOD 83

120

40 20

266 512

274 298 736

347 296 729

p99 p99.9

6.2.5

The preceding experiments deliberately exclude the performance degradation caused by limited reorder buffers to study each scheme’s throughput and OOD behavior. We next show that controlling OOD reduces reorder buffer requirements and enables prompt loss recovery without excessive spurious retransmissions. Controlled OOD sustains throughput with small reorder buffers. Provisioning O(BDP) reorder buffer capacity in hardware is increasingly costly as link bandwidth grows. We therefore vary the shared reorder buffer size from 1.2 MB to 32 KB under 4-to-1 incast with background flows. Figure 14 shows that Rondo sustains 94.8 Gbps of goodput even with a 32 KB buffer because its bounded OOD rarely exhausts the buffer capacity. In contrast, the other multipath transports generate more out-of-order arrivals; as the buffer shrinks, buffer exhaustion causes more packet drops. With a buffer of approximately one BDP (used in Falcon4 ), Rondo achieves 1.15×–3.60× the goodput of the other multipath transports. With a 32 KB buffer, this advantage grows to 1.38×–8.51×. Controlled OOD enables fast and accurate loss detection. A receiver cannot immediately determine whether a PSN gap is caused by packet loss or reordering. Figure 15 compares all schemes with their corresponding reliability mechanisms under a 0.1% random packet-drop rate. Rondo with ER, ECMP, and Falcon with per-path RACK-TLP incur few spurious retransmissions, whereas the other schemes exhibit substantial spurious retransmission ratios. Rondo also achieves a lower retransmission delay by bounding OOD, which enables prompt

1.5

NIC-level OOD

400

180 120 60 WebSearch 50% load

WebSearch 80% load

CacheFollower CacheFollower 50% load 80% load

591

546

1278

790

644

556

Link-delay heterogeneity

Single-path background flows

300 200 100 0

Bandwidth heterogeneity

Figure 12: Rondo improves Figure 13: Rondo’s perfortail FCT while reducing OOD mance under different heterounder realistic workloads. geneous scenarios. it controls both tail and maximum OOD, keeping them significantly lower than those of other multipath transports. Realistic datacenter workloads. Figure 12 shows that Rondo significantly reduces tail flow completion time (FCT) 3 under Web Search and Cache Follower workloads across different loads. This result shows that multipathing also benefits general datacenter workloads. These workloads contain more short flows and are harder to control than ML workloads, but Rondo still guarantees a low OOD. 6.2.4

Benefits of Controlled Reordering

3.0

313 356 844

240

NIC-level OOD

UET AR

0.0

0

0

MP-RDMA Falcon

4.5

60

Speedup

p99 FCT slowdown

Rondo ECMP 80

Heterogeneous Network Conditions

We evaluate these schemes under different heterogeneous scenarios, with the results shown in Figure 13. For bandwidth heterogeneity, we randomly select 10% of the links and reduce their bandwidth to one-quarter of the original capacity 50 µs after flows start. For link-delay heterogeneity, we add 20 µs to 3 FCT slowdown is defined as the ratio of a flow’s actual FCT to its ideal

4 A NIC equipped with DRAM, such as Falcon, can extend its reorder

FCT under line-rate transmission.

buffer into on-NIC DRAM at the cost of higher access latency.

10

75 50 25

1 BDP

32

64

128

100 60

50 30

0

256

512

1200

ER-4

Figure 14: Aggregate goodput with different reorder-buffer sizes.

32 flows join

32 flows leave

0.0 100 50

Probing triggers

0 1

UET

AR

ER

ER-8

Bitmap RACK-TLP

Rondo max queue

0.5

0

SR

40

20

0

19.2

2

Time (ms)

3

4

800

MP-RDMA min queue

20

400

0 0.0

ER OOD 4

1.0

0.5

1.0

0.864

MP-RDMA

0.905

Falcon

0.995

UET

0.999

RTO 4 RTT

0.75

0.906

1 1.25 1.5 2.5 Normalized per-flow throughput

3

Figure 17: Per-flow throughput distribution under incast, normalized by each scheme’s mean throughput.

Rondo maintains stable queues. We evaluate Rondo under the same setup as in §3.2 and present the result in Figure 19. The Rondo controller maintains low and stable queues, keeping the inter-path queue difference always below 30 KB, whereas the difference under other schemes can reach approximately 200 KB.

1.5

1.5

Time (ms)

Figure 18: Rondo converges Figure 19: Rondo maintains quickly.5 stable queues.

Packet scheduler controls OOD under heterogeneity. Figure 20 illustrates how packet scheduling works. We dynamically change the delay of one link while a flow is transmitting. Without the scheduler, Rondo can maintain a stable queue, but it cannot handle the persistent delay difference, causing OOD to increase. The scheduler detects this difference after the delay changes and precisely adjusts the transmission order, thereby maintaining a low OOD.

early retransmission. Rondo does not prescribe a particular loss recovery mechanism and is compatible with most existing recovery schemes. Figure 16 further compares two other loss-recovery mechanisms in Rondo. Early recovery with a four-packet threshold recovers losses in 19.2 µs while incurring only 0.05% spurious retransmissions. Flow-level RACK-TLP achieves a similar recovery delay of 18.2 µs but incurs 1.2% spurious retransmissions. Thus, controlling OOD allows Rondo to use a small reordering threshold for prompt loss recovery without sacrificing detection accuracy.

6.3

0.999

ECMP

without overshooting. These fast and precise reactions enable Rondo to achieve high performance.

200 0 0.0

ER RACK-TLP RACK-TLP OOD 8 flow path

Figure 16: Recovery delay with different recovery mechanisms in Rondo.

Falcon

0.5

18.2

21.8

Rondo

AR

Rondo (zoom)

40

600

22.3

JFI

Equal share

49.3

Falcon

1.5 1.0

ECMP MP-RDMA Falcon

Figure 15: Loss-recovery performance under 0.1% random packet loss.

Per-path queue (KB)

Utilization (%) Queue (MB)

0 Rondo

NIC reorder-buffer size (KB)

Rondo

Spurious retx. ratio

Spurious retx. (%)

UET AR

Mean retx. delay (µs)

100

MP-RDMA Falcon

Mean retx. delay (µs)

Goodput (Gb/s)

Mean retx. delay

Rondo ECMP

Ablation study. Figure 21 shows the contribution of each Rondo component under permutation traffic. Replacing perpath Swift with ARCC reduces the average out-of-order degree by 87.3% and the p99 out-of-order degree by 53.3%, because ARCC maintains stable queues after convergence. Deficit-based path selection further reduces OOD by smoothing packet assignments relative to random packet spray (RPS). Capacity probing then significantly reduces the maximum and p99 OOD by avoiding queue overshoot during flow startup and rapid rate increases. The packet scheduler helps under persistent path heterogeneity, as shown in Figure 20.

Deep Dive into Rondo

Rondo provides fairness for flows sharing the same bottleneck. Figure 17 shows that Rondo provides similar per-flow throughput, with a Jain fairness index of 0.999 under 32-to-1 incast. For multipath transports in this incast scenario, the last-hop switch is the only bottleneck. Under ECMP, however, flow collisions create other bottlenecks even when some paths remain underutilized. The controller achieves fast convergence. Figure 18 shows how Rondo reacts to flows joining and leaving. Initially, 8 flows share the bottleneck. When 32 flows join the bottleneck, Rondo transmits only W0 data packets for each flow and uses probe packets to discover the remaining capacity, avoiding a large queue buildup. ARCC then adjusts its window based on the ACK rate and drains the queue quickly without reducing bandwidth utilization. After these flows leave, Rondo triggers capacity probing and fully utilizes the bandwidth in one step

Scalability. To evaluate the topology scalability of Rondo, we measure its speedup relative to ECMP for all-to-all traffic on a k-ary fat-tree topology, where k ranges from 10 to 16. Figure 22 shows that Rondo’s speedup increases from 3.31× to 3.81×, while its p99 OOD remains zero packets. Parameter sensitivity. Figure 23 shows that Rondo is robust to Tu and τq , maintaining 0.92–1.00× normalized throughput and a maximum OOD of 18–36 packets across all settings under permutation traffic. A smaller Tu enables faster reactions but makes the ACK-rate measurements less stable. A larger τq maintains a larger queue which immediately utilizes newly available capacity but generally increases OOD.

5 Utilization in this figure accounts only for data packets and excludes

Additional evaluations are provided in Appendix E.

Rondo’s probe packets.

11

max

Rondo speedup

Falcon speedup

Rondo OOD

Falcon OOD

0 60 40 20

60

30

4

120

3

90

2

60 30

1

0 20

40

60

80

Time (ms)

Figure 20: Packet scheduling reduces OOD.

0

Per-path Swift+ ARCC+RPS ARCC+deficit RPS w/o Probing PS w/o Probing

Rondo

0

0

Figure 21: Ablation study of Rondo’s components.

250 (k=10)

432 (k=12)

686 (k=14)

τq / max base RTT

15

90

p99 NIC-level OOD

30

0

Default

NUM: normalized throughput 36

0.5

1.00×

1.00×

0.99×

1.00×

0.2

0.98×

1.00×

0.97×

0.96×

0.1

1.00×

1.00×

0.96×

0.96×

28

0.04

0.98×

1.00×

0.93×

0.97×

24

0.01

1.00×

1.00×

0.92×

0.96×

20

0.25

0.5

0.75

1

1024 (k=16)

32

Max NIC-level OOD (packets)

p99

120 +15 μs

+10 μs

Speedup

45

avg

w/o Packet Scheduler +5 μs

NIC-level OOD

OOD (pkts) Skip (pkts)

Rondo +0 μs

Number of servers

Tu / base RTT

Figure 22: Scalability results.

Figure 23: Parameter sweep.

Transport

Congestion control

Path selection

Switch support

In-order delivery1

RoCE AR [31] SRD [61] Falcon [63] MP-RDMA [42] UET [65] MRC [9] EQDS/NDP [22, 52]

DCQCN or ZTR-RTT Delay-based Per-path delay-based Flow-level ECN-based Flow-level ECN + delay Flow-level ECN + delay Receiver-driven

Switch-driven AR Delay-based Maximum-open-window ACK-clocked, OOD-aware ACK/ECN-based ACK/ECN-based Random packet spray

NVIDIA Spectrum Commodity Ethernet Commodity Ethernet Commodity Ethernet Commodity Ethernet Commodity Ethernet Packet trimming

No No Optional2 No Optional (single-path)2 No Yes2

Rondo

Per-path ACK-rate-based CC with probing

Deficit-based

Commodity Ethernet

Yes2 (controllable reorder-buffer demand)

1 In-order delivery refers to whether packets are delivered to the upper layer in order despite out-of-order arrival. 2 UET provides in-order delivery only in single-path mode; the other transports use a reorder buffer.

Table 1: Overview of representative datacenter packet spraying transports.

7

Related Work

dicators and dedicated packets for probing to maintain stable queues and limit reordering. A more comprehensive discussion of datacenter load balancing and congestion control is provided in Appendix F.

Packet-level multipathing. Datacenter transports [9, 22, 42, 61,63,65] and load-balancing schemes [13,18,21,24,35,38,41, 53] exploit path diversity through packet-level multipathing. Table 1 summarizes representative transports. These designs either do not support in-order delivery, fall back to singlepath, or prioritize bandwidth utilization that may introduce significant packet reordering. In contrast, Rondo fully utilizes all available paths while keeping OOD under control. Other methods to reduce reordering. Conweave [64] reorders packets at the ToR switch, but limited switch buffering constrains its scalability and path-switching granularity. QDAPS [26] and FLB [24] select paths based on queueing delays to mitigate reordering, but tend to direct subsequent packets onto a high-delay path to follow an earlier packet. MP-RDMA [42] prunes paths with late ACKs to reduce reordering, leaving slower paths underutilized. DAPS [32] and STMS [62] are MPTCP [68] schedulers that reorder transmissions across subflows, but target low-rate Internet settings and rely on heuristic or reactive schemes that cannot precisely control OOD. Rondo co-designs the multipath controller and packet scheduler for high-throughput datacenter transport. Congestion control. ACC [71] uses ACK rates to determine the pause duration after congestion, but relies on tentative rate increases to reclaim spare bandwidth, resulting in slow ramp-up and queue oscillations. Aeolus [25] exploits spare bandwidth through speculative data transmission, but the packets still interfere with normal data traffic and are more likely to be dropped. Rondo instead uses ACK rates as capacity in-

8

Conclusion

We present Rondo, a multipath transport that controls packet reordering while sustaining high throughput. Rondo codesigns the multipath controller and packet scheduler: the controller limits transient queue differences across paths, while the scheduler compensates for persistent inter-path delay differences. Together, these mechanisms show that highthroughput packet-level multipathing need not incur extensive reordering, providing a practical design point for transports operating with constrained NIC resources.

References [1] Vamsi Addanki, Oliver Michel, and Stefan Schmid. PowerTCP: Pushing the performance limits of datacenter networks. In 19th USENIX symposium on networked systems design and implementation (NSDI 22), pages 51–70, 2022. [2] Anup Agarwal, Venkat Arun, and Srinivasan Seshan. FRCC: Towards Provably Fair and Robust Congestion Control. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pages 2755–2778, 2026. 12

[3] Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. A scalable, commodity data center network architecture. ACM SIGCOMM Computer Communication Review, 38(4):63–74, August 2008.

[10] Serhat Arslan, Yuliang Li, Gautam Kumar, and Nandita Dukkipati. Bolt: Sub-RTT congestion control for UltraLow latency. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 219–236, 2023.

[4] Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson Huang, and Amin Vahdat. Hedera: Dynamic flow scheduling for data center networks. In 7th USENIX Symposium on Networked Systems Design and Implementation (NSDI 10), San Jose, CA, April 2010. USENIX Association.

[11] Aviv Barnea, Itay Ozery, and Barbara Claman. Scaling Zero Touch RoCE Technology with Round Trip Time Congestion Control. https://developer.nvidia.com/blog/scaling-zero-touchroce-technology-with-round-trip-time-congestioncontrol/, December 2021.

[5] Mohammad Alizadeh, Tom Edsall, Sarang Dharmapurikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Matus, Rong Pan, Navindra Yadav, and George Varghese. CONGA: Distributed congestion-aware load balancing for datacenters. In Proceedings of the 2014 ACM Conference on SIGCOMM, SIGCOMM ’14, pages 503–514, New York, NY, USA, August 2014. Association for Computing Machinery.

[12] Giacomo Bernardi, Ratul Mahajan, C. Seshadhri, Enrico Carlesso, Chinchu Merine Joseph, Saurabh Kumar, Pavan Manikonda, Luiza Popa, Randy Ram, Steven Robinson, and Elizabeth Tennent. Rng: Flat datacenter networks at scale, 2026. arXiv:2604.15261. [13] Tommaso Bonato, Abdul Kabbani, Ahmad Ghalayini, Michael Papamichael, Mohammad Dohadwala, Lukas Gianinazzi, Mikhail Khalilov, Elias Achermann, Daniele De Sensi, and Torsten Hoefler. REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure Mitigation. In Proceedings of the European Conference on Computer Systems (EuroSys’26), 04 2026.

[6] Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. Data center TCP (DCTCP). In Proceedings of the ACM SIGCOMM 2010 Conference, pages 63–74, 2010. [7] Thomas Anderson, Andrew Collins, Arvind Krishnamurthy, and John Zahorjan. PCP: Efficient endpoint congestion control. In 3rd Symposium on Networked Systems Design & Implementation (NSDI 06), San Jose, CA, May 2006. USENIX Association.

[14] Lawrence S Brakmo, Sean W O’malley, and Larry L Peterson. Tcp vegas: New techniques for congestion detection and avoidance. In Proceedings of the conference on Communications architectures, protocols and applications, pages 24–35, 1994.

[8] Mina Tahmasbi Arashloo, Alexey Lavrov, Manya Ghobadi, Jennifer Rexford, David Walker, and David Wentzlaff. Enabling programmable transport protocols in {High-Speed}{NICs}. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 93–109, 2020.

[15] Neal Cardwell, Yuchung Cheng, C Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. Bbr: Congestion-based congestion control: Measuring bottleneck bandwidth and round-trip propagation time. Queue, 14(5):20–53, 2016.

[9] Joao Araujo, Alex Chow, Mark Handley, Ryder Lewis, Christoph Paasch, Jitendra Padhye, Michael Papamichael, Greg Steinbrecher, Amin Tootoonchian, Lihua Yuan, S. Anantharamu, Abhishek Dosi, Mohit Garg, Mahdieh Ghazi, Torsten Hoefler, Deepal Jayasinghe, Jithin Jose, Abdul Kabbani, Guohan Lu, Yang Wang, K. Doddapaneni, Murali Garimella, Vipin Jain, Yanfang Le, H. Nagulapalli, S. Narayanan, Rong Pan, Rathina Sabesan, Raghava Sivaramu, Rip Sohan, Eric Davis, Dragos Dumitrescu, Mohan Kalkunte, Bhaswar Mitra, Guglielmo Morandin, Adrian Popa, Costin Raiciu, Eric Spada, John Spillane, Niranjan Vaidya, Aviv Barnea, Idan Burstein, Elazar Cohen, Yamin Friedman, Noam Katz, Masoud Moshref, Yuval Shpigelman, Shahaf Shuler, Shy Shyman, and Sayantan Sur. Resilient AI supercomputer networking using MRC and SRv6, 2026. arXiv:2605.04333.

[16] Cisco. Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide, Release 10.2(x). https://www.cisco.com/c/en/us/td/docs/dcn/nxos/nexus9000/102x/configuration/qos/cisco-nexus9000-nx-os-quality-of-service-configuration-guide102x/m-configuring-queuing-and-scheduling.html. [17] DeepSeek-AI. DeepEP: An efficient expert-parallel communication library. https://github.com/deeps eek-ai/DeepEP. [18] Advait Dixit, Pawan Prakash, Y. Charlie Hu, and Ramana Rao Kompella. On the impact of packet spraying in data center networks. In 2013 Proceedings IEEE INFOCOM, pages 2130–2138, April 2013. [19] Linux Foundation. Data Plane Development Kit (DPDK). https://www.dpdk.org, 2026. 13

[20] Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. RDMA over Ethernet for distributed training at meta scale. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 57–70, 2024.

[29] Weihao Jiang, Wenli Xiao, Yuqing Yang, Peirui Cao, and Shizhen Zhao. Orderlock: A New Type of Deadlock and its Implications on High Performance Network Protocol Design. In Proceedings of the ACM SIGCOMM 2025 Conference, pages 575–591, São Francisco Convent Coimbra Portugal, September 2025. ACM.

[21] Soudeh Ghorbani, Zibin Yang, P. Brighten Godfrey, Yashar Ganjali, and Amin Firoozshahian. DRILL: Micro Load Balancing for Low-latency Data Center Networks. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pages 225–238, Los Angeles CA USA, August 2017. ACM.

[30] Abdul Kabbani, Balajee Vamanan, Jahangir Hasan, and Fabien Duchene. FlowBender: Flow-level Adaptive Routing for Improved Latency and Throughput in Datacenter Networks. In Proceedings of the 10th ACM International on Conference on Emerging Networking Experiments and Technologies, CoNEXT ’14, pages 149–160, New York, NY, USA, December 2014. Association for Computing Machinery.

[22] Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W. Moore, Gianni Antichi, and Marcin Wójcik. Re-architecting datacenter networks and stacks for low latency and high performance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pages 29–42, Los Angeles CA USA, August 2017. ACM.

[31] Sajy Khashab, Albert Gran Alcoz, Alon Gal, Jacky Romano, Rani Abboud, Yonatan Piasetzky, Lior Maman, Amit Nishry, Barak Gafni, Omer Shabtai, Matty Kadosh, Dror Goldenberg, Gilad Shainer, and Mark Silberstein. High-speed Networking for Giga-Scale AI Factories, May 2026. arXiv:2605.21187.

[23] C. Hopps. Analysis of an Equal-Cost Multi-Path Algorithm. Technical Report RFC2992, RFC Editor, November 2000.

[32] Nicolas Kuhn, Emmanuel Lochin, Ahlem Mifdaoui, Golam Sarwar, Olivier Mehani, and Roksana Boreli. DAPS: Intelligent delay-aware packet scheduling for multipath transport. In 2014 IEEE International Conference on Communications (ICC), pages 1222–1227, June 2014.

[24] Jinbin Hu, Wenxue Li, Xiangzhou Liu, Junfeng Wang, Bowen Liu, Ping Yin, Jianxin Wang, Jiawei Huang, and Kai Chen. FLB: Fine-grained Load Balancing for Lossless Datacenter Networks. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), pages 365– 380, 2025.

[33] Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, David Wetherall, and Amin Vahdat. Swift: Delay is Simple and Effective for Congestion Control in the Datacenter. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, pages 514– 528, Virtual Event USA, July 2020. ACM.

[25] Shuihai Hu, Wei Bai, Gaoxiong Zeng, Zilong Wang, Baochen Qiao, Kai Chen, Kun Tan, and Yi Wang. Aeolus: A Building Block for Proactive Transport in Datacenters. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, pages 422–434, Virtual Event USA, July 2020. ACM.

[34] P. Lapukhov, A. Premji, and J. Mitchell. Use of BGP for Routing in Large-Scale Data Centers. RFC 7938, August 2016.

[26] Jiawei Huang, Wenjun Lv, Weihe Li, Jianxin Wang, and Tian He. QDAPS: Queueing Delay Aware Packet Spraying for Load Balancing in Data Center. In 2018 IEEE 26th International Conference on Network Protocols (ICNP), pages 66–76, September 2018.

[35] Yanfang Le, Rong Pan, Peter Newman, Jeremias Blendin, Abdul Kabbani, Vipin Jain, Raghava Sivaramu, and Francis Matus. STrack: A Reliable Multipath Transport for AI/ML Clusters, July 2024. arXiv:2407.15266.

[27] InfiniBand Trade Association. InfiniBand Architecture Specification, volume 1, release 2.1. https://infini bandta.org/ibta-specification/, 2026.

[36] Minghao Li, Alicia Golden, Samuel Hsia, Michael Kuchnik, Adi Gangidi, Xu Zhang, Ashmitha Jeevaraj Shetty, Zachary DeVito, Weiwei Chu, Dong He, Haoci Zhang, Yuchen Hao, Ruoming Pang, James Hongyi Zeng, Ying Zhang, Minlan Yu, and Carole-Jean Wu. ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training, 2026. arXiv:2605.24326.

[28] Wanchun Jiang, Yujie Hu, Kai Wang, Xiao Han, Haoran Wei, Yong Zhang, Jianxi Ye, and Huichen Dai. Odin: Rethinking Congestion Control under All-to-All Traffic. In Proceedings of the ACM SIGCOMM 2026 Conference, pages 1246–1263. Association for Computing Machinery, New York, NY, USA, August 2026. 14

[37] Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, and Minlan Yu. HPCC: High precision congestion control. In Proceedings of the ACM Special Interest Group on Data Communication, pages 44–58, Beijing China, August 2019. ACM.

Zats. TIMELY: RTT-based Congestion Control for the Datacenter. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, pages 537–550, London United Kingdom, August 2015. ACM. [46] Radhika Mittal, Alexander Shpiner, Aurojit Panda, Eitan Zahavi, Arvind Krishnamurthy, Sylvia Ratnasamy, and Scott Shenker. Revisiting network support for RDMA. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, SIGCOMM ’18, pages 313–326, New York, NY, USA, August 2018. Association for Computing Machinery.

[38] Zhexiong Li, Shugui Wei, Puyu Zhao, Feifan Wang, Yuepeng Li, Lin Gu, Deze Zeng, Xiaoliang Wang, and Laiping Zhao. PSN-PATH: When Multipath RDMA Meets Lossy Networks. In Proceedings of the ACM SIGCOMM 2026 Conference, SIGCOMM ’26, pages 1215–1227, New York, NY, USA, August 2026. Association for Computing Machinery.

[47] Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. Homa: A Receiver-Driven LowLatency Transport Protocol Using Network Priorities. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, pages 221–235, August 2018.

[39] Nandor Licker, Kevin Hu, Vladimir Zaytsev, and Lequn Chen. fabric-lib: Rdma point-to-point communication for llm systems. Proceedings of Machine Learning and Systems, 8:169–185, 2026. [40] Vincent Liu, Daniel Halperin, Arvind Krishnamurthy, and Thomas Anderson. F10: A Fault-Tolerant engineered network. In 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13), pages 399–412, Lombard, IL, April 2013. USENIX Association.

[48] The ns 3 project. Network simulator 3. https://www. nsnam.org/, 2025. [49] NVIDIA. ConnectX-6 Dx Firmware Release Notes: Changes and New Feature History. https://networ king-docs.nvidia.com/connectx6dxfwrn/223 52302lts/changes-and-new-feature-history, 2023.

[41] Xiangzhou Liu, Wenxue Li, Zihao Wang, and Kai Chen. Enabling packet spraying over commodity RNICs with in-network support. In Proceedings of the 21st European Conference on Computer Systems, pages 693–709, 2026.

[50] NVIDIA. Linux Kernel Upstream Release Notes: Changes and New Features History. https://netw orking-docs.nvidia.com/kernelupstreamrn/7. 1/changes-and-new-features-history, 2026.

[42] Yuanwei Lu, Guo Chen, Bojie Li, Kun Tan, Yongqiang Xiong, Peng Cheng, Jiansong Zhang, Enhong Chen, and Thomas Moscibroda. Multi-Path Transport for RDMA in Datacenters. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), pages 357–371, 2018.

[51] NVIDIA. Out-of-order Data Placement. https://ne tworking-docs.nvidia.com/doca/archive/3-5 -0/out-of-order-data-placement, June 2026. [52] Vladimir Olteanu, Haggai Eran, Dragos Dumitrescu, Adrian Popa, Cristi Baciu, Mark Silberstein, Georgios Nikolaidis, Mark Handley, and Costin Raiciu. An edgequeued datagram service for all datacenter traffic. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 761–777, 2022.

[43] Changhai Man, Joongun Park, Hanjiang Wu, Huan Xu, Srinivas Sridharan, and Tushar Krishna. Scalable synthesis of distributed llm workloads through symbolic tensor graphs. In Proceedings of the 53rd Annual IEEE/ACM International Symposium on Computer Architecture (ISCA ’26), 2026.

[53] Chenchen Qi, Wenfei Wu, Yongcan Wang, Keqiang He, Yu-Hsiang (Sean) Kao, Zongying He, Chen-Yu Yen, Zhuo Jiang, Feng Luo, Surendra Anubolu, Yanjin Gao, Bingfeng Lin, Wenda Ni, Yiming Yang, Donglin Wei, Boyang Zhou, Jian Wang, and Shan Ding. SGLB: Scalable and Robust Global Load Balancing in Commodity AI Clusters. In Proceedings of the ACM SIGCOMM 2025 Conference, SIGCOMM ’25, pages 626–644, New York, NY, USA, August 2025. Association for Computing Machinery.

[44] Ziming Mao, Yihan Zhang, Chihan Cui, Zhen Huang, Kaichao You, Zhongjie Chen, Zhiying Xu, Zhenyu Gu, Scott Shenker, Costin Raiciu, Yang Zhou, and Ion Stoica. UEP: Portable Expert-Parallel communication. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pages 1107–1123, 2026. [45] Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David 15

[54] Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, Chao Wang, Peng Wang, Pengcheng Zhang, Xianlong Zeng, Eddie Ruan, Zhiping Yao, Ennan Zhai, and Dennis Cai. Alibaba HPN: A Data Center Network for Large Language Model Training. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM ’24, pages 691–706, New York, NY, USA, August 2024. Association for Computing Machinery.

[62] Hang Shi, Yong Cui, Xin Wang, Yuming Hu, Minglong Dai, Fanzhao Wang, and Kai Zheng. STMS: Improving MPTCP Throughput Under Heterogeneous Networks. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 719–730, 2018. [63] Arjun Singhvi, Nandita Dukkipati, Prashant Chandra, Hassan M. G. Wassel, Naveen Kr. Sharma, Anthony Rebello, Henry Schuh, Praveen Kumar, Behnam Montazeri, Neelesh Bansod, Sarin Thomas, Inho Cho, Hyojeong Lee Seibert, Baijun Wu, Rui Yang, Yuliang Li, Kai Huang, Qianwen Yin, Abhishek Agarwal, Srinivas Vaduvatha, Weihuang Wang, Masoud Moshref, Tao Ji, David Wetherall, and Amin Vahdat. Falcon: A Reliable, Low Latency Hardware Transport. In Proceedings of the ACM SIGCOMM 2025 Conference, SIGCOMM ’25, pages 248–263, New York, NY, USA, August 2025. Association for Computing Machinery.

[55] Mubashir Adnan Qureshi, Yuchung Cheng, Qianwen Yin, Qiaobin Fu, Gautam Kumar, Masoud Moshref, Junhua Yan, Van Jacobson, David Wetherall, and Abdul Kabbani. PLB: Congestion signals are simple and effective for network load balancing. In Proceedings of the ACM SIGCOMM 2022 Conference, pages 207–218, Amsterdam Netherlands, August 2022. ACM. [56] Sivasankar Radhakrishnan, Yilong Geng, Vimalkumar Jeyakumar, Abdul Kabbani, George Porter, and Amin Vahdat. SENIC: Scalable NIC for End-Host rate limiting. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 475–488, Seattle, WA, April 2014. USENIX Association.

[64] Cha Hwan Song, Xin Zhe Khooi, Raj Joshi, Inho Choi, Jialin Li, and Mun Choon Chan. Network Load Balancing with In-network Reordering Support for RDMA. In Proceedings of the ACM SIGCOMM 2023 Conference, pages 816–831, New York NY USA, September 2023. ACM.

[57] Feng Ren, Ruoyu Qin, Teng Ma, Shangming Cai, Zheng Liu, Chao Lei, Dejiang Zhu, Ke Yang, Zheming Li, Jialei Cui, Weixiao Huang, Yikai Zhao, Yineng Zhang, Hao Wu, Xiang Gao, Yuhao Fu, Jinlei Jiang, Yongwei Wu, and Mingxing Zhang. TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving, July 2026. arXiv:2604.00368.

[65] Ultra Ethernet Consortium. Ultra Ethernet Specification. Technical Report Version 1.0.3, Ultra Ethernet Consortium, 2026. https://ultraethernet.org/wp-con tent/uploads/sites/20/2026/08/UE-Specifica tion-1.0.3.pdf. [66] Erico Vanini, Rong Pan, Mohammad Alizadeh, Parvin Taheri, and Tom Edsall. Let It Flow: Resilient Asymmetric Load Balancing with Flowlet Switching. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 407–420, 2017.

[58] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. Inside the social network’s (datacenter) network. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, pages 123–137, 2015.

[67] Zilong Wang, Layong Luo, Qingsong Ning, Chaoliang Zeng, Wenxue Li, Xinchen Wan, Peng Xie, Tao Feng, Ke Cheng, Xiongfei Geng, Tianhao Wang, Weicheng Ling, Kejia Huo, Pingbo An, Kui Ji, Shideng Zhang, Bin Xu, Ruiqing Feng, Tao Ding, Kai Chen, and Chuanxiong Guo. SRNIC: A Scalable Architecture for RDMA NICs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 1–14, 2023.

[59] RT-RK. Overview of SiFive 7 series microarchitecture in RISC-V GCC. https://www.rt-rk.com/overvi ew-of-sifive-7-series-microarchitecture-i n-risc-v-gcc/. [60] Ahmed Saeed, Nandita Dukkipati, Vytautas Valancius, Vinh The Lam, Carlo Contavalli, and Amin Vahdat. Carousel: Scalable Traffic Shaping at End Hosts. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, SIGCOMM ’17, pages 404–417, New York, NY, USA, August 2017. Association for Computing Machinery.

[68] Damon Wischik, Costin Raiciu, Adam Greenhalgh, and Mark Handley. Design, implementation and evaluation of congestion control for multipath TCP. In 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11), 2011.

[61] Leah Shalev, Hani Ayoub, Nafea Bshara, and Erez Sabbag. A Cloud-Optimized Transport Protocol for Elastic and Scalable HPC. IEEE Micro, 40(6):67–73, November 2020.

[69] William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at 16

Scale. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 283–294, April 2023.

RDMA WRITE

Throughput (Gbps)

[70] Yongtong Wu, Shaoyuan Chen, Rilin Huang, Yixuan Tan, Yinmin Zhong, Mingxing Zhang, Xin Jin, and Panpan Huang. DualPath: Accelerating Agentic LLM Inference by Harvesting Disaggregated KV-Cache Storage I/O. In Proceedings of the ACM SIGCOMM 2026 Conference, pages 16–29. Association for Computing Machinery, New York, NY, USA, August 2026.

35 30 25

WRITE limit: PSN window = 512

20 READ limit: Inflight requests = 16

15 10

0 200

300

400

500

600

700

800

Out-of-order degree (packets)

Figure 24: ConnectX-7 throughput degrades under excessive reordering.

[71] Yiran Zhang, Qingkai Meng, Chaolei Hu, and Fengyuan Ren. Revisiting Congestion Control for Lossless Ethernet. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 131–148, 2024.

host memory and thus avoids an on-NIC payload buffer. The switch routes selected RoCEv2 packets through artificially delayed paths to vary OOD. We enable Selective Repeat and drop NAKs at the switch, allowing the receiver to accept out-of-order packets.6 Figure 24 shows that RDMA READ throughput drops sharply when OOD exceeds approximately 200 packets; RDMA WRITE exhibits a similar cliff near 500 packets. This occurs because ConnectX-7 limits each queue pair to 16 outstanding READ operations and a 512-packet outstanding PSN range for WRITE [36, 49, 50]. Once either limit is reached, the NIC stalls further transmission. It exhibits the metadata resource constraint for an RNIC. Moreover, the tolerance to reordering also depends on the network BDP. The outstanding PSN window for RDMA WRITE must accommodate both normal inflight packets and reordered packets. Therefore, in high-speed datacenter networks with large BDPs, a substantial portion of this window is consumed by normal inflight, leaving less room for packet reordering. Our testbed uses a smaller BDP (40 Gbps × 10 µs), allowing the outstanding window to be primarily used for absorbing reordering. However, network with a 200-Gbps line rate and a 40-µs RTT has a BDP of approximately 244 packets, consuming nearly half of the 512-packet PSN window before accounting for reordering.

[72] Yiran Zhang, Qingkai Meng, Yifan Liu, and Fengyuan Ren. Revisiting Congestion Detection in Lossless Networks. IEEE/ACM Transactions on Networking, 31(5):2361–2375, October 2023. [73] Zhaochen Zhang, Feiyang Xue, Rui Ning, Keqiang He, Gianni Antichi, Jiaqi Gao, Zhimeng Yin, Kexin Liu, Rui Li, Zhengqi Cui, Zhehao Lin, Peirui Cao, Guihai Chen, and Chen Tian. OSCAR: O(1)-Step convergence and readily-deployable congestion control. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pages 543–570, Renton, WA, May 2026. USENIX Association. [74] Junlan Zhou, Malveeka Tewari, Min Zhu, Abdul Kabbani, Leon Poutievski, Arjun Singh, and Amin Vahdat. WCMP: Weighted cost multipathing for improved fairness in data centers. In Proceedings of the Ninth European Conference on Computer Systems, pages 1–14, Amsterdam The Netherlands, April 2014. ACM. [75] Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. Congestion control for large-scale RDMA deployments. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIGCOMM ’15, pages 523–536, New York, NY, USA, 2015. Association for Computing Machinery.

B

Analysis of OOD and RBO

We derive the OOD, the maximum OOD estimate, and the corresponding RBO used in §3. Packets are transmitted in increasing PSN order. All sending rates below are measured in packets per unit time. OOD derivation. Consider packet i, transmitted at time ti on path pi , that is the next expected packet immediately before its arrival. A later packet j overtakes it if

APPENDIX A

RDMA READ

40

Resource limits in ConnectX-7

t j > ti

To further explore the resource limits in RNICs supporting DDP [51], we measure two hosts equipped with ConnectX-7 NICs and connected through a Tofino2 switch over 40-Gbps links. ConnectX-7 directly places out-of-order payloads into

and t j + D p j (t j ) < ti + D pi (ti )

(9)

6 ConnectX-7 requires an NVIDIA Spectrum switch to enable adaptive routing, which supports out-of-order reception without NAKs. Dropping NAKs at the switch approximates this behavior in our testbed.

17

Let j be the highest-PSN packet that overtakes packet i. Immediately before packet i arrives, OOD equals j − i. Since PSNs increase in transmission order, this gap counts all packets transmitted during (ti ,t j ], including those that have not yet arrived. Using the aggregate sending rate R(u) gives OOD = j − i ≈

each path’s window gives  RBO = j > i : t j + D p j < ti + D pi ≈ ∑

∆D(t) = max D p (t) − min D p (t) p∈P

These packets form a subset of the PSN span counted by OOD, so RBO ≤ OOD. It also motivates our packet scheduler design.

(11)

To obtain the maximum overtaking window, consider packet i transmitted on a maximum-delay path. The overtaking condition then requires t j − ti < ∆D(t). Let g be the maximum gap between consecutive transmissions on a minimum-delay path over this interval. When g < ∆D(t), a packet on that path is transmitted within g before ti + ∆D(t) and arrives before packet i. Thus, the maximum overtaking window satisfies max (t j − ti )

< ∆D(t)

(12)

In high-throughput transport, g is negligible relative to ∆D(t), so the maximum overtaking window is approximately ∆D(t). Letting R̄ denote the average aggregate sending rate over the window, Equation (10) gives OODmax ≈ R̄∆D(t)   = R̄ max D p (t) − min D p (t) p∈P

⇐⇒

t j − ti < D pi − D p j

Proofs of the Propositions

C.1

Model and Assumptions

C.2

Proof of Fairness

Proof of Proposition 1. Both probing and fast-correction are temporary states for ARCC when the set of flows remains unchanged, and its operation uses the EWMA window update described in §4.2.1:

(13)

W f [k + 1] = (1 − γ)W f [k] + γ(T f r̂ f [k] + q f [k + 1])

(16)

During these normal control epochs, the inflight matches the window fast, and we have

This is the estimate in Equation (3). RBO derivation. Consider packet i that is the next expected packet at the receiver immediately before its arrival. At that instant, RBO counts the later packets that have already arrived and are buffered waiting for packet i. Consider steady-state paths with constant forward delays D p and sending rates r p . A later packet j, transmitted at time t j > ti on path p j , overtakes packet i if t j + D p j < ti + D pi

C

Consider a bottleneck of capacity C. We assume each competing flow has one logical path mapped to the physical path passing this bottleneck. In practice, if a flow has multiple logical paths mapped to it, this proof treats them as single flows because ARCC controls per-path windows. We assume an ideal fluid model with a single bottleneck. We write W f , q f , Q f , r̂ f , and T f for the corresponding per-path variables of flow f defined in §4.2.1. Since ARCC updates windows at fixed intervals, we use index k to represent the state after the k-th update, e.g., W f [k]. Flows that share the same bottleneck experience the same queueing delay d. Let I f denote pktInfl, and use a for the additive increase step size as a = q40 .

j overtakes i

p∈P

(15)

p∈P p̸= pi

This yields Equation (2) in the main text. Maximum OOD estimation. Assume path delays remain approximately constant over the overtaking window, and denote the maximum inter-path delay difference by

∆D(t) − g ≤

r p du

= ∑ r p (D pi − D p )+

(10)

ti

p∈P

i

p∈P ti p̸= pi

Z tj

R(u) du

Z ti +(D p −D p )+

W f [k] = (T f + d[k])r̂ f [k]

(17)

We assume every H consecutive updates contain at least one additive-increase update, and flows that share the same bottleneck observe the same queueing delay. Margin converges. Define the normalized disagreement as Φ[k] = 1 −

(14)

min f q f [k] max f q f [k]

(18)

All contenders use the same multiplicative-decrease factor when d[k] > τq :

The overtaking window on path p therefore has length (D pi − D p )+ , where (x)+ = max(x, 0). The RBO immediately before packet i arrives is the number of packets that overtake it. Approximating this count by integrating the sending rate over

q f [k + 1] = 18

τq q f [k] d[k]

(19)

Such an update leaves Φ unchanged. For an additive-increase update q f [k + 1] = min{q0 , q + a}, for 0 < v ≤ u ≤ q0 , we have   min{q0 , v + a} a  v (20) 1− ≤ 1− 1− min{q0 , u + a} q0 u

This update is a convex combination of the current estimated q f [k+1] rate and the target q̄[k+1] r̄[k]. When the margins q f are equal, every flow moves toward r̄[k]. Convergence rate. The mixing coefficients satisfy 0<η≜

Since a = q0 /4, every additive-increase update contracts Φ by at least a factor of 3/4. Therefore,  ⌊k/H⌋ 3 Φ[0] Φ[k] ≤ 4

It means that the margins converge to fairness, and the margin disagreement decreases geometrically. Window and rate dynamics. Under the relation in Equation (17), the window update gives (22) (23)

Since ∑ f r̂ f [k + 1] = ∑ f r̂ f [k] = C, summing Equation (23) yields

Consequently,

q f [k + 1]

d[k + 1] − (1 − γ)d[k] = γ

max r̂ f [k + 1] − min r̂ f [k + 1] f f   2Cq0 ≤ (1 − η) max r̂ f [k] − min r̂ f [k] + Φ[k + 1] f f q− (32) Combining Equations (21) and (32) yields

∑ Tf + d[k + 1] f

r̂ f [k] ∑ Tf + d[k + 1] f

(24)

Define the weighted averages

r̄[k] =

f

1 ∑ Tf + d[k + 1] f

max r̂ f [k] − min r̂ f [k] = O(ρk )

q f [k + 1]

r̂ f [k]

∑ Tf + d[k + 1]

∑ Tf + d[k + 1] , q̄[k + 1]

=

f

f

(33)

f

1

where ρ satisfies

∑ Tf + d[k + 1] f

(25)

 1/H ) 3 ρ = max 1 − η, 4

(26)

Because ∑ f r̂ f [k] = C, every flow’s measured ACK rate therefore converges geometrically to C/N, i.e., every flow’s throughput converges geometrically to C/N.

(

Equation (24) then becomes d[k + 1] − (1 − γ)d[k] = γ

(30)

where q− > 0 and d+ < ∞ denote a lower bound on q f [k] and an upper bound on d[k], respectively. The former follows from the bounded interval between additive increases, and the second one from the network queue bound. If all margins have converged to the same value, Equation (29) moves every r̂ f [k] toward the same weighted mean r̄[k]. This update contracts the rate range by a factor of at most 1 − η. For unequal margins, the deviation from this common target satisfies q f [k + 1] q0 −1 ≤ Φ[k + 1] (31) q̄[k + 1] q−

(21)

(T f + d[k + 1])r̂ f [k + 1]
= T f + (1 − γ)d[k] r̂ f [k] + γq f [k + 1]

γq− ≤ η f [k] < 1 C(maxg Tg + d+ )

q̄[k + 1] r̄[k]

(34)

Substituting this identity into Equation (23) gives r̂ f [k + 1] = r̂ f [k] +

d[k + 1] − (1 − γ)d[k] q f [k + 1]r̄[k] ( − r̂ f [k]) T f + d[k + 1] q̄[k + 1] (27)

Define the coefficient η f [k] =

d[k + 1] − (1 − γ)d[k] T f + d[k + 1]

(28)

Equation (27) can then be written as r̂ f [k + 1] = (1 − η f [k])r̂ f [k] + η f [k]

q f [k + 1] r̄[k] q̄[k + 1]

(29)

19

C.3

Proof of the Out-of-Order Degree Bound

Proof of Proposition 2. We consider a bottleneck with at most Nl controlled flows and an isolated batch of at most NS starting flows. Let Q[k] = Cd[k] denote its physical queue occupancy. Define Tmin = min f T f and Tmax = max f T f . We assume γ ≥ 1/2. Queue bound with controlled flows. We first consider a control phase with a fixed set of at most Nl flows and use Q[k] to represent the bottleneck queue after the k-th update. Flows can leave but do not join in this phase. We assume the inflight matches the window fast enough. Combining equation (17)

and the window update algorithms, we get Q[k] = ∑(W f [k] − T f r̂ f [k])

Substituting into Equation (39) yields (35)

U[k + 1] ≤ (1 − γ)U[k] +

f

Q[k + 1] = (1 − γ)Q[k] + γ ∑ q f [k + 1] + ∑ T f (r̂ f [k] − r̂ f [k + 1]) f

f

When the bottleneck queue is empty, the queue bound holds trivially. We therefore consider epochs with a nonempty queue, during which the work-conserving bottleneck is fully utilized and ∑ f r̂ f [k] = C. Define U[k] as follows and combine it with equation (17): W f [k] Q[k] r̂ f [k] −C = T C ∑ f f f Tf

and

q f [k + 1] Tf f

U[k + 1] = (1 − γ)U[k] + γ ∑

(37)

Q[k] ≤ (38) (39)

U≤

! q f [k] Nl a ∑ Tf −U[k] + Tmin f (40) The margin is initialized as q f = a = q0 /4, and the first update uses the direct window correction described in §4.2.1. After the first update

f

U[1] ≤

1 − γ Nl a γ Tmin

(42)

q f [k]

1 − γ Nl a γ Tmin

(43)

f

Cτq + Nl a + NSW0 Cτq + (Nl + NS )a , U[k] ≤ max Tmin Tmin Cτq + Nl a + NSW0 = Tmin

(49)

 (50)

Since W0 ≥ a = q0 /4. Therefore, throughout the modeled startup episode, Q[k] ≤

Tmax (Cτq + Nl a + NSW0 ) Tmin

(51)

OOD bound. We finally translate the queue bound into a bound on the OOD caused by inter-path queueing-delay differences. We assume that each path traverses at most one congested bottleneck of capacity C. For any considered flow, its aggregate sending rate is at most Rmax . Therefore, the upper bound on the out-of-order degree is

When U[k] ≤ Cτq /Tmin , Equations (39) and (43) give   1 − γ Nl a Nl a U[k + 1] ≤ U[k] + γ + γ Tmin Tmin (44) Cτq + Nl a ≤ Tmin

OOD ≤

When U[k] > Cτq /Tmin , Equation (38) implies Q[k] > Cτq , so all flows perform MD. Moreover, q f [k + 1] Cτq q f [k] = ∑ T Q[k] Tf f f f   Cτq 1 − γ Nl a U[k] + ≤ TminU[k] γ Tmin

Cτq + Nl a + NSW0 Tmin



By mathematical induction,

∑ Tf −U[k] ≤

(48)

Applying Equation (47) to the post-startup stages gives

(41)

Cτq + Nl a Tmin

Tmax (Cτq + Nl a) Tmin

Queue bound when flows join. Adding at most NS flows’ initial windows increases U by at most NSW0 /Tmin . Thus, immediately after their admission,

q f [k + 1] ∑ Tf −U[k + 1] ≤ (1 − γ) f

q f [1]

(47)

Then

Each update increases q f by at most a, so

∑ Tf −U[1] = 0 ≤

Cτq + Nl a Tmin

U[k] ≤

We have U[k]Tmin ≤ Q[k] ≤ U[k]Tmax

(46)

The right-hand side h is convexiin U[k]. For γ ≥ 1/2, its values Cτ Cτ +N a on the interval Tminq , Tqmin l are at most (Cτq + Nl a)/Tmin . Above this interval, it is no larger than U[k]. Combining the two cases therefore gives

(36)

U[k] ≜ ∑

γCτq (1 − γ)Cτq Nl a + 2 U[k] Tmin Tmin

  Rmax Tmax Nl q0 Cτq + + NSW0 . C Tmin 4

(52)

This is a conservative worst-case bound. Equality is achieved only when one path of a flow reaches the maximum queue bound while another path has zero queue. As the number of competing flows increases, this worst-case condition becomes less likely to occur. Meanwhile, with more competing flows, the likelihood that an individual flow transmits at a high rate also decreases.

∑

(45)

20

C.4 Proof of the Packet Scheduling Correctness

selective repeat. TSC-based timers trigger ARCC updates at fixed intervals, which update the per-path windows and the flow-level pacing rate. For each transmission, deficit-based path selection selects the logical path and then packet scheduling decides the packet to transmit. The transport changes the UDP source port field to select the logical path for packets, and transmits packets through rte_eth_tx_burst(). To avoid reverse-path congestion from distorting ACK-rate measurement, ACK packets use the highest priority. Overhead optimizations. We estimate arithmetic overhead based on [59], using a cost model that assigns 1 cycle to an integer addition or subtraction, 3 cycles to an integer multiplication, and 16 cycles to an integer division. We apply two optimizations to reduce this overhead. First, we separate per-packet operations from periodic control operations. ARCC window updates and per-path skip calculations are performed periodically off the critical path. Second, divisions by T and Tu are replaced with multiplications by precomputed reciprocals, and the normalization division in deficit-based path selection is replaced with a shared reciprocal table. After these optimizations, for P paths, path selection requires 6P arithmetic cycles, and selecting the target packet using a precomputed skip requires 1 cycle. Every Tu , ARCC requires approximately (35–45)P cycles, while precomputing cycles. the per-path skips requires at most 5P(P−1) 2

Proof of Proposition 3. Consider all transmission opportunities of a flow before transmission starts. Since each path has a fixed sending rate and forward delay, the transmission time, path, and arrival time of every opportunity are known. Consider an ideal scheduler that sorts these opportunities by their arrival times and assigns consecutive PSNs in this order. This defines a permutation of the PSNs under which packets arrive in order. We show that Rondo reproduces this permutation online. Consider the k-th transmission opportunity, on path p at time t. Its global rank in the ideal scheduler is one plus the number of all opportunities that arrive before it. By the k-th transmission, the first k − 1 opportunities have already consumed their assigned PSNs. Among them, only those that arrive before the current opportunity have smaller PSNs in the ideal assignment. Therefore, the rank of the current opportunity among the remaining unsent PSNs is one plus the number of future opportunities that arrive before it. A future opportunity on path p′ , transmitted at t ′ > t, arrives before the current one if t ′ + D p′ < t + D p

(53)

t ′ − t < D p − D p′

(54)

or equivalently

E

Under the fluid model, path p′ contributes r p′ (D p − D p′ )+ such opportunities. Thus, after rounding the fluid-model result to an integer packet count, the rank of the current opportunity among the remaining opportunities is         +  1 +  ∑ r p′ (D p − D p′ )  (55)

Rondo provides fairness for flows with different RTTs. FRCC [2] shows that fairness is difficult to achieve among flows with different base RTTs. In DCNs, base RTT differences between flows are generally limited, as they primarily stem from differences in path hop count. Figure 25 shows that Rondo keeps fairness among three flows sending to the same receiver from different locations. One sender is in the same rack as the receiver, another is in the same pod, and the third is in a different pod. This result is consistent with Proposition 1. Comparison to EQDS. EQDS [52] is a receiver-driven transport, where the sender transmits data packets only after receiving credits. We compare Rondo with EQDS in Figure 26, with EQDS implemented in RTS mode without packet trimming support. EQDS controls OOD well under incast, but is less effective at handling congestion in the fabric. Therefore, its performance degrades under 4:1 oversubscription and all-to-all traffic. In comparison, Rondo does not face these limitations. Rondo reacts quickly to bandwidth changes. Figure 27 shows how Rondo reacts when the link bandwidth drops to one quarter and later recovers. There are 4 long flows in this experiment. When the bandwidth decreases, transient queue buildup is unavoidable, but ARCC quickly estimates the new capacity and drains the queue without underutilizing the link.

p′ ∈P p′ ̸= p

This is exactly the (skip p + 1)-th smallest unsent PSN selected by Rondo. Therefore, whenever all previous transmission opportunities match the ideal permutation, the current one also receives the same PSN. Starting from the first transmission opportunity, induction shows that Rondo reproduces the entire ideal permutation online. Hence, packets arrive in increasing PSN order and OOD = 0

D

Additional Evaluation

(56)

Implementation Details and Optimizations

Implementation details. In our DPDK prototype, the basic multipath transport supports per-packet path selection, per-packet ACKs, bitmap-based packet arrival tracking and 21

100

ECMP

Flow 2 (via spine)

MP-RDMA

UET

AR

50 25 0

100 200 300 400 500 0

Relative JCT NIC OOD (packets)

100 200 300 400 500 0

200

EQDS (RTS)

1.2 0.8 0.4

Rondo

100 200 300 400 500 0

100 200 300 400 500 0

solid: p99 / pale: max

OS1

OS4

All-to-All

OS1

OS4

Incast

OS1

OS4

100

50 Probing triggers

0 0

Permutation

1

2

100 200 300 400 500

Datacenter load balancing. Datacenter load-balancing schemes operate at different granularities. Flow-level loadbalancing methods [4, 23, 74] cannot fully utilize the network with a small number of active flows. Flowlet-level schemes [5, 30,55,66] require sufficiently large inter-packet gaps to switch paths, which are scarce in the bursty traffic of ML workloads. Packet-level load balancing [13, 18, 21, 24, 35, 38, 41, 53] can better exploit path diversity, but also introduces packet reordering. Packet-level load balancing schemes can be further classified based on their congestion awareness. Congestionoblivious methods [18, 38, 41] do not adapt traffic allocation to path conditions and lead to load imbalance [53]. Locally congestion-aware methods [21] make forwarding decisions using switch-local signals, enabling fast reactions but with limited visibility of end-to-end path conditions. End-to-endcongestion-aware methods [13,35,53,63] make path-selection decisions at end hosts based on end-to-end congestion signals, but still introduce substantial packet reordering. Rondo performs packet-level multipathing based on end-to-end congestion information while keeping the resulting reordering controllable. MP-RDMA [42] introduces OOD-aware path selection, which prunes paths with late ACKs to avoid excessive disorder. But under a high inter-path queue difference, this method frequently prunes paths, resulting in severe oscillations. Even under a persistent delay difference, it hurts utilization. Schedulers. DAPS [32] constructs packet/path schedules heuristically from RTT ratios and congestion windows. However, its coarse schedule cannot precisely mitigate reordering. It is limited to low-rate scenarios with stable congestion windows, and its behavior perturbs the network state established by congestion control. STMS [62] dynamically adapts sending packets allocated to different paths. It focuses on two-path scenarios, tentatively and reactively adjusts a transmission gap based on observed OOD. It faces a trade-off between slow convergence and gap overshoot.

Falcon

Bandwidth reduced to 1/4

100

0

0.0 900 600 300 0

100 200 300 400 500 0

Time (ms) Figure 25: Throughput curves when three flows with different base RTTs step in and out. Queue (KB)

0

Rondo

3

Time (ms)

Figure 26: Comparison between Figure 27: Rondo conRondo and EQDS under typical verges quickly when bandtraffic. width changes. When the bandwidth recovers, capacity probing enables rapid ramp-up without overshooting.

F

Flow 3 (via core)

Falcon

75

Utilization (%)

Throughput (Gbps)

Flow 1 (same ToR)

Rondo

Discussions on Related Works

Congestion Control. Sender-driven CCs [1, 10, 14, 15, 33, 37, 45, 71, 73, 75] mainly focus on legacy single-path transport, while Rondo is designed for multipath transport. Receiverdriven congestion control schemes [22, 25, 47, 52] mitigate performance degradation caused by incast but can not directly capture congestion within the network fabric. Rondo’s ACKbased method provides similar fine-grained control while capturing congestion in both the fabric and the last hop. ACC [71] observes the ACK rate and uses it to determine the stall time after congestion, achieving fast convergence in congestion. But it still relies on TCD [72] to infer congestion, tentatively increases its rate to utilize spare bandwidth, which results in slow rate increasing and queue oscillations. Rondo uses ACK rate throughout its control loop and maintains stable queues. OSCAR [73] achieves O (1) convergence for both rate increasing and reduction. However, it needs a sufficient queue for measuring delay gradient, which introduces a large interpath queue difference in multipathing. This method also cannot work when flows start up. Aeolus [25] allows senders to transmit some low-priority data without credit during the first RTT for probing. However, these packets use the same queue as normal packets, causing a delay difference for existing flows. It can obtain receiver credits, but cannot capture congestion information within the fabric before being dropped. Moreover, these packets are more likely to be dropped, increasing the disorder. The combination of probe packets and ACK-rate-based CC in Rondo addresses both problems. 22

Related documents

• Clear-Weighted Bit Allocation for Satellite Downlinks#423900
• Programmable Packet Scheduling with Dynamic Reordering at Line Rate#10301
• Bidirectional Forwarding Detection (BFD) Encapsulated in Large Packets#144359
• Anomaly Detection Using Network Traffic Characterization#1027285
• Baseline Encoding and Transport of Pre-Congestion Information#501130