CCAaaS: An Orchestration Platform for High-Velocity Design, Development, and Deployment of Data Center Transports at Meta

CCAaaS: An Orchestration Platform for High-Velocity Design, Development, and Deployment of Data Center Transports at Meta Prankur Gupta Meta

Balasubramanian Madhavan Meta

arXiv:2610.33231v1 [cs.NI] 23 Oct 2026

Neil Spring Meta

Rajiv Krishnamurthy Meta

Miao Xu Meta

Prashanth Kannan Meta

Srikanth Sundaresan Meta

Jie Meng Meta

Martin Lau Meta

Theophilus A. Benson Carnegie Mellon University Abstract Today, data center networks run multiple congestion control algorithms (CCAs), each tailored to distinct traffic characteristics, from latency-sensitive intra-region RPCs to highthroughput cross-region AI workloads. The rapid evolution of AI training and inference further accelerates this trend by continually introducing new traffic characteristics that demand new CCAs. This paper introduces CCA-as-a-Service (CCAaaS), our eBPF-based ecosystem for the rapid design, deployment, and operation of production-grade CCAs at hyperscale. We present (1) a modular BPF-TCP framework and general lessons for designing modular CCAs; (2) a deployment and orchestration layer that provides disruption-free CCA updates, along with best practices for managing a zoo of CCAs at scale; and (3) a lightweight observability tool for CCAs and a blueprint for scaling CCA observability. CCAaaS manages over 97% of our server-to-server traffic using three active eBPF-based CCAs; two of the five CCAs we have built have since been deprecated. We show that our eBPF-centric approach to CCAs development delivers the same applicationlevel performance (retransmissions, bandwidth, and tail latency) as their native in-kernel counterparts, and that our custom CCAs add negligible CPU and memory cost over their eBPF baselines. In terms of realized benefits, the CCA built with CCAaaS significantly improved network, and application metrics, for example, one of our CCAs reduced P99 latency for cross-region services by up to 20×.

1

Introduction

Large-scale data-center networks must simultaneously satisfy competing objectives, including low latency, high throughput, and high utilization, for a constantly evolving set of applications [2, 9, 18]. For example, latency-sensitive RPCs for end-user services (e.g., News Feed, Ads ranking) demand predictable tail latency [2], while AI training workloads (e.g., checkpointing and large-scale data replication) require high throughput [8]. In such environments with competing objec-

tives, a single congestion control algorithm (CCA) is insufficient to meet all requirements. As a result, modern hyperscaler networks simultaneously deploy multiple CCAs, each optimized for a specific class of infrastructure, class of traffic, or application characteristics. While prior work has described the deployment of individual CCAs for varying objectives [2, 9, 15, 18, 24, 28, 30, 38], in practice, many of these operators actively manage a growing portfolio of CCAs that coexist within the same infrastructure and must evolve independently over time to address shifting priorities and emerging use cases. Yet, few papers have discussed the systems challenges of developing, operating, and evolving a heterogeneous set of CCAs in production. We believe discussing these challenges is important because we see, in our infrastructure, the trend of rapid CCA development and deployment being accelerated by AI workloads, which rapidly introduce new traffic patterns and use cases that significantly shortens the lifecycle from CCA conception to production deployment (fig. 1). In this paper, we present CCAaaS, a culmination of our experience pursuing in-kernel programmability via eBPF, which shows that operating CCAs raises challenges and aspect of CCA design generally overlooked by recent eBPF-based TCP CCA frameworks [6, 33]: rollout safety, rollback correctness, debugging, and observability. We note that more general production CCA work focus on individual CCA design and not the challenges of managing co-existence [2, 9]. Our experience designing CCAaaS has highlighted four common challenges: • C1, Rapid iteration under tight performance constraints: CCA development demands fast experimentation, but congestion control executes on the critical datapath, where even small abstraction overhead is unacceptable. • C2, Safe and incremental rollout: Certain CCAs require both endpoints to be upgraded (e.g., for ECN negotiation), which complicates rollout. Moreover, new logic must apply retroactively to existing connections, since long-lived flows (e.g., AI training) would otherwise run stale CCA logic until they are reset.

DCNDelay-CCA (intra)

XRegion-CCA (cross)

Coverage (%)

100 97.5

XRegion: 14 regions in 3.7 months

80

94.7

60 DCNDelay: 35% → 94% in 4 months

40 20 0 0

2 4 6 8 10 Months since rollout start

Now

Figure 1: Rollout timelines for our two most recent CCAs: DCNDelay-CCA is rolled out per region, while XRegionCCA is rolled out per region. CCA

Dev. Period

Deploy Period

Status

Best Effort CC BPF-DCTCP [9] DCNDelay-CCA (section 4.3.1) XRegion-CCA (section 4.3.2) BPF-Cubic

6 months 6 months 9 months 6 months 4.5 months

1–2 years 1 year 10 months 6 months 3 months

Deprecated Deprecated Active Active Active†

Table 1: CCAs developed and deployed at Meta using the CCAaaS framework. † At the time of writing, BPF-Cubic was still deployed, but we were actively reducing its usage as XRegion-CCA supersedes it. • C3, Timely and correct rollback: Rollback is complicated by sticky transport state: once ECN is negotiated, it persists even after the CCA changes, and fallback CCAs may misinterpret it. • C4, Lightweight, CCA-aware observability: Operators need visibility into CCA-specific internal state to validate changes and diagnose incidents, but existing tools are too heavyweight (e.g., tcpdump) or too rigid (e.g., Web10G [29]). In this paper, we describe our experience building and operating CCA-as-a-Service (CCAaaS), an eBPF-based ecosystem that enables rapid development, safe evolution and deployment of CCAs coexisting atop the same production infrastructure. With CCAaaS, developers design CCA by implementing programs specified by BPF-TCP hooks, and the ecosystem handles its orchestration (e.g., , rollout, rollback) and monitoring. Our design of CCAaaS builds on three insights: First, to ensure efficiency and performance, CCA interfaces should be specific rather than generic and allow control over various layers to the network stack beyond CCA core control loop. Second, CCA deployment (or rollback on failures) must explicitly account for transport related state or functionality provided by the network, such as negotiated ECN. Third, to scalably observe CCAs, observability should be co-designed with CCA control loop behavior and codebase. CCAaaS builds on NetEdit [3], our production eBPF orchestrator, and extends it to provide CCA-specific support (e.g., CCA-specific state sharing). CCAaaS has been in production for over five years, has been used to develop five CCAs (table 1), and currently man-

Figure 2: CDF of minimum RTT by traffic class: intra-region versus cross-region flows. ages over 97% of server-to-server traffic at Meta through custom eBPF-based CCAs. Our two most recent CCAs provided significant production wins: DCNDelay-CCA reduced retransmissions for major services by 30–40%, while XRegion-CCA, reduced cross-region service P99 latency by up to 20×. We also identify several key lessons for CCA developers and infrastructure administrator: our lessons range from CCA management to eBPF orchestration. This paper makes the following contributions: • Modular BPF-TCP framework (C1): We introduce an eBPF-based framework for developing modular CCAs and evolve it to provide control over other important functions and configurations within the kernel networking stack (section 4) • CCA deployment and orchestration ecosystem (C2 and C3): We extend an existing production-grade eBPF orchestration system to support CCAs, introducing primitives for safe rollback and coordination primitives for safe, disruption-free rollout (section 5). • CCALogger (C4): We provide the first lightweight and comprehensive CCA observability framework, which scales by co-designing its observability primitives with BPFTCP’s modular decomposition and lifecycle (section 6). • Deployment lessons: We distill a set of lessons from the challenges we have faced in designing CCAs, managing CCAs, and evolving eBPF to support an ecosystem of eBPFbased CCAs (section 7).

2

Motivation

In this section, we describe our production environment to highlight why we run multiple CCAs (section 2.1) and to justify our choice of eBPF for building our CCAs (section 2.2). The data in this section were collected across our fleet via FBFlow and queries to our infrastructure database, and span three months in 2026. 2.1

The Production Case for Multiple Simultaneous CCAs Heterogeneous Traffic Conditions. Modern hyperscale networks must support applications with fundamentally different traffic characteristics and requirements. Figure 2 illustrates the gap between the RTTs of intra-region flows (median ∼70 µs)

NIC models

% of hosts 100

36 80

28 24

60

20 16

40

12 8

Hosts (%)

20

4

XRegion-CCA

DCNDelay-CCA

Unclass.

2.2% 1.9% 4.5% 3.9%

11.7% <0.1% <0.1% 94.7%

85.6% 97.5% 95.2% <0.1%

0.5% 0.4% 0.3% 1.2%

0 Other

25

Cubic

Other

XRegion-CCA

Cubic

Other

Share of traffic (%)

100

99 98 97 96

95 90 85

(a) Intra-region traffic

(b) Cross-region traffic

(DCNDelay-CCA).

(XRegion-CCA).

R15

R14

R13

R12

R9

R11

R8

R10

R7

R6

R5

R4

R3

R2

R1

Fleet

R15

R14

R13

R12

R9

R11

R8

R10

R7

80 R6

95 R5

Figure 3: Top-3 NIC vendors by deployment fraction. We use a predefined OCP NIC configuration, but different vendors supply the components; a model is a set of NICs with the same vendor and hardware combination.

DCNDelay-CCA

100

R4

NIC vendor

R2

Vendor C

R1

Vendor B

Fleet

Vendor A

Switches (%)

BPF-Cubic

<0.1% <0.1% <0.1% 0.1%

R3

0

BBR

All traffic Intra-region Intra-DC Cross-region

Table 2: Fleet-wide distribution of CCA usage for server-toserver traffic (by bytes/s). Over 97% of traffic uses custom eBPF-based CCAs. Share of traffic (%)

NIC models

32

Scope

Figure 5: Per-region distribution of CCA usage for server-toserver traffic (by bytes/s). Note the truncated y-axes.

20 15 10 5

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21

0

Switch model

Figure 4: Distribution of switches deployed at Meta. and cross-region flows (median ∼32 ms). This difference of nearly three orders of magnitude translates into distinct CCA requirements, and it is precisely why we run multiple CCAs across our fleet (table 2). Drawbacks of ECN for Intra-Region Traffic. We recently showed that within a region, congestion (and loss) generally occurs at end hosts and in NIC buffers rather than in network switches [11]. However, only switches generate ECN marks, so existing ECN-based CCAs cannot accurately capture this congestion. Additionally, our data centers comprise multiple hardware generations across NIC vendors and switch platforms (figs. 3 and 4), and each generation introduces different queuing and ECN behavior into our network. Thus, selecting ECN configurations that effectively support both incast workloads and bulk transfers is challenging. Because of this configuration complexity and ECN’s limited visibility into host-side congestion, we opt for a delay-based CCA for intra-region traffic (section 4.3.1). The Shift Toward ECN for Cross-Region Traffic. Although we planned to move intra-region traffic to a delaybased CCA, our experience with delay on cross-region traffic showed that delay alone was an insufficient signal: changes in cross-region delay stem either from congestion on crossregion links or from our traffic-engineering system rerouting traffic. We also observed that loss-driven CCAs, e.g., Cubic, perform poorly for our long-lived AI workloads because they react to congestion too late, an observation shared by others.

Thus, while we use delay for intra-region traffic, we design our cross-region CCA around ECN. The Need for Host-Level CCA Coexistence. Although we design our CCAs with different objectives, intra-region and cross-region flows must coexist on the same hosts, NICs, and rack switches. Consequently, aggressive behavior by one class of flows may starve or otherwise harm the other. Moreover, each CCA deployment introduces constraints that limit future options. For example, when we first deployed DCTCP, we could easily tailor ECN marking configurations to it; however, these settings became difficult to maintain once we introduced other CCAs. Table 2 and fig. 5 show the current distribution of CCAs across our fleet. While our two custom CCAs (XRegion-CCA and DCNDelay-CCA) dominate at the fleet level, usage varies across regions, particularly for cross-region traffic, XRegion-CCA requires a kernel upgrade and we are unable to upgrade a subset of hosts and so they still use BPF-Cubic. Managing this portfolio poses two challenges: designing algorithms that coexist safely, and choosing deployment constraints and configurations that leave room for future, unanticipated CCAs. 2.2 Why eBPF for Congestion Control These challenges call for a mechanism for building CCAs that matches the performance of existing in-kernel CCAs while being flexible enough to deploy without disrupting our applications. Kernel Modifications. Given our long history of upstreaming kernel patches, an obvious choice is to modify the kernel directly. However, while this approach provides full flexibility, it is operationally infeasible at our iteration speed: kernel patches are slow to merge, and deploying them requires host reboots that disrupt traffic. Kernel Modules. Although kernel modules are less disruptive than direct kernel modifications, they have two key

drawbacks. First, they are tied to the kernel version they are built for, which complicates kernel upgrades. Second, in certain scenarios, replacing them requires rebooting hosts. Userspace Stacks. Others have shown that moving the network stack, including the CCA, to userspace [22, 37] enables rapid iteration; however, this comes at the cost of rewriting the network stack and often requires specialized hardware (e.g., custom NICs). Moreover, these userspace stacks often sacrifice the maturity and efficiency of the kernel’s TCP implementation. eBPF occupies a sweet spot in this design space: eBPF programs execute in the kernel, and thus provide kernel-level performance, yet they can be loaded and replaced at runtime without reboots. Moreover, the eBPF verifier enforces checks that prevent a broad class of safety issues, such as unbounded loops and invalid memory accesses. Because of this combination of kernel-level performance and continuous delivery, we chose eBPF as the substrate for our CCAs. Since making this choice, we have reimplemented Cubic and DCTCP in eBPF and created new bespoke CCAs (table 1).

3

CCAaaS Overview

Our philosophy is to treat CCA development much like microservice development [23]: modular, rapidly developed, and continuously deployed. The CCAaaS ecosystem (fig. 6) is explicitly designed to support this philosophy and includes the following components: (1) a modular BPF-TCP framework for composing CCAs from reusable eBPF hookpoints (section 4); (2) a deployment and orchestration layer for disruption-free rollout and rollback (section 5); and (3) a lightweight observability tool for CCA-specific introspection (section 6). While CCAaaS leverages NetEdit [3] for eBPF program loading and general lifecycle management, in this paper we extend NetEdit with additional CCA-specific management primitives. CCAaaS splits these components across a control plane and a data plane: Control Plane. The control plane is never involved in perpacket or per-event processing. Instead, it manages the lifecycle of the eBPF objects that a CCA requires to operate correctly. To do this, the control plane identifies which eBPF programs to deploy, installs them (and their related objects) in the kernel, and determines when to safely uninstall and remove them. The control plane also manages the policies that govern when and how a CCA behaves for a specific connection; it installs these policies in eBPF maps for the eBPF programs to use. Data Plane. The data plane consists of the eBPF objects, e.g., maps and programs, that are involved in per-packet and per-event processing.

4

Modular CCA Framework

Our experience with CCAs over the last five years has revealed that production-grade CCAs require control over the core con-

CCAaaS Ecosystem

Userspace Kernel

CCAaaS Global BPF Orchestrator (NetEDIT)

BPF_iter

CCA Selector

TCP Protocol

BPFBPF BPF BPF BPF BPF BPF Programs BPF Programs Programs BPF Programs Programs Programs Programs Programs Programs

Socket Management

BPF BPF BPF Programs Programs Programs

NIC Management TCP network stack

Scope Resolver

Config Store

BPF BPF BPF Programs Programs Programs

CCAaaS Modules

Figure 6: CCAaaS architecture. trol loop (e.g., slow start) and, more importantly, over crosslayer kernel functionality (e.g., TSO segmentation) and internal kernel configurations. Our experience has also yielded design principles for efficiently sharing state within a CCA and between CCAs and other eBPF programs. The current BPF-TCP hookpoint interface (i.e., tcp_congestion_ops) is the culmination of the lessons we learned as we refactored, evolved, and enhanced it to support our use cases. This section describes the BPF-TCP hook interface and its evolution (section 4.1), our state-sharing designs (section 4.2), and two CCAs that we recently deployed with CCAaaS (section 4.3). 4.1

The BPF-TCP Hook Interface

Linux’s BPF-TCP hookpoint, tcp_congestion_ops, is implemented as a struct_ops, which bundles a set of eBPF programs. This struct_ops provides a way to implement the kernel’s underlying CCA interface. Recall that this interface is a struct of functions: developers implement a CCA by implementing the functions exposed in the struct. BPF-TCP lets developers write a set of eBPF programs, each implementing one of these functions; table 3 lists the hookpoints that BPF-TCP exposes. We chose to design BPF-TCP as a bundle of eBPF programs within a struct_ops, rather than as independently defined BPF programs, to ensure that the different functions are always deployed and updated consistently, which is important when CCA code is constantly changing: struct_ops atomically replaces all callbacks as a unit. Unlike other attach types [3, 13] (e.g., sockops, TC, or XDP), which allow many BPF programs to operate on the same event, BPF-TCP allows only one struct_ops (i.e., the programs bundled within it) to handle a connection’s packets or events. Thus, there is no need for ordering, prioritization, or conflict-resolution mechanisms among CCAs applied to a connection (or socket), because only one is applied. 4.1.1

Defining a New CCA

To add a new CCA to CCAaaS (fig. 7), (1) the developer implements the new CCA’s logic as a set of BPF programs that implement a subset of the struct_ops interface (table 3), and (2) operators and service owners specify policies that

CCA Developer

sk_storage

New CCA as a set of eBPF progs (struct_Ops) CCAaaS Global BPF Orchestrator (NetEDIT)

Locality + Scope

Service Owner

CCA Name

PolicyStore (NetEdit Artifact)

CCA policy

Figure 7: Workflow for defining a new CCA. define which connections the new CCA should be applied to. These policies specify both the locality (i.e., cross-region versus intra-region) and the scope (i.e., the services) of the connections. Given these two inputs, CCAaaS manages the lifecycle of the CCA’s programs (i.e., installs them) and configures them (i.e., installs the appropriate configuration in their maps), which allows the CCA to be correctly applied to connections across our fleet. 4.1.2 BPF-TCP Interface Evolution Table 3 shows the hookpoints that each of our CCAs implements; the rows for R1 and R2 show how we extended the interface to support more advanced CCAs. This highlights a core limitation: developers can introduce a new CCA only if they can realize it within the current interface. Limitations. The set of CCAs that can be realized is restricted by the interface that struct_ops exposes. First, the interface supports delay, ECN, and loss signals, but it does not provide richer in-band network telemetry (INT) signals. We note that for a subset of these INT signals, the interface may provide sufficient information to reconstruct them within one of its eBPF programs. Second, the BPF-TCP paradigm enables core CCA functionality and, more recently, cross-layer functionality. However, we found that certain performancecritical socket-level functionality falls outside the core CCA logic and was thus not configurable. While we exposed a subset of this functionality, much of it remains unexposed. As we show below, the community is open to evolving this interface when specific use cases justify it. R0: Core CCA Logic. The top five hooks provide control over the core CCA logic: initialization, state management, and basic congestion response. Our initial eBPF-based CCAs, BPF-Cubic and BPF-DCTCP, used these core hooks as well as cong_avoid and in_ack_event. We were also able to extend Cubic into our transitional CCA by modifying one of its BPF programs to support ECN. R1: Efficient Per-RTT Signals and Control over SocketLevel Functionality. First, we initially designed DCNDelayCCA to infer per-RTT metrics from individual ACK events; however, this incurred significant overhead (compute to calculate per-RTT metrics and memory to maintain this state). We observed that the kernel already precomputes per-RTT metrics (e.g., min_rtt and max_send_rate), so we modified the hook signature to expose them. Second, we noticed that the kernel overrode certain decisions without the CCA’s

New Connection

Scope Resolver

CCA Selector

New Socket

CCA

Figure 8: CCAaaS pipeline for configuring the CCA for a new connection and its socket. knowledge, e.g., forcing a connection into the CWR state on a single ECN mark or imposing send-buffer growth policies. We extended the interface with a new hookpoint that provides programmatic control over this internal socket-level functionality. R2: Control over Cross-Layer Functionality. Finally, we found that we needed control over certain properties not traditionally considered CCA functionality, e.g., tcp_min_tso_segs. Our last change extends the BPF-TCP interface to provide these controls. 4.2 Cross-Program State Sharing To ensure consistent behavior across the programs within a BPF-TCP struct_ops bundle, and to enforce correct policy across our CCA programs and other eBPF programs, we introduced a broad set of state-sharing mechanisms. 4.2.1 Coordination Across Our Fleet CCAaaS dynamically configures and sets the CCA for each connection using a pipeline (fig. 8) of eBPF programs developed by several teams. Reusing these programs ensures consistent policy across teams. • First, NetEdit’s scope resolver (a sockops program) performs a longest-prefix match on the remote address to identify the connection’s locality (i.e., intra-region, cross-region, or unknown) and stores the result in an sk_storage map. • Next, NetEdit’s CCA selector (a sockops program) reads the connection’s scope from the sk_storage map and then calls bpf_setsockopt to set the connection’s CCA, assigning the appropriate struct_ops to the connection’s socket. • The CCA itself (a struct_ops program) reads the scope from the same sk_storage entry to retrieve per-locality parameters.1 • Finally, as services migrate, CCAaaS must ensure that the policies stored in the policy configuration maps2 are updated to reflect the current set of services on a host. To do this, the scope resolver repopulates the scope maps, and the 1 DCNDelay-CCA additionally consumes one-way queuing delay (OWD) measurements written per packet by a separate program owned by a different team. 2 The configuration map is keyed by source IP, destination region, and destination data center, allowing different services on the same host to use different CCAs with different parameters.

Hook

Purpose

Release

BPF-DCTCP

DCNDelay-CCA

XRegion-CCA

BPF-Cubic

init ssthresh set_state undo_cwnd cwnd_event

Initialize CCA state Set slow-start threshold React to TCP state changes Undo spurious CWND reductions React to CWND-related events

R0 R0 R0 R0 R0

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓

cong_avoid in_ack_event pkts_acked cong_control min_tso_segs

Window-based congestion avoidance Extract per-ACK congestion signals Per-RTT delivery-rate statistics Full control over CWND and pacing Override TSO segment count

R0 R0 R1 R1 R2

✓ ✓

✓ ✓ ✓ ✓

✓ ✓ ✓ ✓

✓

Table 3: BPF-TCP hooks and their usage across four CCAs. The top five hooks (R0) are used by all four CCAs; the remaining five are used selectively. CCA selector dynamically re-evaluates each new connection based on the updated scope maps. State

Producer (attach, rate)

Consumer (attach, rate)

Store

Scope CCA name Config

Resolver (sockops, once) Selector (sockops, once) Control plane (minutes)

Selector, CCA (per socket) Kernel TCP (per socket) Selector, CCA (per socket)

sk_storage socket global map

Table 4: State sharing within the CCAaaS CCA configuration pipeline. State Within a CCA struct_ops. We use the sk_storage map type because it is indexed by socket pointer; since CCA struct_ops programs have the connection’s socket pointer, CCA-related lookups take O(1) time. In a microbenchmark on a production host, we found that an sk_storage lookup costs ∼25 ns, versus ∼1 µs for an LPM scope lookup. Additionally, sk_storage maps automatically garbage-collect entries when sockets are destroyed (i.e., when connections close), which eliminates the complex garbage collection that other map types require. State Across a CCA struct_ops and Other eBPF Programs. Shared state is the only way to compose or coordinate CCA BPF programs with other BPF programs. Unfortunately, due to eBPF design constraints, sk_storage maps cannot be used by kernel modules or userspace, so we must build shared storage from other map types to coordinate across programs. In particular, NetEdit and its programs populate and manage the policy configuration maps, and CCAaaS’s CCA programs read and use this configuration. These policy configuration maps are an example of shared state that cannot be stored in sk_storage. 4.2.2 CCA-Specific State-Sharing Edge Cases Section 4.2.1 cleanly assigns different types of eBPF maps to different roles within CCAaaS; however, at scale we encountered a set of edge cases that forced us to make several compromises. Splitting Per-Connection State. In our designs, the CCAprivate (i.e., per-connection) state was much larger than the kernel’s existing data structure can hold (i.e., the 104 bytes provided by icsk_ca_priv). Consequently, we split this state across two data structures: (1) the internal CCA structure and (2) sk_storage. This split lets us support larger (and

more complex) per-connection logic, but at the cost of one additional lookup per read, because we must look up the sk_storage entry in addition to the internal structure. LPM Fallback. CCAaaS relies on one of NetEdit’s programs, the scope resolver, to populate certain maps. If these maps are unavailable, CCAaaS transparently performs an LPM lookup and caches the result in the appropriate maps for future connections. This ensures that a CCA’s correctness is independent of the scope resolver’s availability. We note that we designed NetEdit and its programs to be highly available so that this LPM fallback is infrequent, because it is expensive and may degrade application performance. 4.3

CCAaaS Use Cases

To satisfy our unique deployment constraints, we generally evolve and adapt existing algorithms (e.g., BBR, Swift, DCTCP, and Cubic). In this section, we discuss two CCAs that we recently designed and show how their designs justify extending BPF-TCP beyond the initial interface (i.e., R0). 4.3.1

DCNDelay-CCA: Delay-Based Control for Heterogeneous Data Centers We introduced a new delay-based CCA, DCNDelay-CCA, to replace DCTCP [1]. Two key factors motivated this move. First, correctly configuring ECN across our heterogeneous fleet was challenging, and the fleet’s continual evolution made configuring ECN even more time-consuming. Second, while we experienced significant congestion at the host, many host NICs do not mark ECN bits, leaving DCTCP blind to a significant source of contention. In contrast, delay provides a unified signal of queuing regardless of where it occurs (network switches, NIC buffers, or the host PCI bus) and requires minimal configuration. Algorithm. At a high level, DCNDelay-CCA maintains low queue occupancy by computing a target congestion window from the observed delay and a configured minimum RTT. When the network is free of queuing, the target window grows linearly; when queuing is observed, the algorithm reduces the target window. We deliberately chose a simpler design than other delay-based approaches (e.g., BBR) because we wanted to avoid designing additional heuristics to tailor a complex design to our diverse hardware ecosystem. DCNDelay-CCA

also uses ECN as a fallback to ensure that it is never more aggressive than DCTCP. Further details of the algorithm appear in section C. Hookpoints Used. DCNDelay-CCA’s core algorithm uses three hookpoints (table 3): in_ack_event to compute perACK ECN alpha and delay updates; pkts_acked to capture per-RTT statistics and process loss; and min_tso_segs to dynamically configure the TSO segment count so that the CCA is not bottlenecked by NIC behavior. 4.3.2

XRegion-CCA: ECN-Driven Control for CrossRegion WANs

We replaced Cubic with XRegion-CCA as the CCA for our cross-region traffic. We opted for ECN rather than delay as the cross-region congestion signal because our experience showed that cross-region delays change frequently due to our traffic-engineering system [8], which regularly rebalances virtual paths across physical links. Because this rebalancing is invisible to end hosts, an observed change in delay could be due either to congestion or to rebalancing. We therefore opted for an ECN-based CCA, which we designed to interoperate with DCNDelay-CCA’s shallow buffer occupancy and with flows from other data centers that operate at different RTTs. Algorithm Design. XRegion-CCA builds on Swift’s [18] control loop and adapts it to the ECN signal. Recall that Swift is a delay-based CCA that compares a per-RTT delay measurement against a target. When delay is below the target, Swift grows the window additively; when delay is above the target, Swift reduces the window multiplicatively, in proportion to the overshoot. XRegion-CCA replaces delay with the fraction of ECN-marked ACKs and compares it against a target fraction. Further details of the algorithm appear in section D. Hookpoints Used. XRegion-CCA uses the cong_control hookpoint to control how the kernel processes and reacts to ECN marks.3 Additionally, for efficiency, XRegion-CCA uses the min_tso_segs hookpoint to calculate and maintain efficient segment batching. Finally, XRegion-CCA implements the init hookpoint to minimize convergence time by setting the initial congestion window (we use a default of 100 MSS).

5

CCAaaS Orchestrator

This section describes the challenges of, and our approach to, deploying, updating, and operating CCAs at fleet scale. Our approach inherits certain properties from our eBPF orchestration platform, NetEdit: specifically, disruption-free deployment (section 5.1), retroactive application, and safe rollback. We introduce a transitional CCA that NetEdit can safely roll back to when a CCA deployment fails (section 5.2). We also extend NetEdit to support coordinated CCA orchestration (section 5.3). 3 Recall that the kernel forces a connection into the CWR state on a single

ECN mark; XRegion-CCA must override this behavior to function correctly.

(a) Without object pinning.

(b) With object pinning.

Figure 9: CCA deployment in a region: (a) without disruptionfree rollout, a visible Cubic spike appears during upgrades; (b) with disruption-free rollout, the spike is eliminated. 5.1 Guarantees Provided by NetEdit Disruption-Free Deployment. At Meta, we run an eBPF orchestrator, NetEdit [3], which deploys new versions of our eBPF programs (including our eBPF-based CCAs) to production every week. During an upgrade, NetEdit is briefly unavailable on each host (typically ∼30 seconds), and any new connection established during this window falls back to the kernel’s default CCA (i.e., Cubic). Although this window is short, at scale a large number of connections end up using this default, which performs suboptimally relative to our tailored CCAs. To eliminate this disruption, we apply NetEdit’s BPF pinning mechanism [3], which decouples the lifecycles of eBPF objects (e.g., maps and programs) from the control-plane process that manages them. This decoupling ensures that when an update restarts the control-plane process, the eBPF objects continue running unaffected and remain configured to use the appropriate optimized CCAs. Figure 9 highlights the benefit of building CCAaaS on NetEdit and leveraging its object management. Without object pinning (fig. 9(a)), a large amount of new traffic uses Cubic during the upgrade, producing a visible spike in Cubic traffic; with NetEdit’s object pinning (fig. 9(b)), the spike is eliminated. Retroactive Application. After a new CCA is deployed, we must migrate all existing connections that use the old version to the new CCA. We do this using NetEdit’s TCP iterator (bpf_iter) [3], which enumerates all active connections and applies the new CCA’s logic to each one. As described in NetEdit, this mechanism assumes that CCA logic is idempotent: reapplying initialization or policy logic does not violate correctness.4 The TCP iterator provides an added benefit: because it is applied to all connections, it also updates connections whose shared inputs (e.g., scope, locality, or CCA configuration) have gone stale (section 4.2.2). 5.2 Introducing a Transitional CCA for Safe Rollback When a newly rolled-out CCA is buggy, we want to roll back to a trusted CCA, i.e., a transitional CCA, that can be safely applied to traffic using any other CCA. Designing a transitional CCA is challenging because certain CCAs rely on protocol 4While we expect changes to be nearly instantaneous, certain state variables do not update until further packets are processed; section 9.3 quantifies the transient.

Database Schedule Request

CCA Developer User space

CCA Per-RTT State

CCAaaS Global BPF Orchestrator (NetEDIT)

Kernel LINUX N/W Stack

CCALogger CCALogger CCAaaS Modules

ConfigStore CCALog eBPF Maps

Figure 10: Workflow of CCALogger in production. features negotiated in advance, so rollback must revert to a CCA that is compatible with these prenegotiated features (e.g., once negotiated, ECN persists even after the CCA is changed, so the transitional CCA must support ECN). Unfortunately, Cubic (our default CCA) is not designed for ECN and treats every ECN mark as it would a loss. We therefore extended Cubic into our transitional CCA (BPF-Cubic) by modifying its code to handle ECN signals correctly. In practice, rollback follows the same three broad steps as NetEdit’s rollout procedure: (1) the transitional CCA is loaded as the new active CCA, (2) the TCP iterator retroactively applies it to all existing connections, and (3) the faulty CCA is unregistered. 5.3 Supporting CCA-Centric Orchestration While NetEdit is designed to roll out eBPF programs to hosts within our fleet, this approach fails for CCA rollouts in which both ends of a connection must use the same CCA.5 We therefore extended NetEdit to support coordinated rollouts. For intra-region CCAs, rollout is straightforward and uses existing NetEdit primitives, which upgrade all servers at a specified granularity together. We upgrade all servers within a region at once, ensuring that the source and destination of every intra-region connection use the same CCA. For cross-region CCAs, we added a mechanism to NetEdit that supports region-pair rollouts. We roll out to pairs of regions and update the scope resolver so that cross-region connections between upgraded region pairs use the new CCA.

6

CCALogger

We designed CCALogger to provide fine-grained visibility into eBPF-based CCAs by tracing CCA-specific state and signals over the lifetime of a dynamically selected connection. To scale CCALogger to production environments, our design carefully manages state and control logic across many concurrent connections while working within a strict performance budget and the kernel’s limited support for synchronization primitives. CCALogger addresses these challenges through three generalizable principles: 5 Certain CCAs (e.g., ECN-based ones) require both endpoints to be upgraded simultaneously; e.g., ECN must be negotiated during the TCP handshake.

• Align observability with control timescales. Analyzing CCA internals or evaluating logging conditions at per-packet granularity is prohibitively expensive. Our design builds on the insight that important CCA state evolves at RTT granularity; thus, as long as we log once per RTT, we faithfully capture the control dynamics. Consequently, CCALogger never logs state at a finer granularity than once per RTT, which enables it to scale. • Couple observability with modular CCA components. Given per-RTT logging, the next challenge is deciding when within an RTT to log in order to capture the most accurate snapshot. Because our CCA framework is modularized to invoke eBPF programs at well-defined state transitions and signal changes, the natural observation point is the existing CCA function that performs the per-RTT logic. Consequently, CCALogger integrates into the specific BPF programs that perform per-RTT analysis and control. • Decouple policy from mechanism. The set of connections to monitor varies with workload and context; e.g., our CCA developers often wish to analyze connections based on service identity, endpoints, or previously observed behavior. Unfortunately, supporting this flexibility within the kernel would incur significant overhead, because it requires evaluating predicates (i.e., comparisons and conditional branches) on the datapath. Instead, CCALogger performs all policy logic in user space and uses a simple selection mechanism in the kernel. 6.1

Architecture

CCALogger (fig. 10) realizes these principles by confining policy analysis and decisions to user space while keeping the datapath minimal, predictable, and efficient. This control and data plane split mirrors the philosophy of CCAaaS and more generally eBPF orchestration at Meta (i.e., NetEdit). Control Plane. The control plane decides what to observe. Developers specify the connection, or class of connections, to monitor using a CLI or web interface. This specification is distributed across the fleet via our configuration management system. On each host, a user-space agent translates the specification into compact selection state and updates the corresponding in-kernel data structures (e.g., the configuration map). The agent periodically retrieves the logged CCA state, stores it in a local database, and exposes it to developers through the CLI or web interface. Data Plane. The data plane decides when and how to record state. It comprises a small set of functions inserted into the BPF-TCP programs that are triggered at RTT boundaries and that have access to stable CCA control state. Specifically, we instrument init to record static connection metadata and pkts_acked to capture per-RTT CCA state.6 Each function performs a constant-time ownership check and, if the check succeeds, exports CCA state to a log. The 6 Although pkts_acked may be invoked multiple times per RTT, logging

is coded so that state is written exactly once per RTT.

datapath relies on two shared data structures: a configuration map used for the ownership check and a log used to record state. The ownership check is how CCALogger minimizes datapath overhead: it reduces the computationally expensive selection logic to a single 64-bit comparison.7 Specifically, the function compares the socket cookie of the connection it is processing against a value stored in the configuration map. Socket cookies are compact, stable identifiers that uniquely identify a socket for its lifetime. We chose socket cookies over alternatives, e.g., socket pointer addresses, because they are explicitly exposed to user space (i.e., our control plane). 6.2 CCALogger Use Cases We next briefly describe our two most common uses of CCALogger. Development and Debugging. A key challenge we faced while developing our BPF-based CCAs was validating whether a specific implementation operates correctly. CCALogger exposes a CCA’s internal structures and state values and how they change over time, which also lets us analyze state transitions. For example, during the development of XRegion-CCA, CCALogger revealed that the kernel’s default state machine immediately transitioned a sender from the OPEN to the CWR state upon receiving a single ECN mark. This revealed the need to override the kernel’s state machine by implementing the cong_control hook (section 4.1.2). Operational Validation. We also use CCALogger to monitor our CCAs in production, ensuring that connections use the appropriate CCA and exhibit the intended behavior. For instance, while deploying XRegion-CCA, we used CCALogger to monitor compliance and, specifically, to confirm that over 90% of active connections ran XRegion-CCA rather than falling back to the kernel default.

7

Operational Lessons

Recent work has characterized lessons from managing and scaling eBPF deployments [3, 13, 32]. 8 In light of this work, we focus on CCA-specific challenges: specifically, lessons from modular CCA design (section 7.1), from managing eBPF-based CCAs (section 7.2), and from evolving eBPF to support CCA development (section 7.3). 7.1 Lessons for Designing Modular CCAs We have identified important lessons from experience that research on modular CCAs [5, 6, 14, 16, 25, 26, 33] often does not address. Prefer specialized interfaces over generalized ones. A class of research frameworks argues for generalized eventbased models; however, in our experience, abstract interfaces 7 A naive design would encode selection as predicates over internal CCA state or connection attributes (e.g., the 5-tuple or process identifiers), requiring continuous evaluation and significantly increasing datapath complexity. 8 Portability refers to running the same program across different kernel versions, which is challenging because eBPF support is missing or inconsistent across versions.

force functions to recompute statistics and infer CCA properties, incurring significant overhead (in both compute and storage). Naturally, specialized interfaces introduce some ossification; however, we found the BPF-TCP interface amenable to evolution. Provide control over socket-level functionality, kernel configurations, and cross-layer network stack functions. Research frameworks [6, 25, 26] focus on the core CCA control loop, overlooking kernel behaviors below the CCA (e.g., TSO batching and buffer growth heuristics); however, we found that these layers silently interact with CCA decisions and in certain cases override CCA behavior. Thus, frameworks should provide control over these elements. Evolve CCAs incrementally. In production, new CCAs must be deployed without impacting the performance or availability of existing applications. This is challenging and requires that new CCAs be incrementally deployable and interoperable. Fundamentally, clean-slate approaches, as often proposed in research, are disruptive and require changes to existing applications and infrastructure. A natural path is to take existing CCAs and gradually evolve them to support new requirements and to adapt to lessons learned from production deployments. 7.2 Lessons for Managing CCAs at Scale Unlike the previous lessons, which focus on designing a modular CCA framework, this set of lessons focuses on the operational challenges we faced while continuously rolling out CCAs without impacting applications. Explicitly override the kernel default with your most performant CCA. Our initial approach used the kernel’s default CCA, i.e., Cubic, for connections with unknown scope or locality. However, as we expanded applications into public clouds for inference, these connections had unknown scope and were therefore configured to use Cubic, a CCA known to be suboptimal for WAN traffic. We updated NetEdit to include locality and scope configuration for these cloud infrastructures. More generally, we now select the default based on our most performant CCA rather than the kernel default. Choose your rollout granularity for both problem attribution and blast radius. During our initial rollout of XRegion-CCA, we upgraded a subset of connections to use XRegion-CCA, which meant that connections from the same service used different CCAs. Unfortunately, our network telemetry aggregates metrics (e.g., retransmission rates and ToR buffer occupancy), so we could not debug XRegion-CCA: we could not attribute aggregate behavior to XRegion-CCA or to other CCAs. We enhanced NetEdit’s rollouts to support region-pair granularity, so that all traffic to a given region uses the same CCA. This allowed us to attribute changes in metrics to the CCA change. Plan for network state and bugs. Our initial deployment of XRegion-CCA when it negotiated ECN, the WAN routers incorrectly interpreted it as DSCP. When we rolled back to

BPF-Cubic, we ran into further problems because XRegionCCA had already negotiated ECN. We learned to explicitly account for the network by testing any new network features a CCA uses before deploying it. Similarly, we designed our transitional CCA to support prenegotiated network state. 7.3 Lessons from Evolving eBPF to Support TCP We designed NetEdit for sockops and TC programs; however, extending it to support struct_ops CCAs revealed several issues that, we believe, modern orchestration frameworks must robustly account for as eBPF expands to more hookpoints. Verifier-approved programs can still take down the kernel. In using BPF-TCP, we identified latent kernel bugs that disrupted production traffic. Our programs triggered these bugs despite passing the eBPF verifier’s checks and being considered safe. The bugs ranged from recursion bugs and reference-count corruption to return-value bugs. Most interestingly, these bugs were often nondeterministic, both in when they occurred and in the symptoms they produced. We patched these bugs [13, 17, 20, 21]. Validate that hookpoint semantics hold for new hookpoints. We designed NetEdit to explicitly tackle heterogeneity in kernel semantics by decoupling eBPF behavior from kernel heuristics and introducing an interface that lets NetEdit explicitly manage the eBPF lifecycle. However, our design assumed that hookpoints share certain invariant behaviors that we could design around. This held until we extended NetEdit to struct_ops: we assumed that eBPF programs are managed by a single userspace entity, but struct_ops objects are managed by both userspace and the kernel’s CCA subsystem. Consequently, terminating the userspace process does not remove CCAaaS’s eBPF objects, and a redeployed userspace process fails to load new eBPF objects; when this happened, CCAaaS could not configure the CCA for connections on that host.9 We therefore bound each struct_ops object to a bpf_link, which allows it to be managed entirely by one entity. We are now working to identify other such invariants, and we welcome tools that automatically characterize these invariants and check them as new hookpoints are adopted. Introduce versioning semantics for shared state. When the kernel loads a program, it validates only that the map’s key and value sizes match the program’s definitions. When we rolled back a program, it wrote well-formed but versionincompatible data, which caused our CCA to behave incorrectly for ∼25% of our traffic and degraded performance. We now version all of our maps to avoid such issues. 7.4 Upstream Contributions We upstreamed several kernel patches as part of this project: (1) struct_ops link semantics for correct CCA lifecycle management [20], which allow eBPF orchestrators to auto9 Binding each struct_ops object to a bpf_link owned by NetEdit, so that the CCA is reclaimed when the link’s owner exits, reduced registration errors from ∼4K hosts per week to single digits.

CCA

Core

Headers

CCALogger

Total

re-used lines

BPF-DCTCP DCNDelay-CCA XRegion-CCA BPF-Cubic

209 472 440 182

50 91 350 46

130 141 127 132

389 704 917 360

73 132 253 0

Table 5: Lines of code per CCA and their provenance. matically clean up struct_ops objects when they exit; and (2) a patch that enables live, zero-downtime connection migration between CCAs [17].

8

Implementation

in this section, we briefly describe the implementation of our CCAs: our CCAs are in a combination of C file and header files. Table 5, illustrates the breakdown of the different CCAs as well as the re-use across CCAs due to the modularity that Struct_ops provides. For example, our implementation of DCNDelay-CCA extends on our existing BPF-DCTCP with two additional hooks and ∼340 lines of code. More importantly, we note that a non-trivial fraction of our newer CCAs reuse code from our earlier CCAs enabling the rapid development cycles highlighted in Table 1.

9

Evaluation

Our evaluation of CCAaaS is motivated by the following questions. What are the overheads (both computational and performance) of implementing a CCA in eBPF (section 9.1)? What are the benefits of the new CCAs enabled by CCAaaS (section 9.2)? Do CCAs built with our modular framework achieve production-quality performance (section 9.2)? What are the benefits of the management primitives introduced by CCAaaS (section 9.3)? Evaluation Setup. We collected metrics from running CCAaaS across environments with varying traffic types (streaming and RPC), flow counts, sender counts, NIC speeds (50G and 100G), NIC vendors, and scopes (intra-rack, intradata center, intra-region, and cross-region). For a subset of experiments (section 9.1), we selected a rack on which to explore different CCAs without disrupting production traffic. We evaluated the overhead of eBPF-based CCAs using 49 production hosts running ∼1,100 TCP connections. In each experiment, all hosts use the same CCA. 9.1

Analysis of eBPF-based Modularity

In fig. 11(a) and (b), we compare native kernel and eBPFbased variants of DCTCP and Cubic. We observe negligible differences, which shows that implementing a CCA with the BPF-TCP interface yields no meaningful performance difference compared with implementing it directly in the kernel. However, BPF-Cubic introduces ∼5 pp CPU overhead, attributable to the eBPF JIT dispatch cost being a larger fraction of Cubic’s lightweight arithmetic.

CCA

Metric

Improvement

DCNDelay-CCA

Retransmissions (Ads/Feed/Async) Intra-region congestion window ToR packet discards Host CPU utilization

30–40% reduction >50% reduction Significant reduction ∼5% reduction

XRegion-CCA

Congestion discards (fleet-wide) WRED packet drops (single region) Cross-region service P99 latency GenAI checkpoint P99 latency

>90% reduction ∼40× reduction 4–20× improvement ∼40× (20s → 0.5s)

Table 7: Production benefits of deploying DCNDelay-CCA and XRegion-CCA.

Figure 11: Paired per-host CDFs of CPU and memory differences. We exclude application-level metrics (retransmissions, bandwidth, and P99 latency) because they are virtually identical. CPU graphs are percentage points (PP). 9.2

Analysis of CCAaaS-Enabled CCAs

We present deployment results for our two most recent CCAs to validate the framework’s efficiency and utility. For context, we replaced DCTCP with DCNDelay-CCA as the default for over 94% of our intra-region traffic, and we similarly replaced Cubic with XRegion-CCA for 94.7% of all crossregion traffic. Relative Overheads of New CCAs. We analyze the impact of our extensions to the BPF-TCP interface by comparing the overheads of the new CCAs against eBPF implementations of traditional CCAs (fig. 11(c) and (d)). We observe that DCNDelay-CCA adds only 0.4 pp CPU over BPF-DCTCP, while XRegion-CCA actually reduces CPU by 2.0 pp relative to BPF-Cubic. CCA

CPU Overhead (%) p95 p99

Memory (bytes) p95 p99

DCNDelay-CCA XRegion-CCA

3 <1

45,056 45,056

26 1

269,864 185,964

Table 6: Resource overheads (p95 and p99) of recent CCAs. Overheads of New CCAs. Next, in table 6, we analyze these CCAs’ fleet-wide resource footprint. We observe that both CCAs impose minimal CPU and memory overhead. Because CCAaaS maintains per-connection state,10 the memory variation across our fleet can be attributed to variation in the number of concurrent connections per host. Production Benefits of New CCAs. Table 7 summarizes the production improvements we observed after deploying DCNDelay-CCA and XRegion-CCA. For intra-region traffic, DCNDelay-CCA significantly improved network metrics (retransmissions and packet discards), and these benefits translated into host-level benefits (i.e., lower 10 Each struct_ops CCA has a fixed overhead of ∼12KB (the BPF pro-

gram and counter map) plus 104 bytes per active connection (section 4.2.2).

CPU utilization).11 Additionally, DCNDelay-CCA’s smaller congestion window reduced in-network queuing, creating headroom from which cross-region flows benefit (section B). For cross-region traffic, we also observed significant improvements in network metrics that translated into applicationlevel improvements. Specifically, for the storage service that AI workloads use for checkpoint data, model weights, and configuration: XRegion-CCA significantly improved P99 latency over Cubic, especially for larger objects. For latency-sensitive workloads, we also observed that XRegion-CCA reduced service latency by up to 20×. 9.3

Analysis of Retroactive Application, and Rollback

Retroactive Application. During the initial DCNDelay-CCA rollout, we used TCP iterator to migrate over 6 million existing sockets across one region’s data centers. During this migration, we observed that per-host migration latency averaged approximately 4ms, with a P99 of 125–175ms. More importantly, we observed that TCP iterator produced no measurable impact on host-level metrics. However, we noticed that migrated connections experience a temporary throughput decrease of 13% for 20–30ms while iterator modifies state variables which can then only be used by subsequent packets. Rollback. Rollback generally takes ∼3 minutes end-toend. During the XRegion-CCA rollback, we reverted five regions to BPF-Cubic one region at a time rather than all at once, to prevent side effects from a sudden fleet-wide CCA change; each region completed within 5–10 minutes, covering all servers in its data centers, and all five regions were reverted within one day.

10

Related Work

The work most closely related to CCAaaS falls into four main categories: Modular and Programmable CCA Frameworks. Existing approaches to modular CCAs move logic to userspace [14, 16, 26] to enable rapid prototyping; instead, we use eBPF to enable rapid prototyping within the kernel, which allows us to reuse existing CCA mechanisms. Other work [6, 33] 11 The ∼5% reduction in host CPU utilization, driven by larger effective packet sizes and fewer retransmissions, shows that the eBPF abstraction does not impose meaningful overhead.

uses eBPF to implement CCAs; however, because these approaches use independent programs and focus on core CCA logic, they do not provide the level of management or customization that our CCAs require. Managing CCA Deployment at Scale. Hyperscalers have described their experiences deploying individual CCAs [2, 9, 18, 28, 38]; however, these works generally focus on specific CCAs and their evaluation. Instead, we focus on the ecosystem of tools and frameworks for managing a large and evolving set of CCAs. Additionally, our insights focus on eBPF-based CCAs, whereas others have explored evolvability in the context of userspace stacks (e.g., Google’s Snap [22] and Alibaba’s Luna [37]). eBPF Management and Orchestration. Existing eBPF management systems are general-purpose orchestration frameworks [4, 7, 34] that often focus on TC/XDP programs and do not support struct_ops, i.e., BPF-TCP. Similarly, while our prior work, NetEdit [3], provides general eBPF orchestration, CCAaaS extends it with the CCA-specific mechanisms described in section 5.3. Transport Observability. Traditional TCP tools [12,19,31, 35, 36] diagnose application performance under well-behaved CCAs, whereas we focus on diagnosing incorrectly behaving CCAs. Other work, e.g., Web10G, externalizes limited TCP state or captures external metrics [10,11]; CCALogger instead externalizes a richer set of CCA-internal state. For example CCALogger externalizes the following data: snd_cwnd (a tcp_sock’s current sending congestion window); snd_ssthresh (a tcp_sock’s current slow-start size threshold); ecn_alpha (the fraction of sent bytes that experienced congestion in the previous RTT); is_loss (a flag indicating if a packet loss occurred in this RTT window); is_ece_ack (a flag indicating if an ECEmarked ACK is received in this RTT window); is_tx_start (a flag indicating if a transmit occurred when no packets in this RTT window); is_ssthresh_invoked (a flag indicating if the hook ‘ssthresh‘ is invoked in this RTT window).

11

Discussion

Cloud Data Centers. Today, we run CCAaaS in external cloud environments with little modification. The core issues, we find, arise from the CCAs we run on top of CCAaaS. In particular, a subset of our CCAs are ECN-based, and in public clouds we neither control nor have access to ECN signals. This requires us either to choose CCAs that do not need ECN or to design a new CCA that dynamically detects the available signals and selects an appropriate control loop. Today, we opt for the former and leave the latter as an open research question. Programmable NICs. We focus explicitly on implementing CCAs entirely in software; however, others, e.g., Falcon [30], have explored offloading portions of a CCA onto the NIC hardware. Fundamentally, BPF-TCP works by overriding certain kernel functionality, whereas hardware-assisted

transports offload functionality to modules provided by the hardware. These approaches are not at odds with each other; consequently, CCAaaS could enable hardware-assisted transports in which BPF-TCP overrides some functionality and the rest is offloaded to hardware. This will likely require mechanisms that allow BPF programs to control and configure the hardware, as well as shared memory so that socket or connection state is consistently shared across both. Non-struct_ops CCAs. CCAaaS explicitly manages struct_ops-based CCAs; however, others have designed CCAs using a combination of other eBPF hooks. Our core orchestrator, NetEdit, will support such CCAs, but some of CCAaaS’s benefits, e.g., consistent storage and execution, may be harder to realize. Cross-CCA Fairness. Cross-CCA fairness is traditionally managed through algorithmic design, network infrastructure configuration, or both. CCAaaS focuses on managing CCAs; ensuring fairness is outside the scope of the system. Supporting BBR. The reference BBRv3 implementation extends the underlying tcp_congestion_ops interface in the same direction in which we have evolved BPF-TCP. BPF-TCP exposes every tcp_congestion_ops function as a struct_ops program except the diagnostic get_info; this is sufficient to express BBR. BBRv3, however, adds one new callback, replaces another, and enlarges the per-connection CCA state from 104 to 144 bytes, so supporting it would require upstream interface extensions similar to our own.

12 Conclusion and Open Research Challenges CCAaaS has allowed us to rapidly prototype, test, and deploy five eBPF-based CCAs over the last five years. Today, these CCAs carry over 97% of our internal traffic. In this paper, we describe the evolution of our modular eBPF-based framework, the BPF-TCP hook interface, to support different CCAs, and in doing so we highlight key design lessons for modular CCAs. We also describe the challenges of maintaining application performance and reliability as we evolve our CCAs, and the challenges of extending our eBPF orchestration framework, NetEdit, to support eBPF-based CCAs. Much of our effort has focused on solutions tailored to our unique production environment, and we highlight open challenges for the research community to tackle. Beyond eBPF Verifier (Towards Semantic Verification): Verifier ensure that an eBPF program does not crash the kernel; however, this is not safety. We observed crashes due to bugs in the BPF-Kernel interface and due to semantics of the program. We encourage the research community to look into semantic verification of program behavior . Most efforts focus on designing and engineering for single programs; however, struct_ops are a bundle of programs that interoperate and whose correctness is a function is an emergent behavior. We similarly encourage the research community to investigate designs that look more holistically at program interactions.

References [1] Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. Data center tcp (dctcp). SIGCOMM Comput. Commun. Rev., 40(4):63–74, aug 2010. [2] Serhat Arslan, Yuliang Li, Gautam Kumar, and Nandita Dukkipati. Bolt: Sub-RTT congestion control for UltraLow latency. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 219–236, Boston, MA, April 2023. USENIX Association. [3] Theophilus A. Benson, Prashanth Kannan, Prankur Gupta, Balasubramanian Madhavan, Kumar Saurabh Arora, Jie Meng, Martin Lau, Abhishek Dhamija, Rajiv Krishnamurthy, Srikanth Sundaresan, Neil Spring, and Ying Zhang. Netedit: An orchestration platform for ebpf network functions at scale. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM ’24, page 721–734, New York, NY, USA, 2024. Association for Computing Machinery. [4] bpfman: An eBPF Manager. https://bpfman.io/ main/, 2024. [5] Patrick G. Bridges, Gary T. Wong, Matti Hiltunen, Richard D. Schlichting, and Matthew J. Barrick. A configurable and extensible transport protocol. IEEE/ACM Transactions on Networking, 15(6):1254–1265, 2007. [6] Zhongjie Chen, Qingkai Meng, ChonLam Lao, Yifan Liu, Fengyuan Ren, Minlan Yu, and Yang Zhou. {eTran}: Extensible kernel transport with {eBPF}. In NSDI, 2025. [7] DataDog. eBPF manager. This manager helps handle the life cycle of your eBPF programs. https://github. com/DataDog/ebpf-manager, 2023. [8] Marek Denis, Yuanjun Yao, Ashley Hatch, Qin Zhang, Chiun Lin Lim, Shuqiang Zhang, Kyle Sugrue, Henry Kwok, Mikel Jimenez Fernandez, Petr Lapukhov, Sandeep Hebbani, Gaya Nagarajan, Omar Baldonado, Lixin Gao, and Ying Zhang. Ebb: Reliable and evolvable express backbone network in meta. In Proceedings of the ACM SIGCOMM 2023 Conference, ACM SIGCOMM ’23, page 346–359, New York, NY, USA, 2023. Association for Computing Machinery. [9] Abhishek Dhamija, Balasubramanian Madhavan, Hechao Li, Jie Meng, Shrikrishna Khare, Madhavi Rao, Lawrence Brakmo, Neil Spring, Prashanth Kannan, Srikanth Sundaresan, and Soudeh Ghorbani. A large-scale deployment of DCTCP. In 21st USENIX

Symposium on Networked Systems Design and Implementation (NSDI 24), pages 239–252, Santa Clara, CA, April 2024. USENIX Association. [10] Ruopeng Geng, Jianyuan Lu, Chongrong Fang, Shaokai Zhang, Jiangu Zhao, Zhigang Zong, Biao Lyu, Shunmin Zhu, Peng Cheng, and Jiming Chen. Enabling stateful tcp performance profiling with key event capturing. IEEE Transactions on Network and Service Management, 22(5):4964–4982, 2025. [11] Ehab Ghabashneh, Yimeng Zhao, Cristian Lumezanu, Neil Spring, Srikanth Sundaresan, and Sanjay Rao. A microscopic view of bursts, buffer contention, and loss in data centers. In Proceedings of the 22nd ACM Internet Measurement Conference, IMC ’22, page 567–580, New York, NY, USA, 2022. Association for Computing Machinery. [12] Mojgan Ghasemi, Theophilus Benson, and Jennifer Rexford. Dapper: Data plane performance diagnosis of TCP. In Proceedings of the Symposium on SDN Research, SOSR 2017, Santa Clara, CA, USA, April 3-4, 2017, pages 61–74. ACM, 2017. [13] Prankur Gupta, Takshak Chahande, Miao Xu, Prashanth Kannan, Tejas Birajdar, Lilian Liu, Rajiv Krishnamurthy, and Theophilus A. Benson. BPFComposer: Towards deterministic and safe composition of eBPF programs (and their shared state). In Proceedings of the 20th ACM International Conference on Distributed and Event-based Systems (DEBS ’26), page 7, Lisbon, Portugal, June 2026. Association for Computing Machinery. [14] EunYoung Jeong, Shinae Wood, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. {mTCP}: a highly scalable user-level {TCP} stack for multicore systems. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 489–502, 2014. [15] Glenn Judd. Attaining the promise and avoiding the pitfalls of TCP in the datacenter. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pages 145–157, Oakland, CA, May 2015. USENIX Association. [16] Antoine Kaufmann, Tim Stamler, Simon Peter, Naveen Kr Sharma, Arvind Krishnamurthy, and Thomas Anderson. TAS: TCP acceleration as an OS service. In European Conference on Computer Systems (EuroSys). ACM, 2019. [17] Kui-Feng Lee. Transit between BPF TCP congestion controls. https://lore.kernel.org/all/ 20230323032405.3735486-1-kuifeng@meta.com/, 2023.

[18] Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, David Wetherall, and Amin Vahdat. Swift: Delay is simple and effective for congestion control in the datacenter. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, SIGCOMM ’20, page 514–528, New York, NY, USA, 2020. Association for Computing Machinery. [19] Yuliang Li, Rui Miao, Mohammad Alizadeh, and Minlan Yu. DETER: Deterministic TCP replay for performance diagnosis. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 437–452, Boston, MA, February 2019. USENIX Association. [20] Martin KaFai Lau. Introduce BPF STRUCT_OPS. https://lore.kernel.org/bpf/20200109003453. 3854769-1-kafai@fb.com/, 2020. [21] Martin KaFai Lau. bpf: Allow bpf tcp iter to do bpf_(get|set)sockopt. https://lore.kernel.org/ bpf/20210701200535.1033513-1-kafai@fb.com/, 2021. [22] Michael Marty, Marc de Kruijf, Jacob Adriaens, Christopher Alfeld, Sean Bauer, Carlo Contavalli, Michael Dalton, Nandita Dukkipati, William C Evans, Steve Gribble, et al. Snap: a microkernel approach to host networking. In Symposium on Operating Systems Principles (SOSP). ACM, 2019. [23] Autonomous testing of services at scale. https://engineering.fb.com/2021/10/20/ developer-tools/autonomous-testing/, 2021. [24] Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David Zats. Timely: Rtt-based congestion control for the datacenter. SIGCOMM Comput. Commun. Rev., 45(4):537–550, August 2015. [25] Pedro Mizuno, Kimiya Mohammadtaheri, Linfan Qian, Joshua Johnson, Danny Akbarzadeh, Chris Neely, Mario Baldi, Nachiket Kapre, and Mina Tahmasbi Arashloo. A target-agnostic protocol-independent interface for the transport layer, 2025. [26] Akshay Narayan, Frank Cangialosi, Deepti Raghavan, Prateesh Goyal, Srinivas Narayana, Radhika Mittal, Mohammad Alizadeh, and Hari Balakrishnan. Restructuring endpoint congestion control. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, pages 30–43, 2018.

[27] P. Balasubramanian, Y. Huang, M. Olson. Hystart++: Modified slow start for tcp. https://datatracker. ietf.org/doc/html/rfc9406, 2023. [28] Yajuan Peng, Haoran Wei, Xiaolong Zhong, Junkai Huang, Haohan Xu, Zicheng Wang, Yang Bai, Zhuo Jiang, Jianxi Ye, Xiaoliang Wang, Xiaoming Fu, and Huichen Dai. Barre: empowering simplified and versatile programmable congestion control in high-speed ai clusters. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’25, USA, 2025. USENIX Association. [29] Pittsburgh Supercomputing Center. Web10G:" Taking TCP Instrumentation to the Next Level. https://www. psc.edu/research/networking/web10g/. [30] Arjun Singhvi, Nandita Dukkipati, Prashant Chandra, Hassan M. G. Wassel, Naveen Kr. Sharma, Anthony Rebello, Henry Schuh, Praveen Kumar, Behnam Montazeri, Neelesh Bansod, Sarin Thomas, Inho Cho, Hyojeong Lee Seibert, Baijun Wu, Rui Yang, Yuliang Li, Kai Huang, Qianwen Yin, Abhishek Agarwal, Srinivas Vaduvatha, Weihuang Wang, Masoud Moshref, Tao Ji, David Wetherall, and Amin Vahdat. Falcon: A reliable, low latency hardware transport. In Proceedings of the ACM SIGCOMM 2025 Conference, SIGCOMM ’25, page 248–263, New York, NY, USA, 2025. Association for Computing Machinery. [31] Praveen Tammana, Rachit Agarwal, and Myungjin Lee. Simplifying datacenter network debugging with PathDump. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 233– 248, Savannah, GA, November 2016. USENIX Association. Embracing Netkit [32] Chen Tang and Feng Zhou. Across a Million Servers, What Should We Do? Presented at the eBPF summit, 2024. https: //www.youtube.com/watch?v=0w788CqTp0c&list= PLDg_GiBbAx-m7yn_FYcc41PNrgtxlISBK&index=8. [33] Viet-Hoang Tran and Olivier Bonaventure. Beyond socket options: making the linux TCP stack truly extensible. In 2019 IFIP Networking Conference (IFIP Networking), pages 1–9. IEEE, 2019. [34] l3af. Complete lifecycle management of eBPF programs in the kernel. https://l3af.io/, 2024. [35] Zhe Wang, Huanwu Hu, Linghe Kong, Xinlei Kang, Teng Ma, Qiao Xiang, Jingxuan Li, Yang Lu, Zhuo Song, Peihao Yang, Jiejian Wu, Yong Yang, Tao Ma, Zheng Liu, Xianlong Zeng, Dennis Cai, and Guihai Chen. Diagnosing application-network anomalies for millions of ips in production clouds. In Proceedings of the 2024

USENIX Conference on Usenix Annual Technical Conference, USENIX ATC’24, USA, 2024. USENIX Association. [36] Jianer Zhou, Qinghua Wu, Zhenyu Li, Steve Uhlig, Peter Steenkiste, Jian Chen, and Gaogang Xie. Demystifying and mitigating tcp stalls at the server side. In Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies, CoNEXT ’15, New York, NY, USA, 2015. Association for Computing Machinery. [37] Lingjun Zhu, Yifan Shen, Erci Xu, Bo Shi, Ting Fu, Shu Ma, Shuguang Chen, Zhongyu Wang, Haonan Wu, Xingyu Liao, Zhendan Yang, Zhongqing Chen, Wei Lin, Yijun Hou, Rong Liu, Chao Shi, Jiaji Zhu, and Jiesheng Wu. Deploying user-space TCP at cloud scale with LUNA. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 673–687, Boston, MA, July 2023. USENIX Association. [38] Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. Congestion control for large-scale rdma deployments. ACM SIGCOMM Computer Communication Review, 45(4):523–536, 2015.

Appendices are supporting material that has not been peer-reviewed.

A

B

XRegion-CCA Synthetic Benchmark

Flows

P99 Latency Cubic XR-CCA

Retransmits Cubic XR-CCA

Thruput

12 (5MB) 12 (80MB) 48 (5MB) 48 (80MB)

100ms 765ms 275ms 377ms

2.4K 13.3K 15.7K 12.5K

+7%

87ms 520ms 157ms 331ms

0 0 1.2K 0.6K

+6%

Table 8: Synthetic benchmark: XRegion-CCA vs. Cubic with cross-region senders writing to a single receiver. Latency values are P99; throughput is aggregate improvement.

Mixed-CCA Coexistence

To quantify how our CCAs coexist, we ran a testbed experiment that retraces our deployment history: Cubic everywhere, then DCTCP and later DCNDelay-CCA for intraregion traffic, and finally XRegion-CCA for cross-region traffic. Two cross-region senders (∼32 ms RTT) and two intraregion senders (∼60 µs RTT) send to a single receiver with a 400 Gbps NIC; each run uses 240 flows (60 per sender) of 8 MB request–response transactions for 600 s. Consecutive pairings change only one side’s CCA, so each difference is attributable to a single swap. Table 9 summarizes the results. Fairness. Only the production pairing (DCNDelay-CCA × XRegion-CCA) avoids starving cross-region traffic: Jain’s index rises from 0.48 to 0.94. Within each class, Jain’s index stays between 0.97 and 0.99 in every run, so the unfairness is entirely between classes. Queue occupancy. DCNDelay-CCA holds the lowest intraregion standing queue, 15.6 KB versus 34.8 KB under Cubic, which directly supports the buffer-headroom effect in section 9.2. Cross-impact. Replacing only the intra-region CCA (Cubic → DCTCP → DCNDelay-CCA) raises cross-region goodput from 0.7 to 19.9 Gbps without any change on the crossregion side. Cubic’s cross-region window never exceeds 579 segments versus 5098 for XRegion-CCA: at 32 ms RTT, the bandwidth-delay product far exceeds what shallow ECN and drop thresholds allow, the failure that motivated XRegionCCA (section 4.3.2). The trade-off is that intra-region traffic yields capacity: intra-region goodput falls by 46% and its median transaction time rises from 25.8 to 57.8 ms, while aggregate throughput changes by only 6%. Limitations. Each pairing was run once; cross-region RTTs inflated to 48–54 ms during runs; queue occupancy is inferred from socket state rather than switch telemetry; and the cross- and intra-region senders ran different kernel versions.

Intra × Cross

Cross share

Jain

Cross p50

Intra p50

Intra queue

Cross cwnd

Cubic × Cubic DCTCP × Cubic DCNDelay-CCA × Cubic DCNDelay-CCA × XRegion-CCA

0.3% 2.0% 8.3% 42.3%

0.48 0.51 0.57 0.94

11.5 s 1.6 s 381 ms 74.9 ms

25.8 ms 34.6 ms 37.7 ms 57.8 ms

34.8 KB 25.8 KB 21.5 KB 15.6 KB

19 132 579 5098

Table 9: Mixed-CCA coexistence. Cross share is the cross-region fraction of total goodput; Jain is computed over all 240 flows; p50 is the median transaction time; intra queue is the standing queue inferred from sender socket state; cross cwnd is the mean cross-region congestion window (segments).

C

DCNDelay-CCA Algorithm Details

DCNDelay-CCA augments DCTCP’s ECN response with delay-based proactive congestion window sizing. The ECN response serves as a guardrail: on each ECN mark, DCNDelayCCA includes it in the fraction of marked packets to compute the DCTCP-style alpha value, ensuring it is never more aggressive than DCTCP. The delay component computes an ideal congestion window from the observed send rate and a configured minimum RTT (75µs in-rack, 100µs in-DC, 125µs in-region): target_cwnd = send_rate × min_rtt

(1)

If the current window exceeds the target, DCNDelay-CCA reduces it by 30% of the distance to the target, a gentle reduction that avoids oscillation. When no congestion is detected, the window increases additively (as in Reno). After consecutive periods without congestion signals, the additive increase constant grows, enabling faster ramp-up on uncongested paths. On packet loss, the slow-start threshold is set to approximately 80% of the current window (ssthresh = (cwnd × 819) ≫ 10).

D

XRegion-CCA Algorithm Details

XRegion-CCA computes a congestion ratio after each ACK: # delivered packets marked with CE ce_ratio = × 100% # total delivered packets (2) Hook distribution. XRegion-CCA’s logic is distributed across four hooks: • pkts_acked: Computes the congestion ratio from ECNmarked packets and generates the continuous congestion signal that distinguishes transient from persistent congestion. • cong_control: Implements the INC_CWND/DEC_CWND state machine and the dynamic fairness threshold, bypassing the kernel’s default CWR behavior. • ssthresh: Implements Conservative Slow Start to avoid overshooting upon exiting slow start. • min_tso_segs: Dynamic TSO segment override for batching efficiency.

Fairness via dynamic congestion thresholds. A static congestion threshold leads to unfairness when flows have heterogeneous RTTs: low-rate flows are more likely to exceed a fixed threshold per-ACK, causing them to back off prematurely. XRegion-CCA addresses this with a dynamic threshold that incorporates each flow’s transmission rate. A flow signals congestion when: tx_rate × ce_ratio > C

(3)

where C is a constant representing the targeted ECN rate. Higher-rate flows get a smaller effective threshold, causing them to react to congestion earlier and release bandwidth to lower-rate flows. This mechanism is implemented in the cong_control hook, which provides the delivery-rate context needed for the computation. Congestion response. In many ways, XRegion-CCA is an assembly of well-understood components adapted to our deployment constraints. It combines a CUBIC growth curve with DCTCP-style ECN alpha tracking, but applies gentler ECN-driven reductions (alpha/3 rather than DCTCP’s alpha/2, matching BBR’s ECN factor). A 6.25% minimum floor prevents stagnation when alpha has decayed to near-zero during long uncongested periods. XRegion-CCA implements its own Proportional Rate Reduction (PRR) for smooth window reduction, bypassing the kernel’s default PRR which is coupled to the CWR state machine. The Linux kernel forces a sender into CWR upon a single ECN mark. For cross-region traffic traversing switches with shallow ECN thresholds (configured for intraregion DCNDelay-CCA), this causes premature rate collapse. XRegion-CCA bypasses the kernel’s congestion state machine entirely via cong_control and implements its own two-state machine: • INC_CWND: Increases the congestion window through three stages, exponential Slow Start (cwnd × 2 per RTT), Conservative Slow Start (a HyStart++-inspired [27] stage that transitions on the first ECN mark or delay increase, growing at cwnd × 1.05 per RTT to avoid overshoot), and Congestion Avoidance (CUBIC growth with adaptive additive increase that grows faster when ECN is sparse). The transition from INC_CWND to DEC_CWND is triggered by a strong ECN signal (i.e., the dynamic threshold is exceeded) or packet loss.

• DEC_CWND: Reduces the congestion window to a target value. For ECN-triggered events: target = (1 − ecn_alpha/3) × cwnd, where ecn_alpha is an EWMA of ce_ratio. For loss-triggered events: target = 0.7 × cwnd. The sender uses XRegion-CCA’s own PRR to reach the target within one RTT, then transitions back to INC_CWND via a timer. The timer-driven transition (rather than event-driven) avoids over-reacting to transient congestion.

E

Future Work

Managing Sticky Transport Signals. Modern CCAs rely on multiple congestion signals, some of which are negotiated or configured at different points in a connection’s lifecycle. In particular some signals, notably ECN, are sticky: once negotiated during the TCP handshake, there is no standard mechanism to disable them mid-connection. This creates significant friction during CCA transitions. For example, when rolling back from an ECN-based CCA to a non-ECN CCA (e.g., Cubic), existing connections continue to receive ECNmarked packets. Cubic interprets these marks as implicit loss signals, triggering unnecessary congestion responses and increasing retransmissions. For the other way round, when we needed to migrate the long lived connections from non-ECN based CCA, our current mitigation in production is to transition through an intermediate CCA that gracefully handles but does not require ECN, and to use bpf_iter to conform all active connections fleet-wide. A cleaner solution would allow CCAs to explicitly disable or renegotiate signals on active connections. This requires both kernel support for midconnection signal teardown and CCA-aware mechanisms to drain in-flight marked packets before transitioning. We view signal lifecycle management as a first-class concern for future modular CCA frameworks. Version Management and Compatibility. eBPF-based CCAs lack native support for versioning, creating challenges at multiple levels: tracking CCA versions, ensuring compatibility across versions, and managing shared state with nonCCA programs. CCA versioning. When tuning parameters (e.g., alpha thresholds), adding new signals, or modifying control logic, operators face a choice: register an entirely new CCA under a different name, or overwrite the existing CCA and accept fleet-wide impact. Neither scales gracefully. Registering new CCAs proliferates names and complicates policy configuration; overwriting existing CCAs prevents incremental rollout and A/B testing. Our current workaround, encoding version information in CCA names and manually migrating connections via bpf_iter, is operationally brittle and does not generalize. Version compatibility. Beyond tracking versions, we lack mechanisms to express and enforce compatibility between CCA versions. When rolling out a new version, must it be backward compatible with all prior versions? With the previ-

ous N versions? How do we detect incompatibilities before they cause production impact? Currently, compatibility is validated through manual testing and operational discipline, but there are no explicit compatibility checks enforced at CCA selection or transition time. A robust solution would include version metadata in the CCA registration, compatibility matrices that the selector consults, and automated validation during rollout. Cross-program versioning. CCA programs often share BPF maps with other programs, sockopt handlers, monitoring hooks, scope resolvers, that evolve independently. A version change in the CCA may assume a new map schema or semantic, while a dependent program still expects the old format. This expands version management from a singleprogram concern to an ecosystem-wide coordination problem. As discussed in section 7.3, a non-backward-compatible scope update by an unrelated binary caused widespread CCA misselection. Native support for cross-program version contracts, where shared maps declare their schema version and consumers validate compatibility, would reduce the risk of silent failures. Future directions. We envision extending tcp_congestion_ops and the broader BPF infrastructure to support: (i) Version metadata: CCA registration includes version identifiers and feature flags, exposed to selectors and telemetry.; (ii) Compatibility declarations: CCAs declare which prior versions they can safely transition from, enabling the selector to block incompatible migrations; (iii) Sharedstate contracts: BPF maps declare schema versions; programs that consume them validate compatibility at load time or runtime. These capabilities would transform version management from an operational discipline into an enforceable system property. Coexistence Within the Broader eBPF Ecosystem. eBPFbased CCAs do not operate in isolation. A single host may run dozens of eBPF programs across diverse attach points sockops, sockopt, cgroup, tc, and struct_ops each managed by independent teams with different priorities. Our CCA infrastructure depends on several of these: tcp_congestion_ops for the CCA itself, bpf_iter for fleet-wide connection iteration, sockops for connection-level policy, and sockopt for configuration injection. This diversity introduces subtle conflicts. A sockops program from an unrelated team inadvertently disabled ECN by overwriting shared kernel state. The kernel provides no native ordering guarantees for cgroup-attached eBPF programs beyond insertion order, and no mechanism to express priority or mutual exclusion. We addressed this operationally by building program chaining infrastructure that enforces priority ordering and auditing that surfaces suppressed actions. However, these are workarounds rather than solutions. Native kernel support for eBPF program composition, including explicit ordering, conflict detection, and coordinated state access, would significantly reduce operational complexity and the risk of silent failures. Extending to Cloud Datacenters. Meta is increasingly

adopting cloud datacenters alongside owned infrastructure. Unlike Meta-owned facilities, cloud environments impose constraints we do not control: different network topologies, diverse hardware vendors, and varying feature support. For example, some cloud RSWs do not support ECN and rely on overprovisioning to manage congestion rendering ECN-based CCAs ineffective or counterproductive. Onboarding custom CCAs to cloud datacenters raises several challenges: (i) Feature detection: CCAs must dynamically detect which signals (ECN, delay, loss) are meaningful in a given environment and adapt accordingly; (ii) Monitoring heterogeneity: Our current monitoring infrastructure assumes uniform CCA deployment within a region. Hybrid environments with mixed CCA capabilities require per-path or per-destination observability; (iii) Vendor coordination: Unlike owned infrastructure, we cannot patch cloud network devices. CCA designs must be robust to vendor-specific behaviors and avoid assumptions that hold only in Meta-owned networks. We anticipate that future CCAs will need to be environment-aware, dynamically selecting signal sources and control strategies based on detected path characteristics rather than static configuration.

Related documents

• CCAaaS: An Orchestration Platform for High-Velocity Design, Development, and Deployment of Data Center Transports at Meta#1455712
• DLB: Distributed Load Balancing at Scale for Generative AI Inference#1035049
• DLB: Distributed Load Balancing at Scale for Generative AI Inference#1006827
• A Scalable Digital Twin Framework for Energy Optimization in Data Centers#168279
• DataCenterGym: A Physics-Grounded Simulator for Multi-Objective Data Center Scheduling#31250