Halley: A Predictive Control Plane for Optical AI Data Center Networks

1

Halley: A Predictive Control Plane for Optical AI Data Center Networks Manav Mishra1 and Krishna M. Sivalingam2 1 Prompeteer.ai, Los Angeles, CA 90066, USA 2 Department of Computer Science and Engineering, Indian Institute of Technology Madras, Chennai, India

arXiv:2611.04408v1 [cs.NI] 3 Nov 2026

Emails: manav@prompeteer.com, skrishnam@cse.iitm.ac.in CORRESPONDING AUTHOR: Manav Mishra.

At hundred-thousand-accelerator scale, reliability, network topology, and checkpointing have become first-order design constraints [1]. Each layer faces an online allocation decision under a hard service constraint: how many spare lasers the pool should hold, which circuits the switch should hold and for how long, where each ring’s heater should be set, and which modules receive the few hot spares, each wired to its module in advance. The deployed policies this paper replaces, which we call reactive for short, either fix each answer once in advance or make it only after the event it should have anticipated. Training is bulk-synchronous: every training step ends in a collective, a phase in which all accelerators communicate at once and which also acts as a barrier, so the step finishes only when its slowest link does. A drifted ring, a failed laser, or a late switch reconfiguration therefore leaves every accelerator idle at that barrier. Software-defined networking provisioned for a separate control plane from the data plane. Production optical fabrics use that approach for provisioning. For example, Google’s Jupiter fabric routes on a predicted traffic matrix [2] and TPUv4 reconfigures its optical circuit switches to ride through outages [3], Index Terms—Co-packaged optics, common-cause failure, data [4]. Similarly, Ironwood TPU’s 9,216-chip superpods use them center networks, distributed training, microring modulators, to route around interruptions, the MixNet research system optical circuit switching, optical interconnects, predictive control, reconfigures ahead of a predicted all-to-all [5]. The Opus reliability system reconfigures photonic rails at the boundaries between parallelism phases [6]. Failure prediction for pluggable optical transceivers is now validated on production data [7], [8]. We I. I NTRODUCTION observed that existing control planes do not forecast the coA frontier AI training run usually occupies a single cluster packaged optical hardware itself, a laser’s failure or a ring’s in an AI data center, where a dedicated network moves data drift, and hence do not act before the failure or drift arrives. among tens of thousands of accelerators. Between racks, copper Proposed Halley framework: This work presents Halley, cannot carry that data within its power and reach budget, so a control plane that provides these necessary features. The the network is optical [1], and the industry is now moving the system has multiple components as described below. The Halley optics onto the package itself, next to the switch or accelerator framework provides every layer with a component of its own. silicon, an arrangement known as co-packaged optics (CPO). LightReserve sizes the laser pool to a coverage bound instead This network, the fabric, has an optical data plane of four of a fixed ratio: a floor on the chance that its spares absorb layers, each managed on its own today. A shared pool of every failure in a window, valid for the assumed failure rate external lasers supplies light to many photonic engines, the and correlation. FoldFit schedules the OCS so that, under on-package optical transceivers of CPO. An optical circuit stated conditions, a schedule’s total dwell time, the time its switch (OCS) joins those engines to one another by connecting circuits are held, stays within 11/9 of the shortest possible, a whole light paths instead of forwarding individual packets. bound that leaves out switching delay. ThermaLead sets each Microring modulators, ring resonators a few microns in radius, microring’s heater ahead of the drift. TriageGuard steers a encode the data onto each wavelength. The lasers arrive in limited budget of hot spares to the modules most likely to field-replaceable modules that are swapped in service. fail. Each component predicts its layer’s next state and acts Abstract—A frontier AI model is trained in an AI data center, on tens of thousands of accelerators joined by a dedicated network, and every training step ends with all of them exchanging data at once. Between racks, that network is optical, and each of its four layers, from lasers to switches, runs a fixed or after-the-fact policy. Lasers sharing an enclosure, cooling, and power can fail together. At a fixed spare ratio, correlated failure provably caps a pool’s availability however large it grows. In this paper, we present a comprehensive predictive control plane framework, called Halley, that reads each resource’s next state and acts before the failure occurs. It has four components, each carrying a formal guarantee: LightReserve that sizes the laser pool to provide a coverage bound, FoldFit that schedules the switch within 11/9 of the optimal, ThermaLead that holds a microring with one-step-ahead control, and TriageGuard that protects the highest-risk modules first. Halley gives each layer a predictive controller that defers to the deployed policy when unsure. Composed from simulations at that point, effective goodput, the share of ideal throughput kept, rises from 65.2% to 76.5% if a window that exhausts its spares is lost. If only the dark channels’ bandwidth is lost, it reaches 67.4%. Per megawatt, that is up to 1.03–1.17 times the training; for the same training, up to 3.3–14.7% less data center capacity. Halley’s fallback circuits are formally verified and are shown to achieve close timing on a commercial FPGA, and pass a 130 nm signoff.

2

before it arrives. Each steps aside to the reactive system that it implementation of these components and the hardware design replaces when its forecast is stale or unsure. Hence, for such are available at [13]. a forecast the result is no worse than current systems, though Important Results: At the correlation we assume for a wella confident but wrong forecast has no such floor yet. built package and the stress failure rate of Section III-III-B, a Adding spares is the usual way to keep a shared laser pool with one spare per active laser, the 2× budget cap we pool lit. In a co-packaged pool the lasers are unlikely to fail assume throughout, cannot pass 99.670% availability however independently. NVIDIA’s external laser module holds eight large it grows. A reserve of four spares for 32 channels, the lasers in one temperature-controlled enclosure [9]. Hence, the proportion an analyst reports for a related shipping part, reaches lasers of a module share its heat and often one power supply, only 86.3%. Availability of 99.999% is then out of reach for and a thermal or power fault that stresses one laser stresses its any spare ratio within that cap. For a 32-channel pool at neighbors as well. Failures that arrive together can exhaust the that correlation, 99.999% stays out of reach within the 2× spares. Growing the pool at the same spare ratio, the share of cap whenever a laser’s failure probability per restock window the pool held as spares, does not spread that risk away, since exceeds about 2.4 × 10−4 . At 50 failures per billion hours (50 FIT), the Optical Internetworking Forum (OIF) target for a the next burst of failures is correlated too. At a fixed spare ratio, availability converges as the pool laser source, which we apply to each laser, a laser crosses that grows to a closed-form ceiling A∗ < 1 that depends on the probability once the restock window exceeds about 200 days. With the spare count held at the 23 lasers our sizing rule failure correlation ρ and the per-laser failure probability but not returns for a 99% target, cutting the correlation from ρ = 0.15 on the pool’s size. Raising the spare ratio does lift availability, to ρ = 0.03 moves availability from about 99% to just past but past a certain point so slowly that a high target needs a 99.999%. Under a stated cost model at the stress rate, reaching pool larger than any operator would deploy. The limit itself is 99.999% by separating the pool first and then sizing its spares classical. When exchangeable failures are positively correlated, costs $44,100 in all, against $234,000 for spares alone. the failed fraction of a large pool converges to a random level The overall figure of merit is effective cluster goodput, instead of to its mean [10]. Common-cause models in reliability defined as the fraction of the cluster’s ideal throughput that engineering bound redundant systems the same way [11], [12]. survives the losses of the fabric. At the stress setting, composing What this paper adds is its consequence for a pooled laser the simulated components lifts it from 65.2% to 76.5%, a 17.3% supply: where the cap binds, what it costs in spares, and the relative gain. That figure counts every window whose failures cheaper alternative. However, correlation can be controlled. exhaust the spares as lost, which we call the coverage reading. Separating the lasers into distinct thermal and power domains On the lit-share reading, in which only the dark channels’ and sourcing them from diverse batches raises the ceiling, bandwidth is lost, goodput rises from the same 65.2% baseline which redundancy cannot. to just 67.4%. Most of the gain thus comes from a provisioning Technical contributions: The main contributions are: decision: the larger spare pool that LightReserve sizes for the • First, it carries the classical exchangeable limit over to correlation a static reserve ignores. Holding that pool to the 2× pooled laser supplies and proves that in a pool of many cap costs only 0.2 points, and where no spare count within the modules the correlation between modules, more than cap reaches a target, LightReserve determines the decorrelation within them, sets the cap. Under a stated cost model, it needs. ThermaLead’s simple forecast does slightly worse it also finds that decorrelating before buying spares is the than the deployed dither lock on the stress trace; gated to fall cheaper path across the range of correlations considered back to that lock whenever its recent error is the larger of the plausible. two, it has 2.1% fewer lock losses. At 50 FIT and a 90-day • Second, it builds the proposed Halley control plane restock the fixed reserve already covers nearly every window, framework, that has four major components. Three of so the composed relative gain falls to about 1.3%; with yearly its four components carry a formal guarantee, stated in restocks, 99.999% still exceeds the 2× cap at 50 FIT. idealized form later. The novel aspect is the design for Within each reading, the stress-case gain is an upper bound the failure physics of the AI data center. on the useful compute freed from the same silicon and power, • Third, it composes the components into effective cluster because it treats fabric stall as the only recoverable loss. This goodput under two readings of a spare shortfall, couples bound matters because power and local approval now limit how the laser and thermal losses through a shared common fast new AI data centers are built. A new grid connection can cause, reports the tail as well as the mean, and proves a take five to ten years [14]. Around 45 US data center projects closed form for the correlation-driven part of that tail. worth $68 billion were blocked or delayed at the local level • Fourth, it synthesizes, places, and routes the fallback between April and June 2026 [15]. Every point of goodput that circuits for a commercial field-programmable gate array, the fabric loses is capacity that was built and powered but did carries the same sources through signoff as a SkyWater no useful work. At the bound, what Halley recovers means up 130 nm layout, and formally verifies their register-transfer to 14.7% fewer new megawatts to site and connect for a given design, the part of Halley that can be built today. training target, or up to 1.17 times as much training from the The paper also reports a detailed performance study of the power already in place, with no wait for a new grid connection. proposed mechanisms, based on simulation and mathematical On the lit-share reading the figures are 3.3% and 1.03 times. modeling, and their circuit-level realization: an FPGA build Energy per unit of training falls too, though by less, since a and a signed-off SkyWater 130 nm layout. The software waiting accelerator draws less than full power.

3

Fig. 1. The CPO data plane. Top: its four layers, each with the decision it must make and the policy deployed today. Bottom: the light path. External laser modules supply light to each photonic engine, microring modulators in the engine encode data onto it, and the OCS switches the modulated signals between engines [schematic].

The remainder of the paper is organized as follows. Section II 2) Training Traffic: Data, tensor, and pipeline parallelism presents the background material and related work. Section III split a training job across the accelerators. Mixture-of-experts presents the impact of failure correlation on various metrics (MoE) models add expert parallelism, which spreads specialist such as availability. Section IV presents the control plane parts of the model, the experts, across the job’s worker framework and the four mechanisms proposed in this paper. processes and routes each token to a few of them [5], [22]. Section V presents a detailed performance study of the proposed Expert dispatch is an all-to-all collective. Its demand matrix mechanisms based on simulation and mathematical modeling is dense within each expert group and sparse across groups, studies. Section VI presents the circuit-level realization of and it shifts as some experts draw more tokens than others. the proposed mechanisms, built for a commercial FPGA and FoldFit schedules these matrices in Section IV. signed off as a SkyWater 130 nm layout. Section VII presents Published runs show the scale: Llama 3 was trained on about a detailed discussion of the proposed framework including its 16,000 GPUs [23], MegaScale scaled training past 10,000 [24], limitations. Section VIII concludes the paper. DeepSeek-V3 used 2,048 [25], and Meta reports training on the order of 100,000 GPUs [26]. At this scale hardware is the main threat. Llama 3 attributed about 78% of its unexpected interruptions to confirmed or suspected hardware issues [23]. II. BACKGROUND A study of two large research clusters reports a mean time This section presents the relevant background and related to failure of 7.9 hours for 1,024-GPU jobs; InfiniBand link failures were among the top causes of job failures, and most material. of those came in one short burst from a handful of nodes [27]. Meta’s report on a 32,000-GPU job counts 2.3 interruptions per 1,000 servers per day, 78% of them hardware-related, and Meta A. Data Center Networks estimates one failure every 18 minutes at 100,000 GPUs [26]. The fabric is kept separate from the general-purpose frontend network, and purpose-built production fabrics such as Alibaba’s High Performance Network (HPN) have moved far from the classical data center architecture [16].

B. Optical Data Plane in a Data Center Network

Fig. 1 shows the four optical layers of the data plane. a) The CPO data plane: CPO moves the photonic engines 1) Optical Networking: Optics replaces a wiring problem from the rack edge onto the switch or accelerator package, and with an allocation problem at two levels: inside a single the light source changes with them: a shared pool of external link and across the fabric that joins the links. Inside a link, lasers, in field-replaceable modules, replaces one laser per wavelength division multiplexing (WDM) multiplies capacity engine. Each NVIDIA Quantum-X package uses 18 photonic without adding cable, because each channel travels on its own engines fed by 36 laser inputs [28]. The light comes from wavelength, and that makes capacity allocation a scheduling external laser modules of eight lasers each, and one module problem: which transmitter sends on which wavelength, and powers 32 of the switch’s 576 transmit lanes [9], which implies when. The problem has a long literature on star-coupled and 18 modules and 144 lasers per switch; the 36 laser inputs above mesh WDM networks, covering prediction-based and interval- describe a single package. The OIF implementation agreement based schedulers [17]–[19] and restoration mechanisms for for external laser modules, the external laser small form-factor channel and link failures [20], [21]. pluggable (ELSFP), allows an optional thermoelectric cooler The same problem recurs across the fabric in optical circuit and replacement at the faceplate [29]. An analyst reports switching. An OCS passes the signal through untouched at that each related Spectrum-X package carries 36 photonic any line rate, but demand shifts, so a scheduler must break the engines, 32 active and four held in reserve for redundancy [28]; demand matrix, the traffic each endpoint must send to each NVIDIA’s own description of the single-ASIC version lists 32 other endpoint, into a sequence of switch configurations. engines fed by 16 laser modules and mentions no reserve [9].

4

We apply that proportion of engines to the laser pool, so 32 active channels backed by a four-laser reserve is this paper’s calibration point throughout, and every later use of the four-laser reserve rests on this assumption. Other vendors also use external lasers. Broadcom’s Tomahawk 6-Davisson, sampling since October 2025, places 102.4 Tb/s of CPO behind field-replaceable ELSFP laser modules. Light sources are also changing form. Lightmatter’s Guide is a one-rack-unit light engine whose output would otherwise take about 18 ELSFP modules [30]. Lumentum has announced an eight-wavelength dense-WDM ELSFP with initial availability in the first half of 2027 [31]. How many lasers share an enclosure, the grouping behind the module-level correlation of Section V-V-A, therefore varies by design. The alternative keeps the optics in a pluggable module that can be serviced on its own. Mesh Optical’s linearpluggable optics drop the digital signal processor that retimes and equalizes the signal, and the vendor reports a draw under 10 watts at 1.6 Tb/s [32]. NVIDIA, which reported Spectrum-X Photonics in full production in June 2026, presents it as giving five times the power efficiency and five times the AI uptime of networks built on pluggable transceivers [33]. b) Optical circuit switching and MoE traffic: Jupiter brought optical circuit switching to data center scale under software-defined control [2], [34], and Sirius to nanosecond switching [35]. An earlier system, Helios, paired electrical packet switching with optical circuit switching in modular data centers and reconfigured the circuits as traffic shifted [36]. Other work ties the topology to the training job. TopoOpt optimizes the topology together with the parallelization strategy [37], and MixNet exploits the locality of expert parallelism in MoE training, beating a topology optimizer by 1.5× and 1.3× in its reported comparisons [5]. For scheduling, circuit switches classically use a Birkhoff–von Neumann (BvN) decomposition [38], which meets the busiest-port lower bound on the makespan, the total time the schedule holds its circuits, but does so by splitting demands across as many as (P −1)2 +1 configurations for P ports, each paying a reconfiguration delay. Faster heuristics, such as MixNet’s greedy reconfiguration [5], state no worst-case bound. Research schedulers model the cost of switching explicitly. Solstice schedules hybrid circuit and packet switches by heuristic and lands within 14% of optimal at scale, without a worst-case bound [39]. Eclipse carries a constant-factor guarantee on the traffic a circuit switch serves within a fixed time window [40], and Sunflow finishes a coflow, a group of flows that completes together, within 2× of the optimal time [41]. Solstice and Eclipse halt the whole switch for a delay δ at every change of configuration, and Sunflow halts only the ports being reconfigured. Work from 2025 and 2026 schedules circuit switches for machine-learning jobs specifically. Chronos derives circuit schedules ahead of time from the training program, leaving expert parallelism as the one pattern it must schedule on demand [42]. Spectra schedules one AI demand matrix across parallel circuit switches with the reconfiguration delay in the objective [43], LACE and Flux compile reconfiguration schedules from the structure of the whole workload [44], [45], DELTA engineers one static topology from the job’s dependency graph [46], OMLSched schedules jobs on optical ML clusters [47], and Silverstone

reconfigures the fabric across the phases of disaggregated RL [48]. Amponsah and Addanki show that the many matchings of a BvN decomposition fragment MoE execution into small batches [49]. Coflow schedulers for parallel and multi-core circuit switches now carry worst-case guarantees with the reconfiguration delay in the model [50], [51]; Chen’s makespan ratio grows with the number of flows at a port. Section IV-IV-B sets out how the decomposition works, what it costs, and how FoldFit’s bound compares with these. c) Microring stabilization and module reliability: A microring modulator responds to light only near one particular wavelength, its resonance, and that wavelength moves with temperature. A standard 220-nm silicon ring’s resonance shifts about 77 pm/°C near 1550 nm [52], and our model uses 70 pm/°C as a representative value. In NVIDIA’s design the lasers that feed it sit in a temperature-controlled module [9], so it is the ring’s drift, not the laser’s, that the link must track. Once a ring drifts past the largest offset the receiver tolerates, the receiver loses lock on it. Parts in the field stabilize the ring by reactive dither locking: the controller varies the ring’s setting slightly and reads how the output responds. That response shows which way the resonance has moved, so the correction comes only after an offset has appeared. A ring reacquires lock in roughly 200 µs [53], and its link carries nothing until it does. Integrated thermal stabilization of individual microring modulators has also been demonstrated in hardware [54], modelpredictive and feedforward control of microring modulators has been designed in simulation [55], and locking driven by the bit error rate now tracks 71.4 nm/s with no measurable error-rate penalty under a 40 °C disturbance [56]. Closest to ThermaLead, Yoon et al. show that the compute bursts of MoE training heat a wafer-scale optical interconnect by several kelvin within milliseconds and stall its rings unless the heater is pre-driven from GPU kernel launches or the tracking loop runs much faster [57]. ThermaLead instead forecasts the drift from temperature telemetry. Lumentum has demonstrated two external laser modules delivering 16 channels on a 200 GHz grid at about 24 dBm per channel [58], and Sirius-class tunable lasers switch within 912 ps [35]. However, the open issue for the laser modules is reliability: how many spare lasers a pool must hold. In [59], it was showed that an integrated laser, which cannot be cooled and runs at the 70 °C maximum chip temperature, needs a backup laser to meet a 1% annual failure-rate target, whereas an external source can be cooled by a thermoelectric cooler [59]. Pooling needs far fewer spares than a backup per laser: tunablelaser sparing restores a 32-wavelength link with only 4–8 shared spares under moderate load [21]. Neither analysis models correlated laser failures, so both correspond to the zero-correlation case, ρ → 0, to which the coverage model of Section IV-IV-A reduces. Studies of distributed storage show the cost of that omission. In Google’s storage fleet, failing to model correlated node failures typically overestimated availability by at least two orders of magnitude [60], and random replica placement is almost guaranteed to lose data when a cluster-wide power outage takes many nodes down at once [61]. We found no published vendor pool-sizing rule with a coverage guarantee.

5

d) Formal tools: Coverage bounds, the MULTIFIT packing ratios of Section IV-IV-B, one-step-ahead prediction, and threshold protection are the four formal tools this paper builds on. All four have a long record in optical WDM survivability and scheduling [17]–[21]. That work targeted fiber cuts at gigabit rates. The AI data center adds correlated thermal stress on shared laser pools and on microrings running at 200 Gb/s per lane, and each component here is designed anew for that setting. III. L ASER P OOLING : T HE C ORRELATION C EILING Pooling spares works like insurance, and it fails for the same reason. Ten thousand houses in different towns burn independently, so a small reserve covers the year’s claims. Ten thousand houses on one floodplain produce every claim in a single flood, and insuring more houses there adds exposure without diversifying it. Co-packaged lasers that share cooling and power resemble the floodplain: one shared cause can bring a burst of failures that overwhelms spares sized for scattered ones.

Symbol

Meaning

N n s; σ λ φ; tw ρ a, b F; F Θ AN (σ) A∗ Atgt σmax Acrit s ∗ ; s5 M; L ρwithin , ρacross ρ̄ τ B δ; m σx ; R

total pool size (active plus spare lasers) active channels the pool must feed spare count; spare ratio σ = s/N per-laser failure probability in a window laser FIT rate; window length (hours) pairwise (intraclass) failure correlation Beta parameters of the latent failure level failures in a window; failure fraction F/N latent common-cause failure level, Θ ∼ Beta(a, b) availability of an N -laser pool at spare ratio σ availability ceiling FBeta (σ; a, b) coverage target asked of the sizing algorithm budget cap on the spare ratio (2× pool: 1/2) ceiling at the cap, FBeta (σmax ; a, b) spares LightReserve returns; spares for 99.999% modules per pool; lasers per module correlation within / across modules pool-average pairwise correlation threshold: lock (ThermaLead), risk (TriageGuard) pre-spare budget (TriageGuard) switching delay; circuits per configuration ring-offset std. dev.; lag-1 autocorrelation

TABLE I N OTATION USED THROUGHOUT THE PAPER .

A. Mathematical Model 1) F converges almost surely to a random level Θ ∼ Definitions: We need to know whether the spares in a Beta(a, b), and Var(F ) → λ(1 − λ)ρ > 0. pool cover the lasers that fail in a window. Consider a pool 2) Availability, AN (σ) = Pr[F ≤ σN ] → A∗ (σ; λ, ρ) = of N lasers whose failures are exchangeable, meaning that no FBeta (σ; a, b). laser is special: any set of k lasers is as likely to be the failing That is, as a laser pool operating at a fixed value of σ set as any other. Failures are counted over a fixed window of grows, its availability approaches a limit below one that time: F is the number that fail in one window, and F = F/N does not depend on N . is the failure fraction. The window is the restock interval, the 3) Given a budget cap on the spare ratio, σ ≤ σmax < 1, time a pool runs before its failed modules are replaced and its the availability a pool can approach as it grows is at spares refilled, and every coverage and availability figure in ∗ most A (σ ; λ, ρ) < 1, so for any target above it, this paper is a per-window quantity. One laser’s probability of max every large enough pool held to that cap falls short of failing during one window is its marginal failure probability the target. λ, which later sections also call its per-window failure rate, so 4) For every σ > λ, A∗ → 1 as ρ → 0, so the ceiling of a λ = 0.05 means a one-in-twenty chance per laser per window. 2× pool reaches any target once ρ is low enough. Laser reliability is usually quoted as a FIT rate, failures per 9 10 device hours, and a laser at φ FIT with a window of tw Under independent failures the variance of the failure fraction hours has λ = 1 − exp(−φ tw × 10−9 ). The failure correlation shrinks as 1/N , so a bigger pool is a safer pool. Part (1) says ρ ∈ (0, 1), the pairwise (intraclass) correlation between failures, that under correlation the same variance flattens onto a floor, measures how strongly failures cluster. As ρ → 0 the failures and past a few hundred lasers the extra hardware buys almost become independent. A larger ρ means that a greater share nothing. of failures traces to one cause acting on many lasers at once. Against those failures the pool holds s spare lasers, giving a Proof. Everything turns on one object, the latent commonspare ratio σ = s/N . Deployments cap the ratio, and we write cause level Θ that de Finetti’s theorem [10] guarantees behind the budget cap as σmax ; the cap assumed throughout is the 2× any exchangeable failure process that can be extended to an pool, in which spares equal active lasers. The severity of the arbitrarily large pool. By de Finetti, an infinite exchangeable shared cause varies from window to window, and we model it binary sequence is a mixture of i.i.d. Bernoulli sequences: with the beta distribution whose mean is λ and whose intraclass there is a latent Θ with Xi | Θ i.i.d. Bernoulli(Θ), where correlation is ρ. Its shape parameters are a = λ(1 − ρ)/ρ and Xi indicates the failure of laser i. Read operationally, this b = (1 − λ)(1 − ρ)/ρ. Laser availability, which we also call says that if no laser is special and the pool could be extended coverage, is the probability that the spares cover every failure indefinitely, it must behave as though a hidden severity level in a window, written AN (σ). The rest of the paper states it as Θ were drawn first for the window, after which each laser fails a percentage in the industry’s notation, such as 99.999%, and independently at that level. Correlation is then what happens figures plot it on a logarithmic scale of 1 − AN labelled the when Θ varies between windows, and the beta-binomial model same way. Table I lists the notation used throughout the paper. assumes that a single scalar, ρ, is enough to describe how much Theorem 1 (Correlation ceiling). For the system described it varies. The beta-binomial model is the case Θ ∼ Beta(a, b), above, all of the following hold: with the parameters chosen so that E[Θ] = λ and Var(Θ) =

6

E[Θ(1 − Θ)] + Var[Θ] N λ(1 − λ)[1 + (N − 1)ρ] = , N whose limit is λ(1 − λ)ρ (= 0.0071 at the operating point of Section III-III-B). Part (2). Coverage at spare ratio σ holds if and only if F/N ≤ σ, so, since F/N → Θ almost surely and FBeta is continuous, AN (σ) → Pr[Θ ≤ σ] = FBeta (σ; a, b). That limit stays below one for σ < 1, because Beta(a, b) puts positive mass on (σ, 1]. Part (3). Since AN (σ) is nondecreasing in σ, a pool held to σ ≤ σmax has AN (σ) ≤ AN (σmax ), which converges to FBeta (σmax ; a, b) < 1 by Part (2); the cap bounds the limit, not every finite N . Part (4). As ρ → 0, a + b = (1 − ρ)/ρ grows without bound and Var(Θ) = λ(1 − λ)ρ → 0, so Θ concentrates at its mean λ < σ and A∗ → 1. Var(F/N ) =

Parts (2) and (3) describe the limit, and a finite pool can sit slightly above it: at the stress point, a large pool at the four-laser reserve’s spare ratio of 1/9 approaches 85.1%, below the coverage of the 32-channel reserve itself (Section IV-IV-A). At the 2× cap and ρ = 0.15, every pool size we evaluated approaches the ceiling from below, so there the ceiling of a 2× pool is a true upper limit. Part (4) states only the limit. Concentration alone does not make A∗ monotone in ρ at every σ: for a spare ratio only slightly above λ, a more correlated pool can hold more of its mass below σ, as the left panel of Fig. 3 shows below a spare ratio of about 14%. At σ = 1/2 we verified numerically that ∂A∗ /∂ρ < 0 over ρ ∈ (0, 1) for every λ ≤ 0.3, so at the 2× cap the ceiling A∗ strictly increases as ρ falls. B. Analysis The theorem supplies no value for λ or ρ, and every number below moves with both. The analysis takes a stress operating point of λ = 0.05 and ρ = 0.15. When the perlaser failure probability is small, ρ is the common-cause beta factor of reliability engineering: the share of failures that trace back to a single shared cause. Other industries have estimated it. IEC 61508-6 assigns values from 0.005 to 0.05 for programmable electronics and from 0.01 to 0.10 for field equipment, according to the defenses in place [12], and nuclear practice uses 0.05 to 0.10 for a redundant pair [11], [62]. Where the units share one die, IEC 61508-2 starts from 0.33 and accepts the redundancy as adequately separated only once the estimate is shown to be below 0.25 [63]. Lasers in one module have little physical separation, since they share its enclosure and temperature control and often

10−1

variance of failure fraction

λ(1 − λ)ρ, which makes the pairwise correlation of the Xi equal to Var[Θ]/[λ(1−λ)] = ρ. Each part below is a statement about that latent level. P Part (1). Conditional on Θ, the failure count F = i Xi is Binomial(N, Θ), so the strong law of large numbers gives F/N → Θ almost surely, and unconditionally the limit is Θ. The law of total variance then gives

10−2 10−3 10−4 10−5 10−6 10−7 101

102

103

104

105

pool size N (lasers) Fig. 2. Why correlated failures do not average out: the variance of the failure fraction F against pool size N , on logarithmic axes. With independent failures (falling line) the variance shrinks as 1/N . With correlation (ρ = 0.15, upper curve) it flattens at the floor λ(1 − λ)ρ = 0.0071 [derived].

one power supply, so a pool built from a few such modules sits away from the well-separated end of that range. We therefore take ρ = 0.15, between the 0.10 upper end of the IEC 61508-6 and nuclear ranges and the 0.25 on-die limit, as an estimate for a well-engineered package, a pool-wide value that Section V-V-A splits into within-module and across-module parts. It is an assumption, not a measurement: no field data for these laser form factors pins it down yet, and Section VII-VII-E describes the experiment that would. The choice is pessimistic for availability and favorable to our argument, since a lower ρ would raise the ceiling of Theorem 1 and so weaken the case for decorrelating. We also report the full ρ-sweep, including a plausible band of ρ ∈ [0.08, 0.35] for the cost results. The failure probability needs the same care, because the window turns a FIT rate into a probability. The OIF copackaging framework sets a reliability target of 50 FIT for the laser source of each 3.2 Tb/s engine over a six-year service life [64]. The target covers a whole source, and a source can hold several lasers, as NVIDIA’s eight-laser module does, so applying 50 FIT to each laser is a pessimistic reading of it. An average optical transceiver runs at about 200 to 300 FIT, and low-cost lasers recently deployed in data centers failed at ten times their target rate [65]. Against those figures, λ = 0.05 is a stress setting: it is what a laser at about 1,000 FIT accumulates when it is never replaced over a six-year life. For co-packaged optics, Broadcom reports one million 400G-equivalent porthours without a link flap in Meta’s high-temperature lab [66], which caps the per-port flap rate near 3,000 FIT at 95% confidence, above both 50 FIT and the stress setting. Table II evaluates the same pool at rates nearer the targets. For the 32-channel pool at ρ = 0.15, 99.999% stops fitting inside the 2× cap once λ exceeds about 2.4 × 10−4 per window. That happens when the FIT rate times the restock interval passes about 2.4 × 105 FIT-hours, after about 200 days at 50 FIT and about 40 days at 250 FIT. Fig. 2 shows Part (1) at this operating point: the failure fraction keeps a standard deviation of 0.084 however large the pool grows. In the left panel of Fig. 3, 99.999% availability at this

99.999999%

99.999999%

99.99999%

99.99999%

ceiling at the 2× cap

coverage ceiling

7

99.9999% 99.999% 99.99% 99.9% 99% 90%

99.9999% 99.999% 99.99% 99.9% 99% 90%

0

10

20

30

40

50

60

70

80

spare ratio σ = s/N (%)

90

0.25

0.20

0.15

0.10

0.05

0

failure correlation ρ

Fig. 3. The correlation ceiling. Left: the coverage ceiling against the spare ratio σ, one curve per correlation ρ. The dashed line is the 99.999% target and the hatched band marks spare ratios past the 2×-pool budget cap. At ρ = 0.15 the curve reaches the cap below the target. Right: the ceiling a fixed 2× pool attains as ρ falls (correlation decreases to the right); the filled dot marks the ρ = 0.15 operating point at 99.67%, and the open dot the correlation, about 0.056, at which the pool reaches 99.999%. Spares move along one curve, and only decorrelation moves across curves to 99.999% [derived].

φ (FIT) Window

λ Acrit (%) s5 , ρ = 0.15 s5 , indep.

Coverage ignores how long an exhaustion lasts, and effective availability also counts how quickly it is repaired. Take a 50 90 days 99.9996 25 1 50 1 year 4.4 × 10−4 99.9985 39 2 pool sized for 99.9% coverage, which is 49 spares on the 250 1 year 2.2 × 10−3 99.9922 61 3 32-channel calibration point of Section IV-IV-A. TriageGuard 500 1 year 4.4 × 10−3 99.9839 73 4 (Section IV-IV-D) keeps hot spares pre-wired to the modules 976 6 years 0.05 99.6697 156 10 most likely to fail, so a failure on such a module switches TABLE II to its hot spare in milliseconds instead of lasting the rest of W HERE THE CEILING BINDS , FOR THE 32- CHANNEL POOL . E ACH ROW the window, and those hot spares sit outside the shared pool, TURNS A LASER FIT RATE φ AND A RESTOCK INTERVAL tw INTO THE PER - WINDOW PROBABILITY λ. Acrit IS THE LARGE - POOL CEILING OF A 2× so they are still there when the pool runs out. Only failures POOL (C OROLLARY 1), AND THE 32- CHANNEL POOL REACHES SLIGHTLY on the pre-spared modules are shortened, however. Under the LESS . s5 IS THAT POOL’ S SPARE COUNT FOR 99.999% COVERAGE , exchangeable failures of Theorem 1 every module carries the CORRELATED AND INDEPENDENT; THE 2× CAP ALLOWS 32, WHICH SETS THE 2.4 × 10−4 THRESHOLD . T HE 250 FIT ROW SITS IN THE RANGE OF AN same risk, so hot spares on 4 of the 18 modules see on average AVERAGE OPTICAL TRANSCEIVER , THE 500 FIT ROW IS TEN TIMES THE 4/18, about 22%, of the failures in an exhausted window, OIF TARGET, AND THE LAST ROW IS THE STRESS OPERATING POINT USED and a risk ranking raises that share only as far as module THROUGHOUT [ DERIVED ]. risks differ and are predicted (Section IV-IV-D). Fast recovery therefore narrows the gap to 99.999% by an amount we have not computed, and replacements must also keep arriving to operating point requires a spare ratio of σ = 0.821. Since restock the pool. σ = s/N , the total pool must then be 1/(1 − σ) ≈ 5.6× the The pairing of coverage and fast recovery has two further active laser count, which no deployment would carry; that is limits. First, those 49 spares make a 2.5× pool before the the large-pool limit, and Section IV-IV-A gives the finite-pool hot spares are counted. That is less than a third of the 156 answer in whole lasers. spares the coverage-only route needs, but the pairing is not Corollary 1 (Critical availability). Given a budget cap σmax , shown to reach 99.999%, and even its starting pool sits above the value Acrit = FBeta (σmax ; a, b) acts as a threshold for the 2× budget cap, so fast recovery cannot bring the target large pools: a target below Acrit is reachable with a spare inside the cap. Only decorrelation, priced in Section V-V-B, ratio inside the cap once the pool is large enough, while a fits inside the cap. Second, LightReserve counts spares in target at or above it cannot be reached by any ratio inside the lasers while TriageGuard hands out hot spares a module at a cap, and the pool multiple 1/(1 − σ) it would demand grows time, so combining them in a deployment requires fixing the without bound as the target approaches one. For a 2× pool modules-of-lasers shape of Section V-V-A first. at the stress rate, Acrit = 99.670% at ρ = 0.15, rising above 99.999% once ρ falls below about 0.056. IV. P ROPOSED C ONTROL P LANE D ESIGN 1.1 × 10−4

At this operating point no spare ratio inside the cap reaches 99.999%, and a low enough correlation does, since the ceiling of a 2× pool tends to one as ρ falls (Part (4) of Theorem 1). Section V-V-B prices decorrelation, and the right panel of Fig. 3 shows the rise for a fixed 2× pool. Fast recovery offers a partial second route toward 99.999%, because it shortens an exhaustion that coverage fails to prevent.

This section presents the details of the proposed Halley framework. Four components, one per layer, sit above the optical data plane, as shown in Fig. 4. Table III lists the decision each one makes, its guarantee or, for ThermaLead, its benchmark, and its reactive fallback. All four follow one rule: each acts on a prediction of its layer’s next state, and when its forecast is stale or unsure, it steps aside and the system runs

8

LightReserve

FoldFit

ThermaLead

TriageGuard

sizes the spare pool

schedules the switch

pre-positions ring heaters

pre-spares high-risk modules

coverage bound; flags unreachable

total dwell ≤ 11/9 of optimal

benchmark residual σx2(1 − R 2)

top-B optimal

Shared laser pool

Optical circuit switch

Microring modulators

Field-replaceable modules

by exchange

Fig. 4. The Halley control plane, with one component per data-plane layer: LightReserve over the shared laser pool, FoldFit over the OCS, ThermaLead over the microrings, and TriageGuard over the replaceable modules. Dashed arrows send each layer’s telemetry up into a prediction, and solid arrows send the resulting action back down. The shaded band in each component names its formal guarantee in the idealized form of Table III, or for ThermaLead the benchmark its forecast is measured against [schematic]. Component

Data-plane layer

Decision it makes

Guarantee or benchmark

Reactive fallback

spare count s∗

LightReserve shared laser pool coverage bound; flags unreachable static n+k reserve FoldFit OCS configurations and dwell times total dwell ≤ 11/9 of optimal BvN or greedy decomposition ThermaLead microring modulators heater pre-position each step benchmark residual σx2 (1 − R2 ) reactive dither locking TriageGuard field-replaceable modules which modules get hot spares top-B optimal by exchange uniform sparing from the pool

exactly the deployed policy it replaces. A confident but wrong forecast has no such floor. A. LightReserve: Sizing the Laser Pool LightReserve decides how many spares a shared laser pool carries. Operators fix the number once, as a static n+k reserve, and hold it whatever the failure rate turns out to be. LightReserve computes it from the failure rate and from how strongly the failures are correlated, and returns it with a provable coverage guarantee for those inputs. Algorithm 1 models the total pool N = n + s and not the active set alone, because the spares are themselves failure-prone.

100

100

90

90

coverage (%)

Algorithm 1 LightReserve pool sizing 1: input: active channels n, failure rate λ, correlation ρ, target Atgt 2: model total pool N = n + s; spares are themselves failureprone; coverage holds iff F ≤ s 3: A(s) ← Pr[F ≤ s], F ∼ Binomial(N, λ) or Beta-Binomial(N, a, b) 4: s∗ ← min{ s : A(s) ≥ Atgt } ▷ A(s) increasing; binary search 5: if s∗ /(n + s∗ ) > σmax then 6: report UNREACHABLE and emit the ρ′ that brings Atgt under the ceiling 7: end if 8: output: s∗ spares, or a decorrelation target when the ceiling forbids the goal

coverage (%)

TABLE III T HE FOUR CONTROL - PLANE COMPONENTS . L IGHT R ESERVE , F OLD F IT, AND T RIAGE G UARD CARRY A FORMAL GUARANTEE , SHOWN HERE IN IDEALIZED FORM ; T HERMA L EAD ’ S ENTRY IS A BENCHMARK . F OLD F IT ’ S 11/9 HOLDS UNDER THE THREE CONDITIONS OF S ECTION IV-IV-B AND LEAVES OUT SWITCHING DELAY. T HERMA L EAD ’ S BENCHMARK IS THE RESIDUAL VARIANCE OF THE MINIMUM - MEAN - SQUARE - ERROR FORECAST FROM THE PREVIOUS SAMPLE FOR FIRST- ORDER AUTOREGRESSIVE (AR(1)) DRIFT. I TS MOVING - AVERAGE FORECAST IS CHEAPER , AND ON THE STRESS TRACE OF S ECTION IV-IV-C IT DOES NOT REACH THIS FIGURE . T RIAGE G UARD ’ S TOP -B OPTIMALITY IS CONDITIONAL ON THE RISK ESTIMATE IT IS GIVEN .

80

70

60

80

70

0

5

10

15

20

spare lasers

25

60

0

0.05

0.10

failure rate λ

Fig. 5. LightReserve dimensioning at the 32-channel calibration point. Left: coverage against spare count, one curve per failure rate, with failures assumed independent; the spare count for 99% coverage varies by 3× across these stress rates. Right: at ρ = 0.15, coverage against failure rate for the fixed four-laser reserve and for a pool that LightReserve sizes to a 99.9% target at each rate [derived].

If the spare ratio at s∗ , the smallest count whose coverage meets Atgt , exceeds the budget cap σmax , the algorithm reports UNREACHABLE and, in place of a spare count, emits the correlation ρ′ that would bring the target under the ceiling, which turns an impossible sparing problem into a concrete decorrelation target. Fig. 5 gives numerical results for the calibration point of Section II-II-B. With independent failures (left panel), LightReserve returns the classical small spare counts: 5 spares

9

six 1-slot jobs, then one 3-slot job m = 3 channels 9

1

1

1

8

1

1

7

2

2

6 C1

C2

C3 greedy runs 5/3 = 1.67× optimal

Fig. 6. FoldFit worked example. Left: the demand matrix of a MoE collective, each cell a traffic demand measured in slots. Middle: the schedule FoldFit folds it into, where each filled cell is an active circuit and t gives each configuration’s dwell time. Right: the canonical worst case for greedy list scheduling, which serves demands in arrival order [derived].

Algorithm 2 FoldFit (binary-searched first-fit-decreasing, FFD, demand decomposition) 1: input: demand matrix D, channel capacity C, channel count m, MoE block structure 2: jobs ← nonzero demands of D (in slots at capacity C), each tagged with its (row, col) Plocality 3: Tlb ← the largest of max D, D/m, and the largest row or column sum ofP D 4: Thi ← max{Tlb , D/m + max D} 5: binary-search the deadline T ∈ [Tlb , Thi ], packing jobs by FFD at each candidate T 6: T ∗ ← the feasible T on which the search ends ▷ after q steps, T ∗ ≤ (13/11 + 2−q ) OPT 7: break ties to keep expert-parallel blocks together, so stable configurations are reused 8: output: configurations + dwell times, makespan T ∗ ≤ (11/9) OPT for q ≥ 5, under the conditions stated in the text

cap (Table II), whereas lowering ρ cuts the count at every rate. B. FoldFit: Scheduling the Circuit Switch

FoldFit schedules the OCS that joins the photonic engines to one another. The switch holds only a limited set of connections at a time, so a communication phase is served as a sequence of configurations, each held for a while before the switch is reconfigured. A good sequence is short, because the collective waits while each configuration is held, and stable, because every reconfiguration costs switching time. FoldFit chooses the sequence by recasting the problem as packing: each traffic demand becomes a job, and the jobs are packed onto the parallel circuits the switch can hold at once. The fabric is a single-stage OCS with crossbar semantics. At any moment it holds one configuration, a set of port-disjoint circuits, meaning that no two of them share a switch port. Traffic between two engines flows only while a circuit connects them, so serving a workload means choosing a sequence of configurations and a dwell time for each. Those dwells sum to for 99% coverage at λ = 0.05, 9 at 0.10, and 16 at 0.20. As the makespan of Section II-II-B. Each change of configuration also costs a fixed switching delay δ, which the makespan as ρ → 0 the method reduces to that known result. At λ = 0.05 and ρ = 0.15 the four-laser reserve gives defined here leaves out. The input is the demand matrix D, A(4) = 86.3%, and 99% coverage needs 23 spare lasers with entry Dij the traffic engine i must send to engine j, where independence needed 5. Reaching 99.9% takes 49, and expressed in time slots at the per-channel capacity C. The classical way to choose that sequence is the BvN 99.999% takes 156, a 5.9× pool. The algorithm flags that pool as UNREACHABLE and returns a correlation target in decomposition [38], which pads the demand matrix until every its place: 99.999% within a 2× pool of 32 channels needs row and column sums to the busiest port’s load and writes it as ρ ≤ 0.046, a little below the large-pool value of about a weighted sum of permutation matrices, each a configuration 0.056 in Corollary 1. The flag concerns coverage only; fast whose weight is its dwell. The dwells then meet the busiest-port recovery (Section III-III-B) narrows the effective-availability bound, but over as many as (P − 1)2 + 1 configurations for gap, without a computed guarantee and from a pool still above P ports, most with tiny dwells, each costing a reconfiguration delay that nothing in the decomposition bounds. the cap. The right panel sweeps the failure rate: the fixed reserve’s FoldFit replaces the decomposition with a packing, a problem coverage falls steeply as the rate grows, while a pool that that has carried a worst-case makespan bound since the 1970s. LightReserve sizes to each rate holds its 99.9% target, though FoldFit treats each nonzero demand as an indivisible job above about λ = 0.02 that sizing needs more than the 32 spares of length Dij slots and the fabric as m identical service the 2× cap allows. For 99.999% coverage, which the panel channels, where m is the number of port-disjoint circuits does not plot, the correlation limits the pool more than the rate one configuration can sustain. These channels are the parallel does: at ρ = 0.15, even λ = 0.005, a tenth of the stress rate, machines of the packing problem, not the wavelength channels still needs 76 spares, more than twice the 32 the cap allows. of the laser pool. The target is OPT, the shortest schedule Only a rate below about 2.4 × 10−4 brings 99.999% inside the that serves all jobs on m parallel channels without two circuits

10

sharing a port at the same time. Dropping the port condition One refinement adapts the packing to MoE traffic. Expertleaves the classical makespan problem, whose optimum is no parallel demand is block-structured and drifts slowly as expert longer. popularity shifts. When several placements tie during FFD, Three lower bounds on that optimum are immediate. No FoldFit keeps jobs from the same expert-parallel block in the schedule P beats the largest single demand; none beats the average same configuration, so consecutive scheduling rounds reproduce load ij Dij /m; and none beats the busiest port’s total traffic, the same configurations and the switch can hold them instead of the largest row or column sum of D, because a port carries rebuilding them. The simulation of Section V-V-F replaces first one circuit at a time. Starting from those bounds, Algorithm 2 fit with this preference and, failing it, the least-loaded channel, binary-searches the deadline T , a candidate makespan, running so it evaluates a variant that the 11/9 proof does not cover. FFD at each candidate T . FFD sorts the jobs from longest to Its baseline, a longest-processing-time (LPT) list schedule that shortest and places each in turn on the first channel whose places one demand in five on a random channel, spreads every load would stay within T . If every job fits, T is feasible. The block over every channel. Against it, FoldFit cuts a proxy for deadline T ∗ on which the search ends fixes the assignment of reconfigurations, the number of distinct expert blocks each jobs to channels. channel serves, by 45% at 16 engines. At 8 engines, a single Binary-searched FFD is the MULTIFIT algorithm of multi- block, the two tie. processor scheduling, here with its search started at Tlb , and Fig. 6 works through a 4 × 4 MoE demand matrix, dense it inherits the MULTIFIT theorem. On identical channels, on the expert-block diagonal with entries of 9, 8, 7, and 6 q steps of the search give T ∗ ≤ (1.22 + 2−q ) OPT [67], slots and light off the diagonal. The busiest port carries 11 and Yue tightened the constant to the exact ratio, T ∗ ≤ slots, so no schedule is shorter than 11, and Algorithm 2’s (13/11 + 2−q ) OPT [68]. Both bounds are stated against search stops at that bound, T ∗ = 11. Turning the packing into the classical optimum, which is no longer than OPT. Every port-disjoint configurations costs one more slot: C1 carries the deadline at which FFD fails lies below 13/11 of the classical diagonal entries and dwells 9 slots, C2 the exchanges inside optimum, P the search starts at Tlb ≤ OPT, and its upper end, each two-engine expert block for 2, and C3 the cross-block max{Tlb , D/m + max D}, keeps the search interval no residual for 1. The makespan is 12 slots before switching delay, wider than the classical optimum, so both ratios hold against a ratio of 1.09 to the lower bound; the extra slot is an alignment OPT. They need not hold against the classical optimum itself, loss that the 11/9 ≈ 1.22 guarantee does not cover, and the which the busiest-port bound can exceed several times over. ratio still falls inside it. A BvN decomposition of the same With five or more steps Yue’s guarantee sits below 11/9, the matrix meets the 11-slot bound on dwell by splitting demands; figure we quote throughout. Under the three conditions below, the decomposition we computed uses five configurations to the schedule is then never more than 22% longer than the FoldFit’s three, so it pays the switching delay δ five times optimum however the demand matrix moves, and the exact instead of three. bound narrows that margin toward 18% as q grows. a) How this differs from MixNet: MixNet [5] exploits MULTIFIT is a result about m identical machines, and it the same expert-parallel locality and reconfigures the switch applies to a switch under three conditions. First, the channels in advance from a prediction of the first all-to-all’s demand, have to be interchangeable, which holds while every circuit but gives no worst-case bound, so on an adversarial shift runs at the same per-channel capacity C. Second, each demand its makespan is unconstrained. FoldFit adds a bound that has to be packed as one indivisible job on one channel. Third, holds however MoE traffic moves mid-training, under the three FFD’s assignment of jobs to channels has to be realizable as a conditions above. sequence of port-disjoint configurations. b) How this compares with circuit-switch schedulers: The classical setting does not supply the third condition, Sunflow [41] finishes a single coflow within 2× of the optimal because two jobs sitting on different channels may share a time for any reconfiguration delay δ, and its bound already switch port, and no configuration can run both at once. We call counts the delay of every reconfiguration it makes. FoldFit’s the step that turns the assignment into port-disjoint configura- 11/9 is tighter but narrower. It bounds the dwell time T ∗ , tions the alignment. The busiest-port bound is necessary for an and a schedule with kr reconfigurations takes T ∗ + kr δ in alignment to exist at all, which is why the binary search starts all, a second term the bound does not cover. Commercial no lower than it. When a demand may be split at a configuration OCSs typically switch in 10–20 ms [69], so with the worked boundary, which is the relaxation BvN works in, the alignment example’s three configurations that term stays under 1% of is exact and the 11/9 guarantee carries over untouched. When the dwell only for phases of 3–6 s or longer, and rescheduling demands are strictly unsplittable, the alignment can leave idle within one MoE layer’s dispatch would need microsecond-class gaps that the 11/9 bound does not account for, and the worked or faster switching [35]. Keeping the shorter of the FoldFit example below loses one slot this way. On the workloads swept and Sunflow schedules bounds the total by the smaller of in Section V-V-F, FoldFit’s makespan lands within 6% of the (11/9) OPT + kr δ and twice Sunflow’s optimum, in Sunflow’s larger of the first two lower bounds above on every instance, switch model. Solstice [39] minimizes a similar total, dwell and within about 1% on average from 16 engines up. The time plus δ per configuration, by heuristic and without a worstsimulation packs jobs onto channels without the port condition, case guarantee. Eclipse [40] maximizes the traffic served within so that evidence covers the packing and says nothing about a fixed window, a different objective. alignment gaps; Section VII-VII-D records the unsplittable c) Why the tighter bound matters: Plain LPT list schedulcase as open. ing carries Graham’s 4/3−1/(3m) ratio [70]. That ratio equals

11

Algorithm 3 ThermaLead one-step-ahead microring control 1: input: per-ring temperature history, ring slope 70 pm/°C, lock threshold τ 2: x̂[t+1] ← one-step predictor (8-sample moving average of dT /dt) 3: pre-position the heater to cancel x̂[t+1] 4: residual r[t+1] = x[t+1] − x̂[t+1] ▷ only what the prediction missed

11/9 at m = 3 and is tighter at m = 2, so the quoted 11/9 improves on it only for m ≥ 4, while Yue’s exact 13/11 improves on it for every m ≥ 3. The right panel of Fig. 6 shows the canonical worst case for list scheduling: six unit demands followed by one three-slot demand on m = 3 channels. A greedy scheduler serving demands in arrival order spreads the six unit jobs across all three channels and must then run the long job alone, for a makespan of 5 slots against an optimum of 3. That is 5/3 ≈ 1.67× optimal, exactly Graham’s 2 − 1/m bound for list scheduling at m = 3 [70]. FFD’s decreasing order places the long job first and stays exactly optimal here. Expert popularity shifting mid-training or mid-serving can produce orderings of exactly this shape.

C. ThermaLead: Steadying the Microring The receiver loses lock once the ring’s resonance offset exceeds the lock threshold, the largest offset it tolerates. Parts in the field use the reactive dither locking of Section II-II-B, which corrects an offset only after observing it, so the receiver absorbs each step’s drift before the correction lands. ThermaLead adds a feedforward term. It predicts the next-step drift from a short history and pre-positions the heater one step early, so the receiver sees only the error of that forecast. The heater and its authority are the same in both schemes; only the excursion, the swing of the offset at the receiver, changes. In Algorithm 3, x[t] is the ring’s resonance offset at control step t, in picometers via the 70 pm/°C slope. The 8-sample moving average of line 2 costs an adder and a shift, so it is cheap enough to sit beside the local servo, the circuit that sets the heater from the temperature error, which Section VI builds in hardware. The residual r[t+1] of line 4 is the part of the drift that the predictor missed and that is left for the receiver. ThermaLead’s benchmark is the residual of the best one-step predictor from the previous sample. We model the drift x[t] as wide-sense stationary over the window the predictor sees. Its variance is σx2 (this subsection’s σ is a standard deviation of the resonance offset, not the spare ratio of Section III), and its lag-1 autocorrelation R is the correlation between one step’s offset and the next. By the orthogonality principle of estimation theory, the minimum-mean-square-error predictor leaves residual variance σe2 = σx2 (1 − R2 ) < σx2 , the best any predictor can do from the previous sample alone. The expression is exact for AR(1) drift; the evaluation’s trace adds a periodic heat cycle to an AR(1) process with noise bursts, so on that trace it is an approximation. The moving average of line 2 is a cheaper forecast that extrapolates the recent slope.

It uses eight samples, so the bound above does not cover it, and the simulation below measures what it achieves. Treat the offset as Gaussian, a modeling assumption that the trace below satisfies only approximately. An offset of standard deviation σ then exceeds the lock threshold τ with probability 2Φ̄(τ /σ) on any given step, where Φ̄ is the standard normal upper-tail probability. Shrinking σ from σx to σe therefore strictly lowers the lock-loss rate, and the larger R is, the larger the gain. The lock-loss argument fixes neither the threshold nor the control step. The lock-loss rate depends on the threshold only through the ratio τ /σ, so shrinking σ lowers the rate wherever a particular receiver’s threshold sits. The control step is only a unit of counting: the analysis and the counts below are both per step, and any reduction is a ratio of two per-step counts, so no particular duration is assumed. The step must be short against the thermal time constant, so that a one-step-ahead forecast has something to predict, and long enough for the heater to act. Section VI shows that the heater drive is not the binding constraint; the heater’s own thermal response is. The evaluation models the thermal environment of training as a periodic heat cycle, a sinusoid with an 8 °C swing and a period of 120 control steps. The cycle stands for the bulksynchronous alternation in which each compute phase heats the die and each communication phase lets it cool. On top of it sits an AR(1) residual with coefficient 0.8, whose noise has a standard deviation of 0.55 °C on a random 30% of steps and 0.12 °C on the rest. The lock threshold is 25 pm. We treat this trace as a stress case. In simulation the deployed dither lock cancels every offset once it has observed it, so the receiver sees each step’s full drift. On the stress trace the dither lock loses lock on 462 of every 2000 control steps and ThermaLead on 472, averaged over 30 seeds. Paired seed by seed, ThermaLead loses lock 2% more often (p = 0.003, paired t-test) and does worse on 22 of the 30 seeds. Its residual variance is 1.3% above the drift’s own, so its forecast removes nothing. The closed form above explains part of this. The drift’s lag-1 autocorrelation is only 0.075, so even the best linear forecast from the previous sample could remove less than 1% of its variance. The trace is not unforecastable, however. An oracle forecast that knows the trace’s sinusoid and noise state, a bound and not a controller, would cut the count to 315, 32% below the dither lock. ThermaLead gains as the drift gets smoother. When noise bursts hit 10% of steps instead of 30%, it loses lock 22% less often than the dither lock, and with no bursts 78% less often. A gate on the forecast’s own error removes the stress-trace deficit. Applied only while its mean absolute error over the last 16 steps is below the raw drift’s, the gated controller loses lock on 452 steps, 2.1% fewer than the dither lock and better on 23 of the 30 seeds, and keeps at least 95% of the forecast’s gain on smoother traces. The rule was fixed before any run; other windows and error measures give 448 to 487, so the result depends on it. On the burst sweep the drift’s lag-1 autocorrelation was 0.13 or more wherever ThermaLead helped and 0.075 where it lost, so ring telemetry could tell an operator which case holds. ThermaLead is therefore the least proven of the four: its value depends on how smooth the field’s drift is, and

12

1.00 stale telemetry

risk score

0.75

0.50

fresh telemetry

0.25 oracle 0.00

M7

M2

M5

M9

M1

M4

M6

M3

0

5

10

15

MTTR cut vs uniform sparing (%)

Fig. 7. TriageGuard worked example. Left: the risk scores of eight of the pool’s modules, with the risk threshold τ = 0.35 dashed; the four modules above it get pre-allocated hot spares, and the rest fall back to the shared pool. Right: in simulation, the cut in mean time to recover against uniform sparing of the same four-module budget, for rankings on stale and on fresh telemetry and for an oracle ranking that knows each module’s failure risk [composed].

Algorithm 4 TriageGuard risk-ranked selective protection 1: input: risk scores r1 ≥ · · · ≥ rM , budget B, threshold τ , switch tsw (5 ms), miss tmiss (150 ms) 2: pre-spare the top-B modules with r > τ ▷ fast path 3: route the remainder to the LightReserve pool ▷ dynamic path 4: on a failure, recovery = tsw if pre-spared, else tmiss

Section VII-VII-D shows its ungated deficit growing under abrupt transients. ThermaLead also serves as a sensor. The per-ring temperature telemetry it samples every control step is the newest thermal signal in the package, and TriageGuard’s risk ranking runs on it.

while a failure anywhere else draws from the shared pool at tmiss = 150 ms. Call the share of expected failure probability that lands on protected modules the intercepted mass µ. The mean time to recover (MTTR) is µ tsw + (1 − µ) tmiss , so MTTR falls linearly as the ranking intercepts more of the mass. Uniform sparing of B modules out of M intercepts B/M of it on average, and a risk ranking improves on that only as far as it aims at the modules that will fail. The all-dynamic baseline, in which every failure draws from the shared pool, bounds the MTTR of any ranking from above, because a protected failure can only shorten a repair.

Fig. 7 runs this procedure on the shape of Section II-II-B, 18 field-replaceable modules of 8 lasers each, with hot spares for 4 of the 18; its left panel is a worked example at τ = 0.35. The simulation draws each module’s true risk from its temperature, D. TriageGuard: Protecting the Modules age, and FEC bin, fails each module with a probability that A fixed budget of hot spares cannot protect every module, rises with that risk, and pre-spares the four modules ranked so the budget should go to the modules most likely to fail. highest. The ranking sees the true risk through Gaussian error Deployed systems hand out spares uniformly from the shared with a standard deviation of 0.30 on stale telemetry and 0.05 pool. TriageGuard scores each module on its temperature, its on telemetry that ThermaLead refreshes each step. An oracle age, and its observed forward-error-correction (FEC) bin, the ranks on the true risk itself. Over 30 seeds of 400 episodes, band of corrected-error counts the link reports. The result is with every policy scored on the same draws, the stale ranking the module’s risk score, the estimate of how likely it is to fail. intercepts 0.264 of the failures, the fresh ranking 0.307, and It then pre-spares every module above a risk threshold τ , up the oracle 0.311. Uniform sparing intercepts 0.222, close to its to the pre-spare budget B, the number of modules that can expected 4/18. Averaged over the seeds, mean time to recover hold a hot spare, and routes the rest to the shared pool. falls by 5.1%, 10.4%, and 10.9% against uniform sparing, each Protecting the top-B risks is optimal, by an exchange within about one point at 95% confidence. Fresh telemetry thus argument. The expected number of protected failures is additive recovers nearly all of the oracle’s margin, and that margin is over the protected set, a sum of one term per protected module. modest at a budget of 4 modules in 18. The two error levels Swapping a protected low-risk module for an unprotected are assumptions, not measurements. Field rankings suggest higher-risk one never lowers the sum, and repeating such such concentration is attainable: on a production GPU fleet, a swaps turns any selection into the top-B selection. Greedy learned risk ranking catches 64% of future failures in its top top-B selection is therefore exactly optimal for any fixed risk 5% of nodes [71], and telemetry-based failure prediction for estimate. The argument settles how to spend the budget once optical transceivers reaches an F1 score of 0.89 on production the modules are ranked and says nothing about whether the data [7], [8]. Neither measures the intercepted mass of a coranking is right, so optimality is conditional on the risk estimate, packaged module pool. Under these assumptions the fresh and the freshness of that estimate dominates the result below. telemetry ThermaLead supplies doubles the cut a stale ranking Algorithm 4 links ranking quality to recovery time. A failure achieves, and a field measurement of the intercepted mass is on a pre-spared module switches to its hot spare in tsw = 5 ms, what would confirm it.

13

99.99999%

coverage

99.9999% 99.999% 99.99% 99.9% 99% 90%

Consider a pool of M modules holding L lasers each, in which a single laser fails with marginal probability λ, pairs in the same module are correlated at ρwithin , and pairs in different modules at ρacross , with ρwithin ≥ ρacross > 0. Concretely, a pool-wide level Θ0 ∼ Beta(a, b) is drawn with a and b set by λ and ρacross ; each module j then draws its own level Θj , independently given Θ0 , from a beta law with mean Θ0 and intraclass correlation (ρwithin − ρacross )/(1 − ρacross ); and the lasers of module j fail independently at level Θj .

Theorem 2 (Hierarchical ceiling). For the hierarchical system described above, the following statements hold true: 0 8 16 24 32 40 48 56 64 spare lasers (32 active) 1) As the module size L grows with M fixed, the failure fraction F/N has limiting variance λ(1 − λ)[ρacross + Fig. 8. Hierarchical ceiling for the 32-channel pool. The curves plot coverage (ρwithin − ρacross )/M ], so the within-module term is against the number of spare lasers. The dashed curve is a flat pool in which suppressed by 1/M , and as modules are added the acrossevery pair of lasers has correlation ρ = 0.15. The solid curve is the same module correlation sets the floor. pool with the average correlation of the modeled 18 × 8 module structure, ρ̄ = 0.062, evaluated with the flat model [derived]. 2) As the module size L grows with M fixed,Pthe failure M 1 fraction converges to the average Θ̄ = M j=1 Θj of the per-module failure levels, so availability converges V. P ERFORMANCE E VALUATION to a ceiling Pr[Θ̄ ≤ σ] < 1, exactly as in Theorem 1. 3) The flat model of Section III is the exact special case This section turns the ceiling of Section III into design ρ within = ρacross . guidance and then evaluates the integrated control plane. Theorem 2 says where in the pool to decorrelate, Observation 1 By Part (1), adding modules averages away the correlation prices separation against spare lasers, and Observation 2 shows inside a module and leaves the correlation between modules that the prescription holds across the plausible band of the untouched, so a designer gains more from separating modules correlation. Theorems and propositions are proved, apart from than from separating the lasers within one. the steps their proofs mark as numerical checks; an observation is a computed finding under the cost model stated below. Proof sketch. Part (1) is the law of total variance applied to the Section V-V-D reports the integrated goodput, its ablation, and two exchangeable levels, and the factor 1/M appears because its tail, and Proposition 1 gives a closed form for the part of that the within-module component is averaged over M module tail the correlation drives. The relevant software implementation draws that are independent given Θ0 . Part (2) repeats the lawof these components and the hardware realization is available of-large-numbers argument of Theorem 1 one module at a time: at [13]. condition on the per-module levels Θ1 , . . . , ΘM (indexed by j), and the pool’s failure fraction converges to their average. Part (3) is immediate, since equal correlations collapse the two A. Hierarchical Correlation Ceiling levels into a single beta-binomial, which reproduces the flat Lowering the correlation raises the ceiling, but a single model of Section III digit for digit. pool-wide correlation gives a designer nothing to act on, The pool-average pairwise correlation is because it does not say which lasers to pull apart from which. nsame ρwithin + ncross ρacross Section III gave every pair of lasers the same correlation ρ. ρ̄ = , nsame + ncross Deployments are not built that way. The lasers sit in fieldreplaceable modules, and a laser shares more with the lasers where nsame counts the pairs inside one module
2 and ncross the M n = in its own module than with the rest of the pool. We model pairs spanning two. There are cross 2 L cross-module
a shape of 18 modules of 8 lasers, the module count and pairs against nsame = M L2 same-module pairs, a ratio of module size that NVIDIA’s figures imply for one Quantum- (M − 1)L/(L − 1) that is close to the module count M . At the X switch (Section II-II-B). The active channels and spares 18 × 8 shape that is 9,792 cross-module pairs against 504 sameof Section IV-IV-A are drawn from a population of this module ones, roughly 19 to 1, so ρ̄ tracks the across-module kind. A module is the unit of thermal and power isolation, correlation almost entirely. Evaluated at ρ̄, the flat, single-level and separate modules are kept apart by design. Two lasers beta-binomial model of Section III matches the tail of the true in the same module therefore fail together with correlation nested process to within about one percentage point from 8 ρwithin , and two lasers in different modules with the smaller spares up, and to within 0.63 points from 12 spares up. This ρacross < ρwithin . Reliability engineering treats common causes shortcut, the flat model evaluated at ρ̄, is an approximation, so in the same grouped way. The alpha-factor and multiple-Greek- wherever we use it we also report the nested value. letter models of nuclear safety analysis apply a common One objection to co-packaging is that it increases correlation, cause within a common-cause component group and treat and it does, but only within a module. By Theorem 2 the ceiling separate groups as independent [11], which is the limiting rises with separation across modules, and the field-replaceable case ρacross = 0 of the model here. faceplate modules of the architecture provide that separation.

14

104

200

total cost ($k)

cost to 99.999% ($k)

250

150 100

102

50 0

103

0.15

0.12

0.09

0.06

0.045

0.03

decorrelate to ρ Fig. 9. Optimal split. Under the stated cost model, decorrelating first and buying spares second reaches 99.999% coverage at a 5.3× saving over spares alone. Each bar stacks the two costs of reaching the target, the spare lasers bought and the engineering premium paid for physical separation, and the number above it is the total, in thousands of dollars [derived; stated cost model].

Set ρwithin = 0.30 and ρacross = 0.05, and the pool-average correlation of the 18 × 8 shape becomes ρ̄ = 0.062. Applied to the 32-channel pool at the 2× budget cap, with the spare count unchanged, the shortcut lifts coverage from the 99.623% of the flat model at ρ = 0.15 to 99.993% (Fig. 8), well past the 99.670% that even a large flat pool approaches. The nested model confirms the gain: intrepreted as 8 modules of 8, the 32-channel pool has its own ρ̄ = 0.078, at which the shortcut gives 99.975% and the nested tail 99.985%, so the shortcut errs low. The full 18 × 8 shape at its own 2× cap reaches 99.997%. B. Spares Versus Separation

0

0.05

0.10

0.15

0.20

0.25

0.30

0.35

decorrelate to ρ Fig. 10. ρ-invariance. The curves plot the total cost of reaching the coverage target against how far the pool is decorrelated, one curve for each starting correlation ρ0 . Every curve falls all the way to ρ = 0.03, the best separation the premium is priced for, so the cheapest plan decorrelates as far as the model allows, wherever the pool starts within the plausible band [derived].

model that Algorithm 1 uses, and the reported point is the cheapest plan that meets the 99.999% target. Both paths are priced on coverage alone (spares absorbing every failure), with no credit for the fast recovery of Section IV-IV-D. Reaching 99.999% coverage by spares alone at ρ = 0.15 takes 156 spare lasers, about $234,000. Decorrelating to ρ = 0.03 instead costs about $9,600 in separation premium, which is the 20% figure applied to the $48,000 of active-laser capital in the 32-channel pool. That leaves only 23 spare lasers to buy, at $34,500, for $44,100 in total. Under the stated cost model, separating first and sizing second therefore reaches the same target at a 5.3× saving over spares alone. At this premium the ordering is not close: it would reverse only if separation cost several times the pool’s entire laser capital. The split depends on the separationpremium assumption, and a reader with a different premium can re-solve it with the same coverage model and prices.

How far to take separation depends on prices, and once prices are attached the choice between decorrelating and overprovisioning becomes an optimization with a definite answer. We state the prices as assumptions: $1,500 per spare laser, C. Optimality with Varying ρ since we found no public price for pluggable laser sources, The plan priced above starts from an assumed correlation, and a one-time separation premium of 20% of the active-laser and ρ has not been measured, so the analysis is weakest here. capital to move a pool from poor separation to good separation. We do not defend a particular value of ρ. Neither is a quote. The premium stands in for second sourcing and for independent power and cooling, which IEC 61508-6 Observation 2 (ρ-invariance). Under the stated cost model, rewards with a lower beta factor [12] but does not price. Every for every starting correlation in the plausible band ρ0 ∈ dollar figure below scales with the per-laser price, so the ratio [0.08, 0.35] the minimum-cost path to the 99.999% coverage target decorrelates the pool first, toward ρ ≈ 0.03. The optimal between the two paths does not depend on it. action never reverses direction within the band. Observation 1 (Optimal split). Under the stated cost model, Fig. 10 sweeps the starting correlation across the band, with given a budget spendable on either spares (raising the spare ratio) or separation (lowering ρ at the stated premium), the a partial move priced in proportion to the correlation it removes: minimum-cost path to a 99.999% coverage target at the stress $800 for each 0.01, so the move from 0.15 to 0.03 costs the operating point decorrelates the pool first and buys spares $9,600 above. A measured value of ρ will therefore set the size of the saving, not the choice of action. second. Fig. 9 prices two paths to the same target. A candidate plan is a pair: a spare count s, and a decorrelation target ρ′ at or below the pool’s starting correlation ρ0 . Its cost is s times the per-laser price plus the separation premium for reaching ρ′ . Its availability comes from the same beta-binomial coverage

D. Performance of the Integrated System Each component has been evaluated alone against the baseline it replaces, and here their results are composed into the effective cluster goodput of Section I.

15

At the stress setting of Section III-III-B, composing the simulated component results lifts effective cluster goodput from 65.2% to 76.5%, a rise of 11.3 points, or 17.3% in relative terms (Fig. 11). The full-stack figure is composed by arithmetic on the component results and is not a separate emulation: the 65.2% baseline, a calibration anchor rather than a simulated value, is propagated through the simulated availability, lockloss, and reconfiguration changes under goodput = ideal × availability × (1 − thermal) × (1 − reconfig). Here availability is the laser-pool availability of Section IV-IV-A, thermal is the share of goodput lost to lock-loss stalls, and reconfig is the share lost to switch reconfigurations. The laser pool enters through its per-window coverage, which depends on the window only through λ. That availability factor carries an assumption that sets most of the result. The coverage reading counts a window whose failures exhaust the spares as lost from start to finish. If only the dark channels’ bandwidth is lost instead, the availability factor becomes the expected lit share of the 32 channels; we call this the lit-share reading. That share is 98.0% for the four-laser reserve, against its 86.3% coverage, and 99.98% for the full stack. With the 65.2% anchor held fixed, that reading lifts the full stack only to 67.4%, a rise of 2.2 points, or 3.4% relative. Each reading implies its own ideal goodput. A real job need not fall between the two. When in the window the spares run out and for how long also move the figure, and this work models neither. Within the window model, how a job handles a dark channel is one parameter: if, once the spares run out, it cannot use a share c of its remaining bandwidth, the availability factor is c times the coverage plus 1 − c times the lit share, and the gain rises from 2.2 points at c = 0, a fabric that routes around a dark channel, to 11.3 at c = 1, a synchronous job halted until the restock. Every figure below uses c = 1, the coverage reading. Each component reports its result in its own units, events or milliseconds, and the baseline’s two loss shares, thermal and reconfig, convert the results into goodput. The full stack keeps each share in proportion to its simulated count, 472/462 for lock-loss events against the dither lock and 8.8/16 for the proxy count of reconfigurations, and recovery time has no term in the composition. We have no measured costs, so we set both baseline shares at three percent, and Section VII-VII-E names them among the cheaper things to ground on real hardware. Sweeping the two shares by a factor of two in each direction moves the baseline across 61.2–67.2% and the full stack across 72.8–78.3%, while the full stack’s gain relative to the baseline stays within 16.5–18.9%. The relative gain is therefore robust to the two loss shares, though not to the reading of a shortfall, and the absolute pair should be read with its band. The released composite model computes coverage and lit share from the model of Section III and produces every number in this section. Fig. 12 isolates each component’s contribution to the key performance indicators (KPIs). LightReserve contributes the most, because availability dominates goodput once failures correlate. Composing the availability gain alone, from 86.3% to 99.9%, moves goodput from 65.2% to 75.5%. Credited first, availability therefore accounts for about 10.3 of the 11.3 points,

goodput availability 0

4

8

12

16

change (percentage points) reconfigurations lock-loss events recovery time power per output −50

−40

−30

−20

−10

0

change (%) Fig. 11. The full control plane against the deployed baseline: one bar per system-level KPI at the stress operating point, each showing the change from the baseline. Under the coverage reading, goodput rises 11.3 points and pool availability 13.6 points with a 49-spare pool past the 2× cap (11.1 and 13.3 points held to the cap). Reconfigurations, recovery time, and the power needed per unit of training output fall. Lock-loss events rise 2% [composed].

and the reconfiguration factor adds the remaining 1.0, since the thermal factor barely moves. Credited the other way round, the two factors add 0.9. The availability share is provisioning: it comes from the spare lasers that the coverage bound sizes, 49 of them for 99.9% at the stress point, past the 2× cap of 32. Held to the cap, the pool reaches 99.623% and composed goodput 76.3%, against 76.5% uncapped. 1) A Correlated-Factor Composition: The multiplicative model treats availability and thermal loss as independent, but a shared cause could move both. The coupled model assumes that one thermal event drives both the lock-loss stalls (ThermaLead’s domain) and the correlation ρ (LightReserve’s), so it moves both factors through the latent variable Θ of Theorem 1. Multiplying the two means gives an expected value and understates the tail, where both factors fail together. The coupled model draws Θ ∼ Beta(a, b) for each window. Given the draw, the failures among the full stack’s lasers are binomial at that level, so the pool either covers every one of them or runs out. A window that runs out counts as lost, as the coverage reading assumes. The thermal loss share is the full stack’s share scaled by Θ/λ. Coverage and thermal loss thus keep their stated means, and the model reports the distribution of the goodput each window realizes. The means agree: 76.5% under the coupled model and for the independent point estimate. The coupling changes the tail (Fig. 13). Goodput at the fifth percentile is 67.8%, and at the first percentile it is 59.7%. A cluster provisioned to the mean therefore misses its plan by 8.7 points or more in one window out of twenty, and by 16.8 points or more in one out of a hundred. This spread comes from the ρ of Theorem 1, and the decorrelation that raises the ceiling also tightens the spread. None of the per-component guarantees composes into a worst-case bound on effective cluster goodput, but part of the tail can be proved (Section V-V-E). The stack is worth more under worse conditions. At 50 FIT the four-laser reserve covers 99.974% of 90-day windows and 99.896% of one-year windows. The 90-day figure already exceeds the 99.9% target, so availability is held at that value.

+13.6

0.0

0

+10.3

−14

−5 −45 +2

+13.6

−10

−45

+0.9 +11.3

0

2

4

5

9

16

0.05

5

8

11

17

29

0.15

9

17

23

0.25

12

27

0.35

15

−1

80

cluster goodput (%)

+2

correlation ρ

16

70 60 50 40

−15

0.01 0.03 0.05

Fig. 12. Ablation. The first four rows each run one component alone and the last row runs the full stack; each entry is its change from the deployed baseline. Among the four layer KPIs, a component alone moves only the one its layer owns: lock-loss for ThermaLead, availability for LightReserve, recovery time for TriageGuard, and reconfigurations for FoldFit. TriageGuard alone runs on stale telemetry, since ThermaLead supplies the fresh signal, and its recovery-time cut has no term in the goodput formula, so its goodput and power entries are blank. The availability entry uses a 49-spare pool past the 2× cap, under the coverage reading. LightReserve and FoldFit each add goodput and save power per unit of output. ThermaLead moves goodput by less than 0.1 point and power by less than 0.1% [composed].

0.1

0.2

30

0.05

failure rate λ

0.10

0.15

failure rate λ

Fig. 14. The swept range. Left: the spares needed for 99% coverage of 32 active channels as the failure rate λ and the correlation ρ vary. Hatched cells need more than the 2× cap of 32 spares, and the outlined cell is the stress operating point. Right: goodput of the deployed baseline (dashed) and of the full stack, its pool sized to 99.9% coverage at each rate, as the failure rate grows at ρ = 0.15, with circles at the stress operating point. That sizing already passes the 2× cap at the stress point. Shading marks the gap between the two curves. Both figures use the coverage reading [composed]. 80

quantile Qp(G) (%)

probability density

1.5

1.0

0.5

0.0

75 70 65 60 55

55

60

65

70

75

80

composed cluster goodput (%) Fig. 13. Correlated-factor composition. Histogram of composed goodput, under the coverage reading, when the common driver Θ moves availability and thermal loss together. Vertical lines mark the mean (76.5%), the fifth percentile (67.8%), and the first percentile (59.7%). The mean is within 0.1 point of the independent model’s point estimate, which has no tail. Below the plotted range fall 0.1% of windows, most of them windows whose failures exhaust the spares [composed].

With the thermal and reconfiguration factors at their simulated values, the composed gain falls to about 1.0 point, or 1.3% relative, all of it from the reconfiguration factor. As the failure rate rises at ρ = 0.15, the baseline’s goodput drops sharply while the stack, its pool sized to 99.9% coverage, holds, and the gap between them widens to 41 points at a failure rate of 0.18 per window (Fig. 14). At that rate the target takes 116 spares, far past the 2× cap; held to the cap, the stack keeps 73.2% against the baseline’s 35.1%, a gap of 38 points. E. What Can Be Proved About the Goodput Tail Section V-V-D reported a fifth-percentile goodput of 67.8%, obtained by sampling the common driver Θ. A sampled percentile is a simulation output: run the simulation differently and the number moves. The systematic part of goodput, the part Θ moves, has quantiles in closed form, so that part of the

0

10

20

30

40

50

tail percentile p Fig. 15. Quantile transport. Closed-form quantile of the correlation-driven part of goodput, G(Θ) of the coupled model in Section V-V-D, against the tail percentile p, in percent. The dashed curve is the ρ = 0.15 operating point and the solid curve the same pool after decorrelation to ρ = 0.03. Decorrelation lifts the lower tail, including the fifth percentile; the curves cross near p = 22, above which the more correlated pool sits higher [derived].

tail can be reasoned about as well as observed. The mechanism is the standard monotone transport of quantiles. Proposition 1 (Quantile transport). Let G be the systematic part of the composed goodput, a strictly decreasing function of the single driver Θ ∼ Beta(a, b); the realized goodput is G plus the conditional binomial variation of failures given Θ. Then,
for any p, the p-quantile of G is Qp (G) = G FΘ−1 (1 − p) . For p below a threshold p∗ that depends on λ and ρ and that we locate numerically, ∂Qp /∂ρ < 0, so decorrelation lifts that part of the tail. Proof sketch. The identity is monotone transport of quantiles: a strictly decreasing map sends the (1 − p)-quantile of its input to the p-quantile of its output. The sign follows because, for p < p∗ , raising ρ moves the (1 − p)-quantile of Θ up, which lowers G there; p∗ is where that quantile stops rising with ρ, which we locate numerically. We checked the identity

17

numerically, comparing empirical quantiles of four million Monte Carlo draws of G(Θ) against the closed form, and the two agree to within 2 × 10−4 at the reported percentiles. Located numerically at λ = 0.05, p∗ falls from the 31st percentile at ρ = 0.03 to the 15th at ρ = 0.15 and the 8th at ρ = 0.35, so it covers the fifth percentile across the plausible band. Fig. 15 plots this closed form. Realized goodput varies further: the conditional binomial spread of failures given Θ widens its distribution in both directions. Here G is the goodput expected at a given Θ, a mean and not a minimum. In the coupled model of Section V-V-D, a window’s realized goodput sits slightly above G when its spares cover every failure. It falls to zero in the 0.1% of windows where they run out, so Qp (G) is no floor under realized goodput. At the percentiles reported there the two agree to within 0.02 points: at p = 5 the closed form gives the 67.8% sampled there. Turning Qp (G) into a floor under realized goodput would require bounding the conditional binomial spread as well, which we do not attempt here.

A reproducibility bundle accompanies the paper. It holds the register-transfer sources, testbenches, proofs, build scripts, and 130 nm layout flow behind Section VI; the component simulator behind the results of Section IV, with the ThermaLead baseline and gate studies and the abrupt-transient sweep of Section VII-VII-D; the composite model, which reads the simulator’s output, behind the goodput, recovery-time, and capital figures; the cross-workload simulations of Section VII-VII-C; and the scripts that evaluate the closed-form results and draw the figures. VI. H ARDWARE I MPLEMENTATION OF THE C ONTROL P LANE

Halley’s fallback to reactive control, which overrules a forecast that is stale or unsure, needs a circuit to enforce it. We implemented the two safety-critical parts of Halley as one design for a Lattice iCE40UP5K [72], a small fieldprogrammable gate array (FPGA). The first is the fail-safe watchdog, which enforces the fallback rule of Section IV: cycle by cycle, it decides whether the predictive or the reactive command reaches the actuator. The second is ThermaLead’s local thermal servo, which drives the per-ring heaters and is the only one of the four controllers that closes a loop at F. Simulation Methodology and Reproducibility hardware speed. LightReserve, FoldFit, and TriageGuard decide The simulated and composed results in this and the preceding on slower time scales: LightReserve sizes a pool when it is sections come from a calibrated simulation stack written in provisioned or restocked, FoldFit computes a schedule for each Python with NumPy and SciPy and run with fixed seeds; the demand matrix, and TriageGuard ranks a pool’s modules from derived results are closed-form evaluations. None of them is their telemetry. A supervisory processor, such as the supervisory a hardware measurement. Each figure caption carries a provehost of Section VI-VI-C, can run all three in software, so none nance tag: [derived] for closed-form evaluation, [composed] for of them needs dedicated logic. All four need telemetry and results composed arithmetically from the component outputs, actuation that published laser and switch interfaces provide such as goodput and recovery time, [schematic] for architecture only in part (Section VII-VII-B), which is why the fallback is drawings, and [synth; pnr; verified] for the synthesis, place-andthe part of Halley that can be built and measured today. route, and verification outputs reproduced in Section VI. Some The design was synthesized, placed and routed, and formally captions combine or qualify these base tags, as in [derived; verified with an open toolchain pinned to fixed versions: Icarus stated cost model], [synth; pnr], and [structure matches the Verilog 12.0 [73], Yosys 0.33 [74], nextpnr 0.6 [75], and the synthesized sources]. icetime timing analyzer of Project IceStorm. The method is The configuration of each component follows, for repro- test-driven [76], with an executable specification written before duction; Section IV reports the results. LightReserve runs the circuit. Every resource figure below regenerates exactly Algorithm 1. The 32-channel calibration uses independent from the released register-transfer sources. The timing figures failures at λ = 0.05, 0.10, and 0.20 with a 99% coverage are the recorded results for each placement seed; a rebuild target, and a correlated case at λ = 0.05 and ρ = 0.15 with can land at different frequencies on a given seed, and in our targets of 99%, 99.9%, and 99.999%. FoldFit is simulated on independent rebuild of the core, the servo, the eight-instance block-structured MoE demand, dense within each expert block layout, and the ECP5 port, every seed we rebuilt still closed of 8 engines and sparse across blocks, on 8 channels at 8 to timing. 128 engines. The simulation binary-searches the deadline and, at each candidate, places the jobs longest first on a channel that already holds the job’s expert block when one has room, and A. Circuit Design otherwise on the least-loaded channel with room. It omits the The safety core is one circuit with two halves that share port condition, and with it the alignment step, so makespans a clock and a reset. The first half is the watchdog of are measured against the larger of the largest demand and the Fig. 16, module watchdog_bypass. It lets the predicaverage load, which they meet from 64 engines up. ThermaLead tive path drive the actuator only while the predictor keeps runs on the heat-cycle trace of Section IV-IV-C: over 30 seeds answering and reports confidence. A windowed counter the simulation counts lock-loss events per 2000 control steps restarts on each predictive_kick from the predictor for ThermaLead, its gated variant, the dither lock, and an and otherwise counts up until it saturates at the 1,000-cycle oracle; it also sweeps the share of steps that carry a noise timeout, which a magnitude comparator detects. A registered burst from 30% down to none. TriageGuard is configured as using_predictive flag is set when the counter is in in Section IV-IV-D. window and predictive_confidence_ok is high, and a

18

D-FF windowed counter

comparator

reset on kick, else +1; saturates at 1,000

counter ≥ 1,000 counter < 1,000

&

D-FF 1

MUX 0

Fig. 16. The fail-safe watchdog and bypass multiplexer (MUX); & marks an AND gate and D-FF a register. The counter stops at the timeout, a test that synthesis shares with the comparator [structure matches the synthesized sources].

1 error

2 accumulate

3 clamp

4 control

5 duty

error_r signed 16-bit

isum_r signed 16-bit

integral [0, 8191]

control_r signed 16-bit

duty_r [0, 255]

enable launch

enable valid[0]

enable valid[1]

enable valid[2]

enable valid[3]

PWM counter

comparator

8-bit, +1 per cycle

pwm_counter < duty_r

stage enables valid[4:0]

Fig. 17. The ThermaLead local servo. The five numbered blocks are the pipeline stages of a multiplier-free proportional-integral law, from temperature error to heater duty cycle. A label ≫ n means an arithmetic shift right by n bits, which applies a gain without a multiplier. The strip below is the 256-cycle pulse-width-modulation generator, which drives the per-ring heater and, once per period, launches the stage enables that step one update through the pipeline [structure matches the synthesized sources].

second register raises watchdog_fault one cycle after the window closes. The flag is the select of an output multiplexer that holds no state of its own, and reset and power-up clear it, so the reactive path is chosen unless the flag is set. A crash, hang, low-confidence signal, or reset therefore parks the actuator on the reactive path, which is always live, within at most one trust window, 20 µs at the 50 MHz target clock, plus one clock after the last kick, 20.02 µs in all. After a fresh kick, the predictive path is restored within two clock cycles. The second half is ThermaLead’s local loop, module thermal_pi (Fig. 17), which turns a measured temperature error into heater drive. Five pipeline stages compute error, accumulate, clamp, control, and duty. Together they implement a proportional-integral law whose two gains are arithmetic shifts, one bit for the proportional gain and five for the integral gain. The law therefore needs no multiplier, and it maps onto look-up tables (LUTs) and carry cells instead of a dedicated digital-signal-processing multiplier. The accumulator, clamped to [0, 8191], is the only state carried from one update to the next. The control sum is clamped to [0, 255] and registered

as the duty. A separate free-running 8-bit generator holds pwm_out high while its counter sits below that duty, and that signal drives the per-ring heater. The law updates once per 256-cycle pulse-width period, 5.12 µs at 50 MHz, so the fivecycle pipeline latency is negligible against that period, and the duty stream is bit-identical to a single-cycle implementation of the same law. At that rate the heater drive does not limit ThermaLead’s control step; the heater’s thermal response does (see the closed-loop sweep below). B. Chip Design Table IV gives the toolchain results. The watchdog synthesizes to 36 LUTs and the servo to 97, and the combined core is exactly their sum, 133 LUTs, so either half can be budgeted on its own. Counted at synthesis for the core alone, the 133 LUTs are 2.52% of the device’s 5,280. Under nextpnr, timing closes on every placement seed. Placed and routed behind the on-chip serial test harness, the combined core closes at 68.97–74.69 MHz across ten seeds, a worstcase margin of 38% over the target. The independent icetime

19

Module watchdog_bypass thermal_pi Combined safety core Core + SPI sensor path

LUTs

FFs

36 97 133 161

12 78 90 149

Carries

fmax (MHz)

14 92.54 53 81.98–87.01 67 68.97–74.69 68 > 50

TABLE IV S YNTHESIZED FOOTPRINT AND CLOSED TIMING ON THE I CE40UP5K. L OGIC COUNTS COME FROM YOSYS S Y N T H _ I C E 40, AND PLACEMENT AND ROUTING FROM NEXTPNR ACROSS 10 SEEDS ( THE WATCHDOG ALONE ACROSS 3) AGAINST A 50 MH Z TARGET. FF S ARE FLIP - FLOPS , AND fmax IS THE HIGHEST CLOCK FREQUENCY AT WHICH TIMING CLOSES .

analyzer puts the combined core at 71.33–74.79 MHz on the same placements, so both models clear the target on every seed. The core is paired with a production-grade sensor path on a mode-zero serial-peripheral-interface (SPI) link. With that path the design comes to 161 LUTs and closes on 10 of 10 seeds under both models. The same sources also port to a second FPGA, a Lattice ECP5-25, where they close at 196.97– 211.91 MHz across three seeds, so the result is not tied to one device. Unchanged, the core and its serial test harness also lay out with LibreLane 3.0.14 as a 142 by 153 µm SkyWater 130 nm block, 0.022 mm2 , with 128 flip-flops, the core’s 90 and the harness’s 38, among 995 standard cells besides fill (Fig. 18b). The block meets 50 MHz at all nine signoff corners, with 11.0 ns of worst-case setup slack, and passes design-rule, layout-versus-schematic, and antenna checks; it has not been fabricated. Beyond the tests, four machine proofs by temporal induction in Yosys check the core, each for every input. Temporal induction proves a property outright for a bounded run of cycles, then proves that any two consecutive cycles satisfying it force the next cycle to satisfy it too, so the property holds for every cycle. The first proof checks the watchdog against a one-bitwider golden model, an independently written reference circuit. The second proves the servo’s invariants: the accumulator stays in [0, 8191], the integral and the duty change only on their stage enables, and pwm_out equals the comparison of counter and duty. The third checks the servo pipeline against the one-shot control law. The fourth checks the whole core, watchdog and servo together, against an independent specification core whose clamps are written as comparisons. The first two start checking after eight cycles of warm-up, and the last two check every cycle from power-on. All four report success on the induction step in the released logs, so the bypass path that the control plane relies on is a proven property of the register-transfer design. The proofs show that the circuit computes the law it was meant to compute. They do not show that the law is well behaved once the loop is closed, so the servo was also run in closed loop against a second-order thermal plant with quantization, noise, and one period of sensor delay. Across a gain-and-pole sweep, it converges at all 31 feasible corners, settling in 0.2 to 3.7 ms, and re-settles after a mid-run thermal step in all 27 corners where one was injected. The one infeasible corner asks for 500 counts at 0.6 times the nominal gain, more than the 480 the plant reaches at full duty, and there the servo

correctly holds the duty at 255 and the integral at its cap. Its phase and gain margins stay at or above 22.5° and 14.9 dB across the sweep, and an independent recomputation confirms them. The swept plant time constants include the 30.9 µs and 0.71 ms of a measured microring heater [77] and the 13.8 µs of a metal microheater [78]; doped-silicon microheaters, at 0.98 µs [78], are faster than the sweep covers. C. Packaging The fallback can be built from catalog silicon with an ordinary pinout: the iCE40UP5K in the 48-pad SG48 package. The interface brought out to the package is deliberately narrow (Fig. 18). A deployed part would carry setpoints and telemetry to the supervisory host one level above this device over a management bus such as I2C, in place of the two-pin serial test port, and the wide data buses stay on chip behind that serial test harness. Placed together with the harness, the design occupies 202 of the 5,280 logic cells, or 3.83%, against the 2.52% that synthesis counts for the core alone. To test whether one device can serve several thermal domains, we placed and routed eight instances of the core without the shared telemetry bus that would report them to the supervisory host. They occupy 1,417 of the 5,280 logic cells (26.8%) and close timing on all six seeds under both models, at 55.25–58.59 MHz under nextpnr and 55.91–58.92 MHz under icetime; the bus itself has not been placed. Both safety-critical circuits are placed and routed for a shipping part, close timing under the place-and-route tool on every seed, and use very little area, and the bypass is a proven property of their register-transfer design, so the fallback path can be built. Whether Halley then delivers the reliability outcome still depends on the unmeasured correlation ρ of Section III, which only the field measurement of Section VII-VII-E would settle. VII. D ISCUSSION A. Impact on Capital and Power Costs The goodput gain of Section V-V-D is an upper bound on the useful compute, and so the capital, that Halley frees. The coverage reading multiplies the useful compute the same hardware delivers by up to 1.17, a 17.3% rise, and the litshare reading by up to 1.03. Both factors hold only if fabric stall is the sole recoverable loss, and it is not. Model FLOPs utilization (MFU), the share of peak arithmetic that a training run keeps busy, also counts compute, memory, pipeline-bubble, and data-loading losses that the control plane never touches. The 65.2% goodput here and the 38–43% MFU that Llama 3 reported therefore do not conflict. At that bound, the gain cuts the accelerators, capital, and provisioned power needed for a given training output by up to 14.7%, or 3.3% on the lit-share reading, for a fixed rack design and power-smoothing scheme. Energy falls by less, because accelerators waiting at a barrier draw less than full power [23]. Where power and local approval limit new AI data centers (Section I), the gain becomes more training from existing sites instead of a saving. It is a one-time factor on top of the 30 to 40% yearly improvement in accelerator performance per

20

logic array 5,280 cells

placed design 202 cells, 3.83%

Fig. 18. The device, the placement, and the pins, in the iCE40 and 130 nm builds. (a) Inside the 48-pad SG48 package of the Lattice iCE40UP5K, the inner square is the logic array, and the filled square, drawn to scale by area, is the placed design: 202 logic cells including the serial test harness. Pad positions and the design’s place in the array are schematic [synth; pnr]. (b) The signed-off SkyWater 130 nm layout, drawn from its GDS: cells light blue, the 128 flip-flops darker, pins as laid out [synth; pnr; verified].

the alternative, and each point of availability recovers training time in proportion to the attribution fraction. The second is the tail-latency objective of MoE inference. Generating a token runs the same all-to-all exchange as training; that exchange takes 30–40% of iteration time in DeepSeek-V3 training on H100 before overlap with computation and under 5% after [22]. If decoding falls in the same range, FoldFit’s 11/9 bound caps the worst-case inflation of an iteration, under its stated conditions, at 6.7–8.9% before switching delay and with circuits held across steps, or at about 1% once the exchange is overlapped. The third is capacity planning before ρ is measured. Observation 2 shows that decorrelating first is the cheaper path across the plausible band, and the coupled model plans against the goodput tail that the independent model hides. The fourth is the control interface. The algorithms are cheap and step aside on a flagged forecast, so apart from ThermaLead’s forecast (Section VII-VII-D) the obstacle to deployment is an interface that exists only in part. External laser modules report per-lane bias, power, and alarms and accept per-lane enable [79], and commercial optical circuit switches expose NETCONF and RESTCONF control [80]. Neither the OIF management framework for co-packaged engines nor the ELSFP interface, however, defines per-ring temperature or lock state [79], [81]. One change would unblock both deployment and the field measurement of ρ (Section VII-VII-E): a published interface exposing per-laser health and failure timestamps, perring temperature and lock state, and switch reconfiguration hooks on the management interface the packages already carry. C. Impact on Inference, Post-Training, and RL Traffic

Halley was developed for training, where the failure data watt [14], not a substitute for it. At 50 FIT the same arithmetic of Section II show the need. The same fabric now carries starts from the 1.3% composed gain instead, and these capital inference and RL as well, but of the four components only FoldFit carries a guarantee that transfers cleanly. and power figures fall with it. FoldFit transfers: MoE decoding runs the same all-to-all A control plane can address only the fabric’s share of recoverable stall, the attribution fraction. Scaled by it, the expert dispatch on its critical path when experts are spread coverage reading’s 17.3% rise becomes 13.0%, 8.6%, and 5.8% across the fabric. Serving that disaggregates attention from the at three-quarters, half, and a third, and the lit-share reading’s experts sends M-to-N traffic instead [82]. On inference-shaped rise becomes 2.5%, 1.7%, and 1.1%. Llama 3 gives the fraction demand, across expert-parallel degrees from 8 to 128 and a scale. A tenth of its 419 unexpected interruptions were traced packed without the port condition, the schedule stays within to network switches, cables, and interface cards [23], which 1% of the classical lower bound and never exceeds 11/9, so would put the gain near 1.7% on the coverage reading and 0.3% the same cap on worst-case iteration inflation, before switching on the lit-share reading. That breakdown counts interruptions, delay, applies across that sweep. RL post-training inherits the not stalled time, so it gives a scale rather than a value, and this result through rollout, which is inference and takes from half work does not measure the fraction for a co-packaged fabric. to over 90% of the step in reported systems [83]–[85]; by Amdahl’s law, a hypothetical 10% rollout speed-up would be roughly a 5 to 9% step speed-up. Weight synchronization B. Addressing Practical System Losses between phases is a separate load: under 1.6 s for a 72B model The results bear on four problems that operators pay for inside one cluster, against rollouts of hundreds to thousands today and that deployed systems leave open. of seconds [86], but enough to stall rollouts at terabyte scale, The first is time lost at the synchronization barrier in especially across data centers [87]. hyperscale pretraining. Llama 3 held effective training time Availability and thermal are training-first: Reported above 90% through its 419 interruptions [23]. Laser-pool machine failures run about 13.6× lower per thousand machines exhaustion would add to that loss on a co-packaged fabric, per week for an inference service than for pretraining at scale, and within a budget cap Theorem 1 shows that more spares 2.2 against 30 [88]. If the per-laser rate fell by the same alone cannot remove it. Decorrelation with fast recovery is factor, the same coverage bound would put a fixed four-laser

21

reserve at 99.1% availability, against 86.3% at the stress training rate. The machinery pays off most where failures are frequent and correlated, so we claim no per-component inference win for LightReserve, ThermaLead, or TriageGuard. The case is strongest on the largest jobs, since mean time to failure falls as the GPU count rises [27]; at platform scale, ByteDance logged 38,236 explicit and 5,948 implicit failures across 778,135 training jobs in three months [89]. Disaggregated serving also moves the key-value cache from prefill to decode over the fabric [90], [91], a load that grows with context. No component removes it, and we claim no guarantee for it.

within 4.3%, an empirical limit, not a proof. A deployable ThermaLead needs a forecast matched to the drift and aware of regime changes. Comparisons with the fastest reactive loops now reported [56] and with heater pre-driving from kernel launches [57] remain open, since both need the control step in seconds, which the per-step analysis leaves free. Fourth, several external anchors are vendor-reported or forward-looking, and each is hedged where it appears. Fifth, Proposition 1 covers only the correlation-driven part of goodput, and Section V-V-E states what a genuine floor under realized goodput would also require. Sixth, the composed goodput turns on what a spare shortfall costs a real job, and the two readings of Section V-V-D are cases, not bounds.

D. Limitations and Open Problems E. The Measurement That Turns the Bound into a Forecast The fallback rule of Section IV covers the cases it can Theorem 1 bounds availability in terms of ρ, so a fleetdetect: on a stale, low-confidence, or late signal, the component scale measurement of that correlation would pin down the bypasses itself. It does not catch a confident but wrong forecast, value on which the bound depends, and the measurement on which a controller can reconfigure the fabric more often than needs only instrumentation. Log the failure timestamp of every it should. Gating reconfiguration on confidence and feasibility laser across many identical pools, count how many lasers fail curbs that churn without eliminating it, and a formal bound together in each restock window, and estimate ρ from how on the harm is left to future work. Acting also costs compute far that count is over-dispersed, that is, how much more it for the predictions and energy for the heaters, and spoofed varies than the independent-binomial expectation. Randomtelemetry could force a mis-reconfiguration, so the control effects lifetime models, which already estimate location-driven interface has to be hardened. correlation from GPU fleet failure data [92], offer one way Every numerical result outside Section VI is a calibrated to do it. Correlated failures are rare, so a credible estimate simulation or a closed-form evaluation, and those of Section VI needs exposure on the order of thousands of pool-years, from are synthesis, place-and-route, signoff, and proof outputs, which a hyperscaler instrumenting its own packages or a vendor settle the logic and its static timing but leave the process and interfaces to fabrication; none is a measurement on hardware. publishing co-failure statistics. Two choices sharpen the result. The two loss shares of the goodput model are assumptions, Logging the candidate shared causes lets ρ be attributed to a and most of the stress-point gain comes from provisioning the mechanism. The full per-window distribution should also be spare lasers that the coverage bound sizes (Section V-V-D). reported, because Part (2) of Theorem 1 and Proposition 1 show that coverage and the goodput tail depend on the tail of Θ, not Six limits bound the claims. First, the failure inputs are assumed: the correlation ρ, on only on its variance. Until such data exists, ρ = 0.15 should be which both the sparing result and the ceiling depend, and the read as a prior, and the design should rest on the full ρ-sweep. per-laser failure rate, for which we found no published field Cheaper measurements would narrow other unmeasured inputs: value for the lasers inside external laser modules. The headline one instrumented package would ground the two composition figures use the stress setting λ = 0.05 per window, and 99.999% loss shares, and one real job would establish both the fabric’s stays out of reach inside the 2× cap of the 32-channel pool share of recoverable stall and what a spare shortfall costs it. only while λ exceeds about 2.4 × 10−4 (Table II). We anchor VIII. C ONCLUSIONS ρ in the common-cause beta factors of Section III-III-B but cannot yet calibrate it to field data for these laser form factors, This paper argues that each of the four optical layers can so the recommendation rests on the sweep of Observation 2. be run a step ahead of failure. Halley puts one predictive Second, of the three conditions that carry FoldFit’s 11/9 controller over each of them, though ThermaLead’s simple guarantee onto a switch (Section IV-IV-B), two hold by forecast paid off mainly on smoother drift; its gate kept it near construction here. The third, realizing the packing as port- the dither lock elsewhere. Each controller steps aside when its disjoint configurations, is exact only when a demand may be forecast is flagged stale or unsure, so for such a forecast the split at a configuration boundary and resumed later, and a proof worst case is the reactive system it replaces; a confident but for demands that must each run on one uninterrupted circuit wrong forecast has no such floor yet (Section VII-VII-D). is open. The guarantee also covers dwell time only, not the One theorem shapes the design: correlated laser failures switching delay of each reconfiguration. put a ceiling on the coverage a large pool can reach within Third, ThermaLead’s moving-average forecast does slightly a deployable budget cap. For a 32-channel pool at ρ = 0.15, worse than the dither lock on the stress trace of Section IV-IV-C, 99.999% stays out of reach within the 2× cap whenever a and in the released sweep its deficit grows to 49–67% when laser’s failure probability per restock window exceeds about the heat cycle’s phase jumps randomly every 16 steps or more 2.4 × 10−4 , as it does for a laser at the 50 FIT source target often, and to 17% under random temperature steps of 1 °C restocked once a year. Between that target and the stress level, standard deviation every 4 steps. Gated, both deficits stay correlation raises the spares that 99.999% takes more than

22

tenfold. That target is then reached by decorrelating the pool first and sizing the spares against the correlation that remains. Under the stated cost model, that path costs $44,100 at the stress operating point against $234,000 for the pure-spares alternative, a 5.3× saving, and it stays the cheaper of the two across the whole plausible band of ρ, which the field has not yet measured. Composed from the simulated components, effective cluster goodput at the stress operating point rose 11.3 points on the coverage reading and 2.2 points on the lit-share reading. Most of the first figure comes from the larger spare pool, and holding it to the 2× cap leaves 11.1 points. The margin over today’s reactive stack is narrowest under mild conditions, about 1.3% in relative terms at 50 FIT. How much of that gain a real job would keep depends on the fabric’s share of recoverable stall, which this work did not measure. The part of the gain that a job keeps reduces the accelerators and megawatts its training needs, by up to 3.3–14.7% at the stress operating point. That much data center capacity need not be built. The fallback that early action relies on is synthesized, placed and routed for a shipping FPGA and passes signoff as a 130 nm layout, and its combined core is machine-proven to match an independent specification core on every cycle. An operator should size the pool from a measured failure rate and a measured correlation, spend the next dollar on separating the lasers before buying more of them, and plan capacity against a bad window such as the fifth percentile. Further research can help realize a deployed system. What remains open, ThermaLead’s forecast and the theory, is stated in Section VII-VII-D. Running these controllers on real optics needs the parts of the telemetry and actuation interface that the optics vendors have not yet published, chiefly perring temperature and lock state. Estimating ρ across a fleet needs only enough pools instrumented for long enough. This measurement would turn the ceiling proved here from a bound conditioned on an assumed correlation into a forecast. R EFERENCES [1] D. Patel and D. Nishball, “100,000 H100 clusters: Power, network topology, Ethernet vs InfiniBand, reliability, failures, checkpointing,” SemiAnalysis, Jun. 2024. [Online]. Available: https://newsletter.semiana lysis.com/p/100000-h100-clusters-power-network [2] L. Poutievski et al., “Jupiter evolving: Transforming Google’s datacenter network via optical circuit switches and software-defined networking,” in Proc. ACM SIGCOMM, 2022, pp. 66–85. [3] Y. Zu et al., “Resiliency at scale: Managing Google’s TPUv4 machine learning supercomputer,” in Proc. USENIX NSDI, 2024, pp. 761–774. [4] N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, C. Young, X. Zhou, Z. Zhou, and D. A. Patterson, “TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” in Proc. ACM/IEEE ISCA, Jun. 2023, pp. 1–14. [5] X. Liao et al., “MixNet: A runtime reconfigurable optical-electrical fabric for distributed mixture-of-experts training,” in Proc. ACM SIGCOMM, 2025, pp. 554–574. [6] E. Ding, B. Lyu, B. Kataria, and R. Singh, “Opus: Photonic rail-optimized fabric in ML datacenters,” in Proc. ACM SIGCOMM, Aug. 2026, pp. 922–939. [7] J. Su, D. Fu, Q. Qiu, X. Ding, J. Wang, C. Chen, B. Zhang, P. Wang, X. Chen, Z. He, H. Yu, Z. Feng, W. Hu, and Q. Zhuge, “Analysis and future-guided prediction for optical transceiver failures in AI data center networks,” in Proc. OFC, 2026, Paper W4H.2.

[8] Y. Wang, S. Li, X. Ding, J. Wang, W. Chen, C. Chen, Y. Jiang, Q. Liu, Y. Shao, B. Zhang, Y. Liu, M. Zhang, P. Wang, and D. Wang, “Systematic fault management of million-scale field-deployed optical transceivers in AI data centers,” in Proc. OFC, 2026, Paper Th3B.2. [9] A. Seyedi, “How industry collaboration fosters NVIDIA co-packaged optics,” NVIDIA Technical Blog, Aug. 2025. [Online]. Available: https://developer.nvidia.com/blog/how-industry-collaboration-fosters-n vidia-co-packaged-optics [10] D. J. Aldous, “Exchangeability and related topics,” in École d’Été de Probabilités de Saint-Flour XIII – 1983, ser. Lecture Notes in Mathematics. Berlin, Heidelberg: Springer, 1985, vol. 1117, pp. 1– 198. [11] A. Mosleh, D. M. Rasmuson, and F. M. Marshall, “Guidelines on modeling common-cause failures in probabilistic risk assessment,” U.S. Nuclear Regulatory Commission, Tech. Rep. NUREG/CR-5485 (INEEL/EXT-9701327), Nov. 1998. [12] International Electrotechnical Commission, “IEC 61508-6:2010, Functional safety of electrical/electronic/programmable electronic safetyrelated systems, Part 6: Guidelines on the application of IEC 61508-2 and IEC 61508-3,” 2010, Annex D. [13] M. Mishra and K. M. Sivalingam, “Halley: Implementation of a control plane for optical AI data center networks,” Oct. 2026, encryption password: halley123. [Online]. Available: https: //www.cse.iitm.ac.in/∼skrishnam/.zip [14] International Energy Agency, “Key questions on energy and AI,” IEA, Paris, France, World Energy Outlook Special Report, Apr. 2026. [Online]. Available: https://www.iea.org/reports/key-questions-on-energy-and-ai [15] N. Purnell, “New data centers worth $68 billion disrupted in US, data show,” Bloomberg News, republished by Insurance Journal, Sep. 21, 2026, reports Data Center Watch figures for Q2 2026. [Online]. Available: https: //www.insurancejournal.com/news/national/2026/09/21/886238.htm [16] K. Qian et al., “Alibaba HPN: A data center network for large language model training,” in Proc. ACM SIGCOMM, 2024, pp. 691–706. [17] E. L. Johnson, K. M. Sivalingam, and M. Mishra, “Scheduling in optical WDM networks using hidden Markov chain based traffic prediction,” Photon. Netw. Commun., vol. 3, no. 3, pp. 269–283, Jul. 2001. [18] K. M. Sivalingam, J. Wang, X. Wu, and M. Mishra, “An interval-based scheduling algorithm for optical WDM star networks,” Photon. Netw. Commun., vol. 4, no. 1, pp. 73–87, Jan. 2002. [19] N. M. Bhide, M. Mishra, and K. M. Sivalingam, “Scheduling algorithms for star-coupled WDM networks with tunable transmitter and tunable receiver architecture,” Photon. Netw. Commun., vol. 1, no. 3, pp. 219–234, Nov. 1999. [20] R. Shenai, C. Maciocco, M. Mishra, and K. M. Sivalingam, “Threshold based selective link restoration for optical WDM mesh networks,” in Proc. DRCN, Oct. 2003, pp. 31–38. [21] H. Krishnamurthy, K. M. Sivalingam, and M. Mishra, “Restoration mechanisms for handling channel and link failures in optical WDM networks: Tunable laser-based switch architectures and performance analysis,” Comput. Commun., vol. 28, no. 9, pp. 987–999, Jun. 2005. [22] Z. Yan et al., “Scalable training of mixture-of-experts models with Megatron Core,” arXiv:2603.07685, Mar. 2026. [23] A. Grattafiori et al., “The Llama 3 herd of models,” arXiv:2407.21783, Jul. 2024. [24] Z. Jiang et al., “MegaScale: Scaling large language model training to more than 10,000 GPUs,” in Proc. USENIX NSDI, 2024, pp. 745–760. [25] DeepSeek-AI et al., “DeepSeek-V3 technical report,” arXiv:2412.19437, Dec. 2024. [26] O. Salpekar, R. Varma, K. Yu, V. Ivanov, Y. Wang, A. Sharif, M. Si, S. Xu, F. Tian, S. Zheng, T. Rice, A. Garg, S. Peng, S. Siravara, W. Fu, R. de Castro, A. Gangidi, A. Obraztsov, S. Narang, S. Edunov, M. Naumov, C. Tang, and M. Oldham, “Training LLMs with fault tolerant HSDP on 100,000 GPUs,” arXiv:2602.00277, Jan. 2026. [27] A. Kokolis et al., “Revisiting reliability in large-scale machine learning research clusters,” in Proc. IEEE HPCA, 2025, pp. 1259–1274. [28] S. McDowell, “NVIDIA is rewiring the data center with light,” NAND Research, Jun. 2026. [Online]. Available: https://nand-research.com/nvid ia-is-rewiring-the-data-center-with-light/ [29] Optical Internetworking Forum, “External laser small form factor pluggable (ELSFP) implementation agreement,” OIF, Implementation Agreement OIF-ELSFP-02.0, Jan. 2025. [Online]. Available: https: //www.oiforum.com/wp-content/uploads/OIF-ELSFP-02.0.pdf [30] Lightmatter, “Lightmatter introduces Guide light engine for AI, featuring VLSP technology,” Press release, Jan. 2026. [Online]. Available: https://lightmatter.co/press-release/lightmatter-introduces-guide-light-e ngine-for-ai-featuring-vlsp-technology/

23

[31] Lumentum, “Lumentum to demonstrate DWDM ELSFP laser module for OCI MSA applications at ECOC 2026,” Press release, Sep. 2026. [Online]. Available: https://investor.lumentum.com/financial-news-relea ses/news-details/2026/Lumentum-to-Demonstrate-DWDM-ELSFP-Las er-Module-for-OCI-MSA-Applications-at-ECOC-2026/default.aspx [32] Mesh Optical Technologies, “Alpha C1,” Product page, 2026, Accessed: Oct. 2, 2026. [Online]. Available: https://www.meshoptical.com/alpha-c1 [33] NVIDIA, “NVIDIA GTC Taipei at COMPUTEX: Live updates on what’s next in AI,” NVIDIA Blog, Jun. 1, 2026, entry “NVIDIA SpectrumX Ethernet Photonics ramps to production”. [Online]. Available: https://blogs.nvidia.com/blog/nvidia-gtc-taipei-computex-2026-news/ [34] H. Liu, R. Urata, K. Yasumura, X. Zhou, R. Bannon, J. Berger, P. Dashti, N. Jouppi, C. Lam, S. Li, E. Mao, D. Nelson, G. Papen, M. Tariq, and A. Vahdat, “Lightwave fabrics: At-scale optical circuit switching for datacenter and machine learning systems,” in Proc. ACM SIGCOMM, Sep. 2023, pp. 499–515. [35] H. Ballani et al., “Sirius: A flat datacenter network with nanosecond optical switching,” in Proc. ACM SIGCOMM, 2020, pp. 782–797. [36] N. Farrington et al., “Helios: A hybrid electrical/optical switch architecture for modular data centers,” in Proc. ACM SIGCOMM, 2010, pp. 339–350. [37] W. Wang et al., “TopoOpt: Co-optimizing network topology and parallelization strategy for distributed training jobs,” in Proc. USENIX NSDI, 2023, pp. 739–767. [38] V. Valls, G. Iosifidis, and L. Tassiulas, “Birkhoff’s decomposition revisited: Sparse scheduling for high-speed circuit switches,” IEEE/ACM Trans. Netw., vol. 29, no. 6, pp. 2399–2412, 2021. [39] H. Liu et al., “Scheduling techniques for hybrid circuit/packet networks,” in Proc. ACM CoNEXT, 2015, pp. 41:1–41:13. [40] S. Bojja Venkatakrishnan, M. Alizadeh, and P. Viswanath, “Costly circuits, submodular schedules and approximate Carathéodory theorems,” in Proc. ACM SIGMETRICS, 2016, pp. 75–88. [41] X. S. Huang, X. S. Sun, and T. S. E. Ng, “Sunflow: Efficient optical circuit scheduling for coflows,” in Proc. ACM CoNEXT, 2016, pp. 297–311. [42] S. Renganathan and N. McKeown, “Chronos: Prescheduled circuit switching for LLM training,” in Proc. NAIC Workshop, Sep. 2025, pp. 89–97. [43] K. Liang, L. Qiao, I. Keslassy, and B. Lin, “Scheduling parallel optical circuit switches for AI training,” arXiv:2603.07373, Mar. 2026. [44] B. Liang, Y. Chen, S. Zhao, P. Huang, D. Yang, B. Xu, B. Yang, S. Zhao, and G. Chen, “Breaking the duplex barrier: Lane-granularity OCS scheduling for LLM training,” arXiv:2609.14253, Sep. 2026. [45] A. Troch, S. Lahmer, A. Nada, J. Famaey, and M. Peeters, “Flux: Optimal scheduling of optical circuit switches for LLM training,” arXiv:2609.25949, Sep. 2026. [46] N. Ye, J. Liu, G. Zhu, W. Sun, and W. Hu, “Beyond traffic matrix: DELTA – a DAG-aware OCS logical topology optimization for AIDCs,” arXiv:2603.28096, Mar. 2026, v5, May 2026. [47] Q. Hu, W. Wang, X. Wang, Y. Li, Y. Zhao, Y. Tan, and J. Zhang, “OMLSched: Computation-driven flow scheduling for optical machine learning training clusters,” IEEE Open J. Commun. Soc., vol. 7, pp. 4100–4111, 2026. [48] X. Tan, Y. Feng, Y. Zhou, Y. Jiang, Y. Zhu, and H. Xu, “Dynamic compute and network orchestration for disaggregated RL,” in Proc. ACM SIGCOMM, Aug. 2026, pp. 1109–1126. [49] E. Amponsah and V. Addanki, “Birkhoff decompositions and photonic interconnects wait! Don’t forget the compute!” in Proc. ACM SIGCOMM HotOptics Workshop, Aug. 2026, pp. 2135–2141. [50] C.-Y. Chen, “Non-splitting coflow scheduling with provable guarantees in heterogeneous parallel networks,” arXiv:2501.09293, Jan. 2025, v7, Jun. 2026. [51] X. Wang, H. Shen, H. Tian, and D. Wang, “Scheduling coflows in multicore OCS networks with performance guarantee,” arXiv:2604.08242, Apr. 2026. [52] H. Xu, M. Hafezi, J. Fan, J. M. Taylor, G. F. Strouse, and Z. Ahmed, “Ultra-sensitive chip-based photonic temperature sensor using ring resonator structures,” Opt. Express, vol. 22, no. 3, pp. 3098–3103, 2014. [53] E. Timurdogan et al., “Automated wavelength recovery for microring resonators,” in Proc. CLEO, 2012, Paper CM2M.1. [54] K. Padmaraju, D. F. Logan, X. Zhu, J. J. Ackert, A. P. Knights, and K. Bergman, “Integrated thermal stabilization of a microring modulator,” Opt. Express, vol. 21, no. 12, p. 14342, Jun. 2013. [55] A. J. Cherian, A. Michalka, K. Murray, G. Röll, and K. Graichen, “Control approaches for operating point stabilization of microring resonator modulators under fast perturbations,” in Proc. SPIE Silicon Photonics XIX, G. T. Reed and A. P. Knights, Eds., vol. 12891, Mar. 2024, p. 128910D.

[56] Y. Shi, Q. Huang, Q. Zhang, S. Zhang, S. Zhou, X. Wang, J. Yang, and H. Yu, “Fast wavelength control of silicon PAM4 microring modulators based on an analytical bit-error-rate model,” J. Lightw. Technol., vol. 44, no. 15, pp. 6514–6522, Aug. 2026. [57] S. Yoon, P.-J. Chen, and S. Yu, “Cross-layer analysis of thermal tuning stalls in wafer-scale optical interconnects for LLM MoE training,” arXiv:2608.24637, Aug. 2026, v3, Sep. 2026. [58] Lumentum, “Lumentum demonstrates industry-leading technologies and products for scale-out, scale-up and scale-across AI infrastructure at OFC 2026,” Press release, Business Wire, Mar. 17, 2026. [Online]. Available: https://www.businesswire.com/news/home/20260317080634/en [59] B. Buscaino, E. Chen, J. W. Stewart, T. Pham, and J. M. Kahn, “External vs. integrated light sources for intra-data center co-packaged optical interfaces,” J. Lightw. Technol., vol. 39, no. 7, pp. 1984–1996, Apr. 2021. [60] D. Ford et al., “Availability in globally distributed storage systems,” in Proc. USENIX OSDI, 2010, pp. 61–74. [61] A. Cidon, S. Rumble, R. Stutsman, S. Katti, J. Ousterhout, and M. Rosenblum, “Copysets: Reducing the frequency of data loss in cloud storage,” in Proc. USENIX ATC, 2013, pp. 37–48. [62] F. Hark, R. Ring, S. D. Novack, and P. Britton, “Common cause failure modeling in space launch vehicles,” in Proc. IAASS Conf., May 2016, NASA NTRS 20160007073. [63] International Electrotechnical Commission, “IEC 61508-2:2010, Functional safety of electrical/electronic/programmable electronic safetyrelated systems, Part 2: Requirements for electrical/electronic/programmable electronic safety-related systems,” 2010, Annex E. [64] Optical Internetworking Forum, “Co-packaging framework document,” OIF, Tech. Rep. OIF-Co-Packaging-FD-01.0, Feb. 2022. [Online]. Available: https://www.oiforum.com/wp-content/uploads/OIF-Co-Packa ging-FD-01.0.pdf [65] R. Herrick, “Paths to improved laser reliability,” Compound Semiconductor, vol. 32, no. 2, Mar. 2026. [Online]. Available: https://compoundsemiconductor.net/article/123764/Paths to improved l aser reliability [66] Broadcom, “Broadcom showcases industry-leading quality and reliability of co-packaged optics,” Press release, Oct. 2025. [Online]. Available: https://www.broadcom.com/company/news/product-releases/63616 [67] E. G. Coffman, Jr., M. R. Garey, and D. S. Johnson, “An application of bin-packing to multiprocessor scheduling,” SIAM J. Comput., vol. 7, no. 1, pp. 1–17, Feb. 1978. [68] M. Yue, “On the exact upper bound for the multifit processor scheduling algorithm,” Ann. Oper. Res., vol. 24, no. 1, pp. 233–259, Dec. 1990. [69] R. Urata et al., “Mission Apollo: Landing optical circuit switching at datacenter scale,” arXiv:2208.10041, Aug. 2022. [70] R. L. Graham, “Bounds on multiprocessing timing anomalies,” SIAM J. Appl. Math., vol. 17, no. 2, pp. 416–429, Mar. 1969. [71] D. Ma, C. Pei, Y. Lu, Q. Zhou, Z. Wang, Y. Zhu, D. Jiang, D. Pei, J. Li, and G. Xie, “Don’t predict, prioritize: Rethinking GPU reliability assessment,” in Proc. ACM SIGKDD, Aug. 2026, pp. 7773–7784. [72] iCE40 UltraPlus family data sheet, Lattice Semiconductor, Feb. 2025, FPGA-DS-02008, ver. 2.4: 5,280 four-input look-up tables; SG48 package. [73] S. Williams, “Icarus Verilog, version 12.0,” 2023. [Online]. Available: https://steveicarus.github.io/iverilog/ [74] C. Wolf, “Yosys Open SYnthesis Suite, version 0.33 (git 2584903a060),” 2023. [Online]. Available: https://yosyshq.net/yosys/ [75] D. Shah, E. Hung, C. Wolf, S. Bazanski, D. Gisselquist, and M. Milanović, “Yosys+nextpnr: An open source framework from Verilog to bitstream for commercial FPGAs,” in Proc. IEEE FCCM, 2019, pp. 1–4. [76] K. Beck, Test-Driven Development: By Example. Boston, MA, USA: Addison-Wesley, 2003. [77] X. Xue et al., “Thermal tuning of Kerr frequency combs in silicon nitride microring resonators,” Opt. Express, vol. 24, no. 1, pp. 687–698, Jan. 2016. [78] N. A. Nobile et al., “Time-resolved temperature mapping leveraging the strong thermo-optic effect in phase-change materials,” ACS Photonics, vol. 10, no. 10, pp. 3576–3585, Oct. 2023. [79] Optical Internetworking Forum, “Implementation agreement for external laser small form factor pluggable (ELSFP) CMIS,” OIF, Implementation Agreement OIF-ELSFP-CMIS-01.0, Sep. 2024. [80] HUBER+SUHNER Polatis, “POLATIS Series 7000 384x384 port software-defined optical circuit switch,” Product page, Accessed: Oct. 2026. [Online]. Available: https://www.polatis.com/series-7000-384 x384-port-software-controlled-optical-circuit-switch-sdn-enabled.asp [81] Optical Internetworking Forum, “Management of external light sources and co-packaged optical engines,” OIF, White Paper OIF-MGT-CoPackaging-ELSFP-01.0, Nov. 2022.

24

[82] R. Zhu, Z. Jiang, C. Jin, P. Wu, C. A. Stuardo, D. Wang, X. Zhang, H. Zhou, H. Wei, Y. Cheng, J. Xiao, X. Zhang, L. Liu, H. Lin, L.-W. Chang, J. Ye, X. Yu, X. Liu, X. Jin, and X. Liu, “MegaScale-Infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism,” in Proc. ACM SIGCOMM, Sep. 2025, pp. 592–608. [83] W. Gao et al., “RollPacker: Mitigating long-tail rollouts for fast, synchronous RL post-training,” arXiv:2509.21009, Sep. 2025. [84] T. Hu, X. Liu, Y. Miao, Y. Xiao, H. Zang, Y. Zheng, X. Huang, J. Ding, Y. Zhang, Y. Yang, Y.-K. Zhang, Y. Sun, C. Han, X. Ma, W. Wang, Q. Gu, Y. Sun, Y. Xie, and X. Cai, “DORA: A scalable asynchronous reinforcement learning system for language model training,” arXiv:2604.26256, Apr. 2026. [85] Y. Zhou, J. Li, Y. Su, G. Ramesh, Z. Zhu, X. Long, C. Zhao, J. Pan, X. Yu, Z. Wang, K. Du, J. Wu, X. Sun, J. Liu, Q. Yu, H. Chen, Z. Liu, and E. Barsoum, “APRIL: Active partial rollouts in reinforcement learning to tame long-tail generation,” arXiv:2509.18521, Sep. 2025. [86] G. Sheng et al., “Laminar: A scalable asynchronous RL post-training framework,” arXiv:2510.12633, Oct. 2025.

[87] C. Ye, H. Zhang, M. Han, B. Zhong, X. Li, Q. Chen, X. Zhang, W. Zhang, K. Jiang, W. Zhang, H. Sun, W. Xiao, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau, “TensorHub: Scalable and elastic weight transfer for LLM RL training,” in Proc. ACM SOSP, Sep. 2026, pp. 882–897. [88] Z. Chen et al., “Role-based fault tolerance system for LLM RL posttraining,” arXiv:2512.22492, Dec. 2025. [89] B. Wan et al., “Robust LLM training infrastructure at ByteDance,” in Proc. ACM SOSP, 2025, pp. 186–203. [90] R. Qin et al., “Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot,” in Proc. USENIX FAST, 2025, pp. 155–170. [91] P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in Proc. ACM/IEEE ISCA, Jun. 2024, pp. 118–132. [92] J. M. Clark, J. Min, Y. Wang, Y. Hong, and G. Ostrouchov, “Modeling spatially correlated failure-time data under Euclidean and toroidal distance functions with an application to Titan GPU data,” IEEE Trans. Rel., vol. 75, pp. 2625–2637, 2026.

Related documents

• Throughput-Optimized Networks at Scale#1131552
• Parallel Branch Model Predictive Control on GPUs#629788
• Parallel Branch Model Predictive Control on GPUs#627926
• Hybrid Model Predictive Control with Physics-Informed Neural Network for Satellite Attitude Control#927240
• Learning-enabled Acceleration of Scenario-based Model Predictive Control#973271