Scaling Inference Prefill with High-Radix Photonic Interconnects

Scaling Inference Prefill with High-Radix Photonic Interconnects

arXiv:2609.01821v1 [cs.DC] 1 Sep 2026

Arulselvan Madhavan, Peter Carson, Taylor Groves, Thomas Graham

Abstract—With the rise of inference as today’s dominant AI workload, the industry is transitioning to highbandwidth photonic interconnects to meet the large scale-up requirements of increasingly complex Mixtureof-Experts (MoE) models. This paper quantifies the benefits of 3D-integrated photonic interconnects for inference prefill by analyzing tradeoffs between highconcurrency throughput for Large Language Model (LLM) chat and the large context windows typically required for reasoning and agentic AI. We simulate three MoE models: short context (1K–8K tokens), medium context (128K tokens), and long context (1M tokens). We evaluate this workload across existing copper-based GPU systems and one with high bandwidth integrated photonics. We show 2.1–3.2× latency improvements in the stressed high-batch regimes and 2.8–5.8× improvements over baselines in communication-limited configurations. 3D photonics enable the 1152-GPU footprint required to lower time-to-first-token, yielding 2.2–4.5× speedups across production-grade platforms when electrical systems cross their inherent scale-uppod limits. Index Terms—Silicon Photonics, AI Inference, Photonic Interconnects, 3D-Integrated Photonics, Large Language Models, Disaggregated Inference, Scale-up Pod, Radix

I. Introduction The proliferation of AI-driven applications and broad global adoption have transformed the compute infrastructure landscape; while frontier training remains critical, the industry has matured into an inference-first era, with inference now accounting for 80–90% of compute cycles [1, 2]. The rise of reasoning models and agentic workflows demands ”continuous inference” capabilities, forcing providers to balance the high-concurrency throughput required for LLM chat services against the large context windows required for deep research and agentic AI. In production agentic coding workflows, the median initial context (prefill) length already reaches ∼96K tokens— driven by accumulated tool-call outputs, system prompts, and skill primers—with nearly half of all requests exceeding 128K tokens [3], making prefill the dominant computational phase. This not only places simultaneous pressure on system latency and memory bandwidth but also creates a fundamental engineering dilemma in system optimization between cost-efficiency and user experience. The trajectory of AI hardware follows the path of domain-specific architecture (DSA) evolution described by Patterson and Hennessy [4], moving toward silicon increasingly specialized for the distinct phases of LLM

execution. This specialization has led to a bifurcation in silicon design: for example, NVIDIA’s Rubin roadmap focuses on large parallel compute particularly suited for the prefill phase of inference, while architectures like the Groq Language Processing Unit (LPU) prioritize highspeed, deterministic memory access to optimize the decode phase. However, these refinements do not eliminate system bottlenecks; they disperse and transform them. While raw floating-point operations per second (FLOPS) continue to scale, the ”choke point” for decode remains the memory bandwidth [5]. For inference prefill, the reach and shoreline limits of conventional interconnects are the primary constraint for large-scale distributed systems. In the prefill phase, providers face a fundamental batching dilemma: increasing batch sizes is necessary to maximize GPU utilization and aggregate throughput. However, larger batches proportionately increase the volume of data exchanged in distributed collectives—tensor-parallel allreduces, expert-parallel all-to-alls, and sequence-parallel all-gathers—across the compute cluster [6]. This creates a major networking challenge in the form of large activation tensors continuously exchanging large amounts of data across the compute cluster. With electrical interconnects, which are constrained by a fundamental bandwidth-reach-power envelope, the communication overhead of these workloads becomes a primary bottleneck. As per-lane signaling rates increase to satisfy bandwidth demands, the effective reach of passive copper shrinks to 1m at 224G and to tens of centimeters at 448G [7], effectively restricting high-performance scale-up pods to a single rack, forcing the use of slower scale-out fabrics for multi-rack communication. In distributed inference, communication overhead can account for more than half of total prefill latency—over 65% on bandwidth-limited inference GPUs under tensor parallelism [8]—making interconnect bandwidth a firstorder constraint on inference throughput and latency. Conversely, in reasoning and agentic scenarios (128K to 1M+ tokens), meeting strict time-to-first-token (TTFT) service level agreements (SLAs) requires extreme parallelism across multiple racks encompassing hundreds of nodes. Photonic interconnects enable the expansion of the high-bandwidth scale-up pod to hundreds or thousands of compute packages across multiple racks. This allows inference engines to sustain larger batch sizes for throughput and large context for reasoning while maintaining compute

Accepted at the 33rd IEEE Symposium on High-Performance Interconnects (HotI 2026). Lightmatter. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

utilization. In this work, we quantify the impact of 3D-integrated photonics on AI inference efficiency. We demonstrate how 4x bandwidth and radix up to 1,152 unlock new opportunities for larger context handling at higher batch sizes, specifically detailing: • High Bandwidth Impact: Increased bandwidth between compute nodes allows inference engines to maintain large-scale batch sizes while meeting SLA latency constraints. • Expanded Scale-Up Pods: Extending highbandwidth scale-up pods up to 1152 devices ensures the ability to process reasoning contexts (128K+ tokens) without incurring scale-out penalties. • Large Context Viability: For emerging 1M+ token windows, the bandwidth and scale-up pod size achievable with optical interconnect technology are necessary to achieve fast TTFT metrics. Overall, on the NVIDIA B300 platform with FP4 quantization and using a 42B-active-parameter MoE model, optical interconnects deliver approximately 2.6× improvement in prefill latency for 1K-token contexts, 2.2× for 8K, 2.9× for 128K, and 2.3× for 1M tokens when looking at systems comprising 72 to 1152 processors (Table IV). These ratios depend on the operating regime—defined by batch size, context length, and device count relative to the native scale-up pod—and should not be read as universal improvements: configurations that are computebound show modest gains, whereas those that push batch size, accelerate per-GPU compute, or cross the scale-up pod boundary expose larger communication penalties and benefit more from higher interconnect bandwidth. Racklimited B200 and Rubin baselines show larger improvements at 128K–1M input tokens because communication and scale-up pod limits dominate prefill performance. Baselines with larger hybrid electrical-and-optical scaleup pods reduce the comparative improvement at 128K input tokens, but the all-optical configuration still unlocks substantially larger gains for 1M tokens. II. Background A. LLM Inference Workload Characteristics LLM inference is characterized by a distinct dual-phase execution pattern, each with divergent resource demands. 1) Prefill Phase (Compute-Bound): The prefill phase processes the entire input prompt in parallel to generate the initial Key-Value (KV) cache. Because the full sequence is known a priori, this phase relies on dense matrix multiplications that can saturate GPU compute utilization [9]. The primary performance metric is TTFT, which is directly sensitive to the raw FLOPS the processor can achieve. 2) Decode Phase (Memory-Bound): Following prefill, the system enters the decode phase to auto-regressively generate output tokens one at a time. Each step requires

loading the entire model weights and the active KV cache from memory to produce a single token. This operation has low arithmetic intensity and is fundamentally bound by High Bandwidth Memory (HBM) bandwidth rather than compute capacity [5]. The critical metric is Time Per Output Token (TPOT), which determines the perceived fluidity of generation. B. The Efficiency vs. Interactivity Challenge To achieve high-performance LLM serving, system engineers must navigate a complex landscape of hardware constraints and user expectations. 1) Batch Throughput vs. Response Time: The fundamental tension in inference serving lies in batching. Increasing batch sizes maximizes aggregate system throughput by saturating GPU compute resources, which improves cost efficiency for the provider. However, larger batches simultaneously degrade individual inter-token latency for the user due to increased queuing and iteration times [9]. Providers are thus forced to compromise on throughput to meet strict interactivity SLAs. 2) Cache Pressure and Eviction: Compounding the batching dilemma is memory pressure. Retaining longcontext KV caches is essential for seamless user experiences in multi-turn dialogues or reasoning tasks. However, the large memory footprint of these caches limits the total number of concurrent users the hardware can support [5]. When HBM capacity is exhausted, the system must trigger performance-degrading evictions or offload data to slower CPU memory, creating a “memory wall” that caps scalability [10]. C. Hardware Bottlenecks in the Scale-Up Pod While algorithmic innovations such as continuous batching [9] and disaggregated inference [11] attempt to mitigate the hardware constraints, system performance is ultimately limited by the physical characteristics of the underlying hardware interconnects [7]. 1) The Shoreline Constraint: As computational capability scales with die area, I/O bandwidth is constrained to the chip’s perimeter, or “shoreline” [7]. Large portions of this shoreline are reserved for HBM to ensure signal integrity, significantly reducing the available area for scaleup interconnect serializer/deserializer (SerDes) links [7]. This bottleneck restricts the aggregate bandwidth available to support the large batch sizes required for efficiency. 2) Reach and Energy Limitations: Current scale-up fabrics rely on electrical copper interconnects, but the physics of high-speed electrical transmission imposes a “reach-energy” trade-off. At 224 Gb/s, the maximum reach of passive electrical interconnects shrinks to approximately 1 meter [7]. Extending this reach requires power-hungry retimers that divert power away from computation [7]. Consequently, electrical scale-up pods are effectively limited to a single rack (approximately 72 to 144 GPUs), constraining the size of the logical GPU available for

inference [7]. Scale-out fabrics (e.g., InfiniBand) connect far more devices, but at much lower per-GPU bandwidth (Table III, scale-up (SU) bandwidth vs. scale-out (SO) bandwidth), so they do not substitute for a larger scale-up domain. 3) 3D Integrated Photonics: To overcome these barriers, 3D-integrated photonic interconnects offer a solution by stacking SerDes or host ASICs directly onto the optical engines [7]. This approach decouples I/O from the package shoreline, allowing SerDes to be distributed across the chip area. This architecture enables a massive increase in bandwidth density and radix, allowing for scale-up pods of greater than 1152 GPUs while mitigating the power and cooling challenges inherent in dense clusters [7]. III. Meeting AI Inference Demands with Semiconductor Design Recently, the market has seen the introduction of solutions tailored towards the specific needs of inference workloads. Major silicon providers such as Nvidia are moving away from general-purpose, one-size-fits-all silicon designs towards architectures targeting disaggregated inference, such as Rubin CPX [12] and Groq LPX [13]. Other specialized silicon providers, such as Cerebras and dMatrix, have built silicon optimized for inference [14, 15]. These architectures generally aim to balance the ratios of compute power, memory capacity, memory bandwidth, and interconnect bandwidth to match the needs of the prefill or decode phase (or both) of inference. While compute, memory capacity, and memory bandwidth continue to scale [16], a pivotal shift in interconnect technology is required to increase bandwidth density, overcome the reach limitations of copper, and build more powerful inference systems. A. 3D Photonic Interconnects To address these limitations, the industry is moving toward true 3D integrated photonics, exemplified by the Lightmatter Passage platform [17], which we cite only as a reference point for the bandwidth, radix, and energy figures used below (Section IV-C) and do not directly simulate. Passage utilizes a 3D package architecture where the Electronic Integrated Circuit (EIC) or host ASIC is stacked directly on top of the Photonic Integrated Circuit (PIC). Passage offers distinct advantages for nextgeneration inference hardware: •

Extreme Bandwidth: By placing optical microring modulators (MRMs) just 100 microns beneath the electrical SerDes, Passage eliminates the need for power-hungry equalization and long electrical traces. This vertical integration allows I/O to be distributed anywhere across the chip area—represented as a “grid” of vertical connections—rather than being constrained to the package shoreline. MRMs combine multiple wavelengths (lambdas) of light onto a single

fiber. 3D stacking and wavelength division multiplexing enables unprecedented bandwidth. While current GPU configurations, such as the NVIDIA Blackwell architecture, are equipped with 14.4 Tb/s of bidirectional interconnect bandwidth [18], a Passage-enabled GPU can support greater than 64 Tb/s of bidirectional bandwidth per GPU. • High Radix: With photonic interconnects, data escapes the host GPU or switch using tightly spaced optical fibers. Fibers are 127 µm (compared to 254 µm for 30AWG copper) and combine both transmit and receive signals on a single fiber. A bidirectional fiber replaces four copper wires (a differential pair (DP) for transmit and DP for receive). This allows for 8X increase in radix vs copper-based approaches. Furthermore, optical interconnects overcome the reach limitations of copper cables, allowing for scale-up pods to extend beyond a single rack. This enables both larger scale-up pods and sparser racks, which alleviates power density and cooling constraints. • Energy Efficiency: For inference workloads where power efficiency is paramount, 3D integrated photonics dramatically reduce the energy cost of data movement. The complete link budget—including the PIC, laser, and host SerDes—operates at approximately 4.3 pJ/bit (2.3 pJ/bit for PIC and laser only), comparable to the ∼5 pJ/bit of passive copper and significantly lower than the >20 pJ/bit of traditional pluggable optics [7]. TABLE I Comparison of Passage optics compared to passive copper. Reach (m) Port Pitch (µm) Tbps/Fiber or Wire Power (pJ/b)

Copper

3D Optics

∼1 (224G) 1020, 30AWG, 2 DP 0.224/DP 5 (SerDes)

1000 127 bidirectional fiber 1.79, 16λ 4.3 (2.3 PIC & Laser, 2 SerDes)

Improvement 1000× 8× 8× 0.13×

Table I shows an overview of the benefits of 3D photonics compared to passive copper. In this paper, we quantify the benefits of these photonic interconnects in the AI inference landscape. IV. Methodology A. Modeling LLMs We model the inference of production-grade Machine Learning (ML) models by generating a Multi-Level Intermediate Representation (MLIR), capturing the model partitions (Tensor parallel, Expert parallel, Context parallel). There are two main reasons for engaging in this style of modeling. First, we want to take implementations deployed in production, with model partitions faithfully captured. Second, our focus is on accurately modeling the communication calls. We have the compiler generate these calls, thereby ensuring their implementation is as it would be in a production system. We maintain a fork of the XLA (Accelerated Linear Algebra) compiler [19] that we use to

walk through the MLIR file and capture the compute and communication costs of any model. B. Model Configurations We evaluate three model scales representative of MoE frontier models, as shown in Table II. All three models use Multi-Head Latent Attention (MLA) [20], which significantly reduces KV cache size through low-rank compression compared to Grouped-Query Attention (GQA)-based architectures. TABLE II Model Configurations Evaluated. R1 denotes a divisibility-adjusted variant of DeepSeek-R1; see Section IV-B. Metric

Mini

R1

Next

Layers Experts Attention heads Total params Active params

30 144 144 190B 21B

61 288 144 840B 42B

120 1152 288 13T 201B

The Mini model (21B active parameters) represents the smallest MoE tier; R1 (42B active parameters) is aligned with the DeepSeek-R1 architecture [20]; Next (201B active parameters) represents a larger-scale MoE system. We evaluate both FP4 and FP8 quantization precisions [21]. The R1 variant used in this work applies divisibilitymotivated parameter adjustments (embedding dimension 8064 compared to 7168 in the production DeepSeek-R1 [22], 288 routed experts compared to 256, vocabulary 145 440 compared to 129 280) to satisfy mesh-divisibility constraints across device counts of 8, 16, 24, 48, and 72. This yields a model approximately 14% larger than production DeepSeek-R1 in parameter count while preserving the same communication topology and scaling behavior. Complete configuration details are in Appendix A. C. Hardware Configuration We model four GPU architectures at the per-GPU and scale-up-pod specifications summarized in Table III. Scaleup (SU) is the maximum electrical scale-up pod (GPU count); SU bandwidth (BW) is the unidirectional scaleup interconnect bandwidth (GB/s) per GPU across that pod. Scale-out (SO) bandwidth is the unidirectional crossrack interconnect bandwidth (GB/s) per GPU modeled for devices beyond the SU pod. Rubin denotes a reported roadmap-class Rubin configuration, corresponding to the proposed dual-die Vera Rubin GPU with up to 72 GPUs in a scale-up pod (144 logical dies). R4 denotes a speculative roadmap-class configuration inferred from the SemiAnalysis report; it models a potential quad-die GPU by extrapolating from that dual-die design, with up to 576 GPUs in a scale-up pod (2304 logical dies) [23]. We note that any R4 configuration beyond 72 GPUs reportedly uses optical cables across racks, and our analysis compares

this configuration with an all-optical scale-up pod configuration. Best optical-vs.-electrical prefill comparisons by platform appear in Section V-D. TABLE III Modeled electrical-baseline accelerators. FP8/FP4 columns are dense matmul peak PFLOPS per GPU; SU BW is unidirectional scale-up bandwidth per GPU within the electrical pod; SO BW is unidirectional scale-out (cross-rack) bandwidth per GPU modeled for devices beyond the SU pod. MoE sparse matmuls use an effective 2× speedup relative to the dense compute rate. † R4 specs extrapolate the proposed dual-die Rubin design to a potential quad-die configuration; parameters are inferred from a roadmap report [23] and should be read as projections. Plat. FP8 FP4 Max SU SU BW SO BW HBM HBM dense (PFLOPS) GPUs uni. (GB/s) TB/s GB B200 B300

4.5 7

9 14

72 72

900 900

50 100

7.67 7.94

192 288

Rubin 17.5 R4† 35

25 50

72 576

1800 1800

100 100

22 53

288 288

For each electrical baseline in Table III, we pair a matching optical configuration: identical per-GPU compute and HBM, 4× the scale-up bandwidth per GPU vs. the electrical baseline, and a maximum optical scale-up pod of 1152 GPUs (enabling the sweeps in Section IV-D). The 4× multiplier is a conservative lower bound: Section III notes >64 Tb/s of bidirectional bandwidth per GPU for 3D-integrated photonics [17] vs. 14.4 Tb/s for current electrical NVLink [18], a ratio exceeding 4×. The comparison isolates interconnect bandwidth and radix; we do not model vendor-specific optical products. Appendix C summarizes how Blackwell-class compute is mapped from public ISA documentation for the XLA-based cost model. D. Experimental Sweeps We perform two sweeps to characterize the throughputlatency trade-off space: 1) Device sweep: We fix the batch size per context length (8192 at 1K context length, 1024 at 8K context length, 64 at 128K context length, and 8 at 1M context length), which holds total input tokens near 8M in each case. The device count is swept over valid parallel meshes for each model: typically 8–72 GPUs for 1K and 8K input tokens (rack-scale scale-up pods in Table III), up to 288 GPUs at 128K input tokens, and up to 1152 at 1M input tokens. Prefill latency is reported vs. cluster size for electrical baseline and optical counterpart at each point. 2) Batch sweep: We fix the device count per context length (16 at 1K context length, 72 at 8K context length, 288 at 128K context length, 1152 at 1M context length). The batch size is swept across a range. Input-token throughput and prefill latency use the same overlapped-latency pipeline as the device sweep.

We sweep four context lengths representative of key operating regimes: 1K, 8K, 128K, and 1M tokens. E. Performance comparisons and table entries For each evaluated configuration (model, precision, hardware, context length, sweep type), we compute overlapped prefill latency for each point in the sweep. Overlapped prefill latency takes into account the compute– communication overlap from the XLA-based cost walk. The number reported at any given configuration (e.g., in Table IV) is the ratio of overlapped latency for the electrical baseline to the overlapped latency for the opticallyinterconnected comparison. Values greater than 1 denote an improvement in prefill latency when simulating optical interconnects. For device sweep entries, we vary device count, for each model and GPU, and take the maximum multiplier across all device counts in the sweep. For batch sweep entries, we vary batch size at fixed device count and take the maximum ratio across batch sizes. V. Results We evaluate the impact of optical interconnects on LLM inference across three workload regimes defined by context length: (i) short-context workloads (1K–8K tokens), (ii) medium-context workloads (128K tokens), and (iii) long-context workloads (1M tokens). These regimes stress different system bottlenecks—communication saturation from large batch sizes versus compute saturation leading to benefits from multi-rack scale-out—and therefore expose distinct benefits enabled by higher-bandwidth, largeradix optical interconnects. All results in this section use the DeepSeek R1 variant (42B active parameters, FP4) on the NVIDIA B300 platform unless otherwise stated. Section V-D summarizes results across all three models, four hardware platforms, and both sweep types for FP4 precision; Appendix D (Table VI) extends this to FP8. Sections V-A–V-D use a bulk-prefill sweep (fixed aggregate tokens, swept device count/batch size, up to batch 16,384) to isolate interconnect sensitivity; Section V-E then cross-checks these trends under the bounded per-worker batches of latency-sensitive, interactive serving. A. Low-Context Workloads (1K and 8K Input Tokens) We examine short context workloads in Figures 3 and 4. At short context lengths, the data reveals a counterintuitive but defining pattern: communication can dominate computation as batch size grows. As seen in panel (b) in Figures 3 and 4, serving requests at very large batch sizes maximizes compute utilization per GPU. However, larger batches also create larger activation tensors that must be continuously shuffled across devices. Figure 1(a) isolates this effect for the 8K-token workload at 72 devices: by batch 2048, the electrical baseline spends approximately 40.6 seconds in communication versus 16.8 seconds in computation.

With electrical interconnects, this communication overhead becomes a significant bottleneck to throughput. As batch size increases, the volume of data exchanged in distributed collectives grows quickly enough that the electrical baseline flattens in throughput. Optical interconnects, offering 4× higher bandwidth, directly reduce this communication phase: in Figure 2(a), the same 8Ktoken, batch-2048 point spends approximately 10.2 seconds in communication while compute time remains 16.8 seconds. Consequently, the optical configuration sustains proportionally higher input-token throughput per GPU at large batches, as shown by the solid lines in panel (b) of Figures 3 and 4. The dashed lines in panel (b) highlight the corresponding operational benefit: the optical configuration permits a system to meet a lower-latency SLA at larger batch sizes than the electrical infrastructure. B. Medium-Context Workloads (128K Input Tokens) Figure 5 shows the results for 128K sequence length (288-device batch sweep). First, on a single rack (72 GPUs), continuous inference of a 128K token sequence at batch 64 is bound by computation time. Figure 1(b) shows that at 72 devices, computation requires approximately 56.1 seconds, whereas communication takes 20.3 seconds. In order to achieve a lower TTFT SLA, an operator would be forced to scale out the workload across more devices—for instance, to 288 GPUs. However, resolving the computational bottleneck by scaling to 288 GPUs introduces a severe interconnect penalty for electrical systems. When the device count exceeds the capacity of a single 72-device rack, cross-rack communication links become a bottleneck. In Figure 1(b), the electrical baseline at 288 devices drops compute to approximately 14.3 seconds, but communication increases to 35.5 seconds. Optical scale-up changes the time spent doing communication versus doing computation. By introducing a homogeneous ultra-high-bandwidth scale-up pod up to 1152 devices, optical interconnects enable the compute scaling to 288 GPUs while dropping the communication time to 2.3 seconds, as shown in Figure 2(b). This manifests as a ∼2.8–3.0× improvement at 288 devices for B300 (Table IV). Rack-limited platforms like B200 and Rubin observe even larger ratios (∼4.3–5.8×) due to more pronounced cross-rack penalties. R4, built with a large native electro-optical scale-up pod of 576 GPUs, sees smaller but still material gains at this context length (∼2.0–2.1×). Panel (b) of Figure 5 shows the operational consequence: high scale-up and high bandwidth are necessary for sustained batch sizes at reasonable latencies, with the solid lines showing throughput gains and the dashed lines showing the corresponding latency reductions. C. Large-Context Workloads (1M Tokens) Figure 6 shows the results for a 1M sequence length over a maximum of 1152 devices. At this scale, keeping the 1152

Fig. 1. Communication and compute component latency for representative DeepSeek R1 variant, FP4, NVIDIA B300 electrical-baseline prefill workloads — (a) low-context 8K tokens at 72 devices across batch sizes, (b) 128K tokens at batch 64 across device counts, and (c) 1M tokens at batch 8 across device counts. The panels isolate the raw component times behind the overlapped prefill-latency curves in Figures 4, 5, and 6.

Fig. 2. Communication and compute component latency for the corresponding optical scale-up configuration with 4× per-GPU scale-up bandwidth. Compute time is unchanged relative to the matched electrical baseline; the reduced communication component shows where high-bandwidth optical scale-up converts communication-limited cases back toward compute-limited operation.

devices in a high-bandwidth scale-up pod is essential for low prefill latency. In panel (a) of Figure 6, single-rack (up to 72 GPUs) or lightly distributed scale-up pods are compute-timebottlenecked. To bring processing time to lower levels for 1M tokens, the workload benefits from scaling to a higher device footprint; this analysis sweeps up to 1152 devices at batch 8. At 1152 devices, the compute time decreases to 26.2 seconds per pass, as shown in Figure 1(c). The communication burden increases as the number of GPUs scales, and there is an inflection point when scaling the workload to multiple racks. In the Baseline systems, with up to 72 GPUs in a scale-up pod, the communication overhead partially counters the performance benefits of

scaling the workload beyond 72 devices. The diminishing returns are particularly apparent when scaling from 288 to 576 devices, and again from 576 to 1152 devices. A 1152device footprint results in 39.2 seconds of communication latency per forward pass. The higher scale-up pod and bandwidth enabled in the optical configuration (all 1152 devices in the scale-up pod) allow the workload to continue scaling nearly linearly as more devices are added, reducing communication latency to just 1.6 seconds for 1152 devices in Figure 2(c). This enables an overlapping latency of 27.5 seconds, dominated by compute time, a 2.3× total speedup compared to the electrically connected Baseline on the B300 (Table IV). As shown in panel (b) of Figure 6, optics allow for greater throughput per GPU (solid lines) and

Fig. 3. DeepSeek R1 variant, FP4, NVIDIA B300, 1K tokens — (a) prefill latency vs. number of devices, (b) input-token throughput per GPU (left axis, solid lines) and prefill latency (right axis, dashed lines) vs. batch size. The optical configuration (Table III pairing) achieves approximately 2.58× lower latency at the best device sweep point vs. electrical baseline and sustains larger batch sizes at any latency SLA.

Fig. 4. DeepSeek R1 variant, FP4, NVIDIA B300, 8K tokens — (a) prefill latency vs. number of devices, (b) input-token throughput per GPU (left axis, solid lines) and prefill latency (right axis, dashed lines) vs. batch size. The optical advantage in the device sweep is approximately 2.21× at comparable device counts.

Fig. 5. DeepSeek R1 variant, FP4, NVIDIA B300, 128K tokens — (a) prefill latency vs. number of devices, (b) input-token throughput per GPU (left axis, solid lines) and prefill latency (right axis, dashed lines) vs. batch size. Communication dominates at this context length; on B300 the best device-sweep factor is approximately 2.93×, lower than on B200/Rubin at the same aggregate electrical scale-up bandwidth because faster compute partially offsets the collective bottleneck.

Fig. 6. DeepSeek R1 variant, FP4, NVIDIA B300, 1M tokens — (a) prefill latency vs. number of devices, (b) input-token throughput per GPU (left axis, solid lines) and prefill latency (right axis, dashed lines) vs. batch size. Device sweep best-case improvement vs. electrical baseline is approximately 2.27× on B300 at 1152 devices; B200/Rubin reach ∼3.4–4.5× and R4 reaches 8×+ in Table IV because the latency floor is set more by scale-up limits than by per-GPU FLOPS alone.

lower prefill latency (dashed lines) as batch sizes increase. This large-context, high-batch paradigm is particularly relevant for agentic and reasoning workloads that generate many intermediate tokens per request [24, 25]. D. Summary Across Models, Precisions, and Hardware Table IV consolidates the multiplier sweep across all three model scales, four hardware platforms, FP4 precision, and 36 representative scenarios (Table VI in Appendix D provides the corresponding FP8 numbers). Each cell is the best electrical-to-optical overlapped prefilllatency ratio for that configuration. The header rows list the sequence length, total-token tier, device count, and batch size for each scenario. The total-token tier is important because it fixes the derived batch size at each sequence length; consequently, two columns with the same context length can expose different communication pressure if they represent different aggregate input-token volumes. Several consistent patterns emerge from this analysis: a) GPU type and electrical scale-up pod shape the performance benefits of implementing optical interconnects: At 1K–8K context length, the performance benefits of optical interconnects generally increase from B200 to B300 to Rubin to R4 (Table IV): faster GPUs shrink the compute component of prefill, so the same collective traffic accounts for a larger fraction of latency. FP4 shows larger multipliers than FP8 for the same reason—faster arithmetic exposes communication more—a pattern visible across all platforms in Table VI; this reflects the compute/communication balance, not a change in communication volume. b) Context length, total tokens, and the compute– communication balance: Moving from 1K to 8K at the same total-token tier reduces the derived batch size eightfold and increases per-sequence computation, so the multiplier can dip even though the context is longer, since

communication is less dominant—this is why 8K gains are often lower than 1K gains. Conversely, the low-gain cells around 128K at 16–72 devices and 1M at 16–288 devices have not yet crossed the most expensive scale-up boundary or remain compute-dominated: optical interconnects do not provide a uniform speedup when the baseline is not communication-limited. c) Device count and scale-up-pod transitions: The largest changes occur when the device count needed to reduce compute time exceeds the native electrical scaleup pod. At 128K, the 288-device columns are substantially larger than the 16- and 72-device columns for rack-limited platforms, since the workload has crossed from intrarack scale-up into slower scale-out communication; the same mechanism appears at larger scale at 1M, where electrical baselines pay a large penalty once the footprint spills beyond the high-bandwidth pod. Because the optical configuration improves both bandwidth and radix, the observed speedup can exceed what a simple 4× bandwidth argument alone would predict. d) Hardware generation and scale-out bandwidth: The B200-to-B300 comparison at 128K and 288 devices reflects two countervailing effects: B300’s faster compute would tend to raise the value of communication bandwidth, but its higher modeled scale-out bandwidth modestly reduces the electrical penalty at the 288-device boundary, so B300 shows slightly smaller optical multipliers than compute scaling alone would predict. The relevant variable is thus not per-GPU FLOPS alone, but the ratio among compute rate, scale-up bandwidth, scale-out bandwidth, and required device count. e) Rubin and R4: Rubin illustrates the cost of pairing very high per-GPU compute with a 72-GPU electrical scale-up pod: at 128K and 288 devices, its racklimited baseline incurs a large scale-out communication cost, producing the largest medium-context multipliers

TABLE IV Sweep best multiplier heatmap (FP4 quantization). Device columns use dev ∈ {16, 72, 288, 576, 1152} (capped by seq len); the underlying sweep includes more device counts in summary.csv. Three total-token tiers per column (multiples of 10242 ; batch = tokens/seq len, capped). Tiers depend on dev: [8,72] → 2M/8M/16M; (72,288] → 2M/8M/32M; (288,1152] → 8M/32M/128M. Header seq len uses binary-style 1K = 1024, 8K = 8×1024, 128K = 128×1024, 1M = 10242 . Cell color scales from min to max among these heatmap cells only. For FP8 results see Table VI in Appendix D. Model Hardware Quant

Best multiplier (optical vs. no-optical) seq len 1K 1K 1K 1K 1K 1K 8K 8K 8K 8K 8K 8K 128K 128K 128K 128K 128K 128K 128K 128K 128K tokens 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 32M dev 16 16 16 72 72 72 16 16 16 72 72 72 16 16 16 72 72 72 288 288 288 batch 2048 8192 16384 2048 8192 16384 256 1024 2048 256 1024 2048 16 64 128 16 64 128 16 64 256

mini

r1

next

1M 2M 16 2

1M 8M 16 8

1M 16M 16 16

1M 2M 72 2

1M 8M 72 8

1M 1M 1M 1M 1M 1M 1M 1M 1M 1M 16M 2M 8M 32M 8M 32M 128M 8M 32M 128M 72 288 288 288 576 576 576 1152 1152 1152 16 2 8 32 8 32 64 8 32 64

B200

FP4

2.12 2.23

2.24 2.33 2.41

2.41 1.59 1.61 1.62 2.04 2.09 2.09 1.06 1.07 1.07 1.23 1.23 1.23 3.98 4.27 4.30 1.01 1.01 1.01 1.03 1.03 1.03 1.50 1.51 1.51 2.12 2.13 2.13 3.29 3.32 3.32

B300

FP4

2.34 2.41

2.41 2.59 2.64

2.64 1.65 1.67 1.67 2.21 2.24 2.24 1.07 1.07 1.07 1.25 1.25 1.25 2.75 2.84 2.85 1.01 1.01 1.01 1.04 1.04 1.04 1.28 1.28 1.28 1.60 1.60 1.60 2.20 2.21 2.21

Rubin

FP4

2.48 2.62

2.62 2.71 2.80

2.81 1.81 1.85 1.85 2.38 2.45 2.45 1.10 1.10 1.10 1.33 1.34 1.34 5.16 5.51 5.55 1.01 1.01 1.01 1.05 1.05 1.05 1.71 1.72 1.73 2.59 2.60 2.60 4.24 4.28 4.28

R4

FP4

3.00 3.10

3.11 3.11 3.18

3.19 2.36 2.41 2.42 2.92 2.98 2.99 1.21 1.21 1.22 1.66 1.67 1.67 1.91 1.98 1.99 1.03 1.03 1.03 1.11 1.11 1.11 1.19 1.19 1.19 1.28 1.28 1.28 8.04 8.14 8.15

B200

FP4

2.11 2.21

2.22 2.28 2.35

2.36 1.59 1.62 1.62 2.01 2.06 2.06 1.07 1.07 1.07 1.23 1.23 1.23 4.12 4.42 4.46 1.01 1.01 1.01 1.03 1.03 1.03 1.53 1.53 1.54 2.18 2.19 2.19 3.41 3.45 3.45

B300

FP4

2.32 2.38

2.38 2.53 2.58

2.58 1.65 1.67 1.67 2.18 2.21 2.21 1.07 1.07 1.07 1.24 1.25 1.25 2.83 2.93 2.93 1.01 1.01 1.01 1.04 1.04 1.04 1.29 1.30 1.30 1.63 1.63 1.63 2.27 2.28 2.28

Rubin

FP4

2.45 2.58

2.58 2.64 2.72

2.73 1.81 1.85 1.85 2.34 2.40 2.40 1.10 1.10 1.10 1.33 1.33 1.33 5.34 5.71 5.75 1.01 1.01 1.01 1.05 1.05 1.05 1.75 1.76 1.76 2.68 2.69 2.69 4.41 4.45 4.45

R4

FP4

2.95 3.05

3.06 3.05 3.13

3.13 2.34 2.39 2.40 2.86 2.92 2.92 1.21 1.22 1.22 1.66 1.67 1.67 1.94 2.01 2.02 1.03 1.03 1.03 1.11 1.11 1.11 1.20 1.20 1.20 1.31 1.31 1.31 8.33 8.43 8.44

B200

FP4

2.18 2.23

2.23 2.33 2.36

2.36 1.62 1.63 1.63 2.05 2.07 2.08 1.07 1.07 1.07 1.24 1.24 1.24 4.38 4.52 4.53 1.01 1.01 1.01 1.03 1.03 1.03 1.55 1.55 1.55 2.22 2.23 2.23 3.50 3.51 3.51

B300

FP4

2.38 2.40

2.40 2.57 2.59

2.59 1.68 1.68 1.68 2.21 2.23 2.23 1.07 1.07 1.07 1.25 1.25 1.25 2.95 2.99 2.99 1.01 1.01 1.01 1.04 1.04 1.04 1.31 1.31 1.31 1.66 1.66 1.66 2.33 2.33 2.33

Rubin

FP4

2.53 2.58

2.59 2.69 2.73

2.73 1.85 1.86 1.86 2.38 2.41 2.41 1.10 1.10 1.10 1.34 1.34 1.34 5.67 5.82 5.84 1.01 1.01 1.01 1.05 1.05 1.05 1.79 1.79 1.79 2.74 2.74 2.74 4.51 4.53 4.53

R4

FP4

3.01 3.05

3.05 3.10 3.13

3.13 2.38 2.40 2.40 2.89 2.91 2.92 1.22 1.22 1.22 1.67 1.68 1.68 2.06 2.09 2.10 1.03 1.03 1.03 1.12 1.12 1.12 1.22 1.22 1.22 1.35 1.35 1.35 8.42 8.46 8.47

(∼5.5–5.8×). R4’s 576-GPU hybrid pod already covers this regime, so its optical gains there are smaller (∼2.0– 2.1×); but this larger native pod is insufficient at 1M, 1152 devices, where the all-optical configuration again avoids the scale-out penalty and produces the largest multipliers in the table (∼8.0–8.5×). This shows large scale-up pods matter most for long-context workloads that force device footprints beyond a single rack. f) Model scale: Across Mini, R1, and Next, the table’s qualitative structure is stable: gains are modest in compute-bound cells, increase with faster arithmetic at low context, and peak when long-context workloads exceed the native scale-up pod. The exact multiplier varies with model size, but the dominant predictor remains whether collective communication lies on the overlapped-latency critical path. E. Serving-Level DES Validation The analytical sweep examines bulk-prefill regimes at total token volumes up to 128M, where batch sizes can reach 16,384 at short contexts—realistic for offline, throughput-oriented workloads (e.g., prompt-cache warming, document indexing) but not for interactive chat or agentic serving. In the latter, no system waits for batches that large to accumulate before dispatching: each prefill worker must process requests promptly to meet TTFT SLOs. To quantify how optical scaleup behaves in this complementary small-batch regime, we run a discrete-event simulation (DES) on the same B300 FP4 platform. Six 8-GPU prefill workers operate at max prefill batch size = 64 (∼524K tokens per dispatch at input sequence length (ISL) 8192, roughly the low end of the bulk-sweep range). We deliberately provision a single 24-GPU decode worker to isolate how prefill-side bandwidth improvements propagate to— and interact with—the decode stage. Requests arrive at Poisson(λ=250 req/s) across three in-flight concurrencies (2048, 4096, 6144). Figure 7 plots the p99 operating

points; Appendix E provides the full mean/median/p99 breakdown. Even at these modest per-worker batch sizes, optical scale-up produces compounding TTFT benefits. Shorter per-step prefill time reduces the execution latency for each dispatch; because the six workers are nearly fully utilized at high concurrency (mean batch ≈50 of 64), the prefill queue also drains faster, compressing queue-wait in tandem. Together, these two effects reduce p99 TTFT by 12–20% across all concurrencies (29.4 s → 25.8 s at concurrency 2048; 81.8 s → 65.5 s at concurrency 6144). Inputtoken throughput improves by 1–4% across the same range (13,959 → 14,141 tokens/s/GPU at concurrency 6144), confirming that the prefill workers were already close to their rate ceiling under both configurations. The intentionally single decode worker exposes the downstream consequence. The 24-GPU worker is fully utilized at every concurrency tested (zero scheduler-induced idle time with a non-empty queue). Faster prefill delivers completed requests to the decode queue sooner and in tighter clusters, pushing the steady-state decode batch toward its cap (∼694 → 734 at concurrency 6144). A larger decode batch produces more tokens per iteration but takes proportionally longer per token—the canonical TPOT inflation of a memory-bandwidth-bound worker running at higher load. Consequently, p99 TPOT increases by 77–110% (31.6 ms → 66.3 ms at concurrency 6144). Because TPOT inflation and TTFT improvement act on different phases of end-to-end latency, their interaction at the tail depends on output length: for output sequence length (OSL) 1024, the two effects largely cancel, leaving p99 end-to-end latency approximately neutral (−1 to −5%; 103.4 s → 102.0 s at concurrency 6144). PerGPU output throughput increases only 1–4% (3490 → 3535 tokens/s/GPU at concurrency 6144), confirming that widening the decode stage—not the prefill side—is the lever required to grow total output capacity.

Fig. 7. DeepSeek R1 variant, B300 FP4, disaggregated serving DES — p99 operating points for a 72-GPU layout (six 8-GPU prefill workers, one 24-GPU decode worker), Poisson arrivals at λ=250 req/s, ISL 8192, OSL 1024. Each column is one SLO axis (TTFT, TPOT, end-to-end latency); each datapoint is annotated with its in-flight concurrency (2048, 4096, 6144). Blue: electrical baseline. Coral: optical scale-up (4× per-GPU bandwidth).

Read alongside Table IV, these results clarify the scope of the prefill multipliers: optical scale-up is a TTFT and input-throughput lever whose end-to-end impact is shaped by the decode-side configuration. Operators sizing a disaggregated system should treat the prefill multipliers as available headroom to reduce TTFT, and must codesign decode capacity to determine how much of that headroom converts into end-to-end latency or throughput improvements. VI. Related Work AI infrastructure today combines tightly coupled electrical scale-up fabrics such as NVLink/NVSwitch and Infinity Fabric with scale-out networks such as InfiniBand, but these links face worsening power, reach, and signalintegrity limits as data rates rise [12, 26]. Compute Express Link (CXL) and Universal Chiplet Interconnect Express (UCIe) address adjacent memory-pooling and chipletconnectivity problems, but lack the rack-spanning, GPUto-GPU bandwidth profile the prefill collectives studied here require [27–29]. Silicon photonics and Co-Packaged Optics (CPO) reduce the cost of high-bandwidth datacenter links and have shown benefits for distributed MoE training [7]; this paper extends that analysis to inference, where quantization, continuous batching, disaggregated prefill/decode, and MLA-style KV compression shift the bottleneck among compute, memory, and collectives [9, 11, 16, 20]. VII. Limitations The results are analytical: we extend XLA’s cost model with a projected optical layer not yet validated on deployed photonic hardware, and the optical cases assume ideal 4× scale-up bandwidth without modeling additional link latency, thermal limits, signal-integrity effects, deployment cost, or full Total Cost of Ownership (TCO). The paper is also intentionally prefill-centric: the DES validation in Section V-E shows that decode saturation can dominate end-to-end serving behavior even when optical scale-up reduces TTFT. Consequently, the reported multipliers should be read as interconnect sensitivity results for prefillheavy regimes: they quantify a real TTFT and input-

throughput benefit from optical scale-up, but are not yet a complete serving-system sizing rule until decode capacity is co-designed alongside it. VIII. Future Work Future work should extend this prefill study into full serving-system co-design. Larger optical scale-up pods could support prefix-cache sharing, distributed KV-cache placement, faster prefill-to-decode KV transfer, and topology reconfiguration around failed GPUs; the key open question is how to jointly size optical prefill capacity and memory-bound decode capacity so TTFT reductions also translate into TPOT and end-to-end latency improvements. Running these workloads on optical hardware prototypes, rather than relying only on projected specifications, would help ground these projections. IX. Conclusion This paper presents modeled evidence that siliconphotonics-based optical interconnects can meaningfully benefit AI inference in communication-limited regimes. Across three MoE model scales and four hardware setups, the results suggest optical interconnects improve prefill latency and input-token throughput, particularly at high scale. At 1K–8K tokens, the modeled results indicate 2.1–2.9× prefill improvements in high-batch configurations, where FP4 arithmetic shifts the critical path toward communication. At 128K tokens, rack-limited platforms (B200, B300, Rubin) see ∼4.3–5.8× gains by keeping 288+ GPUs inside a high-bandwidth scale-up pod rather than crossing the scale-out boundary. At 1M tokens, modeled improvements reach 2.2–4.5× on production platforms and up to 8.5× for the speculative R4 configuration. Optical scale-up matters most when it prevents a workload from crossing into a lower-bandwidth scale-out fabric, or when faster per-GPU arithmetic makes communication the limiting component. AI Assistance Disclosure AI tools assisted with editorial and formatting tasks. The authors reviewed and are responsible for all technical claims, data, citations, and conclusions in this submission.

References [1] B. Bamford, “Ai inference costs in 2025: The $255b market’s energy crisis and path to sustainable scaling,” Tensormesh Blog, 2025. [Online]. Available: https://www.tensormesh.ai/blog-posts/ai-inferencecosts-2025-energy-crisis [2] Fullview, “200+ ai statistics & trends for 2025: The ultimate roundup,” https://www.fullview.io/blog/aistatistics, 2025. [3] SemiAnalysis, “The coding assistant breakdown: More Tokens Please,” https : / / newsletter . semianalysis . com / p / the - coding - assistant breakdown - more, 2026, reports that the median input sequence length (P50 ISL) in production agentic coding workloads is ∼96K tokens, driven by tool-call outputs, system prompts, and skill primers; nearly 50% of requests exceed 128K tokens based on ∼432K sampled requests (∼80B tokens). [4] J. L. Hennessy and D. A. Patterson, “A new golden age for computer architecture,” Communications of the ACM, vol. 62, no. 2, pp. 48–60, 2019. [5] R. Pope, S. Douglas, and et al., “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems (MLSys), 2023, discusses the memory-bandwidth vs. compute trade-offs in Transformer decoding. [6] V. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” in Proceedings of Machine Learning and Systems (MLSys), 2023. [Online]. Available: https://arxiv.org/abs/2205.05198 [7] M. Bernadskiy, P. Carson, T. Graham, T. Groves, H. J. Lee, and E. Yeh, “Accelerating frontier moe training with 3d integrated optics,” 2025. [Online]. Available: https://arxiv.org/abs/2510.15893 [8] Q. Li, B. Zhang, L. Ye, Y. Zhang, W. Wu, Y. Sun, L. Ma, and Y. Xie, “Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference,” 2024. [Online]. Available: https://arxiv.org/abs/2412.04964 [9] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’24), 2024. [Online]. Available: https://www.usenix.org/conference/ osdi24/presentation/agrawal [10] H. Li, Q. Mang, R. He, Q. Zhang, H. Mao, X. Chen, H. Zhou, A. Cheung, J. Gonzalez, and I. Stoica, “Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live,” 2025. [Online]. Available: https://arxiv.org/abs/2511.02230 [11] Y. Zhong, S. Liu, and et al., “Distserve:

Disaggregating prefill and decoding for goodputoptimized large language model serving,” Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024, the foundational text on disaggregated inference and prefill-decode interference. [Online]. Available: https://arxiv.org/abs/2401.09670 [12] NVIDIA, “Nvidia rubin cpx accelerates inference performance and efficiency for 1m+ token context workloads,” 2025. [Online]. Available: https://developer.nvidia.com/blog/nvidiarubin - cpx - accelerates - inference - performance - and efficiency-for-1m-token-context-workloads/ [13] G. Inc., “Lpu inference engine: Deterministic performance for llms,” 2024. [Online]. Available: https://groq.com/technology/ [14] C. Systems, “Cerebras wafer-scale engine 3 (wse3): The world’s most powerful ai processor,” 2024. [Online]. Available: https://www.cerebras. net/product-chip/ [15] d Matrix, “d-matrix corsair: In-memory computing for ai inference,” 2024, digital in-memory computing platform optimized for AI inference workloads. [Online]. Available: https://www.d-matrix.ai/ [16] A. Reuther and et al., “Ai and ml accelerator survey and trends,” MIT Lincoln Laboratory, Tech. Rep., 2023. [17] Lightmatter, “Passage: 3d photonics for ai applications,” https://lightmatter.co/products/passage/, 2026. [18] NVIDIA, “NVIDIA GB200 NVL72,” https://www. nvidia.com/en- us/data- center/gb200- nvl72/, 2025, reports rack-scale GB200 NVL72 configuration and NVLink bandwidth. [19] OpenXLA, “Openxla: A unified compiler and runtime for ai accelerators,” 2025. [Online]. Available: https: //github.com/openxla/xla [20] DeepSeek-AI, “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024, describes MultiHead Latent Attention (MLA) architecture for efficient KV cache. [Online]. Available: https://arxiv. org/abs/2412.19437 [21] IEEE Standard for Floating-Point Arithmetic, IEEE Std. 754-2019, 2019, revision of IEEE Std 754-2008; defines binary and decimal floating-point formats including FP8 and FP4 sub-formats adopted by AI accelerators. [22] JAX Team, “Jax llm examples: Deepseek-r1 implementation,” https : / / github . com / jax - ml / jax - llm examples/tree/main/deepseek r1 jax, 2025. [23] SemiAnalysis, “Nvidia: The inference kingdom expands,” https : / / newsletter . semianalysis . com / p / nvidia-the-inference-kingdom-expands, 2025, industry analysis of Vera Rubin-class inference systems; used for projected Rubin and Rubin Ultra hardware parameters. Paywalled newsletter; specs should be

treated as projections. [24] U. Institute, “Reasoning will increase the infrastructure footprint of ai,” https://journal.uptimeinstitute. com / reasoning - will - increase - the - infrastructure footprint-of-ai/, 2025. [25] L. Heim, “Inference compute: Gpt-o1 and ai governance,” https://blog.heim.xyz/inference- compute/, 2024. [26] TokenRing AI, “The speed of light: Silicon photonics and the end of the copper era in AI data centers,” https://markets.financialcontent.com/wral/ article/tokenring-2026-1-2-the-speed-of-light-siliconphotonics-and-the-end-of-the-copper-era-in-ai-datacenters, 2026, company-authored article; no independent peer review. [27] D. Insights, “The ai infrastructure reckoning: Optimizing compute strategy in the age of inference economics,” https://www.deloitte.com/us/en/insights/ topics/technology- management/tech- trends/2026/ ai-infrastructure-compute-strategy.html, 2026. [28] Compute Express Link Consortium, “Compute Express Link Specification 3.0,” https : / / computeexpresslink . org/, 2022, defines CXL 3.0 memory and I/O interconnect capabilities. [29] UCIe Consortium, “Universal Chiplet Interconnect Express Specification 2.0,” https://www.uciexpress. org/specification, 2024. [30] NVIDIA Corporation, NVIDIA Blackwell Architecture Whitepaper: Reimagining the GPU for Generative AI, 2024, accessed: 2024-05-20. Explains 5th Gen Tensor Cores and Tensor Memory (Tmem). [Online]. Available: https: //www.nvidia.com/en-us/data-center/blackwell/ [31] ——, Parallel Thread Execution ISA Version 8.5, 2024, details the tcgen05.mma instruction set for sm 100. [Online]. Available: https://docs.nvidia. com/cuda/parallel-thread-execution/index.html Appendix A DeepSeek R1 Model Configuration We use a slightly modified version of the DeepSeekR1 architecture to ensure divisibility constraints are met across all hardware mesh dimensions [22]. The embedding dimension is set to 8064, the number of attention heads to 144, the number of routed experts to 288, and the vocabulary size to 145,440. These values ensure divisibility by 8, 16, 24, 48, and 72 (the device counts used in the sweep). These modifications result in a model approximately 14% larger than the production DeepSeek-R1 in terms of parameter count, but communication patterns and scaling behavior remain representative. Listing 1. DeepSeek R1 Model Configuration class Config : embed : int = 8064 q_lora_rank : int = 1536 kv_lora_rank : int = 512

num_heads : int = 144 q k _ n o p e _ h e ad _ d i m : int = 128 q k _ r o p e _ h e ad _ d i m : int = 64 v_head_dim : int = 128 vocab_size : int = 145440 num_layers : int = 61 n _ r o u t e d _ e xp e r t s : int = 288 # d i v i s i b l e by 72 n u m _ e x p e r t s _ p e r _ t o k : int = 8 n_group : int = 8 topk_group : int = 4 moe_ffw_size : int = 2048 ffw_size : int = 18432 first_k_dense : int = 3 n _ s h a r e d _ e xp e r t s : int = 1

Appendix B Parallelism Strategy Our modeling supports 3D parallelism: Tensor (T), Expert (E), and Sequence (S) parallelism. The physical device mesh uses three axes: • Axis x: splits the sequence dimension across racks (context parallelism). • Axis y: splits experts and attention heads within the scale-up pod (expert + tensor parallelism). This is the most bandwidth-intensive axis. • Axis z: splits the FFW hidden dimension of dense and expert layers (tensor parallelism). For DeepSeek-R1 with 288 devices, the preferred mesh is 2×144×1 (x×y×z): axis y receives the largest allocation because both expert routing (all-to-all) and attention head partitioning (all-gather/reduce-scatter) benefit most from maximizing the y-dimension. Table V lists the selected mesh shape for each device count in the prefill sweep. Each configuration is the bestperforming mesh found after evaluating hundreds of candidate shapes; the selection criterion is minimum overlapped prefill latency at the corresponding context length. Decode meshes are included for reference. TABLE V Selected prefill mesh shapes. Each entry is the best mesh out of hundreds of candidates evaluated for minimum overlapped prefill latency. Mesh dimensions are expressed as x×y×z, matching the jax.make mesh axis ordering ("x","y","z") in the JAX LLM examples implementation [22]: x = sequence/context-parallel, y = expert+tensor-parallel (most bandwidth-intensive), z = FFW tensor-parallel. Phase

Devices

Mesh (x×y×z)

Prefill

8 16 24 48 72 144 288 576 1152

1×8×1 1 × 16 × 1 1 × 24 × 1 1 × 48 × 1 1 × 72 × 1 2 × 72 × 1 4 × 72 × 1 8 × 72 × 1 16 × 72 × 1

Appendix C GB200 Specifications We derive hardware specifications from the OpenXLA project [19]. Memory capacity (192 GB HBM3e per GPU) and memory bandwidth (8 TB/s) are used unmodified. For compute, we use the tcgen05.mma tensor core instruction in 2-SM coordination mode, which supports 256 × 256 × 16 shapes for BFloat16 (BF16) precision [30, 31]. We apply precision-dependent multipliers: FP8 yields 2× the BF16 rate; FP4 yields 4× the BF16 rate. Appendix D Full Precision Sweep Heatmap (FP4 and FP8) Table VI reproduces the sweep best multiplier heatmap from Section V-D for both FP4 and FP8 quantization precisions. Within each model–hardware block, FP8 is listed first, followed by FP4. Column definitions are identical to Table IV. TABLE VI Full precision sweep best multiplier heatmap (FP8 and FP4). Column layout identical to Table IV. Within each model–hardware block, FP8 is listed first and FP4 second. Cell color scales from min to max among these heatmap cells only. Model Hardware Quant

Best multiplier (optical vs. no-optical) seq len 1K 1K 1K 1K 1K 1K 8K 8K 8K 8K 8K 8K 128K 128K 128K 128K 128K 128K 128K 128K 128K tokens 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 16M 2M 8M 32M dev 16 16 16 72 72 72 16 16 16 72 72 72 16 16 16 72 72 72 288 288 288 2048 8192 16384 2048 8192 16384 256 1024 2048 256 1024 2048 16 64 128 16 64 128 16 64 256 batch

mini

r1

next

1M 2M 16 2

1M 8M 16 8

1M 16M 16 16

1M 2M 72 2

1M 8M 72 8

1M 1M 1M 1M 1M 1M 1M 1M 1M 1M 16M 2M 8M 32M 8M 32M 128M 8M 32M 128M 72 288 288 288 576 576 576 1152 1152 1152 16 2 8 32 8 32 64 8 32 64

B200

FP8 FP4

1.62 1.84 2.12 2.23

1.86 1.80 1.99 2.24 2.33 2.41

2.01 1.40 1.49 1.50 1.67 1.82 1.83 1.06 1.06 1.06 1.20 1.22 1.22 3.46 3.98 4.09 1.01 1.01 1.01 1.03 1.03 1.03 1.47 1.50 1.50 2.08 2.11 2.11 3.20 3.28 3.29 2.41 1.59 1.61 1.62 2.04 2.09 2.09 1.06 1.07 1.07 1.23 1.23 1.23 3.98 4.27 4.30 1.01 1.01 1.01 1.03 1.03 1.03 1.50 1.51 1.51 2.12 2.13 2.13 3.29 3.32 3.32

B300

FP8 FP4

1.98 2.12 2.34 2.41

2.13 2.16 2.27 2.41 2.59 2.64

2.28 1.55 1.59 1.60 1.93 2.01 2.02 1.07 1.07 1.07 1.23 1.24 1.24 2.52 2.74 2.77 1.01 1.01 1.01 1.04 1.04 1.04 1.27 1.28 1.28 1.58 1.59 1.59 2.17 2.20 2.20 2.64 1.65 1.67 1.67 2.21 2.24 2.24 1.07 1.07 1.07 1.25 1.25 1.25 2.75 2.84 2.85 1.01 1.01 1.01 1.04 1.04 1.04 1.28 1.28 1.28 1.60 1.60 1.60 2.20 2.21 2.21

Rubin

FP8 FP4

2.24 2.41 2.48 2.62

2.43 2.43 2.56 2.62 2.71 2.80

2.57 1.73 1.78 1.79 2.19 2.28 2.29 1.09 1.09 1.09 1.32 1.32 1.33 4.82 5.35 5.42 1.01 1.01 1.01 1.05 1.05 1.05 1.70 1.72 1.72 2.57 2.59 2.59 4.18 4.25 4.26 2.81 1.81 1.85 1.85 2.38 2.45 2.45 1.10 1.10 1.10 1.33 1.34 1.34 5.16 5.51 5.55 1.01 1.01 1.01 1.05 1.05 1.05 1.71 1.72 1.73 2.59 2.60 2.60 4.24 4.28 4.28

R4

FP8 FP4

2.79 2.94 3.00 3.10

2.95 2.90 3.00 3.11 3.11 3.18

3.01 2.25 2.33 2.34 2.74 2.83 2.84 1.21 1.21 1.21 1.64 1.65 1.65 1.86 1.95 1.96 1.03 1.03 1.03 1.11 1.11 1.11 1.19 1.19 1.19 1.28 1.28 1.28 7.95 8.08 8.10 3.19 2.36 2.41 2.42 2.92 2.98 2.99 1.21 1.21 1.22 1.66 1.67 1.67 1.91 1.98 1.99 1.03 1.03 1.03 1.11 1.11 1.11 1.19 1.19 1.19 1.28 1.28 1.28 8.04 8.14 8.15

B200

FP8 FP4

1.63 1.84 2.11 2.21

1.85 1.78 1.94 2.22 2.28 2.35

1.97 1.41 1.48 1.50 1.67 1.77 1.81 1.06 1.06 1.06 1.20 1.22 1.22 3.57 4.11 4.22 1.01 1.01 1.01 1.03 1.03 1.03 1.50 1.52 1.53 2.14 2.17 2.17 3.32 3.40 3.41 2.36 1.59 1.62 1.62 2.01 2.06 2.06 1.07 1.07 1.07 1.23 1.23 1.23 4.12 4.42 4.46 1.01 1.01 1.01 1.03 1.03 1.03 1.53 1.53 1.54 2.18 2.19 2.19 3.41 3.45 3.45

B300

FP8 FP4

1.98 2.10 2.32 2.38

2.12 2.12 2.21 2.38 2.53 2.58

2.22 1.56 1.58 1.60 1.91 1.95 1.99 1.07 1.07 1.07 1.23 1.23 1.23 2.59 2.82 2.85 1.01 1.01 1.01 1.03 1.04 1.04 1.28 1.29 1.29 1.62 1.63 1.63 2.24 2.27 2.27 2.58 1.65 1.67 1.67 2.18 2.21 2.21 1.07 1.07 1.07 1.24 1.25 1.25 2.83 2.93 2.93 1.01 1.01 1.01 1.04 1.04 1.04 1.29 1.30 1.30 1.63 1.63 1.63 2.27 2.28 2.28

Rubin

FP8 FP4

2.23 2.38 2.45 2.58

2.40 2.37 2.49 2.58 2.64 2.72

2.50 1.74 1.76 1.79 2.16 2.21 2.25 1.09 1.10 1.10 1.32 1.32 1.32 4.99 5.54 5.62 1.01 1.01 1.01 1.05 1.05 1.05 1.74 1.76 1.76 2.66 2.68 2.68 4.35 4.42 4.43 2.73 1.81 1.85 1.85 2.34 2.40 2.40 1.10 1.10 1.10 1.33 1.33 1.33 5.34 5.71 5.75 1.01 1.01 1.01 1.05 1.05 1.05 1.75 1.76 1.76 2.68 2.69 2.69 4.41 4.45 4.45

R4

FP8 FP4

2.74 2.88 2.95 3.05

2.90 2.83 2.94 3.06 3.05 3.13

2.95 2.24 2.28 2.32 2.68 2.73 2.77 1.21 1.21 1.21 1.63 1.64 1.65 1.88 1.98 2.00 1.03 1.03 1.03 1.11 1.11 1.11 1.20 1.20 1.20 1.30 1.30 1.31 8.22 8.37 8.38 3.13 2.34 2.39 2.40 2.86 2.92 2.92 1.21 1.22 1.22 1.66 1.67 1.67 1.94 2.01 2.02 1.03 1.03 1.03 1.11 1.11 1.11 1.20 1.20 1.20 1.31 1.31 1.31 8.33 8.43 8.44

B200

FP8 FP4

1.73 1.79 2.18 2.23

1.80 1.91 1.96 2.23 2.33 2.36

1.97 1.45 1.47 1.48 1.78 1.81 1.82 1.06 1.06 1.06 1.22 1.22 1.22 3.91 4.27 4.30 1.01 1.01 1.01 1.03 1.03 1.03 1.53 1.54 1.54 2.20 2.21 2.21 3.42 3.47 3.47 2.36 1.62 1.63 1.63 2.05 2.07 2.08 1.07 1.07 1.07 1.24 1.24 1.24 4.38 4.52 4.53 1.01 1.01 1.01 1.03 1.03 1.03 1.55 1.55 1.55 2.22 2.23 2.23 3.50 3.51 3.51

B300

FP8 FP4

2.08 2.12 2.38 2.40

2.13 2.19 2.22 2.40 2.57 2.59

2.23 1.59 1.61 1.61 1.97 1.99 2.00 1.07 1.07 1.07 1.24 1.24 1.24 2.77 2.89 2.90 1.01 1.01 1.01 1.04 1.04 1.04 1.30 1.30 1.30 1.65 1.65 1.65 2.30 2.32 2.32 2.59 1.68 1.68 1.68 2.21 2.23 2.23 1.07 1.07 1.07 1.25 1.25 1.25 2.95 2.99 2.99 1.01 1.01 1.01 1.04 1.04 1.04 1.31 1.31 1.31 1.66 1.66 1.66 2.33 2.33 2.33

Rubin

FP8 FP4

2.34 2.40 2.53 2.58

2.40 2.45 2.50 2.59 2.69 2.73

2.51 1.78 1.80 1.80 2.22 2.25 2.26 1.10 1.10 1.10 1.33 1.33 1.33 5.40 5.67 5.70 1.01 1.01 1.01 1.05 1.05 1.05 1.77 1.78 1.78 2.72 2.73 2.73 4.47 4.51 4.51 2.73 1.85 1.86 1.86 2.38 2.41 2.41 1.10 1.10 1.10 1.34 1.34 1.34 5.67 5.82 5.84 1.01 1.01 1.01 1.05 1.05 1.05 1.79 1.79 1.79 2.74 2.74 2.74 4.51 4.53 4.53

R4

FP8 FP4

2.83 2.89 3.01 3.05

2.89 2.89 2.94 3.05 3.10 3.13

2.95 2.29 2.32 2.33 2.73 2.76 2.77 1.22 1.22 1.22 1.65 1.66 1.66 2.02 2.06 2.07 1.03 1.03 1.03 1.12 1.12 1.12 1.22 1.22 1.22 1.35 1.35 1.35 8.35 8.41 8.41 3.13 2.38 2.40 2.40 2.89 2.91 2.92 1.22 1.22 1.22 1.67 1.68 1.68 2.06 2.09 2.10 1.03 1.03 1.03 1.12 1.12 1.12 1.22 1.22 1.22 1.35 1.35 1.35 8.42 8.46 8.47

Appendix E DES Latency Distribution: Mean, Median, and p99 Full percentile breakdown (mean, median, p99) for the same DeepSeek R1, B300 FP4 disaggregated serving scenario as Figure 7: six 8-GPU prefill workers, one 24-GPU decode worker, Poisson(λ=250 req/s), ISL 8192, OSL 1024. Rows correspond to percentile level; columns to TTFT, TPOT, and end-to-end latency. Each datapoint is annotated with its in-flight concurrency (2048, 4096, 6144). Blue: electrical baseline; coral: optical (4× per-GPU bandwidth).

Fig. 8. DES latency distribution — DeepSeek R1, B300 FP4, mean / median / p99.

Related documents

• Scaling Inference Prefill with High-Radix Photonic Interconnects#822245
• Spectral purity transfer between optical wavelengths at the $10^{-18}$ level#170448
• Limits on compositeness from lepton and photon channels at LEP#210596
• An integrated photonics engine for unsupervised correlation detection.#165864
• Germanium‐Based Mid‐Infrared Integrated Photonics#195776