An Optical KV Cache Network for Disaggregated LLM Serving

An Optical KV Cache Network for Disaggregated LLM Serving Seongwon Yoon and Shimeng Yu

arXiv:2610.18444v1 [cs.NI] 13 Oct 2026

School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA gabrielyoon@gatech.edu

Abstract—Disaggregated LLM serving moves each request’s KV cache from a prefill GPU to a decode GPU before the response begins, and a coding-agent turn often moves the KV of its whole long context. Behind an oversubscribed packet fabric this transfer adds to time to first token (TTFT). An optical circuit switch (OCS) can give every transfer a full-rate circuit for less than an oversubscribed packet fabric costs, but a mechanical OCS needs tens of milliseconds to reconfigure. We hide this delay inside the prefill. A reservation scheduler commits each circuit while the prefill is still computing, so the circuit is ready when the KV is, and each GPU reaches the OCS over its own wavelengthmultiplexed (WDM) optics on one fiber pair. In simulations of GPU clusters on real serving traces with prefill times measured on H100 GPUs, this optical KV cache network matches 400G packet fabrics with adaptive routing at 0.55 to 0.78 times the cost of a 2:1 fabric and tolerates the 50 ms reconfiguration delay of a 384-port mechanical OCS.

I. Introduction Prefill/decode (P/D) disaggregation runs the two phases of LLM serving on separate GPU pools [1], [2], [3], so every request’s KV cache crosses the network and adds to TTFT unless the transfer ends within the prefill. In our analysis of one P/D application in Alibaba’s production trace [4], the hourly mean KV transfer time is a median 1.10 times the prefill time over one week. Deployed P/D systems choose the decode instance by load [3], [5], [6], so a coding-agent turn [7] that appends a tool result to a long context often moves the KV of the whole context behind a prefill of only the new tokens. A non-blocking 400G fabric mostly hides this move, but operators oversubscribe the upper tiers [8], [9], [10], and at half of 400G per GPU the median migrated turn of Llama3.1-70B on H100 GPUs adds 67 ms to TTFT. A non-blocking packet fabric costs 1.27 to 1.31 times a 2:1 fabric, and packet schedulers [11], [8] reduce contention on the same links. An OCS port draws far less power than a packet switch port and passes light without transceivers [12], [13], so a circuit per transfer can give every GPU its full link rate for less than an oversubscribed fabric, yet a mechanical OCS, steered by MEMS mirrors or piezoelectric actuators, needs 10 to 50 ms to reconfigure [13], [14], [15], which we hide inside the prefill. Unlike OCS fabrics that reconfigure per training job, phase or workload mode [13], [12], [16], [17], [18], [19], our circuits follow single KV transfers. Our contributions are an optical KV cache network with GPU-side WDM optics and the requirements it sets on them, one fiber pair per 400G link, up to 50 ms of reconfiguration and 2 dB of link margin (Section III-A, Table I), a reservation scheduler that commits

the prefix circuit at prefill start and the new-KV circuit at the last prefill chunk and reuses standing circuits (Section III-B), and simulations of H100 clusters with measured prefill times on real serving traces (Section IV). II. Background A. Prefill, decode, and KV cache transfer Prefill reads the whole prompt at once and decode generates one token at a time, both through the key and value vectors stored per token and layer, the KV cache. For Llama-3.1-70B in BF16 it holds 320 KiB per token [20], so a 64K-token context occupies about 21 GB. Prefill is compute bound and decode is memory bound, so P/D disaggregation places them on separate pools [1], [2], as in Mooncake, NVIDIA Dynamo and DeepSeek [3], [6], [5]. Mooncake streams the KV cache layer by layer during prefill, which hides the transfer while it ends within the prefill, and the vLLM NIXL connector pulls it after prefill [21], which adds the whole transfer to TTFT because decode emits the leading token. B. Multi-turn agent sessions and KV cache migration Coding agents append a short input, such as a tool result, to a long shared context every turn. Among the steps that fit the 128K window of Llama-3.1-70B in a trace of 4,300 codingagent sessions from 43 developers [7], the median context is 66.7K tokens and the median new input 1.1K tokens, against 88.8K and 1.6K for main-agent turns in the public AgentX Claude Code sessions [22] and a 68K-token median prompt in GitHub Copilot [23]. A turn migrates when the prefill side still holds its prefix [24], [25], [26] but the chosen decode instance does not, and its full KV cache then crosses the network before decode can respond. Mooncake and DeepSeek send the full KV to a decode instance chosen by load every turn, Dynamo routes decode without regard to prefix reuse [3], [5], [6], and in llm-d runs on coding-agent traces about half of the decodeside KV arrived from prefill [27]. In our analysis of Alibaba’s production P/D telemetry [4], the least-utilized decode instance of a group changes in 42% of consecutive 5-minute windows (16.5% by more than one SM point). With session affinity 7.6% of turns still miss decode memory in our modeled replay, and a KV store still sends the KV over the fabric [5]. C. Packet and circuit switching in the datacenter A GPU has three network paths. NVLink joins the GPUs of a server, a 400G RDMA NIC per GPU joins GPUs across

spine

turn 1

spine

turn 2 tool call

agent session other requests prefill start

TTFT leaf

leaf

leaf

leaf

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

prefill rack

prefill rack

decode rack

decode rack

prefill

P

P P P P

decode B

decode A

D D

P

prefill full KV migration over the OCS D D D D

D D D D D D D D D D D D D D

prefill, 1.1K new tokens exposed 67 ms

half of 400G

KV of 66.7K tokens

TTFT

reconfiguration 25 ms OCS circuit

KV

hidden

time

OCS (KV network)

0

(b) agent turn migrating to decode GPU B

(a) EPS for requests, OCS for KV

100

200

(c) one migrated turn on four H100 GPUs (ms)

Fig. 1. Optical KV cache network. (a) Requests, output tokens and control keep the host NICs and their leaf-spine EPS, while the RDMA NIC of every GPU drives WDM optics into an OCS instead of a packet leaf. (b) After a tool call decode GPU A is busy with other requests, so turn 2 lands on decode GPU B and its full KV cache crosses the OCS on a circuit committed during prefill. (c) Median migrated turn of Llama-3.1-70B on four H100 GPUs, modeled with the measured prefill time, where one circuit committed at prefill start reconfigures in 25 ms and ends inside the 176 ms prefill.

Algorithm 1 Reservation with last-chunk commit per request

last chunk P1 prefill P2 prefill

R1 prefill last chunk R2

D1 port

booked

D2 port

R1 head

D3 port

frees earliest

ends after the predicted end

R2 tail extends

R1 tail reconfiguration

booked

time Fig. 2. Reservation calendar with last-chunk commit. The head circuit of R1 is committed at prefill start on the decode port that frees earliest (D2) and its tail at the last chunk, so the reconfiguration delay (grey) overlaps that chunk, and the late tail of R2 extends into free port time.

servers [10], [9], and host NICs carry requests, storage and management [9]. Both NIC paths cross leaf and spine fat trees of electrical packet switches (EPS), whose upper tiers are often oversubscribed [8], [9], [10] and which spread flows by hashing (ECMP), letting large transfers collide [28], or by adaptive routing [29] under RDMA congestion control [30]. An OCS steers light from an input fiber to an output fiber without converting it to an electrical signal [31], [32], [33], as in Google’s TPU and datacenter networks [18], [34], [35]. It does not inspect packets, so one port serves any line rate at 0.39 W against 39.4 W for a 400G packet switch port [12], and at $520 against $583 to $1,090 [12], [13]. In exchange a circuit joins one pair of ports at a time, and changing it takes up to 25 ms in 192-port and 50 ms in 384-port mechanical switches [13], [14], [15] and about a microsecond in silicon photonic switches [36]. III. Optical KV Cache Network A. Architecture Requests, output tokens and control keep the host NICs and their leaf-spine EPS, while the 400G RDMA NIC of each GPU

Admission 1: 𝑇ˆ ← prefill time predicted by the cost model 2: 𝐵ℎ , 𝐵𝑡 ← KV bytes of the prefix and of the new tokens 3: split into head 𝐵ℎ and tail 𝐵𝑡 only if ( 𝐵ℎ + 𝐵𝑡 )/𝑅 + 𝜏 ≤ 0.8 𝑇ˆ Prefill start at 𝑡0 4: 𝑑 ← decode port free earliest, preferring a standing circuit 5: 𝜏ℎ ← 0 if the circuit to 𝑑 stands else 𝜏 6: book head of length 𝜏ℎ + 𝐵ℎ /𝑅 in the earliest gap after 𝑡0 free on both ports Last prefill chunk at 𝑡1 7: 𝐸ˆ ← prefill end re-predicted from observed progress 8: book tail on 𝑑 to end at 𝐸ˆ + 𝑇ℓ + 0.1( 𝐸ˆ − 𝑡1 ) Prefill end at 𝐸 9: if the tail circuit is not up then move the tail to the earliest free gap 10: if a burst outlasts its window then extend it into free port time, displacing bookings without ready data, else cut it and re-book the rest 11: keep the circuit up for reuse

connects to an OCS instead of a packet leaf and carries the KV cache of every migrated turn (Fig. 1(a), (b)). Each prefill and decode GPU drives one 400G link of pluggable or co-packaged WDM optics [37] into the OCS, 45 GB/s at an assumed 90% efficiency as for every fabric. A fiber OCS switches fibers rather than wavelengths, so the GPU-side optics set its size. A 400GBASE-FR4 link carries four wavelengths on one fiber pair, while a parallel-fiber 400GBASE-DR4 link occupies four pairs [38] and needs four times the switch ports. Eight 384-port planes, each joining one GPU position of every 8-GPU node, reach 3,072 GPUs in one stage [15], so KV moves between equal GPU positions. B. Reservation scheduler The controller keeps one calendar per instance with the booked windows of its OCS ports and the peer they point to. A window holds the reconfiguration delay, skipped when the circuit already stands toward that peer, and the transfer, and a booking takes the earliest gap free on both the prefill and the decode port, so windows never overlap. Algorithm 1 lists the four events (Fig. 2), with 𝑅 the circuit rate, 𝜏 the

19%

20

10

0

31%

10% 5%

400G 1:1

400G at 65%

400G 2:1

400G 3:1

(a) KV fabric per H100 GPU

88 3:1

86

2:1

84

800G 2:1 1:1 800G 2:1

2:1 3:1

82

OCS 400G EPS, adaptive routing EPS, ECMP

80 78

KV fabric cost per GPU (k$)

30

16

no network limit

agentic SLO attainment (%)

KV share of mean TTFT (%)

migrated turn, BF16 KV migrated turn, FP8 KV 10% of TTFT

12

8

4

6

8

(b) KV fabric cost per GPU (k$)

10

0.55 to 0.64x

0.56 to 0.65x

0.74 to 1.02x

256

3K

8K

4

0 2

EPS 1:1 EPS 2:1 OCS, WDM OCS, DR4

(c) GPUs in the KV domain

Fig. 3. (a) KV share of mean TTFT with each GPU at its share of 400G, Llama-3.1-70B on four H100 GPUs (modeled, layer-wise), chat turns below 0.1%. (b) Agentic SLO attainment against cost per GPU at 256 H100 GPUs (simulated, pull), the OCS at one or two ports per fiber pair. (c) KV fabric cost per GPU at 400G with the prices of [13], whiskers up to two ports per fiber pair, ratios to EPS 2:1 above each group, three stages at 8K and DR4 on eight ports.

reconfiguration delay and 𝑇ℓ one prefill layer. The head carries prefix KV that prefix caching already holds in the prefill GPU and that this prefill only reads, so it does not depend on the tail, which carries the new tokens’ KV as their layers finish. Committing the head at prefill start makes its start exact, and committing the tail when the last chunk of up to 2,048 tokens starts leaves only the error of the remaining prefill, while a booking at admission would absorb the error of every prefill queued ahead. Preferring a standing circuit (line 4) makes the decode choice circuit-aware and favors reuse. IV. Evaluation A. Setup Design space. A request’s KV demand per GPU 𝐷 is the KV bytes one GPU sends over the prefill time, and 𝑆 is the effective KV bandwidth per GPU, so layer-wise streaming exposes max(0, 1 − 𝑆/𝐷) of TTFT. Prefill times use a cost model fitted to 33 vLLM prefill runs of Llama-3.1-70B that we measured on four H100 GPUs (median error 4.6%), without scaling. We use H100 GPUs because they are the most advanced GPU system available to us, and the method applies in principle to newer GPUs by refitting the cost model. Simulation. A flow-level event-driven simulator models 128 and 256 H100 GPUs as slices of a larger fabric with 32-GPU racks, prefill and decode in separate racks and each decode instance as a KV sink without decode compute, so the results cover the KV fabric and TTFT. As in prior P/D studies [1], [2], [11], request traces drive the simulator. A TraceLab tenant [7], whose decode instances keep no session KV between turns as in Mooncake and DeepSeek, runs beside Azure 2024 chat and code [45], [46] and Mooncake [3] tenants, all on the measured Llama-3.1-70B configuration on four H100 GPUs. The base point gives TraceLab 25% of the GPUs at 60% prefill load, and the grid adds a 50% share and 75% load, in layer-wise and pull transfer. Results average five paired 330 s trace windows, with prediction errors drawn from the measured runs. SLO attainment is the share of requests with TTFT within five times the tenant’s median unloaded prefill time. Without the OCS the KV crosses a two-tier EPS that runs 400G or 800G per GPU at

TABLE I Optical KV cache network requirements and link budget Requirement Value KV bandwidth per GPU 33 GB/s for 10% of mean TTFT Reconfiguration delay 50 ms, every tenant within 5% at 60% load OCS unit 384 ports, 50 ms, 8 planes reach 3,072 GPUs WDM, one fiber pair per 400G link at 45 GB/s GPU-side optics Optics DR4 (parallel) WDM (FR4) OCS ports per link 4 or 8 1 or 2 0.77 to 1.36 0.55 to 0.78 Cost over 2:1 EPS Break-even port price $305 to $842 $1,050 to $3,058 Link budget DR4 WDM one 400G link over 500 m of fiber, transceiver to OCS to transceiver Launch OMA (dBm) −0.8 to +4.2 −0.3 to +3.7 max( −3.9, SECQ −5.3) max( −4.6, SECQ −6.0) Rx sensitivity (dBm) none 2.5 typ, 3.3 max each Mux and demux (dB) OCS (dB) 1.5 typ, 2.7 max Four connectors (dB) 2.0 typ, 3.0 max Fiber at 0.5 dB/km (dB) 0.25 3.75 typ, 5.95 worst Path loss (dB) Loss allocation (dB) 3 4 2.35 typ, 0.15 dB worst 2.55 typ, 0.35 dB worst Margin at TDECQ 3.4 dB ≤ 1.5 dB ≤ 1.7 dB TDECQ for 2 dB margin TDECQ for 2 dB, spliced ≤ 2.4 dB ≤ 2.6 dB Sources are the exposure model, Fig. 4(c), the cost model [13], [12], data sheets [15], [39], [40], [41] and specifications [38], [42], [43], [44]. Margins use the maximum launch and the 1550 nm OCS loss, with mux and demux in the module.

1:1, 2:1 or 3:1 (as in MixNet [13], simulated as 2.91:1 with 11 uplinks per 32 GPUs) with ECMP, with adaptive routing and fair sharing as in RDMA fabrics [29], [30], or with adaptive routing and global SRPT as a strong packet reference [47], [48], and no network limit gives the ceiling. The OCS runs last-chunk commit at a 25 ms delay, which 50 ms changes by at most 0.15 percentage points of SLO attainment at 60% load, and a baseline without reservation holds a circuit once any KV is ready, with the decode chosen by load. Cost. Both designs keep the same host NIC network, so for new clusters that serve each model within one node, as here, the comparison covers the network behind the 400G RDMA NIC of each GPU, built from packet switches or from an OCS. Switch ports, NICs and 400G modules use published prices [13], [12], with an FR4 module at 1.47 times a 400G module as in reseller lists, the EPS the fewest tiers of 51.2T switches, and the OCS one or two ports per fiber pair [12], [39] and a 100G control network.

75 50 25

3.6

OCS, commit at admission

3.4 OCS, lastchunk commit

3.2

EPS, SRPT or ECMP

3.0

0 0

25

50

75

100

(a) delay injected on two H100 (ms)

agentic SLO attainment (%)

100

agentic P99 TTFT (s)

added TTFT (ms)

measured set up after prefill at admission, 34 ms prefill at admission, 65 ms prefill at admission, 130 ms prefill

90

SiPh switch

192-port OCS

384-port OCS

EPS 1:1, SRPT reservation, reuse reservation only no reservation (overload)

80

70

60 0

5

10

15

20

(b) median prediction error (%)

10 µs

100 µs

1 ms

10 ms

100 ms

(c) OCS reconfiguration delay

Fig. 4. (a) TTFT added by a circuit delay injected after prefill or at admission on two H100 GPUs. (b) Agentic P99 TTFT against median prediction error at 1 ms and (c) agentic SLO attainment against reconfiguration delay, simulated at the 128-GPU base point, layer-wise, where no network limit reaches 85%.

B. Performance and cost Migrated turns of Llama-3.1-70B demand a median 28.6 GB/s per H100 GPU, within a 400G link at 45 GB/s, so a non-blocking fabric exposes 5% of mean TTFT, or 10% at the 32.5 GB/s NCCL rate we measured over 400G InfiniBand, and half or a third of 400G per GPU exposes 19% and 31% (Fig. 3(a)). At these rates AgentX turns give 12% and 20%, and FP8 KV cuts the TraceLab shares to 5% and 11%. Under load at 128 GPUs the OCS stays within 0.4 points of the agentic SLO attainment of EPS 1:1 with SRPT at every grid point and mode, with P99 TTFT within 1.8%. Adaptive routing with fair sharing stays within 0.5 points of the OCS even at 3:1, so the loss behind oversubscription comes from ECMP collisions of migrated turns. In pull transfer the OCS gains 3.4 to 6.6 points over 2:1 ECMP and 4.6 to 10.7 points over 3:1 ECMP, and 0.2 to 3.2 points in layer-wise transfer, where it hides 71 to 74% of migrated turns within 10% of their prefill time against 50 to 58% and 43 to 53%. At 256 GPUs the pull gains over ECMP are 2.7 to 6.2 points (Fig. 3(b)). The gain follows the share of migrated turns, 1.3 to 1.8 and 1.8 to 2.7 points in pull when half the turns migrate, as half the decode-side KV does in llm-d [27], and within 0.6 points for every fabric under session affinity (7.6%), where the OCS keeps its cost advantage. With a NIC and a WDM pluggable per link at two sets of published prices [13], [12], the OCS costs 0.55 to 0.78 times a 2:1 EPS and 0.61 to 0.85 times a 3:1 EPS from 256 to 3,072 GPUs, the reach of one OCS stage (Fig. 3(c) shows one price set). Parallel-fiber DR4 costs 0.77 to 1.36 times a 2:1 EPS in one stage (Table I), so WDM optics pay off best. C. What the optical layer must provide Overlap. On our two-H100 testbed (Llama-3.1-8B over NVLink, Fig. 4(a)), an injected delay started at admission stays hidden while it is shorter than the prefill, and one started after prefill costs 0.95 to 1.12 times its length. Prediction. With last-chunk commit agentic P99 TTFT rises 1.5% at the measured 4.6% median error and 3.6% at 10%, while admission commit rises 3.5% and 9.6% (Fig. 4(b)), so P99 stays within 5% up to 11.6% error against 5.9%.

Reconfiguration delay. Without reservation, layer-wise transfers hold decode ports through every prefill, so with the decode chosen by load the Mooncake tenant overloads in every trace window even with a 10 𝜇s switch (agentic SLO attainment 71%), while reservation with the same decode choice holds 85% at 25 ms. Circuit-aware decode choice keeps the same decode for 92% of consecutive agentic requests, with the busiest decode at 1.07 times the mean, so with reuse the SLO attainment stays at 85% up to 100 ms (Fig. 4(c)). Without reuse all tenants keep P99 and SLO within 5% of EPS 1:1 with SRPT up to 50 ms at 60% load, so a mechanical OCS suffices. Link budget. The worst path through four connectors and a 384-port OCS at 2.7 dB [15] loses 5.95 dB, above the 3 and 4 dB channel allocations of DR4 and FR4, so 2 dB of margin needs a launch near the maximum and TDECQ up to 1.5 dB with DR4 and 1.7 dB with WDM, or spliced OCS fibers with typical loss for any compliant TDECQ at the maximum launch. V. Related Work Optical datacenter networks reconfigure for aggregate traffic [31], [32] or training jobs [49], [17], or rotate through traffic-oblivious schedules [50], [51], and KV compression [52], [53] loses to raw transfer above 50 to 110 Gb/s [54]. MoSE reallocates OCS ports per workload mode without modeling reconfiguration delay [16], xSwitch pairs a portreduced EPS with an OCS for scale-up [55], a photonic CXL appliance replaces the P/D hop with a memory pool [56], and Opus names KV cache pipelines as an unevaluated target [12]. VI. Conclusion Carrying the KV cache of each migrated turn over an optical circuit committed during prefill hides the delay of a mechanical OCS and matches 400G packet fabrics with adaptive routing at 0.55 to 0.78 times the cost of a 2:1 fabric. Against the same cluster with a 2:1 or 3:1 packet fabric and ECMP, it raises agentic SLO attainment by 2.7 to 6.6 and 4.1 to 10.7 points in pull transfer. References [1] Y. Zhong et al., “DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proc. OSDI, 2024.

[2] P. Patel et al., “Splitwise: efficient generative LLM inference using phase splitting,” in Proc. ISCA, 2024. [3] R. Qin et al., “Mooncake: trading more storage for less computation – a KVCache-centric architecture for serving LLM chatbot,” in Proc. FAST, 2025. [4] Y. Zheng, L. Yang, Y. Yu, G. Yang, L. Zhang, and M. Yu, “Model-asa-Service in the wild: characterizing production LLM inference clusters at scale,” in Proc. ACM SIGOPS ATC, 2026, trace cluster-trace-v2026maas. [5] Y. Wu et al., “DualPath: accelerating agentic LLM inference by harvesting disaggregated KV-cache storage I/O,” in Proc. ACM SIGCOMM, 2026. [6] NVIDIA, “Disaggregated serving,” Dynamo router documentation, v1.2.1, 2026. [7] K. Zhu et al., “TraceLab: characterizing coding agent workloads for LLM serving,” arXiv:2606.30560, 2026. [8] H.-C. Lin, T.-W. Hsu, C.-E. Ho, and A. Saeed, “ARK: avoiding routing collisions for KV cache transfer in disaggregated LLM inference,” in Proc. ACM SIGCOMM Workshop on Networks for AI Computing (NAIC), 2026, pp. 2067–2073. [9] K. Qian et al., “Alibaba HPN: a data center network for large language model training,” in Proc. ACM SIGCOMM, 2024. [10] A. Gangidi et al., “RDMA over Ethernet for distributed AI training at Meta scale,” in Proc. ACM SIGCOMM, 2024. [11] Y. Sun et al., “Nuska: stage-aware communication scheduling for disaggregated LLM serving,” arXiv:2603.17456, 2026. [12] E. Ding, B. Lyu, B. Kataria, and R. Singh, “Opus: photonic railoptimized fabric in ML datacenters,” in Proc. ACM SIGCOMM, 2026, pp. 922–939. [13] X. Liao et al., “MixNet: a runtime reconfigurable optical-electrical fabric for distributed mixture-of-experts training,” in Proc. ACM SIGCOMM, 2025, pp. 554–574. [14] HUBER+SUHNER Polatis, “POLATIS Series 6000n network optical matrix switch,” data sheet, Rev. 6000n.022022.001, 2022. [15] HUBER+SUHNER Polatis, “POLATIS Series 7000n optical switch,” data sheet, Rev. 7000n.022022.001, 2022. [16] F. Yang, Y. Zhou, B. Wang, Z. Zhou, and J. Li, “MoSE: modeswitching expander for mixed LLM training and inference,” in Proc. Asia Communications and Photonics Conf. (ACP), 2026, arXiv:2609.39138. [17] W. Wang et al., “TopoOpt: co-optimizing network topology and parallelization strategy for distributed training jobs,” in Proc. NSDI, 2023. [18] N. P. Jouppi et al., “TPU v4: an optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” in Proc. ISCA, 2023. [19] X. Tan, Y. Feng, Y. Zhou, Y. Jiang, Y. Zhu, and H. Xu, “Dynamic compute and network orchestration for disaggregated RL,” in Proc. ACM SIGCOMM, 2026. [20] A. Grattafiori et al., “The Llama 3 herd of models,” arXiv:2407.21783, 2024. [21] vLLM Team, “NixlConnector usage guide,” vLLM documentation, 2025. [22] SemiAnalysis, “AgentX v1.0 Claude Code traces,” dataset semianalysisai/cc-traces-weka-062126, Hugging Face, Jun. 2026. [23] B. Liu, H. Qiu, Í. Goiri, R. Fonseca, R. Bianchini, and E. Choukse, “Agentic coding in the wild: characterizing GitHub Copilot at production scale,” arXiv:2608.00101, 2026. [24] W. Kwon et al., “Efficient memory management for large language model serving with PagedAttention,” in Proc. SOSP, 2023. [25] L. Zheng et al., “SGLang: efficient execution of structured language model programs,” in Proc. NeurIPS, 2024. [26] V. Srivatsa et al., “Preble: efficient distributed prompt scheduling for LLM serving,” in Proc. ICLR, 2025. [27] M. Ayoub, Y. Fama, E. Crncevic, T. Smith, R. Joshi, and C. Costa, “Serving GLM-5.2 for agentic workloads on llm-d,” llm-d blog, Jul. 2026. [28] M. Al-Fares et al., “Hedera: dynamic flow scheduling for data center networks,” in Proc. NSDI, 2010. [29] M. Alizadeh et al., “CONGA: distributed congestion-aware load balancing for datacenters,” in Proc. ACM SIGCOMM, 2014. [30] Y. Zhu et al., “Congestion control for large-scale RDMA deployments,” in Proc. ACM SIGCOMM, 2015. [31] N. Farrington et al., “Helios: a hybrid electrical/optical switch architecture for modular data centers,” in Proc. ACM SIGCOMM, 2010.

[32] G. Wang et al., “c-Through: part-time optics in data centers,” in Proc. ACM SIGCOMM, 2010. [33] H. Liu et al., “Lightwave fabrics: at-scale optical circuit switching for datacenter and machine learning systems,” in Proc. ACM SIGCOMM, 2023. [34] L. Poutievski et al., “Jupiter evolving: transforming Google’s datacenter network via optical circuit switches and software-defined networking,” in Proc. ACM SIGCOMM, 2022. [35] R. Urata et al., “Mission Apollo: landing optical circuit switching at datacenter scale,” arXiv:2208.10041, 2022. [36] T. J. Seok, N. Quack, S. Han, R. S. Muller, and M. C. Wu, “Largescale broadband digital silicon photonic switches with vertical adiabatic couplers,” Optica, vol. 3, no. 1, pp. 64–70, 2016. [37] M. Wade et al., “An error-free 1 Tbps WDM optical I/O chiplet and multi-wavelength multi-port laser,” in Proc. OFC, 2021, paper F3C.6. [38] Optical Internetworking Forum, “Implementation agreement for a 3.2 Tb/s co-packaged (CPO) module,” OIF-Co-Packaging-3.2T-Module01.0, 2023. [39] CALIENT Technologies, “CALIENT’s optical circuit switch,” S320 data sheet, 2022. [40] FS, “400G DR4 OSFP PAM4 1310nm 500m MTP/MPO SMF silicon photonics transceiver,” data sheet, 2022. [41] Sumitomo Electric, “Single mode CWDM4 mux/demux coupler, FlexBeamGuidE-WD,” product page, accessed Oct. 2026. [42] IEEE Standard for Ethernet, IEEE Std 802.3-2022, Clause 124 (400GBASE-DR4), and IEEE Std 802.3cu-2021, Clause 151 (400GBASE-FR4). [43] 100G Lambda MSA, “400G-FR4 technical specification,” Rev. 2.0, 2018. [44] Optical Fiber Cabling and Components Standard, ANSI/TIA-568.3-D, 2016. [45] Microsoft, “Azure LLM inference trace 2024,” AzurePublicDataset, GitHub, 2024. [46] J. Stojkovic, C. Zhang, Í. Goiri, J. Torrellas, and E. Choukse, “DynamoLLM: designing LLM inference clusters for performance and energy efficiency,” in Proc. IEEE HPCA, 2025. [47] M. Alizadeh et al., “pFabric: minimal near-optimal datacenter transport,” in Proc. ACM SIGCOMM, 2013. [48] B. Montazeri, Y. Li, M. Alizadeh, and J. Ousterhout, “Homa: a receiverdriven low-latency transport protocol using network priorities,” in Proc. ACM SIGCOMM, 2018. [49] M. Khani et al., “SiP-ML: high-bandwidth optical network interconnects for machine learning training,” in Proc. ACM SIGCOMM, 2021. [50] W. M. Mellette et al., “RotorNet: a scalable, low-complexity, optical datacenter network,” in Proc. ACM SIGCOMM, 2017. [51] H. Ballani et al., “Sirius: a flat datacenter network with nanosecond optical switching,” in Proc. ACM SIGCOMM, 2020. [52] Y. Liu et al., “CacheGen: KV cache compression and streaming for fast large language model serving,” in Proc. ACM SIGCOMM, 2024. [53] Z. Liu et al., “KIVI: a tuning-free asymmetric 2bit quantization for KV cache,” in Proc. ICML, 2024. [54] Z. Liu et al., “KVServe: service-aware KV cache compression for communication-efficient disaggregated LLM serving,” in Proc. ACM SIGCOMM, 2026. [55] W. Wu, X. Mu, X. Chen, B. Tang, X. Cui, Y. Dou, and Z. Zhu, “xSwitch: an adaptive O-E-integrated interconnect for scale-up networks,” in Proc. ACM SIGCOMM Workshop on Hot Topics in Optical Technologies and Applications in Networking (HotOptics), 2026, pp. 2116–2121. [56] J. Ding, Y. Nishant, C. Ambati, J. Kamati, and T. Diep, “A photonicCXL memory appliance for scalable KV cache management in LLM inference,” arXiv:2607.27187, 2026.

Related documents

• An Optical KV Cache Network for Disaggregated LLM Serving#1343953
• BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks#1357911
• BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks#1348847
• MoSE: Mode-Switching Expander for Mixed LLM Training and Inference#1144105
• MoSE: Mode-Switching Expander for Mixed LLM Training and Inference#1136800