Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects

I NTRODUCING C LUSTER E FFICIENCY TO LLM S ERVING : P OOLING E CONOMICAL ACCELERATORS OVER FAST I NTERCONNECTS Youhe Jiang 1 Pavle Glušac 2 Binhang Yuan 3 Eiko Yoneki 1

arXiv:2611.02975v1 [cs.DC] 2 Nov 2026

1

University of Cambridge

2

Tenstorrent

3

HKUST

A BSTRACT Long-context large language model (LLM) serving places substantial demands on compute, memory capacity, and memory bandwidth. Economical accelerators can offer these resources at lower cost, but their limited per-device capacity requires pooling across devices, making communication overhead critical. We study cluster efficiency, measured in tokens per dollar and per joule, and show how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage. Guided by a characterization of coding-agent workloads and accelerator scaling, we realize this design on the Tenstorrent Galaxy, a cluster of Blackhole chips. The serving stack combines integrated communication engines, pipelined ring transfers, communication-computation overlap, and device-side decode replay to reduce collective overhead. Across three dense models in BF16 and 8-bit configurations with 8K-256K-token contexts, the Galaxy achieves 1.07-2.24× the tokens per dollar of an eight-GPU RTX PRO 6000 Blackwell server under a three-year acquisition-and-electricity cost model. Emulating TCP-class communication on the same chips reduces throughput by 23.5-67.4% and eliminates the cost advantage in every evaluated configuration. These results demonstrate how fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving. 1

I NTRODUCTION

Large language models (LLMs) now power a widening range of applications, from conversational assistants to coding agents, and the models behind them have grown to tens and hundreds of billions of parameters (Qin et al., 2025; Zhu et al., 2026; Grattafiori et al., 2024; Yang et al., 2025). A model of this size outgrows a single device, so efficient LLM serving typically runs it over a cluster of accelerators, whether GPUs, TPUs, or NPUs (Pope et al., 2023; Jouppi et al., 2023; Vasiljevic et al., 2021). These accelerators are not interchangeable: they differ in compute throughput, memory bandwidth, memory capacity, and power draw (Choquette, 2023; Lenovo Press, 2026; NVIDIA, 2025c). Which accelerator a serving cluster is built from is therefore a first-order decision for its efficiency. Powerful GPUs such as the B200, H200, and H100 provide high per-device memory capacity and bandwidth (device efficiency), so a large model spans few of them and the links between them are short and fast (NVIDIA, 2023a; 2024; Lenovo Press, 2026). Low-end GPUs such as the RTX A5000 and RTX 4090 provide much less of both (NVIDIA, 2021; 2022). A large model and its KV cache must therefore span several machines of them, and those machines typically reach one another over poor connections, which bottleneck the serving process (Shoeybi et al., 2019; Narayanan et al.,

2021). Our characterization confirms both halves of this picture: a machine of such accelerators lacks the memory and compute for a large model and its KV cache, and the links between such machines are far slower than NVLink (§3, Finding 2). Under this view, a serving cluster is built from the fewest, highest-end GPUs that the model allows. However, our characterization (§3) shows that this preference does not always hold once the cluster is judged by what it costs. In tokens per dollar and tokens per joule, the high-end part is not necessarily the better choice, because its higher per-device capacity comes with a higher price and a higher power draw (CNBC, 2024; Jarvislabs, 2026; Lenovo Press, 2026; NVIDIA, 2023a). Normalized by price, the ranking can reverse (§3, Finding 3): per $1000, a consumer RTX 4090 (NVIDIA, 2022) supplies 413 peak 8-bit TFLOPS of compute, more than three times the 129 TFLOPS of a B200 (CNBC, 2024). The weakness of such a lower-end accelerator lies in its per-device capacity rather than in what it offers for each dollar spent. This observation motivates us to rethink the design of the serving cluster. Rather than asking which device is the fastest (device efficiency), we ask how to build a cluster that serves the most tokens per dollar and per joule (Griggs et al., 2024; Stojkovic et al., 2025), a property that we define as the cluster efficiency of the deployment.

Introducing Cluster Efficiency to LLM Serving

To reach cluster efficiency, we propose a counter-intuitive design: build the cluster from lower-end accelerators, which offer more capacity per dollar and per watt, and connect them with fast links instead of the slow cross-machine connections they are typically given, so that the communication bottleneck of pooling them disappears. Our characterization demonstrates that over many fast links per chip, small memories and compute pool at little cost, with capacity, prefill, and decode scaling with the chip count while collectives stay a small share of each step, whereas over weak links the same chips lose most of the speedup (§3, Finding 4). Once the collectives (i.e., model-parallel communication (Shoeybi et al., 2019; Narayanan et al., 2021)) are cheap, the small memories and compute of many lower-end chips accumulate into a pool1 that keeps both advantages, the per-dollar and per-watt capacity of each chip and the scalability that today belongs to tightly linked high-end clusters. Our characterization further shows that, pooled this way, lower-end chips overtake high-end GPUs of the same tensor-parallel degree from as few as two devices in both tokens per dollar and tokens per joule (§3, Finding 5). We realize this design as a communication system over the chips of the Tenstorrent Blackhole Galaxy (Tenstorrent, 2026c;a), built for cluster efficiency (§4). Each chip is a lower-end part on its own, with a fraction of the compute and memory of a high-end GPU, and the communication system is what turns tens to hundreds of them into one efficient pool: it connects the chips over links integrated on each chip (§4.2), carries a transfer across the ring with little overhead on each hop (§4.3), and shapes model inference around those links so that the links alone set the cost of every collective (§4.4). §5 describes its implementation in the serving stack, from the model code down to the links and the Ethernet-core firmware beneath them. To our knowledge, this paper is the first to introduce the concept of cluster efficiency and to rethink the design of the LLM serving cluster on top of it, pooling lower-end accelerators over fast links in place of fewer high-end GPUs. Our contributions are summarized as follows: Contribution 1: We characterize long-context agent serving on lower-end and high-end accelerators (§3). Five findings (Findings 1-5) show that (i) long-context agent serving needs compute, memory capacity, and memory bandwidth together; (ii) a machine of lower-end accelerators lacks them, and the links between such machines are slow; (iii) per dollar, lower-end accelerators can offer more memory, bandwidth, and compute than high-end GPUs; (iv) over fast links, lower-end chips pool at little cost, whereas over weak links they lose most of the speedup; and (v) pooled over fast 1 A pool is the set of devices that serves one model replica over tensor parallelism; a cluster holds one or more pools and is the unit that a provisioner has to buy and power.

links, lower-end chips beat high-end GPUs in tokens per dollar and per joule. These characterizations motivate the cluster design of utilizing fast links to connect lower-end accelerators for achieving cluster efficiency. Contribution 2: Motivated by the characterization results, we define cluster efficiency and realize a communication system on the Tenstorrent Blackhole Galaxy (Tenstorrent, 2026c), a cluster of lower-end accelerators (§4, §5). The communication system links the chips through ports integrated on every chip (§4.2), carries transfers across the ring with little overhead on each hop (§4.3), and shapes model inference around those links (§4.4), implemented throughout the Galaxy serving stack, from the vLLM serving engine and runtime down to the collective kernels (§5). Contribution 3: We evaluate the Galaxy against an RTX PRO 6000 Blackwell server on three dense models, two precisions, and 8K-256K contexts (§6). The Galaxy delivers 1.07-2.24× the tokens per dollar and 1.00-2.27× the tokens per joule of the GPU server across all configurations (§6.2). An ablation of the Galaxy chips connected over slow links cuts their throughput by 23.5-67.4% and erases every one of its cost-efficiency wins over the GPU server (§6.3).

2

BACKGROUND

LLM serving. Serving a decoder-only LLM runs each request through two phases: prefill, which processes the prompt in one forward pass and writes its KV cache, and decode, which generates one token per step while reading the KV cache of every earlier token (Pope et al., 2023; Kwon et al., 2023). Serving engines batch many requests into one decode step so that each weight read serves all of them, admit and retire requests between steps, and page the KV cache so that memory is allocated per block rather than per maximum context (Yu et al., 2022; Kwon et al., 2023; Zheng et al., 2024; Sun et al., 2024); chunked prefill schedules prompt chunks alongside decode, and prefill and decode may be split across machines (Agrawal et al., 2024; Zhong et al., 2024; Patel et al., 2024; Qin et al., 2025). The cost of serving is therefore set by how much memory and bandwidth a deployment holds and by how many requests share them at each step, and agentic workloads raise both with long sessions whose cached state stays resident (Zhu et al., 2026); this paper judges a serving cluster by what that capacity costs per token served, in dollars and in joules, which we call its cluster efficiency, rather than by what any one of its devices delivers on its own. Parallelism. A model that exceeds one accelerator is split by parallelism. Tensor parallelism shards every layer across devices and synchronizes them at each layer with collectives, so its step time depends on the latency and bandwidth of the links between the devices (Shoeybi et al., 2019; Narayanan

Introducing Cluster Efficiency to LLM Serving

et al., 2021); pipeline parallelism assigns contiguous layers to each device and passes only activations between stages, which tolerates slower links at the cost of pipeline bubbles (Huang et al., 2019; Narayanan et al., 2021); expert and sequence parallelism partition the mixture-of-experts layers and the token dimension (Lepikhin et al., 2021; Korthikanti et al., 2023; Liu et al., 2024). Inference systems combine these with data parallelism across model replicas and choose the combination per model and hardware (Aminabadi et al., 2022; Zheng et al., 2022; Pope et al., 2023). For serving, tensor parallelism is the way to pool the memory and bandwidth of several devices behind one model replica, and the collectives it issues at every layer are the price of pooling; this paper shows that this price, set by the links rather than by the devices, decides whether the per-dollar advantage of lower-end accelerators survives pooling, and builds a communication system that keeps it small. Heterogeneous resources. The hardware that serves LLMs is no longer one kind of device. Within one vendor’s line, successive generations differ in memory, bandwidth, and interconnect, from H100 and H200 to B200 (Choquette, 2023; NVIDIA, 2024; Lenovo Press, 2026), and the same generation ships in data-center and workstation parts that differ mostly in links and price (NVIDIA, 2025c). Beyond GPUs, TPUs (Jouppi et al., 2023), wafer-scale and dataflow accelerators (Lie, 2023; Abts et al., 2022), and Tenstorrent’s Ethernet-connected chips (Vasiljevic et al., 2021; Tenstorrent, 2026a) trade per-device capability for cost and scale-out. Serving systems have begun to exploit this heterogeneity: by mixing GPU types across replicas (Griggs et al., 2024; Jiang et al., 2025b), by placing one model across devices of unequal memory and links (Jiang et al., 2023; 2025c; Mei et al., 2025; Zhang et al., 2025; Wei et al., 2026), by using preemptible or consumer devices (Miao et al., 2024; Borzunov et al., 2023; Sheng et al., 2023), and by characterizing new accelerators for decode (Lim et al., 2027; Rhee et al., 2025). These systems take the devices as given and schedule serving around their differences; this paper rethinks the cluster design itself, and shows that lower-end accelerators connected over fast links beat highend GPUs on the two metrics a provisioner ultimately pays for, tokens per dollar and tokens per joule.

3

C HARACTERIZATION

This section builds the case for cluster efficiency in five steps, each closed by a finding. We first characterize what long-context agent serving demands from hardware, then show why lower-end accelerators look unfit for it one device at a time, why they nevertheless offer more resources per dollar, how many fast links per chip pool their small memories and compute at little cost, and that the resulting pool beats high-end GPUs per dollar and per joule. The workload

is the TraceLab coding-agent trace (Zhu et al., 2026); the scaling experiments run one Llama-3.1-8B replica at 32K context on one, two, four, or eight devices of the Galaxy and of the GPU server (§6 describes both machines). Agent serving needs compute and memory. We analyze 357149 model invocations from 4265 TraceLab codingagent sessions (Table 1); an invocation is one agent step, not a complete user request. The median step adds 875 fresh input tokens to a cached prefix of 118656 tokens and generates 214 output tokens, so cached prefixes supply 95.7% of the context, and the 99th-percentile step holds 822889 tokens. The two phases load different resources. Prefill is compute-bound: one 32K prompt sustains 489 TFLOP/s in FP8 on one GPU, 49% of its dense peak, and 45 TFLOP/s on one Galaxy chip. Decode is memory-bound: the GPU’s effective decode bandwidth, modeled weight and KV bytes divided by measured step time, rises from 1.22 TB/s at one user to 1.48 TB/s at 37 users, 93% of its nominal 1.6 TB/s. The resident state is large: mapping each session’s peak context to Llama-3.3-70B BF16 KV gives a median of 22.6 GB and a 90th percentile of 73.9 GB before weights, and the 36 sessions open at the busiest moment hold 9.6 million context tokens, 3.15 TB of KV cache under that model. Table 1. TraceLab agent-step demand: tokens per step over 357149 steps, and the Llama-3.3-70B BF16 KV cache at each session’s peak context (decimal GB, before weights and runtime buffers). P50 P90 P99 Fresh input (tokens) 875 7119 144688 Cached prefix (tokens) 118656 247836 816627 Output (tokens) 214 1169 5050 Context held in KV (tokens) 124024 256770 822889 Session peak KV, 70B BF16 (GB) 22.6 73.9 139.3

Finding 1. Long-context agent serving needs compute for prefill and memory capacity and bandwidth for decode and for a resident KV cache that grows with every open session. A serving pool must supply both. Lower-end machines lack memory, compute, and links. Conventional provisioning keeps large-model serving on high-end GPUs because a machine of lower-end accelerators falls short in two ways (Table 2). Within the machine, memory and compute are small: an eight-GPU RTX A5000 server holds 8×24 GB and an RTX 5090 card 32 GB, against 80-180 GB per H100 or B200, and their peak compute is a fraction of the high-end parts’. The retained Llama-3.3-70B dimensions imply 141.1 GB of raw BF16 weights before the 73.9 GB session KV of Finding 1, so a long-context pool of such parts must span machines. Across machines, the links are slow: the A5000 server reaches other servers at 25 GB/s of RDMA, about 3 GB/s per GPU, while high-end GPUs share 450-900 GB/s of NVLink inside one chassis. Pooling many small machines over slow links is exactly

Introducing Cluster Efficiency to LLM Serving

what conventional provisioning avoids, so lower-end parts are passed over for large models at any price. Table 2. Per-device resources of high-end and lower-end accelerators (Lenovo Press, 2026; NVIDIA, 2024; 2023a;b; 2025c; 2021; 2025b). 8-bit rates are dense vendor peaks (GeForce FP8 with FP16 accumulation; the RTX PRO 6000 Blackwell rate is derived from its 4 PFLOPS sparse FP4 rating). Link: the fastest link to another device in the same machine, per direction (NVL: NVLink generation; PCIe: PCIe generation). † FP16 is listed, because the RTX A5000 has no FP8 support in hardware. Accelerator Price $ GB GB/s TFLOPS Link, GB/s B200 ∼35000 180 7700 4500 NVL5, 900 H200 ∼35000 141 4800 1979 NVL4, 450 H100 SXM ∼27500 80 3350 1979 NVL4, 450 L40S ∼7500 48 864 733 PCIe4, 32 RTX PRO 6000 8565 96 1597 ∼1000 PCIe5, 63 RTX A5000 2250 24 768 111† PCIe4, 32 RTX 4090 1599 24 1008 661 PCIe4, 32 RTX 5090 1999 32 1792 838 PCIe5, 63

Finding 2. A machine of lower-end accelerators has too little memory and compute for a large model and its KV cache, and the links between such machines are slow. Serving on them means pooling several such machines, and slow links between the machines are what prevent it. Per dollar, lower-end could offer more. Normalized by price, the ranking reverses (Table 3). Per $1000, a Blackhole p150a supplies 22.9 GB of memory, 366 GB/s of memory bandwidth, and 475 peak Block FP8 TFLOPS, against 5.1 GB, 220 GB/s, and 129 TFLOPS for a B200 and 11.2 GB, 186 GB/s, and 117 TFLOPS for an RTX PRO 6000 Blackwell. The consumer RTX 4090 and RTX 5090 follow the same pattern with 15.0 and 16.0 GB, 630 and 896 GB/s, and 413 and 419 TFLOPS, about three to four times the B200 on every axis, while the data-center H100 and H200 sit lowest at 2.9 and 4.0 GB, 122 and 137 GB/s, and 72 and 57 TFLOPS. Prices are the midpoint of NVIDIA’s $3040K estimate for the B200 (CNBC, 2024), midpoints of reported ranges for the H200 and H100 (Jarvislabs, 2026), a reported L40S card price (Modal, 2025), the RTX PRO 6000 Blackwell price (Punch Technology, 2026; Bytehaus, 2026), launch prices for the A5000, RTX 4090, and RTX 5090 (CG Channel, 2021; NVIDIA, 2022; 2025a), and the p150a list price (Tenstorrent, 2026a). The 8-bit peaks cover different number formats and accumulation modes, so they are an upper bound on, rather than a prediction of, the compute that each device delivers on a real workload. Finding 3. Per dollar, lower-end accelerators could offer more memory, memory bandwidth, and compute than high-end GPUs. Their weakness is per device, not per dollar; the comparison that matters is per dollar.

Table 3. Memory (GB), memory bandwidth (GB/s), and peak 8-bit compute (TFLOPS) per $1000, at the prices of Table 2 and the p150a’s $1399 list price (Tenstorrent, 2026a). Accelerator GB GB/s TFLOPS B200 5.1 220 129 H200 4.0 137 57 H100 SXM 2.9 122 72 L40S 6.4 115 98 RTX PRO 6000 11.2 186 117 RTX A5000 10.7 341 49† RTX 4090 15.0 630 413 RTX 5090 16.0 896 419 Blackhole p150a 22.9 366 475

Link speed decides whether pooling pays. We vary tensor parallelism at a fixed model and context and report each quantity relative to one chip, over the Galaxy’s own links and over the same chips with emulated TCP-class links (Table 4). Over the native links, the 8-bit active batch grows from 10 users on one chip to 25, 54, and 113 users on two, four, and eight: capacity grows faster than the chip count because the fixed weight footprint spreads over more memory. The pooled compute speeds up a fixed workload as well: a 32K prefill falls from 18.04 s to 3.11 s (5.80×) and the eight-user decode step from 99.1 ms to 16.8 ms (5.90×), while collectives take only 4.8-6.5% of a profiled decode step. Over TCP-class links the same chips keep the same capacity but lose most of the speedup: the 32K prefill on eight chips takes 8.20 s (2.20×), the eight-user step 42.0 ms (2.36×), and the gain stalls beyond four chips because every collective now pays a link latency of 100-250 µs and communication fills 63% of the step. The capacity is still there, but the added chips mostly wait on one another; §4.1 names the terms behind this loss and §6.3 measures them. Table 4. Llama-3.1-8B 8-bit at 32K on two, four, and eight Galaxy chips, relative to one chip, over fast links (the communication system of §4) and over the same chips with emulated TCP-class links (the bandwidth and 64 KB latency of NCCL over TCP sockets, §6.1). Active users are the memory maximum; speedups are for a fixed workload (one 32K prefill, a one- or eight-user decode step). Collective share is the fraction of a decode step spent in collectives, measured with the device profiler over the fast links and computed from the step-time increase over TCP-class links. Metric TP2 TP4 TP8 Active users (any link) 2.50× 5.40× 11.30× Native: 32K prefill speedup 1.87× 3.37× 5.80× Native: 1-user step speedup 2.72× 4.54× 6.95× Native: 8-user step speedup 2.13× 3.69× 5.90× Native: collective share 5.2% 4.8% 6.5% TCP-class: 32K prefill speedup 1.50× 2.05× 2.20× TCP-class: 1-user step speedup 2.31× 2.61× 2.50× TCP-class: 8-user step speedup 1.87× 2.31× 2.36× TCP-class: collective share 16.8% 40.4% 62.6%

Introducing Cluster Efficiency to LLM Serving

Finding 4. Over many fast links per chip, small memories and compute pool at little cost: capacity, prefill, and decode scale with the chip count while collectives stay a small share of each step. Over weak links, the same chips keep their capacity but lose most of the speedup.

system, and the resources and links of its pools. §4.3 demonstrates its communication system, the links and routers that keep the communication overhead small. §4.4 shows how Galaxy adapts model inference to its fast links to improve cluster efficiency on real inference workloads.

Pool size reverses the ranking. We compare each Galaxy replica with the GPU replica of the same tensor-parallel degree at its maximum batch, at $5000 per chip and $28137 per GPU over three years at $0.15/kWh, with the power of the replica’s own devices (Table 5). One chip loses to one GPU: in 8-bit it delivers 0.63× the tokens per joule and 0.52× the tokens per dollar. The pool reverses the ranking at two devices and widens it from there, reaching 1.25× in tokens per joule and 1.54× in tokens per dollar at eight. The Galaxy replica gains because its users per chip keep rising, from 10 to 14 across the sweep, while the GPU replica’s tokens per dollar peak at two GPUs and fall at eight, where collectives take half of its decode step. BF16 shows the same reversal, reaching 1.32× and 1.59× at eight devices. Several smaller replicas are another valid use of eight GPUs, and the main evaluation adopts them; the device ranking and the pool ranking disagree, and the cluster, not the device, is the unit a provisioner has to buy and power.

4.1

Table 5. Efficiency of one Llama-3.1-8B replica at its maximum batch (32K) on n Galaxy chips over fast links (the communication system of §4) against n RTX PRO 6000 Blackwell GPUs on their own links: tokens per dollar in thousands (prices are given in §6.1) and the Galaxy/GPU ratios in tokens/$ and tokens/J. The power of each device is profiled with its vendor’s management interface during every timed round (Tenstorrent, 2026; NVIDIA, 2026). Precision n Galaxy tok/$ GPU tok/$ tok/$ ratio tok/J ratio 8-bit 1 1020K 1975K 0.52× 0.63× 8-bit 2 2090K 2011K 1.04× 1.07× 8-bit 4 2317K 1926K 1.20× 1.16× 8-bit 8 2367K 1533K 1.54× 1.25× BF16 1 817K 904K 0.90× 0.97× BF16 2 1006K 983K 1.02× 1.00× BF16 4 1166K 979K 1.19× 1.14× BF16 8 1257K 792K 1.59× 1.32×

Finding 5. Pooled over fast links, lower-end chips beat high-end GPUs in tokens per dollar and reported tokens per joule, and the edge grows with the pool. Efficiency must be compared across whole pools, not single devices.

Definition and Communication Impact

This subsection defines cluster efficiency and the terms through which communication lowers it. Objectives for cluster efficiency. Cluster efficiency has two objectives, tokens per joule and tokens per dollar, and both take the pool rather than the device as their unit (Finding 5). For a pool that decodes R tokens/s for active requests while its devices draw P watts, over an active lifetime of H seconds, at an acquisition cost of C dollars and an electricity price of e dollars per kWh, E=

R , P

D=

RH , C + kP

C LUSTER E FFICIENCY AND I TS R EALIZATION ON G ALAXY

This section turns the five findings into a definition and shows how Galaxy realizes it. §4.1 defines cluster efficiency by its two metrics and the terms through which communication lowers it. §4.2 presents Galaxy, a pooled lower-end

eH . 3.6 × 106

(1)

Here E is in tokens/J and D in tokens/$; R counts tokens of active requests only, so padded slots do not inflate it; P is the measured device power, whose telemetry boundary (chip, board, or system) must be stated with every value of E; and k converts watts into lifetime electricity dollars, so the denominator of D is the purchase price plus the electricity the pool draws over its life, and the numerator is every token it produces in that time. The paper uses H = 3 years of continuous use and e = $0.15/kWh, under which electricity is under a tenth of the denominator, so D is set mainly by throughput per dollar of hardware. This denominator is the three-year total cost of ownership that accelerator designers optimize, capital amortized over three years plus the electricity of operation (Jouppi et al., 2021; Barroso et al., 2018), and the cost per generated token that serving-hardware studies report (Peng et al., 2023). Both are provisioning metrics for comparing hardware, not predictions of utilization. Communication impact on cluster efficiency. Communication lowers both objectives through the time a decode step spends in collectives, and that time has two parts. Following the standard α-β model, the collectives of one step cost Tcomm ≈

K  X i=1

4

k=

αi +

Vi  , Bi

(2)

where K is the number of collectives per step, αi is the startup and synchronization latency of the i-th, Vi is the bytes it moves over the bottleneck link, and Bi is the bandwidth of that link. The two terms ask for different link properties: bandwidth shrinks Vi /Bi , but a decode collective moves only kilobytes, so its time is dominated by αi , and K is large (130 to 484 per step for the models in §6). Fast links must therefore be low-latency as well as wide,

Introducing Cluster Efficiency to LLM Serving the tray ring continues into the next Galaxy

ring closure: 14 links, tray 3 back to tray 0

1 2

4-chip pool

3

4 5 6 7

8 9 10 11

12 13 14 15

16 17 18 19

20 21 22 23

Galaxy 0

Tray 3 24 25 26 27

Galaxy 1

Galaxy 2

Galaxy 3

28 29 30 31

8 ports

0

Tray 2

8 ports

8-chip pool

Tray 1

8 ports

Tray 0

8 ports (ring closure)

Figure 1. The communication topology of one Galaxy, from the measured cluster descriptor. Each tray is a 2 × 4 mesh of chips with two links between neighbours (grey, 80 links); the four trays form a ring in which each chip is linked to the same position in the next tray (green, 60 links), and the bracket marks the 14 links that close the ring from tray 3 back to tray 0. An eight-chip pool is one tray and a four-chip pool is a 2 × 2 square inside it.

Figure 2. A four-Galaxy supercluster. The tray ring of each system continues through its edge ports into the next, so the 128 chips form one ring of 16 trays; each green bundle is eight same-position links through 800 GbE QSFP-DD ports.

and that latency is what Finding 4 measures across four pool sizes, from one chip to eight, on one machine.

400 GbE per direction, so the ring construction extends across trays and machines unchanged.

4.2

Beyond one machine. The same wiring continues across machines, which is what makes the Galaxy a template for the lower-end clusters of Finding 2 rather than a single box. The Ethernet ports at the edge of the tray ring are exposed on the chassis, up to 56 QSFP-DD ports at 800 GbE, and are cabled to the matching ports of the next system, so the tray ring of one Galaxy continues into the next and a foursystem supercluster is one ring of 16 trays with no switch on the path (Figure 2) (Tenstorrent, 2026c). The vendor offers this configuration from four to more than 36 systems, and Equation 2 applies to it unchanged: each added hop adds one link latency to αi and nothing to the bytes each rank moves, so the communication time grows by one α per machine on the ring and by nothing in its bandwidth term.

Economical Resources on Galaxy

This subsection presents Galaxy, the pooled lower-end system, and the resources and links of its pools. Resources per chip. The Galaxy starts from cheap resources: each of its 32 Blackhole chips carries 32 GB of GDDR6 at 512 GB/s, 210 MB of SRAM, and 664 peak Block FP8 TFLOPS (Tenstorrent, 2026c;a). At the $5000 the machine’s list price allocates to a chip, this is 6.4 GB, 102 GB/s, and 133 TFLOPS per $1000, about twice the memory and bandwidth per dollar of the GPU server (Finding 3). The links are on the same die: each chip has Ethernet cores that drive its own 400 GbE ports, so a chip reaches its neighbours over its own NoC rather than through a host or a PCIe switch; the measured cluster descriptor shows 8 cabled ports on 20 chips and 10 on the rest. Pools of chips. A pool is the group of chips that serves one model replica, and the Galaxy’s wiring lets a pool be any rectangle of its grid (Figure 1). The 32 chips sit on four trays of eight; within a tray the chips form a 2 × 4 mesh with two links between every neighbouring pair, 80 links in all, and the four trays form a ring in which each chip is linked to the chip at the same position in the next tray, 60 links more, so the whole machine is a 4 × 8 grid with wrap-around along the tray axis. Galaxy places a pool on any rectangle of this grid: a four-chip pool on a 2 × 2 square, an eight-chip pool on one tray, a 16-chip pool on two trays, and a 32-chip pool on the whole machine; it runs each pool’s collectives on a logical ring over its chips and places several pools side by side on disjoint chips. The inter-tray links carry the same

4.3

Fast Links on Galaxy

This subsection demonstrates the communication system of Galaxy: the links and routers that keep the communication overhead small on every hop of the ring. Low startup cost. The communication time of Equation 2 is spent mostly on the startup term αi , so the links must keep that term small, and the Galaxy does so by keeping the host out of the data path. Figure 3 shows the path of one transfer: a worker core writes a tensor over the on-chip NoC to a fabric router that runs on an Ethernet core; the router sends it over the link, the receiving router forwards it toward its destination or delivers it to the local NoC, and the NoC writes it into the addressed buffer. The link runs in TT-link mode, in which hardware adds a 16-byte header, places data

Introducing Cluster Efficiency to LLM Serving

chip A (sender) i

receiving addressed router buffer chip B (receiver) iii iv

one chip of the ring shards from the ring

Ethernet core: fabric router

worker core

fused

router matmul cores all-gather cores

Ethernet core A

NoC

receive RISC-V slots

earlier

time

forward

Figure 3. The path of one transfer between two Galaxy chips. A worker core writes over the on-chip NoC to the fabric router on an Ethernet core (i), the router sends over the link (ii), the receiving router delivers to its local NoC or forwards toward the destination (iii), and the NoC writes the addressed buffer (iv). send RISC-V

hops under the matmul

Wo matmul on shards

ii Ethernet link NoC mesh

unfused hop hop hop Wo matmul

data credits

Ethernet core B slots receive RISC-V forwarded 2 free send RISC-V new, from NoC

Figure 4. Inside the Ethernet cores at the two ends of one link. Each core runs a send and a receive RISC-V processor; data crosses the link only against credits returned by the receiver, and a chip injects new traffic only while two of its receiver buffer slots stay free for forwarded traffic, which keeps the ring free of deadlock.

into receiver buffers, and retransmits with Go-Back-N (TTMetalium Scale-Out Team, 2026); the host configures the transfer but never moves the bytes. A 64 KiB all-gather therefore takes 15.0 µs on a ring of four chips and 21.0 µs on a ring of eight, against 119 and 257 µs for the same collectives over NCCL TCP sockets on the GPU server, the reference for the weak links of §3 and §6.3. Pipelined links. The bandwidth term Vi /Bi stays small when transfers overlap on the ring, which the fabric arranges in hardware and firmware. The installed Blackhole firmware runs separate send and receive RISC-V processors on each Ethernet core, credit-based flow control protects the receiver buffers, and ring deadlock avoidance reserves two free slots for forwarded traffic before a chip injects new traffic (Figure 4). Measured on rings of four and eight chips, a 64 MiB all-gather reaches 85 to 86 GB/s of bus bandwidth, where bus bandwidth is (n−1)V /T for n ranks contributing V bytes each in time T , against 5.5 and 4.3 GB/s over the TCP sockets; each timing is the mean of 20 trace replays after five warm-ups with one link per hop. Achieved bandwidth. What the model sees is the achieved collective bandwidth, which sits below the port rate. The collective helper chooses how many links a collective uses, and specialized kernels set their own parameters (Tenstorrent, 2026b), so a collective rarely drives every port: 86 GB/s per chip is about a fifth of the 400 to 500 GB/s its

Figure 5. The fused all-gather and matmul on one chip of the ring. The fabric router on the Ethernet core delivers the attention-head shards from the ring neighbour, one set of Tensix cores places them, and another set runs the output projection on each shard as it lands, so the hops of the collective run under the matmul instead of before it, on every chip of the ring at once.

cabled ports could carry. The transfer term Vi /Bi of Equation 2 is set by this software rather than by the ports’ full rate, and the headroom above it is a route to a smaller term that needs no new hardware, only firmware and kernel work. 4.4

Adapting Model Inference to the Links

This subsection demonstrates how Galaxy adapts model inference to its fast links. A pool runs one model replica with standard tensor parallelism (Shoeybi et al., 2019), two ring collectives per layer over the links of §4.2, and the adaptation lies in how those collectives are overlapped with compute on the ring and issued without the host, so that the links of §4.3 are the only thing a collective waits for. Overlapping the all-gather with compute. Figure 3 separates the two kinds of core a transfer touches: the fabric router on the Ethernet core moves the data, and the Tensix cores are free while it is in flight. Galaxy’s decode uses that separation to hide a collective behind the matmul that needs it. The all-gather of the attention heads runs on one set of Tensix cores and the output projection on another, and the projection consumes each head shard as the router delivers it from the ring, so the n − 1 hops of the collective run under the matmul instead of before it (Figure 5). This adaptation takes one of the two collectives per layer off the critical path: its αi and Vi /Bi in Equation 2 are paid under compute, and the step sees one collective per layer on its critical path instead of two, on every chip of the pool. Host-free decode steps. The links of §4.3 make a 64 KiB all-gather cost 15 to 21 µs on a ring, but a decode step issues hundreds of collectives, and if the host enqueued each of them from Python the dispatch would be paid on every one, on top of the link latency. Galaxy therefore runs decoding through Metal Trace (Tenstorrent, 2025) in two sub-steps, (i) capture and (ii) replay. For the capture, Galaxy runs one decode step eagerly at start-up while Metal Trace records every operation and collective it issues. For the replay, every later decode step issues that recording from the device’s command queue (Figure 6): the collectives run

Introducing Cluster Efficiency to LLM Serving capture, once: host dispatch before every collective host enqueue device

enqueue all-gather

idle

6

enqueue matmul

idle

reduce-scatter

each collective waits for host dispatch on top of the link latency replay, every step: issued from the device queue host replay device

all-gather matmul reduce-scatter next layer ... back-to-back: each hop pays the link latency alone

time

Figure 6. Capturing and replaying one decode step. The capture pass runs once at start-up, dispatched eagerly from Python: the device waits for the host to prepare and enqueue each operation, so the ring idles for a host dispatch before every collective. Every later decode step, for every request, replays the captured operations from the device queue back-to-back, so each hop pays the link latency alone, with no host in the loop between one collective and the next one on the ring, step after step.

back-to-back, and none waits for the host to dispatch it. This adaptation works on the startup term αi of Equation 2: each hop pays the link latency of §4.3 and no host time, so a decode step is as short as the fabric and the kernels allow, whatever the host is doing in the meantime.

5

I MPLEMENTATION

Serving stack. Galaxy serves through TT inference server 0.25.0 (Tenstorrent, 2026d) on TT-Metal revision 978f93b56e36 (Tenstorrent, 2026e) and firmware 19.11.0. The server’s Llama and Qwen model code shards each layer across the chips of a pool (Shoeybi et al., 2019), keeps the KV cache in paged device memory (Kwon et al., 2023) with the heads that own it, and runs decode as a captured Metal Trace (Tenstorrent, 2025) that every later step replays, with the next token selected on the device. The fabric is brought up in FABRIC 1D RING mode (TTMetalium Scale-Out Team, 2026), one logical ring per pool over the links of §4.2, with every hop forwarded by the Ethernet-core routers of §4.3 rather than by a switch. Communication system. Every collective runs on the Ethernet cores of §4.3, whose router firmware forwards packets around the ring under credit-based flow control and the twoslot injection rule (TT-Metalium Scale-Out Team, 2026). The model issues them through TT-NN’s asynchronous kernels (Tenstorrent, 2026b), the ring all-gather, the minimal reduce-scatter that implements every all-reduce, and the fused all-gather-matmul of §4.4, each tuned per model and pool size in links per hop, worker cores per link, and chunks between synchronizations. Payloads cross the ring in block floating point and land in the width-sharded layout the next matmul reads, and banks of hardware semaphores, doublebuffered per ring axis, let back-to-back collectives proceed without waiting on one another (Tenstorrent, 2026b).

E VALUATION

We design the evaluation to answer two research questions: • RQ1: How does the Galaxy compare with the GPU server in tokens per dollar and tokens per joule? • RQ2: How much of that efficiency do slower links take away at the same active concurrency? 6.1

Experimental Setup

Hardware environment. The lower-end cluster is one Tenstorrent Blackhole Galaxy, 32 chips of 32 GB each joined by the Ethernet links of §4.2, with the serving stack of §5 (Tenstorrent, 2026c). The high-end baseline is one server with eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (96 GB each) connected over PCIe 5.0 ×16 links inside a single chassis (NVIDIA, 2025c). Models and workloads. We evaluate Llama-3.3-70BInstruct, Qwen3-32B, and Llama-3.1-8B-Instruct in BF16 and 8-bit configurations (Grattafiori et al., 2024; Yang et al., 2025); the 8-bit runs use Galaxy BFP8 and GPU FP8 weights and KV cache, formats that differ numerically. We test workloads whose context length varies from 8K to 256K tokens, so that each comparison spans short sessions and long resident ones; K denotes 1024 tokens. Baseline systems. We compare three systems, and the same version of vLLM (Kwon et al., 2023) serves on top of each. (i) GPU server: the RTX PRO 6000 Blackwell server described above. (ii) Galaxy with our communication system: the chips pooled over their native links by the communication system of §4. (iii) Galaxy with TCP-class links: the same chips and batches with every collective raised to the bandwidth of NCCL over TCP sockets, modelling the slow links between machines in a conventional cluster (§6.3). Metrics. We report the two cluster-efficiency metrics of Equation 1, tokens per dollar and tokens per joule. Tokens per dollar uses the selling price of each system, the list price of the Galaxy and the market price of the RTX PRO 6000 Blackwell server (Tenstorrent, 2026c; Punch Technology, 2026; Bytehaus, 2026). We record power through the core-rail telemetry of the Tenstorrent system management interface and the board telemetry of the NVIDIA management library, sampled every one to two seconds throughout every timed round (Tenstorrent, 2026; NVIDIA, 2026). 6.2

End-to-End Efficiency Evaluation (RQ1)

Tokens per dollar. The Galaxy delivers more tokens per dollar than the GPU server in all 36 configurations (Figure 7), by 1.07-2.24×. The gain is largest for Llama-3.3-70B in BF16, 1.75-2.24×, where the memory of the GPU server

Introducing Cluster Efficiency to LLM Serving

1.29× 0.95×

1.27× 0.94×

1.28× 0.95×

1.30× 1.00×

1.32× 0.98×

1.14× 0.85×

64K 128K 256K

1.21× 0.93×

1.1 0.8 0.53 0.27 0

64K 128K 256K

1.13× 0.86×

1.3M 999K 666K 333K 0

1.34× 1.01×

0.57 0.42 0.28 0.14 0

64K 128K 256K

1.21× 0.93×

1.17× 0.88×

694K 521K 347K 174K 0

1.12× 0.85×

1.40× 1.07×

1.32× 0.99×

8K 16K 32K

Llama-3.1-8B

1.21× 0.93×

1.29× 0.98×

1.28× 0.95×

1.21× 0.84×

8K 16K 32K

1.23× 0.94×

7.5 5.6 3.7 1.9 0

8K 16K 32K

1.16× 0.82×

9.9M 7.4M 4.9M 2.5M 0

1.21× 0.90×

3.8 2.8 1.9 0.94 0

8K 16K 32K

1.26× 0.88×

1.15× 0.56×

5.2M 3.9M 2.6M 1.3M 0

1.13× 0.76×

1.55× 0.71×

1.36× 0.68× 1.15× 0.56×

64K 128K 256K

Llama-3.1-8B

1.25× 0.86×

1.54× 0.68×

1.40× 0.66×

1.45× 0.70× 0.72×

1.42×

1.18× 0.60×

0.47 0.35 0.23 0.12 0

64K 128K 256K

1.17× 0.61×

585K 439K 292K 146K 0

64K 128K 256K 1.16×

0.22 0.16 0.11 0.055 0

64K 128K 256K

0.58×

1.09× 0.45×

273K 205K 136K 68K 0

1.18×

1.37× 0.71×

1.22× 0.53×

8K 16K 32K

Galaxy, TCP-class links

Qwen3-32B

0.61×

1.39× 0.67×

1.25× 0.50×

1.24× 0.49× 1.15× 0.51×

8K 16K 32K

1.06× 0.48×

2.6 2 1.3 0.66 0

1.10× 0.43×

3.7M 2.8M 1.9M 931K 0

8K 16K 32K 1.10×

1.4 1 0.68 0.34 0

8K 16K 32K

1.12× 0.48×

1.37× 0.58× 1.35× 0.59×

1.9M 1.4M 926K 463K 0

0.42×

2.03× 0.91×

1.77× 0.74× 1.25× 0.52×

64K 128K 256K

Qwen3-32B

1.05×

2.08× 0.91×

1.75× 0.69×

2.03× 0.85× 2.06× 0.90×

64K 128K 256K

1.24× 0.55×

0.3 0.22 0.15 0.074 0

1.28×

372K 279K 186K 93K 0

64K 128K 256K

0.54×

1.29× 0.54× 1.27× 0.56×

1.02× 0.39×

8K 16K 32K

0.094 0.07 0.047 0.023 0

64K 128K 256K

1.25×

2.23× 1.01×

2.27× 1.04× 1.07× 0.38×

8K 16K 32K

118K 88K 59K 29K 0

0.55×

2.19× 0.92×

2.24× 0.96×

1.81×

BF16 Tokens/$

0.68× 1.74× 0.74× 0.36×

1.08×

8K 16K 32K

1.00×

1.5 1.1 0.76 0.38 0

8K 16K 32K

0.39×

0.6 0.45 0.3 0.15 0

2.3M 1.7M 1.1M 569K 0

8-bit Tokens/J

8-bit Tokens/$

BF16 Tokens/J

815K 611K 408K 204K 0

Galaxy, our communication system

Llama-3.3-70B

0.46×

GPU server

Llama-3.3-70B

64K 128K 256K

Context length (tokens)

Figure 7. Tokens per dollar and tokens per joule of the GPU server, of the Galaxy with our communication system, and of the Galaxy over TCP-class links, for three models in BF16 and 8-bit over 8K-256K contexts. Over TCP-class links every collective is raised to the bandwidth and latency of NCCL over TCP sockets, emulated on the native fabric with the same chips and batches.

limits concurrency: at 16K the Galaxy serves 92 active sequences against 32 on the GPU server and reaches 429K tokens per dollar against 192K. Qwen3-32B BF16 gains 1.24-1.54× and Llama-3.1-8B BF16 1.21-1.29×. Within a series the gain grows with context for the two larger models, from 1.08× at 8K to 1.37× at 256K for 70B 8-bit and from 1.24× to 1.54× for 32B BF16, as longer contexts leave the GPU server fewer sequences in memory to batch at each decode step than the Galaxy keeps in flight. Together, these results show that the pooled chips turn their larger pooled memory, bandwidth, and compute into more tokens per dollar at every model, precision, and context length. Tokens per joule. The Galaxy also delivers more tokens per joule in all 36 configurations, by 1.00-2.27×, with the same ordering across series: 1.74-2.27× for 70B BF16, 1.15-1.55× for 32B BF16, 1.21-1.40× for 8B BF16, and 1.00-1.35×, 1.05-1.18×, and 1.21-1.26× for the three 8-bit series. The one configuration at parity is 70B 8-bit at 8K, where 520 active sequences on the 32 Galaxy chips and 324 on the eight GPUs produce 1.270 and 1.269 tokens per joule. For instance, 70B BF16 at 16K yields 0.333 tokens per joule on the Galaxy and 0.147 on the GPU server, and 8B 8-bit at 32K yields 1.786 against 1.446. Together, the two metrics show that the lower-end cluster, pooled over its own links,

beats the high-end server on both measures a provisioner pays for, the native half of the claim of §1. One chip against one card. One Galaxy chip stays below one RTX PRO 6000 Blackwell on both metrics in every configuration that fits (Figure 8). The chip serving Llama3.1-8B holds 0.21-0.25× the sequences of the card and generates 0.13-0.17× its tokens per second, and with the price of each system divided by its number of devices it reaches 0.73-0.94× the card’s tokens per dollar and 0.740.93× its tokens per joule in the 11 configurations that fit, and runs out of memory at 256K in BF16. Together, these results show that the lower-end device loses on device efficiency while the pooled chips of Figure 7 win on cluster efficiency, so the gain comes from pooling over the fast links, not from the chips on their own. KV-cache capacity. Pooling multiplies the KV-cache capacity of the chips faster than that of the cards and keeps it cheaper per token at every pool size (Table 6). One Galaxy chip holds 135K tokens of Llama-3.1-8B context in BF16 and 349K in 8-bit, 0.24-0.28× one RTX PRO 6000 Blackwell, because the weights take 16 GB of its 32 GB in BF16 and 8 GB in 8-bit. Pooling spreads the weights: a model replica on eight chips holds 1.93M tokens in BF16 and

Introducing Cluster Efficiency to LLM Serving

8 16 32

0.75×

0.74×

64 128256

0.78×

0.88 0.66 0.44 0.22 0

0.74×

1.2M 889K 593K 296K 0

0.74×

0.78×

0.74×

0.73× 0.79×

6 4.5 3 1.5 0

8-bit

8 16 32 0.85×

OOM

64 128256

8.7M 6.5M 4.4M 2.2M 0

0.85×

0.94× 0.89×

64 128256 0.88×

0.42 0.32 0.21 0.11 0

8-bit

OOM

0.90×

8 16 32

543K 407K 271K 136K 0

0.83×

0.94×

0.94× 0.89× 0.93×

3.1 2.3 1.6 0.78 0

BF16

8 16 32

0.84×

Tokens/J

Tokens/$

BF16 4.3M 3.2M 2.1M 1.1M 0

sequences. Together, these results show that with fast links the Galaxy keeps gaining from every chip added to a replica, in tokens per second per chip and in tokens per joule.

One Galaxy chip

0.84×

One RTX PRO 6000 Blackwell

64 128256

Table 7. Throughput per chip and tokens per joule of one Llama3.1-8B replica on one, two, four, and eight Galaxy chips at 32K context; the best value in each column is bold. BF16 8-bit Chips tok/s per chip tok/J tok/s per chip tok/J 1 47.4 0.62 80.6 1.14 2 57.0 0.83 116.5 1.65 4 64.6 0.97 129.4 1.78 8 69.9 1.01 131.9 1.80

Context length (K tokens)

6.3 Figure 8. Tokens per dollar and tokens per joule of one Galaxy chip and one RTX PRO 6000 Blackwell serving Llama-3.1-8B in BF16 and 8-bit over 8K-256K contexts, each at its fastest batch. Tokens per dollar divides each system’s price by its device count.

3.73M in 8-bit, 14.3× and 10.7× one chip, while a model replica on eight cards holds 5.57M and 11.2M, 9.7× and 9.0× one card. Per dollar of acquisition price, with the price of each system divided by its device count, the chips hold 27-48 KV tokens in BF16 and 70-93 in 8-bit against 20-25 and 45-50 for the cards. Together, these results show that the chips hold 1.3-1.9× more context per dollar than the cards, and that the gap widens with pool size. Table 6. KV-cache capacity of one Llama-3.1-8B replica on one, two, four, and eight Galaxy chips and RTX PRO 6000 Blackwell cards, in tokens and in tokens per dollar of acquisition price, with the price of each system divided by its device count; the better of chips and cards at each device count is set in bold. BF16 8-bit KV tokens KV tokens/$ KV tokens KV tokens/$ Devices chips cards chips cards chips cards chips cards 1 135K 572K 27 20 349K 1.25M 70 45 2 383K 1.29M 38 23 832K 2.70M 83 48 4 899K 2.72M 45 24 1.79M 5.55M 90 49 8 1.93M 5.57M 48 25 3.73M 11.24M 93 50

Scaling with pool size. Over fast links, every chip added to a replica raises the Galaxy’s efficiency per chip as well as per joule (Table 7). For one replica of Llama-3.1-8B at 32K context, throughput per chip rises monotonically from 81 to 132 tokens per second in 8-bit and from 47 to 70 in BF16 as the replica grows from one chip to eight, 1.6× and 1.5×, and tokens per joule from 1.14 to 1.80 and from 0.62 to 1.01. Pooling the chips’ memory, bandwidth, and compute does three things at once: the weights’ share of each chip’s memory shrinks, so a larger part of the pool goes to KV cache, which long-context agent serving needs for the resident sessions whose prefixes it reuses (Zhu et al., 2026; Zheng et al., 2024; Qin et al., 2025); the memory I/O and compute available to the replica grow with the chip count; and the memory-bound decode step batches more

Communication-Link Ablation (RQ2)

Throughput loss. Connecting the same chips over TCPclass links cuts their throughput by 23.5-67.4% across the 36 Galaxy configurations, 47.3% on average, with the hardware capacity, the batches, and the acquisition cost unchanged. The 70B and 32B models lose 50.4-67.4% and the 8B model 23.5-32.6%; the larger models issue 257-484 collectives per decode step against 130-131 for 8B, so the added startup latency is paid more often for every token generated. Together, these results show that the link cost alone, with nothing else changed, takes away between a quarter and two thirds of the throughput that the pooled chips deliver. Efficiency. The loss erases the native advantage: over TCPclass links the Galaxy falls below the GPU server in tokens per dollar in every one of the 36 configurations and in tokens per joule in 31 of the 36 (Figure 7). The ratios are 0.360.98× and 0.39-1.07×; the five points that stay at or above parity are 8B and 70B BF16 configurations where the loss is smallest and the pool draws less power while it waits on the links. For instance, 70B BF16 at 16K drops from 429K to 184K tokens per dollar, 0.96× the GPU server, and 70B 8-bit at 8K from 1.90M to 0.62M, 0.36×. Together, these results show that, since nothing but the link cost changes between the two Galaxy bars of Figure 7, the fast links are what let the lower-end chips keep their per-dollar and per-joule advantage once pooled, as §1 claims.

7

C ONCLUSION

This paper asks how an LLM serving cluster should be built when the goal is cluster efficiency, the tokens it serves per dollar and per joule, rather than device efficiency. Across three models, two precisions, and contexts from 8K to 256K, the Galaxy delivers more tokens per dollar and more tokens per joule than an RTX PRO 6000 Blackwell server in all 36 configurations, by 1.07-2.24× and 1.00-2.27×, while a single Galaxy chip stays below a single GPU on both metrics. Connecting the same chips over TCP-class links cuts their throughput by 23.5-67.4% and erases all 36 tokensper-dollar wins and 31 of the 36 tokens-per-joule wins. The

Introducing Cluster Efficiency to LLM Serving

advantage of a lower-end cluster comes from pooling over fast links: start from economical accelerators, connect them with fast links, and judge the cluster by what it serves per dollar and per joule rather than by any one device.

CG Channel. Nvidia Announces Eight New Professional Ampere GPUs, 2021. URL https://www.cgchan nel.com/2021/04/nvidia-announces-eig ht-new-professional-ampere-gpus/. April 2021; accessed 2026-10-29.

R EFERENCES

Choquette, J. NVIDIA hopper H100 GPU: Scaling performance. IEEE Micro, 43(3):9–17, 2023. doi: 10.1109/ MM.2023.3256796. URL https://doi.org/10.1 109/MM.2023.3256796.

Abts, D., Kimmell, G., Ling, A., Kim, J., Boyd, M., Bitar, A., Parmar, S., Ahmed, I., DiCecco, R., Han, D., Thompson, J., Bye, M., Hwang, J., Fowers, J., Lillian, P., Murthy, A., Mehtabuddin, E., Tekur, C., Sohmers, T., Kang, K., Maresh, S., and Ross, J. A software-defined tensor streaming multiprocessor for large-scale machine learning. In Proceedings of the 49th Annual International Symposium on Computer Architecture (ISCA ’22), pp. 567–580. ACM, 2022. doi: 10.1145/3470496.3527405. URL ht tps://doi.org/10.1145/3470496.3527405. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 117–134. USENIX Association, 2024. URL https://www.usenix.org/conference/os di24/presentation/agrawal. Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., and He, Y. DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–15. IEEE, 2022. doi: 10.1109/SC41404.2022.00051. URL https://doi.org/10.1109/SC41404.2022 .00051. Barroso, L. A., Hölzle, U., and Ranganathan, P. The Datacenter as a Computer: Designing Warehouse-Scale Machines. Synthesis Lectures on Computer Architecture. Morgan & Claypool, third edition, 2018. Also published by Springer, doi:10.1007/978-3-031-01761-2. Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., and Raffel, C. Petals: Collaborative Inference and Fine-tuning of Large Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 558–568. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023 .acl-demo.54. Bytehaus. AI servers. https://www.bytehaus.co. uk/pages/ai-servers, 2026. Accessed 2026-1030.

CNBC. Nvidia Shares Close Up After Company Unveils Latest AI Chips, 2024. URL https://www.nbcwas hington.com/news/business/money-repor t/nvidia-stock-dips-after-company-unv eils-latest-ai-chips/3570332/. March 19, 2024; accessed 2026-10-30. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva,

Introducing Cluster Efficiency to LLM Serving

R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., De Paola, B., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Zhang, G. J., Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari,

M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Wang, Y. S., Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The Llama 3 Herd of Models, 2024. URL https://arxiv.org/abs/2407.21783. Griggs, T., Liu, X., Yu, J., Kim, D., Chiang, W.-L., Cheung, A., and Stoica, I. Mélange: Cost efficient large language model serving by exploiting GPU heterogeneity, 2024. URL https://arxiv.org/abs/2404.14527. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips. cc/paper_files/paper/2019/hash/093f6 5e080a295f8076b1c5722a46aa2-Abstract. html. Jarvislabs. NVIDIA H200 GPU Price (October 2026): $4.59/hr, Spot $2.29, 2026. URL https://jarvis labs.ai/blog/h200-price. Published January 8, 2026, updated October 20, 2026; accessed 2026-10-30. Jiang, Y., Li, W., Peng, Y., Zhang, J., Yan, R., Chen, J., Han, X., Fu, F., and Yuan, B. Hexgen-3: A fully disaggregated llm serving framework with fine-grained heterogeneous resource autoscaling. In Forty-third International Conference on Machine Learning.

Introducing Cluster Efficiency to LLM Serving

Jiang, Y., Yan, R., Yao, X., Zhou, Y., Chen, B., and Yuan, B. Hexgen: Generative inference of large language model over heterogeneous environment. In ICML’24: International Conference on Machine Learning, 2023.

Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611– 626, 2023. doi: 10.1145/3600006.3613165. URL https://arxiv.org/abs/2309.06180.

Jiang, Y., Fu, F., Yao, X., He, G., Miao, X., Klimovic, A., Cui, B., Yuan, B., and Yoneki, E. Demystifying costefficiency in llm serving over heterogeneous gpus. In ICML’25: International Conference on Machine Learning, 2025a.

Lenovo Press. ThinkSystem NVIDIA HGX B200 180GB 1000W GPU. Product guide LP2226, 2026. URL https: //lenovopress.lenovo.com/lp2226-thi nksystem-nvidia-b200-180gb-1000w-gpu. Updated 2026-01-13; accessed 2026-10-29.

Jiang, Y., Fu, F., Yao, X., Wang, T., Cui, B., Klimovic, A., and Yoneki, E. Thunderserve: High-performance and cost-efficient llm serving in cloud environments. Proceedings of Machine Learning and Systems, 7, 2025b.

Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021. URL https: //openreview.net/forum?id=qrwe7XHTmYb.

Jiang, Y., Yan, R., and Yuan, B. Hexgen-2: Disaggregated generative inference of llms in heterogeneous environment. In ICLR’25: International Conference on Learning Representations, 2025c. Jiang, Y., Fu, F., and Yoneki, E. Boute: Cost-efficient llm serving with heterogeneous llms and gpus via multiobjective bayesian optimization. MLSys’26: Proceedings of Machine Learning and Systems, 2026. Jouppi, N., Kurian, G., Li, S., Ma, P., Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., Young, C., Zhou, X., Zhou, Z., and Patterson, D. A. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA ’23), pp. 1–14. ACM, 2023. doi: 10.1145/3579371.3589350. URL https: //doi.org/10.1145/3579371.3589350. Jouppi, N. P., Yoon, D. H., Ashcraft, M., Gottscho, M., Jablin, T. B., Kurian, G., Laudon, J., Li, S., Ma, P., Ma, X., Norrie, T., Patil, N., Prasad, S., Young, C., Zhou, Z., and Patterson, D. Ten lessons from three generations shaped Google’s TPUv4i: Industrial product. In Proceedings of the 48th Annual International Symposium on Computer Architecture (ISCA ’21), pp. 1–14. IEEE, 2021. doi: 10.1 109/ISCA52012.2021.00010. URL https://doi.or g/10.1109/ISCA52012.2021.00010. Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. In Proceedings of Machine Learning and Systems, volume 5, pp. 341–353, 2023. URL https://proceedings. mlsys.org/paper_files/paper/2023/has h/80083951326cf5b35e5100260d64ed81-A bstract-mlsys2023.html. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model

Lie, S. Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning. IEEE Micro, 43(3):18–30, 2023. doi: 10.1109/MM.2023.3256 384. URL https://doi.org/10.1109/MM.202 3.3256384. Lim, J., Shin, D., Kim, U., Choi, J., Kim, K. M., Shin, S., Park, S., Koo, G., and Suh, T. Performance characterization and optimization of LLM inference on Tenstorrent AI accelerators. In Euro-Par 2026: Parallel Processing, volume 16782 of Lecture Notes in Computer Science, pp. 466–479. Springer, 2027. doi: 10.1007/978-3-032-35251 -4 32. URL https://doi.org/10.1007/978-3 -032-35251-4_32. First published online August 15, 2026. Liu, H., Zaharia, M., and Abbeel, P. RingAttention with blockwise transformers for near-infinite context. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum ?id=WsRHpHH4s0. Mei, Y., Zhuang, Y., Miao, X., Yang, J., Jia, Z., and Vinayak, R. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via MaxFlow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 586– 602, 2025. doi: 10.1145/3669940.3707215. URL https://arxiv.org/abs/2406.01566. Miao, X., Shi, C., Duan, J., Xi, X., Lin, D., Cui, B., and Jia, Z. SpotServe: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24), pp. 1112–1127. ACM, 2024. doi: 10.1 145/3620665.3640411. URL https://doi.org/10 .1145/3620665.3640411.

Introducing Cluster Efficiency to LLM Serving

Modal. How Much Does It Cost to Run NVIDIA L40S GPUs in 2025?, 2025. URL https://modal.com/ blog/nvidia-l40s-price-article. Accessed 2026-10-29. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’21), pp. 1–15. ACM, 2021. doi: 10.1145/3458817.3476209. URL ht tps://doi.org/10.1145/3458817.3476209.

rchitecture.pdf. Appendix A, Table 3; accessed 2026-10-29. NVIDIA. NVIDIA RTX PRO 6000 Blackwell Server Edition GPU Accelerator Product Brief, 2025c. URL https://dam-cdn.nvd.orangelogic.com /AssetLink/3km2720jiy76r06ctf8mxg217 43p1058.pdf. SP-12355-001 v02, June 27, 2025; accessed October 28, 2026. NVIDIA. NVIDIA management library (NVML) API reference guide. https://docs.nvidia.com/de ploy/nvml-api/, 2026. Last updated September 9, 2026; accessed 2026-10-30.

NVIDIA. NVIDIA RTX A5000 Datasheet, 2021. URL https://www.nvidia.com/content/dam/e n-zz/zh_tw/Solutions/design-visualiza tion/quadro/resource/proviz-print-nvi dia-rtx-a5000-datasheet-a4-nvidia-1 609472-r8-web-zhTW-1724048.pdf. Accessed 2026-10-29.

Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Íñigo., Maleki, S., and Bianchini, R. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132, 2024. doi: 10.1109/ISCA59077.2024.00019. URL https://arxiv.org/abs/2311.18677.

NVIDIA. NVIDIA Delivers Quantum Leap in Performance, Introduces New Era of Neural Rendering With GeForce RTX 40 Series. Press release, 2022. URL https: //nvidianews.nvidia.com/news/nvidia-d elivers-quantum-leap-in-performance-i ntroduces-new-era-of-neural-rendering -with-geforce-rtx-40-series. September 20, 2022; accessed 2026-10-29.

Peng, H., Davidson, S., Shi, R., Song, S. L., and Taylor, M. Chiplet cloud: Building AI supercomputers for serving large generative language models, 2023. URL https: //arxiv.org/abs/2307.02666.

NVIDIA. NVIDIA H100 Tensor Core GPU. Product specifications, 2023a. URL https://www.nvidia.com /en-us/data-center/h100/. Accessed 2026-1029.

Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, volume 5, pp. 606–624, 2023. URL https://proceedings.mlsys.org/pa per_files/paper/2023/hash/c4be71ab8d 24cdfb45e3d06dbfca2780-Abstract-mlsys 2023.html.

NVIDIA. NVIDIA L40S GPU. Product specifications, 2023b. URL https://www.nvidia.com/en-u s/data-center/l40s/. Accessed 2026-10-29. NVIDIA. NVIDIA H200 Tensor Core GPU. Product specifications, 2024. URL https://www.nvidia.com /h200. Accessed 2026-10-29. NVIDIA. NVIDIA Blackwell GeForce RTX 50 Series Opens New World of AI Computer Graphics. Press release, 2025a. URL https://nvidianews.nvi dia.com/news/nvidia-blackwell-geforce -rtx-50-series-opens-new-world-of-a i-computer-graphics. January 6, 2025; accessed 2026-10-29. NVIDIA. NVIDIA RTX Blackwell GPU Architecture. Whitepaper, 2025b. URL https://images.nvi dia.com/aem-dam/Solutions/geforce/b lackwell/nvidia-rtx-blackwell-gpu-a

Peng*, Y., Jiang*, Y., Jiang, W., Wang, C., and Yuan, B. Hexgen-flow: Optimizing llm inference request scheduling for agentic text-to-sql. In ICDE’26: International Conference on Data Engineering, 2025.

Punch Technology. 8 x NVIDIA RTX PRO 6000 Blackwell GPU server. https://www.punchtechnology. co.uk/gpu/8-x-nvidia-rtx-pro-6000-b lackwell, 2026. Accessed 2026-10-30. Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading More Storage for Less Computation—A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pp. 155–170. USENIX Association, 2025. URL https://www.us enix.org/conference/fast25/presentat ion/qin.

Introducing Cluster Efficiency to LLM Serving

Rhee, M., Sim, J., Ahn, T., Lee, S., Yoon, D., Kim, E., Park, K., Joo, Y., and Kim, H. HPU: High-bandwidth processing unit for scalable, cost-effective LLM inference via GPU co-processing. arXiv preprint arXiv:2504.16112, 2025. URL https://arxiv.org/abs/2504.1 6112. Version 2. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094–31116. PMLR, 2023. URL https://proc eedings.mlr.press/v202/sheng23a.html. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. URL https://arxiv.org/abs/1909.0 8053. Stojkovic, J., Zhang, C., Goiri, Íñigo., Torrellas, J., and Choukse, E. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 1348–1362, 2025. doi: 10.1109/HPCA61900.2025.00102. URL https: //iacoma.cs.uiuc.edu/iacoma-papers/hp ca25_2.pdf. Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., and Lin, W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 173–191. USENIX Association, 2024. URL https://www.usenix.org/conference/os di24/presentation/sun-biao. Tenstorrent. Advanced performance optimizations for models. TT-Metalium technical report, 2025. URL https://github.com/tenstorrent/t t- metal/blob/978f93b56e36b68318d1 1ecf7b101085a81d718e/tech_reports/ AdvancedPerformanceOptimizationsFo rModels/AdvancedPerformanceOptim izationsForModels.md. Repository revision 978f93b56e36b68318d11ecf7b101085a81d718e, 202610-13; accessed 2026-10-30.

93b56e36b68318d11ecf7b101085a81d71 8e/tech_reports/CCLs/CCL_Performan ce_Best_Practices.md. Repository revision 978f93b56e36b68318d11ecf7b101085a81d718e, 202610-13. Tenstorrent. Tenstorrent Galaxy: Scalable AI-Compute Server. Product specifications, 2026c. URL https: //tenstorrent.com/hardware/galaxy. Accessed 2026-10-28. Tenstorrent. TT Inference Server, version 0.25.0. GitHub repository tenstorrent/tt-inference-server, 2026d. URL https://github.com/tenstorrent/tt-inf erence-server. Accessed 2026-10-29. Tenstorrent. TT-Metalium and TT-NN: the tenstorrent ai kernel and operator libraries. GitHub repository tenstorrent/tt-metal, revision 978f93b56e36, 2026e. URL https://github.com/tenstorrent/tt-met al. Accessed 2026-10-29. Tenstorrent. TT-SMI: Tenstorrent system management interface. https://github.com/tenstorrent/t t-smi, 2026. Accessed 2026-10-30. Tong*, C., Jiang*, Y., Chen, G., Zhao, T., Lu, S., Qu, W., Yang, E., Ai, L., and Yuan, B. Parallax: Efficient llm inference service over decentralized environment. arXiv preprint arXiv:2509.26182, 2025. TT-Metalium Scale-Out Team. TT-Fabric Architecture Specification. TT-Metalium technical report, version 1.0, revision 2.0, 2026. URL https://github .com/tenstorrent/tt- metal/blob/97 8f93b56e36b68318d11ecf7b101085a81d 718e/tech_reports/TT- Fabric/TT- Fab ric- Architecture.md. Repository revision 978f93b56e36b68318d11ecf7b101085a81d718e, 202610-13.

Tenstorrent. Cards. Product specifications, Blackhole p150 series, 2026a. URL https://tenstorrent.com/ hardware/cards. Accessed 2026-10-30.

Vasiljevic, J., Bajic, L., Capalija, D., Sokorac, S., Ignjatovic, D., Bajic, L., Trajkovic, M., Hamer, I., Matosevic, I., Cejkov, A., Aydonat, U., Zhou, T., Gilani, S. Z., Paiva, A., Chu, J., Maksimovic, D., Chin, S. A., Moudallal, Z., Rakhmati, A., Nijjar, S., Bhullar, A., Drazic, B., Lee, C., Sun, J., Kwong, K.-M., Connolly, J., Dooley, M., Farooq, H., Chen, J. Y. T., Walker, M., Dabiri, K., Mabee, K., Lal, R. S., Rajatheva, N., Retnamma, R., Karodi, S., Rosen, D., Munoz, E., Lewycky, A., Knezevic, A., Kim, R., Rui, A., Drouillard, A., and Thompson, D. Compute substrate for software 2.0. IEEE Micro, 41(2):50–55, 2021. doi: 10.1109/MM.2021.3061912. URL https: //doi.org/10.1109/MM.2021.3061912.

Tenstorrent. CCL Performance Tuning Tips. TT-Metalium technical report, 2026b. URL https://github .com/tenstorrent/tt-metal/blob/978f

Wei, Z., Wang, Y., Yen, J., Xia, M., and Qi, Z. HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators, 2026.

Introducing Cluster Efficiency to LLM Serving

URL https://arxiv.org/abs/2606.29986. Preprint.

rg/conference/osdi22/presentation/zh eng-lianmin.

Yan*, R., Jiang*, Y., Nie, X., Fu, F., Cui, B., and Yuan, B. Hexiscale: Facilitating large language model training over heterogeneous hardware. In Ninth Conference on Machine Learning and Systems.

Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pp. 62557– 62583. Curran Associates, Inc., 2024. URL https:// proceedings.neurips.cc/paper_files/p aper/2024/hash/724be4472168f31ba1c9a c630f15dec8-Abstract-Conference.html.

Yan*, R., Jiang*, Y., Wu, T., Gao, J., Mei, Z., Fu, W., Mai, H., Wang, W., Wu, Y., and Yuan, B. Areal-hex: Accommodating asynchronous rl training over heterogeneous gpus. arXiv preprint arXiv:2511.00796, 2025. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 Technical Report, 2025. URL https: //arxiv.org/abs/2505.09388. Yao, X., Jiang, Y., Badanin, I., Hu, Q., Smith, R. M., Yuan, B., Schlag, I., Yoneki, E., and Klimovic, A. {OpenTela}: Unifying decentralized computing resources for heterogeneous {LLM} serving (operational systems). In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 1821–1838, 2026. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.G. Orca: A distributed serving system for TransformerBased generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538. USENIX Association, 2022. URL https://www.usenix.org/conference/os di22/presentation/yu. Zhang, Y., Shen, H., Yang, R., Tian, D., Luo, Y., Zhang, M., Li, L., Hu, C., Wo, T., Song, C., and Ouyang, J. Cauchy: A Cost-Efficient LLM Serving System through Adaptive Heterogeneous Deployment. In Proceedings of the 2025 ACM Symposium on Cloud Computing, pp. 881–893, 2025. doi: 10.1145/3772052.3772264. URL https://doi.org/10.1145/3772052.3772 264. Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., Gonzalez, J. E., and Stoica, I. Alpa: Automating inter- and IntraOperator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 559–578. USENIX Association, 2022. URL https://www.usenix.o

Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210. USENIX Association, 2024. URL https://www.usenix.org/conference/os di24/presentation/zhong-yinmin. Zhu, K., Jacob, M., Ma, C., Pan, Y., Wang, S., Krishnamurthy, A., and Kasikci, B. TraceLab: Characterizing Coding Agent Workloads for LLM Serving, 2026. URL https://arxiv.org/abs/2606.30560.

Related documents

• FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training#267627
• Just Talk Once: Communication-Efficient Split Federated LLM Fine-Tuning on Edge Devices#821496
• TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching#1035140
• Persistent Internal Standpoints in Artificial Systems: Part 9 - Large Language Models and the Absence of Standpoint#102320
• FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon#973681