arXiv:2604.21072v1 [cs.DC] 22 Apr 2026
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization Jiu Chen∗
Shuangyan Yang∗
Xu Xiong∗
University of California Merced USA
University of California Merced USA
University of California Merced USA
Hexiao Duan
Xinran Zhang
Jie Ren
University of California Merced USA
University of California Berkeley USA
College of William & Mary USA
Dong Li University of California Merced USA
Abstract
A decentralized approach to LLM inference [3, 20, 24, 38] offers a compelling alternative by distributing computation across heterogeneous, independently operated nodes, enabling elastic capacity and improving robustness against single-point power outages. Together, these factors make decentralized AI not merely an architectural choice but a necessary evolution for building scalable, democratic, and highly-accessible AI systems [18, 21]. At the same time, the growing heterogeneity and fragmentation of compute resources across modern data centers and personal devices create an opportunity to enable the decentralized approach. Contemporary cloud environments routinely exhibit substantial pockets of underutilized GPUs and stranding effects caused by placement constraints [6, 28]. In parallel, consumer hardware—ranging from high-end laptops to gaming GPUs (e.g., NVIDIA RTX 5090) —has become powerful enough to execute meaningful slices of LLM inference. By aggregating idle capacity, exploiting locality, and dynamically routing computation across a diverse pool of contributors, decentralized LLM inference transforms what is currently wasted or siloed compute into a scalable, elastic, and cost-effective inference fabric. Decentralized LLM inference distributes transformer workloads across the public internet rather than within a data center’s high-bandwidth fabric. Residential contributors in decentralized settings often operate at only 20–500 Mbps [12], making cross-node communication 500×–2000× slower than typical intra-cluster links. As a result, network transfer—not heterogeneous computation—becomes the primary bottleneck: for example, sending a 5 MB activation tensor takes 25–200 ms on typical home uplinks, far exceeding the 1–3 ms compute time of a transformer block on a desktop-level GPU. This stark bandwidth gap underscores the importance of shifting the performance optimization target to be aggressively communication centric.
Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to lowbandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76×. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced1 .
1
Introduction
The rapid proliferation of large language models (LLM) has intensified the demand for scalable and performant inference infrastructures. However, the increasingly high computational requirements and large model sizes make it challenging to serve them in a cheap and efficient manner. Centralized AI services — dominated by hyperscale providers—create structural bottlenecks [2, 35]: they impose high operational cost [16], introduce single points of failure [1, 5], and fragmented GPU resources across data centers [6, 13, 28, 42]. As model sizes grow and applications diversify, these limitations become untenable. ∗ Equal contribution 1 https://github.com/ai-decentralized/BloomBee
1
Trovato et al.
In this paper, we introduce a framework (named BloomBee) and study how to optimize throughput of distributed inference of LLM at internet scales (characterized with low network bandwidth and heterogeneous compute), with a focus on communication optimizations. The communication overhead has two dimensions: (1) the number of inter-node hops in the inference pipeline and (2) the volume of data each hop carries. The two dimensions are entangled, creating challenges to reduce or hide the communication overhead. In particular, in BloomBee, we explore three optimization techniques: layer assignment, microbatching, and tensor offloading. The layer assignment decides the placement of transformer layers across geo-distributed GPUs, which impacts the overhead (1); the micro-batch decomposes the input batch to smaller batches for scheduling, such that we can overlap communication and computation, which impacts the overhead (2); the tensor offloading uses CPU memory to hold GPU memory spill such that we can use less GPU, hence impacting the overhead (1). The effectiveness of the three techniques interacts with each other because of GPU memory capacity constraint and I/O overhead for tensor offloading (§5). As a result, coordinating the three techniques to maximize inference throughput is challenging. BloomBee addresses this problem by formulating the coordination of the three techniques as an optimization problem, and solves it using dynamic programming. Furthermore, the formulation considers the heterogeneity of GPU and network, and the solving process is lightweight because of the constrained formulation. On top of the synergies of the three techniques, we further push up inference throughput by using lossless compression and speculative decoding. The lossless compression reduces the volume of activations communicated across the internet without impacting the inference accuracy. The existing lossless compression methods either lose floating point structures (e.g., exponent and mantissa) [7, 8] or focus on LLM weights [15] (not activations). We characterize the values of activations, and reveal that by re-organizing data layout of activations, we can expose more repeated patterns for the compression algorithm to tap and hence improve efficiency. Speculative decoding [4, 23, 25] (SD) has been employed in centralized environments to improve the throughput of LLM inference. SD uses a lightweight draft-model to propose candidate tokens and a heavyweight target-model to parallelize the verification of candidate tokens. However, when using SD in a decentralized environment, we face a challenge from slow interconnect: the candidate tokens, when transferring from the draft model to the target model across the internet, substantially amplifies inter-node communication volume, causing SD to yield negative throughput gains compared to standard autoregressive decoding. To make SD useful in the decentralized environment, we introduce a learned classifier placed right before the data communication. The classifier 2
efficiently prunes the candidate tokens to reduce the communication volume while effectively maintaining the token acceptance rate at the target model. We evaluate BloomBee with multiple network environments with diverse bandwidth using a series of LLM models. Compared to the baselines of state-of-the-art decentralized LLM inference systems (i.e., Helix [24] and Petals [3]), BloomBee improves service throughput by up to 1.76× while reducing average latency by up to 43.20%. In summary, we make the following contributions. • We recognize the communication as the major bottleneck for building performant LLM inference at internet scales, and introduce a framework with multi-dimensional communication optimization. • We use a spectrum of techniques to reduce or hide communication overhead in decentralized environment; we identify their interactions and formulate them into an optimization problem to maximize inference throughput. • We customize lossless compression and SD to accommodate unique features of communicating tensors and decentralized environment.
2
Background
Decentralized AI. Across geo-distributed GPUs, the pipeline parallelism is commonly employed in decentralized AI [3, 9, 10, 19, 20, 24, 30, 31, 34, 38]. This means that the LLM is partitioned across GPUs, each of which contains a group of consecutive layers of the LLM. During inference, the inference request proceeds stage by stage, and only the activations at the stage boundaries are transferred between consecutive stages. The pipeline parallelism is a practical approach to enable decentralized AI, because the communication to transfer activation tensors across two pipeline stages is point-to-point, and the communication volume is much smaller (compared with tensor model parallelism [26], another common parallelism scheme for distributed inference), and can be managed to be 𝑂 (100𝐾𝐵) across two stages. Tensor model parallelism is not suited for the internet-scale LLM inference, because it requires collective communications, such as AllReduce, within transformer layers [33]. Such communication patterns are sensitive to latency and bandwidth, which in practice confines tensor model parallelism to single-node or other tightly coupled environments. In the setting of decentralized AI, given the long latency at the internet scale, we focus on token throughput (not latency) and aim to serve offline LLM inference. There are many use cases of offline LLM inference, such as batch processing large document sets [47], automated report generation [40], local photo captioning or organization [27], and offline creative writing assistant [41]. For those use cases, the decentralized AI provides a cost-effective solution with maximized throughput and reasonable latency [43].
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
Tree-based speculative decoding (SD). SD [4] is an acceleration technique for LLM in which a lightweight draft model rapidly proposes a short sequence of candidate next tokens—often accompanied by their probability distributions—and passes these proposed continuations to a larger, more accurate target model for verification. The target model evaluates the draft tokens in parallel, accepting those that fall within its own high-probability predictions and rejecting or correcting any that diverge. By allowing the target model to skip many decoding steps while still ensuring that the final output adheres to its distribution, SD substantially improves throughput without compromising output quality. Tree-based SD [25] extends SD by organizing draft tokens into a tree structure, where each path represents a candidate continuation. The target model verifies all paths in a single forward pass via tree attention masking, substantially increasing the hypothesis space evaluated per step.
3
Time per Decode Step (ms)
120 100 80 60
GPU Compute GPU CPU CPU NIC NIC NIC NIC CPU CPU GPU
10.06
40 20 0
0.74 0.40
0.45 0.09 0.62 2.77
0.24 2.09
41.45
W1 (Intra-DC)
W2 (Cross-Region)
94.4
44.2 0
20
40
60
80
100
Throughput (tok/s)
416.4 416.4
W1 W2
57.91
42.44
W1 W2
0
100
W1 10.8 W2 0
200
300
400
500
60
80
100
Communication Volume (KB)
82.9 20
40
End-to-End Net Latency (ms)
(a) Per-step decoding-time break- (b) Throughput, communication down. volume, and end-to-end latency.
Figure 1. Performance comparison between W1 and W2. 76.5 Mbps from NC to PA. To make the comparison fair, we use Petals for W1 too. Figure 1 summarizes the communication behavior under W1 and W2. We break it down to GPU–CPU communication time (𝑇𝐺𝑃𝑈 ↔𝐶𝑃𝑈 ), CPU–NIC staging time (𝑇𝐶𝑃𝑈 ↔𝑁 𝐼𝐶 ), NIC– NIC communication time (𝑇𝑁 𝐼𝐶↔𝑁 𝐼𝐶 ) exposed to the critical path, and per-stage GPU compute time. We have the following observations. Observation 1. Inter-site communication becomes the dominant component on the critical path once execution moves to the internet scale. With the same communication volume (416 KB per pipeline stage), 𝑇𝑁 𝐼𝐶→𝑁 𝐼𝐶 rises from 10.06 ms in W1 to 57.91 ms in W2. In W1, The communication is a minor cost (17.8% of inference time); in W2, it reaches 1.4× GPU compute time (56.6% of inference time). Observation 2. The local staging overhead is small, compared to the communication time over the internet. In W2, the host-mediated components are limited:𝑇𝐺𝑃𝑈 →𝐶𝑃𝑈 , 𝑇𝐶𝑃𝑈 →𝑁 𝐼𝐶 , 𝑇𝑁 𝐼𝐶→𝐶𝑃𝑈 , and 𝑇𝐶𝑃𝑈 →𝐺𝑃𝑈 are 2.09 ms, 0.24 ms, 0.40 ms, and 0.24 ms, respectively. Their sum is only 2.97 ms, far below the 57.91 ms spent on 𝑇𝑁 𝐼𝐶→𝑁 𝐼𝐶 . The main bottleneck is not local movement among GPU, CPU, and NIC. It is the transfer of activations across internet sites. The above observations point to communication optimization rather than tuning of local computation as the main goal for improving internet-scale LLM inference. Since the communication dominates inference time, any computation (e.g., compression) that can save communication volume would be valuable. Observation 2 rules out local GPU–CPU and CPU–NIC staging as performance optimization targets.
Motivation
Analysis on communication data path. The communication data path at the internet scale across GPUs is fundamentally different from that at the data center scale. At the data center scale, inter-GPU communication across nodes can rely on GPUDirect RDMA. In particular, the data from a GPU memory is first moved through the local GPU interconnect (e.g., NVLink and PCIe) to the network interface. From there, it is handed off to a network adapter-using RDMA-capable technologies (e.g., Infiniband or RoCE) which transports data across data-center fabric with minimum CPU involvement. On the receiving node, the network interface card (NIC) delivers payloads directly into target GPU’s memory space, enabling fast, low-latency communication. In the decentralized environment, the end-to-end GPUDirect RDMA is rarely possible across the open internet because (1) RDMA requires a lossless fabric, but internet routing introduces congestion, packet loss, and variable Maximum Transmission Unit (MTU), which conflict with RDMA; and (2) the security layers (TLS, IPsec, and QUIC) needed by internet are on the CPU (not on NIC or GPU). So in real-world geo-distributed GPU communication, CPU memory must be involved, at least for packetization, encryption, and transport handling, even if the GPU-side data movement is optimized. Communication performance. We examine the role of communication in internet-scale LLM inference. We use LLaMA-30B with batch size 32 and sequence length 128. The model is evenly partitioned across three nodes, each with 20 layers on an NVIDIA RTX 5090 GPU, using pipeline parallelism. We use two environments: (1) W1: a data center connected by 45 Gbps Ethernet; and (2) W2: three nodes distributed in Maryland (MD), North Carolina (NC), and Pennsylvania (PA). We use Petals [3] (a framework for internetscale AI) for W2. Along the forward pipeline path, the bandwidth of inter-stage links is 331.0 Mbps from MD to NC and
4
BloomBee Design Overview
Driven by the observations, BloomBee adopts a communicationcentric design that jointly optimizes three factors: the number of inter-node hops per decoding step, the data volume per hop, and the total number of decoding steps, under GPU memory constraints, as shown in Figure 2. Reducing hops. BloomBee consolidates more transformer blocks onto fewer nodes by adaptively offloading KV cache, 3
Trovato et al.
L₁
L₂
Geo node 1
Client draft model input prompt
𝝉 candidate tree (N)
global batch B pruning micro batch M
slow internet/
L₃
𝑳𝟒 = 𝟎 weaker GPU
Geo node 3
Geo node 2
GPU weights + active KV
GPU weights + active KV
CPU offloaded KV (α₁)
CPU offloaded KV (α₂)
𝑮𝑷𝑼 𝟎 weights + active KV
Geo node 4
𝑮𝑷𝑼 𝟏 weights + active KV
GPU
CPU offloaded KV (α₃)
CPU
Reduce hops / step via CPU offloading (§5)
Reduce data volume per hop via compression (§6)
Reduce decode steps via speculative decoding (§7)
Offload KV to CPU so each node can host more layers
Byte-split + lossless compression shrinks each activation transfer through internet
Draft multiple candidates, then prune before transmission
Bloombee chooses {Lᵢ, αᵢ, B, M, compression, N, 𝝉} under GPU / host memory and Internet bandwidth constraints via dynamic programming
Figure 2. BloomBee with heterogeneous GPU nodes connected over the internet form a pipeline-parallel inference system. BloomBee jointly optimizes layer assignment (𝐿𝑖 ), offloading(𝛼𝑖 ), micro-batching(𝑀), compression, and speculative decoding (𝜏 and 𝑁 ) to reduce communication overhead. Nodes with slow internet or weak GPUs that cannot contribute to system throughput are excluded from the pipeline (𝐿𝑖 = 0). which grows linearly with both batch size and sequence length, from GPU to host CPU memory when GPU capacity is insufficient. This frees GPU capacity for additional blocks and reduces pipeline hops. BloomBee further decomposes each batch into micro-batches to overlap computation and communication across stages. These decisions are coupled through the GPU memory budget, and BloomBee solves their coordination via dynamic programming (§5). Reducing volume. BloomBee applies lossless compression to shrink the activation payload at each hop. The key insight is that byte lanes of serialized floating-point activations have markedly different entropy. A byte-split transform separates them before compression, yielding higher ratios than compressing the raw stream (§6). Reducing steps. BloomBee adopts speculative decoding to reduce the number of decoding steps per request, where a lightweight draft model on the client proposes candidate tokens and the target model deployed across BloomBee’s pipeline verifies them. Unlike single-machine settings, transmitting draft tokens for verification across geo-distributed nodes inflates communication overhead. BloomBee addresses this through draft token pruning, communication-efficient batched transmission, and asynchronous KV cache management (§7).
5
Geo-Distributed High-Throughput Scheduling
5.1
Throughput-Optimized Pipeline Planning
cache footprint. The planner Í assigns each node a contiguous range of 𝐿𝑖 blocks, with 𝑖 𝐿𝑖 = 𝐿, and 𝐿𝑖 ≥ 0. Throughput objective. The pipeline processes a batch of 𝐵 requests, each generating 𝑠 tokens. max Í𝑁
𝐵 ·𝑠
𝑖=1 𝑇comp (𝑖) + 𝑇comm (𝑖)
(1)
where 𝑇comp (𝑖) is the compute time at node 𝑖 and 𝑇comm (𝑖) is the communication time to the next node. Per-node compute time. As described in §4, BloomBee offloads a fraction 𝛼𝑖 ∈ [0, 1] of each node’s KV cache to host CPU memory, splitting attention computation between CPU and GPU while MLP remains on GPU. The per-stage compute time is: 𝑇comp (𝑖) = 𝐿𝑖 · 𝑡 block (𝛼𝑖 ) gpu
cpu
𝑡 block (𝛼𝑖 ) = 𝑡 mlp + (1 − 𝛼𝑖 ) · 𝑡 attn + 𝛼𝑖 · 𝑡 attn
(2)
All latencies are profiled per node. When a node contains multiple GPUs, BloomBee applies intra-node tensor parallelism; the profiled latencies reflect this. Inter-node communication time. Each stage boundary incurs a network hop over the public internet. The per-hop communication time is: 𝑇comm (𝑖) = 𝜆𝑖 +
𝐵 ·𝑑 𝐵 net (𝑖)
(3)
where 𝜆𝑖 is the network propagation latency between node 𝑖 and node 𝑖+1, 𝑑 is the per-request activation size determined by the model’s hidden dimension, and 𝐵 net (𝑖) is the effective link bandwidth. Constraints. The layer assignment, offload ratio, and batch size are coupled through memory. Assigning more layers to a node increases weight memory, which forces a higher offload ratio 𝛼𝑖 or a smaller batch size 𝐵. Formally, Í all blocks must be assigned ( 𝑖 𝐿𝑖 = 𝐿), each node’s GPU must accommodate model weights, activation workspace, and the GPU-resident portion of KV cache, and each node’s host memory must accommodate the offloaded KV cache.
System setup. BloomBee partitions an LLM of 𝐿 transformer blocks across 𝑁 heterogeneous nodes in a linear pipeline (Figure 2). Between nodes, pipeline parallelism forwards activations over the internet; within each node, tensor parallelism splits layers across local GPUs. The scheduling granularity is the transformer block: modern LLMs stack identical blocks (self-attention followed by feed-forward), so each block contributes the same compute cost and the same KV 4
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization split along the batch dimension
time
fill
Sender GPU compute μk Sender CPU DMA + compress + transport Internet transfer activation of μk
μ1
μ2
μ3
μ1 μ1
μ4
μ2 μ2
μ3 μ3
μ1
Receiver CPU receive + decompress Receiver GPU start μk on arrival
activation to CPU memory via DMA, freeing the GPU to immediately begin the next micro-batch. The CPU then handles compression and network transmission in parallel. This overlap exploits the internet-scale data path: since GPU Direct RDMA is unavailable, activations must transit through CPU memory, which naturally decouples GPU computation from network transfer. As the pipeline fills, all stages operate concurrently on different micro-batches. In steady state, the throughput objective with micro-batching becomes:
drain
steady-state overlap
μ4 μ4
μ2
μ3
μ1
μ4
μ2 μ1
μ3 μ2
μ4 μ3
μ4
Figure 3. Micro-batch pipelining in BloomBee. After a node finishes computing 𝜇𝑘 , the activation moves to CPU memory for compression and transport, while the sender GPU immediately starts 𝜇𝑘+1 . In steady state, computation and communication proceed in parallel (Equation 4).
𝐵 ·𝑠 , (𝑀 + 𝑁 − 1) · max𝑖 𝑇cycle (𝑖) 𝑤ℎ𝑒𝑟𝑒 𝑇cycle (𝑖) = max 𝑇comp (𝑖), 𝑇comm (𝑖) max
Decision variables. For a given batch size 𝐵, the planner determines the layer assignment {𝐿1, . . . , 𝐿𝑁 }; the offload ratios {𝛼𝑖 } are then derived from the GPU memory constraint. Setting 𝐿𝑖 = 0 excludes a node and its adjacent internet hops from the pipeline. Layer placement via dynamic programming. The placement problem assigns 𝐿 blocks to 𝑁 nodes in pipeline order. Because each node holds a contiguous range of blocks, the problem reduces to an interval partition, which BloomBee solves exactly via dynamic programming (DP) in 𝑂 (𝑁 · 𝐿 2 ) time. For a pipeline of 𝑁 = 8 nodes and a model with 𝐿 = 80 blocks, the solver completes in under one millisecond. In contrast, Helix [24] formulates placement over arbitrary routing graphs using mixed-integer linear programming, which requires up to hours of solving time with large number of nodes 𝑁 . For each candidate batch size 𝐵, the solver iterates over nodes in pipeline order, evaluating all feasible layer counts per node. For each candidate assignment, it computes𝑇comp (𝑖) and 𝑇comm (𝑖) from a profiled cost table, derives 𝛼𝑖 from the GPU memory constraint, and selects the partition that minimizes total pipeline time. The planner runs the solver for each candidate 𝐵 and selects the configuration with the highest throughput. The DP captures heterogeneity without manually tuned placement rules: nodes with fast PCIe and large CPU memory host more layers because offloading overhead is low, nodes behind slow internet links are skipped to remove costly hops, and nodes with weaker GPUs receive fewer layers to avoid becoming the bottleneck. 5.2
(4)
The per-stage cost becomes the maximum of computation and communication, rather than their sum. The factor (𝑀 + 𝑁 − 1) accounts for pipeline fill and drain. As 𝑀 grows, the fill and drain overhead amortizes and throughput is dominated by the slowest stage. The planner selects 𝑀 jointly with 𝐵 by enumerating candidate values during the dynamic programming solver (§5.1).
6
Communication Compression
We adopt lossless compression, instead of lossy compression (e.g., quantization). The tensors transferred are intermediate activations, rather than static model weights stored offline. This distinction is important: activations are generally more difficult to quantize than weights, in part due to outliers [44]. Prior work also shows that reducing activation precision can introduce noticeable accuracy degradation in the transient states used during inference [46]. Although some systems quantize transmitted activations to reduce latency [14], they alter the hidden states consumed by downstream pipeline stages. Lossless compression preserves tensor values bit-forbit without introducing an accuracy–bandwidth tradeoff. Motivation of our design. Lossless compression employs two main techniques, repetition removal (created in LZ compression [48, 49]) and entropy encoding (e.g., [17, 29]). LZ compressors detect repeated sequences of multiple bytes—usually four or more—and replace them with compact back-references to earlier occurrences, reducing storage requirements. Entropy coders, in contrast, analyze the probability distribution of individual bytes and shrink the data by encoding frequent symbols with shorter bit-level representations. Many widely used compressors, such as ZSTD [7] and zlib [8], combine these two approaches by removing repeated sequences and then applying entropy coding. We study two most popular lossless compressors, ZSTD and zlib and one state-of-the-art, ZipNN [15] for BloomBee. ZSTD and zlib serialize the input data into a byte stream and then detect repeated bytes within a sliding window over the stream. This method is general, but loses floating-point structures (e.g., mantissa and exponent) once the data values
Micro-batch Pipelining
In a geo-distributed pipeline, each node’s GPU is idle while its activation transfers over the internet. Since network transfer dominates stage time, GPU utilization is low. BloomBee splits the batch of 𝐵 requests into 𝑀 micro-batches of size 𝑏 = 𝐵/𝑀, where each micro-batch contains a subset of requests (i.e., partitioned along the batch dimension) to overlap communication with computation, as shown in Figure 3. When a node finishes computing a micro-batch, it copies the output 5
FP16 Element sign exponent (5 bits)
FP16 Element
mantissa (10 bits)
sign exponent (5 bits)
mantissa (10 bits)
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 14
15 14
10 9 8 7
Entropy (bits/byte)
Trovato et al.
0
Byte 1 (High Byte) Byte 0 (Low Byte) raw[0::2] raw[1::2]
10 9 8 7
0
Byte 1 (High Byte) Byte 0 (Low Byte) raw[0::2] raw[1::2]
(a) The original data type
High Byte Stream (8 bits)
Low Byte Stream (8 bits)
Exponent Stream (5 bits)
Sign+Mantissa Stream (11 bits)
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
(b) BloomBee
7.94
7.37
5 0
4.40
Overall (all bytes)
7.70 2.80
Exponent only Sign + Mantissa High byte (bits 14-10)(sign 15, mantissa 9-0) (bits 15-8)
ZipNN
Low byte (bits 7-0)
BloomBee
Figure 5. Entropy of the raw serialized FP16 byte stream and the component streams induced by ZipNN and BloomBee.
(c) ZipNN Table 1. Comparison of lossless compression methods
Figure 4. Comparison of data layouts.
Method ZSTD ZipNN BloomBee
are serialized. ZipNN improves this method by separating exponent bits and mantissa bits (see Figure 4.c), which leads to more repeated patterns. ZipNN focuses on AI model weights. In the context of decentralized LLM inference, we need to compress activations (not weights). Hence, we aim to answer the following two fundamental questions. • Does the separation of exponent bits and mantissa bits in activations reveal more compression opportunities? • How should the separation happen to maximize compression effectiveness? To answer these questions, we profile activations dumped from multiple models (LLaMA-13B/30B/65B, Mixtral-8×7B, and Falcon-40B) during decoding. Since activations use FP16, we study FP16. Figure 5 shows the results for LLaMA-13B using the metric entropy. Entropy in data compression [32] measures the average information content or unpredictability of data. Low-entropy data is highly redundant (e.g., repeated patterns) and compressible, while high-entropy (random) data is not. Figure 5 shows that the entropy of the whole 16 bits is 7.37 bits/byte. Using the method of ZipNN, the separated exponent bits have an entropy of 4.40 bits/byte, which is significantly (40.3%) lower than the whole 16 bits, while the mantissa bits (plus one sign bit) have an entropy of 7.94 bits/byte, which is slightly higher (7.7%). Overall, the separation of exponent and mantissa can bring benefits, which answers the first question. Figure 5.c shows the entropy of another bit-separation method. In this method, the separation happens at the byte binary: we report the entropy for the first byte and second byte of the 16 bits. The first byte (named high byte) includes 1-bit sign, 5-bit exponent, and the most significant two bits of mantissa); the second byte (named low byte) includes the least significant eight bits of mantissa. Comparing Figure 5.b and Figure 5.c, we see that the high and low bytes show lower entropy than exponent and mantissa (plus one sign), revealing more opportunities for compression.
Compressed Size (smaller is better) Total Compression Time 69% 71% 46%
290.0 ms 330.2 ms 300 ms
The rationale behind the improvement from the new separation method is that the most significant two bits of mantissa in activations show repetitiveness. Combining them with repetitive exponent bits, we create longer repetitive sequences, which leads to more effective lossless compression. We have the similar observations in LLaMA-30B, LLaMA65B, Mixtra-8×7B, and Falcon-40B. Compression design in BloomBee. Based on the above discussions, BloomBee introduces a lightweight compressor. Similar to ZSTD and zlib, BloomBee serializes the data into a stream buffer, and applies entropy coding of ZSTD after compression. But different from ZSTD, zlib, ZipNN, BloomBee considers the floating point structure and uses the new bit-separation method. Effectiveness of compression in BloomBee. We compare BloomBee with ZSTD and ZipNN in terms of compressed size in percentage (smaller is better) and compression time. We use activations collected from LLaMA-13B for compression. Table 1 reports the average results per inference among 16 inference requests from the dataset AlpacaEval [22]. In conclusion, BloomBee leads to the smallest compressed size, 33% and 35% smaller than ZSTD and ZipNN respectively, while using similar compression time. We also note that ZSTD slightly outperforms ZipNN even though ZSTD does not consider the floating point structure. We attribute ZSTD’s better performance to its effective entropy coding. BloomBee uses ZSTD’s entropy coding plus the new separation method, hence beating both ZSTD and ZipNN. At the internet scale, the compression time can be a small portion of communication time. For example, In Table 1, given overall activation size of 76.5 MiB, the communication time is 27,911ms with the bandwidth of 20 Mbps. For a network with higher bandwidth (e.g., 500 Mbps), the compression can be turned off when the compression cannot bring performance benefits. 6
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
Table 2. Notation used in the latency model. Symbol 𝐿 𝐷 𝑆 𝑡 rtt 𝑡 comp 𝑚 𝑐 𝑛 𝑁 𝑎 𝑆∗
7
Measure bandwidth S
Definition
Compute break-even bandwidth S*
S ≥ S* ?
Yes
Total tokens to generate Hidden-state payload per token (MB) Network bandwidth (MB/s) Fixed per-transfer latency (ms) Per-worker-node compute time, autoregressive (ms) Compute time ratio, speculative vs. autoregressive Draft model compute time per step (ms) Number of worker nodes Draft tree size (nodes) Average accepted tokens per step Break-even bandwidth (MB/s)
Enable speculation
No (subsequent attempts)
No (initial attempt)
Fallback to autoregressive
Prune tree (↓N)
Figure 6. Bandwidth-aware SD configuration in BloomBee. 𝑛·𝐵·𝐷 𝑇auto = 𝐿 · 𝑛 · 𝑡 comp + + 𝑛 · 𝑡 rtt 𝑆
(5)
Each token in 𝐿 incurs compute, transfer of the batched hidden states, and round-trip latency at every stage. With speculative decoding, the target model performs only 𝐿/𝑎 verification passes on average, but each pass carries 𝑁 candidate token states and requires 𝑚× the compute of one autoregressive pass:
Speculative Decoding over Internet
In BloomBee’s pipeline, each decoding step requires a full round of communication across all geo distributed stages. Reducing the number of decoding steps therefore directly reduces internet communication. Speculative decoding offers this opportunity: a draft model on client side proposes multiple candidate tokens, and the target model deployed with BloomBee geo-distributed verifies them in a single pass. In BloomBee, however, the benefit of fewer target-model passes must be weighed against the cost of sending a larger candidate set over every internet hop. 7.1
Choose N and estimate accept token rate a
𝑇spec =
𝐿 𝑛·𝐵·𝑁 ·𝐷 · 𝑐 + 𝑛 · 𝑚 · 𝑡 comp + + 𝑛 · 𝑡 rtt 𝑎 𝑆
(6)
where 𝑐 is the draft-model generation time per speculative pass. Speculation helps only when 𝑇spec < 𝑇auto , demonstrated as 𝐵 · (𝑁 − 𝑎) · 𝐷 𝑐 + (𝑚 − 𝑎) · 𝑡 comp + (1 − 𝑎) · 𝑡 rtt + <0 𝑆 𝑛 ·𝑎 | {z } {z } | {z } | |{z} data transfer penalty
compute penalty
RTT saving
draft cost
(7)
Rearranging yields the break-even bandwidth:
Speculative Decoding under Network Constraints
𝑆 > 𝑆∗ =
BloomBee’s communication-centric design creates a tension with speculative decoding. As discussed in §5.2, BloomBee batches 𝐵 requests together to amortize the cost of each internet hop. Under autoregressive decoding, each request contributes one token’s hidden state per step, so a batched hop carries 𝐵 · 𝐷, where 𝐷 is the hidden-state size of a single token. Under speculative decoding, each request carries a draft tree with 𝑁 candidate token states, increasing the per-hop payload to 𝐵 · 𝑁 · 𝐷. The same batching that makes internet-scale inference practical therefore amplifies the communication overhead of speculation. Over the internet, the larger payload traverses every distributed stage boundary, and the additional transfer time can exceed the latency saved by reducing the number of targetmodel passes. Whether speculation helps therefore depends on the candidate set size 𝑁 , the average acceptance rate of tokens per speculative pass 𝑎, and the available bandwidth 𝑆. Here, 𝑎 denotes the average number of output tokens committed per speculative pass. Table 2 summarizes the notation. We consider the end-toend latency for generating 𝐿 output tokens across 𝑛 internetdistributed pipeline stages. For simplicity, the model considers one transformer block compute and communication per stage. Under autoregressive decoding, the total execution time is
𝐵 · (𝑁 − 𝑎) · 𝐷 (𝑎 − 1) · 𝑡 rtt + (𝑎 − 𝑚) · 𝑡 comp − 𝑐/𝑛
(8)
Equation 8 exposes why speculative decoding is challenging in the internet setting. The RTT term improves with 𝑎, but data transfer grows with both batch size and candidate set size. When the payload increase dominates the reduction in passes, speculation slows decoding. To reduce the break-even bandwidth 𝑆 ∗ , BloomBee prunes low-probability candidates from the draft tree before data transfer, reducing 𝑁 while preserving most of the progress benefit (§7.2). If the measured bandwidth remains below 𝑆 ∗ after pruning, BloomBee disables peculation and falls back to autoregressive decoding, ensuring that speculation does not increase end-to-end latency. Figure 6 demonstrates this decision process. 7.2
Draft Tree Pruning and Efficient Verification
Equation 6 shows that the data transfer cost of speculation grows linearly with the number of candidate token states sent across each internet hop. Pruning therefore reduces communication directly. The challenge is that pruning can also remove candidates that would have survived verification, reducing the average acceptance ratio of 𝑎. BloomBee addresses this tension with three mechanisms: early pruning at the first distributed stage, padding-free data transfer of the retained states, and asynchronous KV-cache compaction. 7
Trovato et al.
Figure 7. Data transfer for SD in BloomBee. Early pruning from first-stage hidden states. A key observation is that the final-layer hidden states at the first worker node already encode enough context to predict whether a candidate is likely to survive full-model verification. BloomBee therefore scores candidates before they cross the next internet hop. Concretely, let h𝑖 denote the hidden state of candidate token 𝑣𝑖 at the output of the first worker node. BloomBee applies a lightweight proxy LM head 𝑊ˆ and obtains a local distribution estimate 𝑝ˆ𝑖 = softmax(𝑊ˆ h𝑖 ).
8
Evaluation
BloomBee is an extension to Petals. Hence, in terms of computation (e.g., attention and MLP), BloomBee and Petals have the same implementation. BloomBee is open-sourced and has 31,216 lines of code in total. BloomBee has been deployed in multiple realistic use cases. 8.1
(9)
Evaluation Setup
Baselines. We compare BloomBee against two baselines. Neither of them have any communication optimization techniques employed by BloomBee. (1) Petals, an open-source framework for internet-scale LLM fine-tuning and inference. (2) Helix, an LLM serving framework for heterogeneous GPU and network environments on the internet scale. Models. We use LLaMA (13B, 30B, and 65B). Unless otherwise stated, the primary evaluation model is LLaMA-30B with batch size 32, as it is large enough to expose inter-stage communication overhead while still allowing controlled evaluation across multiple network environments. We also evaluate Falcon-7B, Falcon-40B, and Mixtral-8×7B; Their results are reported in Appendix A.2. Cluster setup. We evaluate BloomBee with three types of cluster setups: (1) a single cluster (E1), (2) geo-distributed homogeneous clusters (E2-E5), and (3) geo-distributed heterogeneous clusters (E6). E1 serves as a high-bandwidth (45 Gbps) environment without the internet. E2-E5 are four environments built upon E1 to emulate internet-scale inferences. We control interconnect bandwidth between nodes in E1, aligned with representative geo-distributed internet bandwidth [3, 24, 34]: the bandwidths for E2-E5 are 500 Mbps, 250 Mbps, 125 Mbps, and 20 Mbps respectively. E6 is a realistic internet-scale environment with three GPUs: one NVIDIA A100 in California, one NVIDIA RTX 4090 in New Jersey, and one NVIDIA RTX 4090 in Canada. Table 3 summarizes the bandwidth between nodes in E6. In our evaluation, we use single-GPU nodes in most evaluations, because multi-GPU nodes are harder to allocate consistently in the cloud [37]. BloomBee also supports tensor parallelism across GPUs within the same node when multi-GPU nodes are available.
The proxy head is trained offline to match the full-model distribution 𝑝𝑖 by minimizing the KL divergence 𝐷 KL (𝑝𝑖 ∥ 𝑝ˆ𝑖 ). From 𝑝ˆ𝑖 , BloomBee extracts three scalar features, f𝑖 = [max(𝑝ˆ𝑖 ), 𝑝ˆ𝑖 [𝑣𝑖 ], H (𝑝ˆ𝑖 )] ,
the next inference request and defers compaction until that request arrives. Each worker node keeps its KV cache in three regions: a compact prefix from earlier rounds, a hole region corresponding to previously rejected tokens, and a newly appended region for the current round. The hole region is masked during attention, so the cache remains immediately usable. Once the next request arrives, a background thread compacts the hole region in parallel with the current round’s layer execution. In steady state, this overlap keeps compaction off the latency-critical path.
(10)
which capture distribution peakedness, confidence in the candidate token, and overall uncertainty. A lightweight MLP maps f𝑖 to a retention score 𝑠𝑖 ∈ [0, 1], and BloomBee drops candidate 𝑣𝑖 when 𝑠𝑖 < 𝜏. The threshold 𝜏 controls the pruning ratio at inference time without retraining. The classifier is trained on the Alpaca dataset with binary labels indicating whether the candidate belongs to the full model’s retained top-𝐾 set, and adds only sub-millisecond overhead per decoding step. Padding-free data transfer. After pruning, request 𝑖 retains 𝑁𝑖′ candidate token states, where 𝑁𝑖′ may differ across requests. The post-pruning batch is therefore ragged. Padding each request back to a common length would restore a regular layout, but it would also reintroduce the communication overhead that pruning eliminates. BloomBee instead packs all retained states into a single contiguous tensor of shape Í [1, 𝑖 𝑁𝑖′, 𝐷] and transmits it together with a small offset array that records per-request boundaries, as shown in Figure 7. The next worker node reconstructs the logical batch layout locally and continues the forward pass without extra network messages. Asynchronous KV-cache compaction. Verification reveals which draft tokens were committed only after the client returns the accepted prefix at the beginning of the next decoding round. Eager compaction would therefore either block the current round or require an extra internet round-trip. BloomBee instead piggybacks the acceptance metadata on 8
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
Table 3. Bandwidths (Mbps) in E6.
As bandwidth drops (E2–E4), SD becomes less effective and is disabled. BloomBee therefore disables SD. The model 13B solely relies on compression. For the model 30B, standard autoregressive decoding performs best in E2, while compression with micro-batching becomes the best choice in E3 and E4. For the model 65B in E2 and E3, standard autoregressive decoding performs best; the system overhead of compression and SD exceeds the communication savings at these environments. Under severe bandwidth constraints (e.g., E5 for all model sizes and E4 for 65B), BloomBee consistently applies compression and micro-batching. This configuration reduces perstep communication volume and hides transmission latency behind computation.
Sender → Receiver California New Jersey Canada California New Jersey Canada
— 347 305
312 — 577
280 643 —
Table 4. Best-performing communication optimization techniques in each environment. Model
Environments
Best techniques
13B 13B 13B
E1 (2 nodes, 2 5090 GPUs) E2,E3,E4 (2 nodes, 2 5090 GPUs) E5 (2 nodes, 2 5090 GPUs)
SD Compression Compression + micro-batching
30B 30B 30B 30B
E1(3 nodes, 3 5090 GPUs) E2 (3 nodes, 3 5090 GPUs) E3,E4,E5 (3 nodes, 3 5090 GPUs) E6 (3 nodes, 2 5090 + 1 A100)
SD Autoregressive Compression + micro-batching Compression + micro-batching
8.3
We use LLaMA-30B with batch size 32 and sequence length 128, with compression, micro-batching, and speculative decoding disabled. With offloading, as selected automatically by BloomBee, all KV cache are placed in CPU, and BloomBee can use just 2 GPUs, reducing the GPU count by 1. Both Helix and Petals have to use 3 GPUs. According to common AIspecific cloud vendors (Yotta [45] and VAST [39]), we price each GPU at $0.56 per hour, giving a 3-GPU baseline cost of $1.68/h and a 2-GPU cost of $1.12/h, and report throughput per GPU dollar, which considers the impact of offloading on both throughput and production cost. See Figure 9. Offloading consistently improves cost efficiency across all environments, with the largest gains in bandwidth-limited settings. In E5, offloading achieves 1.82× higher throughput per GPU dollar (41.8 vs. 22.9 tok/s/$/h). In E3 and E4, the improvement is 1.17× and 1.18× respectively. Even in E1 and E2 with high-bandwidth, offloading still provides a minor gain (1.02× and 1.05×). Note that with improvement of cost efficiency, no matter how small the improvement is, longer LLM service time will continuously bring benefits in larger saving of production cost. Figure 9 also shows the variance of throughput after using offloading (see the second y axis). We notice that with offloading, BloomBee even brings 21.6% improvement in throughput in E5, because of saving in communication hops. In another evaluation at E5, we use three GPUs and change the batch size. Using offloading, BloomBee puts 30% of KV cache to CPU, which enables larger batch sizes (up to 128), compared to no offloading with a smaller batch size 32. As a result, offloading brings 14.4% improvement in throughput.
65B E1 (4 nodes, 8 5090 GPUs) SD 65B E2,E3 (4 nodes, 8 5090 GPUs) Autoregressive 65B E4,E5 (4 nodes, 8 5090 GPUs) Compression + micro-batching Autoregressive: no compression, no micro-batching, no SD, and no offloading.
8.2
Evaluation of Offloading
Overall Performance
We evaluate BloomBee in E1-E5 against Petals and Helix. Figure 8 shows the overall results. BloomBee outperforms both Petals and Helix across most environments. For example, in E5 with LLaMA-30B, BloomBee achieves 67 tok/s, demonstrating significant speedups over Petals (1.76×) and Helix (1.46×). In E3 and E4, BloomBee matches or outperforms Petals while consistently outperforming Helix by notable margins. The throughput gains are driven by two complementary effects: pruned SD reduces inter-stage communication volume by 60% while preserving 96% of the acceptance rate in highbandwidth environments, and micro-batching with compression reduces communication overhead in bandwidth-limited environments. As observed in Figure 8, when we change the model size from 13B to 65B, the throughput gap between BloomBee and Helix narrows. This is because of the change in number of layers per GPU. When the number of layers per GPU is larger, the computation per GPU becomes larger, which provides more opportunities to pipeline parallelism to hide communication overhead. Hence, there is less room for BloomBee to optimize communication performance. Table 4 summarizes the communication optimization techniques selected by BloomBee across various model sizes and evaluation environments. In the highest-bandwidth environment (E1), pruned SD is universally the primary driver: sufficient bandwidth allows SD to reduce the number of decoding steps while pruning makes transmission volume manageable.
8.4
Evaluation of Micro-batching
We use LLaMA-30B with batch size 32 and sequence length 128, with compression, offloading, and SD disabled. We use micro-batch size 16 unless otherwise stated. See Figure 10. In E1 and E2 with relatively high bandwidth, BloomBee disables micro-batching automatically, and hence performs similarly to the case without micro-batching. 9
Trovato et al.
145
150
162
150
BloomBee
171
117 104 129
120
129
123
122 104
100
102
105 85
73 68
84
E1
E2
E3
E4
Model Size: 65B
Helix
111
102
98
100
109 84 68
67 38
0
E1
E2
E3
E4
Helix
BloomBee
81
56
E5 Worse
Petals
100
108
95 77
BloomBee
50
50 0
Petals
46
Better
200
Model Size: 30B
Helix
Better
Better
Throughput (Token/s)
Model Size: 13B Petals
75
70 71 52 54 52
50
44 47 44 32
40
45 34
0
E5 Worse
38
20
25
E1
E2
E3
E4
E5 Worse
60
66.3
(1.18x) 58.9
56.8
49.8
(1.82x)
0
41.8
40
−20
22.9
20 −40
0
E1
E2
E3
E4
217
0
4 4
7 7
25 19
45 39
Better
(1.02x)
60.8 62.0
269
200
Throughput (Token/s)
61.9 65.1
20
(1.17x)
w/o Comp w/ Comp
400
Better
(1.05x)
NIC-to-NIC Time (ms)
80
Throughput Variance (%)
40
BloomBee (3 GPU, $1.68/h) BloomBee w/ Offload (2 GPU, $1.12/h) Throughput Variance (%)
100
Better
Better Throughput per GPU Dollar (tok/s per $/hour)
Figure 8. Overall performance across E1–E5 for various model sizes.
E1 E2 E3 E4 E5 Worse (a) NIC-to-NIC transfer time.
200 Baseline w/ Comp. only
150 100
104 104 104
w/ MB only w/ Both
104 102 100 101 102 102 102 95 99 84 87
99 100
50 0
67 38
E1
E2
E3
E4
45
53
E5 Worse
(b) End-to-end throughput.
Figure 11. Compression evaluation with BloomBee. “MB” = “micro-batching”. “Comp” = “compression”.
E5
Worse
The NIC-to-NIC transfer time reduction from compression grows with network constraint: negligible in E1 (4 to 4 ms) and E2 (7 to 7 ms), but substantial in E3 (25 to 19 ms), E4 (45 to 39 ms), and E5 (269 to 217 ms). This pattern drives the end-toend throughput results. In E1 and E2, all four configurations perform similarly, and BloomBee disables compression since there is no benefit. In E5, where network transmission dominates, compression alone improves throughput by 18.4%, micro-batching alone by 39.5%, and the combination by 76.3%. We note that the improvement of the combination is larger than the improvement summation of compression and micro-batching (i.e., 13.2% + 39.5%). This is because the micro-batching reshapes the data to be transferred across hops and accidentally brings more opportunities for effective compression.
Figure 9. Cost efficiency (throughput per GPU dollar) with and without offloading. 200 BloomBee BloomBee w/ micro-batch size=16
Better
150 100
104104
102102
95 100
84
99
50 0
38
E1
E2
E3
E4
53
E5 Worse
Micro batch size=8 Micro batch size=16
150
Better
Throughput (Token/s)
200
104
100 50
102
100
99
50
48
47
48
48 53
E1
E2
E3
E4
E5 Worse
0
Figure 10. Evaluation of micro-batching. From E3 onward, the gain of using micro-batching grows monotonically: +5.3% in E3, +17.9% in E4, and +39.5% in E5. We evaluate the impact of micro-batch size. The size 16 consistently outperforms the size 8 across all environments, with the size 8 plateauing at 47–50 tok/s while size 16 matches or exceeds the no-micro-batching baseline in E1–E2 and substantially outperforms it in E3–E5. The larger micro-batches provide more opportunities for overlap, as the compute time per micro-batch grows 8.5
8.6
Evaluation of Speculative Decoding
We use LLaMA-30B with the batch size 32 and 128 output tokens. To reduce performance variance because of promptdependent generation lengths, we randomly sample 10 batches from the dataset Alpaca [36] and report mean throughput across runs. We compare three configurations: Helix, BloomBee without any communication optimization (named Auto and the baseline), unpruned SD (named SD), and SD with MLPbased pruning (named SD+Prune). Pruned SD yields consistent throughput improvements over the baseline in high-bandwidth environments: +12.5% in E1 (117 vs. 104 tok/s) and +1.0% in E2 (103 vs. 102 tok/s). In E3, pruned SD falls slightly below the baseline (75 vs. 95 tok/s), as the transmission overhead of the draft tree begins to outweigh the reduction in decoding steps at this bandwidth. Unpruned SD performs worse than pruned SD in all three environments, confirming that pruning is essential
Evaluation of Compression
We use LLaMA-30B with batch size 32 and sequence length 128. With this configuration, BloomBee reduces the activation payload per hop from 416.4 KB to 312.6 KB. Since microbatching can hide communication overhead and hence impacts the effectiveness of compression, we selectively add micro-batching for evaluation. We compare four configurations: BloomBee without compression and micro-batching (baseline), with compression only, with micro-batching only, and with compression plus micro-batching. See Figure 11. 10
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
to realizing the benefit of SD under network constraints: by reducing the transmitted tree size by 60% while preserving 96% of the acceptance rate, pruning shifts the operating point closer to the break-even bandwidth derived in §7. We compare the peak throughput of BloomBee using SD with the steady-state throughput (which is also the peak one) of Helix. This comparison reflects the best case of BloomBee against that of Helix. BloomBee with pruned SD achieves 1.56×, 1.46×, and 1.27× higher throughput than Helix in E1– E3 (152.73 vs. 97.6, 112.17 vs. 76.9, and 85.82 vs. 67.6 tok/s respectively) because of effectiveness of SD. Throughput alone, however, understates the benefit of SD in batched inference. Under autoregressive decoding, all samples in a batch complete simultaneously at the wall-clock time, since each step advances every sequence by exactly one token. SD breaks this uniformity: sequences that align well with the draft model accept more tokens per step and finish earlier, while others are held until the batch tail. We evaluate the impact of SD in BloomBee on per-sample completion time. Figure 12 shows the results. The per-sample completion times of SD+Prune spread over a wide range, whereas Auto appears as a single vertical bar. In E1, the median sample completes at 22.8 s—43% faster than the autoregressive baseline of 40.2 s. In E2, the median completion is 30.6 s versus 40.3 s (−24%). In E3, the median is 41.1 s versus 43.1 s (−4%). For LLM serving systems that dispatch completed requests as soon as they finish rather than waiting for the full batch to complete, this reduction in per-sample latency translates directly into lower user-perceived response time—a benefit orthogonal to batch throughput.
for model allocation and request scheduling in dynamic environments. Petals reduces communication overhead via quantization of weights and activations. HexGen [20] is a concurrent work on LLM serving but works only for fixed pipelines. Helix [24] studies model placement and scheduling based on Max Flow formulation and flexible per-request pipeline. Parallax [38] introduces dynamic pipeline construction with a two-phase scheduling strategy. This strategy creates replicas of pipeline stages and stitches layers from different replicas to balance load and improve utilization. Distributed LLM training on geo-distributed GPUs. Sailor [34] co-optimizes resource allocation and 3D parallelization plans for distributed training over dynamic, heterogeneous, and geo-distributed clusters, emphasizing fast planning via accurate simulation and search-space pruning. DiLoCo [10] focuses on data parallelism for distributed training and proposes a new distributed optimizer to reduce frequency of collective communication. SWARM [30] (and the extension of SWARM based on DiLoCo [31]) constructs stochastic training-pipelines dynamically. Each node forwards activations or gradients to a randomly selected node in the next pipeline stage, with probability proportional to the node’s throughput. This dynamically constructed pipeline enables dynamic load balance and fault tolerance. To reduce collective communication overhead, SPARTA [11] only communicates a small random subset of gradients at each step. DeDLOC [9] (and DeDLOC extension, DiLoCo-FSDP2 [19] with int8 all-reduce) use delayed parameter update during training to allow nodes to communicate less frequently and accumulate gradients at each node’s own pace.
10 8.7
In this work, we study LLM inference performance at internet scales, and customize the inference system design for low-bandwidth environments. Different from existing decentralized system designs, our study puts communication optimization as a first-class citizen. Our work sheds lights on how a decentralized LLM system can be designed to maximize throughput.
Heterogeneous Clusters across Internet
We use LLaMA-30B with batch size 32 and sequence length 128. BloomBee achieves the highest end-to-end throughput among all compared systems, shown in Figure 13. Petals reaches 89 tok/s, and Helix reaches 66 tok/s. BloomBee with compression alone further increases throughput to 92 tok/s. BloomBee with micro-batching delivers a much larger gain, reaching 105 tok/s. When both optimizations are enabled, BloomBee achieves the best result at 108 tok/s, outperforming Petals by 21.3% and Helix by 63.6%. The result shows that micro-batching is the main source of improvement in this setting, and compression brings a smaller but positive gain on top of it.
9
Conclusions
Related Work
LLM inference on geo-distributed GPUs. There are existing efforts using geo-distributed GPUs for LLM inferences with heterogeneous clusters and networks. Petals [3] focuses on a pipeline parallel setup, and employs a greedy algorithm 11
Trovato et al.
E2: 500 Mbps, 25 ms
0.05
0.00
P50=23.1s
20
25
30
0.100
0.100
0.075
0.075
0.050 P50=30.6s
0.025
35
40
0.000
45
25
30
Completion time (s)
SD+Prune
Density
0.10
Density
Density
E1: Intra-data-center
35
40
Completion time (s)
Autoregressive
SD+Prune
(a) E1 (Intra-data-center)
45
Autoregressive
(b) E2 (500 Mbps, 25 ms)
50
E3: 250 Mbps, 50 ms
0.050 P50=41.1s
0.025 0.000
35
40
45
50
55
Completion time (s)
SD+Prune
Autoregressive
(c) E3 (250 Mbps, 50 ms)
Throughput (Token/s)
Better
Figure 12. Distribution of per-sample completion latency. The dashed vertical line indicates the wall-clock time of autoregressive (baseline) batch, at which all samples complete simultaneously. The dotted vertical line marks the P50 of SD+Prune. SD spreads completions across a wide range, with the majority of samples finished well before the autoregressive baseline.
100
92.0
89.0
105.0
108.0
BloomBee w/ MB.
BloomBee w/ Both
66.0
50 0
Petals
Helix
BloomBee w/ Comp.
Figure 13. Throughput evaluation on realistic heterogeneous clusters across internet. “MB” = “micro-batching”. “Comp” = “compression”.
12
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
References
[16] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training computeoptimal large language models. arXiv preprint arXiv:2203.15556 10 (2022). [17] David A. Huffman. 1952. A Method for the Construction of MinimumRedundancy Codes. Proceedings of the IRE 40, 9 (1952), 1098–1101. [18] Xiang Hui and Catherine Tucker. 2025. Decentralization, blockchain, artificial intelligence (AI): challenges and opportunities. Journal of Product Innovation Management 42, 5 (2025), 947–957. [19] Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann. 2024. INTELLECT-1 Technical Report. https://arxiv.org/abs/2412. 01152 [20] Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. 2024. HEXGEN: generative inference of large language model over heterogeneous environment. In International Conference on Machine Learning. [21] Qazi Waqas Khan, Anam Nawaz Khan, Atif Rizwan, Rashid Ahmad, Salabat Khan, and Do-Hyeun Kim. 2023. Decentralized machine learning training: a survey on synchronization, consolidation, and topologies. IEEE Access 11 (2023), 68031–68050. [22] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsu-lab/alpaca_eval. [23] Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2023. Online speculative decoding. arXiv preprint arXiv:2310.07177 (2023). [24] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 586–602. doi:10.1145/3669940.3707215 [25] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-Based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (ASPLOS). 932–949. [26] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. In SC ’21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. doi:10.1145/3458817.3476209 [27] Vrajkumar Patel, Aayush Modi, Harsh Mistry, Abhishesh Mishra, Rocky Upadhyay, and Apoorva Shah. 2025. From Alt-text to Real Context: Revolutionizing image captioning using the potential of LLM. International Journal of Scientific Research in Computer Science Engineering and Information Technology 11, 1 (2025), 379–387. [28] Ivy Peng, Ian Karlin, Maya Gokhale, Kathleen Shoga, Matthew Legendre, and Todd Gamblin. 2022. A Holistic View of Memory Utilization on HPC Systems: Current and Future Trends. In Proceedings of the International Symposium on Memory Systems. [29] J. J. Rissanen. 1976. Generalized Kraft Inequality and Arithmetic Coding. IBM Journal of Research and Development 20, 3 (1976), 198– 203. [30] Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. 2023. Swarm parallelism: Training large models can be
[1] Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. 2024. Beyond efficiency: A systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00625 (2024). [2] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). [3] Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2022. Petals: Collaborative Inference and Fine-tuning of Large Models. arXiv preprint arXiv:2209.01188 (2022). [4] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 (2023). [5] Xin Chen, Xiaoyang Wang, Ana Colacelli, Matt Lee, and Le Xie. 2025. Electricity demand and grid impacts of AI data centers: Challenges and prospects. arXiv preprint arXiv:2509.07218 (2025). [6] Arnab Choudhury, Yang Wang, Tuomas Pelkonen, Kutta Srinivasan, Abha Jain, Shenghao Lin, Delia David, Siavash Soleimanifard, Michael Chen, Abhishek Yadav, Ritesh Tijoriwala, Denis Samoylov, and Chunqiang Tang. 2024. MAST: global scheduling of ML training across geo-distributed datacenters at hyperscale. In Proceedings of USENIX Conference on Operating Systems Design and Implementation. [7] Yann Collet and Murray Kucherawy. 2021. Zstandard Compression and the ’application/zstd’ Media Type. RFC 8878. doi:10.17487/ RFC8878 [8] Peter Deutsch and Jean loup Gailly. 1996. ZLIB Compressed Data Format Specification version 3.3. RFC 1950. doi:10.17487/RFC1950 [9] Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, quentin lhoest, Anton Sinitsin, Dmitry Popov, Dmitry V. Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ilia Kobelev, Yacine Jernite, Thomas Wolf, and Gennady Pekhimenko. 2021. Distributed Deep Learning In Open Collaborations. In Advances in Neural Information Processing Systems. [10] Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. 2023. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105 (2023). [11] exo. 2025. SPARTA: Distributed Training with Sparse Parameter Averaging. https://blog.exolabs.net/day-12. [12] Federal Communications Commission. 2024. Inquiry Concerning Deployment of Advanced Telecommunications Capability to All Americans in a Reasonable and Timely Fashion, GN Docket No. 22-270, 2024 Section 706 Report. Technical Report FCC 24-27. Federal Communications Commission. https://docs.fcc.gov/public/attachments/FCC-2427A1.pdf [13] Yanjie Gao, Yichen He, Xinze Li, Bo Zhao, Haoxiang Lin, Yoyo Liang, Jing Zhong, Hongyu Zhang, Jingzhou Wang, Yonghua Zeng, Keli Gui, Jie Tong, and Mao Yang. 2024. An Empirical Study on Low GPU Utilization of Deep Learning Jobs. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). doi:10.1145/ 3597503.3639232 [14] Jan Hansen-Palmus, Michael Truong Le, Oliver Hausdörfer, and Alok Verma. 2024. Communication Compression for Tensor Parallel LLM Inference. arXiv preprint arXiv:2411.09510 (2024). doi:10.48550/arXiv. 2411.09510 [15] Moshik Hershcovitch, Andrew Wood, Leshem Choshen, Guy Girmonsky, Roy Leibovitz, Ilias Ennmouri, Michal Malka, Peter Chin, Swaminathan Sundararaman, and Danny Harnik. 2024. ZipNN: Lossless Compression for AI Models. arXiv preprint arXiv:2411.05239 (2024). doi:10.48550/arXiv.2411.05239 13
Trovato et al.
[48] J. Ziv and A. Lempel. 2006. Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory 24, 5 (2006), 530–536. [49] J. Ziv and A. Lempel. 2006. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory 23, 3 (2006), 337–343.
surprisingly communication-efficient. In International Conference on Machine Learning. [31] Mika Senghaas. 2025. DiLoCo-SWARM. In EPFL Master Research Project. [32] Claude E. Shannon. 1948. A Mathematical Theory of Communication. Bell System Technical Journal 27, 3 (1948), 379–423. [33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053 (2019). [34] Foteini Strati, Zhendong Zhang, George Manos, Ixeia Sánchez Périz, Qinghao Hu, Tiancheng Chen, Berk Buzcu, Song Han, Pamela Delgado, and Ana Klimovic. 2025. Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters. In ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP). doi:10. 1145/3731569.3764839 [35] Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th annual meeting of the association for computational linguistics. 3645–3650. [36] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https:// github.com/tatsu-lab/stanford_alpaca. [37] John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. 2023. Bamboo: Making preemptible instances resilient for affordable training of large {DNNs}. In USENIX Symposium on Networked Systems Design and Implementation (NSDI). [38] Chris Tong, Youhe Jiang, Gufeng Chen, Tianyi Zhao, Sibian Lu, Wenjie Qu, Eric Yang, Lynn Ai, and Binhang Yuan. 2025. Parallax: Efficient llm inference service over decentralized environment. arXiv preprint arXiv:2509.26182 (2025). [39] Vast.ai. 2026. Vast.ai: On-demand GPU Cloud Platform. https://cloud. vast.ai/ Accessed: 2026-04-15. [40] William Walden, Marc Mason, Orion Weller, Laura Dietz, John Conroy, Neil Molino, Hannah Recknor, Bryan Li, Gabrielle Kaili-May Liu, Yu Hou, et al. 2025. Auto-argue: Llm-based report generation evaluation. arXiv preprint arXiv:2509.26184 (2025). [41] Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, et al. 2024. Weaver: Foundation models for creative writing. arXiv preprint arXiv:2401.17268 (2024). [42] Qizhen Weng, Lingyun Yang, Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. 2023. Beware of Fragmentation: Scheduling GPU-Sharing Workloads with Fragmentation Gradient Descent. In 2023 USENIX Annual Technical Conference (ATC). [43] Linyu Wu, Xiaoyuan Liu, Tianneng Shi, Zhe Ye, and Dawn Song. 2025. DeServe: Towards Affordable Offline LLM Inference via Decentralization. arXiv preprint arXiv:2501.14784 (2025). [44] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2022. SmoothQuant: Accurate and Efficient PostTraining Quantization for Large Language Models. arXiv preprint arXiv:2211.10438 (2022). doi:10.48550/arXiv.2211.10438 [45] Yotta Labs. 2026. Yotta Labs: GPU Cloud for AI Training and Inference. https://www.yottalabs.ai/ Accessed: 2026-04-15. [46] Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. 2024. WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More. arXiv preprint arXiv:2402.12065 (2024). doi:10.48550/arXiv.2402.12065 [47] Hengrui Zhang, Yulong Hui, Yihao Liu, and Huanchen Zhang. 2025. ScaleDoc: Scaling LLM-based Predicates over Large Document Collections. arXiv preprint arXiv:2509.12610 (2025).
A
Appendix
A.1
Specification-driven code generation.
Integrating new model architectures (e.g., LLaMA) into BloomBee requires writing a substantial amount of boilerplate code to match the runtime’s expected model interfaces and configuration formats. To reduce this effort, we adopt a specificationdriven code generation approach based on structured model templates, depicted in Figure 14. The model developers provide a YAML specification that describes model-specific parameters (e.g., layer structure and configuration fields). This specification is used to instantiate a set of Jinja2 templates that generate boilerplate model scaffolding, including block.py (stage wrappers), config.py (model configuration), and model.py (model composition), with a consistent structure. The generated code covers repetitive scaffolding such as class definitions and configuration handling, while core computation logic (e.g., forward passes) remains manually implemented. The specification follows a constrained schema, allowing simple structural validation before code generation. This approach simplifies model onboarding and reduces repetitive engineering effort, without changing the underlying execution semantics. We evaluate spec-driven code generation against manual integration in deployment workflows. As shown in Table 5, the approach reduces model integration time by 85% and code review effort by 70%, by eliminating boilerplate and enforcing interface consistency. Deployment error rates decrease by 95%, indicating improved reliability from standardized generation. Automation further yields up to 3× faster end-to-end iteration, as components are regenerated directly from specifications. These results demonstrate improved efficiency and robustness in LLM deployment pipelines, with specification overhead amortized across repeated deployments. Model Specification (YAML)
Template-based Generator
architecture + features deterministic synthesis
Generated Infrastructure block / config / model
Distributed Inference Runtime Figure 14. Specification-driven code generation
14
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
Table 5. Impact of spec-driven code generation on development efficiency and reliability. Results compare templatedriven integration with manual workflows.
Listing 2. Generated block (simplified) 1 2 3
Metric Model integration time Code review effort Deployment error rate End-to-end iteration time Manual integration effort
4
Improvement 85% reduction 70% reduction 95% reduction 3× speedup Significantly reduced
5 6 7 8
1 2 3 4 5 6 7 8 9 10
features : rotary_embeddings : true hf_causal_mask : true kv_cache_reorder : false use_cache : true shared_impl_snippet : " path / to / custom_forward . py "
13 14 15
def forward ( self , x , kv_cache = None , mask = None ) : if config . rotary_embeddings : x = apply_rotary ( x ) # from features . rotary_embeddings
Example. Listing 1 shows a high-level specification for a 9 LLaMA-3-70B Transformer block, while Listing 2 presents 10 the generated implementation. The generation process es- 11 tablishes a structured correspondence between specification 12 if config . hf_causal_mask : fields and code components. Module-level entries such as 13 mask = build_4d_mask ( mask ) # from features . hf_causal_mask attention_class, mlp_class, and norm_class are mapped 14 to concrete module instantiations in the constructor. Feature 15 if config . use_cache : flags including rotary_embeddings, hf_causal_mask, and 16 kv_cache = update_cache ( kv_cache , x ) use_cache are compiled into conditional branches in the # from features . use_cache forward function, controlling execution behavior. Through 17 out = self . attn (x , kv_cache , mask ) this mapping, the generated code is directly driven by the 18 out = self . mlp ( out ) specification rather than manually implemented. This en- 19 20 return out sures that architectural choices and feature configurations expressed in the specification are consistently and faithfully A.3 Micro-batching on Falcon-7B and Mixtral-8×7B reflected in the resulting implementation, enabling interpretability and verifiability. We evaluate micro-batching on Falcon-7B and Mixtral-8×7B to examine its sensitivity to pipeline depth. Table 9 sumListing 1. Model specification for LLaMA-3-70B marizes the throughput with and without micro-batching model_name : " llama -3 -70 b " across E1–E5. While the overhead in E1 is similar for both attention_class : " FlexGenAttention " models (≈18–19%), the benefit in E5 increases with deeper mlp_class : " FlexGenMLP " pipelines: from +0.9% for Falcon-7B (2 stages) to +11.4% for norm_class : " RMSNorm " Mixtral-8×7B (4 stages).
11 12
class TransformerBlock ( nn . Module ) : def __init__ ( self , config ) : super () . __init__ () self . attn = FlexGenAttention ( config ) # from attention_class self . mlp = FlexGenMLP ( config ) # from mlp_class self . norm = RMSNorm ( config . hidden_size ) # from norm_class
distributed : dht_prefix : " llama3 -70 b "
A.2
Transport Breakdown for Additional Model Architectures
We report baseline transport breakdowns for Falcon-7B, Mixtral-8×7B, and Falcon-40B across E1–E5 to assess whether BloomBee’s communication profile generalizes beyond LLaMA. Tables 6, 7, and 8 report the detailed transport breakdowns for Falcon-7B, Mixtral-8×7B, and Falcon-40B, respectively. Across model families, scales, and GPU counts, the results confirm the same communication-dominated behavior observed in the main evaluation. 15
Trovato et al.
Table 6. Transport breakdown across network environments for Falcon-7B (2 GPUs, blocks 0:16 and 16:32) at batch size 32, FP16, micro-batching off. Env. 𝑇𝐺𝑃𝑈 →𝐶𝑃𝑈 𝑇𝐶𝑃𝑈 →𝑁 𝐼𝐶 𝑇𝑁 𝐼𝐶→𝑁 𝐼𝐶 𝑇𝑁 𝐼𝐶→𝐶𝑃𝑈 𝑇𝐶𝑃𝑈 →𝐺𝑃𝑈 (ms) (ms) (ms) (ms) (ms) E1 E2 E3 E4 E5
1.23 1.32 1.26 1.25 1.23
0.25 0.27 0.26 0.26 0.26
8.28 36.02 63.47 98.79 220.90
0.07 0.13 0.08 0.08 0.08
Inference Lat. (ms)
Throughput (tok/s)
267.61 287.34 291.70 327.04 477.22
119.38 111.87 107.02 97.30 66.23
0.60 0.44 0.44 0.46 0.48
Comm. Vol. 𝑇𝐺𝑃𝑈 Compute (KB) (ms) 284.4 284.4 284.4 284.4 284.4
35.47 36.65 35.41 36.15 35.21
Server-side Net Lat. (ms) 8.36 36.15 63.54 98.87 220.98
Table 7. Transport breakdown across network environments for Mixtral-8×7B (4 GPUs, blocks 0:8, 8:16, 16:24, 24:32) at batch size 32, FP16, micro-batching off. Timing metrics are from the first server (S1→S2 push). Env. 𝑇𝐺𝑃𝑈 →𝐶𝑃𝑈 𝑇𝐶𝑃𝑈 →𝑁 𝐼𝐶 𝑇𝑁 𝐼𝐶→𝑁 𝐼𝐶 𝑇𝑁 𝐼𝐶→𝐶𝑃𝑈 𝑇𝐶𝑃𝑈 →𝐺𝑃𝑈 (ms) (ms) (ms) (ms) (ms) E1 E2 E3 E4 E5
1.25 1.24 1.25 1.26 1.32
0.25 0.25 0.25 0.25 0.25
4.01 33.31 62.42 96.60 209.70
0.07 0.08 0.08 0.08 0.11
Inference Lat. (ms)
Throughput (tok/s)
349.89 406.76 502.54 605.42 971.91
92.98 77.93 63.28 52.75 32.70
0.43 0.44 0.45 0.45 0.46
Comm. Vol. 𝑇𝐺𝑃𝑈 Compute (KB) (ms) 256.4 256.4 256.4 256.4 256.4
38.28 38.12 38.54 38.76 39.55
Server-side Net Lat. (ms) 4.08 33.39 62.49 96.68 209.81
Table 8. Transport breakdown across network environments for Falcon-40B (3 GPUs, blocks 0:20, 20:40, 40:60) at batch size 32, FP16, micro-batching off. Timing metrics are from the first server (S1→S2 push). Mean of 5 runs after warmup. Env. 𝑇𝐺𝑃𝑈 →𝐶𝑃𝑈 𝑇𝐶𝑃𝑈 →𝑁 𝐼𝐶 𝑇𝑁 𝐼𝐶→𝑁 𝐼𝐶 (ms) (ms) (ms) E1 E2 E3 E4 E5
1.31 1.35 1.40 1.34 1.34
0.39 0.37 0.39 0.36 0.38
Inference Lat. (ms)
5.64 38.14 72.30 114.66 316.42
Throughput Comm. Vol. 𝑇𝐺𝑃𝑈 Compute (tok/s) (KB) (ms)
506.2 578.1 625.4 718.6 1118.0
63.18 55.32 51.15 44.52 28.62
512.4 512.4 512.4 512.4 512.4
61.80 63.70 62.74 61.76 57.00
Table 9. Micro-batching effect on Falcon-7B (2 GPUs) and Mixtral-8×7B (4 GPUs). All runs use batch size 32, FP16, sequence length 128. Throughput in tokens/s. Falcon-7B (2 GPUs) Env. MB off MB on E1 E2 E3 E4 E5
119.38 111.87 107.02 97.30 66.23
Mixtral-8×7B (4 GPUs) Δ MB off MB on
97.37 −18.4% 91.33 −18.4% 86.75 −18.9% 79.11 −18.7% 66.80 +0.9%
16
92.98 77.93 63.28 52.75 32.70
Δ
75.55 −18.7% 66.35 −14.9% 58.09 −8.2% 49.55 −6.1% 36.42 +11.4%