Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off

Speculation at a Distance: Where Edge-Cloud Speculative Decoding Actually Pays Off Yuan Lyu, Bharath Irukulapati, and Jaya Prakash Champati

arXiv:2606.25091v1 [cs.DC] 23 Jun 2026

University of Victoria [email protected], [email protected], [email protected]

Abstract—Speculative decoding (SD) accelerates LLM inference by 1.5–3× when the draft and target models are co-located. This has motivated a distributed variant (DSD) that places the draft model on an edge device while the target stays in the cloud. We show with closed-form inequalities that DSD’s per-request latency benefit is limited under WAN edge-cloud communication. If the server can host both models, co-located SD has lower latency and communication than synchronous DSD, with the same per-output FLOPs and model-weight memory. Pipelining can make DSD competitive with co-located SD only in low-RTT regimes where the round trip is shorter than the edge drafting time window; at WAN RTTs, the cloud round trip remains too large for pipelined DSD to beat co-located SD. Against cloud autoregressive decoding, DSD can reduce latency only inside a bounded window given the target-model speed, acceptance rate, and RTT. DSD is also infeasible against closed-source APIs without a verifier-only interface. The main case for DSD appears in multi-tenant capacity. Under cross-client overlap, offloading draft compute lets a saturated cloud server sustain (1+γ td /tv )× more concurrent clients at the same per-client rate, where γ is the speculation length and td , tv are the per-step draft and verification times. DSD should therefore be evaluated primarily by multi-tenant capacity and server throughput, not only by single-request latency. Index Terms—Distributed speculative decoding, distributed inference, edge-cloud, LLM serving, latency

I. I NTRODUCTION Speculative decoding (SD) has become a standard tool for accelerating large language model (LLM) inference [1], [2]. The standard co-located SD, which places a small draft model and a large target model on the same hardware, exploits the asymmetry between sequential token generation and parallel verification to deliver 1.5–3× wall-clock speedups [3] while guaranteeing the tokens are generated according to the target distribution [1]. Driven by the proliferation of capable small language models on edge devices and their on-device performance, a growing body of work proposes a distributed variant of SD (DSD): the draft models run on edge devices such as smartphones, gateways, or on-premise servers, while the target models run in data centers, with each verification round involving communication over a network [4]–[9]. Existing DSD systems each attack a different bottleneck in the system, but share a common evaluation gap. DSSD [4] splits verification between edge and cloud, sending the vocabulary distribution on the downlink only when a draft is rejected rather than on every uplink round; TSLT [9] sends only a truncated candidate set of logits on the uplink; PicoSpec [5]

applies asynchronous pipelining to overlap edge drafting with cloud verification; DSD [6] applies an Adaptive Window Control policy that dynamically adjusts the speculation window; SLED [7] batches verification across multiple edge clients to raise server throughput; SpecEdge [10] disaggregates drafting to consumer-grade GPUs on edge devices, hiding communication and computation latency by overlapping proactive edge-side drafting and server-side pipeline-aware scheduling; PipeSD [11] adds token-batch pipeline scheduling and adaptive verification triggering; and [8] proposes resource-aware parallel speculative decoding across heterogeneous devices. Reported gains span 1.1–2.9× latency speedups in DSSD, PicoSpec, PipeSD, and DSD, or 2.2× aggregate throughput in SLED and SpecEdge. Most are measured against cloud autoregressive decoding (AR) or other DSD methods. Two exceptions reinforce our point. DSD [6] compares distributed execution with a fused mode in which the draft and target models are co-located on the cloud server, and distributed execution is faster only when the round-trip time (RTT) is below ∼50 ms; SpecEdge [10], a DSD based approach, adopts co-located SD as a primary baseline, and its inter-token latency gain holds only when RTT is below ∼50 ms; see §IV. To our knowledge, no DSD system demonstrates a latency advantage over co-located SD at WAN round-trip times. We therefore study DSD as a provider or self-hosted-operator deployment choice. Closed-API end users cannot choose co-located SD themselves; for them, DSD would require the verifier API discussed in Rem. 7. We study the edge-cloud WAN setting, with a wide-area network between the edge draft and the cloud target. Our closed-form analysis in §III–§IV separates synchronous and asynchronous DSD. For synchronous systems such as DSSD [4] and DSD [6], per-request latency gains over cloud AR exist only inside a bounded RTT window. The tolerable RTT is essentially the expected per-round target model processing time, minus a few fixed compute overheads. That budget is tight enough to exclude 4G/LTE networks, cross-region cloud, fast modern GPUs, and short-response workloads. For asynchronous systems such as PicoSpec [5], pipelining can mask RTT only while edge draft phase is at least as long as the cloud round trip plus the target verification time; §IV applies this stronger form to the co-located-SD baseline and shows that the latency advantage still does not extend to WAN RTTs. DSD does, however, retain one advantage that our analysis makes precise: under multi-tenant serving, offloading draft

TABLE I A NALYTICAL COMPARISON SUMMARIZING P ROP. 1, P ROP. 2, R EM . 7, AND P ROP. 9. A RROWS INDICATE THE FAVORABLE DIRECTION ; BOLD MARKS THE BEST ENTRY IN EACH ROW.

Metric

Cloud AR

Co-located SD

DSD

Per-request / configuration Per-token latency ↓ Medium Per-token server time ↓ High Total memory ↓ Low Network per round ↓ Low Closed-API support ✓

Low Medium Medium None ✓

High† Low Medium High ׇ

Multi-tenant Concurrent clients ↑

Baseline

High§

—∗

† Medium latency only inside the narrow RTT window of Prop. 2. ‡ See Remark 7. § (1+γtd /tv )× co-located SD under cross-client overlap; see Prop. 9. ∗ The capacity comparison concerns where draft compute runs;

cloud AR has no draft model and lies outside this comparison.

compute to the edge lets the cloud sustain more concurrent clients at the same per-client output token rate than co-located SD. This explains why the more promising direction in recent DSD systems focuses on improving multi-tenant capacity and goodput rather than single-request latency. Our contribution is to make that distinction explicit against the co-located-SD baseline: DSD’s capacity case is stronger than its per-request latency case, whose gains depend on RTT and baseline. Below are our main contributions. A closed-form per-request analysis in §III showing that co-located SD has lower latency and communication than DSD when both baselines are available, while DSD reduces latency over cloud AR only within a bounded latency window. • A multi-tenant capacity result, Prop. 9, that quantifies DSD’s capacity benefit—(1 + γ td /tv )× more concurrent clients per cloud server—and reframes DSD as a systemcapacity technique. • Result-by-result support from published measurements: the 350+ co-located SD experiments of [12] validate our effective-time model; DSSD’s network results trace the cloud-AR break-even window of Prop. 2; SLED and SpecEdge support DSD’s server-capacity benefit, while CoSine supports the same resource-allocation principle in a datacenter setting. •

Table I summarizes the closed-form comparisons in §III.

TABLE II N OTATION USED THROUGHOUT THE PAPER . Symbol

Meaning

Models and distributions Mt , Ms target model and small draft model |Mt |, |Ms | weight-memory footprints in bytes p(· | x<i ) target next-token distribution q(· | x<i ) draft next-token distribution |V | vocabulary size Speculation γ x1 , . . . , xγ ri α A E[A]

speculation length in draft tokens per round proposed draft tokens per-token acceptance prob. min(1, p(xi )/q(xi )) per-position acceptance probability (1) output tokens per round, A ∈ {1, . . . , γ+1} expected output tokens per round (3)

Per-token / per-round timing tar cloud-AR per-token wall-clock time td time per draft token tv time for one forward pass verifying γ tokens coloc , T dsd , T pipe Teff mean wall time per output token (4),(6),(7) eff eff (·) (·) Tround per-round wall time, Teff E[A] w speculative-waste fraction under pipelining Network RTT Ttx (γ) b, bprob R RTTmax

payload-independent round-trip time transmission time, γb/R (5) per-draft payload, bytes per prob. value link bandwidth break-even network budget (8)

Compute (FLOPs) and server throughput CAR , Cdraft , Cverify per-token / per-round FLOPs c draft-to-AR FLOP ratio Cdraft /CAR r required long-run output rate per admitted client Nar , Ncoloc , Ndsd maximum clients sustainable at rate r Hypothetical API cost; see Rem. 8 pin , pout per-token input / output price Fver (γ) per-call verification fee

A. Co-located SD. A small draft model Ms and large target model Mt have weight-memory footprints |Ms | and |Mt | bytes and token distributions q(· | x<i ) and p(· | x<i ) respectively. The draft model proposes γ tokens x1 , . . . , xγ ∼ q; the target model verifies them in one forward pass, accepting each xi with probability ri = min(1, p(xi )/q(xi )). The per-position acceptance probability α is the expectation of this acceptance rule over the draft distribution: i h  X
= min q(x), p(x) α ≜ Ex∼q min 1, p(x) q(x)

(1)

x

II. BACKGROUND AND S ETTING We compare three configurations, all serving the same target model Mt . The simplest is cloud AR: the target runs alone on the cloud server with no drafting and no speculation, generating one token per forward pass. Let tar denote its pertoken wall-clock cost; cloud AR is our per-request baseline throughout the paper. Table II collects the notation used throughout.

Following [1], we assume α is constant across positions. Let A ∈ {1, . . . , γ + 1} be the number of tokens produced per round. It consists of A − 1 accepted draft tokens plus one additional target token. The additional token is a correction after the first rejected draft, or a bonus token when all γ drafts are accepted. The event {A ≥ a} holds iff the first a−1 drafts were all accepted, so by independence and constant α (1): P (A ≥ a) = αa−1 .

(2)

Applying the tail-sum formula E[A] = E[A] =

γ+1 X

αa−1 =

a=1

Pγ+1

a=1 P (A ≥ a):

1 − αγ+1 . 1−α

(3)

Let td denote the time per draft token and tv the time to verify the γ tokens in parallel. For any configuration (·), we (·) define Teff as its expected wall-clock time per generated output token: the wall-clock time of one decoding round divided by the expected number E[A] of output tokens produced in that (·) round. Its reciprocal, 1/Teff , is the corresponding per-request token throughput. For co-located SD, coloc Teff =

γ t d + tv . E[A]

(4)

We adopt the memory-bound assumption tv ≈ tar : a single verification pass over the γ proposed tokens is dominated by HBM-to-SM weight transfer rather than FLOPs, so it runs in roughly the same wall-clock as one autoregressive step. Published single-node measurements support this roundtime decomposition. [12] reports the Tokens-Accepted Rate, or TAR, with TAR≡ E[A]; the time to generate γ draft tokens; the time for the target to verify them; and measured throughput for 350+ experiments on OPT-66B and LLaMA65B targets using 4×A100 GPUs. Substituting their measured draft and verification times for γtd and tv in (4) gives the coloc , which closely matches their predicted throughput 1/Teff measurements. This empirical check uses the measured tv directly and does not require the additional approximation tv ≈ tar . B. DSD (synchronous). When the draft model lives on the edge and the verifier in the cloud, each round traverses the network: the payloadindependent round-trip RTT, which includes propagation plus processing and is ping-measurable, plus a payload-dependent transmission time Ttx (γ) = γ · b/R,

(5)

where b is the per-draft payload size and R is the link bandwidth. The payload size b is set by what the verifier needs in order to check a draft token, which in turn depends on the decoding mode. Under greedy verification the cloud only checks whether each proposed token matches the target’s argmax, so the edge sends bare token IDs: b is a few bytes and the transmission time is negligible relative to the roundtrip, Ttx (γ) ≪ RTT, at common WAN bandwidths. Exact distribution-preserving verification, by contrast, must compare the target and draft probabilities of each proposed token and resample rejected positions from the residual (p − q)+ , which requires probability information over the vocabulary. Naive logit-based protocols therefore send b ≈ |V | bprob bytes per draft token, where |V | is the vocabulary size and bprob the bytes per probability value. At FP16 or BF16, this is 2 bytes per value; the payload is larger by orders of magnitude, and Ttx (γ) can then dominate RTT at lower

bandwidths. Communication-efficient distributed SD introduced by DSSD [4] uploads only token IDs and scalar draft probabilities, and defers the full distribution to a downlink transmission only on rejection, effectively reducing b. We use low-transmission-overhead regime for this communication pattern when its transfer time is small enough to omit from the per-round latency model, written Ttx (γ) ≈ 0. In a normal round, the edge sends proposed token IDs without the fullvocabulary logits. The verifier sends a full vocabulary distribution back only when it rejects a proposal and the edge needs that distribution to resample. The payload is therefore nonzero, but its expected transfer time is small relative to the roundtrip and compute terms. Separating the payload-independent round-trip from the payload-dependent transmission time, the per-token effective time is: dsd Teff =

γ td + RTT + Ttx (γ) + tv . E[A]

(6)

C. DSD (pipelined). Asynchronous DSD systems such as PicoSpec [5] overlap the next round’s edge drafting with the current round’s network transfer and cloud verification. Thus, the sequential sum in (6) becomes the maximum of the drafting and cloudside branches. A rejection can invalidate tokens drafted speculatively for the next round, so we use a speculative-waste fraction w ∈ [0, 1] to represent the resulting extra draft work:
max (1 + w) γ td , RTT + Ttx (γ) + tv pipe . (7) Teff = E[A] Here w = 0 represents perfect overlap without wasted predrafts, while larger w accounts for pipeline flushes caused by rejection. Under the memory-bound assumption tv ≈ tar , synchronous dsd DSD beats the cloud-AR baseline iff Teff < tar , equivalently the network budget is bounded by: RTT + Ttx (γ) < tar E[A] − γ td − tv ≜ RTTmax

(8)

For brevity, in the rest of the paper, we assume Ttx (γ) = 0 and the bound above reduces to RTT < RTTmax . III. P OSITION : T HE G AIN W INDOW I S NARROW We evaluate DSD against the two baselines a provider or self-hosted operator can choose, namely co-located SD, where both models run on the same server, and cloud AR, where only the target model runs and there is no draft model. For each baseline, we examine DSD via closed-form inequalities in a per-request comparison: one active decoding request under fixed service times td , tv , and tar , excluding queueing delays and resource contention from other requests. We compare stream-level latency, per-output compute, and interdevice communication, together with two configuration-level properties: total model-weight memory and API feasibility or cost. Under heavy load, batching consumes more accelerator capacity and speculative computation may cease to improve goodput; serving systems may therefore shorten or disable speculation [13].

We also assume that the draft and target models are the same across all configurations, so that the acceptance rate α and hence the expected number of accepted tokens E[A] are consistent. A. Comparison with Co-Located SD We argue that co-located SD matches or outperforms DSD across all dimensions of this comparison. Proposition 1. Assume the server has memory for both models (realistic, since |Ms | ≪ |Mt |). Then co-located SD matches or beats DSD on every comparison dimension: latency, peroutput compute (FLOPs), total model-weight memory, and inter-device communication. Proof. We establish each metric in turn. (i) Latency. Equation (6) adds the nonnegative terms RTT dsd coloc and Ttx (γ) to (4), so Teff ≥ Teff , strictly for any RTT > 0. (ii) Compute. Per-round FLOPs equal γ Cdraft + Cverify in both configurations, independent of model location. (iii) Memory. Total model-weight memory equals |Mt | + |Ms | in both configurations; only the draft’s residency differs. (iv) Communication. Co-located SD samples the probability distribution p and q from shared GPU memory and incurs zero inter-device communication. DSD transmits at least γ token IDs and one accept-count per round. By (i) and (iv), co-located SD is strictly better on latency and communication; by (ii) and (iii), it ties on compute and memory. Hence DSD has no strict per-request or configuration-level advantage. B. Against Cloud AR Cloud AR is the baseline DSD must beat to justify its existence. We work through latency, compute, API feasibility, memory, and hypothetical API cost in turn. Proposition 2 (Latency bound). DSD has lower latency than cloud AR only if
α tar − γ td + Rb , (9) RTT < 1−α where b/R is the per-draft token uplink transmission time. Proof. Substituting tv = tar under the memory-bound assumption from §II and Ttx (γ) = γb/R in (8) gives RTT < tar (E[A]−1)−γ(td +b/R). Using E[A]−1 = α(1−αγ )/(1− α) ≤ α/(1 − α) yields the bound. □ Remark 3. The bound is finite for α < 1 and tightens linearly with γ at rate td + b/R, which combines draft compute and per-draft uplink transmission. The low-transmission-overhead regime defined in §II, where Ttx (γ) ≈ 0, recovers RTT < α tar /(1 − α) − γ td ; naive logit-based protocols with b ≈ |V | bprob tighten the bound substantially. Table III illustrates the exact break-even condition in (8) for γ = 5, td = 10 ms, tv = tar , and the low-transmissionoverhead regime defined in §II. The feasible window contracts as the cloud target becomes faster or the acceptance rate falls.

TABLE III B REAK - EVEN RTT ( MS ) FROM (8). A DASH DENOTES RTTmax < 0: DSD IS SLOWER THAN CLOUD AR EVEN AT ZERO RTT, SO NO FEASIBLE BREAK - EVEN RTT EXISTS .

Cloud AR tar

α = 0.5

α = 0.7

α = 0.85

α = 0.9

Slow, 100 ms Standard, 50 ms Fast, 30 ms Very fast, 20 ms

47 — — —

144 47 8 —

265 108 45 13

319 134 61 24

For a representative 4G RTT of ∼60 ms, the tar = 100 ms target requires roughly α ≥ 0.7, the tar = 50 ms target requires roughly α ≥ 0.85, and the faster targets have no feasible point in the table. At a cross-region RTT of ∼80 ms, targets with tar ≤ 30 ms are infeasible throughout the displayed acceptance range. Published DSSD measurements exhibit the same failure mode [4]. That paper defines speedup as protocol throughput divided by target-only LLM throughput. At 50 ms nontransmission delay, 10 Mbps bandwidth, and γ = 8, its predecessor DSD method achieves only 43% of the throughput of its OPT-6.7B cloud-AR baseline. DSSD improves this to 2.19× for OPT-6.7B and 1.62× for OPT-13B by transmitting a full vocabulary distribution only on rejection. These results show that communication design can move a DSD system into or out of the cloud-AR break-even window; because the baseline is cloud AR rather than co-located SD, they do not address Prop. 1. Proposition 4 (Compute). Let c = Cdraft /CAR be the pertoken FLOP ratio of the draft model to cloud AR, and assume verifying γ tokens costs Cverify = γ CAR FLOPs. DSD uses strictly more FLOPs per output token than cloud AR whenever E[A] =

1 − αγ+1 < γ(1 + c). 1−α

(10)

This holds for all α once c ≥ 1/γ, and fails only in the corner case c < 1/γ with α → 1. Proof. A DSD round costs γ Cdraft + Cverify = γ(1 + c) CAR FLOPs and yields E[A] tokens. Therefore, its per-token FLOPs exceed cloud AR’s CAR iff γ(1 + c) > E[A]. Using E[A] ≤ γ +1, a sufficent condition for (10) to hold is γ +1 < γ(1+c), which results in c ≥ 1/γ. Remark 5. The corner case is operationally empty. It requires c < 1/γ together with α near 1 — e.g. at γ = 5, and c = 0 it needs α ≈ 0.93, with higher α demanded as c rises toward 1/γ. But c < 1/γ means the draft is nearly free to run on the server as well, so co-located SD’s draft cost is also small and DSD offloads negligible compute even as it “wins.” DSD’s FLOP disadvantage therefore holds in any realistic deployment. Remark 6 (Memory). During inference, cloud AR loads only the target model weights into cloud memory, with footprint |Mt |. DSD loads the same target model weights into cloud

memory and also loads the draft model weights on the edge device, giving a system-wide model-weight footprint of |Mt |+ |Ms |. Thus DSD has no system-wide model-weight memory advantage over cloud AR; it changes where the draft model resides, not whether the draft model exists. Remark 7 (API cost — infeasibility). Distribution-preserving SD against a closed-source commercial API as the verifier requires scoring client-proposed draft tokens, where the client needs to acquire p(xi | x<i ) for each proposed draft token xi from the target model. Mainstream closed-source APIs generally expose logprobs for generated tokens only, not for arbitrary tokens proposed by the client. Thus DSD is not implementable through today’s closed-source APIs without a custom verify-only endpoint. Even if such an endpoint were offered, DSD would have no natural API-cost advantage over provider-side co-located SD: a standard endpoint already charges pout per generated token regardless of whether the provider uses SD internally, while DSD would add a separate verification cost. Pricing that cost low enough to undercut the standard output service would cannibalize provider revenue. Remark 8 (Cost for hypothetical verifier APIs). Suppose a provider offered a verify-only endpoint charging a percall verification fee Fver (γ) (absorbing any bundled correction/bonus), with pin and pout the per-token input and output charges. A DSD round then costs γ pin + Fver (γ) for E[A] tokens, so DSD is cheaper than cloud AR iff E[A] >

γ pin + Fver (γ) . pout

(11)

The right-hand side of (11) is the cost of one verification round, expressed in units of the ordinary output-token price. DSD is economical only when that normalized round cost is smaller than the expected number of output tokens produced by the round. The conclusion therefore depends on a provider’s hypothetical verification cost: charging for all proposed tokens makes break-even difficult, whereas a low flat verification fee makes it possible at moderate acceptance rates. C. Multi-Tenant Server Capacity The single-client per-token times above describe one edge– cloud pair. We now turn to the server side: at a common required per-client output rate, how many concurrent clients can the cloud sustain under each protocol? Proposition 9 (Server-capacity gain under multi-tenant serving). Let Nar , Ncoloc , and Ndsd be the maximum numbers of clients a saturated, work-conserving server can sustain under cloud AR, co-located SD, and DSD at a common perclient output rate r. Assume cross-client overlap: the server fills each client’s edge-drafting and network-transit phase with verification work for other clients, as in batched cloud inference. Then Nar : Ncoloc : Ndsd = 1 :

E[A] tar E[A] tar : . γ t d + tv tv

(12)

Under the memory-bound condition tv ≈ tar , this reduces to Nar : Ncoloc : Ndsd ≈ 1 :

E[A] : E[A], 1 + γ td /tv

(13)

with E[A] from (3). Because saturated aggregate throughput is N r at the common rate, it scales by the same ratios. Proof. Normalize the server’s available occupancy to one. A cloud-AR client producing r tokens per second consumes occupancy rtar , so Nar = 1/(rtar ). A co-located SD round occupies the server for γ td + tv and yields E[A] tokens, so each client consumes r(γ td + tv )/E[A] and Ncoloc = E[A]/[r(γ td + tv )]. Under DSD, drafting moves to the edge and a round occupies the server only for tv , giving Ndsd = E[A]/(rtv ). The DSD occupancy excludes edge drafting, network transit, and transmission because cross-client overlap lets the server process other clients during those phases. Normalizing the three client counts by Nar gives (12); substituting tv ≈ tar gives (13). Note that the overlap assumption in Prop. 9 is across clients — batching independent clients’ verifications — and is weaker than the within-client pipelining of §IV, which fails in the WAN regime; it requires only enough concurrent clients to keep verification work available during each idle phase. With partial overlap, the DSD-over-co-located capacity factor 1+γtd /tv should be read as an upper bound: realized gains degrade when the server cannot fully fill a client’s edge-drafting or network phase with verification work from other clients. With a single client that condition is empty: the round takes γ td +tv under co-located SD versus γ td +RTT+Ttx (γ)+tv under DSD, so DSD merely produces the same E[A] tokens more slowly. Remark 10. Let ρ = tv /tar . Equation (12) gives Ndsd /Nar = E[A]/ρ. The specialization Ndsd /Nar ≈ E[A] therefore requires ρ ≈ 1. MagicDec shows that this condition depends on batch size and context length: at large batch with short contexts, verification becomes compute-bound and ρ rises, reducing or reversing DSD’s capacity advantage over cloud AR [14]. In the idealized compute-bound limit ρ ≈ γ, the ratio approaches E[A]/γ. DSD then supports more clients than AR only in the high-acceptance corner E[A] > γ, and even perfect acceptance gives at most (γ + 1)/γ. Beyond a hardware- and model-dependent critical sequence length, KV-cache loading again dominates and ρ ≈ 1 can hold even at large batch [14]. The comparison with co-located SD does not require tv ≈ tar : Ndsd /Ncoloc = 1 + γ td /tv . This factor approaches 1 specifically when tv /(γ td ) grows, rather than merely whenever tv grows. Remark 11. Under the memory-bound case (13), two readings matter. First, Ndsd /Nar = E[A] while Ndsd /Ncoloc = 1 + γ td /tv : co-located SD alone already captures most of the capacity gain over cloud AR, and distributing the draft adds only the (1 + γ td /tv ) factor on top. Second, this distribution factor requires enough concurrent clients for cross-client overlap; with a single client the condition is absent, and DSD

merely adds network latency over co-located SD, as stated in Prop. 1. Published multi-tenant results support both comparisons in this remark. SLED [7] reports a 2.2× system-throughput improvement for its DSD framework over cloud AR, consistent with the memory-bound Ndsd /Nar ≈ E[A] branch at typical E[A] ≈ 2–3. SpecEdge [10] instead compares against colocated SD: offloading tree drafting raises server throughput by 2.22×, illustrating the separate Ndsd /Ncoloc = 1 + γtd /tv branch at a draft-heavy operating point. CoSine [15] gives complementary datacenter evidence that separating draft and verification resource demands can improve throughput. These measurements support the direction of the capacity result; they do not establish a universal numerical factor because the systems use different models, batching policies, drafting mechanisms, and interconnect regimes. Remark 12. For providers or self-hosted operators, co-located SD has lower latency and communication than DSD and matches DSD on FLOPs and memory by Prop. 1; DSD reduces latency over cloud AR only in the bounded window of Prop. 2. Closed-API end users instead face the API limitation in Rem. 7. Under multi-tenant serving, speculation alone already gives co-located SD a large capacity gain over cloud AR — the server verifies E[A] tokens per forward pass instead of generating one. DSD’s additional capacity benefit over colocated SD is the (1 + γ td /tv )× more clients supported by offloading drafting from the server. Prop. 9 shows that this factor is governed entirely by the on-server draft cost γ td /tv , significant only when drafting is expensive, and shrinking as native multi-token prediction cheapens on-server drafting; see §IV. DSD is not a faster way to serve one user — it is a cheaper way to serve many. IV. W HAT P IPELINING C HANGES (A ND W HAT I T D OESN ’ T ) The latency part of Prop. 1 assumes synchronous DSD. A natural objection is that an asynchronous protocol could hide the RTT behind edge drafting and thereby overturn DSD’s latency disadvantage relative to co-located SD. Equations (6) and (7) make this distinction explicit: synchronous DSD adds edge drafting, network, and verification time, whereas pipelined DSD takes the maximum of the overlapped drafting and cloud-side branches. Most DSD systems we discuss, including DSSD [4] and DSD [6], use the synchronous structure already covered above. PicoSpec [5] and PipeSD [11] are recent asynchronous cases, so the following proposition tests whether this stronger pipelined variant changes the latency pipe ordering in the edge-cloud WAN regime. Here Tround and coloc X X Tround denote per-round wall times, Tround = Teff E[A]; w is the speculative-waste fraction from (7); and the lowtransmission-overhead regime is defined in §II. The condition RTT ≥ γtd describes the case where one cloud round trip takes at least as long as drafting one full speculation round on the edge. In this case, the pipeline cannot hide the round trip inside edge drafting; the network branch is at least as large as the co-located drafting branch before verification is added.

Proposition 13 (DSD with pipelining does not dominate co-located SD in the WAN regime). In the low-transmissionoverhead model obtained by setting Ttx (γ) = 0 and for pipe coloc RTT ≥ γtd , Tround ≥ Tround X X Proof. Write Tround ≜ Teff E[A] for per-round wall time under configuration X. From (7) and (4), after setting Ttx (γ) = 0, pipe Tround = max((1 + w)γtd , RTT + tv ),

(14)

coloc Tround = γtd + tv .

(15)

If RTT ≥ γtd , then the cloud-side branch satisfies RTT+tv ≥ pipe coloc γtd +tv = Tround . Since Tround is the maximum of this branch pipe coloc and the draft branch, Tround ≥ Tround . PipeSD [11] is another example of what pipelining can buy inside the DSD design space. It overlaps edge draft generation and communication using a dynamic-programming tokenbatch scheduler, and it triggers cloud verification with a dualthreshold rule. Its reported 1.16–2.16× speedups are measured against DSD baselines such as Vanilla, HSL, and EdgeLLM, rather than against co-located SD. PipeSD therefore supports the claim that pipelining improves DSD implementations, but it does not change Prop. 13: the relevant comparison for this paper remains pipelined DSD versus co-located SD. Using the same illustrative values as Table III, γ = 5 and td = 10 ms, we have γtd = 50 ms. In this case, the condition RTT ≥ γtd in Prop. 13 holds for 4G networks, where RTT ≈ 60 ms, and for cross-region cloud, where RTT ≈ 80 ms. Pipelined DSD may dominate co-located SD when the network latency is sufficiently low, for instance RTT < γtd ; this can occur on nearby WiFi, metro-edge, or favorable 5G links. Prop. 13 says that this possibility disappears once commodity-WAN RTT exceeds the edge drafting window. Two recent systems exploit the RTT < γtd regime, but only inside the datacenter. CoSine [15] splits the matrix-matrix multiplication and the matrix-vector multiplication parts of LLM inference computation across heterogeneous accelerators connected by a 10 Gbps datacenter network. DSI [16] runs speculative branches in parallel across multiple GPUs and proves that this datacenter-parallel schedule is no slower than non-speculative decoding, and no slower in expectation than standard speculative decoding. Both are datacenter-internal parallel-inference systems operating at sub-millisecond RTT. They are complementary to our scope, rather than latency targets for our WAN comparison, because CoSine relies on a fast datacenter interconnect and DSI on abundant parallel accelerators. Neither removes the WAN round trip. CoSine nevertheless supports our capacity argument by showing that drafting and verification can contend for different accelerator resources under load, and that separating their execution across heterogeneous resources can improve serving throughput. SpecEdge [10] is consistent with the RTT < γtd branch for nearby WiFi, metro-edge, or favorable 5G links. In Prop. 13, γtd is the wall-clock time spent drafting one speculation round. SpecEdge uses tree-based drafting with adaptive depth, so the analogous quantity is the edge draft-phase time, es-

timated as draft depth times per-pass draft latency. In its 32B/1.7B calibration, verification averages 94.2 ms and each draft forward pass takes about 11 ms. SpecEdge therefore uses depths 7, 5, and 4 at RTTs of 15, 40, and 50 ms, giving edge draft times of about 77, 55, and 44 ms. Thus the low-RTT operating points satisfy the same timing condition as RTT < γtd , whereas the 50 ms point is already at the boundary where RTT is no longer hidden by edge drafting. This matches its reported latency behavior: at low RTT, SpecEdge reports 36.5 ms inter-token latency versus 42.4 ms for co-located SD, while its sensitivity analysis places the crossover near 50 ms RTT; at 65 ms RTT, SpecEdge reaches 44.5 ms and is slower than the co-located baseline. Two effects narrow DSD’s remaining capacity advantage. First, MagicDec observes that verification can become compute-bound in large-batch, short-context settings [14]. In that regime, tv /tar rises, so Prop. 9 predicts a smaller DSD capacity gain over cloud AR. Second, native multi-token prediction in frontier models such as DeepSeek-V3 [17] and Qwen3-Next [18] cheapens co-located proposal generation. For the comparison with co-located SD, these target-side proposal heads replace the separate draft-model cost γtd with a smaller proposal overhead on the target. As that overhead falls, the capacity gain from moving draft work to the edge moves closer to no gain. V. C ONCLUSION The analysis in §III–§IV shows that, per request, DSD does not improve latency, compute, or memory over co-located SD, and reduces latency over cloud AR only inside the bounded, tar -dependent window of Prop. 2; see Table III. DSD is also infeasible against closed-source APIs. For the latency comparison with co-located SD, asynchronous pipelining does not make DSD faster in the WAN regime by Prop. 13. The main case for DSD is multi-tenant capacity: under cross-client overlap, offloading draft compute lets a saturated server sustain (1+γ td /tv )× more clients at the same per-client output token rate by Prop. 9. Thus DSD should be evaluated primarily by multi-tenant capacity and server throughput, rather than by single-request latency alone. Latency claims should report the RTT range where they hold and whether the baseline is cloud AR or co-located SD. DSD’s remaining capacity advantage becomes smaller in regimes where verification is computebound, as observed for large-batch, short-context settings, and when native multi-token prediction is adopted in target models that reduce co-located drafting cost. Our reframing implies three reporting practices. First, report the break-even RTT of Prop. 2 together with its parameter setting, so a reader can tell whether a system applies to their deployment. Second, sweep (α, RTT, γ) at several target speeds tar rather than reporting a single operating point: the viable region is a surface, not a point, and single-condition speedups are easily cherry-picked. Third, report multi-tenant server throughput, not just single-user speedup, and specify the communication protocol (greedy vs. logit-based), whose payload differs by orders of magnitude and sets the latency

window. Throughout, co-located SD—not cloud AR—is the baseline to beat by Prop. 1. We leave to future work several deployment-level concerns where DSD may add value but where no controlled comparison yet exists: user-prompt privacy (standard DSD still sends the prefix to the cloud, though the architecture admits encrypted or partial-prefix variants), per-user personalized draft models at scale (where co-located SD would require the server to hold N draft models), perceived time-to-first-token from showing unverified drafts, and graceful degradation under intermittent connectivity. These are legitimate directions, but orthogonal to the latency claims the literature currently makes. R EFERENCES [1] Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in International Conference on Machine Learning (ICML), 2023, pp. 19 274–19 286, arXiv:2211.17192. [2] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” arXiv:2302.01318, 2023. [3] H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui, “Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding,” arXiv:2401.07851, 2024. [4] J. NING, C. ZHENG, and T. Yang, “DSSD: Efficient edge-device deployment and collaborative inference via distributed split speculative decoding,” in ICML Workshop on Machine Learning for Wireless Communications, 2025. [5] Y. Zhang, Z. Gao, S. Yue, J. Li, and R. Wang, “A pipelined collaborative speculative decoding framework for efficient edge-cloud llm inference,” arXiv:2603.19133, 2026. [6] F. Yu, L. Li, B. McDanel, and S. Q. Zhang, “Dsd: A distributed speculative decoding solution for edge-cloud agile large model serving,” arXiv:2511.21669, 2025. [7] X. Li, D. Spatharakis, S. Ghafouri, J. Fan, H. Vandierendonck, D. John, B. Ji, and D. Nikolopoulos, “Sled: A speculative llm decoding framework for efficient edge serving,” arXiv:2506.09397, 2025. [8] J. Koh and H. J. Yang, “Collaborative large language model inference via resource-aware parallel speculative decoding,” arXiv:2511.01695, 2025. [9] C. Zheng, K. Zhang, C. Sun, W. Zhang, Q. Liu, and A. A. Tesfay, “Fast collaborative inference via distributed speculative decoding,” arXiv:2512.16273, 2025. [10] J. Park, S. Cho, and D. Han, “Specedge: Scalable edge-assisted serving framework for interactive llms,” Advances in Neural Information Processing Systems, vol. 38, pp. 92 668–92 694, 2026. [11] Y. Han, Y. Gao, B. Hu, M. B. Mashhadi, Y. Duan, P. Xiao, and Y. Zhang, “Pipesd: An efficient cloud-edge collaborative pipeline inference framework with speculative decoding,” arXiv:2605.13319, 2026. [12] M. Yan, S. Agarwal, and S. Venkataraman, “Decoding speculative decoding,” in North American Chapter of the Association for Computational Linguistics (NAACL), 2025, pp. 6460–6473, arXiv:2402.01528. [13] X. Liu, J. Park, L. Hu, W. Kwon, Z. Li, C. Zhang, K. Du, X. Mo, K. You, A. Cheung, Z. Deng, I. Stoica, and H. Zhang, “Turbospec: Closedloop speculation control system for optimizing llm serving goodput,” arXiv:2406.14066, 2024. [14] R. Sadhukhan, J. Chen, Z. Chen, V. Tiwari, R. Lai, J. Shi, I. E. Yen, A. May, T. Chen, and B. Chen, “Magicdec: Breaking the latencythroughput tradeoff for long context generation with speculative decoding,” in International Conference on Learning Representations (ICLR), 2025. [15] L. Gao, J. Liu, H. Xu, X. Zhang, Y. Liao, and L. Huang, “Collaborative speculative inference for efficient llm inference serving,” arXiv:2503.10325, 2025. [16] N. Timor, J. Mamou, D. Korat, M. Berchansky, O. Pereg, M. Wasserblat, T. Galanti, M. Gordon, and D. Harel, “Distributed speculative inference (dsi): Speculation parallelism for provably faster lossless language model inference,” arXiv:2405.14105, 2024. [17] DeepSeek-AI, “Deepseek-v3 technical report,” arXiv:2412.19437, 2024. [18] Qwen Team, “Qwen3-next: Towards ultimate training & inference efficiency,” Qwen blog, 2025.