ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving Xiangchen Li
Jiakun Fan
Queen’s University Belfast Belfast, Northern Ireland, UK [email protected]
Virginia Tech Blacksburg, Virginia, USA [email protected]
Babar Ali
Hans Vandierendonck
Dimitrios S. Nikolopoulos
Queen’s University Belfast Belfast, Northern Ireland, UK [email protected]
arXiv:2604.09722v1 [cs.DC] 8 Apr 2026
Saeid Ghafouri
Virginia Tech Blacksburg, Virginia, USA [email protected]
Queen’s University Belfast Belfast, Northern Ireland, UK [email protected]
Virginia Tech Blacksburg, Virginia, USA [email protected]
Abstract
1
Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configurationselection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft–target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (𝐾 ∗ =2–10). Secondly, both cost and energy efficiency converge to 𝐾=2 due to a dominant bonus-token effect—with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge–cloud LLM inference.
Deploying Large Language Models (LLMs) closer to end users has become increasingly attractive due to the need to reduce end-toend latency, limit bandwidth usage [14], and preserve data locality and privacy [7]. Many interactive, streaming, and Internet of Things (IoT) applications in the fields of healthcare, industrial IoT, surveillance, etc., benefit from near-source inference execution, where responsiveness and data sensitivity are critical, enabling LLM inference on edge and mobile platforms [17]. For on-device inference, a wide range of model efficiency techniques have been explored, including quantization [15], pruning [5], and architectural simplification, which aim to reduce computation, memory footprint, and energy consumption on resource-constrained hardware. While these techniques can substantially lower inference cost and enable partial deployment of LLM models on edge devices, they face fundamental limits, as aggressive compression often leads to accuracy degradation and diminishing returns [2]. Consequently, fully executing high-capacity LLMs on edge devices alone remains impractical for many workloads, motivating hybrid cloud–edge execution models in which IoT devices offload sensory insights to edge, where edge devices perform lightweight or partial inference while delegating expensive computation to centralized infrastructure [9]. Speculative decoding is a decoding paradigm for autoregressive language models in which a lightweight draft model proposes multiple tokens ahead of time, and a higher-capacity target model subsequently verifies them against the target distribution. The target model accepts correct prefixes and discards the remaining, which eliminates divergence likelihood [8]. Crucially, speculative decoding preserves the output distribution of the target model despite using an auxiliary draft model [1, 8]. This process decouples generation into a drafting phase that prioritizes speed and a verification phase that enforces correctness [8, 12]. By enabling multiple tokens to be proposed and validated in a single step, speculative decoding exposes opportunities to overlap computation, amortize verification cost, and restructure inference across heterogeneous resources. Speculative decoding can substantially improve throughput, system capacity, and cost efficiency in heterogeneous edge environments [10, 11, 16]. By distributing generation across devices with different computational capabilities, these systems achieve better utilization of both edge and server resources while maintaining output quality. However, speculative decoding exposes a fundamental challenge in practice that is orthogonal to mechanism design, namely configuration selection. Designers must decide which draft model variants to deploy on what edge devices, and how to set the
CCS Concepts: • Computing methodologies → Distributed computing methodologies; Distributed artificial intelligence; Natural language processing; • Computer systems organization → Distributed architectures; • General and reference → General conference proceedings. Keywords: Speculative Decoding, Large Language Models, Edge Computing, Distributed Inference, Token Verification ACM Reference Format: Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos. 2026. ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving. In 4th International Workshop on Testing Distributed Internet of Things Systems (TDIS ’26), April 27–30, 2026, Edinburgh, Scotland Uk. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3802513.3803483
This work is licensed under a Creative Commons Attribution 4.0 International License. TDIS ’26, Edinburgh, Scotland Uk © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2608-8/26/04 https://doi.org/10.1145/3802513.3803483
Introduction
TDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos
speculative decoding length, choices that directly impact performance, resource efficiency, monetary cost, and energy consumption across heterogeneous hardware. In realistic deployments, these configurations interact in non-trivial ways with hardware constraints, workload characteristics, and service-level objectives. Larger draft models typically improve token acceptance rates, but consume more compute and memory on the device, potentially limiting concurrency and degrading energy efficiency. No single configuration dominates across devices, workloads, and optimization objectives. Selecting configurations for distributed speculative LLM serving is challenging because system behavior depends on the interaction between draft model family, model size, quantization level, edge hardware, speculative decoding length, and target model alignment. These factors jointly determine verified-token throughput, verification cost efficiency, and energy efficiency, and their effects are often non-intuitive. For example, increasing draft model size improves acceptance rate but may reduce goodput and increase energy per verified token; similarly, platform differences can dominate model-size differences. As a result, configuration decisions cannot be inferred from model accuracy or hardware capability alone, and require systematic profiling and cross-metric evaluation. In this paper, we present ConfigSpec, a framework that combines systematic device profiling, draft–target alignment measurement, and analytical performance modeling to evaluate configuration quality. ConfigSpec characterizes each draft model and edge platform using measurable quantities—drafting throughput, acceptance rate against a target model, and device power draw—and analytically factors cloud verification effects into a set of parameters, mapping these profiles to deployment-relevant metrics including goodput, verification cost efficiency, and energy efficiency. By grounding configuration evaluation in measurable primitives while preserving the structural properties of speculative decoding, ConfigSpec enables fast, repeatable comparison of model families, quantization levels, and edge platforms prior to large-scale deployment. In summary, this paper makes the following contributions: • We identify configurations—specifically the joint choice of draft model, speculative decoding length, and edge platform— as a primary determinant of performance, cost, and energy in distributed edge–cloud speculative LLM serving. • We design ConfigSpec, a profiling-based configuration evaluation framework that measures drafting throughput, draft– target acceptance rate, and device power, and systematically maps them to system goodput, cost, and device energy. • Exhaustive enumeration of the joint (𝑀, 𝑄, 𝐾) configuration space reveals structurally conflicting optima: goodput favours smallest draft at device-dependent 𝐾 ∗ =2–10, while both cost and energy converge to 𝐾=2 via a bonus-token effect—yet cost selects the largest drafter and energy the smallest. These conflicts produce trade-offs of up to 2.9× in goodput, 2.2× in cost, and 7.8× in energy between objectiveoptimal configurations on same device, confirming that no single fixed setting can simultaneously optimise all metrics.
2
Related Work
Related works are categorized into three groups: speculative decoding algorithms, speculative decoding systems for edge and distributed serving, and edge–cloud collaboration for LLM inference.
Speculative Decoding Algorithms. Speculative decoding was originally introduced as a lossless acceleration technique for autoregressive generation, enabling multiple tokens to be proposed in parallel by a lightweight draft model and verified by a larger target model. Leviathan et al. [8] formalize speculative decoding and prove that it preserves the output distribution of standard decoding while achieving substantial speedups without retraining or architectural changes. Subsequent work extends speculative decoding to new regimes and workloads. MagicDec [13] demonstrates that speculative decoding remains effective for moderate to long sequences and high-throughput inference, identifying bottleneck shifts with batch size and proposing adaptive drafting strategies to improve throughput and latency. Speculative Decoding Systems for Edge and Distributed Serving. Recent systems integrate speculative decoding into practical LLM serving architectures. EdgeLLM [16] applies speculative decoding to on-device inference, addressing memory constraints by combining draft models with efficient branch navigation, adaptive fallback, and compute–IO pipelining. SLED [11] reinterprets speculative decoding as a mechanism for collaborative edge computing, allowing heterogeneous edge devices to draft tokens locally while a shared server verifies them in batches. WISP [10] further identifies wasted drafting time and verification interference as key scalability bottlenecks in distributed speculative serving and proposes dynamic drafting and SLO-aware batching to improve system capacity and goodput. Edge–Cloud Collaboration for LLM Inference. Beyond speculative decoding, several works study collaborative inference across edge and cloud resources. CLONE [14] explores algorithm–hardware co-design for latency-aware LLM inference on edge devices, relying on profiling and deployment-driven evaluation to optimize performance and energy efficiency. EdgeShard [18] proposes partitioning LLMs into shards distributed across collaborative edge devices and cloud servers, optimizing device selection and partitioning decisions under heterogeneity and bandwidth constraints. While these approaches demonstrate the benefits of collaboration between edge and cloud, they do not model speculative decoding or address configuration selection across draft variants and speculative lengths.
3
System Framework
ConfigSpec provides a configuration-profiling framework for distributed speculative LLM serving across heterogeneous edge platforms, as illustrated in Figure 1. In the deployment model we consider, multiple edge devices execute inference requests using local draft models, while a centralized verifier hosts a target model that enforces correctness under speculative decoding semantics [8]. ConfigSpec evaluates configurations through profiling combined with an analytical model of speculative decoding rounds.
3.1
Framework
Serving abstraction. An inference request consists of autoregressive token generation. On each edge device, a draft model proposes tokens locally and submits them for verification by the target model. The verifier accepts the longest valid prefix and produces a corrective token upon divergence. Only accepted tokens contribute to final output.
ConfigSpec : Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving
Edge Devices Speculative Draft A
i >
.
i
Speculative Token Block
1
|--------- ]
V
H
el ecti J
A
Accept Prefix:
Token 1, Token 2
Verified Tokens
1
L
e
Target Model
Draft Model B Draft Model C
T ~---|
Table 1. Acceptance rate (𝛼) of draft models against each target model with speculative length of 5. Higher values indicate stronger alignment between draft and target distributions.
Cloud Verifier
""""""""""""
Draft Model
.
il
L
i
Target: Llama-3.1-70B
)
Configuration Evaluation Profiling Inputs =» Estimate Goodput
Draft Throughput v
4 Maximize Goodput
=p Estimate Cost Efficiency
(tokens/s)
=P Estimate Energy per Verified Token
Accepted Tokens
$ Minimize Cost
/ Drafted Tokens
$ Dicost,
TDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Minimize Energy
iifi
Target: Qwen3-32B
Draft Model
𝜶
Draft Model
𝜶
Llama-3.2-1B Llama-3.2-1B-Instruct Llama-3.2-3B-Instruct Llama-3.1-8B
0.462 0.546 0.572 0.622
Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B
0.378 0.466 0.487 0.522
[Frmrrermemmmcmmossmeeess Configuration Evaluation
4 Maximize Goodput
~ $ Minimize Cost
Figure 1. System abstraction used by ConfigSpec. Heterogeneous edge devices generate speculative tokens using local draft models and interact with a centralized cloud verifier. Configuration profiling. For each draft model and edge platform, ConfigSpec profiles: (i) drafting throughput 𝑣𝑑 , (ii) draft–target acceptance rate 𝛼 (𝐾), (iii) device power draw 𝑃. Configuration space and selection. A configuration consists of a target model at the verifier, a draft model variant, and an edge platform. Given the measured profile, ConfigSpec evaluates configurations under objectives such as maximizing goodput, minimizing verification cost per token, or minimizing edge device energy per verified token. 3.2
Performance, Cost, and Energy Model
We model distributed speculative decoding using the same round abstraction employed in our evaluation. In each speculative round, an edge device drafts 𝐾 candidate tokens at rate 𝑣𝑑 (tokens/s), which are verified by a cloud-hosted target model with acceptance rate 𝛼 (𝐾) and verification latency 𝑇verify . Goodput. We define the goodput as the verified token throughput under the speculative decoding round model: 𝐺 (𝐾) =
𝐾 · 𝛼 (𝐾) + 1 𝐾/𝑣𝑑 + 𝑇verify
[tok/s],
(1)
where the numerator counts the expected accepted tokens per round (𝐾𝛼 (𝐾) from the draft plus one bonus token at the first rejection), and the denominator is the round latency comprising local drafting time 𝐾/𝑣𝑑 and remote verification latency 𝑇verify . Verification Cost Efficiency. Under a token-priced billing model with unit price 𝑝 ($/token), each speculative round processes approximately 𝐾 tokens at the verifier. The cost efficiency in accepted tokens per dollar is: 𝜂 cost =
𝐾 · 𝛼 (𝐾) + 1 𝛼 (𝐾) + 1/𝐾 = 𝐾 ·𝑝 𝑝
[tok/$],
(2)
which depends only on the acceptance rate 𝛼 (𝐾), speculative length 𝐾, and verifier price 𝑝. In practice, continuous batching and parallel prefill/decode can influence this cost. Accounting for the existing conflicting objectives and diverse search space, we considered token-priced billing for decoding without batching.
Energy Efficiency. Let 𝑃 denote the average power draw of the edge device during drafting. Since verification occurs in the cloud, only local drafting time contributes to on-device energy consumption. The energy per verified token is: 𝐸=
𝑃 · 𝐾/𝑣𝑑 𝐾 · 𝛼 (𝐾) + 1
[J/tok],
(3)
where the numerator is the drafting energy per round and the denominator is the expected number of accepted tokens. The purpose of ConfigSpec is configuration-level exploration. For each configuration (draft model, device, and target model), ConfigSpec measures 𝑣𝑑 , 𝛼 (𝐾), and 𝑃, and evaluates 𝐺 (𝐾), 𝜂 cost , and 𝐸 using the analytical model above. Because 𝜂 cost is independent of drafting speed and 𝐸 depends only on 𝑣𝑑 and 𝑃, configurations can be compared by profiling edge-side behavior and measuring draft–target alignment, while treating cloud verification latency as a parameter.
4
Evaluation
We evaluate the effectiveness of distributed speculative decoding across heterogeneous edge platforms by profiling a comprehensive set of draft model configurations and speculative lengths. Our analysis proceeds along three axes—goodput, verification cost efficiency, and energy efficiency—and culminates in a unified selection framework that maps deployment constraints to recommended configurations. Experimental Setup. We deploy draft models on three representative edge platforms: Raspberry Pi 4B (RPi 4B, Cortex-A72, 8 GB RAM), Raspberry Pi 5 (RPi 5, Cortex-A76, 8 GB RAM), and NVIDIA Jetson AGX Orin (64 GB unified memory, Ampere GPU). All draft models are served via llama.cpp in GGUF format with quantization variants ranging from Q4_K_M to Q8_0. Two target models are considered: Llama-3.1-70B and Qwen3-32B, both hosted on a remote cloud verifier. We use the instruction prompts from the Databricks Dolly 15K dataset [3] as input prompts. The dataset contains 15,011 instruction-following records created by Databricks contributors. 4.1
Goodput Analysis
4.1.1 Results. Fig. 2 reveals that the optimal speculative length 𝐾 ∗ is not a universal constant but varies significantly with both device speed and draft model size. On the slowest device (RPi 4B), 𝐾 ∗ = 2 almost universally, as the high per-token drafting cost (1/𝑣𝑑 ) penalises additional speculation. In contrast, the Jetson AGX Orin pushes 𝐾 ∗ to 5–8 for Llama and 4.5-7 for Qwen family, since its fast drafting makes the incremental cost of extra tokens negligible
TDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos
Goodput vs Speculative Length (Tverify = 0.5s, ★ = optimal K) Llama-3.1-70B — RPi 5
2.5
Goodput (tok/s)
7.0
3.5 6.5 3.0
1.5
6.0
2.5
5.5
2.0
1.0
5.0
1.5 0.5
4.5
1.0
Qwen3-32B — RPi 4B
Qwen3-32B — RPi 5
4.0
Qwen3-32B — Jetson AGX Orin
3.0
2.0
Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B
6.0
3.5
2.5
Goodput (tok/s)
Llama-3.2-1B Llama-3.2-1B-Inst. Llama-3.2-3B-Inst. Llama-3.1-8B-Inst.
7.5
4.0 2.0
Cost Efficiency of Draft Models (K=5) Target: Llama-3.1-70B Target: Qwen3-32B
Llama-3.1-70B — Jetson AGX Orin
4.5
Accepted Tokens per $
Llama-3.1-70B — RPi 4B
2.5 5.0
2.0 1.0
1.5
4.5
2
3
4
5
6
7
8
9
10
2
3
Speculative Length K
4
5
6
7
8
9
4.0
10
2
3
4
Speculative Length K
5
6
7
8
9
10
Goodput (tok/s)
Target: Llama-3.1-70B (K=5, Tverify =0.5s) RPi 4B
m
k_
4_
-q st
Goodput (tok/s)
m
k_
4_
-q st
m
k_
4_
q e-
k 4_
in b-
_m
st
-q
st
s ba b-
-in 3b
-in
8
1
0
8_
-q
6 4 2
m
k_
_ q4
_0 q8
se ba b-
e-
as -b
7 1.
_m
_k
q4 e-
4b
s ba
6
0.
_0
_m
_k
q4 e-
s ba b-
8 -q
se
0.
6
ba b-
m
0
8_
k_
4_
q e-
s ba b-
8
829K
858K
500K 250K
n3-0
Qwe
B B .6B .7B n3-4 n3-8 n3-1 Qwe Qwe Qwe
-q se
4
ba b-
draft tokens are produced quickly, the system must still wait for the remote server to verify them. As 𝑇verify dominates the roundtrip time, additional drafting speed yields diminishing returns on goodput. On the RPi 4B, all models above 1B fall below 1 tok/s, rendering them impractical for interactive use. These results reveal a consistent structural trend across both model families: on edge devices, goodput is governed by the interplay between drafting speed and verification latency. 4.2
3b
Target: Qwen3-32B (K=5, Tverify =0.5s)
0
7b
Jetson AGX Orin
2
-in 1b
1.
RPi 5
4
0
736K
1224K
Speculative Length K
Figure 2. Goodput vs. speculative length across devices and models.
6
750K
1165K
980K
913K
Figure 4. Cost efficiency of draft models.
1.0
0.5
1128K
1000K
0K uct -1B -3B -8B a-3.2 -Instr lama-3.2 lama-3.1 Llam a-3.2-1B L L m Lla
5.5
1.5
1250K
k
6_
-q
se
ba b-
8
Figure 3. Verified token generation speed. relative to the fixed verification latency 𝑇verify , which is effectively amortised over more candidates. 𝑇verify has been carefully selected based on historical experiments where the considered target models have been observed taking on average 0.5s to verify tokens, and it can vary for different target models and underlying hardware. Moreover, within each device, smaller draft models (e.g., Llama-3.21B, Qwen3-0.6B) consistently favour higher 𝐾 ∗ than their larger counterparts, following the same cost–benefit logic. On fast devices the goodput peaks are broad, meaning the exact choice of 𝐾 is forgiving; on slow devices the curves are nearly flat, making 𝐾 inconsequential. These observations suggest a practical guideline: set 𝐾 = 2 on resource-constrained platforms and 𝐾 ≥ 5 on GPUclass accelerators, with the precise value tuned per draft–target pair via a lightweight sweep. Goodput results. Fig. 3 reports the goodput for Llama- and Qwen-family draft models across three devices. In both families, the smallest draft models achieve the highest goodput: Llama-3.2-1B (Q4_K_M) reaches approximately 6.5 tok/s on the Jetson AGX Orin, while Qwen3-0.6B (Q4_K_M) reaches roughly 5.8 tok/s. Scaling up the draft model consistently degrades goodput despite improving 𝛼; for instance, moving from Llama-3.2-1B to 3.1-8B raises 𝛼 from 0.46 to 0.62 yet cuts goodput by more than 4× on the Jetson. A notable observation is that the Jetson AGX Orin does not outperform the Raspberry Pi devices by as wide a margin as its raw drafting speed would suggest. While the Jetson drafts tokens 6.5–16.2× faster than the RPi 5, its goodput advantage is only 1.5–2×. This is because the verification latency 𝑇verify is shared across all devices—once the
Verification Cost Efficiency
4.2.1 Pricing Data. We adopt publicly listed inference pricing from major providers as representative verification costs. For the Llama-3.1-70B target, we use the Fireworks AI serverless tier for models exceeding 16B parameters, priced at $0.90 per 1M tokens [4]. For the Qwen3-32B target, we use Groq’s on-demand pricing at $0.59 per 1M tokens [6]. 4.2.2 Results. Fig. 4 shows that cost efficiency increases monotonically with draft model capacity in both families, consistent with Eq. (1). The largest drafts—Llama-3.1-8B (913K tokens/$, 𝛼 = 0.62) and Qwen3-8B (1224K tokens/$, 𝛼 = 0.52)—outperform their smallest counterparts by around 19.4% and 24.9%, respectively, driven entirely by higher acceptance rates. The Qwen configurations achieve uniformly higher cost efficiency due to a lower verification price ($0.59/M vs. $0.90/M for Llama). Combined with the goodput results in Section 4.1, this reveals a fundamental Pareto tradeoff: smaller draft models maximize goodput by maximizing 𝑣𝑑 , while larger draft models maximize cost efficiency by maximizing 𝛼. The optimal choice therefore depends on whether the deployment prioritizes latency or monetary cost, a point we revisit in Section 4.4. 4.3
Energy Efficiency
Energy comparison results. Fig. 5 shows that the edge device’s energy per verified token escalates steeply with draft model size on both platforms 1 . The smallest drafts—Llama-3.2-1B-Instruct Q4_K_M (0.63 J/tok on Jetson, 1.28 J/tok on RPi 5) and Qwen3-0.6B Q4_K_M (0.60 J/tok, 0.91 J/tok)—achieve the best energy efficiency across all configurations. Scaling to the largest drafts degrades efficiency by 4.17–5.14× in the Llama family (Llama-3.1-8B: 2.67 J/tok on Jetson, 6.58 J/tok on RPi 5) and up to 10.75× in the Qwen family (Qwen3-8B Q6_K: 4.25 J/tok on Jetson, 9.73 J/tok on RPi 5). The degradation is consistently less severe on the Jetson, whose higher compute throughput better amortizes the fixed power overhead. Speed–energy Pareto front. Fig. 6 visualizes the speed–energy tradeoff with iso-power curves at 15 W, 20 W, 40 W, and 60 W. In 1 Real-time power monitoring is much less accessible on the Raspberry Pi 4B than on
the Raspberry Pi 5, which provides more practical platform-level support for power measurement. Therefore we report power results only for the Raspberry Pi 5.
ConfigSpec : Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving
J / Verified Token
Ll
Jetson AGX Orin
4 2 0
. st In ) B - _M -1 K .2 Q4_ 3 a- (
am
1B 2- ) 3. _M a- K m 4_ a l (Q
L
3B 2- ) 3. _M a- K m 4_ a l (Q
L
3B 2- 0) 3. _ a- (Q8
m la
L
8B 1- ) 3. _M a- K m 4_ a l (Q
L
Energy Efficiency Comparison (Target: Qwen3-32B, K=5)
10
5
0
Table 2. Recommended configurations under different optimisation objectives. 𝐺: goodput (tok/s), 𝜂 cost : cost efficiency (K tokens/$), 𝐸: energy per accepted token (J/tok). Best values per (target, device) group are bolded. Target
Llama-3.1-70B
J / Verified Token
Energy Efficiency Comparison (Target: Llama-3.1-70B, K=5) RPi 5
6
4B 4B 8B 8B 6B 6B 7B 7B 0. ) 0. 0) 1. ) 1. 0) 3- ) 3- 0) 3- ) 3- K) 3- _M 3- 8_ 3- _M 3- 8_ en _M en 8_ en _M en 6_ w K w (Q w K w en K en (Q en K en (Q Q Q4_ Q Q Q4_ Q (Q w 4_ w w 4_ w Q (Q Q Q (Q Q ( (
J / Verified Token
7.5
RPi 5 Jetson AGX Orin
60 W
5.0
20 W
Llama-3.2-1B-Inst. (Q4_K_M) 1.28 J/tok
10 W 5W
Llama-3.2-1B-Inst. (Q4_K_M) 0.64 J/tok
J / Verified Token
10 60 W
5
40 W 20 W
Qwen3-0.6B Qwen3-0.6B (Q4_K_M) 0.91 J/tok (Q4_K_M) 0.60 J/tok
10 W 5W
2 4 6 8 Verified Goodput (tok/s) = (Kα(K) + 1) / (K/tps + Tverify )
Figure 6. Energy efficiency and speed (goodput) comparison. both model families, the smallest-draft model with lowest quantization bit-width on Jetson configurations occupy the Pareto-optimal corner (high goodput, low energy), while all RPi 5 configurations are Pareto-dominated by their Jetson counterparts. This architectural advantage stems from the Jetson’s GPU-accelerated inference, which exploits massive parallelism in matrix operations to deliver substantially higher throughput per watt, whereas the RPi 5’s CPUonly execution serializes these operations across a small number of cores, resulting in longer active inference time and proportionally greater energy expenditure per token. 4.4
Objective
Configuration
𝐺
𝜂 cost
𝐸
RPi 4B
Max Goodput Min Cost/tok Min Energy
Llama-3.2-1B-Inst Q4 2 2.44 Llama-3.1-8B-Inst Q4 2 0.77 no power data
1334K 1401K
— —
RPi 5
Max Goodput Min Cost/tok Min Energy
Llama-3.2-1B-Inst Q4 Llama-3.1-8B-Inst Q4 Llama-3.2-1B-Inst Q4
6 2 2
4.50 1.55 3.76
763K 1401K 1334K
0.84 3.75 0.48
Jetson
Max Goodput Min Cost/tok Min Energy
Llama-3.2-1B-Inst Q4 Llama-3.1-8B-Inst Q4 Llama-3.2-1B-Inst Q4
8 2 2
7.65 4.35 4.60
623K 1401K 1334K
0.85 1.74 0.39
RPi 4B
Max Goodput Min Cost/tok Min Energy
Qwen3-0.6B Q4 Qwen3-8B Q4
2 2.81 2 0.74 no power data
1801K 2048K
— —
RPi 5
Max Goodput Min Cost/tok Min Energy
Qwen3-0.6B Q4 Qwen3-8B Q4 Qwen3-0.6B Q4
7 2 2
3.86 1.49 3.48
828K 2048K 1801K
0.90 3.86 0.41
Jetson
Max Goodput Min Cost/tok Min Energy
Qwen3-0.6B Q4 Qwen3-8B Q4 Qwen3-0.6B Q4
10 2 2
6.21 4.14 4.08
633K 2048K 1801K
0.93 1.88 0.33
𝐾
2 4 6 8 10 Verified Goodput (tok/s) = (Kα(K) + 1) / (K/tps + Tverify )
Speed–Energy Tradeoff (Target: Qwen3-32B, K=5)
0
Device
40 W
2.5 0.0
Qwen3-32B
Figure 5. Energy efficiency of draft models. Speed–Energy Tradeoff (Target: Llama-3.1-70B, K=5)
TDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Optimal Configuration Selection across the Three-Dimensional Space
Sections 4.1–4.3 examined goodput, cost, and energy in isolation while fixing the speculative length at 𝐾 = 5. We now treat all three configuration knobs—draft-model variant 𝑀, quantisation level 𝑄, and speculative length 𝐾—as a joint search space and ask: for a given LLM family and edge device, which triple (𝑀, 𝑄, 𝐾) optimises each objective. 2 Methodology. For every (target, device) pair, we enumerate all feasible (𝑀, 𝑄, 𝐾) triples with 𝐾 ∈ {2, . . . , 10} and evaluate three metrics: (i) goodput 𝐺 (Eq. 1), (ii) cost efficiency 𝜂 cost (Eq. 2), and 2 Considering the power supply on the server is continuous, we only consider the
power consumption and optimization on edge device side
(iii) energy per accepted token 𝐸 (Eq. 3). Table 2 reports the configuration that optimises each metric, together with all three metric values so that the trade-offs are directly visible. As the 𝐾 and acceptance rate (𝛼) have an intricate relation, for each 𝑀, 𝑄, 𝐾, edge device, and target model combination, we computed tailored 𝛼 (𝐾). Observation 1: Goodput favours small, fast drafters with device-dependent 𝐾 ∗ . Across both families, the smallest Q4 quantised draft model—Llama-3.2-1B-Instruct for Llama-70B and Qwen30.6B for Qwen3-32B—delivers the highest goodput on every device. The optimal speculative length 𝐾 ∗ , however, scales with device speed: on the RPi 4B the fixed verification overhead 𝑇verify already dominates at 𝐾 = 2; on the RPi 5 the sweet spot rises to 𝐾 ∗ = 6– 7; and on the Jetson it climbs to 𝐾 ∗ = 8–10. Comparing the Jetson’s goodput-optimal 𝐺 = 7.65 tok/s (Llama) with the RPi 4B’s 𝐺 = 2.44 tok/s reveals only a 3.1× advantage despite a ∼22× raw speed gap, confirming that 𝑇verify compresses the goodput range. The cost of this goodput-maximising strategy is visible in the 𝜂 cost column: on the Jetson it drops to 623K tok/$—less than half the cost-optimal 1401K tok/$. Observation 2: Cost efficiency is device-independent and always peaks at 𝐾 = 2 with the largest drafter. Since 𝜂 cost = (𝛼 (𝐾) + 1/𝐾)/𝑝 depends only on the acceptance rate and verifier price, the cost-optimal configuration is identical across all three devices: 8B-Instruct Q4 at 𝐾 = 2 for Llama-70B and Qwen3-8B Q4 at 𝐾 = 2 for Qwen3-32B. The key is the bonus-token effect: every verification round produces one auto-regressive output token regardless of how many drafted tokens are rejected. At 𝐾 = 2 this free token contributes 1/𝐾 = 0.5 to the per-verified-token yield, whereas at 𝐾 = 10 it contributes only 0.1. Combined with the high acceptance rate of 8B drafters at short sequences (𝛼 (2) ≈ 0.76), 𝐾 = 2 yields 𝛼 (2) + 1/2 = 1.26 accepted tokens per verified token—the maximum across the entire search space. The trade-off is stark: on the RPi 5 the cost-optimal configuration delivers only 𝐺 = 1.55 tok/s
TDIS ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, and Dimitrios S. Nikolopoulos
(2.9× slower than the goodput-optimal 4.50 tok/s) and 𝐸 = 3.75 J/tok (7.8× worse energy than the energy-optimal 0.48 J/tok). Observation 3: Energy efficiency mirrors goodput’s model choice but universally locks to 𝐾 = 2. The energy-optimal draft model is the same small, fast model that maximises goodput—1BInstruct Q4 for Llama, 0.6B Q4 for Qwen—because these draw the least power and minimise drafting time. Yet the optimal speculative length is universally 𝐾 = 2, diverging from goodput’s devicedependent 𝐾 ∗ . Since 𝐸 only counts local drafting energy, setting 𝐾 = 2 minimises the joules spent per round while the bonustoken effect keeps yield high. On the Jetson the energy-optimal 𝐸 = 0.39 J/tok (Llama) is 17% lower than the RPi 5’s 0.48 J/tok despite higher idle power, because the Jetson completes each 2-token draft in a fraction of the time. Notably, the energy-optimal configuration still achieves reasonable goodput—𝐺 = 4.60 tok/s on the Jetson (60% of the goodput-optimal 7.65)—making it an attractive default for battery-constrained deployments where moderate latency is acceptable. Key trade-offs. • Model size: Goodput and energy prefer the smallest drafter (maximum 𝑣𝑑 , minimum 𝑃), while cost prefers the largest (maximum 𝛼). On the RPi 5, switching from the cost-optimal 8B to the goodput-optimal 1B improves throughput by 2.9× and energy by 7.8×, but sacrifices 46% cost efficiency. • Speculative length 𝐾: Goodput benefits from longer speculation on fast devices (more tokens amortise 𝑇verify ), while both cost and energy favour 𝐾 = 2 (the bonus-token effect dominates). Reducing 𝐾 from the goodput-optimal 𝐾 ∗ to 2 costs up to 40% goodput but can halve energy per token. These results demonstrate that profiling-based configuration selection across the joint (𝑀, 𝑄, 𝐾) space is essential: no single fixed setting can simultaneously optimise throughput, cost, and energy.
5
Conclusion
ConfigSpec is a framework for configuration selection in distributed speculative LLM serving across cloud and edge. By modeling heterogeneous edge devices and speculative decoding dynamics, ConfigSpec enables rapid exploration of trade-offs among edge model variants, quantisation levels, and speculative lengths without repeated system deployment. Our analysis of the joint (𝑀, 𝑄, 𝐾) configuration space reveals that the three optimisation objectives impose fundamentally conflicting pressures: goodput favours small, fast drafters at device-dependent 𝐾 ∗ that amortises verification latency, while both cost and energy efficiency converge to 𝐾=2 due to the dominant bonus-token effect—though cost selects the largest drafter to maximise acceptance rate whereas energy selects the smallest to minimise power draw. These structural conflicts confirm that no single fixed configuration can simultaneously optimise throughput, cost, and energy, underscoring the necessity of profiling-based selection. The framework complements deployment-focused speculative decoding systems by supporting principled design-space exploration in disaggregated AI inference infrastructures.
6
Acknowledgements
This work was supported by a research grant from the Department for the Economy, Northern Ireland (grant agreement USI-226),
Virginia Tech College of Engineering (grant Major Grants Initiative Program), and by the National Science Foundation (grant No. 2315851).
References [1] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 (2023). [2] Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. 2025. Efficientqat: Efficient quantization-aware training for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10081–10100. [3] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World’s First Truly Open InstructionTuned LLM. https://www.databricks.com/blog/2023/04/12/dolly-first-opencommercially-viable-instruction-tuned-llm [4] Fireworks AI. 2025. Pricing – Fireworks AI. https://fireworks.ai/pricing. Serverless tier: $0.90 / 1M tokens for models >16B parameters. Accessed: 2025-07-15. [5] Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning. PMLR, 10323–10337. [6] Groq. 2025. GroqCloud On-Demand Pricing. https://groq.com/pricing. Qwen332B: $0.29 / 1M input tokens, $0.59 / 1M output tokens. Accessed: 2025-07-15. [7] Chengzhuo Han, Tingting Yang, Zhengqi Cui, and Xin Sun. 2025. A privacypreserving and trustworthy inference framework for LLM-IoT integration via hierarchical federated collaborative computing. IEEE Internet of Things Journal (2025). [8] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. PMLR, Honolulu, Hawaii, USA, 19274–19286. [9] Hui Li, Xiuhua Li, Qilin Fan, Qiang He, Xiaofei Wang, and Victor CM Leung. 2025. Adaptive model partitioning and pruning for collaborative DNN inference in mobile edge-cloud computing networks. IEEE Transactions on Mobile Computing (2025). [10] Xiangchen Li, Jiakun Fan, Qingyuan Wang, Dimitrios Spatharakis, Saeid Ghafouri, Hans Vandierendonck, Deepu John, Bo Ji, Ali R Butt, and Dimitrios S Nikolopoulos. 2026. WISP: Waste-and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching. arXiv preprint arXiv:2601.11652 (2026). [11] Xiangchen Li, Dimitrios Spatharakis, Saeid Ghafouri, Jiakun Fan, Hans Vandierendonck, Deepu John, Bo Ji, and Dimitrios S Nikolopoulos. 2025. Sled: A speculative llm decoding framework for efficient edge serving. In Proceedings of the Tenth ACM/IEEE Symposium on Edge Computing. 1–8. [12] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2023. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781 1, 2 (2023), 4. [13] Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. 2024. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding. In International Conference on Learning Representations. [14] Chunlin Tian, Xinpeng Qin, Kahou Tam, Li Li, Zijian Wang, Yuanzhe Zhao, Minglei Zhang, and Chengzhong Xu. 2025. CLONE: customizing LLMs for efficient latency-aware inference at the edge. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 34, 23 pages. [15] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning. PMLR, 38087–38099. [16] Daliang Xu, Wangsong Yin, Hao Zhang, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, and Xuanzhe Liu. 2024. Edgellm: Fast on-device llm inference with speculative decoding. IEEE Transactions on Mobile Computing 24, 4 (2024), 3256– 3273. [17] Zhongzhi Yu, Zheng Wang, Yuhan Li, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reddy Bommu, Yang Zhao, and Yingyan Lin. 2024. Edge-llm: Enabling efficient large language model adaptation on edge devices via unified compression and adaptive layer voting. In Proceedings of the 61st ACM/IEEE Design Automation Conference. 1–6. [18] Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. 2024. EdgeShard: Efficient LLM inference via collaborative edge computing. IEEE Internet of Things Journal (2024).