arXiv CSopen access

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

· arxiv_cs

arXiv CS · Papers · License: Open Access

operating systems, kernel, virtualization

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling Weisi Yang

Stephen Xia

Northwestern University Evanston, Illinois, USA [email protected]

arXiv:2609.09662v1 [cs.LG] 9 Sep 2026

Abstract Deploying Large Language Models (LLM) directly on mobile platforms at the edge is gaining a large amount of traction due to a myriad of benefits, such as increased privacy, personalization, and latency. However, LLMs have heavy compute requirements, which are difficult for resource constrained mobile and edge platforms to fulfill. In addition to limited compute resources, mobile and edge systems often have a compact form factor and lack physical mechanisms to dissipate heat generated from high processor usage rates (e.g., fans) to prevent throttling and reduced processing power, which LLMs can easily cause. To mitigate these effects, prior works have proposed various power governing strategies, such as dynamic voltage and frequency scaling (DVFS), for reducing power and heat generation for heavy computational tasks on mobile platforms. Recently, DVFS tailored for mobile LLMs have also been proposed. However, these methods mostly focus on optimizing hardware parameters and processor frequencies, which fall short under some thermally constrained scenarios. Drawing from recent advances in machine learning, we form and take advantage of the key insight that not all tokens require full-depth inference to maintain high quality generation. Motivated by this, we present PELM, a solution that augments traditional DVFS processor frequency tuning with two additional workload-specific knobs: 1) speculative decoding and 2) variable verification depth to expand the optimization space to multiple dimensions for more power efficient on-device LLM inference. In extensive evaluations across different hardware platforms and datasets,reduction in energy consumption, while maintaining comparable task performance. The source code is available at: https://github.com/imec-nu/PELM.

CCS Concepts • Computer systems organization → Embedded systems; • Software and its engineering → Power management; • Computing methodologies → Natural language generation.

Keywords Edge computing; Mobile systems; On-device large language models; Dynamic voltage and frequency scaling; Speculative decoding; Reinforcement learning.

This work is licensed under a Creative Commons Attribution 4.0 International License. SenSys ’26, Saint Malo, France © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2309-4/2026/05 https://doi.org/10.1145/3774906.3802783

ACM Reference Format: Weisi Yang and Stephen Xia. 2026. PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling. In ACM/IEEE International Conference on Embedded Artificial Intelligence and Sensing Systems (SenSys ’26), May 11–14, 2026, Saint Malo, France. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3774906.3802783

Introduction

As mobile devices grow increasingly capable, they are positioned to handle more demanding tasks. The recent development of Large Language Models (LLMs) has attracted significant attention for ondevice deployment, driven by the need for ubiquitous computing, data privacy, and offline accessibility [38]. Despite increasingly powerful mobile devices, running LLMs locally remains challenging due to constrained hardware resources, resulting in low decoding throughput. Furthermore, the small form factor of these devices limits heat dissipation, making them highly vulnerable to thermal throttling and reduced performance when running high workloads, such as LLM inference, for long periods of time. Fortunately, there are promising power governing techniques, such as dynamic voltage and frequency scaling (DVFS), which adjust processor voltage and frequency to maintain timely execution of workloads, while preventing excessive power consumption, heat generation, and throttling [14, 24]. Tailored DVFS solutions have been designed for conventional deep learning (DL) tasks [10, 44] and more recently for LLMs, optimizing and selecting processor frequencies for different generation patterns [43]. However, these hardware-only tuning solutions face a fundamental dilemma: lowering frequency to save power directly throttles throughput and reduces quality of experience (QoE), while raising it exacerbates thermal issues. This makes further optimization of the frequency challenging, especially for LLM workloads where performance and power are highly sensitive to change. To help rectify this problem, our key insight is that running the full model for every token generation is not necessary for maintaining high task performance. For example, speculative decoding leverages this insight for accelerating end-to-end LLM inference without task performance degradation [42] by using a small, fast “draft” model to generate multiple tokens and then verifying them in parallel with a single pass of the larger and slower “verify” model. This allows a single pass of the large model to generate multiple tokens, instead of needing to run separate passes for each token. Self-speculative decoding (SSD) further refines this idea by using a small portion of the model’s own intermediate layers for drafting, eliminating the need for a separate model [7, 35]. However, speculative decoding primarily focuses on improving generation speed, and to the best of our knowledge, has not been explored for power governing on resource-constrained devices.

215 215 215 222 233 249 265 281

165 167 166 171 179 190 202 216 149 150 149 154 161 170 182 193 132 134 133 138 144 152 160 173

198 200 197 206 215 230 244 259 181 183 182 188 196 210 224 238

0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 GPU Freq (GHz)

350 300 Energy (Joules)

13.83 15.90 17.92 19.56 19.80 19.99 20.18 20.26

Layer Used 20 22 24

12.31 14.17 15.97 17.44 17.69 17.86 18.02 18.07

265 266 264 276 289 307 323 346

11.10 12.77 14.37 15.74 15.95 16.13 16.24 16.34

283 282 278 293 305 326 345 369

10.14 11.66 13.17 14.35 14.55 14.70 14.81 14.90

9.28 10.70 12.11 13.23 13.36 13.51 13.61 13.70

Weisi Yang and Stephen Xia

Layer Used 20 22 24

8.59 9.91 11.18 12.20 12.35 12.49 12.59 12.68

Speed (tokens/sec)

32 30

7.00 8.05 9.08 9.95 10.09 10.21 10.32 10.39

6.59 7.59 8.57 9.39 9.52 9.62 9.69 9.77

SenSys ’26, May 11–14, 2026, Saint Malo, France

250 200 150

0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 GPU Freq (GHz)

(a) speed

(b) energy

Figure 1: The speed and energy consumption of large language models (LLMs) vary with GPU frequency and the number of layers executed. Tested with LLaMA-3.1-8B on Jetson AGX Orin.

HumanEval

GSM8K

0.6

Task Score

0.5

0.4 0.3 0.2

0.1 0 23

Exit Layer

Figure 2: Intermediate layer outputs still convey meaningful information that is comparable to the final output of the model. Blue: HumanEval (pass@1). Orange: GSM8K (accuracy). Evaluated on LLaMA-3.1-8B using LayerSkip [7].

To see the impacts of varying the number of layers executed and GPU frequency on energy and generation speed, we conducted a pilot study, as shown in Figure 1, and see that multiple configurations achieve the same QoE (token generation speed). For example, running a 32-layer model at 1.3 GHz yields a similar speed to a 24layer model at 0.7 GHz, yet the latter consumes 41.7% less energy. This “variable” depth or “early-exit” execution paradigm has been shown capable of producing similar task performance with much less memory and latency overhead in recent studies [7, 32]. Our own task-specific evaluations in Figure 2 confirm that running a subset of intermediate layers still retains meaningful and usable information for many tokens. Our key insight exposes a new and larger optimization space by augmenting the traditional control knobs tuned by 1) DVFS (hardware frequency) with two additional workload-based knobs: 2) Self-Speculative Decoding (algorithmic throughput) and 3) Variable Depth (computational workload). However, this creates a new and highly complex challenge: a combinatorial control problem. The optimal joint decision for frequency, speculation length, and layer depth is not static; it must ideally adapt per-token based on transient thermal conditions and token-specific difficulty [25]. Simple heuristics cannot navigate this dynamic joint state space. We present PELM, a Power-Efficient LLM inference framework for Mobile systems, a power governing solution that jointly optimizes these three components to reduce energy consumption. In

addition to tuning hardware frequency parameters, like in traditional DVFS, PELM leverages self-speculative decoding to selectively run a subset of the model’s intermediate layers as the draft model to speed up token generation. Additionally, PELM also introduces an optional third knob (variable depth) that builds on self-speculation, where the number of layers used in the verification model can also be tuned to further reduce energy consumption and latency. First, we formulate this novel joint optimization problem that significantly enlarges the optimization space compared to traditional DVFS methods. Second, to help navigate this large optimization space, we create a Deep Reinforcement Learning (DRL) power governor to tune these knobs and optimize for power and latency at runtime, with comparable task quality as running the full model normally. We conduct comprehensive evaluations with LLMs of different sizes on two Jetson devices under various thermal conditions. These evaluations verify PELM’s robustness to different levels of thermal and computational workloads and its adaptability to different QoE requirements, showcasing its feasibility for real-world deployment. The main contributions are as follows: (1) We formulate a novel optimization problem for power governing on mobile and edge devices that expands the traditional pathways for limiting heat generation and energy consumption (e.g., tuning processor frequencies in DVFS) during LLM inference, with self-speculative decoding for tuning model throughput and variable depth execution for tuning the computational workload. (2) We propose PELM, a Deep Reinforcement Learning-based power governing solution that jointly optimizes this formulation at runtime. By incorporating additional algorithmic and workload-based pathways for reducing energy consumption, while maintaining task performance, PELM has more flexibility in how it chooses to limit energy consumption and heat generation than traditional DVFS solutions. (3) We conduct extensive evaluations on two popular embedded platforms, 5 diverse datasets, demonstrating up to 23.1% speedup and 52.4% reduced energy consumption, while maintaining task performance compared to existing state-of-theart power governors. Our experiments illustrate PELM’s effectiveness and robustness across diverse workloads and thermal conditions.

2 Background, Related Works, and Motivation 2.1 Dynamic Voltage Frequency Scaling for Edge Devices Dynamic Voltage Frequency Scaling (DVFS) is a critical power management technique that adjusts a processor’s voltage (𝑉 ) and frequency (𝑓 ) to match the current workload. DVFS leverages the cubic relationship between power and frequency—arising from 𝑃 ∝ 𝑉 2 𝑓 and the relation 𝑉 ∝ 𝑓 , which yields 𝑃 ∝ 𝑓 3 [34]—so that small frequency reductions produce significant power savings, making it particularly effective for power-constrained edge devices. Conventional DVFS implementations (e.g., Linux governors such as schedutil and ondemand) are generic and typically react to coarsegrained metrics like average CPU/GPU utilization, whereas more recent work proposes application-aware DVFS policies that incorporate task-level Quality of Experience (QoE) to jointly optimize

PELM : On-Device LLM Inference with Speculative Decoding and DVFS

hardware and application objectives—for example, maintaining a target FPS in video rendering while minimizing power consumption [10, 14, 41, 44, 45]. With the rise of on-device LLMs, system-level evaluations have expanded in scope [16, 17, 20, 34], and DVFS schemes tailored to LLM workloads have emerged for both datacenters [12, 13, 29, 33] and mobile devices [26, 27, 31, 40, 43]. Inefficiency of Existing DVFS for LLM Workloads. However, both conventional and recent QoE-aware DVFS designs are insufficient for LLM generation due to a fundamental mismatch between their control granularity and the workload characteristics (see Section 5.1). Existing DVFS approaches primarily regulate processor frequency, assuming performance scales smoothly with frequency adjustments. In contrast, LLM generation exhibits highly irregular and token-dependent computational patterns, where latency is jointly determined by both hardware frequency and algorithmic factors such as speculative depth and exit strategies. This limited control space of frequency-only adaptation prevents DVFS from fully exploring the performance-power trade-off under stringent QoE constraints. As a result, such methods often make suboptimal decisions in practice: conservative settings may fail to meet latency targets, while aggressive frequency boosting can induce thermal throttling, ultimately degrading sustained performance and increasing energy consumption. These limitations motivate the need for a co-designed optimization strategy tailored to LLM workloads.

2.2

Speculative Decoding

Autoregressive LLM inference is primarily bottlenecked by memory bandwidth, as each token is generated sequentially, leading to underutilization of compute units. Speculative Decoding (SD) [19] addresses this by using a small "draft model" to generate a 𝑘-token draft, which is then verified in parallel by the large "target model" in a single, compute-bound pass. This replaces 𝑘 memory-bound steps with a single step of the target model, significantly accelerating generation. The conventional two-model speculative decoding paradigm is impractical on memory-constrained edge devices due to its prohibitive memory overhead. To address this, Self-Speculative Decoding eliminates the auxiliary draft model by reusing a truncated version (e.g., the first 𝑁 layers) of the target model as a lightweight draft generator [42]. Classic draft–then–verify amortizes a full forward pass over multiple tokens via rejection sampling, achieving 2–4× latency reductions while preserving the original model distribution [19, 42]. However, approaches relying on independent draft backbones incur tokenizer coupling and VRAM overhead, motivating integrated designs such as Medusa (multi-head future token prediction) [3], EAGLE-1/2/3 (small draft model with multi-layer hidden states fusion, dynamic tree, and training-time test) [21–23], and layer-skipping methods like SWIFT, LayerSkip, and AdaDecode [7, 35, 37], as well as auxiliary-free lookahead schemes [9]. Building on the LayerSkip-style self-speculative paradigm, we generalize the design by allowing both draft exit depth and verification depth to vary dynamically at each decoding step, while co-optimizing speculative length under a DVFS-aware governor. This joint control sustains QoE while respecting power and thermal constraints on mobile hardware.

SenSys ’26, May 11–14, 2026, Saint Malo, France

2.3

Observations

Here we leverage the key insight: not all tokens require full depth inference, and those processed by shallower depth still maintain useful information while excelling in terms of latency and power consumption. Such insight primarily draws from recent advances in the machine learning field about the study of intermediate layers of LLMs [30, 32], indicating that the output of final layers is not necessarily better than that of the earlier layers. To validate this, we profiled the LLaMA-8B on an AGX Orin across a 2D space of GPU frequency and executed layer depth (Figure 1). We first observe a crucial many-to-one mapping: numerous (Layer, Frequency) combinations achieve the same QoE (token speed), yet have vastly different energy costs. For example, (20 Layers, 0.6 GHz) matches the 10 tokens/s speed of (32 Layers, 1.3 GHz) but consumes 50.9% less energy. This reveals a massive, untapped optimization space. This optimization opportunity is only viable if executing fewer layers does not catastrophically degrade task performance. And Figure 2 confirms that task performance is surprisingly substantial between intermediate (e.g., layer 24) and final layers, retaining meaningful information. While GSM8K exhibits larger fluctuations across layers due to its sensitivity to reasoning consistency, overall performance remains within a comparable range to the final layer. Together, these findings form our core motivation: shallower execution is feasible (quality is preserved) and essential (it saves energy for the same QoE). Existing DVFS methods are blind to this, as they only control the "frequency" knob. This necessitates a new controller that can co-manage both layer depth and frequency to find the better, energy-optimized state for any QoE target.

2.4

Positioning of PELM

PELM bridges two previously separate lines of work: speculative decoding in the ML community and DVFS-based power management in systems research. On one hand, recent SD methods [7, 22, 37] focus on reducing latency by adapting model execution depth or by using lightweight draft generators. For example, the EAGLE series improves speculative decoding by attaching lightweight draft heads to the target model’s hidden states and predicting future tokens, with later versions introducing dynamic drafting and multi-layer feature fusion to improve acceptance rates. These approaches primarily target throughput improvement under fixed hardware configurations rather than energy-constrained QoE control. On the other hand, conventional DVFS schemes adjust hardware frequency based on coarse utilization signals, without visibility into model-level execution depth or decoding dynamics. As a result, they treat the LLM workload as a fixed black box and lack fine-grained control over computational intensity. PELM operates at the intersection of these two spaces. It jointly manages hardware knobs (DVFS frequency) and software decoding parameters (self-speculation depth, verification depth, and speculative horizon) under explicit thermal and power constraints. By coordinating workload shaping with hardware adaptation in a closed loop, PELM opens a new cross-layer optimization space that is inaccessible to decoding-only or hardware-only approaches.

SenSys ’26, May 11–14, 2026, Saint Malo, France

Weisi Yang and Stephen Xia

Method

In this section, we present the design of PELM, with its workflow illustrated in Figure 3. The central component of PELM is the DVFS Governor, which continuously monitors system metrics, including both hardware parameters and software execution metrics, and provides instructions for subsequent LLM generation and hardware configuration. We next formulate the optimization problem that the DVFS Governor is designed to address.

3.1

Problem Formulation

The primary consideration for on-device LLM inference is Quality of Experience (QoE) of the user, which we define as the token decoding speed. In addition to QoE, the process should also address efficiency requirements. Our design principle is to provide a powerefficient service that meets the QoE target while minimizing side effects related to power consumption and overheating. Traditional DVFS solves the following problem to optimize these objectives. Standard DVFS formulation. Let 𝑞 be an input query to an LLM, 𝑄 be a set of queries, 𝐽 be energy consumed, 𝑡 be the time, 𝑇𝑡𝑒𝑚𝑝 (𝑡) be the temperature at time 𝑡, 𝑆 be the decoding speed, the optimization problem could be: min 𝐸𝑞∼𝑄 [𝐽 (𝑞)] s.t. 𝑇temp (𝑡) < 𝑇limit

(1)

𝑆 (𝑞) ≥ 𝑆 target The optimization problem (1) aims at minimizing the expected energy consumption throughout the LLM inference process while meeting thermal and QoE constraints. Since heating is fundamentally related to power dissipation, which is the rate of energy consumption, we could rewrite the objective function as min 𝐸𝑞∼𝑄 [𝑝 (𝑞)× 𝑡 (𝑞)], where 𝑝 (𝑞) is the power consumption of a query 𝑞, and 𝑡 (𝑞) indicates the time used for processing 𝑞. Power and Latency Dilemma. Minimizing either 𝑝 (𝑞) or 𝑡 (𝑞) is viable for optimizing the formulation in Equation 1. However, these two directions indicate a contradiction that makes this optimization challenging to solve. For minimizing 𝑝 (𝑞), given that a processor’s power 𝑝 is proportional to its executing frequency 𝑓 , and more precisely 𝑝 ∝ 𝑉 2 × 𝑓 , where 𝑉 is the working voltage. Since 𝑉 ∝ 𝑓 [34], we have 𝑝 ∝ 𝑓 3 . This suggests that 𝑝 could be minimized by reducing operating frequency 𝑓 . Conversely, to minimize 𝑡 (𝑞), we maximize generation speed 𝑆 (𝑞) to reduce the time needed, which requires raising the frequency 𝑓 , contradicting the former optimization. This inherent 𝑝 (𝑞) and 𝑡 (𝑞) trade-off must be systematically addressed. We incorporate self-speculative decoding and variable depth execution as two additional knobs for the power governor to minimize execution time, 𝑡 (𝑞), discussed next. Self-speculative decoding with variable depth. Unlike conventional speculative decoding, which requires distinctive draft and verify models, self-speculative decoding uses a sub-part of the full model for drafting, providing a memory-efficient way for speculative computing. Let 𝑒𝑥 be the depth of draft model, 𝐿 be the depth of the full model, 𝑛𝑠 be the number of speculations made. For one time speculation, we execute 𝑛𝑠 times of 𝑒𝑥 layers of an LLM to get 𝑛𝑠 draft tokens, and execute a full-depth run to verify these draft tokens, so the number of total layers run is 𝑛𝑠 × 𝑒𝑥 + 𝐿. Depending on the draft quality, the number of accepted tokens is 𝑛𝑎𝑐 ,

𝑛𝑎𝑐 ∈ {1, 2, ..., 𝑛𝑠 + 1}. Theoretically we could compute the speedup ratio 𝑅 as 𝑅 = 𝐿/((𝑛𝑠 × 𝑒𝑥 + 𝐿)/𝑛𝑎𝑐 ) = (𝑛𝑎𝑐 × 𝐿)/(𝑛𝑠 × 𝑒𝑥 + 𝐿). Then the token decoding speed could be expressed as 𝑆 = 𝑅 × 𝑆 𝐵𝑎𝑠𝑒 , where 𝑆 𝐵𝑎𝑠𝑒 is the speed of running full model depths. From our initial experiments (Figure 2), showing that earlyexiting at an LLM’s intermediate layer often attains comparable output quality as running up to the final layer, we further introduce self-speculative decoding with variable depth. Let 𝑣𝑑 indicate the variable depth and 𝑛𝑠 < 𝑣𝑑 ≤ 𝐿 in a single speculation. Only 𝑣𝑑 layers of original model act as the verify model instead of full depth, so the number of total layers run becomes 𝑛𝑠 × 𝑒𝑥 + 𝑣𝑑. Now, the speedup ratio is calculated as 𝑅 = (𝑛𝑎𝑐 × 𝑣𝑑)/(𝑛𝑠 × 𝑒𝑥 + 𝑣𝑑). PELM’s optimization formulation. We can split the objective function min 𝐸𝑞∼𝑄 [𝑝 (𝑞) ×𝑡 (𝑞)] without introducing contradictions between maximizing and minimizing 𝑓 . We minimize 𝑓 specifically for min 𝐸𝑞∼𝑄 [𝑝 (𝑞)] and maximize the speedup ratio from speculative decoding 𝑅 for min 𝐸𝑞∼𝑄 [𝑡 (𝑞)]. Moreover, since 𝑣𝑑 is involved, meaning that the verification model could run partially and produce responses that diverge in target distribution from the full model, we add a new QoE objective – output fidelity 𝜙, which describes the degree to which the generated content matches the output of the original full model. 𝑣𝑑, or the depth of the verification model, is a natural proxy for this new metric. This metric provides a new objective for maximizing fidelity 𝜙, which encourages LLM to run full depth. We also model power 𝑝’s relationship to 𝑓 as 𝑝 (𝑓 ). We scalarize the multi-objective formulation via a weighted summation, where 𝜆𝑝 , 𝜆𝜙 , and 𝜆𝑅 control the trade-offs among power, fidelity, and speedup. Additional practical considerations such as thermal limits and utilization–frequency matching are incorporated in the reward design, as described in the next subsection. Given new and replaced objectives, the original problem with variable depth self-speculation can be rewritten as: h i min 𝐸𝑞∼𝑄 𝜆𝑝 𝑝 (𝑓 , 𝑞) + 𝜆𝜙 (1 − 𝜙 (𝑣𝑑 |𝑞)) + 𝜆𝑅 − 𝑅(𝑒𝑥, 𝑛𝑠 |𝑣𝑑, 𝑞) s.t. 𝑇temp (𝑡) < 𝑇limit 𝑆 (𝑞) ≥ 𝑆 target (2) As such, the original problem (1) is reshaped into a concrete and practical problem (2) with multiple objectives that incorporates both hardware and algorithmic (self-speculative decoding) configurations. Taking inspiration from recent DVFS methods, problem (2) could further be solved in a reinforcement learning manner [14, 24, 45], we next further formalize our problem within the Q-learning context, a widely used RL algorithm. Convert to a Q-learning problem. The problem (2) objective could be replaced by max 𝐸𝜋 [𝑄 𝜋 (𝑠, 𝑎)] 𝜋

(3)

where, 𝑠 is the state of the environment, and 𝑎 is the action taken by the agent. 𝑄 𝜋 (𝑠, 𝑎) is a value function of state 𝑠 and action 𝑎 (details in Section 3.2) given a policy 𝜋. The Q-function is iteratively updated via the Bellman Equation: 𝑄 (𝑠𝑡 , 𝑎𝑡 ) ← 𝑄 (𝑠𝑡 , 𝑎𝑡 ) + 𝛼 (𝑟𝑡 +1 + 𝛾 max𝑎′ 𝑄 (𝑠𝑡 +1, 𝑎 ′ ) − 𝑄 (𝑠𝑡 , 𝑎𝑡 )) to find an optimized policy or Q-function, where 𝑟 is the reward observed given current state 𝑠𝑡 and action 𝑎𝑡 taken, 𝛾 is a discounting factor, and 𝛼 is the

PELM : On-Device LLM Inference with Speculative Decoding and DVFS

SenSys ’26, May 11–14, 2026, Saint Malo, France

LLM Control

LLM Generation Verify Layer Draft Layer

Spec Length

DVFS

System Signals Processors GPU

CPU0

Verify Layer

Acceptance Rate

CPU1

Thermal Zones

Governor

Power

Spec Length

Draft Layer

System Control GPU Freq

CPU0 Freq

CPU1 Freq

QoE

Figure 3: PELM workflow. At each control step, the governor monitors system and decoding signals (utilization, temperature, power, token/s, and acceptance statistics) and selects joint actions: DVFS settings and decoding knobs (exit layer, speculation length, and verification depth). The next window executes with early-exit drafting and variable-depth verification, supported by HQCache for efficient layer skipping. This closed loop co-optimizes hardware frequency and decoding depth to maintain QoE while reducing energy and avoiding thermal throttling, adapting dynamically across workloads. Table 1: RL components and symbols for PELM State 𝑠 Action 𝑎 Reward Function 𝑟

𝑝𝑐 , 𝑝𝑔 ,𝑢𝑐 , 𝑢𝑔 , 𝑇𝑡𝑒𝑚𝑝 ,𝛿𝑡𝑒𝑚𝑝 ,𝑆, 𝑣𝑑, 𝑅(𝑒𝑥, 𝑛𝑠 |𝑣𝑑), 𝑓𝑐 , 𝑓𝑔 𝑓𝑐 , 𝑓𝑔 , 𝑣𝑑, 𝑒𝑥, 𝑛𝑠 𝑟 𝑆 + 𝑟 𝑣𝑑 + 𝑟𝑡ℎ𝑒𝑟𝑚𝑎𝑙 + 𝑟𝑢 + 𝑟 𝑝

learning rate. As a model-free RL framework, Q-learning is free from establishing state transition models. For problems with small state and actions spaces, the Q-function can be stored in a lookup table (e.g., Q-Table). However, in cases where the state and action spaces are continuous or large, using a Q-Table to store Q-values used for evaluating the Q-function is prohibitively expensive. As such, we approximate the Q-function with a neural network or a Deep Q-Network (DQN). We detail the implementation of our DQN in Section 3.4. Here, the Q-function is defined over a reward signal that instantiates the scalarized objective in Eq. (2). Specifically, the per-step reward is constructed as the negative weighted objective, such that maximizing the expected Q-value approximates minimizing Eq. (2). Therefore, the RL formulation is a direct optimization surrogate of the original control objective. Next, we discuss the state, action, and reward of our DRL problem.

3.2

speculation speedup ratio 𝑅(𝑒𝑥, 𝑛𝑠 |𝑣𝑑) as computed in Section 3.1, and current CPU & GPU frequency – 𝑓𝑐 and 𝑓𝑔 .

State, Action, and Reward Design

The states, actions and rewards used to solve problem (3) are listed in Table 1, which we detail in the following section. 3.2.1 State. The state space of PELM consists of power consumption of CPU and GPU – 𝑝𝑐 and 𝑝𝑔 , processor utilization for CPU and GPU – 𝑢𝑐 and 𝑢𝑔 , current temperature and its rate of change – 𝑇𝑡𝑒𝑚𝑝 and 𝛿𝑡𝑒𝑚𝑝 , the self speculation parameters – variable depth 𝑣𝑑, the

3.2.2 Action. On the hardware side, available actions are CPU & GPU operating frequency 𝑓𝑐 and 𝑓𝑔 . On the speculation side, we have exit layer 𝑒𝑥, number of speculations 𝑛𝑠, and variable depth 𝑣𝑑. PELM takes 𝜖-greedy strategy to decide between exploration and exploitation. It leverages actions generated by DQN with probability 𝜖, and does exploration with probability 1 − 𝜖. Next we describe the behavior policy when PELM is in exploration phase. Behavior policy. Instead of adopting random sampling from action space for exploration [14], we provide a simple yet effective knowledge-guided policy design, directly inspired from our optimization problem (Section 3.1). The behavior policy operates on few key metrics: 1) thermal headroom 𝑡ℎ , which is the gap between current temperature 𝑡𝑡𝑒𝑚𝑝 and predefined warning temperature 𝑡 𝑤𝑎𝑟𝑛 ; 2) speedup ratio: 𝑆/𝑆𝑡𝑎𝑟𝑔𝑒𝑡 ; 3) speculative speedup ratio 𝑅(𝑒𝑥, 𝑛𝑠 |𝑣𝑑). Using these metrics, the agent performs an informed action. When it observes a sufficient thermal headroom, it takes aggressive actions, such as increasing 𝑓𝑐 and 𝑓𝑔 to make decoding speed 𝑆 to match 𝑆𝑡𝑎𝑟𝑔𝑒𝑡 . When it observes overspeed (i.e. 𝑆 > 1.2 × 𝑆𝑡𝑎𝑟𝑔𝑒𝑡 ), it reduces the current frequency to mitigate excessive power consumption and overheating. In scenarios where thermal headroom is limited, the agent adjusts the frequencies more conservatively. Specifically, it takes a smaller step to raise frequency when underspeed is observed and a larger step to lower the frequency for overspeed, and also considers reducing variable depth 𝑣𝑑 for faster LLM inference, at the cost of output fidelity. This idea of gradually increasing frequency until an “error” (e.g., overspeed) occurs, and then backing off at an increase rate, is inspired by congestion control algorithms, such as additive increase multiplicative decrease (AIMD), that have shown stable, fair, and efficient resource allocation behavior [6].

SenSys ’26, May 11–14, 2026, Saint Malo, France

Weisi Yang and Stephen Xia

For self-speculation, a feedback loop is employed for exploration. By comparing current speculation speedup ratio 𝑅𝑡 (𝑒𝑥𝑡 , 𝑛𝑠𝑡 |𝑣𝑑𝑡 ) with the previous time step ratio 𝑅𝑡 −1 (𝑒𝑥𝑡 −1, 𝑛𝑠𝑡 −1 |𝑣𝑑𝑡 −1 ), the agent adjusts its next speculation strategy accordingly. For instance, if the current ratio is higher than previous one, it reduces 𝑒𝑥 for shorter draft stage time, and increases the number of speculations 𝑛𝑠 to generate more draft tokens in a single step. Conversely, if the ratio is lower, the agent performs the reverse actions. This knowledgeguided behavior policy enables efficient decision-making to achieve optimized QoE while satisfying efficiency and other constraints. 3.2.3 Reward. We describe each term of reward function in Table 1. To meet decoding speed QoE demand, we define speed ratio 𝑆𝑟 = 𝑆/𝑆𝑡𝑎𝑟𝑔𝑒𝑡 , then speed reward is defined as

𝑟𝑆 =

 𝛼𝑆 (𝑆𝑟 − 0.95)      (𝑆 − 0.95)(1 − 𝑓  𝑟

𝑟 −𝑔𝑝𝑢 )𝑅

 (1.45 − 𝑆𝑟 )(1 − 𝑓𝑟 −𝑔𝑝𝑢 )𝑅     (1.45 − 𝑆 )(𝑓 𝑟 𝑟 −𝑔𝑝𝑢 )𝑅 

if 𝑆𝑟 < 0.95 if 0.95 ≤ 𝑆𝑟 ≤ 1.2 if 1.2 < 𝑆𝑟 ≤ 1.45 if 𝑆𝑟 > 1.45

where 𝑓𝑟 −𝑔𝑝𝑢 is the current GPU frequency ratio, 𝑅 is speculation speedup, 𝛼𝑆 is a penalty factor. The speed within the range of 0.95x to 1.2x the target speed are considered acceptable, allowing for minor underspeed and moderate speedup. Both overspeed and underspeed have decreasing or even negative reward value. It also encourages to use as low GPU frequency as possible and rewards high speculation speedup. By doing this, the agent manages to maintain speed at target level with relatively low GPU frequency. 𝑟 𝑣𝑑 refers to variable depth reward, defined as: ( 𝛼 𝑣𝑑+ · 𝐹𝑡 · (1 − 𝑣𝑑𝑟 ) 𝑟 𝑣𝑑 = 𝛼 𝑣𝑑 − · 𝐹𝑡 · (1 − 𝑣𝑑𝑟 )

1: Initialize DQN parameters 𝜃 , target network 𝜃 − , and replay

buffer D 2: Observe initial system state 𝑠 1 3: Initialize DVFS settings and decoding configuration 4: for 𝑡 = 1, 2, . . . do

Select action 𝑎𝑡 ∼ 𝜋𝜃 (𝑠𝑡 ) Update CPU/GPU frequencies (𝑓𝑐0, 𝑓𝑐1, 𝑓𝑐2, 𝑓𝑔 ) 7: Set decoding parameters (𝑒𝑡 , 𝑛𝑡 , 𝑣𝑡 ) 8: Execute system for control interval Δ𝑡 9: Observe next state 𝑠𝑡 +1 10: Compute reward 𝑟𝑡 = R (𝑠𝑡 , 𝑎𝑡 , 𝑠𝑡 +1 ) 11: if system is in valid operating regime then 12: Store (𝑠𝑡 , 𝑎𝑡 , 𝑟𝑡 , 𝑠𝑡 +1 ) in D 13: end if 14: Update 𝜃 using minibatch from D 15: Update target network 𝜃 − ← 𝜃 (periodically) 16: 𝑠𝑡 ← 𝑠𝑡 +1 17: end for 5: 6:

Table 2: Hardware platforms description. Hardware CPU CPU Frequency CPU Clusters GPU GPU Frequency Memory Typical Power

Jetson AGX Orin

Jetson Orin Nano

12× Arm Cortex-A78AE 0.1–2.0 GHz (29 levels) 3 clusters Ampere (2048 CUDA cores) 0.3–1.3 GHz (11 levels) 64 GB LPDDR5 15–60 W

6× Arm Cortex-A78AE 0.1–1.7 GHz (22 levels) 2 clusters Ampere (1042 CUDA cores) 0.3–1.0 GHz (8 levels) 8 GB LPDDR5 7–15 W

if 𝐹𝑡 > 0 if 𝐹𝑡 ≤ 0

where 𝑣𝑑𝑟 is ratio between variable depth and full depth of an LLM, 𝛼 𝑣𝑑+ and 𝛼 𝑣𝑑 − are reward factors, 𝐹𝑡 = tanh(𝑇𝑡𝑒𝑚𝑝 − 𝑇𝑣𝑑 ), 𝑇𝑣𝑑 is a predefined threshold temperature. This term rewards deep depth 𝑣𝑑 when thermal headroom is adequate, while promoting the use of shallower models when thermal headroom is limited. For thermal consideration, PELM employs a sigmoid function that gives reward based on the gap between current temperature and warning temperature, plus a rate of change consideration: 𝑟𝑡ℎ𝑒𝑟𝑚𝑎𝑙 = −(sigmoid(𝑇𝑡𝑒𝑚𝑝 − 𝑇𝑤𝑎𝑟𝑛𝑖𝑛𝑔 ) + 𝛿𝑡𝑒𝑚𝑝 ). When the temperature is significantly below the warning threshold, the penalty approaches zero; however, as 𝑇𝑡𝑒𝑚𝑝 approaches and exceeds the warning threshold, the penalty increases progressively. PELM also penalizes mismatched workloads and processor frequencies, denoted as 𝑟𝑢 . For each processor 𝑝𝑟𝑜𝑐𝑖 , 𝑖 ∈ {1, 2, .., 𝑁 }, the corresponding penalty value is: ( 𝑢 − 𝑓𝑟𝑖 𝑝𝑢𝑖 = 0

Algorithm 1 PELM Runtime Control

if 𝑢 > 𝑓𝑟𝑖 + 0.05 otherwise

Í𝑁 where 𝑓𝑟𝑖 is the frequency ratio for 𝑝𝑟𝑜𝑐𝑖 . Then 𝑟𝑢 = − 𝑖=1 𝑝𝑢𝑖 , which is the sum of each processor’s penalty. Lastly, we have term Í𝑁 𝑟 𝑝 that penalizes large power consumption: 𝑟 𝑝 = −𝛼 𝑝 × 𝑖=1 (𝑝𝑖 ). The complete updating and control procedure of PELM is presented in Algorithm 1.

3.3

Saving Compute Caused by Missing KV-Cache

Due to variable depth, each time the LLM may use different depths instead of conventional full model for verification. This causes an issue illustrated in Figure 4, when the 4th token has been decoded and its target depth exceeds previous tokens, KV-cache missing happens in layers where previous token decodings did not happen. A naive approach is to concatenate the previous token to recompute from the first layer to the last token’s target layer, which is computationally inefficient. Drawing ideas from previous work [7, 35], we propose to use Hidden Queue Cache (HQCache) for storing hidden states of intermediate layers. When the target depth for generating the current token exceeds the depth used to generate the previous token, we can load the hidden state and KV-cache from the deepest layer used to generate the previous tokens and recompute from there rather than from the first layer. Prior work leverages this idea to cache and reload from select layers to speed up early exit strategies. In this work, we generalize this idea to cache and reload from any layer and apply to both the drafting and verification steps in self-speculative decoding.

3.4

Implementation

Hardware platforms. We deploy PELM on two widely used embedded edge devices – Jetson AGX Orin and Jetson Orin Nano,

PELM : On-Device LLM Inference with Speculative Decoding and DVFS

SenSys ’26, May 11–14, 2026, Saint Malo, France

Legend Skipped

Restored by HQCache

HQCache

Computed

Hidden states

Output token

Cached hidden states

Cached hidden states join the next forward

CPU0 Freq

HQCache Updated Exit Layer Num Spec.

T3 forward pass

Decode T1’

T2’

T3’

Variable Depth

T4’

Verify output

State Figure 4: Hidden Queue Cache mechanism to minimize overhead for recomputing KV Cache at deeper layers for previously generated tokens.

representing high-end and lower-end mobile computing solutions respectively. The detailed hardware specifications are summarized in Table 2. Both devices are installed with Jetpack 6.2.1, running on Ubuntu 22.04.5 LTS. We use Python 3.10.12 for the DVFS monitoring and controlling, leveraging sysfs [28] for system metrics and frequency control. Q-learning implementation. Given the large action space—such as up to 29 frequency configurations for each CPU cluster and 11 for the GPU on the NVIDIA Jetson AGX Orin—we adopt a branch-DQN framework to address the complexity, following the methodology proposed in prior work [24]. Specifically, the high-dimensional action space is decomposed into multiple sub-domains, significantly reducing both model parameters and computational overhead, as illustrated in Figure 5. The resulting lightweight DQN model is implemented using PyTorch 2.8.0 and runs efficiently on CPU cores. PELM updates the control decision every 100 ms. The selected control parameters (CPU/GPU frequency, speculation parameters, and verification depth) are held constant for all tokens generated within this control window, and are updated only at the next control step. Language Model selection. PELM requires access to intermediatelayer outputs (e.g., via early-exit heads) that allow dynamic draft and verification depth control. In our implementation, we adopt LayerSkip [7], an open-source LLM pretrained with an early-exit mechanism, as it naturally exposes such intermediate outputs. This requirement reflects a model capability rather than a model-specific dependency (see discussions in Section 6).

Evaluation Setup

Datasets and LLMs. To evaluate the efficiency of our DVFS method, we construct a mobile-scale dataset by subsampling (uniform sampling with a fixed seed for reproducibility) 40 samples from each of five representative datasets which strikes a balance between evaluation comprehensiveness and the computational constraints of mobile devices: GSM8K [5], focusing on mathematical reasoning (Math); NQ-open [15, 18], for question answering (QA); HumanEval [4], for code generation (Code); WMT14-DE-EN [2], for

FCN Feature Branch Q-values

Figure 5: Branch Q-learning network structure.

machine translation (Tran); and CNN/Daily Mail [11], for text summarization (Sum). This selection ensures a diverse range of domains for evaluation. For LLMs, we select LLaMA-3.2-1B, LLaMA-3.1-8B, and LLaMA-2-13B models pretrained using LayerSkip [7] for evaluation (Section 3.4), covering different sizes that allow us to evaluate the DVFS governor under distinct computational workloads and thermal stress. These models are executed using the HuggingFace Transformers library [36] with a PyTorch backend. The decoding is configured with a batch size of one, approximating the singlerequest behavior of on-device mobile queries. Test-bed. We evaluate our DVFS governor on two mobile/embedded platforms: the NVIDIA Jetson AGX Orin and Orin Nano (details in Section 3.4). To evaluate robustness under varying thermal conditions, we define four cooling conditions by fixing fan PWM levels: 64 (High), 51 (Moderate), 38 (Low), and 26 (Minimum) for the AGX Orin. Due to the Nano’s limited heat dissipation, its levels are set higher: 90 (High), 77 (Moderate), 64 (Low), and 51 (Minimum). To ensure a fair comparison from a consistent thermal baseline, all sessions are initiated at 45◦ C. We define thermal failure thresholds at 78◦ C (AGX Orin) and 73◦ C (Nano); any session reaching its limit is forcibly terminated, marking a failure in the CR metric. The DVFS "warning" temperature is set 5◦ C below this critical point. Finally, we set the target decoding speed for our governor to match the default system’s baseline performance: (AGX Orin) 25 Tokens/s (1B), 10 Tokens/s (8B), and 6 Tokens/s (13B); and (Nano) 18 Tokens/s (1B). Evaluation Metrics. We evaluate our method using two categories of metrics: (1) DVFS-Level Efficiency and (2) Task-Level Quality. Our DVFS-Level Efficiency metrics, the primary focus of this work, quantify the runtime performance of our governor. These include: Decoding Speed (Tokens/s); Energy Consumption (Joules), the total energy consumed per session; Completion Rate (CR), the percentage of samples successfully completed; Layers Per Accepted Token (LPAT), the average number of layers executed per accepted token; Mean Variable Depth (VDm); and Speculation Ratio (SR), which measures the speedup from our speculation method.

SenSys ’26, May 11–14, 2026, Saint Malo, France

Weisi Yang and Stephen Xia

High Cooling Mode

6.0

200

Moderate Cooling Mode 400

7.0 6.0

Low Cooling Mode 400

7.0 6.0

Minimal Cooling Mode 400

5.0 200

4.0

BLTN STSD FUSE zTT PELM DVFS

7.0 6.0 5.0

BLTN STSD FUSE zTT PELM DVFS

4.0

300

12.0 300

12.0

200

10.0 200

10.0

100

8.0 100 6.0 0

8.0

BLTN STSD FUSE zTT PELM DVFS

JPQ

60 40 20 0

BLTN STSD FUSE zTT PELM DVFS

26.0 25.0 24.0 23.0 22.0

BLTN STSD FUSE zTT PELM DVFS

60 40 20 0

BLTN STSD FUSE zTT PELM DVFS

26.0 25.0 24.0 23.0 22.0

BLTN STSD FUSE zTT PELM DVFS

60 40 20 0

BLTN STSD FUSE zTT PELM DVFS

26.0 25.0 24.0 23.0 22.0

TPS

JPQ

BLTN STSD FUSE zTT PELM DVFS

7.0

BLTN STSD FUSE zTT PELM DVFS

60 40 20 0

6.0 26.0 25.0 24.0 23.0 22.0

TPS

JPQ

13B

400

Speed

TPS

Energy/Samples_Run

BLTN STSD FUSE zTT PELM DVFS

(a) AGX Orin

20 0

20.0 40

Moderate Cooling Mode

15.0 20 BLTN STSD FUSE zTT PELM DVFS

10.0

Low Cooling Mode

20.0 40 15.0 20

BLTN STSD FUSE zTT PELM DVFS

10.0

20.0 40

Minimal Cooling Mode

15.0 20 BLTN STSD FUSE zTT PELM DVFS

10.0

20.0 15.0

TPS

JPQ

High Cooling Mode

BLTN STSD FUSE zTT PELM DVFS

10.0

(b) Orin Nano

Figure 6: Energy consumption and decoding speed for each method across different settings. (JPQ: Joules per Query, TPS: Tokens per Second). For Task-Level Quality, we verify that efficiency gains do not lead to catastrophic degradation in output quality by reporting standard task-specific metrics (Accuracy, EM, pass@1, BLEU, and ROUGE-L) separately for each benchmark (Table 4). These metrics directly capture correctness and generation quality, allowing us to detect potential degradation under variable-depth execution. For a comprehensive comparison, we additionally normalize each task score to a 0–100 scale and average them to compute the Task Performance Average (TSA), which serves as a summary indicator. Finally, we define PPJ (Performance per joule, values are scaled by 103 for readability.) as TSA/Energy (J) to quantify the joint efficiency–quality trade-off. Baseline methods. We compare PELM with the following baseline methods: • Built-in (BLTN). The system’s default DVFS governors – schedutil for the CPU and nvhost_podgov for the GPU. • Standard Self-Speculative Decoding (STSD). A static speculative decoding baseline, in which the first quarter of the model functions as the draft model, while the complete model acts as the target model. The draft length is fixed at 3. The default system DVFS governor is used throughout STSD.

• FUSE [43]. A DVFS governor for mobile LLMs. It uses a search-based approach to find optimal frequencies that either minimize energy for a target speed or maximize speed for an energy budget. • zTT [14]. An RL-based DVFS framework for DNN tasks. Its primary goal is to learn a policy that maximizes performance while actively mitigating thermal throttling to maintain QoE. In the next section, we demonstrate how PELM’s co-design of DVFS and speculative decoding outperforms the system built-in governor and conventional QoE-based methods.

5 Results 5.1 Power Governing Performance Overall performance. We first compare PELM with baseline methods in terms of two primary metrics to evaluate their overall performance: energy consumed per query and token decoding speed. As shown in Figure 6a, for LLaMA-13B model running on AGX Orin, PELM has the least energy consumption across all cooling conditions, achieving 29.0% to 52.4% energy savings compared to the other four methods, while keeping the highest average generation speed under three out of four cooling conditions. On 8B models,

0.20

FUSE

1.00

0.46

0.28

0.24

zTT

1.00

0.34

1.00

0.45

High

Moderate

Low

Minimum

Cooling Condition

BLTN

0.28

0.70

0.45

STSD

0.47

1.00

0.54

FUSE

STSD

1.00

0.70

0.43

1.00

High

Moderate

Low

Minimum

DVFS Method

0.19

zTT

0.24

1.00

FUSE

1.00

zTT

1.00

High

Moderate

Low

Minimum

BLTN

1.00

0.79

STSD

1.00

1.00 1.00

1.00

0.84

FUSE

1.00

0.77

1.00

High

Moderate

Low

Minimum

DVFS Method

1.00

zTT

1.00

PELM

BLTN

1.00

DVFS Method

Cooling Condition

(b) AGX Orin – 8B

PELM

(a) AGX Orin – 13B

Cooling Condition

SenSys ’26, May 11–14, 2026, Saint Malo, France

PELM

BLTN

0.34

DVFS Method

0.85

PELM

PELM : On-Device LLM Inference with Speculative Decoding and DVFS

Cooling Condition

(d) Orin Nano – 1B

Figure 7: Completion Rate (CR) for different DVFS under diverse thermal conditions. Table 3: Impact of DVFS Strategies on GPU/CPU Average Frequency Ratios on Jetson Platforms. Avg. Freq Ratio

Orin Nano

AGX Orin

DVFS

High

Moderate

Low

Minimum

GPU

CPU

GPU

CPU

GPU

CPU

GPU

CPU

13B

BLTN STSD FUSE zTT PELM

0.91 0.91 0.83 0.74 0.57

0.33 0.33 1.00 0.77 0.72

0.91 0.91 0.83 0.63 0.47

0.35 0.33 1.00 0.58 0.63

0.91 0.91 0.83 0.47 0.49

0.33 0.33 1.00 0.38 0.65

0.90 0.91 0.83 0.55 0.48

0.33 0.33 1.00 0.50 0.59

BLTN STSD FUSE zTT PELM

0.74 0.68 0.92 0.83 0.62

0.34 0.49 1.00 0.92 0.73

0.74 0.68 0.92 0.80 0.53

0.34 0.49 1.00 0.89 0.72

0.72 0.68 0.92 0.71 0.50

0.35 0.48 1.00 0.69 0.65

0.72 0.67 0.92 0.53 0.40

0.34 0.49 1.00 0.44 0.64

BLTN STSD FUSE zTT PELM

0.28 0.29 0.43 0.61 0.52

0.35 0.50 1.00 0.98 0.70

0.29 0.28 0.43 0.64 0.60

0.35 0.50 1.00 0.98 0.75

0.29 0.30 0.43 0.60 0.57

0.35 0.50 1.00 0.98 0.70

0.28 0.30 0.43 0.77 0.59

0.36 0.49 1.00 0.97 0.68

BLTN STSD FUSE zTT PELM

0.38 0.34 0.50 0.45 0.53

0.39 0.38 0.76 0.74 0.66

0.38 0.34 0.50 0.51 0.49

0.39 0.38 0.76 0.73 0.64

0.38 0.34 0.50 0.53 0.48

0.39 0.38 0.76 0.68 0.65

0.37 0.33 0.50 0.48 0.49

0.39 0.38 0.76 0.54 0.64

PELM consumes 13.4% to 32.5% less energy per query than baselines, with comparable generation speed. For the 1B model on AGX Orin, PELM consumes slightly higher energy (0.4%–7.3% above the lowest-energy baseline) but consistently meets the 25 tokens/s target, which several frequency-only

DVFS methods fail to achieve. This suggests that while hardwarelevel frequency scaling remains effective under light workloads with sufficient thermal headroom, its optimization space is limited. On Orin Nano, where the same 1B model represents a relatively heavier workload for the device, PELM achieves low energy consumption while maintaining competitive speed across cooling modes, demonstrating stable performance under tighter resource constraints. Overall, PELM’s advantage becomes increasingly pronounced as workload intensity grows. Frequency-only governors are bounded by hardware-level tuning and tend to either over-prioritize performance (e.g., BLTN, STSD, FUSE) or conservatively react to thermal throttling (e.g., zTT). By jointly controlling frequency and decoding behavior, PELM consistently achieves the best or comparable performance across thermal conditions while improving energy efficiency. Robustness under different thermal conditions. We evaluate the DVFS thermal robustness by checking the query completion rate (CR) for each case. In Figure 7, the results clearly divide the methods into two groups: those with thermal awareness and those without. Methods without thermal awareness (e.g., BLTN and FUSE) frequently fail to complete the entire query set, with CR dropping as low as 19% in the 13B workload. Although STSD achieves superior token throughput in many scenarios, it underperforms with respect to this critical metric. In contrast, both zTT and PELM demonstrate exceptional robustness, achieving 100% completion rate in all scenarios except one. The single failure case — the extreme 13B workload on minimum cooling — serves as a tie-breaker. In this worst-case scenario, PELM completed 45% of queries before hitting the limit, while zTT completed only 34%. This means PELM processed 32.4% more queries than the closest competitor, proving its superior robustness when the system is under maximum thermal stress. Adaptive Frequency Scaling Behavior. To investigate the mechanisms behind the optimized power and speed, we evaluated the average operating frequency ratios (summarized in Table 3). The results demonstrate PELM’s adaptive behavior, which manifests differently under heavy and light workloads. For heavy and moderate workloads (13B and 8B models on AGX Orin), PELM consistently operates at the lowest GPU frequency among all methods. This directly explains the previously observed energy savings. For instance, in the 13B-High scenario, PELM (PELM) runs at 0.57, which is 37.4% lower than BLTN and STSD (0.91) and 31.3% lower than FUSE (0.83). As thermal pressure increases (e.g., in Low or Minimum cooling), PELM adaptively lowers its frequency further, whereas BLTN and FUSE maintain their high, inefficient frequencies. Conversely, for light workloads (1B model on both platforms), PELM shows a critical adaptation for performance. On Jetson Orin, it operates at a higher GPU frequency (e.g., 0.52 in High mode) compared to the overly conservative BLTN (0.28) and FUSE (0.43) governors while avoiding a substantial increase in power consumption. This finding is crucial: PELM correctly identifies the ample thermal headroom and proactively increases its frequency to meet the QoE (token generation speed) target. This directly explains the "Overall Performance" result where PELM was the only method to

SenSys ’26, May 11–14, 2026, Saint Malo, France

meet the target speed, while others failed precisely because their frequencies were too low. This intelligent, bi-directional adaptation (saving power when heavy, boosting performance when light) is also observed on the Jetson Orin Nano. These results demonstrate that PELM possesses a genuine awareness of both workload demands and thermal conditions, allowing it to adapt its strategy to optimize for either energy efficiency or performance as needed.

Weisi Yang and Stephen Xia

Table 4: Task score comparison between PELM and the baselines. (TSA=Task Score Average, PPJ=Performance per joule) H

13B

Ablations

We create three reduced versions of PELM for ablation studies to evaluate the efficacy of its individual components. A1: In this version, we remove the variable depth control from PELM, fixing the depth to its maximum value. A2: Here, we eliminate the self-speculative decoding component, such that PELM only controls processor frequency and the exit layer (via variable depth) during autoregressive decoding. A3: In this case, we remove frequency control from PELM

Jetson AGX Orin

The variable depth execution in PELM, which dynamically uses a partial LLM, could potentially impact downstream task performance. We now evaluate this impact comprehensively. We compare PELM’s task scores against the vanilla (auto-regressive) decoding used by the baseline methods. The results are shown in Table 4. The primary concern is: does speculative decoding with variable depth execution degrade the task performance? As shown in Table 4, for 8B and 1B models, the overall task scores (TSA) remain statistically comparable to those of the vanilla full-model decoding on both Jetson platforms, with the largest degradation being less than 17.2%. For the 13B model, the output still retains meaningful quality to full depth vanilla baseline. These results show the effectiveness of variable depth execution without sacrificing task performance. Now we turn to performance efficiency. The PPJ column quantifies output quality normalized by energy consumption. In all moderate-to-heavy workloads (13B, 8B@Orin), PELM is significantly more efficient, delivering responses of comparable quality with a lower energy cost at target speed – achieving up to 45.4% higher performance-per-joule than baselines. Finally, the 1B results on both AGX Orin and Orin Nano may appear different at first glance, as PELM does not always achieve the highest PPJ among all baselines. This behavior is expected and consistent with the trends observed in Figure 6 and Figure 7. Under light workloads with minimal thermal pressure, the optimization objective shifts from aggressive energy minimization to QoE-aware performance control. In this regime, PELM intentionally allocates slightly more power to reliably meet the target decoding speed (25 tokens/s) while maintaining competitive energy efficiency. On Orin Nano, due to its more constrained compute capability, PELM operates under a tighter power–performance envelope and adopts a more balanced tradeoff between power and speed, whereas several “more efficient” baselines reduce power but fail to consistently sustain the desired throughput. These results demonstrate that PELM remains effective even in low-demand scenarios by adaptively balancing speed and efficiency, rather than over-optimizing for a single objective.

5.3

DVFS

Math

Code

Tran

Sum

TSA

PPJ

High

BLTN FUSE STSD zTT PELM

2.94 2.50 5.00 2.50 5.00

30.00 28.75 28.75 28.75 15.00

0.00 0.00 0.00 0.00 0.00

21.51 22.73 22.71 22.73 7.52

26.50 25.96 25.96 25.96 26.12

16.19 15.99 16.49 15.99 10.73

37.30 34.75 40.65 36.92 48.82

Moderate

BLTN STSD FUSE zTT PELM

9.09 7.14 7.69 2.50 10.00

20.00 30.95 30.95 28.75 12.50

0.00 0.00 0.00 0.00 0.00

23.97 22.08 22.72 22.73 9.68

27.71 27.37 26.68 25.96 26.92

16.15 17.51 17.61 15.99 11.82

44.48 50.76 46.63 39.41 49.32

Low

BLTN STSD FUSE zTT PELM

0.00 0.00 0.00 2.50 5.00

18.18 18.18 18.18 28.75 10.00

0.00 0.00 0.00 0.00 0.00

26.67 27.38 27.03 22.73 6.59

30.03 29.25 29.25 25.96 26.41

14.98 14.96 14.89 15.99 9.60

38.60 42.68 40.66 43.44 45.39

Minimum

BLTN STSD FUSE zTT PELM

0.00 0.00 0.00 9.09 7.69

12.50 10.00 18.18 20.00 16.67

0.00 0.00 0.00 0.00 0.00

26.17 26.17 26.67 23.97 10.52

31.33 31.33 31.33 27.71 25.09

14.00 13.50 15.23 16.15 11.99

36.02 39.56 46.95 50.23 52.42

High

BLTN STSD FUSE zTT PELM

20.00 22.50 20.00 20.00 15.00

22.50 23.75 22.50 22.50 22.50

27.50 27.50 27.50 27.50 25.00

18.76 18.76 18.76 18.76 18.51

21.75 21.44 21.75 21.75 22.57

22.10 22.79 22.10 22.10 20.71

100.72 126.19 98.93 105.05 133.26

Moderate

BLTN STSD FUSE zTT PELM

20.00 22.50 20.00 20.00 27.50

22.50 23.75 22.50 22.50 22.50

27.50 27.50 27.50 27.50 25.00

18.76 18.76 18.76 18.76 14.61

21.75 21.44 21.75 21.75 23.83

22.10 22.79 22.10 22.10 22.69

100.22 125.63 98.47 105.43 160.91

Low

BLTN STSD FUSE zTT PELM

23.33 22.50 23.33 20.00 25.00

23.33 23.75 23.33 22.50 18.75

23.33 27.50 23.33 27.50 27.50

14.49 18.76 14.49 18.76 12.23

22.54 21.44 22.54 21.75 23.33

21.41 22.79 21.41 22.10 21.36

108.32 124.87 105.80 108.63 149.90

Minimum

BLTN STSD FUSE zTT PELM

30.77 25.00 30.77 20.00 25.00

21.43 19.57 20.00 22.50 17.50

27.27 26.09 28.57 27.50 20.00

16.43 15.76 17.01 18.76 11.29

21.23 22.45 21.35 21.75 22.20

23.43 21.77 23.54 22.10 19.20

117.18 126.09 116.01 110.66 128.99

High

BLTN STSD FUSE zTT PELM

2.50 2.50 2.50 2.50 0.00

3.75 3.75 3.75 3.75 3.75

7.50 7.50 7.50 7.50 10.00

10.21 10.35 10.21 10.21 9.36

22.10 21.53 22.10 22.10 20.19

9.21 9.13 9.21 9.21 8.66

175.90 159.89 170.52 157.82 162.11

Moderate

BLTN STSD FUSE zTT PELM

2.50 2.50 2.50 2.50 2.50

3.75 3.75 3.75 3.75 3.75

7.50 7.50 7.50 7.50 7.50

10.21 10.35 10.21 10.21 9.50

22.10 21.53 22.10 22.10 21.34

9.21 9.13 9.21 9.21 8.92

177.45 158.13 167.45 154.49 159.89

Low

BLTN STSD FUSE zTT PELM

2.50 2.50 2.50 2.50 2.50

3.75 3.75 3.75 3.75 3.75

7.50 7.50 7.50 7.50 7.50

10.21 10.35 10.21 10.21 9.99

22.10 21.53 22.10 22.10 21.92

9.21 9.13 9.21 9.21 9.13

175.00 160.61 168.75 158.60 163.27

Minimum

BLTN STSD FUSE zTT PELM

2.50 2.50 2.50 2.50 2.50

3.75 3.75 3.75 3.75 3.75

7.50 7.50 7.50 7.50 7.50

10.21 10.35 10.21 10.21 9.42

22.10 21.53 22.10 22.10 21.13

9.21 9.13 9.21 9.21 8.86

172.85 161.78 168.48 143.90 163.12

High

BLTN STSD FUSE zTT PELM

2.50 2.50 2.50 2.50 2.50

3.75 3.75 2.50 3.75 2.50

7.50 7.50 7.50 7.50 7.50

10.21 10.21 8.49 10.21 8.49

22.10 21.48 21.54 22.10 21.54

9.21 9.09 8.51 9.21 8.51

309.82 337.87 279.44 300.97 297.56

Moderate

BLTN STSD FUSE zTT PELM

2.50 2.50 5.00 2.50 5.00

3.75 3.75 2.50 3.75 2.50

7.50 7.50 5.00 7.50 5.00

10.21 10.21 5.93 10.21 5.93

22.10 21.48 20.58 22.10 20.58

9.21 9.09 7.80 9.21 7.80

307.54 334.87 253.92 301.37 283.03

Low

BLTN STSD FUSE zTT PELM

2.50 2.50 2.50 2.50 2.50

3.75 3.75 3.75 3.75 3.75

7.50 7.50 7.50 7.50 7.50

10.21 10.21 10.21 6.51 6.51

22.10 21.48 22.10 21.13 21.13

9.21 9.09 9.21 8.28 8.28

305.44 331.98 298.33 264.39 315.67

Minimum

BLTN STSD FUSE zTT PELM

2.50 2.94 0.00 3.03 0.00

3.75 4.29 3.75 4.55 3.75

7.50 9.68 10.00 9.68 10.00

10.21 10.42 7.38 10.45 7.38

22.10 21.94 20.58 22.64 20.58

9.21 9.85 8.34 10.07 8.34

313.31 363.56 280.62 326.03 302.58

Task Performance

Jetson Orin Nano

5.2

Cooling

PELM : On-Device LLM Inference with Speculative Decoding and DVFS

SenSys ’26, May 11–14, 2026, Saint Malo, France

Cooling

DVFS

SSD

Energy

Speed

LPATm

VDm

TSA

PPJ

High

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

43724 66247 52732 59446

6.40 5.60 6.10 6.80

31.00 35.97 32.63 32.39

31.77 40.00 32.64 33.18

1.02 1.11 1.00 1.02

10.73 16.06 10.23 11.07

48.82 48.23 38.59 37.05

Moderate

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 0.53

47696 71966 44803 27809

5.80 5.30 6.20 7.00

31.95 37.85 31.54 31.76

32.72 40.00 31.53 33.02

1.02 1.05 1.00 1.04

11.82 16.07 9.86 9.13

49.32 44.43 43.80 34.82

Low

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 0.71 0.71 0.33

42084 39957 30603 17785

6.00 5.20 5.90 6.90

30.76 36.77 32.40 32.34

31.58 40.00 32.40 34.10

1.03 1.09 1.00 1.05

9.60 16.23 9.23 12.02

45.39 57.26 42.52 44.59

Minimum

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

0.45 0.37 0.45 0.23

20363 20283 17720 14207

5.50 4.90 5.90 6.90

33.49 39.09 32.73 32.49

34.29 40.00 32.74 34.51

1.02 1.02 1.00 1.06

11.00 17.01 8.66 13.76

52.42 62.07 43.48 44.54

High

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

30934 32237 35151 35311

9.20 9.40 9.00 11.00

26.07 27.28 28.93 27.29

29.62 32.00 28.88 29.85

1.14 1.18 1.00 1.10

20.71 22.60 22.82 21.97

133.26 139.51 129.22 123.80

Moderate

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

28059 30500 31680 35705

9.60 9.10 9.60 11.60

24.84 26.54 27.05 25.61

28.55 32.00 27.04 27.80

1.15 1.20 1.00 1.09

22.69 22.89 20.26 21.04

160.91 149.34 127.23 117.29

Low

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 0.86

28360 31752 30829 28554

9.20 8.60 9.60 11.50

23.95 26.87 25.99 25.73

26.93 32.00 25.99 28.27

1.12 1.19 1.00 1.10

21.36 22.78 17.20 20.35

149.90 142.79 111.03 122.59

Minimum

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 0.96 0.83 0.53

29617 33071 24233 17695

9.30 8.00 9.40 11.50

23.32 27.55 26.36 25.93

26.86 32.00 26.34 28.29

1.15 1.16 1.00 1.09

19.20 22.82 16.85 21.22

128.99 133.16 115.43 127.12

High

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

10629 11131 11257 9440

25.80 25.50 22.20 27.20

12.95 13.89 15.01 12.91

14.89 16.00 15.02 14.05

1.15 1.15 1.00 1.09

8.66 9.13 8.51 8.65

162.11 163.15 150.51 182.43

Moderate

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

11098 10993 11162 9227

25.20 25.20 23.80 26.70

13.82 13.46 14.30 12.93

15.70 16.00 14.30 15.02

1.14 1.19 1.00 1.16

8.92 9.13 8.75 8.22

159.89 165.25 155.99 177.23

Low

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

11130 10850 11772 9995

25.90 25.20 22.90 26.20

13.03 13.67 14.81 13.60

15.48 16.00 14.84 15.68

1.19 1.18 1.00 1.15

9.13 9.16 9.35 8.88

163.27 168.05 158.13 176.74

Minimum

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

10807 11209 11558 9761

25.50 23.60 23.00 25.90

13.56 14.21 14.89 13.69

15.72 16.00 14.91 15.18

1.16 1.12 1.00 1.11

8.86 9.16 8.27 8.71

163.12 162.63 142.34 177.67

High

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

5688 5926 6005 5652

17.60 16.30 16.60 19.50

13.13 14.64 14.85 14.30

14.42 16.00 14.83 15.64

0.91 1.09 1.00 1.10

8.51 9.18 9.64 8.97

297.56 308.42 319.50 315.65

Moderate

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

5486 5988 6009 5263

19.60 16.90 17.00 20.70

11.96 14.54 14.32 12.89

13.05 16.00 14.33 13.53

0.92 1.10 1.00 1.05

7.80 9.14 9.00 8.85

283.03 303.78 298.15 334.64

Low

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 1.00

5218 6010 5362 5253

19.80 16.70 17.90 20.70

11.98 14.11 13.69 12.80

12.69 16.00 13.69 13.67

0.94 1.14 1.00 1.06

8.28 9.13 9.70 9.14

315.67 302.26 360.09 346.40

PELM PELM_A1 PELM_A2 PELM_A3

Y N Y Y

Y Y N Y

Y Y Y N

1.00 1.00 1.00 0.99

5486 5906 5209 4937

17.10 14.20 16.60 21.20

11.96 13.92 12.94 12.19

13.19 16.00 12.94 13.01

0.91 1.15 1.00 1.06

8.34 9.07 7.52 7.44

302.58 305.77 287.44 298.47

Jetson AGX Orin

13B

Jetson Orin Nano

Minimum

and allow the BLTN governor to manage it, leaving PELM responsible solely for self-speculative decoding with variable depth. Table 5 provides an explicit component-wise breakdown of PELM. The three columns (DVFS, VD, SSD) correspond to three orthogonal control dimensions. The full PELM enables all three (Y/Y/Y), while A1, A2, and A3 each disable exactly one component, forming a complete leave-one-out ablation matrix. Results are grouped by thermal regimes to reveal component sensitivity under distinct hardware constraints. Removing Variable Depth (A1) universally degrades performanceper-joule (PPJ) in most scenarios. By being forced to use the maximum model depth, A1 consistently performs more work per token (higher LPATm) and is universally slower than the full PELM system. This indicates that dynamically selecting an earlier exit layer is critical for computational efficiency. The removal of Speculative

5.4

Query-Locality Impact

Table 6: Metrics across different localities. (TSA=Task Score Average, LPATm=layers per accepted token mean value, SR=Speculation Ratio, PPJ=Performance per joule) H (#P)

AGX Orin (8B)

Decoding (A2) causes the most significant performance degradation. This purely autoregressive version is simultaneously slower and consumes substantially more energy in every tested scenario (e.g., 8B Moderate: 31680 J vs. PELM’s 28059 J), resulting in a significant drop in PPJ. This demonstrates that speculative decoding is essential to PELM’s speed and energy savings. Finally, removing Frequency Control (A3) reveals a critical trade-off. For the light 1B workload on AGX Orin, A3 achieves superior PPJ by aggressively prioritizing speed. However, this naive strategy fails on all moderate-to-heavy workloads. In those cases, A3 consumes disproportionate energy for its speed gains (e.g., 35705 J vs. 28059 J for 8B Moderate), severely hurting efficiency. In conclusion, this study highlights the importance of PELM’s components.

Orin Nano (1B)

Table 5: Ablation study of PELM components. (TSA=Task Score Average, SR=Speculation Ratio, PPJ=Performance per joule)

Cooling

Locality

Energy

Speed

LPATm

VDm

TSA

PPJ

High

1 10 20 40

30934 27867 27547 30868

9.2 9.6 9.5 10.4

26.07 25.55 24.93 24.85

29.62 27.86 29.32 29.94

1.14 1.09 1.18 1.20

20.71 20.40 20.84 21.19

133.26 145.68 150.55 136.60

Moderate

1 10 20 40

28059 28562 28753 29470

9.6 10.0 9.8 9.9

24.84 24.63 26.14 25.63

28.55 29.22 30.15 30.31

1.15 1.19 1.15 1.18

22.69 23.47 20.51 22.01

160.91 163.55 141.95 148.66

Low

1 10 20 40

28360 26551 27626 26357

9.2 10.0 9.6 10.6

23.95 23.28 23.76 23.22

26.93 26.68 27.43 27.61

1.12 1.15 1.15 1.19

21.36 20.30 22.48 19.05

149.90 152.16 161.93 143.86

Minimum

1 10 20 40

29617 27272 27711 27258

9.3 9.9 9.9 10.7

23.32 23.07 23.61 23.00

26.86 26.02 26.90 26.68

1.15 1.12 1.14 1.16

19.20 19.13 19.84 18.83

128.99 139.55 142.47 137.45

High

1 10 20 40

5688 5681 5514 5562

17.6 17.4 17.7 18.3

13.13 13.31 12.74 12.96

14.42 14.86 14.25 14.18

0.91 1.11 1.12 1.10

8.51 9.88 8.68 8.48

297.56 346.14 313.36 303.29

Moderate

1 10 20 40

5486 5729 5251 5321

19.6 17.6 18.4 17.9

11.96 13.36 12.37 12.70

13.05 15.06 13.42 14.25

0.92 1.12 1.09 1.12

7.80 9.08 8.46 9.42

283.03 315.24 320.74 352.36

Low

1 10 20 40

5218 5445 5267 5096

19.8 18.3 18.3 19.6

11.98 12.50 11.86 11.38

12.69 13.80 13.35 12.85

0.94 1.10 1.12 1.12

8.28 8.70 8.83 7.59

315.67 318.02 333.44 296.55

Minimum

1 10 20 40

5486 5487 5309 4925

17.1 16.9 18.4 18.6

11.96 11.69 11.30 11.14

13.19 13.09 12.68 12.80

0.91 1.12 1.12 1.15

8.34 7.65 8.13 8.15

302.58 277.42 304.81 329.46

While all previous evaluations used a randomized query dataset, real-world scenarios often exhibit temporal locality, with consecutive queries belonging to the same task type [8]. To investigate the impact of locality on end-to-end performance, we reordered the query dataset to form clusters of queries from the same task domain and did evaluations on 8B@AGX Orin and 1B@Orin Nano. The results are in shown in Table 6. PELM intelligently adapts to and exploits query locality. Rather than a static system, it learns from the locality in query patterns for more effective speculations. The primary finding is that speculation speedup ratio (SR) likely benefits directly from clustered query sequence. For instance, 18

SenSys ’26, May 11–14, 2026, Saint Malo, France

out of 24 cases for larger locality (10, 20, 40) show an increase in SR compared to fully randomized order (Locality 1), which contributes to lower LPATm and improved end-to-end decoding speed. As a result, the total energy consumption is reduced for larger locality for most cases. However, the results also reveal complexities, particularly for 1B model on the Nano. In that setting, 8 out of 12 higher-locality cases do not result in higher speedups, in contrast to 8B model. Moreover, task performance (TSA) exhibits greater variance than 8B on Jetson. We hypothesize this is due to the constraints of the small (1B) model, which makes correct speculation more challenging. While SR may increase on plausible tokens, the final task quality could be impacted. This, in turn, negatively affects the PPJ score. Despite these complex and non-monotonic behaviors, PELM does not exhibit catastrophic performance degradation. This overall stability underscores its robustness and usability across diverse query locality patterns.

5.5

System Overhead

We empirically measure the runtime, power, and memory overhead of PELM, and show that its control and caching mechanisms introduce negligible system cost. PELM’s DQN agent is lightweight and CPU-only. The policy network (BranchQNet) is a small MLP with two 32-wide fully connected layers and shallow linear heads (5,687 parameters, ≈22 KB in FP32), and all action selection and training run on CPU without launching GPU kernels. At a 10 Hz control rate (100 ms period), action selection takes 0.087–0.922 ms and training updates 9–18 ms, remaining well within the control window. End-to-end measurements show that PELM does not increase CPU power relative to baselines (≈1.8–2.2 W), while substantially reducing GPU power (e.g., 12.39 W vs. 20–21 W on 8B@AGX, Moderate), indicating that DRL overhead is negligible compared to the energy savings it enables. HQCache stores hidden states (not KV cache) and incurs small latency and memory cost. On AGX Orin, HQCache resume/load latency is 0.090–0.320 ms for token lengths 1–1024, whereas a single decoder-layer forward at the same length takes 4.195–20.416 ms (≈46–60× slower), making cache loading far cheaper than recomputation. For an 8B LLaMA-style model with hidden size 𝑑 = 4096 in FP16, one checkpoint of length 𝐿 costs 𝐿 × 𝑑 × 2 bytes (16 MiB at 𝐿 = 2048; 128 MiB for 𝐾 = 8), typically < 1% of FP16 model weights and modest relative to Jetson unified memory.

Discussion

Limitations. First, the effectiveness of PELM depends on access to LLMs that expose intermediate-layer outputs through early-exit supervision (e.g., LayerSkip [7]). Currently, publicly available models with robust early-exit heads are concentrated within the LLaMA family. Extending PELM to other model families requires enabling intermediate layer supervision during pretraining or fine-tuning to expose early-exit logits. This modification is confined to the model training pipeline through intermediate layer supervision (SFT); the system-level components of PELM–— including the DVFS interface, runtime telemetry, and closed-loop governor —– remain unchanged. When early-exit capability is unavailable, PELM degrades to a reduced configuration that combines DVFS control with dynamic

Weisi Yang and Stephen Xia

self-speculative decoding using full-depth verification (see ablation study in Section 5.3). In this case, verification depth control is disabled, while remaining speculation control, hardware-level adaptation, and QoE-aware regulation continue to operate. Quantization is widely adopted for on-device LLM deployment and represents a complementary optimization to our approach. In this work, we focus on unquantized checkpoints to isolate and analyze the cross-layer control behavior of PELM. Integrating quantized models is feasible but requires additional model preparation and validation, particularly to ensure consistent intermediate-layer outputs under variable-depth execution and HQCache state resumption. From a control perspective, quantization primarily reshapes the latency–power characteristics of the model, which the closedloop governor is designed to adapt to in principle; however, a rigorous quantized evaluation remains future work. We view this as a promising direction to further expand the applicability of PELM to larger-scale models within edge device constraints. Our evaluation is also confined to NVIDIA Jetson platforms and their DVFS interfaces; portability to other SoCs may require remapping control knobs (e.g., heterogeneous CPU/GPU/NPU governors) and re-tuning policy targets. Future Directions. Promising future directions include making the governor more input- and output-aware. Token-level difficulty signals, such as entropy or probability margin, could enable finergrained depth control, while lightweight task classification could guide verification depth selection based on task complexity, improving the balance between hardware constraints and task-level quality under tight thermal and power budgets. Another direction is to improve decoding algorithms for on-device LLMs through hardware-software co-design. This includes more effective training of early-exit LLMs to improve exit reliability and reduce verification cost, extending such schemes to other model families such as Phi [1] and Qwen [39], and developing new methods to accelerate on-device generation. An important future direction is to evaluate PELM under corunning background workloads and platform-wide DVFS policies. In practical deployments, system-level governors and concurrent services may introduce additional contention and dynamic frequency scaling interactions; studying joint control or coordination mechanisms would further strengthen robustness in multi-tenant edge environments. Lastly, many orthogonal yet useful techniques such as quantization, and memory offloading could be incorporated into PELM for deploying LLM on broader platforms such as Raspberry Pi-class SBCs, smartphones, and NPUs, which will also require adapting to heterogeneous DVFS controls and thermal envelopes.

Conclusion

This paper proposes PELM, a novel power-efficient optimization framework co-designing DVFS and speculative decoding for ondevice LLM inference. PELM bridges the gap between prior decodingagnostic DVFS methods and hardware-agnostic speculative decoding. By introducing variable depth for verification execution, it enlarges the optimization space for further efficiency gains. We evaluate it with diverse datasets across different devices, hardware constraints, and QoE requirements, verifying its efficiency.

PELM : On-Device LLM Inference with Speculative Decoding and DVFS

References [1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024). [2] Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve SaintAmand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation. 12–58. [3] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 (2024). [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021). [6] Vincent Dumas, Fabrice Guillemin, and Philippe Robert. 2002. A Markovian analysis of additive-increase multiplicative-decrease algorithms. Advances in Applied Probability 34, 1 (2002), 85–111. [7] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. 2024. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 12622–12642. doi:10.18653/v1/ 2024.acl-long.681 [8] Ophir Frieder, Ida Mele, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, and Nicola Tonellotto. 2024. Caching Historical Embeddings in Conversational Search. ACM Trans. Web 18, 4, Article 42 (Oct. 2024), 19 pages. doi:10.1145/3578519 [9] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057 (2024). [10] Jiawei Geng, Zongwei Zhu, Weihong Liu, Xuehai Zhou, and Boyu Li. 2024. PowerLens: an adaptive DVFS framework for optimizing energy efficiency in deep neural networks. In Proceedings of the 61st ACM/IEEE design automation conference. 1–6. [11] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems 28 (2015). [12] Andreas Kosmas Kakolyris, Dimosthenis Masouros, Petros Vavaroutsos, Sotirios Xydis, and Dimitrios Soudris. 2025. throttLL’eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 1363–1378. [13] Andreas Kosmas Kakolyris, Dimosthenis Masouros, Sotirios Xydis, and Dimitrios Soudris. 2024. Slo-aware gpu dvfs for energy-efficient llm inference serving. IEEE Computer Architecture Letters 23, 2 (2024), 150–153. [14] Seyeon Kim, Kyungmin Bin, Sangtae Ha, Kyunghan Lee, and Song Chong. 2022. zTT: Learning-based DVFS with zero thermal throttling for mobile devices. GetMobile: Mobile Computing and Communications 25, 4 (2022), 30–34. [15] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453–466. [16] Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. [n. d.]. Mobile and edge evaluation of large language models. In Workshop on Efficient Systems for Foundation Models II@ ICML2024. [17] Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. 2024. Melting point: Mobile evaluation of language transformers. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 890–907. [18] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Korhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 6086–6096. doi:10.18653/v1/P19-1612 [19] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. PMLR, 19274–19286. [20] Xiang Li, Zhenyan Lu, Dongqi Cai, Xiao Ma, and Mengwei Xu. 2024. Large language models on mobile devices: Measurements, analysis, and insights. In Proceedings of the Workshop on Edge and Mobile Foundation Models. 1–6.

SenSys ’26, May 11–14, 2026, Saint Malo, France

[21] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 7421–7432. doi:10.18653/v1/ 2024.emnlp-main.422 [22] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 (2024). [23] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=4exx1hUffq [24] Chengdong Lin, Kun Wang, Zhenjiang Li, and Yu Pu. 2023. A workload-aware DVFS robust to concurrent tasks for mobile devices. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking. 1–16. [25] Daria Lioubashevski, Tomer M. Schlank, Gabriel Stanovsky, and Ariel Goldstein. 2025. Looking Beyond the Top-1: Transformers Determine Top Tokens in Order. In Forty-second International Conference on Machine Learning. https://openreview. net/forum?id=2B11W1Z6ID [26] Kaiyuan Liu, Xiaobo Zhou, and Li Li. 2025. m 2 LLM: A Multi-Dimensional Optimization Framework for LLM Inference on Mobile Devices. IEEE Transactions on Parallel and Distributed Systems (2025). [27] Qunyou Liu, Darong Huang, Marina Zapater, and David Atienza. 2025. GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving. arXiv preprint arXiv:2508.16449 (2025). [28] Patrick Mochel. 2005. The sysfs filesystem. In Linux Symposium, Vol. 1. The Linux Foundation San Francisco, CA, USA, 313–326. [29] Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar, and Ravishankar K Iyer. 2024. Power-aware deep learning model serving with {𝜇 -Serve } . In 2024 USENIX Annual Technical Conference (USENIX ATC 24). 75–93. [30] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems 35 (2022), 17456–17472. [31] Dongjoo Seo, Juhee Sung, Jaekoo Lee, and Nikil Dutt. 2025. GOLD: Green Optimization of Language Models Serving on Devices. In 2025 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 1–2. [32] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. Layer by Layer: Uncovering Hidden Representations in Language Models. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=WGXb7UdvTX [33] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca, Josep Torrellas, and Ricardo Bianchini. 2025. Tapas: Thermal-and poweraware scheduling for LLM inference in cloud platforms. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1266–1281. [34] D Suleiman, Muhammed Ibrahim, and Ibrahim Hamarash. 2005. Dynamic voltage frequency scaling (DVFS) for microprocessors power and energy reduction. In 4th International Conference on Electrical and Electronics Engineering, Vol. 12. 4. [35] Zhepei Wei, Wei-Lin Chen, Xinyu Zhu, and Yu Meng. 2025. AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=VnO2GEpmlb [36] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Qun Liu and David Schlangen (Eds.). Association for Computational Linguistics, Online, 38–45. doi:10.18653/v1/2020. emnlp-demos.6 [37] Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. 2024. Swift: Onthe-fly self-speculative decoding for llm inference acceleration. arXiv preprint arXiv:2410.06916 (2024). [38] Mengwei Xu, Dongqi Cai, Wangsong Yin, Shangguang Wang, Xin Jin, and Xuanzhe Liu. 2025. Resource-efficient algorithms and systems of foundation models: A survey. Comput. Surveys 57, 5 (2025), 1–39. [39] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [40] Zicong Ye, Kunming Zhang, and Guoming Tang. 2025. AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimization. arXiv preprint arXiv:2508.01744 (2025). [41] Amir Yeganeh-Khaksar, Mohsen Ansari, Sepideh Safari, Sina Yari-Karin, and Alireza Ejlali. 2020. Ring-DVFS: Reliability-aware reinforcement learning-based DVFS for real-time embedded systems. IEEE Embedded Systems Letters 13, 3

SenSys ’26, May 11–14, 2026, Saint Malo, France

(2020), 146–149. [42] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168 (2023). [43] Zongpu Zhang, Pranab Dash, Y Charlie Hu, Qiang Xu, Jian Li, and Haibing Guan. 2025. Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency. arXiv preprint arXiv:2507.02135 (2025).

Weisi Yang and Stephen Xia

[44] Ziyang Zhang, Yang Zhao, Ming-Ching Chang, Changyao Lin, and Jie Liu. 2025. E4: Energy-Efficient DNN Inference for Edge Video Analytics Via Early Exiting and DVFS. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 1165–1173. [45] Ziyang Zhang, Yang Zhao, Huan Li, Changyao Lin, and Jie Liu. 2024. DVFO: Learning-based DVFS for energy-efficient edge-cloud collaborative inference. IEEE Transactions on Mobile Computing 23, 10 (2024), 9042–9059.

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

Related documents