arXiv CS open access

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

2026 · arxiv_cs

arXiv CS · Papers · 2026

distributed-systems internet networking protocols

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

arXiv:2608.05926v1 [cs.NI] 6 Aug 2026

Guanqiao Qu, Graduate Student Member, IEEE, Shuo Chen, Graduate Student Member, IEEE, Qian Chen, Member, IEEE, Kin K. Leung, Fellow, IEEE, Xianhao Chen, Member, IEEE

Index Terms—Large language models, edge inference, autoregressive decoding, speculative decoding, user scheduling, task throughput maximization.

I. I NTRODUCTION In next-generation mobile networks, large language model (LLM) inference at the network edge is emerging as an important paradigm for delivering LLM services to users [1]. In a typical edge LLM inference system, users upload their prompts to an edge server, which performs inference using the hosted LLM and returns the generated responses [2]. By placing LLMs on edge servers closer to end users, edge LLM inference can reduce transmission latency, compared with cloud-based serving [3], [4]. As such, edge LLM inference is becoming a key enabler for real-time LLM applications [5]. Autoregressive decoding (AD) and speculative decoding (SD) are two representative approaches in LLM inference. Guanqiao Qu, Qian Chen, and Xianhao Chen are with the Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China (e-mail: gqqu@eee.hku.hk; qchen@eee.hku.hk; xcheneee@hku.hk). Shuo Chen is with the Electrical and Electronic Engineering Department, Imperial College, London SW7 2BT, U.K. (e-mail: shuo.chen22@imperial.ac.uk). Kin K. Leung is with the Electrical and Electronic Engineering, and Computing Departments, Imperial College, London SW7 2BT, U.K. (e-mail: kin.leung@imperial.ac.uk). The work was supported in part by the Research Grants Council of Hong Kong under Grant 27213824, Grant 17207826, and Grant CRS HKU702/24. Kin K. Leung is supported by the EPSRC grant EP/Y037243/1. (Corresponding author: Xianhao Chen.)

AD SD 40.3%

5 4 39.8%

3 2 1 0

37.2% 200

600

Output length (tokens)

1000

Required GPU memory (GB)

7 6

End-to-end latency (s)

Abstract—Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output tokens sequentially, resulting in long latency; Speculative decoding (SD) accelerates inference by using a small language model (SLM) to generate multiple draft tokens for LLM verification, but incurs extra memory costs. Due to this latency-memory tradeoff, neither approach alone can efficiently serve users with heterogeneous demands under limited edge computing resources. To address this challenge, we propose a hybrid autoregressive-speculative inference (BALANCE) framework for edge LLM inference. In BALANCE, an edge server hosts both an SLM and an LLM, assigns each user to AD or SD, and performs the two modes simultaneously. To maximize the number of served users, we formulate a task throughput maximization problem to jointly determine user scheduling and computing resource allocation between AD and SD under user latency requirements and server memory constraints. Since the problem is NP-hard, we develop a polynomial-time algorithm that transforms the original problem into two sub-problems and obtains a sub-optimal solution with a constant approximation guarantee. Experiments demonstrate that BALANCE consistently outperforms conventional AD and SD and significantly improves task throughput.

AD SD

11 18.0%

18.3%

18.6%

200

600

1000

Output length (tokens)

(a) End-to-end latency vs. out(b) Required GPU memory vs. put length. output length. Fig. 1. Comparison between AD and SD when serving a single user using an NVIDIA GeForce RTX 4090 GPU, where the output length varies, and the input length is fixed at 500 tokens. In this experiment, the end-to-end latency denotes the time from feeding the prompt to the model to completing output generation. Llama-2-7B is used as the LLM in both AD and SD, and TinyLlama-1.1B is used as the SLM in SD.

In AD, an LLM generates the response token by token, producing one new token in each forward pass based on the input and previously generated tokens [6], [7]. However, the sequential decoding process of AD incurs high response latency [8]. This issue becomes more severe for tasks with long output responses and stringent latency requirements [8]. To reduce decoding latency, SD has been proposed [9]. In SD, a small language model (SLM) first generates multiple draft tokens via AD, and the target LLM verifies them in parallel within a single forward pass. By accepting multiple verified draft tokens at once, SD reduces the number of serial LLM forward passes. This reduction can significantly reduce response latency, since each forward pass of the SLM is typically faster than the target LLM [10]. Meanwhile, SD maintains output quality, since the verification preserves the target LLM’s output distribution [11]. When applying SD to edge LLM inference, the SLM and LLM can be co-deployed on the edge server to serve users [12]. However, despite its latency advantage, SD introduces higher memory overhead due to the coexistence of the SLM and LLM as well as the memory for parallel draft verification [13]. Challenge. Neither AD nor SD alone can efficiently serve users with heterogeneous demands on resource-constrained edge servers. Serving users with AD alone is memory-efficient but may fail to satisfy stringent latency requirements, whereas relying solely on SD reduces latency but may exceed available memory resources [14], [15]. As illustrated in Fig. 1, when serving a single user, SD reduces the average end-to-end (E2E) latency by 39.1% compared with AD, but increases the GPU memory consumption by 18.3%. This latency-memory tradeoff raises a key challenge: how to efficiently serve users with heterogeneous demands by jointly exploiting AD and SD under

constrained computing and memory resources on edge servers. Solution. To address the above challenge, we propose a hybrid autoregressive-speculative inference (BALANCE) framework. In BALANCE, an edge server hosts both an LLM and an SLM, enabling the simultaneous support of AD and SD for different users. To serve users with heterogeneous demands under limited computing resources, the edge server jointly selects users for service, assigns each selected user to AD or SD, and allocates computing resources between the two modes. Based on this framework, we formulate a joint user scheduling and computing resource allocation problem to maximize the number of served users in edge LLM inference while satisfying users’ E2E latency requirements and the server’s memory constraint. This problem is combinatorial because different assignments of users with heterogeneous demands between AD and SD incur different latency and memory costs, and we show that it is NP-hard. To address this problem, we further design a polynomial-time algorithm with a constant approximation guarantee. To the best of our knowledge, this is the first work to study hybrid autoregressive-speculative inference for edge LLM inference. The main contributions are as follows. 1) We propose a hybrid autoregressive-speculative inference framework where an edge server simultaneously supports AD and SD for multiple users. We formulate a joint user scheduling and computing resource allocation problem to maximize the number of served users under E2E latency requirements and resource constraints. We show that this problem is NP-hard. 2) To facilitate efficient solution finding, we decompose the original problem into AD and SD sub-problems and design a polynomial-time algorithm with a constant approximation guarantee. 3) Experimental results show that the proposed BALANCE framework significantly improves the system performance compared with conventional AD and SD. The rest of this paper is structured as follows. Section II reviews the related work. Section III proposes the BALANCE framework and formulates the task throughput maximization problem. Section IV designs the algorithm and provides the theoretical analysis. Section V provides the experiment results, and Section VI concludes the paper. II. R ELATED W ORK Existing edge LLM inference systems commonly adopt AD to provide services to end users. To improve system throughput, prior studies have investigated batching techniques that group users requesting the same LLM into one inference batch [16]. Moreover, user scheduling and resource allocation for multi-user edge inference systems under heterogeneous user demands and constrained server resources have also been studied [17], [18]. Some works further explore collaborative inference, where an LLM is partitioned across end devices, edge servers, and cloud servers to exploit distributed computing resources [19]. These studies provide important system and scheduling foundations for edge LLM inference under AD.

To reduce response latency, SD has been introduced into edge LLM inference. Similar to AD, existing studies investigate batching, user scheduling, and resource allocation for SD-based edge inference [20], [21], as well as collaborative SD at the network edge, where the SLM and the LLM are hierarchically deployed across end devices, edge servers, and cloud servers [22]. To further improve the efficiency of SD, some techniques, such as tree-structured draft generation [23] and parallel draft generation and verification [24], have been proposed. Together, these studies advance the efficient deployment of SD in edge networks. Most existing works focus on either AD or SD alone. Although several studies have explored hybrid LLM inference, they mainly consider distributed deployment of SLMs and LLMs [25], request routing between SLMs and LLMs [26], and selective LLM involvement during SLM token generation [27]. These hybrid inference methods coordinate SLMs and LLMs rather than jointly supporting AD and SD. Moreover, some works combine different decoding techniques, such as AD for early tokens followed by parallel decoding for subsequent tokens [28] and retrieval-based draft generation within SD [29]. However, such combinations represent a new inference paradigm rather than joint coordination of AD and SD under resource constraints. Therefore, user scheduling and resource allocation for hybrid autoregressive-speculative inference remain largely unexplored in edge LLM inference. III. F RAMEWORK A. Network Scenario We consider a multi-user wireless edge network with a single edge server covering a set of users K, as shown in Fig. 2. Each user k ∈ K generates an inference task with an E2E latency requirement Tk and requests the edge server to perform inference by uploading the task data to the edge server. The edge server hosts both an LLM and an SLM and simultaneously supports the AD mode, where the LLM generates output tokens autoregressively, and the SD mode, where the SLM generates draft tokens and the LLM verifies them. To serve heterogeneous user demands, the edge server selects users and schedules each selected user to AD or SD mode. To indicate the user scheduling, we define binary variables xk and yk , where xk = 1 if and only if user k is scheduled to the AD mode, and yk = 1 if and only if user k is scheduled to the SD mode. xk = yk = 0 indicates that user k is not served. To improve task throughput, users scheduled to the same mode are batched for inference. After inference, the generated outputs are returned to the users. B. BALANCE Framework The procedure of BALANCE is shown in Fig. 3. Before task execution, each user uploads its input length LIk and expected output length LO k , measured in tokens, to the edge server through a control signal1 . Based on the reported information, 1 Reporting input and expected output lengths is commonly adopted in scheduling for LLM inference [30]. LO k can be specified by the task requirements or obtained by efficient prediction methods, such as lightweight predictors [31] and distribution-based prediction [32].

γLO

Here, ∆τkdown = B̄k R̄kk , where B̄k and R̄k denote the downlink bandwidth and spectral efficiency of user k, respectively.

LLM Token decoding for AD and draft verification for SD

SLM Draft generation for SD

Draft tokens Verification results

Batched inputs for the SD mode

Batched inputs for the AD mode

Edge server

Users scheduled to the AD mode

Users scheduled to the SD mode

SD mode Input uploading Output downloading

AD mode Input uploading Output downloading

Fig. 2. Illustration of BALANCE, where an edge server simultaneously supports the AD and SD modes for different users. The blue and orange lines denote the data flows of the SD and AD modes, respectively. Procedure of the proposed BALANCE framework (i): Input uploading for AD τad,up User 1 … User k1

Input uploading Δτ1up … Input uploading Δτup

(ii-a-i): LLM prefilling for the AD mode τad,p

AD decoding step 1 τ1ad,a

…

AD decoding step l τlad,a

…

AD decoding (iii): Output ad,O − 1 downloading step Lmax

Timeline of the AD mode

(ii-a-ii) (i): Input uploading for SD τsd,up User k2 …

Input uploading Δτup k2

…

up User K Input uploading ΔτK

SD iteration 1

SD iteration n

sd SD iteration Nmax

SLM draft LLM draft LLM prefilling for SLM prefilling for SLM draft LLM draft generation verification … generation verification … SLM draft LLM draft (iii): Output the SD mode the SD mode generation verification downloading τsd,v τsd,a τsd,L,p τsd,S,p τsd,a τsd,v 1

(ii-s-i)

(ii-s-ii)

Timeline of the SD mode

Fig. 3. The procedure of the proposed BALANCE framework.

the edge server determines the user scheduling and sends the scheduling results to the users. The scheduling problem will be formulated in Section III-G. (i) After receiving the scheduling results, the scheduled users upload their input prompts. Once the prompts of users scheduled for the same inference mode are received, the server processes them in a batch using that mode. (ii-a-i) In the AD mode, the prompts for AD are first prefilled by the LLM. (ii-a-ii) Then, the LLM generates the output tokens through multiple decoding steps. (ii-s-i) In the SD mode, the prompts for SD are first prefilled by the LLM and then by the SLM. (ii-s-ii) Then, the SLM and LLM generate output tokens over multiple iterations. (iii) Once one mode is completed, the corresponding outputs are returned to the users scheduled to that mode. C. Communication Model We consider orthogonal frequency-division multiple access for both uplink and downlink transmissions. The data uploadγLI ing time of user k is ∆τkup = Bk Rkk , where Bk and Rk denote the uplink bandwidth and uplink spectral efficiency of user k, respectively. γ is the number of bits per token, and γLIk is the bit size of user k’s input prompt tokens. Thus, the data uploading times of the AD and SD modes are given by τ ad,up = max ∆τkup xk = 1 , (1) and τ sd,up = max ∆τkup

D. Computing Model We model the computing procedures and workloads using a common Transformer architecture. Following previous works [33], we focus on matrix multiplication operations in workload analysis, including query, key, and value (QKV) projections, multi-head attention (MHA) computation, output projection, and feed-forward computation using a feed-forward network (FFN), while ignoring minor contributors. 1) Computing procedures and workload analysis of the AD mode: After all user input prompts of the AD mode arrive, the edge server starts the AD mode. These prompts are batched and prefilled by the LLM in one forward pass, which generates the first output token for each request. We define the LLM workload for prefilling a single input prompt of LI tokens as ΓL,p LiI = h 2 M L 8LI DL,m DL,h H L + 4 LI DL,h H L + 4LI DL,m DL,f , where DL,m , DL,h , H L , DL,f , and M L denote the LLM hidden dimension, the hidden dimension of an attention head, the number of attention heads, the FFN hidden dimension, and the number of layers of the LLM, respectively. The three terms inside the bracket represent the workloads of the QKV and output projections, attention computation, and feed-forward computation in each layer, respectively. Thus, the LLM prefilling workload in the AD mode is given by X W ad,p = ΓL,p Lad,I xk , max (4) k∈K

xk is the number of users scheduled to the AD I mode, and Lad,I xk = 1 is the maximum input max = max Lk where

k∈K

length of users scheduled to the AD mode2 . After prefilling, the LLM generates one token for each unfinished user request at each decoding step. Since prefilling has generated the first token, user k requires LO k − 1 decoding steps for output generation. Therefore, the total number of decoding steps in the AD mode is Lad,O max − 1, where O ad,O Lmax = max Lk xk = 1 (5) k

is the maximum output length of users in the AD mode. We define the LLM workload for generating one token at the l-th decoding step for I = a single prompt of L tokens as ΓL,a LI l L,minput L,h L,h L I L L,m L,f L M 8D D H + 4 L + l D H + 4D D , where the attention workload depends on the sequence length LI + l, since the query of the current token attends to the KV cache of the previous LI + l − 1 tokens as well as the key and value of the current token. Thus, the LLM workload at the l-th decoding step in the AD mode is given by X Wlad,a = xk I{l≤LO −1} ΓL,a Lad,I max , l (6) k k∈K

yk = 1 ,

(2)

respectively. The output downloading time of user k is τkdown = (xk + yk ) ∆τkdown .

(3)

where users with l > LO k − 1 are excluded at decoding step l. 2 Following previous works [34], we consider padding for efficient GPU matrix multiplication, as supported and recommended by PyTorch [35].

2) Computing procedures and workload analysis of the SD mode: After all input prompts of users scheduled to the SD mode are received, the edge server starts the SD mode. These prompts are batched, first prefilled by the LLM, and then prefilled by the SLM. Following (4), the LLM prefilling workload in the SD mode is given by X W sd,L,p = ΓL,P Lsd,I yk , max (7) k∈K

yk is the number of users scheduled to the SD I mode, and Lsd,I yk = 1 is the maximum max = max Lk k∈K input length of users scheduled to the SD mode. Additionally, following ΓL,p LI , we define the SLM workload for I prefilling a single input prompt of LI tokens as ΓS,p L h i = 2 M S 8LI DS,m DS,h H S + 4 LI DS,h H S + 4LI DS,m DS,f , where DS,m , DS,h , H S , DS,f , and M S denote the SLM hidden dimension, the hidden dimension of an attention head, the number of attention heads, the FFN hidden dimension, and the number of layers of the SLM, respectively. Thus, the SLM prefilling workload in the SD mode is expressed as X W sd,S,p = ΓS,p Lsd,I yk . max (8) where

k∈K

After prefilling, SD proceeds over multiple iterations. In each iteration, the SLM autoregressively generates draft tokens for unfinished requests, and the LLM verifies the batched draft sequences in one forward pass. After verification, the accepted draft tokens and one additional token are retained for each request, and the KV cache of rejected draft tokens is removed. The additional token is used as the input token for the next draft generation. Let Ak denote the expected acceptance rate that a draft token is accepted by the LLM for user k’s task. The number of accepted draft tokens for user k’s task is h i LD

Ak 1−(Ak )

LO k LA k +1

. User k requires Nk = ⌈ given by LA ⌉ k = 1−Ak SD iterations, where term 1 represents the additional token. Therefore, the total number of SD iterations is denoted by sd Nmax = max {Nk | yk = 1} . k∈K

(9)

where users with n > Nk are excluded in iteration n. We define the LLM workload for verifying LD tokens in iteration n for a single input prompt, given LPn prefix tokens available at n, as ΓL,v LPn , LD = M L# n " the beginning of iteration D L P P 8LD DL,m DL,h H L + 4 Ln + l DL,h H L + 4LD DL,m DL,f . l=1

Thus, the LLM workload for verification in iteration n is P D  yk I{n≤Nk } ΓL,v Lsd,P , n=1  1 1 ,L Wnsd,v = k∈K P D  yk I{n≤Nk } ΓL,v Lsd,P  n n , L + 1 , n ≥ 2. k∈K

(12) In the first iteration, the LLM verifies LD draft tokens for each request. When n ≥ 2, the verification further includes the additional token generated in the previous iteration. E. Memory Model Following previous works [36], we consider the memory required for storing model weights and the KV cache during inference when analyzing memory consumption. 1) Memory for hosting the LLM and SLM: The GPU memory required to store the LLM and SLM weights is q L = M L θ 4DL,m DL,h H L + 2DL,m DL,f , (13) and q S = M S θ 4DS,m DS,h H S + 2DS,m DS,f ,

(14)

respectively, where θ denotes the number of bytes required to represent one floating-point number [36]. 2) Memory for the AD mode: In the AD mode, the ad,p required = KV cache memory for prefilling is q P L ad,I L,h L L,h L M θ xk Lmax 2D H , where 2D H denotes the k∈K

number of KV cache elements required by one token across all attention heads in each layer. During decodad,a ing, the = peak KV cache memory is denoted by q P O xk Lad,I 2DL,h H L . Therefore, the peak M Lθ max + Lk k∈K

KV cache memory in the AD mode is given by q ad = max q ad,p , q ad,a = q ad,a .

(15) Moreover, let Lsd,P be the prefix length of the retained tokens n before draft generation in iteration n, which is given by 3) Memory for the SD mode: In the SD mode, the required  sd,I KV cache memory for prefilling is given by q sd,p = q sd,L,p + Lmax , n = 1, A A sd,S,p sd,I Lsd,P = , where q sd,L,p and q sd,S,p denote the KV cache memory q Lk + 1 + max Lk n Lmax + (n − 2) max , n ≥ 2. k∈K k∈K required by the LLM and SLM during prefilling, yk =1 yk =1 respectively. P (10) L,h L yk Lsd,I Following q ad,p , q sd,L,p = M L θ max 2D H and For n = 1, the retained prefix only consists of the input prompt k∈K P tokens. For n ≥ 2, it further includes the accepted draft tokens S,h S ad,a q sd,S,p = M S θ yk Lsd,I , the max 2D H . Following q and additional tokens retained from previous iterations. k∈K L,a I Following Γl L , we define the SLM workload for peak SLM KV cache memory during draft generation is generating one token for a single input prompt at the given by q sd,a = M S θ P y Lsd,P + LD 2DS,h H S . Durk Nk l-th decoding step of iteration n, given LPn prefix tokens k∈K S,a P the peak LLM KV cache is given by available at the beginning of iteration n, as Γn,l Ln = ing verification, P sd,v L D M S 8DS,m DS,h H S + 4 LPn + l DS,h H S + 4DS,m DS,f . q =M θ yk Lsd,P + L + 1 2DL,h H L . Therefore, Nk k∈K Thus, the SLM workload at decoding step l of iteration n is the peak KV cache memory in the SD mode is X sd,a sd,P yk I{n≤Nk } ΓS,a L , Wn,l = n n,l (11) q sd = max q sd,p , q sd,a + q sd,v = q sd,a + q sd,v . (16) k∈K

F. User E2E latency To model the inference latency under shared GPU computing resources, we consider the computing resource allocation between the AD and SD modes. Let z denote the fraction of GPU computing resources allocated to the AD mode, and thus 1 − z is allocated to the SD mode3 . 1) Latency of the AD mode: Following previous studies [39], the LLM prefilling time in the AD mode is τ ad,p = αL W ad,p + β L , where αL and β L are model- and hardwarez dependent constants measured offline. The time of decoding αL W ad,a step l in the AD mode is τlad,a = z l + β L . Thus, the time from prefilling to the completion of step l in the AD mode is τlad = τ ad,p +

l X

τlad,a . ′

(17)

l′ =1

2) Latency of the SD mode: The prefilling times of the LLM and the SLM in the SD mode are given by τ sd,L,p = S W sd,S,p αL W sd,L,p + β L and τ sd,S,p = α 1−z + β S , respectively. For 1−z iteration n, the time required by the SLM to generate LD draft tokens for all users scheduled to the SD mode is  D LP −1 αS W sd,a   n,l S  , n = 1,  1−z + β (18) τnsd,a = Ll=1 D sd,a  P αS Wn,l  S sd  , 2 ≤ n ≤ Nmax ,  1−z + β l=1

where the first iteration requires only L − 1 steps due to the prefilling. Moreover, the LLM verification time in iteration αL W sd,v n is denoted by τnsd,v = 1−zn + β L . Thus, the time from prefilling to the completion of iteration n in the SD mode is n X τnsd = τ sd,L,p + τ sd,S,p + + τnsd,v . (19) τnsd,a ′ ′

yk ∈ {0, 1} , ∀k ∈ K,

(21f)

z ∈ Z.

(21g)

Here, xk ∈ X, yk ∈ Y, and z are the decision variables for AD-mode user scheduling, SD-mode user scheduling, and GPU computing resource allocation, respectively. In (21a), X + Y denotes the element-wise sum of X and Y. Constraint (21b) assigns each user to at most one inference mode. Constraints (21c) and (21d) enforce the user latency and GPU memory capacity constraints, respectively. Here, Q is the available GPU memory capacity, and the indicator function I{z<1} equals 1 if z < 1 and 0 otherwise. When z = 1, the SD mode is inactive, so the SLM weights are not loaded into GPU memory. Constraints (21e) and (21f) are binary constraints. Constraint (21g) restricts z to Z = {0, z1 , . . . , zI , 1}, where 0 < z1 < · · · < zI < 1. The values in Z represent the GPU computing resource allocation ratios supported by the system. IV. A LGORITHM D ESIGN This section proves the NP-hardness of P1 and develops a polynomial-time constant-approximation algorithm. A. Problem Mapping The NP-hardness of P1 is stated below. Proposition 1. P1 is an NP-hard problem. Proof. The proof sketch is as follows. Consider a special case of P1 with fixed z. The problem then reduces to user scheduling under latency requirements and memory resource constraints, which can be mapped to a multi-dimensional multiplechoice knapsack problem (MMKP). Since the MMKP is NPhard [40], [41], P1 is NP-hard, completing the proof.

n′ =1

3) E2E latency: Based on (1), (2), (3), (17), and (19), the E2E latency of user k is expressed as sd tEk = xk τ ad,up + τLadad,O −1 + yk τ sd,up + τN + τkdown . sd max max (20) G. Problem Formulation To maximize the task throughput of BALANCE, i.e., the number of served users satisfying their latency requirements, we formulate the following optimization problem. X P1 : max r (X + Y) = xk + y k (21a) X,Y,z

k∈K

s.t. xk + yk ≤ 1, ∀k ∈ K, tEk ≤ Tk ,

∀k ∈ K,

(21b) (21c)

I{z<1} q S + q L + q ad + q sd ≤ Q,

(21d)

xk ∈ {0, 1} , ∀k ∈ K,

(21e)

3 In practice, GPU computing resource allocation can be implemented by partitioning GPU resources, e.g., streaming multiprocessors and GPU engines, across multiple tasks, which has already been supported by NVIDIA [37]. For simplicity, we refer to it as “GPU computing resource allocation” in this paper. Such allocation has been adopted in existing studies [38].

B. Equivalent Problem Transformation To facilitate the solution design, we introduce an auxiliary variable λ to represent the fraction of available GPU memory allocated to the AD mode. Since the memory consumption of the AD and SD modes can be characterized separately, P1 is equivalently transformed into the following P2. X P2 : max r (X + Y) = xk + yk (22a) X,Y,z,λ

k∈K

s.t. (21b), (21c), (21e), (21f), (21g), ad

(22b)

≤ Q1 ,

(22c)

q sd ≤ Q2 ,

(22d)

λ ∈ Λ.

(22e) S

Here, Q1 = λ Q −q L − I{z<1} q and Q2 = L (1 − λ) Q − q − I{z<1} q S denote the memory reserved for the AD and SD modes, respectively. Constraint (22e) restricts λ to Λ = {0, λ1 , . . . , λJ , 1}, where 0 < λ1 < · · · < λJ < 1. The set Λ contains all supported ratios for partitioning GPU memory between the AD and SD modes, ensuring that every feasible memory split can be represented by some λ ∈ Λ. Next, we state that solving P2 is equivalent to solving P1.

Proposition 2. Solving P1 is equivalent to solving P2.

D. Solution to P3.1: User Scheduling for the AD Mode

Proof. After reserving memory for the model weights, Q1 and Q2 can be obtained according to λ. By the construction of Λ, any feasible solution to P1 corresponds to a feasible solution to P2. Conversely, any feasible solution to P2 satisfies the memory constraint in P1 since Q1 and Q2 are given. Thus, P1 and P2 are equivalent, completing the proof.

We obtain an optimal solution to P3.1 by enumerating the ad,O ad ad,up feasible values of Lad,I , and identifying max , Lmax , t , and τ the served users under each setting. When z = 0 or λ = 0, the AD mode is inactive, and thus the optimal solution to P3.1 is X̂ = 0. The following analysis considers z > 0 and λ > 0. ad,O ad First, given Lad,I = t1 , and max = L1 , Lmax = L2 , t ad,up τ = τ , the candidate user set for the AD mode is 1 up ′ K1 = k k ∈ K, LIk ≤ L1 , LO ≤ L , t ≤ T , ∆τ 2 1 k k k ≤ τ1 . For each candidate user k ∈ K1 , its latency contribution is

To solve P2, we fix z and λ, and have the below proposition. Proposition 3. Given z and λ, solving P2 is equivalent to solving P3.

P3 : max r (X + Y) =

xk + y k

(23a)

s.t. (21b), (21c), (21e), (21f), (22c), (22d).

(23b)

X,Y

k∈K

Proof. For fixed z and λ, constraints (21g) and (22e) can be removed, reducing P2 to P3 and completing the proof.

ad,a LX 2 −1 αL W̃l,k αL W̃kad,p ad + . t̃k = z z l=1

ad,a Here, W̃kad,p and W̃l,k denote the contributions of user k to the LLM workloads during prefilling and at the l-th decoding step in the AD mode, respectively, where W̃kad,p = ΓL,p (L1 ) ad,a and W̃l,k = I{l≤LO −1} ΓL,a l (L1 ). Moreover, the contribution k of user k ∈ K1 to the total memory consumption of the AD mode is given by

q̃kad = q̃ ad = M L θ (L1 + L2 ) 2DL,h H L . C. Problem Decomposition Given the NP-hardness of P3, we decouple P3 into the following two sub-problems, P3.1 and P3.2, which schedule users in the AD and SD modes, respectively. We have X P3.1 : max r (X) = xk (24a) X

k∈K

(26)

(27)

Second, let Π1 = {1, . . . , d1 . . . , |K1 |} denote a permutation of the users in K1 sorted in a non-decreasing order of ad ad t̃ad k , i.e., t̃Π1 (d1 ) ≤ t̃Π1 (d1 +1) . Therefore, the number of served users under L1 , L2 , t1 , and τ1 is given by   d1   X Q 1 ≤ ∆t , (28) t̃ad d∗1 = max d1 d1 ≤ ⌊ ad ⌋, 1 ′ Π1 (d1 )   q̃ ′ d1 =1

s.t. (21e), (22c)

(24b)

xk tad ≤ Tk′ , ∀k ∈ K,

(24c)

where tad = τ ad,up + τLadad,O −1 denotes the latency of the AD max mode excluding output downloading, and Tk′ = Tk − ∆τkdown denotes the remaining latency budget of user k after reserving the output downloading latency. Also, we have X P3.2 : max r′ (Y) = yk (25a) Y

k∈K\Kad

s.t. (22d),

(25b)

yk tsd ≤ Tk′ , ∀k ∈ K \ Kad ,

(25c)

yk ∈ {0, 1} , ∀k ∈ K \ Kad ,

(25d)

sd where tsd = τ sd,up +τN denotes the latency of the SD mode sd max excluding output downloading, and Kad = {k | xk = 1} is the set of users served by the AD mode. The following subsections will sequentially solve P3.1 and P3.2 to determine the optimal user scheduling for the AD and SD modes, respectively. The performance gap between the sequential solution and the optimal solution to P1 will be analyzed in Section IV-F.

where ∆t1 = t1 − τ1 − ∆τ1 and ∆τ1 = L2 β L denotes the fixed latency term of one forward pass in prefilling and L2 − 1 forward passes in decoding. Based on (28), the served user set for the AD mode under L1 , L2 , t1 , and τ1 is K̂ad = {Π1 (1) , . . . , Π1 (d∗1 )}, and the corresponding user scheduling decision x̂k ∈ X̂ is ( 1, if k ∈ K̂ad , x̂k = (29) 0, otherwise. By enumerating all feasible L1 , L2 , t1 , and τ1 , we can obtain the maximum number of served users and the corresponding user scheduling decision under given z and λ for P3.1. Algorithm 1 summarizes the above procedures, and its optimality is established in the following proposition. Proposition 4. Algorithm 1 obtains an optimal solution to P3.1 under given z and λ. Proof. The proof sketch is below. For fixed z and λ, every feasible solution to P3.1 corresponds to a tuple (L1 , L2 , t1 , τ1 ). For each tuple, all candidate users in K1 have unit profit, identical memory contributions, and fixed latency contributions. Thus, (29) leads to the optimal user scheduling for this tuple. Since Algorithm 1 enumerates all feasible tuples, it obtains an optimal solution to P3.1, completing the proof.

Algorithm 1: AD-Mode User Scheduling Algorithm Input: z and λ. Output: X̂ and K̂ad . ad 1 Initialize: r1 = 0, X̂ = 0, K̂ = ∅. 2 if z = 0 or λ = 0 then 3 Return X̂ = 0 and K̂ad = ∅. 4 end S I 5 for L1 ∈ {0} S OLk k ∈ K , S L2 ∈ {0} LkSk ∈ K , t1 ∈ {0} {Tk′ |k ∈ K} do up 6 for τ1 ∈ {0} {∆τk | k ∈ K} do 7 Calculate d∗1 with (28). 8 if d∗1 > r1 then 9 r1 = d∗1 and K̂ad = {Π1 (1) , . . . , Π1 (d∗1 )}. 10 end 11 end 12 end 13 Calculate X̂ with (29).

Algorithm 2: SD-Mode User Scheduling Algorithm Input: z, λ, and K̂ad . Output: Ŷ and K̂sd . ′ ad sd 1 Initialize: K = K \ K̂ , r2 = 0, Ŷ = 0, K̂ = ∅. 2 if z = 1 or λ = 1 then 3 Return Ŷ = 0 and K̂sd = ∅. 4 end S I 5 for L3 ∈ {0} L k ∈ K′ , S A k S L4 ∈ {0}S Lk k ∈ K′ , Ñ1 ∈ {0} {Nk |k ∈ K′ }, t2 ∈ {0} {Tk′ S | k ∈ K′ } do 6 for τ2 ∈ {0} {∆τkup | k ∈ K′ } do 7 Calculate d∗2 with (32). 8 if d∗2 > r2 then 9 r2 = d∗2 and K̂sd = {Π2 (1) , . . . , Π2 (d∗2 )}. 10 end 11 end 12 end 13 Calculate Ŷ with (33).

E. Solution to P3.2: User Scheduling for the SD Mode We solve P3.2 similarly by enumerating the feasible values sd sd,up sd sd,A , and identifying the of Lsd,I max , Lmax , Nmax , t , and τ served users under each setting. When z = 1 or λ = 1, the SD mode is inactive, and thus the optimal solution to P3.2 is Ŷ = 0. The following analysis considers z < 1 and λ < 1. sd,A sd sd First, given Lsd,I max = L3 , Lmax = L4 , Nmax = Ñ1 , t = t2 , sd,up and = τ2 , the candidate user set for the SD mode is K2 = n τ up ′ k k ∈ K′ , LIk ≤ L3 , LA k ≤ L4 , Nk ≤ Ñ1 , t2 ≤ Tk , ∆τk ≤ τ2 }, where K′ = K \ Kad . The contribution of each candidate user k ∈ K2 to the total latency of the SD mode is αL W̃ksd,L,p αS W̃ksd,S,p + 1−z 1−z D sd,a sd,v sd,a Ñ1 X Ñ1 LX −1 S LD X α W̃1,l,k X αL W̃n,k αS W̃n,l,k + + + . 1−z 1−z 1−z n=2 l=1 n=1 l=1 (30) Here, W̃ksd,L,p and W̃ksd,S,p are the contributions of user k to the total LLM and SLM workloads during prefilling in the SD mode, respectively, where W̃ksd,L,p = ΓL,p (L3 ) and sd,a sd,v W̃ksd,S,p = ΓS,p (L3 ). Additionally, W̃n,l,k and W̃n,k denote the contributions of user k to the total SLM workload at the l-th decoding step of iteration n and the total LLM workload during verification in iteration n, respectively. Let L̃sd,P denote the prefix length of the retained tokens before n draft generation in the n-th iteration under L3 and L4 , where sd,P L̃sd,P + n = L3 when n = 1, and L̃n = L3 + (n − 2) (L4+ 1) sd,a S,a sd,P L4 when n ≥ 2. Thus, W̃n,l,k = I{n≤Nk } Γl L̃n , sd,v sd,v D W̃n,k = I{n≤Nk } ΓL,v L̃sd,P when n = 1, and W̃n,k = 1 1 ,L D I{n≤Nk } ΓL,v L̃sd,P when n ≥ 2. Moreover, the n n ,L + 1 contribution of user k to the total memory consumption of the SD mode is given by t̃sd k =

q̃ksd = q̃ksd,a + q̃ksd,v . (31) D Here, q̃ksd,a = M S θ L̃sd,P 2DS,h H S denotes the contriNk + L

bution of user k to the total memory consumption during SLM draft generation. q̃ksd,v denotes the contribution of user k to the total memory consumption during LLM draft verification, D L,h L where q̃ksd,v = M L θ L̃sd,P + L 2D H when Nk = 1, and Nk sd,P sd,v q̃k = M L θ L̃Nk + LD + 1 2DL,h H L when Nk ≥ 2. Second, let Π2 = {1, . . . , d2 . . . , |K2 |} denote a permutation of the users in K2 sorted in a non-decreasing order sd sd sd sd of t̃sd k , i.e., t̃Π2 (d2 ) ≤ t̃Π2 (d2 +1) . Since both t̃k and q̃k are monotonically non-decreasing with Nk , sorting users by t̃sd k also preserves the non-decreasing order of q̃ksd . Therefore, the number of served users under L3 , L4 , Ñ1 , t2 , and τ2 is   d2 d2   X X sd sd d∗2 = max d2 q̃Π t̃ ≤ ∆t , ≤ Q , 2 2 ′ ′ Π2 (d2 ) 2 (d2 )   ′ ′ d2 =1

d2 =1

where ∆t2 = t2 − τ2 − ∆τ2 and ∆τ2 =

(32) Ñ1 + 1 β L +

Ñ1 LD β S denotes the fixed latency term in the SD mode. Based on (32), the served user set of the SD mode under L3 , L4 , Ñ1 , t2 , and τ2 is K̂sd = {Π2 (1) , . . . , Π2 (d∗2 )}, and the corresponding user scheduling decision ŷk ∈ Ŷ is ( 1, if k ∈ K̂sd , ŷk = (33) 0, otherwise. By enumerating all feasible L3 , L4 , Ñ1 , t2 , and τ2 , we can identify the maximum number of served users and the corresponding user scheduling decision under given z and λ for P3.2. Algorithm 2 summarizes the above procedures, and its optimality is established in the following proposition. Proposition 5. Algorithm 2 obtains an optimal solution to P3.2 under given z, λ, and K̂ad . Proof. The proof is similar to that of Proposition 4. Since t̃sd k and q̃ksd are non-decreasing in Nk , (33) maximizes the number of served users for each tuple. Enumerating all feasible tuples yields an optimal solution to P3.2, completing the proof.

Algorithm 3: BALANCE User Scheduling Algorithm Input: K, Q, and Tk . Output: Ẋ, Ẏ, ż, and λ̇. 1 Initialize: r3 = 0, Ẋ = 0, Ẏ = 0, ż = 0, λ̇ = 0. 2 for z ∈ Z, λ ∈ Λ do 3 Obtain X̂ and K̂ad using Algorithm 1 with z and λ. 4 Obtain Ŷ and K̂sd using Algorithm 2 with z, λ, and K̂ad . 5 if K̂ad + K̂sd > r3 then 6

r3 = K̂ad + K̂sd , Ẋ = X̂, Ẏ = Ŷ, ż = z,

Proof. The proof sketch is as follows. Algorithm 1 enumerates O K 4 tuples (L1 , L2 , t1 , τ1 ). For each tuple, constructing Π1 and computing d∗1 in Line 7 require O (K log K) time. Therefore, the time complexity of Algorithm 1 is O K 5 log K . Similarly, the time complexity of Algorithm 2 is O K 6 log K . Algorithm 3 enumerates O (|Z| |Λ|) pairs (z, λ), where |Z| and |Λ| denote the cardinalities of Z and Λ, respectively. For each pair, Lines 3 and 4 require O K 5 log K and O K 6 log K time, respectively. Since |Z| and |Λ| are constants, Algorithm 3 has time complexity O K 6 log K , completing the proof.

and λ̇ = λ.

V. N UMERICAL R ESULTS This section evaluates the performance of BALANCE under different system settings, compares the algorithm running time, and conducts ablation studies.

F. Algorithm Outline

A. Experimental Setup We adopt a hybrid testing method that combines device measurements and numerical simulations to evaluate the performance of the proposed BALANCE framework. Specifically, computing latencies are measured on a real server, while the communication latencies in multi-user scenarios are obtained through simulations. The inference is performed on an edge server using an AMD Ryzen Threadripper PRO 5975WX CPU, an NVIDIA GeForce RTX 4090 GPU with 24 GB of GPU memory, and 256 GB of DDR4 RAM. TinyLlama-1.1B and Llama-2-7B are used as the SLM and LLM, respectively, with FP16 precision. Accordingly, Q = 24 GB. The coverage radius of the edge server is set to 250 m. The uplink and downlink bandwidths allocated to user k are set to Bk = 1 MHz and B̄k = 2 MHz, respectively. The transmit power spectral densities of the users for uplink transmission and of the edge server for downlink transmission are set to -50 dBm/Hz and -29 dBm/Hz, respectively. The number of users K ranges from 20 to 70. The user latency requirement satisfies Tk ∈ [Tmin , Tmax ], ranging from 0.5 s to 13 s. The input I I I length length LO k and output k are set within Lmin , Lmax L O O and Lmin , Lmax , respectively, both ranging from 1 to 1000 tokens. For SD, the draft token length LD is set to 1, and the acceptance rate Ak is set within [0.6, 0.8]. We compare BALANCE with the following baselines. • AD-only [17]. In this baseline, all users are served only by the AD mode, and user scheduling is determined by Algorithm 1. By Proposition 4, this baseline achieves the maximum task throughput under the AD-only mode. • SD-only [20]. In this baseline, all users are served only by the SD mode. User scheduling is determined by Algorithm 2. By Proposition 5, this baseline achieves the maximum task throughput under the SD-only mode. • Exhaustive search. This baseline solves the original problem P1 by exhaustively searching all feasible solutions, thereby incurring exponential time complexity. We use this baseline as an exact solution method to evaluate the running time efficiency of the proposed algorithm. To evaluate system performance, we use normalized task throughput, defined as the ratio of the number of served users to the total number of users.

end 8 end 7

Algorithm 3 summarizes the overall procedure for solving P1. It enumerates all feasible pairs of (z, λ). For each pair, it first solves P3.1 using Algorithm 1 to schedule users for the AD mode, and then solves P3.2 using Algorithm 2 to schedule the remaining users for the SD mode. The solution that serves the maximum number of users over all pairs gives the user scheduling decision Ẋ for the AD mode, user scheduling decision Ẏ for the SD mode, and GPU computing resource allocation ż. The approximation guarantee of Algorithm 3 is characterized in the following theorem. Theorem 1. Algorithm 3 achieves a constant approximation guarantee. Specifically, the produced solution Ẋ and Ẏ satisfy r Ẋ + Ẏ ≥ 12 r (X∗ + Y∗ ), where X∗ and Y∗ denote the optimal user scheduling decisions of P1. Proof. The proof sketch is as follows. Since X and Y are disjoint, we have r (X + Y) = r (X) + r (Y) and r′ (Y) = r (Y). Let z ∗ and λ∗ denote the computing and memory resource allocations in an optimal solution. Since Algorithm 3 enumerates all feasible z and ∗ ∗ λ, it also examines z and λ , and thus r Ẋ + Ẏ = r X̂ + Ŷ λ = λ̇, z = ż ≥ r X̂ + Ŷ λ = λ∗ , z = z ∗ . By Proposition 4, r X̂ λ = λ∗ , z = z ∗ ≥ r (X∗ ). ∗ ∗ We decompose Y into disjoint Y1 and Y2∗ , where Y1∗ includes the users in Y∗ that are also selected by K̂ad λ=λ∗ ,z=z∗ , and Y2∗ includes the remaining users. By Proposition 5, r Ŷ λ = λ∗ , z = z ∗ ≥ r (Y2∗ ). Since Y1∗ only contains users already selected by X̂, we ∗ have r (Y1 ) ≤ r X̂ λ = λ∗ , z = z ∗ . Thus, r (Y∗ ) ≤ r X̂ λ = λ∗ , z = z ∗ +r Ŷ λ = λ∗ , z = z ∗ . Then, we have r (X∗ + Y∗ ) ≤ 2r X̂ + Ŷ λ = λ∗ , z = z ∗ ≤ 2r Ẋ + Ẏ , completing the proof. The time complexity of Algorithm 3 is stated below. Theorem 2. Algorithm 3 has a polynomial time complexity of O K 6 log K , where K is the total number of users in K.

0.5 0.4 0.3 20

0.7 0.6 0.5 BALANCE (proposed) SD-only AD-only

0.4 30

0.3

Number of users

Maximum latency requirement

0.9

Normalized task throughput

1.0

0.9 0.8 0.7 0.6 0.5 0.4

BALANCE (proposed) SD-only AD-only

0.4

150

5.0x103

0.0 10

80 10 10

0.0 10

BALANCE (proposed) SD-only AD-only

0.8 0.7 0.6 0.5

(tokens)

(a) Results vs. maximum input length LImax .

0.3 100 200 300 400 500 600 700 800 900 1000

Maximum output length

(tokens)

(b) Results vs. maximum output length LO max .

Fig. 5. Performance of BALANCE under varying LImax and LO max . The O default values of LImin , LImax , LO min , Lmax are set to 1, 1000, 1, and 500, respectively. Tmin and Tmax are set to 2 s and 12 s, respectively. The default value of K is same as Fig. 4.

B. Performance Evaluation Figs. 4 and 5 evaluate the normalized task throughput of BALANCE under varying system settings. BALANCE consistently outperforms both AD-only and SD-only baselines. In Fig. 4, the throughput decreases with the number of users and increases under looser latency requirements. On average, BALANCE improves throughput over AD-only and SD-only baselines by 37.9% and 33.4%, respectively, under varying K, and by 31.5% and 27.3%, respectively, under varying Tmax . In Fig. 5, BALANCE achieves average gains over ADonly and SD-only baselines of 38.8% and 23.9%, respectively, under varying LImax , and 30.4% and 27.3%, respectively, under varying LO max . C. Algorithm Running Time Comparisons Fig. 6 compares the algorithm running time and task throughput of the proposed algorithm with exhaustive search under varying K. To run the exhaustive search efficiently, K is varied from 10 to 16. As shown in Fig. 6(a), the running time of the proposed algorithm grows nearly linearly with K, while exhaustive search increases nearly exponentially. The proposed algorithm achieves an average acceleration of about 5,100 times and up to 21,800 times at K = 16. Moreover, Fig. 6(b) shows that the proposed algorithm achieves near-optimal task throughput, with only about 0.9% average degradation. D. Ablation Study We conduct ablation studies to evaluate the contribution of each component in the proposed algorithm. We compare the proposed algorithm with three variants, including equal computing resource allocation (ECRA), equal memory allocation

0.998

Exhaustive search Proposed algorithm 0.99

0.99

0.985

0.986

0.987

0.988

0.98

0.97

Number of users

(a) Running time vs. number of (b) Normalized task throughput users K. vs. number of users K. Fig. 6. Comparisons of algorithm running time and task throughput between the proposed algorithm and the exhaustive search under varying K. The system parameters are the same as those in Fig. 4(a). 1.0

BALANCE (proposed) ECRA ( ) EMA ( ) ECRMA (

0.9 0.8

1.0

)

0.7 0.6 0.5 0.4 20

0.9 0.8 0.7 0.6 0.5 0.4

0.4

0.3 100 200 300 400 500 600 700 800 900 1000

Maximum input length

0.8

(s)

(a) Results vs. number of users (b) Results vs. maximum latency K. requirement Tmax . Fig. 4. Performance of BALANCE under varying K and Tmax . The default values of K, Tmin , Tmax are set to 30, 0.5, and 4, respectively. LImin , LImax , O LO min , and Lmax are set to 1, 300, 1, and 100, respectively. 1.0

1.0x104

Normalized task throughput

0.6

0.8

Normalized task throughput

0.7

1.00

Exhaustive search Proposed algorithm

0.9

Running time (s)

0.8

1.5x104

1.0

Normalized task throughput

BALANCE (proposed) SD-only AD-only

0.9

Normalized task throughput

1.0

Number of users

0.3

BALANCE (proposed) ECRA ( ) EMA ( ) ECRMA (

)

Maximum latency requirement

(s)

(a) Ablation results vs. number (b) Ablation results vs. maxiof users K. mum latency requirement Tmax . Fig. 7. Ablation study results for the proposed algorithm. The system parameters are the same as those in Fig. 4.

(EMA), and equal computing resource and memory allocation (ECRMA). In ECRA, z = 0.5, while λ and user scheduling are optimized using Algorithm 3. In EMA, λ = 0.5, while z and user scheduling are optimized using Algorithm 3. In ECRMA, both z and λ are 0.5, and only user scheduling is optimized using Algorithm 3. Fig. 7 illustrates the performance gains of the proposed algorithm over ECRA, EMA, and ECRMA under varying K and Tmax . Specifically, in Fig. 7(a), the proposed algorithm improves the task throughput by 5.6%, 18.7%, and 23.7% on average compared with ECRA, EMA, and ECRMA, respectively. The corresponding average performance improvements are 9.0%, 4.9%, and 18.4% in Fig. 7(b). VI. C ONCLUSION In this paper, we investigated a hybrid autoregressivespeculative inference (BALANCE) framework for edge LLM inference to balance the latency-memory tradeoff between conventional autoregressive decoding (AD) and speculative decoding (SD). In BALANCE, an edge server hosts both a small language model and a large language model, and simultaneously supports AD and SD for different users with heterogeneous demands. To maximize the number of served users, we formulated a joint user scheduling and computing resource allocation problem under end-to-end latency requirements and server memory constraints. Since this problem is an NP-hard problem, we transformed it into two sub-problems and developed a polynomial-time algorithm with a constant approximation guarantee. The experiment results showed that BALANCE consistently outperforms the conventional AD and SD schemes and achieves significant improvements in task throughput. Overall, BALANCE provides an effective and practical solution for resource-constrained edge LLM inference systems.

R EFERENCES [1] Z. Chen, B. Zhu, J. Wang, H. Shin, A. Nallanathan, and D. T. Niyato, “Network edge inference for large language models: Principles, techniques, and opportunities,” ACM Comput. Surv., vol. 58, no. 12, pp. 1–35, May 2026. [2] G. Qu, Q. Chen, W. Wei, Z. Lin, X. Chen, and K. Huang, “Mobile edge intelligence for large language models: A contemporary survey,” IEEE Commun. Surveys Tuts., vol. 27, no. 6, pp. 3820–3860, Dec. 2025. [3] M. Hu, Q. He, and D. Wu, “QLLMS: Quantization-adaptive LLM scheduling for partially informed edge serving systems,” in Proc. IEEE Int. Conf. Comput. Commun. (INFOCOM), May 2025, pp. 1–10. [4] S. Jang and R. Morabito, “Edge-first language model inference: Models, metrics, and tradeoffs,” in Proc. IEEE 45th Int. Conf. Distrib. Comput. Syst. Workshops (ICDCSW), Jul. 2025, pp. 309–314. [5] D. Kafetzis, R. Khalili, and I. Koutsopoulos, “Large language model partitioning for low-latency inference at the edge,” in Proc. 23rd Int. Symp. Model. Optim. Mobile, Ad Hoc, Wireless Netw. (WiOpt), Aug. 2025, pp. 1–8. [6] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in LLM inference with sarathi-serve,” in Proc. 18th USENIX Conf. Oper. Syst. Des. Implement. (OSDI), Jul. 2024. [7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. Adv. Neural Inform. Process. Syst. (NeurIPS), Long Beach, CA, USA, Dec. 2017, pp. 5998–6008. [8] A. K. Jaiswal, B. Hu, L. Yin, Y. Ro, T. Chen, S. Liu, and A. Akella, “FFN-SkipLLM: A hidden gem for autoregressive decoding with adaptive feed forward skipping,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), Miami, Florida, USA, Nov. 2024, pp. 16 943– 16 956. [9] H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui, “Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding,” in Proc. Findings Assoc. Comput. Linguistics: ACL 2024, Aug. 2024, pp. 7655–7671. [10] M. Yin, M. Chen, K. Huang, and M. Wang, “A theoretical perspective for speculative decoding algorithm,” in Proc. Adv. Neural Inform. Process. Syst. (NeurIPS), vol. 37, Dec. 2024, pp. 128 082–128 117. [11] M. Yan, S. Agarwal, and S. Venkataraman, “Decoding speculative decoding,” in Proc. Conf. Nations Americas Chapter Assoc. Comput. Linguistics: Human Lang. Technol., Albuquerque, New Mexico, Apr. 2025, pp. 6460–6473. [12] R. Svirschevski, A. May, Z. Chen, B. Chen, Z. Jia, and M. Ryabinin, “SpecExec: Massively parallel speculative decoding for interactive LLM inference on consumer devices,” in Proc. Adv. Neural Inform. Process. Syst. (NeurIPS), vol. 37, Dec. 2024, pp. 16 342–16 368. [13] R. Tiwari, H. Xi, A. Tomar, C. R. C. Hooper, S. Kim, M. Horton, M. Najibi, M. W. Mahoney, K. Keutzer, and A. Gholami, “QuantSpec: Self-speculative decoding with hierarchical quantized KV cache,” in Proc. 42nd Int. Conf. Mach. Learn. (ICML), Jul. 2025, pp. 59 668– 59 686. [14] J. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra, “Draft & verify: Lossless large language model acceleration via selfspeculative decoding,” in Proc. 62nd Annu. Meeting Assoc. Comput. Linguistics (ACL), Bangkok, Thailand, Aug. 2024, pp. 11 263–11 282. [15] H. Xia, Y. Li, J. Zhang, C. Du, and W. Li, “SWIFT: On-the-fly selfspeculative decoding for LLM inference acceleration,” in Proc. Int. Conf. Learn. Represent. (ICLR), Apr. 2025, pp. 1–24. [16] J. Jiang, Z. Chen, H. Shin, and A. Nallanathan, “Edge inference for large language models with pipeline parallelism and batching,” IEEE Trans. Commun., vol. 74, pp. 8390–8406, Apr. 2026. [17] X. Zhang, J. Nie, Y. Huang, G. Xie, Z. Xiong, J. Liu, D. Niyato, and X. Shen, “Beyond the cloud: Edge inference for generative large language models in wireless networks,” IEEE Trans. Wireless Commun., vol. 24, no. 1, pp. 643–658, Jan. 2025. [18] C. Liu and J. Zhao, “Resource allocation in large language model integrated 6G vehicular networks,” in Proc. IEEE 99th Veh. Technol. Conf. (VTC), Sep. 2024, pp. 1–6. [19] K. Zhang, H. He, S. Song, J. Zhang, and K. B. Letaief, “Communicationefficient distributed on-device LLM inference over wireless networks,” IEEE J. Sel. Topics Signal Process., vol. 19, no. 7, pp. 1301–1317, Oct. 2025. [20] B. Zhu, Z. Chen, L. Zhao, H. Shin, and A. Nallanathan, “Efficient LLM inference over heterogeneous edge networks with speculative decoding,” arXiv preprint arXiv:2510.11331, 2025.

[21] Y. Xu, S. Zhou, and Z. Niu, “DiP-SD: Distributed pipelined speculative decoding for efficient LLM inference at the edge,” arXiv preprint arXiv:2604.20919, 2026. [22] X. Li, D. Spatharakis, S. Ghafouri, J. Fan, H. Vandierendonck, D. John, B. Ji, and D. S. Nikolopoulos, “SLED: A speculative LLM decoding framework for efficient edge serving,” in Proc. 10th ACM/IEEE Symp. Edge Comput. (SEC), Dec. 2025. [23] J. Wang, Y. Su, J. Li, Q. Xia, Z. Ye, X. Duan, Z. Wang, and M. Zhang, “OPT-tree: Speculative decoding with adaptive draft tree structure,” Transactions of the Association for Computational Linguistics, vol. 13, pp. 188–199, Feb. 2025. [24] T. Liu, Y. Li, Q. Lv, K. Liu, J. Zhu, W. Hu, and X. Sun, “PEARL: Parallel speculative decoding with adaptive draft length,” in Proc. Int. Conf. Learn. Represent. (ICLR), vol. 2025, May 2025, pp. 1085–1104. [25] A. Albaseer, E. Bentafat, M. Hamood, M. Abdallah, A. Al-Fuqaha, and M. Hamdi, “Think fast, infer smart: A hybrid distributed LLMs inference at the wireless edge,” in Proc. IEEE 36th Int. Symp. Pers., Indoor Mobile Radio Commun. (PIMRC), Sep. 2025, pp. 1–6. [26] D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. Lakshmanan, and A. H. Awadallah, “Hybrid LLM: Cost-efficient and quality-aware query routing,” in Proc. Int. Conf. Learn. Represent. (ICLR), Vienna Austria, May 2024, pp. 1–19. [27] S. Oh, J. Kim, J. Park, S.-W. Ko, J. Choi, T. Q. Quek, and S.-L. Kim, “Communication-efficient hybrid language model via uncertainty-aware opportunistic and compressed transmission,” IEEE Trans. Commun., early access 2026. [28] H. Huang, F. Yang, Z. Liu, Y. Xu, J. Li, Y. Liu, X. Yin, D. Li, P. Ren, and E. Barsoum, “Jakiro: Boosting speculative decoding with decoupled multi-head via MoE,” arXiv preprint arXiv:2502.06282, 2025. [29] M. Fang, Z. Fu, Q. Zhao, and J. Wang, “When, what, and how: Rethinking retrieval-enhanced speculative decoding,” arXiv preprint arXiv:2511.01282, 2025. [30] H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-s. Chang, and J. Seo, “ExeGPT: Constraint-aware resource scheduling for LLM inference,” in in Proc. 29th ACM Int. Conf. Archit. Support Program. Lang. Operating Syst. (ASPLOS), Apr. 2024, p. 369–384. [31] H. Qiu, W. Mao, A. Patke, S. Cui, S. Jha, C. Wang, H. Franke, Z. T. Kalbarczyk, T. Başar, and R. K. Iyer, “Efficient interactive LLM serving with proxy model-based sequence length prediction,” in Proc. 5th Int. Workshop Cloud Intell./AIOps, vol. 5, San Diego, CA, USA, Apr. 2024, pp. 1–7. [32] R. Gong, S. Bai, S. Wu, Y. Fan, Z. Wang, X. Li, H. Yang, and X. Liu, “Past-future scheduler for LLM serving under SLA guarantees,” in Proc. 30th ACM Int. Conf. Archit. Support Program. Lang. Oper. Syst. (ASPLOS), Mar. 2025, pp. 798–813. [33] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for {Transformer-Based} generative models,” in Proc. 16th USENIX Symp. Operating Syst. Design Implement. (OSDI), Jul. 2022, pp. 521–538. [34] O. Khattab and M. Zaharia, “ColBERT: Efficient and effective passage search via contextualized late interaction over BERT,” in Proc. 43rd Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval (SIGIR), Mar. 2020, pp. 39–48. [35] PyTorch, “Reference API.” [Online]. Available: https://docs.pytorch. org/docs/stable/generated/torch.nn.utils.rnn.pad sequence.html [36] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “FlexGen: High-throughput generative inference of large language models with a single GPU,” in Proc. 40th Int. Conf. Mach. Learn. (ICML), Honolulu, Hawaii, USA, Jul. 2023, pp. 31 094–31 116. [37] NVIDIA, “NVIDIA multi-instance GPU user guide release r580,” 2025. [Online]. Available: https://docs.nvidia.com/datacenter/tesla/pdf/ MIG User Guide.pdf [38] F. Xu, J. Xu, J. Chen, L. Chen, R. Shang, Z. Zhou, and F. Liu, “iGniter: Interference-aware GPU resource provisioning for predictable DNN inference in the cloud,” IEEE Trans. Parallel Distrib. Syst., vol. 34, no. 3, pp. 812–827, Mar. 2023. [39] W. Shi, S. Zhou, Z. Niu, M. Jiang, and L. Geng, “Multiuser coinference with batch processing capable edge server,” IEEE Trans. Wireless Commun., vol. 22, no. 1, pp. 286–300, Jan. 2023. [40] H. Shojaei, T. Basten, M. Geilen, and A. Davoodi, “A fast and scalable multidimensional multiple-choice knapsack heuristic,” ACM Trans. Design Autom. Electron. Syst., vol. 18, no. 4, pp. 1–32, Oct. 2013. [41] M. Hifi, M. Michrafy, and A. Sbihi, “Heuristic algorithms for the multiple-choice multidimensional knapsack problem,” J. Oper. Res. Soc., vol. 55, no. 12, pp. 1323–1332, Jul. 2004.

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

Analyze & explore related research in Conceptio

Related documents