arXiv CSopen access

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

arXiv:2604.22906v1 [cs.DC] 24 Apr 2026

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities ZHIXIONG CHEN, Queen Mary University of London, United Kingdom BINGJIE ZHU, Xidian University, China JIANGZHOU WANG, Southeast University, China HYUNDONG SHIN, Kyung Hee University, Republic of Korea ARUMUGAM NALLANATHAN, Queen Mary University of London, United Kingdom DUSIT NIYATO, Nanyang Technological University, Singapore Large language models (LLMs) have advanced rapidly, emerging as versatile tools across ﬁelds thanks to their exceptional language understanding, generation, and reasoning capabilities. However, performing LLM inference at the network edge remains challenging due to their large memory and compute demands. This survey outlines the challenges speciﬁc to LLM edge inference and provides a comprehensive overview of recent progress, covering system architectures, model optimization and deployment, and resource management and scheduling. By synthesizing state-of-the-art techniques and mapping future directions, this survey aims to unlock the potential of LLMs in resource-constrained edge environments. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing. Additional Key Words and Phrases: Large language models, edge intelligence, mobile network architecture, resource optimization ACM Reference Format: Zhixiong Chen, Bingjie Zhu, Jiangzhou Wang, Hyundong Shin, Arumugam Nallanathan, and Dusit Niyato. 2026. Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities. 1, 1 (April 2026), 35 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 Introduction 1.1

Background

In recent years, large language models (LLMs) have become a cornerstone of artiﬁcial intelligence (AI) progress, demonstrating exceptional performance across a wide range of language-related tasks. Building on foundational advances, a vibrant ecosystem of LLMs has emerged, including the GPT-family [16], LLaMA-family [149], and open models such as OPT [198] and Mistral [75], along with specialized variants ﬁne-tuned for coding, reasoning, and multimodal understanding. From machine translation and sentiment analysis to question answering and text generation, these models exhibit outstanding capabilities in natural language comprehension, synthesis, reasoning, and generation. This progress has reshaped both research and industry, enabling the development of widely used applications such as ChatGPT [3] Authors’ Contact Information: Zhixiong Chen, [email protected], Queen Mary University of London, London, United Kingdom; Bingjie Zhu, [email protected], Xidian University, Xian, China; Jiangzhou Wang, [email protected], Southeast University, Nanjing, China; Hyundong Shin, [email protected], Kyung Hee University, Yongin-si, Republic of Korea; Arumugam Nallanathan, [email protected], Queen Mary University of London, London, United Kingdom; Dusit Niyato, [email protected], Nanyang Technological University, Nanyang Avenue, Singapore. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or commercial advantage and that copies bear this notice and the full citation on the ﬁrst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speciﬁc permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM

Chen et al.

and DeepSeek [97]. There is a growing consensus regarding this trajectory as a substantial step toward artiﬁcial general intelligence (AGI), though its timeline and scope remain active topics of debate [109]. Driven by scaling laws [81, 122], the development of LLMs has trended toward ever-increasing parameter counts. Nowadays, LLMs with tens and even hundreds of billions of parameters are becoming increasingly common. The sheer size of LLMs imposes heavy demands on memory, computation, and energy resources, creating deployment and inference bottlenecks not only on resource-constrained devices but also on advanced accelerators. For example, a LLaMA model [149] with 70B parameter requires roughly 140 GB just to hold 16-bit ﬂoating-point (FP16) weights, excluding the additional headroom needed for the key-value (KV) cache and activations during inference. Such requirements exceed the capacity of most edge devices and even data center GPUs, such as NVIDIA’s A100 80 GB or H100 80 GB. Moreover, once the model is loaded, long-context prompts and multi-user concurrency can quickly exhaust the available memory. Owing to their substantial computational and memory demands, mainstream commercial LLMs, e.g., ChatGPT [3] and DeepSeek [97], are deployed in cloud data centers to provide inference services for users. However, cloud-centric inference has inherent limitations in interactive, latency-critical, privacy-sensitive, and bandwidth-constrained scenarios. First, because each request must traverse the wide-area network to reach the cloud, end-to-end latency is susceptible to network delay and jitter, resulting in longer response times and unstable service quality. This worsens under peak load or on congested links, such as mobile networks during commutes, stadium events, or campus rush hours. Second, transmitting prompts and context to the cloud raises data-residency, privacy, and compliance risks. This is especially problematic in privacy-sensitive sectors such as healthcare and ﬁnance, where clinical notes, imaging summaries, and transaction data are governed by laws (e.g., GDPR [124]) and strict internal policies. Transmitting them oﬀ device risks noncompliance. Third, bandwidth costs rise with long contexts and multimodal inputs (e.g., video clips for vision-language analysis) and are further ampliﬁed when many users are served concurrently. Finally, reliance on remote cloud infrastructure undermines availability in weak-connectivity environments such as rural areas and disaster zones. These realities motivate techniques that compress, place, and schedule LLMs to run on consumer-grade devices while preserving user experience. To deploy LLMs on consumer-grade devices, a new wave of lightweight yet capable models and toolchains has emerged, co-optimizing model size, runtime, and hardware utilization. On the model side, Google’s Gemini Nano runs locally inside Android’s AICore on supported phones such as Pixel 8 Pro, with 1.8B and 3.25B variants designed for lowlatency on-device tasks [17]. Apple’s Apple Intelligence ships on iPhone, iPad, and Mac, executing models on device when possible and selectively using private cloud for heavier requests [7]. Chip vendors are broadening coverage, e.g., Qualcomm and Meta announced support for running LLaMA models on Snapdragon-powered phones and PCs, and LLaMA 3.2 introduced mobile-optimized 1B and 3B variants aimed at edge and handset use [127]. Toolchains such as ONNX Runtime Mobile [116] further streamline deployment of quantized models on iOS and Android with low memory overhead. 1.2

Motivation and Challenges

According to the above discussion, cloud-centric inference has been crucial for bringing LLMs to scale, but bandwidth, latency, privacy, and coverage constraints make it diﬃcult to meet practical application requirements. On-device LLMs minimize latency and keep data local, yet lightweight models typically underperform relative to state-of-the-art cloud models. Even when deployment succeeds, many devices cannot sustain long prompts, multi-turn sessions, or concurrent requests. These limitations motivate shifting toward edge serving that blends device, near-edge, and cloud resources. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

Edge serving avoids the extremes of cloud-centric and on-device inference by deploying LLM services close to user equipment at the network edge (for example, base stations and on-premises edge servers) while retaining on-demand access to near-edge and cloud resources. Compared with cloud-centric inference, edge serving reduces dependence on the wide-area network, shortens network paths, and lowers end-to-end latency. Compared with on-device inference, it provides more compute and memory headroom, enabling larger models, longer contexts, and greater concurrency. Consequently, moving from cloud-centric or on-device architectures to edge serving reframes the objective from ﬁtting everything on a single tier to co-designing model, memory, computation, and communication resources across tiers. The result is a practical pathway to sustaining quality of experience (QoE) with short response times, high throughput, and robust operation under real-world networks, without constant cloud backhaul or unrealistic on-device headroom. However, deploying LLM inference at the edge remains a complex challenge at the intersection of LLM serving, mobile computing, and wireless communications. Although cloud-centric and purely on-device LLM inference, as well as edge inference for conventional deep neural networks (DNNs), are well studied, their techniques do not scale or transfer directly to edge LLM serving due to LLM- and edge-speciﬁc constraints. Relative to cloud and on-device deployments, LLM edge inference faces distinctive challenges: 1) Device Heterogeneity: cloud-centric and on-device techniques often assume relatively uniform hardware and stable interconnects, whereas edge devices are heterogeneous across communication, computation, and memory. 2) Geo-Distributed Resources: edge devices are geographically dispersed and oﬀer fragmented resources, making coordination and scheduling harder than in centralized clouds or single-device setups. 3) Time-Varying Wireless Channels: because LLM edge inference depends on the radio access network, eﬀects like fading, interference, mobility, and handovers cause time-varying channel quality, resulting in bursty latency, uneven throughput, and packet loss. Relative to conventional DNN inference, LLMs introduce extra challenges: 1) Stateful Inference Process: unlike traditional DNNs that run a single forward pass per request, LLMs generate token by token, requiring multiple forward passes and needing to maintain persistent states (mainly KV cache that grows and shrinks dynamically) across multiple invocations. 2) Uncertain Resource Demand: conventional DNNs often assume ﬁxed-size inputs and outputs, whereas LLM prompts vary widely and output length is unpredictable, memory usage and processing strategies must adapt on the ﬂy. 3) Large-Scale Model Parameters: even small LLMs occupy gigabytes, stressing device storage, bandwidth, and computation more than DNNs. 4) Two-Phase Asymmetry: the inference of LLMs involves two asymmetric phases, i.e., compute-bounded preﬁll and memory-bounded decoding, which requires phase-aware system designs. In view of these challenges, enabling LLM inference at the wireless network edge requires transformative cross-layer designs spanning models, system architectures, and algorithms for managing and scheduling communication, computation, and memory resources. To this end, this survey focuses on a comprehensive review of emerging techniques in this area and provides practical guidance for research and deployment.

1.3

Comparisons with Prior Surveys and Our Contributions

Prior surveys on LLM inference, such as [82, 90, 92, 107, 118, 167], primarily focus on cloud data centers and synthesize advances in model compression, memory management, and cluster orchestration under relatively homogeneous GPU-rich infrastructure. These works overlook the resource constraints, device heterogeneity, and variability of wireless communications. Moreover, surveys on on-device LLMs, e.g., [174, 206], focus on single-device deployments for phones and personal computers. They emphasize compression and intra-device serving, but rarely address multi-user edge environments. In addition, existing surveys on LLM edge inference, e.g., [125, 175], remain high level, oﬀering Manuscript submitted to ACM

Chen et al.

architectures or application taxonomies but lacking a detailed treatment of LLM-speciﬁc challenges, cross-layer codesign, and reproducible evaluation. This survey ﬁlls these gaps by adopting a cross-layer perspective that integrates model optimization and deployment, system architecture design, and resource management to enable communication-, compute-, and memory-aware LLM inference at the network edge. The main contributions of this paper are summarized as follows: • We provide a comprehensive overview of LLM inference and the unique challenges at the wireless network edge. Building on this foundation, we present reference architectures for LLM edge serving that span the device, near-edge, and cloud tiers, covering distributed deployment and inference frameworks. • We provide a comprehensive review of optimization techniques for LLM inference at the network edge, covering model optimization and deployment, and resource management across communications, computation, and memory. We also curate the metrics, models, datasets, and platforms needed for rigorous, reproducible evaluation and to support real-world deployment. • We identify several crucial research directions for LLM edge inference, including scalable serving for multimodel and multimodal LLMs, secure and privacy-preserving execution, and green inference. This survey is organized as follows. Section 2 illustrates the fundamentals of LLM edge inference. Section 3 presents system architectures for LLM edge inference and their associated challenges, which deﬁne the execution topology and the resulting communication and state-transfer patterns. Building on this, Section 4 reviews model optimization and deployment techniques that make each architecture practical and eﬃcient under edge constraints. Section 5 then covers joint optimization of communication, computation, and memory for improved eﬃciency and user-perceived service quality. Section 6 details evaluation methodology. Section 7 outlines future research opportunities. The conclusion is drawn in Section 8. 2 Fundamentals of LLM Edge Inference 2.1

Generative LLM Architecture

LLMs are typically Transformer-based neural networks with massive parameters. Depending on the Transformer components used and the attention-masking scheme, LLMs typically have three primary structures: • Encoder-only LLMs: These models consist of a stack of Transformer encoder blocks [151] that map input text to contextual vector representations, without an explicit decoding phase for free-form generation. They are typically pretrained using a masked language modeling paradigm, i.e., predicting masked words in a sentence using bidirectional self-attention (no causal mask) conditioning on both left and right context. Canonical examples include BERT [31], RoBERTa [99], and ALBERT [86]. • Encoder-decoder LLMs: These models pair a bidirectional encoder with an autoregressive decoder, linked via cross-attention from the decoder to the encoder. The encoder converts the input text into context-rich embeddings, while the decoder generates the target sequence left-to-right using causal self-attention plus crossattention to the encoded representations output by the encoder. A canonical example is T5 [128], with closely related variants such as mT5 [177] and FLAN-T5 [159]. • Decoder-only LLMs: These models are composed of a stack of Transformer decoder blocks and are pretrained using an autoregressive language modeling paradigm with causal self-attention to generate the next token given the preceding tokens. For each token, the attention mechanism makes only the preceding tokens visible, enforcing causality consistent with natural human speech. This makes them natural generators for tasks such Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

as open-ended text generation, dialogue, summarization, code generation, and question answering. They also support in-context learning (ICL) [35] and are often adapted with instruction tuning [101] for general-purpose assistance. Representative models under this architecture include GPT-3 [16], OPT [198], and LLaMA [149]. Early in the development of LLMs, decoder-only models were less popular than encoder-only and encoder-decoder models. However, the release of the game-changing GPT-3 [16] shifted the landscape, spurring a rapid rise of decoderonly architectures, which now dominate modern LLM design. Output token

Prefilling

Transformer block output

Linear GeLU Linear

Decoding strategy

Output logits Linear

Multi-head attention input

Matmul

Dropout

Transformer Decoder Block 1

Multi-Head Masked Attention

Softmax

Matmul K V Linear

Multi-Head Attention

Feed Forward Network

...

Transformer block input

...

Layer Normalization

Embedding layer Tokenizer

Layer 1

...

Positional Encoding

Layer 1

Linear

Layer Normalization

Transformer Decoder Block 2

Layer 1

Iteration 2

...

Feed Forward Network

Iteration 1

...

Transformer Decoder Block N

...

Layer Normalization

Dropout

Decoding

ABCD

Layer N

<EOS>

Multi-head attention input

KV-Cache Input sentence

(a) Architecture of decoder-only LLMs

(b) Inference process of decoder-only LLMs

Fig. 1. Architectures and inference process of LLMs.

2.2

Inference Process of LLMs

Given the prevalence of decoder-only LLMs, we provide a concise overview of their architecture and inference process. In fact, encoder-only inference can be viewed as a special case of the decoder-only pipeline, consisting only of the preﬁll stage and processing the input once to produce contextual representations. Encoder-decoder models add an explicit encoding step, ﬁrst run a single pass through the encoder to produce ﬁxed representations, then perform left-to-right decoding with causal self-attention while cross-attending to the static encoder memory at each step. Other model variants, such as vision-language models (VLMs), extend the text-only pipeline with a modality front-end. In VLMs like Qwen2-VL [148] and LLaVA-1.5 [98], a vision encoder converts images or video frames into feature sequences, projects them into the LLM embedding space, and combines them with text tokens (or uses them as cross-attention memory). The LLM then follows the same preﬁll and autoregressive decoding as in text-only inference, while the modality encoder, typically Transformer-based, resembles an encoder-only model at inference time. The architecture of decoder-only LLMs is illustrated in Fig. 1(a), which consists of a tokenizer, an embedding layer, a positional encoding module, multiple stacked Transformer decoder blocks, and a decoding head composed of a linear projection followed by a softmax operator. Each Transformer decoder block follows the design of the decoder component in the original Transformer model [151] (or one of its modern variants) and typically contains three core submodules: a multi-head self-attention (MHSA) mechanism with causal masking, a FFN, and layer normalization (LN). Residual connections are applied around each submodule to stabilize training and preserve information ﬂow. The blocks are stacked sequentially, where the output of one block serves as the input to the next. Decoder-only LLMs rely on an autoregressive generation strategy, producing sequences in a strictly sequential manner by generating one token at a time, as shown in Fig. 1(b). At each generation step, the model processes the Manuscript submitted to ACM

Chen et al.

entire sequence observed so far, including both the original input tokens and the tokens it has already generated, to predict the next token. While conceptually straightforward, this process becomes increasingly expensive as sequence length grows, since naively recomputing self-attention over all prior tokens at every step leads to quadratic growth in computation and memory. To address this challenge, modern LLMs adopt the KV caching technique [65], in which the key (K) and value (V) vectors produced within the MHSA blocks are stored after their ﬁrst computation and then reused in subsequent decoding steps. Building on this, the inference process of decoder-only LLMs can be conceptually divided into two stages:

• Preﬁll Stage: In this stage, the LLM processes the entire initial input (the prompt) to compute hidden states and build the KV cache in each self-attention layer, as shown in Fig. 1(b). Concretely, the prompt is tokenized (text to token IDs), mapped to embeddings, combined with positional encodings, and passed through the stacked Transformer decoder blocks. The output of the last Transformer block is fed into the LLM head to produce output logits. A decoding method (e.g., greedy, top-k/top-p sampling with temperature, beam search, and contrastive search [9]) then selects the ﬁrst output token based on the logits. • Decoding Stage: After preﬁll, the LLM enters an iterative loop that generates one token at a time, as shown in Fig. 1(b). At each step, the previously generated token is fed directly into the embedding layer without tokenization, since it is already a token rather than raw text. The resulting embedding is combined with positional encodings and passed through the Transformer stack and the LM head to produce the next token. During this process, the LLM reuses the existing KV cache from prior steps and appends new keys and values for the freshly generated token, avoiding recomputation over the entire history. The procedure continues until a stop condition is met, namely when an end-of-sequence token is generated or the conﬁgured maximum sequence length is reached. Note that without a KV cache, each step would require reprocessing all past tokens with the new token appended.

2.3

Edge Inference

Edge inference executes trained AI models at or near the data source, with on-demand access to nearby edge and cloud resources, rather than relying solely on distant cloud infrastructure or underpowered local devices. It oﬀers: (1) low latency and high responsiveness by avoiding wide-area round trips and backhaul jitter, (2) stronger privacy by keeping sensitive data (e.g., voice, medical text, personal documents) local, and (3) oﬄine capability in disconnected or low-connectivity settings such as rural areas, underground transit systems, and disaster zones. These advantages make edge inference a key enabler for a broad spectrum of AI applications including on-device metaverse perception and rendering, autonomous driving with millisecond-scale perception and planning loops, and real-time smart-city video analytics.

3 System Architectures and Challenges for LLM Edge Inference This section introduces system architectures for LLM edge inference and highlights the associated research challenges. As shown in Fig. 2, we categorize LLM edge inference into four architectural patterns. We describe each architecture and its typical characteristics below, with an overview in Table 1. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities Device-edge

Edge-cloud

Device-cloud

Cloud server

Device-edge-cloud

Edge-edge

Cloud server

Complete LLM Edge server

Edge server Edge server

Edge server

Edge servers

Edge device

Device-device Edge servers

Edge devices

Edge device

(a)

Edge device

(b)

Edge device

Edge devices

(c)

(d)

Fig. 2. Architectures of LLM edge inference: (a) single-edge-node inference, (b) vertical collaborative inference, (c) horizontal collaborative inference, (d) hybrid collaborative inference.

3.1

Single-Edge-Node Inference

This architecture deploys an LLM on a single edge server to perform inference computations, serving multiple users in its coverage area and, when needed, nearby regions via low-latency metro or backhaul links, as shown in Fig. 2(a). Unlike on-device inference that serves requests on a single device, single-edge-node inference runs on a shared access or metro node that receives user prompts, performs LLM inference, and returns the results to their devices. In line with this architecture, [200] deploys a quantized LLM on an edge server and adopts batching to increase inference throughput while meeting latency targets. Recognizing that users may require diﬀerent LLMs for diverse tasks, [210] and [176] deploy multiple LLMs on each edge server and optimize their placement to match server resource constraints. [58] further optimizes LLM task oﬄoading and resource allocation via active inference. Overall, single-edge-node inference oﬀers low coordination overhead, reduces backhaul traﬃc, and strengthens privacy, but it is constrained by the server’s memory, compute, and power budget. It works well when quantized or distilled LLMs ﬁt with suﬃcient KV-cache headroom, while very large models often exceed a single node’s capacity. 3.2

Vertical Collaborative Inference

This architecture partitions an LLM across devices, edge servers, and the cloud to meet latency, throughput, and privacy goals within each tier’s resource budget, enabling support for very large LLMs and bursty traﬃc. It is typically realized as device-edge, edge-cloud, device-cloud, or device-edge-cloud collaboration, as shown in Fig. 2(b), with split points adjusted at runtime based on link quality, queueing delay, and compute load. This ﬂexibility improves perceived latency and elasticity while preserving privacy. In line with this architecture, prior work uses DRL [23, 42] or dynamic programming [110] to select split points for deploying an LLM across the user device and the edge server. In [171], the LLM is partitioned into three submodels, with the input and output submodels running on the device and the middle submodel, which contains most decoder layers, hosted in the cloud. By pooling resources across devices, edge servers, and the cloud, vertical collaborative inference can lower perceived latency, provide elastic capacity, and strengthen privacy. However, it increases orchestration complexity and can be bandwidth-sensitive, especially for long prompts, due to repeated activation transfers across tiers. 3.3

Horizontal Collaborative Inference

This architecture adopts multiple peer edge devices in the same tier to cooperate on LLM inference by splitting work across devices, as shown in Fig. 2(c). It can employ tensor parallelism to shard weights and run layers in lockstep, pipeline parallelism to place consecutive layer stages on diﬀerent devices, and data-parallel replication to smooth bursts Manuscript submitted to ACM

Chen et al.

and raise throughput. Towards this architecture, [201] partitions the LLM across end devices to enable collaborative inference and optimizes model assignment policy via linear programming. In [79] and [196], the LLM is partitioned across heterogeneous devices or edge servers, and dynamic programming is employed to optimize the pipeline layout for maximizing throughput. [202] combines model quantization with phase-aware partitioning across multiple edge devices to further improve throughput. This architecture scales beyond a single device’s memory and compute limits, improving elasticity and availability. However, it introduces orchestration complexity, increases sensitivity to interdevice link quality, and raises state-management overhead. It is most useful when an LLM cannot ﬁt on one device and when nearby edge nodes oﬀer suﬃciently fast links.

3.4

Hybrid Collaborative Inference

This architecture combines vertical collaboration across the device, edge, and cloud with horizontal collaboration among peer nodes within each tier, as depicted in Fig. 2(d). Compared with using only vertical or only horizontal collaboration, it can serve larger models, absorb bursts, and reduce perceived latency by anchoring decoding near users while borrowing cloud capacity for heavy preﬁll or long-context retrieval. For example, [15] aggregates idle computation resources from multiple research groups and volunteers to execute LLM inference, eﬀectively accelerating inference. The trade-oﬀ is greater orchestration and state-management complexity, since KV caches and session state may need to be partitioned, compressed, and migrated across tiers and peers without violating latency targets. Arch.

Placement

SingleEdge-Node

Single edge server

Vertical Collaborative

Combination between device, edge, cloud

Horizontal Collaborative

Peer edge nodes

Hybrid Collaborative

Vertical + horizontal

3.5

Table 1. An Overview of Architectures for LLM Edge Inference Strength Limits Enabling techniques Compression (4.1), Decoding acceleration Lightweight Simple (4.2), Caching/selection (4.3.2), Resource LLMs only scheduling (5) Compression (4.1), Decoding acceleration Strong and (4.2), Partition & placement (4.3.1), Backhaul elastic Disaggregated preﬁll/decoding (4.3.3), bandwidth capacity Resource scheduling (5) Compression (4.1), Decoding acceleration Higher Inter-device (4.2), Partition & placement (4.3.1), throughput bandwidth Resource scheduling (5) Support Compression (4.1), Decoding acceleration Orchestration very large (4.2), LLM deployment strategies (4.3), complexity LLMs Resource scheduling (5)

Rep. Examples [58, 176, 200, 210]

[23, 42, 110, 171]

[79, 196, 201, 202]

[15]

Challenges for LLM Edge Inference

As discussed in Section 2, LLM inference diﬀers from conventional DNN inference in both computational and systems behavior. On resource-constrained edge platforms, this divergence introduces distinct challenges: 3.5.1

Stateful Inference Process. Unlike stateless DNN inference (a single forward pass with a relatively ﬁxed

memory footprint), LLM inference is autoregressive and performs many forward passes, one per generated token. At each step the system maintains a per-request KV cache at every layer to avoid recomputation, and this cache grows with context length and is released only when the request completes [118]. This statefulness complicates edge serving, since schedulers must anticipate per-request memory growth, reserve headroom to prevent out-of-memory (OOM) errors, and manage costly preemption or migration when KV caches must be transferred or rebuilt. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities 3.5.2

Two-Phase Asymmetry. LLM inference proceeds in two phases, i.e., preﬁll and decoding. In preﬁll, the model

processes the entire prompt, builds the KV cache, and computes full self-attention. The workload is dominated by large dense matrix multiplications (Q/K/V projections and FFNs) and quadratic attention score products. Because weights are reused across all prompt tokens, arithmetic intensity is high with strong on-chip reuse, making preﬁll largely computebound and primarily limited by compute throughput [90]. In contrast, the decoding phase generates one token per step, repeatedly reading the full KV cache and appending new token’s KV entries, with most computation reduced to matrixvector operations. Arithmetic intensity drops and memory traﬃc dominates, so decoding depends more on memory bandwidth than compute throughput and is typically memory-bound [9]. This asymmetry complicates edge serving, requiring compute-focused optimizations for preﬁll and memory bandwidth-centric optimizations for decoding. 3.5.3

Large-Scale Parameter Counts. Mainstream LLMs contain billions or even trillions of parameters. For in-

stance, the LLaMA-70B model [149] has 70 billion parameters, while the GPT-3 model [16] scales up to 175 billion parameters. This scale drives up computation demand, memory capacity requirements, and memory bandwidth pressure during inference. Even with quantization, weight footprints can exceed on-device memory, increasing reliance on model partitioning, weight oﬄoading, or aggressive compression. 3.5.4

Device Heterogeneity. Unlike cloud data centers with relatively uniform hardware and stable interconnects,

edge devices are often resource limited and heterogeneous in computing, communication, memory capabilities [24]. The scale of modern LLMs frequently exceeds the capacity of a single device, and inference typically requires spanning multiple edge devices or cooperating with the cloud. This makes deployment, request scheduling, resource management, and memory allocation substantially more complex. 3.5.5

Data Heterogeneity. Unlike the conventional DNN inference with ﬁxed-size inputs, LLM input prompts vary

widely in length and structure [206]. Moreover, the inference requests come from diﬀerent users and have diﬀerent diﬃculty, leading to uneven preﬁll cost, decoding time, and KV-cache growth. This variability complicates batching and scheduling, which must balance throughput against latency, pick batch windows that do not penalize short prompts, and mitigate head-of-line blocking from long prompts. 3.5.6

Uncertain Resource Demand. Unlike conventional DNN inference with ﬁxed-size outputs, LLM inference

has stochastic generation lengths, making workloads variable and unpredictable [118, 206]. As a result, the computation, memory, and communication demands are not known in advance, complicating edge scheduling and capacity planning. Systems need to provision headroom for KV-cache growth and adapt resources dynamically during a request. In edge-cloud settings, they should also adjust oﬄoading to network and queueing conditions and prefetch or migrate KV state to avoid OOM. 3.5.7

Time-Varying Wireless Channels. Most cloud-centric and on-device LLM inference studies (e.g., [119, 173])

model device-cloud and inter-device links as bit pipes with constant or stationary random rates, which improves tractability and is often reasonable when core-network latency or high-speed on-device links dominate. In edge deployments, however, the radio access network induces time-varying rate, loss, and jitter due to fading and interference [49]. This variability matters because LLM edge inference is communication-coupled, with prompts or context and intermediate activations traversing wireless links, which increases end-to-end and tail latency, reduces token throughput, and can stall pipelines or batches. In streaming settings, it can also make token delivery bursty and trigger SLO-driven Manuscript submitted to ACM

Chen et al.

adaptations such as prompt or context truncation, switching to smaller or cached models, repartitioning, or more aggressive compression or early-exit, thereby trading quality for latency. Channel dynamics similarly aﬀect distributed training and ﬁne-tuning across edge nodes, where intermittent uplinks create stragglers and stale updates, forcing synchronization waits or participant dropout and slowing or destabilizing convergence. 3.5.8

Mobility and Intermittent Connectivity. In wireless edge environments, user mobility can trigger frequent

handovers across base stations or serving edge nodes, while deep fading and coverage holes may cause transient disconnections [49]. These dynamics are particularly problematic because LLM inference is stateful, and maintaining session continuity may require migrating runtime states (e.g., KV caches and conversation context), re-establishing transport connections, and re-optimizing placement and scheduling under new radio conditions. If not handled carefully, handovers and outages can interrupt service, increase tail latency, and waste compute and radio resources due to retries or restarted decoding. This challenge is prominent in high-mobility and unreliable-link scenarios such as high-speed rail, unmanned aerial vehicle (UAV) swarms, and rural connectivity, motivating handover-aware state management and graceful degradation mechanisms. 4 Model Optimization and Deployment This section reviews model optimization and deployment techniques for resource-eﬃcient, low-latency LLM inference at the network edge, including model compression, decoding mechanisms, and deployment strategies. These techniques are tightly coupled with the collaborative architectures in Section 3. In particular, the architecture determines where computation and state reside and what must be transferred across the network, while the techniques in this section provide the concrete levers to make each architecture feasible and eﬃcient. 4.1

Model Compression

The formidable size and compute demands of LLMs make practical deployment challenging in resource-constrained wireless edge environments. A natural and widely adopted solution is model compression, which reduces the storage and computational footprint of LLMs while preserving accuracy as much as possible and complying with edge resource constraints. Although compression techniques are broadly applicable to conventional DNNs, their design and eﬀectiveness for LLM edge inference are inﬂuenced by LLM-speciﬁc properties, such as pronounced weight/activation outliers and prompt-dependent activation ranges. This subsection systematically reviews compression methods for LLMs, including quantization, pruning, knowledge distillation, and low-rank factorization. Table 2 summarizes their key characteristics. 4.1.1

Quantization. Quantization is a key LLM compression technique that reduces model size with only modest

degradation in inference performance. Depending on when it is applied, quantization is categorized as post-training quantization (PTQ), which quantizes a pretrained model without retraining, and quantization-aware training (QAT), which simulates quantization eﬀects during training or ﬁne-tuning to improve robustness at low bit-widths. Furthermore, quantization can be grouped by the target component, i.e., weight-only quantization, and weight-activation quantization. 1) Weight-only Quantization: This category compresses LLMs by representing model parameters at reduced numerical precision while typically keeping activations in higher precision. It is particularly eﬀective for edge LLM inference Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

Technique Quantization

Pruning Knowledge Distillation Low-rank Factorization

Table 2. Comparison of LLM Compression Techniques Key Idea Advantages Limitations Reduce Reduces model size; Accuracy degradation at low weight/activation supported by many precision; may require precision accelerators retraining Remove redundant Produces sparse models; Unstructured pruning needs weights, neurons, heads, reduces computation special hardware; structured or layers and memory footprint pruning harm accuracy Train a smaller Produces compact Requires costly teacher “student” to mimic a dense models with good training; student may underﬁt larger “teacher” accuracy retention complex reasoning tasks Approximate large Compression limited by Signiﬁcant parameter weight matrix with intrinsic rank; accuracy loss if reduction low-rank matrices rank too small

Rep. Examples AWQ [95], Q-BERT [138], LLM.int8() [29], PACT [25] SparseGPT [43], Wanda [144], LLM-Pruner [103] MiniLLM [53], FUSELLM [152], PromptMix [134] ASVD [190], LASER [136], DSFormer [18]

because weights dominate the memory footprint, lowering weight precision reduces memory traﬃc and can significantly improve token throughput. However, compared with conventional DNNs, LLMs often exhibit pronounced outlier weights that disproportionately contribute to quantization error, motivating outlier-aware PTQ designs. For example, AWQ [95] preserves only 1% of salient weights identiﬁed via activation statistics to mitigate quantization error, achieving more than 3× speedup over FP16 implementation on both desktop and mobile GPUs. In [30, 54, 100, 161], outliers are retained by sacriﬁcing nearby low-importance values. In QAT settings, prior works, e.g., Q8BERT [191] and mixed-precision schemes such as Q-BERT [138], show that training-aware or sensitivity-aware bit-width assignment can better preserve accuracy at low precision, and subsequent methods (e.g., TernaryBERT [199] and BinaryBERT [10]) further push quantization to ternary or even binary regimes. 2) Weight-Activation Quantization: This category lowers the precision of both weights and activations to reduce computation and data movement costs beyond weight-only schemes. Activation quantization is typically more challenging for LLMs because activations exhibit large dynamic ranges and can shift with prompts, sequence length, and token position. Naively quantizing them may destabilize attention and MLP computations and amplify error during autoregressive decoding. ZeroQuant [184] pioneers weight-activation quantization for LLMs with group-wise weight quantization and token-wise INT8 activation quantization. To handle outlier activations, LLM.int8() [29] routes a small fraction of outlier channels through higher precision to stabilize INT8 inference without ﬁne-tuning, while SmoothQuant [170] redistributes activation magnitudes into the weights via complementary rescaling, enabling robust W8A8 and even W4A8 across diverse LLMs. Moreover, PTQ methods typically set quantization parameters via calibration methods such as min-max ranges, percentile clipping, or KL-divergence minimization [74, 83]. For tougher settings or domain shift, QAT integrates fake-quantization modules into the training loop so that networks explicitly learn robust scaling factors and clipping thresholds, with methods such as PACT [25] and LSQ [38] widely recognized for stabilizing low-bit training. 4.1.2

Pruning. Pruning is another widely used LLM compression technique that removes unnecessary or redundant

parameters. In general, pruning approaches can be divided into two categories, i.e., unstructured pruning and structured pruning. 1) Unstructured Pruning: This form of pruning removes individual weights throughout the model without respecting architectural boundaries. Compared with classic magnitude pruning used in conventional DNNs, unstructured LLM pruning is often calibration-driven and layer-wise, because Transformer blocks are sensitive to perturbations and errors may propagate through residual connections during long-horizon generation. For instance, SparseGPT [43] is Manuscript submitted to ACM

Chen et al.

a one-shot method that removes a large fraction of weights while approximately reconstructing each layer’s outputs using calibration data, achieving over 50% sparsity in models like OPT and LLaMA without retraining. Wanda [144] combines weight magnitudes with input activation norms to drop weights that are both small and weakly activated, improving over simple magnitude pruning. OWL [188] further reﬁnes pruning decisions by assigning higher pruning budgets to layers with more activation outliers through non-uniform hierarchical sparsity. Despite these advances, unstructured pruning inherently produces irregular and non-uniform weight distributions, which are poorly aligned with standard dense linear algebra kernels [150, 168]. Thus, realizing speedups in practice often requires specialized hardware (e.g., sparse tensor cores) or optimized software libraries (e.g., cuSPARSE1 and oneMath2 ). 2) Structured Pruning: This form of pruning removes entire architectural units in a model, such as neurons/channels in MLPs, attention heads, or even whole layers, rather than individual weights. Because the remaining tensors stay dense and regularly shaped, structured pruning maps well to commodity kernels (e.g., general matrix-matrix multiplication (GEMM) and convolution) and thus delivers practical speedups on CPUs, GPUs, and NPUs. It is especially suitable for LLMs since the Transformer architecture oﬀers natural pruning granularity, and removing heads/hidden dimensions can further reduce attention cost and potentially shrink the KV-cache footprint, a key memory bottleneck for long-context decoding. A representative example is LLM-Pruner [103], which performs task-agnostic compression by estimating component importance with ﬁrst-order gradients and approximate Hessians, pruning structural groups, and then restoring accuracy with lightweight LoRA ﬁne-tuning. Beyond attention heads and FFNs, some studies further explore pruning hidden dimensions (e.g., embedding layers, layer norm) for additional gains [84, 146]. ShearedLLaMA [169] combines targeted structured pruning with dynamic batch loading to reshape pretrained models into target architectures. By adapting data sampling ratios across domains, it maintains quality while using about 3% of the computation needed to train from scratch, making it a highly cost-eﬀective approach. 4.1.3

Knowledge Distillation. Knowledge distillation transfers the capabilities of a large “teacher” model to a

smaller “student” model, producing a more eﬃcient model that reduces size and deployment cost while aiming to retain performance. Distillation methods are commonly categorized as open-box or closed-box. 1) Open-Box Knowledge Distillation: This approach exploits full access to the teacher model’s architecture and parameters, transferring knowledge through both internal representations (e.g., hidden states, attention maps) and output logits. Early eﬀorts such as MiniLM [156] extract self-attention and value relations from the ﬁnal Transformer layer to alleviate the complexity of strict layer-to-layer mapping, enabling smaller students to retain over 99% of the teacher’s accuracy with roughly half the parameters. To address teacher-student capacity mismatch, MiniLLM [53] and TED [94] improve distillation robustness using reverse KL divergence and layer-wise alignment, respectively. FUSELLM [152] further boosts performance by fusing knowledge from multiple teachers at multiple granularities. 2) Closed-Box Knowledge Distillation: This approach uses only the teacher’s outputs and requires no access to its internal architecture or parameters, which suits proprietary or closed-source teachers (e.g., commercial LLMs). The student is trained to mimic the teacher’s predictions or responses. PromptMix [134] generates labeled examples through prompting, where borderline examples improve knowledge transfer from teacher models such as GPT-3.5 to student models. For chain-of-thought reasoning, Fine-tune-CoT [62] demonstrates that large models (over 100B parameters)

1 https://docs.nvidia.com/cuda/cusparse/

2 https://github.com/uxlfoundation/oneMath

Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

can distill their reasoning into students with as few as 0.3B parameters. Instruction following is another major focus, with Lion [78] achieving ChatGPT-level performance using a 13B-parameter model by adversarially generating complex instructions. 4.1.4

Low-Rank Factorization. Low-rank factorization approximates a matrix W ∈ R𝑚×𝑛 with the product of

two lower-rank matrices, A ∈ R𝑚×𝑟 and B ∈ R𝑟 ×𝑛 , i.e., W ≈ AB, where 𝑟 ≪ min(𝑚, 𝑛). This reduces parameters from O (𝑚𝑛) to O ((𝑚 + 𝑛) × 𝑟 ). ASVD [190] is the ﬁrst work applying factorization to compress LLMs, explicitly managing activation outliers by adapting weight matrices to the activation distribution and iteratively calibrating layers according to their decomposition sensitivity. LASER [136] further reveals that selectively removing higher-order components of weight matrices can surprisingly improve model performance while reducing complexity. DSFormer [18] factorizes weights into a semi-structured sparse matrix times a small dense matrix, yielding up to 40% higher compression than standard low-rank factorization methods. Overall, low-rank factorization reduces memory footprint and computation. Since memory access often bottlenecks LLM decoding, it also lowers the number of parameters to load, speeding up generation. 4.1.5

Compression, accuracy, and trustworthiness trade-oﬀ. While quantization, pruning, distillation, and low-

rank factorization facilitate edge deployment by reducing memory and computation, they can also change the generation behavior of LLMs. As compression becomes more aggressive (e.g., lower bitwidths, higher pruning ratios, or smaller distilled students), task performance may degrade and the impact is often uneven across tasks. Prompts that require strong factuality or multi-step reasoning are typically more sensitive. For generative applications, standard accuracy metrics should therefore be complemented with trustworthiness evaluations, including hallucination and factuality as well as bias and toxicity measurements [45, 96, 111]. Because edge deployments commonly rely on compact or heavily quantized models, compression schemes should be designed and selected with a multi-objective view that explicitly balances inference eﬃciency (latency, memory, and energy) with task quality and trustworthiness. Moreover, although this survey focuses on inference rather than training, many compression techniques still require calibration or ﬁne-tuning, such as PTQ calibration or QAT. When these updates are learned from distributed edge data, federated learning and other decentralized training frameworks [113, 121] are particularly useful, as they can enable privacy-preserving adaptation while limiting data movement. 4.2

Decoding Mechanisms

This subsection reviews decoding mechanisms, as illustrated in Fig. 3, that improve the eﬃciency and eﬀectiveness of LLM edge inference. A comparison of diﬀerent decoding mechanisms is provided in Table 3. 4.2.1

Non-autoregressive Decoding. Non-autoregressive decoding departs from standard autoregressive genera-

tion by generating multiple tokens in parallel, as shown in Fig. 3(a). It relaxes sequential dependencies by assuming a certain degree of conditional independence across output positions, removing the per-token feedback loop and markedly speeding up inference. However, non-autoregressive methods often lag in accuracy because they do not explicitly model token dependencies. To mitigate this limitation, knowledge distillation [133] and source-target alignment [56] approaches are proposed to narrow the performance gap with autoregressive models. Moreover, semiautoregressive decoding further extends non-autoregressive methods by modeling output dependencies [48, 52] or iteratively reﬁning predicted tokens [89], with the goal of approaching autoregressive-level quality. Manuscript submitted to ACM

Chen et al.

can

help

you

can can

Transformer Block N

help help

Transformer Block 2

Transformer Block N you you Transformer Block 3 I Transformer Block 2

Transformer Block 1

Transformer Block 3

How

can

How

can can

(a)

help help

you you

(b)

Target model: Parallel verification

Draft model:

can can

Autoregressive generation

can can

hold hold

Model L

Input: How

Model router

Transformer Block

Model 2 can can

Transformer Block

Transformer Block Transformer Block

Transformer Block How How

can can

Transformer Block

I can can

can can

hold hold

Transformer Block

How

(c)

help hold

Transformer Block hold help

Model 1 help can can I hold

can can

Transformer Block

can

Transformer Block Transformer Block Transformer Block How How

can can

(d)

Fig. 3. Decoding strategies: (a) Non-autoregressive. (b) Early exiting. (c) Speculative decoding. (d) Cascade inference. Mechanism Nonautoregressive decoding Early exiting Speculative decoding Cascade inference Reasoning & agentic multi-LLM

4.2.2

Table 3. Comparison of decoding mechanisms for LLM edge inference Key Idea Advantages Limitations Generate tokens in parallel Stop at intermediate layers when conﬁdence is high Small draft model proposes, large model veriﬁes in parallel Route queries across small→large models; escalate if needed Add deliberation, tool/agent loops

Rep. works

Lower decoding latency, higher throughput

Possible quality drop, may need distillation/reﬁnement

[48, 52, 89]

Less compute per token; saves energy/latency

Needs reliable conﬁdence/exit policy

[28, 37, 135, 193]

Speedup without retraining target model

Depends on draft accuracy, extra draft cost

[13, 57, 145, 203, 209]

Lower average cost/latency, reduces oﬄoading/backhaul Improves correctness; reduces hallucinations

Requires routing + multiple models; misrouting overhead Higher latency/cost, coordination issues

[32, 93, 210, 211] [22, 155, 157, 160, 182, 183]

Early Exiting. Early exiting reduces latency and computational cost by skipping the computation of later lay-

ers instead of always executing the full model. It is commonly implemented by attaching lightweight exit heads at multiple depths and triggering an early stop based on conﬁdence signals such as entropy or margin, as shown in Fig. 3(b). Prior work in [47, 141, 154, 172] observed that hidden states often saturate at intermediate layers for some tokens, so early exits can match the full model’s top-1 prediction, motivating decoder-side early exiting. Building on this insight, CALM [135] performs token-wise adaptive compute during generation using conﬁdence- and saturation-based signals to select exit depth. However, token-level exiting can be hard to deploy with batched decoding because diﬀerent Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

sequences may require diﬀerent depths, creating stragglers and KV-cache complications. SkipDecode [28] addresses this by designing exit behavior that remains compatible with batching and KV caching while achieving substantial speedups. More recently, LayerSkip [37] trains LLMs for early-exit inference via layer dropout and an early-exit objective, and introduces self-speculative decoding that exits early and veriﬁes with remaining layers without a separate draft model. Beyond heuristic thresholds, ConsistentEE [193] learns exit policies via reinforcement learning to balance compute savings and accuracy. In edge deployments, exit policies should be co-optimized with compute and communication scheduling to meet end-to-end SLOs. For a broader discussion of early-exit design choices and trade-oﬀs, we refer readers to the surveys in [104]. 4.2.3

Speculative Decoding. Speculative decoding mitigates the sequential bottleneck of autoregressive generation

by having a small draft model propose multiple tokens per step, then letting the target LLM verify them in parallel, as shown in Fig. 3(c). Only tokens that pass the target LLM’s veriﬁcation are accepted, preserving generation quality. This approach can signiﬁcantly reduce latency without retraining or modifying the original LLM. Recent systems deploy draft and target models across heterogeneous edge nodes [209], split drafting on-device with veriﬁcation at the base station while transmitting only sparse logits [203], or use edge drafting with cloud veriﬁcation plus output sparsiﬁcation and quantization to cut end-to-end latency [13, 57]. To increase draft-token acceptance rate, [145] uses batch veriﬁcation and resampling over multiple draft sequences, preserving the target distribution and delivering an additional 1.37× speedup over standard speculative decoding. However, the eﬀectiveness of speculative decoding depends strongly on draft accuracy and the acceptance rate of predicted tokens, i.e., low-quality drafts may lead to frequent rejections and limiting the speedup of speculative decoding. Thus, balancing speed gains with reliability is essential. 4.2.4

Cascade Inference. Cascade inference handles varying user query diﬃculty by routing requests through a

hierarchy of LLMs with increasing capacity, exiting early when a smaller model is suﬃciently conﬁdent, as shown in Fig. 3(d). A trivial cascade runs a lightweight front-end LLM ﬁrst and escalates to larger LLMs when conﬁdence falls below a threshold. Several representative approaches demonstrate the eﬀectiveness of this paradigm. For instance, [32] leverages a learning-based router to adaptively route queries between a small LLM on the end device and a large LLM in the cloud, reducing calls to the large LLM by up to 40% without degrading response quality. Given limited memory and computation resources at the edge server, [93, 210] deploys multiple LLMs on the edge server to meet diverse user demands and uses a DRL-based method to learn the task scheduling policy for minimizing serving delay. A concurrent study [211] jointly optimizes model multiplexing and query caching, providing theoretical analysis of optimality in minimizing inference cost. Overall, cascade inference is a promising way for improving the eﬃciency for LLM edge inference, where most queries can be handled locally by a compact model, while only diﬃcult cases are oﬄoaded to more powerful devices or the cloud. Nonetheless, designing accurate dispatching mechanisms remains a key challenge, as poor routing decisions can compromise generation quality. 4.2.5

Reasoning and Agentic Multi-LLM Inference. While model compression and faster decoding improve eﬃ-

ciency, generation quality and reliability remain critical, especially at the mobile edge where hallucinations and inconsistent outputs can undermine user trust and downstream actions. Recent work augments decoding with reasoningenhanced inference that allocates extra deliberation to improve correctness, including step-by-step decomposition of problems such as chain-of-thought [160], sampling multiple reasoning traces and aggregating for consistency [157], searching over alternative reasoning trajectories [182], and grounding responses through retrieval or tool use [183]. Manuscript submitted to ACM

Chen et al.

These methods can reduce hallucinations through intermediate checks and evidence grounding, but often increase cost via longer contexts, multiple rollouts, or tool calls. Building on reasoning, agentic AI extends LLM serving from single-turn generation to autonomous multi-step task execution via iterative plan-act-observe loops with memory and tools. Multi-LLM agents further improve capability and eﬃciency by orchestrating multiple specialized models through delegation, routing, and iterative reﬁnement. For instance, [155] aggregates outputs from multiple agents in layered stages to enhance generation quality, while [22] proposes a scalable framework for discovering and coordinating heterogeneous agents in a distributed, internet-like setting. These approaches naturally align with edge intelligence by enabling tiered orchestration across device, edge, and cloud resources, where lightweight on-device agents handle local and privacy-sensitive steps and more capable edge or cloud agents perform retrieval or compute-intensive reasoning, balancing quality, latency, bandwidth, and privacy. However, they also introduce new challenges, including multi-step latency accumulation, coordination over time-varying wireless links, long-lived state and memory management, and governance for autonomous actions. 4.3

LLM Deployment at the Network Edge

In this subsection, we review LLM deployment techniques for the wireless network edge, including Transformer partitioning and placement, LLM caching and routing, and the disaggregation of preﬁll and decoding. 4.3.1

LLM Partition and Placement. Deploying LLMs under collaborative inference architectures (vertical, hori-

zontal, hybrid) in Section 3 requires partitioning the LLM and placing its components across devices, edge servers, and the cloud to meet latency and throughput targets under memory, bandwidth, and energy constraints. At the edge, eﬀective partitioning depends not only on FLOPs and compute capacity, but also on wireless channel quality, mobility, backhaul limits, and tight memory budgets. In practice, LLM partitioning strategies range from coarse Transformer block-level splits to ﬁne-grained layer-level placements. Transformer block-wise partitioning is the most widely used technique in LLM inference, splitting the model at the granularity of Transformer blocks, with a single Transformer block as the minimal unit. For example, [67, 79, 196] proposes dynamic programming-based methods that partition an LLM into consecutive Transformer blocks and deploy the resulting partitions in vertical device-to-cloud and horizontal edge server collaborative architectures, respectively. For horizontal settings, [201, 202] formulate LLM partitioning and placement as an integer linear program solved with Gurobi3 to match segments to device capabilities, while [120] casts partitioning as a matching game and ﬁnds a stable, low-latency assignment. To further improve system performance, ﬁne-grained techniques partition LLMs within Transformer layers, such as separating attention and MLP sublayers to ﬁt heterogeneous memory and bandwidth budgets [80], or sharding tensor dimensions of large projection matrices across devices, trading memory relief with extra inter-node communication [140]. Mixture-of-experts (MoE)-based LLMs further improve the quality-latency trade-oﬀ by activating only a few experts per token, but edge deployment is harder because expert weights are large, activations are skewed, and expert dispatch can amplify communication and load imbalance. These issues motivate communication-aware expert placement and caching [132, 142, 178], which partition MoE models at the expert level and distribute experts across edge devices to balance memory and compute demand. In addition, routing and batching should be expert-aware to reduce switching and maximize reuse, and load balancing is crucial to mitigate stragglers under heterogeneous wireless or interconnect conditions.

3 https://www.gurobi.com/solutions/gurobi-optimizer/

Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities 4.3.2

LLM Caching and Selection. In practice, edge deployments seldom rely on a single LLM due to the varying

user preferences and model capabilities. A portfolio of models with diﬀerent sizes and providers can be spread across end devices, edge servers, and the cloud. Under tight memory, compute, and bandwidth budgets, LLM caching and routing jointly decide which LLM instances stay resident at each tier, when to prefetch or evict, and how to dispatch each request to meet latency-accuracy targets under wireless and capacity constraints. Unlike conventional data caching that stores immutable content objects and serves exact matches by key, LLM caching means keeping selected LLM service instances (weights, segments, and possibly adapters) resident to avoid loading and cold-start overheads and to reduce cross-tier traﬃc. Its objective and constraints are therefore dominated by GPU memory residency, model loading or migration cost, and accuracy-latency trade-oﬀs, and it is naturally coupled with request routing and model selection under heterogeneous and time-varying edge resources. Towards this direction, [12] and [32] propose semantic-aware and learning-based routers that orchestrate inference between a lightweight LLM on the device and a larger LLM at the edge server to balance latency and quality, while [180, 210] develop DRL-based routers that dispatch user requests to appropriate edge-hosted LLMs. Moreover, [106] partitions an LLM into segments, replicates segments across edge devices, and formulates routing as a max-ﬂow problem with a heuristic that improves throughput by up to 3.3× over no-replication baselines. 4.3.3

Disaggregated Prefill and Decoding. Motivated by the two-phase asymmetry of LLM inference (Section 3.5),

disaggregated preﬁll and decoding separates preﬁll and decoding phases so each can be provisioned, parallelized, and scheduled on hardware matched to its bottlenecks. DistServe [207] routes preﬁll to throughput-oriented devices and decoding to memory-eﬃcient devices, handing oﬀ prompt state to reduce cross-phase interference and improve goodput. [185] proposes intra-sequence pipeline parallel planning for eﬃcient preﬁll and outline-based parallel decoding across collaborative edge nodes to overcome single-device limits, while SplitWiser [1] keeps both phases on the same device to avoid inter-device transfers and improve cache reuse, reducing network overhead and end-to-end latency. To handle ﬂuctuations in prompt and output lengths, Arrow [166] dynamically adjusts preﬁll and decoding instances using real-time cluster metrics, substantially improving resilience to traﬃc spikes and load variation. Overall, phaseaware disaggregation with lightweight state movement improves throughput, tail latency, and hardware eﬃciency. It is particularly eﬀective at the edge, where preﬁll can burst to near-edge or cloud resources while decoding runs locally to deliver low token latency, save energy, and preserve privacy. 5 Resource Management and Scheduling Beyond model optimization and deployment, eﬃcient LLM edge inference hinges on coordinated resource management and scheduling. This section surveys cross-layer techniques spanning parallel computing, memory management, communication optimization, and their joint optimization. 5.1

Computation Scheduling

This subsection reviews two computation scheduling techniques, namely batching and parallel computing. 5.1.1

Batch Optimization. Batching groups multiple input requests for simultaneous processing, fully exploiting

the parallelism of modern accelerators such as GPUs and TPUs to boost throughput and hardware utilization. Common batching strategies are summarized below, with a comparison in Table 4. Manuscript submitted to ACM

Chen et al.

Technique Static batching Dynamic batching Continuous batching Chunked preﬁll

Table 4. Comparison of batching techniques for LLM edge inference Goal Key Idea Overheads High queueing delay, poor Wait until batch ﬁlls, then Max throughput under bursty arrivals and run wireless jitter Head-of-line delay with Balance latency and Run when batch size or variable lengths, sensitive to throughput timeout is met arrival variability High utilization Admit/ﬁnish requests at any More complex scheduling and (variable lengths) decoding step KV management Needs chunk-size tuning, Better long-prompt Split long prompts and extra scheduler/kernel eﬃciency overlap preﬁll with decoding overhead

Rep. works –

[137, 164, 194] [63, 85, 189] [4, 34]

1) Static Batching: This form waits for a ﬁxed number of requests to arrive and ﬁlls the batch before processing begins. In general, static batching works well in scenarios with controlled request rates or oﬄine inference scenarios where all inference tasks are arrived before inference starts, but it can cause unbounded delays in the online inference scenarios where requests typically arrive irregularly. 2) Dynamic Batching: Dynamic batching bounds the delay of static batching by launching a batch when either a target batch size is reached or a time window expires. The batch size and time limit can be conﬁgured to balance throughput and latency according to workload demands. It is widely used in DNN inference with stochastic arrivals (e.g., [137, 194]) and extended to LLM serving frameworks such as vLLM [85] to balance responsiveness and throughput. However, these approaches batch requests in First-Come-First-Served (FCFS) order, which can be suboptimal because early long requests block shorter ones and inﬂate latency. To this end, [164] adopts a skip-join multi-level feedback queue that prioritizes high-priority requests and preempts long-running tasks to accelerate short ones, improving throughput by up to 31.4% under latency constraints over vLLM. A key limitation is that new requests can be admitted only after the current batch ﬁnishes. Dynamic batching works well when requests have similar runtimes, as in conventional DNNs or encoder-only LLMs. For autoregressive LLMs, highly variable output lengths force all requests in a batch to wait for the longest one, causing idle GPU time and reducing eﬃciency. 3) Continuous Batching: This technique extends dynamic batching by allowing requests to join or exit a batch at any decoding step, rather than waiting for the entire batch to complete. It is well suited to autoregressive LLMs with highly variable output lengths, since it prevents shorter queries from being blocked by longer ones. By overlapping preﬁll and decoding and reusing cached KV states, continuous batching improves GPU utilization and reduces head-of-line blocking compared to static or dynamic batching. Orca [189] implements this approach via iteration-level scheduling and selective batching, where the former constructs batches at every decoding step to accommodate new arrivals, while the latter targets only batchable transformer operations, achieving 36.9× higher throughput than conventional batching at the same latency level. Modern LLM serving frameworks such as vLLM [85] and DeepSpeed-FastGen [63] have adopted continuous batching as a core mechanism to achieve high throughput while maintaining interactive latencies. However, continuous batching requires sophisticated scheduling to handle challenges such as memory fragmentation in the KV cache and fairness across heterogeneous requests. 4) Chunked-Preﬁll: This technique targets the ineﬃciency of processing long input prompts by splitting them into smaller segments that are processed incrementally. The decoding for the ﬁrst segment can start immediately while subsequent segments undergo preﬁll, allowing these phases to overlap and reducing peak memory usage. Building on this idea, Sarathi-Serve [4] splits prompts into near-equal sized chunks and creates stall-free schedules that add new Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

requests in a batch without pausing ongoing decoding, achieving 3.7× higher serving capacity than vLLM. [34] further proposes a dynamic adaptive chunk-based prefetching strategy to overcome the resource and latency limitations inherent in static chunking approaches, improving throughput by 24.4% over Sarathi-Serve. Chunked-preﬁll is especially promising in edge environments because it enables serving long-context queries without overwhelming limited hardware resources. However, too many small chunks can increase scheduler and kernel overhead, making chunk-size tuning critical. 5.1.2

Parallel Computing. LLMs often have billions to trillions of parameters, making compute and memory key

barriers to deployment at the resource-constrained edge. Parallel computing mitigates these limits by exploiting hardware and algorithmic concurrency, distributing work across cores or devices to accelerate inference. Below, we review parallelization strategies, as shown in Fig. 4, with a comparison in Table 5. Technique DP TP PP Hybrid parallelism

Table 5. Comparison of parallelism computing techniques Key Idea Overheads Replicate model, split Needs load balancing, routing Scale throughput requests across replicas overhead across nodes Communication-heavy, Fit/accelerate large Split matrix operations performance depends on layers across devices interconnect quality Fit model across Split layers into stages, Bubble overhead, activation devices, overlap stages micro-batch pipeline transfers between stages Combine scalability + Combine DP/TP/PP to More complex orchestration, ﬂexibility match heterogeneity higher coordination overhead Goal

Rep. works [114, 129] [123, 140, 153] [79, 120, 196] [40, 205]

1) Data Parallelism (DP): DP replicates the full model across multiple devices and distributes inference requests such that each device processes its assigned inputs independently [114, 129], as illustrated in Fig. 4(a). It is simple to implement with minimal framework changes, but edge deployments require careful workload scheduling and batching to meet performance targets under the constraints in Section 3.5. 2) Tensor Parallelism (TP): TP shards compute-intensive LLM operations (e.g., matrix multiplications) across devices, then merges the partial results using collectives such as all-reduce or all-gather [123, 140, 153]. For instance, Fig. 4(b) shows a case where three devices collaboratively computing X × A = Y by sharding A across the devices in a columnwise manner (row-wise is also possible) and then aggregating the partial results. By distributing large computations, TP accelerates inference and reduces per-device memory footprint. Nevertheless, frequent inter-device communication can impose signiﬁcant overhead, and poor partitioning strategies may lead to load imbalance and reduced eﬃciency. 3) Pipeline Parallelism (PP): PP partitions an LLM into multiple stages, with each stage comprising one or more consecutive model layers. Each device is assigned a pipeline stage and is responsible for executing its corresponding layers [68, 186, 202], as shown in Fig. 4(c). During inference, input requests are partitioned into micro-batches that sequentially traverse the pipeline stages, with each device forwarding its computed activations to the device responsible for the next stage. This arrangement reduces per-device memory usage by distributing model layers across devices and can improve throughput by overlapping computation across micro-batches. Motivated by these advantages, PP has been widely investigated for horizontal edge collaborative inference scenarios, such as [79, 120, 196]. 4) Hybrid Parallelism: Hybrid parallelism combines DP, TP, and PP to leverage their complementary strengths while alleviating individual limitations [40, 205]. For instance, within a device group, TP can be applied to split large matrix multiplications across GPUs, while PP distributes layers across groups, and DP replicates these groups to process independent requests in parallel. This multi-dimensional design enables scaling to extremely large LLMs that exceed Manuscript submitted to ACM

Chen et al.

the memory capacity or compute limits of any single parallelism method alone. At the wireless edge, hybrid schemes help handle device heterogeneity and network variability by balancing work within high-bandwidth local clusters with TP or PP while using DP across geographically distributed nodes to raise throughput. Scheduler

Request queuing

Batch 2

Batch 3

Y Scheduler

Request queuing

Device 2

Device 3

Device 1

Batch 1

All reduce

Device 1

Scheduler

Batch 1

Device 2

Request queuing

Batch 1

Batch 2 Batch 1

Device 3

Device 2

Device 1

Device 2

Batch 2

Device 1

1 2 3

Device 2

Batch 3

Device 3

Device 3 Device 3

Time

Model replica 1

Model replica 2

Model replica 3

(a) Data Parallelism

(b) Tensor Parallelism

Fig. 4. Comparison of diﬀerent parallelism methods.

5.2

Memory Management

The runtime memory footprint in LLM inference mainly stems from model weights, activations, and the KV cache. Weights are static once deployed and activations are relatively small and predictable, whereas the KV cache grows with context length during autoregressive decoding and often becomes the dominant term. Eﬀective memory management is therefore essential, otherwise context length, batch size, and throughput are capped. Below we review two complementary mechanisms, i.e., memory allocation and memory oﬄoading, with a comparison in Table 6. Technique Fixed KV pre-allocation Paged KV cache Paged KV kernel and layout co-design Preﬁx reuse / tree attention Token-level KV management Memory oﬄoading

5.2.1

Table 6. Comparison of memory management techniques Key Idea Overheads Wastes memory, lowers Avoid OOM, Reserve contiguous KV for concurrency with variable simplify runtime max length lengths Reduce Store KV in noncontiguous Allocator/bookkeeping overhead, fragmentation “pages” irregular access Goal

Improve paged eﬃciency Reuse shared KV Fine-grained KV utilization Fit larger models/ context

Co-design KV layout and attention kernels

Implementation complexity, hardware-speciﬁc tuning

Share preﬁxes across candidates/branches Manage KV at token granularity to reduce waste Place weights/activations/KV across devices

Extra control logic, workload dependent Higher bookkeeping overhead, gains depend on workload Data transfer overhead, sensitive to bandwidth

Rep. works [67, 79, 132, 140, 178, 196] [85] [187] [108] [19, 165, 208] [69, 139]

Memory Allocation. Considering the KV cache grows and shrinks dynamically during decoding, many LLM

edge inference studies, e.g., [67, 79, 80, 132, 140, 142, 178, 195, 196], pre-allocate a contiguous piece of memory with a ﬁxed maximum sequence length to avoid OOM errors. However, this wastes signiﬁcant memory by holding reserved memory until each request completes, blocking reuse and reducing eﬀective system capacity. To mitigate fragmentation and overprovisioning, vLLM [85] introduces PagedAttention, which partitions the KV cache into noncontiguous pages and stores it in a paged format similar to an operating system, improving throughput by 2-4× over Orca. To counter the irregular memory accesses introduced by paged layouts, FlashInfer [187] co-designs KV-cache data layouts and attention-kernel access patterns for paged caches, eﬃciently mapping virtual to physical pages to restore contiguous, vectorized loads. SpecInfer [108] uses tree attention with depth-ﬁrst traversal to reuse shared preﬁxes across multiple Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

output sequences and eliminate redundant KV allocations. [19, 165, 208] push granularity further by introducing tokenlevel KV cache management to reduce waste. However, these fragmented memory management introduces allocator and bookkeeping overheads. When other optimizations already increase batch size, the additional throughput gains can be limited while latency overhead becomes signiﬁcant. 5.2.2

Memory Oﬀloading. While memory allocation reduces the footprint on a single device, the memory demands

of LLM inference often exceed the capacity of one device. Consequently, memory oﬄoading that distribute weights, activations, and the KV cache across tiers such as CPU memory, local storage, and even remote devices are essential to meet capacity goals [77]. On a single machine, FlexGen [139] formulates a linear program to place weights, activations, and KV across GPU, CPU, and disk, enabling single-GPU inference of OPT-175B under tight budgets and achieving a generation throughput of 1 token/s with a batch size of 144 for the ﬁrst time. TightLLM [69] uses an adaptive oﬄoading policy that balances data transfers with compute, recomputes parts of the KV cache to overlap data movement, and slices weight loading across batches to amortize loading overhead, signiﬁcantly improving throughput. Beyond a single device, weights, activations, and KV caches can be oﬄoaded to peer edge devices or even to the cloud. 5.3

Communication Eﬀiciency Enhancement

For LLM edge inference, communication is often the main bottleneck for latency and energy targets because prompts, retrieved context, and intermediate activations are transmitted among devices, edge servers, and the cloud. The bandwidthlimited uplinks and time-varying channels aﬀected by fading, interference, mobility, and blockage can increase delay and retransmissions. These constraints motivate communication-aware designs beyond generic compression. Below, we review three promising techniques to address this bottleneck. 5.3.1

Semantic Communication. In LLM edge inference, end devices may need to transmit high-volume data, such

as prompts and activations, to peer devices or edge servers, leading to substantial communication overhead, especially with long prompts and large preﬁll activations. Semantic communication reduces this cost by transmitting compact, task-aware representations of raw inputs, intermediate features, or textual context instead of source-agnostic bitstreams, while largely preserving downstream utility and generation quality [33, 204]. In extended reality (XR) applications and multimodal mobile assistants, this helps because compact semantics shrink uplink payloads and reduce end-to-end latency. In vehicle-to-everything (V2X) perception and cooperative sensing, this helps because exchanging only task-relevant information scales better under multi-user contention. However, designing semantic encoders and decoders that handle data and device heterogeneity, time-varying wireless conditions, and privacy constraints while meeting strict latency, accuracy, and energy targets remains challenging and warrants further investigation. 5.3.2

Over-the-Air Computation. Over-the-air computation (AirComp) exploits the analog superposition property

of wireless multiple access channels, allowing many devices to transmit simultaneously so that the receiver directly obtains a desired function of their signals, typically a weighted sum. Recent work in [195] applies AirComp to distributed on-device LLM inference with tensor parallelism and shows that analog all-reduce can signiﬁcantly reduce end-to-end inference delay compared with digital baselines. AirComp is particularly useful for industrial IoT and smart factories, where many devices can aggregate updates or intermediate results in one shot, reducing scheduling overhead and latency. For UAV swarms and V2X collaboration, the simultaneous transmissions can accelerate collective inference as many participants must synchronize frequently. However, AirComp is sensitive to synchronization errors, Manuscript submitted to ACM

Chen et al.

imperfect channel estimates, and hardware nonidealities. It therefore requires careful calibration, power control, and device grouping.

5.3.3

Spatial-domain Technologies. In addition to semantic communication and AirComp, spatial-domain tech-

nologies such as multiple-input multiple-output (MIMO) [88, 102], reconﬁgurable intelligent surfaces (RIS) [71], and programmable antennas [163] can shape wavefronts to increase spectral eﬃciency, enhance reliability, and shorten end-to-end transmission time between edge devices. These gains translate directly to faster transmission of prompts and activations between edge devices, and lower energy consumption. They are particularly valuable in dense, multiuser deployments, where massive MIMO enables spatial multiplexing of numerous uplink transmissions, and in highfrequency or blockage-prone environments, where RIS and antenna reconﬁguration can mitigate blockages for mobile XR or V2X links. However, spatial-domain techniques increase system complexity because they require accurate and timely channel state acquisition, pilot and control overhead, RIS reconﬁguration latency, and hardware calibration, all of which must be carefully managed to preserve end-to-end latency beneﬁts.

5.4

Joint Optimization of Communication, Computation, and Memory

LLM edge inference is constrained by tightly coupled budgets of communication, computation, and memory. Communication cost depends on what is transmitted (prompts, activations, KV state) and how often, computation depends on batching, parallelism, and per-token workload, and memory limits model size, context length, and concurrency. Optimizing any single resource in isolation often harms another. For example, larger batches improve accelerator utilization but increase KV memory pressure and queueing delay, while oﬄoading reduces on-device compute but can raise communication latency. These couplings motivate coordinated optimization across all three resources to meet latency, throughput, and energy targets. Prior work in [209] combines speculative decoding with pipeline parallelism to overlap drafting and veriﬁcation across concurrent tasks, and then uses dynamic programming to jointly optimize speculation length, batching, and communication and memory allocation. In [200], batching, model quantization, and communication and computation provisioning are co-optimized to improve throughput on resource-constrained edge servers. [50] further overlaps communication and computation by splitting each batch into two wave-aware token subsets. Recent works also show that phase disaggregation enables more ﬂexible joint optimization. DistServe [207] separates preﬁll and decode onto diﬀerent GPU pools and co-optimizes phase-speciﬁc resource allocation and parallelism under latency constraints while accounting for disaggregation-induced communication. Follow-up work further explores ﬁne-grained scheduling and resource partitioning for phase-disaggregated serving [41, 64]. Despite the progress, joint optimization of communication, computation, and memory for LLM edge inference remains at an early stage. Further work is needed on online and robust optimization under time-varying wireless channels, cost models that capture preﬁll and decode asymmetry and KV growth, and cross-layer mechanisms that enforce strict latency and energy targets without sacriﬁcing service quality.

6 Evaluation Methodology This section outlines the evaluation methodology for LLM edge inference, covering metrics, models, datasets, and platforms to support rigorous evaluation and deployment. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

6.1

Metrics

6.1.1

Latency. Latency measures how quickly an LLM inference system responds and delivers results to a user re-

quest. The key metrics to measure latency are: 1) Time to First Token (TTFT): TTFT [1, 119, 143, 207] is the elapsed time from when an LLM request is accepted by the serving system to when the ﬁrst output token is generated (or begins streaming to the user). It captures how quickly the system starts responding. TTFT typically includes (i) request queueing, (ii) preﬁll/encoding (e.g., tokenization and attention over the full prompt to build the KV cache), and (iii) network/coordination delays in distributed edge execution. TTFT generally increases with prompt length since the full input must be processed before decoding can begin. 2) End-to-End Latency (E2EL): E2EL [1, 119, 164, 189, 202, 205] is the elapsed time from request submission to receiving the complete response, and thus directly determines perceived responsiveness. It spans the full serving pipeline, including queueing, batching/scheduling, preﬁll and decoding, post-processing, and any network latency in split or multi-edge deployments. Notably, a low TTFT does not imply a low E2EL, since slow token generation can still dominate the time to completion. 3) Token Generation Time (TGT): TGT [187] is the elapsed time to stream all output tokens after the ﬁrst token is produced. It excludes TTFT and measures only the steady generation phase. Formally, TGT = E2EL − TTFT. 4) Time per Output Token (TPOT): TPOT [1, 4, 164, 207] is the average inter-token latency during the steady generation phase, i.e., after the ﬁrst token is produced. It captures how quickly the serving system emits subsequent tokens. Lower TPOT corresponds to faster token production and higher throughput. 6.1.2

Throughput. Throughput characterizes how much useful work an LLM inference system delivers per unit time

[4] and is typically reported in three forms: 1) Tokens per Second (TPS): TPS [139, 164, 189, 196, 202] quantiﬁes system throughput, i.e., the number of tokens processed per second across all concurrently served requests over a measurement window. In practice, TPS often reported separately for preﬁll (prompt processing) and decode (iterative generation), as the two phases exhibit diﬀerent bottlenecks. TPS generally increases with larger batch sizes and higher parallelism, but may come at the cost of higher queueing delay and worse tail latency. 2) Requests per Second (RPS): RPS [58, 67, 110, 200] measures the number of requests an LLM serving system completes per second over a measurement interval, reﬂecting its capacity to sustain concurrent workloads. Unlike TPS, which scales with response length, RPS is request-centric and thus depends on the mix of prompt/output lengths, decoding settings (e.g., max tokens), and system policies such as batching, scheduling, and admission control. 3) End-to-End Throughput (E2ET): E2ET [189, 200] quantiﬁes the number of tasks completed per unit time. It reﬂects the eﬀective throughput of the entire serving pipeline, from request arrival to response completion, including admission and queueing, batching and scheduling, preﬁll and decoding, post-processing, and any communication overhead in split or multi-edge deployments. 6.1.3

Goodput. Goodput extends the classic networking notion of “useful throughput” to LLM serving by measuring

how many requests per second the LLM successfully completes while meeting predeﬁned SLOs, such as TTFT, endto-end latency, and timeout constraints [158, 207]. By construction, goodput ≤ throughput, since it excludes errors, timeouts, policy blocks, and responses that violate the SLO. Goodput is particularly important at the edge because limited compute and memory, constrained bandwidth, energy budgets, and network variability can cause queueing Manuscript submitted to ACM

Chen et al.

and tail-latency spikes, reducing the share of SLO-compliant requests even when raw TPS or RPS is high. Improving goodput therefore requires SLO-aware control rather than simply maximizing utilization. Common approaches include admission control and load shedding to avoid queue buildup, latency-aware batching and scheduling, KV-cache-aware concurrency limits and cache management, adaptive model or precision selection and speculative decoding to speed generation under tight budgets, and oﬄoading or split execution that trades compute savings for communication delay. 6.1.4

Energy Eﬀiciency. Energy eﬃciency quantiﬁes how much electrical energy is consumed to produce a re-

sponse, which can be measured as following metrics: 1) Tokens per Joule (TPJ): It characterizes the number of output tokens delivered per unit energy [162]. 2) Requests per Joule (RPJ): It measures the number of fully completed requests per unit energy [91]. 3) Total Energy Consumption: It measures the absolute electrical energy consumed by the LLM inference system to process a given workload [70, 143, 181]. Conceptually, TPJ and RPJ are the energy-normalized counterparts of TPS and RPS, i.e., TPS/RPS express work per unit time, whereas TPJ/RPJ express work per unit energy. For completeness, energy analogs of latency metrics can also be deﬁned, such as energy to ﬁrst token and energy per request, as well as composite energy-delay measures. These metrics are crucial because edge deployments are constrained by battery capacity and local power budgets. ETSI MEC standards [39] treat energy and delay as coequal KPIs for evaluating edge systems, underscoring the importance of energy-eﬃcient LLM serving. 6.1.5

Memory. Unlike compute limits, memory pressure aﬀects feasibility (whether the model and KV cache can

reside on-device), speed (decoding stages that are bandwidth- and cache-bound), and stability (OOM failures). Below, we summarize the key memory metrics that govern edge deployability and performance. 1) Model Footprint: It is the static memory required to store model parameters on-device [1, 85, 139]. It determines basic deployability, aﬀects cold-start time, and limits the memory available for runtime state. The footprint is governed by model architecture (e.g., hidden size and number of layers), parameter precision (e.g., FP16, INT8), and compression methods such as quantization, pruning, and low-rank factorization. 2) KV Cache Footprint: It is the memory required to store key and value tensors for all processed tokens during inference [1, 85, 139]. It scales roughly linearly with context length (prompt plus generated tokens), the number of concurrent requests, model architectural factors, and it is proportional to the storage precision. The footprint can be moderated by KV compression, oﬄoading to slower memory, and eviction or sliding-window policies that cap growth at the cost of long-range context. 3) Peak Inference Memory: It characterizes the maximum resident memory observed during the two main phases of LLM serving, i.e., preﬁll and decoding [1, 85, 139]. Preﬁll peaks are largely driven by activations and temporary kernel workspaces (e.g., attention and GEMM buﬀers), while decoding peaks are typically dominated by the KV cache and attention buﬀers. Peak inference memory therefore determines the minimum device capacity needed to avoid OOM and constrains the achievable concurrency for a given context length. 6.1.6

Inference Quality. Beyond eﬃciency (latency, memory, energy), LLM edge inference systems must also be

evaluated for output quality. Because deployment optimizations and compression can change model behavior, designs should explicitly balance eﬃciency and quality. We summarize key quality metrics below. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

1) Task quality metrics: These metrics quantify how often model outputs match the expected answer under a benchmark protocol. For classiﬁcation-style benchmarks, accuracy [26, 60, 192] is the fraction of examples predicted correctly. For question answering, Exact Match (EM) [5, 27] is the fraction of predictions that exactly match the reference answer after normalization, and token-level F1 [11] is the harmonic mean of token precision and recall between prediction and reference. For generation tasks, ROUGE and BLEU [11, 61, 112] measure 𝑛-gram overlap between generated and reference texts. For code generation, pass@k [8, 20] is the probability that at least one of 𝑘 sampled programs passes unit tests. 2) Hallucination: Hallucinations are ﬂuent outputs that are incorrect or unsupported by the given context [96]. They are often measured by hallucination rate (the fraction of outputs with false or unsupported claims) or dataset-speciﬁc factuality scores (the fraction judged factually correct), and are especially important for compact or aggressively compressed edge models. 3) Bias and toxicity: Bias evaluations quantify stereotypical associations or performance disparities across demographic groups, often reported as a stereotype score or group disparity (performance gap between groups) [111]. Toxicity is typically reported as a toxicity rate, deﬁned as the fraction of outputs ﬂagged as toxic by a classiﬁer or human annotation [45]. Optimizing a single metric in isolation can be misleading for edge LLM inference because the metrics are tightly coupled. Batching and parallelism may increase TPS or RPS, but often increase queueing delay and worsen tail TTFT or E2EL. Higher concurrency or longer context increases memory demand, raising TPOT and potentially causing latency spikes or OOM failures. Higher utilization can boost raw throughput while reducing goodput when more requests violate SLOs. Compression and acceleration can improve latency, throughput, or energy eﬃciency, but may degrade output quality or robustness depending on the conﬁguration. Oﬄoading or split execution can reduce ondevice compute and energy, yet introduces communication delay and variability that can harm tail latency. These interactions motivate SLO- and resource-constrained optimization to identify Pareto-eﬃcient operating points that balance latency, goodput, energy, and memory under edge hardware and network constraints. Table 7. Representative Open LLMs for Edge Inference Evaluation Scales (B) Provider Key Features General-purpose baselines with broad tooling support, LLaMA / Variants [149] 1-70 Meta long-context variants available 7-8 (dense), Mistral / Mixtral [75, 76] Mistral AI Optimized attention mechanisms, MoE architecture 8×7 (MoE) Compact architectures with optimized kernels, Gemma / Gemma 2 [147] 2-27 Google compatible with speculative decoding Multilingual and tool-use capabilities, long-context Qwen / Qwen2 [73, 148, 179] 0.5-72 Alibaba variants available Phi-2 / Phi-3 [2] 2-7 Microsoft Distilled-style models, friendly for edge devices OPT [198] 1-66 Meta Transparent architectures and training details TinyLLaMA / Variants [197] 1-3 Community Mobile/embedded focus, support aggressive quantization Code-specialized models with domain-adapted DeepSeek-Coder [55] 1.3, 6.7, 33 DeepSeek tokenizers, long-context variants available Model Family

6.2

Models and Datasets

The choice of models and datasets is central to evaluating LLM inference on edge platforms, since both directly determine the representativeness and comparability of results. Manuscript submitted to ACM

Chen et al. On the model side, most evaluation studies focus on widely adopted open-source families such as LLaMA [149],

Mistral (including Mixtral) [75, 76], Gemma [147], and Qwen [73, 148, 179]. Table 7 lists representative models commonly used for inference evaluation, all available on Hugging Face. Their architectural features make them particularly relevant for diﬀerent aspects of inference evaluation on edge platforms. For example, the availability of long-context variants supports systematic testing of retrieval, summarization, and memory scaling. Lightweight distilled or quantized models are well suited for evaluating trade-oﬀs between accuracy, latency, and energy eﬃciency under mobile or embedded constraints. MoE models (e.g., Mixtral [76]) highlight tokens-per-watt eﬃciency and reveal how expert activation aﬀects KV-cache growth and scheduling, making them valuable for throughput and concurrency evaluation. In addition to models, evaluation of LLM inference at the edge relies on datasets that capture both task semantics and workload characteristics. General reasoning and knowledge are typically measured using MMLU [60], HellaSwag [192], and ARC [26], which feature short prompts and short outputs suited to classiﬁcation-style evaluation. Mathematical reasoning datasets such as GSM8K [27] and MathQA [5] often require CoT prompting, producing longer outputs that stress both accuracy and inter-token latency. Code generation benchmarks like HumanEval [20] and MBPP [8] follow a short-input/long-output pattern, making them appropriate for assessing throughput. For long-context evaluation, suites such as LongBench [11] and RULER [66] substantially extend prompt length, enabling systematic study of memory footprint and KV-cache scaling. Summarization tasks (e.g., CNN/DailyMail [61], XSum [112]) represent long-input/long-output scenarios that reﬂect document-level generation workloads. Table 8 summarizes representative datasets frequently used in evaluation. While not exhaustive, this collection highlights the most widely adopted benchmarks and illustrates how diﬀerent dataset types align with speciﬁc evaluation dimensions. Table 8. Representative Datasets for LLM Inference Evaluation Notes: Sequence classes are deﬁned based on input and output length. Short input: prompt length ≤ 1000 tokens; long input: > 1000. Short output: output length ≤ 128 tokens, long output: > 128. Accordingly, 𝑆 ↔ 𝑆 denotes short input & short output, 𝑆 ↔ 𝐿 short input & long output, 𝐿 ↔ 𝑆 long input & short output, and 𝐿 ↔ 𝐿 long input & long output.

6.3

Dataset

Task Focus

Prompt length / Output length

MMLU [60] HellaSwag / PIQA [192] ARC-C / ARC-E [26] TruthfulQA [96] XSum [112] GSM8K [27] MathQA [5] HumanEval [20] MBPP [8] LongBench (Retrieval/QA) [11] LongBench (Summ.) [11]

Knowledge/reasoning Commonsense Grade-school reasoning Factuality Summarization Math word problems Math QA Code generation Code (beginner) Long-context retrieval Summarization

50-400 / 1-10 50-200 / 1-5 50-300 / 1-10 50-200 / 5-50 400-1k / 20-80 30-200 / 50-300 (CoT) 50-300 / 30-200 30-200 / 50-300 30-150 / 50-250 2k-16k / 20-150 4k-16k / 150-400

RULER [66]

Context-length stress

4k-32k+ / 5-200

CNN/DailyMail [61]

Summarization (news)

600-1.5k / 120-300

Sequence Class 𝑆 ↔𝑆 𝑆 ↔𝑆 𝑆 ↔𝑆 𝑆 ↔𝑆 𝑆 ↔𝑆 𝑆 ↔𝐿 𝑆 ↔𝐿 𝑆 ↔𝐿 𝑆 ↔𝐿 𝐿 ↔𝑆 𝐿↔𝐿 𝐿 ↔𝑆 / 𝐿↔𝐿 𝐿↔𝐿

Evaluation Platforms

Evaluating LLM edge inference requires both simulation frameworks and real-device testbeds, as each oﬀers complementary insights. Simulations enable rapid, low-cost exploration of design alternatives under controlled conditions, whereas physical deployments validate performance under realistic constraints such as wireless resources, energy, memory. Table 9 summarizes the main evaluation platforms. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

Table 9. Representative Evaluation platforms and software stacks for edge LLM inference Category Rep. tools/platforms Evaluation focus gem5 [14], Timeloop [117], Processor/accelerator pipelines and memory Hardware simulation NVSim [36] hierarchy behavior TVM [21], PyTorch proﬁlers, Kernel scheduling, operator fusion, quantization System simulation or proﬁling MLPerf [105] eﬀects, backend behavior Split/distributed inference under bandwidth, RTT, Network emulation ns-3 [59], Mininet [87] and packet loss constraints Smartphones; Raspberry Pi; End-to-end performance under realistic compute, NVIDIA Jetson Orin Nano/AGX; Real-device testbeds (hardware) memory, energy, and thermal constraints GPUs/accelerators llama.cpp [46], Execution eﬃciency and system-level behavior Real-device software stack TensorRT-LLM [115], vLLM [85] across backends and accelerators

Simulation-Based Evaluation: Simulation is widely used to explore hardware and system design choices prior to deployment. At the hardware level, simulators such as gem5 [14], Timeloop [117], and NVSim [36] enable detailed modeling of processor pipelines, accelerator dataﬂows, and memory hierarchies, providing cycle-accurate estimates of latency and energy consumption. At the system level, frameworks including TVM [21], MLPerf [105], and proﬁling tools within PyTorch support evaluation of kernel scheduling, quantization eﬀects, and operator fusion strategies across heterogeneous backends. For distributed or split inference scenarios, network emulators such as ns-3 [59] and Mininet [87] are widely adopted to study sensitivity to bandwidth limitations, round-trip time (RTT), and packet loss. Real-Device Evaluation: It is indispensable for capturing end-to-end performance, energy consumption, and user experience under realistic operating conditions. Representative hardware platforms span a wide spectrum of form factors: smartphones, edge AI boards (e.g., Raspberry Pi and NVIDIA Jetson Orin Nano), and edge servers equipped with GPUs or accelerators. On the software side, a variety of inference engines are employed, including llama.cpp [46], TensorRT-LLM [115], vLLM [85], and ONNX Runtime [116]. These are often complemented by vendor-speciﬁc SDKs such as Apple Core ML [6] and Qualcomm AI Engine Direct[126], which provide optimized execution paths for on-device accelerators. Benchmarking harnesses such as MLPerf [105], Hugging Face Optimum [72], and custom token-latency proﬁlers are widely used to quantify key metrics, including TTFT, TPS, and memory footprint across standardized datasets. 7 Furture Research Directions As LLM inference at the network edge rapidly evolves, it confronts a set of open challenges that deﬁne the next wave of research. Below we outline three future promising directions. Scalable Inference for Multi-Model and Multi-modal LLMs: Most existing LLM serving frameworks are optimized for a single, text-only model, which limits perception and application scope. In practice, edge deployments increasingly require multimodal LLMs, e.g., Qwen2-VL [148], LLaVA-1.5 [98], that integrate text with images, audio, video, and sensor streams, as well as multi-model pipelines that combine a general-purpose LLM (e.g., LLaMA [149]) with compact specialists for retrieval or tool use. Supporting these settings calls for end-to-end co-design across model selection, data pipelines, orchestration, scheduling, and runtime placement to meet tight edge budgets on memory, energy, and compute while maintaining service quality. From a wireless communications perspective, scalable multi-model/multimodal inference can enable emerging services such as XR and interactive agents, V2X and intent understanding, and UAV/robot coordination. It can also enhance network intelligence by jointly reasoning over heterogeneous telemetry (e.g., logs, traﬃc traces, and spectrum measurements) for anomaly detection and policy-driven conﬁguration. Manuscript submitted to ACM

Chen et al. Secure LLM Edge Inference: Security threats widely observed in cloud LLM serving, such as prompt injection,

jailbreaks, and data leakage, are often ampliﬁed at the wireless edge, where devices are user-owned, physically accessible, intermittently connected, and frequently shared in multi-tenant settings [130]. From an edge threat model perspective, attacks can be classiﬁed into: 1) Integrity attacks that manipulate behavior (e.g., prompt injection, jailbreaks, and tool abuse). 2) Privacy leakage of prompts, retrieved context, outputs, and runtime states (e.g., KV caches), including leakage via logs or improper state retention. 3) Model extraction and theft via repeated queries or compromised storage of weights/adapters. and 4) Side-channel leakage (e.g., timing and cache/memory contention) that is exacerbated by co-resident multi-tenancy. Conventional data-center defenses such as role-based access control [44] and multi-factor authentication [131] provide a baseline, but edge deployments additionally require edge-tailored mitigations, including prompt/retrieval sanitization with strict tool permissioning, encrypted communication and secure storage, zeroization with explicit retention policies, and side-channel-aware resource management. These protections are particularly critical in regulated domains such as healthcare and ﬁnance, and are a prerequisite for deploying LLMs in wireless workﬂows such as enterprise edge copilots and network automation. Green LLM Edge Inference: With the ubiquitous deployment of LLMs, the inference demand is ever-expanding. Since these models are both compute- and memory-intensive, energy availability is now a primary bottleneck for capacity growth in data centers. This problem is further ampliﬁed at edge environments as energy is the binding limit for edge devices such as phones and embedded boards. A public estimation [51] report that an average ChatGPT query consumes about 0.34 Wh, and that daily usage is comparable to the electricity consumed by roughly 180000 U.S. households, underscoring a looming “energy wall" for LLM serving. This makes green, sustainable LLM edge inference a necessity, otherwise models that meet latency targets in controlled tests will throttle in the ﬁeld, drain batteries, or become too expensive and carbon-intensive to scale. From a wireless communications perspective, green LLM edge inference is essential for enabling always-on, sustainable on-device applications such as radio access network automation, intelligent network controllers, and wearables and body area networks. 8 Conclusion This survey presents a comprehensive overview of recent advances in enabling LLM edge inference, systematically exploring system architectures, including single-edge-node, vertical, horizontal, and hybrid collaboration. Meanwhile, we review emerging techniques across architecture, model optimization, deployment, and resource management that improve performance under tight memory, bandwidth, and energy budgets. We also curate methodologies for rigorous, reproducible evaluation and highlight future directions toward greater scalability, security, and sustainability. References [1] Asad Aali, Adney Cardoza, and Melissa Capo. 2025. Splitwiser: Eﬃcient LM inference with constrained resources. arXiv preprint arXiv:2505.03763 (2025). [2] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024). [3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [4] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, et al. 2024. Taming Throughput-Latency Tradeoﬀ in LLM Inference with Sarathi-Serve. In Proc. USENIX OSDI 24. 117–134. [5] Aida Amini, Saadia Gabriel, et al. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proc. Conf. North American Chapter of ACL. 2357–2367. [6] Apple. 2017. Core ML Tools. https://github.com/apple/coremltools [7] Apple. 2024. Private Cloud Compute: A new frontier for AI privacy in the cloud. https://security.apple.com/blog/private-cloud-compute/. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

[8] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). [9] Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, et al. 2024. Beyond eﬃciency: A systematic survey of resource-eﬃcient large language models. arXiv preprint arXiv:2401.00625 (2024). [10] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2021. BinaryBERT: Pushing the Limit of BERT Quantization. In Proc. ACL. 4334–4348. [11] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, et al. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. In Annual Meeting of the ACL. 3119–3137. [12] Rui Bao, Nan Xue, Yaping Sun, and Zhiyong Chen. 2025. Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks. In IEEE Int. Conf. Commun. China. 1–6. [13] Payel Bhattacharjee, Fengwei Tian, et al. 2025. Conformal Sparsiﬁcation for Bandwidth-Eﬃcient Edge-Cloud Speculative Decoding. In Proc. NeurIPS Workshop: AI and ML for Next-Generation Wireless Commun. and Netw. [14] Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K. Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R. Hower, et al. 2011. The gem5 simulator. SIGARCH Comput. Archit. News 39 (2011), 1–7. [15] Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, et al. 2023. Distributed Inference and Fine-tuning of Large Language Models Over The Internet. In Adv. Neural Infor. Process. Syst., Vol. 36. 12312–12331. [16] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. Language Models are Few-Shot Learners. In Adv. Neural Infor. Process. Syst., Vol. 33. 1877–1901. [17] Dave Burke. 2023. A New Foundation for AI on Android. https://android-developers.googleblog.com/2023/12/a-new-foundation-for-ai-on-android.html. [18] Rahul Chand, Yashoteja Prabhu, and Pratyush Kumar. 2023. Dsformer: Eﬀective compression of text-transformers by dense-sparse weight factorization. arXiv preprint arXiv:2312.13211 (2023). [19] Junyi Chen, Shihao Bai, Zaijun Wang, Siyu Wu, Chuheng Du, Hailong Yang, Ruihao Gong, Shengzhong Liu, et al. 2025. Pre3 : Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation. In Proc. ACL. [20] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [21] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, et al. 2018. TVM: An Automated End-toEnd Optimizing Compiler for Deep Learning. In Proc. USENIX OSDI. 578–594. [22] Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, et al. 2024. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. arXiv preprint arXiv:2407.07061 (2024). [23] Yuxuan Chen et al. 2025. Adaptive layer splitting for wireless large language model inference in edge computing: a model-based reinforcement learning approach. Frontiers of Infor. Technol. & Electronic Eng. 26, 2 (2025), 278–292. [24] Zhixiong Chen, Wenqiang Yi, Yuanwei Liu, and Arumugam Nallanathan. 2023. Knowledge-Aided Federated Learning for Energy-Limited Wireless Networks. IEEE Trans. Commun. 71, 6 (2023), 3368–3386. [25] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, et al. 2018. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 (2018). [26] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, et al. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018). [27] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, et al. 2021. Training veriﬁers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021). [28] Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, et al. 2023. Skipdecode: Autoregressive skip decoding with batching and caching for eﬃcient llm inference. arXiv preprint arXiv:2307.02628 (2023). [29] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 35. 30318–30332. [30] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, et al. 2023. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078 (2023). [31] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. ACL. 4171–4186. [32] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, et al. 2024. Hybrid llm: Cost-eﬃcient and quality-aware query routing. In Proc. Int. Conf. Learn. Represent. (ICLR). 1–23. [33] Guangyao Ding, Huiguo Gao, Shengli Liu, and Guanding Yu. 2025. Multi-Stage Semantic Communication for Low-Latency Edge Inference. IEEE Trans. Cogn. Commun. and Netw. (2025), 1–1. [34] Yu Ding, Jingxuan Zhao, Zhengong Cai, et al. 2025. Adaptoserve: An Eﬃcient System for Supporting Adaptive Chunked-Preﬁlls in LLM Inference. In Proc. IEEE Int. Conf. High Perf. Comput. and Commun. (HPCC). 1–9. [35] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022). Manuscript submitted to ACM

Chen et al.

[36] Xiangyu Dong, Cong Xu, et al. 2012. NVSim: A Circuit-Level Performance, Energy, and Area Model for Emerging Nonvolatile Memory. IEEE Trans. Computer-Aided Design Integr. Circuits Syst. 31, 7 (2012), 994–1007. [37] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, et al. 2024. Layerskip: Enabling early exit inference and self-speculative decoding. In Proc. ACL. 12622–12642. [38] Steven K Esser, Jeﬀrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. 2019. Learned step size quantization. arXiv preprint arXiv:1902.08153 (2019). [39] ETSI ISG. 2017. Mobile Edge Computing; Market Acceleration; MEC Metrics Best Practice and Guidelines. ETSI GS MEC-IEG 006 V1.1.1. ETSI. https://www.etsi.org/deliver/etsi_gs/mec-ieg/001_099/006/01.01.01_60/gs_mec-ieg006v010101p.pdf [40] Shiqing Fan, Yi Rong, et al. 2021. DAPPLE: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 431–445. [41] Jingqi Feng, Yukai Huang, Rui Zhang, et al. 2025. WindServe: Eﬃcient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling. In Proc. Annual Int. Symp. Comput. Arch. (ISCA). 1283–1295. [42] Zideng Feng, Lu Lu, Qin Li, Yuhao Chai, Zhenyu Zhang, et al. 2025. Distributed Inference Optimization for Large Language Model in Edge-Cloud Collaborative Networks. In Proc. IEEE Int. Conf. Commun. 6161–6166. [43] Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 202. 10323–10337. [44] Aadil Gani Ganie. 2025. Securing AI Agents: Implementing Role-Based Access Control for Industrial Applications. arXiv:2509.11431 (2025). [45] Samuel Gehman, Suchin Gururangan, Maarten Sap, et al. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Proc. EMNLP. 3356–3369. [46] Georgi Gerganov. 2023. llama.cpp. https://github.com/ggml-org/llama.cpp [47] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680 (2022). [48] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324 (2019). [49] Andrea Goldsmith. 2005. Wireless communications. Cambridge university press. [50] Raja Gond, Nipun Kwatra, and Ramachandran Ramjee. 2025. TokenWeave: Eﬃcient Compute-Communication Overlap for Distributed LLM Inference. arXiv preprint arXiv:2505.11329 (2025). [51] Suchi Gopal. 2024. The AI Gold Rush: Can Utilities Keep Up with the Energy Demand? https://ﬂoodlightglobal.com/the-ai-gold-rush-can-utilities-keep-up-with-the-energy [52] Jiatao Gu and Xiang Kong. 2020. Fully non-autoregressive neural machine translation: Tricks of the trade. arXiv preprint arXiv:2012.15833 (2020). [53] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge Distillation of Large Language Models. In Proc. Int. Conf. Learn. Represent. (ICLR). [54] Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, et al. 2023. OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization. In Proc. Annual Int. Symp. Comput. Arch. [55] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [56] Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019. Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input. Proc. AAAI 33, 01 (2019), 3723–3730. [57] Zixu Hao et al. 2024. Hybrid SLM and LLM for Edge-Cloud Collaborative Inference. In Proc. EdgeFM. 36–41. [58] Ying He et al. 2024. Large Language Models (LLMs) Inference Oﬄoading and Resource Allocation in Cloud-Edge Computing: An Active Inference Approach. IEEE Trans. Mobile Comput. 23, 12 (2024), 11253–11264. [59] Thomas R Henderson, Mathieu Lacage, George F Riley, Craig Dowell, and Joseph Kopena. 2008. Network simulations with the ns-3 simulator. SIGCOMM demonstration 14, 14 (2008), 527. [60] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In Proc. Int. Conf. Learn. Represent. (ICLR). [61] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, et al. 2015. Teaching Machines to Read and Comprehend. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 28. [62] Namgyu Ho et al. 2023. Large Language Models Are Reasoning Teachers. In Proc. ACL. 14852–14882. [63] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, et al. 2024. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference. arXiv preprint arXiv:2401.08671 (2024). [64] Ke Hong, Lufang Chen, Zhong Wang, Xiuhong Li, et al. 2025. semi-pd: Towards eﬃcient llm serving via phase-wise disaggregated computation and uniﬁed storage. arXiv preprint arXiv:2504.19867 (2025). [65] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, et al. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 37. 1270–1303. [66] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models?. In Proc. Conf. Language Modeling. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

[67] Yang Hu, Connor Imes, Xuanang Zhao, Souvik Kundu, Peter A. Beerel, et al. 2022. PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices. In Proc. DSD. 298–307. [68] Yang Hu, Connor Imes, Xuanang Zhao, Souvik Kundu, Peter A Beerel, Stephen P Crago, and John Paul N Walters. 2021. Pipeline parallelism for inference on heterogeneous edge computing. arXiv preprint arXiv:2110.14895 (2021). [69] Yitao Hu, Xiulong Liu, Guotao Yang, Linxuan Li, Kai Zeng, et al. 2025. TightLLM: Maximizing Throughput for LLM Inference via Adaptive Oﬄoading Policy. IEEE Trans. Comput. 74, 7 (2025), 2195–2209. [70] Sheng Hua, Yong Zhou, Kai Yang, Yuanming Shi, and Kunlun Wang. 2021. Reconﬁgurable Intelligent Surface for Green Edge Inference. IEEE Trans. Green Commun. and Netw. 5, 2 (2021), 964–979. [71] Chongwen Huang, Alessio Zappone, et al. 2019. Reconﬁgurable Intelligent Surfaces for Energy Eﬃciency in Wireless Communication. IEEE Trans. Wireless Commun. 18, 8 (2019), 4157–4170. [72] Hugging Face. 2021. Optimum. https://github.com/huggingface/optimum [73] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). [74] Benoit Jacob, Skirmantas Kligys, et al. 2018. Quantization and Training of Neural Networks for Eﬃcient Integer-Arithmetic-Only Inference. In Proc. IEEE Conf. Comput. Vis. and Pattern Recog. (CVPR). [75] Albert Q. Jiang, Alexandre Sablayrolles, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023). [76] Albert Q Jiang, Alexandre Sablayrolles, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024). [77] Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. 2024. Neo: Saving gpu memory crisis with cpu oﬄoading for online llm inference. arXiv preprint arXiv:2411.01142 (2024). [78] Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial Distillation of Proprietary Large Language Models. In Proc. Conf. Empirical Methods in Natural Language Process. 3134–3154. [79] Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, et al. 2024. HexGen: Generative Inference of Large Language Model over Heterogeneous Environment. In Proc. Int. Conf. Machine Learn. (ICML), Vol. 235. 21946–21961. [80] Dimitrios Kafetzis, Ramin Khalili, and Iordanis Koutsopoulos. 2025. Large Language Model partitioning for low-latency inference at the edge. arXiv preprint arXiv:2505.02533 (2025). [81] Jared Kaplan et al. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). [82] Mahsa Khoshnoodi, Vinija Jain, Mingye Gao, Malavika Srikanth, and Aman Chadha. 2024. A comprehensive survey of accelerated generation techniques in large language models. arXiv preprint arXiv:2405.13019 (2024). [83] Raghuraman Krishnamoorthi. 2018. Quantizing deep convolutional networks for eﬃcient inference: A whitepaper. arXiv preprint arXiv:1806.08342 (2018). [84] Eldar Kurtić, Elias Frantar, and Dan Alistarh. 2023. ZipLM: Inference-Aware Structured Pruning of Language Models. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 36. 65597–65617. [85] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al. 2023. Eﬃcient Memory Management for Large Language Model Serving with PagedAttention. In Proc. SOSP. 611—-626. [86] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019). [87] Bob Lantz, Brandon Heller, and Nick McKeown. 2010. A network in a laptop: rapid prototyping for software-deﬁned networks. In Proc. ACM SIGCOMM Workshop on Hot Topics in Netw. [88] Erik G. Larsson, Ove Edfors, Fredrik Tufvesson, and Thomas L. Marzetta. 2014. Massive MIMO for next generation wireless systems. IEEE Commun. Mag. 52, 2 (2014), 186–195. [89] Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative reﬁnement. arXiv preprint arXiv:1802.06901 (2018). [90] Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. 2024. LLM Inference Serving: Survey of Recent Advances and Opportunities. In Proc. IEEE High Perf. Extreme Comput. Conf. (HPEC). 1–8. [91] Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. 2024. Sprout: Green Generative AI with Carbon-Eﬃcient LLM Inference. In Proc. Conf. Empirical Methods in Natural Language Process. 21799–21813. [92] Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, and othres. 2024. A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412.19442 (2024). [93] Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2020. Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade. arXiv preprint arXiv:2012.14682 (2020). [94] Chen Liang, Simiao Zuo, Qingru Zhang, et al. 2023. Less is More: Task-aware Layer-wise Distillation for Language Model Compression. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 202. 20852–20867. [95] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, and othres. 2024. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. In Proc. Mach. Learn. Syst. (MLSys), Vol. 6. 87–100. [96] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proc. ACL. 3214–3252. Manuscript submitted to ACM

Chen et al.

[97] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [98] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved Baselines with Visual Instruction Tuning. In Proc. IEEE CVPR. 26296– 26306. [99] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, et al. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). [100] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, et al. 2023. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 (2023). [101] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, et al. 2023. The Flan Collection: Designing Data and Methods for Eﬀective Instruction Tuning. In Proc. Int. Conf. on Mach. Learn. (ICML), Vol. 202. 22631–22648. [102] Lu Lu, Geoﬀrey Ye Li, A. Lee Swindlehurst, Alexei Ashikhmin, and Rui Zhang. 2014. An Overview of Massive MIMO: Beneﬁts and Challenges. IEEE J. Sel. Topics in Signal Process. 8, 5 (2014), 742–758. [103] Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. LLM-Pruner: On the Structural Pruning of Large Language Models. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 36. 21702–21720. [104] Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. 2022. Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges. ACM Comput. Surv. 55, 5 (2022). [105] Peter Mattson, Vijay Janapa Reddi, Christine Cheng, Cody Coleman, et al. 2020. MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance. IEEE Micro 40, 2 (2020), 8–16. [106] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proc. ASPLOS, Vol. 1. 586–602. [107] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, et al. 2025. Towards Eﬃcient Generative Large Language Model Serving: A Survey from Algorithms to Systems. ACM Comput. Surv. 58, 1 (2025), 1–37. [108] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, and othres. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Veriﬁcation. In Proc. ASPLOS. 932–949. [109] Melanie Mitchell. 2024. Debates on the nature of artiﬁcial general intelligence. Science 383, 6689 (2024), eado7069. [110] Akrit Mudvari, Yuang Jiang, and Leandros Tassiulas. 2024. Splitllm: Collaborative inference of llms for model placement and throughput optimization. arXiv preprint arXiv:2410.10759 (2024). [111] Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proc. ACL, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). 5356–5371. [112] Shashi Narayan and othres. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Conf. Empirical Methods in Natural Language Process. 1797–1807. [113] Van-Dinh Nguyen, Symeon Chatzinotas, et al. 2022. FedFog: Network-Aware Optimization of Federated Learning Over Wireless Fog-Cloud Systems. IEEE Trans. Wireless Commun. 21, 10 (2022), 8581–8599. [114] NVIDIA Corporation. 2021. FasterTransformer. https://github.com/NVIDIA/FasterTransformer [115] NVIDIA Corporation. 2023. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. https://github.com/NVIDIA/TensorRT-LLM [116] ONNX Runtime. 2025. Get started with ONNX Runtime Mobile. https://onnxruntime.ai/docs/get-started/with-mobile.html. [117] Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, et al. 2019. Timeloop: A Systematic Approach to DNN Accelerator Evaluation. In Proc. IEEE ISPASS. 304–315. [118] Sihyeong Park, Sungryeol Jeon, Chaelyn Lee, Seokhun Jeon, et al. 2025. A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Eﬃciency. arXiv preprint arXiv:2505.01658 (2025). [119] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, et al. 2024. Splitwise: Eﬃcient Generative LLM Inference Using Phase Splitting. In Proc. ACM/IEEE Annual Int. Symp. Comput. Arch. (ISCA). 118–132. [120] Benedetta Picano, Dinh Thai Hoang, and Diep N. Nguyen. 2025. A Matching Game for LLM Layer Deployment in Heterogeneous Edge Networks. IEEE Open J. the Commun. Soc. 6 (2025), 3795–3805. [121] Francesco Piccialli, Diletta Chiaro, Pian Qi, Valerio Bellandi, and Ernesto Damiani. 2025. Federated and edge learning for large language models. Information fusion 117 (2025), 102840. [122] Tomer Porian, Mitchell Wortsman, Jenia Jitsev, et al. 2024. Resolving Discrepancies in Compute-Optimal Scaling of Language Models. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 37. 100535–100570. [123] Rohan Baskar Prabhakar, Hengrui Zhang, and David Wentzlaﬀ. 2024. Kraken: Inherently Parallel Transformers For Eﬃcient Multi-Device Inference. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 37. 7957–7980. [124] Formerly Data Protection. 2018. General data protection regulation (GDPR). Intersoft Consulting, Accessed in October 24, 1 (2018). [125] Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. 2025. Mobile Edge Intelligence for Large Language Models: A Contemporary Survey. IEEE Commun. Surveys Tuts. (2025), 1–1. [126] Qualcomm Innovation Center, Inc. 2024. QAI AppBuilder: Quick AI Application Builder. https://github.com/quic/ai-engine-direct-helper Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

[127] Qualcomm Technologies, Inc. 2024. Qualcomm Enables Meta Llama 3 to Run on Devices Powered by Snapdragon. https://www.qualcomm.com/news/releases/2024/04/qualcomm-enables-meta-llama-3-to-run-on-devices-powered-by-snapd. [128] Colin Raﬀel, Noam Shazeer, Adam Roberts, Katherine Lee, et al. 2020. Exploring the Limits of Transfer Learning with a Uniﬁed Text-to-Text Transformer. J. Mach. Learn. Res. (JMLR) 21, 140 (2020), 1–67. [129] Samyam Rajbhandari, Jeﬀ Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations Toward Training Trillion Parameter Models. In Proc. Int. Conf. High Perfor. Comput., Netw., Storage and Analysis. 1–16. [130] Pasika Ranaweera, Anca Delia Jurcut, and Madhusanka Liyanage. 2021. Survey on Multi-Access Edge Computing Security and Privacy. IEEE Commun. Surveys Tuts. 23, 2 (2021), 1078–1124. [131] Abdul Rehman, Kamran Ahmad Awan, Mahmood Ul Hassan, et al. 2025. CLAF-IoT: Context-Aware LLMs-Enhanced Authentication Framework for Internet of Things. IEEE Int. Things J. (2025). [132] Jiaqi Ren, Chao Wang, Yihan Zhong, Shaohua Cao, Danyang Zheng, and Xiaojun Cao. 2025. Towards Expert Models Deployment Cost Optimization in Edge Computing Networks. In Proc. IEEE Int. Conf. Commun. (ICC). 838–843. [133] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. FastSpeech: Fast, Robust and Controllable Text to Speech. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 32. [134] Gaurav Sahu, Olga Vechtomova, et al. 2023. PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation. In Proc. Conf. Empirical Methods in Natural Language Process. 5316–5327. [135] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Conﬁdent Adaptive Language Modeling. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 35. 17456–17472. [136] Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. 2024. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. In Proc. Int. Conf. Learn. Represent. (ICLR). [137] Yechao She, Tuo Shi, Jianping Wang, and Bin Liu. 2024. Dynamic Batching and Early-Exiting for Accurate and Timely Edge Inference. In Proc. IEEE Veh. Technol. Conf. (VTC-Spring). 1–6. [138] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. Proc. AAAI 34, 05 (2020), 8815–8821. [139] Ying Sheng, Lianmin Zheng, Binhang Yuan, et al. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 202. 31094–31116. [140] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019). [141] Antoine Simoulin and Benoit Crabbé. 2021. How Many Layers and Why? An Analysis of the Model Depth in Transformers. In Proc. ACL. 221–228. [142] Qiuchen Song, Shusen Jing, Shuai Zhang, Songyang Zhang, and Chuan Huang. 2025. Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Function. In Proc. IEEE Int. Conf. Commun. (ICC). [143] Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Eﬃciency. In Proc. IEEE HPCA. 1348–1362. [144] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A Simple and Eﬀective Pruning Approach for Large Language Models. In Proc. Int. Conf. Learn. Represent. (ICLR). [145] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, et al. 2023. SpecTr: Fast Speculative Decoding via Optimal Transport. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 36. 30222–30242. [146] Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023. Structured Pruning for Eﬃcient Generative Pre-trained Language Models. In Proc. ACL. 10880–10895. [147] Gemma Team, Thomas Mesnard, Cassidy Hardin, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024). [148] Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2 (2024). [149] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, et al. 2023. Llama: Open and eﬃcient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [150] Shikhar Tuli and Niraj K. Jha. 2023. AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference With Transformers. IEEE Trans. ComputerAided Design Integr. Circuits Syst. 42, 11 (2023), 4038–4051. [151] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 30. [152] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge Fusion of Large Language Models. In Proc. Int. Conf. Learn. Rep. (ICLR). [153] Haoran Wang, Lei Wang, Haobo Xu, Ying Wang, Yuming Li, and Yinhe Han. 2024. PrimePar: Eﬃcient Spatial-temporal Tensor Partitioning for Large Transformer Model Training. In ACM ASPLOS. 801—-817. [154] Jue Wang, Ke Chen, Gang Chen, Lidan Shou, and Julian McAuley. 2022. SkipBERT: Eﬃcient Inference with Shallow Layer Skipping. In Proc. Annual Meeting of the ACL). 7287–7301. [155] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2024. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692 (2024). Manuscript submitted to ACM

Chen et al.

[156] Wenhui Wang, Furu Wei, Li Dong, et al. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 33. 5776–5788. [157] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, et al. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). [158] Zhibin Wang, Shipeng Li, Yuhang Zhou, Xue Li, Rong Gu, Nguyen Cam-Tu, Chen Tian, and Sheng Zhong. 2024. Revisiting slo and goodput metrics in llm serving. arXiv preprint arXiv:2410.14257 (2024). [159] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned Language Models are Zero-Shot Learners. In Proc. Int. Conf. Learn. Represent. (ICLR). [160] Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, Vol. 35. 24824–24837. [161] Xiuying Wei, Yunchen Zhang, Yuhang Li, et al. 2023. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145 (2023). [162] Patrick Wilhelm, Thorsten Wittkopp, and Odej Kao. 2025. Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference. In Proc. Workshop on Machine Learn. and Sys. (EuroMLSys). 208–215. [163] Kai-Kit Wong et al. 2021. Fluid Antenna Systems. IEEE Trans. Wireless Commun. 20, 3 (2021), 1950–1962. [164] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920 (2023). [165] Wei Wu et al. 2025. TokenSelect: Eﬃcient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection. In Proc. Conf. Empirical Methods in Natural Language Process. 21275–21292. [166] Yu Wu, Tongxuan Liu, Yuting Zeng, Siyu Wu, et al. 2025. Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture. arXiv preprint arXiv:2505.11916 (2025). [167] Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, et al. 2024. Unlocking eﬃciency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851 (2024). [168] Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, et al. 2023. Flash-llm: Enabling cost-eﬀective and highly-eﬃcient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309.10285 (2023). [169] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In Proc. Int. Conf. Learn. Represent. (ICLR). [170] Guangxuan Xiao, Ji Lin, Mickael Seznec, et al. 2023. SmoothQuant: Accurate and Eﬃcient Post-Training Quantization for Large Language Models. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 202. 38087–38099. [171] Zuan Xie, Yang Xu, Hongli Xu, Yunming Liao, and Zhiwei Yao. 2025. A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Models. arXiv preprint arXiv:2503.18989 (2025). [172] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. arXiv preprint arXiv:2004.12993 (2020). [173] Daliang Xu, Wangsong Yin, Hao Zhang, Xin Jin, et al. 2025. EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding. IEEE Trans. Mobile Comput. 24, 4 (2025), 3256–3273. [174] Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. 2024. On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00088 (2024). [175] Minrui Xu, Hongyang Du, Dusit Niyato, Jiawen Kang, et al. 2024. Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services. IEEE Commun. Surveys Tuts. 26, 2 (2024), 1127–1170. [176] Xinyi Xu, Gang Feng, Yijing Liu, Shuang Qin, Jian Wang, and Yunxiang Wang. 2025. Joint Inference Oﬄoading and Model Caching for Small and Large Language Model Collaboration. IEEE Trans. Mobile Comput. (2025), 1–16. [177] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raﬀel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proc. ACL. 483–498. [178] Nan Xue, Yaping Sun, Zhiyong Chen, Meixia Tao, et al. 2025. WDMoE: Wireless Distributed Mixture of Experts for Large Language Models. IEEE Trans. Wireless Commun. (2025), 1–1. [179] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. 2025. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383 (2025). [180] Jin Yang, Qiong Wu, Zhiying Feng, Zhi Zhou, Deke Guo, and Xu Chen. 2025. Quality-of-Service Aware LLM Routing for Edge Computing With Multiple Experts. IEEE Trans. Mobile Comput. 24, 12 (2025), 13648–13662. [181] Yang Yang and M. Cenk Gursoy. 2025. Collaborative Inference in RIS-Assisted MEC Networks under Computing Backlog Constraints. IEEE Trans. Commun. (2025), 1–1. [182] Shunyu Yao, Dian Yu, Jeﬀrey Zhao, Izhak Shafran, et al. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems, Vol. 36. 11809–11822. [183] Shunyu Yao, Jeﬀrey Zhao, Dian Yu, Nan Du, et al. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. Manuscript submitted to ACM

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities

[184] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, et al. 2022. ZeroQuant: Eﬃcient and Aﬀordable Post-Training Quantization for Large-Scale Transformers. In Adv. Neural Infor. Process. Sys., Vol. 35. 27168–27183. [185] Shengyuan Ye, Bei Ouyang, Liekang Zeng, et al. 2025. Jupiter: Fast and Resource-Eﬃcient Collaborative Inference of Generative LLMs on Edge Devices. In Proc. IEEE Conf. Comput. Commun. (INFOCOM). 1–10. [186] Shengyuan Ye, Liekang Zeng, Xiaowen Chu, et al. 2024. Asteroid: Resource-Eﬃcient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices. In Proc. ACM MobiCom. 312—-326. [187] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, et al. 2025. FlashInfer: Eﬃcient and Customizable Attention Engine for LLM Inference Serving. In Proc. Annual Conf. Machine Learn. and Sys. (MLSys). [188] Lu Yin, You Wu, Zhenyu Zhang, et al. 2024. Outlier weighed layerwise sparsity (OWL): a missing secret sauce for pruning LLMs to high sparsity. In Proc. Int. Conf. Machine Learn. (ICML). 57101–57115. [189] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for TransformerBased Generative Models. In Proc. USENIX OSDI. 521–538. [190] Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821 (2023). [191] Oﬁr Zafrir, Guy Boudoukh, et al. 2019. Q8BERT: Quantized 8Bit BERT. In Proc. EMC2-NIPS. 36–39. [192] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. In Proc. ACL. 4791–4800. [193] Ziqian Zeng, Yihuai Hong, Hongliang Dai, et al. 2024. ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference. Proc. AAAI 38, 17 (Mar. 2024), 19506–19514. [194] Deyu Zhang, Yunzhen Luo, Yaobo Wang, Xiaoyan Kui, and Ju Ren. 2024. BatOpt: Optimizing GPU-Based Deep Learning Inference Using Dynamic Batch Processing. IEEE Trans. Cloud Comput. 12, 1 (2024), 174–185. [195] Kai Zhang, Hengtao He, Shenghui Song, Jun Zhang, and Khaled B Letaief. 2025. Distributed on-device llm inference with over-the-air computation. In Proc. IEEE Int. Conf. Commun. (ICC). [196] Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. 2025. EdgeShard: Eﬃcient LLM Inference via Collaborative Edge Computing. IEEE Int. Things J. 12, 10 (2025), 13119–13131. [197] Peiyuan Zhang et al. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385 (2024). [198] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022). [199] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. TernaryBERT: Distillation-aware Ultra-low Bit BERT. In Proc. Conf. Empirical Methods in Natural Language Process. (EMNLP). 509–521. [200] Xinyuan Zhang, Jiangtian Nie, et al. 2025. Beyond the Cloud: Edge Inference for Generative Large Language Models in Wireless Networks. IEEE Trans. Wireless Commun. 24, 1 (2025), 643–658. [201] Junchen Zhao, Yurun Song, Simeng Liu, Ian G. Harris, and Sangeetha Abdu Jyothi. 2024. LinguaLinked: Distributed Large Language Model Inference on Mobile Devices. In Proc. ACL). 160–171. [202] Juntao Zhao, Borui Wan, Chuan Wu, Yanghua Peng, and Haibin Lin. 2024. POSTER: LLM-PQ:Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization. In Proc. ACM PPoPP. 460—-462. [203] Ce Zheng and Tingting Yang. 2025. Communication-Eﬃcient Collaborative LLM Inference via Distributed Speculative Decoding. In Proc. Int. Conf. Wireless Commun. and Signal Process. (WCSP). [204] Guangyuan Zheng, Miaowen Wen, et al. 2025. Computation-Aware Oﬄoading for DNN Inference Tasks in Semantic Communication Assisted MEC Systems. IEEE Trans. Wireless Commun. 24, 4 (2025), 2693–2706. [205] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, et al. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In Proc. USENIX OSDI. 559–578. [206] Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen. 2025. A Review on Edge Large Language Models: Design, Execution, and Applications. ACM Comput. Surv. 57, 8 (2025), 1–35. [207] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, et al. 2024. DistServe: Disaggregating Preﬁll and Decoding for Goodput-optimized Large Language Model Serving. In Proc. USENIX OSDI. 193–210. [208] Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. 2024. Dynamickv: Task-aware adaptive kv cache compression for long context llms. arXiv preprint arXiv:2412.14838 (2024). [209] Bingjie Zhu, Zhixiong Chen, Liqiang Zhao, Hyundong Shin, and Arumugam Nallanathan. 2025. Eﬃcient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding. arXiv preprint arXiv:2510.11331 (2025). [210] Bingjie Zhu, Zhixiong Chen, Liqiang Zhao, Hyundong Shin, and Arumugam Nallanathan. 2025. Joint Caching and Inference for Large Language Models in Wireless Networks. In Proc. IEEE Int. Conf. Commun. (ICC). 6285–6290. [211] Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark Barrett, et al. 2023. Towards Optimal Caching and Model Selection for Large Model Inference. In Proc. Adv. Neural Infor. Process. Syst. (NeurIPS), Vol. 36. 59062–59094.

Manuscript submitted to ACM

Record · ID 138905 · SHA-256 9d4de0aaa652bb7a
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.