A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware Maysam Khatib1 , Moysis Symeonides1 , Demetris Trihinas2 , George Pallis1 , and Marios D. Dikaiakos1 University of Cyprus, Nicosia, Cyprus {mkhati01,msymeo03,pallis,mdd}@ucy.ac.cy 2 University of Nicosia, Nicosia, Cyprus [email protected]
arXiv:2609.08307v1 [cs.DC] 8 Sep 2026
1
Abstract. Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest computeside latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services. Keywords: Large Language Models · Edge Computing · Benchmarking
1
Introduction
Large language models (LLMs) are rapidly moving from centralized cloud services to deployments that span the Edge continuum, from on-device execution to nearby edge servers and remote data centers [7]. This shift is driven by practical requirements such as tighter latency budgets, improved privacy, reduced bandwidth usage, and the need to operate under intermittent connectivity. At the same time, edge deployments introduce a new set of constraints. Hardware is heterogeneous, memory is limited, and power budgets are tight, especially on embedded accelerators and battery-powered nodes [8]. As a result, deploying an
M. Khatib et al.
LLM in real Edge-to-Cloud systems is no longer a single choice of model, but a sequence of decisions about which model to run, where to run it, and which trade-offs are acceptable for a given application and query [14]. In this setting, practitioners face competing objectives that rarely align. Higher accuracy is often associated with larger models, which in turn increase model footprint and can degrade responsiveness [14]. Quantization can reduce memory needs and enable execution on edge devices, yet it can introduce overheads and quality shifts that depend on both the model and the serving stack [20]. Moreover, the latency experienced by users is not only shaped by inference time. Even small network delays between the client and the server can offset computeside latency advantages and change which model and deployment node are optimal for end-to-end performance [14, 13]. Lastly, energy usage further complicates these decisions, since higher-performance hardware may offer lower latency at higher power, while slower devices can be more energy-efficient [3]. These challenges reveal a gap not fully addressed by existing benchmarking, serving-system, quantization, and edge-LLM studies. Standardized benchmarks such as MLPerf Inference and MLPerf Power provide comparable performance and energy reporting across systems [17, 19], while application-level benchmarks such as MMLU evaluate model reasoning across diverse subjects [5]. However, these benchmarks are not designed to determine where an LLM-backed intelligent web service should execute across the edge continuum. Similarly, LLM serving systems such as Orca, vLLM, and Sarathi-Serve optimize batching, scheduling, memory management, and throughput-latency trade-offs within serving infrastructures [21, 10, 1], while quantization studies focus on compression methods for efficient inference [20, 11]. Lastly, Edge-oriented work such as CLONE further shows that LLM deployment feasibility depends on model design, system behavior, and hardware constraints [18], without examining the multi-objective tradeoffs. In contrast, our work provides a controlled measurement-driven methodology for comparing model family, quantization, execution platform, latency, model footprint, and energy, together with a Pareto-based sensitivity analysis of streamed-token delivery overhead for intelligent web services. Specifically, this paper presents a controlled measurement study of LLM inference across selected edge and near-edge deployments. Rather than aiming to provide an exhaustive benchmark of all edge-LLM serving configurations, our goal is to quantify how model choice, quantization, execution platform, and streaming delivery overhead interact when selecting an LLM deployment under constraints on accuracy, responsiveness, model footprint, and energy. To this end, this paper makes four contributions: (i) a containerized benchmarking pipeline for self-hosted LLM inference that records accuracy, model footprint, prefill and per-token decoding latency, and the overall execution energy; (ii) we use this pipeline to compare edge and near-edge deployments, including on-device execution on Jetson Orin, near-edge GPU serving, and near-edge CPU serving, under a controlled interactive question-answering workload; (iii) we quantify how model family, quantization level, and execution platform jointly affect the accuracy-latency-energy trade-off, showing that model size alone is not
LLM Inference Trade-offs Across the Edge Continuum
Fig. 1. Benchmarking Pipeline Overview
a reliable setup heuristic; and (iv) we use Pareto-frontier analysis to identify dominated setups in the accuracy-latency space and to study how possible pertoken streaming delivery overheads can affect deployment choices between local and server-side inference. The complete experimental configuration, including model identifiers and filenames, runtime/container versions, hardware settings, and prompt templates, is available in the versioned repository [9]. Our results highlight that no single off-the-shelf model is best across accuracy, latency, model footprint, and measured energy. GPU servers deliver the lowest per-token latency, while on-device execution on Jetson Orin shows lower measured energy, consistent with its lower platform power. Architectural differences and quantization overheads can outweigh parameter count when comparing practical latency and quality outcomes. Finally, we show that introducing possible per-token delivery overheads in a sensitivity analysis reshapes the accuracylatency relation by reducing the latency advantage of server-side GPU inference and making on-device GPU inference more competitive for delay-sensitive settings, while server GPU remains preferable for higher-accuracy configurations. Together, these findings provide practical guidance for selecting LLMs and execution nodes under user and system constraints, and they can serve as a basis for preference-aware LLM routing in intelligent web services. The rest of the paper is organized as follows. Sec. 2 presents the benchmarking pipeline, setup, and methodology. Sec. 3 reports the results and Pareto-frontier analysis. Sec. 4 and Sec. 5 cover related work and the conclusion, respectively.
2
Reference Benchmarking Architecture
To enable efficient and fully automated benchmarking, we design a pipeline that orchestrates the experiment workflow from configuration to result export. Fig. 1 shows the overall architecture, which converts user-specified configurations into repeatable, measurable experiments that include accuracy, latency, and energy.
M. Khatib et al.
The workflow starts from a simple configuration file that defines the selected model, dataset, and target device. A lightweight Parser ingests these parameters and forwards them to the Request Generator, which coordinates the execution. The Request Generator interacts with the LLM Serving API, which is responsible for instantiating the chosen LLM. Upon initialization, the serving API downloads the required model weights from an Online LLM Repository and loads them into the runtime before accepting inference requests. All components are deployed as containerized services to ensure portability and repeatability across heterogeneous edge platforms, with the serving module packaged as a multi-architecture image to support different CPU/GPU environments. Once the model is ready, the Request Generator streams prompts from the selected dataset, normalizes them into a consistent request format, and submits them to the LLM Serving API. During execution, it records responses and timestamps to derive latency-related metrics and, for datasets that include expected outputs, computes task-level metrics such as accuracy. In parallel, the monitoring subsystem collects power measurements through device-appropriate mechanisms, including external smart plugs or software-exposed counters. These metrics are exposed via a Power Exporter and ingested by a Monitoring Power Storage that stores time-aligned power metrics over the benchmark window. After completion, the pipeline collects logs and monitoring traces, aggregates per-prompt and per-configuration summaries, and exports the results into a unified results.csv artifact. This output enables comparison across LLMs, datasets, and devices, allowing post-benchmark analysis. 2.1
Deployment and Implementation
Hardware. In order to highlight how LLM design choices interact with hardware constraints, we evaluate two self-hosted deployment platforms and a cloud baseline. The first platform is an NVIDIA Jetson AGX Orin developer kit, an Arm-based edge system that integrates a 12-core Arm Cortex A78AE CPU with an NVIDIA Ampere GPU (2048 CUDA cores and 64 Tensor Cores) and 64 GB of 256-bit LPDDR5 memory, which makes it well suited for low-power on-device deployments while remaining sensitive to model size and runtime memory pressure. The second platform is a near-edge server with an Intel Xeon Gold 6230 CPU, 96 GB RAM, and an NVIDIA T4 Tensor Core GPU with 2,560 CUDA cores, 320 Tensor Cores, and 16 GB of GDDR6 memory, providing higher throughput and lower inference latency than the Jetson-class device. Finally, we include GPT-4o as a cloud-hosted reference point to provide a reference accuracy baseline, noting that cloud executions do not expose device-level metrics comparable to our on-premise measurements. Model Serving & LLMs. For the LLM Serving API on edge nodes, we rely on the Ollama service [15] to host and query self-hosted models, while model weights are retrieved from the Hugging Face Hub [6], from which we select the evaluated set of open-weight LLMs. Specifically, we focused on variations of Llama 3.2 (Instruct model) with 1B and 3B parameters and F16, Q5_K_M, Q4_K_M quantization; Mistral 7B with Q5_K_M and Q4_K_M quantization; TinyLlama
LLM Inference Trade-offs Across the Edge Continuum
1.1B Chat v1.0 with Q5_K_M and Q4_K_M; and Phi v2.0 with Q5_K_M and Q4_K_M quantized weights stored in GGUF format. We selected these quantization schemes because Q4_K_M and Q5_K_M represent a practical balance between resource usage and model quality. Q4_K_M provides strong compression (approximately 4.8 bits per weight [4]), while Q5_K_M retains higher precision at increased memory cost. Both formats are widely adopted in practice. Specifically, Q5_K_M typically better preserves the model’s original performance and is considered preferable when hardware constraints permit, whereas Q4_K_M remains a reliable alternative for lower-memory environments. Compared to older legacy formats, the “K” variants reduce quantization error by using block-wise, importance-aware bit allocations, which can improve accuracy and, in some CPU-based settings, inference speed. Workload Generator. We designed a workload generator as a Python script using modular abstractions that can operate across multiple datasets. For this study, we evaluate all models using the MMLU (Massive Multitask Language Understanding) validation split, a widely used benchmark covering 57 academic subjects ranging from mathematics and computer science to the humanities [5]. MMLU provides a controlled multiple-choice workload for comparing factual knowledge and reasoning accuracy across the evaluated configurations. We generate a total of 1531 prompts, issuing requests via the Ollama API for selfhosted models and the OpenAI API for the cloud baseline. Each prompt is formatted into a strict multiple-choice question to ensure consistent evaluation across quantization levels, device types, and model families. Since this workload asks each model to return a single multiple-choice option, it primarily evaluates short-output interactive inference; therefore, sustained long-form generation, long-context behavior, and stable inter-token streaming are outside the scope of our experiments. All evaluations are initiated locally, with the workload generator deployed on the same node as the LLM on Orin or the GPU server. Thus, self-hosted models incur no WAN latency, as both request handling and inference are performed on the same device, ensuring that the measured latency captures only model computation and local serving overhead. OpenAI API calls naturally include Internet latency, which is reported as part of the cloud baseline. This setup allows us to isolate device-level performance and eliminates effects from network variability. Moreover, we intentionally evaluate a sequential, single-request workload to model interactive LLM use cases in which users submit individual queries. This setting allows us to isolate the effect of model choice, quantization, execution device, and access latency without introducing batching and queueing effects. We note that we do not claim that the same ranking will hold under concurrent or multi-tenant serving. Parallel request streams, dynamic batching, and queue-aware scheduling can change both throughput and tail latency, and are treated as complementary dimensions for our future work. Reproducibility and Parametrization details. All self-hosted experiments use the same serving stack, decoding parameters, and answer-extraction logic across devices and models. The answer generation is performed with the LLM temperature set to 0 and a fixed maximum output length sufficient to produce
M. Khatib et al.
at least a single multiple-choice answer. Each prompt is formatted to request exactly one answer among the available options, and responses are parsed by extracting the first valid option label. Specifically, we use a zero-shot prompt, preserve the original option ordering, aggregate accuracy across all 1,531 questions, and count malformed responses as incorrect; the exact prompt and parser are available in [9]. Before measurement, each model is loaded once and 10 warmup prompts are executed to avoid including model download, initialization, and first-token warm-up effects in the reported latency. The measured interval begins immediately before request submission and ends when the complete model response is received by the workload generator. Recorded metrics. For each trial, the workload generator records the LLM outputs and computes accuracy as the fraction of correctly predicted multiple-choice answers. It also records latency from request submission to response completion, together with prefill latency, decoding latency, and derived per-token latency metrics based on the data provided by the Ollama server and the number of input and generated tokens. Specifically, prefill latency per token is computed as prompt-evaluation duration divided by the number of input tokens, while decode latency per token is computed as generation duration divided by the number of generated tokens. The resulting measurements are exported to a CSV file. Power metrics are collected separately through Prometheus [16], which acts as our Monitoring Power Storage and periodically scrapes exporters running on the physical devices and stores the observed power values. Once the workload completes, the system queries monitoring storage to estimate the average power consumption P̄ during execution. It then computes energy per trial as E = P̄ × T , where T is the workload runtime, and stores it alongside the other captured metrics. Compute-side latency vs. client-observed streamed-token latency. We distinguish between compute-side latency and client-observed streamed-token latency. Compute-side latency per token is measured at the serving node and captures the time required by the model and serving stack to generate each output token. However, in interactive LLM applications, responses are often streamed token by token so that users can begin consuming the output before the full response is complete. In remote deployments, each generated token must also become available at the client, introducing additional transport, serialization, buffering, and client-side receipt costs. For our sensitivity analysis (Sec. 3), we approximate client-observed streamed-token latency as Lstream = Lcompute + ddelivery , where Lcompute is the measured compute-side decoding latency per generated token, and ddelivery is an effective per-token delivery overhead. The term ddelivery abstracts the combined effects of pipelined delivery, buffering, persistent connections, and possible overlap between computation and communication into a single effective delivery penalty. We use this term to test how sensitive the accuracy-latency Pareto frontier is to remote-delivery effects. The baseline self-hosted measurements report Lcompute , and, to study the sensitivity of deployment choices to delivery overhead, we add two fixed effective penalties, ddelivery = 30 ms and ddelivery = 60 ms, to the server-side setups and
LLM Inference Trade-offs Across the Edge Continuum
recompute the accuracy-latency frontier. These values should be interpreted as controlled what-if parameters rather than measured network delays. Power measurement. In our evaluation, power measurements are interpreted as deployment-level indicators rather than as a calibrated cross-platform power audit. Specifically, on Jetson Orin, we use socket-level measurements [12], which capture total device draw during inference, including the board and surrounding platform components. On the server, we combine GPU power reported by nvidia-smi with CPU package power reported by Intel RAPL. This estimate excludes some platform-level components, including memory, storage, fans, motherboard losses, and power-supply losses. Hence, the server’s reported energy is a partial, lower-bound estimate, while the Orin’s is a full-system measurement. Moreover, we report average power over the execution interval and do not subtract idle power. Consequently, background platform consumption remains included. Because the Jetson measurement is socket-level whereas the server estimate covers CPU-package and GPU telemetry only, absolute cross-platform energy values should be interpreted as indicative rather than directly equivalent. Because telemetry mechanisms differ across platforms, we focus primarily on robust qualitative trends rather than exact cross-platform energy equivalence. Moreover, because the OpenAI API does not expose hardware-level utilization or energy measurements, we exclude this model from the energy-related analysis. Variability. Each trial is executed five times; we report mean values and use repeated runs to check whether the main ranking trends remain stable. Our goal is to compare central tendencies across deployment configurations; tail behavior such as p95/p99 latency, queueing delay, and request-level power transients are outside the scope of this sequential workload. Moreover, across repeated runs, the relative ranking of the main deployment configurations remained stable. The latter aspects are important for production serving systems, but they require separate experiments with parallel requests, batching, and tail-latency analysis.
3
Observations
Using our benchmarking pipeline, we evaluate multiple LLMs across five repeated runs and report average values. Next, we present the most notable results to examine key research questions on inference latency, model size, and energy usage for Edge-enabled LLM deployments. RQ1: Which LLM setup provides the best trade-off between accuracy, latency, and model size for inference? The results of our benchmarking are presented in Fig. 2. Specifically, Fig. 2 (left) illustrates each model’s inference accuracy in relation to its size (middle) and observed latency (right) on the Jetson Orin platform. We also include GPT-4o as a cloud baseline. As expected, GPT-4o achieves the highest accuracy, reaching approximately 80% correct answers. Among the self-hosted models, larger models such as Mistral 7B exhibit higher accuracy, albeit at the cost of significantly increased parameter counts and slower response times. Notably, the Llama 3.2 F16 model, despite its larger weight-file size than the quantized
M. Khatib et al.
Fig. 2. Performance of Different LLMs on Jetson Orin
variants, demonstrates slightly lower accuracy compared to smaller quantized models like Mistral Q5_K_M, and similar accuracy to its own quantized variants. In terms of latency (Fig. 2 (right)), however, the Llama 3.2 F16 model performs better than other 7B quantized models, suggesting that the observed latency is affected not only by model-file size, but also by model architecture, precision format, and serving-backend behavior. These results show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Model family, precision/quantization, serving stack, and execution platform jointly shape the measured outcome. For instance, the quantized versions of the Phi 2 are comparable in size to the quantized Llama 3.2 3B variants, yet the latter offer superior accuracy and latency performance. Observation: Model size alone is an insufficient deployment heuristic. In our measurements, models with comparable footprints occupy different positions in the accuracy-latency space. This indicates that edge LLM selection should be based on measured configurations and workload-specific trade-offs rather than on model size alone. For intelligent services, this means that deployment decisions should consider model family, quantization, serving stack, and hardware together. RQ2: How does the choice of deployment platform influence LLM latency? The left plot of Fig. 3 reports average latency per generated token for a set of self-hosted LLMs, grouped by the device where inference runs. The x-axis shows three deployment options, Orin GPU, Server CPU, and Server GPU. The y-axis shows milliseconds per token, so lower bars mean faster token generation and therefore a more responsive model during decoding. Each colored bar corresponds to a specific model and weight format, including multiple quantizations such as Q4 and Q5 and in some cases F16 precision. A clear pattern is that the deployment platform dominates the latency. Across all models, Server GPU yields the lowest per-token latency, with bars clustered close to the bottom of the plot. Orin GPU is consistently slower than the server GPU but still remains lower than the server CPU deployment, indicating that GPU-accelerated inference on the Jetson provides reasonable token generation despite its more constrained compute and memory bandwidth. In contrast, Server CPU exhibits the highest per-token latency, with several models rising to the top of the chart. This means that even on a powerful server, moving inference from a GPU to CPU execution can increase token-level response time substantially.
LLM Inference Trade-offs Across the Edge Continuum
Fig. 3. Decode latency, prefill latency, and energy across deployment platforms
Moreover, the middle plot of Fig. 3 shows prefill latency, meaning how long the system needs to process the input prompt before it starts generating the output. The main observation is that the Server CPU has by far the highest prefill latency, especially for the larger models. For example, the 7B models and the larger quantized variants reach roughly 17–22 ms per input token, while smaller models remain lower but still clearly above the GPU results. This means that prompt processing on CPU becomes expensive as model size increases. The Orin device has much lower prefill latency than the Server CPU, mostly around or below 1 ms per token. This suggests that Orin handles the prefill phase relatively efficiently for these models, despite being an edge device. The Server GPU has the lowest prefill latency overall, with almost negligible values in the plot. This is expected because the prefill phase is highly parallelizable, and GPUs can process prompt tokens much more efficiently than CPUs. Overall, it shows that prefill is strongly affected by the execution device and model size. GPUs are clearly the best option for fast prompt processing, CPUs suffer significantly for larger models, and Orin provides a strong edge alternative with much lower prefill latency than the Server CPU. Within each device group, the multiple variants for the same base model show the effect of quantization. The Q4 and Q5 versions typically reduce memory traffic compared to F16, which usually results in faster token generation. On GPUs, Q4 and Q5 generally differ less, and all variants tend to converge to low values on the server GPU due to its higher throughput. Observation: In this workload, execution platform is the largest observed factor for both decoding and prefill latency. Server GPU execution offers the lowest latency across the LLMs, while Orin GPU remains a competitive edge deployment option, offering substantially lower latency than CPU-only execution under tighter power and memory constraints. In contrast, Server CPU exhibits the highest latency in both phases, especially during prefill for larger models, making prompt processing and token generation slower without providing an accuracy benefit. Quantization affects performance within each device group, although its impact is less pronounced on the Server GPU where all variants converge to low latency. Thus, for the evaluated LLMs, server hardware, and our workload, CPU-only execution can be deprioritized when latency is the primary objective. RQ3: How do execution platform and model choice affect measured energy consumption? The right plot of Fig. 3 shows the average energy consumed per trial for each evaluated model, grouped by the device where inference runs. The x-axis again
M. Khatib et al.
shows the three deployment options Orin GPU, Server CPU, and Server GPU. The y-axis shows energy per trial in kilojoules, so lower bars indicate that the system consumes less total energy over the benchmark trial. First, we observe that the deployment platform strongly shapes energy needs. The CPU-only execution has the highest values by a large margin for most models, with several bars reaching multiple kilojoules per trial. This indicates that CPU-based inference is energy-expensive in our setup because longer execution times, especially during the prefill stage, increase the accumulated energy over the benchmark interval. Furthermore, under our measurement setup, Orin shows the lowest measured energy per trial among the evaluated self-hosted setups, while the Server’s consumption is also low, but it appears higher than Orin’s. However, because the server measurement excludes memory, storage, fans, and PSU losses, the server’s true total energy would be higher than reported. This suggests that the GPUenabled server can provide high throughput and low latency, while an embedded edge device may still be attractive when lower platform power is prioritized. Additionally, within each device group, model choice and quantization still matter, but the device remains the dominant factor. Larger models and heavier variants tend to increase energy, especially on the CPU where higher compute costs directly extend generation time. Finally, lower-bit quantization variants, such as Q4 relative to Q5, reduce model size and memory traffic and may also reduce energy consumption, although the gains depend on the hardware and workload. Observation: Energy consumption is governed by the interaction between generation time and platform power. Under our evaluated server configuration and specific workload, CPU-only inference shows higher measured energy because its substantially longer runtime outweighs its lower instantaneous device power. Orin achieves lower measured energy values in this workload, plausibly because its lower platform power offsets its slower decoding speed relative to the GPU server3 . RQ4: How can Pareto frontier analysis be used to identify and eliminate LLM-device configurations that are suboptimal in terms of accuracy and latency? Our accuracy-latency trade-off analysis highlights that no single off-the-shelf LLM performs optimally across the evaluated configurations. Deploying a “onesize-fits-all” model can require practitioners to compromise between quality and responsiveness, with cost and energy requiring additional constraints or objectives. Here, we examine how the Pareto front can be used as a tool for selecting the most efficient configuration within a deployment. Specifically, Fig. 4 (left) plots each evaluated configuration as a point in the accuracy versus latency space, where the y-axis shows the percentage of correct responses and the x-axis shows average latency per generated token in milliseconds. Marker shapes encode the deployment platform, with "diamond" points for Orin GPU, "x" for GPU-enabled deployment, and the "triangle" for CPU-enabled deployment. The dotted red curve highlights the Pareto front, that is, the set of configurations 3
We should note that, because Orin and server power are measured via different mechanisms, the absolute energy values should be interpreted cautiously, while the qualitative trend remains informative.
LLM Inference Trade-offs Across the Edge Continuum
Fig. 4. Accuracy-latency Pareto frontiers under different per-token delivery overheads.
not dominated by any other option, meaning that no other point achieves both higher accuracy and lower latency simultaneously. GPU-enabled Server deployments concentrate on the far left, indicating the lowest per-token latency, typically within a few milliseconds, while spanning a moderate range of accuracies. Additionally, Orin points are in the low latency region but with higher latency than the server GPU for many models, and they cover a wider spread in accuracy. By contrast, CPU-enabled deployment points appear further to the right, often at tens of milliseconds per token, showing that CPU execution incurs a large responsiveness penalty without a corresponding accuracy gain. Within our evaluated accuracy–latency configuration space, this makes all CPU-only trials dominated, since similar accuracies are achievable on GPU deployments at substantially lower latency. Moreover, the Pareto front starts from very low latency with lower accuracy, then climbs toward higher accuracy as latency increases, and ends at a high accuracy point that requires noticeably more latency than the fastest options. On the Pareto front itself, almost all points come from the GPU-enabled server, and only one Orin point provides high accuracy and relatively low latency. Observation: There is no universally best LLM, so selection should be Paretodriven and dominated options, such as TinyLlama, Phi, or CPU-enabled deployments, can be deprioritized for the evaluated workload when optimizing only accuracy and per-token latency. RQ5: How does streamed-token delivery delay affect the accuracylatency Pareto frontier between on-device inference and edge-server LLM serving? Continuing our analysis, we study how potential streaming delivery overheads can change the apparent performance of different deployment options. The previous measurements for the edge server reflect compute-side decoding latency and exclude the additional time required for generated tokens to become visible to a remote client. In many interactive LLM-enabled web services, responses are streamed token by token so that users can start reading before the full response is complete. Under this serving model, generated tokens may experience effective delivery overheads caused by transport, serialization, buffering, server-side flushing, and client-side receipt, although these costs can overlap with computation in real deployments. To approximate this effect, we add two fixed effective per-token delivery overheads, namely 30 ms and 60 ms, to the serverside per-token latency before recomputing the Pareto frontier. These values are not intended to represent a fixed network round-trip delay for every token, nor measurements from a specific production edge network. Instead, they are used
M. Khatib et al.
as controlled sensitivity parameters that represent moderate and high effective streaming overheads. This choice is consistent with prior cloud-edge latency measurements showing that latency differences between cloud and edge deployments often fall within the tens-of-milliseconds range [2]. The sensitivity analysis is also motivated by recent cloud-edge LLM inference studies, which identify communication overhead as a key contributor to inference latency [22]. We do not apply this adjustment to Orin configuration, since it represents on-device inference with no additional access-network hop. Introducing effective per-token delivery overhead changes the shape of the Pareto frontier, but it does not remove the server GPU from the efficient set, as shown in Fig. 4. As the added overhead increases to 30 ms and 60 ms, the server GPU points move to the right, reducing their advantage in terms of clientobserved token arrival delay. Under these overheads, local Orin execution becomes increasingly relevant in the low-delay region because it does not include the modeled remote delivery penalty. However, server GPU setups remain on the Pareto frontier for higher-accuracy models, showing that the best deployment depends on the target accuracy-delay trade-off. Therefore, this analysis should be viewed as a controlled sensitivity study of client-observed token arrival delay, rather than a measurement of a specific production network. Observation: For streaming LLM services, deployment decisions should consider client-observed token arrival latency, not only compute-side inference time. As delivery overhead increases, server GPU setups lose part of their latency advantage, making local GPU-enabled devices more attractive for latency-sensitive workloads. However, server GPU remains Pareto-efficient for higher-accuracy configurations, so the best deployment depends on the accuracy-delay trade-off.
4
Related Work
Standardized benchmarking efforts have shaped how inference performance and efficiency are reported across platforms. MLPerf Inference defines representative tasks together with a common load generator, accuracy checking, and prescriptive run and submission rules that make results comparable across diverse hardware and software stacks [17]. MLCommons further complements this direction with the MLPerf Power methodology, which specifies measurement and reporting procedures to enable consistent power and energy comparisons across systems [19]. In addition, MMLU is widely used as an application-level benchmark for evaluating the reasoning and knowledge capabilities of language models across diverse academic subjects [5]. Unlike such benchmarks, we do not introduce new tasks, but provide a methodology for edge LLM serving that reports accuracy, model footprint, compute-side latency, and measured energy, while also exploring possible streaming-delivery overheads through sensitivity analysis. Focusing on making LLM inference feasible under tight memory limits, another large body of work targets LLM compression and quantization. Specifically, SmoothQuant proposes a training-free transformation that smooths the activation outliers by offline migrating the quantization burden from activations to weights through a mathematically equivalent transformation, while causing only
LLM Inference Trade-offs Across the Edge Continuum
minimal accuracy loss [20]. In addition, AWQ [11] targets low-bit weight-only quantization by identifying salient channels using activation statistics, protecting a fraction of weights, and pairing the method with a deployment-oriented runtime to accelerate on-device inference. Rather than proposing a new quantizer, our study treats quantization as a trial factor in an end-to-end pipeline and captures its effect on the underlying metrics. LLM serving systems have improved throughput by rethinking scheduling around model compute and memory behavior and infrastructure capabilities. For instance, Orca [21] introduces iteration-level scheduling and selective batching to balance latency and throughput for large transformers. vLLM [10] further improves efficiency with PagedAttention, reducing KV-cache waste and increasing effective batch sizes, while Sarathi-Serve [1] addresses the throughput-latency tradeoff through chunked prefills and stall-free scheduling, enabling larger batches with lower latency impact and fewer pipeline bubbles. Recent edge-oriented work shows that deployment feasibility depends jointly on model design, system behavior, and hardware limits. For example, CLONE [18] quantifies the challenges of running LLMs on edge devices and proposes an energy- and latency-aware algorithm-hardware co-design. Unlike prior work, we present a measurementdriven methodology for quantifying accuracy and system-level trade-offs across a self-hosted edge continuum, enabling principled local-versus-remote selection.
5
Conclusion, Limitations & Future Work
This paper presented a controlled measurement study of LLM inference across edge and near-edge platforms, focusing on the trade-offs that arise when intelligent web services must choose between local, near-edge, and cloud-hosted execution. Using a containerized benchmarking pipeline, we evaluated multiple openweight LLMs and quantization variants on Jetson Orin and on a near-edge server in GPU-enabled and CPU-only modes, while using GPT-4o as a cloud-hosted reference. The results show that LLM deployment is inherently multi-objective: accuracy, latency, model size, and measured energy do not always improve together. GPU-enabled server execution provides the lowest compute-side latency, Jetson Orin shows lower measured energy, and CPU-only execution is dominated in the accuracy-latency space for the studied workload. The results further show that parameter count alone does not reliably predict deployment performance, highlighting the need for empirical evaluation across model architectures and quantization settings. Our Pareto-frontier analysis shows that deployment choices depend on whether latency is measured only at the compute side or at the client-observed streamed-token level. When per-token streaming delivery overhead is included, some server-side configurations become less attractive despite faster decoding. This finding is particularly important for latency-sensitive interactive web services, where user-perceived responsiveness depends not only on token generation speed but also on how quickly generated tokens arrive at the client. Overall, our study provides evidence that preference-aware LLM routing across the edge continuum can benefit from jointly considering model quality, compute-side latency, model size, and energy.
M. Khatib et al.
Limitations. This study focuses on sequential interactive question answering and does not evaluate concurrent request streams, batching, queueing, or tail latency. Therefore, the reported rankings should not be generalized directly to high-throughput multi-tenant serving. Moreover, the workload is based on MMLU, a multiple-choice benchmark that captures factual and reasoning accuracy but does not cover long-context dialogue, retrieval-augmented generation, tool use, or multimodal tasks. In addition, because the MMLU workload produces short multiple-choice responses, the decoding and streaming results should not be generalized to long-form generation without additional experiments. Energy measurements rely on platform-specific telemetry with different measurement boundaries across devices; therefore, they should be interpreted as indicative deployment-level measurements rather than fully calibrated cross-platform comparisons. Finally, the streamed-token delivery analysis uses controlled delay injection rather than measurements over production edge networks. These limitations motivate the broader evaluation planned in future work. Future work. We plan to extend this controlled study along several directions. First, we aim to evaluate a broader set of edge-continuum configurations, including additional accelerators, near-edge servers, serving stacks, and memory systems, to assess how robust the observed deployment rankings are across different hardware and software environments. Second, we plan to expand the workload beyond sequential interactive question answering to include concurrent serving, long-context dialogue, retrieval-augmented generation, and agentic tool-use scenarios. Moreover, we intend to replace the controlled delay injection with measurements over real edge networks, capturing latency variability and its impact on model placement and routing decisions. Lastly, our analysis can serve as a foundation for prompt routing systems that combine user preferences with edge-to-cloud deployment performance to optimize end-to-end execution. Acknowledgment. This work is supported by the EU Commission through the AI-DAPT project (HORIZON-CL4-2023-HUMAN-01-01, GA: 101135826). Language refinements were performed at the sentence level using ChatGPT. All original content and ideas are solely those of the authors.
References 1. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B.S., Tumanov, A., Ramjee, R.: Taming throughput-latency tradeoff in llm inference with sarathi-serve. In: Proceedings of the 18th OSDI. USENIX Association, USA (2024) 2. Charyyev, B., Arslan, E., Gunes, M.H.: Latency comparison of cloud datacenters and edge servers. In: GLOBECOM 2020 - 2020 IEEE Global Communications Conference. pp. 1–6 (2020) 3. Georgiou, J., Symeonides, M., Pallis, G., Dikaiakos, M.D.: Automating multitenancy performance evaluation on edge compute nodes. In: 2025 IEEE International Conference on Edge Computing and Communications (EDGE). pp. 103–114. IEEE, Piscataway, NJ (2025) 4. ggml-org contributors: llama.cpp quantize README. https://github.com/ggmlorg/llama.cpp/blob/master/tools/quantize/README.md (2026) 5. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) (2021)
LLM Inference Trade-offs Across the Edge Continuum 6. Hugging Face: Hugging face: The ai community building the future. https://huggingface.co/ (2026), accessed: 2026-03-13 7. Jiang, S., Zhou, X., Zhang, M., Xu, C., Liao, G., Chen, J., Cao, J.: Edge large language models: a comprehensive survey. CCF Trans. Pervasive Comp. Interact. pp. 181–210 (2026) 8. Kasioulis, M., Symeonides, M., Ioannou, G., Pallis, G., Dikaiakos, M.D.: Energy modeling of inference workloads with ai accelerators at the edge: A benchmarking study. In: 2024 IEEE International Conference on Cloud Engineering (IC2E). pp. 189–196. IEEE, Piscataway, NJ (2024) 9. Khatib, M., et al.: LLM Edge Continuum Benchmarking Dataset and Analysis. https://github.com/UCY-LINC-LAB/Edge-LLM-Inference-Benchmark (2026) 10. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the 29th Symposium on Operating Systems Principles. p. 611–626. SOSP ’23, Association for Computing Machinery, New York, NY, USA (2023) 11. Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., Han, S.: AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. GetMobile: Mobile Comp. and Comm. 28(4), 12–17 (Jan 2025) 12. Meross Technology Limited: Smart plug for google home - meross mss310eu. https://www.meross.com/en-gc/smart-plug/smart-plug-google-home/6 (2026), accessed: 2026-03-13 13. Mohammed, T., Joe-Wong, C., Babbar, R., Francesco, M.D.: Distributed inference acceleration with adaptive dnn partitioning and offloading. In: IEEE INFOCOM 2020 - IEEE Conference on Computer Communications. pp. 854–863. IEEE, Piscataway, NJ (2020) 14. Morabito, R., Jang, S.: Smaller, smarter, closer: The edge of collaborative generative artificial intelligence. IEEE Internet Computing 29(4), 7–15 (2025) 15. Ollama Inc.: Ollama. https://ollama.com/ (2026), accessed: 2026-03-13 16. Prometheus Authors: Prometheus: Open source metrics and monitoring for your systems and services. https://prometheus.io/ (2026), accessed: 2026-03-13 17. Reddi, V.J., et al.: Mlperf inference benchmark. In: 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). pp. 446–459. IEEE, Piscataway, NJ (2020) 18. Tian, C., Qin, X., Tam, K., Li, L., Wang, Z., Zhao, Y., Zhang, M., Xu, C.: Clone: customizing llms for efficient latency-aware inference at the edge. In: Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference. USENIX ATC ’25, USENIX Association, USA (2025) 19. Tschand, A., et al.: MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from µWatts to MWatts for Sustainable AI. In: 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). pp. 1201–1216 (2025) 20. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S.: Smoothquant: accurate and efficient post-training quantization for large language models. In: Proceedings of the 40th ICML. ICML’23 (2023) 21. Yu, G.I., Jeong, J.S., Kim, G.W., Kim, S., Chun, B.G.: Orca: A distributed serving system for Transformer-Based generative models. In: 16th USENIX OSDI. pp. 521– 538. USENIX Association, Carlsbad, CA (Jul 2022) 22. Zhang, M., Shen, X., Cao, J., Cui, Z., Jiang, S.: Edgeshard: Efficient llm inference via collaborative edge computing. IEEE Internet of Things Journal 12(10), 13119– 13131 (2025)