arXiv CSopen access

Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence

Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence Chanwoo Cho

Wooseok Kim

Yonglak Son

[email protected] Korea University Seoul, Republic of Korea

[email protected] Korea University Seoul, Republic of Korea

[email protected] Korea University Seoul, Republic of Korea

Young Seo Lee∗

Young Geun Kim∗

[email protected] Soongsil University Seoul, Republic of Korea

[email protected] Korea University Seoul, Republic of Korea

arXiv:2607.07046v1 [cs.DC] 8 Jul 2026

Abstract

Typically, LLM-based services batch inference requests from tens of millions of users in a centralized data center [34, 59, 60]. To accelerate the large volume of inference requests, state-of-the-art inference serving frameworks such as vLLM [34] are used to maximize the size of batched LLM inference requests. However, such centralized execution has several limitations for end-users. First, the response latency can be prolonged by communication overhead depending on wireless network conditions and congestion in core network, which adversely impacts a user QoS — the communication overhead can further be increased depending on the subscription option of the service [19, 48]. Second, the inference requests may contain user private data, posing a security concern [61]. To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices by leveraging the available computational resources recently featured in mobile SoCs — on-device execution of LLM inference is expected to open up new personalized applications, such as smart home agent [6], personal AI robot [1], etc. However, it is still not feasible to execute large models whose required memory is beyond the device memory capacity. To enable on-device LLM, many researchers have focused on designing smaller LLMs (sLLMs) (e.g., MobileLLM [39] and TinyLlama [65]) or applying model compression techniques such as quantization [17, 38, 62] and pruning [40, 42]. However, such techniques inevitably cause accuracy drop due to information loss. To break through the memory constraint, we first propose to leverage multiple nearby edge devices for LLM inference execution. According to [13], mobile users typically use multiple devices (3.6, on average) at the same time, including smartphones, tablets, and smartwatches. Furthermore, there can be various IoT (Internet of Things) devices, such as smart speakers and smart kitchen appliances, available at the smart

Large language models (LLMs) are widely used in intelligent services due to their remarkable capability in generative tasks. Typically, LLM-based services process the inference requests of the users in a centralized data center. Unfortunately, such centralized execution has limitations for end-users, such as increased response latency with communication overhead and privacy leakage risk. To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices. However, the limited resources of a single edge device impose restrictions on achievable accuracy of LLMs. To overcome the issue, we first propose to leverage multiple user-end devices available at the edge for LLM inference, enabling the execution of larger models. Specifically, we propose Voltron, a novel on-device LLM inference framework that elastically utilizes multiple user-end devices for LLM inference execution while adapting to diverse real-world edge environments. In our evaluation, Voltron achieves up to 16.5% higher accuracy than state-ofthe-art LLMs that can be executed on a single edge device, satisfying user QoS requirements.

1

Introduction

Large language models (LLMs) have shown a remarkable capability in generative tasks such as question answering, text generation, and summarization. Many intelligent services are utilizing the capability of LLMs as a key functionality. For example, personal assistants (e.g., Siri [7] and Alexa [6]) use LLMs to improve their conversational abilities, AI chatbots (e.g., ChatGPT [11] and Gemini [18]) provide information through question-and-answer interactions using LLMs, and personal agents (e.g., OpenClaw [54], ManusAI [26]) rely on LLMs to autonomously plan and execute complex user tasks. ∗ Co-corresponding author.

1

home. By exploiting such available multiple devices simultaneously, it is possible to execute larger LLMs achieving higher inference accuracy. To enable multi-device execution of LLM inference, distributed inference methods, such as tensor parallelism [46], can be used. However, our characterization reveals that adopting the distributed inference methods in edge environment poses two key challenges as follows: – Heterogeneity: Edge environments inherently exhibit multiple forms of heterogeneity. Devices in edge clusters often have highly heterogeneous computational and memory capabilities, making the execution characteristics of the distributed inference methods vary across the devices. Furthermore, the characteristics are also affected by LLM architectures, and mixed precision configurations that are natural in resource-constrained edge environment. These factors make it challenging to apply a single distributed inference method as a one-size-fits-all solution in edge. – Runtime variance: Edge LLM inference execution is stochastic by nature. The computational characteristics of LLM inference can vary across inference phases and conversations with various input/output length. Wireless network variability also significantly affects the communication overhead of distributed inference, due to signal strength fluctuations caused by user movement. Such runtime variance makes it challenging to efficiently employ appropriate distributed inference methods in continuously changing execution environments. To tackle the aforementioned challenges, we propose Voltron, a novel on-multi-device LLM inference execution framework that enables elastic execution of LLM across edge devices. Whenever a user types a new prompt for LLM inference, Voltron observes the current execution environment and quickly determines an appropriate execution strategy for that environment. During LLM inference, Voltron adapts to changing environments by adjusting the execution strategy to satisfy QoS requirements while maximizing accuracy. The key contributions of this work include: – We present an in-depth performance and accuracy characterization of multi-device LLM inference at the edge. The results show that performance and accuracy of multidevice LLM inference can significantly vary under multiple forms of heterogeneity and runtime variance (Section 3). – We propose Voltron, a novel LLM inference execution framework that elastically utilizes multiple user-end devices available at the edge. Voltron efficiently determines the appropriate execution strategy for the current environment and continuously adjusts it to adapt to changing environments, maximizing accuracy while satisfying QoS requirements (Section 4).

Figure 1: LLM architecture and inference process. – We implement Voltron1 across various combinations of edge clusters and models. In our evaluation, Voltron achieves 10.2% higher accuracy on average (up to 16.5% higher accuracy) than state-of-the-art LLMs that can be executed on a single device, while satisfying the QoS requirements even in the presence of runtime variance (Section 5).

2 Background 2.1 Large Language Model LLM Architecture: An LLM generally has an architecture where identically structured decoders (depicted in Fig. 1(a)) are stacked. When a user enters an input prompt, the prompt is converted into LLM inputs (i.e., tokens) through the tokenizer. The tokens are transformed into high-dimensional vectors (i.e., input embeddings), and then pass through the attention and feed-forward layers [58]. In the attention layer (ATTN) (highlighted with blue color in Fig. 1(a)), each attention head independently computes an attention score with scaled dot-product operations between query (Q), key (K), and value (V) matrices. By using multiple attention heads, the attention mechanism projects Q, K, and V into different subspaces, which allows the LLM to focus on different parts of an input sequence simultaneously. The attention outputs are then passed through an output linear (O) and subsequently processed by the feed-forward network (FFN), which projects them into higher-dimensional feature spaces. The feed-forward network further applies nonlinear transformations that refine each token representation and improve the model reasoning capability. Recent LLM architectures are also employing several variants of ATTN and FFN for efficient parameter scaling [4, 10, 51]. For example, group query attention (GQA) reduces computational load of multi-head attention (MHA) by sharing K and V projections across multiple query heads, while sliding window attention (SWA) reduces the load by restricting 1We plan to open-source Voltron upon acceptance to expedite the adoption

of multi-device LLM execution for empowered edge intelligence. 2

— TTFT and TPOT are normalized to respective QoS constraints. sLLM is a model designed with a small number of parameters to execute effectively in resource-constrained environments. Though sLLM may meet the QoS constraints (i.e. TTFT of 10s [30] and TPOT of 400ms2 ) due to reduced computation load, it exhibits much lower accuracy compared to the LLM (i.e., 14B LLM) [25]. On the other hand, quantization reduces the numerical precision of parameters from high precision (e.g., FP16) to lower bit representations (e.g., INT4), thereby reducing both the computation load and memory footprint. The quantized LLM, however, exhibits lower accuracy compared to the non-quantized LLM (i.e., 14B LLM), due to the fine-grained information loss [33]. It also fails to meet the QoS constraints since the number of parameters remains unchanged. Pruning reduces memory usage and computational load by removing a subset of weights (e.g., attention heads or layers). Although the pruned LLM satisfies the QoS constraints, it shows the lowest accuracy due to significant information loss, as the pruned heads or layers may contain important information. These results demonstrate that, given the limited resources of a single edge device, it is challenging to provide satisfactory accuracy while meeting the QoS requirements solely with the algorithm-side techniques.

Figure 2: Accuracy and performance of on-device LLM techniques. Existing techniques fail to satisfy accuracy and QoS requirements. attention to a fixed-size local window. For FFN, mixture of experts (MoE) is widely used to reduce the load by activating only a subset of expert networks for each token. LLM Inference: Fig. 1(b) shows the LLM inference process. The inference of LLM is divided into two phases: the prefill and decode phases. In the prefill phase, the LLM processes the tokens of a user’s input prompt (e.g., "Ask LLM" in Fig. 1(b)) in parallel and produces the first output token (e.g., "for" in Fig. 1(b)). The LLM saves the calculated K and V tensors of the input tokens in the KV cache. The latency to complete the prefill phase is called time-to-first-token (TTFT). In the decode phase, the LLM generates one output token per iteration in an autoregressive manner. A new token (e.g., "help" in Fig. 1(b)) is jointly predicted based on the tensors of the generated token (e.g., "for" in Fig. 1(b)) and those stored in the KV cache (e.g., "Ask LLM" in Fig. 1(b)). Meanwhile, K and V tensors of the most recently generated token (e.g., "for" in Fig. 1(b)) are accumulated in the KV cache to be used for the next token generation. This process repeats until the “<EOS>” token is generated as the output. The latency of the decode phase is called time-per-output-token (TPOT).

2.2

3 LLM Execution on Multi-edge Devices 3.1 Opportunities and Benefits To break through the limitations of single-device LLM inference execution, we first propose to leverage multiple nearby edge devices for LLM inference execution. According to [13], mobile users typically interact with multiple devices (3.6, on average)—such as smartphones, tablets, and smartwatches— at the same time. The rapid growth of IoT technologies has also resulted in the spread of numerous smart home devices, including TVs, speakers, and sensors. Such devices are typically co-located and connected through low-latency wireless network, which enables fast data exchange across the devices [47]. The multiple devices can thus effectively form a local compute cluster. Along with the advancements of distributed inference execution methods, such as model parallelism (MP) and tensor parallelism (TP), the multi-device cluster opens new opportunities to cooperatively execute larger LLM models. The multi-device execution offers several benefits for ondevice LLM inference. By aggregating the memory and compute resources across devices, it becomes feasible to execute larger LLMs and improve inference accuracy compared to single-device execution. Moreover, the aggregated memory

On-Device LLM

LLM inference has traditionally been executed in centralized data centers due to their large memory and computation requirements. Recently, however, there is a growing demand for executing LLM inference locally on edge devices, driven by the emergence of LLM applications that require long contexts containing privacy sensitive user information, such as private personal agents [66] and autonomous household robots [1]. To enable LLM inference on a single edge device with limited resources, prior works have focused on reducing the computational and memory demands of LLM inference through algorithm-side techniques such as designing sLLMs [39, 65], quantization [38, 62], and pruning [40, 42]. However, such approaches often result in low accuracy, along with long TTFT and TPOT. Fig. 2 shows the accuracy, TTFT, and TPOT of LLMs achieved through representative algorithm-side techniques

2 Prior studies show that users expect mobile devices to respond in several

hundred milliseconds, and their satisfaction quickly drops as latency grows beyond around 400ms [12]. 3

Figure 3: Parallelism methods for multi-device LLM inference execution and TPOT breakdown of multidevice LLM inference under the homogeneous cluster.

Figure 4: TPOT breakdown of multi-device LLM inference under different clusters and parallelism methods. sequential execution of layers while input/output activations are transmitted only once per device. TP, on the other hand, splits each layer of an LLM into fine-grained shards and distributes the shards across devices (Fig. 3(b)). In each layer, the main device broadcasts the input activations to the rest of the devices. Then, all the devices simultaneously perform computations of the assigned shards with the input activations. After the devices complete the computation, the results are aggregated to the main device. This process is repeated across all layers to generate output tokens. TP typically exhibits short computation time and high communication time (Fig. 3(c)), due to the concurrent shard execution of layers with frequent communication (i.e., broadcasting and aggregation repeated at every layer).

resources also enable long-context applications (e.g., personalized LLM agents [66]) to be executed locally at the edge by accommodating larger KV caches. Executing LLM locally also allows users to include richer private information into the context without incurring privacy-leakage risks, enabling more personalized responses compared to server-side LLMs. These advantages highlight the strong potential of multi-device execution for enabling scalable and privacypreserving LLM inference on edge devices.

3.2

Challenges

To enable LLM inference across multiple edge devices, distributed inference methods such as MP and TP can be used. However, naively adopting the distributed inference methods can result in inefficient execution of LLM inference, due to their computation-communication characteristics (Section 3.2.1), device, model, and precision heterogeneity (Section 3.2.2) and runtime variance (Section 3.2.3). In this section, we characterize and quantify the impact of each factor on multi-device LLM execution3 . 3.2.1 Characteristics of Distributed Inference Methods. In this section, we characterize two representative distributed inference methods, MP and TP, on edge execution environment. These methods collaboratively execute LLM inference on the multiple edge devices, while sharing the intermediate data (e.g., output activations) via wireless network (e.g., Wi-Fi Direct). MP partitions an LLM into subsets of layers and allocates contiguous subsets of layers across devices (Fig. 3(a)). In the beginning, a device (typically main device held in user hand, such as smartphone) executes the first subset of layers allocated to it. After the device finishes executing the layers, it transmits the output activations to another device that holds the next layers. This process continues sequentially until all layers are executed. MP typically exhibits long computation time and low communication time (Fig. 3(c)), as it requires

Takeaway: MP and TP exhibit distinct computationcommunication characteristics, making it difficult to employ one as a one-size-fits-all method. 3.2.2 Heterogeneity. Edge LLM execution inherently exhibits multiple forms of heterogeneity. Edge devices have highly fragmented computational and memory capabilities, introducing a severe degree of performance heterogeneity. A variety of layer architectures and numerical precisions used by LLM also affect the latency and memory footprint of multi device execution, leading to model and precision heterogeneity. In this subsection, we characterize the impact of the heterogeneity on multi-device LLM inference. Device Heterogeneity: Fig. 4 shows TPOT breakdown of multi-device LLM inference under two heterogeneous clusters commonly observed in realistic deployment scenarios4 . As shown in Fig. 4, naively adopting MP and TP on heterogeneous clusters fails to provide sufficient execution results. In case of Cluster 1 (Fig. 4(a)), TPOT does not satisfy the QoS requirements. This is because, despite distinct performance capabilities of the edge devices, MP and TP, which are originally designed for identical GPU nodes in large-scale server execution, allocate computations to them evenly —

3 Note TTFT/TPOT are normalized to respective QoS constraints throughout 4 The detailed configurations of the clusters are explained in Section 5.

the characterization results. 4

Figure 5: Latency and memory footprint of various LLM layers. Note latency is normalized to that of MP in MHA of ATTN.

Figure 6: (a) Memory usage and accuracy of fixed- and mixed-precision LLMs and (b) layer latency across different precision levels. Note latency is normalized to MP in FP16 of ATTN.

the device with the lowest computation capability (i.e., Pixel 5 in Fig. 4(a)) thus becomes the performance bottleneck in both MP and TP. Even worse, in case of Cluster 2 (Fig. 4(b)), both MP and TP fails to execute the LLM due to the OOM error. This is also because MP and TP evenly distribute weight tensors and KV cache across the devices without considering the heterogeneous memory budget of the edge devices—the device with the lowest memory budget (i.e., Pixel 2 XL in Fig. 4(b)) cannot accommodate the allocated tensors.

parallelism strategy (Fig. 5(b)). This makes it difficult to employ per-layer parallelism strategy solely relying on the performance characteristics. For example, in case of MoE variant, MP exhibits higher performance than TP but incurs a significantly larger memory footprint — among the ATTN variants, GQA and SWA also exhibit similar performance-memory trade-off. This implies that, when choosing parallelism strategies for each layer, it is crucial to consider both performance characteristics and memory requirements. Takeaway: Model heterogeneity further complicates the parallelism method selection problem, since different layer variants exhibit distinct performance and memory characteristics under MP and TP.

Takeaway: Naively employing MP and TP for heterogeneous edge clusters may lead to the failure (and/or QoS violation) of the LLM execution. This calls for the need of new parallelism method tailor-designed for the edge execution environment.

Precision Heterogeneity: To achieve the highest accuracy given limited memory budget, mixed precision is widely used in LLM inference execution [28]. Fig. 6(a) compares the accuracy and memory requirements of fixed precision execution and the mixed precision execution with three different precisions (FP16, INT8, and INT4). Higher precisions (i.e., FP16 and INT8) typically exhibit higher accuracy, at the expense of larger memory footprint. Such larger memory footprint often incurs OOM errors on the resource-constrained edge devices. Employing lower precision (i.e., INT4) can mitigate the memory issues, but the accuracy can be significantly degraded. By employing different precisions for each layer (considering its weight distribution and importance), it is possible to achieve reasonably high accuracy, within the limited memory budget. This makes the precision heterogeneity natural in the edge execution environment. Such precision heterogeneity even complicates the parallelism method selection problem. As shown in Fig. 6(b), MP/TP performance characteristics of the same layer can even vary depending on the precision. Lower precision reduces the computational workloads of each shard, increasing

Model Heterogeneity: Recently, state-of-the-art LLMs are adopting hybrid architectures composed of several different ATTN and FFN variants. Such layer variants exhibit distinct performance characteristics when executed with MP and TP on heterogeneous edge devices. Fig. 5(a) shows the latency of various layers when they are allocated to different devices based on MP and TP. As shown in Fig. 5(a), the best performing parallelism strategy varies across the layer types due to their distinct performance characteristics. In case of MHA which has the largest computation load among the ATTN variants, TP shows better performance compared to MP. On the other hand, in case of GQA and SWA which are designed to reduce the computation load of ATTN, MP (on the highend) shows better performance than TP. Similar patterns can be observed in the FFN variants — TP outweighs MP for dense FFN while MP outweighs TP for the MoE variant which is designed to exploit sparsity for efficiency. Those layer variants also exhibit significantly heterogeneous memory footprint on each device, depending on the 5

Figure 8: (a) Wi-Fi Direct RSSI under user mobility and (b) the performance and accuracy of multi-device LLM execution under varying wireless signal strength. Note latency is normalized to MP with -30 dBm of signal strength.

Figure 7: (a) Input/output length and (b) the number of tokens accumulated in the KV cache across conversation turns, and (c) layer latency across inference phases and conversation turns. Note latency is normalized to MP with 0k tokens in the KV cache.

The performance characteristics of MP/TP also varies depending on the inference phase. As shown in Fig. 7(c), in the prefill phase, MP achieves lower latency than TP regardless of the number of tokens in KV cache. This is because the prefill phase typically processes many tokens together with batched execution, which diminishes the advantages of TP over MP. On the other hand, in the decode phase, TP can achieve lower latency than MP beyond a certain number of accumulated tokens (right top plot of Fig. 7(c)) — since the decode phase processes the increased computation load of the ATTN layer with limited batching, TP can better amortize its communication overhead over the computation load, achieving better performance than MP.

the relative impact of communication time in TP execution. Accordingly, the performance benefits of TP becomes smaller at lower precision, while MP remains less affected due to its lower communication overhead. Given that the memory requirements of a layer also scales with the precision, the selection of a parallelism strategy for each layer also requires the consideration of the precision heterogeneity. Takeaway: Mixed precision, which is natural in resource-constrained edge environment, introduces additional challenges in the parallelism selection. 3.2.3 Runtime Variance. Edge execution is stochastic by nature. LLM inference itself has computational dynamics due to increasing number of tokens iteratively accumulated into the KV cache, and inference phase changes. Wireless network also exhibits significantly varying signal strength as the user moves. In this subsection, we examine the impact of runtime variance in edge inference execution. Computational Dynamics: LLM inference exhibits inherent variability in the size of the KV cache, due to the dynamic nature of user conversations. As shown in Fig. 7(a), input and output lengths vary significantly across requests (e.g., standard deviation of input/output length of per-user requests is 160.1/381.7, on average in LMSYS-Chat-1M dataset [67]). This affects the number of tokens accumulated in the KV cache for each token generation. The KV cache size is also gradually enlarged as conversation progresses, as shown in Fig. 7(b). This accumulation increases the computational load of the ATTN, leading to higher latency, as shown in Fig. 7(c). Since TP distributes the growing computational load across devices, its latency increases more slowly than that of MP. As a result, better performing parallelism strategy shifts for ATTN in the decode phase.

Takeaway: LLM inference has computational dynamics. Since the layer-wise performance characteristics of the parallelism methods can vary with such dynamics, a static parallelism strategy is not sufficient to consistently achieve efficient LLM execution. Stochastic Wireless Network: Network variability also affects communication overheads, shifting the performance characteristics of the per-layer parallelization strategies. Fig. 8(a) shows an example of the sources of network variability. As a user moves within the smart home environment, the distance between the devices changes, affecting the wireless signal strength of their wireless connections. Such signal strength variation significantly affects the communication overhead, as shown in Fig. 8(b). In case of ATTN, MP consistently outweighs TP due to the relatively small computation load — the communication overhead does not account for a large portion of total execution time. In contrast, in case of FFN, the performance characteristics of MP/TP vary with the wireless signal strength. When signal strength is sufficiently strong (e.g., up to -60 dBm), TP achieves better latency 6

than MP. However, when the Wi-Fi Direct RSSI drops to 70dBm, TP incurs substantial communication overhead due to bandwidth limitations and increased latency. In such cases, MP becomes a better option. When the Wi-Fi Direct signal strength falls below a certain threshold, the connection can be even lost, as illustrated in Fig. 8(a). In this case, the loss of connectivity prunes the weight tensors assigned to the disconnected device, degrading accuracy (Fig. 8(b)).

features such as device type, layer type, precision, input token length, and the number of tokens accumulated in the KV cache [16]. The communication time is estimated based on the size of transmission data and wireless network conditions (e.g., available network bandwidth and signal strength) [31]. Voltron determines the layer allocation for the selected parallelism strategy as follows. (1) MP: Voltron first attempts to place the layer on the highest-performance device to minimize execution time. If the layer does not fit within the device’s memory budget, Voltron selects the next available device that satisfies the memory requirement, while having the best performance. (2) TP: Voltron greedily assigns shards (i.e., partitions of a layer) to devices based on their performance to balance the execution time across devices, thereby minimizing the impact of stragglers. 4.2.2 Importance-aware Mixed Precision. After the execution strategy is determined, Voltron adjusts the precision for each layer to maximize accuracy while satisfying latency and memory requirements. Since each layer has different impact on inference accuracy [22], Voltron first profiles the accuracy impact of each layer in an offline phase. Specifically, Voltron calculates an importance score for each layer using channellevel importance metric [43]. For each layer, we first evaluate the perplexity of the full model with the highest precision (i.e., FP16), by using the validation dataset. After that, we re-evaluate the perplexity while pruning channels within the layer one by one. The perplexity difference measured while pruning each channel is then identified as the importance score of the channel—the higher difference is, the larger its importance is. We then add up the importance scores of all channels within each layer to identify the importance score of the layer. Since layer importance patterns can vary across tasks, we perform the offline profiling separately for each task. Given that the precision changes can affect the layerwise performance characteristics, Voltron re-searches for the better execution strategy with the execution plan module (Section 4.2.1) based on the adjusted precision configuration. When allocating precisions to the layers by considering the layer-wise parallelism strategies and the memory budget, it is infeasible to enumerate all the possible allocations at runtime especially in the resource-constrained edge devices — the complexity of the enumeration is exponential in the number of layers in the worst case. To efficiently explore the precision allocations by exploiting the monotonic relationship between the precision, memory usage, and inference latency (that we observe in Section 3.2.2), Voltron uses binary search of which complexity is 𝑂 log(𝑛) — quantitative overhead analysis is presented in Section 5. Voltron performs the binary search starting from the full-precision (i.e., FP16) model and progressively scales layer precision in the ascending order of importance. At each step, it constructs a candidate mixed-precision model and checks whether the

Takeaway: The wireless network variability also needs to be carefully considered, when deciding the parallelism strategies.

4 4.1

Voltron Overview

To address the challenges of edge execution, we propose a novel multi-device LLM inference framework, Voltron (Fig. 9), which enables elastic execution of LLM across edge devices. At the beginning of each conversation turn, Voltron determines model execution plan, including layer-wise parallelism strategy and precision (Section 4.2). For overcoming the limitations of the conventional parallelism methods, Voltron employs a novel hybrid parallelism (HP) method, tailor-designed for edge execution environment. During the LLM execution, Voltron continuously monitors execution environment, and elastically adjusts the execution strategy by adapting to the runtime variance (Section 4.3).

4.2

Model Execution Plan

At the beginning of each conversation turn, Voltron determines a model execution plan that specifies the layer-wise parallelism strategy and precision configuration, to maximize accuracy while satisfying QoS requirements under the current execution environment. Voltron first searches for the fastest parallelism strategy for each layer based on the estimated execution time (Section 4.2.1) while meeting the memory budget. It then adjusts the precision configurations of each layer to better satisfy the QoS requirements while maximizing accuracy (Section 4.2.2). Since layer-wise performance characteristics of MP/TP can differ across inference phases, as discussed in Section 3.2.3, it explores separate execution strategies for the prefill and decode phases. 4.2.1 Layer-wise Hybrid Parallelism. Voltron sequentially assigns a parallelism strategy (i.e., MP or TP) to each layer, while considering the memory budget of each device. For each layer, Voltron estimates the execution time (i.e., sum of computation and communication time) of both strategies and selects the one with lower latency. The computation time is estimated exploiting a regression model (of which average accuracy is 90% that is sufficient to correctly identify the better-performing parallelism method) based on runtime 7

Figure 9: Overview of Voltron Framework. resulting configuration satisfies both the memory budget and QoS requirements. Then, it adjusts the search direction accordingly until the final configuration is determined.

4.3

plan module to generate new execution plan (i.e., precisionaware layer-wise parallelism strategy) the most suitable for the changed execution environment — the overhead of the model execution plan module only accounts for 0.2% of TPOT (i.e., less than a few hundred microseconds even in the worst case), thereby making it feasible to run the module for tokenby-token modifications. Once the new execution plan is determined, Voltron updates only the layers that require precision scaling. Since these updates may incur non-trivial I/O overhead, we discuss an approach to mitigate this overhead in Overhead Mitigation. Pruning: Since pruning may incur severe accuracy drop, Voltron adopts importance-aware pruning strategy, which follows the similar principle used in an importance-aware precision allocation (Section 4.2.2). Specifically, it adopts shard-wise pruning in order to minimize the information loss. Voltron first determines the number of pruning shards based on the total memory budget. Note Voltron prunes the same number of shards from all layers, to ensure the balanced loss across the layers — this practice has been widely adopted in a number of pruning works to minimize the accuracy loss [40]. Voltron then prunes the shards in the ascending order of importance score for each layer. After that, Voltron modifies the execution plan based on the newly estimated execution time of per-layer parallelization strategies and memory budget of respective devices, by incorporating the pruned layers. Each device reflects the plan modification by 1) reloading the newly allocated layers from the storage and 2) removing the pruned shard weights from the layers allocated in the memory if needed. The overhead of the former can be mitigated by the Overhead Mitigation approach while that of the latter can be mitigated via shard-wise memory allocations — the pruned shard weights can easily be removed from the layers allocated in the memory, and/or from the layers that are newly allocated for the replacement. Overhead Mitigation: When Voltron replaces shards (or layers) stored in the memory with new ones from storage, additional I/O operations are required. Performing I/O operations during the LLM inference can severely degrade performance due to the limited bandwidth of mobile storage (e.g.,

Elastic Model Execution

As we observe in Section 3.2.3, edge execution of LLM inference is inherently stochastic due to computational dynamics and wireless network variability. To adapt to the runtime variance, Voltron continuously adjusts the execution strategy. Specifically, Voltron adapts to the computational dynamics of LLM inference by adjusting the execution plan (Section 4.3.1). Voltron also adapts to the wireless network variability via selective activation quantization (Section 4.3.2). 4.3.1 Computation Scaling. To adapt to computational dynamics, Voltron first monitors the KV cache size and the currently available memory budget5 for each token generation. When the memory budget is expected to be insufficient for the next token generation, due to the increase of KV cache size or device disconnection, Voltron first adopts precision scaling. If solely adopting precision scaling is not sufficient to resolve the memory budget shortage, Voltron additionally adopts selective pruning — as we demonstrate in Fig. 2(a) pruning results in more severe accuracy drop compared to the adopting INT4 precision to entire model. When memory budget headroom becomes available again (e.g., due to KV cache flush or connection recovery), Voltron re-executes the model execution plan module (Section 4.2) to gradually recover the layers and their precisions. Precision Scaling: The goal of precision scaling is to find the best layer-wise precision that meets the memory budget and QoS constraint while minimizing the accuracy loss. To achieve the goal, Voltron re-executes the model execution 5 In our implementation, the communication scaling module is executed

in advance to the computation scaling module in order to prevent communication overhead being a major performance bottleneck — the QoS margin is already minimized by the communication overhead in such case, so that computation scaling has a limited accuracy protection room. Note, if some shards in layers are already pruned by communication scaling, the computation scaling module preserves the pruning results and checks the memory budget based on the pruned architecture. 8

Table 1: Mobile Device Specification Device Name Lenovo Legion Y700 Galaxy S22 Ultra Google Pixel 7 Google Pixel 5 Google Pixel 2 XL Nexus 5X

DRAM / Available 12GB / 8.3GB 12GB / 7.3GB 8GB / 4.7GB 8GB / 4.7GB 4GB / 2.3GB 2GB / 1.2GB

Table 2: Cluster of Devices

SoC Performance Level Snapdragon 8+ Gen 1 High-end Snapdragon 8 Gen 1 High-end Google Tensor G2 Mid-end Snapdragon 765G SM7250 Mid-end Snapdragon 835 MSM8998 Low-end Snapdragon 808 MSM8992 Low-end

Cluster Device1 Car Mobile (Cluster1) (Galaxy S22 Ultra) Office Mobile (Cluster2) (Galaxy S22 Ultra) Home Mobile (Cluster3) (Galaxy S22 Ultra)

UFS). To minimize the I/O overhead, Voltron overlaps I/O operations with ongoing LLM computation by exploiting idle computing resources within each device. Specifically, while executing the LLM computations, Voltron utilizes idle computing resources to preload the new shards (or layers) from storage. By overlapping these data transfers with LLM computations, Voltron effectively hides most of the I/O latency — the detailed analysis on the effectiveness of the overhead mitigation is presented in Section 5.2.4. 4.3.2 Communication Scaling. To adapt to the wireless network variability, Voltron estimates the expected communication overhead for each token generation under the current wireless network signal strength. When the expected communication overhead becomes too large to push the LLM latency beyond the QoS requirement, Voltron determines how much the cross-device transmission data needs to be reduced to satisfy the QoS constraint. To achieve this reduction, Voltron lowers the precision of activations (required to be transmitted across devices) via quantization. Since activation quantization may incur accuracy loss, Voltron employs an importance-aware approach. Specifically, Voltron computes the importance of each activation as the sum of the importance of the weight shards that are used for generating it, and lowers the precision of activations in the ascending order of importance score. If lowering all activations to the lowest precision (i.e., INT4) is still insufficient to achieve the required reduction in the amount of transmitted data, Voltron further decreases communication by progressively pruning transmission activations (as well as the corresponding shards) based on their importance. When the wireless network status improves, Voltron gradually restores the pruned activations and their precisions, to recover the model accuracy.

4.4

Device2 Device3 Automotive SoC Infotainment SoC (Google Pixel 7) (Google Pixel 5) Tablet Smart Watch (Lenovo Legion Y700) (Google Pixel 2 XL) Smart Watch IoT Device (Google Pixel 2 XL) (Nexus 5X)

Table 3: Large Language Models Family gemma-3

Qwen1.5

Qwen2.5

Name gemm-3-4b-it gemm-3-12b-it Qwen1.5-4B Qwen1.5-7B Qwen1.5-14B Qwen1.5-MoE-A2.7B Qwen2.5-4B Qwen2.5-7B Qwen2.5-14B

# of Parameters 4B 12B 4B 7B 14B 14B 3B 7B 14B

ATTN GQA, SWA(1024) GQA, SWA(1024) MHA MHA MHA MHA GQA GQA GQA

FFN Dense FFN Dense FFN Dense FFN Dense FFN Dense FFN MoE FFN Dense FFN Dense FFN Dense FFN

and the latency constraint). The energy optimization module then exploits this margin to reduce the power consumption. Specifically, it reduces the voltage and frequency of the processing units of each device, as long as the latency constraint is satisfied — the effectiveness of the energy module is presented in Section 5.2.3.

5 Evaluation 5.1 Experimental Setup 5.1.1 Devices. We perform experiments on six mobile devices spanning a wide range of performance levels and memory budgets: two high-end devices representing tablets and smartphones, two mid-end devices representing automotive and infotainment SoCs, and two low-end devices representing smartwatches and IoT devices. Table 1 summarizes their specifications. We configure three clusters with the devices (Table 2), reflecting realistic use case scenarios [3, 50, 53]: Cluster 1 (Car), Cluster 2 (Office), and Cluster 3 (Home). In each cluster, the devices are connected via Wi-Fi Direct — other wireless network, such as Bluetooth, can also be used.

Energy Optimization

5.1.2 Workloads. We use three open-source state-of-the-art LLM families (i.e., gemma-3 [57], Qwen1.5 [8], and Qwen2.5 [49]) with various parameter sizes and layer types (Table 3). To support the mixed precision, we adopt k-quant [32], which provides implementation of multiple precision formats including FP16, INT8, and INT4. We use input traces of LMSYSChat-1M dataset [67] to emulate realistic conversational workloads along with varying input/output length. We use representative LLM benchmarks for accuracy evaluation: MMLU (0-shot), Hellaswag (0-shot), GSM8K (8-shot, CoT), and MATH (4-shot, CoT) [14, 23, 24, 64].

Although the multi-device LLM inference improves model accuracy while satisfying QoS requirements, it may increase the energy consumption of the participating devices. If the devices are all battery-powered, alleviating the energy increase might get crucial. To minimize the energy consumption in such cases, Voltron includes an energy optimization module. When energy saving is prioritized, the energy module further reduces latency of LLM inference by employing computation scaling (i.e., precision scaling and/or pruning). This creates a latency margin (i.e., the gap between the reduced latency 9

Figure 10: Normalized TTFT/TPOT and accuracy for Voltron and baselines across clusters and models. Table 4: Accuracy on LLM Benchmarks

5.2 gemma-3 Family 4B

12B

Qwen1.5 Family

Qwen2.5 Family

14B

7B

7B

MoE-A2.7B (14B)

14B

Benchmark Single MP/TP Voltron Single MP/TP Voltron MP/TP Voltron Single MP/TP Voltron MMLU Hellaswag GSM8K MATH

0.5753 0.7428 0.7703 0.4044

0.7158 0.8195 0.8946 0.4752

0.715 0.8191 0.8946 0.476

0.5748 0.654 0.7658 0.7905 0.5959 0.7051 0.215 0.332

0.6586 0.7936 0.7324 0.377

0.6094 0.773 0.6262 0.2784

0.6087 0.7727 0.6164 0.2754

0.7047 0.7693 0.7818 0.8223 0.8211 0.862 0.4536 0.5162

0.7747 0.8285 0.8681 0.5226

Average

0.6232 0.7263

0.7262

0.5379 0.6204

0.6404

0.5718

0.5683

0.6903 0.7425

0.7485

5.1.3 Execution Scenarios. We implement Voltron on top of open-source LLM framework llama.cpp, incorporating the proposed model execution plan and elastic model execution. To evaluate the effectiveness of Voltron, we compare it against 1) single-device LLM execution6 and 2) device heterogeneity-aware MP/TP [27, 55], which allocates shards or layers based on device performance while considering memory budget, under various execution scenarios. We evaluate the performance using two latency metrics: TTFT and TPOT. Following prior works [12, 30], we adopt QoS targets of 10s and 400ms for TTFT and TPOT, respectively. We evaluate the accuracy on the benchmark tasks described in Section 5.1.2. We also evaluate energy consumption of the devices by using an external power measurement device [44]. 5.1.4 Runtime Variance Scenarios. To demonstrate that Voltron can maintain high accuracy while satisfying the QoS requirements under runtime variance, we construct experimental scenarios that emulate realistic edge execution conditions and evaluate the performance and accuracy during multidevice LLM inference. For computational dynamics, we conduct experiments on Cluster 2 (Office) while increasing the number of conversation turns. For network variability, we emulate a scenario (depicted in Fig. 8(a) in Section 3.2.3) where a user moves within an office while carrying a mobile device and a smartwatch, with a tablet placed on a desk. In this setup, we collect cross-device wireless signal strength variations from the real deployment environment. 6 For the single-device execution, we run the LLM on the highest-performing

mobile device in the cluster. 10

Experimental Results and Analysis

5.2.1 Result Overview. Fig. 10 shows the TTFT/TPOT normalized to the respective QoS constraints, and average accuracy on various clusters of edge devices with three LLM families. Across all clusters and models, Voltron satisfies the QoS requirements for both TTFT and TPOT, as it allocates appropriate parallelism method to each layer with hybrid parallelism, by carefully considering the cluster capabilities (i.e., memory budgets and compute capabilities), model architecture, and the computational characteristics that vary across inference phases — the detailed analysis on the parallelism allocation is presented in Section 5.2.2. In addition, Voltron achieves significantly higher accuracy compared to the baselines. The accuracy gain comes from two key factors: 1) the ability to execute larger LLMs by leveraging multiple devices and 2) the use of importance-aware mixed precision, which preserves higher precision for importance layers (improving accuracy by up to 1.8% as in Table 4) as long as the memory budget allows. Voltron achieves 10.7% and 1.0% higher average accuracy than single-device execution and MP/TP, respectively. Note, although Voltron allocates higher precision than MP/TP on average, the validation accuracy is slightly lower in some cases — the difference is less than 0.3% though. MP/TP, however, are not satisfying the QoS constraints (or even result in execution failure) in those cases while Voltron robustly satisfies the QoS constraints. 5.2.2 Adaptability Analysis. Adaptability to Device and Model Heterogeneity: Voltron can adapt to device and model heterogeneity, by selecting efficient hybrid parallelism strategies which satisfy the latency constraints and memory budget. Fig. 11 shows the execution strategies used by Voltron across two heterogeneous clusters and four model types. Voltron determines the execution plans tailored to heterogeneous device capabilities across clusters and model types. For example, in Cluster 1 (Fig. 11(a)), most layers are executed with MP in the decode phase whereas some layers

Figure 11: Execution strategies determined by Voltron on the Cluster 1 and Cluster 2 across different models. Note Qwen2.5 14B Decode Phase in Cluster2 informs which layers are allocated with MP/TP.

Figure 12: Adaptability to (a) precision heterogeneity, (b) computation dynamics, and (c) network variability.

are executed with TP in Cluster 2 (Fig. 11(b)). This is because performance gap across devices is higher in Cluster 1, compared to Cluster 2, which makes TP less performing compared to MP due to the severe straggler problem. Voltron also executes each layer with an appropriate parallelism method according to the layer type. As shown in Fig. 11(b), Voltron executes most ATTNs, which have relatively low computational load, with MP, while executing most FFNs, which have relatively high computational load, with TP — in case of Qwen1.5-MoE-A2.7B, on the other hand, most FFNs are executed with MP due to sparse computation of the MoE. Adaptability to Precision Heterogeneity: By using mixed precision, Voltron can maximize accuracy while satisfying latency constraints. Fig. 12(a) shows the accuracy and latency under fixed- and mixed-precision for Qwen1.5, along with the execution strategy determined by Voltron. In the INT4 fixed-precision setting, Voltron satisfies the latency constraints but exhibits lower accuracy due to information loss. In the INT8 fixed-precision setting, Voltron achieves higher accuracy but fails to satisfy the QoS latency requirements due to the increased computational load. In contrast, by using mixed precision, Voltron achieves higher accuracy while satisfying the QoS latency constraints by effectively exploiting the latency-accuracy trade-off across precisions — as shown in Fig. 12(a), Voltron reduces memory usage by lowering the precision of certain layers, while still allocating high precisions to important layers, thereby achieving higher accuracy within the QoS constraint.

Adaptability to Computational Dynamics: Voltron can adapt to computational dynamics. Fig. 11(b) shows that Voltron employs different execution strategies across inference phases. In the prefill phase, Voltron uses MP for most layers regardless of the cluster and model type. This is because batched execution reduces the per-token computation time, amortizing the overhead of MP’s sequential execution. In contrast, during the decode phase, Voltron appropriately assigns MP or TP to each layer according to the execution environment. Voltron can also handle large KV cache size (i.e., the number of tokens accumulated in the KV cache) while satisfying latency constraints and maintaining high accuracy. Fig. 12(b) presents the accuracy and latency of the baselines and Voltron under the varying KV cache size increases. When the KV cache size is 0k, all baselines and Voltron satisfy the latency constraints because the computational load of ATTN is relatively low. However, when the KV cache size increases to 50k, none of the baselines satisfy the latency constraints, whereas Voltron still meets the latency constraints through computation scaling while achieving high accuracy. This is because 1) the importance-aware precision allocation enables Voltron to preserve high accuracy even when most layers are scaled to INT4 due to the reduced memory budget, and 2) the execution plan module appropriately generates layer-wise parallelism strategies for the heterogeneous precisions meeting the QoS constraint. This result implies that Voltron can strike the balanced memory-latency-accuracy trade-off point even for the long context applications. 11

accounts for 0.03% of TPOT. The precision allocation module also exhibits a small overhead (average of 0.8ms in our experiments) thanks to the low complexity of binary search — naive enumeration takes up to 24.1ms in the worst case though. Voltron also does not incur severe overhead for the actual plan modifications. Precision scaling and pruning incur I/O overhead, but Voltron mitigates the I/O overhead 1) by overlapping I/O operations with on-going computations, and 2) with shard-wise memory allocations. This decreases the modification overhead from 31.7ms to 4.1ms per output token. As a result, the total overhead accounts for only 1.2% of TPOT, making it feasible for Voltron to determine an adequate execution plan token-by-token.

Figure 13: Impact of Voltron Energy Optimization Adaptability to Network Variability: Voltron maintains high accuracy satisfying the latency constraints under various wireless signal strength. We consider a scenario where a table is located in a room while a user carrying a watch and a mobile devices moves away from the tablet (which is illustrated in Fig. 8(a) of Section 3.2.3). Fig. 12(c) shows the accuracy and latency of baselines and Voltron as the distance between the user and the tablet changes in the scenario. When the distance between the user and the tablet is small ( 1 ), Voltron and MP/TP satisfy the QoS constraint. However, as the user moves away from the tablet (i.e., from 1 to 3 ), TP violates the latency constraints due to the increased communication overhead under weakened signal strength, whereas Voltron still satisfies the latency constraint through communication scaling while maintaining nearly the same accuracy. When the user moves even farther, the device becomes disconnected from the cluster ( 4 ). In this case, both MP/TP cannot execute the full LLM due to the reduced memory budget, resulting in severe accuracy degradation. Even in this case, Voltron sacrifices only a small amount of accuracy by exploiting computation scaling — changing all layers to INT4 and additionally applying pruning, thereby enabling execution under the reduced memory budget.

6

Related Work

Distributed Inference Method: With the growing size of LLMs, there have been increasing pushes to execute LLM inference across multiple compute nodes (e.g., GPUs) in clusters. To run LLM inference on the multiple compute nodes, prior works have explored various distributed inference techniques, such as tensor parallelism [46] and model parallelism [29, 35], which partition model weights and computations across multiple compute nodes. However, most existing approaches are adopting a single parallelism technique uniformly across all layers of an LLM. Although these techniques are effective in homogeneous data center clusters, they offer limited flexibility in heterogeneous or resource constrained environments [27, 41, 55]. In addition, modern LLM architectures consist of diverse layer types with different computational and memory characteristics [2, 8, 49, 57], making the parallelism methods have different performance characteristics and memory footprint [29, 35, 68]. Different from the above techniques, Voltron enables flexible layer-wise hybrid parallelism that dynamically adapts execution strategies to heterogeneous devices and resource-constrained environments, thereby enabling efficient distributed LLM inference across edge devices. On-Device LLM inference: There have been increasing pushes to execute LLM inference at the edge, due to privacy concerns. To enable on-device LLM inference, several algorithm-level techniques, such as sLLMs [39, 65], pruning [40, 42], and quantization [17, 38, 62], have been proposed. sLLMs reduce model size by designing lightweight architectures tailored for edge environments. sLLMs are typically trained from the scratch [39, 65] or distilled from the larger LLMs [20]. Pruning partially removes parameters in a structured [37] or unstructured [36] manner, based on their importance [42]. Quantization reduces memory and computation by representing model parameters in low precision. Quantization adopts low precision (e.g., INT4) to entire

5.2.3 Energy Analysis. When Voltron adopts the energy optimization with different compression ratio (i.e., how much parameters of a LLM is reduced via precision scaling and pruning), the energy consumption can be significantly reduced via DVFS — the energy consumption can even be reduced by 59.0% which is almost the same as that of the single execution baseline (less than 4.4% of difference). Voltron does not incur a severe accuracy drop even in those cases (only 1.9% even under the 30% compression ratio), by adopting the importance-aware principle to the precision scaling and pruning. This result implies that Voltron can even strike better accuracy-energy trade-off point compared to the single device execution, making the multi-device LLM execution more promising for future applications. 5.2.4 Overhead Analysis. Voltron elastically adjusts the layerwise execution strategy and precisions only with a negligible overhead. The execution plan module incurs a negligible overhead (average of 0.1ms in our experiments), which only 12

parameters [9] or partial parameters in various granularity [15, 21, 56]. Though the above techniques effectively reduces resource requirements, they often incur non-trivial accuracy loss due to the reduced model capacity or information loss [45]. In contrast, Voltron enables on-device LLM inference through a system-level approaches that flexibly leverages multiple edge devices to execute larger models while minimizing accuracy loss. Several system-level works have tried to enable on-device execution of LLMs, of which parameter size is beyond the single device memory [5, 52, 63], at the expense of the increased latency. Voltron can be complementarily adopted along with such works, to further maximize the accuracy of LLM inference at the edge while satisfying the QoS constraints.

7

Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL] https://arxiv.org/abs/2404.14219 [3] Shohin Aheleroff, Xun Xu, Yuqian Lu, Mauricio Aristizabal, Juan Pablo Velásquez, Benjamin Joa, and Yesid Valencia. 2020. IoT-enabled smart appliances under industry 4.0: A case study. Advanced Engineering Informatics 43 (2020), 101043. doi:10.1016/j.aei.2020.101043 [4] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245 [cs.CL] https://arxiv.org/abs/2305.13245 [5] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 12562–12584. [6] amazon. 2023. Alexa. https://alexa.amazon.com/about [7] Apple. 2023. Siri. https://www.apple.com/siri/ [8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report. arXiv:2309.16609 [cs.CL] https://arxiv.org/ abs/2309.16609 [9] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. 2019. Post-training 4-bit quantization of convolution networks for rapiddeployment. arXiv:1810.05723 [cs.CV] https://arxiv.org/abs/1810. 05723 [10] Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150 [cs.CL] https: //arxiv.org/abs/2004.05150 [11] ChatGPT. 2025. ChatGPT. https://openai.com/index/chatgpt/ [12] Xiantao Chen, Moli Zhou, Renzhen Wang, Yalin Pan, Jiaqi Mi, Hui Tong, and Daisong Guan. 2019. Evaluating Response Delay of Multimodal Interface in Smart Device. In Design, User Experience, and Usability. Practice and Case Studies, Aaron Marcus and Wentao Wang (Eds.). Springer International Publishing, Cham, 408–419. [13] Cisco. 2020. Cisco Annual Internet Report. https://www.cisco.com/c/ en/us/solutions/collateral/executive-perspectives/annual-internetreport/white-paper-c11-741490.html. [14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] https://arxiv.org/abs/2110.14168 [15] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339 [cs.LG] https://arxiv.org/abs/2208.07339 [16] Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. 2025. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling. In Proceedings of the 52nd

Conclusion

To break through memory limitations of on-device LLM inference execution, in this paper, we propose Voltron, a novel on-device LLM inference framework which elastically exploits multiple devices adapting to the execution environments. Our characterization reveals that the performance and accuracy of multi-device LLM inference are strongly affected by characteristics of parallelism methods, diverse edge heterogeneity, and runtime variance. To address the issues, Voltron adopts layer-wise hybrid parallelism and mixed precision allocation, and elastically adjusts them adapting to the execution environment changes. Voltron achieves 10.2% higher accuracy, on average, compared to the baselines while satisfying QoS constraints. We believe Voltron will empower the edge intelligence for future applications.

References [1] 1X. 2026. NEO Home Robot. https://www.1x.tech/neo. [2] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang 13

[31] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. SIGARCH Comput. Archit. News 45, 1 (April 2017), 615–629. doi:10.1145/3093337. 3037698 [32] I. Kawrakow and contributors. 2023. k-quants. https://github.com/ggml-org/llama.cpp/pull/1684. [33] Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, and Tijmen Blankevoort. 2024. Pruning vs Quantization: Which is Better? arXiv:2307.02973 [cs.LG] https://arxiv.org/abs/2307.02973 [34] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [35] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668 [cs.CL] https://arxiv.org/abs/2006.16668 [36] Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. 2022. Parameter-Efficient Sparsity for Large Language Models Fine-Tuning. arXiv:2205.11005 [cs.AI] https://arxiv. org/abs/2205.11005 [37] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing 461 (2021), 370–403. doi:10.1016/j.neucom. 2021.07.045 [38] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for OnDevice LLM Compression and Acceleration. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 87–100. https://proceedings.mlsys.org/paper_files/paper/2024/ file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf [39] Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. 2024. MobileLLM: optimizing sub-billion parameter language models for ondevice use cases. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 1316, 24 pages. [40] Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. LLM-pruner: on the structural pruning of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 950, 19 pages. [41] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association for Computing Machinery, New York, NY, USA, 586–602. doi:10.1145/3669940.3707215 [42] Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better than One?. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/ 2c601ad9d2ff9bc8b282670cdd54f69f-Paper.pdf

Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 1283–1295. doi:10.1145/3695053.3730999 [17] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/ abs/2210.17323 [18] Google. 2025. Gemini. https://deepmind.google/technologies/gemini/ [19] Google. 2025. Gemini Developer API Pricing. https://ai.google.dev/ gemini-api/docs/pricing [20] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [21] Liwei Guo, Wonkyo Choe, and Felix Xiaozhu Lin. 2023. STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Vancouver, BC, Canada) (ASPLOS 2023). Association for Computing Machinery, New York, NY, USA, 791–803. doi:10.1145/3575693.3575698 [22] Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. 2024. What Matters in Transformers? Not All Attention is Needed. arXiv:2406.15786 [cs.LG] https://arxiv.org/abs/2406.15786 [23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. arXiv:2009.03300 [cs.CY] https://arxiv.org/abs/2009.03300 [24] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874 [cs.LG] https://arxiv.org/abs/2103.03874 [25] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. arXiv:2203.15556 [cs.CL] https://arxiv.org/abs/2203.15556 [26] Xiao Hong. 2026. ManusAI. https://www.manus.im. [27] Yang Hu, Connor Imes, Xuanang Zhao, Souvik Kundu, Peter A. Beerel, Stephen P. Crago, and John Paul Walters. 2022. PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices. In 2022 25th Euromicro Conference on Digital System Design (DSD). 298–307. doi:10.1109/DSD57027.2022.00048 [28] Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. 2025. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models. arXiv:2405.14917 [cs.LG] https: //arxiv.org/abs/2405.14917 [29] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. arXiv:1811.06965 [cs.CV] https://arxiv.org/abs/1811.06965 [30] Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. 2025. SAGESERVE: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling. Proc. ACM Meas. Anal. Comput. Syst. 9, 3, Article 61 (Dec. 2025), 24 pages. doi:10.1145/3771576

14

[57] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, JeanBaptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. 2025. Gemma 3 Technical Report. arXiv:2503.19786 [cs.CL] https://arxiv.org/abs/2503.19786 [58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010. [59] Xiong Wang, Linghe Kong, Zucheng Wu, Long Cheng, Chenren Xu, and Guihai Chen. 2020. SLoRa: towards secure LoRa communications with fine-grained physical layer features. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (Virtual Event, Japan) (SenSys ’20). Association for Computing Machinery, New York,

[43] Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better than One? arXiv:1905.10650 [cs.CL] https://arxiv.org/ abs/1905.10650 [44] Monsoon. [n. d.]. High Voltage Power Monitor. ([n. d.]). https: //www.msoon.com/high-voltage-power-monitor [45] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. 2021. A White Paper on Neural Network Quantization. arXiv:2106.08295 [cs.LG] https://arxiv.org/abs/2106.08295 [46] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 58, 15 pages. doi:10.1145/3458817.3476209 [47] Sangeun Oh, Ahyeon Kim, Sunjae Lee, Kilho Lee, Dae R. Jeong, Steven Y. Ko, and Insik Shin. 2019. FLUID: Flexible User Interface Distribution for Ubiquitous Multi-device Interaction. In The 25th Annual International Conference on Mobile Computing and Networking (Los Cabos, Mexico) (MobiCom ’19). Association for Computing Machinery, New York, NY, USA, Article 42, 16 pages. doi:10.1145/3300061.3345443 [48] Openai. 2025. ChatGPT Pricing. https://openai.com/chatgpt/pricing/ [49] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL] https://arxiv.org/abs/2412.15115 [50] Samsung. 2021. Samsung SmartThings Integrates Matter into Ecosystem, Bringing Matter Device Control to Multiple Samsung Products. https://www.samsungmobilepress.com/articles/samsungsmartthings-integrates-matter-into-ecosystem-bringing-matterdevice-control-to-multiple-samsung-products. [51] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538 [cs.LG] https://arxiv.org/abs/1701.06538 [52] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning. PMLR, 31094–31116. [53] Karan Singh. 2025. Tesla’s MCU Infotainment Computer vs FSD Computer: What’s the Difference? https://www.notateslaapp.com/news/2963/tesla. [54] Peter Steinberger. 2026. OpenClaw. https://openclaw.ai/. [55] Ki Tae Kim, Seok-Ju Im, and Eui-Young Chung. 2025. AsymGroup: Asymmetric Grouping and Communication Optimization for 2D Tensor Parallelism in LLM Inference. IEEE Access 13 (2025), 120591–120602. doi:10.1109/ACCESS.2025.3587387 [56] Chen Tang, Haoyu Zhai, Kai Ouyang, Zhi Wang, Yifei Zhu, and Wenwu Zhu. 2022. Arbitrary Bit-width Network: A Joint Layer-Wise Quantization and Adaptive Inference Approach. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM ’22). Association for Computing Machinery, New York, NY, USA, 2899–2908. doi:10.1145/3503161.3548001 15

NY, USA, 258–270. doi:10.1145/3384419.3430770 [60] Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 911–927. https://www.usenix.org/conference/osdi24/ presentation/wu-bingyang [61] Xiaodong Wu, Ran Duan, and Jianbing Ni. 2024. Unveiling security, privacy, and ethical concerns of ChatGPT. Journal of Information and Intelligence 2, 2 (2024), 102–115. doi:10.1016/j.jiixd.2023.10.007 [62] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient PostTraining Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 38087–38099. https://proceedings.mlr.press/v202/ xiao23c.html [63] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv:2406.06282 [cs.LG] https://arxiv.org/abs/2406. 06282 [64] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?

arXiv:1905.07830 [cs.CL] https://arxiv.org/abs/1905.07830 [65] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL] https://arxiv.org/abs/2401.02385 [66] Weizhi Zhang, Xinyang Zhang, Chenwei Zhang, Liangwei Yang, Jingbo Shang, Zhepei Wei, Henry Peng Zou, Zijie Huang, Zhengyang Wang, Yifan Gao, Xiaoman Pan, Lian Xiong, Jingguo Liu, Philip S. Yu, and Xian Li. 2025. PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time. arXiv:2506.06254 [cs.AI] https: //arxiv.org/abs/2506.06254 [67] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 46595–46623. https://proceedings.neurips. cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832Paper-Datasets_and_Benchmarks.pdf [68] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. arXiv:2201.12023 [cs.LG] https://arxiv.org/abs/2201.12023

16