TrimMoE A communication aware and adaptive depth framework for distributed edge inference

1 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

TRIMMOE: A COMMUNICATION-AWARE AND ADAPTIVE-DEPTH FRAMEWORK FOR DISTRIBUTED EDGE INFERENCE Ning Li, Shuting Bai, Xin Yuan, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Fellow, IEEE, Haijun Zhang, Fellow, IEEE

Abstract—Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission. The existing approaches mainly focus on how to reach a remote expert faster. However, in this paper, we instead consider whether a given layer, and the layers after it, need to be executed at all. To this end, a communicationaware adaptive-depth framework is proposed in this paper, termed TrimMoE, which couples layer skipping and confidencebased early exit with substitute execution and server-expert selection under a unified quality budget. Specifically, in the offline stage, TrimMoE freezes the backbone, trains the lightweight perlayer exit heads, calibrates the per-layer importance thresholds, and allocates the expert replicas by a skip/exit-aware redundancy benefit. In the online stage, a transition-aware look-ahead anticipates the token movement, so that the depth reduction targets the costliest transmissions, and besides, two feedback rules adapt the delay-quality weights and the exit threshold. Moreover, we prove that the substitution-and-skipping proxy degradation never exceeds the configured budget, and that the early exit is admitted only under a calibrated confidence gate. On a heterogeneous 10-server testbed with Switch-Base-8E, QwenMoE-A2.7B, and Mixtral-8x7B, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load, while keeping the task-quality degradation within a 2% bound. Index Terms—Edge Intelligence, Edge LLMs, Distributed Inference, Early exit, Layer skip

I. INTRODUCTION Large language models (LLMs) have been widely used and have shown impressive performance in many intelligent applications, such as conversational agents, code generation, multimodal perception, interactive content creation, etc. [1]. However, because the parameter scale and inference cost of LLMs are always large, it is difficult to directly execute them on resource-limited end devices or individual edge servers [2,3]. To address this issue, edge intelligence has emerged as an appealing direction, in which the inference workloads are jointly carried out by terminals and nearby edge infrastructure This paragraph of the first footnote will contain the date on which you submitted your paper for review, which is populated by IEEE. This work was supported in part by the grant from NSFC Grant no. 62571156, 62101159, 52475009, NSF of Shandong Grant no. ZR2021MF055, ZR2025QC666, the Research Grants Council of Hong Kong under the Areas of Excellence scheme grant AoE/E-601/22-R, and also the Opening Project of the Key Laboratory of Advanced Manufacturing and Intelligent Technology (Ministry of Education) at Harbin University of Science and Technology (KFKT202306). (Corresponding author: Xin Yuan). Ning Li and Haijun Zhang are with the School of Artificial Intelligence, University of Science and Technology Beijing, China (e-mail: ningli_polyuhk@outlook.com, zhanghaijun@ustb.edu.cn). Xin Yuan is with the School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China (e-mail: xin.yuan@hit.edu.cn).

to lower response time, save backbone bandwidth, and better protect user data. Among the different LLM frameworks, the mixture-of-experts (MoE) architecture is especially suitable for edge deployment, since each token only activates a small portion of the experts, which achieves a good balance between representational capacity and per-token computation [4,5]. To deploy LLMs on the edge, the existing works generally follow two routes. The first route is model compression, such as quantization, structured pruning [6,7], knowledge distillation [8], and low-rank factorization [9,10], which reduces the model footprint and computation cost before inference. However, model compression often reduces the inference accuracy, and for extremely large LLMs the compressed model can still exceed the memory and computation limits of a single edge node [11]. The second route is distributed inference, in which a large model is split and cooperatively executed over several servers [12]. Compared with model compression, distributed inference has better scalability, because the memory and computation can be amortized across multiple nodes [13]. Therefore, cooperative inference across edge servers becomes indispensable once a single server can neither hold the full model nor provide adequate inference throughput. Distributed serving of large models has been extensively studied in cloud data centers, where the partitions of a model are executed over many servers to improve throughput [14,15]. In the cloud, the servers are usually connected by high speed transmission technologies, such as RDMA [16], InfiniBand [17], so the communication cost of cross-server execution can be well controlled. However, it is hard to directly apply these cloud-oriented designs to the edge. First, unlike the tightly coupled cloud, the geographically dispersed edge servers are usually connected over the public Internet rather than dedicated high-bandwidth links [18], so the inter-server bandwidth is much more limited, the latency is higher, and the time-varying network state further improves the complexity of this problem. Second, the edge servers differ widely in memory budget, computation capability, and load, which makes it more difficult to distribute LLMs across them [19]. These edge-specific properties make distributed MoE Shuting Bai is with the School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China (e-mail: bstican@163.com). Wenchao Xu is with the Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong (e-mail: wenchaoxu@ust.hk) Athanasios V. Vasilakos is with the Center for AI Research (CAIR), University of Agder (UiA), Grimstad, Norway (e-mail: th.vasilakos@gmail.com). Song Guo is with the Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (e-mail: songguo@cse.ust.hk) Mentions of supplemental materials and animal/human rights statements can be included here. Color versions of one or more of the figures in this article are available online at http://ieeexplore.ieee.org

2 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID serving particularly difficult. In our previous work, the samelayer expert similarity is leveraged, so that a token whose routed expert is not available locally can be served by a functionally similar local substitute instead of being forwarded to a remote server, which already reduces a large fraction of cross-server transmissions [37]. However, there are still two kinds of interserver traffic. First, every token is still pushed through all MoE layers, even though many layers contribute marginally to a given token, and a non-trivial part of these layers needs a remote hop to execute. Second, every token is still processed to the full depth, even when its representation has already stabilized at a much shallower layer. Most existing works concentrate on how to reach the remote computation unit more efficiently, but seldom consider whether it is necessary to traverse that layer, or the remaining layers, at all. Therefore, for distributed edge MoE inference, eliminating the unnecessary layer executions and the transmissions they trigger, rather than only accelerating them, is still an open and appealing problem. Fortunately, recent works reveal that the computation of an MoE model is highly redundant along the depth for individual tokens [20]. First, the MoE layers are not equally important: the gating response of the routed expert provides a per-token and per-layer importance signal, and the layers with weak importance can be bypassed through the residual path with little impact on the final prediction [21]. Second, many tokens reach a representation that already matches the full-depth result well before the last layer, so a lightweight confidence estimator can decide that the remaining layers are unnecessary and terminate the inference early [22]. These two observations are particularly valuable in the distributed edge scenario. On the one hand, if a low-importance layer is one that would otherwise force a crossserver hop, skipping it removes the transmission entirely. On the other hand, if a token can confidently exit early, all the remaining transmission-prone layers are removed at once. In other words, layer skipping and early exit change the question from “how to transmit a token to the right remote expert” into “whether that layer, and the layers beyond it, need to be executed on the remote expert at all”. To the best of our knowledge, this is the first work which jointly exploits communication-aware layer skipping and confidence-based early exit, coordinated with server-expert selection along the token trajectory, to further suppress the cross-server transmission in distributed edge MoE inference. Nevertheless, it is far from trivial to turn this idea into a practical framework. First, the skip decision, the exit decision, and the server-expert choice are tightly coupled along the token’s path, i.e., a skipped layer keeps the token on the current server while consuming part of the quality budget, an early exit cancels all the remaining layers once the confidence is met, and the server selected for an executed layer determines where the next layer sources the token. Therefore, handling these decisions in isolation is not enough. Second, skipping a layer or exiting early reduces the communication but may reduce the inference quality, so the framework must respect a controllable per-user quality degradation budget, while ensuring that the early exit never terminates on an under-refined representation. Third, a sound decision at the current layer depends on the transmission cost and the remaining budget of the layers that have not been computed yet, which needs a look-ahead mechanism to anticipate the future token movement without

actually executing those layers. Finally, because the edge servers are heterogeneous in memory and computation, the offline deployment must concentrate the redundant expert replicas on the experts that genuinely reduce transmission after skipping and early exit are enabled, rather than on the experts that are frequently bypassed or rarely reached. To address the above challenges, a communication-aware adaptive-depth framework for distributed edge MoE inference is proposed in this paper, termed TrimMoE. Specifically, in the offline stage, TrimMoE freezes the backbone and trains one lightweight per-layer exit head to produce a calibrated confidence score and calibrates a per-layer importance threshold so that only low-importance layers become skip candidates within a budget-derived tolerance. Then, it collects the expert transition statistics together with a compact tokensimilarity reference set for runtime prediction. Based on the similarity-aware deployment inherited from our previous work, TrimMoE further refines the redundant placement by a skipexit-aware redundancy benefit, so that improving the efficiency of edge memory. In the online stage, for each token at each layer, TrimMoE jointly determines whether to skip, to exit, or to execute, and on which server with which expert. The purpose is to minimize the sum of a normalized per-step cost and a lookahead cost-to-go estimated from the transition statistics. Besides, a pair of runtime feedback rules continuously adapts the delay-quality weights and the exit confidence threshold to the changing network and workload conditions. In this way, TrimMoE removes both the unnecessary layer executions and their induced transmissions over the bandwidth-limited and dynamic edge links, while keeping the inference quality within a controllable budget. The main contributions of this paper can be summarized as follows. l We formulate the distributed edge MoE inference problem as a joint optimization of layer skipping, early exit, and server-expert selection over the token trajectory. Different from the existing formulations that treat every layer as mandatory and every token as full-depth, the proposed formulation allows a token to skip the transmission-inducing low-importance layers and to terminate early under a confidence requirement, while updating the token location after each executed layer and respecting a per-user quality degradation budget. l We design an offline calibration and communication-aware deployment scheme. Without fine-tuning the backbone, we train the per-layer exit heads on the cached representations to obtain a quality-preserving exit indicator, calibrate the layer importance thresholds from the empirical skipdegradation relationship, and refine the redundant expert deployment by a skip-exit-aware redundancy benefit, which prioritizes the experts whose replication genuinely reduces cross-server transmission. l We develop a communication-aware online skip-exit and server-expert selection algorithm. The algorithm jointly weighs the transmission delay, the computation delay, the substitution and skip-induced quality loss, and the lookahead cost of future token movement. It decides not only whether a layer should be skipped, executed, or exited, but also which server and expert should serve an executed layer, and it self-calibrates its weights and exit threshold from the

3 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID runtime observations. l We implement TrimMoE on a real distributed edge testbed. The experimental results demonstrate that, compared with the representative baselines, TrimMoE markedly reduces the average latency, the tail latency, the cross-server traffic, and the remote execution ratio, while keeping the introduced inference quality degradation small and bounded by the configured budget. The remainder of this paper is organized as follows. Section II surveys the related work. Section III describes the system model and formulates the optimization problem. Section IV presents the offline exit-head training and communicationaware deployment scheme. Section V details the communication-aware online skip-exit and server-expert selection algorithm. Section VI reports the experimental evaluation. Finally, Section VII concludes this paper. II. RELATED WORKS In this section, we review the existing studies related to this paper from three aspects: 1) collaborative inference and distributed deployment of large models at the edge; 2) MoE serving optimization and communication reduction; and 3) layer skipping and early exit for efficient inference. Although these studies have improved the efficiency of large-model inference from different perspectives, they cannot effectively address the problem studied in this paper, because none of them jointly exploits communication-aware layer skipping and confidence-based early exit, coordinated with substitute-aware server-expert selection along the token trajectory, to remove the cross-server transmission triggered by the unnecessary layers in distributed edge MoE inference. A. Collaborative Inference and Distributed Deployment of Large Models at the Edge The deployment and inference acceleration of large models in edge environments have been studied in many previous works. The early works mainly focus on model partitioning, collaborative inference, task offloading, and resource allocation, so as to reduce the end-to-end latency and improve the efficiency of resource-constrained edge intelligence. For conventional DNNs, Mohammed et al. [23] formulate the adaptive DNN partitioning and offloading as a matching game to accelerate the distributed inference, and PArtNNer [24] proposes a platform-agnostic adaptive edge-cloud DNN partitioning method, which automatically locates the optimal partition point to minimize the end-to-end latency. Besides, a comprehensive survey in [25] summarizes the split-computing and early-exiting techniques, together with their modeling and optimization objectives, for cooperative DNN inference at the edge. With the rapid development of LLMs, the recent works have extended the collaborative inference from conventional DNNs to large generative models. Edge-LLM [26] proposes a collaborative framework for LLM serving in edge computing, and EdgeShard [12] partitions an LLM into shards and jointly optimizes the device selection and the model partition for collaborative edge inference. To further reduce the communication cost of tensor-parallel inference, a communication-efficient framework for distributed on-device LLM inference over wireless networks [27] partitions the

weight tensors of an LLM across multiple devices and designs an over-the-air all-reduce scheme. Besides, Jupiter [28] introduces a fast and resource-efficient pipelined architecture, which exploits the different characteristics of the prefill and decoding phases for collaborative LLM inference on edge devices. However, these works are fundamentally different from this paper. First, they mainly consider model partitioning, tensor partitioning, or inference assignment, and do not exploit the internal expert similarity or the depth redundancy of MoE models as a deployment and execution dimension. Second, they assume the exact model execution, i.e., when a model component is assigned to a remote node, the corresponding token representation must still be transmitted to that node for execution. Third, although these works improve the efficiency of distributed inference, they only accelerate the necessary transmissions, rather than considering whether it is necessary to execute a layer, or the remaining layers, at all. Therefore, they cannot achieve the additional communication reduction studied in this paper. B. MoE Serving Optimization and Communication Reduction The efficiency of MoE inference has attracted increasing attention in recent years, because the sparsity of MoE models introduces not only computation challenges but also new communication and memory challenges. Specifically, when the activated experts of the tokens are distributed across different devices or servers, the expert dispatch and communication may become a dominant bottleneck. Several works optimize MoE inference through expert offloading, caching, prefetching, replication, and communication scheduling. AdapMoE [29] adopts an adaptive sensitivity-based expert gating and management method to reduce the on-demand loading overhead of the sparsely activated experts. SlimCaching [30] investigates the edge caching of MoE experts for distributed low-latency inference, which stores the frequently activated experts on devices and caches the remaining experts across the edge network. DiffMoE [31] proposes a priority-driven differential expert caching hierarchy, together with a lightweight predictor, to improve the batched MoE inference. PROBE [32] further co-balances the computation and communication through real-time predictive prefetching, dynamic expert replication, and token assignment. MegaScale-Infer [33] disaggregates the attention and expert modules and adopts the disaggregated expert parallelism for efficient large-scale MoE serving. Besides, some recent works also investigate the distributed or wireless MoE deployment at the edge: WDMoE [34] proposes a wireless distributed MoE architecture, which collaboratively deploys the experts across base-station edge servers and mobile devices, and MoE2 [35] optimizes the collaborative inference for edge LLMs under the energy and latency constraints. Together with the sparselyactivated MoE models themselves [4] and the recent MoE survey [5], these works demonstrate that communication-aware expert management is important to the efficiency of MoE inference. However, the above works still differ from this paper in several key aspects. First, their objective is mainly to access the exact target experts more efficiently through better caching, replication, prefetching, dispatch, or scheduling. In contrast, this paper considers whether the remote target-expert

4 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID invocation itself can be avoided. Second, most existing works assume that the routed target expert is the only valid execution and must always be executed, so they neither exploit the functionally similar local substitutes nor allow a lowimportance layer to be skipped or the inference to terminate early. Therefore, they reduce the communication by transporting the tokens faster, whereas this paper reduces the communication by eliminating the layer executions and the cross-server transmissions they would otherwise trigger. C. Layer Skipping and Early Exit for Efficient Inference Another line of research reduces the inference cost along the depth dimension, which is motivated by the observation that the per-token computation of deep models is highly redundant. For the layer-level redundancy, ShortGPT [20] introduces a blockinfluence metric and shows that many layers of LLMs can be removed with negligible accuracy loss. Mixture-of-Depths [21] learns to dynamically allocate the computation to a subset of tokens at each layer, and bypasses the remaining tokens through the residual path under a fixed compute budget. For the early exit, Confident Adaptive Language Modeling [22] dynamically allocates different amounts of computation per token and exits the decoding once a calibrated confidence measure is met, with provable quality guarantees, and LayerSkip [36] combines the layer-dropout training with an early-exit loss and selfspeculative decoding to enable accurate exit at earlier layers. However, these works differ from this paper in two essential aspects. First, they are designed for single-device inference and aim to reduce the FLOPs or the decoding steps. They are not communication-aware and are not formulated for distributed multi-server edge MoE serving, so they ignore the fact that skipping or exiting a layer can additionally remove a crossserver transmission. Second, they treat layer skipping and early exit as standalone compute-saving mechanisms, and do not couple them with the server-expert selection along the token trajectory, nor enforce a per-user quality-degradation budget jointly with the server selection and the substitute-execution candidates. In contrast, this paper jointly exploits communication-aware layer skipping and confidence-based early exit, coordinated with substitute-aware server-expert selection, to remove both the unnecessary layer executions and their induced cross-server transmissions, while keeping the inference-quality degradation within a controllable budget. Ⅲ. NETWORK MODEL AND PROBLEM STATEMENT In this section, we first briefly introduce the distributed edge MoE deployment and the similarity-based substitute execution model established in our previous work [37]. Then, we introduce the expert importance and skip-layer model, the confidence-aware early-exit model, the inference delay model, and the unified quality degradation model that are used in this paper. Based on these models, the problem solved in this paper is described in detail. A. Network Model and Distributed MoE Deployment We consider the same distributed edge intelligence system as in [37]. Let 𝐷 = {𝑑! , 𝑑" , … , 𝑑# } denote the set of mobile users and 𝑆 = {𝑆! , 𝑆" , … , 𝑆$ } denote the set of edge servers. For user 𝑑% , its associated edge server is 𝑆&(%) , which receives the request and serves as the initial execution server. The edge servers are geographically distributed and connected through

ordinary Internet links. Let 𝐵),+ and 𝜏),+ denote the available bandwidth and the propagation-plus-queuing delay between 𝑆) and 𝑆+ , both of which differ with different server pairs. The servers are heterogeneous, where 𝐶) and 𝐹) denote the available GPU memory capacity and computation capability of 𝑆) , respectively.

Fig.1 Network model

The MoE model deployed at the edge is denoted by ℳ , which contains 𝑁 MoE layers. In the 𝑙-th MoE layer, the expert set is 𝐸, = {𝑒,! , 𝑒," , … , 𝑒, ! } , where expert 𝑒,. uses GPU memory 𝑝,. and incurs computation workload 𝑐,. for processing a token representation. Since the whole model cannot be deployed on a single server, the experts are distributed across the edge servers. Let 𝑥),,,. = 1 indicate that expert 𝑒,. is deployed on 𝑆) . The deployment satisfies the GPU memory constraint and the coverage constraint: -! . ∑0 ∀𝑆) ∈ 𝒮 (1) ,/! ∑./! 𝑥),,,. 𝑝, ≤ 𝐶) , ∑$ 𝑥 ≥ 1 , ∀𝑙 ∈ [1, 𝑁], 𝑗 ∈ [1, 𝐸 ] (2) )/! ),,,. , To reduce the cross-server transmission caused by remote expert invocation, our previous work [37] groups functionally similar same-layer experts and disperses them across edge servers, so that each server can preserve local substitute candidates. For a target expert 𝑒,1 selected by the router, let 𝑉,,1 = {𝑒,1 } ∪ 𝑅,,1 denote its feasible execution expert set, where 𝑅,,1 is the set of feasible substitute experts whose routerinduced similarity with 𝑒,1 exceeds a layer-aware threshold [37]. When a substitute expert 𝑒,2 ∈ 𝑅,,1 is executed instead of the target expert, a substitution quality loss is introduced: 0, 𝑘 = 𝑟, (𝑞), 𝒬, (𝑟, (𝑞), 𝑘) = I !3456 !(1!(8),2) (3) 𝜙L M , 𝑘 ≠ 𝑟, (𝑞), " where 𝜙(⋅) is a non-decreasing function. The detailed grouping, deployment, and similarity computation follow [37]. B. Expert Importance and Skip-Layer Model For an inference request of user 𝑑% , let 𝑄% denote its token set, and 𝑟, (𝑞) denote the target expert selected by the router for token 𝑞 in the 𝑙-th MoE layer. Since the routers are small and replicated on all servers [37], the routing decision of each layer is available at the server where the token currently resides, before the expert is actually executed. We exploit the router output not only to obtain the target expert, but also to estimate how important this layer is for the token. At layer 𝑙, the router activates a top-k expert set for token 𝑞, denoted 𝐸,(8) , where 𝑔,2 (𝑞) is the gating weight of the k-th activated expert. Models with a shared expert treat the shared expert as always locally resident; only routed experts in 𝐸,(8) are subject to placement and selection. The layer importance is the aggregate gating mass of the activated set: 𝐼%,8,, = ∑2∈-!(#) 𝑔,2 (𝑞), 𝐼%,8,, ∈ (0,1] (4)

5 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID A larger 𝐼%,8,, means the router is more confident that this specific expert is decisive for the token, so omitting it would distort the representation more. Conversely, a small 𝐼%,8,, indicates that the activated expert contributes little, and the layer can be bypassed through the residual path with limited impact. Since 𝐼%,8,, is computed from the router output alone, the importance of a layer is known without executing its expert, which makes the skip decision feasible at runtime. Let the skip decision be 𝛿%,8,, ∈ {0,1} , where 𝛿%,8,, = 1 means the expert computation of token 𝑞 in the 𝑙-th MoE layer is skipped and the token representation is forwarded through the identity (residual) path. When a layer is skipped, the predicted quality degradation is modeled as a nondecreasing function of its importance: Λ%,8,, = 𝜌V𝐼%,8,, W (5) where 𝜌(⋅) is a non-decreasing function. The skip-layer mechanism is meaningful only when it removes cross-server transmission. If a feasible expert in 𝑉,,1!(8) is already available on the current server, the layer can be processed locally by exact or substitute execution at almost no transmission cost, and there is no need to skip it. Therefore, skipping is preferentially considered for a layer whose feasible experts all reside on remote servers, i.e., whose execution would otherwise require the token to be transmitted across servers. In this way, a skip simultaneously saves the computation of an unimportant expert and the transmission that would be needed to reach it. C. Confidence-Aware Early-Exit Model Besides skipping individual layers, a token may not need to traverse all 𝑁 layers. Once the representation of a token is already sufficient for the task requirement, the remaining layers can be terminated, which avoids both the residual computation and any cross-server transmission on the remaining trajectory. To describe the depth that each token actually traverses, we introduce the active indicator 𝑦%,8,, ∈ {0,1} , where 𝑦%,8,, = 1 means token 𝑞 is still active when entering the 𝑙 -th MoE layer. Since a terminated token is never reactivated, the active indicator is monotonically non-increasing along the depth: 𝑦%,8,! = 1, 𝑦%,8,, ≥ 𝑦%,8,,:! , ∀𝑙 ∈ [1, 𝑁 − 1] (6) The exit layer of token 𝑞 is then 𝐿%,8 = ∑0 𝑦 , and ,/! %,8,, layers 𝑙 > 𝐿%,8 are not executed. Because the ground-truth output is not available during inference, the sufficiency of a token cannot be measured by accuracy directly. Instead, we attach a lightweight exit head ℎ, (⋅) to each layer. Let 𝐡%,8,, ∈ ℝ; denote the hidden representation of token 𝑞 of user 𝑑% at the output of the 𝑙-th MoE layer, where 𝑑 is the model hidden dimension. The exit head maps this representation to a confidence score: 𝜉%,8,, = ℎ, (𝐡%,8,, ) ∈ [0,1] (7) where 𝜉%,8,, reflects how stable and decisive the current representation is, and is consistent with established early-exit Transformer practice. Let 𝑝<% denote the task confidence requirement of user 𝑑% . Token 𝑞 is allowed to exit after the 𝑙-th MoE layer only if its confidence already meets the requirement: 𝑦%,8,,:! = 0 ⇒ 𝜉%,8,, ≥ 𝑝<% (8) Equation (8) guarantees that early exit is triggered only when the token is confident enough, so that early exit is qualitypreserving by construction rather than a source of uncontrolled

error. It should be noted that the skip-layer mechanism and the early-exit mechanism may act on the same layer, and their interaction must be specified to keep the confidence judgment well defined. When the 𝑙-th MoE layer is skipped (𝛿%,8,, = 1), the expert is not actually executed, and the hidden representation 𝐡%,8,, is merely propagated from the input through the identity (residual) path rather than refined by an expert. In this case the confidence score produced by the exit head does not reflect any newly performed computation and is therefore unreliable for exit judgment. To avoid making an exit decision on a representation that has not been updated, the exit judgment is performed only at layers that are actually executed, while a skipped layer never triggers early exit: 𝛿%,8,, = 1 ⇒ 𝑦%,8,,:! = 𝑦%,8,, (9) According to (9), when a layer is skipped, the active state is passed unchanged to the next layer, so that skipping and early exit cannot be triggered simultaneously at the same layer. This rule ensures that the confidence condition (8) is always evaluated on a representation refined by an executed expert, and that the two depth-reduction mechanisms remain mutually consistent along the token trajectory. When a token is skipped or exited, its representation is propagated to the dependent computations through the residual path, so that the dependencies of the subsequent tokens remain well defined. In this paper, we focus on the placement and invocation of MoE experts, which dominate the memory occupation and crossserver transmission overhead. Attention placement. Attention sublayers, routers, and shared experts are dense and comparatively small; one copy of each is deployed on every participating server, with the KV cache colocated on the token’s current server. Consequently, attention is always computed locally and never triggers cross-server transmission; all cross-server traffic is induced by routed-expert (FFN) execution when a required expert is unavailable locally. A layer skip bypasses only the FFN sublayer, while the attention sublayer of that block is still computed locally. Under this placement, the latency model treats local attention as a fixed per-layer computation term and attributes all cross-server delay to remote expert execution. D. Token Trajectory and Inference Delay Model Let 𝑢%,8,,,),2 = 1 indicate that token 𝑞 of user 𝑑% is processed by expert 𝑒,2 on server 𝑆) in the 𝑙 -th MoE layer. A serverexpert pair is selected only for a layer that is active and not skipped. Let 𝜀%,8,, = 𝑦%,8,, (1 − 𝛿%,8,, ) denote the executed indicator. Then, -! ∑$ (10) )/! ∑2/! 𝑢%,8,,,),2 = 𝜀%,8,, , ∀𝑑% , 𝑞, 𝑙 2 𝑢%,8,,,),2 ≤ 𝑥),,,2 , 𝑢%,8,,,),2 = 0 if 𝑒, ∉ 𝑉,,1!(8) (11) -! The execution-server indicator is 𝜇%,8,,,) = ∑2/! 𝑢%,8,,,),2 . Let ℓ%,8,, denote the server on which the token resides before entering the 𝑙-th MoE layer. The initial location is the access edge server, and the location evolves according to whether the layer is executed or skipped: ℓ%,8,! = 𝑎(𝑖) (12) arg ) (𝜇%,8,,,) = 1), 𝜀%,8,, = 1 ℓ%,8,,:! = h (13) ℓ%,8,, , 𝛿%,8,, = 1 According to (12), a skipped layer keeps the token at its current server, so it incurs no transmission. The transmission

6 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID delay of token 𝑞 in the 𝑙-th MoE layer is only paid when the layer is executed on a server different from the current location: trans 𝑇%,8,, = ∑$ )/! 𝜇%,8,,,) 𝟙V 𝑚 ≠ ℓ%,8,, W o>

=! ℓ&,#,! ,(

+𝜏ℓ&,#,!,) q (14)

where 𝑧, is the size of the intermediate token representation and 𝟙(⋅) is the indicator function. The computation delay is incurred only by executed layers: -

comp

@)

! ! 𝑇%,8,, = ∑$ )/! ∑2/! 𝑢%,8,,,),2 ⋅ A

(15)

(

A skipped layer (𝛿%,8,, = 1) and an inactive layer (𝑦%,8,, = 0) contribute zero to both (14) and (15). Therefore, the total delay of token 𝑞 and the total inference delay of user 𝑑% are: 𝑇% = ∑8∈B& ∑0 (16) ,/! 𝑇%,8,, comp trans where 𝑇%,8,, = 𝑇%,8,, + 𝑇%,8,, . Equation (16) shows the source of the additional reduction in this paper: a skipped remote layer removes one transmission term in (14), and early exit removes all the delay terms of the layers beyond 𝐿%,8 . E. Unified Quality Degradation Model In this paper, the inference quality of a token is affected by three approximation behaviors: substitute execution, layer skipping, and early exit. By the confidence condition (8), early exit is only triggered when the task requirement is already met, so it is modeled as a hard feasibility constraint rather than a quality loss. The quality degradation that must be budgeted therefore comes from substitution and skipping. For token 𝑞 in the 𝑙-th MoE layer, the per-layer degradation is: -! $ 𝐷%,8,, = ∑ 𝛿%,8,, Λtv (17) )/! ∑2/! 𝑢%,8,,,),2 𝑄, (𝑟, (𝑞), 𝑘) + st %,8,, stttttttttutttttttttv tut substitution

CD5E

The first term is the substitution quality loss inherited from [37], and the second term is the skip-induced degradation defined in (5). Both degradation sources are expressed on a common probabilistic scale. 𝑄, (𝑟, 𝑘) is the fraction of calibration tokens whose final prediction changes when expert 𝑒,1 is replaced by substitute 𝑒,2 ; Λ, (𝑙) is the fraction whose prediction changes when layer 𝑙 is skipped at importance 𝐼 . Both lie in [0,1] on the same calibration distribution, so they share a common unit and can be summed within the budget w (𝑖). 𝐷 The total quality degradation of user 𝑑% is: 𝐷% = ∑8∈B& ∑0 (18) ,/! 𝐷%,8,, To guarantee inference quality, the total degradation of each user should not exceed an acceptable budget: 𝐷% ≤ 𝐷%6FG , ∀𝑑% ∈ 𝐷 (19) The above model indicates that the server selection, expert selection, skip decision, and exit decision are jointly related to inference delay and quality degradation. For example, if the feasible experts of the next MoE layer all reside on remote servers, the token can be transmitted to a remote server for exact or substitute execution, which preserves quality but increases transmission delay; alternatively, if the layer is of low importance, the token can skip the layer and stay on the current server, which removes the transmission delay but consumes part of the quality budget. Moreover, once the token confidence reaches the task requirement, terminating the remaining layers removes both the residual computation and the remaining crossserver transmission. Therefore, it is necessary to jointly consider inference delay and quality degradation when determining the deployment, the token trajectory, the skip

decision, and the exit decision. F. Problem Statement From the above subsections, we obtain the inference delay 𝑇% and the quality degradation 𝐷% of user 𝑑% . The purpose of this paper is to minimize the inference delay and the quality degradation by jointly optimizing the expert deployment strategy 𝑿, the server-expert selection strategy 𝑼 = {𝑢%,8,,,),2 }, the skip-layer strategy 𝚫 = {𝛿%,8,, } , and the early-exit strategy 𝒀 = {𝑦%,8,, }, where the expert grouping and the feasible substitute sets are obtained as in [37]. Since the inter-server links and the computation resources are shared, the per-server workload within a scheduling slot Δ𝑡 is bounded. Let 𝐷(𝑡) denote the set of requests served in slot 𝑡 . The computation workload assigned to 𝑆) in slot 𝑡 is: -! 2 𝑊) (𝑿, 𝑼; 𝑡) = ∑;&∈H(I) ∑8∈B& ∑0 (20) ,/! ∑2/! 𝑢%,8,,,),2 ∙ 𝑐, Then, the problem P0 is expressed as: 𝐏𝟎: min{∑#%/! 𝑇% , ∑#%/! 𝐷% } s.t.: -! . ∑0 (c.1) ,/! ∑./! 𝑥),,,. 𝑝, ≤ 𝐶) , ∀𝑆) ∈ 𝒮 ∑$ (c.2) )/! 𝑥),,,. ≥ 1 , ∀𝑙 ∈ [1, 𝑁], 𝑗 ∈ [1, 𝐸, ] 𝑥),,,. ∈ {0,1}, 𝑢%,8,,,),2 ∈ {0,1} (c.3) 𝛿%,8,, ∈ {0,1}, 𝑦%,8,, ∈ {0,1} (c.4) 𝑦%,8,! = 1, 𝑦%,8,, ≥ 𝑦%,8,,:! , ∀𝑙 ∈ [1, 𝑁 − 1] (c.5) 𝛿%,8,, = 1 ⇒ 𝑦%,8,,:! = 𝑦%,8,, , ∀𝑑% , 𝑞, 𝑙 (c.6) -! $ ∑)/! ∑2/! 𝑢%,8,,,),2 = 𝑦%,8,, (1 − 𝛿%,8,, ), ∀𝑑% , 𝑞, 𝑙 (c.7) 2 𝑢%,8,,,),2 ≤ 𝑥),,,2 , 𝑢%,8,,,),2 = 0 if 𝑒, ∉ 𝑉,,1!(8) (c.8) 𝑦%,8,,:! = 0 ⇒ 𝜉%,8,, ≥ 𝑝<% , ∀𝑑% , 𝑞, 𝑙 (c.9) 𝐷% ≤ 𝐷%6FG , ∀𝑑% ∈ 𝐷 (c.10) 𝑊) (𝑿, 𝑼; 𝑡) ≤ 𝐹) Δ𝑡, ∀𝑆) ∈ 𝑆, ∀𝑡 (c.11) In P0, constraints (c.1) to (c.3) are the deployment and binary-selection constraints inherited from [37]. Constraint (c.4) means that the skip and exit decisions are binary. Constraint (c.5) guarantees that the active indicator is monotonically non-increasing along the depth, so that a terminated token is never reactivated. Constraint (c.6) is the skip-exit coordination constraint: when a layer is skipped, its representation is not refined by an executed expert, so the active state is passed unchanged to the next layer and early exit cannot be triggered at the same layer. Constraint (c.7) means that a server-expert pair is selected if and only if the layer is active and not skipped. Constraint (c.8) guarantees that an executed expert is deployed on the selected server and is either the target expert or a feasible substitute expert. Constraint (c.9) guarantees that early exit is triggered only when the task confidence requirement is satisfied. Constraint (c.10) bounds the total quality degradation of each user, and (c.11) bounds the per-server workload within each scheduling slot. This problem is difficult to solve for the following reasons. First, the objectives are conflicting: skipping a remote layer or exiting early reduces transmission and computation delay, but skipping consumes the quality budget, while exact remote execution preserves quality at the cost of extra transmission delay. Second, the deployment, selection, skip, and exit decisions are all discrete, which makes the solution space extremely large. Third, the decisions of different layers are coupled along the token trajectory: the execution server selected in one layer determines the source server of the next

7 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID executed layer, and skipping or exiting at one layer changes both the residing server and the remaining trajectory. Fourth, the importance 𝐼%,8,, and the confidence 𝜉%,8,, of a layer are only revealed when the token reaches that layer, so the skip and exit decisions must be made in an online and predictive manner. Therefore, finding the optimal solution of P0 is difficult. Since the objectives in P0 are different, a weight-based ‰% = 𝐷% / approach is introduced. Let 𝑇‡% = 𝑇% /𝑇%ref and 𝐷 𝐷%6FG denote the normalized inference delay and the normalized quality degradation of user 𝑑% , where 𝑇%ref is the inference delay under the pure full-depth exact-execution strategy on the same deployment. The utility function is: ‰% 𝑈(𝑿, 𝑼, 𝚫, 𝒀) = 𝜔J ∑#%/! 𝑇‡% + 𝜔H ∑#%/! 𝐷 (21) where 𝜔J and 𝜔B are the weights of inference delay and substitution quality loss, respectively, and 𝜔J + 𝜔B = 1. Thus, P0 is transformed into the single-objective problem: 𝐏𝟏: min 𝑈(𝑿, 𝑼, 𝚫, 𝒀) 𝑿,𝑼,𝚫,𝒀

According to (21), a larger 𝜔J favors low latency through more aggressive skipping and earlier exit, while a larger 𝜔H favors quality preservation through more exact and full-depth execution. Therefore, the weight-based approach can flexibly achieve the tradeoff between inference delay and inference quality. Ⅳ. OFFLINE EXIT-HEAD TRAINING AND COMMUNICATION AWARE DEPLOYMENT In Section III, the joint optimization problem P1 has been formulated. In this section, we therefore present the offline procedure that produces these calibration quantities and refines the deployment strategy 𝑋 in a communication-aware manner.

Fig.2. Offline exit-head training and communication aware deployment

A. Exit-Head Training The early-exit decision relies on the confidence score 𝜉%,8,, = ℎ, (𝐡%,8,, ) defined in (7). For the exit decision to be qualitypreserving as required by constraint (c.9), the confidence score must be a faithful indicator of whether the current representation already yields the same prediction as the fulldepth model. We therefore train one lightweight exit head per layer on the calibration set. The exit head ℎ, (⋅) is a single linear layer followed by a sigmoid activation, which maps the hidden representation 𝐡%,8,, ∈ ℝ; to a scalar in [0,1] . Its parameter count is negligible compared with an expert, so the head incurs almost no memory and computation overhead. For each calibration token 𝑞 ∈ 𝒬cal , let 𝑜•8,, denote the prediction obtained by applying the model output head to 𝐡8,, , and

let 𝑜8∗ denote the full-depth prediction of the same token. The supervision label indicates whether the layer-𝑙 representation is already consistent with the full-depth output: 𝑏8,, = 𝟙(𝑜•8,, = 𝑜8∗ ) (22) Each exit head is then trained to predict this consistency by minimizing the binary cross-entropy loss over the calibration set: ! ℒ, = − ∣𝒬 ∣ ∑8∈𝒬cal ’𝑏8,, log 𝜉8,, + (1 − 𝑏8,, )log (1 − 𝜉8,, )• (23) cal

The backbone model is frozen during this stage, and only the exit heads are trained, so the original routing behavior and expert outputs are not altered. Consequently, the confidence score learns to approximate the probability that exiting at layer 𝑙 preserves the full-depth prediction. This is exactly the semantic required by constraint (c.9): when 𝜉%,8,, ≥ 𝑝<% , the representation is consistent with the full-depth output with probability at least 𝑝<% , so the exit is quality-preserving by construction rather than a source of uncontrolled error. Note that the proposed exit-head training does not fine-tune the backbone model. The MoE experts, routers, and attention layers are all frozen throughout, and only the per-layer linear exit heads are trained on cached hidden representations. As a result, no gradient is back-propagated through the backbone, no optimizer state of the backbone is maintained, and the original routing decisions and expert outputs remain bit-identical to the pre-trained model. The total number of trainable parameters introduced by all exit heads is approximately 𝑁 ⋅ 𝑑, which is negligible compared with a single expert. Consequently, both the offline training cost and the runtime memory and latency overhead introduced by the exit heads are marginal. B. Layer Importance Threshold Calibration The skip decision 𝛿%,8,, admits the skip action only when the layer importance in (4) is below a layer-specific threshold 𝜃, . The role of 𝜃, is to bound the degradation introduced by skipping, so that the runtime budget enforcement in Algorithm 2 (in the next Section) is rarely the binding constraint and the per-step skip cost stays meaningful. We calibrate 𝜃, offline from the empirical relationship between layer importance and skip-induced degradation. On the calibration set, for each layer 𝑙 we measure the actual quality degradation caused by skipping the layer for tokens with different importance values. Let Δ, (𝐼) denote the average degradation observed when layers with importance in a small neighborhood of 𝐼 are skipped, where the degradation is measured as the consistency drop between the skipped-path prediction and the full-depth prediction and Λ%,8,, = Δ, (𝐼%,8,, ). Since a higher importance corresponds to a larger degradation, Δ, (𝐼) is also non-decreasing in 𝐼, which is consistent with the modeling assumption of (5). The threshold 𝜃, is then selected as the largest importance value whose skip degradation does not exceed a per-layer tolerance 𝜀, : 𝜃, = max {𝐼 ∈ [0,1] ∣ Δ, (𝐼) ≤ 𝜀, } (24) The per-layer tolerance is derived from the user budget so that the offline threshold and the online budget are mutually consistent. Letting 𝐷¯ denote the average per-token budget over all users and 𝜅¯ the expected number of skipped layers per token observed on the calibration set, the tolerance is set as: H̄ 𝜀, = (25) S̄

8 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID Equation (25) distributes the per-token budget evenly across the expected skipped layers, so that a token that skips 𝜅¯ layers, each within tolerance 𝜀, , accumulates an expected degradation no larger than 𝐷¯ . This makes the offline threshold a conservative pre-filter: it removes high-importance layers from the skip candidates before the runtime budget check in (19), so that the two mechanisms work in the same direction and never contradict each other. We emphasize that 𝜃, is only an admission pre-filter; the hard guarantee of constraint (c.10) is still enforced online by the budget condition in (19), so the quality constraint holds regardless of any calibration mismatch. C. Communication-Aware Redundant Deployment In our previous work [37], it guarantees that every expert is deployed at least once, so the coverage constraint (c.2) and the memory constraint (c.1) are satisfied; when edge servers still have free memory after the baseline deployment, important experts can be replicated to further reduce runtime cross-server transmission. However, in this paper the skip and exit mechanisms change how often each expert is actually executed, so replicating an expert that is frequently skipped or that lies beyond the typical exit layer wastes memory without reducing transmission. We therefore redefine the redundancy benefit to be aware of the skip and exit behavior. Let 𝜋, denote the probability that a token is still active when entering layer 𝑙, estimated from the calibration exit distribution: ! 𝜋, = ∣𝒬 ∣ ∑8∈𝒬cal 𝟙V𝐿8 ≥ 𝑙W (26) cal

Since the active indicator is monotonically non-increasing along the depth by (6), 𝜋, is non-increasing in 𝑙, which means deeper experts are reached by fewer tokens. Let 𝜎,2 denote the probability that expert 𝑒,2 is actually executed rather than skipped, given that the token is active at layer 𝑙 . This is estimated as the fraction of calibration tokens routed to 𝑒,2 whose layer importance exceeds the skip threshold: 𝜎,2 = Pr (𝐼8,, > 𝜃, ∣ 𝑟, (𝑞) = 𝑘) (27) 2 The expected execution frequency of expert 𝑒, is then 𝜆2, = 𝜋, ∙𝜎,2 ∙𝑎,2 , where 𝑎,2 is the activation frequency of 𝑒,2 from [37]. This product form assumes the early-exit attrition 𝜋, , skip attrition 𝜎,2 , and activation frequency 𝑎,2 are approximately independent. In practice they are weakly positively correlated, which slightly overestimates 𝜆2, for deep experts. This only inflates the redundancy benefit marginally and never produces an infeasible replication decision; the memory constraint is enforced independently. Replicating 𝑒,2 on a server 𝑆) that currently must fetch the expert from a remote server saves the corresponding transmission for every token that resides on 𝑆) and executes 𝑒,2 . Accordingly, the redundancy benefit of placing a replica of 𝑒,2 on 𝑆) is defined as: = 𝐵),,,2 = 𝜆2, ⋅ 𝜌),, ⋅ L ¯ ! + 𝜏¯) M (28) >(

where 𝜌),, is the fraction of tokens expected to reside on 𝑆) before layer 𝑙 (estimated from the access distribution and the baseline trajectories), and 𝐵¯) and 𝜏¯) are the average bandwidth and latency from 𝑆) to the servers currently hosting 𝑒,2 . The benefit is large only when the expert is both frequently executed ( 𝜆2, high, i.e., shallow and frequently activated and rarely skipped) and currently expensive to reach from 𝑆) . Experts that are usually skipped (small 𝜎,2 ) or lie

beyond the typical exit depth (small 𝜋, ) receive a low benefit and are therefore not replicated, so the free memory is concentrated on the experts that genuinely cut transmission. The replicas are then placed greedily: among all feasible (expert, server) pairs that satisfy the memory constraint (c.1), the pair with the largest redundancy benefit is replicated first, the residual memory is updated, and the process repeats until no feasible replica remains. Since replication only adds copies of already-deployed experts, the coverage constraint (c.2) and the memory constraint (c.1) continue to hold throughout, so the refined deployment 𝑋 remains feasible. F. Offline Algorithm Based on the above models, the offline exit-head training and communication-aware deployment algorithm is summarized in Table I. Table I Alaogirtim1: Offline exit head training and communication aware deployment algorithm Input: MoE model ℳ , edge server set 𝑆, calibration token set 𝒬cal , baseline grouping 𝒢 and deployment from [37], memory capacities {𝐶! } , bandwidths (𝐵!,# * , latencies (𝜏!,# * , per-layer skip tolerances {𝜀$ }. Output: Exit heads {ℎ$ } , importance thresholds {𝜃$ } , transition statistics {𝑃$ }, token-similarity reference set, refined deployment 𝑋. 1. Run the full-depth model on 𝒬cal and record hidden representations, routing targets, gating weights, and full-depth predictions 2. for each MoE layer 𝑙 = 1 to 𝑁 do 3. Build consistency labels 𝑏%,$ by (22) 4. Train exit head ℎ$ by minimizing the loss (23) with the backbone frozen 5. Measure skip degradation curve Δ$ (𝐼) and calibrate threshold 𝜃$ by (24) to (25) 6. Cluster layer-𝑙 representations into centroids with average exit layers 7. end for 8. Estimate active probabilities 𝜋$ by (26) and execution probabilities 𝜎$& by (27) 9. Initialize 𝑋 from the baseline deployment of [37] 10. while some server has free memory and a feasible replica exists do 11. Compute redundancy benefit 𝐵!,$,& by (28) for all feasible (expert, server) pairs 12. Replicate the pair with the largest benefit and update residual memory 13. end while 14. return {ℎ$ }, {𝜃$ }, {𝑃$ }, token-similarity reference set, and 𝑋.

The algorithm contains three parts. Lines 1-9: Per-layer calibration. A single forward pass over the calibration set 𝒬cal is performed with the backbone frozen, and all intermediate quantities required by the offline procedure are recorded simultaneously, avoiding multiple passes over the data. For each MoE layer 𝑙 , the hidden representations at the layer output, the routing targets, the gating weights, and the full-depth predictions are extracted and stored. The consistency label 𝑏8,, of each calibration token is then built by comparing the intermediate prediction obtained from 𝐡8,, against the full-depth prediction, and the exit head ℎ, is trained from scratch on these labels with only the head parameters updated. Because the backbone is strictly frozen throughout this stage, the routing logits and expert outputs are identical to those seen at runtime, so the transition counts and the calibration representations are not contaminated by any training-induced drift. For each layer, the skip degradation curve Δ, (𝐼) is measured by evaluating the

9 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID consistency drop when tokens of different importance levels are forced to skip the layer, and the importance threshold 𝜃, is derived from this curve using the budget-derived tolerance 𝜀, in (25), ensuring that the offline threshold and the online budget constraint point in the same direction. Lines 10-11: Execution-frequency estimation. Once the per-layer calibration is complete, the effective execution frequency of each expert is estimated by composing two independently calibrated quantities. The active probability 𝜋, captures the depth-wise attrition of tokens due to early exit: it is the fraction of calibration tokens that are still active when entering layer 𝑙, and it is non-increasing in 𝑙 by construction, so deeper experts automatically receive a lower weight in the redundancy benefit. The execution probability 𝜎,2 captures the additional attrition due to skip decisions: it is the fraction of calibration tokens routed to expert 𝑒,2 whose importance exceeds the calibrated threshold 𝜃, , meaning that the expert would actually be executed rather than bypassed. The product 𝜆2, then gives the expected fraction of inference tokens that will actually invoke expert 𝑒,2 at runtime, accounting simultaneously for early exit and layer skipping. This quantity replaces the raw activation frequency used in [37], so that the subsequent replication step is not misled by an inflated frequency estimate for experts that are rarely reached or frequently skipped. Lines 12-16: Communication-aware replication. Starting from the feasible baseline deployment inherited from [37], which already satisfies the coverage constraint (c.2) and the memory constraint (c.1), each edge server's residual memory is computed. The redundancy benefit 𝐵),,,2 in (28) is then evaluated for every (expert, server) pair that is not yet deployed and does not violate the memory constraint. The benefit explicitly weights the skip-exit-aware execution frequency 𝜆2, against the expected transmission cost saved by placing a replica on 𝑆) , so that experts that are deeply buried beyond the typical exit layer, or that are almost always skipped, contribute negligible benefit regardless of how large their raw activation frequency is. The greedy loop selects the pair with the largest benefit, places the replica, and updates the residual memory of the corresponding server before the next iteration. The loop terminates when no feasible pair remains, either because no server has sufficient residual memory or because all beneficial replicas have already been placed. Because each iteration only adds a copy of an already-deployed expert and the memory check is enforced before placement, the constraints (c.1) to (c.3) remain satisfied at every iteration, and the final deployment 𝑋 is guaranteed feasible. The refined deployment and all calibration outputs are then returned together, forming the complete offline input to Algorithm 2. V. COMMUNICATION AWARE ONLINE SKIP EXIT SERVER-EXPERT SELECTION

In Section IV, the offline expert grouping result 𝒢 and the deployment strategy 𝑋 have been obtained. Based on these results, the remaining runtime tasks are threefold: determining which layers to skip, determining when to exit early, and selecting the server-expert pair for each executed layer. Therefore, in this section, we propose a communication-aware online strategy that jointly determines the skip decision, the exit

decision, and the server-expert pair for each token at each layer, as illustrated in Fig. 3.

Fig.3 Communication aware online skip exit and server-expert selection

A. Online Decision State and Per-Step Cost The online strategy processes tokens layer by layer. For token 𝑞 of user 𝑑% , let 𝑠%,8,, denote the server on which the token representation resides before entering the 𝑙-th MoE layer. The initial location is the access edge server of the corresponding user, i.e., 𝑠%,8,! = 𝑎(𝑖), and the location is updated only when the layer is executed: 𝑚, if layer 𝑙 executed on 𝑆) 𝑠%,8,,:! = h 𝑠 , (29) if layer 𝑙 skipped %,8,, At the 𝑙-th MoE layer, three mutually exclusive actions are available for an active token (𝑦%,8,, = 1): 1) Execute on server 𝑆) by expert 𝑒,2 : the token is transmitted to 𝑆) if 𝑆) ≠ 𝑠%,8,, , then processed by 𝑒,2 . 2) Skip: the expert is bypassed through the residual path; the token stays on 𝑠%,8,, and no expert is invoked. 3) Exit: all layers from 𝑙 onward are terminated; this action is available only if 𝜉%,8,,3! ≥ 𝑝<% (i.e., the confidence requirement is already satisfied after the previous executed layer). When the token is executed on server 𝑆) by expert 𝑒,2 , the per-step delay is the sum of the transmission delay from the current location to 𝑆) and the computation delay on 𝑆) : 𝑑%,8,, (𝑚, 𝑘) = 𝟙(𝑠%,8,, ≠ 𝑚) o>

@)

=!

*&,#,! ,(

+𝜏T&,#,!,) q + A!

(

(30)

When the layer is skipped, no transmission is incurred and no computation is performed, so the skip delay is zero. The perstep cost of executing the server-expert pair (𝑚, 𝑘) is defined as the normalized weighted sum of delay and quality degradation: ;

(),2)

𝑓%,8,, (𝑚, 𝑘) = 𝜔J ⋅ &,#,! ref ;&

+ 𝜔H ⋅

B! (1! (8),2) H̄&

(31)

where 𝑑%ref is the per-step reference delay of user 𝑑% under fulldepth local exact execution on the same deployment, and 𝐷¯% is the per-token quality degradation budget defined in the next subsection. Both terms are dimensionless and share the same weights as the global utility (21), which ensures that the online per-step cost is consistent with the offline optimization objective. The per-step cost of a skip action is defined analogously. Since a skip incurs zero delay but consumes a skip-induced degradation Λ%,8,, = 𝜌V𝐼%,8,, W = 𝛥, V𝐼%,8,, W, the skip cost is: skip

U&,#,!

𝑓%,8,, = 𝜔H ⋅ H̄

&

(32)

An exit action incurs neither delay nor degradation beyond

10 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID the current layer, so its incremental cost is zero. The exit action is therefore taken only when it is feasible (the confidence requirement is met) and the look-ahead cost of the remaining layers is positive, making exit strictly beneficial. B. Quality-Budget-Aware Feasible Action Set To enforce the per-user quality degradation constraint (c.10) in an online manner, the user-level budget 𝐷%6FG is evenly allocated across the tokens of user 𝑑% : H+,𝐷¯% = & (33) ∣B& ∣

-%,8,, denote the accumulated quality degradation of Let 𝐷 token 𝑞 before the 𝑙 -th MoE layer, initialized as zero. The runtime feasible action set of token 𝑞 at the 𝑙-th MoE layer is: 𝒜%,8,, = ¯(𝑚, 𝑘)°𝑥),,,2 = 1, 𝑒,2 ∈ 𝑉,,1!(8) , -%,8,, + 𝑄, (𝑟, (𝑞), 𝑘) ≤ 𝐷¯% ± ∪ 𝐷 -%,8,, + Λ%,8,, ≤ 𝐷¯% ± ∪ ¯skip °𝐷 {exit °𝜉%,8,,3! ≥ 𝑝<% } (34) The first subset contains all execute candidates: expert 𝑒,2 must be deployed on 𝑆) , must be the target expert or a feasible substitute, and the resulting accumulated degradation must not exceed the per-token budget. The second subset contains the skip action, which is feasible only if the skipinduced degradation does not exhaust the remaining budget. The third subset contains the exit action, which is feasible only if the confidence requirement has already been satisfied at the previous executed layer. Since each expert is deployed on at least one server by 1 (8) constraint (c.2), the target expert 𝑒, ! is always available somewhere, and its substitution quality loss is zero by definition. Therefore, the exact-execution candidate always satisfies the budget condition, which guarantees that 𝒜%,8,, always contains at least one execute candidate and is never empty. The exit action is a hard feasibility gate controlled by the confidence score, not a soft preference, so it never forces premature termination. This construction ensures that the online feasible set is always consistent with the constraints (c.6) to (c.9) of P1. C. Importance-Guided Skip Decision with Look-Ahead The skip decision trades a fraction of the quality budget for the elimination of a cross-server transmission. To make this tradeoff principled rather than greedy, the strategy estimates the transmission cost that would be incurred if the layer were executed, and compares it against the skip cost. Transmission necessity score. For the 𝑙 -th MoE layer of token 𝑞 , define the transmission necessity score as the minimum transmission delay over all execute candidates: Γ%,8,, =

min

exec (),2)∈𝒜&,#,!

𝟙(𝑠%,8,, ≠ 𝑚) o

=!

>*&,#,! ,(

+𝜏T&,#,!,) q

(35)

exec where 𝒜%,8,, is the execute subset of 𝒜%,8,, . If the minimum is zero, there exists a locally available execute candidate and the token need not be transmitted; the skip action in this case provides no transmission benefit and is thus suppressed. This formalizes the design principle in Section III-B that skipping is preferentially applied only to layers whose execution would require cross-server transmission. Importance threshold. Even when a layer requires crossserver transmission, skipping is appropriate only if the layer importance is sufficiently low. The skip action is admitted into

the decision only when 𝐼%,8,, ≤ 𝜃, , where 𝜃, is a layer-specific importance threshold calibrated offline (detailed in Section IV). When 𝐼%,8,, ≤ 𝜃, is violated, the layer is deemed too important to skip, and the decision reduces to a server-expert selection exec among the execute candidates in 𝒜%,8,, . Look-ahead skip evaluation. Even when the skip action is admitted, its impact on future layers must be assessed. Skipping keeps the token on the current server 𝑠%,8,, , which may be favorable for future layers if the subsequent experts are also hosted locally, or unfavorable if the skip merely defers a crossserver transmission to a later, more expensive hop. To capture this, the skip cost-to-go is estimated by the same look-ahead mechanism described in the next subsection, with the token location fixed at 𝑠%,8,, after the skip. D. Confidence-Guided Exit Decision with Token-Similarity Prediction The exit decision eliminates all remaining layers once the confidence of the current representation meets the task requirement. However, the confidence score 𝜉%,8,, is computed from the exit head ℎ, (⋅) and is only available after an executed layer; it is not available after a skipped layer by constraint (c.6). The exit decision therefore interleaves only with executed layers. Confidence-based exit gate. After executing the 𝑙-th MoE layer, if 𝜉%,8,, ≥ 𝑝<% , the exit action becomes feasible. The decision of whether to exit or continue is then made by comparing the incremental cost of the remaining layers against the zero incremental cost of exit. If the estimated cost-to-go of the remaining layers is positive, exit is taken; if it is zero or negative (which occurs only when all remaining layers are locally available and nearly cost-free), execution continues. Token-similarity-based exit-point prediction. To estimate the confidence of future layers without actually computing them, we exploit the observation that tokens with similar hidden representations at a given layer tend to exit at similar layers. Let 𝒞, (𝑞) denote the set of calibration tokens whose hidden representation at layer 𝑙 is closest to 𝐡%,8,, under cosine similarity. The predicted exit layer of token 𝑞 given that it is still active at layer 𝑙 is: ! 𝐿´%,8,, = round L|𝒞 (8)| ∑8.∈𝒞!(8) 𝐿8. M (36) !

where 𝐿8. is the actual exit layer of calibration token 𝑞Y . The predicted exit layer 𝐿´%,8,, is used to estimate the number of remaining executed layers and thus the expected future delay. If 𝐿´%,8,, ≤ 𝑙, the prediction indicates that the token should have already exited, which strengthens the evidence for triggering exit at the current layer. Adaptive exit threshold. In practice, the calibration-based prediction may be biased under distribution shift. To maintain calibration, the exit confidence threshold is adapted using a running estimate of the observed false-exit rate within the latest observation window: 𝑝<% ← ΠZ[,!\ L𝑝<% +𝜂V𝜉¯exit −𝑝<% WM (37) where 𝜉¯exit is the average confidence score at which exits were triggered in the observation window, 𝜂 is a small step size, and ΠZ[,!\ (⋅) projects the value onto ¸𝑝, 1¹ with a minimum

11 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID confidence floor 𝑝 to prevent premature exit. The update pushes 𝑝<% toward the observed exit confidence, so that the threshold self-corrects when the actual exit behavior deviates from the calibration. E. Look-Ahead Joint Selection For each active and non-exited token, the per-layer decision is made by selecting the action in 𝒜%,8,, that minimizes the sum of the current-step cost and an estimated look-ahead cost-to-go over the following 𝐻 layers. Expert transition statistics. Since the routing targets of future layers are not known before their representations are computed, the look-ahead relies on expert transition statistics collected on the calibration set. Let 𝑃, (𝑘 → 𝑘 Y ) denote the empirical probability that a token routed to expert 𝑒,2 in the 𝑙-th 2. MoE layer is subsequently routed to expert 𝑒,:! in the next layer: 𝑃, (𝑘 → 𝑘 Y ) = ^

+! (2→2 . )

+ (2→2 .. ) ).. !

(38)

where 𝑛, (𝑘 → 𝑘 Y ) is the transition count observed on the calibration set. The transition statistics capture the layer-wise routing correlation, so that the look-ahead can estimate the distribution of future execution targets without computing future representations. Cost-to-go recursion. The look-ahead cost-to-go of token 𝑞 being at server 𝑆) after taking action (𝑚, 𝑘) at the 𝑙-th MoE layer is defined by a depth-𝐻 recursion. The base case (_) is Φ%,8,, (𝑚, 𝑘) = 0, and for ℎ ≥ 1: (`)

Φ%,8,, (𝑚, 𝑘) = ∑2 . 𝑃, (𝑘 → 𝑘 Y ) ∙ min

&. ∈𝒜&,#,!/0 Y

(`3!)

’𝑓%,8,,:! (𝑎Y )°𝑠%,8,,:! = 𝑚 + Φ%,8,,:! (𝑚, 𝑎Y )• (39)

where 𝑎 ranges over all feasible actions (execute, skip, or exit) in 𝒜%,8,,:! , and the per-step cost 𝑓%,8,,:! (𝑎Y ) is evaluated with the source location fixed at the server 𝑆) inherited from the current step. For the skip action 𝑎Y = skip , the location skip remains 𝑚 and the cost is 𝑓%,8,,:! . For the exit action 𝑎Y = exit, both the cost and the subsequent cost-to-go are zero. This recursive definition ensures that the look-ahead is always selfconsistent with the token trajectory: the estimated future transmission is always measured from the server actually chosen at the current step, and the skip-exit coordination constraint (c.6) is respected within the recursion by suppressing exit at layers that are predicted to be skipped. Action selection. Combining the current-step cost and the look-ahead cost-to-go, the action for token 𝑞 at the 𝑙-th MoE layer is selected as: (a3!) 𝑎∗ = arg minF∈𝒜&,#,! ’𝑓%,8,, (𝑎) + Φ%,8,, (𝑎)• (40) When 𝑎∗ is an execute action (𝑚∗ , 𝑘 ∗ ) , the decision variable 𝑢%,8,,,)∗,2 ∗ = 1 is set, the token is executed on 𝑆)∗ by ∗ expert 𝑒,2 , and the token state is updated as: -%,8,,:! = 𝐷 -%,8,, + 𝑄, (𝑟, (𝑞), 𝑘 ∗ ) 𝑠%,8,,:! = 𝑚∗ , 𝐷 (41) ∗ When 𝑎 is skip, 𝛿%,8,, = 1 is set, the token stays on 𝑠%,8,, , and the accumulated degradation is updated as: -%,8,,:! = 𝐷 -%,8,, + Λ%,8,, 𝑠%,8,,:! = 𝑠%,8,, , 𝐷 (42) ∗ When 𝑎 is exit, 𝑦%,8,,. = 0 is set for all 𝑙Y > 𝑙, and the token contributes no further cost to any subsequent layer. When 𝐻 = 1, the look-ahead term vanishes and the strategy

degenerates to a myopic per-layer decision. A larger 𝐻 produces decisions that are more aware of future transmission costs and remaining budget, at the cost of additional runtime computation. The horizon 𝐻 therefore provides a tunable tradeoff between decision quality and computational overhead. F. Dual-Weight Adaptation The weights 𝜔J and 𝜔H control the tradeoff between inference delay and quality degradation. Since the runtime network condition and server workload vary over time, the weights are periodically updated according to the observed average normalized delay 𝑇¯ and the observed average budget utilization rate 𝐷¯ within the latest observation window: 𝜔H ← Π[_,!] V𝜔H +𝜅(𝐷¯−𝑇¯)W, 𝜔J ← 1 − 𝜔H (43) where 𝜅 is a small positive step size and Π[_,!] (⋅) clips the value into [0,1]. The update increases 𝜔H when the budget utilization rate is higher than the normalized delay, directing the strategy to be more conservative about quality degradation; conversely, it increases 𝜔J when the delay is dominant, directing the strategy toward more aggressive skipping and earlier exit. Since both 𝑇¯ and 𝐷¯ are dimensionless quantities normalized in the same way as (21), the update direction is always physically meaningful, and the constraint 𝜔J + 𝜔H = 1 is maintained at every update step. The exit threshold adaptation (37) runs on the same observation window, so the two adaptation rules share a common runtime feedback loop without interfering with each other. E. Online Selection Algorithm Based on the foregoing models, the communication-aware online skip-exit and server-expert selection algorithm is summarized in Table II. Table II Algorithm2: Communication aware online skip exit and server-expert selection Input: Deployment strategy 𝑋, grouping result 𝒢, exit heads {ℎ$ }, layer importance thresholds {𝜃$ } , transition statistics {𝑃$ } , look-ahead horizon 𝐻, quality budgets {𝐷'()* }, initial confidence thresholds (𝑝+' *, network and computation parameters. Output: Server-expert selection 𝑈, skip decisions Δ, exit decisions 𝑌, and token trajectories. 1. for each user 𝑑' and each token 𝑞 ∈ 𝑄' do 2. Set initial location 𝑠',%,, = 𝑎(𝑖) by (29) H',%,, = 0 and per-token 3. Set accumulated degradation 𝐷 budget 𝐷¯' by (33) 4. end for 5. for each MoE layer 𝑙 = 1 to 𝑁 do 6. for each user 𝑑' and each token 𝑞 ∈ 𝑄' with 𝑦',%,$ = 1 do 7. Obtain target expert 𝑟$ (𝑞) from the local router - (%) 8. Compute layer importance 𝐼',%,$ = 𝑔$ ! (𝑞) by (4) 9. Compute transmission necessity score Γ',%,$ by (35) 10. Build feasible action set 𝒜',%,$ by (34), admitting skip only if Γ',%,$ > 0 and 𝐼',%,$ ≤ 𝜃$ 11. for each action 𝑎 ∈ 𝒜',%,$ do 12. Compute per-step cost 𝑓',%,$ (𝑎) by (31) or (32) (01,)

Compute look-ahead cost-to-go Φ',%,$ (𝑎) by the depth𝐻 recursion (39) 14. end for (01,) 15. Select 𝑎 ∗ = arg min 3∈𝒜",$,! [𝑓',%,$ (𝑎) + Φ',%,$ (𝑎)] by (40) ∗ ∗ ∗ , 16. if 𝑎 is execute (𝑚 𝑘 ) then ∗ 17. Set 𝑢',%,$,!∗,& ∗ = 1; execute token 𝑞 on 𝑆!∗ by 𝑒$& 18. Compute confidence 𝜉',%,$ = ℎ$ (𝐡',%,$ ) by (7) H',%,$6, by (41) 19. Update 𝑠',%,$6, and 𝐷 20. else if 𝑎 ∗ is skip then 13.

12 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID H',%,$6, by (42) 21. Set 𝛿',%,$ = 1; update 𝑠',%,$6, and 𝐷 22. else {𝑎 ∗ is exit} 23. Set 𝑦',%,$& = 0 for all 𝑙 7 > 𝑙 24. end if 25. end for 26. end for 27. Periodically update 𝜔 8 , 𝜔9 by (43) and 𝑝+' by (37) from runtime observations 28. return 𝑈, Δ, 𝑌, and token trajectories.

The algorithm contains three parts. Lines 1-4: Initialization. Before the layer-by-layer loop begins, each token's runtime state is initialized. The initial server location 𝑠%,8,! is set to the access edge server 𝑎(𝑖) of the corresponding user, which is the server to which the user's device is attached and where the token representation first -%,8,! is arrives. The accumulated quality degradation 𝐷 initialized to zero, reflecting the fact that no decision has yet been made. The per-token quality budget 𝐷¯% is derived by evenly distributing the user-level budget 𝐷%6FG over all tokens of user 𝑑% according to (33). This even allocation is a sufficient condition for the user-level constraint 𝐷% ≤ 𝐷%6FG : as long as each individual token does not exceed its share 𝐷¯% , the sum over all tokens is bounded by 𝐷%6FG , and the constraint is satisfied regardless of the actual exit layers or skip patterns realized at runtime. Lines 5-28: Layer-by-layer joint decision. The outer loop advances from layer 𝑙 = 1 to 𝑁 . Tokens that have already exited (𝑦%,8,, = 0) are excluded from processing, so the active set shrinks monotonically as tokens exit, reducing the per-layer computational load as inference progresses. For each active token, the local router is queried to obtain the target expert 𝑟, (𝑞), which is available on the current server without any cross-server communication because the router is replicated on every server following [37]. The layer importance 𝐼%,8,, is then computed from the gating weight of the target expert, and the transmission necessity score Γ%,8,, is computed by scanning exec the execute candidates in 𝒜%,8,, for any locally available option. The feasible action set 𝒜%,8,, is then assembled: the execute subset includes all deployed (server, expert) pairs that are substitute-feasible and within the remaining budget; the skip action is admitted only when Γ%,8,, > 0 and 𝐼%,8,, ≤ 𝜃, , jointly ensuring that skipping is considered only when execution would incur a genuine transmission cost and the layer importance is low enough that the skip degradation stays within the calibrated tolerance; the exit action is admitted only when the confidence score of the last executed layer already meets the task requirement 𝑝<% , so that exit is never triggered on a representation that has not been sufficiently refined. For each candidate action, the per-step cost is computed, and the lookahead cost-to-go is evaluated by the depth-𝐻 recursion (39), which propagates the expected future transmission cost forward from the server that would result from the current action. The action with the minimum total cost is then selected. In the event that no substitute-execute or skip candidate satisfies the quality budget, the exact-execution candidate with 𝑘 ∗ = 𝑟, (𝑞) is selected as the fallback, so that the accumulated degradation never exceeds 𝐷¯% regardless of the routing outcome. The token state is updated according to the chosen action: for execute,

both the server location and the accumulated degradation are updated; for skip, only the degradation is updated and the location is inherited unchanged; for exit, the active flags of all subsequent layers are cleared. Lines 29-30: Adaptive feedback and output. After each observation window, the dual weights 𝜔J and 𝜔H are updated by (43) based on the observed average normalized delay and budget utilization, keeping the online cost function calibrated to the current runtime environment. The exit confidence threshold 𝑝<% is separately adapted by (37) based on the average exit confidence observed in the same window, so that the threshold self-corrects when the calibration-time and runtime token distributions diverge. The two adaptation rules share the same observation window but operate on disjoint parameters, so they do not interfere with each other. The complete serverexpert selection 𝑈 , skip decisions Δ , exit decisions 𝑌 , and resulting token trajectories are then returned. F. Properties of the Proposed Algorithms In this section, the feasibility, the quality and confidence guarantees, and the complexity of the two algorithms are analyzed in detail. Property 1 (Deployment feasibility): If the total edge memory can store every expert at least once, Algorithm 1 outputs a deployment 𝑋 satisfying constraints (c.1) to (c.3). Proof. We need to show that the output 𝑋 of Algorithm 1 satisfies constraints (c.1), (c.2), and (c.3). For (c.3), every deployment variable 𝑥),,,2 is set to either 0 or 1 in both the baseline initialization step and the greedy replication loop, so the binary constraint holds throughout. For (c.2), the baseline deployment from [37] guarantees that every expert 𝑒,2 is placed on at least one server before Algorithm 1's replication loop begins. The replication loop in Lines 12–16 only adds additional copies of existing experts to servers that have residual memory; it never removes an existing replica. Therefore, after the replication loop terminates, every expert is still placed on at least one server, and (c.2) is satisfied. For (c.1), the baseline deployment from [37] satisfies the memory constraint by assumption. In each iteration of the replication loop, the feasibility check: 𝑝,2 + ∑,.,2 . 𝑥),,.,2 . ∙ .

𝑝,2. ≤ 𝐶) is performed before placing any replica. A replica is placed only if this check passes, meaning that the memory footprint on server 𝑆) after placement does not exceed 𝐶) . Since the check is enforced at every iteration and the loop only adds replicas that pass the check, the memory constraint on every server is maintained throughout, and (c.1) holds in the final output. Since (c.1), (c.2), and (c.3) all hold, Algorithm 1 outputs a feasible deployment strategy. █ Property 2 (Quality-budget guarantee): For any user 𝑑% , the strategy produced by Algorithm 2 satisfies 𝐷% ≤ 𝐷%6FG . Proof. Fix any user 𝑑% and any token 𝑞 ∈ 𝑄% . We show by -%,8,, satisfies induction on 𝑙 that the accumulated degradation 𝐷 ¯ 𝐷%,8,, ≤ 𝐷% for every layer 𝑙 ∈ [1, 𝑁 + 1]. -%,8,! = 0 ≤ 𝐷¯% . Base case. At initialization, 𝐷 Inductive step. Suppose 𝐷%,8,, ≤ 𝐷¯% for some layer 𝑙 at which -%,8,,:! ≤ 𝐷¯% . the token is active. We show 𝐷 ∗ ∗ , Execute action (𝑚 𝑘 ). The execute candidate is admitted -%,8,, + 𝑄, (𝑟, (𝑞), 𝑘 ∗ ) ≤ 𝐷¯% by the budget into 𝒜%,8,, only if 𝐷

13 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID -%,8,,:! = 𝐷 -%,8,, + 𝑄, (𝑟, (𝑞), 𝑘 ∗ ) ≤ condition in (34). Therefore 𝐷 ¯ 𝐷% . -%,8,, + Skip action. The skip action is admitted only if 𝐷 -%,8,,:! = 𝐷 -%,8,, + Λ%,8,, ≤ 𝐷¯% . Λ%,8,, ≤ 𝐷¯% . Therefore 𝐷 Exit action. No further layers are processed, - %,8,,. remains unchanged for all 𝑙Y > 𝑙 , and the so 𝐷 -%,8,,. ≤ 𝐷¯% holds trivially. bound 𝐷 It remains to verify that at least one feasible action always exists in 𝒜%,8,, , so that Algorithm 2 is never forced to violate the budget. By Property 1, every expert is deployed on at least one server, so the exact-execution candidate (𝑚_ , 𝑟, (𝑞)) with 𝑘 ∗ = 𝑟, (𝑞) exists and satisfies 𝑄, (𝑟, (𝑞), 𝑟, (𝑞)) = 0 by definition. -%,8,, + 0 ≤ 𝐷¯% , Therefore, the budget condition reduces to 𝐷 which holds by the inductive hypothesis. Hence the exactexec execution candidate is always in 𝒜%,8,, , and 𝒜%,8,, is never empty. -%,8,0:! ≤ 𝐷¯% for every token 𝑞 . Summing By induction, 𝐷 over all |𝑄% | tokens of user 𝑑% and applying the definition 𝐷¯% = -%,8,0:! ≤ |𝑄% | ⋅ 𝐷¯% = 𝐷%6FG . 𝐷%6FG /|𝑄% |: 𝐷% = ∑8∈B& 𝐷 Therefore, the user-level quality constraint (c.10) is satisfied for every user 𝑑% . Crucially, this guarantee holds for any realization of the routing targets and any calibration accuracy of 𝜃, , because the hard budget check in (34) is performed online at every step independently of any offline approximation. █ Property 3 (Confidence-preserving exit): Every early exit produced by Algorithm 2 satisfies the confidence requirement, i.e., constraint (c.9) holds. Proof. We need to show that whenever Algorithm 2 sets 𝑦%,8,,:! = 0 (i.e., token 𝑞 exits after layer 𝑙), the confidence constraint 𝜉%,8,, ≥ 𝑝<% holds, i.e., constraint (c.9) is satisfied. By the construction of the feasible action set in (34), the exit action is admitted into 𝒜%,8,, if and only if 𝜉%,8,,3! ≥ 𝑝<% has been observed at the previous executed layer. Consequently, the algorithm can only select the exit action—and thus set 𝑦%,8,,:! = 0, when this condition is satisfied, and (c.9) holds by construction. We additionally verify that the confidence score is always evaluated on a representation that has been updated by an executed expert, rather than a stale residual representation from a skipped layer. By constraint (c.6) and its implementation in Algorithm 2, when a layer is skipped (𝛿%,8,, = 1), the exit action is not admitted into 𝒜%,8,, ; the active flag 𝑦%,8,,:! is passed unchanged from 𝑦%,8,, without triggering an exit check. Therefore, 𝜉%,8,,3! in the exit admission condition always refers to the confidence score produced after the most recent executed layer, and the confidence evaluation is never performed on a representation that was not refined by an expert computation. Finally, by the exit-head training objective (23), the output 𝜉8,, of exit head ℎ, approximates the probability that the representation 𝐡8,, is already consistent with the full-depth prediction. Under this calibration, 𝜉%,8,, ≥ 𝑝<% implies that the representation at layer 𝑙 agrees with the full-depth output with empirical probability at least 𝑝<% on the calibration distribution. When the runtime distribution is close to the calibration distribution, this probabilistic guarantee carries over to runtime;

when distribution shift occurs, the adaptive threshold update (37) moves 𝑝<% toward the observed exit confidence, restoring calibration over time. Therefore, every exit decision produced by Algorithm 2 satisfies (c.9), and the exit is quality-preserving by construction. █ Property 4 (Complexity): Let ∣ 𝒬cal ∣ be the calibration set size and 𝐸¯ the average number of experts per layer. Algorithm 1 runs in 𝑂(𝑁 |𝒬cal | 𝐸¯ ) time for the single calibration pass and the per-layer statistics, plus 𝑂(𝑀 𝑁 𝐸¯ log (𝑀𝑁𝐸¯ )) for the greedy replication. For Algorithm 2, the per-token per-layer decision evaluates at most ∣ 𝒜%,8,, ∣≤ 𝑀𝐸¯ + 2 actions, each requiring a depth-𝐻 look-ahead with branching bounded by the support of the transition distribution 𝐸¯ , so the worst-case perdecision cost is 𝑂((𝑀𝐸¯ ) ⋅ 𝐸¯ a3! ). Proof. We analyze the complexity of Algorithm 1 and Algorithm 2 separately. Algorithm 1. The dominant cost comes from four steps. (i) Calibration pass. Running the frozen model on |𝒬cal | tokens requires a single forward pass. For each of the 𝑁 MoE layers, extracting the hidden representations and routing targets takes 𝑂(|𝒬cal | ⋅ 𝑑) time, where 𝑑 is the hidden dimension. Across all layers the total cost is 𝑂(𝑁 ⋅ |𝒬cal | ⋅ 𝑑). (ii) Exit-head training. Each exit head ℎ, is a single linear layer with 𝑑 parameters, trained for a fixed number of epochs on ∣ 𝒬cal ∣ labels. The training cost per head is 𝑂(|𝒬cal | ⋅ 𝑑), and summing over 𝑁 layers gives 𝑂(𝑁 ⋅ |𝒬cal | ⋅ 𝑑) . Since 𝑑 is a model constant, this is 𝑂(𝑁 ⋅ |𝒬cal |) in terms of the problemsize parameters. (iii) Transition statistics and threshold calibration. For each layer 𝑙 , computing the 𝐸, × 𝐸,:! transition count matrix requires scanning |𝒬cal | tokens, giving 𝑂(|𝒬cal |) per layer and 𝑂(𝑁 ⋅ |𝒬cal |) in total, where 𝐸, is the number of experts at layer 𝑙 . Computing the skip degradation curve and deriving 𝜃, also requires 𝑂(|𝒬cal |) per layer. The centroid clustering with a fixed number of centroids 𝐾@ runs in 𝑂(|𝒬cal | ⋅ 𝐾@ ) per layer. (iv) Greedy replication. Each replication iteration evaluates 𝐵),,,2 for at most 𝑀 ⋅ 𝑁 ⋅ 𝐸¯ (server, expert) pairs, where 𝐸¯ is the average number of experts per layer and 𝑀 is the number of servers. With at most 𝑀 ⋅ 𝑁 ⋅ 𝐸¯ total replication decisions, the replication loop runs in 𝑂(𝑀" 𝑁𝐸¯ " ) time in the worst case, or 𝑂(𝑀𝑁𝐸¯ log (𝑀𝑁𝐸¯ )) if the benefit values are maintained in a max-heap. Combining all steps, the total complexity of Algorithm 1 is 𝑂(𝑁 ⋅ |𝒬cal | ⋅ 𝑑 + 𝑀𝑁𝐸¯ log (𝑀𝑁𝐸¯ )). Since both terms are polynomial in the problem parameters, Algorithm 1 has polynomial-time complexity. Algorithm 2. For each active token 𝑞 at each layer 𝑙, the feasible action set 𝒜%,8,, contains at most 𝑀𝐸¯ + 2 actions (at most 𝑀𝐸¯ execute candidates, plus skip and exit). For each action, the depth-𝐻 look-ahead recursion (39) is evaluated. At each recursion level, the sum over 𝑘 Y in (39) has at most 𝐸¯ nonzero terms (the support of the transition distribution), and the inner minimization is over 𝑀𝐸¯ + 2 candidates. Therefore the cost of one complete look-ahead evaluation is bounded ¯ +tv ¯ a3! ¯ +tv (𝑀𝐸 (𝑀𝐸 by: 2) ∙ 𝐸 suv ∙ st 2) = st tut tut d1&+@`%+e &I ,

((𝑀𝐸¯ )" ⋅ 𝐸¯ a3! ).

TfdTg8fg+I ,ghg,T

%++g1 65i [g1 +j;g

14 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID Summing over all tokens and layers, the total cost of Algorithm 2 is 𝑂(𝑁 ⋅ |𝑄| ⋅ (𝑀𝐸¯ )" ⋅ 𝐸¯ a3! ), where |𝑄| is the total number of tokens across all users. This is polynomial in all problem parameters for any fixed horizon 𝐻. In practice, the transition distribution 𝑃, (𝑘 → 𝑘 Y ) is sparse because each token is routed to a small number of experts per layer, so the effective branching factor at each look-ahead level is much smaller than 𝐸¯ . When 𝐻 = 1 the look-ahead term vanishes entirely and the per-token per-layer cost reduces to 𝑂(𝑀𝐸¯ ) , which is the same order as a simple myopic selection. Therefore, the horizon 𝐻 controls the complexityquality tradeoff continuously, and the algorithm remains practically efficient for the small values of 𝐻 used in the experiments. █ VI. PERFORMANCE EVALUATION In this section, we evaluate the proposed TrimMoE framework by experiment and analyze its behavior from the perspectives of latency, communication, depth reduction, and inference quality.A. Experimental Setup Experimental setup. TrimMoE is implemented on PyTorch and deployed on a geo-distributed edge testbed of 10 physical edge servers. The servers are heterogeneous in both computation and memory: their computation capacities 𝐹) span 18 to 120 TFLOPS and their GPU memory capacities 𝐶) span 12 to 48 GB. The servers are interconnected over ordinary Internet links rather than a dedicated fabric, with available bandwidths 𝐵),+ ranging from 0.8 to 10 Gbps and propagationplus-queuing latencies 𝜏),+ ranging from 0.3 to 22 ms, so that the bandwidth-limited and time-varying nature of edge interconnections is reproduced. Each user is attached to one access edge server following a non-uniform spatial distribution, and requests arrive at a default rate of 40 requests per second unless stated otherwise. Models, tasks, and calibration. We evaluate TrimMoE on three MoE models of increasing scale, Switch-Base-8E, QwenMoE-A2.7B, and Mixtral-8x7B. Inference quality is measured on WikiText-103 (perplexity), SQuAD (F1), and GSM8K (accuracy). All offline quantities required by Algorithm 1 (the per-layer exit heads, the layer-importance thresholds, the expert transition statistics, and the token-similarity reference set) are computed once on a held-out calibration set with the backbone strictly frozen. Unless otherwise stated, the look-ahead horizon is H=3, the per-user quality-degradation budget is set to 2%, the confidence requirement is 𝑝<% = 0.9, and the memory budget ratio of the redundant deployment is 2.0. Baselines. We compare TrimMoE with five representative methods that cover the relevant design space: 1) EdgeShard [12], 2) SlimCaching [30], 3) MoE² [35], 4) Mixture-of-Depths (MoD) [21], and 5) Substitute-Only [37], our previous similarity-based substitute-execution framework without depth reduction. We further include four internal variants, i.e., TrimMoE without skip, without early exit, without look-ahead, and without communication-aware redundancy, for the ablation study. The metrics are the average inference latency, the tail (P99) latency, the cross-server communication volume, the remote execution ratio, the throughput, and the inference quality. B. Overall Performance Comparison

We first compare the overall performance on Mixtral-8x7B, as shown in Fig. 4, which reports the average latency, the crossserver traffic per one thousand tokens, the remote execution ratio, and the throughput.

Fig. 4 Overall performance

As shown in Fig. 4(a), TrimMoE attains the lowest average latency of 142ms, which is 62.8% lower than EdgeShard, 39.6% lower than MoE², and 24.5% lower than our prior SubstituteOnly scheme. The additional gain over Substitute-Only is the direct effect of the two depth-reduction mechanisms: a lowimportance layer that would otherwise require a remote hop is skipped on the spot, and once the confidence requirement is met the remaining layers, together with their potential cross-server transmissions, are terminated at once. Fig. 4(b) shows that the cross-server traffic of TrimMoE is only 1.05 GB per one thousand tokens, 77.2% below EdgeShard and 30.0% below Substitute-Only, and Fig. 4(c) shows that its remote execution ratio drops to 16%, far below the 71% of EdgeShard and the 26% of Substitute-Only. Consistently, Fig. 4(d) shows that TrimMoE delivers the highest throughput of 1620 tokens per second, 2.35× that of EdgeShard. It is worth noting that MoD reduces latency relative to EdgeShard but remains clearly worse than the MoE-aware schemes, because it skips layers without being aware of which executions are cross-server, so it often drops cheap local layers while keeping the expensive remote ones. C. Tail Latency and Execution Behavior Fig. 5 reports the latency distribution and the expert execution-type breakdown. Fig. 5(a) plots the empirical CDF of the per-request latency, and Fig. 5(b) decomposes all MoE layer-level executions into local exact, local substitute, remote exact, remote substitute, skipped, and early-exited.

Fig. 5 Tail latency and execution behavior

From Fig. 5(a), TrimMoE not only shifts the whole distribution to the left but also compresses the tail: its P99 latency is 268ms, 41.1% lower than that of MoE² and 64.0%

15 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID lower than that of EdgeShard. The compressed tail comes from tokens, which indicates that a large fraction of tokens do not the look-ahead selection, which avoids moving a token onto a need the full depth once their representation has stabilized. server that is favorable now but expensive for the subsequent From Fig. 7(b), the skip ratio is low in the shallow layers, where layers, thereby eliminating the worst-case multi-hop trajectories. experts are more specialized and important, and rises in the From Fig. 5(b), TrimMoE removes 23% of the layer executions middle-to-deep layers, where the routed expert contributes entirely, 11% skipped and 12% terminated by early exit, and marginally and can be bypassed through the residual path. This among the remaining executions only 16% are remote. In depth-aware behavior is consistent with the layer-importance contrast, EdgeShard executes every layer exactly and incurs 71% model and explains why the quality loss remains bounded even remote executions, while Substitute-Only still runs to full depth though nearly a quarter of the layer executions are eliminated. and leaves 26% remote executions. This confirms that F. Effect of the Look-Ahead Horizon TrimMoE attacks the two residual sources of cross-server traffic that substitute execution alone cannot remove. D. Inference Quality and the Quality Budget Fig. 6 evaluates the inference quality. Fig. 6(a) reports the quality of representative methods on the three datasets, and Fig. 6(b) shows how the latency and the actual quality degradation of TrimMoE vary with the per-user quality budget.

Fig. 8 Effect of the look-ahead horizon

Fig. 6 Inference quality and quality budget

From Fig. 6(a), the quality of TrimMoE stays close to the exact methods: compared with the full centralized model, the perplexity on WikiText-103 rises by only 0.38, the SQuAD F1 drops by only 0.8, and the GSM8K accuracy drops by only 1.1. This small and controlled loss is a consequence of the design: early exit is gated by the confidence condition and is therefore quality-preserving by construction, while skipping is admitted only for low-importance layers within the offline-calibrated tolerance and the online per-token budget. By comparison, MoD skips layers without a confidence guarantee and suffers a larger GSM8K drop of 2.4. From Fig. 6(b), as the quality budget is relaxed from 0.5% to 3%, TrimMoE trades quality for latency in a smooth and monotone manner, and the measured degradation always stays strictly below the budget bound, which is exactly the hard guarantee established in Property 2. The knee of the curve is around a 2% budget, where the latency is already close to its minimum while the degradation is still well controlled, which justifies the default setting. E. Depth-Reduction Behavior

Fig. 7 Depth reduction behavior

To examine how TrimMoE allocates depth, Fig. 7 reports the exit-layer distribution and the per-layer skip ratio on Mixtral8x7B. From Fig. 7(a), most tokens are confidently terminated in the deep layers, with the distribution peaking around layers 20 to 22 and a residual mass at the final layer for the hardest

Fig. 8 evaluates the effect of the look-ahead horizon H on Mixtral-8x7B, with H varied from 1 to 5, where H=1 reduces to the myopic per-layer decision. As H grows from 1 to 3, both the average latency and the cross-server traffic decrease quickly, by 17.0% and 29.1% respectively, because anticipating more future layers prevents the strategy from skipping or relocating a token in a way that merely defers an expensive transmission to a later hop. Beyond H=3 the two curves saturate, while the per-token decision overhead keeps growing because the costto-go recursion expands more candidate trajectories. Thus H=3 offers a good balance between decision quality and runtime overhead, in agreement with the complexity analysis in Property 4. G. Comparison Across MoE Models

Fig. 9 Comparison across MoE models

Fig. 9 compares TrimMoE with the representative baselines on the three MoE models. As shown in Fig. 9(a), TrimMoE consistently achieves the lowest latency, and its advantage widens as the model scales: the latency reduction relative to MoE² grows from 40.4% on Switch-Base-8E to 39.6% on Mixtral-8x7B, while the absolute saving grows from 21ms to 93 ms. Larger models have more layers and more experts, which both increases the number of cross-server-prone layers that skipping and early exit can remove and enlarges the pool of local substitutes. As shown in Fig. 9(b), the quality retention of TrimMoE stays above 99.2% on all three models, confirming that the latency gains do not come at the cost of inference quality.

16 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID H. Ablation Study

Fig. 10 Ablation Study

Fig. 10 isolates the contribution of each component on Mixtral-8x7B by removing the skip mechanism, the early exit, the look-ahead, and the communication-aware redundancy in turn. From Fig. 10(a) and Fig. 10(b), removing any component increases both latency and cross-server traffic. Disabling the skip mechanism causes the largest latency increase, from 142ms to 176ms, because the token must then traverse every layer and pay every cross-server hop that an importance-based skip would have removed; disabling early exit and look-ahead also degrades performance, though to a smaller extent. From Fig. 10(c), the quality degradation of all variants stays small and even decreases slightly when skipping is removed, which shows that the depth-reduction components mainly serve to cut latency and communication rather than to preserve quality—the quality is already protected by the confidence gate and the per-user budget regardless of which component is enabled. I. Performance Under Different Request Loads

TrimMoE substantially lowers average and tail latency as well as cross-server traffic while preserving task quality and remaining robust under heavy load. Our analysis delineates the guarantee scope clearly: the substitution-and-skipping proxy degradation is hard-bounded, while exit quality is governed by the confidence gate and adaptive calibration. Future work includes skip-conditioned transition modeling to tighten lookahead accuracy and extending the attention-placement model to settings where attention is sharded across servers. ACKNOWLEDGMENT This work was supported in part by the grant from NSFC Grant no. 62571156, 62101159, 52475009, NSF of Shandong Grant no. ZR2021MF055, ZR2025QC666, the Research Grants Council of Hong Kong under the Areas of Excellence scheme grant AoE/E-601/22-R, and also the Opening Project of the Key Laboratory of Advanced Manufacturing and Intelligent Technology (Ministry of Education) at Harbin University of Science and Technology (KFKT202306). Additionally, the authors used an AI-based language assistance tool to improve the clarity and readability of parts of the manuscript, particularly the Abstract and Introduction, and all technical content, analysis, and conclusions were developed and carefully verified by the authors. REFERENCES [1] [2] [3] [4] [5]

Fig. 11 Performance Under Different Request Loads

Finally, Fig. 11 evaluates robustness to the request load, varied from 10 to 80 requests per second. As shown in Fig. 11(a), the average latency of all methods grows with the load because of increased GPU queueing and link contention, but TrimMoE grows the slowest, since fewer and shorter cross-server transmissions relieve both the busy links and the GPU queues. As shown in Fig. 11(b), where the SLA satisfaction ratio is defined as the fraction of requests completed below a 300ms target, TrimMoE still satisfies 71% of requests at 80 requests per second, whereas the baselines fall below 50% under the same load. This confirms that, by removing unnecessary layer executions and the transmissions they trigger, TrimMoE remains markedly more robust under heavy load. VII. CONCLUSION This paper reframed distributed-edge MoE serving from accelerating remote-expert access to eliminating unnecessary layer executions and the transmissions they trigger. TrimMoE jointly optimizes communication-aware layer skipping, confidence-based early exit, substitute execution, and serverexpert selection under a quality budget, with a transition-aware look-ahead and a skip/exit-aware expert replication. Across three MoE models on a heterogeneous 10-server testbed,

[6] [7] [8]

[9] [10] [11] [12] [13] [14] [15] [16]

W. Zhao, K. Zhou, et al., "A survey of large language models," Frontiers of Computer Science, vol. 20, 2026, pp: 1-144. Z. Zhou, X. Ning, K. Hong, et al., "A survey on efficient inference for large language models," arXiv:2404.14294, 2024, pp: 1-36. Y. Zheng, Y. Chen, B. Qian, X. Shi, Y. Shu, and J. Chen, "A review on edge large language models: Design, execution, and applications," ACM Computing Surveys, vol.57, no.8, 2025, pp: 1-35 A.Q. Jiang, A. Sablayrolles, A. Roux, et al., "Mixtral of experts," arXiv:2401.04088, 2024, pp: 1-13. W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang, "A survey on mixture of experts in large language models," IEEE Transactions on Knowledge and Data Engineering, vol.37, no.7, 2025, pp: 3896-3915. E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, "GPTQ: Accurate post-training quantization for generative pre-trained transformers," ICLR, 2023, Kigali, Rwanda, pp: 1-16. X. Ma, G. Fang, and X. Wang, "LLM-Pruner: On the structural pruning of large language models," NeurIPS, 2023, New Orleans, USA, pp: 120. C. Yang, Y. Zhu, W. Lu, Y. Wang, Q. Chen, C. Gao, B. Yan, Y. Chen, "Survey on knowledge distillation for large language models: Methods, Evaluation, and Application," ACM Transactions on Intelligent Systems and Technology, vol.16, no.6, 2025, pp: 1-27. X. Wang, Y. Zheng, Z. Wan, and M. Zhang, "SVD-LLM: Truncationaware singular value decomposition for large language model compression," ICLR, 2025, Singapore, pp: 1-21 E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, et al., "LoRA: Low-rank adaptation of large language models," ICLR, 2022, Virtual, pp: 1-26. K. Alizadeh et al., "LLM in a flash: Efficient large language model inference with limited memory," ACL, 2024, Bangkok, Thailand, pp. 12562–12584. M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, "EdgeShard: Efficient large language model inference via collaborative edge computing," IEEE Internet of Things Journal, vol.12, no. 10, 2025, pp: 13119-13131. Z. Li，L. Zheng, Y. Zhong, V. Liu, et al., "AlpaServe: Statistical multiplexing with model parallelism for deep learning serving," USENIX OSDI, 2023, Boston, USA, pp. 663-679. G.I. Yu, J. S. Jeong, G.W. Kim, S. Kim, and B.G. Chun, "Orca: A distributed serving system for transformer-based generative models," USENIX OSDI, 2022, Carlsbad, USA, pp. 521-538. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, "Megatron-LM: Training multi-billion parameter language models using model parallelism," arXiv:1909.08053, 2019, pp: 1-15. C. Guo et al., "RDMA over commodity Ethernet at scale," ACM

17 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID [17]

[18]

[19] [20] [21] [22] [23] [24]

[25] [26] [27]

[28]

[29] [30] [31] [32] [33] [34] [35]

[36] [37]

SIGCOMM, 2016, Florianopolis, Brazil, pp. 202-215. G. F. Pfister, T. Cortes, H. Jin, et al., "An introduction to the InfiniBand architecture," High Performance Mass Storage and Parallel I/O: Technologies and Applications, IEEE Press, 2002, pp. 617-632. DOI: 10.1109/9780470544839.ch42 Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, "A survey on mobile edge computing: The communication perspective," IEEE Communications Surveys & Tutorials, vol. 19, no. 4, 2017, pp. 23222358. X. Zhang and S. Debroy, "Resource management in mobile edge computing: A comprehensive survey," ACM Computing Surveys, vol. 55, no. 13s, 2023, pp. 1–37. X. Men, J. Nabende, E. Shutova, M.T. Pilehvar, "ShortGPT: Layers in large language models are more redundant than you expect," ACL, Vienna, Austria 2025, pp: 20192-20204. D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro, "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models," arXiv:2404.02258, 2024, pp: 1-14. T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, D. Metzler, "Confident adaptive language modeling," NeurIPS, New Orleans, USA, 2022, pp:1-17. T. Mohammed, C. Joe-Wong, R. Babbar, and M.D. Francesco, "Distributed inference acceleration with adaptive DNN partitioning and offloading," IEEE NFOCOM, 2020, Toronto, Canada, pp. 854-863. S. K. Ghosh, S. K. Raghavan Unnithan, I. Diaconita, A. Raha, V. Raghunathan, and A. Raghunathan, "PArtNNer: Platform-agnostic adaptive edge-cloud DNN partitioning for minimizing end-to-end latency," ACM Transactions on Embedded Computing Systems, vol. 23, no. 1, 2024, pp. 1-38. Y. Matsubara, M. Levorato, and F. Restuccia, "Split computing and early exiting for deep learning applications: Survey and research challenges," ACM Computing Surveys, vol. 55, no. 5, 2023, pp. 1-30. F. Cai, D. Yuan, Z. Yang, and L. Cui, "Edge-LLM: A collaborative framework for large language model serving in edge computing," IEEE ICWS, Shenzhen, China, 2024 pp: 1-11. K. Zhang, Y. Mao, S. Leng, S. Maharjan, and Y. Zhang, "Communication-efficient distributed on-device LLM inference over wireless networks," IEEE Journal of Selected Topics in Signal Processing, vol.19, no.7, 2025, pp: 1301-1317. S. Ye, B. Ouyang, L. Zeng, T. Qian, X. Chu, J. Tang, and X. Chen, "Jupiter: Fast and resource-efficient collaborative inference of generative LLMs on edge devices," IEEE ICC, Montreal, Canada, 2025, pp: 1-10. S. Zhong, L. Liang, Y. Wang, R. Wang, R. Huang, and M. Li, "AdapMoE: Adaptive sensitivity-based expert gating and management for efficient MoE inference," IEEE/ACM ICCAD, NJ, USA, 2024, pp: 1-9. Q. Chen, X. Chen, and K. Huang, "SlimCaching: Edge caching of mixture-of-experts for distributed inference," IEEE Transactions on Mobile Computing, vol.25, no.7, 2026, pp: 10924-10938. K. Li, W. Huang, Q. Wang, L. Zheng, X. Liao, H. Jin, and J. Xue, "DiffMoE: Efficient batched MoE inference with priority-driven differential expert caching," ACM SC, ST. LOUIS, USA, 2025, pp: 1951-1965. Q. Zhu, X. Ye, Y. Liu, H. Ouyang, and C. Song, "PROBE: Co-balancing computation and communication in MoE inference via real-time predictive prefetching," arXiv:2602.00509, 2026, pp: 1-14. R. Zhu, Z. Jiang, C. Jin, P. Wu, et al., "MegaScale-Infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism," ACM SIGCOMM, Coimbra, Portugal, 2025, pp: 592-608. N. Xue, Y. Sun, Z. Chen, M. Tao, X. Xu, L. Qian, S. Cui, and P. Zhang, "WDMoE: Wireless distributed mixture of experts for large language models," IEEE GLOBECOM, Cape Town, South Africa, 2024, pp: 1-10. L. Jin, Y. Zhang, Y. Li, S. Wang, H. H. Yang, J. Wu, and M. Zhang, "MoE2: Optimizing collaborative inference for edge large language models," IEEE Transactions on Networking, vol.34, 2026, pp: 46374651. M. Elhoushi et al., "LayerSkip: Enabling early exit inference and selfspeculative decoding," ACL, Bangkok, Thailand, 2024, pp. 12622– 12642. X. Yuan, N. Li, Q. Chen, W. Xu, A.V. Vasilakos, S. Guo, H. Zhang, “OrderMoE: An Expert Similarity Driven Distributed Edge MoE Inference,” arXiv: 2607.17154, 2026, pp: 1-17.

Related documents

• Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization#126492
• LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices#1061659
• X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths#1119285
• BIDENT: Heterogeneous Operator-level Mapping for Efficient Edge Inference#1051469
• FedSceneX: Time-to-Target Orchestration for Same-Scene Multimodal Federated Edge Learning#441739