MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks Sokratis Nikolaidis† , Stylianos I. Venieris‡ , Leonidas Malachias† , and Iakovos S. Venieris† † National Technical University of Athens, Athens, Greece, ‡ Samsung AI Center, Cambridge, UK
arXiv:2609.14660v1 [cs.DC] 13 Sep 2026
Email: [email protected], [email protected], [email protected], [email protected] Abstract—Split computing constitutes a widely used distributed inference approach, where a lightweight head model is onloaded onto the device and a heavier tail model resides on an edge server, leveraging the growing computational capabilities of modern System-on-Chips while alleviating server load. As intelligent indoor environments such as smart offices grow increasingly populated with diverse IoT devices, a single edge server must simultaneously assist multiple devices, each competing for the same shared inference resources. Without a principled mechanism to manage this shared load, the server is quickly overwhelmed, causing latency SLO violations and rendering server-assisted inference ineffective. In this work, we present MANE, a distributed inference framework that equips the server with a multipath tail architecture, enabling a dynamic accuracy–throughput trade-off at runtime. By introducing a novel multi-path model architecture, a three-stage training scheme featuring a Joint Head Network Distillation loss and a hysteresis-based scheduler with an equitable device-fallback policy, MANE maintains over 80% SLO satisfaction rate where state-of-the-art onloading methods fail completely, while preserving accuracy 6pp higher than ondevice alternatives, across up to 40 concurrent devices.
I. I NTRODUCTION The proliferation of mobile and Internet-of-Things (IoT) devices has driven a growing demand for deploying sophisticated deep neural networks (DNNs) in smart environments [1]. Conventionally, to alleviate the heavy computational demands of state-of-the-art DNNs, developers have resorted to cloud or edge offloading, pushing as much computation as possible to a remote server. However, the emergence of modern Systemon-Chips (SoCs), equipped with progressively more powerful processors, has expanded the capabilities of embedded devices. This shift has led to the new distributed inference paradigm of onloading [2]. Contrary to offloading, onloading allows server-based DNN applications to deliberately push computation onto the edge devices in order to exploit their growing local compute. In the context of split computing, the server onloads the early stages of computation onto the device, reducing both the server’s processing burden and the dimensionality of the intermediate feature maps transmitted over the network. While existing split-computing approaches have demonstrated latency and communication benefits, the vast majority assume a static environment where a single device has exclusive access to the server. In realistic smart environments, a single edge server must provide inference assistance to multiple devices concurrently [3]. As the number of assisted devices grows, the aggregate arrival rate of inference requests inevitably surpasses the processing throughput of the server’s tail model. In a naive deployment, this contention leads to
unboundedly growing queues, causing severe latency servicelevel objective (SLO) violations. Without a dynamic mechanism to manage the server’s load, the system is eventually forced into an indiscriminate fallback to on-device execution, negating the accuracy benefits of server-assisted inference. To address the limitations of static onloading, we propose MANE,1 a novel distributed inference framework designed specifically for multi-tenant edge servers. MANE overcomes rigid throughput constraints by transforming the server-side architecture into a multi-path network, enabling a dynamic trade-off between accuracy and speed. MANE onloads a highly compressed, path-agnostic head to the client devices. During inference, these devices transmit intermediate feature maps to the server, which dynamically routes them through varying depths of the pretrained backbone model based on instantaneous load. To manage this architecture, we introduce a server-side scheduler that continuously monitors request queues, scaling down to faster execution paths during traffic surges, and enforcing a targeted, equitable fallback to ondevice execution when maximum capacity is exceeded. The main contributions of the paper are the following: A novel split-computing architecture that pairs a highly compressed, path-agnostic on-device head with a multipath server-side tail, introducing a dynamic accuracy–throughput trade-off to edge onloading. • A three-stage training methodology featuring a novel Joint Head Network Distillation loss that mathematically ensures that the onloaded head learns a universally compatible intermediate representation for all server paths. • A dynamic scheduler that employs hysteresis-based path switching and an equitable device-fallback policy to maximise system-wide accuracy while meeting latency SLOs. •
II. R ELATED W ORK Split Computing & Onloading. The distribution of DNN inference between mobile devices and edge servers has been extensively studied within the scope of split computing [4]. After Neurosurgeon [5] introduced the partitioning of DNN computation at layer granularity, a large body of work followed, introducing onloading [2] and tackling challenges including dynamic split-point selection [6], [7], model partitioning and resource allocation [8]–[10], and energy-aware inference and feature compression [11]–[13]. 1 Multi-Path Adaptive Network for Edge Onloading.
Server
Input
Path Adapter 1
Tail Part 1
Path Adapter 2
Tail Part 2
Shared Head Path Adapter n Device
Scheduler
Tail Part n Prediction
Fig. 1. Multi-path adaptive architecture for edge onloading.
Knowledge Distillation for Edge Inference. Knowledge distillation (KD) [14] transfers knowledge from a large teacher model to a compact student by training on the teacher’s soft output distributions rather than hard labels alone. Tailored to split computing, Head Network Distillation (HND) [15] distills the on-device head of a bottleneck-injected DNN, minimising both on-device computation and the size of intermediate feature maps transmitted to the server while preserving accuracy. In MANE, we enhance HND in our three-stage training scheme. Subsequent work has further advanced bottleneckbased split computing through improved training strategies, compression benchmarking, and multi-task extensions [16]– [18]. Ladon [18] proposed a single shared encoder compatible with multiple task-specific heads, comprising a parallel to MANE’s path-agnostic head design. Multi-Device Scheduling. The majority of split computing works assume a single device with exclusive access to a server, an assumption that does not hold in real-world, multi-device smart environments. Several works have addressed multiuser inference scheduling at the edge through batching, early exiting, and latency-aware model switching [19]–[22]. More related to MANE, MultiTASC [3] proposed a multi-tenancyaware scheduler for cascade architectures that adaptively controls per-device forwarding decisions to maximise throughput while satisfying latency SLOs. MANE focuses on the splitcomputing setting with a richer accuracy–throughput trade-off enabled through a server-side, multi-path architecture. To the best of our knowledge, this paper is the first to jointly address multi-path split computing, bottleneck distillation, and multitenant scheduling in a unified framework. III. S YSTEM M ODEL & P ROBLEM F ORMULATION In this section, we define the system architecture of the Multi-Device Onloading setting. An edge server seeks to provide DNN inference assistance to a set of M IoT devices D = {d1 , . . . , dM }, all performing the same task. Rather than processing raw inputs, the server onloads the early stages of computation onto the devices, delegating execution of a lightweight head model to each device and reserving its resources for the more computationally demanding tail. Single-Device Onloading. Let x ∈ X be a raw input sample on an IoT device. The server onloads a lightweight head model fh : X → Z onto the device, which executes it locally to produce a compressed intermediate representation z = fh (x) ∈ Z. This representation is transmitted to the
server, where the tail model ft : Z → [0, 1]K completes the inference, producing a probability distribution over K classes, with predicted label ŷ = arg max ft (z). Formally: onloadfh ,ft (x) = ft (fh (x))
(1)
From the server’s perspective, onloading reduces the volume and dimensionality of incoming data — the server receives compressed representations z ∈ Z rather than raw inputs x ∈ X — alleviating both communication overhead and the server’s preprocessing burden. From the device’s perspective, the head model is substantially lighter than the full DNN, and raw input data never leaves the device, preserving privacy. Multi-Device Onloading. To capture the more realistic setting of multi-device smart environments, we extend the singledevice model to M devices, each running the same head model fh and transmitting compressed representations concurrently. Let xm ∈ X denote the input sample of device dm . The inference result for device dm is: onloadfh ,ft (xm ) = ft (fh (xm )),
∀ dm ∈ D
(2)
All intermediate representations are placed in a shared request queue Q, from which the server draws batches of size B and processes them using the shared tail model ft at a throughput of µB (samples/sec). Each device dm generates requests at an arrival rate λm , yielding an aggregate arrival PM rate Λ = m=1 λm . As the number of assisted devices grows, Λ increases and the server’s ability to process requests within a given latency budget degrades. Problem Formulation. The server’s objective is to maximise accuracy α across all assisted devices while satisfying a latency SLO L∗ for each processed sample. Formally: max α s.t. lhead,m + ltransfer,m + ltail (B) ≤ L∗ , ∀ dm ∈ D (3) where lhead,m is the on-device head execution time for device dm , ltransfer,m is the network transmission time of the compressed representation from device dm , and ltail (B) is the server-side tail inference time as a function of batch size B, including queueing delay. While lhead,m and ltransfer,m remain approximately constant regardless of the number of assisted devices, ltail (B) grows with Λ as requests accumulate in Q and queueing delay increases. IV. P ROPOSED S YSTEM The multi-device onloading setting demands a server-side architecture capable of dynamically adapting its accuracy– throughput operating point in response to fluctuating demand. To this end, we propose MANE, whose system architecture is illustrated in Fig. 1. Building upon the Head Network Distillation scheme comprising a custom lightweight head with a bottleneck, MANE enhances it by introducing multiple tail adapter paths on the server side, offering multiple options to balance the accuracy–latency trade-off during inference. To ensure stability, high accuracy, low latency and scalable system throughput, MANE uses a three-stage training process and a runtime scheduler that dynamically chooses the execution path and whether to selectively apply on-device execution.
Stage 1
LJHND =
N X
wi · ∥his − hit ∥2
(4)
LiHND = ∥his − hit ∥2
(5)
By isolating each path and freezing the shared representation learned during Stage 1, this second stage allows each adapter to specialise towards its respective teacher signal without interfering with the learned weights of other paths. Third Stage. In the third stage, the shared head remains frozen and the pretrained tail is attached to the pipeline. Each path is then fine-tuned individually using a standard KD loss [14], combining KL divergence between the student and teacher output distributions with a cross-entropy term against the ground-truth labels, balanced by a hyperparameter b ∈ [0, 1] and scaled by temperature T . Following this fine-tuning, each path reaches its peak accuracy, matching the performance that would be achieved if trained independently from the outset. B. Scheduler To effectively utilise MANE’s multi-path capabilities, we design a server-side scheduler that dynamically adapts the accuracy–throughput trade-off by continuously monitoring system load and via two novel policies: path switching and device fallback. Upon initialisation, the scheduler calculates the largest batch size that can be processed by each path p within the latency SLO, defining this as the path’s capacity (Cp ). At run time, the scheduler evaluates the system state by comparing the current request queue length (QLcurrent ) against the capacity of the active path (Ccurrent ). Path Switching. Path switching enables the server to dynamically adapt to fluctuating arrival rates. To prevent rapid oscillation between paths due to instantaneous load spikes, the scheduler employs a hysteresis mechanism by means of a growth counter GC and a drain counter DC.
Path 2 Teacher 2 Path n
Stage 2
Input
Path 1
HND
Teacher 1 Path 2
HND
Teacher 2 Path n
and hit
HND
Teacher n
Stage 3
Input
Shared Head
denote the feature maps produced by the student and the respective teacher for path i, and wi is a scalar weight controlling each path’s contribution to the total loss. This joint optimisation encourages the shared head to learn a path-agnostic intermediate representation. Training utilises early stopping on the validation set. Second Stage. With the universally compatible representation established, the shared head is now frozen and the pretrained tail remains detached. Each path adapter is trained individually using standard HND loss against its respective teacher head:
Teacher 1
Teacher n
i=1
where his
Shared Head
Input
Path 1
Shared Head
Fig. 2 depicts MANE’s carefully orchestrated training process for multi-path model splitting, comprising three stages. First Stage. The aim of the first stage is for the student model to learn to produce intermediate feature maps that satisfy the requirements of all paths simultaneously. Rather than training each path to independently mimic its respective teacher, we introduce the Joint Head Network Distillation (JHND) loss. JHND jointly optimises the shared head and all path adapters via a weighted combination of per-path HND losses [15]:
Joint HND
A. Training Scheme
Path 1
Tail 1
Tail 2
Tail n
Path 2 KD 1
KD 2
KD n
Path n
Full Teacher Mode
Fig. 2. MANE’s three-stage training process.
The GC counter tracks sustained queue growth. When the queue length exceeds the current path’s capacity and has grown since the last batch was processed, GC is incremented by one; otherwise it is reset to zero: GC + 1 if QLcurrent > Ccurrent (6) GC = and QLcurrent > QLlast 0 otherwise where QLcurrent is queue length at scheduling time, Ccurrent the active path’s capacity, and QLlast queue length at the previous batch’s time of processing. When GC≥GL, where GL is the growth limit, the scheduler switches to the next fastest path. Conversely, the drain counter DC tracks sustained queue draining. When the queue length falls below the capacity of the next more accurate path (Caccurate ) and has decreased since the last batch, DC is incremented; otherwise it is reset to zero: DC + 1 if QLcurrent < Caccurate DC = (7) and QLcurrent < QLlast 0 otherwise where Caccurate is the capacity of the next more accurate path. When DC ≥ DL, where DL is the drain limit, the scheduler switches to the more accurate path to maximise accuracy. Device Fallback. Device fallback is triggered when server demand surpasses the processing capacity of even the fastest path, using the same growth counter logic as path switching with the additional condition that the current path must already be the fastest. When GC≥GL, a fraction ϕ of devices —
TABLE I
TABLE II
C OMPRESSION A NALYSIS
PATH S PECIFICATIONS
Model
Path
Param. Reduction Feature Map Reduction
Model
Path
Accuracy Accel.
ResNet152
Accurate Balanced Fast
213× 1550× 1922×
42.7× 21.3× 21.3×
ResNet152
Accurate Balanced Fast
83.98% 80.79% 78.03%
1.23× 1.85× 2.11×
Accurate ConvNeXt Small Balanced Fast
102× 1329× 1930×
16× 8× 8×
Accurate ConvNeXt Small Balanced Fast
89.39% 85.43% 80.39%
1.35× 2.12× 2.43×
vs. Raw Input (all paths)
—
16×
prioritised by sample submission volume to ensure fair resource allocation — is instructed to process their workloads locally using a lightweight fallback model, directly reducing the total arrival rate Λ and restoring the server’s ability to meet the latency SLO. Fallback devices are periodically rotated to maintain equitable quality of service under prolonged load, and are reintroduced to server-side onloading one at a time once the queue falls below the fastest path’s capacity and exhibits sustained draining, avoiding sudden load oscillations. Unlike path switching, fallback enforcement is subject to a minimum cooldown period Tcooldown between successive engagements. V. E VALUATION A. Experimental Setup MANE’s multi-path model architecture was implemented using PyTorch 2.11.0. We target an edge server equipped with a 10GB NVIDIA RTX 3080 GPU and an AMD Ryzen 5 3600 3.6GHz CPU, for both training and deployment. For the client devices, we target Samsung S20 FE, a dated flagship smartphone to simulate a median-capability user device, with device-side models deployed using PyTorch Mobile targeting the device’s NPU. For device–server communication, we employ the AMQP protocol, following standard practice for IoT device messaging. We target 200-class image classification using the Tiny ImageNet dataset, comprising 100k samples with an 80%–10%–10% training–validation–test split. All deployment results are reported on the held-out test set. Hyperparameters. All stages use the AdamW optimiser with early stopping on the validation set with a patience of 3. The learning rate is set to 10−3 for Stages 1 and 2, and reduced to 10−4 for Stage 3. Stage-specific hyperparameters are set as follows: wi = N1 for Stage 1’s JHND, b = 0.9 and T = 4 for Stage 3’s KD. The scheduler growth GL and drain limits DL are set to 3 and 5, respectively, and ϕ to 0.2. Evaluation Protocol. We evaluate MANE on ResNet152 [23] and ConvNeXt Small [24] adaptations, targeting latency budgets ranging from 100ms to 200ms with a varying number of devices. A three-path structure was implemented for each backbone model. The initial split point was chosen to preserve full accuracy, while the two additional tail injection points were selected based on the accuracy–throughput curve of candidate injection points to realise an accurate–balanced–fast path scheme. For ResNet152, the balanced and fast injection points were placed 14 and 18 blocks after the initial split point, respectively, while for ConvNeXt Small at 12 and 18 blocks.
Each device is assigned the same subset of 1000 images, randomly sampled from the test set using a fixed seed. Each experiment is repeated across three different seeds, reporting average, minimum, and maximum values. The sample order is shuffled before each run, and inter-arrival delays are sampled from a Poisson distribution with a mean of 20 ms to emulate realistic device usage patterns. Device inference latency measurements use a batch size of 1, averaged across 500 runs. The evaluation metrics are system throughput, average accuracy across devices, and SLO satisfaction rate, which is the percentage of samples processed within a latency target. Baselines. We compare MANE against two baselines: i) server-only execution, where samples are processed by the original pretrained model without any path switching or fallback, and ii) on-device execution, where all inference is performed locally using MobileNetV2 [25]. We also use two ablated versions of MANE to showcase the importance of each component: 1) MANE-NPS, where we disable both path switching and fallback from the scheduler, allowing only the accurate path to be used, and 2) MANE-NF, where we enable path switching and remove fallback. B. Model Evaluation Compression Analysis. Table I shows the compression achieved by MANE’s shared head relative to the original backbone models. The head model comprises approximately 12.6k parameters, representing a reduction of 213× and 102× over the accurate-path split point of ResNet152 and ConvNeXt Small, respectively. The intermediate feature maps produced by the head have a shape of [12 × 28 × 28], yielding a transmission cost reduction of ≈ 16× relative to the raw input and up to ≈ 42.7× relative to the naive split point of ResNet152’s fast path. As a result, MANE significantly reduces both the on-device compute cost and the communication overhead, independently of which server-side path is active. Per-Path Performance. Table II reports the accuracy and acceleration of each path over the pretrained backbone, using a batch size of 128. The three paths offer a consistent accuracy– throughput trade-off across both model families. The threestage training scheme manages to preserve accuracy across all paths. Notably, the accurate path matches or slightly exceeds the pretrained backbone’s accuracy, demonstrating that the JHND-based training in Stage 1 successfully guides the shared head towards a representation that serves all paths without compromising accurate. Balanced and fast modestly trade accuracy for substantial speedup, with fast delivering up to 2.11× for ResNet152 and 2.43× for ConvNeXt Small.
100
TABLE III
E ND - TO - END M ODEL P ERFORMANCE
Device Device
— 73.13%
5ms 25ms
ResNet152 MANE ResNet152
Server Server
82.64% 83.98%
15ms 13ms
ConvNeXt Small Server MANE ConvNeXt Small Server
89.35% 89.39%
13ms 11ms
End-to-end Model Performance. Table III compares MANE against the original pretrained models in terms of accuracy and inference latency, measured with a batch size of 1. MANE’s accurate path matches or exceeds the accuracy of the original backbone in both cases, while also achieving lower latency. This is attributed to the bottleneck injection and path adapter design, which allows the tail to begin inference at a deeper point in the network, reducing the total computation performed on the server. The MANE shared head adds only 5 ms of ondevice latency, which is substantially lower than the 25 ms required by the MobileNetV2 fallback model, demonstrating that the onloaded head induces minimal on-device overhead. C. Full System Evaluation SLO Satisfaction Rate & Accuracy. Fig. 3 evaluates MANE against our baselines on SLO satisfaction rate and accuracy under a challenging latency target of 100 ms. With the exception of device-only execution, all baselines eventually fail as the number of devices increases. Server-only execution using the pretrained ConvNeXt Small model collapses to a satisfaction rate of 0% at 15 devices, with MANE-NPS and MANE-NF following at 20 and 30 devices, respectively. The difference in failure point correlates directly with the throughput of each pipeline: the original ConvNeXt Small has the lowest throughput, followed by the accurate path used in MANE-NPS, and finally the fast path which MANENF eventually settles on before failing. In contrast, MANE maintains a satisfaction rate of around 80% even when all other baselines have completely failed, showcasing its adaptability under demanding conditions. On-device execution achieves 100% satisfaction rate throughout, as each sample is processed in approximately 25 ms, but at a substantially lower accuracy. On accuracy, MANE and MANE-NF begin trading off accuracy to sustain the satisfaction rate at around 20 devices. At 30 devices, MANE continues to trade accuracy via the fallback policy, while MANE-NF has exhausted its ability to do so, having already switched to the fastest path. Even under the challenging 100 ms target with 40 devices, MANE achieves a mean accuracy of 78% with low inter-device deviation, a direct consequence of the democratic fallback rotation policy. Similar behaviour is observed in Fig. 4, where MANE uses ResNet152 as the server-side model. The baselines fail at 15, 20, and 35 devices, respectively, consistent with the ConvNeXt results. A more pronounced satisfaction rate drop is visible at intermediate device counts due to ResNet152’s lower throughput compared to ConvNeXt Small. Despite this,
40
Accuracy (%)
MANE Shared Head MobileNetV2
MANE ConvNeXt Small MobileNetV2 MANE-NPS MANE-NF
60
0
MANE ConvNeXt Small MobileNetV2 MANE-NPS MANE-NF
85 80 75
20
70 0
5
10
15
20
25
30
Number of Devices
35
40
0
5
10
15
20
25
30
Number of Devices
35
40
Fig. 3. SLO Satisfaction Rate and Accuracy for ConvNeXt Small. 86
100
84
80
82
MANE ResNet152 MobileNetV2 MANE-NPS MANE-NF
60 40
Accuracy (%)
Accuracy Latency
Satisfaction Rate (%)
Loc.
Satisfaction Rate (%)
Model
90 80
20 0
MANE ResNet152 MobileNetV2 MANE-NPS MANE-NF
80 78 76 74 72 70
0
5
10
15
20
25
30
Number of Devices
35
40
0
5
10
15
20
25
30
Number of Devices
35
40
Fig. 4. SLO Satisfaction Rate and Accuracy for ResNet152.
MANE maintains a satisfaction rate above 75% across all tested device counts up to 40. On accuracy, the same tradeoff pattern is observed, with MANE trading more aggressively once the fast path is insufficient and fallback is engaged. Even in the most demanding scenario, MANE achieves an accuracy of 76%, substantially higher than on-device execution. System Throughput. Fig. 5 presents system throughput as the number of devices grows. Since each device contributes 1000 samples to the system, the ideal behaviour is a linear increase in throughput with the number of devices. Full server onloading and MANE-NPS plateau at around 20 devices, indicating the system has reached its maximum processing capacity. MANE-NF sustains linear throughput growth up to 30 devices before plateauing, demonstrating the benefit of the multi-path architecture in extending the system’s effective capacity. MANE is the only server-assisted method that continues to increase throughput beyond this point without plateauing, reflecting the combined effect of path switching and the fallback policy in avoiding server overload. The comparison with on-device execution is particularly instructive. Despite displaying linearly increasing throughput, on-device execution achieves substantially lower throughput values than server-assisted schemes at low device counts, as the server can process samples more efficiently by leveraging batched inference. This explains why even the more conservative baselines, i.e. server-only execution and MANE-NPS, outperform on-device execution in throughput when the device count is small. As device count grows enough that MANE begins reverting a significant portion to fallback, its throughput growth rate converges towards that of on-device execution, with the two curves becoming approximately parallel. Relaxed Latency Target. Fig. 6 shows SLO satisfaction rates and accuracy for all methods under a relaxed latency
Throughput (samples/sec)
1200
MANE ResNet152 MobileNetV2 MANE-NPS MANE-NF
1000 800
1000 800
600
600
400
400
200
200
0
0
5
10
15
20
25
30
Number of Devices
35
0
40
R EFERENCES
MANE ConvNeXt Small MobileNetV2 MANE-NPS MANE-NF
1200
0
5
10
15
20
25
30
Number of Devices
35
40
35
40
Fig. 5. Throughput for ResNet152 and ConvNeXt Small. 100
MANE ConvNeXt Small MobileNetV2 MANE-NPS MANE-NF
60 40
Accuracy (%)
Satisfaction Rate (%)
90 80
80 75
20 0
MANE ConvNeXt Small MobileNetV2 MANE-NPS MANE-NF
85
70 0
5
10
15
20
25
30
Number of Devices
35
40
0
5
10
15
20
25
30
Number of Devices
Fig. 6. SLO Satisfaction Rate and Accuracy with 200ms time target.
target, where the scheduler can sustain a higher capacity per path and therefore trade off accuracy less aggressively. On the satisfaction rate side, server-only execution and MANE-NPS fail completely at 15 and 20 devices, respectively, consistent with the 100 ms experiment. MANE begins trading off accuracy at 25 devices and maintains satisfaction rate above 80% throughout, mirroring its behaviour under the stricter SLO. A subtle but notable difference is observed in MANE-NF, which retains satisfaction rate of approximately 20% at 30 devices rather than collapsing to 0% as in the 100 ms setting, reflecting the additional headroom due to the relaxed latency budget. The most significant difference between the two setups is visible on accuracy. MANE leverages the relaxed latency target to preserve accuracy above 80% across a wider range of device counts, compared to the 78% floor observed under the 100 ms target. In this setting, MANE also consistently achieves higher accuracy than MANE-NF across all device counts. VI. C ONCLUSION This paper presents MANE, a novel distributed inference framework that enables split-computing systems to operate effectively in demanding multi-device edge environments. By equipping the server with a multi-path tail architecture and a Joint Head Network Distillation scheme, we introduce a dynamic accuracy–throughput trade-off that existing static onloading approaches cannot provide. MANE continuously adapts the active inference path in response to server load, while its equitable fallback policy ensures that no single device monopolises server resources, sustaining high satisfaction rates and accuracy across a growing number of concurrent devices. MANE’s adaptability and scalability represent a step towards making split computing a viable and robust solution for the intelligent indoor environments of the future.
[1] S. Laskaridis, S. I. Venieris, A. Kouris, R. Li, and N. D. Lane, “The Future of Consumer Edge-AI Computing,” IEEE Pervasive Computing, vol. 23, no. 3, 2024. [2] M. Almeida, S. Laskaridis, S. I. Venieris, I. Leontiadis, and N. D. Lane, “DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device,” ACM Trans. Embed. Comput. Syst., 2022. [3] S. Nikolaidis, S. I. Venieris, and I. S. Venieris, “MultiTASC: A MultiTenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge,” in ISCC, 2023. [4] Y. Matsubara, M. Levorato, and F. Restuccia, “Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges,” ACM Comput. Surv., 2022. [5] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge,” in ASPLOS, 2017. [6] A. Bakhtiarnia, N. Milošević, Q. Zhang, D. Bajović, and A. Iosifidis, “Dynamic Split Computing for Efficient Deep EDGE Intelligence,” in ICASSP, 2023. [7] L. Zhang, L. Chen, and J. Xu, “Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning,” in WWW, 2021. [8] J. Yan, S. Bi, and Y.-J. A. Zhang, “Optimal model placement and online model splitting for device-edge co-inference,” IEEE Transactions on Wireless Communications, 2022. [9] J. Huang, C. Samplawski, D. Ganesan, B. Marlin, and H. Kwon, “CLIO: Enabling Automatic Compilation of Deep Learning Pipelines across IoT and Cloud,” in MobiCom, 2020. [10] X. Hou, Y. Guan, T. Han, and N. Zhang, “DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices,” in IPDPS, 2022. [11] Y. G. Kim and C.-J. Wu, “AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning,” in MICRO, 2020. [12] M. Sung, S. Im, D. Bang, I.-M. Kim, S. Yun, and J.-M. Kang, “Why should the server do it all?: A scalable, versatile, and model-agnostic framework for server-light DNN inference over massively distributed clients via training-free intermediate feature compression,” arXiv, 2025. [13] D. Cha, J. Lee, D. Jung, and S. Pack, “Fast and fair split computing for accelerating deep neural network (DNN) inference,” ICT Express, 2025. [14] G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” in NeurIPS, 2014. [15] Y. Matsubara, D. Callegaro, S. Baidya, M. Levorato, and S. Singh, “Head Network Distillation: Splitting distilled deep neural networks for resource-constrained edge computing systems,” IEEE Access, 2020. [16] Y. Matsubara, D. Callegaro, S. Singh, M. Levorato, and F. Restuccia, “BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing,” in WoWMoM, 2022. [17] Y. Matsubara, R. Yang, M. Levorato, and S. Mandt, “SC2 benchmark: Supervised compression for split computing,” Transactions on Machine Learning Research, 2023. [18] Y. Matsubara, M. Mendula, and M. Levorato, “A Multi-Task Supervised Compression Model for Split Computing,” in WACV, 2025. [19] W. Shi, S. Zhou, Z. Niu, M. Jiang, and L. Geng, “Multiuser Co-Inference With Batch Processing Capable Edge Server,” IEEE Transactions on Wireless Communications, 2023. [20] Z. Liu, Q. Lan, and K. Huang, “Resource Allocation for Multiuser Edge Inference With Batching and Early Exiting,” IEEE Journal on Selected Areas in Communications, 2023. [21] A. Kouris, S. I. Venieris, S. Laskaridis, and N. D. Lane, “Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs,” in ICCAD, 2023. [22] V. Paramanayakam, A. Karatzas, D. Stamoulis, and I. Anagnostopoulos, “Ecomap: Sustainability-Driven Optimization of Multi-Tenant DNN Execution on Edge Servers,” IEEE Transactions on Computers, 2025. [23] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in CVPR, 2016. [24] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” in CVPR, 2022. [25] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in CVPR, 2018.