Replica-Aware Split Inference for Edge IoT Systems

Replica-Aware Split Inference for Edge IoT Systems Zied Jenhani, Nasrin Razmi, and Admela Jukan

arXiv:2611.04779v1 [cs.DC] 3 Nov 2026

Technische Universität Braunschweig, Germany; {zied.jenhani, n.razmi, a.jukan}@tu-braunschweig.de

Abstract—Split-Inference (SI) addresses the hardware resource limitations of Internet of Things (IoT) devices by distributing tiny Machine Learning (TinyML) model segments across multiple IoT devices. However, a device failure can interrupt the inference pipeline. This paper proposes a replica-aware recovery approach that deploys replicas of TinyML model segments originally hosted on failed devices onto surviving devices. We formulate the joint selection of replica hosts and communication protocols as a mixed-integer linear programming (MILP) problem that minimizes post-reconfiguration inference latency, accounting for computation and transmission latencies, under flash memory and runtime memory constraints. Simulations using MobileNetV2 on heterogeneous IoT networks compare the proposed approach with four baselines. We implement the model in a network of 15 IoT devices with a maximum link-rate reduction of 75%, and show that MILP approach achieves an average modeled end-to-end inference latency that is 6.8% lower than that of Computation Greedy and 18.3% lower than that of Random. Index Terms—TinyML, split inference, Internet of Things, fault tolerance, replica placement.

I. I NTRODUCTION Tiny Machine Learning (TinyML) enables on-board inference on resource-constrained Internet of Things (IoT) devices, reducing dependence on remote processing [1]. However, larger models may exceed the processing, flash memory, and runtime memory budgets of individual devices. Model optimization and memory-aware inference address these limitations [2], [3]. Complementing these methods, Split-Inference (SI) distributes sequential model segments across Internet of Things (IoT) devices that exchange intermediate activations, combining their memory and processing resources [4]. The resulting pipeline depends on the availability of all participating devices; a device failure therefore prevents inference from completing. Replica-based approaches improve inference resilience through model redundancy [5]–[8]. However, assigning replicas to surviving IoT devices that already host primary segments while satisfying memory constraints and minimizing computation and transmission latency remains a challenge. In this paper, we address this challenge with a replicaaware recovery approach that deploys replicas on demand on surviving devices through over-the-air (OT A) updates, preserving the original partitioning and the placement of surviving primary segments. We formulate the joint selection of replica hosts and communication protocols as a MixedInteger Linear Programming (MILP) problem that minimizes post-reconfiguration inference latency, accounting for computation and transmission latencies, subject to flash memory application-slot and runtime memory capacity constraints. Simulations compare the approach with Random, Computation

Greedy, SRAM Worst-Fit, and Flash First-Fit Decreasing across network sizes and failure scenarios. II. R ELATED W ORK Recent work has studied efficient model partitioning for SI across resource-constrained IoT devices to support TinyML inference close to sensors [9], [10]. For example, DeepThings combines fused tile partitioning (FTP) with dynamic task allocation across IoT edge devices to reduce execution latency and memory usage [11]. Neuron-level partitioning of convolutional and linear layers enables inference across networked Microcontroller Units while reducing per-device peak random-access Memory usage [12]. In addition, DISNET combines vertical and horizontal model partitioning to optimize inference latency and energy consumption across IoT devices [13]. These approaches generally assume that all participating devices remain available. Additional recovery mechanisms are therefore needed to sustain distributed inference under device failures. For example, the CONTINUER framework [14] selects recovery mechanisms after edge-node failures based on latency, accuracy, and service downtime requirements. Other resilience approaches include model repartitioning [15], earlyexit mechanisms [16], and skip connections enabled by gating networks inserted between layers [17]. Another approach estimates the predictions of failed devices at the edge using classifier weights and feature vectors from surviving devices, without model replication [18]. Beyond these recovery mechanisms, replica-based approaches have also been studied to improve the resilience of distributed Deep Neural Network (DNN) execution. The FRIM framework maintains redundant copies of DNN partitions across available nodes and adaptively adjusts the redundancy level to preserve complete inference paths under device failures [5]. A related approach has also been explored for distributed DNN training [19]. Rather than adapting redundancy after deployment, this approach predicts the fault probability of each model partition, determines the required number of replicas accordingly, and assigns these replicas to edge devices before anticipated failures. Other approaches introduce redundancy at a finer granularity or adapt it to specific model architectures. RobustDiCE [6] combines partial replication of important neurons with importance-aware partitioning to improve Convolutional Neural Network resilience. At the model level, RoCoIn [7] uses knowledge distillation to create compact student models and deploys redundant copies across heterogeneous edge devices. EASTER [8] applies a similar redundancy concept to Transformer models by replicating a

Model preparation

Failure monitoring

Recovery management

Training · partitioning Heartbeat-based availability Failure detection Edge server E Quantization · firmware build Heartbeats / recovery control

MILP: hosts + protocols Flash + runtime memory constraints

Differential OTA Update

a1 · BLE

IoT devices

Device 1

Device 2

Device 4

Device 3

r2,4

Sensor input

S1

a1 · UDP

Failure-free path

S2

Failed

a2 · BLE

Recovery path

S3

Failed

a3 · TCP

S4

Primary ML segment

Device 5 a2 · UDP a3 · TCP a4 · UDP

r3,5

S5

Prediction Deployed ML replica

Figure 1: Architecture of the replica-aware split-inference framework.

selected fraction of important connections while distributing the remaining connections across devices. Prior work improves resilience through pre-deployed replicas, redundant sub-models, or replication of neurons and connections. We address post-failure recovery of vertically partitioned TinyML inference across heterogeneous IoT devices by deploying replicas on demand through differential OT A updates. Replica hosts and communication protocols are jointly selected to minimize post-reconfiguration inference latency under flash memory and runtime memory constraints. III. S YSTEM M ODEL A. IoT Network Configuration and Replica-Aware Recovery We consider an IoT network consisting of an edge server E and a set D = {1, 2, . . . , D} of D geographically static, heterogeneous IoT devices that communicate wirelessly to perform an SI task. The fixed distance between any two connected devices i, j ∈ D is denoted by di,j . The edge server prepares the TinyML model by performing model training, model partitioning, post-training quantization, firmware generation, and OT A deployment to IoT devices. It is assumed that the SI model has already been partitioned between the IoT devices. In this regard, the TinyML model is divided into K vertically partitioned sequential segments, each of which may contain one or more layers depending on the selected split points. These segments are assigned to the K IoT devices, with K ≤ D. The segment assigned to the k-th IoT device is denoted by sk and referred to as its primary TinyML model segment. In the proposed workflow, the source IoT device collects the sensor data and executes its assigned primary TinyML model segment, sk . It then transmits the resulting intermediate activations, ak , obtained from the output of the final layer of segment sk , to the next connected device, which uses ak as input to execute its assigned primary TinyML model segment. This process is repeated across the participating IoT devices until the destination device produces the final inference result. For simplicity, we assume that the source IoT device, which hosts the sensor and the first segment s1 , does not fail. After deployment, the edge server monitors IoT devices through heartbeat messages transmitted at a predefined interval Tb . An IoT device is considered to have failed if no heartbeat message is received within a timeout To ≥ Tb . Upon detecting a failure, the edge server selects a suitable active IoT device based on its resource constraints and characteristics, including

flash memory, runtime memory, and computational latency, as well as network information such as transmission latency. Let rk,k′ denote the replica of the primary TinyML model segment sk , originally assigned to a failed device k ∈ F , that is deployed to a surviving device k ′ ∈ S for failure recovery, where F ⊆ D denotes the set of failed IoT devices, and S = D \ F denotes the set of surviving active devices. The replica segment is deployed to the selected device via a differential OT A update Fig. 1. Furthermore, let RF denote the set of all feasible replica-placement configurations for the failed-device set, where each configuration specifies how the TinyML model segments associated with the failed devices are assigned as replicas to the surviving devices in S. B. Hardware resources model a) Flash memory: Each IoT device k ∈ D uses a dualslot OT A scheme. Its application-slot capacity MkOTA accommodates the fixed flash memory allocation Mkbase , the primary TinyML segment Mkprim , and replicas with a total footprint of Mkrep . The fixed allocation includes the baseline firmware and space unavailable for model storage. The flash memory footprint of the TinyML model segment sk , following [20], is given by X mk = [mem(wℓ , Qℓ ) + Mℓ ] , (1) ℓ∈Lk

where Lk denotes the set of layers in segment sk , wℓ is the weight tensor of layer ℓ, and Qℓ denotes its bit precision. The term mem(wℓ , Qℓ ) represents the flash memory footprint of the quantized weights, while Mℓ accounts for biases, quantization scales, zero points, and other static parameters associated with layer ℓ. Accordingly, Mkprim = mk for every device hosting a primary segment, and Mkprim = 0 otherwise. Since a replica rk,k′ is an exact copy of sk , its footprint is also mk . For a given replica-placement configuration in RF , let Fk′ = {k ∈ F | rk,k′ is assigned to device k ′ }

(2)

denote the set of failed devices whose primary segments are replicated on the surviving device k ′ ∈ S. The total memory footprint of the replicas hosted by device k ′ is then X Mkrep = mk , ∀k ′ ∈ S. (3) ′ k∈Fk′

The total flash memory usage of each surviving device must remain within its application partition capacity as Mkbase + Mkprim + Mkrep ≤ MkOTA , ′ ′ ′ ′

∀k ′ ∈ S.

(4)

b) Runtime memory: Each IoT device is constrained by its memory capacities Mkr , where r ∈ {SRAM, PSRAM} indexes the memory pool: Static Random-Access Memory (SRAM) or pseudo-static random-access memory (PSRAM). At runtime, this working memory is partitioned between (i) the TinyML execution arena, which maintains intermediate activation tensors, temporary operator scratch buffers, and persistent model states, and (ii) the system and communication overhead. Following the SRAM decomposition proposed

in [21], we characterize the runtime tensor-arena requirement scr state of the TinyML segment sk as uk = uact , where k + uk + uk act uk denotes the peak memory occupied by live activation tensors, uscr k denotes the temporary memory required by the operator kernels, and ustate denotes the persistent runtime state k and arena-resident metadata allocated by the inference runtime for sk . Since a replica rk,k′ is an exact copy of sk , it has the same tensor-arena requirement uk on its host device k ′ . The activation term is estimated from the worst-case workingmemory requirement during the execution of sk as X uact size(x), (5) k ≈ max τ

x∈Xk (τ )

where τ denotes the discrete execution step of the layers in sk , Xk (τ ) is the set of activation tensors simultaneously alive in the tensor arena at step τ , and size(x) is the storage footprint of tensor x. We extend the segment-level model to each surviving device. For each host considered, urk denotes the segment’s memory requirement in pool r under that host’s fixed arena allocation, including per-segment overhead. The total memory required by replicas hosted on device k ′ ∈ S is X Ukrep,r = urk , ∀k ′ ∈ S. (6) ′ k∈Fk′

The total memory usage of device k ′ is then Ukr′ = Uksys,r + Ukcomm,r + Ukprim,r + Ukrep,r , ′ ′ ′ ′

(7)

where Uksys,r is reserved for system and application tasks, ′ Ukcomm,r is allocated to wireless communication buffers, and ′ Ukprim,r denotes the memory required by the primary segment ′ of device k ′ , i.e., Ukprim,r = urk′ if k ′ hosts a primary segment ′ prim,r and Uk′ = 0 otherwise. All quantities refer to pool r. Each memory pool must satisfy Ukr′ ≤ Mkr′ ,

∀k ′ ∈ S, r ∈ {SRAM, PSRAM}.

(8)

C. Computation latency The computation latency of the primary TinyML model segment sk executed on its assigned IoT device k is estimated relative to a reference model Θ as   vk ω k , (9) Tkcomp = ϕk ωΘ where vk is the data volume of the input activation ak−1 of segment sk , ϕk is the computing rate of device k, and ωk and ωΘ denote the numbers of computational operators in segment sk and the reference model Θ, respectively. When device k ∈ F fails, its segment runs as the replica rk,k′ on the surviving host device k ′ ∈ S. Since rk,k′ is an exact copy of sk , it has the same input volume vk and operator count ωk , while its execution speed is determined by the host device. Accordingly, the computation latency of rk,k′ is   vk ωk comp Tk,k′ = , ∀k ∈ F, k ′ ∈ S. (10) ϕk′ ωΘ With vertical, layer-wise partitioning, each segment processes the full input feature map. Thus, the data volume of the input

activation ak−1 of segment sk is vk = Hk Wk Ck b, where Hk , Wk , and Ck denote the height, width, and channel count of ak−1 , and b is the number of bytes per tensor element, with b = 1 under INT8 quantization. The computing rate of the k-th IoT device, expressed in bytes per second, is ϕk = fckk , where fk and ck denote the clock frequency in cycles per second and the number of clock cycles required to process one byte. D. Transmission latency The transmission latency represents the time required to transfer intermediate activations ak from segment sk to sk+1 . In the failure-free case, ak is sent from device k to device k + 1. If device k fails, sk is executed as the replica rk,k′ on the surviving device k ′ , which then becomes the sender; similarly, if device k + 1 fails, sk+1 is executed as the replica rk+1,k′′ on the surviving device k ′′ , which then becomes the receiver. Let P denote the set of considered communication protocols. For each pair of distinct surviving devices i, j ∈ S, we define Pi,j ⊆ P as the set of protocols supported by both devices and usable over the direct link from i to j. These sets are assumed known to the edge server and fixed during the evaluated inference. An inter-device activation transfer can use only a protocol p ∈ Pi,j . If Pi,j = ∅, direct communication between the two devices is unavailable. When consecutive segments execute on the same device, their activation is passed locally with zero communication latency. Assuming a direct wireless link using an available protocol p, the transmission latency is  i = j,  0, (11) Tktr,p = vk+1  p , i ̸= j, p ∈ Pi,j , Bi,j where vk+1 denotes the size of the intermediate activation ak p and Bi,j denotes the effective throughput (bytes/s) between the corresponding transmitter i and receiver j under protocol p. Here, i = k if device k is active, and i = k ′ if device k has failed and its segment is executed on the surviving replica device k ′ . Similarly, j = k + 1 if device k + 1 is operational, and j = k ′′ if the segment is executed on the surviving replica device j = k ′′ . For i = j, no wireless transfer is required. IV. P ROBLEM F ORMULATION The objective of the Replica-Aware recovery approach is to assign each failed segment to a surviving device hosting its replica, minimizing end-to-end (E2E) inference latency while satisfying the flash memory and runtime memory constraints. Let F K = F ∩ {1, . . . , K} denote the set of failed devices that hosted a primary segment. The replica placement is described by the binary decision variables xk,k′ ∈ {0, 1}, where xk,k′ = 1 if the replica rk,k′ of segment sk , k ∈ F K , is deployed on the surviving device k ′ ∈ S, and xk,k′ = 0 otherwise. The placement configuration is denoted by x = [xk,k′ ], and the set of replicas hosted by device k ′ is Fk′ (x) = {k ∈ F K | xk,k′ = 1}. Let p = (p1 , . . . , pK−1 ) denote the communication choices at the segment boundaries, where pk is the protocol used to transfer activation ak between

distinct hosts. For co-located segments, we use pk = local to indicate that no wireless transfer is required. The E2E inference latency of the reconfigured pipeline is X X X comp T e2e (x, p) = Tkcomp + xk,k′ Tk,k ′ k∈F K k′ ∈S

k∈{1,...,K}\F K

+

K−1 X

Tktr,pk (x),

k=1

(12) where the first term is the computation latency of the primary segments on the active devices, the second term is the computation latency of the replicas on their selected host devices, and the third term is the transmission latency of the intermediate activations along the reconfigured pipeline, as defined in (11). For each inter-device transfer, pk must belong to Pi,j , where i and j are the actual hosts determined by placement x. For co-located segments, the transmission term is zero. For a given placement x, let A(x) denote the set of admissible communication-choice vectors compatible with the replica-placement. The replica-placement and protocolselection problem is formulated as min T e2e (x, p) x,p P K ′ s.t. k′ ∈S xk,k = 1, ∀k ∈ F , P Mkbase + Mkprim + k∈F K xk,k′ mk ′ ′ ≤ MkOTA , ∀k ′ ∈ S, ′

K

|×|S|

,

yk,i,j ≤ hk,i ,

(13c)

where constraint (13b) ensures that each failed segment is replicated on exactly one surviving device. Constraints (13c) and (13d) guarantee that the primary segment and the hosted replicas of each device fit within its flash memory and runtime memory capacity, respectively. The feasible set RF consists of placements satisfying (13b)–(13e) with at least one compatible protocol assignment. An optimal solution satisfies x∈RF p∈A(x)

i∈Hk j∈Hk+1 p∈Pi,j j̸=i

(13b)

(13f)

(x⋆ , p⋆ ) ∈ arg min T e2e (x, p).

which is either a constant or an existing placement variable and thus introduces no new decisions. For each segment boundary k = 1, . . . , K − 1 and distinct candidate hosts i ∈ Hk , j ∈ Hk+1 , we introduce zk,i,j,p ∈ {0, 1} for each available protocol p ∈ Pi,j . This variable equals one when activation ak is transferred from device i to device j using protocol p. The transmission latency of activation ak then becomes X X X vk+1 zk,i,j,p p , Tktr,pk (x) = Bi,j

(13a)

(13e)

p ∈ A(x),

(16)

with zero communication latency for co-located hosts (i = j), consistent with Eq. (11). The products hk,i hk+1,j are bilinear only when both consecutive segments belong to F K ; otherwise one factor is constant and the term is already linear. For each bilinear product we apply the standard linearization of Glover [22], introducing an auxiliary variable yk,i,j ∈ [0, 1] with

P Uksys,r + Ukcomm,r + Ukprim,r + k∈F K xk,k′ urk ′ ′ ′ ≤ Mkr′ , ∀k ′ ∈ S, r ∈ {SRAM, PSRAM}, (13d) x ∈ {0, 1}|F

and, for i ∈ Hk , the host indicator ( 1, k∈ / F K , i = k, hk,i = xk,i , k ∈ F K ,

(14)

V. S OLUTION The problem defined in Eq. (13) can be reformulated as an MILP by linearizing the placement-dependent communication terms. When both consecutive segments sk and sk+1 require replicas, their link latency depends jointly on the selected hosts, introducing bilinear terms into the transmission latency in Eq. (13a). To consider all cases of Eq. (11) uniformly, we define for each segment sk the candidate-host set ( {k}, k ∈ / FK, Hk = (15) S, k ∈ FK,

yk,i,j ≤ hk+1,j ,

yk,i,j ≥ hk,i + hk+1,j − 1, (17) which enforces yk,i,j = hk,i hk+1,j at any feasible binary point. For the remaining candidate-host pairs, where at least one host indicator is constant, we define yk,i,j = hk,i hk+1,j directly; this relation is already linear. Thus, yk,i,j represents the selected host pair at every segment boundary. PProtocol selection is linked to the host-pair selection by p∈Pi,j zk,i,j,p = yk,i,j , for all k = 1, . . . , K − 1, i ∈ Hk , j ∈ Hk+1 with i ̸= j. When yk,i,j = 1, exactly one available protocol is selected. When yk,i,j = 0, all corresponding protocol-selection variables are zero. These constraints encode p ∈ A(x). For an inter-device transfer, the selected protocol pk is recovered from zk,i,j,p = 1; for a local transfer, pk = local. Since the transfer coeffip cients vk+1 /Bi,j are known constants, the resulting objective and constraints are linear in x, y, and z. VI. P ERFORMANCE E VALUATION A. Implementation and Device Parameters We ran simulations on an Intel Core i9-14900 workstation with 64 GB of RAM. Memory capacities and OTA slot sizes for IoT devices follow hardware specifications [23]–[26] and partition layouts [27]–[30] summarized in Table I. For ESP32S3 devices, the simulated fixed flash memory allocation leaves 1.3 MiB for primary and replica TinyML segments. Compatible differential-OTA software is assumed [31]. Segment storage and intermediate activation sizes follow the model partitions. Tensor arenas [32] use PSRAM on ESP32-S3 and internal SRAM on Arduino, Pico, and STM32 devices, with an assumed 2 KiB SRAM overhead per hosted segment. Latency models use assumed device computing rates for computation,

SRAM 512 256 520 786

PSRAM 8192 – – 8192

Internal Flash – 1024 – 2048

External Flash 16384 – 4096 65536

OTA slot 7680 448 1744 832

and activation sizes and effective link rates referenced to ESP32-S3 benchmarks for transmission [4]. The simulation scripts and input datasets used in the evaluation are available in the accompanying GitHub repository 1 . B. Network and Model Configuration We evaluate networks comprising D ∈ {5, 10, 15} distributed IoT devices. The device counts, ordered as ESP32-S3 / Arduino / Pico / STM32, are (2, 1, 1, 1) for D = 5, (3, 3, 2, 2) for D = 10, and (4, 4, 4, 3) for D = 15. Intermediate activations are exchanged directly using Wi-Fi (TCP/UDP), BLE, or ESP-NOW, with ESP-NOW restricted to ESP32-S3 pairs. Differential OT A updates use the server’s Wi-Fi access point for Wi-Fi-capable devices and BLE for the Arduino Nano 33 BLE Sense. We assume that each device is connected to the server through a wired monitoring interface for periodic status reports and recovery control. This infrastructure is assumed to remain available and is excluded from D and the inference-device resource budgets. The workload is an ImageNet-pretrained MobileNetV2 model with α = 0.35, input dimensions of 224×224×3, and INT8 quantization. We set K = D and keep the initial partitioning fixed throughout the evaluation. C. Failure Scenarios and Baselines
For each network size, we evaluate all D−1 failure comF binations for F = |F| ∈ {1, . . . , min(5, D − 1)}. The dataacquisition device is excluded from the failure set, and the initial deployment remains fixed across all failure combinations. Four baselines preserve surviving primaries and enforce flash memory and runtime memory constraints. Random places failed segments in random order on uniformly selected eligible surviving devices; results aggregate three seeds per scenario. Computation Greedy and SRAM Worst-Fit place segments in pipeline order, respectively minimizing computation latency and maximizing the remaining SRAM fraction after placement. Flash First-Fit Decreasing (FFD) processes segments by decreasing flash memory demand, selecting the first eligible device in ascending device-ID order. Each feasible heuristic placement uses the highest-throughput compatible protocol for transfers between distinct hosts. All methods share input parameters and feasibility checks. D. Evaluation Metrics and Results We evaluate three metrics: E2E inference latency T e2e (x, p), the algorithm computation time, and recovery success rate, defined as the percentage of attempts that find a valid recovery plan. Latency is evaluated at the original link 1 https://github.com/Zied-Jenhani/Replica-Aware-Split-Inference-for-EdgeIoT-Systems

Average end-to-end latency (ms)

Table I: Device memory and OTA slot sizes (KiB) Device ESP32-S3-WROOM-1-N16R8 Arduino Nano 33 BLE Sense Rev2 Raspberry Pi Pico 2 W STM32 B-U585I-IOT02A

5 devices

10 devices

15 devices

2,700 2,400 2,100 1,800 1,500 1,200 MILP

Computation Greedy

SRAM Worst-Fit

Flash FFD

Random

Figure 2: Average E2E inference latency at the original link rates.

rates and under link-specific reductions ranging from zero to a maximum of 25%, 50%, or 75%. Results are averaged over three link-rate configurations per reduction level. The average latency is computed only for failure scenarios that every method successfully recovers. For the MILP, computation time includes problem construction, optimization, and verification; for the heuristics, it includes replica placement, protocol selection, and feasibility verification. We report median workstation execution times with interquartile bands over all completed attempts, including unsuccessful attempts. Fig. 2 shows that the MILP achieves the lowest average E2E latency across all evaluated network sizes. Results nearly coincide with the deterministic heuristics for five devices, with larger differences for ten and fifteen devices. For fifteen devices, the MILP achieves 2393.78 ms, compared with 2482.93 ms for Computation Greedy and 2699.78 ms for Random. The MILP considers computation and transmission latency jointly when selecting replica hosts. The evaluated heuristics instead use random, computation-based, or memory-based placement rules, followed by selection of the fastest compatible protocols. Fig. 3 shows that average E2E latency increases for all methods as link rates decrease. For fifteen devices at a maximum link-rate reduction of 75%, latency reaches 3361.92 ms for the MILP, 3607.94 ms for Computation Greedy, and 4114.66 ms for Random. Lower rates increase transmission latency, making communication costs more influential in replica placement. Fig. 4 shows increasing median MILP computation time with failure count for ten and fifteen devices, reaching 22.69 ms and 50.33 ms, respectively, at five failures, where the evaluated heuristics are faster. For five devices with four failures, recovery is infeasible under the memory constraints, with a median MILP computation time of 1.95 ms. At the original link rates, the MILP and deterministic heuristics achieve identical recovery success rates: 73.3%, 94.2%, and 100% for D = 5, 10, 15, respectively. VII. C ONCLUSION Our approach restores SI across heterogeneous IoT devices by deploying replicas through differential OT A updates, preserving the original partitioning and surviving primary placements. The proposed MILP optimizes replica placement while accounting for compatible communication protocols to

MILP

Computation Greedy

Flash FFD

Random

SRAM Worst-Fit

Average end-to-end latency (ms)

4,500 4,000 Greedy vs MILP 246.02 ms

3,500 3,000 2,500 2,000 0

25

50

75

Maximum link-rate reduction (%)

Figure 3: Average E2E inference latency under link-rate reductions for 15 devices.

Algorithm computation time (ms)

MILP

Computation Greedy

5 devices

SRAM Worst-Fit

Flash FFD

10 devices

4.5

30

3.0

20

50

1.5

10

25

0.0

0

1.95 ms

1

2

3

Failed devices, F

4

22.69 ms

1

2

3

4

Failed devices, F

Random

15 devices

5

75

50.33 ms

0

1

2

3

4

5

Failed devices, F

Figure 4: Median algorithm computation time versus failure count, with interquartile bands.

minimize post-reconfiguration E2E inference latency under flash memory and runtime memory constraints. Simulations show lower average latency than four placement heuristics, with larger gains under reduced link rates in the 15-device network, while quantifying recovery-plan computation time. Future work will extend the optimization to OTA update latency and device energy consumption. R EFERENCES [1] C. R. Banbury et al., “MLPerf Tiny benchmark,” CoRR, vol. abs/2106.07597, 2021. [Online]. Available: https://doi.org/10.48550/ arXiv.2106.07597 [2] A. Capotondi, M. Rusci, M. Fariselli, and L. Benini, “CMix-NN: Mixed low-precision CNN library for memory-constrained edge devices,” IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 67, no. 5, pp. 871–875, 2020. [3] J. Lin, W.-M. Chen, Y. Lin, C. Gan, S. Han et al., “MCUNet: Tiny deep learning on iot devices,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 11 711–11 722, 2020. [4] Z. Jenhani, M. Bensalem, J. Dizdarevic, and A. Jukan, “Optimizing split learning latency in tinyml-based IoT systems,” 2026, version 2. [Online]. Available: https://arxiv.org/abs/2507.16594v2 [5] W. Liu, Z. Chen, and Y. Gong, “Towards failure-aware inference in harsh operating conditions: Robust mobile offloading of pre-trained neural networks,” Electronics, vol. 14, no. 2, p. 381, 2025. [6] X. Guo, Q. Jiang, A. D. Pimentel, and T. Stefanov, “RobustDiCE: Robust and distributed CNN inference at the edge,” in Proc. 29th Asia South Pacific Design Automat. Conf. (ASP-DAC), 2024, pp. 26–31. [7] L. Wang, L. Li, L. Xu, X. Peng, and A. Fei, “Failure-resilient distributed inference with model compression over heterogeneous edge devices,” IEEE Trans. Mobile Comput., vol. 23, no. 12, pp. 12 680–12 692, 2024. [8] X. Guo, et al., “EASTER: Learning to split transformers at the edge robustly,” IEEE Trans. Comput.-Aided Design Integr. Circuits Syst., vol. 43, no. 11, pp. 3626–3637, 2024. [9] F. Bove, S. Colli, and L. Bedogni, “Performance evaluation of split computing with TinyML on IoT devices,” in Proc. IEEE 21st Consum. Commun. Netw. Conf. (CCNC), 2024, pp. 1–6.

[10] H. Hafi, B. Brik, P. A. Frangoudis, A. Ksentini, and M. Bagaa, “Split federated learning for 6G enabled-networks: Requirements, challenges, and future directions,” IEEE Access, vol. 12, pp. 9890–9930, 2024. [11] Z. Zhao, K. M. Barijough, and A. Gerstlauer, “DeepThings: Distributed adaptive deep learning inference on resource-constrained IoT edge clusters,” IEEE Trans. Comput.-Aided Design Integr. Circuits Syst., vol. 37, no. 11, pp. 2348–2359, 2018. [12] J. Lu, S. Suresh, H. Liu, Q. Hong, and Q. Wang, “Split CNN inference on networked microcontrollers,” 2026. [Online]. Available: https://arxiv.org/abs/2605.09357 [13] E. Samikwa, A. Di Maio, and T. Braun, “DISNET: Distributed microsplit deep learning in heterogeneous dynamic IoT,” IEEE Internet Things J., vol. 11, no. 4, pp. 6199–6216, 2024. [14] A. A. Majeed, P. Kilpatrick, I. Spence, and B. Varghese, “CONTINUER: Maintaining distributed DNN services during edge failures,” in Proc. IEEE Int. Conf. Edge Comput. Commun. (EDGE), 2022, pp. 143–152. [15] ——, “NEUKONFIG: Reducing edge service downtime when repartitioning DNNs,” in Proc. IEEE Int. Conf. Cloud Eng. (IC2E), 2021, pp. 118–125. [16] S. Teerapittayanon, B. McDanel, and H. Kung, “BranchyNet: Fast inference via early exiting from deep neural networks,” in Proc. 23rd Int. Conf. Pattern Recognit. (ICPR), 2016, pp. 2464–2469. [17] X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez, “SkipNet: Learning dynamic routing in convolutional networks,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 420–436. [18] O. Jamshidi, M. Abbasi, A. Ramazani, A. Salimi Shahraki, and A. Taherkordi, “Modifier guided resilient CNN inference enables faulttolerant edge collaboration for IoT,” Sci. Rep., vol. 15, p. 45458, 2025. [19] T. Sen and H. Shen, “Fault tolerant data and model parallel deep learning in edge computing networks,” in Proc. IEEE 21st Int. Conf. Mobile AdHoc Smart Syst. (MASS), 2024, pp. 460–468. [20] M. Rusci, A. Capotondi, and L. Benini, “Memory-driven mixed low precision quantization for enabling deep network inference on microcontrollers,” in Proc. Mach. Learn. Syst., vol. 2, 2020, pp. 326–335. [21] N. S. Vummaneni, A. Mittal, I. Kumar, S. Bharadwaj, and H. Varshney, “Trend-aware Tinyml co-design on microcontrollers: Quantization, structured sparsity, and on-device adaptation under tight memory-energy budgets,” in Proc. SoutheastCon, 2026, pp. 1–7. [22] F. Glover, “Improved linear integer programming formulations of nonlinear integer problems,” Manage. Sci., no. 4, pp. 455–460, 1975. [23] Espressif Systems, ESP32-S3-WROOM-1 and ESP32-S3WROOM-1U Datasheet, version 1.8. Accessed: Oct. 4, 2026. [Online]. Available: https://documentation.espressif.com/ esp32-s3-wroom-1_wroom-1u_datasheet_en.pdf [24] Arduino, Arduino Nano 33 BLE Sense Rev2 Datasheet, product ABX00069. Accessed: Oct. 4, 2026. [Online]. Available: https: //docs.arduino.cc/resources/datasheets/ABX00069-datasheet.pdf [25] Raspberry Pi, Rpi Pico 2 W Datasheet, 2026. [Online]. Available: https://datasheets.raspberrypi.com/picow/pico-2-w-datasheet.pdf [26] STMicroelectronics, “B-U585I-IOT02A Discovery Kit: Product Specifications,” accessed: Oct. 4, 2026. [Online]. Available: https://www.st.com/en/evaluation-tools/b-u585i-iot02a.html [27] Espressif Systems, “Arduino-ESP32 Documentation: Partition Table,” accessed: Oct. 4, 2026. [Online]. Available: https://docs.espressif.com/ projects/arduino-esp32/en/latest/tutorials/partition_table.html [28] Raspberry Pi, “Pico Examples: Wi-Fi OTA Update Partition Layout,” accessed: Oct. 4, 2026. [Online]. Available: https://github.com/raspberrypi/ pico-examples/blob/master/pico_w/wifi/ota_update/main.json [29] STMicroelectronics, Getting started with STM32CubeU5 TFM application, 2023, uM2851, Rev. 4. [30] MCUboot Project, “MCUboot Design,” accessed: Oct. 4, 2026. [Online]. Available: https://docs.mcuboot.com/design.html [31] Espressif Systems, “ESP Delta OTA,” version 1.1.4. Accessed: Oct. 4, 2026. [Online]. Available: https://components.espressif.com/ components/espressif/esp_delta_ota/versions/1.1.4/readme [32] TensorFlow, “Memory management in TensorFlow Lite Micro,” accessed: Oct. 7, 2026. [Online]. Available: https://github.com/tensorflow/tflite-micro/blob/main/tensorflow/ lite/micro/docs/memory_management.md

Related documents

• Hardware-assisted Machine Learning in Resource-constrained IoT Environments for Security: Review and Future Prospective#133994
• Smart Adaptive Computing Across the Continuum: LLMs in IoT-Edge-Cloud Resource Management#987105
• Artificial Intelligence and Machine Learning for EDGE Computing#10787
• Availability-Aware and Efficiency-Driven AI Service Chain Provisioning in Multi-Domain Edge Intelligence Cloud#259396
• EdgeCraft: Automated Model Crafting for Edge IoT#1129764