arXiv CSopen access

EdgeCoInfer: Hierarchical Collaborative Inference for On-Device Multimodal Large Models

EdgeCoInfer: Hierarchical Collaborative Inference for On-Device Multimodal Large Models Lin Tan†‡ , David K. Y. Yau† , Songtao Guo‡ , † Singapore University of Technology and Design

arXiv:2607.17143v1 [cs.DC] 19 Jul 2026

‡ Chongqing University

Abstract—Modern mobile applications predominantly execute concurrent Multimodal Large Language Models (MLLMs) to provide ubiquitous intelligence. However, satisfying this demand within edge environments faces significant challenges due to multi-task concurrency and strictly coupled hard constraints. To address these issues, we propose EdgeCoInfer, a framework enabling granularity-adaptive deployment by co-optimizing intermodel functional module sharing and intra-model fine-grained partitioning. We solve the underlying Mixed-Integer Non-Linear Programming (MINLP) problem via a Hybrid Evolutionary Hierarchical Reinforcement Learning (HE-HRL) paradigm, which synchronizes a Genetic Algorithm (GA) for discrete model placement with a Soft Actor-Critic (SAC) agent for continuous resource allocation. To navigate the sparse feasible region, we introduce a feasibility-guided constructive execution mechanism, integrating a constructive cut-step decoder with pre-act pruning and a two-phase curriculum strategy for stable adaptation. Experimental results demonstrate that EdgeCoInfer ensures a 100% task completion rate in high-concurrency scenarios, achieving a 76% reduction in system cost and 71.88% memory savings compared to state-of-the-art baselines. Index Terms—Edge Artificial Intelligence, Edge Computing, Multimodal Large Language Models, Hierarchical Collaborative Inference

I. I NTRODUCTION A. Background and Challenges The surge of large language models (LLMs) and multimodal models marks a major milestone in artificial intelligence (AI) [1]. At present, these models are predominantly deployed in centralized cloud infrastructures, a paradigm that introduces several challenges, including excessive end-to-end latency, high bandwidth costs, and heightened data-privacy risks [2]. A recent shift seeks to place models directly on edge devices (EDs), enabling computation closer to where data are generated [3]. By reducing reliance on remote transmission, this approach can substantially decrease communication overhead and improve responsiveness. However, the practical deployment of large-scale on-device models faces two fundamental tensions. First, many state-ofthe-art AI models are extremely large, with parameter counts that far exceed the memory capacity of any single ED, making local loading and execution infeasible [4]. Second, modern EDs are often required to run multiple AI applications concurrently (e.g., voice assistants, real-time object detection, and AIenhanced photography). Naively loading a full model for each application can quickly exhaust scarce memory resources and may even compromise system stability [5]. These conflicting

requirements necessitate a shift from standalone execution to collaborative inference paradigms, where resources across multiple devices are orchestrated to support heavy-duty multitask workloads. B. Related Work and Limitations Existing research on distributed edge collaborative inference primarily evolves along two directions. The first is Intra-Model Partitioning, which involves partitioning a single large model into multiple partitions distributed across devices to accelerate execution speed. The second is Inter-Model Sharing, which involves identifying and reusing common model modules among different tasks to conserve memory. We discuss the limitations of each approach below. 1) Intra-Model Partitioning: This approach focuses on a single, monolithic model. It involves finely partitioning the model, often layer-by-layer, and distributing these partitions across multiple collaborative devices. Typical approaches include Deep Neural Network (DNN) [6], [7] partitioning, and Transformers/Large Language Model (LLM) partitioning methods [8]–[10]. The goal is to optimize performance metrics including latency and throughput for a single task. A prime example is [10], which uses dynamic programming to find an optimal partition and deployment strategy for LLMs on heterogeneous EDs. Approaches of this type are essentially optimized for single-model single-task scenarios. In a realistic edge environment requiring multiple AI capabilities, applying such approaches would necessitate deploying separate, fully partitioned models for each task, leading to massive resource redundancy and inefficiency. They overlook the substantial redundancy that may exist among multiple models in concurrent multi-task environments. Furthermore, its fine-grained partitioning may create significant communication overhead, which may become a bottleneck. 2) Inter-Model Sharing: This approach targets multi-task scenarios. It decomposes different models into their constituent functional modules (e.g., vision encoder, text encoder, taskspecific head) and shares common modules across various tasks [11]. For example, different multimodal tasks such as Visual Question Answering (VQA) and image-text retrieval can reuse the same vision and text encoders to reduce memory and computation [12]. In contrast, task-specific heads, such as the VQA classifier or retrieval scoring head, contain large, task-unique parameters. This strategy significantly reduces the total memory footprint. Recent work [13] exemplifies

this by proposing a split-and-share architecture to efficiently serve multiple multimodal tasks on the edge. However, approaches of this type fail when a single module (e.g., a 7B-parameter LLM head) exceeds the memory capacity of any EDs. Although intra-module partitioning is theoretically compatible, no concrete mechanisms exist to manage the resulting scheduling, resource allocation, or optimization challenges. They cannot adapt to runtime variability, exploit intertask redundancies, or balance multi-task resource demands efficiently. Consequently, the existing strategy is insufficient in complex hybrid edge environments. In summary, real-world user requirements are inherently hybrid and complex, and edge systems must support multiple concurrent AI applications. Therefore, we propose Edge Collaborative Inference (EdgeCoInfer). C. Research Challenges The proposed EdgeCoInfer aims to build an efficient hierarchical collaborative inference system for MLLMs. By combining coarse-grained module decomposition and sharing with fine-grained model partitioning for distributed inference, EdgeCoInfer enables practical deployment of MLLMs on EDs. At the core of EdgeCoInfer is an innovative central brain that performs unified, adaptive, and near-optimal decisionmaking across collaboration, resource allocation, and scheduling. This design is directly motivated by the need for coordinated control in multi-device, multi-application edge environments. However, building such an intelligent decision-making system requires overcoming three significant and intrinsically interdependent challenges. 1) Trade-off between Resource Efficiency and Performance: The heterogeneity of edge capabilities creates a fundamental tension between inter-model sharing and intramodel partitioning. While sharing functional modules minimizes memory footprint, it may introduce computation bottlenecks. Conversely, partitioning accelerates inference but incurs communication overheads. Navigating this multiobjective trade-off to optimize placement is non-trivial [3]. 2) Hierarchical Resource Contention: Concurrent multitasking induces novel multi-granularity conflicts [14]. A single device’s resources might be simultaneously requested as a standalone host for a small, complete module (a coarsegrained request) and as a worker node within a distributed cluster for a fraction of a large, partitioned model (a finegrained request). Managing and prioritizing these multigranularity resource requests to avoid scheduling conflicts and ensure QoS for all tasks introduce a highly complex scheduling challenge. 3) Intractability of the Solution Space: The joint optimization of module sharing, partitioning, and resource allocation across heterogeneous nodes leads to a combinatorial explosion [15]. The search space scales exponentially with the number of devices and model layers. The core difficulty lies in making rapid, near-optimal scheduling decisions without exhaustively evaluating every possible configuration.

D. Contributions To address these multifaceted challenges, we propose EdgeCoInfer, a novel hierarchical collaborative inference framework for on-device MLLMs. The primary contributions of this paper can be summarized as follows. • We propose a novel hierarchical collaborative inference architecture that enables granularity-adaptive deployment. Uniquely, EdgeCoInfer supports both inter-model functional module sharing and intra-model fine-grained partitioning. This dual-granularity design breaks the memory wall of single devices, allowing complex multimodal workflows to run efficiently on EDs. • We propose a Hybrid Evolutionary Hierarchical Reinforcement Learning (HE-HRL) framework,, which introduces a methodological shift based on hierarchical decomposition and feasibility-guided learning. Specifically, we employ a genetic algorithm to explore the discrete model placement space and a soft actor-critic Soft Actor-Critic (SAC) agent to optimize continuous resource allocation in two stages. • We design a two-stage constraint-aware execution mechanism to overcome the sparsity of valid solutions. In Stage 1, we implement a constructive cut-step decoder with preact pruning to guarantee model placement validity. In Stage 2, we introduce a two-phase strategy, transitioning from a deterministic feasibility-preserving controller to a learned differentiable policy, which stabilizes training warm-up and enables fine-grained performance refinement. • We conduct extensive simulations on an edge cluster running concurrent multimodal tasks. Experimental results demonstrate that EdgeCoInfer significantly outperforms state-ofthe-art baselines, achieving a 76% reduction in system cost and 71.88% memory savings. The remainder of the paper is organized as follows. We present the system model and the problem formulated in Section II. Section III describes the details of our proposed optimization framework. The experimental results are analyzed in Section IV. Finally, Section V concludes the paper. II. S YSTEM M ODEL AND P ROBLEM F ORMULATION In this section, we present the execution flow and system model of EdgeCoInfer. Then, we analyze the memory, energy consumption, and time required for the multimodal model, and then formulate the optimization problem. A. System Model 1) System Overview: As illustrated in Fig. 1, the proposed EdgeCoInfer framework decomposes complex multimodal models into a cohesive sequence of functional modules. This abstraction explicitly categorizes components into standard modules (e.g., vision encoders), which typically execute a single forward pass, and generative modules (e.g., LLM heads), which involve computationally intensive autoregressive decoding. Crucially, each functional module is treated as a shareable and partitionable entity, enabling flexible distributed inference across heterogeneous EDs.

-

Profiling

Three Layer Architecture

(Input Arrival) Layer 1: Task & Model Layer (Logical View)

Image

(VQA)

Input

Decision

(Partition & Share)

Task B (Image Captioning) Iterative

DAG

Task A

+ Closed-Loop Flow DAG

Decoding

Vision

LLM

Encoder

Head

Vision

Classifier

Encoder Output

Task B

Image

Output

Input

Sharing Candidate

C Server / Central Brain

Layer 2: Decision Layer

ME

Decision Maker (Core)

Profiler (Analysis Engine)

Partitioned

Inference

(Execution & Aggregation)

LL FLOPs

Share

(

Allocator

Resource Mapping)

Instruction Dispatching

LM

Memory

Layer 3: Physical Infrastructure Layer Memory Budget

Input

Static

Weights

Shared

KV-Cache

∞

Shared

Shared

r

Vision Encode

1

1-10)

Intermediate

(Layer

Activation

Transmission

Mo

bile

Laptop

∞ Jetson

2

11-32)

LLM Part

LLM Part (Layer

∞ MEC

Server

Fig. 1. Overview of the EdgeCoInfer framework. The system features a three-layer architecture: (1) The Task and Model Layer abstracts concurrent multimodal tasks into DAGs comprising shareable standard modules and generative modules; (2) The Decision Layer acts as the central brain, integrating profiling, optimization (partitioning and sharing), and resource allocation and mapping; (3) The Physical Infrastructure Layer executes distributed inference on heterogeneous EDs. The solid arrows illustrate the continuous closed-loop flow from task arrival to result aggregation.

The EdgeCoInfer framework features a three-layer architecture governed by a continuous closed-loop flow. The process commences at the Task and Model Layer, where concurrent multimodal tasks are abstracted into Directed Acyclic Graphs (DAGs) comprising potentially shareable standard and generative modules. These logical requirements are transmitted to the Decision Layer, where a profiler first characterizes the input workload in terms of FLOPs and memory usage. Based on this profiling, a decision engine solves a joint optimization problem to determine optimal partitioning and sharing strategies, balancing inter-model redundancy against intra-model computational loads. Subsequently, the Allocator enforces the resource distribution strategy. Instructions are then dispatched to the Physical Infrastructure Layer, where EDs (e.g., mobiles, laptops, and Mobile Edge Computing (MEC) servers) execute the assigned workloads within strict memory constraints [5]. The cycle concludes with inference aggregation, as intermediate activation tensors are transmitted across the network and final results are consolidated back to the source. 2) System Components and Notation: The system comprises one MEC server and N EDs. We denote N = {0, 1, 2, . . . , n, . . . , N } as the set of compute nodes, where 0 represents the MEC server. a) Device Resource: Each node n ∈ N is characterized by the maximum operating frequency of its on-chip accelerator GPU or Neural Processing Unit (NPU), denoted as fnmax (Hz), and its memory budget Mnmax . The memory constraint is paramount for large models, limiting the maximum number of parameters and KV-cache blocks a device can host. Moreover, we consider EDs with System-on-Chip architectures, where the CPU and on-chip accelerators GPU or NPU access a unified main memory. Therefore, we do not explicitly model hostto-device data transfer overhead arising from PCIe transfers that are typical in discrete server GPUs. The remaining onchip memory effects, including bandwidth contention and syn-

chronization overhead, are implicitly captured by the effective compute parameters in our execution-time and energy models. b) Inference Task: We denote Q = {1, . . . , Q} as the set of concurrent multimodal inference tasks. Let K denote the universal set of all unique functional module types. Distinct from traditional linear models [6], each task q ∈ Q is modeled as a Directed Acyclic Graph (DAG) Gq = (Kq , Eq ). • Functional Modules (Kq ): The nodes Kq ⊆ K in the DAG represent the set of functional modules required by task q. If multiple tasks map to the same module type, they can potentially share the same static weight instance in memory. To further capture the distinct execution patterns of multimodal models, we partition the module set Kq into two disjoint subsets: Kq = Kqstd ∪ Kqgen , with Kqstd ∩ Kqgen = ∅. Kqstd represents modules that execute a single forward pass per inference request (e.g., Convolutional Neural Network (CNN) backbones, ViT encoders, or simple classifier heads). Kqgen represents modules involved in auto-regressive generation (e.g., LLM decoders). These require a prefill phase followed by iterative decoding. • Dependencies (Eq ): The edges represent the data flow dependencies. An edge (u, v) ∈ Eq indicates that the output activations of module u are required as input for module v. c) Fine-grained Layer Partitioning: To support intramodule partitioning, each module k ∈ Kq is further partitioned into a sequence of layers Lk = {1, . . . , Lk }. For a specific layer l of module k, we define: • wk,l : The computational workload. • δk,l : The size of the output activation tensor to be transmitted if a cut occurs after this layer. stat • µk,l : The static memory requirement for model weights. dyn • µq,k,l : The dynamic memory requirement (e.g., KV-cache), which may vary by task q (e.g., depending on sequence length). d) Decision Variables: The decision engine of EdgeCoInfer involves determining the physical placement of

these logical layers. We define the binary decision variable xq,k,l,n = 1 indicating whether layer l of module k in task q is assigned to node n. Additionally, we introduce a placement and sharing indicator σk,l,n ∈ {0, 1} for each unique module. If σk,l,n = 1, the static weights of layer l of module type k are loaded into the memory of node n, allowing it to serve multiple tasks simultaneously without duplicating memory usage. B. Computational Model The computational process involves executing the specific layers of functional modules on edge nodes. We first quantify the workload based on the module type and then formulate the processing latency. 1) Workload Quantification: The computational workload wk,l depends on the specific architecture of layer l in module k. We categorize modules into Standard Functional Modules and Generative Functional Modules. a) Standard Functional Modules (e.g., Vision Encoders): • CNN-based Layers: For a convolutional layer with kernel size Kl , input channels Clin , and output channels Clout . We denote the output feature-map resolution of layer l as Hlout × Wlout . The workload is determined by the output feature map size [16]: cnn wk,l = 2Hlout Wlout Clin Clout Kl2 , (1) • ViT-based Layers: For Vision Transformers, the sequence length is sk,l = Hlimg · Wlimg /ρ2 , where Hlimg × Wlimg is the input image resolution and ρ is the patch size. The workload for a self-attention block is [17]: vit wk,l = 24sk,l h2 + 4sk,l 2 h, (2) where h is the hidden dimension. It is worth noting that our framework can be readily extended to support other neural architectures beyond CNNs and ViTs by simply integrating their specific computational and memory profiling functions. b) Generative Functional Modules (e.g., LLM Decoders): For generative modules, the workload varies dynamically with the sequence length. We distinguish between the prefill and decode phases. in • Prefill Phase: The entire prompt of sequence length sk,l is processed in parallel. The workload exhibits quadratic complexity due to the full self-attention mechanism [18]: pre 2 in 2 wk,l = 24sin (3) k,l h + 4(sk,l ) h. • Decode Phase (with KV-Cache): The model generates tokens auto-regressively and sequence length grows dynamically. Let sout k,l denote the target or maximum number of tokens to be generated. For the i-th decoding step (1 ≤ i ≤ sout k,l ), the effective context length includes the prompt and previously generated tokens sk,l [i] = sin k,l +i−1. Leveraging the KV-cache, the attention computation is reduced to linear complexity relative to the current context length: dec wk,l (i) = 24h2 + 4sk,l [i]h. (4) 2) Computational Latency and Energy: It is critical to distinguish between memory sharing and computation sharing. While static weights can be shared across tasks by storing a single copy, computation is inherently additive. Even if a functional module k is reused by multiple tasks, the accelerator

must execute the corresponding operations separately for their distinct inputs. We first denote ψn as the effective throughput coefficient of node n in FLOPs per Hz, which captures device-level effects such as parallelism, kernel efficiency, on-chip memory, memory bandwidth limitations and bandwidth contention. Let fq,n denote the effective accelerator operating frequency allocated to task q on node n. This quantity can be interpreted as an equivalent value that jointly reflects Dynamic Voltage and Frequency Scaling (DVFS) and time-sliced scheduling on a shared accelerator. The effective compute throughput allocated to task q on node n is then ψn fq,n in FLOPs per second. Accordingly, the computation latency for executing layer l of module k for task q on node n is given by wk,l xq,k,l,n . (5) tcomp q,k,l,n = ψn fq,n We further model the computational power on on-chip accelerators using a DVFS-aware formulation consisting of a static leakage term and a frequency-dependent dynamic term. The computational power associated with task q running on node n is given by [19] α comp Pq,n = Pnsta + κn (fq,n ) , (6) where Pnsta is the static power, κn is a device-dependent coefficient, and α is a hardware-dependent exponent. Therefore, the computational energy for executing layer l of module k for task q on node as  n is obtained  wk,l xq,k,l,n comp comp comp Eq,k,l,n = Pq,n tq,k,l,n = Pnsta + κn (fq,n )α . ψn fq,n (7) C. Memory Model Memory capacity is the primary bottleneck for deploying large multimodal models on edge nodes. We model the memory footprint as the sum of static model parameters and dynamic runtime states. 1) Static Memory Modeling (Inter-Model Sharing): The static memory consumption corresponds to the storage of model weights and biases. A key feature of EdgeCoInfer is the ability to share functional modules. The total static memory footprint on node n is calculated based on the unique module layers loaded: X X Mnstat = σk,l,n µk,l , (8) k∈K l∈Lk

The parameter size µk,l is determined by the specific layer architecture and data precision b in bytes per element parameter. By adjusting b, the system can dynamically trade accuracy for memory/compute footprint, thereby adapting the runtime demand to the available device resources and maintaining deployment feasibility. • CNN-based Layers: For a convolutional layer, the parameters consist of the kernel weights and biases:
in out 2 out µcnn b. (9) k,l = Cl Cl Kl + Cl • Transformer-based Layers (ViT/LLM): For a standard Transformer block, the parameters are concentrated in the Linear projections (Q, K, V, O) and the Feed-Forward

Network (FFN). For a hidden dimension h, the parameter size is approximated as [17]: µtrans = 12h2 b. (10) k,l Note that different from computational workload, the static memory of a Transformer layer is independent of the sequence length. 2) Dynamic Memory Modeling (Intra-Module Partitioning): Dynamic memory consists of intermediate activation tensors and, crucially for LLMs, the Key-Value (KV) cache. This consumption is strictly task-specific and additive, meaning it cannot be shared across tasks. a) Activation Memory: For standard layers, node n must allocate buffer space to accommodate the peak activation tensor generated during the forward pass. Relating this to the output transmission size δk,l , the required buffer space for layer l is modeled as: µact (11) k,l = ηk,l · δk,l , where ηk,l is an architecture-dependent expansion factor, which is obtained via lightweight profiling. b) KV-Cache Memory: For generative modules, the KVcache grows linearly with the sequence length to support autoregressive decoding. The reserved KV-cache size is formulated as [20]: in out µkv (12) q,k,l = 2(sk,l + sk,l )hb. The constant 2 accounts for both Key and Value matrices. Consequently, the total dynamic memory on node n is the summation of requirements X X Xfor all assigned task layers as kv xq,k,l,n (µact (13) Mndyn = k,l + µq,k,l ). q∈Q k∈Kq l∈Lk

D. Communication Model The collaborative inference requires transmitting intermediate activation tensors between edge nodes when adjacent layers are partitioned across different devices. We first calculate the data size and then formulate the transmission latency. 1) Intermediate Data Size: a) Standard Functional Modules: The intermediate activation to be transmitted after layer l equals the output feature map for CNNs or the token sequence for ViTs, and can be expressed as cnn δk,l =Hlout Wlout Clout b (14) vit δk,l =sk,l hb. (15) b) Generative Functional Modules (LLM): For LLMs, the communication overhead differs drastically between phases due to the auto-regressive nature. • Prefill Phase: The output of layer l contains the hidden states for all tokens in the prompt. For a sequence length s, the transmission volume is pre δk,l (s) = shb. (16) • Decode Phase: In each step of token generation, only the hidden state of the newest token needs to be transmitted to the next layer (as previous keys/values are already cached at the destination). Thus, the transmission volume is minimal and constant, given by dec δk,l = 1hb. (17)

2) Transmission Latency: Data transmission occurs strictly when layer l and layer l+1 are assigned to different nodes. The total bandwidth B is divided into N orthogonal sub-channels to prevent interference. Therefore, the data transmission rate is calculated as:   gn,m Pntrans B , (18) log2 1 + Rn,m = N β2 where B is the bandwidth, Pntrans is the transmission power, gn,m represents the channel gain, and β 2 denotes the background Gaussian noise power. Therefore, the transmission latency for the output of layer l (assigned to node n) to layer l + 1 (assigned to node m) is 8δk,l xq,k,l,n xq,k,l+1,m ttrans , (19) q,k,l,n,m = Rn,m where the constant 8 converts the data size from bytes to bits. The product xq,k,l,n xq,k,l+1,m acts as a binary indicator that is 1 only if the partition cut occurs exactly between layer l and l + 1. 3) Transmission Energy: The transmission energy is determined by the transmission power Pntrans and the duration of data transfer. Based on the transmission latency ttrans q,k,l,n,m , the energy consumed by node n to transmit the output of layer l to node m is given by 8Pntrans δk,l xq,k,l,n xq,k,l+1,m trans . Eq,k,l,n,m = Pntrans ttrans q,k,l,n,m = Rn,m (20) Similar to latency, transmission energy is only incurred when adjacent layers are partitioned across different nodes (n ̸= m). E. Energy Consumption Model Energy efficiency is paramount for battery-powered EDs. The total energy consumption must account for the repeated execution of the decoding phase. For any specific layer l of module k in task q, the total processing energy is the sum of its computational energy on the assigned node and the transmission energy the subsequent X tocomp X node as follows: trans Eq,k,l = (Eq,k,l,n + Eq,k,l,n,m ) (21) n∈N

m∈N ,m̸=n

Task execution encompasses both the Standard Phase (executing non-generative modules) and the Generative Phase (comprising prefill and iterative decoding). Specifically, the total energy consumption for completing task q is the aggregation of computation and transmission energy across all involved nodesX and layers: X Eqtotal = Eq,k,l (wk,l , δk,l ) k∈Kqstd l∈Lk

|

{z

Standard Phase (Non-generative)

X

+

X

}

pre pre Eq,k,l (wk,l , δk,l )

k∈Kqgen l∈Lk

|

{z

Prefill Phase (One-time)

}

sout k,l

+

X X

X

dec dec Eq,k,l (wk,l (i), δk,l ),

i=1 k∈Kqgen l∈Lk

|

{z

Decode Phase (Iterative)

}

(22)

For standard tasks, the second and third terms naturally become zero, reducing the model to the classical DAG energy summation. F. Inference Latency Model The end-to-end inference latency is the primary metric for QoE. We formulate the total latency by aggregating the processing time across the task’s specific execution phases: the one-time encoding/prefill and the iterative decoding. For any specific layer l of module k, the processing delay Tq,k,l consists of execution and transmission time. Crucially, the values depend on X the specific phaseX given by Tq,k,l = (tcomp + ttrans (23) q,k,l,n,m ). q,k,l,n n∈N

m∈N ,m̸=n

Since task q is modeled as a DAG, standard modules may run in parallel branches. The total inference latency for completing task q is the aggregation of computation and transmission delays across all involved nodes and layers given as follows:   X Ltotal = max  Tq,k,l (wk,l , δk,l ) q p∈Pqstd

(k,l)∈p

{z

|

}

Standard Path (Non-generative)

X

+

X

pre pre Tq,k,l (wk,l , δk,l )

k∈Kqgen l∈Lk

|

{z

Prefill Phase (One-time)

}

sout k,l

+

X X

X

dec dec Tq,k,l (wk,l (i), δk,l ),

(24)

i=1 k∈Kqgen l∈Lk

|

{z

Decode Phase (Iterative)

}

where Pqstd represents the set of paths in the subgraph formed by Kqstd . G. Problem Formulation We introduce four sets of optimization variables {X, Σ, F} to orchestrate the collaborative inference. Σ = {σk,l,n } represents the module placement and sharing decisions. X = {xq,k,l,n } denotes the partitioned layer decisions. F = {fq,n } denotes the allocation of computational resources. The system objective is to minimize the total system cost, defined as the weighted sum of energy consumption and inference latency across all concurrent tasks. This objective allows the system to balance execution speed and energy efficiency based on user preferences, which can be formulated as: X P1 : min [λq Eqtotal + (1 − λq )υLtotal ] (25) q Σ,X,F

q∈Q

Subject Xto xq,k,l,n = 1, ∀q ∈ Q, k ∈ Kq , l ∈ Lk

(26)

n∈N

xq,k,l,n ≤ σk,l,n , ∀q ∈ Q, k ∈ Kq , l ∈ Lk , n ∈ N

(27)

Mnstat + Mndyn ≤ Mnmax , ∀n ∈ N X fq,n ≤ fnmax , ∀n ∈ N q∈Q

(28) (29)

Ltotal ≤ Dq , ∀q ∈ Q (30) q where λq ∈ [0, 1] is the weighting factor balancing performance, and υ is a normalization coefficient to align the scales of time and energy. Constraint (26) ensures structural integrity, guarantees that every layer l of every module k in task q is assigned to exactly one compute node. Constraint (27) indicates that if layer l of module type k is used by any task q on node n, the corresponding static weights must be loaded into memory. Constraint (28) ensures that the sum of unique static weights and additive dynamic KV-caches is constrained by the device budget Mnmax . Constraints (29) ensure that the allocated computational resources do not exceed the device capacity. Constraint (30) imposes the QoS constraint, ensuring the latency satisfies the specific deadline Dq . III. H YBRID E VOLUTIONARY H IERARCHICAL R EINFORCEMENT L EARNING F RAMEWORK A. Complexity Analysis and Methodological Rationale The formulated Problem P1 constitutes a Mixed-Integer Non-Linear Programming (MINLP) challenge, which is NPhard due to the combinatorial explosion of the joint discretecontinuous decision space and the severely restricted feasible region caused by coupled hard constraints on memory and latency. Conventional heuristics often is trapped in local optima, while standard DRL approaches struggle to converge due to the high dimensionality and sparsity of valid solutions. To tackle this intractability, we propose the HE-HRL framework, which introduces a methodological shift based on hierarchical decomposition and feasibility-guided learning. As shown in Fig. 2, the architecture consists of four synergistic components where the Perception module extracts spatio-temporal features and the Decision module generates hierarchical policies. Subsequently, the Execution module ensures constraint satisfaction through pruning and repair, while the Learning module optimizes the policy via a hybrid evolutionary-gradient paradigm. Further details are provided below. B. Spatio-Temporal State Encoder The Perception module constructs a comprehensive latent representation s from the system state. We design a hybrid encoder to capture three distinct views of the edge environment. 1) Spatial State via GCN: To capture the topologydependent resource heterogeneity, we represent the edge network infrastructure as a graph G = (N , E), where N is the set of computing nodes and E represents network links between nodes. Each node n possesses a feature vector yn , including memory budget, peak frequency, and current load, which can be expressed as: X yn = [Mnmax , fnmax , ψn , Mn , fq,n ]T . (31) q We employ a Graph Convolutional Network (GCN) [21] to aggregate neighborhood information via the adjacency matrix A:   1 1 (32) zGCN = σ D̂− 2 ÂD̂− 2 YWG ,

Perception Layer （State Encoding） GC

N

（Z）

Stage 1

Feature

Latent

Placement

Fusion

State

s

Policy Head

(Device & Task DAGs)

TC

N

（Z）

r

Masked

Sampling

Heuristic

Repair / Constraint

Check

Feasible

→ Final

Stage

MLP

2

x

Placement

Softma

Resource

Encode History State

Decoding & Execution Layer (Constraint-Aware)

chical Policy Network (Decision)

Infeasible → Migrate

r

Encode Graph State

e

Hi rar

Allocation Head

j

Pro ection

Resource Allocation

X,Σ

Physical

Metrics:

Environment

Latency,

System Cost

Simulation

Energy,

Reward R

(Evaluation)

Violations

F

(Workload Seq) Evolution Update

(Selection & Mutation)

Population Memory

Gradient Update

(SAC, Replay Buffer)

Hybrid Evolutionary Training Loop (Learning)

Fig. 2. Architecture of the proposed HE-HRL framework. It integrates a spatio-temporal state encoder, a two-stage hierarchical policy for joint partitioning and resource allocation, and a constraint-aware execution mechanism. The policy is trained via a hybrid paradigm alternating between evolutionary strategies and gradient descent SAC to handle the coupled discrete-continuous action space.

where Y is the node feature matrix obtained by stacking all yn , Â = A+I incorporates self-loops, D̂ is the degree matrix of Â, and WG denotes the learnable weight matrix. 2) Temporal State via TCN: Since edge dynamics exhibit strong temporal correlations, we encode the sliding window history Ht = [Yt−K , . . . , Yt ] using a Temporal Convolutional Network (TCN) [22], where K is the length of the historical window. The TCN employs dilated causal convolutions to expand the receptive field, extracting a trend embedding zT CN . 3) Sequential Context Embedding: Since the placement strategy adopts a constructive Cut-Step mechanism (which is detailed in the next section), the decision process is inherently sequential. To ensure the state representation satisfies the Markov property, the agent requires explicit awareness of the current position within the task DAG Gq . We encode the decomposition status as a progress vector pt as follows: pt = [k, l, Lpart , Lrem ]T , (33) where k denotes the topological index of the current functional module, l indicates the starting layer of the current partition, Lpart is the length of the current partition, and Lrem represents the number of remaining layers to be assigned. This vector is projected into a dense embedding zprog via an a Multi-Layer Perceptron (MLP), providing the necessary temporal context for the sequential decision-making. 4) Feature Fusion: The final latent state s is obtained by concatenating and projecting these embeddings: s = MLP(zGCN || zTCN || zprog ). (34) C. Hierarchical Policy Network The decision module serves as the core mapping engine, transforming the latent state s into the complete set of optimization variables {X, Σ, F} defined in Problem P1. We employ a two-headed neural architecture πθ (a|s) to decouple the discrete placement decisions from the continuous resource allocation. 1) Head 1: Placement Policy: The high-level policy π1 (·|s) constructs the binary placement variables X via a sequential Cut-Step mechanism. Instead of outputting the highdimensional placement matrix directly, it iteratively generates partitions. At each step, the policy network produces two sets of unnormalized scores, denoted as logits L. First is Cut Logits Lcut ∈ RLrem , which is a vector of scores over the layer slots. These logits determine the partition length ℓ (i.e., how many consecutive layers to group) by selecting the cut position with

the highest score. Second is Node Logits Lnode ∈ RN . A vector of scores over the N available computing nodes. These logits identify the destination node n for the current partition via a sampling operation. In addition, this head determines the module sharing variables Σ toW implement memory deduplication. We calculate σk,l,n = q∈Q xq,k,l,n , ensuring that the static weights of module k are instantiated only once onWnode n, even when requested by multiple concurrent tasks. denotes the logical OR operation. This logical aggregation effectively minimizes memory redundancy, allowing the agent to satisfy strict resource constraints. 2) Head 2: Resource Allocation Policy: The low-level policy π2 (·|s) governs the allocation of resources to satisfy the computation and communication demands established by Stage 1. To facilitate stable training under strict constraints, we design a deterministic controller to provide feasible feedback for the high-level placement policy in the early training stage. Once Stage 1 stabilizes, we activate the neural network to perform fine-grained optimization. Specifically, regarding computational resources (F), the policy outputs a continuous score matrix Lfreq ∈ RN ×Q , where N is the number of nodes and Q is the number of concurrent tasks. D. Constraint-Aware Action Decoding Directly sampling from the policy network often leads to invalid actions in constrained edge environments [15]. This module implements the Feasibility-Guided Decoding mechanism. 1) Pruning and Repair for Stage 1: We treat the placement generation as a constructive process. To ensure feasibility, we integrate model-based checks directly into the decoding loop. First is the Pre-Act Pruning (Action Masking). Before committing a candidate partition of length ℓ to node n, we compute a validity mask M(ℓ, n). We define the memory feasibility condition as: Mn + ∆M (ℓ) ≤ Mnmax , (35) where Mn denotes the current memory occupancy of node n, calculated as the sum of all layers previously assigned to this node. ∆M (ℓ) represents the incremental memory cost of the new candidate partition. Moreover, the estimated cumulative latency must not exceed the task deadline Dq . Tqela + Tqest (ℓ, n) ≤ Dq , (36)

where Tqela denotes the elapsed time, and Tqest (ℓ, n) represents the estimated computation and transmission latency to node n, derived from a computable approximation bound. The mask M(ℓ, n) is set to −∞ if this condition is violated or if the constraint bound is breached. The policy distribution is then re-normalized: P (ℓ, n|s) = Softmax ((Lcut ⊕ Lnode ) + M) , (37) where ⊕ denotes the broadcasting operation, resulting in a matrix of size S × N . Second is the Post-Act Repair. If a complete placement X still violates deadlines or causes Out-Of-Memory errors during iteration, a heuristic repair operator is triggered. It iteratively migrates bottleneck partitions to the MEC server or nodes with maximal resource slack. 2) Deterministic Controller and Differentiable Projection for Stage 2: To strictly satisfy the capacity constraint, we employ a two-phase curriculum strategy that transitions from a heuristic controller to a differentiable projection. During the initial training phase, we employ a feasibilitypreserving heuristic to compute F directly from (X, Σ). We adopt a Deadline-Aware Load Balancing rule. The frequency fq,n is allocated proportional to the urgency given as wq,n aq,n . (38) , aq,n = fq,n = fnmax · PQ Dq a i=1 i,n where wq,n denotes the workload of task q placed on node n. In the refinement phase, the policy outputs frequency logits which are decoded via masked softmax projections (f ) Let ln,q denote the unnormalized logit for task q on node n. We apply a standard softmax projection to determine the operating frequency fq,n : (f ) eln,q fq,n = fnmax · P . (39) (f ) Q ln,i e i=1 This formulation allocates the node’s total computing capacity fnmax proportionally among the assigned tasks based on their learned scores, ensuring that the capacity constraint is strictly satisfied. Consequently, we obtain the complete set of decision variables (X, Σ, F), from which the total system cost is derived. The final reward R is defined as the negative of the total system cost. E. Hybrid Evolutionary Training Since the decision variables contain both discrete placement and continuous resource allocation, the overall optimization problem is non-differentiable. We decouple the training into two synchronized paradigms. 1) Evolutionary Path (Stage 1 Optimization): The highlevel policy πθ1 generates placement logits, which are mapped to discrete actions (X, Σ) via the masking and sampling process. Since gradients cannot backpropagate through these discrete and heuristic operations, we employ a gradient-free GA. We maintain a population of P actors P = {θ11 , . . . , θ1P }. In each generation g, the fitness of actor i is evaluated as its average episodic return J(θ1i ) = E[R].

For selection operator in GA, we rank actors by fitness and select the top-k elites, denoted as Pelite ⊂ P. For mutation operator in GA we use Parameter Space Noise. To explore the combinatorial solution space, we generate an offspring parameter θ̃1 for each elite θ1 ∈ Pelite by injecting random perturbations: θ̃1 = θ1 + ϵ, ϵ ∼ N (0, ϑ2 I), (40) where ϵ is the Gaussian noise vector, ϑ is the mutation strength, and I is the identity matrix. This zero-order optimization allows the agent to escape local optima in the rugged discrete landscape. 2) Gradient Path (Stage 2 Optimization): The low-level policy πθ2 generates the resource allocation. We train θ2 via SAC [23] to maximize " T the entropy-regularized objective: # X J(θ2 ) = Eτ ∼πθ2 γ t (r(st , at ) + ςH(πθ2 (·|st ))) , (41) t=0

where τ = {(st , at )}Tt=0 denotes the trajectory induced by the policy over a finite horizon T , γ is the discount factor, and ς determines the relative importance of the entropy term H(·). To optimize this objective, we use the transitions (s, a, r, s′ ) stored in a replay buffer B. Therefore, the SAC optimization alternates between evaluating the policy via soft Q-value estimation (Critic) and improving the policy via gradient ascent (Actor). We introduce this in detail below. First is the Critic Update. We maintain two Q-functions parameterized by ϕa,b to mitigate overestimation. The parameters are updated by minimizing the Bellman   residual: LQ (ϕi ) = EB (Qϕi (s, a) − y)2 , for i ∈ {a, b}, (42) where the target y is calculated using the target networks ϕ̄ and θ̄2 : (43) y = r + γ min Qϕ̄j (s′ , a′ ) − α log πθ̄2 (a′ |s′ ). j=a,b

Second is the Actor Update. The policy parameters θ2 are updated by minimizing the Kullback-Leibler (KL) divergence. Moreover, the reparameterization trick a = fθ2 (s, ξ), ξ ∼ N (0, I) is employed to allow gradients to propagate through the stochastic sampling process. Thus, the loss function can be expressed as:  Lπ (θ2 ) = Es∼B,ξ∼N α log πθ2 (fθ2 (s, ξ)|s)  − min Qϕj (s, fθ2 (s, ξ)) .

(44)

j=a,b

Once trained and deployed, the learned policy functions as a lightweight, real-time decision engine. In contrast to traditional iterative solvers that suffer from high computational latency, our approach directly maps real-time graph states to near-optimal partitioning and sharing decisions in short time. Crucially, thanks to the GCN-based structural perception, the algorithm generalizes well to unseen task topologies and fluctuating network conditions, enabling edge clusters to dynamically adapt to different workloads to achieve on-device MLLMs inference. F. Computational Complexity Analysis To theoretically validate the real-time capability of EdgeCoInfer, we analyze the computational complexity of the

TABLE I. M ODULE S PECIFICATIONS Functional Module

Model

Params

Precision

Audio Encoder LLM Decoder Text Encoder Visual Encoder Classification Head Alignment Head

ViT-B [25] Vicuna-7B [26] OpenCLIP TRF [27] ViT-L/14 [28] Linear Classifier Cosine Similarity

86 M 7B 120 M 304 M 2M –

FP16 FP16 FP16 FP16 FP16 FP16

TABLE II. TASK T YPES AND M ODULE C OMPOSITIONS Task Type

Module Compositions

Image Classification ViT-L/14 → Linear Classifier Visual Question Answering ViT-L/14 → Vicuna-7B Image Captioning ViT-L/14 → Vicuna-7B Automatic Speech Recognition (ASR) ViT-B → Vicuna-7B Audio-Visual Alignment ViT-L/14 + ViT-B → Cosine Similarity Image-Text Retrieval ViT-L/14 + OpenCLIP TRF → Cosine Similarity

inference phase, which comprises state encoding, policy generation, and action decoding. Specifically, the perception layer processes graph and temporal states. For the GCN encoder with Lg layers, the complexity is O(Lg (|E|d+|N |d2 )), where |N | and |E| denote the number of nodes and links, and d is the feature dimension. The TCN encoder, operating on a history window K, incurs O(|N | · K · d2 ), meaning the perception complexity scales linearly with the network size |N | and |E|. Subsequently, regarding policy generation, the decision module employs MLPs with hidden dimension H. The Stage-1 placement head generates logits with a complexity of O(H 2 + H(Lrem + |N |)), while the Stage-2 resource head incurs O(H 2 + H|N |Q). Since H, Lrem , and Q are fixed hyperparameters during inference, the policy generation operates in O(1) relative to optimization iterations, scaling linearly with system size. Furthermore, the Cut-Step decoder sequentially determines partition points. For a task with L layers, it performs at most L steps, where feasibility masking and sampling require traversing candidate nodes, taking O(L|N |). In summary, the total inference complexity per decision step is dominated by graph operations, approximating O(|E| + |N |(d2 + L)). This polynomial complexity ensures millisecond-level responsiveness, contrasting sharply with the exponential complexity of combinatorial solvers. IV. E XPERIMENTAL S TUDY 1) Experimental Settings: Our simulation constructs a realworld edge computing environment consisting of varying numbers of nodes N = 5, 10, . . . , 50, utilizing empirical profiling data from multimodal models. Task generation follows a Poisson process. The detailed specifications of all functional modules are provided in Table I. We evaluate our system on six types of multimodal AI task as shown in Table II, which summarizes the module compositions for each task type. The network architectures and algorithmic hyperparameters are primarily adapted from [21], [23], [24]. The MEC server is configured based on NVIDIA Jetson AGX Orin specifications [29], operating at a maximum frequency of fnmax = 1.3 GHz with a memory budget of Mnmax = 16 GB and peak computational throughput of ψn fnmax = 10.6 TFLOPS. EDs have heterogeneous configurations: maximum frequencies are uniformly sampled from [0.3, 1.2] GHz, memory budgets are randomly selected from

{4, 6, 8} GB, and peak throughput is set to 2.0 TFLOPS. For power consumption, the MEC server has a static power Pnsta = 15 W and transmission power Pntrans = 2 W, while EDs have static power uniformly sampled from [0.8, 3.0] W and transmission power uniformly sampled from [0.8, 2.0] W. The dynamic voltage and frequency scaling (DVFS) parameters are uniform across all devices: κn = 10−27 and α = 3.0 for the power model Pdyn = κn f α . All neural network modules use FP16 precision with b = 2 bytes per parameter. Task deadlines are Dq = 3.0 s for standard tasks and Dq = 8.0 s for tasks involving LLM modules (VQA, image captioning, and ASR). The energy weight in the system cost objective is set to λq = 0.6, the latency normalization factor υ = 1.0. The activation expansion coefficient is uniformly set to ηk,l = 2.0 for all layers. The communication bandwidth is B = 30 MHz, and the noise power is β 2 = −113 dBm. Our Stage1 hierarchical policy network uses an embedding dimension of 128, with a GCN encoder with hidden dimension 128 and a TCN encoder with 3 levels, kernel size 3, hidden dimension 128. The history window length is K = 8. For the ERL framework, the population size is P = 16 for the Stage1 evolutionary path, with elite number k = 4 and mutation strength ϑ = 0.02. The replay buffer capacity is 200,000 transitions, with batch size 128 and warmup steps 128. We perform 16 Stage2 RL updates per iteration. For Stage2 RL, the learning rate is 3 × 10−4 , the soft update coefficient is τ = 0.005, and the entropy coefficient is ς = 0.2. The horizon length T is set to 1 for single-step decision making. The Actor and Critic networks both use hidden dimension 256. The effectiveness of EdgeCoInfer is evaluated using the metrics of system cost and inference task completion rate. We do not include accuracy as a comparative metric in this study, because we assume that partitioning does not affect inference accuracy: intermediate activation tensors are transmitted losslessly, preserving the original results. Investigating the accuracy loss is left for future work. 2) Comparison Methods: We compare our proposed method against the following methods: • Baselines: Full Local, all inference tasks are performed locally; Full MEC, all tasks attempt to be offloaded to MEC; Greedy, where all tasks offloads to the node with the most abundant resources. • RL Comparisons: TD3 [24], a twin-delayed deep deterministic policy gradient algorithm for continuous control; PPO [30], a proximal policy optimization algorithm with clipped policy updates. All methods use the same parameter settings as ours. • Ablations: DP Placement, where using dynamic programming decoding replaces cut step in Stage1; No Pruning, which disables Stage1 feasibility-guided pruning; No Controller, which disables Stage2 load-proportional controller. 3) Convergence Analysis: Figure 3 presents training convergence comparison, showing that HE-HRL achieves the lowest system cost with rapid and stable convergence. For instance, HE-HRL achieves a 76% cost reduction at N = 10.

N=15

System Cost

N=10

76%

40

N=20

N=25

150

300

100

200

50

100

100

25

20

50

6 0

50

0

100 150 200 250 300

50

PPO

HE-HRL

TD3

50

0

100 150 200 250 300

DP Placement

N=35

N=30 System Cost

0

100 150 200 250 300

No Pruning

300

300

300

200

200

200

200

100

100

100

100

50

0

100 150 200 250 300

50

Train epoch

0

100 150 200 250 300

50

Train epoch

100 150 200 250 300

N=45

Optimal Initialization N=40

300

0

50

No Controller

stable

0

100 150 200 250 300

50

Train epoch

100 150 200 250 300

Train epoch

Fig. 3. Convergence performance comparison. 1200

800

No Controller

PPO

Full Local

TD3

Full MEC

DP Placement

Greedy

0.8

Completion Rate

System Cost

1000

1 HE-HRL

100 80

No Pruning

60

600

40

400

35

40

45

0.4

0.2

200 0 10

0.6

15

20

25

30

35

Number of Nodes

40

45

50

Fig. 4. System Cost performance comparison.

PPO and TD3 exhibit oscillatory curves with cost spikes, as uniform exploration wastes budget on infeasible configurations violating constraints. No Pruning consistently shows the highest costs and very slow convergence, particularly at larger N , with costs starting around 300 and declining only gradually—this unequivocally demonstrates that feasibility-guided pruning is essential for efficient search space exploration. DP Placement shows gradual cost reduction but plateaus at much higher levels, indicating that DP alone lacks adaptive optimization capabilities and cannot dynamically adjust to changing system loads. No Controller performs closer to PPO/TD3 but still at higher costs, showing that the load-proportional controller contributes to fine-tuning resource allocation in Stage2. 4) System Cost Analysis: Figure 4 shows that HE-HRL consistently achieves the lowest system cost across all evaluated scales, with the performance gap widening as the system scale increases. The baseline methods exhibit fundamental limitations. Full MEC incurs the highest cost due to excessive communication overhead and MEC server bottleneck; Greedy ignores global resource optimization and energylatency trade-offs; Full Local shows higher costs as local EDs lack sufficient computational capacity for LLM-based tasks. The RL algorithms PPO and TD3 achieve better performance than baselines but consistently fall short of HE-HRL, as the vast state-action space and lack of feasibility guidance

0 10

No Controller Local Computing Full MEC Greedy Offloading

HE-HRL PPO TD3 DP Placement No Pruning

15

20

25

30

35

40

45

50

Number of Nodes

Fig. 5. Task completion rate performance comparison.

make exploration challenging for general-purpose RL. The ablation studies reveal critical insights: No Pruning exhibits the highest cost, demonstrating that feasibility-guided pruning is essential for efficient search space exploration; DP Placement shows higher costs than HE-HRL, indicating that DP alone is insufficient without the integrated framework; No Controller performs competitively with PPO/TD3 but still inferior to HEHRL, showing that the Stage2 load-proportional controller contributes to fine-tuning resource allocation. 5) Task Completion Rate Analysis: Figure 5 demonstrates that HE-HRL achieves a perfect 100% completion rate across all scales, while all other methods experience significant degradation as the system scales. Both PPO and TD3 start high but decline rapidly, revealing that general-purpose RL struggles with the problem’s complexity and large state-action space. PPO outperforms TD3 owing to its on-policy learning nature and clipped objective function, which provide more stable policy updates and better sample efficiency in the discrete action space of device placement decisions. The ablation studies demonstrate each component’s necessity. 6) Memory Optimization Analysis: Table III presents a detailed analysis of memory optimization achieved through EdgeCoInfer for the N = 40 configuration. The table listed the memory requirements in real world against the actual loaded memory with HE-ERL. Firstly, the algorithm reduces

TABLE III. M EMORY O PTIMIZATION A NALYSIS : R EQUIRED VS . ACTUAL L OADED R ESOURCES (N = 40) Model Instance

Unit Mem

Requirement

EdgeCoInfer Actual Loaded

Qty Mem (GB)

Qty Partitioned Mem (GB)

Sav. (%)

ViT-L/14 576MB 27 Vicuna-7B 13GB 10 OpenCLIP TRF 162MB 6 CLAP-Audio 162MB 5 Cosine Similarity 2MB 11 Linear Classifier – 6

15.19 130.00 0.95 0.79 0.02 –

3 3 2 2 1 1

14 19 2 2 0 0

1.68 88.94 39.00 70.00 0.32 66.32 0.32 59.49 0.002 90.00 – –

Total

146.95

–

–

41.32 71.88

–

–

the total number of loaded model instances from 65 to 12 instances using sharing mechanism. Secondly, the algorithm employs strategic layer partitioning to distribute large models across multiple devices, enabling partitioned loading and further memory optimization. Notably, Vicuna-7B has 3 loaded instances and is partitioned a total of 19 times, enabling the system to load only the necessary layers for each task rather than complete models. ViT-L/14, used by all 27 tasks, reducing memory from 15.19 GB to 1.68 GB. Vicuna7B, used by 10 LLM-based tasks, achieves 70% savings, demonstrating that even large models can be efficiently shared when tasks share similar computational requirements. The smaller modules also show significant savings, though their absolute memory contributions are smaller. The total memory reduction from 146.95 GB to 41.32 GB with 71.88% savings demonstrates HE-HRL’s significant advantage in deploying large-scale multimodal models on resource-constrained EDs. 7) Computational Efficiency Analysis: Beyond optimization performance, the decision-making latency is critical for real-time edge scheduling. We evaluated the average inference time of the trained HE-HRL policy on the MEC server. The results show that our framework generates a valid joint strategy in only 15.17 ms on average. Compared to the task execution deadlines, this decision overhead is negligible. This confirms that EdgeCoInfer supports online scheduling with millisecondlevel responsiveness. V. C ONCLUSION In this paper, we have presented EdgeCoInfer, a framework that successfully orchestrates concurrent MLLMs on edge clusters. By unifying inter-model module sharing and intramodel fine-grained partitioning, we have overcome the singledevice memory wall, transforming the intractable deployment and scheduling problem into a manageable task. To support this, we developed the HE-HRL algorithm, which efficiently navigates the discrete-continuous solution space via a feasibility-guided hybrid evolutionary strategy. Our results have confirmed that EdgeCoInfer is highly effective, delivering a 71.88% reduction in memory footprint and a 76% decrease in system cost compared to state-of-the-art baselines. R EFERENCES [1] G. K. Walia, M. Kumar, and S. S. Gill, “AI-empowered fog/edge resource management for IoT applications: A comprehensive review, research challenges, and future perspectives,” IEEE Communications Surveys & Tutorials, vol. 26, no. 1, pp. 619–669, 2024.

[2] S. Tang, Y. Yu, H. Wang, G. Wang, W. Chen, Z. Xu, S. Guo, and W. Gao, “A survey on scheduling techniques in computing and network convergence,” IEEE Communications Surveys & Tutorials, vol. 26, no. 1, pp. 160–195, 2024. [3] L. Tan, P. Zhou, S. Guo, J. Zhao, Z. Kuang, D. Qiao, and L. Yang, “Partitioned collaborative inference for on-device models via evolutionary reinforcement learning,” in 2025 IEEE 45th International Conference on Distributed Computing Systems (ICDCS), pp. 714–724, 2025. [4] B. Kar, W. Yahya, Y.-D. Lin, and A. Ali, “Offloading using traditional optimization and machine learning in federated cloud–edge–fog systems: A survey,” IEEE Communications Surveys & Tutorials, vol. 25, no. 2, pp. 1199–1226, 2023. [5] C. Zhang, Z. Yang, J. Liu, Y. Li, Y. Han, X. Chen, Z. Huang, B. Fu, and G. Yu, “Appagent: Multimodal agents as smartphone users,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, (New York, NY, USA), Association for Computing Machinery, 2025. [6] T. Mohammed, C. Joe-Wong, R. Babbar, and M. Di Francesco, “Distributed inference acceleration with adaptive dnn partitioning and offloading,” in IEEE INFOCOM 2020-IEEE conference on computer communications, pp. 854–863, IEEE, 2020. [7] H. Li, X. Li, Q. Fan, Q. He, X. Wang, and V. C. Leung, “Distributed dnn inference with fine-grained model partitioning in mobile edge computing networks,” IEEE Transactions on Mobile Computing, vol. 23, no. 10, pp. 9060–9074, 2024. [8] A. Borzunov, M. Ryabinin, A. Chumachenko, D. Baranchuk, T. Dettmers, Y. Belkada, P. Samygin, and C. A. Raffel, “Distributed inference and fine-tuning of large language models over the internet,” Advances in neural information processing systems, vol. 36, pp. 12312– 12331, 2023. [9] P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132, IEEE, 2024. [10] M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “Edgeshard: Efficient llm inference via collaborative edge computing,” IEEE Internet of Things Journal, 2024. [11] R. Han, Q. Zhang, C. H. Liu, G. Wang, J. Tang, and L. Y. Chen, “Legodnn: block-grained scaling of deep neural networks for mobile vision,” in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, pp. 406–419, 2021. [12] X. Sun, R. Panda, R. Feris, and K. Saenko, “Adashare: Learning what to share for efficient deep multi-task learning,” in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33, pp. 8728–8740, Curran Associates, Inc., 2020. [13] J. Yoon, J. Lee, T. He, N. Choi, and B. Ji, “S2m3: Split-and-share multimodal models for distributed multi-task inference on the edge,” in 2025 IEEE 45th International Conference on Distributed Computing Systems (ICDCS), pp. 868–878, IEEE, 2025. [14] M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, and F. Yang, “Analysis of {Large-Scale}{Multi-Tenant}{GPU} clusters for {DNN} training workloads,” in 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 947–960, 2019. [15] L. Tan, Z. Kuang, L. Zhao, and A. Liu, “Energy-efficient joint task offloading and resource allocation in ofdma-based collaborative edge computing,” IEEE Transactions on Wireless Communications, vol. 21, no. 3, pp. 1960–1972, 2022. [16] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510–4520, 2018. [17] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. [18] V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” Proceedings of Machine Learning and Systems, vol. 5, pp. 341–353, 2023. [19] Q. Wang, L. Li, W. Luo, Y. Zhang, and B. Wang, “Dso: A gpu energy efficiency optimizer by fusing dynamic and static information,” in 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pp. 1–6, 2024. [20] Y. Su, Y. Zhou, Q. Qiu, J. Li, Q. Xia, P. Li, X. Duan, Z. Wang, and M. Zhang, “Accurate KV cache quantization with outlier tokens tracing,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, eds.), (Vienna, Austria), pp. 12895– 12915, Association for Computational Linguistics, July 2025. [21] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016. [22] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” arXiv preprint arXiv:1803.01271, 2018. [23] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, pp. 1861–1870, PMLR, 10–15 Jul 2018. [24] L. Tan, S. Guo, P. Zhou, Z. Kuang, S. Long, and Z. Li, “Multiuav-enabled collaborative edge computing: Deployment, offloading and

resource optimization,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 11, pp. 18305–18320, 2024. [25] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. [26] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. LMSYS. [27] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, et al., “Openclip.” https: //doi.org/10.5281/zenodo.5143773, 2021. [28] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” arXiv preprint arXiv:2103.00020, 2021. [29] NVIDIA Corporation, “Nvidia jetson agx orin developer kit,” 2022. [Online]. Available: developer.nvidia.com/embedded/jetson-agx-orin. [30] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.