IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, XX 2026
1
Generation Quality-Latency Tradeoff-Aware Inference Offloading for Multimodal LLMs in Cloud-Edge Continuum
arXiv:2606.15210v1 [cs.DC] 13 Jun 2026
Zhongxiao Wang, Yueshen Xu, Member, IEEE, Wei Xi, Member, IEEE, Xinkui Zhao, Tom H. Luan, Fellow, IEEE, Wei Shao, Member, IEEE, Rui Li, Member, IEEE
Abstract—Beyond pure cloud, some efforts are being made to deploy Large Language Models (LLMs) in edge to accelerate inference response. So the deployment of LLMs in cloud-edge continuum becomes a promising paradigm, where the tasks involving multimodal data occupy a large part of requests. Under this continuum, users usually concern about multiple Quality-ofService (QoS) attributes, but it is always intractable to jointly optimize them. In this paper, we propose to study the joint optimization of those attributes and focus on two key representatives, i.e., content generation quality and response latency. We propose to study the offloading technology to achieve a tradeoff between the two objectives in the cloud-edge collaborative Multimodal LLM (MLLM) system. However, it is highly difficult to predict generation quality and inference latency for MLLM inference tasks while optimizing this offloading process. To address these unprecedented difficulties, we propose a Quality-Latency TradeoffAware MLLM Inference Offloading (QLMIO) framework to make decisions that optimally balance generation quality and response latency. QLMIO consists of two novel capabilities: 1) MLLM Inference Latency Prediction (MILP) and 2) MLLM Generation Quality Prediction (MGQP), which proactively predict inference latencies and generation correctness probabilities, respectively. Meanwhile, recognizing the absence of publicly available datasets tailored to the MLLM inference offloading problem, we constructed a real-world cloud-edge collaborative MLLM system and subsequently collected an MLLM Inference Offloading Benchmark (MIOBench) to comprehensively evaluate our framework and facilitate the study of this problem. Extensive experimental results demonstrate that the QLMIO framework reduces latency by up to 58.14% compared to baselines, while simultaneously matching the task completion rate achieved under the case that executes all requests exclusively on a cloud server. The dataset and codes are available at Github1 . This paper is funded by National Key Research and Development Program of China (2023YFF0905100), National Natural Science Foundation of China (62472338), Shaanxi Province Qinchuangyuan “Scientist+Engineer” Team Development Program (2024QCY-KXJ-165), and Open Foundation of Yunnan Key Laboratory of Software Engineering (2023SE301) (Corresponding author: Yueshen Xu, and Yueshen Xu contributes equally with Zhongxiao Wang, so he is also the co-first author). Zhongxiao Wang, Yueshen Xu, and Rui Li are with the School of Computer Science and Technology, Xidian University, Xi’an 710126, China. E-mails: [email protected], [email protected], and [email protected]. Wei Xi is with the School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China (e-mail: [email protected]). Xinkui Zhao is with the School of Software Technology, Zhejiang University, Ningbo 315048, China. E-mail: [email protected]. Tom H. Luan is with the School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an 710049, China. E-mail: [email protected]. Wei Shao is with the School of Computer Science and Engineering, University of New South Wales, Sydney 2052, Australia. E-mail: [email protected]. 1 https://anonymous.4open.science/r/MIOBench
Index Terms—Multimodal LLM, Offloading, Generation Quality, Inference Latency, Cloud-Edge Continuum
I. I NTRODUCTION
I
N recent years, with the unprecedented expansion of Large Language Models (LLMs) [1] around the world, the demand for LLM inference services is exponentially proliferating. In the current production environment, predominant LLM inference systems (e.g., Qwen2 and ChatGPT3 ) conventionally deploy hundred-billion-parameter LLMs purely on cloud servers to provide LLM inference services. However, this cloud-based deployment paradigm faces serval serious challenges, where two representatives are as follows: 1) excessive response latency caused by overwhelming concurrent user requests within the same time slot, and 2) unsustainable resource and cost overheads arising from provisioning numerous hundred-billion-parameter LLM instances. Meanwhile, edge computing has emerged as a distributed computing paradigm that can partially offload computation workloads from cloud centers to edge nodes, thereby reducing response latency and resource costs [2]. This raises a promising solution, that is, deploying LLMs with heterogeneous parameter scales across the cloud-edge continuum to collaboratively deliver LLM inference services [3]. As LLM inference tasks are becoming increasingly multimodal, Multimodal Large Language Models (MLLMs) [4] have emerged as a preferred choice for users owing to their advanced cross-modal processing capabilities. In practical applications of cloud-edge collaborative MLLM systems, due to the significant computational capacity disparity between cloud and edge servers, the parameter scales of deployable MLLMs exhibit substantial heterogeneity. For instance, a cloud server equipped with an RTX5090 GPU featuring 32GB VRAM can effortlessly deploy the 30-billion-parameter Qwen3-VL4 model, while an edge device with merely 8GB VRAM (e.g., Jetson Orin Nano Super5 ) can only deploy the 2-billionparameter Qwen3-VL model. This heterogeneity in device computational capabilities and model parameter scales inevitably leads to significant variations in generation quality 2 https://www.qianwen.com/ 3 https://chatgpt.com/ 4 https://huggingface.co/collections/Qwen/qwen3-vl, VL is short for VisionLanguage 5 https://www.nvidia.com/en-us/autonomous-machines/embeddedsystems/jetson-orin/nano-super-developer-kit/
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, XX 2026
2
and response latency among different nodes within cloud-edge collaborative MLLM systems. To visually demonstrate performance disparities among heterogeneous nodes, we randomly sampled 300 inference tasks from the MMBench6 dataset [5] and executed them on three distinct devices (RTX5090: 32GB, RTX3090Ti: 24GB, and Jetson Orin Nano Super: 8GB) hosting Qwen3-VL variants with disparate parameter scales (30B, 8B, and 2B). The generation quality and response latency results are shown in Fig. 1. As observed in Fig. 1(a), on low-computing devices such as the Jetson Orin Nano Super, the generation accuracy of the MLLM is only 66.67%, with a high proportion of time-out samples reaching 26.33%. In contrast, on the high-performance device RTX5090, the generation accuracy approaches 90%, and no time-out samples occur. This phenomenon clearly demonstrates a significant disparity in generation accuracy when the same task is performed on different devices equipped with different MLLMs. Fig. 1(b) displays the histograms and the Kernel Density Estimation (KDE) plots of response latency across different devices.
process sophisticated tasks, leading to longer inference latency [6]. This substantial heterogeneity across nodes makes the offloading of MLLM inference an imperative technology for the efficient operation of cloud-edge collaborative MLLM systems. Compared to traditional edge computing offloading problems [7] [8], the offloading for MLLM inference presents two unprecedented challenges: 1) there are no existing methods can predict the generation quality of a specific task on one given node, making it exceptionally difficult to guarantee the overall generation accuracy in cloud-edge collaborative MLLM systems, and 2) the response latency of MLLM inference cannot be simply derived from the data volume and the devices’ computational capacities, which introduces significant uncertainty into the response time of cloud-edge collaborative MLLM systems. To address these challenges and ensure low-latency, high-reliability inference in cloud-edge collaborative MLLM systems, this paper makes the following primary contributions: • To the best of our knowledge, we are one of the pioneers to address MLLM inference offloading for cloud-edge continuum, considering both generation quality and response latency. We formulate this problem as a multi-objective optimization problem and propose a Quality-Latency Tradeoff-Aware MLLM Inference Offloading (QLMIO) framework to efficiently address this problem, thereby guaranteeing the system’s performance. • To address the challenge that the generation quality and inference latency are hard to predict, caused by heterogeneous cognitive capabilities across nodes, we design an MLLM Generation Quality Prediction (MGQP) module and an MLLM Inference Latency Prediction (MILP) module. MGQP and MILP can estimate task-node compatibility by extracting semantic features from user input prompts and node-level features, enabling the prediction of generation quality and inference latency for specific tasks on designated nodes. • Given the current absence of a public dataset that contains the actual generation quality and response latency of various MLLM inference tasks across nodes with diverse configurations, we collected and created an MLLM Inference Offloading Benchmark (MIOBench). MIOBench is collected from a real-world cloud-edge collaborative MLLM system constructed by this paper using a variety of devices and MLLMs. • Given the high costs of deploying MLLMs in cloudedge environments, we developed a Cloud-Edge Collaborative MLLM System simulator (CEMLLM-Sim) based on our MIOBench dataset for training and evaluating MLLM inference offloading algorithms. We open-sourced the MIOBench dataset, the CEMLLM-Sim simulation framework, and all source codes on GitHub7 . The subsequent sections are arranged as follows. Section II provides an overview of the related work. Section III presents the system model and formulates the optimization