From Cloud to Crowd: Democratizing LLM Service with Decentralized Edge Collaboration for RAG

1

From Cloud to Crowd: Democratizing LLM Service with Decentralized Edge Collaboration for RAG

arXiv:2608.00922v1 [cs.DC] 2 Aug 2026

Jiaxing Li, Hengzhi Wang, Feng Wang, Senior Member, IEEE, Chi Xu, Danyang Song, Ruixiao Zhang, Edith C. H. Ngai, Senior Member, IEEE, Jiangchuan Liu, Fellow, IEEE

Abstract—The rapid advancement of large language models (LLMs) has increased demand for scalable and cost-effective deployment, especially for mobile and edge devices. Cloud-hosted LLMs are powerful but expensive and difficult to scale due to vendor lock-in and high resource needs, resulting in high expenses and unstable performance under load. Recent efforts focus on deploying small language models (SLMs), distilled or pruned from LLMs, on resource-constrained edge devices to reduce costs and improve scalability. However, edge-based SLMs face limited knowledge coverage and notable accuracy gap compared to cloud-based LLMs. To address this, we present DEFRAG, a decentralized edge collaboration system for retrievalaugmented generation (RAG) that optimizes both retrieval and generation across heterogeneous edge devices. For retrieval, DEFRAG compresses and shares knowledge graphs, using hybrid retrieval to expand knowledge coverage. For generation, DEFRAG introduces an optimizer that adaptively selects SLMs and RAG parameters per query, balancing accuracy and cost. We implement DEFRAG on a heterogeneous edge testbed and evaluate it on benchmark QA datasets. We also test it under mobile route stress, non-uniform data placement, and a domainspecific QA workload. The results show that DEFRAG maintains stable service quality and cost efficiency under these broader settings. Results show that DEFRAG narrows the SLM–LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services. These findings demonstrate the potential of DEFRAG for democratized LLM services at the edge. Index Terms—Edge computing, decentralized systems, large language model applications, retrieval-augmented generation.

I. I NTRODUCTION © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Accepted for publication in IEEE Transactions on Mobile Computing. This work is partly supported by an NSERC Discovery Grant. (Corresponding author: Jiangchuan Liu.) Jiaxing Li, Chi Xu and Jiangchuan Liu are with the School of Computing Science, Simon Fraser University, Burnaby, BC V5A 1S6, Canada (e-mail: {jla641, chix, jcliu}@sfu.ca). Hengzhi Wang is with the College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China (e-mail: whz@szu.edu.cn). Feng Wang is with the Department of Computer and Information Science, The University of Mississippi, University, MS 38677, USA (e-mail: fwang@cs.olemiss.edu). Danyang Song and Edith C. H. Ngai are with the Department of Electrical and Electronic Engineering, University of Hong Kong, Pok Fu Lam, Hong Kong (e-mail: dysong@connect.hku.hk, chngai@eee.hku.hk). Ruixiao Zhang is with the Department of Computer Science, The University of Illinois Urbana-Champaign, Champaign, IL 61820, USA (e-mail: ruixiao.cs.zhang@gmail.com).

Computing Resources Sharing Mobile/Edge Device

Lightweight Knowledge Graphs (KGs) Sharing

Edge Server

Vector Databases (DBs) Sharing Mobile/Edge Device

Mobile/Edge Device Vector DBs Mobile/Edge Device

Relevant Data Retrieved Context Shared KGs DB

Edge Server Generated

Query

Optimizer

Output

Final Answer

Figure 1: Overview of the Decentralized Edge Collaboration For RAG (DEFRAG) system architecture.

T

HE remarkable progress in large language models (LLMs) has accelerated the proliferation of AI-powered applications across a range of domains, including content creation, intelligent assistants, and collaborative robotics [1]. Increasingly, these applications are accessed from mobile and edge clients (e.g., smartphones, laptops, wearables, and IoT gateways), where usage is often frequent and interactive. As a result, per-query and per-token costs become a first-order barrier to scalability and accessibility. However, the prevailing paradigm for deploying LLMs remains largely cloud-centric, relying on centralized supercomputing resources to deliver state-of-the-art performance [2]. While this approach enables access to powerful models, it is constrained by two critical limitations. These limitations are particularly costly for mobile and edge clients, where frequent interactive queries make per-token charges add up quickly. First, the vendor-locked nature of cloud-based LLM services raises the barriers to entry for widespread adoption. Subscription and token-based pricing schemes make high-quality language models prohibitively expensive for many users, hindering broader accessibility [3], [4]. Furthermore, monolithic model architectures complicate the rapid integration of newly emerging knowledge, rendering real-time personalization and timely adaptation challenging and inefficient [5], [6]. Second, centralized frameworks suffer from limited scalability. As the demand for LLM services grows, vertical scaling—adding capacity within individual data centers—leads to steeply increasing costs across hardware, software, and energy consumption [7]. These costs are further exacerbated by inherent resource utilization imbalances. During off-peak periods, resources are underutilized; during peak demand, oversubscription leads to both compute and network congestion, causing unpredictable tail latency and limiting scalability [8], [9].

2

To address the high cost and limited scalability of centralized LLM services, leveraging idle computing resources on edge devices for distributed deployment has emerged as a promising approach [10]. Existing research mainly follows two directions. The first partitions large-parameter LLMs across edge nodes, assigning inference tasks based on bandwidth and processing capacity to reduce the load on centralized systems [4], [11]. However, this method introduces significant coordination and communication overhead, resulting in increased latency and possible inference stalls [4]. The second direction deploys language models of varying sizes, especially small language models (SLMs), independently on edge devices. SLMs are typically obtained by distilling or pruning larger LLMs [12], making them lightweight variants of LLMs with much lower resource requirements [2]. Integrating retrievalaugmented generation (RAG) at the edge further expands SLM knowledge coverage by retrieving external database content, narrowing the knowledge gap between SLMs and LLMs, and improving answer accuracy [13], [14]. Our measurement shows that SLMs on individual edge devices are still constrained by resource capacity (Section II). Most mobile edge devices can support SLMs up to 4B parameters; without further optimization, the accuracy gap with a 235B LLM can reach 38%. Access to external databases via RAG can reduce this gap to 16.5%, enabling edge-based SLMs to approach the accuracy of cloud-based LLMs. Nevertheless, single-device resource limits remain a bottleneck. Edge collaboration is therefore essential to narrow the accuracy gap and enhance cost efficiency and scalability. In this context, we propose the Decentralized Edge Collaboration For RAG (DEFRAG) system architecture, which organizes heterogeneous edge nodes into a unified network that shares both data storage and computational resources. To make LLM-quality service affordable and scalable for mobile and edge clients, decentralized RAG must address several challenges. First, device heterogeneity complicates system coordination. Edge devices vary greatly in computational capacity, supporting different model sizes and quantization precisions, making unified SLM inference challenging. Second, queryaware adaptation is challenging. Aligning RAG parameters and SLM selection with each query’s demands is nontrivial, as both excessive and insufficient retrieval can degrade performance, and SLMs differ in their ability to process retrieved content. Third, network-wide knowledge fragmentation limits coverage. Comprehensive knowledge bases cannot be stored on individual edge devices; thus, efficient knowledge sharing and collaboration across the network are essential for maintaining inference quality, yet this remains an open challenge. To address these challenges, we design the DEFRAG system architecture as shown in Figure 1. First, to overcome device storage limits and knowledge fragmentation, we compress knowledge graphs into lightweight entity-relation representations without embeddings, enabling efficient caching and sharing across edge devices. For data in vector databases, sparse retrieval over these lightweight graphs identifies relevant entities and fetches the corresponding text and embeddings from peer devices, supporting broad and deep knowledge through crossdevice collaboration. Second, motivated by our data analysis,

we introduce a query-adaptive optimizer based on Bayesian online learning. Unlike linear-contextual bandits, Bayesian optimization better models complex, nonlinear interactions among queries, retrieval contexts, and generation configurations, providing robust accuracy even with limited data. The optimizer updates predictions through online feedback, dynamically selecting RAG parameters and SLM configurations to balance accuracy and cost efficiency. Finally, to handle resource heterogeneity, devices are grouped by computational capacity, and the optimizer assigns models of appropriate size and precision to peer devices, enabling coordinated and efficient SLM inference. These advances allow SLMs to approach the accuracy of centralized LLMs while improving cost efficiency and scalability, supporting broader and more affordable access to LLMs. We implemented and evaluated DEFRAG on a testbed with heterogeneous edge devices, analyzing its performance across accuracy, cost, and scalability. For accuracy, the optimizer’s adaptive selection of RAG parameters and SLM configurations narrowed the gap between edge-based SLMs and cloud-based LLMs. On two benchmark QA datasets, edge accuracy improved from 61.5% and 76.6% to 67.8% and 81.4%, reducing the accuracy gap from 12.7% and 7.1% to 6.4% and 2.3%. For cost, DEFRAG reduced expenses by up to 98.4% with full peer resource sharing. For scalability, DEFRAG achieved up to 97.8% higher peak throughput than centralized services as concurrent user numbers increased. We also test DEFRAG under mobile route stress, non-uniform data placement, and a domain-specific QA workload. The results show that DEFRAG maintains stable service quality and cost efficiency under these broader settings. These results show that DEFRAG effectively leverages heterogeneous edge resources to deliver accurate, cost-efficient, and scalable LLM services. DEFRAG connects edge devices in a decentralized network, enabling users to share and access models and knowledge. This approach shifts LLM service from centralized cloud delivery to collaborative provisioning by the crowd, advancing the democratization of LLM capabilities. Our contributions can be summarized as follows: • We propose a lightweight edge-collaborative RAG retrieval strategy that compresses and distributes knowledge graphs for efficient coverage under storage constraints. • We design an optimizer that adapts RAG generation parameters per query and updates online from historical data, balancing accuracy and cost efficiency across heterogeneous edge devices. • We implement and deploy the DEFRAG prototype on a heterogeneous edge devices testbed. Extensive experiments demonstrate that DEFRAG significantly narrows the SLM–LLM accuracy gap, achieves substantial cost savings, and provides strong scalability compared to centralized baselines. II. M OTIVATION AND O BSERVATION In this section, we comprehensively evaluate recent edge SLMs and their gap with LLM on benchmark QA datasets, analyzing performance across model scales, quantization, query

3

Table I: Qwen3 models memory usage across different parameter size and quantization precisions. INT4

INT8

FP16

Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B Qwen3-235B

1.4 GB 2.6 GB 5.2 GB 9.3 GB 20 GB 142 GB

2.2 GB 4.4 GB 8.9 GB 16 GB 35 GB 250 GB

4.1 GB 8.1 GB 16 GB 30 GB 66 GB 470 GB

75

NQ Dataset

0 100

HotpotQA Dataset

75 50 25

Orin Nano S25 Ultra

100

0

7 1. b7 IN 1. b- T 7b IN 4 - T 4b FP 8 - 1 4b IN 6 T 4b -IN 4 -F T8 8b P - 1 8b IN 6 T 8b -IN 4 14 -F T8 b P 14 -I 16 N 14 b-I T4 b NT 32 -F 8 b P1 -I 6 N 23 T4 5B

RTX5090 RTX4090 AGX Orin

1.

Tokens/s

200

GraphRAG

25

400 300

RAG

50

Accuracy (%)

Models

LLM-only 100

Models

1.

7b 1. -IN 7 T 1. b-I 4 7b NT -F 8 4b P1 -I 6 4b NT - 4 4b INT -F 8 8b P1 -I 6 8b NT - 4 8b INT -F 8 14 P b 16 14 -IN b T 14 -IN 4 b T8 32 FP b 16 -I N T4

0

Figure 3: Accuracy comparison on NQ and HotpotQA with direct generation, RAG, and GraphRAG.

Models

Figure 2: Average throughput (tokens/s) of various SLMs across heterogeneous edge devices. types, devices, and RAG strategies. The results reveal key challenges and opportunities for improvement. A. Preliminary We evaluate both SLM and LLM models from the Qwen3 series [12], as it provides high-quality, fully open-source models at multiple parameter sizes for fair comparison. The SLMs include 1.7B, 4B, 8B, and 14B variants, with Qwen332B as an edge-server reference point and Qwen3-235B as the reference LLM. For comprehensive comparison, we consider 4-bit, 8-bit, and half-precision quantizations, implemented via llama.cpp [15]. Table I reports GPU memory requirements for each configuration. To assess practical performance on heterogeneous edge devices, we evaluate five representative edge devices: Jetson Orin Nano, Jetson AGX Orin, Galaxy S25 Ultra (Snapdragon 8 Elite), RTX 4090, and RTX 5090. These devices cover both PC-class and mobile platforms for applications such as smartphones, laptops, AR glasses, and robots. For questionanswer (QA) evaluation, we use Natural Questions (NQ) [16] and HotpotQA [17], which contain single-hop and multihop queries, respectively. To ensure consistency in RAG and GraphRAG evaluation, we sample top 500 QA pairs from each dataset by selecting questions supported by Wikipedia pages referenced more than 10 times. By leveraging RAG, SLMs can access the same breadth of knowledge as LLMs, thereby narrowing the accuracy gap caused by knowledge limitations. To measure the current gap between SLMs and LLMs, we adopt two advanced RAG methods: standard dense retrieval [18] and a knowledge graphbased approach adapted from GraphRAG [19]. B. Data-Driven Analysis Edge computing environments comprise a diverse array of devices, from high-performance servers to lightweight mobile

hardware, with significant variability in practical performance. Prior research has seldom provided fine-grained evaluation of the capabilities of these devices, particularly for SLMs of different sizes and quantization precisions. To address this, we systematically measure the performance of representative edge devices. As shown in Figure 2, these devices can be grouped by their token generation rates: edge servers (such as RTX 4090 and RTX 5090) can stably serve larger models (e.g., 14B), while edge devices (such as Jetson Orin Nano, Jetson AGX Orin, and Galaxy S25 Ultra) typically support smaller models (e.g., 1.7B). Edge servers provide stable, consistent performance, whereas edge devices display greater variability. Effectively coordinating such heterogeneous devices, each with distinct capabilities, to collaboratively serve SLMs as a unified system remains an open challenge. Recent work has explored combining RAG with SLMs on edge devices [13], [14], [20], but systematic evaluations across model sizes, quantization precisions, RAG strategies, and query types remain limited—particularly regarding the accuracy gap between SLMs and LLMs. To address this, we use the Qwen3 series to evaluate the SLM–LLM accuracy gap on Natural Questions and HotpotQA using standard RAG and GraphRAG. Accuracy is assessed by GPT-4.1 [21] on 500 QA pairs, using a binary scoring method [22]. As shown in Figure 3, for Natural Questions, standard RAG with a static TOP-K=7 strikes a good balance between context sufficiency and overload, significantly narrowing the accuracy gap between SLMs and LLMs. In contrast, on the multihop HotpotQA dataset, standard RAG fails to capture logical relationships, resulting in limited accuracy improvements. We therefore adopt a simplified version of GraphRAG, since the dense retrieval results of the GraphRAG approach often exceed the processing capacity of SLMs [20]. Specifically, we retain only essential information—the title and description of the top 10 entities, the source, target, and description of the top 10 relationships, and the text of the top 2 data chunks, each selected using static heuristics. This streamlined retrieval is better suited to SLM capacity, improving reasoning accuracy and further reducing the SLM–LLM gap on HotpotQA.

4

Table II: Coverage of QA pairs and mean SLMs accuracy at different entity relevance score thresholds (Dataset: HotpotQA, Method: GraphRAG) Threshold

θ ≥ 0.50

θ ≥ 0.55

θ ≥ 0.65

Coverage

459/500 (91.8%)

363/500 (72.6%)

77/500 (15.4%)

Accuracy (1.7B) Accuracy (4B) Accuracy (8B) Accuracy (14B)

65.0% 72.7% 76.4% 78.2%

67.6% 75.2% 78.8% 80.9%

80.1% 83.5% 83.5% 88.7%

Table III: Storage sizes of the Knowledge Graph database (KG DB) and Vector database (Vector DB) for the NQ and HotpotQA datasets. Dataset

KG DB

Vector DB

Scale

Natural Questions [16] HotpotQA [17]

27.42MB 40.38MB

956.41MB 1433.6MB

∼35× ∼36×

These results indicate that RAG can substantially narrow the accuracy gap between SLMs and LLMs. However, the optimal RAG configuration, including the number of entities, relationships, and data chunks, varies by query type (e.g. single-hop vs. multi-hop). Standard RAG can be considered a special case of GraphRAG where no entities and relationships are retrieved. Achievable accuracy also depends on the SLM chosen. Therefore, dynamically selecting both SLMs and RAG parameters for each query remains a key challenge. As shown in Table II, raising the relevance score1 threshold for retrieved content allows SLMs to maintain higher accuracy, though with reduced coverage2 . This suggests that leveraging relevance scores to characterize query patterns and guide RAG generation parameters tuning can further close the gap to LLM accuracy on suitable queries. Adaptive selection of SLMs and RAG configurations based on query relevance score is therefore a promising approach for accurate edge inference. Another observation is that a complete graph database, including both the knowledge graph and vector database, can be large. Even when limited to popular Wikipedia pages from NQ and HotpotQA, database sizes reach 983.83MB (139 pages) and 1473.98MB (231 pages), respectively. In practice, each edge device stores only a subset, limiting knowledge coverage. Queries involving out-of-scope content must be broadcast to neighboring devices, which is inefficient and difficult to handle at scale. Efficient data sharing and collaboration among edge devices therefore remains a key challenge. We observe that the knowledge graph database without embeddings (entities and relationships only) is much smaller—about 1/35 the size of the vector database (see Table III). This suggests an effective strategy: cache shared, lightweight knowledge graphs on each edge device, and use sparse retrieval [18] to identify relevant content in neighboring vector databases, enabling accurate extraction of data chunks and embeddings. This approach efficiently expands both the breadth and depth of knowledge accessible to each device. 1 Indicates how well retrieved content relates to the query. 2 Proportion of queries with average relevance score above threshold.

C. Challenges and Opportunities Based on our data-driven analysis, several key challenges and corresponding opportunities emerge in advancing distributed LLM services with the DEFRAG system architecture, as summarized below: Challenge 1: Coordinated Generation on Heterogeneous Edge Devices. Edge environments consist of diverse device types with significant variability in computational capacity, supporting different model sizes and quantization precisions (Figure 2). Effectively coordinating these heterogeneous devices to serve SLMs as a unified system remains a challenge. Opportunity: Systematic grouping of devices based on capability, together with dynamic resource scheduling that assigns models of suitable size and precision to corresponding hardware, can improve overall resource efficiency. This approach enables coordinated and effective SLM inference, facilitating integrated utilization of edge resources. Challenge 2: Query-Adaptive Selection of RAG Generation Parameters. As shown in Figure 3, aligning RAG retrieval parameters with SLM capacity is crucial for inference accuracy and efficiency. Both excessive and insufficient information can degrade performance, and SLMs differ in their ability to process retrieved content. Dynamically selecting parameters for each query, therefore, remains a key challenge. Opportunity: As shown in Table II, analyzing relevance scores between the query and retrieved context enables dynamic selection of SLM configurations and RAG parameters for each query. This approach allows edge devices to achieve high accuracy while maintaining efficient resource utilization. Challenge 3: Scalable Knowledge Retrieval for RAG Across Edge Devices. Comprehensive knowledge bases, including knowledge graphs and vector databases, are typically too large for individual edge devices to store in full, resulting in limited knowledge coverage per device. Efficient crossdevice knowledge sharing and processing remain a challenge. Opportunity: Compressing the knowledge graph into a lightweight representation of essential entities and relationships enables efficient caching across edge devices, ensuring broad knowledge coverage (Table III). Sparse retrieval can then dynamically locate relevant entities and access corresponding vector data from peer devices to address depth requirements. This cross-device collaboration efficiently expands knowledge scope and improves overall coverage. III. S YSTEM D ESIGN AND M ODELING In this section, we present DEFRAG, a decentralized RAG architecture that enables collaboration among edge devices to address the challenges above. We first provide an overview of the architecture, followed by details on the modeling of its core components. A. Methodology Overview DEFRAG enhances both retrieval and generation across a group of edge devices. For retrieval, it expands knowledge access by sharing lightweight knowledge graphs among peers. For generation, it adaptively tunes SLMs and RAG parameters

5

Table IV: Summary of frequently used notations. Shared KGs DB ... ED2 ED3 ED1 Peers Shard KGs Local KG

Fetch Data Chunks and Embeddings from Vector DBs

Local ED1 ED2 ED3 Offline

Query

Sparse Retrieval

Vector DBs

TOP-K ED4 Entities Dense Retrieval Peer Edge Devices (EDs)

Relevance Score Context

(a) Edge-Collaborative Hybrid Retrieval Multi-objective Optimizer Accuracy Efficiency Context

Action Set

Max Utility Prediction Model

Utility Weight

Online Update

Action <SLM config, RAG parameters> Yes

No

Local Compute Check

Local ED1 Final Answer

ED2

ED4

Peer Edge Devices

Notation

Explanation

ct , at C, A E m (a) E R (a) E(a) α(ct , a) U (ct , a) γ ĒW ᾱW αC

Context vector and action at query round t Context space and combinatorial action space Model efficiency of action a (by model-cost) Input efficiency of action a (by input length) Efficiency indicator of action a Accuracy indicator of action a under context ct Utility combining accuracy and efficiency Weight of accuracy in the utility (0 ≤ γ ≤ 1) Average efficiency over the latest |W | queries Average accuracy over the latest |W | queries Accuracy of centralized SLMs/LLM baseline

(b) Optimizer-Driven Adaptive Generation

Figure 4: Design of the DEFRAG system architecture and end-to-end workflow, combining edge-collaborative hybrid retrieval with optimizer-driven adaptive generation and optional peer offloading.

based on query-specific context features extracted during retrieval. This design balances accuracy and efficiency, enabling the system to approach cloud LLM accuracy while running SLMs on heterogeneous edge devices. Figure 4 shows the overall workflow of DEFRAG. Below, we detail its retrieval and generation optimizations. Edge-collaborative hybrid retrieval. To broaden retrieval, DEFRAG extends the knowledge scope of edge-based RAG by sharing lightweight knowledge graphs with peer devices. To support breadth data coverage, these graphs are designed without dense vector representations. Following hybrid retrieval strategies [18], the system adopts a two-stage process, further enhancing depth through collaboration among devices. In the first stage, queries use coarse-grained sparse retrieval [23] over the shared knowledge graphs database (KGs DB) with term-based methods such as BM25 to identify the TOP-K relevant entities. In the second stage, the system locates the corresponding vector databases (Vector DBs) on peer edge devices based on the retrieved entities, fetching data chunks and their embeddings. If a target device is offline, retrieval proceeds to the next nearest device by proximity until the entity is found or retrieval is abandoned. Fine-grained dense retrieval [18] further reranks the TOP-K relevant entities, enhancing retrieval depth. As shown in Table II, relevance scores from dense retrieval can characterize query patterns. These scores are summarized into context features—including count, mean, standard deviation, maximum, and minimum for entities and data chunks—which are passed to the optimizer to support adaptive generation (see Section III-B for details). Optimizer-driven adaptive generation. To support efficient and accurate generation, DEFRAG employs a multiobjective optimizer to select generation parameters for each query. These parameters include the choice of SLM size, quantization precision, and a combination of structured and unstructured RAG retrieval outputs. Structured information (entities and relationships) is extracted from shared knowledge graphs, while unstructured evidence (data chunks) is obtained

from vector databases on local and peer devices. The optimizer dynamically adjusts its strategy based on context features derived during retrieval, which reflect the structure and complexity of each query. For example, when a query involves multiple high-relevance entities, indicating a need for multi-entity reasoning, the optimizer selects highercapacity models and richer structured input. In contrast, for a query focused on a single high-confidence entity, often corresponding to a single-hop question, it chooses smaller models and minimal input to improve efficiency. Each candidate action receives a utility score balancing expected accuracy and efficiency, and the action with the highest score is selected. If the local device cannot execute it, the request is forwarded to the nearest capable peer device; otherwise, generation is performed locally. All interactions are logged to support online optimizer updates and ongoing refinement of the accuracy and efficiency trade-off. By jointly optimizing retrieval and generation, DEFRAG leverages heterogeneous edge resources to democratize LLM services through decentralized collaboration among the crowd. B. Contextual Multi-Objective Multi-Armed Bandit Problem To better understand and address the Multi-objective Optimizer, we formalize it as a Contextual Multi-Objective MultiArmed Bandit (CMOMAB) problem tailored to our DEFRAG system architecture. Frequently used notations are summarized in Table IV for reference. Context space. We formally define the context at each query round t as: ct := [net , c̄et , c̃et , ĉet , čet , ndt , c̄dt , c̃dt , ĉdt , čdt ] ∈ C, where net and ndt are the numbers of entities and data chunks retrieved after initial sparse filtering. c̄et and c̃et denote the mean and standard deviation of relevance scores for entities, while ĉet and čet are their maximum and minimum scores. Analogously, c̄dt , c̃dt , ĉdt , and čdt are the mean, standard deviation, maximum, and minimum relevance scores for selected data chunks. Here, C denotes the context space. The context design and its impact are evaluated through ablation studies in Section V-C. Action space. We define the action at each query round t as a configuration consisting of four distinct parameters: at := [mt , et , rt , dt ] ∈ A := M × E × R × D, where M denotes the set of selected SLMs with different sizes and quantization precision; E, R, and D are the TOP-K

6

selected entities, relationships, and data chunks, respectively. The action space A is combinatorial, spanning all generation parameters (RAG and SLM configurations). Utility indicators. To select the most suitable action at at each query round t, we use two utility indicators. The first is action accuracy. It is conditioned on the current query context ct and denoted as α(ct , at ). If the selected SLM’s response matches the reference answer, α(ct , at ) = 1; otherwise, α(ct , at ) = 0. However, it is computationally infeasible to exhaustively evaluate all actions by running each set of RAG generation parameters and comparing outputs. To address this, we use Bayesian online learning to estimate the probability of accuracy for each action given the current query context and update the model as outcomes are observed. This enables the system to adapt dynamically to changing query patterns (see Section IV for details). The model policy is therefore learned from observed context-action feedback rather than from model size, so a smaller or quantized model can be selected when its estimated accuracy is higher for the current query context. The second indicator is the efficiency of each action a, denoted as E(a), which reflects both the efficiency of the selected SLM and the chosen RAG parameters, and is defined below with higher value indicating greater efficiency: E(a) = wm E m (a) + wr E R (a),

a∈A

γ

1−γ

[α(ct , a)] · [E(a)]

s.t. 1 where ᾱW = |W |

lim ĒW

|W |→∞

lim ᾱW ≥ αC − ε

(3)

|W |→∞

P

accuracy over t∈W α(ct , a) is the average P 1 |W | queries, ĒW = |W | t∈W E(at ) is

the most recent the average efficiency, αC is the centralized SLMs or LLM baseline accuracy, and ε is a tolerance constant. The long-term objective guides the short-term trade-off by adaptively adjusting the weighting parameters γ, wm , and wr . When sliding window accuracy ᾱW drops below the target αC , the system increases γ to emphasize accuracy; if accuracy exceeds the target, γ is decreased to prioritize efficiency. If the average input token count or model-cost indicator exceeds its predefined threshold, wm and wr are adjusted. IV. BAYESIAN O NLINE L EARNING S OLUTION In this section, we propose the Decentralized Edge Collaboration For RAG Bayesian Online Learning (DEFRAGBOL) algorithm to address the CMOMAB problem above. We also discuss deployment strategies to reduce computational overhead and analyze the regret bound of the algorithm.

(1)

where E m (a) = 1/κm , and κm is an offline-profiled modelcost coefficient for the selected SLM configuration. The coefficient is computed from model memory footprint (size and quantization), FLOPs-level workload, and response latency. The term E R (a) = (Tmax − T (a))/Tmax represents the input’s efficiency coefficient, where T (a) is the input token length and Tmax is the maximum allowed. Larger RAG parameter values increase input length, which raises prefill and decode cost and latency [24], thereby reducing E R (a). The weights wm and wr (with wm +wr = 1) balance model and input efficiency. Higher wm discourages actions with high model-cost coefficients, while higher wr avoids long input lengths caused by large RAG parameters. These weights are dynamically adjusted online, allowing the system to balance model and input efficiency over time. Short-term objective. At each query round t, the shortterm objective is to select the action at from the action space A that yields the highest utility for the current query context ct . Utility combines accuracy α(ct , a) and efficiency E(a), aggregated by a weighted geometric mean [25] for a smooth and interpretable trade-off: max

max

(2)

Here, γ ∈ [0, 1] is a weighting parameter that balances accuracy and efficiency, and is adjusted online to align with the long-term objective. Long-term objective. While the short-term objective balances accuracy and efficiency for each query, the long-term objective dynamically adjusts this trade-off to sustain performance. The aim is to keep average accuracy above a target threshold, measured over a sliding window, while maximizing efficiency. Formally:

A. Algorithmic Framework Initially, we explored linear-contextual bandit methods such as LinUCB, Hybrid UCB [26], and LinkUCBGlobal [27], but these showed limited performance, converging to accuracies well below αC due to non-linear relationships between contexts and actions. To address this, we adopt a Bayesian online learning approach, modeling the utility function with a Gaussian Process (GP) [28] over the joint context-action space. As non-parametric estimators, GPs capture non-linear dynamics and provide uncertainty estimates, enabling a balance between exploration and exploitation. Gaussian process and kernel design. We use Gaussian Processes as non-parametric approximators to estimate the accuracy component of the utility function, modeling each context-action pair x ∈ X = C × A as a sample from GP(µ(x), k(x, x′ )), where µ(x) is the mean and k(x, x′ ) is the kernel. The predictive uncertainty σ(x), given by the kernel, reflects the variance of the GP prediction for each contextaction pair. Based on observed non-linearities and varying smoothness in accuracy across different contexts and actions (Section II-B), we adopt a stationary, anisotropic Matérn kernel (smoothness parameter 2.5), which fits the smooth yet nonlinear nature of the utility function [29]. To ensure stability, kernel hyperparameters are fixed during execution, selected through robustness testing on prior data [30]. Mathematical details are given in [28]. Exploration and exploitation. As delineated in Algorithm 1, DEFRAG-BOL operates in two phases: exploration and exploitation. In the exploration phase (t = 1, . . . , T0 ), the system retrieves the context ct for each query and uniformly samples an action at from the action space, executing it without utility prediction. The observed accuracy α(ct , at ) is recorded to

7

Algorithm 1: Decentralized Edge Collaboration For RAG Bayesian Online Learning (DEFRAG-BOL) Input: Action space A, total query rounds T . Hyperparams: Kernel k(·, ·), exploration params T0 , βt , window size Wmax , K, target accuracy αC , step sizes δγ , δw , efficiency limits Llim , Rlim . 3 Definitions: Let lt denote the input token length and rt indicate model-cost usage at query round t. 4 Initialize: GP ← ∅, history W ← ∅, weights γ, wm , wr . Compute initial E(a) via Eq. 1. 5 for t = 1, . . . , T do 6 Observe query context ct 7 if t ≤ T0 then // Phase I: Random Exploration 8 at ∼ Unif(A) 9 end 10 else // Phase II: UCB Exploitation 11 for a ∈ A do 12 µt−1 , σt−1 ← GP.Predict(c t , a) √ 13 α b(ct , a) ← µt−1 + βt σt−1 14 U (ct , a) ← [b α(ct , a)]γ · E(a)1−γ 15 end 16 at ← arg maxa∈A U (ct , a) 17 end 18 Execute at , observe accuracy αt and metrics lt , rt 19 Update W ← {(cτ , aτ , ατ , lτ , rτ )}tτ =max(1,t−Wmax +1) 20 Update GP posterior using data in W 21 if t > T0 and t (mod K) ≡ 0 then 22 Compute means ᾱ, L̄, R̄ from window W 23 if ᾱ < αC then γ ← min(1, γ + δγ ) 24 else γ ← max(0, γ − δγ ) 25 if L̄ > Llim then 26 wr ← min(1, wr + δw ); wm ← 1 − wr 27 end 28 else if R̄ > Rlim then 29 wm ← min(1, wm + δw ); wr ← 1 − wm 30 end 31 Update E(a) via Eq. 1 with new wm , wr 32 end 33 end 1 2

update the GP posterior. Uniform sampling ensures unbiased exploration and avoids early exclusion of potentially optimal actions, which could hinder convergence [31]. In the exploitation phase (t = T0 + 1, . . . , T ), the system uses the GP posterior to predict the accuracy for each candidate action. Specifically, it computes the mean µt−1 (ct , a) and uncertainty σt−1 (ct , a) to formulate an Upper Confidence Bound (UCB) estimate α b(ct , a). This accuracy estimate is integrated with the pre-computed efficiency E(a) to derive a composite utility score U (ct , a) = [b α(ct , a)]γ · E(a)1−γ . The optimizer selects the action maximizing this utility for execution. Upon observing the actual accuracy αt and efficiency metrics (lt , rt ), the history W is updated as a sliding window

retaining the most recent Wmax samples. Its observed accuracy is then used to update the GP, allowing the model to adapt as new data become available. To ensure long-term performance, the algorithm employs a periodic adaptation mechanism. Every K rounds, statistics (ᾱ, L̄, R̄) are computed over a window W to recalibrate the weighting parameters. The accuracy weight γ is increased when the moving average ᾱ falls below the target αC , prioritizing accuracy, and decreased otherwise to improve efficiency. Meanwhile, the efficiency weights wr and wm are adjusted to improve resource utilization efficiency: wr is increased if the average input length L̄ exceeds Llim , penalizing long contexts, while wm is raised when model-cost usage R̄ exceeds Rlim , discouraging high-cost model configurations. The efficiency table E(a) is then recomputed to reflect these updated priorities in subsequent decisions. Practical issue and acceleration. A key limitation of Gaussian Processes is the O(N 3 ) time complexity for posterior computation, making real-time fitting impractical as N grows [28], where N is the number of historical data used for training. To improve efficiency, we adopt two strategies: (1) fit the GP model periodically instead of after every query, using fast O(N ) prediction in between; and (2) restrict GP fitting to a fixed-size sliding window of recent queries, which also mitigates overfitting to outdated data. Together, these strategies reduce optimizer overhead while preserving the GP’s ability to model non-linear dependencies and maintain accuracy. B. Regret Bound Analysis Given Algorithm 1 and the short-term objective in Eq. 2, we derive the following theoretical results. Theorem 1. Let δ ∈ (0, 1), γ ∈ (0, 1), and define βt = 2 log (6|A|/(δπ 2 t2 )). Assume the utility function over contextaction pairs is sampled from a Gaussian Process with zero mean and Matérn kernel k(·, ·). Then, with probability at least 1 − δ, the cumulative regret RT of Algorithm 1 after T rounds satisfies s RT ≤

3−γ 4Emin βT λI T , log(1 + σ −2 )

where Emin = mina E(a), λI = maxct ,a I(α(ct , a); α̂(ct , a)) is the maximal mutual information gain, and σ 2 is the observation noise variance. √ In particular, for fixed hyperparameters, we have RT = O( T λI ). Proof. If a random variable x ∼ N (0, 1), then for θ > 0 √ Z ∞ −x2 /2 Pr{x > θ} = 1/ 2π e (4) θ Z ∞ √ 2 2 = 1/ 2π e−(x−θ) /2−(x−θ)θ−θ /2 (5) 0 2

2

≤ e−θ /2 Pr{x > 0} = e−θ /2 /2.

(6) √

Let x = (α̂(ct , a) − µt−1 (ct , a))/σt−1 (ct , a) and θ = βt , then it holds that p Pr{|α̂(ct , a) − µt−1 (ct , a)| > βt σt−1 (ct , a)} ≤ e−βt /2 .

8

Apply union bound on both a ∈ A and t ∈ T and set βt = 2 log (6|A|/(δπ 2 t2 )). Then, p |α̂(ct , a) − µt−1 (ct , a)| ≤ βt σt−1 (ct , a), ∀a, ∀t (7) holds with probability at least 1 − δ. According to line 17 in Algorithm 1, it holds that p [µt−1 (ct , a) + βt σt−1 (ct , a)]γ [E(a)]1−γ p ≥[µt−1 (ct , a∗ ) + βt σt−1 (ct , a∗ )]γ [E(a∗ )]1−γ ≥[α̂(ct , a∗ )]γ [E(a∗ )]1−γ = U (ct , a∗ ).

Edge Device 5 (Jetson Agx Orin)

Rt = U (ct , a ) − U (ct , a) ∗

(9)

γ

1−γ

The second inequality is by the fact that aγ − bγ < a − b when γ ∈ (0, 1) and a > b > 0. Obviously, µt−1 (ct , a) + βt σt−1 (ct , a) > α̂(ct , a) holds by Eq. 7. The last inequality is also by Eq. 7. Since βt is a non-decreasing variable with t, it holds that 3−γ 3−γ 2 2 Rt2 = 4Emin βt σt−1 (ct , a) ≤ 4Emin βT σ 2 σ −2 σt−1 (ct , a) 3−γ 4Emin βT 2 log(1 + σ −2 σt−1 (ct , a)). log (1 + σ −2 )

(12)

2 The last inequality holds by σ −2 σt−1 (ct , a) log(1 + σ −2 ) ≤ −2 −2 2 σ log(1 + σ σt−1 (ct , a)). Due to the information theory [32], the information gain about α̂(ct , a) for the decision a ∈ A can be measured by the mutual information I(α(ct , a); α̂(ct , a)) between α̂(ct , a) and the actual observation α(ct , a). Assuming α(ct , a) − α̂(ct , a) ∼ N (0, σ 2 ), it holds by induction that T

I(α(ct , a); α̂(ct , a)) =

Figure 5: DEFRAG prototype testbed.

(11)

1−γ

= [α̂(ct , a )] [E(a )] − [α̂(ct , a)] [E(a)] p ≤ ([µt−1 (ct , a) + βt σt−1 (ct , a)]γ − [α̂(ct , a)]γ )[E(a)]1−γ p ≤ (µt−1 (ct , a) + βt σt−1 (ct , a) − α̂(ct , a))[E(a)]1−γ p 1−γ ≤ 2Emin βt σt−1 (ct , a),

≤

Edge Devices 2-4 (Jetson Orin Nano)

(10)

∗

γ

Edge Device 1 (S25 Ultra)

(8)

where Emin = mina {E(a)}. The second inequality is by Eq. 7. Then, the regret Rt is ∗

Edge Device 6 (RTX 5090)

1X 2 log(1 + σ −2 σt−1 (ct , a)). (13) 2 t=1

Further, by the Cauchy-Schwarz inequality, we have s r 3−γ XT XT 4Emin βT λ I T Rt ≤ T Rt2 ≤ , t=1 t=1 log (1 + σ −2 )

(14)

where λI = maxct ,a I(α(ct , a); α̂(ct , a)) and is upper bounded by O(T d(d+1)/(2v+d(d+1)) log T ) with Matérn kernals according to [29].√The cumulative regret of Algorithm 1 is upper bounded by O( T λI ), which completes the proof.

glasses. Edge Device 5 is an NVIDIA Jetson AGX Orin, representing higher-capacity platforms typical of autonomous robotics. Edge Device 6 is a workstation with an NVIDIA RTX 5090 GPU, emulating a cellular base station or edge server. All devices are connected via Wi-Fi, allowing evaluation of network bottlenecks and peer-to-peer communication. In DEFRAG, each device acts both as server and client: as a server, it hosts a local vector database and provides SLM inference services via Flask-based RESTful APIs [33] and Ollama APIs [34]; as a client, it accesses data and computational resources from peer devices. To simulate user-specific knowledge preferences and non-uniform data distribution, we partition Wikipedia pages from each of the NQ and HotpotQA datasets into ten balanced clusters using TF-IDF vectors and a balanced K-means algorithm. Each of the five peer devices stores two clusters, while Device 6 stores all clusters and serves as a fallback for retrieval. The action space is defined by |M| = 12 and |E| = |R| = |D| = 11, yielding |X | = 12 × 113 ≈ 1.6 × 104 possible actions. Given the additional variability from different query contexts, such complexity underscores the necessity of a dataefficient learning mechanism. Initial efficiency weights wm and wr are set to 0.5, as there is no strong prior evidence favoring either model size or input length in terms of cost efficiency. Given that accuracy is the primary long-term objective, we initialize the accuracy weight γ to 0.8. These weights are updated every 20 queries based on recent performance. The Gaussian Process model is also updated every 20 rounds, using up to the latest 1500 samples to balance computational overhead and prediction accuracy. Unless specified, lines show cumulative averages up to each round, and shaded areas indicate the difference between cumulative and moving averages over the most recent 500 rounds. B. Experimental Result

V. E XPERIMENTAL E VALUATION A. Experimental Setup Our prototype testbed, shown in Figure 5, comprises six heterogeneous edge devices to emulate realistic edge deployment scenarios. Edge Device 1 is a Samsung S25 Ultra smartphone, representing mobile clients with limited resources. Edge Devices 2 to 4 are NVIDIA Jetson Orin Nano boards, emulating lightweight inference nodes such as laptops or AR

We evaluate DEFRAG and all baseline methods on the heterogeneous prototype testbed. For the single-hop NQ dataset, baseline SLMs and LLMs are evaluated with standard RAG; for the multi-hop HotpotQA dataset, GraphRAG is used to enable relational reasoning, as explained in Section II-B. Each method is assessed on 500 QA pairs using a welltrained optimizer (with analysis provided in the following section). Evaluation metrics include answer accuracy, total

9

Accuracy

Cost (Edge + Cloud)

Cost (All Tiers)

Cost (Cloud Only)

) B B 5B (E) (A 1.7 -14 G G -23 A M A M L M R R L L S F F S L DE DE Methods

5

Cost ($)

74.2 3.02 3.02 3.02

0

1.23 1.18 0.06

81.4

5

×10−3

83.7 3.84 3.84 3.84

67.8 1.54 1.48 0.06

61.5 0.80 0.59 0.06

61.6 0.95 0.95 0.00 76.4

76.6 0.66 0.42 0.06

0

1.29 1.29 0.01

100

×10−3

HotpotQA Dataset 61.8

0

0.49 0.01 0.01

Accuracy (%)

100

47.2 0.33 0.00 0.00

NQ Dataset

0

Tokens/s

Figure 6: Comparison of accuracy and cost between DEFRAG and centralized SLM/LLM baselines augmented with RAG. The top panel presents results on the NQ dataset using standard RAG, while the bottom panel shows results on the HotpotQA dataset using GraphRAG.

C. Ablation Studies

20 10 Centralized

0 0

67.8% and 81.4%, narrowing the gap with LLM-235B from 12.7% and 7.1% to 6.4% and 2.3%, respectively. For cost, DEFRAG achieves reductions of up to 98.4% compared to LLM-235B when mutual resource sharing is enabled among peer devices. In cost-oriented mode, DEFRAG (E) maintains accuracy comparable to SLM-14B baseline, while reducing total cost by up to 48.8% when all device models (1.7B to 235B) are included, and by 67.4% when considering only the edge and cloud server models (14B and 235B). As shown in Figure 7, DEFRAG demonstrates strong scalability as concurrency increases. While the centralized baseline on a single edge server quickly saturates and incurs queuing delays with more users, DEFRAG distributes retrieval and inference across heterogeneous devices, supporting near-linear throughput growth. In our experiments, DEFRAG achieves up to 97.8% higher peak throughput than the centralized approach, with further gains as more edge devices join. These results confirm that while SLMs still lag behind LLMs in accuracy with RAG, DEFRAG substantially narrows this gap, reduces costs, and improves scalability by leveraging heterogeneous edge resources to provide decentralized and democratized LLM services from cloud to crowd.

5

15 25 Concurrent Users

DEFRAG

35

45

Figure 7: Comparison of average throughput (tokens/s) under varying numbers of concurrent users for centralized and our DEFRAG (decentralized) system architecture.

cost (covering inference and network overhead) to reflect cost efficiency, and system scalability. Figure 6 compares DEFRAG with static and centralized SLM and LLM baselines on both datasets, evaluating both accuracy and total cost. Baselines use SLMs and LLM with corresponding RAG parameters, and accuracy is measured accordingly. Total cost includes model inference (calculated using official Qwen API pricing3 ) and network overhead (estimated from AWS public data transfer rates4 ). For SLM1.7B and SLM-14B baselines, inference is performed locally on the edge server, with costs including computation and user-to-device data transfer. The LLM-235B baseline is cloudhosted, with similar cost components. For DEFRAG, costs include computation, user-to-device data transfer, and network transfer for coordination and peer data exchange. Experimental results show that under the accuracy-oriented configuration (DEFRAG (A), αC = 0.99), the accuracy gap between edge-based SLMs and cloud-based LLMs is significantly reduced. On two benchmark QA datasets, edge accuracy improves from 61.5% and 76.6% (SLM-14B) to 3 https://www.alibabacloud.com/help/en/model-studio/models 4 https://aws.amazon.com/ec2/pricing/on-demand/

We first evaluate the effect of different initial exploration steps (T0 ) on the convergence of DEFRAG in terms of accuracy and efficiency. Experiments are conducted over 5,000 query rounds (10 iterations per dataset). Due to dynamic adjustment of efficiency weights, we report model efficiency (Etm ) and input efficiency (EtR ), as shown in Figure 8. Among the tested settings, model efficiency peaked at T0 = 20 for both datasets, as the system favored smaller models, resulting in lower accuracy and limited convergence. This illustrates the risk of insufficient exploration, which may introduce bias and hinder convergence. At T0 = 100, accuracy approached that of the cloud-scale SLM baseline (SLM14B), though efficiency could still improve. For T0 ≥ 200, both accuracy and efficiency converged. After about 2,000 iterations, accuracy stabilized, while model selection gradually shifted toward smaller models to further improve efficiency. Ultimately, model efficiency converged with continued iterations, balancing accuracy and resource use. Based on these results, we select T0 = 200 as the final setting. Notably, NQ accuracy increased steadily, while HotpotQA showed greater fluctuations with varying exploration steps. This is because HotpotQA relies more on common-sense knowledge, allowing even partial relevant information to support correct answers [35]. In contrast, NQ queries often involve obscure facts outside pre-training data, making them more dependent on retrieval [36], and resulting in a lower baseline accuracy than HotpotQA. Table V shows how different context feature sets affect average accuracy and efficiency on the NQ and HotpotQA datasets. By incorporating features that capture query structure and complexity, the optimizer can dynamically adjust its actions. Richer representations of relevance score distributions help the optimizer better interpret queries and select actions.

10

T0=20

Model Efficiency (Etm)

1.0 Accuracy (αt)

0.6 0.4

NQ Dataset

0.8 0.6 0

HotpotQA Dataset 1000 2000 3000 4000 5000 Query Round (t)

T0=100 NQ Dataset

0.5 0.0 1.0

HotpotQA Dataset

0.5 0.0 0

1000 2000 3000 4000 Query Round (t)

T0=200

T0=400

1.0 Input Efficiency (EtR)

SLM-14B with RAG (Baseline Accuracy)

0.5 0.0 1.0

HotpotQA Dataset

0.5 0.0 0

5000

NQ Dataset

1000 2000 3000 4000 Query Round (t)

5000

Feature set c

m

α (%)

R

E

E

NQ Dataset Mean+Std Mean+Std+Num Mean+Std+Num+Min+Max

60.3 60.8 61.5

0.24 0.28 0.32

0.60 0.55 0.52

Decision Time (s)

Original

72.9 75.4 76.6

PeriodicFit

0.28 0.19 0.41 DEFRAG

2000 3000 Query Round (t)

Centralized

4000

5.0 2.5

1.57s 0.56s

LowEmbedded power edge edge

Edge server

0.0

DEFRAG(A) DEFRAG(E)

Trace-level overhead

Figure 10: Model residency and switching overhead across serving-peer routes.

0.55 0.42 0.52

10.0

1000

2.5

Model residency

20.0

0.0 0

Warm

5.0

0.0

Hotpot Dataset Mean+Std Mean+Std+Num Mean+Std+Num+Min+Max

Cold

Penalty (s)

Table V: Comparison of average accuracy and efficiency across context features (c) on HotpotQA and NQ datasets.

Load time (s)

Figure 8: Comparison of DEFRAG accuracy and efficiency across initial exploration steps (T0 ) on NQ and HotpotQA datasets.

5000

Figure 9: Comparison of decision delay (retrieval plus optimizer prediction) across different optimization strategies. While mean and standard deviation reflect central tendency and dispersion, they may not distinguish between a few strong matches and many moderate ones. Including count (num) indicates the density of relevant information, while adding min and max reveals the score range and the presence of outliers. Figure 9 compares decision delay, including retrieval and optimizer prediction time, across different acceleration strategies. Without optimization (Original), decision delay is significantly higher, exhibiting O(N 3 ) complexity. Periodic GP fitting (PeriodicFit) mitigates delay to some extent but does not eliminate the exponential growth. Combining periodic GP fitting with window restriction (DEFRAG) substantially reduces decision delay, resulting in near-linear time complexity. The centralized baseline (Centralized) measures delay on an edge server without network overhead. DEFRAG achieves a similar delay, despite additional network latency from distributed peer database access, indicating that network communication is not a bottleneck under stable testbed conditions. Table VI compares seven HotpotQA settings under the same prototype deployment. Each setting uses 500 queries and

reports retrieval-plus-generation latency. Fixed-CollabRAG uses fixed RAG parameters and a fixed 14B model. Adaptive Retrieval changes only the RAG parameters. Adaptive Model changes only model selection. Both improve over Fixed-CollabRAG, but both remain below DEFRAG(A). This shows that RAG parameter selection and model selection are both needed. Compared with Fixed-CollabRAG, DEFRAG(A) raises accuracy from 77.2% to 81.4% and lowers normalized cost from 0.312 to 0.292. DEFRAG(E) keeps similar accuracy and lowers normalized cost to 0.158. Heuristic-Fastest and Heuristic-Strongest show the two single-objective choices: lower cost with lower accuracy, or higher accuracy with higher cost. Table VI therefore shows how DEFRAG balances accuracy, latency, and cost. D. Robustness and Generalization Analysis We also test DEFRAG under four controlled settings that affect edge-collaborative RAG: model loading, mobile route instability, skewed data placement, and another QA domain. These tests examine whether DEFRAG remains effective when model execution, routes, data placement, or the task domain changes. The prototype setting is the reference, and each experiment changes one factor at a time. Figure 10 quantifies model loading and switching overhead in dynamic model selection. Figure 10(a) profiles cold and warm execution with HotpotQA retrieval prompts on three route classes. Low-power edge routes represent lightweight peers such as Jetson Orin Nano boards and mobile terminals. Embedded edge routes represent higher-capacity embedded AI platforms, such as Jetson AGX Orin-class devices used in robotics. Edge-server routes represent workstation-class edge

11

Table VI: HotpotQA 500-query system comparison. RAG parameters

Model policy

Accuracy

Latency

Norm. cost

Adaptive Adaptive Fixed Adaptive Fixed Fixed Fixed

Edge adaptive Edge adaptive Local 14B Local 14B Edge adaptive Rule-based Rule-based

81.4% 76.6% 77.2% 79.6% 78.0% 60.8% 84.2%

6.233s 7.490s 3.322s 3.057s 6.102s 5.577s 6.355s

0.292 0.158 0.312 0.305 0.295 0.078 1.000

100

1.23

Cost

50

1.5

1.22

75

0.96 79.6

79.6

69.4

25 0

Stable

With fallback

Mean

×10−3

No fallback

1.0 0.5

15

0.0

13.4

12.7

10

7.5

6.6

13.3

7.5

5 0

Stable

With fallback

Description Accuracy-focused setting Efficiency-focused setting Fixed retrieval, local generation Adaptive RAG parameters only Adaptive model route only Fastest feasible route Strongest feasible route

Full

P95

No fallback

Peer evidence (%)

Accuracy (%)

Accuracy

Latency (s)

DEFRAG(A) DEFRAG(E) Fixed-CollabRAG Adaptive Retrieval Adaptive Model Heuristic-Fastest Heuristic-Strongest

Cost ($)

Setting

Fallback

Partial

None

100 75 50 25 0

59.6

53.2

40.4

43.2

With fallback

No fallback

100.0

Stable

Figure 11: Paired mobile route stress on HotpotQA: accuracy/cost, latency, and evidence availability.

Accuracy

Remote fetch

Latency

2 50

80.2 76.6 2.03

80.4

90.5 2.24

81.4

1

61.3 0.85

0

Balanced

Capacityaware

Requesteraware

Latency (s)

Rate (%)

100

0

Figure 12: Impact of skewed data placement on accuracy, remote fetch, and retrieval latency.

servers such as the RTX 5090 node. Cold execution loads a non-resident model, while warm execution reuses a resident or cached model. Cold loading takes 2.16 s on low-power edge routes, 4.92 s on embedded edge routes, and 2.25 s on edge-server routes. Warm reuse stays below 0.28 s in all three classes. This shows that model residency has a direct effect on action cost. Figure 10(b) replays the HotpotQA action traces of DEFRAG(A/E) with the measured loading and switching penalties. Loading and switching add 1.566 s/query to DEFRAG(A), or 23.09% of retrieval-plus-generation latency. They add 0.560 s/query to DEFRAG(E), or 6.39%. The lower penalty of DEFRAG(E) comes from more reuse of resident models, with a 91.4% warm-resident ratio compared with 69.3% for DEFRAG(A). This result shows that dynamic model selection has measurable I/O cost, and that efficiency-oriented selection can reduce this cost by reusing resident or cached models more often. Figure 11 evaluates DEFRAG under mobile route instability. We use paired HotpotQA runs with the same queries, balanced placement, and DEFRAG(A) action sequence. The Stable run has no injected route loss. In the stress runs, the requester is a mobile device, and its local shard remains available. We inject a 30% timeout/drop rate only on non-local peer data and

model routes. In the fallback setting, the edge server serves as a redundant data and model route. In the no-fallback setting, the system skips missing peer evidence, and an unavailable model route is replaced by local 1.7B generation. Figure 11(a) reports accuracy and per-query cost. The fallback setting keeps the same 79.6% accuracy as Stable, with similar cost ($0.001225 vs. $0.001226 per query). The nofallback setting has lower cost, but the reduction comes from degraded service: missing peer evidence is skipped, local 1.7B generation is used more often, and accuracy drops to 69.4%. Figure 11(b) shows that fallback increases mean retrievalplus-generation latency from 6.618 s to 7.513 s. This is the measured latency cost of route recovery. Figure 11(c) explains why accuracy is preserved. With fallback, all queries keep full peer evidence: 40.4% through the original peer routes and 59.6% through edge-server fallback. Without fallback, only 43.2% of queries keep full peer evidence. Another 53.2% keep partial peer evidence, and 3.6% receive no peer evidence. In the same no-fallback run, 30.4% of queries also use local 1.7B generation after a modelroute failure. These results show that edge-server redundancy preserves evidence completeness and model-route continuity under mobile-like route stress. Figure 12 evaluates non-uniform data placement. We compare three placements. Balanced uses the original group assignment. Capacity-aware places frequently requested evidence groups on higher-capacity peers. Requester-aware places evidence groups closer to the requester origins that most often need them. All runs use the same HotpotQA queries and the same DEFRAG(A) selected actions. Thus, the comparison isolates the effect of data placement. The three placements give comparable answer accuracy, indicating that collaborative retrieval still finds usable evidence when data placement is non-uniform. We then measure the communication overhead of this retrieval process from the requester origin. Capacity-aware placement is a capacity-

12

Table VII: Comparison of accuracy and normalized cost on the HarryPotter QA task.

No retrieval RAG GraphRAG DEFRAG(E) DEFRAG(A) Cloud GraphRAG

RAG

Model

Accuracy

Cost

None Fixed Fixed Adaptive Adaptive Fixed

Local 14B Local 14B Local 14B Edge adaptive Edge adaptive Cloud 235B

40.6% 47.2% 56.0% 47.8% 53.4% 62.2%

0.0680 0.2290 0.4062 0.0341 0.0802 1.0000

Accuracy (%)

No retrieval DEFRAG(A) RAG

DEFRAG(E) GraphRAG 235B GraphRAG

100

50

0

−1

0

10 10 Normalized cost per query

(a) Accuracy–cost tradeoff.

Low-power edge Embedded edge Edge server

Route share (%)

Setting

100

50

0

A E DEFRAG variant

(b) Route distribution.

Figure 13: HarryPotter QA accuracy–cost tradeoff and route distribution.

oriented skewed baseline. It assigns frequently requested evidence groups to higher-capacity peers, yielding a 90.5% requester-origin remote-fetch rate and 2.24 s mean retrieval latency. Requester-aware placement uses requester demand as a placement signal. It places evidence closer to the requester origins that most often need it, reducing remote fetch to 61.3% and mean retrieval latency to 0.85 s, compared with 76.6% and 2.03 s for Balanced. Together, these results show that DEFRAG preserves answer quality without a perfectly balanced partition. The requester-aware result also demonstrates the potential of using requester demand to reduce communication overhead under skewed storage. Finally, we add HarryPotter QA as a domain-specific generalization task. The task uses a local seven-book narrative corpus with reference QA pairs. It adds domain-specific evidence and narrative entities to the benchmark suite. Reference answers allow a controlled accuracy–cost comparison. Table VII and Fig. 13(a) compare HarryPotter QA operating points across no retrieval, fixed RAG, fixed GraphRAG, DEFRAG(A/E), and cloud GraphRAG. The strongest local baseline is 14B GraphRAG, which reaches 56.0% accuracy at normalized cost 0.4062. DEFRAG(A) reaches 53.4% accuracy at cost 0.0802, or about 95% of the local 14B GraphRAG accuracy at about one fifth of its cost. It also reaches 86% of the cloud GraphRAG accuracy at about 8% of the cloud cost. DEFRAG(E) moves further toward efficiency: its normalized cost is 0.0341, about 15% of fixed RAG cost and 3% of cloud GraphRAG cost. Figure 13(b) explains this tradeoff through the selected model tiers. DEFRAG(A) routes 76% of queries to the 14B edge-server tier, while DEFRAG(E) routes 79% to the 1.7B low-power tier. This mirrors the NQ and HotpotQA results: the

accuracy objective uses stronger models more often, while the efficiency objective keeps a lower-cost operating point. The HarryPotter result therefore supports DEFRAG’s accuracy– cost tradeoff on an additional domain-specific QA corpus. VI. R ELATED W ORK Cost-Effective Large Language Model Usage. Reducing LLM deployment costs is an active research focus. Techniques like model quantization [37], [38] and pruning [39], distillation [40] cut costs but may reduce performance and require specialized hardware. Other strategies include caching LLM responses for frequent queries [41], [42] or reusing keyvalue states during inference [43], [44]. Additionally, model multiplexing dynamically selects an appropriate model based on real-time query analysis [3], [45], [46]. However, these approaches primarily focus on reducing resource consumption during inference. They do not adequately address deploying LLMs effectively on edge devices with idle resources, particularly in achieving performance comparable to cloud-scale models. Retrieval-Augmented Generation (RAG) Enhancements. RAG integrates external textual knowledge to enhance language model outputs [6], [47]. Early implementations, termed Naive RAG, primarily employ sparse retrieval methods like TF-IDF and BM25, directly concatenating passages into prompts [23]. Later developments, or Advanced RAG, incorporate dense retrieval such as Dense Passage Retrieval (DPR) [18] and iterative refinement introduce iterative feedback and complexity-based query refinement, improving multi-hop reasoning [5], [46], [48], [49]. Modular RAG frameworks emphasize interchangeable modules for retrieval, reranking, and generation, facilitating hybrid strategies and external integrations [50]. Furthermore, Graph-based RAG methods utilize knowledge graph structures to enhance relational reasoning and structured retrieval for complex tasks [19], [51], [52]. While RAG methods can enhance accuracy, they often do so at the cost of increased resource consumption. Identifying effective trade-offs and selecting suitable RAG strategies for efficient deployment on edge nodes remains a key challenge. LLM Resource Allocation Strategies in Edge Computing. Research on deploying LLMs in edge computing emphasizes optimal resource allocation by balancing latency, energy efficiency, and computational capabilities through techniques like task offloading, resource scheduling, and cloudedge collaboration [1], [53]. Multi-stage inference, for example, initially employs smaller models deployable on edge devices to handle general queries, escalating to larger LLMs when confidence is insufficient [2]. Active inference strategies dynamically offload resource-intensive LLM inference tasks across multiple edge nodes based on network bandwidth and processing capabilities [4], [11]. Distributed on-device inference systems such as PRIMA.CPP demonstrate fast 30B– 70B LLM inference on heterogeneous low-resource home clusters [54]. Incorporating RAG techniques at the edge further enhances LLM performance through federated multi-domain knowledge bases, adaptive retrieval and inference strategies, and by explicitly integrating logical relationships derived from

13

knowledge graphs into context [13], [14], [20], [55]. Edge computing with RAG has also been explored in emerging domains such as next-generation networks [10]. Nevertheless, due to memory constraints on edge devices, knowledge coverage remains significantly limited. Therefore, efficient collaboration among decentralized edge nodes remains an open challenge and is crucial for achieving cloud-scale LLM performance. Prior work typically optimizes retrieval or generation separately at the edge. By contrast, DEFRAG jointly optimizes both components, explicitly modeling the interaction between retrieval strategies and generation model capabilities. This unified system architecture enables more effective adaptation to resource constrained edge devices. VII. F UTURE W ORK This work takes an important step toward scalable and accessible LLM services at the edge. Looking ahead, several promising directions merit further exploration. Advancing dynamic integration and grouping mechanisms will be critical for supporting evolving edge networks, including automatic onboarding of new devices and efficient handling of cold-start scenarios. As knowledge bases and network scale increase, optimizing local caching and update strategies for lightweight knowledge graphs and vector stores will be essential; adaptive cache management that responds to storage constraints and access patterns will be a valuable avenue. In addition, advancing inference efficiency and seamless system integration on mobile devices will remain crucial to enable practical applications facing users. VIII. C ONCLUSION In this paper, we have introduced DEFRAG, a decentralized edge collaboration architecture for RAG that enables accurate, cost-efficient, and scalable LLM deployment at the edge. By compressing and distributing lightweight knowledge graphs for retrieval and dynamically tuning generation parameters with Bayesian online learning, DEFRAG narrows the accuracy gap between edge-based SLMs and cloud-based LLMs while reducing costs and improving scalability. By enabling collaborative sharing of models and knowledge among the crowd, DEFRAG advances the democratization of LLM services from cloud to crowd. R EFERENCES [1] G. Qu, Q. Chen, W. Wei, Z. Lin, X. Chen, and K. Huang, “Mobile edge intelligence for large language models: A contemporary survey,” IEEE Communications Surveys & Tutorials (ICST), 2025. [2] Y. Wang, K. Chen, H. Tan, and K. Guo, “Tabi: An efficient multi-level inference system for large language models,” in European Conference on Computer Systems (EuroSys), 2023. [3] L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,” Transactions on Machine Learning Research (TMLR), 2025. [4] M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “Edgeshard: Efficient llm inference via collaborative edge computing,” IEEE Internet of Things Journal (IOT), 2024. [5] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-rag: Learning to retrieve, generate, and critique through self-reflection,” in International Conference on Learning Representations (ICLR), 2024.

[6] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2023. [7] S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” in IEEE High Performance Extreme Computing Conference (HPEC), 2023. [8] K. Zhao, P. Goyal, M. Alizadeh, and T. E. Anderson, “Scalable tail latency estimation for data center networks,” in USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2023. [9] B. Reidys, P. Zardoshti, Í. Goiri, C. Irvene, D. S. Berger, H. Ma, K. Arya, E. Cortez, T. Stark, E. Bak et al., “Coach: Exploiting temporal patterns for all-resource oversubscription in cloud platforms,” in International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2025. [10] X. Huang, Y. Tang, J. Li, N. Zhang, and X. S. Shen, “Toward effective retrieval augmented generative services in 6g networks,” IEEE Network, 2024. [11] Y. He, J. Fang, F. R. Yu, and V. C. Leung, “Large language models (llms) inference offloading and resource allocation in cloud-edge computing: An active inference approach,” IEEE Transactions on Mobile Computing (TMC), 2024. [12] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025. [13] P. Shojaee, S. S. Harsha, D. Luo, A. Maharaj, T. Yu, and Y. Li, “Federated retrieval augmented generation for multi-product question answering,” in International Conference on Computational Linguistics (COLING), 2025. [14] T. Ouyang, G. Hong, K. Zhao, Z. Zhou, W. Wu, Z. Lv, and X. Chen, “Adarag: Adaptive optimization for retrieval augmented generation with multilevel retrievers at the edge,” in IEEE International Conference on Computer Communications (INFOCOM), 2025. [15] G. Gerganov and contributors, “llama.cpp: Port of facebook’s llama model in c/c++,” https://github.com/ggml-org/llama.cpp, 2023. [16] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., “Natural questions: a benchmark for question answering research,” Transactions of the Association for Computational Linguistics (TACL), 2019. [17] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “HotpotQA: A dataset for diverse, explainable multi-hop question answering,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. [18] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [19] D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson, “From local to global: A graph rag approach to query-focused summarization,” arXiv preprint arXiv:2404.16130, 2024. [20] T. Fan, J. Wang, X. Ren, and C. Huang, “Minirag: Towards extremely simple retrieval-augmented generation,” arXiv preprint arXiv:2501.06713, 2025. [21] OpenAI, “Introducing gpt-4.1 in the api,” https://openai.com/index/ gpt-4-1/, 2025. [22] Y. Wang, Z. Yu, W. Yao, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang et al., “Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization,” in International Conference on Learning Representations (ICLR), 2024. [23] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrievalaugmented generation for knowledge-intensive nlp tasks,” International Conference on Neural Information Processing Systems (NeurIPS), 2020. [24] Y. Zhu, J.-C. Gu, C. Sikora, H. Ko, Y. Liu, C.-C. Lin, L. Shu, L. Luo, L. Meng, B. Liu et al., “Accelerating inference of retrieval-augmented generation via sparse context selection,” in International Conference on Learning Representations (ICLR), 2025. [25] T. Lan, D. Kao, M. Chiang, and A. Sabharwal, “An axiomatic theory of fairness in network resource allocation,” in IEEE International Conference on Computer Communications (INFOCOM), 2010. [26] L. Li, W. Chu, J. Langford, and R. E. Schapire, “A contextual-bandit approach to personalized news article recommendation,” in International World Wide Web Conferences (WWW), 2010.

14

[27] N. Cesa-Bianchi, C. Gentile, and G. Zappella, “A gang of bandits,” International Conference on Neural Information Processing Systems (NeurIPS), 2013. [28] C. K. Williams and C. E. Rasmussen, Gaussian processes for machine learning. MIT press Cambridge, MA, 2006. [29] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger, “Gaussian process optimization in the bandit setting: No regret and experimental design,” in International Conference on Machine Learning (ICML), 2010. [30] A. D. Bull, “Convergence rates of efficient global optimization algorithms.” Journal of Machine Learning Research (JMLR), 2011. [31] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in International Conference on Learning Representations (ICLR), 2016. [32] T. M. Cover and J. A. Thomas, Elements of information theory. WileyInterscience, 1991. [33] A. Ronacher and the Pallets team, “Flask documentation,” https://flask. palletsprojects.com/, Pallets Projects, 2025. [34] O. Team, “Ollama api documentation,” https://github.com/jmorganca/ ollama/blob/main/docs/api.md, 2025. [35] D. Groeneveld, T. Khot, A. Sabharwal et al., “A simple yet strong pipeline for hotpotqa,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [36] B. Jin, J. Yoon, J. Han, and S. O. Arik, “Long-context llms meet rag: Overcoming challenges for long inputs in rag,” in International Conference on Learning Representations (ICLR), 2025. [37] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning (ICML), 2023. [38] G. Park, M. Kim, S. Lee, J. Kim, B. Kwon, S. J. Kwon, B. Kim, Y. Lee, D. Lee et al., “Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models,” in International Conference on Learning Representations (ICLR), 2023. [39] X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” International Conference on Neural Information Processing Systems (NeurIPS), 2024. [40] Y. Gu, L. Dong, F. Wei, and M. Huang, “Minillm: Knowledge distillation of large language models,” in International Conference on Learning Representations (ICLR), 2024. [41] B. Zhu, Y. Sheng, L. Zheng, C. Barrett, M. I. Jordan, and J. Jiao, “On optimal caching and model multiplexing for large model inference,” in International Conference on Neural Information Processing Systems (NeurIPS), 2023. [42] J. Li, C. Xu, F. Wang, I. M. von Riedemann, C. Zhang, and J. Liu, “Scalm: Towards semantic caching for automated chat services with large language models,” in International Symposium on Quality of Service (IWQoS), 2024. [43] Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan et al., “Cachegen: Kv cache compression and streaming for fast large language model serving,” in International Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM), 2024. [44] J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang, “Cacheblend: Fast large language model serving with cached knowledge fusion,” arXiv preprint arXiv:2405.16444, 2024. [45] S. Kim, K. Mangalam, S. Moon, J. Malik, M. W. Mahoney, A. Gholami, and K. Keutzer, “Speculative decoding with big little decoder,” International Conference on Neural Information Processing Systems (NeurIPS), 2023. [46] S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J.-C. Park, “Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity,” in Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024. [47] A. Singh, A. Ehtesham, S. Kumar, and T. T. Khoei, “Agentic retrievalaugmented generation: A survey on agentic rag,” arXiv preprint arXiv:2501.09136, 2025. [48] O. Yoran, T. Wolfson, O. Ram, and J. Berant, “Making retrievalaugmented language models robust to irrelevant context,” in International Conference on Learning Representations (ICLR), 2024. [49] S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling, “Corrective retrieval augmented generation,” arXiv preprint arXiv:2401.15884, 2024. [50] J. Jin, Y. Zhu, Z. Dou, G. Dong, X. Yang, C. Zhang, T. Zhao, Z. Yang, and J.-R. Wen, “Flashrag: A modular toolkit for efficient retrievalaugmented generation research,” in Companion Proceedings of the ACM on Web Conference, 2025.

[51] B. Peng, Y. Zhu, Y. Liu, X. Bo, H. Shi, C. Hong, Y. Zhang, and S. Tang, “Graph retrieval-augmented generation: A survey,” arXiv preprint arXiv:2408.08921, 2024. [52] L. Liang, Z. Bo, Z. Gui, Z. Zhu, L. Zhong, P. Zhao, M. Sun, Z. Zhang, J. Zhou, W. Chen et al., “Kag: Boosting llms in professional domains via knowledge augmented generation,” in Companion Proceedings of the ACM on Web Conference, 2025. [53] Y. Wang, C. Yang, S. Lan, L. Zhu, and Y. Zhang, “End-edge-cloud collaborative computing for deep learning: A comprehensive survey,” IEEE Communications Surveys & Tutorials (ICST), 2024. [54] Z. Li, T. Li, W. Feng, R. Xiao, J. She, H. Huang, M. Guizani, H. Yu, Q. Ho, W. Xiang, and X. Liu, “Prima.cpp: Fast 30–70B LLM Inference on Heterogeneous and Low-Resource Home Clusters,” in International Conference on Learning Representations (ICLR), 2026. [55] J. Li, C. Xu, L. Jia, F. Wang, C. Zhang, and J. Liu, “Eaco-rag: Edgeassisted and collaborative rag with adaptive knowledge update,” arXiv preprint arXiv:2410.20299, 2024.

Jiaxing Li is currently a Ph.D. candidate in the School of Computing Science at Simon Fraser University, British Columbia, Canada, under the supervision of Prof. Jiangchuan Liu. He received his B.Sc. degree in Computing Science from Simon Fraser University in 2023. His research interests include large language model applications, cloud edge computing, and distributed machine learning systems.

Hengzhi Wang (Member, IEEE) is currently serving as an assistant professor in the College of Computer Science and Software Engineering at Shenzhen University in Shenzhen, China. He obtained his B.S. degree in software engineering from Jilin University in Changchun, China, in 2017, and his Ph.D. degree in computer science from Jilin University in Changchun, China, in 2023. During his doctoral studies, he also worked as a visiting Ph.D. student at the School of Computing Science, Simon Fraser University in British Columbia, Canada. His research interests primarily focus on spatial crowdsourcing, federated learning, and privacy protection.

Feng Wang (Senior Member, IEEE) received both the Bachelor’s degree and Master’s degree in Computer Science and Technology from Tsinghua University, Beijing, China in 2002 and 2005, respectively. He received the PhD degree in Computing Science from Simon Fraser University, Burnaby, British Columbia, Canada in 2012. He is currently an Associate Professor in the Department of Computer and Information Science at the University of Mississippi, University, MS, USA. He is a Senior Member of IEEE. He is a recipient of IEEE ICME Quality Reviewer Award (2011), ACM BuildSys Best Paper Award (2018), ACM/IEEE IWQoS Best Paper Award Runner-up Award (2022), IEEE CloudCom Special Section Best Paper Award (2025), and IEEE INFOCOM Distinguished TPC Member Awards (2020-2022). He is a Technical Committee Member of Elsevier Computer Communications. He served as Program Vice Chair in International Conference on Internet of Vehicles (IOV) 2014, as TPC Co-chair in IEEE CloudCom 2017 for Internet of Things and Mobile on Cloud track, as Publication Co-chair in ACM/IEEE IWQoS 2024. He also serves as TPC member in various international conferences such as IEEE INFOCOM, IEEE/ACM IWQoS, ACM Multimedia, IEEE ICC, IEEE GLOBECOM and IEEE ICME.

15

Chi Xu received the B.Sc. degree in software engineering from Xidian University, Xi’an, China, and the M.Sc. degree in computing science from Simon Fraser University. He is currently pursuing his Ph.D. degree in computing science at Simon Fraser University, Canada. His research spans applied artificial intelligence and networked systems, with a recent focus on Physical AI for challenging field environments.

Danyang Song is currently a Ph.D. student in the Department of Electrical and Electronic Engineering at the University of Hong Kong. Prior to HKU, he received his MSc degree in Computing Science from Simon Fraser University in 2023 and Dual B.E degree in Computer Science from Zhejiang University and Simon Fraser University in 2018. His research interests include Artificial Intelligence, and edge computing under industrial scenarios.

Ruixiao Zhang now is a Research Scientist at ByteDance Multimedia Lab in San Diego. Previously, he was a postdoctoral researcher at the University of Illinois Urbana-Champaign (supervised by Prof. Klara Nahrstedt), the University of Hong Kong (supervised by Prof. Chuan Wu), and the Hong Kong University of Science and Technology (supervised by Prof. Bo Li). He also collaborates closely with Prof. Jiangchuan Liu at Simon Fraser University. He received his Ph.D. in Computer Science from Tsinghua University in 2022, advised by Prof. Lifeng Sun, and his B.E. in Electronic Engineering from Tsinghua University in 2017. His research focuses on networking-driven optimization of multimedia systems, with an emphasis on robust and efficient video delivery over realworld networks under packet loss, bandwidth constraints, and time-varying conditions.

Edith C.H. Ngai (Senior Member, IEEE) is currently an Associate Professor in the Department of Electrical and Electronic Engineering, The University of Hong Kong. Before joining HKU in 2020, she was an Associate Professor in the Department of Information Technology, Uppsala University, Sweden. Her research interests include Internet-of-Things, edge intelligence, and smart cities. She was a VINNMER Fellow awarded by Swedish Governmental Research Funding Agency VINNOVA in 2009. Her co-authored papers received a Best Paper Award in QShine 2023, Best Paper Runner-Up Awards in IEEE IWQoS 2010 and ACM/IEEE IPSN 2013, and Best paper candidate in ACM BuildSys 2024. She was an Area Editor of IEEE Internet of Things Journal from 2020 to 2022. She is currently an Associate Editor in IEEE Transactions of Mobile Computing, IEEE Network, and IEEE Transactions of Industrial Informatics. She served as a program chair in ACM womENcourage 2015 and a TPC cochair in IEEE SmartCity 2015, IEEE GreenCom 2022, IEEE/ACM IWQoS 2024, IEEE CloudCom 2025. She received a Meta Policy Research Award in Asia Pacific in 2022. She was selected as one of the N²Women Stars in Computer Networking and Communications in 2022. She is a Distinguished Lecturer in IEEE Communication Society in 2023-2024. She is an ACM Distinguished Member in 2025.

Jiangchuan Liu (Fellow, IEEE) is a Professor in the School of Computing Science, Simon Fraser University, British Columbia, Canada. He is a Fellow of Royal Society of Canada, a Fellow of The Canadian Academy of Engineering, an IEEE Fellow, and an NSERC E.W.R. Steacie Memorial Fellow. He was an EMC-Endowed Visiting Chair Professor of Tsinghua University (2013-2016) and is a Distinguished Guest Professor of Tsinghua Shenzhen International Graduate School (2022-). In the past he worked as an Assistant Professor at The Chinese University of Hong Kong (2003-2004) and as a research fellow at Microsoft Research Asia. He received the BEng degree (Magna cum laude) from Tsinghua University in 1999, and the PhD degree from The Hong Kong University of Science and Technology in 2003, both in computer science. He is a co-recipient of the inaugural Test of Time Paper Award of IEEE INFOCOM (2015), IEEE ICDCS Distinguished Paper Award (2024), ACM SIGMM TOMCCAP Nicolas D. Georganas Best Paper Award (2013), and ACM Multimedia Best Paper Award (2012). His research interests include intelligent multimedia computing and networking, cloud and edge computing, social networking, online gaming, and wireless mobile and space networking. He has served on the editorial boards of IEEE/ACM TON, IEEE TNSE, TMM, TBD, COMST, and IOTJ. He was a Steering Committee member of IEEE TMC, and Steering Committee Chair of IEEE/ACM IWQoS (2015-2017). He was TPC Chair of IEEE INFOCOM 2021 and General Chair of INFOCOM 2024.

Related documents

• Integrity of peer-to-peer distributed LLM inference under malicious nodes#394351
• AI-Native Orchestration in the 6G Continuum: Evolving Operator Platforms with Agentic AI#985794
• Edge deployment of cloud-originated machine learning and artificial … — Armada Systems Inc. (US12033006B1)#607542
• M2M Networking Architecture for Data Transmission and Routing#116543
• LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication#10325