DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments

DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments Reza Farahani∗ , Zoha Azimi Ourimi† , Mario Colosi‡ , Lauri Lovén§ , Christian Timmerer† , Schahram Dustdar∗ ∗ Distributed Systems Group (DSG), TU Wien, Austria † Christian Doppler Laboratory ATHENA, Department of Information Technology (ITEC), University of Klagenfurt, Austria ‡ MIFT Department, University of Messina, Italy

I. I NTRODUCTION Large language models (LLMs) power modern AI services like chatbots and code-generation systems, processing queries with diverse structures, reasoning at varying levels of complexity, and meeting varying latency requirements [1]. In edge environments, this challenge is amplified by hardware heterogeneity, where model instances, spanning different families, parameters, and quantizations, show distinct latency-qualityresource trade-offs [2]–[4]. Thus, LLM orchestration becomes a fine-grained, per-query decision problem over model-devicequantization configurations rather than a simple model or device selection task [5], [6]. To quantify these interactions, we evaluate representative configurations on two edge devices: a constrained Raspberry Pi 4 (RP) and a more capable NVIDIA Jetson ORIN AGX (JOA). Each configuration is denoted as (x, y), where x is the model size (in billions of parameters) and y is the quantization level. Fig. 1 reports average accuracy and response time for Mathematics [7] and TruthfulQA [8] queries, revealing three key observations: 1) Model-family impact: On JOA (Fig. 1b), Gemma3(12, 4) has the highest accuracy on Mathematics, whereas

Math. TfullQA

0.8

Accuracy

0.8 0.6

Accuracy

Math. TfullQA

0.6

0.4

0.4 0.2 0 Gemma3 Gemma3 Gemma3 Tinyllama (0.27, 4) (0.27, 16) (1, 4) (1.1, 4)

0.2 0

Gemma3 (12,4)

Qwen3 (0.6,4)

Qwen3 (0.6, 16)

(a) Average accuracy on RP.

(b) Average accuracy on JOA. Response Time (s)

Abstract—Large language model (LLM) services increasingly process heterogeneous queries with diverse latency, accuracy, and resource requirements. While edge deployment reduces response time, the heterogeneity of devices and the diversity of model families, parameter scales, and quantization levels make efficient LLM query orchestration challenging. This paper introduces DRLM, a Deep Reinforcement Learning-based LLM query orchestration framework in edge environments. DRLM integrates two lightweight predictors: (i) a class-conditioned quality estimator that maps queries to semantic categories and infers model performance, and (ii) a feature-driven latency predictor that estimates inference time across model-device configurations. These predictions, combined with system state, feed a factorized Proximal Policy Optimization (PPO) agent that performs stateaware orchestration decisions. To enable data-driven orchestration, we construct a large-scale benchmarking dataset with 223 835 measurements spanning 1258 queries, 6 query classes, 8 model families (32 deployed instances), 5 quantization levels, and heterogeneous edge devices. Evaluation on a 64-node edge cluster and comparison with three baselines and two state-of-the-art methods show that DRLM reduces inference latency by up to 51 % and queuing delay by up to 67 %, while incurring at most 8 % accuracy loss. It improves latency under increasing workloads up to 61.4 %, demonstrating robust and stable orchestration. Index Terms—LLM Inference; Edge Computing; Deep Reinforcement Learning; Query Orchestration; Edge Clusters.

Response Time (s)

arXiv:2609.00442v1 [cs.DC] 31 Aug 2026

§ Center for Ubiquitous Computing, University of Oulu, Finland

300

Math. TfullQA

200 100 0 Gemma3 Gemma3 Gemma3 Tinyllama (0.27, 4) (0.27, 16) (1, 4) (1.1, 4)

(c) Average response time on RP.

300

Math. TfullQA

200 100 0

Gemma3 (12,4)

Qwen3 (0.6,4)

Qwen3 (0.6, 16)

(d) Average response time on JOA.

Figure 1: Latency-quality of LLM configurations on RP and JOA edge devices for Mathematics and TruthfulQA queries.

Qwen3(0.6, 4) and (0.6, 16) outperform others on TruthfulQA. This indicates that model architecture, not just size, determines task-specific performance. 2) Scaling impact: On RP (Fig. 1a), Gemma3(1, 4) improves Mathematics accuracy over (0.27, 4) but degrades performance on TruthfulQA. Similarly, on JOA, Gemma3(12, 4) improves Mathematics but does not consistently dominate across tasks. Thus, scaling benefits are workload-dependent rather than monotonic. 3) Quantization impact: On RP (Figs. 1a-1c), Gemma3(0.27, 16) improves TruthfulQA accuracy over (0.27, 4) but increases latency. On JOA (Fig. 1d), Qwen3(0.6, 16) incurs higher latency than (0.6, 4) with marginal accuracy gains. These results show that quantization effects are highly configuration- and device-dependent, and must be selected jointly with model and system characteristics rather than assumed to improve latency or accuracy. Implication: Effective LLM orchestration on edge clusters needs joint, state-aware modeling of query semantics, model configurations, and runtime dynamics, which cannot be captured by static or single-objective policies. Existing approaches address LLM query orchestration from largely decoupled perspectives. Inference-aware systems such as ExeGPT [9] optimize execution by modeling input/output token lengths to control batching and parallelism, while recent

schedulers such as Bullet [10] improve throughput and latency through KV-cache-aware memory management and dynamic prefill-decode co-scheduling. At the infrastructure level, DynamoLLM [11] adapts system parameters such as instance scaling, parallelism, and GPU frequency to meet latency constraints under dynamic workloads. Model selection has been explored via learning and optimization. OptLLM [12] selects models from a Pareto frontier based on predicted performance under fixed constraints. RouteLLM [13] reduces routing to a binary decision between a fixed strong-weak model pair. Recent extensions incorporate system-level realism: RouterWise [14] jointly models query routing and resource allocation, Robust Batch-Level Routing [15] enforces cost and capacity constraints at the batch level, and TRouter [16] improves robustness under cold-start query distributions. Edgeoriented works focus on deployment and resource distribution rather than per-query orchestration. Jang et al. [17] proposes device-level selection strategies to balance load, while EdgeLLM [18] and EdgeShard [19] optimize collaborative inference and model partitioning across devices. State-of-the-art limitations: Existing solutions either optimize execution for fixed models or perform simplified model selection, treating latency and quality as static. They do not jointly capture query semantics, model configuration (family, scale, quantization), and dynamic system state (utilization, queueing), leading to load-coupled, query-dependent behavior and limiting fine-grained perquery orchestration in heterogeneous edge environments. This paper introduces DRLM, a Deep Reinforcement Learning-based framework for fine-grained LLM query orchestration in heterogeneous edge clusters. DRLM leverages (i) a query profiler that extracts semantic and structural features, (ii) lightweight predictors that estimate class-conditioned quality and configuration-dependent latency, and (iii) a factorized Proximal Policy Optimization (PPO)-based agent that combines these predictions with runtime system state information (e.g., resource utilization and queue dynamics) to make perquery orchestration decisions. To enable data-driven learning, we construct a large-scale benchmarking dataset with 223 835 measurements over 1258 queries and 6 classes, capturing finegrained query-model-device interactions. We evaluate DRLM on a realistic Kubernetes-based edge cluster with 64 heterogeneous nodes and 32 LLM instances spanning 8 model families, 11 parameter scales, and 5 quantization levels. Comparison with three baselines and two state-of-the-art methods shows that DRLM reduces inference latency by up to 51 % and queuing delay by up to 67 %, while incurring at most 8 % accuracy loss. Under increasing workload, it further reduces latency by up to 61.4 %, showing robust orchestration. II. P ROBLEM F ORMULATION We formulate LLM query orchestration in edge clusters as a Markov Decision Process (MDP) defined by the tuple (S, A, π, r), where S is the state space, A the action space, π

the policy, and r the reward function. At each decision step, the agent observes the system state and assigns an incoming query to a model-device configuration. 1) Environment: We consider an edge cluster with N heterogeneous devices E = {e1 , . . . , eN }, hosting K deployed LLM instances M = {m1 , . . . , mK }. Each instance mi ∈ M corresponds to a unique configuration (f, p, z), where f ∈ F denotes the model family, p ∈ P the parameter scale, and z ∈ Z the quantization level. Thus, configurations differing in family, size, or quantization are treated as distinct deployable instances. Incoming queries X = {x1 , x2 , . . . } arrive online according to a stochastic arrival process. Each query xi must be assigned to a feasible model-device pair (m, e), subject to deployment and resource constraints. To enable efficient decision-making, the environment provides predictive feedback via lightweight models: (i) inference latency b tinf = T (xi , m, e) and response quality yb = A(xi , m). These predictors approximate execution outcomes without performing full inference, enabling fast candidate evaluation. 2) State space: We define the state si ∈ S at the arrival b, b of query xi as si = ϕ(xi ), y tinf , u, q , where ϕ(xi ) denotes the extracted query representation capturing semantic and structural properties. The vectors u = {u(e)}e∈E and q = {q(e)}e∈E capture device resource utilization and queue state, respectively, where q(e) encodes the queue length and remaining processing time of active requests. This state representation enables joint reasoning over query characteristics, model behavior, and system dynamics. 3) Action space: The action space A comprises feasible assignments a = (m, e), selecting a model configuration and execution device, constrained by deployment placement and device compatibility. 4) Reward function: The reward balances response quality and end-to-end serving latency via r(si , a) = α· b̄ y−β·Φ(b te2e ), where b̄ y is the normalized predicted response quality and b te2e is the end-to-end latency defined as b te2e = b tinf (xi , m, e) + twait (e), where twait (e) denotes the waiting time before query xi can start execution on device e, as induced by the current queue state and ongoing executions. The function Φ is a normalized exponential penalty that increases the cost of high-latency assignments more aggressively than a linear formulation. Both b̄ y and b te2e are normalized via min-max scaling across candidate actions. The coefficients α and β control the trade-off between response quality and latency. 5) Policy learning: We learn a stochastic policy πθ (a|s) using PPO, which maximizes the expected cumulative reward. At each step, the agent samples an action ai = (m, e) from πθ (a|si ) and receives reward r(si , ai ). The policy is optimized using the PPO objective: h
i L(θ) = E min ρi (θ)Âi , clip(ρi (θ), 1 − ϵ, 1 + ϵ)Âi (1) −c1 Lvalue + c2 H(πθ ) i) , Âi is the advantage estimate, Lvalue where ρi (θ) = ππθθ (a(ai |s i |si ) old is the value-function loss, and H(πθ ) denotes the entropy

Query Profiler Query Embedding Generator Feature Extractor

Performance Predictor Query Classifier

Quality Latency

Runtime Monitor

Orchestration Agent

Model Instance Manager

State Encoder

Edge Edge Cluster Cluster State State Manager Manager

Factorized Policy Network

Edge Cluster

...

Figure 2: DRLM architecture; models are indicated by family f , parameter scale p, and quantization level z, and executed on edge device e. bonus encouraging exploration. The coefficients c1 and c2 control the contribution of the value loss and entropy regularization. We employ an actor-critic architecture, where the policy πθ (actor) is trained jointly with a value function Vψ (s) (critic). Training relies on trajectories generated by the predictors T and A, avoiding execution of real LLM inference and enabling efficient learning under dynamic system conditions. III. DRLM D ESIGN Fig. 2 illustrates the DRLM architecture for fine-grained LLM query orchestration in heterogeneous edge clusters. The system consists of five modules that jointly enable efficient per-query model-device selection. 1) Query profiler: transforms each incoming query into a structured representation capturing both structural and semantic properties. This includes lightweight features (e.g., token length, syntactic complexity) and a dense embedding generated via a compact sentence encoder. 2) Query classifier: maps each incoming query to a semantic class distribution over a set of query categories. Implemented using a lightweight encoder-classifier pipeline, it provides a low-dimensional abstraction of query intent. For low-confidence or out-of-distribution inputs, the system falls back to a default class associated with robust model configurations, ensuring stable behavior under uncertainty. The class set can be extended by retraining on new workloads. This abstraction reduces prediction variance while preserving query-dependent performance patterns. 3) Performance predictor: estimates query-dependent model behavior, enabling fast evaluation of candidate model-device assignments without executing inference. a) Quality: infers expected model performance from query classes, avoiding noisy per-query accuracy prediction. It captures consistent patterns across semantic categories, enabling stable quality estimation. b) Latency: predicts inference time using query features, model configuration, and device characteristics. It captures execution behavior across devices and configurations, enabling inference-time estimation under varying system conditions. 4) Runtime monitor: The runtime monitor maintains the dynamic system state required for orchestration. a) Model instance manager: tracks the deployment and availability of model instances across devices. b) Edge cluster state manager: continuously collects device-level metrics, including resource utilization and execution state. In particular, it tracks current load and remaining processing time of active requests, forming the system state (u, q) used by the orchestration agent.

5) Orchestration agent: The agent integrates predictive signals and system state to perform query-conditioned decisions. a) State encoder: Constructs the decision state by combining query representation, predicted performance, and runtime state into a compact latent representation. b) Factorized policy network: learns a stochastic mapping πθ (a|s) over model-device assignments. To handle the combinatorial action space, we adopt an autoregressive factorization method over model configuration and device selection: πθ (a|si ) = πf (f, p|si ) · πz (z|si , f, p) · πe (e|si , f, p, z), (2) where (f, p, z) defines the selected model instance m with family f , parameter scale p and quantization level z. The policy is implemented using separate output heads for each decision stage (model family/scale, quantization, and device), each conditioned on the state and preceding selections. Invalid choices are dynamically masked based on deployment and compatibility constraints. This factorization reduces the effective action space and improves sample efficiency by enabling structured credit assignment across model-device selection. IV. E VALUATION S ETUP This section describes the setup, including the testbed, model deployment, workload, and evaluation methodology. 1) Edge testbed: We evaluate DRLM on an edge cluster comprising 64 heterogeneous nodes across 5 device classes: Raspberry Pi 3/4 (RP, 2 to 12 CPU cores, 1 GB to 4 GB RAM, 38 nodes), Jetson Nano (NJN, 4 cores, 4 GB RAM, 128-core GPU, 6 nodes), Jetson Orin Nano (JON, 6 cores, 8 GB RAM, 1024-core GPU, 2 nodes), Jetson Orin AGX (JOA, 12 cores, 64 GB RAM, 2048-core GPU, 2 nodes), and KVM-based VMs (2 to 24 vCPU, 16 GB to 32 GB RAM, 16 nodes). The cluster runs Kubernetes with containerd, hosting containerized LLM inferences. We collect runtime telemetry using Prometheus and cAdvisor for CPU/memory, and tegrastats for GPU utilization on Jetson devices. Metrics are aggregated to construct system state, including utilization, active load, and processing backlog. All model instances are exposed via REST, enabling dynamic query dispatch to valid model-device pairs. 2) LLM models: We deploy 32 LLM instances from 8 model families, spanning parameters from 0.27 B to 20 B with diverse quantizations, resulting in a heterogeneous space of model-configuration variants. Our deployment is deviceaware, i.e., highly quantized lightweight models are placed on constrained devices (RP, NJN), while larger and higherprecision models are hosted on more capable nodes (JON,

Math. Hist. TfulQA

4000 1.2K 0.6K

40K

2000

0 0.2 0.4 0.6 0.8 1 0

(a) Prompt complexity (KDE).

3000

1000

20K

Prompt Complexity Score

Count

Bio. CSQA Geo.

Count

Math. Hist. TfulQA

Density

Density

Bio. CSQA Geo.

140K 120K 100K 80K 60K

Accuracy

1

Prompt Token Count (log)

(b) Prompt token count (KDE).

(c) Accuracy.

Inference Time (log)

(d) Inference time.

Figure 3: Distribution of query, accuracy, inference time in the benchmarking dataset. Table I: Summary of deployed models on the edge devices. Model family GPT-OSS Gemma2 Gemma3 Gemma3 Gemma3 Llama3.1 Llama3.2 Llama3.2 Qwen3 Qwen3 Tinyllama Mistral

Parameter scale 20 B 2B 0.27B 12B 1B 8B 1B 3B 0.6B 1.7B 1.1B 7B

Quantization z4 z4 z4 , z16 z4 z3 , z4 z8 , z16 z4 , z8 z2 , z3 , z4 z4 , z8 , z16 z4 z2 , z3 , z4 , z8 z8

Edge device JOA, VM NJN, RP, JOA, JON, VM NJN, RP, JOA, JON, VM JOA, VM NJN, RP, JOA, JON, VM JOA, VM NJN, RP, JOA, JON, VM RP, JOA, JON, VM JOA, VM JOA, VM NJN, RP, JOA, JON, VM JOA, VM

JOA, VM). This induces diverse latency-quality trade-offs and non-uniform actions. Table I summarizes the deployment. 3) Query workload: We evaluate DRLM using 1258 queries from four benchmark datasets spanning diverse domains: MMLU [20] (biology, world history, geography), GSM8K [7], CommonsenseQA [21], and TruthfulQA [8]. These queries are grouped into 6 semantic categories used for classconditioned modeling. Categories capture semantic query types (not dataset identities) and are assigned via embeddingbased classification, mitigating dataset-specific bias. To ensure balanced workloads, GSM8K, CommonsenseQA, and TruthfulQA are subsampled to 200 queries each. All datasets provide ground-truth answers, enabling evaluation of response quality across heterogeneous query types. Query arrivals follow a Poisson process with rate λ ∈ {0.5, 1, 2}, modeling varying workload intensities. 4) Benchmarking dataset: We construct a benchmarking dataset with 223 835 measurements spanning queries, model configurations, quantization levels, and edge devices. Each record corresponds to a query-model-device tuple and includes inference latency, resource utilization (CPU/GPU/memory), and response quality, enabling supervised training of the classifier and predictors. Fig. 3 summarizes key distributions. Kernel Density Estimation (KDE) plots (Figs. 3a-3b) reveal multi-modal variability in query complexity and length. Fig. 3c shows a bimodal accuracy distribution due to binary and softscored tasks, while Fig. 3d exhibits heavy-tailed latency (log scale), reflecting heterogeneity across models and devices. Quality labels are binarized using a 0.5 threshold to ensure consistent supervision for class-conditioned estimation. 5) Query classifier: We encode each query via all-MiniLM-L6-v2 sentence encoder, a 6-layer Transformer producing 384-dimensional embeddings. A

Random Forest classifier (200 trees, max depth 20) maps embeddings to 6 semantic classes, enabling low-dimensional abstraction of query semantics. For low-confidence or outof-distribution queries, the system falls back to a predefined class associated with high-capability GPT-OSS 20B, which performs well across query categories (Fig. 4a), ensuring reliable behavior under uncertainty. 6) Quality estimator: We estimate response quality using class-conditioned empirical statistics derived from the benchmarking dataset. Specifically, model performance is aggregated per query class, providing stable estimates of expected accuracy. The estimator is implemented as a precomputed class-model lookup table and queried at runtime, enabling constant-time evaluation of candidate assignments. 7) Latency predictor: We model inference latency using a gradient-boosted regressor (LightGBM) trained on query, model, and device features. The model uses 600 trees, 63 leaves, and a learning rate of 0.05. Input features include: (i) query structure (length, token/word/sentence counts, lexical diversity), (ii) model configuration (family, parameter scale, quantization, layers, attention heads), and (iii) edge device characteristics (hardware type, CPU/GPU execution). 8) DRL training: We train the orchestration policy using PPO on trajectories generated by the predictors, avoiding costly LLM execution while preserving system dynamics. The query set is split into disjoint training (80 %) and evaluation (20 %) subsets. The actor-critic network shares a two-layer MLP encoder (128 units per layer). The actor is factorized into three heads, and the critic outputs a scalar value estimate. Training runs for 1000 episodes, each simulating 100 requests with exponential inter-arrivals (rate 1). We use Adam (lr 3 × 10−4 ), discount factor 0.95, entropy coefficient 0.02, clipping 0.2, and 6 update epochs per rollout. Reward weights are set to α = 0.8 and β = 0.3, empirically calibrated to balance quality and latency under normalized rewards. 9) Comparison methods: We compare DRLM against three baselines and two state-of-the-art methods under the same testbed, deployment, and workload: (i) Random: uniformly samples a feasible (m, e), ignoring query and system state. (ii) High-Acc: selects arg maxm A(c, m), ignoring latency and load. (iii) Fastest: selects (m, e) minimizing predicted inference latency, ignoring quality. (iv) RouteLLM-style: adapted [13] by defining a single strong model (GPT-OSS 20 B) and a weak pool (all others). Queries are routed to the

Bio.

50

CS. Geo.

40

Math.

20

30

Tfull.

10

Biology CommonSense Geography Mathematics

History

TruthfulQA

(a) Accuracy of LLMs across queries.

His.

Tfull.

Math.

CS.

Geo.

Bio.

His.

0

Predicted

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

Gemma3 (12B) Llama3.1 (8B) Qwen3 (1.7B) Mistral (7B)

GPT-OSS (20B) Llama3.2 (1B) Qwen3 (0.6B) Llama3.2 (3B)

Predicted

Accuracy

Gemma3 (1B) Gemma3 (0.27B) Gemma2 (2B) Tinyllama (1.1B)

Actual

Actual

(b) RF query classification.

(c) LGBM inference time prediction.

Reward

Critic Loss

0.4 0.3 0.2 0.1

Critic Loss

h nt. e Siz Arc Qua . t. th lex uc lass ng mp C Str Le Co 0 10 20 30 40 50 60 70 Importance (%)

12 10 8 6 4 2

Reward

Features Prompt Model Device

Figure 4: Predictive modeling performance in DRLM.

0 0 200 400 600 800 1000 Episode

(a) Latency predictor feature importance. (b) Reward and loss convergence.

Random Fastest High-Acc

RouteLLM-Style OptLLM-Style DRLM

CDF

Orchestration Overhead (ms/ request)

Figure 5: Training dynamics in DRLM.

dom cc test tyle tyle LM Ran High-A Fas LLM-S LLM-S DR Opt Route

(a) Orchestration time.

0.01

Waiting Time (s) - Log Scale

(b) Waiting time (log-scale).

Figure 6: Orchestration and waiting time across baselines. strong model if its predicted advantage exceeds 0.5, otherwise to the best weak model based on A(c, m). (v) OptLLM-style: adapted [12] by replacing cost with latency and restricting decisions to deployed (m, e) pairs; selection is performed from the Pareto frontier of quality-latency trade-offs. 10) Evaluation metrics: We evaluate all methods using the following metrics: (i) Response quality: average correctness against ground truth, capturing task-level accuracy across heterogeneous queries. (ii) Inference latency: model execution time, reflecting efficiency of selected model-device configurations. (iii) Waiting time: queueing delay before execution, capturing load-induced congestion. (iv) Orchestration overhead: per-query decision time of the orchestration policy. V. E XPERIMENTAL R ESULTS This section evaluates DRLM against three baselines and two state-of-the-art methods on the same edge testbed, reporting the average and standard deviation over 10 independent runs. 1) Predictive modeling analysis: Fig. 4a shows variability in model accuracy across query categories, with no single model dominating across all tasks. This non-uniform behavior confirms that model effectiveness is inherently querydependent, motivating class-conditioned quality estimation rather than global model ranking. The RF-based query classifier (Fig. 4b) achieves high consistency in mapping queries to semantic categories, as reflected by the strongly diagonal

confusion matrix with limited cross-class leakage. This behavior stems from the semantic coherence of queries within each category, allowing embedding-based representations to effectively cluster similar queries. Rather than enforcing strict boundaries, the classifier provides a stable abstraction of query types suitable for downstream class-conditioned estimation. The latency predictor (Fig. 4c) achieves consistent alignment between predicted and measured inference times on a log scale (R2 = 0.75), capturing relative latency differences across model-device configurations. This level of accuracy is sufficient for query orchestration, as the policy depends on preserving the ordering of candidate configurations rather than minimizing absolute prediction error. Its inference overhead is negligible (0.78 ms per instance), enabling real-time evaluation of candidate assignments. Feature importance analysis (Fig. 5a) shows that prompt-level features dominate latency prediction, with input length and structural complexity contributing the largest share, followed by model configuration, while device features have a lower impact. This is because devices are partially entangled with model deployment, leading to shared explanatory power; thus, model configuration captures most of the variance associated with execution cost. 2) DRL convergence and orchestration time: Fig. 5b shows the training dynamics of the PPO-based DRLM agent. The reward increases rapidly within the first 200 episodes and stabilizes at a high-value plateau, indicating fast and stable convergence. In parallel, the critic loss drops sharply and remains near zero, confirming accurate value estimation and low training variance. Fig. 6a compares orchestration overhead across methods. DRLM achieves approximately 35 ms/query, reducing decision latency by about 60 % compared to OptLLMstyle and RouteLLM-style, and by about 30 % compared to Fastest. High-Acc incurs similar overhead to optimizationbased methods due to full candidate evaluation. This efficiency is enabled by the factorized policy design, which decomposes decision-making into sub-actions, reducing the effective action space and avoiding exhaustive evaluation. 3) Comparison with baselines: Fig. 7a shows the distribution of query assignments across model-device configurations. DRLM learns a load-aware policy that distributes queries across both lightweight and high-capacity configurations, assigning simple queries to quantized models on constrained devices (RP, NJN) while reserving larger models on powerful nodes (VM, JON, JOA) for complex queries. In contrast,

Mistral (7B) Gemma3 (12B) Gemma3 (1B) Llama3.1 (8B)

Llama3.2 (3B) Tinyllama (1.1B) Qwen3 (0.6B) Llama3.2 (1B)

NJN

JON

Random Fastest

High-Acc DRLM

Random Fastest

OptLLM-Style RouteLLM-Style

Accuracy

DRLM RouteLLMStyle OptLLMStyle Fastest

High-Acc DRLM Inf. time

OptLLM-Style RouteLLM-Style Wait. time

Latency (s)

GPT-OSS (20B) Gemma2 (2B) Gemma3 (0.27B)

High-Acc Random RP

VM

JOA

(a) Model-device selection distribution.

(b) Average response quality.

(c) End-to-end latency (log-scale).

Figure 7: Comparison of model selection, response quality, and end-to-end latency across baselines under varying workload intensities. High-Acc concentrates assignments on large models, creating hotspots, while Fastest collapses to aggressively quantized configurations. Random exhibits no structure. OptLLM- and RouteLLM-style partially balance this trade-off but remain less adaptive to device-level heterogeneity. Fig. 6b shows the CDF of waiting time (log scale) in different schemes. DRLM shows a left-shift with a steeper rise, indicating lower median and tail latency. In contrast, High-Acc, OptLLM, and RouteLLM show heavy-tailed behavior, with a large fraction of queries experiencing delays above 100 s. Fastest achieves low waiting time but at the cost of degraded quality (Fig. 7b). Fig. 7b reports average response quality under increasing workload (nreq ∈ {25, 50, 100}, λ ∈ {0.5, 1, 2}). HighAcc and RouteLLM achieve the highest accuracy (0.54 to 0.56), followed by OptLLM. DRLM maintains stable performance around 0.50 to 0.52, outperforming Fastest and Random. DRLM exhibits minor variation across workload intensities, indicating stable behavior under increasing load. Fig. 7c shows end-to-end latency (log scale) under varying workloads. As load increases, latency rises sharply for accuracy-oriented baselines due to queuing effects caused by over-selection of large models. In contrast, DRLM mitigates congestion via stateaware orchestration, distributing queries across model-device configurations. Under the highest load (100 requests, λ = 2), DRLM reduces inference time by 19.2 % and 38 % compared to OptLLM and RouteLLM, and reduces waiting time by 52 % and 62 %, with at most 8 % accuracy loss. VI. C ONCLUSION AND F UTURE W ORK This paper presented DRLM, a DRL-based framework for fine-grained LLM query orchestration in heterogeneous edge clusters. DRLM jointly models (i) query semantics via embedding-based classification, (ii) class-conditioned quality and feature-driven latency prediction, and (iii) dynamic system state (resource utilization and queuing), enabling stateaware model-device selection. We constructed a large-scale benchmarking dataset with 223 835 measurements, capturing query-dependent latency-quality trade-offs across models and edge devices. Evaluation on a 64-node edge testbed shows that DRLM shortens inference latency by up to 51 % and queuing delay by up to 67 %. Future work will extend DRLM toward multi-objective optimization, incorporating cost and energy. R EFERENCES [1] R. Farahani, Z. Azimi Ourimi, C. Timmerer, and R. Prodan, “Towards AI-assisted Sustainable Adaptive Video Streaming Systems: Tutorial and Survey,” ACM Computing Surveys, 2026.

[2] R. Farahani, M. Colosi, I. Murturi, S. Nastic, M. Villari, S. Dustdar, and R. Prodan, “ClusterLess: Deadline-Aware Serverless Workflow Orchestration on Federated Edge Clusters,” 2026. [3] Z. Azimi Ourimi, R. Fahrani, R. Prodan, and C. Timmerer, “ELLMPEG: An Edge-based Agentic LLM Video Processing Tool,” in Proc. of the ACM Multimedia Systems Conf., 2026. [4] L. Lovén, R. Farahani, I. Murturi, S. Sigg, and S. Dustdar, “Agentic Edge Intelligence: A Research Agenda,” in Proc. of the 18th IEEE/ACM Intl. Conf. on Utility and Cloud Computing, 2025. [5] R. Farahani, Z. Azimi, M. Colosi, and S. Dustdar, “LMEdge: QoSAware LLM Inference Orchestration on Edge Clusters,” arXiv preprint arXiv:2607.17175, 2026. [6] R. Farahani and R. Prodan, “Serverless Large Language Models: Edge vs. Cloud Deployment Trade-offs,” in Proc. of Workshop on AI for Sustainable Energy Systems and Green AI, 2025. [7] K. Cobbe et al., “Training Verifiers to Solve Math Word Problems,” arXiv preprint arXiv:2110.14168, 2021. [8] S. Lin et al., “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” https://github.com/sylinrl/TruthfulQA, 2025. [9] H. Oh et al., “ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference,” in Proc. of the 29th ACM Intl. Conf. on Architectural Support for Programming Languages and Operating Systems, 2024. [10] Z. Lin et al., “Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration,” in Proc. of the 31st ACM Intl. Conf. on Architectural Support for Programming Languages and Operating Systems, 2026. [11] J. Stojkovic et al., “DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency,” in 2025 Intl. Symp. on High Performance Computer Architecture. IEEE, 2025. [12] Y. Liu et al., “OptLLM: Optimal Assignment of Queries to Large Language Models,” in 2024 IEEE Intl. Conf. on Web Services. IEEE, 2024. [13] I. Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data, 2024,” https:// arxiv.org/ abs/ 2406.18665, 2025. [14] H. Hosseini Kasnavieh et al., “RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving,” arXiv preprint arXiv:2604.10907v1, 2026. [15] J. Markovic-Voronov et al., “Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints,” arXiv preprint arXiv:2603.26796, 2026. [16] H. Liu et al., “Task-Aware LLM Routing with Multi-Level TaskProfile-Guided Data Synthesis for Cold-Start Scenarios,” arXiv preprint arXiv:2604.09377v1, 2026. [17] S. Jang et al., “Edge-First Language Model Inference: Models, Metrics, and Tradeoffs,” 2025 IEEE 45th Intl. Conf. on Distributed Computing Systems Workshops (ICDCSW), 2025. [18] F. Cai et al., “Edge-LLM: A Collaborative Framework for Large Language Model Serving in Edge Computing,” in 2024 IEEE Intl. Conf. on Web Services. IEEE, 2024. [19] M. Zhang et al., “Edgeshard: Efficient LLM Inference via Collaborative Edge Computing,” IEEE Internet of Things Journal, 2024. [20] D. Hendrycks et al., “Measuring Massive Multitask Language Understanding,” arXiv preprint arXiv:2009.03300, 2020. [21] A. Talmor et al., “CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge,” arXiv preprint arXiv:1811.00937, 2018.

Related documents

• Smart Adaptive Computing Across the Continuum: LLMs in IoT-Edge-Cloud Resource Management#987105
• AI-Native Orchestration in the 6G Continuum: Evolving Operator Platforms with Agentic AI#667937
• To Train or Not to Train: Balancing Efficiency and Training Cost in Deep Reinforcement Learning for Mobile Edge Computing#792607
• To Train or Not to Train: Balancing Efficiency and Training Cost in Deep Reinforcement Learning for Mobile Edge Computing#788607
• LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters#386838