CEMA-Router: Infrastructure-Aware Batch Routing for LLM Inference in Cloud-Edge Systems

CEMA-Router: Infrastructure-Aware Batch Routing for LLM Inference in Cloud–Edge Systems Xijia Wang, TianZhang He, Mohammad Goudarzi*

arXiv:2610.26171v1 [cs.DC] 19 Oct 2026

Faculty of Information Technology, Monash University, Melbourne, Australia

Abstract—Large language models (LLM) are increasingly integrated into pervasive applications spanning heterogeneous cloud and edge resources. These heterogeneous resources differ in computational capacity, energy efficiency, network conditions, and deployed model quality, creating trade-offs among response quality, monetary cost, energy consumption, and batch completion time when routing inference requests. Existing LLM routers typically make per-query routing decisions based primarily on model quality and token cost, whereas cloud–edge schedulers often overlook quality differences among deployed models. Consequently, these objectives remain insufficiently coordinated. We present CEMA-Router, an infrastructure-aware batch router for LLM inference in pervasive cloud–edge systems. CEMA-Router formulates batch routing as a multi-objective assignment problem that jointly minimizes model-quality loss, monetary cost, energy consumption, and straggler-dominated batch completion time. To efficiently explore the assignment space, it employs a routingspecific memetic algorithm that combines largest-request-first, load-aware seeding with probabilistic, evaluator-guided Lamarckian repair. Routing decisions are informed by deploymentspecific node profiles, dynamic backlog estimates, and modelquality signals. We further implement CEMA-Router as an endto-end system comprising a request manager, routing service, and profiler, supporting both real cloud–edge deployment and tracedriven simulation. We evaluate the system using RouterBench traces and GSM8K and MMLU workloads on a heterogeneous six-node cloud–edge testbed. Compared with state-of-the-art baselines, CEMA-Router reduces the weighted infrastructureaware objective by up to 86.3% for GSM8K and 80.5% for MMLU, while reducing energy consumption by up to 33.8% and 19.5% and batch completion time by up to 27.1% and 30.1%, respectively. Quality-oriented baselines remain useful reference points for raw response quality, while CEMA-Router explicitly exposes the stronger deployment policy by jointly accounting for quality, cost, energy, and batch completion time. Index Terms—LLM inference, cloud-edge system, pervasive systems, routing, memetic algorithm, energy-aware scheduling

I. I NTRODUCTION LLM-backed applications are rapidly transitioning from centralized interactive services toward pervasive deployments that span high-performance cloud GPUs and resourceconstrained edge nodes close to users. While this shift offers benefits in wide-area latency reduction and data privacy, it introduces a highly complex routing problem. Existing LLM routers typically focus on model-level quality and token cost * Corresponding author Emails: wangxijia99@gmail.com, mohammad.goudarzi@monash.edu.

tianzhang.he@monash.edu,

Fig. 1. Quality-metric routing vs. multiple-metric routing in the existing sixnode deployment. Values are normalized within each metric for readability.

in isolation, abstracting away the real infrastructure. Consequently, they fail to account for critical deployment constraints such as hardware resource utilization, node-level queuing delay, network bandwidth, and energy consumption. To address this gap, this paper introduces CEMA-Router, an infrastructure-aware LLM request routing system formulated as a batch-level assignment problem. Rather than making myopic, per-query decisions, CEMA-Router jointly routes short batches of requests. It optimizes a comprehensive multiobjective function encompassing profiled quality loss, explicit monetary cost, energy consumption, and batch completion time. To solve this NP-hard assignment efficiently online, we design a customized Memetic Algorithm (MA) equipped with continuous-batching queue models and a lightweight Retrieval-Augmented Generation (RAG) quality predictor. The term infrastructure-aware captures the vital interaction between model capability and node capacity: the most accurate model becomes the wrong target if its hosting node is a bottleneck, or if another endpoint provides a superior latencyenergy trade-off. A multiple-metric objective significantly improves the overall operational efficiency, whereas conventional quality-metric routing incurs latency and energy penalties to achieve only marginal raw quality gains (Fig. 1). Also, to mitigate inference contention, Fig. 2 reports a micro-benchmark validating our batch-level intuition. When requests are routed independently, multiple queries often crowd the same locally attractive endpoint, causing severe queuing delays. However, joint batch routing orchestrates globally and jointly assigns query batches and reduces average makespan by 57.3%. We ask whether a deployable LLM router can leverage dynamic node states and RAG-based quality profiles to produce superior cloud-edge assignments while maintaining sub-second decision overhead. To answer this, we evalu-

Fig. 2. Per query vs batch LLM routing.

ate CEMA-Router through a synthesis of controlled RouterBench [1] traces and a real six-node cloud-edge testbed running GSM8K [2] and MMLU [3] workloads. This methodology isolates algorithmic performance while validating the end-to-end request management and profiling paths. The main contributions are: • We formulate the LLM inference routing problem as a multi-objective optimization problem that jointly considers inference quality, monetary cost, energy consumption, and response latency, incorporating a FIFO queuing model to capture realistic request scheduling delays in cloud-edge environments. • We design a customized Memetic Algorithm-based routing framework that modifies the MA pipeline for cloudedge LLM serving through routing-aware chromosome encoding, mixed heuristic-random initialization, analytical multi-objective fitness evaluation, probabilistic Lamarckian crossover, and domain-specific local search operators including worst-gene repair and 2-opt swap. • We develop a RAG-based quality prediction module that estimates request-model quality from similar historical prompts using sentence-transformer embeddings and FAISS indexing, avoiding trial inference or task-specific predictor training during routing, allowing quality to be combined with real-time infrastructure metrics. • An evaluation using RouterBench traces and real GSM8K/MMLU workloads on a six-node cloud– edge deployment, showing that CEMA-Router reduces the weighted infrastructure-aware objective relative to UniRoute and simple deployment policies while exposing the quality–infrastructure trade-off. II. BACKGROUND AND R ELATED W ORK A. LLM Serving and Model Routing LLM serving focuses on optimizing the execution of trained language models under stringent memory, compute, and latency constraints. The Transformer architecture [4] underlies modern LLMs such as GPT-3 [5], where each request consumes accelerator memory, prefill/decode compute, and keyvalue (KV) cache capacity. State-of-the-art systems such as ORCA [6], Sarathi-Serve [7], DistServe [8], Splitwise [9], and vAttention [10] have made major progress in batching, phase disaggregation, and memory management. However, these systems excel at optimizing the execution path only after a request has already reached a specific model endpoint. Conversely, model routing addresses the decision of which model or provider should process a request before execu-

tion begins. Recent frameworks such as RouteLLM [11], Hybrid LLM [12], PolyRouter [13], RouterBench [1], and UniRoute [14] offer preference-based, quality-cost, or learned routing policies. Other routers employ tabular adaptation, ranking models, and mathematical optimization [15]–[25]. Crucially, this line of work primarily focuses on per-query model-level quality and token-cost trade-offs, often treating the underlying infrastructure as an abstracted, infinite-capacity black box. They typically ignore how routing a batch of requests to the same destination affects local queuing delays or real energy consumption. B. Cloud–Edge and Pervasive AI Systems Cloud–edge computing provides a distributed deployment model where cloud servers, nearby edge nodes, and user-side environments jointly host application services. Pervasive AI systems leverage this architecture to bring inference closer to data sources, thereby reducing wide-area latency and preserving privacy [26], [27]. Recent pervasive systems increasingly emphasize runtime adaptation across heterogeneous devices and energy constraints. For instance, ECORE [28] routes deep learning requests to energy-conscious edge device-model pairs; MERCI [29] adapts multi-expert inference for dynamic vision perception; and ERUN [30] optimizes memory-efficient Edge FaaS runtimes. These systems demonstrate that hardware heterogeneity, energy bounds, and runtime load are central concerns in pervasive deployments. Recent cloud–edge LLM systems have begun to explore adaptive serving policies across heterogeneous deployment scales [31]–[36]. They bring LLM inference closer to pervasive environments and motivate the need for routers that can seamlessly integrate model-level quality metrics with deployment-level resource states. Table I compares representative LLM routing and pervasive AI systems across deployment scales, request granularity, modeled heterogeneity, and target performance metrics (cost, quality, latency, and energy). C. Positioning of CEMA-Router The comparison in Table I identifies a critical gap in the current literature: existing solutions typically address only one side of the coin. LLM routers optimize for response quality and API cost, but completely abstract away the physical endpoint—ignoring node energy consumption, network path limits, and queuing bottlenecks induced by concurrent batch requests. Conversely, cloud–edge scheduling systems meticulously adapt to physical resource constraints but rarely incorporate prompt-specific LLM quality or token economics into their routing objectives. CEMA-Router bridges this gap directly. It treats each deployed node–model pair as a concrete, resource-constrained destination. By combining an RAG-style quality predictor to estimate prompt-specific quality loss with token cost tables, calibrated endpoint throughput, network measurements, energy profiles, and real-time backlog states, CEMA-Router performs batch-level optimization. Consequently, it minimizes a unified,

TABLE I C OMPARISON OF LLM ROUTING T ECHNIQUES Work

[31] [15] [16] [32] [13] [12] [17] [18] [11] [33] [34] [35] [19] [20] [36] [21] [22] [23] [24] [25] [14] CEMA-Router

Environment Architecture Cloud Nodes

Edge Nodes

Multiple Single Single Multiple Single Single Single Single Single Multiple Multiple Multiple Single Single Multiple Single Single Single Single Single Multiple Multiple

Multiple None None None None None None None None Multiple Multiple Multiple None None None None None None None None None Multiple

Incoming Request Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Single Batch

Aspect of Heterogeneity

Scalability

Technique

LLM Model

LLM Size

Computation Nodes

✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓

Performance Metrics

Destination

Request

Type

Cost Quality Latency Energy

✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓

✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓

System Hybrid Hybrid System Hybrid Hybrid Hybrid Hybrid System System ML System Hybrid System System Hybrid Hybrid Hybrid Hybrid Hybrid ML Hybrid

✓ ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓

Heuristic DQN Heuristic DDQN BERT-based BERT-based Elo-based GNN-based Ensemble-based Bandit-based Heuristic Bandit-based Mathematical BERT-based Mathematical BERT-based Bandit-based MLP-based Learning-based IRT-based Heuristic MA-based

✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓

DQN = Deep Q-Network; DDQN = Double DQN; JCT = Job Completion Time; BERT = Bidirectional Encoder Representations from Transformers; MLP = Multi-Layer Perceptron; GNN = Graph Neural Network; ML = Machine Learning; IRT = Item Response Theory; QNC = Quality-Neutral Cost.

weighted objective (i.e. fitness) covering quality, monetary cost, energy consumption, and batch completion time, ensuring that every assignment respects both the semantic capability of the model and the physical limitations of the pervasive infrastructure. III. S YSTEM A RCHITECTURE AND ROUTING M ODEL Figure 3 presents CEMA-Router as a runtime routing architecture with four main parts: the Request Manager, the Router, the Profiler, and the heterogeneous LLM endpoint pool. The Request Manager admits user requests and dispatches the selected routing policy. The Router contains the Request Scheduler and the memetic algorithm that evaluates candidate batch assignments. The Profiler groups the Request Monitor, Network Profiler, Node Profiler, pre-trained RAG Quality Store, and Node Static Attributes, and feeds their routing inputs to the BatchEvaluator. This section defines the system state and objective used by the router; Section IV then explains how the memetic algorithm searches over candidate assignments and obtains the best routing policy. A. Request Management The Request Manager is the online entry and exit path for user workloads. Its admission component operates in discrete routing windows [T0 , T1 ) of length ∆. All requests r arriving within the window are grouped into an admitted batch R, with size R ≤ Rm , and each request carries the fields needed by the evaluator: prompt content, input-token count, predicted output-token count, and network context. The admitted batch is forwarded to the Request Scheduler. After the router returns the best routing policy, the dispatcher sends each request to its assigned destination d in the heterogeneous endpoint pool. Responses are returned through the same management path,

Router

Users Requests send request retur n response

Request Manager ? A?

Admission

Dispatcher

for m request batch Request Scheduler dispatch request

notify completion make decision

Profiler ? B?

? D? Heterogeneous LLM Endpoint Pool

Memetic Algorithm ? C?

RAG Quality Store (pre-tr ained) Node Static Attr ibutes

repor t nodes status

Population I nitialization

feed inputs

Batch Evaluator Selection

Networ k Profiler Crossover Node Profiler

Request M onitor

M utation

Best Routing Policy

Fig. 3. CEMA-Router architecture overview.

while dispatch and completion notifications are recorded by the Request Monitor inside the Profiler. B. Profiler and Routing Context The Profiler provides the routing context consumed by the BatchEvaluator in Section III-C. Static node attributes capture deployment properties such as model placement, node type, token-price settings, calibrated single-request and saturatedbatch throughput, network goodput, and power characteristics. The Node Profiler reports endpoint health and node status, while the Network Profiler measures router-to-endpoint latency and network conditions. The Request Monitor records dispatch and completion events and maintains the in-flight backlog estimate Bd for each destination as carry-over service time from previously dispatched work. After each routing window, it updates Bd by subtracting the elapsed window ∆ from the previous backlog and adding newly assigned

service duration, retaining only unfinished work. This systemside tracking is necessary because the endpoint vary in model family, model size, hardware architecture, and serving stack. CEMA-Router therefore maintans a common in-flight view from its own dispatch and completion records rather than replying on endpoint-specific queue or telemetry interfaces. The pre-trained RAG Quality Store is the quality predictor for routing. The router must make an assignment quickly, so it cannot send the query to every candidate endpoint and wait for observed response quality before choosing a destination. In the real deployment, the quality store is populated offline by profiling each workload across the available endpoints and scoring outputs automatically. Representative prompts are embedded with all-MiniLM-L6-v2 and indexed in FAISS [37]. At runtime, the incoming prompt embedding retrieves the topK nearest profiled prompts. For each destination d, the score is the similarity-weighted average of the historical quality scores of those neighbors, with higher-similarity prompts contributing more to q̂(r, d) (Eq. 3). The BatchEvaluator then uses 1 − q̂(r, d) as the request–destination quality-loss entry. The quality store provides prompt-specific quality estimates before dispatch without requiring trial inference. The same retrieved quality evidence is supplied to CEMARouter and UniRoute in the deployment comparison. CEMARouter combines this evidence with Request Monitor backlog, calibrated batch throughput, network measurements, token cost, and energy profiles, while UniRoute uses it mainly through its quality-cost objective. Holding the quality source fixed isolates the effect of infrastructure-aware assignment. C. Request Scheduler and Batch Evaluation The Request Scheduler passes each admitted batch and the latest Profiler context to the Router. It assembles the request set, destinations, profiler signals, and objective weights, and then asks the memetic algorithm in Sect. IV to find the best destination per request. The BatchEvaluator scores complete batch assignments. For a batch R and destinations D, an assignment maps each request to one concrete node–model destination in the current endpoint pool. The evaluator computes following terms for each assignment. Response-quality loss Q is computed by the similarityweighted historical score from the RAG Quality: P i∈NK (r) si · q(pi , d) P . (1) q̂(r, d) = i∈NK (r) si where NK (r) is the retrieved neighbor set, si is the cosine similarity between r and profiled prompt pi , and q(pi , d) is the recorded response-quality score of pi at destination d. If no neighbor exceeds the similarity threshold, the predictor uses the destination’s mean profiled quality. Monetary cost C combines destination token-pricing entries with any metered transfer cost; in self-hosted deployment runs the external API price is zero for every endpoint. Cpred = Cinfer + Ctrans .

(2)

Energy consumption E is decomposed into processing and communication energy. Processing energy uses the profiled service time of the selected node–model destination together with its board-power and utilization assumptions. Communication energy uses the upload and download payload sizes, link goodput, and transmit/receive power profile for the router-toendpoint path. Thus the same model can contribute different energy under different node placements or network paths. Epred = Einfer + Etrans .

(3)

Completion time T is the batch makespan. For each request, the evaluator estimates inference time from model size, predicted token length, and calibrated single-request throughput; estimates transfer time from router-to-endpoint latency and uplink/downlink goodput; and estimates FIFO queuing from Bd plus the cumulative work of earlier requests assigned to the same destination. T̂r = T̂infer (r, dr ) + T̂trans (r, dr ) + T̂queue (r, dr ).

(4)

To account for continuous batching, the cumulative work is divided by the destination’s saturated-batch speedup, while the prediction is bounded so a request is not treated as completing faster than its own service time. This makes the evaluator sensitive to interactions among requests: several individually attractive assignments can still create a poor batch if they concentrate work on the same endpoint and turn it into the straggler. For the backlog update after dispatch, the newly assigned service duration of a destination is approximated by the larger of its longest individual request service time and its total serial-equivalent work divided by the saturated-batch speedup. The next-window backlog is then the unfinished remainder after subtracting ∆. The saturated-batch throughput profile is used here rather than peak hardware FLOPS, so the service-time reflects the measured serving stack for the deployed model and precision. Weighted Cost CEMA-Router’s primary objective is to minimize a weighted infrastructure-aware cost combining normalized response-quality loss, monetary cost, energy consumption, and batch completion time. Each weight reflects the relative importance of a metric to the deployment operator. By optimising these metrics jointly over the entire request batch, CEMA-Router selects the assignment that best satisfies the configured operational priorities rather than optimising any single metric in isolation.

e e e J(x) = ωQ Q(x) + ωC C(x) + ωE E(x) + ωT Te(x),

(5)

where x is a complete request-to-destination assignment. Exact weights ω are part of the evaluation configuration. All terms are normalized to [0, 1] before aggregation, Metrici,norm =

Metrici − Metricmin . Metricmax − Metricmin

(6)

Algorithm 1 CEMA-Router Batch Assignment

Algorithm 2 Routing-Aware Population Initialization

1: Input: batch R, destinations D 2: Input: context C, weights ω, search budget (P, G) 3: Output: best assignment chromosome g ∗ and decision metrics 4: Remove unavailable destinations and build legal sets {Dr } 5: Load quality estimates, destination profiles, and backlogs {Bd } 6: P ← InitPopulation(R, D, C, P ) 7: for t = 1 to G do 8: Evaluate (Q, C, E, T ) for each g ∈ P using BatchEvaluator 9: Normalize metrics within P and compute scalar J(g) 10: P′ ← ∅ 11: while |P ′ | < P do 12: Select parents g (1) , g (2) by 3-tournament selection 13: Create h(1) , h(2) with Algorithm 3 14: Mutate active genes of h(1) , h(2) with probability pm 15: Add h(1) , h(2) to P ′ 16: end while 17: Evaluate (Q, C, E, T ) for each h ∈ P ′ using BatchEvaluator 18: Normalize metrics within P ∪ P ′ and compute scalar J 19: P ← best P chromosomes from P ∪ P ′ by J 20: end for 21: Return g ∗ = arg ming∈P J(g)

1: Input: batch R, legal destination sets {Dr } 2: Input: context C, population size P 3: Output: initial population P 4: P ← ∅; initialize seed chromosome g 5: Set provisional load Ld ← Bd for each d ∈ D 6: Sort requests in R by predicted token length in descending order 7: for each request r in sorted order do 8: For each d ∈ Dr , estimate Ad = Ld + T̂infer (r, d) + T̂trans (r, d) 9: Choose d∗ = arg mind∈Dr Ad and set gr ← d∗ 10: Update Ld∗ ← Ld∗ + T̂infer (r, d∗ ) 11: end for 12: Add g to P 13: while |P| < P do 14: Add a random legal chromosome sampled from {Dr } 15: end while 16: Return P

D. Heterogeneous Endpoint Pool The heterogeneous endpoint pool receives dispatcher assignments and executes LLM requests. Operationally, if node i hosts Mi models, CEMA-Router flattens all hosted node– model pairs into a destination set D. Destination-specific parameters, including token price, calibrated throughput, power, bandwidth, and latency, are keyed by that flattened destination. CEMA-Router uses a runtime-neutral endpoint abstraction based on dispatch, response, node-status, and completion records. Endpoints return responses to the Request Manager, report node status through the Node Profiler, and notify completion to the Request Monitor. The Profiler converts these observations into a common routing context for the BatchEvaluator, giving the router consistent inputs across GPU servers, edge devices, and remote endpoints with different runtime software and telemetry support. IV. CEMA-ROUTER D ESIGN This section describes how CEMA-Router turns the system context in Fig. 3 into a batch-level routing decision. Its goal is to assign a whole request batch to heterogeneous cloud–edge destinations while accounting for quality, token charge, energy, network delay, calibrated throughput, and queue backlog. A. Design Rationale CEMA-Router uses memetic search because the decision is both multi-objective and assignment-dependent. With R admitted requests and M legal destinations, the router faces M R possible assignments. The router must trade off profiled quality loss, monetary cost, energy consumption, and completion time, while the completion time term changes with the whole batch assignment: moving one request can alter endpoint service duration, queueing delay, and makespan. CEMA-Router therefore evaluates complete candidate assignments and captures interactions among requests in the same batch. Each candidate assignment is encoded as a chromosome. Each active gene corresponds to one admitted request and stores its selected destination; inactive positions beyond R are ignored by the evaluator. Legal destinations come from the current deployment state: unhealthy endpoints can be

removed, destination profiles provide calibrated throughput and power, the Request Monitor supplies Bd , and the quality store supplies request–destination quality-loss estimates. The encoding is compact enough for online routing while still supporting crossover, mutation, and local repair. B. Online Workflow Algorithm 1 gives the online workflow. For each batch, the router materializes the context used by the BatchEvaluator: request features, destination profiles, backlog estimates, profiled quality values, and objective weights. It then initializes a population with Algorithm 2, combining one load-aware chromosome with random legal assignments for diversity. The load-aware chromosome schedules longer requests first and assigns each to the currently earliest finishing destination. Each generation evaluates whole-batch assignments using BatchEvaluator, which computes and normalizes quality, cost, energy, and completion time terms to obtain J(g). Elitist selection retains the best assignments, and the router returns the best assignment found within the search budget. Although tuning indicated that population size and generation count can scale with batch size, the reported experiments use the fixed configuration in Section V, making the scalability results reflect workload size under a constant search budget. Algorithm 2 makes this seeding step explicit. The first chromosome uses request size, endpoint backlog, profiled inference time, and network delay before the random chromosomes are added. The resulting initial population contains one congestion-aware assignment for reducing stragglers together with random legal assignments for exploration. Ordering requests by predicted token length gives the largest requests first access to all feasible destinations, before the fastest endpoint has accumulated a large provisional queue. The earliest-finish score Ad combines inherited backlog Bd , profiled inference time τr,d , and network delay νr,d , so the seed chromosome is guided by estimated completion impact under the current deployment state. T̂infer (r, d) and T̂trans (r, d) denote the evaluator’s inference and network transmission time (Alg. 2, Line 8), derived from throughput and network profiles. C. Memetic Search and Local Repair Algorithm 3 defines the probabilistic Lamarckian repair operator used by CEMA-Router. After two-point crossover

Algorithm 3 Probabilistic Lamarckian Crossover and Repair 1: Input: parents g (1) , g (2) , legal sets {Dr } 2: Input: evaluator E, pc , pLS , repair budget L 3: Output: offspring h(1) , h(2) 4: if rand() < pc then 5: Create h(1) , h(2) by two-point crossover on active genes 6: if rand() < pLS then 7: for each offspring h do 8: Worst-gene repair: Identify bottleneck r ∗ = arg maxr (T̂infer + T̂trans + T̂queue )

9: Evaluate up to L/2 alternative legal destinations for r ∗ 10: 2-opt swap: Sample up to L/2 random request pairs (ra , rb ) 11: Evaluate fitness J after swapping destinations hra ↔ hrb 12: Write the single best improving move (repair or swap) back into h 13: end for 14: end if 15: else 16: Copy parents into h(1) , h(2) 17: end if 18: Return h(1) , h(2)

creates offspring, the router may invoke a bounded repair step on the active request genes. The repair searches two routing-specific neighborhoods. Worst-gene repair identifies the request whose predicted completion time dominates the current makespan and tests alternative legal destinations for that request. A 2-opt swap tests whether exchanging the destinations of two requests reduces queue imbalance or energy cost. All trial moves are scored by the current BatchEvaluator, and only the best improving move is written back into the offspring. In memetic algorithm terminology, the operator is Lamarckian because the locally improved genotype is returned to the population. Coupling repair to crossover confines exploitation to newly recombined assignments and bounds the extra evaluator calls added per offspring. The local repair step targets batch-level interactions that are not visible from an individual request alone. Several requests may each prefer the same fast endpoint, but the combined assignment can increase its service duration and dominate the batch makespan. Worst-gene repair moves work away from the request that currently limits batch completion, while 2opt swap can correct paired imbalance when two destinations would be better exchanged. Mutation adds exploration by changing active genes with probability pm . This repair policy is routing-specific: standard crossover mixes two parent assignments through position-level inheritance, whereas CEMA-Router uses the deployment objective to filter the local moves introduced after recombination. The child passed to survivor selection already contains the accepted reassignment, so useful local improvements can propagate without increasing the population size. This bounded form matches online routing, where the router must improve placement quality within a limited decision budget. D. Suitability for Batch-Level LLM Routing CEMA-Router fits batch-level LLM request routing for three reasons. First, its chromosome directly matches the router-service output: one destination per request in the current batch. Second, the evaluator uses the same deployment terms as the system model, including profiled quality, explicit token cost, endpoint power, calibrated batch throughput, network latency, and backlog. Third, the search is anytime, so the

TABLE II C LOUD – EDGE PROFILE USED FOR ROUTER B ENCH SIMULATION Role

Class

TFLOPS

Power

Models

Cloud Cloud Cloud Edge Edge

H100 A100 A100 A10 T4

312 312 250 125 65

300 W 300 W 250 W 150 W 70 W

GPT-4, Claude-v2 GPT-3.5, Claude-v1, Claude-instant Llama-2-70B, Mixtral-8x7B Yi-34B, Code-Llama-34B WizardLM-13B, Mistral-7B

TABLE III N ODE CONFIGURATION FOR REAL DEPLOYMENT. Role

GPU

Model

F1

FB

Cloud Edge Edge Edge Edge Cloud

A100 40GB A40 24GB A40 24GB A30 24GB A30 24GB V100 16GB

Qwen2.5-14B Phi-3.5-mini Llama-3.1-8B Qwen2.5-7B Mistral-7B SmolLM2-1.7B

4.02 1.47 1.70 2.23 2.22 0.682

287.5 82.9 167.0 158.6 167.4 36.37

router can return the best assignment found so far if the online decision budget is exhausted. The search cost is controlled by three explicit parameters: population size P , generation count G, and the local-repair budget L. For each generation, the dominant work is evaluating complete chromosomes under the four-term objective; repair adds a bounded number of evaluator calls around sampled legal reassignments. The router returns both the selected assignment and decision-time metrics, allowing Section V to report routing overhead alongside objective value. V. P ERFORMANCE E VALUATION A. Experimental Setup Datasets & Workloads: We utilize three datasets. The first is RouterBench [1], providing 36,497 trace-driven prompts evaluated across 11 LLMs with pre-computed ArmoRM [38] quality scores and token costs. RouterBench is commonly used to evaluate contemporary multi-LLM routing approaches, including Eagle [17], MixLLM [35], and UniRoute [14]. UniRoute is also our main quality–cost routing baseline, so RouterBench provides a shared basis for reproducible, isolated algorithmic comparison. For the real-world deployment, we evaluate 300 sampled questions from GSM8K [2] (scored by exact math match) and 600 from MMLU [3] (scored by exact choice match) using a zero-shot chain-of-thought format. Testbed & Infrastructure Profiles: For the controlled RouterBench stage, the 11 models are mapped to a simulated cloud–edge profile (Table II), with cloud nodes providing high throughput (up to 312 TFLOPS) but higher latency (40– 60 ms) and token costs, whereas edge nodes offer lower power and latency (5–8 ms). For the testbed, we instantiate a heterogeneous six-node testbed (Table III), hosting models ranging from SmolLM2-1.7B to Qwen2.5-14B. It is governed by real-time health checks, measured path TCP latencies (up to 197 ms for the remote V100), and calibrated saturated-batch throughput (FB ), dictating the continuous batching speedup. B. Experimental Procedure We compare CEMA-Router against a recent LLM routing reference and several simple deployment policies:

CEMA-Router: the proposed infrastructure-aware batch router, optimizing the normalized weighted objective over quality loss, monetary cost, energy consumption, and batch completion time. • UniRoute [14]: a quality-cost LLM router that clusters prompts by embedding similarity, estimates per-cluster error rates for each model, and scores each prompt–model choice using error rate plus a normalized cost term. In the RouterBench stage, we use λ = 0.05 for its qualitycost score; in the deployment stage, UniRoute uses the same profiled quality source as CEMA-Router but does not score queueing, endpoint power, or batch makespan. • Round Robin: a simple policy that assigns requests sequentially across currently available destinations, following a commonly evaluated routing baseline [31], [32]. • Least Loaded: a simple policy that selects the destination with the lowest tracked outstanding work under the same Request Monitor state, analogous to shortest-queue loadaware routing [32]. • Cloud Only and Edge Only: placement policies that restrict routing to cloud or edge destinations, respectively, and then choose within that destination set; both are established cloud–edge comparison strategies [31], [33]. All methods are invoked through the same router service interface. Request Manager forms batches, the method switch selects the routing policy, and the selected assignment is evaluated using the same BatchEvaluator. In RouterBench stage, all methods use the trace-driven profile in Table II. In testbed stage, they use destination profiles in Table III and the quality context from Section III-B. As noted in Sect. III-B, the testbed study focuses on profiled-quality routing. The real-deployment main comparison uses 10 runs per method on each workload. Request-scalability experiments use batch sizes R ∈ {10, 30, 50} with 10 runs per method and dataset. To stress endpoint-pool scaling beyond the testbed size, the controlled destination-scalability experiment fixes R = 30 and varies the available destinations as M ∈ {11, 22, 44, 88} over 30 independent RouterBench batches by replicating the profiled node–model destination catalogue. The router configuration uses a one-second batching interval, a maximum batch size of 128, and default objective weights (ωQ , ωC , ωE , ωT ) = (0.25, 0.25, 0.25, 0.25). In the testbed, the same weight vector is kept for implementation consistency across controlled and deployed experiments, while the result discussion emphasizes quality, energy, completion time, and decision overhead. The complete MA configuration and selection rationale are reported in Sect. V-C1. Performance is evaluated using the following metrics. Lower values are better for all reported metrics. A) Weighted Objective the normalized weighted objective J optimized for a given dataset, weight profile, and comparison setting. B) Quality Loss: raw profiled answer-quality loss, Qraw . C) Monetary Cost: explicit per-token charge, Craw . In the RouterBench simulations, monetary cost is computed using the TOKEN_COSTS model-pricing table from the RouterBench benchmark code [1], which specifies per-token prompt and •

TABLE IV CEMA-ROUTER HYPERPARAMETER CONFIGURATION . Parameter Population size P Generations G Crossover rate pc Mutation rate pm Local-search mode Local-search probability pLS Objective weights ω

Tested range

Selected

10–240 20–120 0.8–0.9 0.05–0.2 3 variants 0–1 -

30 20 0.9 0.1 Probabilistic Lamarckian 0.5 (0.25, 0.25, 0.25, 0.25)

completion prices, and optional fixed per-request charges, in USD. The self-hosted deployment assigns zero external API price to all endpoints, so the cost term is retained for objective compatibility but is non-discriminative in the deployed experiments; the deployment result tables therefore focus on quality, energy, completion time, and routing overhead. D) Energy Consumption: estimated processing and communication energy, Eraw , computed from the fixed node-power profile and modeled completion time. E) Completion Time: estimated batch completion time after dispatch, Traw . F) Routing Overhead: routing decision time measured separately from batch completion time. The raw terms are reported alongside main objective J so that the trade-off is auditable; a lower J can coexist with different movements in the raw metrics. C. Experimental Results 1) Hyperparameter Configuration: Table IV summarizes preliminary tuning in heterogeneous cloud–edge scenarios. Here, R is the number of requests in a batch. Population sizes near R–1.5R gave the best quality–runtime balance; larger populations improved quality by less than 2% while approximately doubling search time. With the default R = 30, we chose the lower end of this range, P = 30, to limit online overhead. Most runs converged within 5–23 generations, and generation 15 captured 93.5–98.8% of the reduction reached at generation 20, supporting G = 20. This P = 30, G = 20 configuration achieved mean fitness J = 0.079 with 0.325 s decision time in the controlled RouterBench evaluation. The operator sweep selected stable pm = 0.1 and probabilistic Lamarckian search, which gave the best mean fitness and reduced mean search time from 1.02 s to 0.18 s; pc = 0.9, repair probability 0.5, tournament size 3, and an eight-trial repair budget retain exploration within the online budget. No weight profile dominated every raw metric. 2) Controlled RouterBench Results: To evaluate CEMARouter under repeatable trace-driven conditions, we first use RouterBench with the weighted objective. This stage fixes the prompt set, RouterBench quality and cost data, and the fivenode cloud–edge profile in Table II. The controlled run uses R = 30 and reports the mean over five sampled batches. The results are presented in Table V. CEMA-Router ahieves the lowest weighted objective, while also reducing cost, energy consumption, and completion time relative to UniRoute. These results show that jointly optimizing the infrastructureaware objective can produce routing outcomes beyond qualitycost routing. The trace-driven evaluation isolates the sched-

TABLE V C ONTROLLED ROUTER B ENCH SIMULATION . W EIGHTED O BJECTIVE , e E, e C, e Te), AND RAW RESULTS N ORMALIZED (Q, Method CEMA-Router UniRoute Edge Only Least Loaded Round Robin Cloud Only

J 0.087 0.212 0.369 0.562 0.562 0.817

e Qraw Q 0.186 0.022 1.000 0.477 0.477 0.268

0.370 0.317 0.627 0.465 0.465 0.400

e C

Craw

0.097 0.061 0.000 0.598 0.598 1.000

0.0073 0.0061 0.0039 0.0260 0.0260 0.0413

e E

Eraw

Te

Traw

0.064 390.2 J 0.000 0.197 545.0 J 0.569 0.056 393.8 J 0.419 0.646 1033.4 J 0.526 0.646 1033.4 J 0.526 1.000 1438.4 J 1.000

0.456 s 1.441 s 1.188 s 1.347 s 1.347 s 2.281 s

TABLE VI R EAL - DEPLOYMENT CEMA VS . U NI ROUTE MAIN RESULTS . Data

Method

GSM8K GSM8K MMLU MMLU

CEMA UniRoute CEMA UniRoute

J

Qraw

Eraw

Traw

Dec.

0.0669 0.4894 0.0951 0.4886

0.3670 0.3362 0.3643 0.2746

12928 J 19541 J 12947 J 16087 J

2.443 s 3.351 s 2.342 s 3.351 s

1.059 s 0.523 s 1.757 s 1.612 s

Fig. 5. Raw metric trade-offs between CEMA-Router and UniRoute in the real deployment. It shows the different raw-metric priorities induced by modellevel and infrastructure-aware routing. Fig. 4. Real-deployment main comparison on GSM8K and MMLU under the weighted infrastructure-aware objective (R=30, 10 runs).

uler, BatchEvaluator, and memetic search path, whereas the real deployment also evaluates real endpoint health checks, path-latency measurement, request dispatch, and completion monitoring. The trace-driven results further show that CEMARouter incurs lower routing overhead, requiring 0.325 s on average compared with 0.481 s for the UniRoute baseline. 3) Real Deployment Results: Figure 4 and Table VI exhibit the real-deployment comparison on GSM8K and MMLU, exercising the full runtime paths defined in Sect. III. On both datasets, CEMA-Router achieves substantially lower values in the weighted objective J. These real-deployment values show where CEMA-Router’s additional routing context matters. On GSM8K, the weighted objective decreases by 86.3%, while energy consumption decreases by 33.8% and completion time decreases by 27.1%. On MMLU, the same pattern holds: the objective decreases by 80.5%, energy decreases by 19.5%, and completion time decreases by 30.1%. Both routers use the same profiled-quality input, but CEMA-Router also accounts for endpoint load, network delay, saturated batch throughput, and energy profiles. A semantically strong destination can have an unfavorable queuing or energy profile for the current batch; batch-level assignment allows CEMA-Router to choose a lower-straggler or lower-energy placement because of the weighted objective. 4) Metric Trade-off Analysis: Figure 5 decomposes the CEMA–UniRoute trade-off. CEMA-Router routes more work toward endpoints that reduce energy consumption and completion time under the deployment profile. UniRoute instead follows its model-level quality-cost objective. The result is an objective-level improvement, with different raw metrics moving in different directions. This decomposition clarifies the role of each comparator. UniRoute is a strong reference for model-level qualitycost routing. CEMA-Router adds queuing, calibrated batch throughput, endpoint power profiles, and straggler completion time to the decision, exposing how deployment terms change

the selected assignment under the configured objective. 5) Request Scalability Results: Figure 6 and Table VII show request scalability for R = 10, 30, 50. This experiment stresses the connection between Section III-A, which admits larger batches, and Section IV, which searches over longer assignment chromosomes. CEMA-Router preserves a lower weighted objective (lower is better) than UniRoute across both datasets and all batch sizes. The same experiment shows that its decision time grows with batch size and is higher than UniRoute in the real deployment. 6) Destination Scalability Results: Figure 7 shows controlled RouterBench destination scalability for M = 11, 22, 44, 88 available destinations with R = 30. CEMARouter achieves the lowest weighted objective at all scales, improving from J = 0.164 at M = 11 to J = 0.106 at M = 88, whereas UniRoute increases from J = 0.214 to 0.376. Its bounded search budget keeps decision time subsecond, increasing from 0.077 s to 0.171 s as M grows, compared with approximately 0.46–0.47 s for UniRoute. D. Discussion CEMA-Router demonstrates the value of treating LLM routing as a joint infrastructure-aware assignment problem rather than considering model quality or system efficiency in isolation. Across the controlled RouterBench study and the sixendpoint deployment, CEMA-Router achieves lower weighted objective values than UniRoute and the deployment-oriented heuristics. These results indicate that combining requestspecific quality estimates with node throughput, network conditions, energy profiles, and dynamic backlog information can identify operating points that are not visible to quality–cost or load-only policies. Nevertheless, the proposed weighted objective represents a configurable deployment preference rather than universal dominance across every raw metric. The raw results expose the underlying optimisation tradeoff. CEMA-Router reduces estimated energy consumption and straggler-dominated completion time by directing requests away from congested or energy-intensive destinations, but it may accept a higher quality loss than the quality-oriented

Fig. 7. Controlled RouterBench destination scalability with increasing available destinations. Shaded bands denote 95% confidence intervals over 30 independent batches.

Fig. 6. Request scalability in the real deployment (10 runs). TABLE VII R EQUEST SCALABILITY SUMMARY FOR CEMA AND U NI ROUTE . Dataset

R

CEMA J

UniRoute J

CEMA dec.

UniRoute dec.

GSM8K GSM8K GSM8K MMLU MMLU MMLU

10 30 50 10 30 50

0.0005 0.0669 0.3506 0.0123 0.0951 0.4035

0.5239 0.4894 0.4826 0.5389 0.4886 0.4658

0.437 s 1.324 s 2.175 s 0.435 s 1.313 s 2.548 s

0.179 s 0.529 s 0.978 s 0.339 s 1.083 s 1.758 s

UniRoute policy. In the controlled evaluation, it can also incur a slightly higher token cost. Thus, the preferred assignment depends on application requirements: latency- or energy-sensitive pervasive services may tolerate a small quality reduction, whereas safety-critical or accuracy-sensitive applications may require a minimum-quality constraint. The objective weights allow operators to express these preferences, while a quality threshold can be incorporated when degradation beyond a specified level is unacceptable. Because all endpoints in the self-hosted deployment have zero external API cost, the deployed comparison primarily reflects the trade-off among quality, energy, and completion time. The results also illustrate the benefit of batch-level placement. A per-query policy may independently assign several requests to the same apparently attractive endpoint without considering the contention created by their combined execution. CEMA-Router instead evaluates complete assignments using inherited backlog, predicted request length, network delay, and calibrated batch throughput. Its load-aware seed initially distributes longer requests according to expected finishing time, while the repair and swap operators subsequently correct bottlenecks at the batch level. This advantage is most pronounced when endpoint capabilities and request sizes are heterogeneous and several requests compete for the same high-performance destination. When batches are small or one endpoint dominates all metrics, the additional benefit of joint placement is expected to decrease. This additional coordination introduces routing overhead. The controlled simulation primarily operates on precomputed quality and infrastructure profiles, whereas the deployed pipeline additionally performs online prompt embedding,

FAISS-based quality retrieval, profiler access, context construction, and router-service communication. These operations, together with the memetic search, explain why deployment decision time is higher and more workload-dependent than controlled simulation time (Fig. 6 and 7). For small batches, the overhead remains within the batching interval; for larger batches, it can exceed that interval and offset part of the execution-time saving. CEMA-Router is therefore most beneficial when the avoided queuing and inference delay exceeds its routing cost. Practical optimisations include caching prompt embeddings and retrieved profiles, vectorising fitness evaluation, applying adaptive population and generation budgets, terminating search when improvement stalls, and overlapping routing for the next window with current-batch execution.

VI. C ONCLUSION AND F UTURE W ORK This paper presented CEMA-Router, an infrastructure-aware batch router for LLM inference in cloud–edge systems. The system seamlessly integrates request batching, dynamic backlog monitoring, and RAG-profiled quality signals with a customized memetic assignment optimizer. Across controlled RouterBench traces and a real six-node GSM8K/MMLU deployment, CEMA-Router outperforms the quality-cost-centric UniRoute and standard deployment heuristics. Notably, it reduces the overall infrastructure-aware objective by over 80%, effectively cutting energy consumption by up to 33.8% and batch completion time by 27.1%. The decision-time measurements identify practical operating regimes for populationbased batch routing and support scalable deployment mechanisms such as early-stop budgets, asynchronous planning, and adaptive batching. Overall, the results demonstrate that treating infrastructure heterogeneity, response quality, energy consumption, and continuous-batching queueing as joint runtime signals is crucial. By moving beyond isolated model selection, CEMA-Router provides a vital step toward sustainable and latency-bounded pervasive AI. Future work will extend CEMA-Router toward more dynamic pervasive environments, including mobility-driven wireless variation, endpoint churn, and online profile refresh. We also plan to integrate continuous bandwidth and power telemetry from deployed edge devices to evaluate routing stability under broader long-term workloads.

R EFERENCES [1] Q. J. Hu, J. Bieker, X. Li, N. Jiang, B. Keigwin, G. Ranganath, K. Keutzer, and S. K. Upadhyay, “RouterBench: A benchmark for multiLLM routing system,” arXiv preprint arXiv:2403.12031, 2024. [2] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. [3] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in International Conference on Learning Representations, 2021. [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [5] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901. [6] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “ORCA: A distributed serving system for Transformer-based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 2022, pp. 521–538. [7] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 2024. [8] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodputoptimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 2024. [9] P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024. [10] R. Prabhu, A. Nayak, J. Mohan, R. Ramjee, and A. Panwar, “vattention: Dynamic memory management for serving LLMs without PagedAttention,” arXiv preprint arXiv:2405.04437, 2024. [11] I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, “RouteLLM: Learning to route LLMs with preference data,” in Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. [12] D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Rühle, L. V. S. Lakshmanan, and A. H. Awadallah, “Hybrid LLM: Cost-efficient and quality-aware query routing,” in The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. [13] D. Stripelis, S. M. Preum, H. Ekbia, and J. L. Ambite, “PolyRouter: A multi-LLM querying framework,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. [14] W. Jitkrittum, H. Narasimhan, A. S. Rawat, J. Juneja, Z. Wang, C.-Y. Lee, P. Shenoy, R. Panigrahy, A. K. Menon, and S. Kumar, “Universal model routing for efficient LLM inference,” arXiv preprint arXiv:2502.08773, 2025. [15] S. Jha et al., “Learned LLM request routing with deep Q-networks,” in Proceedings of the IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2024. [16] Y. Wang, K. Chen, H. Tan, and K. Guo, “TABI: An efficient multi-level inference system for large language models,” in Proceedings of the 18th European Conference on Computer Systems. ACM, 2023, pp. 233–248. [17] Z. Zhao, S. Jin, and Z. M. Mao, “Eagle: Efficient training-free router for multi-LLM inference,” in Workshop on Machine Learning for Systems at NeurIPS, 2024. [18] T. Feng, Y. Zhu, Z. Ma, X. Tan, T. Liu, and J. Li, “GraphRouter: A graph-based LLM router for semantic analysis,” arXiv preprint arXiv:2410.03834, 2024. [19] S. Mei et al., “OmniRouter: Global constrained routing for LLM inference via Lagrangian decomposition,” in Proceedings of the ACM Web Conference, 2025. [20] M. Všakota, M. Peyrard, and B. Faltings, “Fly-swat or cannon: Costeffective language model choice via meta-modeling,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.

[21] D. Ding et al., “Best-of-n routing for large language models,” arXiv preprint, 2025. [22] Z. Li et al., “Bandit-based LLM routing for cost-effective inference,” arXiv preprint, 2025. [23] L. Pan et al., “Route any question: MLP-based routing for large language models,” arXiv preprint, 2025. [24] Y. Zhang et al., “Query-aware LLM routing with learning-based predictors,” arXiv preprint, 2025. [25] W. Song, Z. Huang, C. Cheng, W. Gao, B. Xu, G. Zhao, F. Wang, and R. Wu, “IRT-Router: Effective and interpretable multi-LLM routing via item response theory,” arXiv preprint arXiv:2506.01048, 2025. [26] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and challenges,” IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637–646, 2016. [27] M. Satyanarayanan, “The emergence of edge computing,” Computer, vol. 50, no. 1, pp. 30–39, 2017. [28] D. K. Alqahtani, M. A. Rodriguez, M. A. Cheema, H. Rezatofighi, and A. N. Toosi, “ECORE: Energy-conscious optimized routing for deep learning models at the edge,” in Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 2026, pp. 1–10. [29] W. Kim, M. Jeong, S. Shin, J. Yi, and Y. Lee, “MERCI: Adaptive multiexpert inference for dynamic and large-vocabulary vision perception,” in Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 2026, pp. 1–10. [30] D. Mvondo, B. Teabe, and N. Parlavantzas, “Efficient memory usage for edge FaaS platforms,” in Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 2026, pp. 1–10. [31] J. Yu et al., “Efficient LLM inference routing over cloud-edge networks,” arXiv preprint, 2025. [32] A. Jain et al., “Intelligent LLM serving with deep reinforcement learning,” arXiv preprint, 2024. [33] Z. Yang, Y. Yang, C. Zhao, Q. Guo, W. He, and W. Ji, “PerLLM: Personalized inference scheduling with edge-cloud collaboration for diverse LLM services,” arXiv preprint arXiv:2405.14636, 2024. [34] Q. Shi et al., “Inference dynamics: Adaptive cloud-edge scheduling for large language model services,” arXiv preprint, 2025. [35] X. Wang, Y. Liu, W. Cheng, X. Zhao, Z. Chen, W. Yu, Y. Fu, and H. Chen, “MixLLM: Dynamic routing in mixed large language models,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025, arXiv:2502.18482. [36] Y. Mei, Y. Zhuang, X. Miao, J. Yang, Z. Jia, and R. Vinayak, “Helix: Distributed serving of large language models via max-flow on heterogeneous GPUs,” arXiv preprint arXiv:2406.01566, 2024. [37] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with GPUs,” IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535– 547, 2021. [38] H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang, “Interpretable preferences via multi-objective reward modeling and mixture-of-experts,” arXiv preprint arXiv:2406.12845, 2024.

Related documents

• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference#189269
• LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices#1061659
• Smart Adaptive Computing Across the Continuum: LLMs in IoT-Edge-Cloud Resource Management#987105
• The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices#999925
• Scheduling the Unschedulable: Taming Black-Box LLM Inference at Scale#2550