Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving Takeshi Yoshimura∗
arXiv:2604.15732v1 [cs.DC] 17 Apr 2026
IBM Research - Tokyo Tokyo, Japan [email protected]
Valentijn Dymphnus van de Beek∗ Delft University of Technology Delft, Netherlands [email protected]
Tatsuhiro Chiba IBM Research - Tokyo Tokyo, Japan [email protected]
Abstract
1
Distributed LLM serving systems optimize per-request latency and throughput. However, under long-context workloads, inference accuracy becomes more variable. When incorrect responses trigger retries, accuracy directly translates into cumulative user-visible delay that is not captured by single-shot latency metrics. In this work, we argue that under long-context serving, accuracy becomes speed through retry dynamics. We introduce Time-to-Correct-Answer (TTCA), a metric that measures the wall-clock time required to obtain the first correct response. Our measurement study shows that prompt characteristics such as length and language amplify accuracy variance, which inflates TTCA. We demonstrate Lightweight Accuracy-Aware Routing (LAAR), a capability-based routing design that reduces TTCA. Our results suggest that in longcontext distributed serving, accuracy should be treated as a first-class systems objective.
Large language models (LLMs) are increasingly used in longcontext settings, such as retrieval-augmented generation [9], code assistants [4], tool-using agents [23], and document understanding [16]. Recent systems work demonstrates that serving long prompts with hundreds of thousands or even millions of tokens is now technically feasible [24, 25]. Long-context workloads fundamentally change the performance characteristics of LLM inference. Prefill computation dominates cost [25], memory bandwidth becomes a primary bottleneck [19], and cache management plays a critical role in overall efficiency [11]. These factors make per-request load and cache locality more important in long-context serving. Distributed LLM serving systems have introduced loadaware, session-affinity, and cache-affinity routing strategies to optimize latency and throughput [8, 15, 26]. However, long-context workloads affect not only performance but also accuracy. Our measurements show that accuracy varies with prompt length, model, and language under long inputs. This variation complicates routing among heterogeneous instances in a cluster. When incorrect answers trigger retries or escalation to larger models, routing mistakes inflate cumulative end-to-end latency. In long-context distributed serving, accuracy therefore becomes a system-level performance factor, and one-shot latency alone no longer captures user-visible delay. In this sense, higher accuracy can directly reduce end-to-end latency. Accuracy therefore effectively becomes a form of speed. To capture this effect, we introduce Time-to-Correct-Answer (TTCA) as a complementary objective for evaluating routing strategies in long-context distributed serving. TTCA is the wall-clock time required to obtain the first correct answer. This metric highlights a systems challenge: routers must account for both correctness and routing overhead when they rely on prompt characteristics. Prior work explores semantic or content-aware routing that analyzes prompt intent or reasoning requirements with embedding-based classification and related prompt analysis techniques [21]. Semantic routing is well suited for environments where task heterogeneity is high and intent-level differentiation directly determines inference strategy (e.g., reasoning vs. non-reasoning paths, specialist vs. generalist models). In contrast, our setting emphasizes stability under
CCS Concepts: • Computer systems organization → Cloud computing; • Computing methodologies → Distributed artificial intelligence; Natural language processing. Keywords: Distributed LLM serving, Long-Context LLMs, Accuracy-Aware Routing, Content-Aware Scheduling,Timeto-Correct-Answer (TTCA) ACM Reference Format: Takeshi Yoshimura, Valentijn Dymphnus van de Beek, and Tatsuhiro Chiba. 2026. Accuracy Is Speed: Towards Long-ContextAware Routing for Distributed LLM Serving. In The 6th Workshop on Machine Learning and Systems (EuroMLSys ’26), April 27–30, 2026, Edinburgh, Scotland Uk. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3805621.3807652 ∗ These authors contributed equally to this work.
This work is licensed under a Creative Commons Attribution 4.0 International License. EuroMLSys ’26, Edinburgh, Scotland Uk © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2605-7/26/04 https://doi.org/10.1145/3805621.3807652
Introduction
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk
long-context accuracy variability. In this setting, capability mismatches must be avoided and control-plane overhead must remain bounded. Rather than replacing semantic routing, we explore complementary and lower-complexity designs. We focus on retryable, task-oriented workloads in which a request has a stable target outcome and response correctness can be determined programmatically or via a reliable evaluation metric. This assumption matches many existing LLM benchmarks and defines the scope in which TTCA is meaningful. To isolate the system-level effect of accuracyinduced retries, our evaluation uses controlled key-value retrieval workloads that sweep context length and language. Although synthetic, this workload provides a clean probe of context-length-dependent accuracy degradation that also appears in realistic settings such as document QA, summarization verification, and tool-based agents. In this work, we introduce Lightweight Accuracy-Aware Routing (LAAR), a capability-based routing design for longcontext workloads. LAAR improves TTCA while keeping control-plane overhead bounded. LAAR uses only lightweight prompt-derived features, such as prompt length and language, without invoking additional models or full-sequence semantic analysis. This work makes the following contributions: 1. Characterizing long-context accuracy variability. We demonstrate that prompt length, language, and model selection amplify inference accuracy variance under longcontext workloads. 2. Revisiting routing objectives through TTCA. We introduce Time-to-Correct-Answer (TTCA) as a metric that captures retry-induced latency inflation, and argue that under long-context serving, accuracy becomes a systemlevel performance factor. 3. Lightweight Accuracy-Aware Routing (LAAR). We introduce a capability-based routing design that stabilizes TTCA. It reduces capability mismatches without relying on semantic intent inference. 4. Empirical validation in long-context settings. We show that LAAR reduces mean TTCA in most evaluated settings across context lengths and languages. It does so while keeping control-plane overhead bounded.
2
Background and Related Work
2.1
Routing in Distributed LLM Serving
Routing and scheduling algorithms are critical for lower latency and higher throughput in distributed LLM serving. Performance-aware LLM load balancing [8] introduces heuristic- and RL-guided routing to adapt to workload characteristics. Deferred Prefill [17] mitigates decode-phase stalls by optimizing prompt departure timing. ShapeShifter [12] formulates LLM cluster management as a bin-packing problem and dynamically rebalances multi-dimensional resource
Takeshi Yoshimura, Valentijn Dymphnus van de Beek, and Tatsuhiro Chiba
utilization to improve GPU utilization. SkyWalker [22] extends routing to multi-region settings with prefix-aware traffic control. Semantic routing approaches classify prompts by their reasoning requirements to guide inference-mode selection in vLLM-based stacks [21]. Collectively, these systems treat routing as a resource allocation and load-balancing problem. They optimize throughput, tail latency, or resource utilization. However, existing routing policies assume that model correctness is stable once a request is assigned. They therefore optimize for computational efficiency without modeling how correctness instability under long contexts can trigger retries and inflate user-visible delay.
2.2
Long-Context Accuracy Variability
A growing body of work systematically studies how LLM performance degrades under long contexts. Lost in the Middle [14] reveals strong positional bias and accuracy drops when relevant information appears in the middle of long inputs. SCBench [13] shows that long-context workloads often involve shared contexts, KV-cache reuse, and multiround interactions that are missed by single-request evaluation. RULER [6] demonstrates that the effective context length of models is often far below their advertised maximum, with task-dependent degradation. LongBench [2] and Ada-LEval [20] further report accuracy declines as input size grows, while MLNeedle [5] highlights language-dependent sensitivity in multilingual retrieval. These prior studies establish that long-context accuracy is highly sensitive to position, task type, language, and model architecture. However, most prior work evaluates long-context performance as a capability or benchmark problem rather than a serving-time systems problem. They characterize degradation across positions, lengths, languages, or request patterns, but do not examine how accuracy variability across requests affects retry probability in real serving systems. In distributed LLM serving, such variability can amplify user-visible delay when incorrect responses trigger retries. Together, these gaps motivate our systems view: we treat long-context accuracy variability as a routing-time concern, quantify how it changes across models, languages, and context lengths, and evaluate routing with TTCA.
3
Analysis of Model Accuracy and Latency
To understand how accuracy variability impacts retry behavior and routing decisions, we analyze single-shot model accuracy and latency under different context lengths and languages. Prior sections focused on degradation within a single model. Here, we emphasize cross-model variability, ranking changes, and their implications for multi-model serving.
Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk
3.1
Granite3.1-2B
Llama3.1-Swallow-8B
Granite3.1-8B
Phi3-medium
Single-shot experiments
3.2
Routing Implications
In summary, accuracy degradation slopes and effective context thresholds vary by model and language, whereas latency ranking is comparatively stable. The best routing choice is therefore not stationary: model rankings change with prompt length and language, so routers must be contextand language-aware rather than fixed. Model size does not monotonically predict long-context accuracy; smaller models such as Phi3-mini outperform larger 8B-class models in several ranges, particularly at mid-length contexts (8K– 16K). Language further amplifies this non-monotonicity, as
Accuracy (%)
100 80 60 40 20 0 4K
8K
16K
32K
64K
Context Length
(a) English prompts Granite3.1-2B
Llama3.1-Swallow-8B
Granite3.1-8B
Phi3-medium
Phi3-mini
Accuracy (%)
100 80 60 40 20 0 4K
8K
16K
32K
64K
Context Length
(b) Japanese prompts Granite3.1-2B
Llama3.1-Swallow-8B
Granite3.1-8B
Phi3-medium
Phi3-mini
100
Accuracy (%)
We run experiments with vLLM v0.16.0 on NVIDIA A100 GPUs (80GB VRAM). Our dataset consists of 100 modified SCBench KV-lookup queries, split into two disjoint sets of 50. This section reports single-shot results on the first split, which we also use to fit the offline estimators used by LAAR in Section 5. Every original query consists of a large context and a small question. The contexts start with the prefix string “JSON data: ” followed by a large JSON dictionary of random UUID key-value pairs. The question is an English sentence that asks for the value associated with a key (e.g., “Key: 6ab6ea3e-f288-4f33-ba46-7f42bb75b03f. The value associated with the specified key is:”). We truncate the original contexts into 4K, 8K, 16K, 32K, and 64K tokens and translate the contexts and questions into Japanese and Chinese. We run them sequentially on a single vLLM server that hosts Granite3.1-2B, Granite3.1-8B [7], Phi3-mini, Phi3-medium [1], or Llama3.1-Swallow-8B [3, 18]. We reuse the original SCBench script to identify correct answers and estimate accuracy for each model. Figure 1 reports accuracy across models, languages, and context sizes. Phi3-mini was often the most accurate across context lengths and notably outperformed Phi3-medium. Granite3.1-2B underperformed Granite3.1-8B at smaller context lengths, but outperformed it at 32K and 64K. Llama3.1Swallow-8B exhibited a clear threshold-like failure: it remained competitive up to 16K (and was often strong at 4K– 16K), but collapsed sharply at 32K and further at 64K across languages. We did not observe the single best model in terms of accuracy. From a routing perspective, secondary model choices also matter when serving multiple queries concurrently in a cluster, but we still did not observe a consistent choice. The parameter scale did not help predict accuracy in long-context KV lookups. Figure 2 shows the latency of each model with 64K tokens. In contrast to the accuracy results, latency is consistent across context lengths and languages, but it is affected by the model. Phi3-mini showed higher latency than Granite3.1-2B and Llama3.1-Swallow-8B. The latency ranking is therefore more stable than the accuracy ranking.
Phi3-mini
80 60 40 20 0 4K
8K
16K
32K
64K
Context Length
(c) Chinese prompts
Figure 1. Mean accuracy of KV lookups with different context lengths and languages. Accuracy degradation is modeland language-dependent, and smaller models (e.g., Phi3mini) can outperform larger ones.
a model that performs best in English is not necessarily optimal in Japanese or Chinese. A policy that always selects the single highest-accuracy model can therefore be brittle and may concentrate load. In practice, routers should prefer competitive models that jointly balance accuracy and latency, which motivates TTCA-aware routing.
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Average Latency (s)
English
Japanese
Takeshi Yoshimura, Valentijn Dymphnus van de Beek, and Tatsuhiro Chiba
5
Chinese
Lightweight Accuracy-Aware Routing
Lightweight Accuracy-Aware Routing (LAAR) ranks candidate models with a lightweight proxy for expected time-tosuccess. The proxy combines estimates of success probability and serving latency under control-plane constraints.
30
20
10
5.1
0
Given a request 𝑥 and candidate model 𝑚, let 𝑄 (𝑚, 𝑥) denote the expected success probability and 𝐿(𝑚, 𝑥) the expected latency. LAAR uses the following practical heuristic:
B 1-2B 1-8B dium ow-8 -me ite3. ite3. wall Phi3 Gran Gran .1-S 3 a Llam
i
-min
Phi3
Routing Objective and Score
𝐿(𝑚, 𝑥) . 𝑄 (𝑚, 𝑥) Under an idealized geometric retry model with stationary per-attempt success probability and latency, this score corresponds to expected time-to-success; in practice, LAAR uses 𝐿(𝑚, 𝑥)/𝑄 (𝑚, 𝑥) as a lightweight proxy for TTCA that is simple to evaluate online. This approximation assumes independent retries with stationary success probability. At routing time, LAAR computes this cost for each model and selects: 𝑚 ∗ = arg min 𝑐𝑜𝑠𝑡 (𝑚 | 𝑥). 𝑐𝑜𝑠𝑡 (𝑚 | 𝑥) =
Figure 2. Mean latency for 64K contexts of KV lookups with five models. We omit other context sizes because they showed the same latency ranking among models.
4
Time-to-Correct-Answer (TTCA)
From a user perspective, latency is not merely the time to receive an answer, but the time to receive a correct answer. Under short-context workloads with stable accuracy, latency can often be approximated by a single inference time. However, under long-context workloads, accuracy variability increases, and incorrect responses may trigger retries—either explicitly by users or implicitly by upstream systems. In such cases, the effective delay experienced by users accumulates across attempts. To capture this retry-induced latency inflation, we introduce Time-to-Correct-Answer (TTCA): the wall-clock time elapsed from the first attempt until the first correct answer is obtained. For attempt 𝑖, let latency be ℓ𝑖 and correctness be 𝐶𝑖 ∈ {0, 1}. Correctness is determined by task-specific automatic evaluation (e.g., exact match in needle-in-a-haystack tasks). If the first correct attempt is 𝐾 = min{𝑖 | 𝐶𝑖 = 1}, then TTCA is defined as TTCA =
𝐾 ∑︁
ℓ𝑖 .
𝑖=1
Intuitively, TTCA measures how long a user waits until they obtain a correct answer under retries. In practice, retries are capped at 𝑅 attempts. If no correct answer appears within 𝑅 attempts, the request is treated as failed and TTCA is rightÍ𝑅 censored at 𝑖=1 ℓ𝑖 . Unlike conventional per-request latency metrics, TTCA explicitly models retry dynamics induced by accuracy variability and therefore better reflects user-visible delay under long-context workloads. We use TTCA as an evaluation objective rather than a production telemetry metric.
𝑚
This trade-off allows a slower model with higher success probability to outrank a faster but less reliable one. We deliberately adopt simple estimators to keep routing efficient and scalable in the control plane. To account for retry behavior, LAAR applies a penalty to models that have been previously selected for the same request. This prevents repeated selection of the same model when its initial attempt fails. It also encourages exploration of alternative models and improves time-to-success in multi-try scenarios. This retry penalty is therefore a pragmatic extension beyond the idealized geometric interpretation above. This is critical in practice because deterministic decoding can otherwise lead to repeated failures. 5.2
Estimating Success Probability
To estimate the expected success probability 𝑄 (𝑚, 𝑥), LAAR uses a lightweight capability model that estimates the probability that a model produces a correct response given coarse, easily extractable request features. Each request is mapped to a small set of features, such as language, task type, and input length bucket. These features are inexpensive to extract and do not require deep semantic analysis. In our current evaluation, we focus on a single task (KV lookup), so the task-type feature remains constant and is not explicitly used. For each model, we train a logistic regression model offline that predicts the probability of success based on these features. We choose logistic regression due to its simplicity, interpretability, and low inference overhead. The resulting model is compact and can be evaluated efficiently at runtime. In practice, this yields a function:
Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving
𝑄 (𝑚, 𝑥) ∈ [0, 1], which reflects the model’s capability under the given request conditions. This capability model serves as a static prior over model capabilities. It complements the runtime latency estimation in Section 5.1. It is independent of runtime load and can therefore be precomputed and reused without introducing additional control-plane overhead. 5.3
Estimating Latency
To estimate 𝐿(𝑚, 𝑥), we model latency as a function of the request size and the current load of the model. Let 𝑐 (𝑚) denote the empirical seconds per generated token for model 𝑚, obtained from offline measurements. Let 𝑇 (𝑥) denote the estimated number of tokens for request 𝑥, derived from the same length bucket used in Section 5.2. We further denote by 𝑅(𝑚) the number of tokens currently being processed or waiting in the queue at model 𝑚. The expected latency is then approximated as: 𝐿(𝑚, 𝑥) = 𝑐 (𝑚) · 𝑇 (𝑥) + 𝛼 · 𝑅(𝑚) , where 𝛼 is a constant that captures the impact of ongoing and queued work on latency. We use 0.7 for our experiments in Section 6. This formulation captures two key effects: (1) longer inputs incur higher processing cost, and (2) queueing and contention increase effective latency. Because all inputs to this model are directly observable at runtime, it avoids expensive prediction pipelines and remains compatible with controlplane constraints. This design deliberately avoids complex latency predictors. It favors robustness and low overhead in the control plane over extra predictive accuracy. 5.4
Implementation
We implement LAAR as an Envoy Endpoint Picker (EPP) policy via the external processing filter. For a given request, our EPP computes a score for each candidate endpoint that serves models. It then uses llm-d’s MaxScorePicker [10] to forward the request to the endpoint with the maximum score. At request time, Envoy invokes our extension. The extension extracts lightweight request features, evaluates the success probability 𝑄 (𝑚, 𝑥) and the expected latency 𝐿(𝑚, 𝑥) for each model, and selects the model that minimizes the cost 𝑐𝑜𝑠𝑡 (𝑚 | 𝑥), i.e., maximizes the inverted cost for MaxScorePicker. Our routing logic relies on lightweight CPU-side computations within the request processing path. Feature extraction parses a short sampled substring of the user text. Language is inferred from character classes (ASCII vs. CJK and Hiragana/Katakana). Thus, the core decision is realized as a compact if-else cascade to avoid semantic parsing or auxiliary model inference.
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk
All computations are constant-time per target model. Where M is the set of candidate models evaluated at routing time, the overall complexity is therefore 𝑂 (|M |). The router relies only on locally available information and does not require cross-backend coordination or global state. This design satisfies control-plane boundedness: routing decisions incur minimal overhead and do not become a bottleneck even under high request rates. To support retry-aware routing, the router requires information about previously selected models for the same request. We propagate this information via request metadata: the router returns the selected model identifier to the client, which includes it in subsequent retry requests. This enables the router to apply penalties to previously attempted models without any server-side session state.
6
Preliminary Evaluation
Our primary goal in this evaluation is to examine whether LAAR improves TTCA under long-context workloads. For comparison, we evaluate two representative baselines, loadaware routing and session-affinity routing, by directly invoking the llm-d implementation [15]. We replace only the scoring logic to keep the gateway and forwarding path identical among all the experiments. 6.1
Experimental Setup
We evaluate a distributed serving stack composed of an Envoy-based EPP router and a pool of vLLM instances. Our primary metric is TTCA (Section 4), with up to ten request retries, i.e., a retry cap 𝑅 = 10. All experiments run vLLM v0.16.0 on NVIDIA A100 GPUs (80GB VRAM) interconnected via a 10Gbps network. The cluster contains five vLLM instances, which run the five models described in Section 3. The number of concurrent requests is set to eight. We use deterministic decoding (temperature = 0) to reduce output variance and isolate routing effects. Workloads consist of SCBench key-value lookups with different context sizes and languages, as shown in Section 3. To eliminate cross-request KV-cache reuse effects, we carefully iterate over different query sets. This evaluation uses only the held-out second 50-query split. The first split from Section 3 is used offline to fit the success probability model 𝑄 (𝑚, 𝑥) and estimate latency 𝐿(𝑚, 𝑥) as described in Section 5. 6.2
TTCA and Success Rate
Figure 3 shows the relationship between TTCA and success rate across retries for different routing policies. Overall, retries allowed the system to improve the success rate at the cost of increased TTCA, i.e., retry-induced latency. As illustrated in the figure, regardless of routing strategy or
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk
Takeshi Yoshimura, Valentijn Dymphnus van de Beek, and Tatsuhiro Chiba vs. load
LAAR
4
4
2
2
2
0
0
0
0
20 40 60 80 100
0
20 40 60 80 100
0
(b) 4K tokens, Japanese
4
4
2
2
2
0 0
20 40 60 80 100
20 40 60 80 100
10.0
10.0
7.5
7.5
7.5
5.0
5.0
5.0
2.5
2.5
2.5
0.0
0.0 0
20 40 60 80 100
(g) 16K tokens, English
0.0 0
20 40 60 80 100
0
(h) 16K tokens, Japanese 40
40
30
30
30
20
20
20
10
10
10
0 0
20 40 60 80 100
(j) 32K tokens, English
0 0
20 40 60 80 100
0
(k) 32K tokens, Japanese 150
150
100
100
100
50
50
50
0 0
20 40 60 80 100
(m) 64K tokens, English
20 40 60 80 100
(l) 32K tokens, Chinese
150
0
20 40 60 80 100
(i) 16K tokens, Chinese
40
0
20 40 60 80 100
(f) 8K tokens, Chinese
10.0
0 0
20 40 60 80 100
(n) 64K tokens, Japanese
30 20 10 0 EN JA ZH EN JA ZH EN JA ZH EN JA ZH EN JA ZH 4K 8K 16K 32K 64K
0
(e) 8K tokens, Japanese
40
−10
0 0
(d) 8K tokens, English
20 40 60 80 100
(c) 4K tokens, Chinese
4
0
vs. session
50
4
(a) 4K tokens, English
mean TTCA (sec)
session
TTCA speedup (%)
load
0
20 40 60 80 100
(o) 64K tokens, Chinese
Correct Ratio (%)
Figure 3. TTCA and success rate for retryable UUID keyvalue lookups in 4K, 8K, 16K, 32K, and 64K contexts under English, Japanese, and Chinese with load-aware, sessionaffinity, and LAAR routing. Retries monotonically increase both TTCA and the success rate, and we run up to ten retries. However, LAAR finishes within at most five attempts because it avoids reusing failed models.
language, shorter contexts consistently achieved higher success rates, while success rates degraded as context length increased. LAAR sometimes started with a lower first-attempt success rate than the other methods, yet achieved lower TTCA over the retry process. This indicates that TTCA depends on the sequence of model choices and their latency, not only
Figure 4. TTCA improvement ratio for UUID key-value lookups in 4K, 8K, 16K, 32K, and 64K contexts under English (EN), Japanese (JA), and Chinese (ZH) compared to loadaware and session-affinity routing.
on first-attempt success in isolation. As retries proceeded, LAAR’s success rate steadily improved and reached the highest final success rate among all methods. In contrast, both session-affinity routing and load-aware routing benefited less from additional retries than LAAR. These approaches often routed repeated attempts to the same model, yielding prefix reuse but limited exploration of alternatives. By explicitly avoiding previously failed models, LAAR was able to cover a broader set of candidate models within a small number of retries, which led to higher final success rates in most settings, although some 64K cases remained unsolved by all models. Figure 4 presents the relative TTCA improvement of LAAR at the final attempt (up to 10 retries). As shown in the figure, LAAR achieved up to 31% improvement over load-aware routing and 49% over session-affinity routing. For load-aware routing, performance became relatively closer to LAAR as context length increased. This was because larger contexts increased overall system load. Load balancing then improved latency, which directly reduced TTCA. In the 64K-token case, load-aware routing even achieved lower absolute TTCA than LAAR. This highlights that latencydominant regimes can favor load balancing. However, as observed in Figure 3, its final success rate was lower. This result reflects the trade-off between lower latency and lower correctness. In contrast, session-affinity routing degraded relative to LAAR as context length increased. This is primarily due to its policy of consistently routing requests within the same session to the same model. That policy becomes suboptimal when model accuracy varies with context length. While all models exhibit high accuracy for short contexts, longer contexts introduce divergence in model performance. That divergence amplifies the advantage of LAAR’s adaptive routing and leads to larger improvements.
Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving
7
Discussion and Future Directions
The experimental results in Section 6 highlight that TTCA is a meaningful objective for routing and scheduling in LLM serving systems. At the same time, they suggest that the LAAR algorithm may require further adaptation depending on factors such as available models, task types, context length, and language. Although LAAR introduces additional computation, its overhead remains sufficiently small compared to model inference latency (e.g., on the order of milliseconds even for 64K-token inputs). While this overhead is not explicitly included in TTCA, it may need to be incorporated into the latency term 𝐿(𝑚, 𝑥) when extending the approach to more complex routing logic. Finally, we evaluate load-aware and session-affinity routing as baselines, but hybrid approaches that combine them with LAAR may further improve performance. For example, the relatively low latency of load-aware routing in the 64K-token setting suggests that integrating load-awareness into LAAR could yield better TTCA under high-load conditions. Session-affinity may provide benefits in workloads not considered here, such as multi-turn interactions or heterogeneous task sequences. Prefix locality reduces prefill cost, but strict stickiness may repeatedly query an incompatible model under deterministic decoding and hurt TTCA. Conversely, aggressively switching to accuracy-compatible endpoints may sacrifice cache reuse. Designing a multi-objective router that jointly optimizes correctness and cache locality remains future work. Although we focus on single-intent, retryable workloads, TTCA can be extended to more general conversational settings. For example, TTCA can be applied at the level of individual turns, or more broadly to measure time-to-goal under a higher-level notion of conversational success. An important direction for future work is to define such success signals, potentially through proxy metrics such as LLM-asa-judge or human feedback.
8
Conclusion
Long-context LLM serving brings routing quality and correctness into closer interaction because a poor model choice can trigger expensive retries. TTCA makes this coupling explicit by measuring time-to-correct-answer rather than single-shot latency. From this perspective, we introduced LAAR, a lightweight routing design for routing across heterogeneous models. LAAR combines success-probability and latency estimates. In our preliminary key-value retrieval evaluation, LAAR improved TTCA without semantic routing or auxiliary model inference in the control plane. Extending the approach to broader open-ended tasks (e.g., repoqa, summary) remains future work because correctness criteria become more taskspecific.
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk
References [1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv:2404.14219 [cs.CL] https://arxiv.org/abs/2404.14219 [2] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. arXiv:2308.14508 [cs.CL] https://arxiv.org/abs/2308.14508 [3] Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual Pre-Training for CrossLingual LLM Adaptation: Enhancing Japanese Language Capabilities. arXiv:2404.17790 [cs.CL] https://arxiv.org/abs/2404.17790 [4] Muhammad Haseeb. 2025. Context Engineering for Multi-Agent LLM Code Assistants Using Elicit, NotebookLM, ChatGPT, and Claude Code. arXiv:2508.08322 [cs.SE] https://arxiv.org/abs/2508.08322 [5] Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2024. Multilingual Needle in a Haystack: Investigating LongContext Behavior of Multilingual Large Language Models. arXiv:2408.10151 [cs.CL] https://arxiv.org/abs/2408.10151 [6] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https://arxiv.org/abs/2404.06654 [7] IBM Granite Team. 2024. Granite 3.0 language models. [8] Kunal Jain, Anjaly Parayil, Ankur Mallick, Esha Choukse, Xiaoting Qin, Jue Zhang, Íñigo Goiri, Rujia Wang, Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. 2025. Performance Aware LLM Load Balancer for Mixed Workloads. In Proceedings of the 5th Workshop on Machine Learning and Systems (World Trade Center, Rotterdam, Netherlands) (EuroMLSys ’25). Association for Computing Machinery, New York, NY, USA, 19–30. doi:10.1145/3721146.3721947 [9] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O. Arik. 2024. LongContext LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. arXiv:2410.05983 [cs.CL] https://arxiv.org/abs/2410.05983
EuroMLSys ’26, April 27–30, 2026, Edinburgh, Scotland Uk [10] Kubernetes SIG Network. 2026. Gateway API Inference Extension. https://github.com/kubernetes-sigs/gateway-api-inferenceextension GitHub repository. Accessed: 2026-04-10. [11] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [12] Ruiqi Lai, Siyu Cao, Leqi Li, Luo Mai, and Dmitrii Ustiugov. 2025. Manage the Workloads not the Cluster: Designing a Control Plane for Large-Scale AI Clusters. In Proceedings of the 5th Workshop on Machine Learning and Systems (World Trade Center, Rotterdam, Netherlands) (EuroMLSys ’25). Association for Computing Machinery, New York, NY, USA, 246–253. doi:10.1145/3721146.3721937 [13] Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu. 2025. SCBench: A KV Cache-Centric Analysis of Long-Context Methods. arXiv:2412.10319 [cs.CL] https: //arxiv.org/abs/2412.10319 [14] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172 [cs.CL] https://arxiv.org/abs/2307.03172 [15] llm-d Project. 2026. llm-d Inference Scheduler. https://github.com/llmd/llm-d-inference-scheduler GitHub repository. Accessed: 2026-04-10. [16] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. 2024. MMLONGBENCH-DOC: benchmarking long-context document understanding with visualizations. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 3041, 48 pages. [17] Moonmoon Mohanty, Gautham Bolar, Preetam Patil, UmaMaheswari Devi, Felix George, Pratibha Moogi, and Parimal Parag. 2025. Deferred prefill for throughput maximization in LLM inference. In Proceedings of the 5th Workshop on Machine Learning and Systems (World Trade Center, Rotterdam, Netherlands) (EuroMLSys ’25). Association for Computing Machinery, New York, NY, USA, 100–106. doi:10.1145/3721146.3721962 [18] Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a Large Japanese Web Corpus for Large Language Models. arXiv:2404.17733 [cs.CL] https://arxiv.org/abs/2404. 17733 [19] Pol G. Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Torres, and Josep Ll. Berral. 2025. Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference . In 2025 IEEE 18th International Conference on Cloud Computing (CLOUD). IEEE Computer Society, Los Alamitos, CA, USA, 277–287. doi:10.1109/CLOUD67622.2025.00036 [20] Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. Ada-LEval: Evaluating long-context LLMs with lengthadaptable benchmarks. arXiv:2404.06480 [cs.CL] https://arxiv.org/abs/ 2404.06480 [21] Chen Wang, Xunzhuo Liu, Yuhan Liu, Yue Zhu, Xiangxi Mo, Junchen Jiang, and Huamin Chen. 2025. When to Reason: Semantic Router for vLLM. arXiv:2510.08731 [cs.ET] https://arxiv.org/abs/2510.08731 [22] Tian Xia, Ziming Mao, Jamison Kerney, Ethan J. Jackson, Zhifei Li, Jiarong Xing, Scott Shenker, and Ion Stoica. 2025. SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference. arXiv:2505.24095 [cs.DC] https://arxiv.org/abs/2505.24095
Takeshi Yoshimura, Valentijn Dymphnus van de Beek, and Tatsuhiro Chiba [23] Haoyuan Xu, Chang Li, Xinyan Ma, Xianhao Ou, Zihan Zhang, Tao He, Xiangyu Liu, Zixiang Wang, Jiafeng Liang, Zheng Chu, Runxuan Liu, Rongchuan Mu, Dandan Tu, Ming Liu, and Bing Qin. 2026. The Evolution of Tool Use in LLM Agents: From Single-Tool Call to MultiTool Orchestration. arXiv:2603.22862 [cs.SE] https://arxiv.org/abs/ 2603.22862 [24] Amy Yang, Jingyi Yang, Aya Ibrahim, Xinfeng Xie, Bangsheng Tang, Grigory Sizov, Jeremy Reizenstein, Jongsoo Park, and Jianyu Huang. 2025. Context Parallelism for Scalable Million-Token Inference. arXiv:2411.01783 [cs.DC] https://arxiv.org/abs/2411.01783 [25] Shang Yang, Junxian Guo, Haotian Tang, Qinghao Hu, Guangxuan Xiao, Jiaming Tang, Yujun Lin, Zhijian Liu, Yao Lu, and Song Han. 2025. LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention. arXiv:2502.14866 [cs.CL] https://arxiv.org/abs/2502.14866 [26] Ying Yuan, Pengfei Zuo, Bo Wang, Zhangyu Chen, Zhipeng Tan, and Zhou Yu. 2026. DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving. arXiv:2602.06502 [cs.DC] https://arxiv.org/abs/2602.06502