arXiv CSopen access

GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources

GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources

arXiv:2605.16867v1 [cs.DC] 16 May 2026

Boxiao Du1 Boning Huangfu1 Yizhou Luo1 Chen Chen1∗ Zijun Li1 Minchen Yu2 Xiaoyi Fan3 Minyi Guo1 1

2

Shanghai Jiao Tong University The Chinese University of Hong Kong, Shenzhen 3 Shenzhen MSU-BIT University

Abstract Large Language Models (LLMs) play a critical role in emerging agentic applications, where the timely completion of each entire inference is critical. Meanwhile, agentic LLM inferences are increasingly served on heterogeneous GPUs in operators’ resource pools. Therefore, it is crucial to route incoming inference requests to appropriate GPUs so that their end-to-end latency requirements are satisfied whenever possible, thereby achieving high goodput. In this paper, we propose GoodServe, a goodput-optimized serving system for agentic inferences over heterogeneous resources. GoodServe performs inference routing in a predict-and-rectify manner. It estimates the request output lengths as well as the GPU serving status in an accurate and also practical manner. Based on information from both the demand and resource sides, it then makes high-quality routing decisions using a just-enough instance selection heuristic. It also periodically monitors SLO-violation risks of active requests and triggers runtime request migrations to address unexpected dynamics. Our evaluations show that GoodServe improves goodput by up to 27.4% over existing routing methods.

1

Introduction

Agentic applications driven by Large Language Models (LLMs), like code generation [18] and database management [22], are increasingly popular. Unlike chatbot-style inference serving [23], agentic applications often require a fully-formed output before downstream tasks can proceed [39, 15]; therefore, service quality is crucially affected by the end-to-end inference latency rather than time-tofirst-token or time-per-output-token latency. Users often associate end-to-end latency requirements with agentic LLM inferences [37, 39, 5, 3, 1], and it is desirable for the serving system to attain high goodput, i.e., to complete as many inferences within their E2E-SLOs as possible. In the meantime, agentic LLM service providers often aggregate all GPUs they own—even across heterogeneous generations—into a resource pool to serve a massive number of requests [20, 1]; each request, upon arrival, is routed by the service proxy to a specific GPU instance for execution. Given the output uncertainty and resource heterogeneity, it is challenging to make routing decisions that achieve the best goodput performance. Regarding multi-instance request routing, existing methods are not designed to optimize the end-toend SLO performance of agentic LLM inferences, failing to attain high goodput. For example, some routing strategies, like random [25], round-robin [25], least-request [25] and Llumnix [30], seek to balance the loads on different instances; some other methods, like prefix-cache [25] and Preble [29], seek to maximize the local execution efficiency of individual requests. In essence, without awareness ∗ Chen Chen is the corresponding author.

Preprint.

of end-to-end request SLOs, those methods lack the flexibility to route less-urgent requests to inferior instances; that is, they are incapable of making locally-suboptimal yet globally-optimal routing decisions. Therefore, when serving agentic LLM inferences, we need to exploit information from both the demand and resource aspects, so as to perform SLO-adaptive routing to optimize overall goodput. In this paper, we propose GoodServe, a goodput-centric routing system for agentic LLM inferences over heterogeneous GPUs. To that end, we need to acquire the SLO performance of each request if it were routed to any GPU instance, which requires several pieces of future information such as the request decode length, the expected GPU queuing time, and the prefill/decode speed. While it is possible to estimate such information, it is, however, impossible to make fully accurate predictions due to the built-in algorithm and system dynamicity. Therefore, we adopt a predict-and-rectify methodology. Specifically, GoodServe consists of three core designs. First, to predict request decode length in an accurate yet also light-weight manner, given that the agentic task type is essentially an implicit precondition crucially affecting the output length, we design a Mixture-of-Experts style prediction model, which ensembles multiple simple-yet-professional MLPs to make a weighted prediction. Second, in estimating request execution efficiency after being routed to a given GPU, for practicality, we design an EMA-smoothed, black-box profiling method, which turns out to be sufficiently accurate for production use. Third, given the estimated demand and resource information, to work out a high-goodput request routing scheme, we propose the just-enough instance selection heuristic, which is efficient and also of high quality; to further address the potential demand-prediction error and system-status drift, GoodServe periodically re-evaluates the SLO-violation risk of active requests, and launches light-weight, token-ID based request migration when necessary. We have implemented GoodServe in 2.5k LoC, and further evaluated its performance with both testbed measurements and large-scale simulations. Our measurements over heterogeneous GPUs show that, when serving popular agentic LLM workloads, GoodServe consistently achieves the best goodput performance under diverse SLO scales, surpassing the second best by up to 27.4%. Meanwhile, our deep-dive experiments do confirm the effectiveness of each GoodServe innovation. Moreover, large-scale simulations emulating a 512-instance cluster show that GoodServe’s routing overhead is still negligible even under a very high load intensity (5 ms at 10,000 requests per second).

2

Background and Motivation

2.1

Serving Agentic LLM Inferences in Multi-Instance GPUs

In the coming era of agentic AI, LLM inference has become a workhorse workload supporting emerging agentic applications like mathematical reasoning [35], code generation [18] and database management [22]. Compared with conventional LLM inferences supporting chatbot conversations [2], the LLM inferences supporting agentic applications—which we call agentic LLM inferences— exhibit two distinct characteristics. First, agentic LLM inferences typically have much longer decode lengths than chatbot inferences, which is highly uncertain a priori. For instance, reasoning models like DeepSeek-R1 [6] generate itecture GPU Arch extensive Chain-of-Thought tokens, where decode lengths vary significantly depending on task difficulty. Second, Figure 1: Inference latency across four while for chatbot inferences it is the Time-to-First-Token GPU architectures under varying batch (TTFT) or Time-Per-Output-Token (TPOT) that matters sizes, for a fixed sequence comprising most, for agentic LLM inferences it is the end-to-end in- 100 input tokens and 200 output tokens. ference latency (i.e., Time-to-Last-Token or TTLT) that truly matters for the application-level performance. This shift arises because the downstream agentic executors require fully formed outputs for execution, making their utility binary upon completion [39, 15]. In practice, agentic LLM inferences are often associated with explicit end-to-end latency (i.e., E2E-SLO) requirements [37, 39, 5, 3, 1]. When 10

Latency (s)

8

6

4

h

tc

Ba

64 32 16 8

ze Si

2

4

2

1

A40

V100

A800

H800

serving agentic LLM inferences, it is increasingly significant for the serving system to maximize goodput (i.e., the number of requests that satisfy their E2E-SLO requirements). Meanwhile, confronting booming LLM inference demands, production service providers often maintain a pool of LLM serving instances for parallel request processing; upon the arrival of each LLM inference request, the LLM service proxy needs to route it to one inference backend for execution. In particular, the inference serving capabilities across different instances are often inconsistent—due to static hardware heterogeneity and dynamic resource fluctuations. Regarding hardware heterogeneity, since LLM service providers need to persistently purchase new GPUs and the available GPU types keep evolving, the resultant GPU resources available for agentic LLM serving naturally become heterogeneous [20, 1]. For example, Microsoft reports maintaining a heterogeneous cluster comprising diverse (e.g., NVIDIA H100, A100 and AMD MI300X) GPUs to serve its LLM inference workloads supporting Office 365 Copilot function [1]. Regarding resource fluctuations, depending on the instantaneous request intensity (which affects the request batch size served in one iteration), the inference serving speed on a GPU instance is also inconstant at runtime. We note that both the GPU hardware type and batch size configuration do affect the per-iteration inference time; this can be demonstrated by Fig. 1, which shows the per-iteration LLM inference latency across different GPU architectures (A40, V100, A800, and H800) under varying batch sizes. In summary, when serving agentic LLM inferences in typical multi-instance clusters, given the potential service-backend heterogeneity and the inherent decode-length uncertainty, it is crucial yet also challenging to optimize the request routing policies for the best E2E-SLO goodput. Next, we study the effectiveness of existing request routing methods with regard to that objective.

Lessons Learned from Existing Request-routing Methods

For request routing, in practice a series of methods has already been proposed. For example, the random (Power of Two Choices) [24] and round-robin [38] strategies are classical routing methods that seek to evenly distribute requests across instances. Meanwhile, least-request and prefix-cache are also core routing strategies supported in production frameworks like AIBrix [31]: the former routes incoming requests to the backend with the minimum number of pending requests, and the latter preferentially routes requests to the server with a high cache hit ratio. Besides, the lowest-TPM strategy, which is adopted by the LiteLLM system [4], dispatches requests to the backend with the minimum Tokens-Per-Minute (TPM) utilization. The Preble-style method [29, 23] further jointly considers prefix cache hits and compute load when selecting the best-performing instance for a request. Moreover, Llumnix [30] is a more advanced scheduler that preferentially routes requests to the instance with the maximum available memory, which also supports runtime migration to alleviate load imbalance during the execution process.

7.51

8

Goodput (RPS)

2.2

6

5.04

4.75 3.82

4 2 0

Least Requests

Round Robin

Llumnix

Oracle

Figure 2: Performance inferiority of existing routing strategies. In total, 600 requests (with an arrival rate of 10 requests per second) are jointly served by four heterogeneous (V100, A40, A800, H800) GPUs. Each request has 100 input tokens and has its output token length randomly sampled from [100, 500]. The E2E-SLO is set to 6s.

However, the above routing strategies all fall short for our problem in that they are agnostic to the end-to-end SLO requirement of agentic LLM inferences. Specifically, given the highly-uncertain inference output length as well as the dynamic backend status, existing strategies essentially choose to implement simple heuristics that are hardware-aware only (e.g., for balanced hardware utilization). In that sense, they treat all the incoming requests indiscriminately, failing to identify and prioritize those requests that have more urgent SLO requirements. Consequently, given a set of agentic LLM requests with their respective SLO requirements, greedily routing each request to the currently most efficient (lightly loaded) serving instance often leads to suboptimal performance in terms of the overall goodput. To demonstrate this, we conduct a testbed experiment with 4 heterogeneous GPUs. As shown in Fig. 2, compared to an oracle request router that has the ground-truth knowledge of 3

the inference generation length and hardware processing capability (routing policy detailed later in Sec. 3.4), those existing routing policies perform substantially worse as measured by goodput. Given the above study, we learn that to attain good SLO performance for agentic LLM inferences, instead of relying on fixed heuristics, we need to perform SLO-aware request-adaptive routing—by properly exploiting auxiliary information from both demand (i.e., request decode length) and resource (i.e., GPU inference capability) aspects. However, given the inherent dynamicity of inference length and GPU capability aforementioned, it is challenging to make near-optimal routing decisions in practical systems. We next present our solution to address that challenge.

3

Solution

3.1

Overview

Problem formulation. For clarity, we first mathematically formulate our research problem (a detailed symbol table is included in Appendix A.1). Given the request set R and the GPU instance set G, we let xr,g ∈ {0, 1} indicate whether request r is routed to GPU-g, and also let T (r, g) be the end-to-end inference latency of request-r when served on GPU-g. Then our optimization objective is to maximize the overall goodput, i.e.,   X X max I xr,g T (r, g) ≤ Dr  , (1) {xr,g }

r∈R

g∈G

where Dr is the E2E-SLO of request-r. P Meanwhile, there are two types of constraints. First, a request can only be served by one GPU, i.e., g∈G xr,g = 1, ∀r ∈ R. Second, the number of requests that can be concurrently served on each GPU is bounded by its memory capacity [32] (for symbolic clarity we skip the formula here, but this constraint is always complied with in our system). We further take a closer look at T (r, g), which is jointly affected by the demand volume of request-r as well as the execution status of GPU-g. Specifically, regarding the request demands, we let Lin r and Lout respectively be the input and output lengths of request-r; regarding the GPU status, we r let pg and dg respectively be the average per-token prefill/decode latencies on GPU-g, and qg be its average request queuing delay. We also take the effect of prefix caching into consideration, which can eliminate the prefill delay upon a cache hit and is therefore commonly adopted in production systems [23]. Suppose Hr,g is the hit prefix token length, then we obtain out T (r, g) ≜ qg + pg · (Lin r − Hr,g ) + dg · Lr .

(2)

GoodServe workflow. To solve the above optimization problem, a prerequisite is to obRequest Flow Client Control Flow tain the coefficients in T (r, g), i.e., qg , pg , dg Migration ① User Request out and L . We note that it is possible to estiProxy Router (Control Plane) mate the demand volume and hardware status GPU Status Request Status Predictor Estimator Monitor Monitor in advance [14, 33, 7], yet, on the other hand, GPU ( , , ) Req ( Prompt , , , SLO ) it is impossible to make 100% accurate prediction. Therefore, in this paper we propose ( , , ) GPU Req ( Prompt , , , SLO ) GoodServe, which routes agentic LLM infer③ Update ③ Update ences in a predict-and-rectify manner for the Request / GPU Request / GPU ② Route / Re-route Status Status highest goodput. As shown in Fig. 3, GoodRequest Instance 1 Instance N Serve introduces a RequestStatusMonitor Request Queue Request Queue and a GPUStatusMonitor on the proxy router ⑤ Output ④ Migrate ⑤ Output Inference Engine Inference Engine (GPU) (GPU) that respectively maintain the demand- and resource-side information. In particular, the RequestStatusMonitor predicts request out- Figure 3: GoodServe architecture and workflow. put token lengths (Lout ) with an accurate and light-weight Mixture-of-Expert-style predictor (Sec. 3.2); the GPUStatusMonitor estimates the average execution status of each GPU (qg , pg , dg ) in an EMA-smoothed, black-box manner (Sec. 3.3). For each incoming request, the proxy first routes it to the GPU following an effective just-enoughselection heuristic, and may also trigger request migrations later based on the refreshed demand and 4

1

1

n

m

resource status to ensure SLO attainment (Sec. 3.4). Next we will elaborate on each component in greater detail. 3.2

Mixture-of-Expert-style Output-length Prediction

Output-length prediction is essential for goodput optimization; while 100% accurate prediction is impossible, it is still important to predict as accurately as possible. Compared with conventional chatbot inferences, agentic inferences typically have a much broader output length range (due to diverse task types); meanwhile, given the high request-arrival rate on a proxy router, output length prediction must be simultaneously made light-weight. In the literature, multiple output-length prediction methods have already been proposed. The S 3 work [14] predicts with another fine-tuned large language model (DistilBERT), the Past-Future work [7] predicts by referring to historical requests, and the STAR work [33] predicts with a 4-layer MLP model. However, all those works fall short for our problem: the DistilBERT-based prediction method incurs high training and inference overheads, whereas history-based and single-MLP-based methods fail to attain high prediction accuracy; to predict the output length of agentic inferences, we need to avoid introducing complex Transformer-like architectures, yet at the same time maintain the flexibility to adapt to diverse task types. To that end, in GoodServe we design a Mixture-of-Experts-style predictor. Given the remarkable impact of the inference task type on the output-length distribution, for accuracy we need to treat the task type as a prediction condition: for example, inferences generating database-operating commands [22] usually have short outputs, whereas those for code-generation [18] usually have long outputs. Moreover, such preconditions (sensible from prompts) should be handled implicitly, without assuming the recurrence of a fixed task set. Therefore, for high accuracy and efficiency, we use an ensemble of small models and automatically select the best ones for runtime prediction. We note that this aligns well with the design philosophy of Mixture-of-Experts architecture [27], which has been widely adopted in modern LLMs [6] given its Pareto-superiority in model capability and compute efficiency. As shown in Fig. 4, our output-length prediction model contains a gating router (R(·)) and multiple (K) experts. Given the feature vector hr (obtained by conducting TF-IDF vectorization [28] over the token window so far) for request r, the router outputs a probability distribution pr ∈ ∆K over experts: pr = softmax(R(hr )), and the final prediction is obtained via a weighted combination of expert outputs: PK ŷr = k=1 pr,k Ek (hr ). Specifically, the gating router is a two-layer MLP and each expert is a fourlayer MLP; in total there are only 45.1M parameters.

Prompt: Solving this problem ... TF-IDF 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Gateway

... Route

...... MLP1

MLP2

MLP n

output length: 500

Moreover, that predictor is trained in two phases. In the first phase, we partition one half of the dataset into Figure 4: MoE-style output-length predictor. K distinct subsets—by discretizing query (input) and √ answer (output) lengths into K tiers (in practice we set K to 9), with each MLP expert trained separately using a specific data subset. In the second phase, with expert parameters frozen, we train the router on the other half of the dataset. Our later evaluations in Sec. 4.3 demonstrate that such a MoE-style predictor can perform well in terms of both accuracy and efficiency. 3.3

EMA-smoothed, Black-box Instance-Capability Estimation

Apart from request-demand prediction, for goodput-optimal serving, we also need to estimate the average execution efficiency after a request is routed to a GPU instance. Specifically, we need to perform comprehensive instance status monitoring, covering all the hardware-related coefficients in Eq. 2: qg , pg and dg . However, there are essentially two challenges. First, given the batched serving mode in mainstream LLM frameworks like vLLM [32], the actual execution efficiency of request-r on GPU-g is jointly affected by the shape (i.e., Lin and Lout in Eq. 2) of request-r and the competing requests on GPU-g, making it hard to calculate precisely with the instance configuration 5

information. Second, an agentic LLM inference may have many decode steps, during which the processing capability of the instance is usually not a constant. We note that execution efficiency estimation does not need to be highly accurate (it suffices if the instance preference order is valid), but must be practical, i.e., easy to implement and adopt in a production environment. Therefore, we make some necessary approximations, trading accuracy slightly for much better practicality. First, since a GPU instance batches many inferences in one iteration and its local configuration rarely changes, given the Law of Large Numbers [26], we can view a GPU’s per-iteration time as relatively stable during a short period. Therefore, the estimation of qg , pg and dg can be made in a black-box manner—directly from their recent past values, instead of relying on runtime request specifics or engine configurations (e.g., batch size, GPU type and queuing policy). That said, to tackle temporal system jitter, we still adopt exponential moving average (EMA) to obtain a smoothed estimation. As shown in Fig. 5, when replaying the realistic Mooncake trace [23] with a mixed BIRD-bench [16] and LiveCodeBench [10] dataset on an NVIDIA A40 GPU, such an EMA-smoothed, black-box estimation method indeed works quite well. This way, we can achieve comprehensive and practical estimation of instance serving efficiency.

3.4

Just-enough Instance Selection with Runtime Migration

TPOT (ms)

Queue Time (ms)

Even after we have obtained both the demandand resource-side information, it is still challengActual Predicted ing to find the goodput-optimal request routing 100 scheme. First, the exact optimization problem behind Eq. 1 is NP-hard: with binary routing variables and bounded GPU memory/compute capaci0 0 20 40 60 80 100 120 ties, it becomes an integer linear program that couples request assignments across GPUs, and thus 100 cannot be directly solved in polynomial time [39]. Second, our previous request length prediction and resource capability estimation may not be per0 0 100 200 300 400 sistently accurate, and a one-shot routing scheme Time (s) may turn out to fail to meet the E2E-SLO requirement. Now we address these two challenges in Figure 5: Effect of the EMA-smoothed, blackturn, with the overall algorithm summarized in box estimation method on queuing time and Appendix A.2. TPOT. First, we propose the just-enough heuristic for instance selection to efficiently obtain a near-optimal solution to Problem 1. In fact, while we notice from Fig. 1 that higher-end GPUs in general yield better efficiency for all the requests, completing a request faster than its E2E-SLO requirement, however, does not help improve the overall goodput, which is essentially a waste of high-end resources. Therefore, among the set of candidate instances that can all meet the E2E-SLO requirement of a request, GoodServe chooses to route that request to the instance with the worst computing capability, thereby opportunistically facilitating the fast completion of other SLO-urgent requests. The per-request routing-decision complexity in this way is only O(M ) in an M -instance cluster. Second, to tackle the demand prediction error and runtime resource drift, after making the initial routing decision, we continuously track each active request’s SLO-violation risk. Specifically, after every τ (set to 50 by default) inference iterations, we re-estimate the future generation length as well as the GPU processing speed with the methods respectively in Sec. 3.2 and Sec. 3.3. For each ongoing request, if its expected finish time based on the latest estimations exceeds the E2E-SLO, the LLM proxy reroutes that request to another GPU with stronger processing capability (still following the just-enough heuristic). In particular, when migrating a pending request to another instance, we choose to transfer the token IDs instead of the intermediate KV cache states. Since typical agentic LLM inferences have large context lengths and massive decoding iterations [34], the KV cache volume is substantial yet the prefilling time portion is negligible. Therefore, by transferring token IDs instead of the KV cache, we can trade a small prefill overhead for substantially lower communication cost and achieve higher migration efficiency (which we will confirm later in Fig. 9). 6

/HDVWUHTXHVW

/OXPQL[

  







6/26FDOH





3UHEOH

6/29LRODWLRQ5DWH

*RRGSXW536

*RRG6HUYH

5DQGRP

5RXQGURELQ

/RZHVW730

    







6/26FDOH





(a) Performance results on Llama3.1-8B-Instruct /HDVWUHTXHVW

/OXPQL[

  







6/26FDOH





3UHEOH

6/29LRODWLRQ5DWH

*RRGSXW536

*RRG6HUYH

5DQGRP

5RXQGURELQ

/RZHVW730

    







6/26FDOH





(b) Performance results on Qwen2.5-14B-Instruct

Figure 6: End-to-end performance under different request routing methods.

4

Evaluation

4.1

Experimental Setup

Implementation. We implemented GoodServe in around 2.5k LoC, and the source code is attached as a supplementary file. The demand predictor was trained on 8680 samples, for 500 epochs with a training time of 25 minutes. Meanwhile, for each ongoing request, its future output length is repredicted directly on its assigned GPU instance, and all such predictions are batched; such measures can help mitigate the prediction overhead (Fig. 11). Hardware and models. Our experiments are conducted on a heterogeneous cluster of four instances, each equipped with distinct NVIDIA GPUs: H800 (80GB), A800 (80GB), A40 (48GB), and V100 (32GB, with a TP level of 2). These instances are interconnected by a 10Gbps Ethernet network, and the proxy router is hosted on the A40 instance. We use two representative instruction-tuned models, Llama3.1-8B-Instruct [8] and Qwen2.5-14B-Instruct [36], as backend LLMs. Workloads. We construct a mixed workload suite from three popular agentic benchmarks: BIRDbench [16], SWE-bench [13] and LiveCodeBench [10]. We replay the Mooncake production traces [23] to set the request submission time. To set the E2E-SLOs, following established methodologies [17, 21, 40], we first measure the median request execution time by running each request alone on a mid-tier GPU (NVIDIA A800), and then scale that base latency by multiple relaxation factors (1×, 1.5×, 2×, 2.5× and 3×) to emulate diverse SLO urgency levels. The temperature is set to 0 to ensure output consistency across the deadline-setting stage and the testbed-measurement stage. Baselines and Metrics. We compare GoodServe with diverse baselines described in Sec. 2.2: the built-in random and least-request methods in AIBrix [31], as well as the round-robin [38], lowestTPM [4], Preble [29] and Llumnix [30] methods. Meanwhile, we adopt two metrics: (1) goodput, which is the average number of requests completing within their E2E-SLO per second, and (2) SLO violation ratio, which is the overall ratio of requests failing to satisfy the E2E-SLO requirements. 7

2.5 2.0 1.5 1.0 0.5

1.0

1.5

2.0 2.5 SLO Scale

3.0

Go

3 2 1 0

5.0 2.5 0.0

rve ased -MLP ased -b gle ory-b LLM t Sin His

Se

od

7.5

Latency (ms)

3.0

105

4 Latency (ms)

GoodServe GoodServe w/o Migration GoodServe w/o Prediction

3.5

Normalized MAE

Goodput (RPS)

4.0

(a) Prediction error.

e

erv

dS

o Go

sed

-ba

LLM

LP

-M

gle

Sin

KV-Cache based Migration Token-ID based Migration

103

0 0 0 0 0 50 100 150 200 300 Sequence Length

(b) Prediction latency.

Figure 9: Average migraFigure 7: Effectiveness of prediction and migration Figure 8: Accuracy and overhead comparison among tion latency under different state transferring methods. innovations in GoodServe. different prediction methods.

4.2

End-to-End Performance

In Fig. 6, we depict the end-to-end results across different SLO scales for both 8B and 14B models. We repeat each experiment five times and report the max/min values as error bars. As shown in Fig. 6, GoodServe consistently delivers the best goodput and the lowest SLO violation rate. For example, for the 8B model at a medium SLO scale of 2, GoodServe surpasses the second-best (Preble) by 27.4% in goodput. Meanwhile, we note that under looser SLO requirements the benefit of GoodServe generally becomes larger: as the SLO scale increases from 1 to 3 for the 14B model, the goodput improvement of GoodServe over the second best increases from 3.1% to 16.6%. This is because a looser SLO requirement allows for more locally-suboptimal yet globally-optimal routing decisions, which existing SLO-unaware methods all fail to make. 4.3

Deep Dive Experiments

In this part, we conduct a series of deep-dive experiments. Unless otherwise specified, we use the 8B model at an SLO scale of 3. Ablation study. Recall that we have adopted an MoE-style request length predictor and also enabled runtime migration to improve robustness. To evaluate their necessity, we create two variants of GoodServe: (1) GoodServe without prediction, which replaces the original predictor with a historybased one [7] (for our problem, prediction itself cannot be disabled), and (2) GoodServe without migration, which disables runtime request migration. As shown in Fig. 7, both innovations are indispensable. For example, at an SLO scale of 3, removing the MoE-style predictor reduces the goodput by 32.8%, and removing the runtime migration functionality reduces the goodput by 18.0%. We note that their importance increases under looser SLOs, because a looser SLO allows for more flexible routing optimizations. Superiority of our MoE-style demand predictor. We check the superiority of our MoE-style predictor over the other ones discussed in Sec. 3.2, in terms of both accuracy and overhead. Fig. 8 shows the normalized Mean Absolute Error (MAE) for different predictors. It suggests that our MoE-style predictor adopted in GoodServe can yield the best accuracy—with an error reduction of 1.4× compared to the LLM-based one [14], and up to 3.8× compared to the history-based one [7]. Meanwhile, its prediction latency (around 2.5 ms per request) is lower than the LLM-based one. Furthermore, replacing our predictor with the LLM-based, Single-MLP, and history-based ones reduces the goodput by 13.8%, 17.0%, and 18.1%, respectively. Superiority of our Token-ID based request migration method. Recall that in Sec. 3.4, for request migration we choose to transfer the Token-IDs instead of the KV cache. To evaluate its superiority, we refer to Fig. 9, which shows the average migration latency for requests of different lengths. Compared with the KV-cache-based method, token-ID based migration achieves 7.1× to 15.3× lower latency. Hyper-parameter sensitivity analysis. GoodServe introduces two hyper-parameters: the number of experts K in the predictor (Sec. 3.2), and the status checking frequency τ (Sec. 3.4). Here we evaluate GoodServe’s performance by varying their values. As shown in Fig. 10a, while adopting more experts can yield higher accuracy and consequently higher goodput, K = 9 can already yield sufficiently good performance, much better than K = 4 yet comparable to K = 16. Meanwhile, regarding τ , Fig. 10b suggests that a higher checking frequency does improve goodput. Overhead analysis with large-scale simulation. In production clusters, there may be hundreds of instances. Since our core innovation lies in the proxy router instead of on the serving engine, to check 8

60

2.0

40

1.8

20

1.6

2.6

0.12 0.09

2.4

0.06

2.2 0.03 2.0

1

4

9

16

0

Number of Experts

20

50

100

200

500 no-check

0.00

Inter-check Iterations

8 instances 32 instances Average Routing Overhead (ms)

2.2

Goodput Overhead

2.8

Goodput (RPS)

80

Amortized Per-iteration Check Overhead (ms)

100 Goodput Model Size

2.4

Model Size (M)

Goodput (RPS)

2.6

128 instances 512 instances

5.0 4.0 3.0 2.5 1000

3000

5000

10000

RPS

(a) Number of experts.

(b) Status recheck interval.

Figure 10: Hyper-parameter sensitivity analysis.

Figure 11: Routing overheads at varying cluster size and request intensity.

GoodServe’s scalability we resort to large-scale simulations. Specifically, we configure a set of virtual IPs each corresponding to a simulated local inference engine. We respectively simulate 8, 32, 128 and 512 instances, and for each case we vary the RPS from 1000 to 10000—all requests handled by a single router. As shown in Fig. 11, thanks to the lightweight prediction model (Sec. 3.2), the simple heuristic (Sec. 3.4) as well as the implementation optimizations (distributed and batched predictor execution), the routing overhead of GoodServe is indeed quite small: the per-request routing latency is still marginal (5.0 ms) even when serving highly-intense workloads (RPS=10000) over 512 nodes.

5

Additional Related Works and Discussions

Related works on cost-efficient provisioning of heterogeneous GPU resources. While we focus on request routing over heterogeneous GPUs, in the literature some other works have explored how to compose a heterogeneous cluster for the best cost-efficiency. Specifically, Mélange [9] formulates GPU selection as a cost-aware bin-packing problem, and recent MILP-based studies [12] co-optimize GPU composition, deployment configuration, and workload assignment. SageServe [11] and llmd [19] further use traffic forecasts or autoscaling signals to decide how many model variants or GPU instances should be provisioned. GoodServe addresses a complementary problem where the GPU composition is an input condition rather than an optimization target. It minimally intrudes on the instance-level LLM service engine. Moreover, GoodServe is also the first routing-optimization work to target agentic LLM inferences for which TTLT is the primary efficiency concern. Compatibility of GoodServe with diverse scenarios: for inferences without E2E-SLOs, in homogeneous clusters, and for PD-disaggregated setups. While in this paper we focus on serving agentic LLM inferences with E2E-SLO requirements, GoodServe can work smoothly for inferences without E2E-SLOs. For those LLM inferences, we can still follow the conventional routing methods, which do not affect the functionality of each GoodServe component. Moreover, by proactively scheduling non-urgent agentic LLM inferences to inferior instances, we can in fact help to attain better TTFT or TPOT performance for chatbot-style inferences. Meanwhile, we note that our solution can also be used in homogeneous clusters, because at each instant, the local queuing time and batch size may still be inconsistent across different GPU instances, rendering their computing capacities still heterogeneous for an individual request. Besides, PD-disaggregated serving is also a popular technique that can avoid prefill-decode interference [23]. While we assume PD-multiplexed setups in this paper, we note that GoodServe can be extended to work in PD-disaggregated setups: by modifying Eq. 2, we can respectively model the true prefill (decode) time on the prefill-only (decode-only) instance for each candidate routing scheme, and our just-enough heuristic can be similarly applied. Limitation: no admission control mechanism. For requests that are impossible to meet the SLO requirements based on the predicted information, GoodServe chooses to route them to the most capable instance to provide best-effort service. However, since completing a request later than its E2E-SLO does not help improve goodput, a better choice is simply to eliminate it from being served. We do not implement that mechanism because we admit that our predictions are not always accurate, and thus admission control itself can be risky. We will explore that in the future. 9

6

Conclusion

In this paper, we present GoodServe, a goodput-oriented serving system for agentic LLM inferences over heterogeneous GPUs. GoodServe works in a predict-and-rectify manner. It first uses an MoEstyle predictor to predict the request output lengths in a light-weight and accurate manner. It then adopts an EMA-smoothed, black-box method to comprehensively and also practically estimate the instance serving efficiency. Finally, it applies the just-enough instance selection heuristic to optimize the overall service goodput, which is strengthened by a runtime request migration mechanism for robustness against prediction errors and system instability. Evaluation results show that GoodServe can enhance the overall goodput by up to 27.4%, with negligible overhead even in large-scale clusters.

References [1] Zain Asgar, Michelle Nguyen, and Sachin Katti. Efficient and scalable agentic ai with heterogeneous systems. arXiv preprint arXiv:2507.19635, 2025. [2] Tina Babu, Rajesh Sharma, et al. Ai-powered chat agent: Revolutionizing online shopping. In 2024 2nd International Conference on Signal Processing, Communication, Power and Embedded System (SCOPES), pages 1–5. IEEE, 2024. [3] Agrim Bari, Parikshit Hegde, and Gustavo de Veciana. Optimal scheduling algorithms for llm inference: Theory and practice. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(3):1–43, 2025. [4] BerriAI. LiteLLM: Python sdk and proxy server for unified llm api access. https://github. com/BerriAI/litellm, 2026. GitHub repository. Accessed: 2026-04-14. [5] Will Chow. Slice: Slo-driven scheduling for llm inference on edge computing devices. arXiv preprint arXiv:2510.18544, 2025. [6] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [7] Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, and Xianglong Liu. Past-future scheduler for llm serving under sla guarantees. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 798–813, 2025. [8] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [9] Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M\’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527, 2024. [10] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. [11] Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, et al. Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling. arXiv preprint arXiv:2502.14617, 2025. [12] Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Guoliang He, Xupeng Miao, Ana Klimovic, Bin Cui, Binhang Yuan, and Eiko Yoneki. Demystifying cost-efficiency in llm serving over heterogeneous gpus. arXiv preprint arXiv:2502.00722, 2025. [13] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. 10

[14] Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. S3 : Increasing gpu utilization during generative inference for higher throughput. Advances in Neural Information Processing Systems, 36:18015–18027, 2023. [15] Hyungjun Kim et al. Kairos: Power-aware serving of agentic ai workloads. arXiv preprint arXiv:2604.16682, 2026. [16] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330–42357, 2023. [17] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663–679, 2023. [18] Siwei Liu, Jinyuan Fang, Han Zhou, Yingxu Wang, and Zaiqiao Meng. Sew: Self-evolving agentic workflows for automated code generation. arXiv preprint arXiv:2505.18646, 2025. [19] llm-d Project. Workload variant autoscaler. https://llm-d.ai/docs/architecture/ Components/workload-variant-autoscaler, 2026. Accessed: 2026-05-05. [20] Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Serving large language models over heterogeneous gpus and network via max-flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 586–602, 2025. [21] You Peng, Youhe Jiang, Wenqi Jiang, Chen Wang, and Binhang Yuan. Hexgen-flow: Optimizing llm inference request scheduling for agentic text-to-sql. arXiv preprint arXiv:2505.05286, 2025. [22] Xuan-Quang Phan, Tan-Ha Mai, Thai-Duy Dinh, Minh-Thuan Nguyen, and Lam-Son Lê. Askdb: An llm agent for natural language interaction with relational databases. arXiv preprint arXiv:2511.16131, 2025. [23] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving {LLM} chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155–170, 2025. [24] Ray Project. Ray serve documentation. https://docs.ray.io/en/latest/serve/index. html, 2026. Accessed: 2026-01-25. [25] Ray Project. Ray serve llm routing policies. https://docs.ray.io/en/latest/serve/ llm/architecture/routing-policies.html, 2026. Accessed: 2026-01-25. [26] Pál Révész. The laws of large numbers, volume 4. Academic Press, 2014. [27] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [28] Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21, 1972. [29] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for llm serving. arXiv preprint arXiv:2407.00023, 2024. [30] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pages 173–191, 2024. [31] The AIBrix Team, Jiaxin Shan, Varun Gupta, Le Xu, Haiyang Shi, Jingyuan Zhang, Ning Wang, Linhui Xu, Rong Kang, Tongping Liu, et al. Aibrix: Towards scalable, cost-effective large language model inference infrastructure. arXiv preprint arXiv:2504.03648, 2025. 11

[32] vLLM Project. vllm: A high-throughput and memory-efficient inference and serving engine for llms. https://github.com/vllm-project/vllm, 2026. Accessed: 2026-01-25. [33] Zhibin Wang, Zetao Hong, Xue Li, Zibo Wang, Shipeng Li, Qingkai Meng, Qing Wang, Chengying Huan, Rong Gu, Sheng Zhong, et al. Adaptive rescheduling in prefill-decode disaggregated llm inference. arXiv preprint arXiv:2510.13668, 2025. [34] Haoran Wu, Can Xiao, Jiayi Nie, Xuan Guo, Binglei Lou, Jeffrey TH Wong, Zhiwen Mo, Cheng Zhang, Przemyslaw Forys, Chengyang Ai, et al. Combating the memory walls: Optimization pathways for long-context agentic llm inference. arXiv preprint arXiv:2509.09505, 2025. [35] Yibo Yan, Shen Wang, Jiahao Huo, Philip S Yu, Xuming Hu, and Qingsong Wen. Mathagent: Leveraging a mixture-of-math-agent framework for real-world multimodal mathematical error detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 69–82, 2025. [36] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. [37] Jiahuan Yu, Mingtao Hu, Zichao Lin, and Minjia Zhang. Superinfer: Slo-aware rotary scheduling and memory management for llm inference on superchips. arXiv preprint arXiv:2601.20309, 2026. [38] Shibo Yu, Mohammad Goudarzi, and Adel Nadjaran Toosi. Efficient routing of inference requests across llm instances in cloud-edge computing. arXiv preprint arXiv:2507.15553, 2025. [39] Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. Jitserve: Slo-aware llm serving with imprecise request information. arXiv preprint arXiv:2504.20068, 2025. [40] Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. Jitserve: Slo-aware llm serving with imprecise request information. 2025.

12

A

Appendix

A.1

Notations used in GoodServe

Notation

Description

r Request index g GPU index R Set of requests G Set of available GPU backends Dr End-to-end latency deadline (SLO) of request r Lin Input (prompt) length of request r r Lout Predicted output tokens of request r r Hr,g Reusable prefix cache length on backend g qg Estimated queuing delay on backend g pg Average per-token prefill latency on backend g dg Average per-token decode latency on backend g xr,g Binary routing variable: 1 if request r is assigned to backend g T (r, g) Predicted end-to-end latency if routing request r to backend g I[·] Indicator function, equals 1 if the condition holds Table 1: Notation used in the routing formulation of GoodServe.

Table 1 summarizes the key notations used in the routing problem formulation and instance selection described in Section 3.1. A.2

Scheduling Algorithm

Algorithm 1 Backend Selection Algorithm 1: hr ← B UILD F EATURES (r); Lout ← P(hr ) r 2: C ← ∅ 3: for all g ∈ G do 4: wg ← AVG WAIT T IME(g) 5: qg ← αwg + (1 − α)qg 6: Hr,g ← R EUSE P REFIX(r, g) out 7: T (r, g) ← qg + pg · (Lin r − Hr,g ) + dg · Lr 8: if T (r, g) ≤ Dr then 9: C ← C ∪ {g} 10: end if 11: end for 12: if C ̸= ∅ then 13: g ⋆ ← arg maxg∈C dg 14: else 15: g ⋆ ← arg ming∈G (T (r, g) − Dr ) 16: end if 17: return g ⋆

As discussed in Section 3.4 and shown in Algorithm 1, given an incoming request r, GoodServe first constructs the request feature vector hr and uses the predictor P(·) to estimate the remaining output length Lout r . For each backend g ∈ G, the scheduler continuously maintains an estimated queueing delay qg using exponential smoothing over the observed waiting time. It then computes the reusable prefix length Hr,g and estimates the prefill latency as T prefill (r, g) = pg (Lin r − Hr,g ). Based on the estimated queueing delay, prefill latency, and decoding cost, the scheduler predicts the end-to-end latency as T (r, g) = qg + T prefill (r, g) + dg · Lout r . A backend is considered feasible if T (r, g) ≤ Dr . Among all feasible backends, GoodServe selects the backend with the largest TPOT dg to maximize serving efficiency while satisfying the latency 13

deadline. If no feasible backend exists, the scheduler falls back to the backend that minimizes the deadline violation, i.e., arg ming∈G (T (r, g) − Dr ).

14