Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines Marcel Wagenländer∗† Guo Li†
Otto White∗† Britannio Jarrett‡ Huanzhou Zhu† Llúis Vilanova†
∗ Equal contribution
† Imperial College London
arXiv:2604.15186v1 [cs.DC] 16 Apr 2026
1
Problem
Yanda Tao†
‡ Independent Researcher
Abstract Agentic workflows carry out complex tasks by orchestrating multiple large language models (LLMs) and tools. Serving such workflows at a target throughput with low latency is challenging because they can be defined using arbitrary agentic frameworks and exhibit unpredictable execution times: execution may branch, fan-out, or recur in data-dependent ways. Since LLMs in workflows often outnumber available GPUs, their execution also leads to GPU oversubscription. We describe Scepsy, a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic workflows onto a GPU cluster. Scepsy exploits the insight that, while agentic workflows have unpredictable end-to-end latencies, the shares of each LLM’s total execution times are comparatively stable across executions. Scepsy decides on GPU allocations based on these aggregate shares: first, it profiles the LLMs under different parallelism degrees. It then uses these statistics to construct an Aggregate LLM Pipeline, which is a lightweight latency/throughput predictor for allocations. To find a GPU allocation that minimizes latency while achieving a target throughput, Scepsy uses the Aggregate LLM Pipeline to explore a search space over fractional GPU shares, tensor parallelism degrees, and replica counts. It uses a hierarchical heuristic to place the best allocation onto the GPU cluster, minimizing fragmentation, while respecting network topology constraints. Our evaluation on realistic agentic workflows shows that Scepsy achieves up to 2.4× higher throughput and 27× lower latency compared to systems that optimize LLMs independently or rely on user-specified allocations.
Pedro Silvestre† Peter Pietzuch†
GEN
...
VER
Beam 0 Active Beams
GEN
VER
GEN
VER
GEN
VER
...
Beam k
Top-k
...
Done?
Result
Fig. 1: Beam search as an agentic workflow (Beam search uses inference-time scaling, which uses LLMs to explore multiple reasoning paths as a search tree.)
enable novel capabilities and improve accuracy-cost tradeoffs [5]. These workflows are diverse and often tailored to specific tasks; common features include using LLMs of different sizes for different roles, or leveraging inference-time scaling [43] techniques to enable structured reasoning (e.g., beam [42] and Monte Carlo tree search [65]). They also augment LLMs with external tools [41], which provide access to up-to-date information (e.g., web search and databases), or enable LLMs to offload exact, deterministic computation (e.g., to command-line utilities and language interpreters). Fig. 1 shows a typical beam-search workflow [42]: starting from a user prompt, the workflow repeatedly expands candidate reasoning steps (beams) with a generator LLM (GEN), scores steps with another verifier LLM (VER), and retains the most promising beams for the next iteration. This shows the key features of modern agentic workflows: they combine multiple LLMs of different sizes (a smaller generator and a larger verifier), intersperse LLM invocations with non-LLM code, and exhibit data-driven LLM parallelism with fan-out and fan-in invocation patterns. Hyperscalers have massive data centers and performance engineering teams to optimize agentic workflow serving [2, 10, 38], but smaller institutions have a much harder time running self-managed deployments efficiently: managing smallto-medium scale GPU clusters means that small inefficiencies matter a lot more, and having smaller performance engineering teams means that they lack the knowledge necessary for the rapidly changing agentic landscape. Yet self-managed deployments are still desirable for privacy [14] and potential cost savings [31]. We therefore identify four challenges in managing self-managed agentic workload deployments:
Introduction
Large language models (LLMs) can generate high-quality text, follow sophisticated instructions and solve problems based on only natural-language prompts [2, 10, 38]. Yet, due to their large size and autoregressive nature, LLMs are slow and expensive to execute, relying on parallelization over GPUs for acceleration. Furthermore, standalone LLMs are insufficient to carry out complex tasks: they are constrained by finite context windows [48], lack access to external or domain-specific knowledge [22], and struggle with hallucinations during extended reasoning [52]. As a result, developers increasingly build agentic workflows, which orchestrate multiple LLMs and external tools to
Arbitrary agentic programs. Developers use a variety of agentic frameworks, such as LangChain [20, 21], AutoGen [55] or Camel [23] and other libraries (schema validation, 1
numerical, databases) to define workflows [51]. Thus, a serving system cannot assume a single programming model, API or the definition of workflows. Unpredictable execution. The latency of an LLM execution is highly variable because generation proceeds token-bytoken until the end of the sequence. Since LLMs may invoke tools and other LLMs, the workflow itself may branch, iterate, and recur in data-dependent ways. Together, these properties make predicting the throughput and latency of an agentic workflow based on a given GPU allocation difficult. Conflicting optimization objectives. Users want agentic workflows with high throughput and low latency. LLM throughput can be increased by exploiting data parallelism with parallel model replicas, but this reduces the GPU resources available for tensor parallelism, which lowers LLM latency. While serving a single LLM navigates one such tradeoff [18, 24], agentic workflows with multiple LLMs compose several objectives: adjusting the allocation of one LLM may affect another, or shift bottlenecks. The heterogeneity of GPU interconnects only worsens this problem [37], as interconnect availability constrains tensor parallelism. Oversubscribed GPUs. GPUs are a scarce and expensive resource [31], and even small inefficiencies bear a large cost impact in small-to-medium scale clusters. Compounding this, agentic workflows have a highly heterogeneous mix of LLMs (embedding, generator, reward, etc.), where coarse LLM-toGPU allocations quickly lead to poor GPU utilization. Current solutions fail to address these challenges. Some serving approaches for agentic workflows [3, 25, 28] exploit the workflow structure to optimize execution, e.g., using graphaware batching, KV-cache sharing, or reducing head-of-line blocking. However, several of these approaches [3, 25] are constrained by framework-specific programming models, and all are limited to single-LLM workflows. More fundamentally, they rely solely on temporal request scheduling to GPUs, and the complexity of managing the unpredictable execution of agentic workflows leaves the allocation of LLMs to GPUs to users. Other approaches [8, 18, 24, 56, 59] focus on multi-LLM serving through temporal or spatial multiplexing without assuming any agentic workflow structure; this makes them more flexible, but the lack of workflow awareness leads to suboptimal performance, since they schedule each LLM in isolation. We empirically observe that, while end-to-end execution times of a workflow vary greatly across requests, the fraction of a request’s total execution time that is spent on each LLM is substantially more stable. Intuitively, despite being unpredictable, workflows have underlying structure that creates stable steady-state behavior: e.g., in beam search, a generator execution is always followed by a verifier execution. Thus, our key insight is that, for the scheduling of requests, a system does not need to model the control flow and execution time variability in agentic workflows exactly
(with data-dependent token generation, branches, fan-out, and loops), but can instead reason about the fraction of aggregate demand put on each LLM to decide on GPU allocations. Under such stable aggregate demand, the familiar model of a pipeline can model performance well: since non-LLM tool and orchestration time is typically negligible (at most a few milliseconds in our measurements), workflow-level throughput is governed by whichever LLM becomes the workflow’s bottleneck. In addition, the end-to-end latency can be approximated as the accumulated LLM latency time (adjusted by the average amount of LLM parallelism). In this paper, we describe Scepsy, an agentic workflow serving system that efficiently schedules LLMs onto GPUs according to their relative performance contributions. Scepsy makes the following novel technical contributions: (1) Modeling LLM workflows as the Aggregate LLM Pipeline (§4). Scepsy makes GPU allocation decision based on aggregate per-LLM statistics collected over many executions. To support arbitrary agentic workflow frameworks [20, 21, 23, 55], Scepsy extracts these statistics through a combination of tracing low-level LLM invocations and LLM performance profiling, each without knowledge of the workflow itself. It then leverages these statistics to construct an Aggregate LLM Pipeline, a simplified representation of the resources used by the workflow, which enables pipeline-style latency and throughput predictions of agentic workflows under different GPU allocations. (2) Joint throughput/latency optimization of GPU allocations (§5). Agentic workflows combine LLMs with different resource needs, which are coupled across workflows. Scepsy’s scheduler therefore performs joint, throughput and latency aware, GPU allocation based on a novel search space that considers model parallelism together with fractional GPU allocations: for each LLM, Scepsy chooses (i) how many model replicas it deploys; (ii) what tensor parallel degree each replica uses; and (iii) what fraction of a GPU is given to each replica. Fractional GPU allocation allows Scepsy’s scheduler to model GPUs as quasi-continuous resources. This ensures that allocated model replicas are not overdimensioned, while maximizing GPU utilization. Considering these three dimensions is sufficient to optimize the throughput/latency trade-off: data parallelism using model replicas adds serving capacity to bottleneck LLMs; tensor parallelism reduces latency for LLMs on the critical path; and fractional GPU allocation increases GPU utilization. (3) Topology-aware fractional GPU placement (§6). With multiple heterogeneous LLM replicas and tensor-parallel instances, it is important to place them on GPU fractions in a topology-aware fashion, i.e., keep tensor-parallel instances in the same NVLink domain. This is an NP-hard problem, and it is not supported by existing GPU cluster orchestrators, such as Kubernetes [18]. In addition, poor placements can lead to cluster fragmentation, which makes future placements 2
Multiplexing Autoscaling
Multi-LLM serving Agentic Workflow
System AlpaServe [24] MuxServe [8] Prism [59] Aegaeon [56] Kubernetes HPA [18] AIBrix APA [47] llm-d WVA [30] vLLM Production Stack [49] Parrot [25] Ayo [46] TokenCake [3] Autellix [28] JITServe [61] Scepsy
Workflow-level Automatic GPU optimization Allocation Multi-LLM ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ Tab. 1: State-of-the-art systems comparison.
infeasible even with sufficient aggregate capacity. Scepsy performs topology-aware fractional GPU placement on top of Kubernetes: (1) Scepsy uses a heuristic that places large tensor-parallel LLMs fractions first across cluster nodes; (2) it then favors well-balanced tensor-parallel instance allocations within a GPU node, while packing smaller fractions tightly; (3) it produces Kubernetes deployment files, which a modified device plugin realizes as hierarchical placements; and (4) it uses Nvidia MPS to enforce isolation between the LLM fractions placed on GPUs. Our prototype implementation of Scepsy1 is implemented in 25K lines of Python code, and supports any agentic framework [20, 21, 23, 55] and most LLM execution engines [19, 63]. We evaluate Scepsy on three agentic workloads (RAG with reranking, beam search, and a combined workload) on a 16-GPU cluster. Compared to multi-LLM serving using Kubernetes [18] or Aegaeon [56], and workflow serving using Ayo [46], Scepsy reduces average workflow latencies by up to 27× at the same throughput, or increases peak throughput by up to 2.4× for the same GPU budget.
2
Optimization target SLO throughput SLO SLO configurable configurable configurable configurable latency latency latency latency SLO latency + t’put
workflows leverage additional compute resources (i.e., more LLM invocations and generated tokens) to structure reasoning steps and improve output quality. Therefore, methods such as beam search [42], Monte Carlo Tree Search [65], and multi-agent collaboration [23] explicitly introduce dependencies and control flow between LLM inference invocations. Agentic workflows thus have three properties: (1) they are composite, i.e., they contain arbitrary components including LLM agents, tools and databases; (2) they are multi-LLM, including different LLMs that must be adequate in size and performance for their roles; and (3) they have unpredictable execution, because their execution times and paths vary drastically depending on the request input. For example, our beam search trace shows that the number of generator LLM invocations for a single request spans from 24 to 844, and the end-to-end request latency varies from 9 to 264 seconds. Diversity of agentic programming models. In tandem with the diversity of emerging agentic workflows, many agentic frameworks have appeared [21, 23, 27, 51]. Frameworks provide different programming models for the development of complex agentic workflows, specifying for particular patterns. For example, LangGraph [21] defines agentic workflows as stateful graphs, with agents as nodes and control flow as edges between them; LlamaIndex [27] specializes in using retrieval-augmented generation (RAG) [22]; frameworks such as Autogen [55] and Camel [23] focus on enabling multi-agent collaboration through agent communication to solve complex problems. Each such agentic framework comes with its own APIs, resulting in agentic workflows that are expressed differently in terms of their programming model. The properties of agentic workflows and the diversity of programming models across agentic frameworks present significant challenges for serving systems. An efficient serving system must address four key requirements: (1) workflowlevel optimization that accounts for cross-LLM dependencies and unpredictable LLM invocations to improve performance
Agentic Workflow Serving
In this section, we describe the key properties of agentic workflows and their associated challenges (§2.1), and survey existing approaches for serving requests to multiple LLMs, explaining why they fall short for agentic workflows (§2.2). We then discuss recent approaches for handling dependencies between LLM invocations in agentic workflows (§2.3), and highlight their limitations (§2.4). 2.1
Unrestricted prog. model ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✓
Challenges when serving agentic workflows
Properties of agentic workflows. In agentic workflows, LLM agents interact with tools [41, 57], databases [22] and other agents [55] to solve complete complex tasks. For example, inference-time scaling [43] is a common technique: 1 Available post-acceptance at https://github.com/anon/Scepsy
3
beyond optimizing LLMs in isolation; (2) automatic GPU allocation that dynamically assigns GPU resources to LLMs without requiring manual user configuration; (3) multi-LLM support that handles workflows that comprise heterogeneous LLMs; and (4) unrestricted programming models that serve agentic workflows written in any framework without imposing custom APIs. Tab. 1 compares state-of-the-art agentic workflow serving systems against these four challenges. The table also includes, in the optimization target column, the performance metrics each system optimizes for (e.g., latency, SLO attainment, throughput, or some combination thereof). 2.2
2.3
Approaches for agentic workflow serving
Recent systems have focused on the problem of serving agentic workflows (Tab. 1). In contrast to multi-LLM serving, these approaches exploit workflow-level knowledge to address challenges specific to agentic execution. However, they generally lack support for heterogeneous LLMs and do not provide mechanisms for optimal GPU allocation. These systems can be classified into two categories: static analysis approaches and prediction-based scheduling approaches. Static analysis systems propose custom abstractions for agentic programming, in order to expose static dataflow graphs of agentic workflows. Analyzing workflow graphs enables them to apply new optimizations. For example, Parrot [25] applies graph-aware request batching and KV cache sharing across requests to reduce inference latency. Ayo [46] iteratively transforms the workflow graph with optimizations, such as splitting the prefilling phase for partial prompts and pipelining decoding of semantically dividable tasks. However, the proposed abstractions restrict the supported programming models, and prevent static analysis systems from supporting workflows with dynamic control flow that are commonplace today, such as beam search [42], where the generator LLM is invoked only on beams with the most promising scores as determined by the verifier LLM. Prediction-based scheduling systems formalize agentic workflows as dynamic graphs, and use workflow-level statistics to forecast request completion time and make request scheduling decisions. For example, Autellix [28] uses cumulative LLM execution time to assign request priorities, and dynamically preempts low-priority requests to avoid head-of-line blocking and improve overall latency. Similarly, JITServe [61] estimates an upper bound on response length, and gradually refines these estimates to schedule serving bandwidth with respect to latency SLOs. However, these systems lack support for multi-LLM agentic workflows: all agents in an agentic workflow must use the same LLM. In addition, these systems still leave the resource allocation decisions for each LLM agent to the user, limiting their ability to optimize serving performance.
Approaches for multi-LLM serving
Some existing approaches to serving agentic workflows focus on multi-LLM serving. These systems execute inference for multiple LLMs concurrently, treating each model as an independent service. As shown in Tab. 1, optimizing metrics at the individual LLM level enables multi-LLM serving systems to support flexible programming models, accommodate heterogeneous LLMs, and automatically allocate GPUs across the models within an agentic workflow. These systems fall into two categories: autoscaling and multiplexing. Autoscaling systems satisfy the resource requirements of serving multiple LLMs by adapting the number of serving engine replicas on demand for each LLM. These systems make scaling decisions either based on observed resource metrics [18, 47, 49] (e.g., GPU utilization, GPU cache usage), or configurable autoscaling policies that optimize towards a performance target [30, 47] (e.g., latency, Service-Level Objectives (SLOs) or throughput). When serving agentic workflows, autoscaling systems can suffer from oscillations in GPU allocation: a scaling decision for one LLM may cascade to other data-dependent LLMs, and prevent the workflow from meeting its configured performance targets. Multiplexing systems co-locate multiple LLMs on the same set of GPUs to share computational resources, either through temporal multiplexing [24], spatial multiplexing [59] or a mix of both [8, 56]. Compared to autoscaling, they allocate temporal or spatial slices of GPUs to serve multiple LLMs by targeting a single fixed performance objective such as SLO attainment or throughput. Multiplexing systems suffer from the unpredictable LLM invocations in agentic workflows, as they provide a fixed GPU multiplexing strategy for each LLM despite drastic variations in LLM execution times across workflow requests, leading to performance degradation. Overall, multi-LLM serving systems fall short for agentic workflows because they do not consider key properties of agentic workflows, such as the dependencies between LLM invocations and the unpredictability of LLM invocations. This prevents them from optimizing workflow-level performance metrics, such as end-to-end latency and throughput.
2.4
Limitations of end-to-end workflow statistics
When serving agentic workflows with multiple heterogeneous LLMs, it is important to allocate more resources to LLMs that have a larger contribution to the overall latency and throughput. Absolute end-to-end performance statistics, such as workflow execution latencies, however, provide limited information for this purpose. Fig. 3a shows the distribution of execution latencies for a generator LLM and a verifier LLM across 500 beam search requests. Since this agentic workflow involves dependencies between the two LLMs and individual LLM invocations are unpredictable, there is a wide distribution of end-to-end execution times, which would make it difficult to decide how to allocate GPU 4
Workflow GEN
VER …
programs Input
…
GEN
VER
Traces and Profiles
1
Conditional Output
Aggregate LLM pipeline
2
GEN
trace profile
Allocation
Latency
GPU fractions
Workflow API 4
Hierarchical placement
send
VER
synthetize
deploy
GEN Allocation
VER Allocation
Throughput
Allocation Generator with Pruning
Replicas
TP=1, GPU=1.0 TP=1, GPU=2.0 TP=2, GPU=2.0
GEN VER
Tensor parallelism Resource Scheduler
Topology-aware fractional placement
TP=1, GPU=1.0
3
Fig. 2: Scepsy Overview 1.0 Execution time fraction
Execution time (s)
1200 1000 800 600 400 200 0
Generator
Verifier
(a) Absolute
considering relative throughput and latency metrics across LLMs. Scepsy uses these relative performance statistics to construct an Aggregate LLM Pipeline, which represents the entire workflow as an abstract pipeline that aggregates independent, per-LLM profile traces. The Aggregate LLM Pipeline has good predictive power for the end-to-end throughput and latency of a workflow for a given number of LLM replicas and tensor-parallel instances within a given resource budget. This enables the Scepsy’s GPU scheduler to reason about the performance of different GPU allocations: using the Aggregate LLM Pipeline it can equalize the relative LLM throughput, and minimize the end-to-end latency in the Aggregate LLM Pipeline, without further profiling. We give an overview of the Scepsy system in Fig. 2. Scepsy builds and uses Aggregate LLM Pipelines to schedule one or more agentic workflows on a GPU cluster, while co-optimizing workflow throughput and latency. At a high level, Scepsy carries out the following steps: 1 it profiles each workflow and its constituent LLMs from execution traces; 2 it synthesizes the collected performance statistics into a per-workflow Aggregate LLM Pipeline; 3 a GPU scheduler then searches for a fractional resource allocation that fulfills the target arrival rate of all provided Aggregate LLM Pipelines (using LLM replication), and uses the remaining resources to best reduce their end-to-end latency (using tensor-parallel LLM instances); and 4 it places the selected GPU allocations using a topology-aware approach on a GPU cluster. As described in §2, existing agentic workflow serving systems require developers to express workflows in a particular programming model. In contrast, Scepsy can execute workflows defined in an arbitrary agentic framework because it captures workflow-level traces and uses these to express per-LLM performance statistics ( 1 ). Scepsy profiles the execution of a sequence of workflow-level requests, and for each trace, it extracts telemetry such as per-LLM prompts or cross-LLM serial/parallel request relationships. Scepsy then obtains per-LLM performance statistics by replaying all the traced LLM-level requests of a given LLM, across
0.8 0.6 0.4 0.2 0.0
Generator
Verifier
(b) Relative
Fig. 3: Distribution of absolute/relative end-to-end execution times for LLMs in beam search (Generator LLM is LLaMA-3.21B; Verifier LLM is LLaMA-3.2-8B; relative distributions are up to 4× more stable.)
resources to each LLM. Hence, prediction-based scheduling, as adopted by Autellix [28] and JITServe [61], struggles to generalize to multi-LLM agentic workflows. In contrast, relative statistics, such as the fraction of execution time spent in each LLM (see Fig. 3b), remain more stable across requests. Since the different LLMs in an agentic workflow are coupled, it is unsurprising that they are affected similarly by the execution cost of a request. Therefore, we can exploit this observation to identify bottleneck LLMs as part of the agentic workflow execution. Based on this, it becomes possible to predict the overall performance of an agentic workflow at steady state from these per-LLM relative statistics (e.g., throughput and latency), and allocate GPU resources to each LLM accordingly.
3
Scepsy Overview
Serving multi-LLM agentic workflows requires reasoning about how the allocation of GPU resources impacts workflow performance, but that is challenging because each request involves dynamic control flow and variable LLM input/output sequence lengths, and heterogeneous LLMs. Scepsy addresses this by leaning on our observation in §2.4: agentic workflows have more stable steady-state behavior when 5
all workflow-level requests, using various arrival rates and parallelism configurations. Scepsy then predicts the performance of the agentic workflow with the help of an Aggregate LLM Pipeline ( 2 ), which abstracts the workflow as a pipeline over LLMs using aggregate invocation counts, latency contributions, and throughput. This pipeline abstraction is sufficient because sequential and parallel LLM requests can be folded into a single aggregate LLM stage by adjusting latencies according to the average number of sequential invocations and the average degree of parallelism. For example, in Fig. 2, the looped and parallel LLM requests in the workflow are aggregated into the GEN and VER LLM stages of the pipeline (stage ordering does not matter; see §4). Scepsy then uses the Aggregate LLM Pipeline to predict workflow performance: it adds the latency contributions and considers the bottleneck throughput of each pipeline stage. In this way, Scepsy can provide reliable performance predictions of GPU allocations in terms of GPU resources, tensor parallelism, and model replica counts. Given LLM heterogeneity, speeding up one LLM can shift a workflow’s bottleneck to another LLM. The Scepsy scheduler therefore maximizes cluster efficiency and workflow performance by optimizing resource allocations jointly across all LLMs in one or more workflows ( 3 ). It does so by searching over fractional GPU resources, tensor parallelism, and replica assignments for each LLM, and looks for allocations that yield the lowest latency within a target workflow-level arrival rate. The scheduler provides an assignment within a reasonable time (seconds), using the Aggregate LLM Pipeline to predict the impact of each allocation. It prunes the search space using a combination of performance statistics, hardware topology information, and symmetries. Given a GPU resource allocation, Scepsy deploys and serves the workflows after placing the fractional LLM models on cluster GPUs ( 4 ). Scepsy supports both the co-location through fractional GPU allocation and distributed inference through tensor parallelism, which enables small LLMs to share GPUs while latency-critical LLMs can scale across tightly interconnected GPU nodes. Scepsy enforces the assigned fractional GPU allocations, and supports KV-cacheaware load balancing and GPU-aware topologies, on top of a Kubernetes orchestrator.
4
arbitrary agentic workflow application. Given an Aggregate LLM Pipeline, Scepsy can then quickly predict end-to-end throughput and latency for a given schedule. 1 - Workflow Tracing. Scepsy deploys the agentic workflow with a sequence of representative workflow-level requests and captures the resulting LLM-level request contents as a set of execution traces. Scepsy does this in a workflowand engine-agnostic way by deploying an HTTP proxy in front of the completions API of each LLM serving engine. The proxy captures the request and response contents of each LLM-level request, and associates them with start and end timestamps and a workflow identifier. Tracing is not used to capture the actual performance of this deployment, and can be parallelized by executing each workflow-level request in an independent deployment. 2 - Statistical Workflow Aggregation. For every LLM 𝑚, Scepsy extracts two workflow statistics: the average number of invocations per workflow request, 𝑛𝑚 , and its average request-level parallelism, 𝑝𝑚 . Request-level parallelism measures the average number of LLM requests that are running in parallel within a single workflow request, and is determined by overlapping timestamps in the execution traces; for example, the generator LLM shows three beams concurrently, which this step would capture as 𝑝𝐺𝐸𝑁 ≈ 3 and 𝑝𝑉 𝐸𝑅 ≈ 2 when the generator produces the same value on two beams. These statistics capture dynamic behavior such as loops, branches, and fan-out, without requiring static analysis of the application. Non-LLM components (e.g., tools, databases) are discarded at this stage, since they do not consume GPU resources and are typically negligible relative to LLM invocation costs. Importantly, these statistics determine the relationship between LLM- and workflow-level load; e.g., a workflow arrival rate of 𝜆𝑊 induces an arrival rate of 𝜆𝑚 = 𝜆𝑊 · 𝑛𝑚 for LLM 𝑚. 3 - LLM Profiling. Scepsy produces per-LLM throughput– latency profiles to predict how each LLM performs under a given allocation. It does so by replaying all the requests for a given LLM across all execution traces, at varying arrival rates, and maintaining inter-request dependencies within each trace. Replaying real traces preserves important characteristics such as prefill-heavy or decode-heavy request distributions, realistic token lengths, and KV-cache behavior (e.g., high prefix cache hit rates in the multi-turn beam search workflow). For each LLM, Scepsy sweeps the arrival rate from low load to saturation (i.e., maximum throughput), and for each point records latency at various percentile points and on average. The replay is repeated at multiple tensor parallelism degrees (e.g., TP = 1, 2, 4) to capture the empirical, non-linear scaling of each LLM under the target workload. Each LLM can be profiled in parallel, without a sweep over the number of replicas (replica count does not affect latency, and throughput scales linearly with replicas).
Aggregate LLM Pipeline
Scepsy uses Aggregate LLM Pipelines to reason about all possible workflow executions accurately, despite their dynamic and data-dependent behavior, while at the same time addressing the many shortcomings of previous systems [25, 28, 46]: it supports any programming model and LLM serving engine, multiple LLMs, provides GPU allocations rather than temporal request scheduling, and optimizes allocations for throughput-latency tradeoffs. Fig. 4 shows the steps that Scepsy follows to build an Aggregate LLM Pipeline from an 6
Beam workflow
Deploy and Execution traces trace LLM requests
Statistical workflow aggregation
2
Workflow statistics Invocations: 9 Parallelism: 3
...
"Let x = y - 3, ..."
Invocations: 6 Parallelism: 2
Aggregate LLM Pipeline
2-stage pipeline
LLM profiles
...
1 Example requests
(Normalize to workflow units)
4
Synthesize
5 Workflow Prediction
3 Profile LLMs by replaying traces
"If all As are B and ..." "A fox, a chicken ..."
Fig. 4: Construction of an Aggregate LLM Pipeline for beam search workflow. 4 - Aggregate LLM Pipeline Synthesis. Scepsy synthesizes the workflow statistics and per-LLM profiles into a pipeline of unique LLMs that captures the agentic workflow’s throughput–latency characteristics. Scepsy does so by transforming each LLM’s throughput–latency profile from LLM-level to workflow-level units. In the resulting pipeline, the latency contribution of each LLM 𝑚 to the latency of workflow 𝑤 incorporates both the number of invocations and request-level parallelism statistics of model 𝑚:
𝐿𝑤 (𝜆𝑤 ) =
∑︁
𝐿𝑤𝑚 (𝜆𝑤 ) =
∑︁
𝑚
𝐿𝑚 (𝜆𝑤 · 𝑛𝑚 ) ·
𝑚
𝑛𝑚 𝑝𝑚
then retrieves the corresponding workflow-level throughput– latency graph of each LLM based on 𝑇 𝑃𝑚 and 𝑃, and calculates 𝐿𝑤𝑚 (𝜆𝑤 ). For multiple replicas 𝑑𝑚 , the arrival rate is scaled to 𝜆𝑤 · 𝑛𝑚 /𝑑𝑚 . The complete prediction for a candidate allocation combines the per-LLM values using eqs. (1) and (2), requiring simple profile lookups and arithmetic. Aggregate LLM Pipelines therefore make prediction cost negligible compared to profiling the workflow for every possible resource allocation. This enables the Scepsy scheduler (§ 5) to rapidly explore large numbers of allocations.
(1)
5
For each agentic workflow, the challenge is to allocate the right amount of GPU resources to each LLM. In particular, an independent per-LLM allocation would not minimize latency while serving the workflow at a target arrival rate. However, the scheduling space is large due to the number of LLMs, GPUs, and degrees of parallelism in a multi-LLM agentic workflow, so the GPU scheduler must find a good solution in a reasonable amount of time. At the high level, the GPU scheduler enumerates candidate allocations with a specific configuration, such as allocated GPUs and tensor parallelism. During the search, the scheduler utilizes the Aggregate LLM Pipeline to predict the utility of the candidate allocations. The scheduler picks the allocation with the highest utility and Scepsy deploys the allocation with the topology-aware fraction placement. The GPU scheduler prunes the enumeration with workflow-level statistics such as the LLM latency ratios, e.g., so that LLMs with higher latencies get more resources. Agentic workflows can exhibit LLM heterogeneity (e.g., small embedding LLM and large generative LLM in RAGReranker). Consequently, the GPU demand of each LLM can differ substantially. To accommodate this, the Scepsy scheduler uses fractional GPU allocation to co-locate LLMs and increase the resource granularity from whole GPUs to GPU fractions. For instance, with an embedding LLM of latency 1 s and a generative LLM of latency 12 s, the embedding LLM should receive roughly 1/13 of the GPU resources, but with, e.g., , 4 GPUs, it would receive at least 1/4 of a GPU. Fig. 5 shows how the GPU scheduler searches for the best GPU allocation. It starts by extracting the latency ratios from
where 𝜆𝑤 is the target workflow-level arrival rate, and 𝐿𝑚 is the average per-request latency of LLM 𝑚 at the LLM arrival rate 𝜆𝑤 · 𝑛𝑚 . The ratio 𝑛𝑚 /𝑝𝑚 scales the LLM’s latency to account for its parallelism. The workflow latency 𝐿𝑤 is therefore the sum over all LLMs, while the maximum workflow throughput 𝑇𝑤 is the minimum across the throughput that each LLM can sustain, normalized to workflow requests: 𝑇𝑤 = min 𝑚
𝑇𝑚 𝑛𝑚
GPU Scheduling
(2)
where 𝑇𝑚 is the maximum throughput of LLM 𝑚. Requestlevel parallelism (𝑝𝑚 ) and arrival rate (𝜆𝑤 ) are not used here, since parallel requests do not reduce the total number of requests for an LLM, and the maximum throughput is independent of the arrival rate (respectively). The resulting throughput-latency curve of LLM 𝑚 is expressed in terms of workflow-level throughput (𝜆𝑤 ) and workflow-level latency contribution (𝐿𝑤𝑚 ), which is also parameterized by the degree of tensor parallelism and target latency percentile when selecting the LLM profile data (not shown for clarity). Once all LLMs are expressed using the workflow-level metrics in an Aggregate LLM Pipeline, its structure directly reveals which LLM dominates latency and which LLM is the throughput bottleneck. 5 - Workflow Predictions. To predict the effect of a resource allocation on workflow latency and throughput, Scepsy takes as input a workflow arrival rate 𝜆𝑤 , a tensor parallelism degree 𝑇 𝑃𝑚 , and replica count 𝑑𝑚 for each LLM, and the percentile 𝑃 at which latency should be evaluated. Scepsy 7
Search Allocation Space LLM A
LLM B
ALP Latency Ratios 3.5 s
1.6 s
3.6 s
GPU Fractions
1
GPU Packing
2
Parallelism LLM A
B
≥
LLM A
Latency
3
2 Packing fractions onto GPUs. The output of GPUfraction enumeration is a per-LLM GPU fraction, e.g., LLM A gets 1.66 GPUs. The GPU scheduler must map these fractions to actual GPUs, since the number of GPUs spanned by each LLM is needed to reason about parallelism configurations later. It packs GPU fractions contiguously onto GPUs. For example, 1.66 GPUs are packed as a 1.0 fraction on GPU 1 and a 0.66 fraction on GPU 2. This ordering ensures that the highest-latency LLMs have less partitioning across GPUs. 3 Resolving parallelism. Given the number of GPUs per LLM, the goal is to determine which parallelism configurations are feasible. The GPU scheduler considers two dimensions of parallelism: tensor parallelism and data parallelism through replica count. At a high level, it enumerates each combination and verifies that it fits on the available GPUs. Feasible combinations must satisfy the constraint that the tensor parallelism degree times replica count evenly divides the number of GPUs. Otherwise, the allocation would underutilize GPU resources and hurt performance. In addition, configurations where tensor parallelism would exceed the degree supported by the high-bandwidth GPU interconnect are pruned, since higher tensor parallelism degrees would incur excessive communication overhead. Scheduling multiple workflows. Users may often want to deploy more than one workflow at a time. To support multiworkflow deployments, the GPU scheduler takes multiple workflows and their arrival rates as input and constructs a multi-workflow resource allocation. This adds an additional level to the search hierarchy to determine how many GPUs to allocate to each workflow. For each workflow and its assigned GPUs, the GPU scheduler searches for the per-workflow allocation independently. It combines the per-workflow utility into an overall utility. Scepsy uses an egalitarian welfare policy to aggregate the workflow utilities, but can support other fairness policies, such as social welfare.
LLM B
0.66 0.66 ratio 1.00 1.00 pruning … … 2.33 2.66 3.0-0.33 4.0-0.66-0.33
0.33 0.66 … 2.33 2.66
Packing Allocations to GPUs
contiguous allocations only LLM C
Resolve Parallelism
3
Replica Replica Tensor Parallelism Utility
≥
LLM C
2
LLM C
Enumerate Allocations
1
LLM C
Allocation
Aggregate LLM Pipeline Predictor
TP
LLM C
LLM A
Replicas
Replicas
T1R1 T1R2 T2R1 T2R2
TP
underutilizing resources
LLM B
T1R1 T1R2
Replicas TP T1R1
T2R1 T2R2 exceeding resources
Fig. 5: Overview of GPU scheduling in Scepsy
the Aggregate LLM Pipeline. Based on the ratios, the GPU scheduler 1 enumerates possible assignments of LLMs to GPUs fractions; 2 maps each LLM’s GPU fraction to GPUs, determining how many GPUs each LLM would be deployed on and co-locations; and 3 it then uses the LLM-to-GPU mapping to explore the tensor parallelism degree and number of replicas to infer potential parallelism configurations. 1 Enumeration of GPU fractions. The first scheduling step is the enumeration of all mappings from LLMs to GPU fractions. Given 5 LLMs, 16 GPUs, and 10 fractions per GPU, the number of possible mappings is large: 16∗10+5−1 ≈ 5−1 29 million. To make this enumeration feasible, Scepsy uses three pruning strategies: (i) giving more resources to LLMs with a higher profiled latency because it is a proxy for the LLM’s resource demand; (ii) only allocating GPUs contiguously, because it exploits the symmetry of GPU allocations; and (iii) limiting the degree of tensor parallelism to the highbandwidth inter-GPU degree, because for higher degrees the communication overhead becomes prohibitive. The main pruning strategy is to order the LLMs by their latency ratios, since latency is a proxy of compute demand, ensuring that LLMs with a higher latency ratio get more GPU fractions. By following the ratios, the scheduler leaves enough freedom to move the boundary between GPU fractions of LLMs while excluding allocations in which small LLMs (e.g., an embedding LLM) get more resources than large LLMs (e.g., generative Mixture-of-Expert LLM). In addition, the GPU scheduler assigns each LLM the minimum number of GPU fractions required to load the LLM parameters and initialize the KV cache. This creates a lower bound of fractions per LLM. For the upper bound, the scheduler subtracts the sum of the minimum GPU fractions of all lower-latency LLMs. Together, these pruning strategies limit the per-LLM GPU fractions and further reduce the search space.
6
Topology-Aware Fractional Placement
The Scepsy scheduler determines how much GPU compute each LLM should receive (number of replicas, tensor-parallel instances, GPU fraction sizes), but not cluster placement. In particular, tensor parallelism requires topology-aware placement to find suitable NVLink domains [37] (often not symmetric), and fractional allocation requires strict resource limits on each LLM instance. However, existing cluster orchestrators do not simultaneously support topology-aware placement and fractional GPUs [36]. Scepsy therefore computes placements ahead-of-time, using simpler orchestrators such as Kubernetes and enforcing fractional allocations at runtime. Hierarchical placement algorithm. Scepsy must avoid cluster fragmentation to make placements feasible under pernode capacity and topology constraints. For example, if a number of small fractions is placed before a large one, they 8
may consume capacity on the few nodes that can host larger tensor-parallel models within an NVlink domain. The optimal solution is NP-hard [7], but Scepsy uses a hierarchical heuristic that first places LLMs into nodes while prioritizing larger instances (inter-node stage), and then assigns GPU fractions within a node (intra-node stage). Scepsy uses a most-constrained-first heuristic that places tensor-parallel models before non-tensor-parallel models and, within each category, places larger allocations first. This ensures NVLink-connected capacity is prioritized for large tensor-parallel allocations, and then spreads the smaller allocations across the remaining cluster capacity. For each node, Scepsy enumerates NVLink domains, scores those with capacity by the imbalance between their largest and smallest per-GPU fractions. Among the most balanced candidates, Scepsy prefers the one with the least capacity, preserving large capacity NVLink domains when possible. For sub-GPU fractions, Scepsy instead packs them onto already occupied GPUs first. Deployment into Kubernetes. Scepsy realizes its placement by producing Kubernetes deployment files that “lock” inter-node placement decisions, and a Nvidia device plugin we extended to implement intra-node GPU-fraction placement. Kubernetes provides the remaining control-plane mechanisms for deployment, restart, and health management. Scepsy uses Nvidia MPS [34], with static limits on SMs and memory, to enforce resource limits for each GPU fraction. Scepsy relies on standard serving components, using one vLLM [19] or SGLang [63] engine per replica and an SGLang router [63] per workflow for load- and KV-cache-aware routing.
7
use the Model Gateway v0.3 of SGLang [63]. For comparability with the Ayo [46] baseline, we use vLLM [19] v0.2. Baselines. We compare Scepsy to (1) Kubernetes [18] using MicroK8s [4] v1.32 as autoscaler baseline, (2) Aegaeon [11, 56] for multi-LLM multiplexing, and (3) Ayo [32, 46] as workflow-aware baseline. Workloads. We use RAG+reranker [33, 40] and beam search [15, 16] as workflows to benchmark the systems. In addition, we combine RAG+reranker and beam search as combined workflows. For RAG+reranker the LLMs are e5-base-v2 and Llama-3-8B. For beam search, we use Llama-3.2-1B and Llama3.1-8B-PRM. For compatibility with the older vLLM version, we use Llama-2-7b and math-shepherd.
7.2
First, we evaluate the end-to-end throughput–latency curve to explore the benefits of Scepsy’s resource allocation. For each workflow and across different cluster sizes, we execute the workflows at different arrival rates. Autoscaling. In this experiment, we compare Scepsy to the autoscaler baseline Kubernetes [18]. We evaluate the systems for RAG+reranker and beam search for the cluster sizes of 4, 8, and 16 GPUs. For RAG+reranker, for 4 GPUs, we vary the arrival rate from 1–28 req/s, for 8 GPUs, the arrival rates are 2–56 req/s, for 16 GPUs, the arrival rates are 4–112 req/s. Fig. 6 shows the achieved throughput on the x-axis and the workflow request latency on the y-axis. For all workflows and cluster sizes, Scepsy outperforms Kubernetes autoscaler for throughput and latency. For beam search, Scepsy achieves 2.4× higher throughput for 4 GPUs, 1.5× for 8 GPUs, and 1.8× for 16 GPUs. The latency improvements are 1.4×– 7.6× for 4 GPUs, 1.3×–5.2× for 8 GPUs, and 1.7×–10.4× for 16 GPUs. For RAG+reranker, there is a 1.5×, 1.2×, and 1.2× throughput improvement for 4, 8, and 16 GPUs. The latency improvements are 1.9×–14.9×, 1.2×–7.2×, and 1.4 × −27×, respectively. Scepsy outperforms Kubernetes autoscaler across the board, because the Aggregate LLM Pipeline does accurate predictions with which the GPU scheduler finds a well-performing allocation. Kubernetes, with autoscaling, is hurt by scaling up too much and then scaling down because the request queues are empty, therefore ending up oscillating between allocations. Additionally, the autoscaler doesn’t know about parallelisms such as tensor parallelism to reduce latency if the resource budgets allow. LLM multiplexing. We compare Scepsy against Aegaeon [56], a multi-model serving system that pools GPU resources across models via token-level auto-scaling. Aegaeon disaggregates serving into dedicated prefill and decode instances; we serve non-LLM models (rerankers, embeddings) via its prefill instances. We test three per-node prefill/decode splits:
Evaluation
We evaluate Scepsy for two workflows: RAG+reranker and beam search. For comparison, we benchmark against the autoscaler of Kubernetes [18], multi-LLM serving of Aegaeon [56], and workflow-aware serving of Ayo [46] (§7.2). We do an ablation study to show how each contribution to the overall performance (§7.4). Additionally, to showcase the ability to combine multiple workflows, we simultaneously run RAG+reranker and beam search (§7.3), and we benchmark the GPU scheduling search (§7.5). 7.1
End-to-end workflows
Experimental Setup
The experiments have the following setup: Cluster. We conduct on-premise experiments with 16 GPUs (4 machines with 4 GPUs each). Each machine has an AMD EPYC 7402P CPU, 4× NVIDIA RTX A6000 GPUs, and PCIe 4.0. The machines are interconnected by 100 Gbps InfiniBand, and the GPUs are connected in two pairs using 3rd generation NVLink. Software. For Scepsy, we use MicroK8s [4] v1.32. As an LLM engine, we use vLLM [19] v0.17, and as a request router, we 9
4 GPUs
Average Latency (s)
4 2
30
2.5
5.0
7.5
2
10.0 12.5 15.0 17.5
5
30
10
15
20
25
30
35
Throughput (req/s)
00.0
0.5
1.0
1.5
2.0
Throughput (req/s)
2.5
10
20
30
40
50
60
70
Throughput (req/s)
80
15 10
5 00
3.0
2
20
10
5
4
25
15
10
6
30
20
15
8
00
40
Scepsy K8s HPA
25
20
Beam Search
4
Throughput (req/s)
Scepsy K8s HPA
25
6
00
Scepsy K8s HPA
10
8
Average Latency (s)
Average Latency (s)
RAG + Reranker
6
00.0
Scepsy K8s HPA
10
8
16 GPUs
12
Average Latency (s)
Scepsy K8s HPA
10
Average Latency (s)
8 GPUs
12
Average Latency (s)
12
1
2
3
4
Throughput (req/s)
5
00
6
Scepsy K8s HPA 10
5 2
4
6
Throughput (req/s)
8
Fig. 6: Throughput–latency comparison across workloads and 4, 8, 16 GPUs 8 GPUs
7.5
Aegaeon 2P2D Aegaeon 1P3D Scepsy
Average Latency (s)
125 100 75 50 25 0 0.125
00
10.0 12.5 15.0 17.5
Achieved Throughput (req/s)
0.25
0.5
1
Achieved Throughput (req/s)
2
5
10
15
20
25
30
Achieved Throughput (req/s)
150
35
Aegaeon 4P4D Aegaeon 2P6D Scepsy
125 100 75 50 25 0
0.25
0.5
1
2
Achieved Throughput (req/s)
4
8 GPUs 20
Ayo Scepsy
Average Latency (s)
Average Latency (s)
40
15
15
10
175 150 125 100 75 50 25 00.0
Ayo Scepsy
10
5 00
Average Latency (s)
5.0
5
Beam Search
2.5
150
Beam Search
RAG + Reranker
10
5 00.0
Average Latency (s)
10
20
Aegaeon 6P2D Aegaeon 2P6D Scepsy
15
Average Latency (s)
Average Latency (s)
RAG + Reranker
Aegaeon 3P1D Aegaeon 1P3D Scepsy
15
4 GPUs
20
2
4
6
Throughput (req/s)
8
00.0
10
Ayo Scepsy
0.1
5
175 150 125 100 75 50 25 00.0
2.5
5.0
7.5 10.0 12.5 15.0 17.5 20.0
Throughput (req/s)
Average Latency (s)
4 GPUs 20
0.2
0.3
Throughput (req/s)
0.4
0.5
0.2
0.4
0.6
0.8
Throughput (req/s)
Ayo Scepsy 1.0 1.2
Fig. 7: Scepsy vs Aegaeon throughput-latency comparison across workloads and cluster scales
Fig. 8: Scepsy vs Ayo throughput-latency comparison across workloads and cluster scales with vLLM v0.2.2.
3P/1D, 2P/2D, and 1P/3D (four instances per node). For 8GPU experiments (two nodes), we use the same per-node configurations. Fig. 7 shows the results for RAG and beam search on 4 and 8 GPUs. Across all configurations and workloads, Scepsy outperforms Aegaeon. For beam search, Scepsy achieves 7.3× higher throughput on 4 GPUs and 6.8× on 8 GPUs, with latency improvements of 14.1× and 12.9×, respectively. For RAG, Scepsy achieves 2.5× higher throughput on 4 GPUs and 2.9× on 8 GPUs, with latency improvements of 3.4× and 4.5×, respectively. Among the tested configurations, every Aegaeon split falls short of Scepsy. Aegaeon lacks prefix caching, which particularly penalizes beam search where multiple candidate solutions share long common prefixes. Additionally, prefill instance swapping between models is a significant source of overhead. Scepsy
does not have these disadvantages and leverages workflowawareness to achieve better performance. Agentic workflow. To compare Scepsy to a agentic workflow system, we use Ayo [46] as the baseline. We evaluate for RAG+reranker and beam search workflows and use cluster sizes 4 and 8 GPUs. For RAG+reranker, for 4 GPUs, we vary the arrival rate from 1 − 12 req/s, for 8 GPUs, the arrival rates are 2 − 24 req/s. Fig. 8 follows the same setup as Fig. 6. Ayo has comparable latencies to Scepsy in the low-throughput region. In the throughput-bound region of higher arrival rates, Ayo soon reaches the throughput limit while Scepsy can still serve the requests. For beam search, Scepsy achieves 3.2× higher throughput for 4 GPUs and 8.2× for 8 GPUs. For RAG+ reranker, there is a 2.1× and 2.4× throughput improvement for 4 and 8 GPUs. 10
0.00
5 0
1
2
3
Throughput (req/s)
4
(a) RAG + Reranker
15
7.5
10
5.0
5
2.5
00
0.0
5
10
15
20
25
Throughput (req/s)
30
35
(b) Beam Search
8
8 GPUs 12
Naive Scepsy Parallelism Only Fractional Only Scepsy
4 2
25 20
2.5
5.0
7.5
8 6 4 2 00
10.0 12.5 15.0 17.5
Throughput (req/s)
30
Naive Scepsy Parallelism Only Fractional Only Scepsy
25 20
15
5
10
15
20
25
30
Throughput (req/s)
35
40
5
6
Naive Scepsy Parallelism Only Fractional Only Scepsy
15
10
10
5 00.0
Naive Scepsy Parallelism Only Fractional Only Scepsy
10
6
00.0
Average Latency (s)
Beam Search
For a few of the data points in the latency bound region, Ayo can slightly beat Scepsy with their optimizations, such as request batching. But overall Scepsy has lower latencies over a much wider range of arrival rates. That’s because Scepsy can adapt the GPU allocations based on the arrival rate and optimizes for latency when it has enough resources and switches to optimizing for throughput when necessary. Therefore, showing that request-level scheduling is not enough and does not have the main impact on throughput-latency of agentic workflows.
0.5
1.0
1.5
2.0
2.5
Throughput (req/s)
3.0
3.5
5 00
1
2
3
4
Throughput (req/s)
Fig. 10: Ablation study showing the contribution of Scepsy’s key optimizations across workloads and cluster scales.
parallelism. When both optimizations are disabled, Scepsy experiences cumulative latency degradation and achieves only a lower throughput compared to the full system. The results show that both optimizations are required and contribute significantly to overall performance and demonstrate that Scepsy’s co-location and parallelism mechanisms work synergistically to deliver good performance.
Combined workflows
Scepsy supports serving multiple workflows. To demonstrate that Scepsy can adapt the allocation to different arrival rates, we fixed the arrival rate of one workflow and varied the arrival rate of the second workflow. Fig. 9a shows a static arrival rate for beam search and varies the arrival rate for RAG+reranker. You can see that in the left half, both workflows have a low latency, and when the cluster utilization goes up, together with the arrival rate, latencies increase. Fig. 9b shows similar behavior for a static arrival rate for RAG+reranker and varying arrival rate for beam search. Scepsy can support multiple workflows and adapt to different workflow demands, i.e., arrival rates. That’s because the GPU scheduler considers multiple workflows and ensures fairness between the workflows without starving one of them to find an allocation that accommodates multiple workflows. 7.4
10
30
Fig. 9: Effect of combined workflows on per-workload performance on 8 GPUs.
7.3
12
Average Latency (s)
50
10.0
Average Latency (s)
0.5
4 GPUs
12.5
Average Latency (s)
100
RAG Beam @ 1 req/s
20
RAG + Reranker
1.0
25
Beam Average Latency (s)
1.5
150
RAG Average Latency (s)
200
RAG @ 5 req/s Beam
Beam Average Latency (s)
RAG Average Latency (s)
2.0
7.5
Scheduler Latency
The search time of Scepsy could be very high because of the many combinations of GPU allocations. To verify that the pruning enables finding a solution in a reasonable time, we do a micro-benchmark and measure the GPU scheduler search time, and vary the search parameters, such as the number of LLMs, GPUs, and fractions per GPU. Fig. 11 shows how the search time changes with the number of LLMs. It’s with 16 GPUs and 10 fractions each. The search time grows exponentially with the number of LLMs, but for all numbers of LLMs the search time stays below 35 s. In Fig. 11, the shape of the curve for the number of GPUs is exponential, too. It uses 3 LLMs and 10 fractions per GPU. Similarly, the search time goes up to 70 s for 128 GPUs. For the number of fractions per GPU with 3 LLMs and 16 GPUs, the search time goes up to 1 s. The experiments show that even though search times increase with larger dimensions, the time stays below a minute. Therefore, Scepsy finds a good allocation in a reasonable time.
Performance impact of contributions
To understand the contribution of Scepsy’s key optimizations, we conduct an ablation study that evaluates the throughput-latency curve with different features disabled. We run experiments in four configurations: the full Scepsy system, Scepsy without parallelism, Scepsy without co-location (fractional), and Scepsy with both optimizations disabled. That’s done for both workflows, RAG+reranker and beam search for 4 and 8 GPUs each. Fig. 10 shows the throughput-latency curve. For RAG+ reranker for 4 and 8 GPUs, co-location is the biggest contributor to the throughput improvement. For beam search for 4 and 8 GPUs, the reduction in latency is due to the tensor
8
Related Work
LLM serving engines optimize performance when executing LLM inference requests. vLLM [19] reduces KV cache fragmentation via PagedAttention to maximize throughput. Sarathi-Serve [1] batches prefill and decode requests via chunked prefill to improve GPU utilization. NanoFlow [66] and PodAttention [17] further improve intra-device resource utilization by overlapping operations with complementary 11
1.00 0.10 0.01 2
3
Number of LLMs
5
Search Time (s)
Search Time (s)
Search Time (s)
10.00
60 40 20 0
0
50
100
Number of GPUs
0.8 0.6 0.4 0.2 0.0
0
10
20
Fractions per GPU
Fig. 11: Search time as scheduling parameters scale for the combined workflow.
resource demands and fusing prefill and decode attention, respectively. In contrast, Scepsy is an orchestration layer between agentic workflows and LLM serving engines, and is compatible with different engines. Distributed LLM systems. Considerable research has focused on distributed LLM serving [6, 13, 39, 45]. DistServe [64] decouples prefill and decode to scale them independently, and LoongServe [53] extends this with elastic scheduling across heterogeneous clusters. Serverless approaches [9, 58, 60] optimize cold-start latency via caching. CornServe [29] disaggregates models into independently scaled components. However, all of these systems optimize single-request LLM performance and lack workflow awareness, leading to suboptimal performance for agentic workloads. GPU sharing techniques. Prior work has explored sharing GPU resources [26, 50, 54, 62] through temporal and spatial multiplexing. Temporal multiplexing shares a GPU by context-switching between jobs. For example, Nvidia vGPUs [35] and AlpaServe [24] both time-slice a GPU across multiple models. However, these systems incur significant context-switching overhead for LLM serving. Spatial multiplexing instead allows multiple processes to run simultaneously, eliminating this overhead. Nvidia MPS [34] provides each application with dedicated memory and streaming multiprocessors; Orion [44] co-schedules non-interfering kernels for fine-grained spatial sharing; and Reef [12] pads realtime kernels with best-effort kernels to maximize utilization. Scepsy leverages Nvidia MPS to co-locate LLMs.
9
scheduling. Rather than fitting a single aggregate distribution, Scepsy could classify traces into a small set of recurring usage patterns, build a separate Aggregate LLM Pipeline for each, and combine them during prediction as it already does for multiple workflows. This would let the predictor reason about each mode separately while still allocating the underlying LLM replicas jointly. Non-negligible tool execution times. Most tools used by agentic workflows are short-running, which allows Scepsy to ignore them in the Aggregate LLM Pipeline. Some workflows, however, include long external operations, such as monitoring experiments, that can dominate end-to-end latency. To support such cases, the Aggregate LLM Pipeline could include non-optimizable tool stages that add a fixed latency bias but are unaffected by GPU allocations.
10
Conclusion
We described Scepsy, a system for serving multi-LLM agentic workflows on GPU clusters. Scepsy is built on the observation that, although individual workflow executions are highly dynamic, their aggregate per-LLM behavior is substantially more stable across requests. Scepsy captures this behavior with the Aggregate LLM Pipeline. This compact abstraction predicts end-to-end workflow throughput and latency from traced executions and per-model profiling, and uses it to jointly optimize fractional GPU shares, tensor parallelism, and replica counts across all LLMs in a workflow. Its topology-aware runtime then efficiently realizes these allocations. Our evaluation shows that it is sufficient to turn unpredictable workflows into good GPU allocations, yielding substantial throughput and latency improvements. Broadly, Scepsy exhibits that aggregate steady-state reasoning is a practical foundation for efficient, framework-agnostic systems supporting dynamic agentic workflows.
Discussion
Inter-LLM parallelism. Scepsy’s parallelism adjustment on latency handles the common case of fan-out on the same LLM. If a workflow instead fans out across different LLMs, the Aggregate LLM Pipeline would incorrectly model that as a serial pipeline. A solution is for Scepsy to better model only the critical-path contributions of each LLM by tracking overlapping traces in the Aggregate LLM Pipeline. Bimodal load distributions. When an LLM is reused across workflows, or in distinct roles within a workflow, it can induce a bimodal load distribution that leads to suboptimal
References [1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/conference/osdi24/presentation/ 12
[18] Kubernetes. 2014. Kubernetes. https://kubernetes.io/. Accessed: 2026-03-08. [19] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [20] LangChain. 2022. LangChain. https://www.langchain.com/. Accessed: 2026-03-08. [21] LangChain. 2024. LangGraph. https://www.langchain.com/langgraph. Accessed: 2026-03-26. [22] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrievalaugmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages. [23] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Thirty-seventh Conference on Neural Information Processing Systems. [24] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 663–679. https://www.usenix.org/conference/ osdi23/presentation/li-zhouhan [25] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM-based Applications with Semantic Variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 929–945. https://www.usenix. org/conference/osdi24/presentation/lin-chaofan [26] Zejia Lin, Hongxin Xu, Guanyi Chen, Zhiguang Chen, Yutong Lu, and Xianwei Zhang. 2026. Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (USA) (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 290–306. doi:10.1145/3779212.3790135 [27] Jerry Liu. 2022. LlamaIndex. https://github.com/jerryjliu/llama_index [28] Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. 2025. Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv:2502.13965 [cs.LG] https: //arxiv.org/abs/2502.13965 [29] Jeff J. Ma, Jae-Won Chung, Jisang Ahn, Yizhuo Liang, Akshay Jajoo, Myungjin Lee, and Mosharaf Chowdhury. 2025. Cornserve: Efficiently Serving Any-to-Any Multimodal Models. arXiv:2512.14098 [cs.LG] https://arxiv.org/abs/2512.14098 [30] Abhishek Malvankar, Lionel Villard, Mohammed Abdi, Evgeny Shindin, Braulio Dumba, Vishakha Ramani, Asser Tantawi, and Tamar Eilam. 2026. WVA: A Global Optimization Control Plane for llmd. arXiv preprint arXiv:2603.09730 (2026). [31] Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et al. 2025. Artificial intelligence index report 2025. arXiv preprint arXiv:2504.07139 (2025). [32] NetX-lab. 2025. Ayo Artifact. https://github.com/NetX-lab/Ayo. Accessed: 2026-03-20.
agrawal [2] Anthropic. 2023. Claude. https://www.anthropic.com/claude. Accessed: 2026-03-08. [3] Zhuohang Bian, Feiyang Wu, Teng Ma, and Youwei Zhuo. 2025. Tokencake: A KV-Cache-centric Serving Framework for LLM-based MultiAgent Applications. arXiv:2510.18586 [cs.DC] https://arxiv.org/abs/ 2510.18586 [4] Canonical. 2026. MicroK8s. https://canonical.com/microk8s. accessed 2026-03-22. [5] Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research (2024). https: //openreview.net/forum?id=cSimKw5p6R Featured Certification. [6] Rongxin Cheng, Yuxin Lai, Xingda Wei, Rong Chen, and Haibo Chen. 2025. KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving. arXiv:2412.18169 [cs.DC] https://arxiv.org/abs/2412.18169 [7] János Csirik, Johannes Bartholomeus Gerardus Frenk, Martine Labbé, and Shuzhong Zhang. 1990. On the multidimensional vector bin packing. Acta Cybernetica 9, 4 (1990), 361–369. [8] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 11905–11917. https://proceedings.mlr.press/v235/duan24a.html [9] Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: LowLatency Serverless Inference for Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 135–153. https: //www.usenix.org/conference/osdi24/presentation/fu [10] Google. 2023. Gemini. https://gemini.google.com. Accessed: 2026-0308. [11] Alibaba Group. 2025. Aegaeon Artifact. https://zenodo.org/records/ 16673199. Accessed: 2026-03-20. [12] Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 539– 558. https://www.usenix.org/conference/osdi22/presentation/han [13] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arXiv:2401.08671 [cs.PF] https://arxiv. org/abs/2401.08671 [14] Hanbo Huang, Yihan Li, Bowen Jiang, Bo Jiang, Lin Liu, Ruoyu Sun, Zhuotao Liu, and Shiyu Liang. 2025. A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality. arXiv:2410.11182 [cs.LG] https://arxiv.org/abs/2410.11182 [15] Hugging Face. 2024. Scaling Test-Time Compute. https://huggingface. co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute. Accessed: 2026-03-20. [16] Hugging Face. 2024. Search and Learn. https://github.com/ huggingface/search-and-learn. Accessed: 2026-03-20. [17] Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. Association for Computing Machinery, New York, NY, USA, 897–912. https://doi.org/10.1145/3676641.3715996
13
[33] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document Ranking with a Pretrained Sequence-to-Sequence Model. arXiv:2003.06713 [cs.IR] https://arxiv.org/abs/2003.06713 [34] NVIDIA. 2022. NVIDIA MPS. https://docs.nvidia.com/deploy/mps/ index.html. Accessed: 2026-03-08. [35] NVIDIA. 2022. Time-Sliced NVIDIA vGPU. https://docs.nvidia.com/aienterprise/release-7/latest/infra-software/vgpu/overview.html# nvidia-vgpu-architecture-overview. Accessed: 2026-03-20. [36] NVIDIA. 2026. NVIDIA Kubernetes device plugin: README, With CUDA Time-Slicing. https://github.com/NVIDIA/k8s-device-plugin/ blob/44345e3d/README.md#with-cuda-time-slicing. GitHub repository documentation, commit 44345e3d, accessed 2026-03-22. [37] NVIDIA Corporation. 2023. NVLink™ Fabric High-Speed Interconnect White Paper. Technical Report. NVIDIA Corporation. https://www. nvidia.com/en-gb/data-center/nvlink/ White paper; GPU-GPU and switch interconnect. [38] OpenAI. 2022. ChatGPT. https://openai.com/chatgpt. Accessed: 2026-03-08. [39] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCachecentric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, 155–170. https://www.usenix.org/conference/fast25/ presentation/qin [40] RUC-NLPIR. 2024. FlashRAG. https://github.com/RUC-NLPIR/ FlashRAG. Accessed: 2026-03-20. [41] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems 36 (2023), 68539–68551. [42] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2025. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=A6Y7AqlzLW [43] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024). [44] Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece) (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 1075–1092. doi:10.1145/3627703.3629578 [45] Qidong Su, Wei Zhao, Xin Li, Muralidhar Andoorveedu, Chenhao Jiang, Zhanda Zhu, Kevin Song, Christina Giannoula, and Gennady Pekhimenko. 2025. Seesaw: High-throughput LLM Inference via Model Re-sharding. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. https://proceedings.mlsys.org/paper_files/paper/2025/file/ cbc4ab80cd77aa0eb87da062fbcddb46-Paper-Conference.pdf [46] Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards Endto-End Optimization of LLM-based Applications with Ayo. Association for Computing Machinery, New York, NY, USA, 1302–1316. https: //doi.org/10.1145/3676641.3716278 [47] The AIBrix Team, Jiaxin Shan, Varun Gupta, Le Xu, Haiyang Shi, Jingyuan Zhang, Ning Wang, Linhui Xu, Rong Kang, Tongping Liu, Yifei Zhang, Yiqing Zhu, Shuowei Jin, Gangmuk Lim, Binbin Chen, Zuzhi Chen, Xiao Liu, Xin Chen, Kante Yin, Chak-Pong Chung, Chenyu Jiang, Yicheng Lu, Jianjun Chen, Caixue Lin, Wu Xiang, Rui Shi, and Liguang Xie. 2025. AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure. arXiv:2504.03648 [cs.DC] https://arxiv.org/abs/2504.03648
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). [49] vllm-project. 2025. vLLM Production Stack. https://github.com/vllmproject/production-stack. Accessed: 2026-03-08. [50] Jiali Wang, Yankui Wang, Mingcong Han, and Rong Chen. 2025. Colocating ML inference and training with fast GPU memory handover. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 98, 19 pages. [51] Yanlin Wang, Xinyi Xu, Jiachi Chen, Tingting Bi, Wenchao Gu, and Zibin Zheng. 2025. An empirical study of agent developer practices in ai agent frameworks. arXiv preprint arXiv:2512.01939 (2025). [52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. [53] Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (Austin, TX, USA) (SOSP ’24). Association for Computing Machinery, New York, NY, USA, 640–654. doi:10.1145/3694715.3695948 [54] Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Transparent GPU Sharing in Container Clouds for Deep Learning Workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 69– 85. https://www.usenix.org/conference/nsdi23/presentation/wu [55] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First conference on language modeling. [56] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. 2025. Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (Lotte Hotel World, Seoul, Republic of Korea) (SOSP ’25). Association for Computing Machinery, New York, NY, USA, 1030–1045. doi:10.1145/3731569.3764815 [57] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations. [58] Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, Haoran Yang, and Yu Ding. 2025. Torpor: GPU-enabled serverless computing for lowlatency, resource-efficient inference. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC ’25). USENIX Association, USA, Article 36, 16 pages. [59] Shan Yu, Jiarong Xing, Yifan Qiao, Mingyuan Ma, Yangmin Li, Yang Wang, Shuo Yang, Zhiqiang Xie, Shiyi Cao, Ke Bao, Ion Stoica, Harry Xu, and Ying Sheng. 2025. Prism: Unleashing GPU Sharing for CostEfficient Multi-LLM Serving. arXiv:2505.04021 [cs.DC] https://arxiv. org/abs/2505.04021 [60] Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. BLITZSCALE: fast and live large model autoscaling with O(1) host caching. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 16, 19 pages. [61] Wei Zhang, Zhiyu Wu, Yi Mu, Rui Ning, Banruo Liu, Nikhil Sarda, Myungjin Lee, and Fan Lai. 2025. JITServe: SLO-aware LLM Serving with Imprecise Request Information. arXiv:2504.20068 [cs.DC] https: 14
Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193– 210. https://www.usenix.org/conference/osdi24/presentation/zhongyinmin [65] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 2572, 23 pages. [66] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, ChienYu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2025. NanoFlow: towards optimal large language model serving throughput. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI ’25). USENIX Association, USA, Article 41, 17 pages.
//arxiv.org/abs/2504.20068 [62] Yihao Zhao, Jiadun Chen, Peng Sun, Lei Li, Xuanzhe Liu, and Xin Jin. 2025. SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference. arXiv:2504.15720 [cs.DC] https://arxiv.org/abs/2504.15720 [63] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2000, 27 pages. [64] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and
15