arXiv:2607.02942v1 [cs.DC] 3 Jul 2026
A Workflow-Aware Serving Layer for Agentic Applications Jiayi Qian∗
Zishen Wan∗
Hanchen Yang
Georgia Institute of Technology Atlanta, GA, USA
Columbia University New York, NY, USA
Georgia Institute of Technology Atlanta, GA, USA
Chun Tao
Souvik Kundu
Tushar Krishna
Intel Santa Clara, CA, USA
Intel Los Angeles, CA, USA
Georgia Institute of Technology Atlanta, GA, USA
Abstract
1
Agentic AI applications form an emerging serving workload in which a request creates a workflow: a directed acyclic graph of LLM and tool calls that exposes per-node model choices and optional quality operators such as verifiers. This workload falls between two existing layers. Model-serving engines execute individual calls efficiently but cannot see workflow structure, while agent frameworks fix the workflow but cannot see backend load, so neither jointly chooses each node’s model, verifier, and backend under serving-time conditions. We present Dyserve, a workflow-aware serving layer that fills this gap. Dyserve compiles each workflow’s pernode (model, verifier) choices in one integer linear program (ILP) over a heterogeneous backend pool, priced by skillconditioned offline profiles that transfer across workflows. This couples with hardware entering only through per-model throughput sweeps, and weighted to concentrate strong models and verification on the nodes whose errors propagate the furthest. Because no single latency-quality preference fits every workload mix, Dyserve pre-solves the program at several pressure levels at admission and shifts a workflow’s uncommitted suffix among these strategies under load, keeping the solver off the load-shift path; a failed tool call triggers a one-time residual re-solve that preserves committed work. On LiveCodeBench, GAIA, ComplexFuncBench, and SWE-bench, Dyserve’s compiled strategies achieve the highest accuracy on every workload, 3 to 10 points above the highest-accuracy baseline, at 1.1 to 6.8× lower latency. Under multi-tenant bursts, a balanced compiled plan gains 4.7 accuracy points over round-robin routing at 2.5× lower burst tail latency, and the precomputed ladder restores an oversubscribing plan’s SLO goodput from 18 to 67%, within 6.5 points of the best measured static plan, while leaving stable plans untouched. Event-driven recovery rescues 84% of injected tool failures against 55 for a flat retry, and admission compilation takes under 60 ms per request at the 95th percentile.
Large language models are increasingly applied to complex tasks in code generation, scientific reasoning, and toolaugmented problem solving, yet a single model invocation rarely suffices. Agentic systems address this by orchestrating multiple model and tool calls into a structured computation. These systems follow two broad paradigms. Dynamic agents such as ReAct [42] and AutoGPT [30] construct the next step only after observing the previous result. Predefined agents instead execute an operator graph authored manually [11, 33–35, 41] or generated by an LLM planner [38, 44, 45], run by frameworks such as LangGraph [2], AutoGen [41], and DSPy [16] that resolve branches and unroll bounded iterations into a materialized graph of dependent LLM and tool calls. We focus on this latter setting, where the workflow is known at admission, and treat the workflow rather than the single call as the unit of serving optimization: logical structure is preserved while the physical choices that execute it adapt. Workflow structure makes final accuracy an end-to-end property rather than the property of any one call, so different nodes warrant different serving choices. A difficult synthesis node may require a strong model or verification, whereas a peripheral node whose error has little downstream effect tolerates cheaper execution. The same workflow can therefore reach similar final accuracy through many serving strategies, each assigning a model, backend, and optional verifier to every node. We call this property resource elasticity: one logical workflow exposes strategies spanning an order of magnitude in latency and cost (Section 3). Exploiting this elasticity at scale is a fleet-level scheduling problem. At admission, the system must compile each workflow into a physical plan that balances expected quality against critical-path latency and serving cost. During execution, the assumptions that priced the plan can stop holding (Section 3), and the system must revise the uncommitted work under the assumptions that actually hold, rebalancing quality against fleet-level latency and SLO goodput, the fraction of workflows correct and completed within the latency target.
∗ Corresponding authors: Jiayi Qian ([email protected]) and Zishen Wan ([email protected]).
1
Introduction
User
1. Workflow Synthesis
2. Orchestrator
(AFlow, MaAS, EvoAgentX, FlowReasoner...) Offline/Online search/planning
(AutoGen, MetaGPT, LangGraph, ...) resolves branches; materializes the DAG; submits ready nodes
DAG + metadata
Workflow Program (DAG)
Admission-Time Policy Compilation (Section 5) jointly selects the model and verification policy for each node using a cost-based integer linear program (ILP). Workflow decomposition makes nodes small enough to classify by dominant skill, so an offline profile can directly measure each (model, verification policy, skill class) combination while remaining hardware-independent: a new GPU requires only a per-model throughput sweep. Topology weighting captures downstream reach, a vulnerability penalty prices node failure, and an effort knob scales how strongly quality concentrates on high-reach nodes. Runtime Repair (Section 6) keeps the plan responsive without placing a solver on the load-shift path. On a tool failure, Dyserve solves a residual ILP over the not-yet-executed suffix once, preserving committed work. For load, the pressureaugmented program is pre-solved at several rungs at admission, off the request path, and a hysteresis controller over perclass backend queues swaps the uncommitted suffix among them: a plan that oversubscribes the fleet regains a tuned plan’s tail latency during bursts, while a stable plan is never touched. We implement Dyserve over a shared pool of 4B, 9B, and 27B model backends and evaluate it on LiveCodeBench [13], GAIA [23], ComplexFuncBench [48], and SWE-bench [14]. Dyserve achieves the highest accuracy of any compared system, 3 to 10 points above the highest-accuracy baseline on each workload, at 1.1 to 6.8× lower latency. Under multitenant bursts, a balanced compiled plan improves accuracy by 4.7 points over a round-robin routing baseline at 2.5× lower burst tail latency. The strategy ladder restores an oversubscribing plan’s burst p95 latency from 823 to 136 seconds and its SLO goodput from 18.1 to 67.2%, within 6.5 points of the best measured static plan. Event-driven recovery rescues 84% of injected tool failures against 55 for a flat retry, and the full admission compile takes 56 milliseconds per request at the 95th percentile. Our contributions are:
node results
Dyserve 3a. Admission-Time Policy Compilation WorkflowAware Serving Layer
Topology & Vulnerability
Skill-conditioned Profiles
Quality-Latency Trade-off at Workflow Level
ILP Solver
Effort Knob
4B +Ver.
9B +Ver.
27B
4B
Per-node Serving Plan (model + verifier + backend)
3b. Runtime Repair Load shift → switch among pre-solved strategies (no solver) Tool failure → one-time residual re-solve
4. Tool Call
Committed Prefix (done)
Uncommitted Suffix
5. Model Serving Engine (vLLM, SGLang...)
Web
Code
File
Individual LLM scheduling w/ batching, KV cache management, etc.
Figure 1. Agentic AI serving stack with Dyserve: It lies between the orchestration framework that submits ready workflow nodes, and the model-serving engines. At admission, Dyserve compiles each workflow into a per-node serving strategy; at runtime, it shifts the uncommitted suffix among pre-solved strategies under load and re-solves it once on a tool failure.
No component in the conventional stack owns this decision throughout execution (Figure 1): the orchestrator knows the workflow but not the live cost of the serving fleet, while serving engines receive the workflow as a stream of calls and do not control its optional quality operators. The physical plan that connects the two layers, and its revision under load, falls between them. Existing work supplies pieces of this missing layer: routing chooses a model [4, 25], agent-aware systems such as Murakkab [3] schedule calls using application structure, and verification systems such as Sherlock [27] place quality operators within a workflow (Section 2). None jointly owns the per-node model-verifier pair while adapting the uncommitted workflow to live backend state, and the separation matters because the two choices price each other and a changing workload re-prices both (Section 3). We present Dyserve, a workflow-aware serving layer between the agent orchestrator and a heterogeneous serving fleet (Figure 1): it accepts a logical workflow and the available model backends, compiles a physical serving plan for its nodes, and revises the plan’s uncommitted portion as execution and fleet state evolve, through two mechanisms.
• We identify and build a workflow-aware serving layer that owns the per-node physical plan throughout execution, with orchestrator adapters for Flow and LangGraph (Section 4). • We design Admission-Time Policy Compilation: a cost-based ILP over skill-conditioned, hardwareportable profiles of model-verifier pairs, with topology and vulnerability weighting and an operator-facing effort knob (Section 5). • We design Runtime Repair: a strategy ladder presolved at several pressure rungs and shifted by a hysteresis controller over live per-backend telemetry, plus event-driven suffix re-solving on tool failures (Section 6). 2
Accuracy
0.8
2
Background and Related Work
2.1
The Agent Workflow Serving Stack
0.5
0.71
0.58
0.6 0.4
The stack serving a predefined agent workflow exposes three layers (Figure 1). A workflow synthesis layer, either a developer or a planner such as AFlow [45] or FlowReasoner [9], produces the logical operator graph. An orchestration framework (AutoGen [41], MetaGPT [11], LangGraph [2]) interprets that graph and dispatches ready nodes. A model-serving engine (vLLM [17], SGLang [47]) executes the resulting calls, optimizing batching, KV-cache reuse, and queueing. Once the logical graph is fixed, serving it still requires a physical plan: which model and backend execute each LLM node, which optional verifier follows it, and how those choices change as fleet state evolves. These decisions couple end-to-end accuracy with critical-path latency and serving cost, and they sit above engine-level batching and cache management. As a running example used throughout the paper, consider a coding workflow: a planner decomposes the problem, a file-search tool gathers context, a code generator and a unit-test generator consume the plan, a second tool executes the tests, and an answer node synthesizes the response. Its logical graph fixes the roles and dependencies; its physical decisions remain open: which model serves each node, whether a verifier follows it, and how the plan responds to congestion. 2.2
0.7
0.83
Accuracy Latency Oracle
0.42 4.2s 4B Base
177.8s
414.5s
400 244.1s
0.56
27B Base
200 50
16.9s 4B Verify-all
600
27B Verify-all
Oracle
Latency (s)
0.9
• We evaluate Dyserve on four agentic workloads against routing, uniform-verification, Sherlock-style, and workflowsearch baselines; the burst results replicate across three arrival seeds (Section 7).
10 0
Figure 2. Agent workflows expose serving headroom. On LiveCodeBench (hard), base strategies pin the model with no verification and verify-all strategies attach each node’s accuracy-maximizing verifier for its skill; the oracle selects, for every problem, the best verified outcome among the strategies executed on it. It exceeds 27B verify-all’s accuracy by 12 points at 1.7× lower mean latency (244.1 s vs. 414.5 s). and the model backends, focusing on placement and scheduling rather than per-node verification or measured quality substitutions. Speculative actions [43] overlap likely future actions with verification. These scheduling and speculation mechanisms are complementary to the workflow-quality decision Dyserve makes. Verification methods (Self-Refine [22], Self-Consistency [36], LLM-as-a-Judge [46], Debate [7]) spend additional calls to improve reasoning quality. Sherlock [27], the closest reliability system, estimates node vulnerability and places verification within agentic workflows; the generator and serving conditions stay fixed. Archon [28] searches inference-time generator-verifier compositions for accuracy, offline and per task family; Dyserve makes this choice per request and prices it with serving-time latency and fleet state. None of these systems sees the workflow DAG, controls both the model and verification axes, and remains active as backend state changes; Table 1 summarizes the visibility and control mismatch. Whether this gap matters, and how much, is the subject of the next section.
Existing Systems and the Missing Control Boundary
Workflow synthesis (AFlow [45], MaAS [44], ADAS [12], GPTSwarm [49], ScoreFlow [40], EvoAgentX [38], MASZero [15]) improves the logical workflow against task-level reward, but does not assign it to a live fleet; Dyserve starts after that decision and is complementary. Model routing and cascades (FrugalGPT [4], RouteLLM [25], AutoMix [1], HybridLLM [6], Tabi [37]) select among models per request, and the serving engines optimize batching and memory after a call arrives; neither owns the cross-node physical plan or the substitution between model strength and verification. Agent-aware schedulers (Llumnix [31], Parrot [19], Autellix [21]) migrate, prefix-share, or reorder calls using application structure, and Ayo [32] optimizes an application’s dataflow execution end to end; all leave each call’s model and verification fixed by the caller. Murakkab [3], the closest serving system, optimizes workflow deployment and the mapping of components to models and hardware, but the verification choice stays fixed by the application. Orla [29] also argues for a dedicated layer between the orchestrator
3
Motivation
Section 2 identifies a missing owner for the serving strategy, but a new systems layer is justified only if the missing decision is valuable and cannot be recovered by composing single-axis mechanisms, whose choices are interdependent: verification changes the value of a model choice, workflow position changes the value of both, and load changes the cost of each. We therefore measure the available serving headroom, the coupling between the model and verification choices, the variation in node impact, and the stability of admission-time plans; the answers turn this gap into concrete requirements for Dyserve. Agent workflows expose substantial serving headroom. A predefined workflow fixes the logical computation, 3
Layer
State visible
Decisions owned
Missing for the physical plan
Workflow synthesis Orchestrator Serving engine
task reward, candidate operators graph, dependencies, node outputs queues, KV cache, device state
graph, prompts, logical operators live fleet state and execution-time events node readiness and application control flow joint model, verifier, and backend allocation call scheduling and backend-local execution workflow semantics and optional quality work
Table 1. The physical serving plan falls across existing ownership boundaries. No existing layer sees both workflow-level quality choices and live fleet state while controlling the remaining workflow. Dyserve fills this gap without taking over the serving engine’s batching, KV-cache, or device-level execution.
but it does not fix the physical serving strategy. Every LLM node can be served by a different model and optionally followed by a verifier, producing many strategies with the same workflow semantics and different accuracy-latency points. This is the systems opportunity behind resource elasticity: a workflow is not merely a chain of calls to execute efficiently, but a graph whose optional quality work can be spent in different places. This flexibility also creates a combinatorial planning problem beyond hand-tuning. With 𝐾 LLM nodes, 𝑀 candidate models, and 𝑉 verification policies, a workflow exposes up to (𝑀𝑉 ) 𝐾 assignments; the running example’s four LLM decision nodes and our three-model, eight-policy pool already expose 244 ≈ 3.3 × 105 , and the choice repeats for each arriving workflow. We quantify this headroom with a measured per-problem oracle: for every problem in the LiveCodeBench set, we select in hindsight the best verified outcome among the strategies executed on it (Figure 2). Accuracy and latency are measured workflow outcomes, not profile estimates, and the oracle bounds selection only within the executed candidate set rather than serving as an online baseline. Fixed strategies span 0.42 at 4.2 seconds to 0.71 at 414.5 seconds; the oracle reaches 0.83 at 1.7× lower mean latency than the strongest fixed plan, and no single fixed strategy attains it: the best plan varies across requests. There is a large gap between treating the workflow as a fixed recipe and serving it as a configurable plan.
where the small model is already near its ceiling. A sequential policy, routing first and placing verifiers second, has already discarded the relevant joint points; the serving strategy must choose model and verifier jointly at each node. In the running example, joint selection can protect the planner with a strong model, verify a 9B code generator, and leave the tool-checked test generator unverified. Topology alone does not determine how much a node matters. Joint selection still leaves a placement question: the value of improving a node is not its local accuracy gain but the effect of that gain on the final answer, and topology captures only part of that effect. An error at an early planning or synthesis node can contaminate every dependent node, whereas an isolated leaf has no downstream consumers; yet structural reach is only an upper bound on impact. Downstream computation may absorb an error, and the node’s required skill affects whether its output can be reconstructed, checked, or safely consumed. Nodes with the same topological role can therefore have sharply different failure consequences. We quantify this distinction with counterfactual fault injection. For each cell defined by topological role, required skill, and fan-in, we perturb held-out nodes and compare the terminal workflow outcome with an unperturbed execution, restricted to pairs whose baseline workflow is correct; vulnerability 𝑣 is the resulting drop in terminal correctness (𝑣=0.54: a 54-point drop). Figure 3 shows that topological role alone is not a sufficient vulnerability proxy. Among mid-workflow nodes with fan-in one or two, reasoning drops terminal correctness by 0.39, compared with 0.18 for tool use. The contrast is larger among leaves: math reaches 0.54, while code is only 0.02. Even at the root, reasoning and code differ by more than 6× (0.13 versus 0.02). Dyserve therefore separates structural influence from measured vulnerability and conditions the latter on both workflow role and semantic node type.
The model and verification axes are coupled. The headroom above cannot be closed by adding a model router and a verifier placer as independent components. Both decisions buy the same local property, namely a lower chance that a node emits an error that propagates downstream, but with different resource profiles: a larger model improves the base answer but charges every generated token, while a verifier leaves the base model unchanged and pays only for the checking and repair calls. The two are substitutable, and the value of either depends on whether the other already provides sufficient reliability. In Figure 2, 4B with verification matches unverified 27B. The size of the substitution also varies by the skill required at the node: in our skill profiles, the best verifier lifts the 4B model by 25 points on reasoning probes but under 5 on code,
The admission pricing stops holding. Even a workflowaware assignment can be priced under assumptions that stop holding, and three mismatches arise in practice. The first is node-level events. Node risk is only estimated at admission from offline profiles and request features; during execution, a tool timeout or malformed tool output reveals that a particular node, and the suffix depending on it, is riskier than 4
Math
Leaf | Math Leaf | Reasoning Leaf | Reasoning 0.02 Leaf | Code Mid | Reasoning Mid | Tool-use Root | Reasoning 0.02 Root | Code 0.0
Reasoning
Tool-use
0.19 0.18
Code
fan-in 0
fan-in 1-2
0.24
Compile
0.13
0.2
0.4
(skill profiles), (throughput), (topology, vuln.)
rungs
Runtime Execution and Adaptation (§ ) load shift Runtime uncommitted Execution committed suffix tool failure
Figure 3. Node vulnerability depends on more than topology. Each point is the terminal-correctness drop after fault injection, conditioned on a correct unperturbed execution; color denotes required skill, marker shape fan-in. Cells within the same topological role differ sharply across semantic node types.
swap suffix residual re-solve
Figure 4. The life of one request inside Dyserve. The orchestrator’s materialized DAG is wrapped into the workflow abstraction (circles are LLM nodes with candidate sets, squares are fixed tool calls); admission compiles it once into a strategy 𝜋 plus strategies pre-solved at 𝐾 pressure rungs; execution then adapts only the uncommitted suffix. A load shift installs a pre-solved rung by pointer swap, with no solver on that path; a tool failure triggers a one-time residual re-solve. Figure 1 shows where Dyserve sits in the stack; this figure shows what happens to one request.
the profile predicted. The second is workload drift. The preference weights that place a compiled plan on its frontier are calibrated against a workload mix: in our measurements, the same plan that sustains a burst p95 of 135 seconds on one trace mix degrades to 292 seconds when the mix shifts toward long code tasks, with the plan unchanged (Section 7.4). The third is preference fragility. A quality-leaning weight setting concentrates load on the strongest models and leaves no slack, so it is structurally fragile under bursts: the same fleet that holds a 135-second burst tail under a balanced setting collapses past 800 seconds under the quality-leaning one (Section 7.4). These signals arrive while the workflow is executing. If the 27B queue congests after the planner commits, moving a remaining node to 9B may require adding verification to preserve quality: the serving engine sees the queue but not this workflow-level trade-off, and the orchestrator sees the suffix but not its serving cost. Adaptation therefore belongs in the missing layer, revising only the uncommitted portion of the workflow and preserving work already performed. Together, these observations refine the gap in Section 2 into three requirements. R1: Joint compilation of a pernode (model, verifier) assignment: neither axis is meaningful in isolation. R2: Workflow-aware allocation of quality work by structure and required skill: error impact varies sharply across nodes. R3: Runtime adaptation of only the uncommitted work: node-level events and calibration drift make the admission plan stale. The next section presents Dyserve, the serving layer that implements these requirements.
4
metadata
Admission-Time Policy Compilation (§ ) joint ILP over strategy
0.6
, skill
tool nodes: fixed call
DAG
0.39
Vulnerability / correctness drop
LLM nodes:
Workload Wrap
0.54
per-node metadata, and activation events from the orchestrator; its lower interface receives a catalog of compatible model backends and consumes the service rates and load signals they export. From these inputs, Dyserve produces a per-node physical serving plan. The ownership boundary is strict: the orchestrator retains prompts, operator semantics, and control flow, while each engine retains batching, caching, and queue discipline. Figure 4 shows the resulting flow: admission compiles the workflow into a serving strategy (Section 5), the runtime dispatches it while monitoring load signals and execution events, and adaptation touches only the uncommitted suffix (Section 6). 4.1
Workflow Abstraction
Dyserve represents each materialized request as a directed acyclic graph 𝐺 = (𝑉 , 𝐸) over operator nodes. Applicationlevel branch predicates and loops remain opaque: the orchestrator resolves the selected branch and unrolls bounded iterations before compilation, so each planning instance is a finite concrete DAG. Dyserve neither invents operators nor changes dependencies; dynamic graph expansion starts a new compilation instance and is outside the current design. Each node 𝑛 ∈ 𝑉 is either an LLM node or a tool-call node. An LLM node exposes an execution-option set O𝑛 of (model, backend) candidates and a verification-policy set P𝑛 , and carries a dominant skill class 𝑠𝑛 ∈ {code, math, reasoning, tool_use}, the last tagging LLM nodes whose input is freshly retrieved external context (profile handling in Section 5.1). A tool-call node has the single execution option of its tool invocation and no verification axis; it contributes a fixed per-tool duration estimate to the critical path. The abstraction wraps any
The Dyserve Layer
Dyserve is a workflow-aware serving layer placed between the agent orchestrator and a shared fleet of model backends (Figure 1). Its upper interface receives a logical workflow, 5
DAG the orchestrator emits: a previously unseen workflow is priced by tagging its nodes, with no re-profiling and no per-application training, and supporting a new orchestrator is an adapter (Section 6.4 describes ours for Flow [24] and LangGraph [2]). 4.2
where 𝑚(𝑜) is the option’s model, 𝑞b is the measured absolute skill score, b 𝑟 is the measured probability of an incorrect or unusable local result, and b 𝜅 records the call structure and token demand. For tool_use nodes, which have no dedicated probes, the lookup substitutes reasoning; vulnerability stays conditioned on the tool-use class. The coefficient builder composes each pair’s quality from these measurements: the option’s absolute base quality plus the skill-conditioned verifier lift, taken at the pool’s strongest model. Two approximations keep profiling tractable: a node’s candidate models share that lift, and two nodes with the same dominant skill reuse the same measurements, which makes profiling independent of the number of workflow templates; the profiles are built once from held-out skill probes covering all eight policies of Table 5 (datasets in Section C). Output-token demand, not wall-clock latency. Observed latencies would bind the profile to one fleet. The profiler instead records the output-token demand of every call, including the base call, organized into the serial stages S𝑝 of policy 𝑝, with 𝑔𝑎,𝑗 the output tokens of call 𝑗 in stage 𝑎. For option 𝑜 = (𝑚, 𝑑), this device-independent demand converts into nominal service time through model 𝑚’s decode throughput on device 𝑑 at a calibrated operating point 𝑏 0 : ∑︁ 𝑔𝑎,𝑗 (𝑠𝑛 , 𝑚, 𝑝) 𝜏 (𝑠𝑛 , 𝑜, 𝑝) = max + 𝜏pre, (2) 𝑗 ∈ C𝑎 𝜃 dec (𝑚, 𝑑, 𝑏 0 )
Policy Axes
Each LLM node exposes two admission-time axes: an execution option selecting the model and backend for the base call, and a verification policy deciding how to validate or improve its output. Dyserve optimizes the two jointly because they are substitutable routes to node quality at different serving costs (Section 3). Eight verification policies are exposed at every applicable node, spanning self refinement, 𝑘-sample self-consistency, judge gates, and multi-round debate; each is described by a call structure, serial stages of parallel sub-calls that determine how token cost and wall latency aggregate (Table 5, Section A). Treating a policy as one black-box cost would lose the per-stage accounting the cost model needs (Section 5.1). Runtime retry. Retry is deliberately not a third admissiontime axis: whether a node needs retrying is unknown until its tool invocation fails. Dyserve treats it as a runtime action with a bounded attempt budget (Section 6.3).
𝑎∈ S𝑝
4.3
Layer Responsibilities
where C𝑎 holds stage 𝑎’s parallel calls, 𝜏pre is an aggregate prefill correction, and gated policies record expected demand over measured gate rates: each stage charges its slowest member, the sum preserves serial dependencies, and monetary cost sums all calls. Runtime pressure is excluded from 𝜏 and enters only through the pressure rungs of Section 6.2; adding a GPU requires only a per-model throughput sweep, never a re-run of the model-verifier-skill grid.
A serving strategy assigns every active LLM node one execution option and one verification policy. Owning it throughout execution requires three capabilities: compile the abstraction, profiles, and backend service rates into one joint assignment; execute and observe, dispatching expanded call structures and watching outputs, failures, and backend pressure; and adapt, revising only the uncommitted suffix. Section 5 presents the compiler and Section 6 the runtime.
5.2
5
Admission-Time Policy Compilation
The coefficient builder specializes the profile to the request. Quality at node 𝑛 is weighted by downstream reach: with 𝑅(𝑛) the set of nodes reachable from 𝑛, including 𝑛 itself, ( | −1 1 + 𝛽𝑒 · |𝑅|𝑉(𝑛)| −1 , |𝑉 | > 1, 𝜌 (𝑛) = (3) 1, |𝑉 | = 1,
At admission, Dyserve compiles the workflow abstraction into a serving strategy: it estimates candidate choices from reusable skill-conditioned profiles (Section 5.1), specializes them to the request’s topology and backend service rates, and selects all per-node choices jointly in one integer linear program (ILP) (Section 5.2). 5.1
Workflow-Aware Compilation
where 𝛽 bounds the topology emphasis and 𝑒 ∈ [0, 1] is the operator-facing effort knob: 𝑒 controls where quality concentrates, whereas 𝜆ℓ controls how much latency the operator pays overall (Section 7.4 examines mispairing). The knob carries no optimality guarantee; sweeping 𝑒 traces a one-dimensional family of operating points (Section 7.5). Structural reach alone does not determine impact: the vulnerability coefficient 𝑣𝑛 comes from the counterfactual fault injection of Section 3 (following Sherlock [27]), conditioned on role, skill, and fan-in (Figure 3); the risk of a concrete pair is 𝑓𝑛,𝑜,𝑝 = 𝑣𝑛 b 𝑟𝑛 (𝑜, 𝑝), which falls when a stronger model or an
Skill-Conditioned Profiling
The space of possible workflows is open-ended, so quality and cost cannot be assigned by a lookup on the whole graph. The node is the unit of estimation: for an execution option 𝑜 ∈ O𝑛 and a verification policy 𝑝 ∈ P𝑛 , the offline profile Φ supplies quality, residual-error, call-demand, and cost estimates (b 𝑞𝑛 (𝑜, 𝑝),b 𝑟𝑛 (𝑜, 𝑝), b 𝜅𝑛 (𝑜, 𝑝), b 𝑐𝑛 (𝑜, 𝑝)) = Φ(𝑠𝑛 , 𝑚(𝑜), 𝑝), (1) 6
values predict better outcomes. The full pseudocode is listed in Section B. The ILP scales linearly with graph edges and candidate pairs; the evaluated workflows (up to 11 nodes, three models, eight policies) stay below 270 joint-choice variables. Across the evaluated workflows the full compile, coefficient construction included, takes 27 milliseconds at the median and 56 at the 95th percentile, and each ladder rung adds one more solve of 12 to 15 milliseconds, paid off the request path and cached per unique workflow (Section 6.2). Figure 5 shows the compiled strategy for the running example; no model-first or verifier-first pass reliably recovers this combination of per-node substitutions.
Example Coding task code reasoning
tool call
Plan
File Search
+27B model no verifier
Code Gen
+9B model +Verif. tool call
code
Run Tests
Test Gen
+27B model
reasoning Answer +4B model +Verif.
Figure 5. The compiled strategy for the running coding example. The high-reach planner gets the strong model without a verifier; the code generator substitutes a 9B model plus a verifier for a stronger model; the test generator takes the strong model unverified, since the test-execution tool already exercises its output; the answer node takes the cheapest model with a light check. No single global routing or verification rule recovers this plan.
6
Runtime Execution and Adaptation
Admission prices the strategy from static profiles, but concurrent workflows change backend pressure and execution reveals failures. Dyserve therefore adapts only uncommitted work: load moves a request among strategies pre-solved at admission (Section 6.2); a tool failure triggers a one-time residual re-solve (Section 6.3).
effective verifier cuts the residual error. The two terms play different roles, 𝜌 (𝑛) scaling delivered quality and 𝑣𝑛 pricing failure, and Section 7.5 ablates them separately. Backend specialization combines the profiled demand with the selected backend’s nominal service rate. Transient queue pressure never enters this solve, so a plan is never re-priced because another request happens to be in flight; pressure instead selects among pre-solved strategies at runtime (Section 6.2). The program selects the option and policy together, since the measured coefficients belong to the (𝑜, 𝑝) pair. For each node it chooses exactly one pair 𝑦𝑛,𝑜,𝑝 ∈ {0, 1}, and continuous start times 𝑎𝑛 and makespan 𝑇 schedule the critical path: ∑︁ ∑︁ 𝑦𝑛,𝑜,𝑝 = 1 ∀ 𝑛 ∈ 𝑉 ,
6.1
Execution and Runtime Events
Let 𝜋𝑡 (𝑛) = (𝑜, 𝑝) denote the choice currently assigned to node 𝑛. The orchestrator tracks dependencies and submits ready nodes, possibly concurrently; on submission, Dyserve reads 𝜋𝑡 (𝑛), dispatches the base call to the backend selected by 𝑜, and expands 𝑝 into its call structure (Section 4.2). A decision becomes committed when its call is dispatched: Dyserve never migrates or cancels an in-progress call, and strategy changes take effect only at node boundaries, where alreadydispatched work keeps its choice and not-yet-started nodes read the latest strategy. This boundary is the runtime’s entire consistency rule. Two signals can invalidate the uncommitted remainder. Per-replica telemetry (queue depth, active requests, latency) is aggregated into a pressure gauge per model class whose only consumer is the ladder controller below; no load statistic feeds an online solve. And tool-call nodes are issued as structured invocations, so a timeout, malformed argument, or explicit error is classified and activates bounded recovery (Section 6.3).
𝑜 ∈ O𝑛 𝑝 ∈ P𝑛
𝑎𝑛 ≥ 𝑎𝑢 + 𝑑 (𝑢) ∀ (𝑢, 𝑛) ∈ 𝐸, 𝑇 ≥ 𝑎𝑛 + 𝑑 (𝑛), (4) Í with 𝑑 (𝑛) = 𝑜,𝑝 𝜏𝑛,𝑜,𝑝 𝑦𝑛,𝑜,𝑝 the selected duration. Latency thus enters through the longest finish-time path, never a flat sum. The objective maximizes topology-weighted quality against cost, risk, and the critical path: ∑︁ ∑︁ ∑︁ max 𝜌 (𝑛)𝑞𝑛,𝑜,𝑝 − 𝜆c𝑐𝑛,𝑜,𝑝 − 𝜆f 𝑓𝑛,𝑜,𝑝 𝑦𝑛,𝑜,𝑝 𝑝 𝑛∈𝑉 𝑜 (5) − 𝜆ℓ 𝑇 .
6.2
Load Adaptation: the Precomputed Strategy Ladder
At an adaptation point, let 𝐶𝑡 be the committed nodes and 𝑈𝑡 = 𝑉 \ 𝐶𝑡 those whose choices may still change. The exact load-aware repair re-solves the admission objective 𝐽0 of Equation (5), restricted to 𝑈𝑡 , with a demand charge at pressure level 𝐿 ≥ 0: ∑︁ ∑︁ 𝐽 (𝐿) = 𝐽0 𝑈𝑡 − 𝜆ld 𝐿 𝜏𝑛,𝑜,𝑝 𝑦𝑛,𝑜,𝑝 , (6)
Because every (𝑜, 𝑝) pair carries its own composed quality, the objective can prefer a small verified model at one node and a strong unverified model at another. The weights are selected on held-out workflows and reused across experiments; the program trades quality through weights rather than a hard floor, and realized accuracy accompanies every efficiency result. The objective is a ranking surrogate, not a claim that end-to-end success decomposes into independent node accuracies; Section 7 validates that higher surrogate
𝑛∈𝑈𝑡 𝑜,𝑝
under Equation (4) on the residual makespan, with 𝜆ld selected with the other weights (expanded form in Section E). 7
up-shift
: gauge
: preferred (27B-heavy)
next class free
threshold, so an oversubscribing plan regains a tuned plan’s tail during saturation while a stable plan is left as the operator priced it (Section 7.4).
: shed (9B-heavy)
: mixed
6.3
flip-back: jump to lowest drained rung (dwell) install pointer swap,no solver
committed
Recovery and Multi-Tenancy
Recovery. On a tool failure, Dyserve classifies the error and locates the anchor LLM node, the failed call’s immediate predecessor or the earlier node that selected the failing argument, and re-executes it with the error as diagnostic feedback under bounded escalation: one step up the model-size ladder, then the highest-quality verifier, then a re-run of the maxed choice. A valid replacement is committed and the residual program is re-solved once (Section E); an exhausted budget returns the failure to the orchestrator. Multi-tenancy. Across tenants, replicas of a model are exposed as separate backends; Dyserve spreads requests round-robin, pinning each request’s node chain to one replica for prefix-cache locality. Workflows are optimized independently but coupled through the queues they create. A joint ILP over all in-flight workflows would grow with concurrency and need rebuilding on every arrival; shared per-class telemetry couples workflows instead, and each adaptation stays local to one suffix.
uncommitted suffix
Figure 6. The strategy ladder. The pressure-augmented program is pre-solved over the full graph at 𝐾 rungs at admission; pressure triggers stepwise up-shifts and a direct flipback to the lowest drained rung. A shift installs the selected rung’s restriction to the uncommitted suffix; committed work never moves, no solver runs on this path. At 𝐿 = 0 this is the admission program restricted to the suffix; as 𝐿 grows the solution sheds toward lower-demand pairs and lighter verification: 𝜆ℓ prices the request’s own critical path, while 𝐿 prices the aggregate demand that saturates a fleet, so higher rungs are demand-shedding fallbacks. Why pressure is pre-solved. An earlier design ran this solve online at every load event; measurement rejected it: calm-period p95 rose by half to double from solver interference alone, and the online path never flipped back (Section 7.6). Dyserve therefore pre-solves Equation (6) over the full graph at a bounded set of pressure levels at admission; a load shift only installs a pre-solved restriction, and tool failure is the one event that still triggers an exact residual solve (Section 6.3). The strategy ladder. At admission, the compiler is invoked once per rung 𝐿 ∈ {0 = 𝐿0 < · · · < 𝐿𝐾 −1 }, yielding pre-solved strategies 𝜋 (0) , . . . , 𝜋 (𝐾 −1) over the full graph (Figure 6), with 𝜋 (0) the admission plan itself, cached per unique workflow and built off the request path. A shift to rung 𝑘 atomically installs 𝜋𝑡 +1 = 𝜋𝑡 |𝐶𝑡 ∪ 𝜋 (𝑘 ) 𝑈𝑡 by replacing the strategy reference. Installing a full-graph solve’s restriction is not identical to re-solving the residual graph at rung 𝑘; the ladder trades that per-shift optimality for a shift path with no solver on it. Hysteresis rung control. A fleet-level selector maintains the active rung from the per-class gauges, using each rung’s primary class, the class receiving the largest share of its planned demand, annotated at pre-solve time; the up-shift, flip-back, and dwell conditions are as in Figure 6 (flip-back jumps directly to the lowest drained rung, since stepwise descent stalls when adjacent rungs share a class). An arriving request starts on the rung the selector currently designates, the admission strategy whenever the fleet is calm: pressure selects among the pre-solved strategies and never re-prices one; an in-flight request adopts a new rung only at its next node boundary. The ladder does not aim to beat a static plan whose weights match the workload: the controller stays silent there, firing only when a queue crosses its saturation
6.4
Implementation
Dyserve is about 10K lines of Python plus a 4.5K-line profiler built on the lm-eval harness [10]; the compiler uses PuLP with CBC [8], and each request’s state lives in context-local storage so concurrent requests share one process. Because everything above the orchestrator boundary consumes the workflow abstraction of Section 4.1, supporting an orchestrator is an adapter, not a fork. We implement two: one for Flow [24] and one for LangGraph [2], the latter in 2.1K lines with no change to the compiler or the runtime. A LangGraph application annotates each node with a decorator carrying its kind, skill, and verifier candidates; the adapter reads the unmodified StateGraph into the abstraction, and the application swaps LangGraph’s executor for Dyserve’s runner at a single call site. Inside a node, one helper returns an LLM client already pointed at the ILP-selected backend and wrapped with the selected verification policy, and reinvocations during recovery carry the diagnostic feedback automatically. The contract is the one Section 4.1 already states: nodes are registered and pure (recovery re-executes the anchor LLM node, never a tool), and conditional branches resolve before compilation; the adapter rejects unresolved branching with an explicit error rather than miscompiling. The integration is covered by 33 adapter tests and an end-toend demo.
7
Evaluation
Our evaluation asks whether Dyserve improves the end-toend quality-latency frontier, whether its plans hold up under 8
steady multi-tenant load, whether the runtime layer protects mispriced plans and recovers failed tools, which parts of the formulation produce the gains, and what the control path costs. We isolate admission-time policy quality before steadystate serving and adaptation: compilation should produce an efficient plan without load information, and the runtime layer matters only when its assumptions fail. 7.1
Experimental Setup
Workloads. We evaluate Dyserve on four workloads spanning distinct agentic skills and workflow structures: LiveCodeBench (LCB) [13] for code generation, GAIA [23] for multi-step reasoning and tool use, ComplexFuncBench (CFB) [48] for function calling, and SWE-bench [14] for repository-level software engineering. We use 55 (LCB), 35 (GAIA), 100 (CFB), and 40 (SWE-bench) instances and report each benchmark’s official task metric. A workflow is successful when it satisfies the benchmark-specific correctness predicate; timed-out and incomplete requests are counted as incorrect. For latency, we measure wall-clock time from workflow admission until the final output, including queueing, model calls, verification, tool execution, and retry. We additionally report total model service time where it helps distinguish computation saved from queueing avoided; on a self-hosted fleet, monetary cost tracks service time, so we do not report it separately. Models, verifiers, and hardware. The serving fleet contains 4B, 9B, and 27B model backends on NVIDIA H200 GPUs. Each applicable workflow node exposes the verification policies described in Section 4.2. Dyserve profiles model-verifier pairs by skill and predicts their latency from output-token demand and the measured throughput of the target GPU (Section 5.1). Unless stated otherwise, the quality and vulnerability profiles are collected on a disjoint calibration set, and all reported requests are held out from profiling.
and identify it separately from the original trained system. Murakkab and Orla are not apples-to-apples baselines: neither selects the per-node model-verifier pair, which is the decision under study, so the single-axis ILPs of the ablation study (Section 7.5) serve as the mechanism-level comparison. All systems use the same task inputs, model endpoints, timeout, and, where applicable, verifier implementations. Dyserve and the serving baselines execute the same materialized workflow; MAS-Zero is a broader end-to-end baseline that may synthesize a different workflow. Multi-tenant methodology. Serving experiments replay balanced mixed traces drawn from LCB, GAIA, and CFB (SWE-bench is excluded: repository setup and tool execution dominate GPU serving time) under open-loop Poisson arrivals, with deadlines generated before execution so overload is not hidden by coordinated omission; every policy sees the same arrival sequence. We run one trace per operating point and report request-level 95% bootstrap confidence intervals; the headline burst comparison is additionally replicated on three independent arrival seeds. Serving fleets are four-GPU compositions of 27B and 9B replicas, written (𝑎+𝑏) for 𝑎 27B and 𝑏 9B backends; the 4B model participates in the singlerequest admission experiments. Unless otherwise noted, the headline fleet is (2+2) and the secondary fleet (3+1), and the base and Sherlock-style baselines run on the homogeneous (4+0) fleet. Metrics. Our primary quality metric is the official benchmark score, and our primary runtime metric is normalized SLO goodput (goodput hereafter): the fraction of requests arriving in a window that are both correct and complete within an end-to-end latency target. Requests are assigned to windows by arrival time, so a calm-window request can still queue behind pulse backlog. Goodput is evaluated on a deadline grid from 60 to 300 seconds; 𝐷 = 180 is a representative cut selected from the original balanced-trace operating points (Section 7.4 summarizes the grid). Uncertainty on goodput and accuracy is reported as 95% bootstrap confidence intervals over the requests of a trace, cross-checked with a 30-second block bootstrap that preserves queue correlation and, since every policy replays the same arrival sequence, with paired sign-flip permutation tests over perrequest outcomes. All three are within-trace analyses; three independent arrival seeds replicate the burst comparison, with per-arm overall accuracy varying by at most 2.3 points across seeds and the ladder’s burst-tail advantage over base and over the unprotected plan reproducing on every seed. The balanced trace gives each workload equal arrival mass, and we report overall accuracy across the mix.
Baselines. We compare against the following alternatives. Fixed-Model assigns every node to one model without verification; Verify-All uses a fixed model and attaches the strongest profiled verifier to every applicable node; both execute on the 27B model, and the single-request figure (Figure 7) labels Fixed-Model as base, distinct from the serving sections’ base routing baseline. MAS-Zero searches for a workflow-level agent design, but does not optimize its physical serving plan; it natively supports LiveCodeBench and SWE-bench, and its generated designs do not support the tool interfaces of GAIA and ComplexFuncBench. On SWE-bench its searches exceed the 4000-second per-task budget, so it is reported as not available there. Our base baseline routes every call round-robin across all replicas of its fleet, blind to model class. Our Sherlock-style baseline follows Sherlock’s topology-aware verifier placement and utility ordering with the same available models and verifiers as Dyserve; because the released system does not include the trained selector for our workloads, we use a training-free approximation
7.2
End-to-End Quality and Latency
Figure 7 compares end-to-end accuracy and wall-clock latency on all four workloads. Base execution misses difficult nodes; Verify-All spends verification where it does not pay; 9
0.0
4.4s Base
0.32 9s
40
8.6s
N/A V-All SHER. MAS-Z Ours
0.6 0.4
20
0.2
0
0.0
0.67
45.1s 19.3s
24.9s
6.8s
Base
N/A V-All SHER. MAS-Z Ours
SHERLOCK 1.0
50 40 30 20 10 0
0.8 0.6
0.56
0.4 177.8s 0.2 0.0
MAS-Zero
LiveCodeBench
Base
0.71 0.71 513s
Ours 0.81 600
0.61
414.5s
269s
400
252.1s
200 V-All SHER. MAS-Z Ours
0
0.6 0.5 0.4 0.3 0.2 0.1 0.0
SWE-Bench 807s 0.45
0.50 1000 800
0.32 0.47 221s
600
288s 400
295s N/A
Base
V-All SHER. MAS-Z Ours
200
Latency (s)
0.2
0.43
0.63 0.64 0.64
Verify-all
Latency (s) Accuracy
0.34
Latency (s) Accuracy
Accuracy
0.4
0.8
0.51 60
58.6s
0.6
Base
ComplexFuncBench
Latency (s) Accuracy
Star: accuracy
GAIA
0
Figure 7. End-to-end accuracy and latency on the four workloads. Bars report mean wall-clock latency (right axis), stars the official task accuracy (left axis). Dyserve has the highest accuracy on every workload at substantially lower latency than uniform verification and, where supported, MAS-Zero, which lacks the tool interfaces of GAIA and ComplexFuncBench and exceeds the per-task budget on SWE-bench. offered load 1 req/s
78
76
74
Better!
72
Accuracy (%)
Accuracy (%)
76
placement raises verification work without pricing its latency and pays up to 291 seconds. Second, on these fleets heterogeneity is only profitable with compilation. Replacing a 27B replica with a 9B replica makes base worse on both axes (127/196 vs 93/172 seconds), because one quarter of its calls land on a weaker model regardless of the node. The compiled plan on the same fleet is the best configuration measured (56/86 seconds at higher accuracy): it sends lowvulnerability nodes to the 9B replica and reserves the 27B replicas for the nodes that decide the answer. In a sweep on the heterogeneous fleet under the same sustained loads, the latency weight behaves as a preference rather than a parameter with one correct value: 𝜆ℓ = 0.05 saturates the fleet (883 to 1179 seconds p95), 𝜆ℓ = 0.1 sits at the figure’s operating point, and 𝜆ℓ = 0.2 pays 3 to 8 accuracy points for its shorter tail; which is right depends on the operator and, as the next section shows, the workload mix.
offered load 2.5 req/s
78
74 72
70
70
68
68
0
50
100 150 P95 Latency (s)
200
Better! 0
50
base Sherlock base (3 + 1) Dyserve (3 + 1) Dyserve (2 + 2) Dyserve (4 + 0)
100 150 200 P95 Latency (s)
300
Figure 8. Steady-state multi-tenant serving at 1.0 and 2.5 requests/s (mixed trace, overall accuracy; up and left is better). Base and Sherlock run on the (4+0) fleet. Heterogeneity moves the compiled policy left at equal or better accuracy, while it makes base worse on both axes; Sherlock-style placement pays for unpriced verification in tail latency. Sherlock-style placement cannot substitute model strength for verification strength; MAS-Zero improves the logical workflow but not the physical plan, reaching 0.61 in 513 s on LiveCodeBench and exceeding the 4000-second per-task budget on SWE-bench. Dyserve achieves the highest accuracy on every workload, 3 to 10 points above the highest-accuracy baseline at 1.1 to 6.8× lower latency. On GAIA it lifts Verify-All’s 0.43 to 0.51 while cutting latency from 58.6 to 8.6 s; on LiveCodeBench it reaches 0.81 in 252.1 s against 0.71 in 269 s for the strongest Sherlock-style operating point; ComplexFuncBench and SWE-bench follow the same pattern (Figure 7). Selective model-verifier assignments avoid uniform verification cost while preserving strong choices at the nodes that decide the answer. 7.3
7.4
Serving under Steady Multi-Tenant Load
Figure 8 compares base, Sherlock-style placement, and Dyserve’s compilation on three four-GPU fleets at offered loads of 1.0 and 2.5 requests/s; every workflow keeps its initial plan, and all arms use round-robin replica routing. Two effects stand out. First, on the homogeneous fleet, compilation matches or beats base on accuracy while cutting the loaded tail (112 vs 172 seconds at 2.5/s); Sherlock-style 10
Bursts, Mispriced Preferences, and the Strategy Ladder
Steady-state load does not stress the admission plan; we therefore drive each fleet with a double-pulse burst: two oneminute pulses raise the arrival rate from a calm 0.3 requests/s to 4.0 on the headline fleet (13×) and 2.5 on the secondary (8×). We compare base, the quality-leaning admission plan (𝜆ℓ = 0.05; admission only in Table 2), its strategy ladder (rungs 𝐿 ∈ {0, 1.5, 3}), and the best measured static preference from the same sweep (the balanced plan, 𝜆ℓ = 0.1). Table 2 isolates what the runtime layer adds, with burst goodput at the headline deadline. Tuned plans degrade gracefully; a mispriced plan collapses, and the ladder restores it. The balanced plan degrades gracefully (burst p95 135 seconds against base’s 336, with base 4.7 points lower in overall accuracy; the secondary fleet reproduces the margin), and the online re-solver of Section 7.6 only hurts it: a correctly tuned plan needs no help. The quality-leaning plan instead concentrates load on the 27B replicas and saturates: its burst p95 reaches 823 seconds. The ladder, starting from the same preference, shifts
Rung
Rolling p95 (s, log)
pulse window
admission only
+ ladder
1000
+ ladder 136 s best static 135 s
500 200
best static
plan’s 75.4 ([70.8, 80.1]), about eight points, while deadline attainment rises from 23.1 to 96.8 percent. Goodput nets the two: 18.1 percent ([14.3, 22.2]) unprotected against the ladder’s 67.2 ([62.0, 72.2]), non-overlapping under the block bootstrap ([9.5, 25.7] against [64.7, 72.2]) and concordant with the within-trace paired permutation test (+49.1 points, 𝑝 < 10−4 ), within 6.5 points of the tuned static’s 73.7. In this trace the ladder also achieves the highest calm-period goodput of any arm (74.1 percent against the tuned static’s 70.7) while retaining its burst goodput, an operating point none of the measured static settings provides; the shift audit shows how, with calm periods served at the quality preference and pulses on the latency-leaning rungs. The rescue is not an artifact of the 180-second cut: across the 60 to 300 second deadline grid, the ladder’s burst goodput exceeds the unprotected plan’s at every deadline, by 27 to 60 points here, 11 to 33 on the code-heavy mix, and within 1.5 points on the fleet the pulses do not saturate. The guardrail fires at the saturation threshold. On the secondary fleet the quality-leaning plan degrades (229 seconds against the balanced 72) but never collapses, and the controller stays silent: zero shifts, with paired permutation tests detecting no difference against either neighbor (𝑝 ≥ 0.26; Table 2, middle block). The guardrail holds on a code-heavy mix. The bottom block of Table 2 repeats the comparison on a trace mix shifted toward long code tasks. The balanced setting remains the best static, so the optimum itself does not move; what fails is the calibrated performance: the same balanced plan’s burst p95 degrades from 135 to 292 seconds with the plan unchanged, and the quality-leaning plan no longer delivers even the accuracy it was priced for (66.0 against the balanced plan’s 68.0) because its requests die in queue. The ladder, with gentle rungs (𝐿 ∈ {0, 0.5, 1}), reaches 57.3 percent burst goodput, within 3.5 points of the best static, with two shifts. Across the three settings the ladder lands within 3.5 to 6.5 points of the best measured static without moving the preference weights, while the unprotected preference falls to 18 to 36 percent wherever the fleet saturates; the one remaining configuration choice is the rung set (aggressive on the original mix, gentle on the code-heavy one), and both sets rescue their saturating plan. We run a second event experiment for tool failure: the first call of each eligible tool site is forced to fail (49 sites across the frozen GAIA workflows, one per trial). A flat retry repeats the same action; the recovery ladder re-executes the anchor node with the error as feedback under escalation. The ladder recovers 84% of the forced failures against 55% for flat retry and none without retry, at 37% additional mean latency; end-to-end correctness rises from 14% to 20%, and the gain concentrates where flat retry fails (web-fetch: 76% against 14%).
823.8 s first shift 137.8 s
100 50 2 1 0
flip-back 13 shifts total 2 0 0
60
120
180
Elapsed time (s)
240
300
340
Figure 9. The strategy ladder under the double-pulse burst on the (2+2) fleet. Admission only executes the compiled quality plan unchanged; + ladder is that plan with pre-solved rungs; best static is the static sweep’s best preference on this trace (𝜆ℓ =0.1). Top: rolling p95 over 60-second arrival windows (log scale); annotations: burst-window p95. Bottom: the fleet’s active rung, with rung 0 the preferred 𝜋 (0) . The first shift lands inside the first pulse; after each drain the controller flips directly back (13 shifts). The trace’s 400 arrivals end at 339 seconds, truncating the second pulse window. Acc. (%)
p95 (s) Goodput (%)
mixed trace, (2+2) base admission only + ladder
70.8 336 75.8 823 69.2 136 ( − 687)
mixed trace, (3+1) base admission only + ladder
72.5 76.2 75.0
25.7 18.1 67.2 (+49.1)
201 229 185
62.9 71.0 72.6
code-heavy mix, (2+2) admission only 66.0 599 + ladder† 64.3 313 ( − 286)
35.7 57.3 (+21.6)
Table 2. Runtime-layer ablation under the double-pulse burst (pulses at [120, 180] and [300, 360] seconds; peak 4.0/s on (2+2) and 2.5/s on (3+1)). base routes round-robin over the block’s fleet without compilation; admission only executes the compiled plan unchanged; + ladder adds the presolved rungs. Accuracy is overall; p95 and goodput are burstwindow values at the 180-second deadline; parentheses give the change against admission only. The pulses saturate (2+2) but not (3+1), where the controller never fires and the differences are within run noise. † Gentle rung set 𝐿 ∈ {0, 0.5, 1}.
to cheaper rungs during the pulses and returns afterward (Figure 9 audits the shifts), holding the burst tail to 136 seconds, matching the correctly tuned plan’s 135. The insurance has a bounded cost paid during the pulses, where the ladder operates at the latency-leaning point: pulse-period accuracy is 67.8 percent ([62.9, 72.8]) against the unprotected 11
Configuration
Accuracy Service time (s)
Joint axes and topology (two-model exhaustive setup) model-only ILP 0.62 verifier-only ILP (27B pinned) 0.72 verifier-only ILP (4B pinned) 0.56 uniform topology weights 0.64 Dyserve (joint, topology-weighted) 0.74
35.4 195.5 12.2 37.1 37.2
Current-pool ablations no vulnerability term skill-collapsed profile Dyserve (full)
247.6 242.8 252.1
0.729 0.760 0.815
Operation
p95
Admission compilation (full) 27 ms 56 ms Per-rung re-solve 12 ms 15 ms Coefficient construction 0.2 ms 0.2 ms Profile load (once per process) 10 ms
Table 4. Dyserve control-path overhead over the 92 frozen evaluation workflows. Admission compilation is end to end: template conversion, coefficient construction, and the CBC solve; a ladder rung repeats only the solve.
Table 3. Ablating the admission formulation on LiveCodeBench. Upper block: the two-model exhaustive setup (every per-node assignment of a ten-problem subset executed), latency as total model service time; the model-only ILP fixes verification to the per-skill best policy. Lower block: the current pool at the canonical operating point (𝑁 =54), mean wall-clock latency; the full row matches Figure 7.
7.5
p50
on the operating point: topology weighting limits the collapse’s damage at the table’s anchor, the same collapse leaves per-node choices identical at the topology-dominated GAIA anchor, and ComplexFuncBench’s canonical plan is already homogeneous. The latency model is validated at the granularity the compiler uses it, ranking strategies for one workflow: predicted durations rank measured latencies with a median per-workflow Spearman correlation of 0.67, rising to 90% pairwise agreement for strategies separated by at least 3× (full breakdown in Section D). The same semantic profile has served three accelerator types (H200 here, GH200 and L40S in development) with only per-device throughput sweeps.
Understanding Dyserve’s Gains
Joint optimization. Table 3 first separates the model and verification axes on an exhaustive setup that crosses 4B and 27B with the profiled verifiers at every decision node of a ten-problem LiveCodeBench subset and executes every assignment end to end. Model-only optimization loses 12 points at matched service time (0.62 against 0.74 near 36 seconds); verifier-only optimization holds 0.72 under the pinned strong model but spends 5.3× the joint plan’s service time, and caps at 0.56 under the pinned small one. The joint formulation reaches 0.74 at 37.2 seconds: the two axes must be compiled together, not applied as independent passes.
Sensitivity. The latency weight is swept 4× throughout (Section 7.4), and sweeping the effort knob 𝑒 from 0 to 1 at the canonical anchor moves accuracy from 0.71 into a 0.76 to 0.81 band as mean latency grows from 167 to 244 seconds: the operating point is not a brittle single setting. Limitations. Our serving measurements cover four-GPU single-node fleets and one model family; the mechanisms act per model class, but larger and mixed fleets are unmeasured. Dyserve plans materialized DAGs, so dynamic graph expansion starts a new compilation instance, and plan quality rests on the dominant-skill tags and the residual-makespan approximation of Section E.
Workflow structure and vulnerability. Replacing topology weights with uniform node weights drops 10 points at the same service time (0.64 against 0.74): quality work lands on locally difficult nodes whose outputs barely affect the final answer. Removing the vulnerability term costs 8.6 points at matched wall clock (0.729 against 0.815): risk-blind choices spend the same budget on configurations whose failures propagate (Section 3). The exhaustive block also validates the score as a selector: every strategy there is measured, the full objective’s argmax recovers the highest accuracy in the bounded set, and each ablated objective selects a measurably worse strategy; the score is a ranking surrogate, not a calibrated probability of success.
7.6
Control-Path Overhead
Table 4 reports control-path overheads over the 92 frozen workflows (up to 11 nodes): the full compile stays under 60 ms at p95, each rung repeats only the 12 to 15 ms solve off the request path, a shift is a pointer swap, and the tool-failure residual re-solve is no larger than the admission program. The pre-solved design is grounded in measuring its online predecessor, which re-solved the residual program in the serving process at every load event. With routing shares unchanged, its calm-period p95 was 445 and 338 seconds against 221 for the identical static plan, pure interference between solver and serving loop, and after the pulses it settled at 685 and 715 seconds against 184 because the offload never flipped back. Pre-solving removes both by construction.
Skill-conditioned and portable profiles. Collapsing all nodes into a global model-verifier average costs 5.5 points at the operating point of Table 3 (0.760 against 0.815) and 12.7 points at a profile-bound point with no topology weighting (0.600 against 0.727), where the collapsed profile makes 9B look adequate everywhere. Which term binds depends 12
8
Conclusion
[8] Iain Dunning, Stuart Mitchell, and Michael O’Sullivan. 2011. PuLP: A Linear Programming Toolkit for Python. Technical Report. Department of Engineering Science, The University of Auckland. https: //optimization-online.org/2011/09/3178/ [9] Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, and Tianyu Pang. 2025. FlowReasoner: Reinforcing Query-Level Meta-Agents. arXiv:2504.15257 [cs.AI] https: //arxiv.org/abs/2504.15257 [10] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. The Language Model Evaluation Harness. doi:10.5281/zenodo.12608602 [11] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A MultiAgent Collaborative Framework. In International Conference on Learning Representations. https://openreview.net/forum?id=VtmBAGCN7o [12] Shengran Hu, Cong Lu, and Jeff Clune. 2025. Automated Design of Agentic Systems. arXiv:2408.08435 [cs.AI] https://arxiv.org/abs/2408. 08435 [13] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https://arxiv.org/abs/2403.07974 [14] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In International Conference on Learning Representations. https://openreview.net/forum? id=VTF8yNQM66 [15] Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Ryan Chin, Caiming Xiong, and Shafiq Joty. 2026. MAS-ZERO: Designing MultiAgent Systems with Zero Supervision. arXiv:2505.14996 [cs.CL] https: //arxiv.org/abs/2505.14996 [16] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714 [cs.CL] https://arxiv.org/abs/2310.03714 [17] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, 611–626. doi:10.1145/3600006.3613165 [18] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s Verify Step by Step. arXiv preprint arXiv:2305.20050 (2023). [19] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLMbased Applications with Semantic Variable. arXiv:2405.19888 [cs.LG] https://arxiv.org/abs/2405.19888 [20] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv.org/abs/2305.01210 [21] Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E.
Dyserve treats the agent serving strategy as a structured optimization variable: per-node (model, verifier) selection over a workflow abstraction is compiled by a profile-driven ILP, and a runtime layer swaps the uncommitted suffix among strategies pre-solved at admission under load and re-solves it once on node-level failures. Across four agentic workloads, the compiled strategies improve accuracy by 3 to 10 points over the highest-accuracy baseline at 1.1 to 6.8× lower latency; the heterogeneous fleet becomes beneficial only under compilation; and under bursts, a tuned plan degrades gracefully while the ladder returns an oversubscribing plan to within 6.5 points of the best measured static plan’s goodput. These results follow from three choices: compiling modelverifier substitutions jointly, modeling verification as explicit call stages, and removing the solver from the load-shift path through pre-solved pressure rungs.
Acknowledgments This work was supported in part by CoCoSys, one of seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.
References [1] Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam. 2025. AutoMix: Automatically Mixing Language Models. arXiv:2310.12963 [cs.CL] https://arxiv.org/abs/ 2310.12963 [2] LangChain AI. 2024. LangGraph: Building stateful, multi-actor applications with LLMs. https://github.com/langchain-ai/langgraph. [3] Gohar Irfan Chaudhry, Esha Choukse, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Adam Belay, and Ricardo Bianchini. 2025. Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms. arXiv:2508.18298 [cs.MA] doi:10.48550/arXiv.2508.18298 [4] Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research (2024). https: //openreview.net/forum?id=cSimKw5p6R [5] Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. 2026. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs. (2026). arXiv:2605.00674 [cs.CL] https://arxiv.org/abs/ 2605.00674 [6] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. arXiv:2404.14618 [cs.LG] https://arxiv.org/abs/2404. 14618 [7] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 11733–11763. https://proceedings.mlr. press/v235/du24e.html 13
arXiv:2510.05077 (2025). [36] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. SelfConsistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations. https: //openreview.net/forum?id=1PL1NIMMrw [37] Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An Efficient Multi-Level Inference System for Large Language Models. In Proceedings of the Eighteenth European Conference on Computer Systems. Association for Computing Machinery, 233–248. doi:10.1145/ 3552326.3587438 [38] Yingxu Wang, Siwei Liu, Jinyuan Fang, and Zaiqiao Meng. 2025. EvoAgentX: An Automated Framework for Evolving Agentic Workflows. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Ivan Habernal, Peter Schulam, and Jörg Tiedemann (Eds.). Association for Computational Linguistics, Suzhou, China, 643–655. doi:10.18653/v1/2025.emnlpdemos.47 [39] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro: A More Robust and Challenging MultiTask Language Understanding Benchmark. arXiv:2406.01574 [cs.CL] https://arxiv.org/abs/2406.01574 [40] Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, and Bryon Aragam. 2025. ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization. arXiv:2502.04306 [cs.CL] https://arxiv.org/ abs/2502.04306 [41] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Ahmed Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. In Conference on Language Modeling. https://www.microsoft.com/en-us/research/publication/autogenenabling-next-gen-llm-applications-via-multi-agent-conversationframework/ [42] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X [43] Naimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu, Kostis Kaffes, and Tianyi Peng. 2025. Speculative Actions: A Lossless Framework for Faster Agentic Systems. arXiv:2510.04371 [cs.AI] https://arxiv.org/abs/2510.04371 [44] Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. 2025. Multi-agent Architecture Search via Agentic Supernet. In International Conference on Machine Learning. https: //openreview.net/forum?id=imcyVlzpXh [45] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. 2025. AFlow: Automating Agentic Workflow Generation. In International Conference on Learning Representations. https://openreview.net/ forum?id=z5uVAKwmjf [46] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36 (2023), 46595– 46623. [47] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104 [cs.AI] https://arxiv.org/abs/2312.07104
Gonzalez, and Ion Stoica. 2025. Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv:2502.13965 [cs.LG] https: //arxiv.org/abs/2502.13965 [22] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems, Vol. 36. 46534–46594. [23] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A Benchmark for General AI Assistants. In International Conference on Learning Representations. https://openreview.net/forum?id=fibxvahvs3 [24] Boye Niu, Yiliao Song, Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, and Tongliang Liu. 2025. Flow: Modularized Agentic Workflow Automation. In International Conference on Learning Representations. https://openreview.net/forum?id=sLKDbuyq99 [25] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to Route LLMs from Preference Data. In International Conference on Learning Representations. https://openreview. net/forum?id=8sSqNntaMr [26] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022 [cs.AI] https://arxiv.org/abs/2311.12022 [27] Yeonju Ro, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Aditya Akella, Zhangyang Wang, Mattan Erez, and Esha Choukse. 2025. Sherlock: Reliable and Efficient Agentic Workflow Execution. arXiv:2511.00330 [cs.MA] doi:10.48550/arXiv.2511.00330 [28] Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, et al. 2024. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254 (2024). [29] Rana Shahout, Hayder Tirmazi, Minlan Yu, and Michael Mitzenmacher. 2026. Orla: A Library for Serving LLM-Based Multi-Agent Systems. arXiv preprint arXiv:2603.13605 (2026). [30] Significant Gravitas. 2023. AutoGPT: An Autonomous GPT-4 Experiment. https://github.com/Significant-Gravitas/AutoGPT. Software. [31] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. arXiv:2406.03243 [cs.AR] https://arxiv.org/ abs/2406.03243 [32] Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards End-toEnd Optimization of LLM-based Applications with Ayo. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’25). ACM, 1302–1316. doi:10.1145/3676641.3716278 [33] Zishen Wan, Che-Kai Liu, Jiayi Qian, Hanchen Yang, Arijit Raychowdhury, and Tushar Krishna. 2026. Reason: Accelerating probabilistic logical reasoning for scalable neuro-symbolic intelligence. arXiv preprint arXiv:2601.20784 (2026). [34] Zishen Wan, Hanchen Yang, Jiayi Qian, Ritik Raj, Joongun Park, Chenyu Wang, Arijit Raychowdhury, and Tushar Krishna. 2025. Compositional AI Beyond LLMs: System Implications of Neuro-SymbolicProbabilistic Architectures. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Pittsburgh, PA, USA) (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 67–84. doi:10.1145/3760250.3762235 [35] Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie, Tushar Krishna, Vijay Janapa Reddi, and Yilun Du. 2025. Slm-mux: Orchestrating small language models for reasoning. arXiv preprint 14
Policy
Behavior
Calls
none self_refine self_refine_iter advanced_refine self_consistency judge_gate gated_refine debate
pass-through feedback → rewrite multi-round, halt on STOP: parallel critique + rewrite 𝑘 samples + vote pointwise verdict only gate; escalate on REJECT opponent/proponent + select
1 2 up to 𝐾 𝑘 par. + 1 𝑘 par. + 1 1 1 or 3 2𝐾+1
C
The skill profiles are built from held-out probe datasets: HumanEval+ and MBPP+ [20] for code; MATH-500 [18], AIME24 [5], and AIME25 [5] for math; and GPQA-Diamond [26] and MMLU-Pro [39] for reasoning. These instantiate reusable skill probes rather than application-specific profiles; a new workflow joins the profile through each LLM node’s dominant skill tag, with no re-profiling.
D
Table 5. Verification policies and total call structures, including the base call. The profiler separately records the base call and each verification stage so wall latency follows serial/parallel structure while token cost sums all calls.
E
Verification Policy Call Structures
Residual Formulation
At an adaptation point with committed nodes 𝐶𝑡 and uncommitted nodes 𝑈𝑡 = 𝑉 \ 𝐶𝑡 , Dyserve constructs the residual workflow 𝐺𝑡 = 𝐺 [𝑈𝑡 ]. A committed node may be complete or still in flight; either way its choice is fixed. Incoming edges from completed nodes are satisfied, while in-flight nodes remain fixed boundary dependencies whose completion the orchestrator enforces; the residual makespan spans only undispatched work, so waiting on a still-executing committed predecessor is enforced at dispatch rather than modeled, which can understate the suffix’s completion time when a long in-flight branch gates it. The repaired strategy is 𝜋𝑡 +1 = 𝜋𝑡 |𝐶𝑡 ∪ Compile 𝐺𝑡 , Φ, 𝜃, 𝐿 , (7)
Table 5 lists the eight verification policies exposed at every applicable LLM node with their call structures. Each policy is a sequence of serial stages of parallel sub-calls; wall latency follows the serial/parallel structure (Equation (2)) while token cost sums all calls. Treating a policy as a single black-box cost would overstate the wall latency of parallel policies such as self-consistency by up to the stage fan-out, so the profiler tags each sub-call and keeps per-stage statistics.
B
Latency-Ranking Validation
Across the 55 frozen LiveCodeBench workflows (a median of 16 distinct executed strategies each), predicted durations rank measured wall-clock latencies with a median per-workflow Spearman correlation of 0.67. Pairwise ordering accuracy grows with separation: 75% over all strategy pairs, 86% for pairs whose measured latencies differ by at least 2×, and 90% at 3×; errors concentrate among near-ties. Absolute wall-clock additionally reflects output-length variance and queueing that the nominal operating point excludes, which is why the compiler consumes rankings rather than absolute predictions.
[48] Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. 2025. ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario. arXiv:2501.10132 [cs.CL] https://arxiv.org/abs/2501.10132 [49] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Language Agents as Optimizable Graphs. arXiv:2402.16823 [cs.AI] https://arxiv.org/ abs/2402.16823
A
Profiling Probe Datasets
Compilation Pseudocode
Algorithm 1 Admission-time policy compilation for one workflow. Require: 𝐺=(𝑉 , 𝐸); skill profile Φ; backend throughputs 𝜃 ; request features; weights 𝜆ℓ , 𝜆c, 𝜆f , 𝛽; effort 𝑒 Ensure: Serving strategy 𝜋 1: for 𝑛 ∈ 𝑉 , 𝑜 ∈ O𝑛 , 𝑝 ∈ P𝑛 do 2: (𝑞, 𝜏, 𝑐, 𝑓 )𝑛,𝑜,𝑝 ← Coef(𝑠𝑛 , 𝑜, 𝑝, Φ, 𝜃, 𝐺, request) 3: end for 4: 𝜌 ← Topology(𝐺, 𝛽𝑒) 5: Build constraints (4) 6: Solve objective (5) with PuLP [8] 7: return per-node (𝑜, 𝑝) assignments 𝜋
where the second term solves Equation (6) on 𝐺𝑡 . Define the per-choice runtime score 𝑠𝑛,𝑜,𝑝 (𝐿) = 𝜌 (𝑛)𝑞𝑛,𝑜,𝑝 − 𝜆c𝑐𝑛,𝑜,𝑝 − 𝜆f 𝑓𝑛,𝑜,𝑝 − 𝜆ld 𝐿 𝜏𝑛,𝑜,𝑝 . The expanded residual objective, with 𝑛 ∈ 𝑈𝑡 , is then ∑︁ 𝐽 (𝐿) = 𝑠𝑛,𝑜,𝑝 (𝐿)𝑦𝑛,𝑜,𝑝 − 𝜆ℓ𝑇𝑡 . 𝑛,𝑜,𝑝
The merge rejects any overlap between the two sets rather than silently overwriting an executed choice. The tool-failure path solves this program once per event; the load path never solves it online, installing pre-solved rung restrictions instead (Section 6.2).
The strategy ladder of Section 6.2 repeats the solve of line 6 once per pressure rung, with the demand charge of Equation (6) added to the objective; the residual re-solve of Section E runs the full procedure on the residual graph. 15