VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
arXiv:2605.06068v1 [cs.AI] 7 May 2026
Keisuke Kamahori∗ University of Washington
Shihang Li∗ University of Washington
Simon Peter University of Washington
Baris Kasikci University of Washington
Abstract For years, we have built LLM serving systems like any other critical infrastructure: a single general-purpose stack, hand-tuned over many engineer-years, meant to support every model and workload. In this paper, we take the opposite bet: a multiagent loop that automatically synthesizes bespoke serving systems for different usage scenarios. We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end. VibeServe uses an outer loop to plan and track the search over system designs, and an inner loop to implement candidates, check correctness, and measure performance on the target benchmark. In the standard deployment setting, where existing stacks are highly optimized, VibeServe remains competitive with vLLM, showing that generation-time specialization need not come at the cost of performance. More interestingly, in non-standard scenarios, VibeServe outperforms existing systems by exploiting opportunities that generic systems miss in six scenarios involving non-standard model architectures, workload knowledge, and hardware-specific optimizations. Together, these results suggest a different point in the design space for infrastructure software: generationtime specialization rather than runtime generality. Code is available at https: //github.com/uw-syfi/vibe-serve.
1
Introduction
LLM serving systems are critical software infrastructure for an economy increasingly dependent on generative AI. Open-source stacks such as vLLM [36], SGLang [80], and TensorRT-LLM [52] provide efficient abstractions across a broad range of models and hardware. Yet their designs are shaped primarily by mainstream deployments, such as decoder-only Transformers on NVIDIA GPUs serving generic chat workloads. As a result, emerging model families (e.g., multimodal models or hybrid state-space architectures), along with new hardware accelerators and atypical workloads, often suffer from suboptimal performance or even require substantial new implementation effort [31, 77, 64, 28, 17, 44]. As the space of model–hardware–workload combinations continues to expand, a one-size-fits-all serving stack is becoming increasingly difficult to sustain. In this work, we explore a different point in the design space: rather than maintaining a single generalpurpose runtime, can we generate a bespoke serving system for each combination of model, hardware, and workload? Per-deployment specialization is a longstanding idea in computer systems [46, 12, 48, 8, 49, 47], but it rarely pays off in practice since per-target engineering cost dwarfs the gain in most cases. However, coding agents are changing this calculus: their demonstrated effectiveness on individual components [56, 68, 51, 61, 22] and system policies [81] suggests that per-target ∗ Equal contribution.
Preprint.
WORKLOAD + MODEL
Generic serving today
VibeServe's approach
VibeServe
Generic runtimes cover common cases.
One bespoke serving system per (workload, model, hardware) target.
outer loop
Workload A Model α
Workload B Workload C Workload D Model β
Model γ
Model δ
Workload E
A
B C
E
A C
E
B D
E
Model ε
α
β
ε
α
ε
β
ε
γ
γ
δ
inner loop
HARDWARE
HW X
HW Y
HW Z
HW X
HW Y
(E,ε,Z)
(D,δ,Z)
(B,β,Z)
(E,ε,Y)
(C,γ,Y)
(A,α,Y)
(E,ε,X)
(C,γ,X)
Generic Framework
(B,β,X)
(A,α,X)
Bespoke Serving Systems SERVING SYSTEM
generates
HW Z
Figure 1: Motivation for VibeServe. General-purpose serving frameworks target common deployments; VibeServe instead generates systems specialized to each model–hardware–workload target.
specialization could now be feasible at scales where engineering costs were previously prohibitive (Figure 1). Generating an end-to-end serving system, however, is a long-horizon, multi-component task that existing agentic optimization does not address: prior systems operate on a much smaller code surface, e.g., a single GPU kernel, an isolated algorithm, or a single policy embedded in an otherwise fixed system [56, 68, 51, 61, 42, 22, 32, 81]. Designing and optimizing an end-to-end system exceeds the context window of any single agent. The standard recourse, compaction [5, 38], induces drift in both performance and correctness [6, 71, 43, 11]. Evolutionary frameworks sidestep this drift via a population of scored programs [51, 61, 42], but a scalar score cannot encode the planning state an end-to-end system needs. Multi-agent loops carry richer state across roles but do not reset agent context windows [22, 32], inheriting limitations from compaction. Long-horizon harnesses of coding agents sustain state across sessions but produce incorrect systems that underperform state-of-the-art baselines [9, 41]. We present VibeServe, a multi-agent system that synthesizes bespoke LLM serving runtimes from scratch. To let agents target the open-ended space of model–hardware–workload deployments, VibeServe exposes two extensible surfaces: a small set of user-provided artifacts (model and reference implementation, accuracy checker, workload benchmark, and target hardware), and an Agent Skills library [7] of serving-systems knowledge distilled from existing engines. New model families, hardware platforms, and optimization techniques enter as new skill entries, so coverage extends beyond the combinations supported by hand-engineered code paths in existing runtimes. For each target, VibeServe factors the work along two axes. An outer loop plans across iterations based on git-recorded optimization history, picking the next optimization and dispatching one concrete task to the inner loop. Its planning state is structured and persistent (e.g., issues, a long-term memory file, the commit history), which is richer than a scalar score and not confined to a single agent’s context, enabling the separation of design failures from implementation flaws. An inner loop executes each task through a coding-agent harness. Implementer, Accuracy Judge, and Performance Evaluator agents take turns in fresh contexts, working over a read-only reference implementation and checker. The outer loop only considers correct implementations: performance naturally varies as agents explore different design choices, but incorrect candidates cannot derail subsequent rounds. §3 gives more details of the design. We evaluate VibeServe across six scenarios (§4). On a standard setting (Llama-3.1-8B-Instruct [19] on H100), VibeServe reaches near-parity with vLLM [36] and SGLang [80], confirming the agentic pipeline can match a hand-tuned baseline on a mainstream scenario. More importantly, VibeServe works effectively in cases where generic systems fall short, which we validate by targeting nonstandard workload patterns (e.g., aggressive speculative decoding for code editing with predicted output, workload-aware prompt cache design), model architectures (e.g., hybrid attention models, multimodal models with complex architecture), or hardware backends (e.g., MacBook). These specialized systems reach 5.95× speedup for predicted-output code editing, 3.45× throughput for hybrid prompt caching, 1.69× lower latency for streaming speech recognition, 2.6× speedup for 2
MacBook JSON decoding, and 6.27× speedup for multimodal model inference on MacBook and 21.4% on H100. In summary, we contribute the following: 1. We make the case that per-target bespoke LLM serving is now feasible given long-horizon coding agents. 2. We build VibeServe, a multi-agent loop with an outer planner and an inner Implementer/Judge/Evaluator that synthesizes complete serving runtimes against a target-agnostic interface. 3. We demonstrate vLLM parity on a standard deployment and concrete wins across six non-standard scenarios spanning workload, model architecture, and hardware.
2
Motivation
Why LLM serving needs bespoke systems. Modern LLM serving stacks [36, 80, 52] achieve strong performance across many models and hardware platforms through various optimization techniques [78, 36, 76, 10]. However, use cases are diversifying rapidly: new model architectures, hardware accelerators, and application interfaces introduce execution structures that challenge runtime abstractions designed for the standard case. This creates persistent long-tail scenarios where a generalpurpose stack may work suboptimally, miss optimizations that a bespoke system could implement, or be unable to run the workload at all. In other words, generic abstractions impose a portability tax on non-standard models, hardware, and applications [12, 49]. Building bespoke systems can solve this problem. For example, knowing workload characteristics at design time can enable optimizations that a workload-agnostic runtime cannot safely assume. RAG-like applications with long shared prefixes can amortize prefill through prompt caching [16, 30], while aggressive speculative decoding based on predicted outputs is possible for some applications like code editing [54, 13, 74, 66]. Similarly, tailoring for a particular model architecture can expose state and execution patterns that fall outside standard decoder-only assumptions. As an example, hybrid state-space/attention models require cache-management strategies different from those used for decoder-only Transformers [40, 53, 57, 65, 79]. Many modern multimodal models also have complex architectures that require significant serving-system effort, such as modality-specific scheduling, memory management, and cross-component execution [31, 77]. Finally, knowing the target hardware can inform the right runtime design: Apple Silicon, for example, exposes a unified memory model that differs from that of CUDA-centric serving stacks [23, 26]. This makes the missed opportunity fundamentally system-level. Exploiting deployment-specific structure often requires coordinated decisions across GPU kernel implementation, memory management, request scheduling, and the external interface. Optimizing only one component is insufficient if the rest of the runtime continues to enforce the generic execution contract. A bespoke serving system, in contrast, can make the deployment contract explicit. Such systems can specialize entire layers to target scenarios, rather than preserving compatibility with unrelated deployments. Why bespoke systems are possible now. Computer systems have long explored specialization as a way to remove abstraction overhead, from extensible operating systems and code specialization to unikernels [12, 48, 8, 49, 46, 47]. This idea is attractive for LLM serving as well, but historically impractical since the engineering cost of building and maintaining a new runtime for every model– hardware–workload combination would normally dominate the performance gains. However, recent coding agents suggest a different cost model. They are increasingly effective at writing software, including real-world bug fixes and performance-critical GPU kernels or scheduling policies [29, 71, 11, 56, 68, 51, 61, 22, 81]. Still, end-to-end system generation remains much more challenging because it requires complex, long-horizon reasoning over a large codebase and coordination across multiple components at all layers [35, 63, 11, 67, 6, 41, 9, 32]. We argue that LLM serving can be the first domain in which agents successfully generate useful systems end-to-end, since there is a broad need for specialization, the optimization objective is concrete and numeric (e.g., throughput or time-to-first-token latency), and correctness can be checked against a reference implementation. This motivates VibeServe, which generates a serving system specialized to a given deployment target. 3
Figure 2: VibeServe architecture. User-provided artifacts define a target deployment. The outer loop plans over validated git checkpoints and dispatches a single-round task to the inner loop, where an Implementer, Accuracy Judge, and Performance Evaluator collaborate on a shared workspace using the execution environment and a skills library of serving-systems knowledge.
3
Design
VibeServe generates a serving system specialized to a user-specified model, hardware platform, and workload, rather than relying on general-purpose runtimes to cover every case. Figure 2 shows the overall architecture: an outer planning loop and an inner implementation loop iteratively produce an end-to-end serving system from a small set of user-provided artifacts. The framework itself is target-agnostic, and specialization enters through three surfaces: per-target inputs (§3.1) that define the model, hardware, and workload; an agentic pipeline (§3.3) that creates and optimizes a bespoke LLM serving system specialized to the target; and an extensible skills library (§3.4) through which agents learn about new model families, hardware platforms, and system optimization techniques. 3.1
Inputs
VibeServe takes a small set of user-provided artifacts that define the target deployment. First, the user provides the model weights and a reference implementation, such as a Hugging Face Transformers [69] model. The reference implementation is assumed to be accurate but not efficient. Second, the user provides an accuracy-checking script that compares a candidate serving system against the reference implementation. In this paper, we treat the user-provided checker as the source of truth for correctness. Completely verifying the semantic accuracy of serving systems is an open research problem beyond our scope [24, 18], but our setting mirrors human-engineered systems, where continuous integration tests serve as the executable correctness gate. Third, the user provides a benchmark script that exercises the target workload and emits the numerical metric to optimize, such as latency, throughput, or time-to-first-token. Finally, the user provides natural-language instructions describing the high-level target, including the hardware platform and any expected shape of the deliverable system, such as an HTTP API or benchmark harness interface. Together, these inputs form the per-target contract that parameterizes the rest of the framework: every subsequent design choice is grounded in the model, hardware, and workload they specify. 3.2
Workspace
Each candidate runs in an isolated workspace that mounts the user-provided artifacts read-only and exposes the target execution environment (a local or cloud GPU) along with platform-specific profilers (Nsight Systems and the PyTorch profiler on NVIDIA). Agents can only edit the serving-system code 4
they generate, and read-only mounts prevent the Implementer from bypassing this by editing the checker or reference implementation. 3.3
Multi-agent pipeline
The pipeline factors the problem along two axes (Figure 2). Across rounds, an outer loop plans what to optimize next over validated git checkpoints, dispatching a single task per round to an inner loop. Within a round, the inner loop employs multiple agents to separate the code edit proposal from validation. VibeServe wraps existing coding-agent harnesses with three pieces of shared infrastructure: a Model Context Protocol (MCP) [3] server whose schema is defined by the outer-loop policy and through which inner-loop agents return structured information back to the policy, a skills library of operational knowledge loaded into the agent context, and an execution environment that issues build, run, and measure calls. Outer loop. The outer loop’s search policy is modular, exposing a single per-round operation: it reads prior state, hands the inner loop a starting commit and a task, and receives the resulting commit with the performance metric. Two shared mechanisms support coordination beyond this contract. First, every accepted build is a git commit, so any policy can revert cheaply when a later round passes correctness but regresses on the headline metric. Second, inner-loop agents need a structured channel back to the policy during execution. Each policy defines its own MCP server schema, and inner-loop agents return information to the policy by calling the MCP tools the policy exposes. We implement three policies: evolutionary search [51], the Ralph loop [27], and the issue-tracker policy used in our evaluation (§4). The issue-tracker policy maintains a backlog of structured issues, using the MCP server tool interface to define and enforce the issue schema. Inner-loop agents file issues over the predefined contract, and the Orchestrator agent picks the next issue to dispatch at each round, optionally requesting to revert to an earlier checkpoint, and updates a long-term memory of optimization directions. The memory is maintained as a markdown file that the Orchestrator reads on entry and edits at the end of each round. The selected issue, including its acceptance criteria, is the contract handed to the inner loop. The long-term memory allows the Orchestrator to distinguish implementation failures from evidence that a direction is unsuitable for the workload; a failed attempt may signal that the implementation needs debugging or a narrower scope, rather than that the technique should be discarded. Inner loop. In the inner loop, Implementer, Accuracy Judge, and Performance Evaluator agents work in sequence, revising the codebase until it passes correctness checks. This separation keeps implementation, correctness, and performance reasoning in independent contexts: a combined agent can weaken its correctness criteria to land a hard optimization, while the Judge inspects diffs and runtime behavior with a fresh context, and the Evaluator runs only after correctness is gated. Each agent is implemented by a coding-agent harness, such as Codex CLI, Claude Code, or DeepAgents, that can read and edit files, run commands, and return structured results [55, 4, 37]. The Implementer produces and revises the candidate serving system in the workspace. It receives the task and pass criteria from the outer loop along with pointers to the reference implementation and model weights, and consults the serving-systems skills library (§3.4). The Accuracy Judge gates overall correctness of the model implementation. This includes end-toend model accuracy, the correctness of the Implementer’s per-round changes, and the absence of reward-hacking patterns that exploit the test setup rather than improve the model. For accuracy, it runs the user-provided accuracy checker against the candidate server. For the changes, it verifies any per-round pass criteria from the outer loop (for the issue-tracker policy, the issue’s acceptance criteria). For reward hacking, it inspects the candidate’s source and runtime behavior for common patterns, including schema-only synthesis, prompt-keyed completion caches, constant templates, and fast paths that bypass model inference. If any of these fail, the Judge returns actionable feedback to the Implementer, and the inner loop iterates. If the Implementer fails to produce a passing build within a retry budget, the round fails, and control returns to the outer-loop Orchestrator. Once an implementation clears the Judge, the Performance Evaluator profiles it and generates performance hints for subsequent rounds. It starts with end-to-end performance on the user-provided benchmark, then drills down with the platform-specific profilers from the workspace when finer measurements are needed, drawing on the skills library (§3.4) for profiler-specific guidance. For 5
targeted investigations, the Evaluator can insert temporary instrumentation around specific code blocks or commit microbenchmarks for repeated measurement; the inner loop then returns the headline metric, trace analysis, hints, and feedback to the outer loop. 3.4
Skills library
VibeServe provides agents with a serving-systems skills library in the Agent Skills format [7]. The skills are created from the source code of mature serving engines and the surrounding research literature, organized along the abstraction layers an engineer works through when building a serving engine: model architectures, serving algorithms, programming frameworks, backend libraries, hardware platforms, reference engines, and tooling. This lets an agent retrieve focused guidance for a task, such as how continuous batching changes scheduler state, how to use FlashInfer or FlashAttention without reimplementing kernels [76, 10], how MLX differs from PyTorch on Apple Silicon, or where a mechanism lives in vLLM, SGLang, or TensorRT-LLM. The library is also an extensibility surface. New model families, hardware platforms, frameworks, backend libraries, or reference engines can be added as new skill entries under the corresponding layer. Because these axes interact, algorithm skills include compatibility notes that connect the technique to supported backends, hardware, and engines; for example, the continuous-batching skill records which paged-KV implementations are available on which hardware backends. The library stops at the serving-system boundary: agents use existing kernel libraries and serving abstractions, while custom CUDA, Triton, or CUTLASS kernel authoring is delegated to GPU-kernel skills. Providing reference-engine skills is not meant to hide the task by asking agents to copy an existing system: agents may inspect existing implementations, just as human systems engineers do, but the target deployments require specialization to the given model, workload, and hardware. As §4 shows, reusing baselines does not achieve competitive performance in long-tail scenarios.
4
Evaluation
Our central question is whether bespoke serving systems generated by VibeServe achieve competitive performance compared with human-engineered systems and address niche yet important use cases where general-purpose systems fall short. We evaluate this question on six scenarios spanning the three axes introduced in §1: workload pattern, model architecture, and hardware. Each scenario pairs a setting in which a generic serving system is suboptimal with a VibeServe-generated implementation specialized for the model, hardware, and workload. 4.1
Setup
All scenarios follow the interface in §3.1: VibeServe receives model weights, a reference implementation, correctness and performance harnesses, and natural-language deployment instructions. We verify generated systems against the reference implementation and report the workload-relevant performance metric, such as token throughput, latency, or time-to-first-token (TTFT). Across all scenarios, the Implementer, Accuracy Judge, and Performance Evaluator are each instantiated with Codex CLI [55], and the outer loop uses the issue-tracker policy (§3.3). We evaluate the following scenarios. §A gives more details. • Scenario A: Standard LLM serving. We serve Llama-3.1-8B-Instruct [19] on an NVIDIA H100, stress-testing VibeServe in a mature setting where existing systems are heavily optimized. We verify greedy-decoding outputs and measure generation throughput across arrival rates. • Scenario B: Code editing with predicted outputs. We serve Qwen3-32B [73] on an NVIDIA H100 using a predicted-outputs interface [54]. Code-editing workloads often exhibit large overlap between the input context, such as the original file, and the generated edit [13, 74, 66]. We generate a system to exploit this via speculative decoding from user-provided predictions, a capability absent from standard serving systems. We measure single-batch latency on CodeEditorBench [20]. #workload • Scenario C: Hybrid-architecture prompt caching. We serve Olmo-Hybrid-7B [50] on an NVIDIA L4 GPU with prompt caching. The model combines Gated DeltaNet layers [75] with 6
Figure 3: On Llama-3.1-8B-Instruct (H100), VibeServe matches vLLM and exceeds SGLang by 5% (TTFT by 3%) over 60 agentic-loop iterations. Panels show the ratio of VibeServe’s token throughput, mean TTFT, and mean TPOT to vLLM’s; 1.0 is parity, higher is better. Each line corresponds to one of four request rates (8, 32, 64, 128 req/s); the agent introduced higher rates after plateauing on the previous one. attention layers, which makes efficient prompt caching difficult with limited GPU memory [57]. We use a RAG-like synthetic workload in which requests share a 32k-token prefix, append a 128-token unique suffix, and generate 128 output tokens. We measure generation throughput. #model #workload • Scenario D: Streaming ASR. We serve Moonshine Streaming medium [34] for streaming automatic speech recognition (ASR) on an NVIDIA L4 GPU. Unlike conventional ASR models such as Whisper [59], Moonshine uses sliding-window attention in the speech encoder to reduce TTFT in streaming applications, which requires system-level support missing from existing serving systems. We measure TTFT at concurrency 32 in a streaming setting where clients send audio chunks every 2 seconds and compare against a vLLM plugin baseline. #model #workload • Scenario E: Local constrained decoding. We run Llama-3.1-8B-Instruct [19] on a MacBook for JSON generation with constrained decoding. We measure single-batch latency on JSONSchemaBench [15]. JSON schemas fix long deterministic token spans (e.g., object keys, delimiters, fixed value prefixes), so a specialized decoder can avoid the generic per-step sampling and token-filtering overhead that general serving stacks pay on every output token. #workload #hardware • Scenario F: Local image generation. We run Show-o2 [72] on a MacBook for image generation. This is a unified vision-language model with a complex architecture that combines a discrete tokenizer, a continuous diffusion head, and an autoregressive language model in a single forward pass and is not supported by vLLM or vLLM-Omni. #model #hardware 4.2
Results
We present results in scenario order. Iteration-level details (which optimization landed when, which alternatives the agent tried and reverted) are taken from VibeServe’s own logs. Scenario A: parity on a heavily optimized setting. Figure 3 traces 60 VibeServe iterations on Llama-3.1-8B-Instruct (H100). The generated system reaches vLLM parity on token throughput and TPOT at all four request rates and lands within 5% on TTFT; it exceeds SGLang by 5% on throughput and 3% on TTFT. VibeServe pursued throughput first, reaching parity by iteration 30 with latency roughly flat, then shifted to latency, with TTFT and TPOT improving sharply over iterations 30–60. The four request rates (8, 32, 64, 128 req/s) were not pre-specified: VibeServe introduced each higher rate after plateauing, escalating to 128 req/s on its own. Scenario B: predicted-output speculative decoding. Figure 4a traces 15 iterations on Qwen332B/CodeEditorBench against two vLLM baselines: vanilla autoregressive (1.0×) and draft-model speculative decoding (≈ 3.0×, dashed). Iteration 2 adds CUDA-graph capture (1.35×); iteration 3 7
(a) Scenario B.
(b) Scenario C.
(c) Scenario D.
Figure 4: Workload- and model-specific scenarios. Each panel shows speedup of the VibeServegenerated system over a baseline across VibeServe iterations; dashed line at 1.0 is parity, higher is better. (a) Qwen3-32B on CodeEditorBench, vs. vLLM without/with draft-model speculative decoding. (b) Olmo-Hybrid-7B token-throughput on a 32k-token shared-prefix workload, vs. vLLM. (c) Moonshine Streaming medium TTFT at concurrency 32, vs. a vLLM plugin baseline. introduces the predicted-output verifier in 16-token blocks, proposing tokens from the user-supplied prediction and verifying them in a single target-model forward, reaching 2.9×, already on par with vLLM’s draft-model speculative decoder at zero draft-model compute. Block sizing and acceptance bookkeeping reach 5.95× by iteration 14, 2.0× over vLLM-with-spec-dec. Scenario C: hybrid-architecture prompt caching. Figure 4b shows token-generation throughput vs. vLLM on Olmo-Hybrid-7B (L4) over 15 iterations. Iterations 1–6 fail accuracy gates while VibeServe wires up the dual cache: attention KV blocks plus per-DeltaNet recurrent-state snapshots at the prefix boundary. Iteration 7 lands continuous batched decode against the shared state (2.45×); iteration 9 adds CUDA-graph capture (3.25×); the system plateaus near 3.45×. The vLLM baseline cannot share DeltaNet state across requests, so the 32k prefix is recomputed per request. Scenario D: streaming ASR. Figure 4c shows TTFT speedup over a vLLM-Moonshine plugin at concurrency 32 on L4 over 16 iterations. Iteration 5 reaches a working but sub-baseline configuration (0.84×) by aligning the per-stream encoder cache with Moonshine’s sliding-window attention; iteration 10 adds CUDA-graph capture (1.1×); iteration 13 adds a paged KV cache for per-stream encoder state (1.69×, holding through iteration 16). The improvement comes from giving the encoder layer first-class per-stream cache management, which the plugin path does not expose. Scenario E: constrained JSON decoding on a MacBook. Figure 5a traces the trajectory from a 22.1 s vanilla autoregressive baseline. VibeServe first adds XGrammar-based constrained decoding [39] (16.9 s), then layers speculative decoding with a Llama-3.2-1B-Instruct-4bit draft against the 8B-8bit target at K=4, reaching 9.3 s; a larger 3B-4bit draft was slower, since the 1B’s lower per-step cost outweighed its lower acceptance rate. Bumping mlx_lm’s prefill_step_size from 512 to 2048 prefills our ∼1300-token prompts in one chunk, yielding 8.6 s (2.6×); K/V quantization, alternative K, and mx.compile did not help. Scenario F: Show-o2 on H100 and MacBook. On H100 (Figure 5b), p50 latency falls from 873 ms to 687 ms (21.4%) over 20 iterations. Gains are front-loaded: iteration 1 contributes 9.7% (CUDA-graph replay/prewarm, VAE/postprocess layout); iteration 2, 5.4% (trim inactive diffusion tokens, restrict AdaLN to the active image span); iteration 6, 3.1% (Qwen tail trim); iterations 11–12, 1.7% combined. Subsequent passes map the limits: aggressive trimming and naive batching regress quality, FlashAttention-2/GQA/torch.compile/fp16 alter outputs or produce NaNs, and Qwen prefix reuse yields no gain (the text prefix is tiny next to the 730-token image span). On MacBook (Figure 5c), VibeServe first ports the Qwen2.5-1.5B body and 10-block diffusion head to MLX and elides a redundant SigLIP und_trans pass on noisy latents (2.4×). Cross-step redundancy then dominates: prefix-KV caches on the body and head, plus a prefill trim to [0, image_end), bring warm latency to 3.5×, with the body at ∼92% of the fp16 compute peak. Quantization regresses on the compute-bound body; only int4 on the bandwidth-bound head survives. A classifier-free-guidance (CFG) stride at K=16 that skips the unconditional branch on K−1 of every K steps and reuses the 8
Ours Baseline
3.0
2.0 Constrained
Prefill 2048
decoding
1.0
Speedup
Speedup
2.5
1.5
Speculative decoding 1
2
3
6×
Speedup (ours) Baseline fp16 kernel-peak (6.7×)
5×
prefix-KV cache
7×
4
Iteration
5
CFG stride
4× MLX rewrite 3× 2×
6
1×
7
(a) Scenario E. Constraineddecoding speedup over baseline across 7 iterations.
(b) Scenario F. Show-o2 1.5B-HQ 432×432 text-to-image speedup over 20 iterations.
0
2
4
6
8
Iteration
10
12
14
(c) Scenario F. Show-o2 speedup over 14 iterations; dotted line is the fp16 kernel-peak ceiling (6.7×).
Figure 5: Hardware- and workload-specific scenarios where existing serving systems lack a fast path or do not run. Each panel shows speedup over a baseline across VibeServe iterations; dashed line at 1.0 is parity, higher is better. (a) Llama-3.1-8B-Instruct JSON decoding on JSONSchemaBench, MacBook (Apple M3 Pro, 36 GB). (b) Show-o2 1.5B-HQ 432×432 text-to-image on H100. (c) Showo2 on the same MacBook; the dotted line marks the fp16 kernel-peak ceiling (6.7×).
cached v_uncond reaches 15.54 s (6.27× over PyTorch-MPS), within ∼7% of a 14.5 s physics floor obtained by replacing each per-step component with its fp16 kernel-perfect time.
5
Related Work
Agentic optimization systems use a few search paradigms, none of which have been applied to greenfield end-to-end system synthesis. Evolutionary search selects among agent-generated candidates by measured performance [51, 61, 42, 68, 21]; multi-agent iteration has agents hypothesize, experiment, and refine across rounds within a single context window [22, 32]; autoresearch [33] puts one long-running agent in charge of the search, tracking candidates across git branches. All three target a bounded code scope (e.g., a marked region) or use a scalar score or a single conversation that cannot encode the bottleneck information driving an end-to-end system’s next step. VibeServe is the first agentic system to design a multi-component serving system end-to-end. VibeServe sits within a broader literature on long-horizon coding agents [35, 63, 11, 6, 27, 70, 1]. The standard recourse when a task exceeds a context window is compaction [5, 38], whose lossy summarization causes drift in performance and correctness. Industrial prototypes from Cursor and Anthropic show agent harnesses can build end-to-end systems via an explicit handoff design that passes work between fresh agent sessions through task abstractions over shared repository state [41, 9], but stop short of optimizing performance. Building on this design, VibeServe targets performant code: agents get direct profiler access, role-based agents fold performance analysis into every implementation change, and skills package context about the platform, optimization techniques, and profiling methodology (Appendix C).
6
Conclusion
We argue for a different point in the LLM serving design space: rather than a single general-purpose runtime, generate a bespoke serving system for each deployment target. VibeServe demonstrates that the agentic loop matches vLLM in a standard setting and yields concrete wins across six non-standard scenarios spanning workload, architecture, and hardware, two of which cannot run on any generic stack. Our work has limitations: single-seed runs, a user-supplied correctness checker, and a non-trivial per-target compute budget (§A). Natural extensions are curriculum bootstrapping from a simpler target and branching exploration of divergent outer-loop strategies, both plugging into the inner-loop interface. As coding agents improve, generation-time specialization will beat runtime generality [46, 12, 48] in more domains where generic abstractions cost performance. 9
References [1] Thomas Anderson, Ratul Mahajan, Simon Peter, and Luke Zettlemoyer. Self-defining systems. 2025. URL https://foci.uw.edu/papers/whitepaper2025-sds.pdf. [2] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster machine learning through dynamic Python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24). ACM, 2024. doi: 10.1145/3620665.3640366. [3] Anthropic. Introducing the model context protocol. model-context-protocol, 11 2024.
https://www.anthropic.com/news/
[4] Anthropic. Claude Code. https://docs.anthropic.com/en/docs/claude-code/overview, 2025. Accessed: 2026-05-06. [5] Anthropic. Effective context engineering for ai agents. https://www.anthropic.com/engineering/ effective-context-engineering-for-ai-agents, 2025. Anthropic Engineering blog. Accessed: 2026-05-06. [6] Anthropic. Effective harnesses for long-running agents. https://www.anthropic.com/engineering/ effective-harnesses-for-long-running-agents, 11 2025. Anthropic Engineering blog. Accessed: 2026-05-06. [7] Anthropic and Contributors. Agent skills: A standardized way to give AI agents new capabilities and expertise. https://agentskills.io, 2025. Open standard; https://github.com/agentskills/ agentskills. [8] Brian N Bershad, Stefan Savage, Przemyslaw Pardyak, Emin Gün Sirer, Marc E Fiuczynski, David Becker, Craig Chambers, and Susan Eggers. Extensibility safety and performance in the spin operating system. In Proceedings of the fifteenth ACM symposium on Operating systems principles, pages 267–283, 1995. [9] Nicholas Carlini. Building a C compiler with a team of parallel Claudes. https://www.anthropic.com/ engineering/building-c-compiler, 2 2026. Anthropic Engineering blog. Accessed: 2026-05-06. [10] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022. [11] Xiang Deng, Jeff Da, Edwin Pan, Yannis Y. He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-bench pro: Can AI agents solve long-horizon software engineering tasks?, 2026. URL https://openreview.net/forum?id= 9R2iUHhVfr. [12] Dawson R Engler, M Frans Kaashoek, and James O’Toole Jr. Exokernel: An operating system architecture for application-level resource management. ACM SIGOPS Operating Systems Review, 29(5):251–266, 1995. [13] Fireworks AI. How cursor built fast apply using the speculative decoding api. https://fireworks.ai/ blog/cursor, 2024. Accessed: 2026-05-05. [14] Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. Perfbench: Can agents resolve real-world performance bugs?, 2025. URL https://arxiv.org/abs/2509.24091. [15] Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. Jsonschemabench: A rigorous benchmark of structured outputs for language models, 2025. URL https://arxiv.org/abs/2501.10868. [16] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6: 325–338, 2024.
10
[17] In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A programmable serving system for emerging llm applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages 415–430, 2025. [18] Raja Gond, Aditya K Kamath, Ramachandran Ramjee, and Ashish Panwar. Llm-42: Enabling determinism in llm inference with verified speculation. arXiv preprint arXiv:2601.17768, 2026. [19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [20] Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, et al. Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprint arXiv:2404.03543, 2024. [21] Ping Guo, Chenyu Zhu, Siyuan Chen, Fei Liu, Xi Lin, Zhichao Lu, and Qingfu Zhang. EvoEngineer: Mastering automated CUDA kernel code evolution with large language models. ArXiv, abs/2510.03760, 2025. URL https://api.semanticscholar.org/CorpusID:281842469. [22] Pouya Hamadanian, Pantea Karimi, Arash Nasr-Esfahany, Kimia Noorbakhsh, Joseph Chandler, Ali ParandehGheibi, Mohammad Alizadeh, and Hari Balakrishnan. Glia: A human-inspired ai for automated systems design and optimization, 2026. URL https://arxiv.org/abs/2510.27176. [23] Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on Apple silicon, 2023. URL https://github.com/ml-explore/mlx. [24] Horace He and Thinking Machines Lab. Defeating nondeterminism in llm inference. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250910. https://thinkingmachines.ai/blog/defeatingnondeterminism-in-llm-inference/. [25] Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. Swe-perf: Can language models optimize code performance on real-world repositories?, 2025. URL https://arxiv.org/abs/2507.12415. [26] Paul Hübner, Andong Hu, Ivy Peng, and Stefano Markidis. Apple vs. oranges: Evaluating the apple silicon m-series SoCs for HPC performance and efficiency, 2025. URL https://arxiv.org/abs/2502.05317. [27] Geoffrey Huntley. Everything is a ralph loop. https://ghuntley.com/loop/, 1 2026. Blog post. Accessed: 2026-05-06. [28] Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, et al. Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(3):1–24, 2025. [29] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? ArXiv, abs/2310.06770, 2023. URL https://api.semanticscholar.org/CorpusID:263829697. [30] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmented generation. ACM Transactions on Computer Systems, 44(1):1–27, 2025. [31] Keisuke Kamahori, Wei-Tzu Lee, Atindra Jha, Rohan Kadekodi, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. Voxserve: Streaming-centric serving system for speech language models. arXiv preprint arXiv:2602.00269, 2026. [32] Pantea Karimi, Kimia Noorbakhsh, Mohammad Alizadeh, and Hari Balakrishnan. Improving coherence and persistence in agentic AI for system optimization, 2026. URL https://arxiv.org/abs/2603.21321. [33] Andrej Karpathy. autoresearch: An autonomous LLM research loop. https://github.com/karpathy/ autoresearch, 2026. Accessed: 2026-05-06. [34] Manjunath Kudlur, Evan King, James Wang, and Pete Warden. Moonshine v2: Ergodic streaming encoder asr for latency-critical speech applications. arXiv preprint arXiv:2602.12241, 2026. [35] Thomas Kwa, Ben West, Joel Becker, et al. Measuring AI ability to complete long tasks. https: //metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/, 3 2025.
11
[36] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [37] LangChain. DeepAgents. https://github.com/langchain-ai/deepagents, 2025. Accessed: 202605-06. [38] Kangwook Lee. Investigating how Codex context compaction works. https://x.com/Kangwook_Lee/ status/2028955292025962534, 2026. Accessed: 2026-05-07. [39] Linzhang Li, Yixin Dong, Guanjie Wang, Ziyi Xu, Alexander Jiang, and Tianqi Chen. Xgrammar-2: Efficient dynamic structured generation engine for agentic llms, 2026. URL https://arxiv.org/abs/ 2601.04426. [40] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Haim Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avshalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamba language model. ArXiv, abs/2403.19887, 2024. URL https://api.semanticscholar.org/CorpusID:268793596. [41] Wilson Lin. Scaling long-running autonomous coding. https://cursor.com/blog/scaling-agents, 1 2026. Cursor blog. Accessed: 2026-05-06. [42] Shu Liu, Mert Cemri, Shubham Agarwal, Alexander Krentsel, Ashwin Naren, Qiuyang Mang, Zhifei Li, Akshat Gupta, Monishwaran Maheswaran, Audrey Cheng, Melissa Pan, Ethan Boneh, Kannan Ramchandran, Koushik Sen, Alexandros G. Dimakis, Matei Zaharia, and Ion Stoica. SkyDiscover: A flexible framework for AI-driven scientific and algorithmic discovery, 2026. URL https://skydiscover-ai. github.io/blog.html. [43] Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023. [44] Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E Gonzalez, et al. Autellix: An efficient serving engine for llm agents as general programs. arXiv preprint arXiv:2502.13965, 2025. [45] Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. Swe-fficiency: Can language models optimize real-world repositories on real workloads?, 2025. URL https://arxiv.org/abs/2511.06090. [46] Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft. Unikernels: Library operating systems for the cloud. ACM SIGARCH Computer Architecture News, 41(1):461–472, 2013. [47] Anil Madhavapeddy, Thomas Leonard, Magnus Skjegstad, Thomas Gazagnaire, David Sheets, Dave Scott, Richard Mortier, Amir Chaudhry, Balraj Singh, Jon Ludlam, et al. Jitsu:{Just-In-Time} summoning of unikernels. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pages 559–573, 2015. [48] Henry Massalin and Calton Pu. Threads and input/output in the synthesis kernal. In Proceedings of the twelfth ACM symposium on Operating systems principles, pages 191–201, 1989. [49] Dylan McNamee, Jonathan Walpole, Calton Pu, Crispin Cowan, Charles Krasic, Ashvin Goel, Perry Wagle, Charles Consel, Gilles Muller, and Renauld Marlet. Specialization tools and techniques for systematic optimization of system software. ACM Transactions on Computer Systems (TOCS), 19(2):217–251, 2001. [50] William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, et al. Olmo hybrid: From theory to practice and back. arXiv preprint arXiv:2604.03444, 2026. [51] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and algorithmic discovery, 2025. URL https://arxiv.org/abs/2506.13131. [52] NVIDIA. TensorRT-LLM. https://github.com/NVIDIA/TensorRT-LLM, 2023.
12
[53] NVIDIA, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhsh, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, Andrew Tao, Anna Shors, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Brian Butterfield, Bryan Catanzaro, Carlo del Mundo, Chengyu Dong, Christine Harvey, Christopher Parisien, Dan Su, Daniel Korzekwa, Danny Yin, Daria Gitman, David Mosallanezhad, Deepak Narayanan, Denys Fridman, Dima Rekesh, Ding Ma, Dmytro Pykhtar, Dong Ahn, Duncan Riach, Dusan Stosic, Eileen Long, Elad Segal, Ellie Evans, Eric Chung, Erick Galinkin, Evelina Bakhturina, Ewa Dobrowolska, Fei Jia, Fuxiao Liu, Gargi Prasad, Gerald Shen, Guilin Liu, Guo Chen, Haifeng Qian, Helen Ngo, Hongbin Liu, Hui Li, Igor Gitman, Ilia Karmanov, Ivan Moshkov, Izik Golan, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jarno Seppanen, Jason Lu, Jason Sewall, Jiaqi Zeng, Jiaxuan You, Jimmy Zhang, Jing Zhang, Jining Huang, Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jon Barker, Jonathan Cohen, Joseph Jennings, Jupinder Parmar, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Luna, Keshav Santhanam, Kezhi Kong, Kirthi Sivamani, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Lawrence McAfee, Leon Derczynski, Lindsey Pavao, Luis Vega, Lukas Voegtle, Maciej Bala, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Markus Kliegl, Marta Stepniewska-Dziubinska, Matthieu Le, Matvei Novikov, Mehrzad Samadi, Michael Andersch, Michael Evans, Miguel Martinez, Mike Chrzanowski, Mike Ranzinger, Mikolaj Blaz, Misha Smelyanskiy, Mohamed Fawzy, Mohammad Shoeybi, Mostofa Patwary, Nayeon Lee, Nima Tajbakhsh, Ning Xu, Oleg Rybakov, Oleksii Kuchaiev, Olivier Delalleau, Osvald Nitski, Parth Chadha, Pasha Shamis, Paulius Micikevicius, Pavlo Molchanov, Peter Dykas, Philipp Fischer, Pierre-Yves Aquilanti, Piotr Bialecki, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi, Rahul Kandu, Ran El-Yaniv, Raviraj Joshi, Roger Waleffe, Ruoxi Zhang, Sabrina Kavanaugh, Sahil Jain, Samuel Kriman, Sangkug Lym, Sanjeev Satheesh, Saurav Muralidharan, Sean Narenthiran, Selvaraj Anandaraj, Seonmyeong Bak, Sergey Kashirsky, Seungju Han, Shantanu Acharya, Shaona Ghosh, Sharath Turuvekere Sreenivas, Sharon Clay, Shelby Thomas, Shrimai Prabhumoye, Shubham Pachori, Shubham Toshniwal, Shyamala Prayaga, Siddhartha Jain, Sirshak Das, Slawek Kierat, Somshubra Majumdar, Song Han, Soumye Singhal, Sriharsha Niverty, Stefania Alborghetti, Suseella Panguluri, Swetha Bhendigeri, Syeda Nahida Akter, Szymon Migacz, Tal Shiri, Terry Kong, Timo Roman, Tomer Ronen, Trisha Saar, Tugrul Konuk, Tuomas Rintamaki, Tyler Poon, Ushnish De, Vahid Noroozi, Varun Singh, Vijay Korthikanti, Vitaly Kurin, Wasi Uddin Ahmad, Wei Du, Wei Ping, Wenliang Dai, Wonmin Byeon, Xiaowei Ren, Yao Xu, Yejin Choi, Yian Zhang, Ying Lin, Yoshi Suhara, Zhiding Yu, Zhiqi Li, Zhiyu Li, Zhongbo Zhu, Zhuolin Yang, and Zijia Chen. Nemotron-H: A family of accurate and efficient hybrid mamba-transformer models, 2025. URL https://arxiv.org/abs/2504.03624. [54] OpenAI. Predicted outputs. https://platform.openai.com/docs/guides/predicted-outputs, 2024. OpenAI API documentation. Accessed: 2026-05-05. [55] OpenAI. OpenAI Codex CLI. https://github.com/openai/codex, 2025. Accessed: 2026-05-06. [56] Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels? ArXiv, abs/2502.10517, 2025. URL https://api.semanticscholar.org/CorpusID:276408165. [57] Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Ravi Netravali, and Yida Wang. Marconi: Prefix caching for the era of hybrid LLMs. ArXiv, abs/2411.19379, 2024. URL https://api.semanticscholar.org/CorpusID:274367849. [58] Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel K. Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, Nathanael Bosch, Eli Meril, Albert Steppi, Arman Zharmagambetov, Fangzhao Zhang, David Perez-Pineiro, Alberto Mercurio, Ni Zhan, Talor Abramovich, Kilian Lieret, Hanlin Zhang, Shirley Huang, Matthias Bethge, and Ofir Press. Algotune: Can language models speed up general-purpose numerical programs?, 2025. URL https://arxiv.org/abs/2507. 15887. [59] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), 2023. URL https://arxiv.org/abs/2212.04356. [60] Atharva Sehgal, James Hou, Swarat Chaudhuri, Jennifer J. Sun, and Yisong Yue. Formulacode: Evaluating agentic superoptimization on large codebases. In ICML 2025 Workshop on Programmatic Representations for Agent Learning, 2025. URL https://openreview.net/forum?id=CMdtl83aZF. [61] Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URL https:// github.com/algorithmicsuperintelligence/openevolve. [62] Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. Gso: Challenging software optimization tasks for evaluating swe-agents, 2025. URL https://arxiv.org/ abs/2505.23671.
13
[63] Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios, 2026. URL https://arxiv.org/abs/ 2512.18470. [64] Prabhu Vellaisamy, Thomas Labonte, Sourav Chakraborty, Matt Turner, Samantika Sury, and John Paul Shen. Characterizing and optimizing llm inference workloads on cpu-gpu coupled architectures. In 2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 49–61. IEEE, 2025. [65] vLLM Team. Hybrid KV cache manager — vLLM documentation. https://docs.vllm.ai/en/ stable/design/hybrid_kv_cache_manager/, 2024. [66] Peiding Wang, Li Zhang, Fang Liu, Yinghao Zhu, Wang Xu, Lin Shi, Xiaoli Lian, Minxiao Li, Bo Shen, and An Fu. Efficientedit: Accelerating code editing via edit-oriented speculative decoding. arXiv preprint arXiv:2506.02780, 2025. [67] Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun, Haorui Wang, Shuibai Zhang, Wenjie Hu, Mya Schroder, Bilge Mutlu, Dawn Song, and Robert D Nowak. The long-horizon task mirage? diagnosing where and why agentic systems break, 2026. URL https://arxiv.org/abs/2604.11978. [68] Nina Wiedemann, Quentin Leboutet, Michael Paulitsch, Diana Wofk, and Benjamin Ummenhofer. KernelFoundry: Hardware-aware evolutionary GPU kernel optimization, 2026. URL https://arxiv.org/ abs/2603.12440. [69] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. [70] Junde Wu, Minhao Hu, Jiayuan Zhu, Jiazhen Pan, Yuyuan Liu, Min Xu, and Yueming Jin. Git context controller: Manage the context of llm-based agents like git, 2026. URL https://arxiv.org/abs/2508. 00031. [71] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024. [72] Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models, 2025. URL https://arxiv.org/abs/2506.15564. [73] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [74] Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487, 2023. [75] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024. [76] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for llm inference serving, 2025. URL https://arxiv.org/abs/2501.01005. [77] Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, et al. vllm-omni: Fully disaggregated serving for any-to-any multimodal models. arXiv preprint arXiv:2602.02204, 2026. [78] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-1. URL https://www.usenix.org/conference/ osdi22/presentation/yu. [79] Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, et al. Jenga: Effective memory management for serving llm with heterogeneity. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages 446–461, 2025.
14
[80] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https://arxiv.org/abs/2312.07104. [81] Yusheng Zheng, Yanpeng Hu, Wei Zhang, and Andi Quinn. Towards agentic OS: An LLM agent framework for linux schedulers, 2025. URL https://arxiv.org/abs/2509.01245.
A
Detailed Evaluation Scenarios
This appendix provides additional details for each evaluation scenario. Across scenarios, the agentic loop receives model weights, a HuggingFace Transformers reference implementation [69], accuracychecking scripts, performance-evaluation scripts, and natural-language instructions. The generated system is evaluated against the reference implementation for correctness and against one or more baseline serving systems for performance. Scenario A: Standard LLM serving on H100. Architecture. Llama-3.1-8B-Instruct [19] is a dense decoder-only Transformer with grouped-query attention (32 query heads sharing 8 key/value heads), RoPE positional encodings, SwiGLU MLPs, and a 128k-token context window. This configuration is the design center of every modern serving stack: dense decoder-only inference on data-center GPUs is precisely what vLLM, SGLang, and TensorRT-LLM are tuned for, and the standard optimization stack is by now well-known — paged KV cache [36], continuous batching [78], CUDA graphs, FlashAttention/FlashInfer kernels [10, 76], and operator fusion. Workload and metric. An open-loop synthetic load generator drives the system at four request rates (8, 32, 64, 128 req/s). Request arrival times follow a Poisson distribution (exponential inter-arrival times), and requests are launched independently of completion; each rate is run for 60 seconds with seed 42. Prompts are sampled uniformly from a predefined prompt pool, and output length is capped at max_tokens=128; most requests emit exactly 128 chunks, and generation uses temperature 0. We report token-generation throughput, mean TTFT, and mean TPOT relative to vLLM. Greedy decoding outputs are checked against the Hugging Face Transformers reference. Scenario B: Code editing with predicted outputs. Architecture and workload. Qwen3-32B [73] is a dense decoder-only Transformer (with Q/K-norm and grouped-query attention) served on an NVIDIA H100. The workload is code editing under OpenAI’s predicted-outputs interface [54]: each request carries both an instruction and a string of predicted output tokens representing the most likely answer. This prediction is naturally available for code editing tasks, since the pre-edit file is typically a near-prediction of the post-edit file, and prior work and deployed systems show that the overlap is large in practice [13, 74, 66]. We report single-batch latency on CodeEditorBench [20]. Optimization opportunity. The predicted-outputs interface is a degenerate case of speculative decoding in which the draft is the user-supplied prediction at zero draft-model cost. The serving system feeds a window of K predicted tokens through the target model in a single forward pass and commits the longest prefix whose argmax matches the prediction; on a mismatch, it falls back to ordinary autoregressive decoding for one token and resumes from the prediction. With high overlap, latency drops by nearly a factor of K with no additional compute. Why generic systems cannot exploit this. Standard systems like vLLM or SGLang support speculative decoding, but do not support the predicted-outputs interface. Predicted outputs are a different request type: the engine needs a per-request token stream, a verifier loop that consumes from that stream until divergence, and well-defined fallback semantics. Adding this to vLLM means non-trivial changes to the scheduler, sequence-group state, and sampler. A bespoke system can build the request lifecycle directly around the predicted-outputs API. Scenario C: Prompt caching for a hybrid architecture. Architecture. Olmo-Hybrid-7B [50] interleaves Gated DeltaNet layers [75] with standard self-attention layers. Gated DeltaNet is a linear-attention/SSM-style layer, and its per-sequence state is a fixed-size matrix that is updated recurrently as tokens arrive, in contrast to attention’s KV cache, which grows linearly with sequence length. Each layer, therefore, carries a different kind of state, and the cache layout, eviction policy, and sharing semantics differ per layer type. 15
Workload and metric. A RAG-like workload in which every request shares a 32k-token system prefix, appends a 128-token request-specific suffix, and produces 128 output tokens. We report token-generation throughput at concurrency 20 on an NVIDIA L4 (24 GB), where memory pressure rules out keeping uncompressed per-request state copies. Optimization opportunity. Prefix sharing across requests is a standard technique, but a hybrid model needs two cache mechanisms in parallel: KV blocks for attention layers and a snapshot of the recurrent state at the prefix boundary for each DeltaNet layer. By having knowledge of the workload at design time, the agent can optimize the system for a particular case and reduce the overhead of supporting prompt caching across generic workloads. Why generic systems cannot exploit this. vLLM and SGLang were architected around the attention KV cache; first-class hybrid-KV support is recent and limited [65, 57]. Sharing the recurrent state across requests requires snapshotting at prefix boundaries, which incurs significant memory overhead to support generic cases, especially on hardware with limited memory capacity like L4. Scenario D: Streaming ASR with sliding-window encoder attention. Architecture. Moonshine Streaming medium [34] is an encoder-decoder ASR model designed for low-latency streaming. The encoder uses sliding-window attention over audio frames, so previously-encoded frames remain valid as new audio arrives; only the new tail needs to be encoded. The decoder is a small autoregressive Transformer that emits text tokens conditioned on encoder outputs. In contrast, Whisper [59], the standard ASR baseline, encodes a full clip in a single pass and is not designed for incremental encoding. Workload and metric. 32 concurrent streaming clients, each sending a 2-second audio chunk every 2 seconds. We report time-to-first-token (TTFT) per chunk, which captures responsiveness for interactive transcription. We compare against a vLLM-plugin Moonshine baseline. Optimization opportunity. Sliding-window attention permits encoder-output caching: each chunk encodes only the new tail and reuses the previous encoder outputs to feed the decoder. The system needs (i) a per-stream encoder cache aligned with the sliding window, (ii) eviction synchronized with the window’s stride, and (iii) a scheduler that batches per-chunk encoder work alongside per-token decoder work across many concurrent streams. Why generic systems cannot exploit this. vLLM can support Moonshine Streaming model via a plugin, but cannot support encoder prompt caching without a significant modification in the system code, leading to redundant computation for streaming applications. In contrast, the bespoke system can optimize around Moonshine’s specific sliding-window attention and expose the encoder layer to per-stream cache management. Scenario E: Local constrained decoding for JSON generation. Architecture and target. Llama3.1-8B-Instruct [19] (8-bit MLX quantization) on a MacBook Pro (Apple M3 Pro, 36 GB unified memory) running macOS 26.5 (build 25F5042g). We optimize single-stream end-to-end latency at T =0 on JSONSchemaBench [15], a corpus of ∼9,558 real-world JSON schemas drawn from jsonschema-corpus, GlaiveAI function-call schemas, and Kubernetes schemas, which is used to measure both the speed and the schema-feature coverage of constrained-decoding engines. The schemas are partitioned into 10 splits along two axes (domain and complexity): function-calling (GlaiveAI-2K, 1,707), operational/resource-access APIs (Snowplow 403, Washington Post 125), Kubernetes API configurations (1,064), a curated JSONSchemaStore set (492), and five GitHub-sourced Misc tiers graded by constraint complexity (Trivial 444, Easy 1,943, Medium 1,976, Hard 1,240, Ultra 164); the distribution is skewed toward GitHub Easy/Medium and function-call schemas, with progressively rarer Hard/Ultra tails that stress less common JSON Schema features. The workload is held fixed across seven VibeServe iterations and we report p50 latency. Optimization opportunity. Three techniques compose. First, JSON-schema-constrained decoding with XGrammar [39] masks tokens that would violate the schema and applies jump-forward to skip over deterministic tokens implied by the schema. Second, speculative decoding uses Llama-3.2-1BInstruct (4-bit, MLX) as the draft against the 8B-8bit target with K=4 draft tokens per step; the smaller draft is preferred because its lower per-step cost outweighs its lower acceptance rate. Third, raising mlx_lm’s prefill chunk size from the default 512 to 2048 lets a ∼1300-token prompt prefill in a single chunk. 16
Why generic systems cannot exploit this. vLLM and SGLang implement constrained decoding well, but only on CUDA backends; mlx_lm runs on Apple Silicon but lacks both XGrammar integration and a speculative-decoding pipeline. Beyond the missing backend, the wins here demand integration deeper than a generic structured-output API allows. The schema must be enforced inside the decoder via a per-token XGrammar bitmask with grammar-aware termination, and that bitmask must coexist with speculative decoding’s rollback semantics: draft tokens may be partially accepted, so a constrained decoder that assumes monotonic token consumption silently drifts on rejection (VibeServe hit two such MLX correctness bugs during the study). The performance levers are similarly non-generic: prefill_step_size=2048 is an MLX-specific fix for the interaction between ∼1300token prompts and cache chunking; XGrammar’s any_whitespace=False and compact separators change token counts and thus latency; and several plausible generic optimizations (larger draft, different K, KV quantization, forced-token jump-forward) regressed in this configuration. Residual failures (unbounded patternProperties, nested arrays, approximate oneOf/anyOf unions) need schema-aware decoding behavior that off-the-shelf APIs do not expose.
Scenario F: Local image generation with a unified vision-language model. Architecture. Showo2 [72] is a unified vision-language model whose forward pass interleaves autoregressive text-token generation (a Qwen2.5-1.5B body) with diffusion-style image-token refinement (a 10-block diffusion head with SigLIP-based image conditioning). Each generation step is partly an AR decode (text body, with prefix-KV cache) and partly a denoising step (head, with classifier-free guidance over conditional and unconditional branches). The control flow does not match either a pure decoder-only LLM or a pure diffusion image model. Workload and targets. We evaluate text-to-image generation at 432×432 resolution with 20 sampler steps in two deployments. (i) MacBook Pro (Apple M3 Pro, 36 GB unified memory) running macOS 26.5 (build 25F5042g): single-stream warm-min latency, baseline is the Show-o2 PyTorch-MPS reference implementation. (ii) NVIDIA H100: single-request latency at fixed prompt and seed, with the baseline’s PyTorch implementation wrapped as a server; the benchmark client and server share the same container over the loopback interface and exchange raw PPM frames, so the measured target is serving latency rather than network or encoding overhead. Accuracy gate. Bitwise reproduction of the baseline is too restrictive when quantization, kernel substitution, or step-skipping is on the table. We provide VibeServe with a custom checker that compares each generated image against the baseline at a fixed prompt, seed, step count, guidance scale, and device profile; the checker accepts an image if it matches the baseline’s 432×432 dimensions and meets a quality bar of MAE ≤ 2, PSNR ≥ 35 dB, and local luminance SSIM ≥ 0.98. The same checker gates both the H100 and MacBook variants. Optimization opportunity. VibeServe ports the body and head to the target backend (MLX on MacBook), elides a redundant SigLIP encode of noisy latents on every step, adds prefix-KV caches on body and head, trims prefill to the active image span, and applies a CFG stride that skips the unconditional branch on K−1 of every K denoising steps and reuses the cached v_uncond. Quantization is restricted to the bandwidth-bound head; weight quantization on the compute-bound body regresses latency. See §4.2 (H100) and §4.2 (MacBook) for iteration-level breakdowns. Why generic systems cannot exploit this. There is no generic serving stack for Show-o2: vLLM does not implement diffusion paths, vLLM-Omni does not include this model, and the reference is a research-grade PyTorch implementation. The AR/diffusion interleaving and the body/head/sampler co-design needed for the wins above do not generalize across models, so adding Show-o2 to a generic stack would be model-specific work that competes for engineering attention with every other model the stack supports. A bespoke system can wire the loop around exactly this control flow.
Per-role agentic-loop breakdown. Table 1 reports per-role LLM-call counts and active time across all six scenarios. The Implementer dominates active time in every run (47–60%), reflecting that producing and revising candidate code is the most compute-intensive step. The Accuracy Judge is the next-largest contributor (20–30%) and is especially heavy on Scenario C, where the dual KV/recurrent-state cache machinery makes correctness review more involved. The Performance Evaluator runs less often because performance work is gated on a passing accuracy round, and the Orchestrator is consistently a small share (3–7%) since it only selects the next issue and updates long-term memory. 17
Table 1: Per-role LLM-call breakdown across evaluation scenarios. “Calls” counts agent invocations, “Duration” is cumulative active time, “Share” is the fraction of the scenario’s total active time, and “Avg/call” is mean wall time per invocation. Roles correspond to the inner-loop Implementer, Accuracy Judge, and Performance Evaluator (§3) plus the outer-loop Orchestrator.
B
Scenario
Role
Calls Duration (h)
Share
Avg/call (s)
A: Llama-3.1-8B standard (25.0 h)
Orchestrator Implementer Judge Perf. Evaluator
120 90 90 60
1.40 5.6% 13.35 53.4% 6.08 24.3% 4.18 16.7%
42 534 243 251
B: Qwen3-32B code edit (8.66 h)
Orchestrator Implementer Judge Perf. Evaluator
47 36 36 23
0.61 7.0% 4.55 52.5% 1.98 22.8% 1.53 17.6%
46 455 198 239
C: Olmo-Hybrid prefix caching (12.54 h)
Orchestrator Implementer Judge Perf. Evaluator
47 33 33 23
0.44 3.5% 5.94 47.4% 3.77 30.1% 2.39 19.1%
34 648 411 373
D: Moonshine streaming (8.71 h)
Orchestrator Implementer Judge Perf. Evaluator
49 46 45 24
0.55 6.3% 5.25 60.3% 1.74 20.0% 1.17 13.4%
41 411 139 175
E: JSON constrained decoding (3.0 h)
Orchestrator Implementer Judge Perf. Evaluator
11 13 13 7
0.13 4.3% 1.42 47.3% 0.85 28.3% 0.60 20.0%
43 393 235 309
F: Show-o2 H100+MBP (14.0 h)
Orchestrator Implementer Judge Perf. Evaluator
70 53 51 34
0.85 6.1% 6.85 48.9% 3.05 21.8% 3.25 23.2%
44 465 215 344
Existing Assets and Licenses
Table 2 lists the third-party models, datasets, frameworks, and coding-agent harnesses used in this paper, together with their licenses. All assets are used in accordance with their published terms. Table 2: Existing assets used in this paper, with versions, licenses, and source URLs. Citations point to the paper or release we used; see §4 and §A for how each asset enters the evaluation. Asset
Version
License
Llama-3.1-8B-Instruct [19] Model Llama-3.2-1B-Instruct (4-bit, MLX) [19] Model Qwen3-32B [73] Model Olmo-Hybrid-7B [50] Model Moonshine Streaming medium [34] Model Show-o2 1.5B-HQ [72] Model
n/a n/a n/a n/a n/a n/a
Llama 3.1 Community License https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct Llama 3.2 Community License https://huggingface.co/mlx-community/Llama-3.2-1B-Instruct-4bit Apache 2.0 https://huggingface.co/Qwen/Qwen3-32B Apache 2.0 https://huggingface.co/allenai/Olmo-Hybrid-7B MIT https://huggingface.co/UsefulSensors/moonshine-streaming-medium Apache 2.0 https://github.com/showlab/show-o
CodeEditorBench [20] JSONSchemaBench [15]
Dataset Dataset
n/a n/a
Apache 2.0 No license specified
https://github.com/CodeEditorBench/CodeEditorBench https://github.com/guidance-ai/jsonschemabench
vLLM [36] SGLang [80] TensorRT-LLM [52] HuggingFace Transformers [69] MLX / mlx_lm [23] PyTorch [2] FlashAttention [10] FlashInfer [76] XGrammar [39]
Framework Framework Framework Framework Framework Framework Library Library Library
v0.19.1 v0.5.11 v1.2.1 v5.5.2 v0.31.2 v2.10 fa4-v4.0.0.beta4 v0.6.6 v0.2.0
Apache 2.0 Apache 2.0 Apache 2.0 Apache 2.0 MIT BSD-3-Clause BSD-3-Clause Apache 2.0 Apache 2.0
https://github.com/vllm-project/vllm https://github.com/sgl-project/sglang https://github.com/NVIDIA/TensorRT-LLM https://github.com/huggingface/transformers https://github.com/ml-explore/mlx https://github.com/pytorch/pytorch https://github.com/Dao-AILab/flash-attention https://github.com/flashinfer-ai/flashinfer https://github.com/mlc-ai/xgrammar
Codex CLI [55] Claude Code [4] DeepAgents [37]
Coding-agent harness Coding-agent harness Coding-agent harness
v0.125 v2.1.122 v0.4.11
Apache 2.0 Anthropic ToS (proprietary) MIT
https://github.com/openai/codex https://www.anthropic.com/claude-code https://github.com/langchain-ai/deepagents
C
Type
URL
Extended Related Work
This appendix expands the discussion in §5. Using agents to optimize performance has attracted substantial attention, with recent benchmarks measuring this ability across kernels, numerical routines, repositories, and performance bugs [56, 18
58, 25, 45, 60, 14, 62]. Agentic optimization systems organize around a few search paradigms, none of which have been applied to greenfield end-to-end system implementation and optimization. Evolutionary search maintains a population of agent-generated candidates and selects among them by measured performance: the score numerically encodes the optimization goal, and selection within the population carries that goal forward without summarization, sidestepping the drift that compactionbased handoffs incur. AlphaEvolve [51], OpenEvolve [61], and SkyDiscover [42] provide general outer-loop frameworks, and KernelFoundry [68] and EvoEngineer [21] apply this style to GPU kernel generation under paired correctness/performance gates. As implemented today, these frameworks evolve only small components, e.g., user-marked code regions inside an otherwise-fixed file; a scalar score is sufficient at that scope but cannot encode much of what an end-to-end system needs, e.g., prerequisite dependencies between optimizations (one technique often requires another to be in place first) or internal-bottleneck information that drives the next step, since which component bottlenecks is itself shaped by the agent’s prior design choices. Multi-agent iteration replaces the population with agents that hypothesize, experiment, and refine designs across rounds, carrying richer reasoning forward to drive the next decision: Glia [22] and Engram [32] use this approach to tune systems policies and heuristics, enabling gains on LLM-serving routing and autoscaling, among other tasks. This reasoning, however, lives within a single context window; Glia’s multi-context variant runs independent instances in parallel rather than passing strategic state forward. A third approach simplifies the loop further: autoresearch [33] puts one long-running agent in charge of the entire search, tracking candidate ideas across git branches, but is prone to drift within the agent’s single context. Adjacent agentic-synthesis work targets similarly bounded scopes, e.g., SchedCP [81] generates Linux scheduling policies without modifying the kernel via LLM-driven techniques. Across these approaches, the agent’s output is a bounded policy, heuristic, or module within a larger system. VibeServe is, to our knowledge, the first agentic system under any of these paradigms to do the multi-file coding work needed to design a system itself, creating bespoke LLM serving systems with multiple interconnected internal components. VibeServe sits within a broader literature on agents performing long-horizon tasks [35, 63, 11, 6, 27, 1]. The standard recourse when a task exceeds a context window is compaction [5, 38], where a session distills its state into a handoff to a fresh successor; lossy summarization causes drift in both performance and correctness over many rounds, and across optimization sessions an agent must additionally remember which bottleneck to target next, which directions have been tried and discarded, and which platform quirks have surfaced. VibeServe is inspired by industrial prototypes for long-horizon coding agents from Cursor and Anthropic, which introduce explicit task abstractions, shared repository state, and custom agent loops so fresh coding-agent sessions can execute small units of work within multi-week autonomous projects [41, 9]; these showcase agent harnesses that can build end-to-end systems from scratch, but stop short of optimizing them. Git primitives such as commits and branches have also been used to manage agent context and explore distinct strategies across sessions [70]. VibeServe exposes a version-controlled repository interface that allows flexible outer-loop strategies, including the issue-driven loop used in our evaluation. VibeServe couples each task with domain-specific correctness and performance gates, so progress is tracked over validated system designs rather than unconstrained repository edits.
19