arXiv CSopen access

Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation

arXiv:2606.11690v1 [cs.DC] 10 Jun 2026

Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation Chitral Patil∗ Independent Researcher [email protected] ORCID: 0009-0007-9169-0930 June 2026 headline finding: because public calculators take utilization as a user-supplied input (or silently assume full utilization), any utilization-naive estimate understates true cost by exactly 1/U — a factor we measure at 2.5–24× across low-tomoderate enterprise offered loads (1–10 rps) and up to 36.3× at idle—systematically mispricing self-hosting across traffic regimes, most severely over-selling it for low-traffic workloads. We additionally show that FP8 quantization benefits the MoE architectures we tested roughly 2.2– 2.4× more than the dense model (+69 to +74% vs. +31% peak throughput; n=3 architectures, broader validation needed), and our data are consistent with active parameter count—not total model size—being a primary predictor of saturation economics.

Abstract Every public LLM cost calculator we surveyed treats GPU utilization as a fixed input—entered by the user, baked in as a preset, or silently assumed at 100%—never measured against the operator’s actual load. We show that this assumption is the dominant source of error: on identical H100 hardware, effective cost spans $0.21 to $15.25 per million output tokens—an underutilization penalty of 2.5–24× across lowto-moderate enterprise offered loads (1–10 rps) and up to 36.3× near idle (λ=1 rps)1 —driven by one operator-controlled variable (offered request rate λ, which sets in-flight concurrency via Little’s Law) that no open-source calculator exposes and that prior academic work has examined only in narrow domains [7, 6]. We propose a measurement methodology that parameterizes the relationship as Ceff = f (H, M, Q, λ, L), validate it with 42 benchmarks across dense, ultrasparse MoE, and sparse MoE models, and release vllm-cost-meter—an open-source operational cost meter that attaches to a live vLLM server and reports real $/M-tokens against the operator’s own traffic, not a vendor’s benchmark. The

To rule out single-hardware confounding we repeat the core sweep on NVIDIA A100 80GB PCIe (56 runs across 8 configurations). The load-driven spread reproduces at 7.0–11.4× per configuration (compressed by cheaper, slower silicon), and the active-parameters-beat-totalparameters ordering survives at FP8. The MoE– FP8 advantage persists on MoE architectures but the dense-FP8 advantage inverts on silicon without native FP8 tensor cores—a hardwareconditional caveat that Q-as-first-class-input in Ceff (H, M, Q, λ, L) already accommodates.

∗

This research was conducted independently and does not represent the views of my employer. 1 At λ=1 the per-configuration penalty spans 17.5× (Mixtral 8x7B FP16; the cost ratio between λ=1 and saturation) to 36.3× (Qwen3-30B-A3B FP8, measured the same way); 36.3× is the idle-edge extreme of the underutilization penalty.

1

1

lationship or provided a reusable framework for practitioners. Put simply: once hardware, model, quantization, and the latency SLO are fixed, the real cost of self-hosted LLM inference is dominated by the offered request rate—and the in-flight concurrency it produces via Little’s Law—the one axis public cost calculators we surveyed do not expose. This paper measures the magnitude of that error, formalizes a better cost model, and releases an open-source tool that accounts for it.

Introduction

The real cost of self-hosted LLM inference depends heavily on a variable that public LLM cost calculators ignore: the offered request rate the GPU is actually serving and the in-flight batch size that rate produces. At 1 request/second on an H100, a Mixtral 8x7B FP16 deployment costs $15.25 per million tokens—more expensive than Claude Sonnet 4.6. At 25 rps, the same hardware costs $0.87. Same GPU. Same model. A 17.5× cost ratio between λ=1 rps and saturation. Organizations plan multi-figure monthly LLM infrastructure budgets using calculators (Helicone, LiteLLM, YourGPT, llm-prices.com) and academic analyses [1, 2] that treat utilization as a static, user-supplied number (100% in Pan et al., 80% in SLaM) and therefore miss the load-driven cost spread entirely. This paper measures the error, formalizes a better cost model, and releases the tool. The prevailing approach to cost estimation treats this as simple arithmetic: divide GPU rental cost by theoretical token throughput, compare to API per-token pricing, and identify a break-even point. This methodology underpins the majority of publicly available LLM cost calculators—Helicone, LiteLLM, YourGPT, llmprices.com, and others—as well as recent academic analyses including Pan et al. [1] and Irugalbandara et al. [2]. The fundamental flaw in this approach is the assumption of full or near-full GPU utilization. In production enterprise deployments, GPU utilization for LLM inference is governed by the interaction of offered request rate, latency SLO constraints, model architecture, and deployment configuration (tensor parallelism, replica count). A GPU serving a dense 8B-parameter model at an offered rate of 10 rps with a 200ms TTFT SLO operates at a fundamentally different utilization—and therefore cost-per-token—than the same GPU serving the same model at 200 rps with a relaxed 500ms SLO. Industry practitioners have observed this qualitatively—noting that a GPU at 10% load transforms a $13/MTok cost into $130/MTok [11]—but no existing tool or publication has systematically quantified the re-

Contributions. This paper makes four contributions: 1. We demonstrate empirically that effective cost-per-million-tokens varies by up to 36.3× on identical hardware (17.5–36.3× across configurations) as a function of offered request rate λ, using 42 systematic vLLM benchmark runs (6 server configurations × 7 arrival-rate levels; 3 models × 2 precisions) on H100 GPUs. 2. We formalize a concurrency-aware cost attribution formula that models effective cost as a function of hardware, model architecture, quantization, offered arrival rate, and SLO—treating λ as the empirical sweep axis for operator-facing cost attribution, rather than as an optimization variable as in systems-scheduling frameworks (Vidur [3], Mélange [4], SageServe [5]). 3. We show that existing break-even analyses systematically produce misleading crossover points by ignoring utilization dynamics, and we provide corrected, utilizationadjusted crossover curves. 4. We release vllm-cost-meter, an opensource real-time cost estimation tool backed by empirical benchmark data, and validate it with live deployment on H100 hardware. To our knowledge, no other open-source tool computes offered-load-adjusted dollars-permillion-output-tokens in real time against a live LLM inference server; vllm-cost-meter is built for practitioners who need a defensible cost num2

ber under their own traffic rather than a vendor- reduction. SageServe [5] handles diurnal traffic patterns with ARIMA-based capacity forecastpublished benchmark. ing and differentiated SLAs; Llumnix [33] complements this with runtime rescheduling of in2 Related Work flight requests across replicas for tail-latency and utilization control. Building on the continuous2.1 Token-Volume Cost Models batching foundations established by Orca [30], The majority of LLM cost estimation tools and Splitwise [25], DistServe [26], and Sarathiacademic analyses employ a token-volume model: Serve [27] attack the throughput–latency tradeT C = Θmax × PGPU , where T is token count, Θmax off from the scheduler/engine side: by disaggreis theoretical maximum throughput, and PGPU gating prefill and decode across GPUs, by disaggregating the KV cache across a dedicated pool is the GPU hourly price. Pan et al. [1] present a break-even frame- (Mooncake [31]), or by chunking prefill, they work comparing on-premise TCO against com- raise the achievable Θmax at a given SLO. These mercial API subscription fees. Their model are complementary to our contribution: they assumes full utilization during 160 operational optimize the engine ceiling at a fixed operathours per month, and their accompanying calcu- ing point, while Ceff = f (H, M, Q, λ, L) charlator accepts model selection, GPU selection, and acterizes how cost per token traverses the enGPU count—but no utilization, concurrency, or tire λ axis below that ceiling. §5.9 provides workload parameters. The SLaM framework [2] measured cross-hardware and TP-scaling eviassumes 80% utilization on a single T4 GPU dence that complements Vidur’s simulated costand computes a direct cost-per-token compari- overhead predictions and Mélange’s allocationobjective premises. son against GPT-4 API pricing. A survey of 15+ public calculators (Helicone, BentoML’s llm-optimizer [13] supports concurLiteLLM, YourGPT, llm-prices.com, Mem0.ai, DocsBot, and others) confirms that over 90% ac- rency sweeps with Pareto frontier visualization cept only model name and token counts as in- but does not output cost-per-token. SemiAnalyputs. The LLM Inference TCO Calculator [12] sis InferenceX (formerly InferenceMAX) [14] prois the most comprehensive existing web-based vides nightly cross-hardware benchmarks with tool, accepting concurrent users, deployment cost attribution at a market-aggregate level but shape, and latency factors as inputs; however, focuses on hardware-vendor reference configuits throughput values are heuristic presets rather rations rather than same-GPU load/cost ecothan empirically measured, and it explicitly dis- nomics; vllm-cost-meter measures the operator’s own server under their own trafclaims queueing theory modeling. fic. These are complementary, not competing. GuideLLM [15] from the vLLM project auto2.2 Systems-Level Cost Optimization mates concurrency sweeps but outputs only laSeveral systems papers model concurrency and tency and throughput metrics without cost comdeployment shape as optimization variables. putation. NVIDIA’s TCO methodology [10] outVidur [3] (MLSys 2024) searches over tensor lines a clean benchmarking-and-sizing workflow parallelism, pipeline parallelism, GPU SKU, but does not release a standalone open-source and scheduling policy to optimize QPS-per- calculator. dollar subject to latency SLO constraints— These are systems optimization or benchmarkdemonstrating that using the wrong workload’s optimal configuration results in up to 2× cost ing tools—their contribution is finding cheaper overhead. Mélange [4] formulates GPU alloca- configurations or measuring performance, not tion as a cost-aware bin-packing ILP across het- providing a cost estimation methodology that erogeneous GPU types, achieving up to 77% cost connects offered load to dollars for practitioners. 3

2.3

posed $/M-tokens as a benchmark metric.2 No prior work combines all four: a parameterized cost function calibrated across architectures and hardware families, a live-server instrument that reads the operator’s own Prometheus metrics, and open data and tool. This paper does.

Inference Economics

Erdil [6] builds a theoretical roofline model for LLM inference economics, producing Pareto frontiers of serial speed versus cost-per-token, explicitly modeling how tensor parallelism trades utilization for speed. WiNGPT (Zhuang et al.) [7] is the closest prior empirical work: they demonstrate a 2.6× cost reduction from concurrency 8 to 48 on A800 GPUs using vLLM, and articulate three empirical principles—diminishing marginal cost, diminishing returns to scale, and an optimal cost-effectiveness zone—that are qualitatively consistent with the Ceff (λ) curves reported here. However, their sweep starts at concurrency 8 (already partially batched, missing the steep cold-start regime below λ=5 that drives our headline spread), covers only medical-domain models on a single hardware family, and releases neither code nor data. We generalize across three architectural classes, two hardware families (H100 NVL, A100 PCIe), provide a parameterized five-variable cost function, and release an open operational meter. Concurrent request scheduling in LLM serving has also been studied with analytical cost models [8], but with a focus on scheduler-level prefill/decode cost rather than per-deployment operational cost measurement.

2.4

3

The Concurrency-Aware Cost Framework

3.1

Problem Formulation

We define the effective cost-per-million-tokens Ceff as a function of four swept variables with one held implicit: Ceff = f (H, M, Q, λ ; L)

(1)

The semicolon separates the four axes we empirically sweep (H, M , Q, λ) from the latencySLO parameter L, which we hold implicit in the sweeps reported in this paper (all configurations are measured without an SLO cutoff, and L would enter as an admission-control filter on the measured distribution). Incorporating L as an active sweep dimension is future work; §6.9 discusses the scope implications. where: • H = Hardware specification (GPU type, count, memory, interconnect)

Gap Statement

• M = Model architecture (parameter count, dense vs. MoE, attention heads)

Prior work splits cleanly into four camps, none of which closes the loop for a practitioner. Token-volume calculators and break-even analyses [1, 2] treat utilization as a user-supplied input. Systems-optimization frameworks [3, 4, 5] find cheaper configurations but do not emit $/Mtokens for a given deployment. Benchmark harnesses and industry dashboards [13, 15, 14, 10] measure throughput or publish market-aggregate TCO but do not instrument an operator’s own server. Prior work has modeled inference cost theoretically [6], observed offered-load–cost behavior empirically in narrow domains [7], benchmarked engines under SLO constraints without emitting cost-per-token [13, 15], and pro-

• Q = Quantization precision (FP16, FP8, INT8, INT4) • λ = Offered request arrival rate (requests/sec; average and peak). The number of in-flight requests resident in the engine is an emergent quantity governed by Little’s Law (N ≈ λ · W for mean residence time W ), not a directly swept variable in this study • L = Latency SLO (target TTFT in milliseconds) 2

kubernetes-sigs/inference-perf Issue #139 (July 2025) scopes exactly this feature; vllm-cost-meter can be read as a standalone realization of that RFE.

4

production operator would observe.

Offered load vs. in-flight concurrency. Throughout this paper we use λ to denote the operator-controlled offered request rate (requests/sec arriving at the server), not the inflight batch size of the inference engine. The two quantities are related but distinct: in-flight concurrency is what the GPU actually batches at any instant and is set by λ together with perrequest residence time W via Little’s Law. The methodology is “concurrency-aware” in the sense that it exposes the operator’s offered-load operating point as a first-class input to cost—rather than assuming a fixed GPU utilization—and reports how the realized in-flight batch (and therefore Ceff ) responds.

3.2

The Utilization Function

The core insight is that GPU utilization U is not an independent parameter to be assumed (as in prior work) but is itself a dependent variable determined by the interaction of H, M , Q, λ, and L:

U (λ, L | H, M, Q) =

Θachieved (λ, L) Θmax (H, M, Q)

(2)

At low offered load, GPU compute units are underutilized—the GPU waits for memory transfers and cannot fill its execution pipeline. As offered load increases, realized in-flight concurrency improves utilization, but latency also increases due to queuing and batch contention. The latency SLO L acts as a constraint that caps the achievable utilization. The effective cost then becomes:

Variables held constant. Five additional parameters could in principle enter (1) but are held constant here to isolate the effect of λ. (i) Inference engine: vLLM defaults with continuous batching and PagedAttention [9]; cross-engine benchmarks [3, 14] report that engine choice shifts Θmax but not the shape of U (λ, L). (ii) Pricing mode: PGPU is the Azure on-demand list price; reserved and spot instances multiply Ceff by a roughly constant scalar (0.3–0.7×) and do not alter offeredload–cost dynamics. (iii) Input:output token ratio: uniform 512:256, drawn from the median of ShareGPT-derived distributions used in serving workloads [4]; variable ratios affect prefill-vs-decode balance but not the underutilization penalty. (iv) Parallelism strategy: tensor-parallel degree (TP=1 for dense models, TP=2 for Mixtral) is treated as part of H. (v) Request arrival distribution: Poisson at rate λ. Production LLM traces follow Gamma or Weibull inter-arrival distributions with coefficient of variation exceeding 1 [34, 35], making Poisson (CV=1) a regularity assumption. Our supplementary Gamma probe (§5.7) found negligible cost impact at CV=2 on C4, but was limited to one configuration; bursty real-world workloads may amplify the penalty reported here rather than diminish it. We therefore interpret Ceff under these held-constant factors as a conservative lower bound on the cost dispersion a

Ceff =

PGPU × 106 3600 × Θachieved (λ, L)

(3)

This differs fundamentally from the tokenvolume model, which assumes: Cnaive =

PGPU × 106 3600 × Θmax (H, M, Q)

(4)

The ratio Ceff /Cnaive = Θmax /Θachieved = 1/U captures the underutilization penalty—the multiplicative factor by which naive estimates understate true costs.

3.3

Architecture-Dependent Cost Behavior

We hypothesize—and demonstrate experimentally in Section 5—that the shape of the utilization function U (λ, L) differs systematically between dense and mixture-of-experts (MoE) architectures. Dense models exhibit more predictable scaling with realized in-flight batch size because all parameters are activated for every token. MoE models activate only a subset of parameters per token, leading to different memory 5

Table 1: Hardware configuration.

access patterns and batch efficiency characteristics. This means a single cost formula calibrated on one architecture will produce incorrect estimates for another, even on the same hardware. Model architecture must be a first-class variable in any cost estimation framework.

3.4

Component

Specification

GPU VRAM Interconnect Host Cloud SKU Cost

2× NVIDIA H100 NVL 96GB 96 GB HBM3 per GPU (192 GB total) NVLink 900 GB/s 80 CPU cores, 640 GB RAM Standard_NC80adis_H100_v5 $13.96/hr node ($6.98/GPU)

The Crossover Analysis

Given the utilization-adjusted cost Ceff , we de- Software environment. All benchmarks were fine the self-host vs. API crossover as the offered run on a single Azure Machine Learning H100 NVL node (NVIDIA driver 535.274.02, arrival-rate threshold λ∗ where: CUDA 12.8 runtime, Linux kernel 6.8.0-1044azure). To avoid ABI conflicts between the ∗ Ceff (λ , L | H, M, Q) = CAPI (model_tier) (5) two serving stacks, we maintain two isolated Python virtual environments: a vllm-bench This crossover point shifts based on all five in- environment pinned to vLLM 0.19.0 on Pyput variables. Prior work reports a single break- Torch 2.10.0+cu128, and an llm-bench eneven (e.g., “self-hosting is cheaper above 50M to- vironment pinned to SGLang 0.5.10 on Pykens/month”), but the actual crossover is a sur- Torch 2.9.1+cu128. Both stacks are compiled face in the (λ, L, H) space, not a single point. against CUDA 12.8. The benchmark harness (commit 1c40823) drives both engines through a unified sweep runner that enforces identical re4 Experimental Setup quest streams, sampling parameters, and measurement windows.

4.1

Hardware Configurations

All experiments are conducted on a single compute node equipped with two NVIDIA H100 NVL 96GB GPUs (Standard_NC80adis_H100_v5, Azure ML). This configuration was chosen deliberately to demonstrate that the framework can be validated with modest, widely-available infrastructure—a single cloud node at $13.96/hr on-demand—without requiring multi-node clusters or specialized hardware. Dense models (Llama 3.1 8B) and ultra-sparse MoE models (Qwen3-30B-A3B) each use a single H100 (TP=1); the sparse MoE model (Mixtral 8x7B) uses both H100s with tensor parallelism (TP=2). Single-GPU configurations (C1–C4) are costed at $6.98/hr per GPU, which is $13.96/hr/node amortized across the two H100s under the assumption that the second GPU is independently utilized by a co-tenant; a dedicated-node operator should double C1–C4 numbers. This denominator choice is made explicit in §5.

4.2

Model Selection

We select three models representing a gradient of architectural sparsity, all well-supported by the vLLM inference engine: The dense model (Llama 3.1 8B Instruct [17], 8B active parameters) runs on a single H100 GPU and represents the most common singleGPU deployment. Qwen3-30B-A3B-Instruct [16] (released July 2025) is an ultra-sparse MoE that activates only 3B of its 30B parameters per token, also fitting on a single H100. Mixtral 8x7B Instruct [18] activates 12.9B of its 46.7B parameters and runs on both H100 GPUs with TP=2. This three-model selection creates a sparsity gradient—dense (8B/8B), ultra-sparse (3B/30B), and moderate-sparse (12.9B/46.7B)— enabling analysis of how architectural sparsity affects the offered-load–cost relationship. These three models are selected to span architectural sparsity regimes, not to represent the frontier of 6

Table 2: Models selected for benchmarking. Model

Total

Active

Arch

Precision

Llama 3.1 8B Instruct Qwen3-30B-A3B Instruct Mixtral 8x7B Instruct

8B 30B 46.7B

8B 3B 12.9B

Dense Ultra-sparse MoE Sparse MoE

FP16, FP8 FP16, FP8 FP16, FP8

model quality at submission time: newer releases decode kernel keeps the GPU nominally busy shift the absolute serving economics but not the while it is arrival-limited at batch size 1), so methodological point—that hardware, model arwe never treat it as a proxy for U chitecture, quantization, offered load, and the la• GPU Memory Utilization—VRAM occutency SLO must enter cost estimation as firstpancy class inputs. Each configuration runs 100 warmup requests (discarded) followed by 500 measured re4.3 Benchmarking Protocol quests. We use synthetic random-token prompts The headline numerical results in this paper with fixed input length 512 tokens and outcome from vLLM [9], which fixes the pricing de- put length 256 tokens, with per-run random nominator and eliminates cross-engine variance seeds to prevent cross-run prefix cache contamifrom the reported figures.3 We sweep across nation. Prefix caching is explicitly disabled on all seven offered arrival-rate levels: servers (–no-enable-prefix-caching) to isolate offered-load effects from cache artifacts. Arrivals follow a Poisson process (–burstiness 1.0). λ ∈ {1, 5, 10, 25, 50, 100, 200} requests/sec Deliberate Design Choices. Several methodological decisions deserve explicit justification, as they may draw scrutiny but are intentional. Random synthetic tokens instead of ShareGPT or conversational traces. We use uniform random token sequences rather than real chat datasets for three reasons: (1) ShareGPT and similar datasets contain repeated system prompts and shared prefixes that produce prefix-cache hits even with APC nominally disabled, biasing throughput upward in ways that don’t generalize across deployments; (2) fixed-length synthetic workloads give exact control over the input/output length distribution, enabling perfectly reproducible results that any reader can replicate without access to proprietary data; and (3) the cost dynamics we study—how GPU utilization changes with offered request rate and the realized in-flight batch it creates—are governed by the batch-filling behavior of the scheduler, not by prompt semantics. A reviewer who argues that “real workloads would show different

For each (λ, H, M, Q) configuration, we measure: • TTFT (Time to First Token)—P50, P90, P99 • TPS (Tokens Per Second)—aggregate output throughput • E2E Latency—total request completion time (P50, P90, P99) • GPU occupancy—nvidia-smi utilization.gpu, the wall-clock fraction of time a kernel is resident. Reported for completeness only; it is not the economic utilization U =Θachieved /Θmax defined in §3 and can read high even at λ=1 (a single resident 3

SGLang [28] runs share the same apparatus (see §4.1 for the two pinned venvs, vllm-bench and llm-bench) and the S1–S6 rows in the companion repository reproduce our methodology on that engine; we exclude them from headline numbers to keep the pricing denominator single-engine. TensorRT-LLM is out of scope. Crossengine benchmarks [3, 14] report that engine choice shifts Θmax but not the shape of U (λ, L), so a single engine is sufficient to characterize the underutilization-penalty structure we report.

7

MIT license; see §7.1.

numbers” is correct: production workloads with prefix sharing and caching enabled would show lower penalties. Our numbers are intentional upper bounds. Prefix caching and speculative decoding disabled. We disable automatic prefix caching (–no-enable-prefix-caching) to isolate pure offered-load effects on GPU utilization. Enabling APC would artificially inflate throughput for repeated prefixes, masking the underlying relationship between request rate and compute efficiency. Our goal is to measure the cost structure, not to optimize it. Note that APC savings accrue in regimes where prefill is a meaningful fraction of batch time (moderate-to-high λ with shared prefixes); at the near-idle λ=1 rps extremum that drives the headline underutilization penalty, the cost is primarily GPU-rental arithmetic against a trickle of tokens, and is structurally insensitive to prefix-cache state regardless of workload. Poisson arrivals (–burstiness 1.0). This is the standard queueing-theory arrival model for open-loop load generation and is used by the majority of LLM serving benchmarks including vLLM’s own benchmark harness. It models independent, memoryless arrivals with no request batching at the client side; burstier-than-Poisson production arrivals are treated empirically in §5.7. These design choices—random tokens, no prefix caching, no speculative decoding, no chunkedprefill tuning—deliberately establish a conservative performance floor. Production workloads with repeated system prompts, shared prefixes, or optimized inference pipelines would achieve higher throughput and therefore lower cost-pertoken than reported here. All cost figures in this paper should be interpreted as upper bounds on achievable cost; real-world deployments with standard optimizations enabled would only improve on these numbers. Configuration matrix: 1 hardware node × 3 models × 2 precisions = 6 server configurations. Each swept at 7 arrival-rate levels produces 42 benchmark runs (approximately 4.5 hours of total GPU time, under $65 at on-demand pricing). All code, data, and a reusable cost estimation tool are released as open-source under an

Raw saturation vs. goodput. We report Θmax as the sustained tokens-per-second achievable at peak offered load under a fixed I/O shape with no latency bound imposed. In the goodput literature [22, 14], this is the raw saturation throughput, distinct from goodput, which would additionally constrain on a TTFT/TPOT/E2EL SLO. Raw saturation is an upper bound on goodput: any meaningful production SLO will reduce the achievable arrival rate, and therefore the effective tokens per dollar. We intentionally measure raw saturation to isolate the engineplus-hardware ceiling from SLO-policy choices; goodput-bounded extensions are a natural direction for community contribution, and our companion tool’s workload_protocol schema is designed to accommodate crowd-sourced curves with self-declared SLO bounds.

4.4

Derived Metrics

From raw measurements, we compute: • Effective cost-per-million-tokens: Ceff = PGPU ×106 TPS×3600 ($/M output tokens). Throughout, TPS is aggregate output-token throughput, so every Ceff figure is dollars per million output tokens; this matches the output-token basis of the API list prices we compare against (§5.6). Input (prefill) tokens consume compute but are not in the denominator. • Utilization rate: U =Θachieved (λ)/Θmax , the throughput ratio defined in §3 (distinct from the nvidia-smi GPU-occupancy metric) • Underutilization penalty: Ceff /Cnaive at each operating point • Crossover arrival rate: the λ where Ceff equals API pricing for GPT-5.5, Gemini 3.1 Pro, and Claude Sonnet 4.6

5

Results

We report results across all 42 benchmark configurations. GPU cost follows the denominator convention introduced in §4 (and Table 1): 8

of peak-throughput cost. 10 The naive cost model underestimates selfhosted costs by 17.5–36.3×. That is the 1/U 17.5x-36.3x underutilization penalty at  = gap between a utilization-naive estimate (any calculator that assumes the GPU runs at its Θmax ) and what you actually pay when the en1 gine is arrival-limited at 1 rps. Prior informal estimates put the error at 3–10×; ours shows they were themselves off by another 3–10×. Even at 10 rps—an arrival rate most practitioners would 10 10 10 not call “low”—the naive model is still wrong by Offered load  (req/s) 2.5× on dense Llama and 3.8× on ultra-sparse Figure 1: Ceff vs. offered load λ across six H100 Qwen. The naive model is not merely optimistic; configurations (log-log); solid lines are FP16, it omits the dominant cost term. dashed FP8. Effective cost drops by roughly 20× over the first 25 rps, then flattens as the 5.2 Architecture-Dependent Scaling engine saturates. Calculators that omit λ and assume a fixed utilization or peak throughput The three FP16 solid lines in Figure 1 isoAt peak therefore misreport cost by close to an order of late the architectural families. throughput, Llama 3.1 8B achieves 6,238 tok/s magnitude in the low-λ regime. At λ ≥ 50 the Qwen3-30B-A3B achieves server is queue-limited; throughput and cost use ($0.311/MTok), 5,319 tok/s ($0.364/MTok), and Mixtral 8x7B completed-request statistics within the measurement window. Note the architecture signal: at on TP=2 achieves 4,454 tok/s at $0.871/MTok; λ=1 the two single-GPU FP16 models (Llama, the three curves converge to within 3× at Qwen) sit at an identical ≈$7.60/MTok—idle saturation. Qwen’s lower throughput than Llama—despite throughput is arrival-limited (≈255 tok/s) and activating only 3B parameters—reflects two hence model-independent—so Mixtral’s ∼ 2×higher idle cost reflects its two-GPU ($13.96/hr) structural costs: (1) MoE routing dispatch overhead per token, and (2) the full 30B paramdenominator, not its parameter count. eter set occupying VRAM, constraining memory bandwidth. The shape of the scaling curve $6.98/hr per GPU for single-GPU models (C1– is, however, comparable between the two: the C4, assuming co-tenant utilization of the second ultra-sparse Qwen model’s cost cliff from λ=1 H100) and $13.96/hr for Mixtral (C5–C6, TP=2, to λ=10 rps (9.4× drop) is marginally shalboth GPUs occupied). The shape of Ceff (λ) is lower than dense Llama’s (9.7× drop), indicatunchanged under the alternative dedicated-node ing that routing overhead does not meaningfully denominator; only the absolute level of C1–C4 shift the per-request amortization curve at this scale. The cost-structure differences between doubles. dense and ultra-sparse MoE therefore manifest primarily as level shifts (higher idle cost, differ5.1 Offered Load vs. Effective Cost ent saturation floor) rather than slope shifts, but Figure 1 shows Ceff across all six con- are large enough in magnitude that a single cost figurations. At λ=1 rps, effective costs formula calibrated on dense models will systemrange from $7.60/MTok (Llama FP16/FP8) atically misestimate MoE costs. This active-parameter dominance holds in the to $15.25/MTok (Mixtral FP16)—the near-idle GPU cost amortized over a trickle of tokens. As batched decode regime measured here (λ ≥ 1 rps, offered load increases, Ceff falls precipitously: by batch sizes reaching hundreds), where per-token λ=25 rps all single-H100 models are within 20% arithmetic scales with active parameters while Effective cost ($ / M output tokens)

Llama 3.1 8B FP16 Llama 3.1 8B FP8 Qwen3-30B-A3B FP16

0

1

Qwen3-30B-A3B FP8 Mixtral 8x7B FP16 Mixtral 8x7B FP8

2

9

Peak sustained throughput (output tokens / s)

12000

rather than total model size, appear to govern the saturation cost. We treat this as suggestive— it rests on a single architecture pair with a ∼12% margin—and defer the larger-dense (e.g. Llama 3.1 70B) plus third-sparsity-ratio validation that would license a general claim to §7.

Inversion at FP8: Qwen3-30B-A3B outperforms Llama 3.1 8B (9,271 vs 8,155 tok/s; $0.209 vs $0.238/MTok)

FP16 FP8

10000

+74% +31% +69%

8000

6000

4000

2000

5.4

The Underutilization Penalty

0 Llama 3.1 8B (Dense)

Qwen3-30B-A3B (Ultra-sparse MoE)

Mixtral 8x7B (Sparse MoE)

Table 3: Underutilization penalty and latency tails for the dense reference model (Llama 3.1 8B FP16, single H100, $6.98/hr). Cnaive = $0.311/MTok at peak throughput. TTFT and TPOT percentiles are measured (vLLM benchmark_serving); at λ ≥ 50 the server is queue-limited, so throughput, cost, and latency use completed-request statistics within the measurement window. At the λ=50 cost weight-memory movement is amortized across floor both tails exceed the example SLO of §6.4 the batch. At small batch sizes or in prefill- (TTFTp99 ≤ 300 ms, TPOTp99 ≤ 50 ms). dominated regimes, total parameters drive memory bandwidth cost and the active-vs-total disλ TTFTp50 TTFTp99 TPOTp99 Ceff Penalty (ms) (ms) (ms) ($/MTok) tinction narrows [18]—a caveat that applies to (rps) 1 37.9 54.3 7.0 7.60 24.4× the λ=1 edge of our sweep and to any deploy5 44.1 713.3 16.4 1.54 5.0× ment with long prompts and short outputs. 10 49.4 84.5 12.3 0.79 2.5× Figure 2: FP8 gives the dense model +31% peak throughput and the two MoE models +69% (Mixtral) and +74% (Qwen)—a roughly 2.2–2.4× larger win for the MoE architectures. Quantization behaves as an MoE-first optimization that also helps dense models.

5.3

25 50 100 200

Quantization Impact

FP8 quantization reduces Ceff at every arrival-rate level (Figure 2). Peak throughput gains are: +31% for Llama 3.1 8B (6,238→8,155 tok/s), +74% for Qwen3-30BA3B (5,319→9,271 tok/s), and +69% for Mixtral 8x7B (4,454→7,524 tok/s). The larger FP8 gain for MoE models is consistent with their memory-bandwidth-bound profile: sparse activation requires many small, scattered parameter reads, and halving parameter precision directly doubles effective bandwidth. A notable emergent result: at peak offered load, Qwen3-30B-A3B FP8 ($0.209/MTok) is cheaper than Llama 3.1 8B FP8 ($0.238/MTok) despite having 30B total parameters. On this dense/ultra-sparse pair, the ordering inverts the common intuition that smaller models are cheaper to serve: active parameter count and memory-access pattern,

113.5 417.9 2,561 3,748

259.0 994.8 5,011 7,437

37.8 71.3 76.1 76.8

0.37 0.32 0.31 0.31

1.18× 1.02× 1.00× 1.00×

Table 3 quantifies the penalty for our dense reference model. At λ=1 rps, effective cost is 24.4× the naive estimate. The penalty declines non-linearly: by λ=25 rps it falls to 1.18×, and beyond λ=50 rps throughput approaches the hardware ceiling. Importantly, at λ ≥ 100 rps, throughput plateaus at ∼6,230 tok/s while TTFT P50 rises to 2,561–3,748 ms, indicating queue saturation. This defines the model’s saturation threshold —the arrival rate above which additional load degrades latency without improving cost. The latency tails confirm that this cost floor is a no-SLO floor: at the λ=50 minimumcost point TTFTp99 has already reached 994.8 ms and TPOTp99 71.3 ms—both well beyond the example production contract of §6.4 (TTFTp99 ≤ 300 ms, TPOTp99 ≤ 50 ms)—and the C2 repeat campaign shows TTFTp50 run-to-run CV rising

10

Llama 3.1 8B FP16

24.4x

5.0x

2.5x

1.2x

1.0x

1.0x

1.0x

35

Llama 3.1 8B FP8

31.9x

6.5x

3.3x

1.4x

1.1x

1.0x

1.0x

Qwen3-30B-A3B FP16

20.9x

4.3x

2.2x

1.1x

1.0x

1.0x

1.0x

Qwen3-30B-A3B FP8

36.3x

7.4x

3.8x

1.8x

1.3x

1.0x

1.0x

Mixtral 8x7B FP16

17.5x

3.6x

1.9x

1.1x

1.0x

1.0x

1.0x

Mixtral 8x7B FP8

29.5x

6.0x

3.1x

1.4x

1.1x

1.0x

1.0x

1

5

10

25

50

100

200

Effective cost ($ / M output tokens)

30

20 15

Penalty (x)

25

10 5

Request rate (req/s)

Figure 3: Underutilization penalty (idle-edge Ceff over saturation Csat ) by config and offered load. The penalty lives almost entirely in the λ=1 column (17.5–36.3×) and collapses to ≈1.0× by λ=50 across every config. Within each model the FP8 variant carries the larger penalty—quantization lowers the saturation floor more than the idle-edge cost—peaking at the darkest cell, 36.3× for Qwen3-30B-A3B FP8.

Saturation floor

1.6

≤

(SLA-infeasible)

SLA-feasible cost floor

1.4 1.2 1.0 ≤

0.8 0.6 ≤

0.4

≤

≤

≤

0.2 0.0 .1

3 ma

Lla

8B

FP

16 .1

Lla

3 ma

8B

FP

8

P1

BF

8

6 3B

-A3

en

Qw

3

B -30

en

Qw

P8

6

P1

FP

B-A

0 3-3

M

BF

x7

l8

a ixtr

BF

x7

l8

a ixtr

M

Figure 4: SLA-feasible cost floor (solid, perconfig color) vs. the unconstrained saturation floor Csat (grey, hatched) under a single fixed SLA: TTFT p99 ≤ 300 ms and TPOT p99 ≤ 50 ms. Annotations give the highest SLA-feasible offered load. Csat is unreachable under this SLA—it lands at λ=100–200 where TTFT p99 runs to multiple seconds (Table 4)—so the SLAto 8.73% at the same load (§5.8). An operator feasible floor, not Csat , is the cost an SLA-bound bound by such an SLO must back off to a lower operator actually pays. arrival rate, where Ceff is strictly higher; the raw-saturation Cmin reported here is therefore Operating a lower bound on goodput-constrained cost— 5.5 SLO-Conditioned Points equivalently, an upper bound on the tokensper-dollar an SLO-bound operator can achieve— The cost floors quoted so far are unconstrained exactly as the goodput-vs-saturation distinction saturation values: they are the cheapest C a eff in §4 and the operational workflow in §6.5 pre- config reaches at any λ. In production they are scribe.4 typically unreachable, because the latency-L ar-

Figure 3 extends this to all configurations. The maximum penalty is 36.3× for Qwen3-30B-A3B FP8 at λ=1 rps. Penalties at λ=1 rps range from 17.5× (Mixtral FP16) to 36.3× (Qwen FP8), converging to ≈1.0× by λ=50 rps across all configurations. The consistent pattern across architectures and precisions confirms that the underutilization trap is a structural property of discrete GPU rental cost, not an artifact of any specific model.

4

The λ=5 row’s tails (TTFTp99 713.3 ms, TPOTp99 16.4 ms) are inflated by a handful of outlier requests on this single, unrepeated run, with the bulk of the distribution unaffected (TTFT P50/P90 of 44.1/56.8 ms); monotone tail growth with load resumes by λ=10 (TTFTp99 84.5 ms, TPOTp99 12.3 ms). The P50, throughput, and cost columns are unaffected.

gument of Ceff = f (H, M, Q, λ, L) is not a free variable—it is a commitment. An operator who signs a service-level agreement (a B2B latency SLA, an internal interactive-product target) fixes L, and a fixed L caps the offered load the deployment may sustain before its tail breaches, which in turn pins the operating point on the Ceff (λ) curve. The SLA is therefore a price: tightening it forces a lower λ and a higher cost per token. To make this concrete on the existing corpus we evaluate one fixed SLA used as the running example in §6.4: TTFT p99 ≤ 300 ms and TPOT p99 ≤ 50 ms. For each config we take the highest λ in the 7-point sweep that satisfies both bounds and report its Ceff (Table 4, Figure 4). Two readings follow. First, the advertised floor is a fiction for any latency-bound deployment: Llama 8B FP16’s $0.311/MTok floor lives

11

Table 4: Cost of honoring a fixed SLA (TTFT p99 ≤ 300 ms, TPOT p99 ≤ 50 ms). λmax is the highest SLA-feasible offered load; the premium is the SLA-feasible cost over the unconstrained saturation floor Csat . Every Csat here lives at λ=100–200 with a multi-second TTFT p99 (2.2– 15.7 s), i.e. SLA-infeasible. Config Llama 8B FP16 Llama 8B FP8 Qwen3-A3B FP16 Qwen3-A3B FP8 Mixtral FP16 Mixtral FP8

SLA λmax $/MTok (rps) at SLA 25 25 25 50 10 25

Csat premium

0.368 0.311 0.337 0.238 0.411 0.364 0.263 0.209 1.666 0.871 0.732 0.515

1.18× 1.42× 1.13× 1.26× 1.91× 1.42×

Sonnet 4.6 ($15.00/M output tokens). At λ=5 rps it falls to $3.16/MTok, below all API tiers. The naive analysis concludes Mixtral self-hosting is economical at any traffic level; the adjusted analysis reveals a crossover threshold near 1.5–2 rps below which the lower-priced reference tiers (Gemini 3.1 Pro, Claude Sonnet 4.6) are actually cheaper. For Llama and Qwen models, the lower singleGPU cost means the crossover is below 1 rps— self-hosting is cost-effective even at minimal traffic, if the GPU is already provisioned—but only under the $6.98/hr co-tenant denominator of §4. Under the dedicated-node $13.96/hr rate the entire single-GPU Ceff (λ) curve doubles, roughly doubling the low-λ cost (e.g. Llama/Qwen idle rises from ≈$7.60 to ≈$15.20/MTok) and pushing the API crossover to a higher arrival rate. Operators on a dedicated node should read the C1–C4 numbers at the doubled rate. (The λ<1 region is extrapolated outside our 7-point sweep; the exact crossover location below 1 rps is a modeled continuation of the measured Ceff (λ) curve, not a directly observed operating point.) However, the effective cost at 1 rps ($7.60/MTok) is 24× the naive estimate, compressing the apparent cost advantage over the lowest-priced reference tier (Gemini 3.1 Pro, $12.00/M output) from 98% (naive) to roughly a third at λ=1 rps.5 An organization planning infrastructure on the naive model will be surprised to find that at realistic low-traffic operating points, most of the expected advantage evaporates.

at λ=200, where TTFT p99 is 7.4 seconds— no interactive product ships that. Its cheapest shippable cost under the SLA is $0.368/MTok, at λ=25. Second, the SLA premium over the (already-unreachable) floor ranges from 1.13× to 1.91×, largest on Mixtral 8x7B FP16, whose strict per-GPU memory budget forces the SLAfeasible load down to λ=10. This is a deliberately conservative slice of L’s effect: it compares two points on the same curve. Loosening the SLA toward batch-grade latency walks the operating point rightward toward Csat , and tightening it past our interactive example walks it leftward into the steep part of Figure 1, where the penalty is the 17.5–36.3× already reported. The practical statement is that the SLA an operator commits to in a contract is, through λ, a directly priced quantity—and one no token-volume calCaveat on the API comparison. Figure 5 culator can express. compares self-hosted Ceff against list outputtoken prices for the commercial APIs, to keep 5.6 Corrected Crossover Analysis the comparison reproducible from public pricing Figure 5 plots Ceff (λ) against commercial API pages. Three real-world discounts are not apreference prices. Under the naive model, all plied and would move the API lines down: (i) list configurations appear permanently cheaper than API alternatives—Llama FP8 at $0.238/MTok is over 50× cheaper than Gemini 3.1 Pro ($12.00/M output tokens [21]). Under the utilizationadjusted model, the picture differs. At λ=1 rps, Mixtral FP16 costs $15.25/MTok—more expensive than Claude

5

At the specific λ=1 operating point: 1−7.60/12.00 ≈ 37%. The exact figure moves with the tier’s list price, which changes on vendor schedule—and under the dedicated-node denominator (≈$15.20/MTok idle) the advantage inverts against the $12 tier and vanishes against the $15 tier. The qualitative point—expected savings compress sharply at low-traffic operating points—is what matters for infrastructure planning.

12

input-token prices are typically ∼5× lower than output prices, so a workload whose output:input ratio is smaller than our 512:256 assumption has a lower blended API cost; (ii) prompt-caching tiers now offered by Anthropic, OpenAI, and Google can reduce cached-input token cost by roughly an order of magnitude for workloads with substantial prefix sharing (system prompts, RAG preambles, agentic loops); (iii) batch APIs (∼50% off) and volume/enterprise contracts further compress the effective API price. In all three cases the comparison in Figure 5 is conservative in favour of self-hosting—applying these discounts would push the crossover threshold to higher λ, not lower.

5.7

prefix-free inputs, turning APC on adds overhead without adding any cache hits. The direction reverses on production workloads with real prefix sharing—shared system prompts, agentic loops, RAG preambles—where APC, KV-cache disaggregation across tiers (Mooncake [31]), and KV-cache reuse systems (LMCache [32]) deliver substantial savings; our random-token protocol deliberately removes that effect to isolate the offered-load cost structure. The point we need for this paper is unchanged: the main-sweep Ceff (λ) numbers are not artifacts of APC state, and the conservative-lower-bound framing in §4 is supported by data. I/O shape (Figure 7). Holding the arrival

Sensitivity to Held-Constant Fac- process Poisson and APC off, we replaced tors the 512:256 chat shape with a RAG shape

§3 enumerated five factors held constant in the main sweep: engine, pricing mode, input:output ratio, parallelism, and arrival distribution. Two of these—the I/O ratio and the arrival distribution—admit direct empirical probes on the same hardware, as does a third heldconstant protocol choice: prefix caching (disabled in the main sweep). We ran a targeted supplementary sweep across these three factors to quantify how far the headline underutilization penalty generalizes when those knobs are unfrozen. We report the results here rather than relegating them to an appendix because they materially bound the scope of our headline claim. Prefix caching (Figure 6). On our randomtoken workload no two prompts share a prefix, so vLLM’s automatic prefix cache (APC) [9] cannot hit and therefore cannot save any prefill compute—its hit-rate is ∼0 by construction. What the data actually show is the cost of running APC’s hashing and block-bookkeeping path with no corresponding savings: across all offeredload levels for C2 (Llama 3.1 8B FP8) and C4 (Qwen3-30B-A3B FP8), the pc-on/pc-off cost ratio lies in [0.79, 1.11] with a median near 1.00, and in the worst tail—C2 at λ ∈ {100, 200}— APC-on is up to +11% more expensive than APC-off. That is the correct direction: on

(4096:1024) and an agentic shape (1024:4096) at λ∈{1, 25, 100}. The RAG/chat cost ratio is non-monotonic in λ: at λ=1 RAG is actually cheaper than chat (the long 4K-token prefill raises effective tokens-per-second at a nearidle GPU, amortising the fixed cold-start cost over more tokens and lowering $/MTok). At the mid-load λ=25 RAG is roughly 2.4–2.7× the chat baseline (C4 Qwen 2.39×, C2 Llama 2.73×), and at the saturation-adjacent λ=100 it peaks at 3.2–3.7× while TTFT p50 is pushed into the hundreds of seconds (queued prefill on 4Ktoken prompts). The agentic shape—dominated by output tokens—lands between chat and RAG throughout. Crucially, the shape of the Ceff (λ) curve is preserved across I/O mixes: every shape still exhibits the concave fall-off from cold-start cost to saturation that motivates the framework. What changes is the y-axis offset, not the qualitative penalty structure. The underutilization story is robust to I/O mix; the absolute cost at a given λ is not. Arrival burstiness. Production LLM traces exhibit bursty, non-Poisson arrivals [34, 35]. On C4 chat with APC off, we reran λ∈{10, 50, 100} under Gamma inter-arrivals at CV=2 (–burstiness 0.25) against a matched Poisson (CV=1) baseline—same harness version,

13

same 120 s time-based measurement window, same prefix-cache and I/O-shape settings, varying only the inter-arrival distribution. The effect is negligible at all three arrival rates: Gamma-CV=2 is 0.3% cheaper at λ=10 (0.997× ratio), 0.5% cheaper at λ=50 (0.995×), and 0.8% more expensive at λ=100 (1.008×). All three deltas sit well inside run-to-run noise observed across repeat measurements of the same configuration. The practical reading is that vLLM’s continuous-batching scheduler absorbs a factor-of-two increase in arrival-process variance without a measurable cost signature at these arrival rates: the KV-cache and the scheduler’s batch-formation window are large enough, relative to the mean inter-arrival gap, that short bursts do not meaningfully displace the steady-state packing a Poisson stream would achieve. A production workload whose arrivalprocess CV deviates from 1.0 by this much therefore neither amplifies nor compresses the headline underutilization penalty in any material way; we preserve our conservative-lower-bound characterization of Ceff under Poisson assumptions. Scope: the Gamma probe is run on C4 only; the burstiness-invariance claim is therefore bounded to this configuration. A spot-check on ultra-sparse MoE at TP>1 (where routing dispatch interacts with the scheduler differently) is deferred to future work. Variable-length arrivals. The three probes above use fixed-length I/O shapes; a natural concern is whether the offered-load cost cliff survives when prompt lengths vary within a single run, as they do in production traffic. We generated a synthetic variable-length dataset with log-normal distributed input and output lengths (input median ∼400 tokens, p10/p90 = 120/906; output median ∼200 tokens, p10/p90 = 68/408) and swept λ ∈ {1, 10, 50, 100} on C2 and C4. The result is unambiguous: the load-driven spread under variable-length arrivals is wider than under fixed 512:256—39.3× on C2 (vs. 31.9× fixed) and 47.6× on C4 (vs. 36.3× fixed). At λ=1, variablelength Ceff matches the fixed-length baseline to within 0.1% ($7.59/MTok); at saturation

(λ=100) the variable-length workload achieves higher throughput because the scheduler can interleave short and long requests more efficiently. The cliff is steeper, not shallower, under realistic mixed-length traffic. This rules out the concern that fixed-length workloads artificially inflate the headline spread. Prefix caching with real cache hits. The prefix-cache probe in the preceding paragraph tested APC on random tokens where it cannot hit by construction. To test whether APC changes the shape of the Ceff (λ) curve when it actually helps, we ran a shared-prefix workload (10 distinct 200-token prefixes, each reused across ∼50 prompts, 312-token suffixes, 256-token output) on C2 and C4 at λ ∈ {1, 10, 50, 100} with APC on and off. At low offered load (λ ≤ 10) APC has negligible effect (ON/OFF ratio ≈ 1.00): the cache hit rate is high but throughput is memorylatency-bound regardless. At saturation (λ = 100) APC reduces Ceff by 20–22% (ON/OFF ratio 0.78–0.80): the saved prefill compute lets the scheduler pack more decode tokens into each batch. Crucially, both APC-on and APC-off exhibit the same characteristic cliff from λ=1 to λ=50, with spreads of 30–47× in both cases. APC shifts the cost level downward at high λ but does not change the shape of the offered-load cost curve. The underutilization story is robust to prefix caching; the absolute cost at saturation is not. Interpretation. None of the five probes—I/O shape, arrival burstiness, prefix caching (no-hit), variable-length arrivals, and prefix caching (real hits)—overturn the main finding. The headline underutilization penalty across the 7-point λ sweep is a structural property of the continuousbatching scheduler, not an artifact of the I/O ratio, arrival process, length distribution, or APC state used to measure it.

5.8

Measurement Stability

A natural reviewer question for any single-run benchmarking study is whether the reported spread is a structural property of the system or

14

an artifact of measurement noise. We address this with a focused repeat-run campaign on the C2 configuration (Llama 3.1 8B FP8 on a single H100 NVL, 1xGPU deployment). We re-ran the random-token workload at four offered-load points λ ∈ {1, 10, 50, 100} with three independent repeats each (12 runs total, distinct seeds per repeat), holding all other knobs fixed. The driver targets a ≥ 60 s steady-state measurement window: prompt count is 60 · λ clamped to [500, 6000] and warmup is 30 · λ floored at 100, producing {500, 600, 3000, 6000} prompts and {100, 300, 1500, 3000} warmups for λ ∈ {1, 10, 50, 100} respectively. At λ=50 and λ=100 the server is queue-limited and only ∼2,100 and ∼2,020 of the prompted requests complete within the measurement window; the CV computation uses the completed-request statistics reported by the harness. The vLLM server process stays resident across all 12 runs—a single server instance, so allocator and scheduler state are reused— which is consistent with how an operator would run back-to-back measurements and is the regime we are claiming stability for. Table 5 reports the coefficient of variation (CV) across repeats on throughput, effective cost, and TTFTp50 . Table 5: C2 repeat-run measurement stability, based on n=3 independent repeats per λ (12 runs total). Throughput and effective cost reproduce to within 0.31% CV across the three repeats at every tested arrival rate. TTFTp50 is similarly tight at under-saturated loads (λ=1, 10) but widens at saturation-adjacent loads (λ=50), reflecting queue-tail sensitivity to arrival randomness rather than instrument error. Ceff CV values are near-identical to throughput CV because GPU-hour cost is a constant multiplier.

is roughly four orders of magnitude larger than the measurement noise floor on Ceff when compared on the natural unit (cost-per-token in dollars): the between-run standard deviation on Ceff at λ=1 is $0.0004/MTok against a crossconfiguration spread of ∼$15/MTok. Stated as a CV, reproducibility is ≤ 0.31% on Ceff at every tested λ. Assuming the C2 noise profile transfers to other configurations—a simplification, since MoE routing dispatch and TP=2 collectives may carry different run-to-run variance— propagating the 0.31% CV would imply approximate 95% CIs of [17.3, 17.7]× for Mixtral FP16 and [35.9, 36.7]× for Qwen FP8. We report these as indicative rather than measured; perconfiguration repeat sweeps on the MoE headline configurations are the clean way to confirm them, and are noted as future work. The wider TTFTp50 CV at λ=50 is not a stability concern for the cost claims but a real secondary observation: tail latency becomes queue-tail-limited once the scheduler is saturation-adjacent, and a single run reports one realization of that tail. Operators who care about TTFT SLOs near saturation should plan for run-to-run spread at the percent level on tail latency, even when throughput and cost are reproducible to within ∼ 0.3%.

5.9

Cross-Hardware Validation (A100 80GB PCIe)

A second obvious reviewer question is whether the load-driven spread we report is a property of H100 NVL specifically—its HBM3 bandwidth, its native FP8 tensor cores, its NVLink fabric—or a structural property of the offeredload λ axis that would reproduce on a different GPU family. We repeat the core sweep on a second, architecturally distinct platform: λ Throughput (tok/s) Ceff ($/MTok) TTFTp50 NVIDIA A100 80GB PCIe (no native FP8, Gen4 (rps) mean CV% mean CV% CV% PCIe in place of NVLink, 1.94 TB/s HBM2e 1 255.4 0.01 7.5920 0.01 0.33 vs. H100’s 3.9 TB/s HBM3), at a list-price rate 10 2501.8 0.31 0.7750 0.31 0.22 of $3.67/hr/GPU on Azure’s NC_A100_v4 fam50 7576.6 0.11 0.2559 0.11 8.73 100 7419.5 0.12 0.2613 0.12 3.24 ily (roughly 1.90× cheaper than H100 NVL per GPU). We run the identical measurement harness—same prompt corpus, same arrival-rate Takeaway. The economic signal this paper ladder λ ∈ {1, 5, 10, 25, 50, 100, 200}, same tworeports—a 17.5× to 36.3× variation in Ceff — gate health check, same steady-state window— 15

across six paired configurations: Llama 3.1 8B FP16 and FP8 (vLLM, TP=1), Qwen3-30B-A3B FP16 and FP8 (vLLM, TP=1), Mixtral 8x7B FP16 (vLLM, TP=2), and Llama 3.1 8B FP16 (SGLang, TP=1). A third Mixtral condition— TP=4—was added on a 4×A100 node to exercise a tensor-parallel dimension the 2×H100 apparatus could not. This TP=4 run (M3) is reported separately in Result 4 and is excluded from the six-configuration 7.0–11.4× spread quoted above; its own spread of 12.3× is discussed in context. An eighth A100 configuration—Mixtral 8x7B FP16 on SGLang TP=2 (M2)—is present in the public 56-run corpus but excluded from Table 6 because the SGLang engine did not scale on this hardware: throughput flatlined near 647 tok/s across every λ, consistent with a PCIeinterconnect bottleneck on TP=2 Mixtral that did not manifest on the NVLink-connected H100. The vLLM TP=2 row (M1) is reported in its place.

Table 6: Per-configuration Ceff at saturation (Cmin , $/MTok) and spread (Cmax /Cmin , unitless × multiplier) on both hardware families. Cost-per-token at saturation is within a factor of two between platforms for every configuration— and on Mixtral TP=2 they round to the same $0.87/MTok at two decimal places (H100 $0.8706 vs. A100 $0.8688, a 0.2% difference rather than a fundamental equality)—while the spread is always narrower on A100. H100 NVL A100 PCIe Config.

Cmin

spr. Cmin

spr.

Llama 8B FP16 (vLLM) Llama 8B FP8 (vLLM) Qwen 30B FP16 (vLLM) Qwen 30B FP8 (vLLM) Mixtral FP16 TP=2 (vLLM) Llama 8B FP16 (SGLang)

0.31 0.24 0.36 0.21 0.87 0.63

24.4 31.9 20.9 36.3 17.5 12.0

11.4 9.8 8.2 11.3 9.3 7.0

0.35 0.41 0.49 0.36 0.87 0.57

Result 2: the FP8 MoE-first pattern reproduces on the architecture, but inverts on dense when the hardware lacks native FP8. On H100 NVL, FP8 quantization cuts Cmin by 23% on Llama (0.31 → 0.24), 42% on Qwen3-30B-A3B (0.36 → 0.21), and 40% on Mixtral TP=2 (0.87 → 0.52). On A100 80GB PCIe— which lacks native FP8 tensor cores on its SM80 silicon and therefore runs FP8 kernels through a software-emulated path—FP8 worsens Llama’s saturation cost (0.35 → 0.41, +17%) while still improving Qwen’s (0.49 → 0.36, −27%). The architectural signal—that MoE models benefit from FP8 more than dense models do—survives the hardware change. What does not survive is the unconditional assumption that “FP8 is cheaper”: on an accelerator without native FP8, the kernel emulation overhead dominates the memory-bandwidth savings for a dense 8B model, and the operator pays a small penalty for the quantization. Any planning framework that claims FP8 is a free lunch needs an explicit hardware caveat; this paper’s framework already accommodates that caveat because Q is a first-class input to Ceff (H, M, Q, λ, L).

Result 1: the load-driven spread reproduces. Figure 8 plots Ceff vs. λ on log-log axes for each paired configuration, H100 (solid blue circles) overlaid on A100 (dashed orange squares). Every panel shows the same characteristic cliff: a steep drop from λ=1 to λ≈25, followed by a plateau as the engine saturates. Quantitatively, the A100 data exhibits a 7.0– 11.4× per-configuration spread (Table 6), narrower than the 17.5–36.3× observed on H100. Two mechanisms explain the compression: (i) the A100’s lower peak throughput brings the saturation plateau closer to the idle-edge cost, and (ii) the A100’s 47% lower hourly rate proportionally shrinks the numerator of Ceff at idle. The shape of the curve—a cliff at low λ followed by saturation—is hardware-independent; the absolute magnitude of the spread tightens on the cheaper, slower accelerator. This is the portable finding: any calculator that omits λ and assumes a fixed utilization will be wrong by close to an order of magnitude on A100 and by over an order Result 3: the active-parameters-beatof magnitude on H100. total-parameters claim survives the hard16

ware change. On H100 NVL the paper’s headline ordering at saturation is Qwen3-30B-A3B FP8 (3B active, $0.21/MTok) cheaper than Llama 3.1 8B FP8 (8B active, $0.24/MTok)— active parameters, not total model size, drive the saturation cost. On A100 80GB PCIe the ordering holds: Qwen3-30B-A3B FP8 at $0.36/MTok is cheaper than Llama 3.1 8B FP8 at $0.41/MTok, and on the FP16 side Qwen330B-A3B at $0.49/MTok is only modestly above Llama at $0.35/MTok despite activating 3.75× fewer parameters per token. Two hardware families now support the same claim: when a sparseMoE model’s active-parameter count is small relative to a dense competitor, the sparse-MoE model will reach a competitive cost-per-token at saturation, even carrying 3–6× more total weight in VRAM. Operators choosing between architectures should budget on active parameters. Result 4: the TP=2 vs. TP=4 inversion on Mixtral. An additional A100 configuration on a 4×A100 node lets us probe a tensor-parallel scaling question the H100 apparatus could not. On A100, Mixtral 8x7B at TP=2 (engine: vLLM) saturates at $0.8688/MTok; at TP=4 the same model saturates at $1.3016/MTok— approximately 1.50× more expensive per token (49.8% at a single run per TP; a repeat-stability sweep on TP=4 is deferred to camera-ready) despite doubling the GPU count. Throughput does scale with TP (A100 TP=2 peaks at 2,347 tok/s, TP=4 at 3,133 tok/s, a 1.33× gain) but the 2× GPU-cost multiplier overwhelms the sub-linear throughput gain. This is a concrete instance of the paper’s general claim that $/MTok is decided by the denominator, not the numerator: adding GPUs to a configuration that already has sufficient capacity inflates Ceff even though it increases peak throughput. Cost-aware operators should pick the smallest TP that fits the model and the SLO, and scale horizontally via replicas rather than vertically via TP.

hardware—reproduces on a second, architecturally distinct hardware family. The absolute spread tightens when moving to cheaper, slower silicon; the FP8-advantage pattern depends on whether the silicon has native FP8 kernels; the active-parameters-dominate claim survives. The cross-hardware exercise is, in effect, a falsification test: if H100 were a special case, the A100 curves would be flat. They are not. The recommendation to measure Ceff against one’s own traffic and hardware is now supported by data on two architecturally distinct hardware families.

6

Discussion

6.1

Why Existing Calculators Fail

Every calculator we surveyed—15+ of them— makes the same mistake: it treats GPU utilization as an input parameter the user is trusted to guess (or quietly assumes 100%). Utilization is not an input. It is the output of the interaction between arrival rate, model architecture, quantization, and SLO. A calculator that accepts utilization as a field is asking its users to solve the paper they came to the calculator to avoid solving. That is why the headline error is an order of magnitude and not, say, 2×: the entire class of tools has the causal direction of its own input backwards.

6.2

Practical Implications

Beyond the calculators themselves, the offeredload axis is largely absent from how self-hosted inference is budgeted in practice: capacity plans and infrastructure reviews reason about GPUhour and per-token rates as fixed inputs, leaving the variable that empirically moves realized cost the most—offered load—out of the cost model entirely. Restoring it as a first-class input, the framework enables three practical decisions: 1. Given a target workload profile, what is the realistic cost of self-hosting?

Takeaway. The framework’s central prediction—that Ceff moves by nearly an 2. At what traffic volume does self-hosting become cheaper than a specific API tier? order of magnitude across λ on identical 17

3. For a hybrid strategy, where should the rout- input (parallel prefill) or output (sequential deing boundary be set between self-hosted and code). This asymmetry has a concrete implication: the effective API cost for generation-heavy API-served traffic? workloads is far higher than aggregate per-token The penalty is also a provisioning signal, pricing suggests. Self-hosting, by contrast, imnot only a calculator error. The largest poses no separate input-token and output-token multipliers in this paper occur at offered loads tariff—though the input/output mix still moves (notably λ=1 rps) at which a cost-aware operator the realized cost per output token through the would not hold a dedicated accelerator idle: the prefill/decode balance (§5.7), so a like-for-like structural remedy for a deployment sitting deep comparison prices the API at its asymmetric perin the cost cliff is to change the provisioning— token rates and the self-hosted deployment at its consolidate tenants, scale replicas to the offered measured Ceff under the same workload shape. As an illustrative back-of-the-envelope (not load, coalesce requests, or fall back to a servera measured benchmark), a code-generation less endpoint below the crossover—not merely to read a more accurate number. The framework pipeline with 100-token inputs and 500-token 5 supports both readings: an accurate Ceff (λ) is outputs ( 6 of tokens are output) pays approxithe diagnostic that tells an operator they are mately (100×$5+500×$30)/600 ≈ $25.80/MTok over-provisioned, and the offered-load axis is the aggregate at GPT-5.5 list pricing—well above dimension along which autoscaling and consoli- the nominal $5.00/M input rate typically quoted. dation act. This also explains why the minute- The same tokens served on self-hosted Qwen3level dispersion in §6.7 is a transient of a fixed 30B-A3B FP8 carry no such output-token predeployment rather than a budget line: once the mium: its $0.209/MTok saturation cost (meafleet size is itself elastic, cost-per-token at an in- sured at the 512:256 shape) bills input and outstantaneous λ is an attribution, not a monthly put tokens at the same GPU-time rate, though the realized figure still shifts with workload shape rate. Contribution (3) connects directly to cost-aware (§5.7). The asymmetric pricing effect amplirouting systems. A routing controller that dy- fies the self-hosting advantage precisely for the namically switches between self-hosted inference workloads—code generation, long-form writing, and API fallback needs an accurate real-time cost agentic tool use—where enterprises are most model—precisely what this framework provides. likely to consider infrastructure investment. Our framework currently models aggregate token throughput; a natural extension is to sepa6.3 The Asymmetric Token Pricing rate input and output token economics and inAdvantage corporate the asymmetric pricing structure into A structural advantage of self-hosting that exist- crossover analysis. ing analyses consistently overlook: commercial API providers employ asymmetric token pricing, 6.4 The Serverless-vs-Dedicated Falcharging substantially more for output tokens lacy than input tokens. GPT-5.5 charges $5.00/M input vs. $30.00/M output [19]; Claude Son- A recurring question from infrastructure net 4.6 charges $3.00/M input vs. $15.00/M decision-makers is: “Why not just use a serveroutput [20]; Gemini 3.1 Pro charges $2.00/M less API at $X per million tokens?” On its input vs. $12.00/M output [21]. Output to- face, this appears to be a straightforward cost kens are priced 5–6× higher because generation comparison. It is not. Published serverless is compute-intensive and sequential—each token per-token prices are unconditioned on any requires a full forward pass. production service-level objective. Commercial Self-hosted models incur identical GPU infras- providers do not publish enforceable per-request tructure cost regardless of whether tokens are guarantees for time-to-first-token or inter-token 18

latency at the p99 tail; best-effort availability is the operative contract. A dedicated self-hosted deployment that must meet a production SLO (for example, TTFT p99 ≤ 300 ms, TPOT p99 ≤ 50 ms) operates in a different cost regime entirely, because provisioning headroom to absorb arrival-rate variance without breaching the tail is precisely what drives Ceff away from Csat . Concretely: under our six-phase live workload (1–50 rps), the best minute-level effective cost a Mixtral-8x7B FP16 deployment on 2×H100 achieves is $8.23/MTok (Table 7, “Best” column, observed at the 50 rps saturation minute)— still roughly 9.5× the theoretical saturation floor (Csat =$0.87) because the averaging window covers the full arrival-rate traversal including idle sub-intervals. An operator pinned below saturation to preserve TTFT p99 headroom would sit strictly worse than this figure. The published commercial API price for a comparable class of model is often quoted at $12–$15 per million output tokens (the lower reference tiers of §5.6). The two numbers are not comparable: one reflects a best-effort stateless endpoint with no latency SLA, the other reflects a dedicated deployment that has committed to a p99 latency contract. Choosing between them based on headline pertoken price alone is a category error, analogous to comparing spot cloud instance pricing against reserved-instance pricing without acknowledging availability and preemption differences.

6.5

Using This Framework Operationally

The framework is designed to be actionable by a single infrastructure owner in a three-step workflow. Step 1 — Deploy your chosen configuration. Select the model, quantization, and hardware that match your deployment constraints. The benchmark corpus released with this paper (the six H100 configurations C1–C6, plus the A100 cross-hardware and sensitivity sweeps; §7.1) provides a starting reference, but is not a substitute for on-hardware measurement under your own workload shape. Step 2 — Load-test under your own workload and SLO. Run a sweep (for example, with vllm bench serve, inference-perf, LLMPerf, or GenAI-Perf[22, 23, 24]) using your real prompt-length and output-length distribution, your expected arrival pattern, and your declared TTFT/TPOT/E2EL SLO. Record the highest arrival rate λ at which your SLO is preserved; this is the goodput of your deployment, distinct from the raw saturation Θmax reported here. Step 3 — Plug observed goodput into the calculator. Feed the goodput-at-SLO into the Ceff formula (Eq. 3) to obtain the effective cost of your deployment under your actual operating contract. Compare that number—not Csat , and not the raw Θmax from a no-SLO benchmark—to any alternative pricing. This workflow inverts the usual procurement order. Rather than picking a price point and then discovering latency consequences, the operator declares the SLO first and derives the cost second. The vllm-cost-meter tool supports the third step directly: its live scrape of vLLM’s Prometheus endpoint surfaces observed goodput in real time against user-declared SLO budgets, and computes Ceff continuously as arrival rate varies.

The framework presented in this paper makes this mismatch visible. An organization with a production SLO should compute Ceff at the arrival rate its SLO permits (Table 4 works this through for a fixed example SLA), then compare to serverless pricing while explicitly acknowledging that the serverless path surrenders tail-latency control. Our companion tool, vllm-cost-meter, gates its API-comparison feature behind an --accept-slo-mismatch flag for 6.6 The Open-Source Tool exactly this reason: the comparison is meaningful only when the user has consciously accepted vllm-cost-meter is the concurrency-aware replacement for every calculator critiqued in Secthat serverless offers no SLO counterpart. 19

tion 2.

6.7

Why a meter, not a calculator. This is a deliberate design choice, not a missing feature. A calculator asks its user to supply the model’s true maximum throughput on their hardware—precisely the SLO-aware, carefully measured quantity this paper argues is almost never obtained correctly, and that in practice is guessed or copied from a vendor benchmark. A meter inverts the dependency: by scraping the live server’s own Prometheus metrics it derives the operating-point throughput from observed traffic, so the hard-to-get denominator is measured rather than assumed. We therefore release an instrument that reads ground truth instead of a form that propagates a guess. The pre-computed curves for the six configurations characterized here serve as a bounded lookup for those exact deployments; we intentionally do not extrapolate them into a general-purpose calculator, because a calculator populated with presets for unmeasured models would reproduce the same heuristic-preset, utilization-naive error (cf. the preset-based tool in §2) that this paper exists to correct. A community calculator seeded by crowd-sourced meter outputs—real observations, never presets—is a natural future direction. Three things ship today: • Benchmark runner. One command sweeps vLLM across a chosen arrival-rate grid and emits a structured CSV. • Cost estimator. Given a target λ, interpolates the empirical curve to produce a utilization-adjusted $/MTok and a crossover point against any API price the user supplies. • Live dashboard. Scrapes vLLM’s /metrics Prometheus endpoint and plots Ceff in real time as arrival rate shifts. Pre-computed curves from this paper’s 42 runs ship in the repository; a practitioner running one of these benchmarked configurations can obtain a corrected cost estimate in under a minute without running a single benchmark, while unmeasured deployments require the on-hardware sweep of §6.5.

Live Validation

To validate the framework beyond controlled benchmarks, we deployed vllm-cost-meter on the same H100 NVL hardware alongside live vLLM inference servers running all six benchmark configurations (C1–C6). The tool is an objective live instrument: on every tick it scrapes vLLM’s Prometheus /metrics endpoint, computes the live Ceff from observed generation tokens per second and declared hardware cost, and surfaces the instantaneous offered-load/cost relationship. A six-phase enterprise workload simulation ramped request rates from 1 to 50 rps and back, exercising the full range of arrival rates that a production deployment would traverse across a day. Table 7 summarizes the results. In one hour of real traffic the same deployment traversed a 653× cost swing between its cheapest and most expensive minutes—which we present as an illustration of minute-window dispersion, not as a reproducible validation datum (per the caveat below). This is an extreme ratio: it contrasts the idle-valley instantaneous minute against the peak-load minute, not a time-weighted mean. Mixtral FP16 peaked at $5,379 per million tokens during the idle valley (when only a handful of tokens were billed against the full GPU-hour) and dropped to $8.23 at 50 rps. Averaged over the hour the effective cost is dominated by the high-λ minutes; the 653× figure is best read as a stresstest artifact of minute-window cost attribution near idle—a near-singular ratio that drops as the window lengthens—rather than as a monthly billing multiplier. We quote it to make the minute-level dispersion concrete; the underlying per-minute Prometheus ticks are not checked into the public corpus (Table 7 records the aggregated Best/Worst columns), so the 653× should be read as a directional signal rather than a reproducible data point. Any cost number quoted without a λ attached—even averaged—is meaningless to a degree most practitioners badly underestimate. The live validation demonstrates two things. First, the concurrency-aware framework produces actionable real-time estimates: the same

20

deployment traces a recognizable Ceff curve as λ varies, matching the shape predicted by the benchmark sweep. Second, the underutilization penalty is not merely a theoretical artifact of offline benchmarking—it manifests immediately in live operation whenever λ falls below the arrival rate at which the engine saturates. The curve the tool surfaces is the same curve an operator would trace by re-running the benchmark sweep on their own workload; the tool makes this visible without a separate measurement pass. The validation we do claim is one of agreement: the live, Prometheus-derived Ceff traces the same offeredload curve, at the same operating points, as the offline benchmark sweep on identical hardware— confirming that the meter reports the cost structure an operator would otherwise have to reconstruct by re-running the sweep.

6.8

Scope and Future Work

We deliberately scope the headline sweep to synthetic fixed-length workloads on a single H100 NVL node—with a targeted A100 80GB PCIe validation sweep (§5.9)—for reproducibility and to avoid exposing proprietary production data. The main benchmark sweep is run with prefix caching and speculative decoding disabled, establishing a conservative performance floor and cost upper bound; our sensitivity probes (§5.7) partially relax these assumptions, enabling real prefix-cache hits and varying the arrival process and I/O shape. Production deployments with prefix caching, speculative decoding, and workload-specific tuning would achieve lower effective costs, making the absolute costs reported here upper bounds rather than typicalcase estimates. Several extensions are natural: validating with variable-length conversational traces (e.g., ShareGPT, LMSYS-Chat-1M [29]), incorporating SLO constraints as a first-class optimization target rather than an observed output, modeling diurnal traffic patterns and autoscaling economics, and extending to additional hardware generations (B200, MI300X) and inference engines (SGLang, TensorRT-LLM). The framework’s structure is designed to accommodate these extensions, though each would require

its own empirical validation to confirm the shape of U (λ, L) under the new conditions.

6.9

Limitations

Our benchmarks use synthetic fixed-length workloads with uniform 512-token inputs and 256-token outputs; production workloads with variable-length conversational distributions will exhibit different utilization curves. We test three model architectures (dense, ultra-sparse MoE, sparse MoE); additional architectures (e.g., state-space models, encoder-decoder) may show different cost behaviors. The headline vLLM numbers are conducted on H100 NVL; we add a cross-hardware validation sweep on A100 80GB PCIe (§5.9) to rule out single-hardware confounding, and the load-driven spread reproduces there, though still-older hardware (V100) or newer generations (B200, MI300X) would shift the curves further and are left as future work. The headline numbers use a single engine (vLLM); engine choice preserves the qualitative cliff but shifts its magnitude—the same Llama 3.1 8B FP16 configuration spreads 24.4× on vLLM versus 12.0× on SGLang (Table 6), as a lower Θmax compresses the idle-to-saturation ratio—so the absolute multiplier should be read as engine-conditional. We vary offered request rate while observing latency as an output rather than constraining it as an SLO input, leaving SLO-constrained optimization for future work. We do not model multi-tenant GPU sharing, which can improve utilization for providers serving multiple customers. GPU pricing changes frequently; the framework’s value is in the methodology, not specific dollar amounts.

7

Conclusion

The headline number is 17.5–36.3×. On identical H100 hardware, holding model, precision, and GPU allocation fixed and varying only the offered request rate, effective cost per token moves by 17.5–36.3× within a single configuration; across all six configurations and the full load range it spans $0.21 to $15.25 per million output tokens. That spread is not a tail case—it is the dominant

21

Table 7: Live validation: effective cost ($/MTok) observed during six-phase workload simulation (1–50 rps) on H100 NVL. 32,401 requests, 100% success. Config

Precision

Csat

Best

Worst

C1: Llama-8B (1 GPU) C2: Llama-8B (1 GPU) C3: Qwen-30B (1 GPU) C4: Qwen-30B (1 GPU) C5: Mixtral (2 GPU) C6: Mixtral (2 GPU)

FP16 FP8 FP16 FP8 FP16 FP8

0.31 0.24 0.36 0.21 0.87 0.52

1.74 1.79 1.63 1.60 8.23 6.24

3,496 226 202 353 5,379 472

term in the cost equation, and it is invisible to 7.1 Data and Artifact Availability every production cost calculator we surveyed. The companion vllm-cost-meter tool and the raw per-run CSVs are released under an MIT Three practical implications. First: if an in- license at https://github.com/pChitral/ frastructure plan is based on a per-token calcu- vllm-cost-meter: the 140-run core corpus— lator, the number can be wrong by more than the 42-run H100 vLLM headline sweep, the 42an order of magnitude—always in the direction run H100 SGLang companion sweep, and the 56that makes self-hosting look cheaper than it is. run A100 80GB PCIe cross-hardware sweep— Second: on the three architectures tested (n=3; together with the I/O-shape, prefix-caching, dense, ultra-sparse MoE, sparse MoE), active pa- and burstiness sensitivity sweep (§5.7) and the rameter count and memory-access patterns, not measurement-stability campaign (§5.8). All sattotal parameter count, appear to drive satura- uration points, Ceff spreads, underutilization tion economics: Qwen3-30B-A3B FP8 is cheaper penalties, and stability CVs in §5 are re-derivable per token at saturation than Llama 3.1 8B FP8, directly from these CSVs; the variable-length inverting the size-based intuition. The same and real-prefix-hit probes of §5.7 are summarized ordering survives the A100 replication in §5.9, in-text. A version-tagged Zenodo DOI will be so two silicon families now support the claim, minted at camera-ready. The repository is inbut it still rests on three architectures; broader tended as a community substrate for extending validation—in particular a larger dense model the λ-axis sweep to additional hardware, model (e.g., Llama 3.1 70B) and a third MoE spar- architectures, and SLO-bounded regimes. sity ratio—is required before the pattern can be treated as an architectural-scaling law. Third: FP8 is not a dense-model optimization—on our three models the two MoE architectures benefit 2.2–2.4× more than the dense one (+69 to +74% vs. +31% peak throughput), a structural signal that should shift quantization priorities pending wider confirmation. The tool is vllm-cost-meter. It scrapes a running vLLM server’s Prometheus endpoint and surfaces the real cost curve in real time. If today’s answer to “what does a million tokens cost us?” is a single number, replace it with a curve before the next infrastructure review. 22

Llama 3.1 8B (FP8)

Llama 3.1 8B API $0.2/out

Llama 3.1 8B FP8

100

150

Offered load  (req/s) Self-hosted (FP8) Qwen3-30B-A3B FP8

200

250

Mixtral 8x7B FP8

Figure 5: Both axes use cost per million output tokens. Self-hosted Ceff is GPU $/hr divided by output throughput; dashed API lines are each provider’s list output-token price only (the “/out” suffix in the inline labels is a reminder of this). Input-token billing, prompt-caching, and batch-API discounts are not applied to the API lines—all three would shift them down (see §5.6 caveat). Plotted self-hosted lines are the FP8 variants (the FP16 curves sit within < 1% of FP8 at λ=1, so the crossover visual is unchanged); prose in §5.6 quotes the FP16 value ($15.25/MTok) for consistency with the H100 headline sweep. The λ⋆ ≈1.6 annotation on the Mixtral FP8 curve marks its log-y-interpolated crossover against the Gemini 3.1 Pro output tier ($12/MTok), the lowest-priced non-open-source reference line in the panel; the Claude Sonnet 4.6 tier at $15/MTok is crossed at λ≈1, while the GPT-5.5 tier ($30/MTok) sits above the Mixtral curve at every measured λ. Under naive utilization math every self-hosted line sits below every API line; under real utilization math Mixtral at 1 rps is more expensive than Claude Sonnet 4.6’s output tier. The crossover is not a point; it is a threshold that moves with traffic and with the reference tier. API prices are public list prices as of 2026-06-09 [19, 20, 21], included only as external economic reference tiers; the benchmark corpus itself is fixed for reproducibility.

$ / M output tokens

1.2 1.0 0.8 0.6 0

50

100

150

200

100

1.2 1.0 0.8 0.6 0

50

Request rate (req/s)

100

150

200

Request rate (req/s)

Prefix cache OFF

Prefix cache ON

[0.79, 1.11] observed range

Figure 6: Prefix-cache OFF vs. ON on randomtoken workloads (no shared prefixes). APC is inert by construction; the small +11% overhead visible at λ∈{100, 200} on Llama C2 is APC’s hashing bookkeeping, not a serving pathology. Production workloads with real prefix sharing go the OTHER way. The main-sweep numbers are not artifacts of prefix-cache state.

Llama 3.1 8B (FP8)

Qwen3-30B-A3B (FP8) 7.60

7.59

Effective cost ($ / M output tokens)

50

100

Ratio (prefix-cache on / off)

$ / M output tokens

1

0

Qwen3-30B-A3B (FP8)

Claude Sonnet 4.6 $15/out Gemini 3.1 Pro $12/out

* ≈ .

10

Ratio (prefix-cache on / off)

Cost ($ / M output tokens)

GPT-5.5 $30/out

1.93

1.92

100

0.84

0.84

0.75

0.75

0.55

0.52

0.45

0.45

0.37

0.34 0.24

λ=1

0.44

0.44

λ=25

chat 512:256

0.22

λ=100

λ=1

rag 4096:1024

λ=25

λ=100

agentic 1024:4096

Figure 7: Effective cost at λ∈{1, 25, 100} across three I/O shapes (chat 512:256, RAG 4096:1024, agentic 1024:4096) for C2 and C4 with prefix caching off and Poisson arrivals. The RAG/chat cost ratio is non-monotonic: RAG is cheaper than chat at λ=1 (long prefill amortises coldstart cost), 2.4–2.7× at λ=25, and peaks at 3.2– 3.7× at λ=100; prompt-heavy workloads dominate cost at high load while preserving the offered-load sensitivity structure of the framework.

23

Llama 3.1 8B FP16 (vLLM, TP=1)

Llama 3.1 8B FP8 (vLLM, TP=1)

100

100

101

Qwen3-30B-A3B FP8 (vLLM, TP=1)

101

100

102

Offered load  (req/s)

Mixtral 8x7B FP16 (vLLM, TP=2)

H100 (C4) A100 (A4)

101

Llama 3.1 8B FP16 (SGLang, TP=1) H100 (S1) A100 (AS1)

($/MTok)

101

102

Offered load  (req/s)

H100 (C5) A100 (M1)

($/MTok)

($/MTok)

100

100

102

H100 (C3) A100 (A3)

($/MTok)

100

Offered load  (req/s)

100

Qwen3-30B-A3B FP16 (vLLM, TP=1)

H100 (C2) A100 (A2)

($/MTok)

($/MTok)

H100 (C1) A100 (A1)

100 10 100

101

102

Offered load  (req/s)

0

100

101

102

Offered load  (req/s)

100

101

102

Offered load  (req/s)

Figure 8: Effective cost Ceff vs. offered load λ on H100 NVL (solid blue) vs. A100 80GB PCIe (dashed orange), log-log, for six paired configurations. The saturation cliff reproduces on A100; the absolute spread is narrower (7.0–11.4× vs. 17.5–36.3×) because A100’s lower peak throughput and lower hourly rate compress the numerator-denominator ratio. At low λ (near-idle), A100’s lower hourly rate makes it cheaper per-million-tokens; above each panel’s crossover (typically λ ≳ 25 for dense/FP8, later for Mixtral TP=2), H100’s throughput advantage wins despite the premium hourly rate.

24

References

//developer.nvidia.com/blog/llm-inf erence-benchmarking-how-much-does-y our-llm-inference-cost/. Accessed 202604-23.

[1] G. Pan, V. Chodnekar, A. Roy, and H. Wang. A cost-benefit analysis of onpremise large language model deployment: Breaking even with commercial LLM ser- [11] Introl. Inference unit economics: The true cost per million tokens. Introl Blog, Februvices. arXiv preprint arXiv:2509.18101, ary 2026. https://introl.com/blog/infe 2025. rence-unit-economics-true-cost-per-m [2] C. Irugalbandara, A. Mahendra, R. Liyanillion-tokens-guide. Accessed 2026-04age, et al. Scaling down to scale up: A cost23. benefit analysis of replacing OpenAI’s LLM with open source SLMs in production. In [12] acnicessc. LLM Inference TCO Calculator v2.4. https://acnicessc.github.io/ll IEEE ISPASS, 2024. mcalc/, 2025. Accessed 2026-04-23. [3] A. Agrawal, N. Kedia, J. Mohan, et al. VIDUR: A large-scale simulation framework [13] BentoML. llm-optimizer: Benchmark and optimize LLM inference across frameworks. for LLM inference. In MLSys, 2024. https://github.com/bentoml/llm-optim [4] T. Griggs, X. Liu, J. Yu, et al. Mélange: izer, 2025. Accessed 2026-04-23. Cost efficient large language model serving by exploiting GPU heterogeneity. arXiv [14] SemiAnalysis. InferenceX (formerly Inferpreprint arXiv:2404.14527, 2024. enceMAX): Open-source continuous inference benchmarking. https://github.com [5] S. Jaiswal et al. SageServe: Optimizing /SemiAnalysisAI/InferenceX, 2025. AcLLM serving on cloud data centers with cessed 2026-04-23. forecast aware auto-scaling. arXiv preprint arXiv:2502.14617, 2025. [15] vLLM Project. GuideLLM: Evaluate LLM deployments for real-world inference. http [6] E. Erdil. Inference economics of language s://github.com/vllm-project/guidellm, models. arXiv preprint arXiv:2506.04645, 2025. Accessed 2026-04-23. 2025. [16] Qwen Team. Qwen3 Technical Report. [7] B. Zhuang, J. Qiao, M. Liu, et al. Beyond arXiv preprint arXiv:2505.09388, 2025. benchmarks: The economics of AI inference. arXiv preprint arXiv:2510.26136, 2025. [17] A. Grattafiori, A. Dubey, A. Jauhri, et al. The Llama 3 Herd of Models. arXiv preprint [8] K. Kim, J. Li, K. Hong, and A. AilaarXiv:2407.21783, 2024. maki. Faster LLM inference using DBMSinspired preemption and cache replacement [18] A. Q. Jiang, A. Sablayrolles, A. Roux, et policies (INFERMAX). arXiv preprint al. Mixtral of Experts. arXiv preprint arXiv:2411.07447, 2024. arXiv:2401.04088, 2024. [9] W. Kwon, Z. Li, S. Zhuang, et al. Ef- [19] OpenAI. API Pricing. https://develope ficient memory management for large lanrs.openai.com/api/docs/pricing, 2026. guage model serving with PagedAttention. Accessed 2026-06-09. In SOSP, 2023. [20] Anthropic. Claude API Pricing. https:// [10] NVIDIA. LLM inference benchmarking: claude.com/pricing, 2026. Accessed 2026How much does your LLM inference cost? 06-09. NVIDIA Developer Blog, June 2025. https: 25

[21] Google. Gemini API Pricing. https://ai [29] L. Zheng, W.-L. Chiang, Y. Sheng, et al. LMSYS-Chat-1M: A large-scale real-world .google.dev/gemini-api/docs/pricing, LLM conversation dataset. In ICLR, 2024. 2026. Accessed 2026-06-09. [22] Kubernetes SIG Serving. inference-perf: [30] G.-I. Yu, J. S. Jeong, G.-W. Kim, et al. GenAI inference performance benchmarking Orca: A distributed serving system for tool. https://github.com/kubernetes-s transformer-based generative models. In igs/inference-perf, 2025. Accessed 2026OSDI, 2022. 04-23. [31] R. Qin, Z. Li, W. He, et al. Moon[23] Anyscale. LLMPerf: A tool for benchmarkcake: A KVCache-centric disaggregated aring LLMs. https://github.com/ray-pro chitecture for LLM serving. arXiv preprint ject/llmperf, 2024. Accessed 2026-04-23. arXiv:2407.00079, 2024. [24] NVIDIA. GenAI-Perf: Benchmark genera- [32] Y. Liu, Y. Cheng, J. Yao, et al. LMtive AI models. https://github.com/tri Cache: An efficient KV cache layer for ton-inference-server/perf_analyzer, enterprise-scale LLM inference. arXiv 2024. Accessed 2026-04-23. preprint arXiv:2510.09665, 2025. [25] P. Patel, E. Choukse, C. Zhang, et al. Split[33] B. Sun, Z. Huang, H. Zhao, et al. Llumwise: Efficient generative LLM inference usnix: Dynamic scheduling for large language ing phase splitting. In ISCA, 2024. model serving. In OSDI, 2024. [26] Y. Zhong, S. Liu, J. Chen, et al. Dist[34] Y. Wang, Y. Chen, Z. Li, et al. BurstServe: Disaggregating prefill and decoding GPT: A real-world workload dataset to opfor goodput-optimized large language model timize LLM serving systems. arXiv preprint serving. In OSDI, 2024. arXiv:2401.17644, 2024. [27] A. Agrawal, N. Kedia, A. Panwar, et al. Taming Throughput-Latency Tradeoff in [35] Y. Xiang, X. Li, K. Qian, et al. ServeGen: Workload characterization and generation of LLM Inference with Sarathi-Serve. In OSDI, large language model serving in production. 2024. arXiv preprint arXiv:2505.09999, 2025. [28] L. Zheng, L. Yin, Z. Xie, et al. SGLang: Efficient execution of structured language model programs. In NeurIPS, 2024.

26