Scaling Shapley Energy Attribution to Large LLM Batches

arXiv:2611.13802v1 [cs.DC] 10 Nov 2026

Scaling Shapley Energy Attribution to Large LLM Batches Qi Luo

Ziwen Wang

Kunlin Li

qluo615@connect.hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

zwang578@connect.hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

kli082@connect.hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Changxi Liu

Yun Chen∗

changxiliu@hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

yunchen@hkust-gz.edu.cn The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Abstract

1

User A

Summarize this document.

LLM services have become essential infrastructure for a wide range of applications, such as chat assistants and coding agents. Usually, service providers need to attribute resource consumption to individual requests for purposes such as billing. Among these resources, energy is a key metric for assessing the cost of serving each request. However, serving engines batch requests together, and GPUs report only aggregate energy consumption, making it difficult to attribute energy use to individual requests. Providers typically estimate each request’s share in proportion to its token consumption, but this approach can produce substantially unfair estimates, especially when requests vary significantly in length or computational complexity. The Shapley value provides a principled solution to fair energy allocation, but computing it directly requires 2𝑁 − 1 GPU runs for 𝑁 requests, which is infeasible for large batches. To address this problem, we propose StepSplit, an efficient method for attributing energy consumption to individual LLM requests. Across GPU generations, we observe that the energy consumed by each inference step, a forward pass over a batch, consists of a shared energy, largely due to reading model weights, and a request-specific energy. Based on this observation, we derive a closed-form Shapley value: each request is charged its own energy plus an equal part of the shared energy of every step in which it participates. A one-time microbenchmark lasting 2.5–17.8 minutes calibrates both parts of the step energy for each model and GPU configuration. Under static batching, StepSplit uses this closed form in a single pass over the scheduler log, which records the requests served at each step. Under continuous batching, removing a request can change how the remaining requests are batched, so StepSplit adds a correction to the closed form, computed by replaying the scheduler on a CPU. We evaluate StepSplit against Shapley values measured using 529,513 GPU runs. The results show that, under static batching with batch sizes of 8–256, StepSplit achieves an 𝐿1 error of 0.4–4.3%, compared with 13.4–61.3% for token-proportional allocation. Under continuous batching with 8–128 requests, the corresponding errors are 0.5–6.4% and 20.6–71.7%.

Token bill

356.3 J User A

126.8 J

4,157 input tokens 124 output tokens

Write a long story.

User B

Fair share

>

987 input tokens 413 output tokens

Batched LLM server

User B Token bill

227.6 J

Fair share

<

356.8 J

Figure 1: Comparison of token bills and fair Shapley shares.

providers need to attribute resource consumption to individual requests for purposes such as billing [25], carbon reporting [10, 36] and scheduling [41]. Among these resources, energy is a key metric for assessing the cost of serving each request, as it is a major cost of LLM inference [29, 30]. However, the energy of a single request cannot be measured directly [37]. A serving engine such as vLLM [18] runs many requests together in each forward pass. The GPU draws approximately 75% of the power of a server [5, 8, 19, 31] and reports only their total energy. How to split this total fairly among the requests is not obvious. In practice, the cost of a request is usually measured by its tokens. OpenAI and Anthropic price requests per token [4, 28], a commercial service bills energy by the share of each request in the in-flight tokens [25], and fair schedulers weight requests by their tokens [42]. As shown in Figure 1, splitting by tokens does not give each request its fair share. The reason is that each forward pass reads the weights once, however many requests it holds. As we show in §4, the energy of a pass is the shared energy of this read plus the own energy of each request. The read takes 11.0–89.8% of the energy, depending on the workload, and all requests in the pass share it. The own energy of a request grows with its tokens, but its share of the reads depends on how many passes it joins and how many requests share each pass, not on its tokens. The standard fair split is the Shapley value [39]. It charges each request the extra energy it causes, averaged over every order in which the requests could join. Energy and carbon attribution use it as their reference [9, 16, 21]. However, computing it requires 2𝑁 − 1

Introduction

LLM services have become essential infrastructure for many applications, such as chat assistants and coding agents [32, 33, 44]. Service ∗ Corresponding author.

1

• We validate StepSplit against Shapley values measured in 529,513 GPU runs and show that it is more accurate than both splitting by tokens and a trained regression in every configuration. • We build an online ledger that hooks into vLLM without changing it and bills each request when it finishes, at 47 𝜇s per step.

separate GPU runs for the subsets of the 𝑁 requests, 255 for 8 requests but 4.3 × 109 for 32, which is infeasible for large batches. In practice, it is measured only for small groups such as 8 requests [23, 38]. Scaling to larger groups needs a sampled estimate [6], whose number of runs still grows with the group. Yet a serving engine batches hundreds of requests, so neither approach reaches the batch sizes of real serving. Prior work instead approximates the value with a regression trained on measured values [23], a game over time periods rather than requests [16], or Shapley-inspired splits of idle energy [40]. Providers thus have a scalable split that is not fair and a fair split that does not scale. To address this problem, we propose StepSplit. We call each forward pass a step. We measure the two-part energy on three GPU generations, and the shared energy of a step stays the same however many requests are in it. The energy of any subset can therefore be computed from its steps instead of measured in a separate GPU run. Under this structure, the Shapley value has a closed form known from classic cost-sharing problems [11, 22]. Each request pays its own energy plus an equal part of the shared energy of every step it joins. We call this rule step sharing. A microbenchmark of 2.5–17.8 minutes calibrates both parts of the step energy once per model and GPU configuration. Under static batching, StepSplit uses this closed form in one pass over the scheduler log of the run that serves the requests, without enumerating subsets. Under continuous batching, removing a request changes when the other requests run, so a subset goes through different steps from those in the scheduler log, and the closed form on the logged steps no longer gives the Shapley value. StepSplit therefore adds a correction to the closed form, computed by replaying the scheduler of the serving engine on a CPU. The closed form does not give this correction, so StepSplit samples it [6], and for each sampled subset it computes the energy of the replayed and of the logged steps on the CPU instead of measuring it on the GPU. We also build an online ledger that hooks into vLLM without changing it. As each step ends, the ledger adds to every scheduled request its share of the step, at 47 𝜇s per step, so each request is billed when it finishes. Under continuous batching, more accurate bills need a CPU replay after the run. We compare StepSplit with Shapley values (exact for 8 requests, sampled for more) measured on the GPU across 529,513 runs, seven models, three GPU generations and a distributed deployment. Under static batching with batch sizes of 8–256, StepSplit has an 𝐿1 error of 0.4–4.3%, against 13.4–61.3% for splitting by tokens and 8.6–26.6% for a regression trained on measured Shapley values [23]. For 8– 128 requests under continuous batching, it has 0.5–6.4%, against 20.6–71.7% and 8.0–56.0%. In summary, we make the following contributions: • We characterize the energy of batched LLM inference on three GPU generations and show that the energy of each step has two parts: a shared energy, mostly for reading the model weights, and the own energy that each request adds. • We propose StepSplit, which computes the Shapley value of each request in one pass over the scheduler log of one run, with no extra GPU runs. Under continuous batching and in a distributed deployment, it replays the scheduler and the router on a CPU. The same rule extends to MoE experts and cached prefixes.

2 Background 2.1 Cooperative Games and the Shapley Value A fair split charges each request for the energy it causes. However, requests served together share part of the work, so the energy that a request adds depends on which other requests are served with it. Cooperative game theory defines this split with the Shapley value [39]. The players are the 𝑁 requests N served together. For a coalition 𝑆 ⊆ N , the characteristic function 𝑣 (𝑆) is the energy of serving only the requests in 𝑆, with 𝑣 (∅) = 0. The Shapley value of request 𝑖 averages the energy that 𝑖 adds to the requests before it over all orders in which the requests could join,

i 1 ∑︁ h 𝜎 𝑣 𝑃𝑖 ∪ {𝑖} − 𝑣 𝑃𝑖𝜎 , 𝜑𝑖 = (1) 𝑁! 𝜎 where 𝑃𝑖𝜎 is the set of requests that precede 𝑖 in the order 𝜎. It is the only split that satisfies four axioms. Its shares add up to 𝑣 (N ) (efficiency), treat interchangeable requests equally (symmetry), give nothing to a request that never adds energy (null player), and add up over games (additivity). Since each 𝑣 (𝑆) is a separate GPU run, the definition needs 2𝑁 − 1 runs, 255 for 8 requests but about 4.3 × 109 for 32.

2.2

Batched LLM Inference

A serving engine such as vLLM [18] runs an LLM on GPUs and serves a batch of requests one step at a time. Each step is one forward pass of the model. A request 𝑖 has a prompt of 𝑝𝑖 input tokens and generates 𝑑𝑖 output tokens. Each request goes through two phases, prefill and decode. Prefill processes the prompt and stores the result in a KV cache. Decode then generates one output token per step and reads the KV cache each time. A step handles all of its requests together, whether in prefill or in decode. In every step the GPU reads the model weights from its memory once, for all of its requests. This read is the work that the requests of a step share. Static batching serves requests that are all available at the start. They enter the batch together, so 𝑁 is the batch size, and the number of requests in a step, 𝑚, starts at 𝑁 and falls as requests finish. Continuous batching [47] serves requests that arrive over time, as in online serving, and lets them join and leave the running batch between steps, so 𝑚 changes from step to step.

2.3

The Energy of a Coalition

We split the busy energy of the GPUs, which is the total energy that they consume while they serve requests, including the idle power they draw with the model loaded even without work. The gaps without any request, such as while the engine waits for arrivals, and host stalls (§6.1) are left out, because no request causes their energy. The energy law of §5.1 includes this idle power in its coefficients. In our experiments on three GPU generations, the idle power contributes 24.4–46.2% of the busy energy. 2

group

Defining 𝑣 (𝑆) also needs the number of tokens that each request generates in a coalition. Batched arithmetic is not bit-identical across batch compositions, so even with greedy decoding the output length of a request can change with the other requests. If output lengths were left free, the energy a request adds would include the energy of length changes in other requests, which it did not cause. We therefore define 𝑣 (𝑆) with every request generating the same number of tokens as in the actual run, which is also the number an online attribution system observes.

3

𝒅𝒊

Token

Solo

JCalib

Shapley

0 1 2 3 4 5 6 7

120 1060 987 1314 4157 173 1232 204

14 397 413 176 124 110 399 67

14.2 227.1 227.6 163.6 356.3 53.9 240.7 40.2

11.4 304.5 315.5 138.5 112.8 83.4 306.7 50.9

21.3 335.1 351.4 103.1 135.3 31.2 341.8 4.3

7.6 315.8 356.8 113.9 126.8 50.2 323.5 29.0

0.454

0.126

0.090

–

𝐿1 vs. Shapley

Energy / energy at 32

Token L1

0.8

0.4

e T ode onv. ncak stGP re c re c Bur Moo Azu

Azu

Token

(a) Trace

0.8 0.6 0.4 0.2 0.0 32

64

128

256

(b) Batch size

Figure 2: (a) Token 𝐿1 for each group of 8 requests, by trace. (b) Fair share of each of the same 32 requests as the batch grows, relative to a batch of 32.

Table 1: Input and output tokens of 8 requests in one batch and the energy (J) attributed to each. 𝒑𝒊

fair share 1.0

0

Motivation

Request

mean

1.2

and it differs across workloads. The mean Token 𝐿1 is 18.5% on the Azure code trace and 98.7% on the Mooncake agent trace [33]. As shown in Figure 2b, the fair share of a request also depends on the batch it runs in. We draw 256 requests from the Azure conversation trace and serve the first 32 of them in a batch of 32 and then in batches of 64 to 256. In the batch of 256, their fair shares fall to 0.29–0.86 of those in the batch of 32. We compute these fair shares with the method of §5, which matches the sampled Shapley values of these batches (§6.3). In the larger batch, more requests share each step, so each request pays a smaller part of the shared energy of the step. Requests with long outputs, whose energy is mostly shared, thus fall the most. Token cannot follow these changes. It lowers the charge of every one of the 32 requests to the same 0.43 of its charge in the batch of 32, because only the energy per token of the batch changes, not the tokens of the request. None of the three rules gives fair shares. Computing the Shapley value without running every subset on the GPU thus needs an analytical model of the energy of a subset. §4 characterizes how the energy of a step depends on the requests in it, and §5 builds this model from it.

Because the exact Shapley value is expensive, the energy of a batch is often split with cheaper rules. Token splits it in proportion to the tokens of each request. Following the pricing of Anthropic and OpenAI [4, 28], we count the tokens of request 𝑖 as 𝑝𝑖 + 5𝑑𝑖 . Solo splits it in proportion to the energy of each request run alone. JCalib [23] is a regression that predicts shares from features of the token counts of each request and its batch, trained on the measured Shapley values of other batches. Table 1 compares these rules with the exact Shapley value of one batch of 8 requests from the Azure conversation trace [32], measured with all 255 runs of Qwen2.5-7B on an H100 under static batching. We measure the error of a rule Í by the normalized 𝐿1 distance 𝑖 |𝜑ˆ𝑖 − 𝜑𝑖 |/𝑣 (N ) between its shares 𝜑ˆ𝑖 and the Shapley values 𝜑𝑖 (Eq. (1)). Token has the largest 𝐿1 error, 45.4%. It overcharges requests with many input tokens and undercharges those with many output tokens. Request 4, with 4,157 input and 124 output tokens, is charged 356.3 J against a Shapley share of 126.8 J. Solo is closer at 12.6%, but it needs one extra GPU run per request, which is impractical in serving. It charges the short requests 0, 5 and 7 1.5–1.8× their shares, because alone each pays for the weight reads of all its steps, which it shares with most of the other requests in the batch. JCalib comes closest at 9.0%. However, its features describe token counts, not how many requests share each step. All three rules miss the shared energy that each step consumes once for all its requests, mainly to read the weights. In this batch, the shared energy of its 413 steps is 83.7% of the total. The fair shares thus depend mainly on which steps each request is in, and the output lengths set these steps, not the input lengths. The error of Token is not limited to this batch. As shown in Figure 2a, it appears in every group of 8 requests that we measure,

4

Characterizing the Energy of a Step

Section 4.1 gives the setup, which §6 also uses. Section 4.2 presents five insights that guide the design of StepSplit.

4.1

Experimental Setup

Platforms and models. We evaluate three NVIDIA GPU generations: Hopper H100, Ampere A800, and Blackwell RTX PRO 6000 (PRO 6000). All experiments run on vLLM 0.30 with greedy decoding, driven one step at a time by a harness that logs the duration of every step and the tokens of every scheduled request. We test seven models spanning three architectures: the dense Qwen2.5-7B, Qwen2.5-14B, and Qwen2.5-72B [34] and Gemma-4-12B [13]; the MoE models gpt-oss-20b [27] and Qwen3-30B-A3B [46]; and the linear-attention hybrid Qwen3.8-27B [35]. Models run on a single GPU or with tensor parallelism across two or four GPUs at default clocks and power limits. Weights are in BF16, except for the MXFP4 experts of gpt-oss-20b. We also evaluate a distributed deployment serving Qwen2.5-72B across two nodes behind a router that dispatches each arriving request to the less loaded node, where each node runs one vLLM engine on four H100s. 3

(b) shared energy: split per step

(a) inside a step: shared a + own ei,k tokens in the step

decode step

A

1 token

B

1 token

A

1 token

B

1 token

C

pC tokens

weights, read once per step

energy of the step:

engine steps in time order

Step sharing

Ek = a + ∑i ei,k step

shared a, split in (b)

×

own KV cache, read in each decode step

c1 pC + c2 pC2

×

W

ei = c1 pi + c2 pi2 + prefill, once

ℓ pi

pi

pi

pi

pi

pi

2

3

4

5

6

7

8

φA = eA + 21a/8 φB = eB + 17a/8 φC = eC + 7a/8 φD = eD + 13a/24 φE = eE + 7a/24 φF = eF + 7a/24 φG = eG + 3a/8 φH = eH + 7a/8

A B C D E F G H

W b + κ ℓ each

step with a prefill

1

b di + κ ctxi

each output token

d (d + 1) ctxi = total area = pi di + i i 2

output tokens 1 … di

shared energy: a per step

prefill a 2

each pays

a 2

a 3

a 4

a 6

a 8

a 4

decode

a 2

Figure 3: Step sharing, 𝜑𝑖 = 𝑒𝑖 + 𝑠𝑖 . (a) Inside step 𝑘, each scheduled request 𝑖 adds its own energy 𝑒𝑖,𝑘 , while the step incurs the shared energy 𝑎 once. (b) The shared energy 𝑎 of each step is split equally among the requests in it, however long the step lasts. H100

4.2

Energy of one step (J)

35

A800

30

PRO 6000 0.5

intercept = shared energy

25

14B

20 15

7B

10 5 0

Shared energy per GB (J)

Workloads. We take request lengths and arrival times from four production traces: the Azure LLM inference traces for conversation and code [32], BurstGPT [44], and the Mooncake agent trace [33]. Because these traces omit prompt text, we truncate ShareGPT [3] conversations to the input length of each request and fix each output to its trace length. We run 8 to 256 requests under static batching and 8 to 128 requests under continuous batching. We measure the exact Shapley value over all subsets for 8 requests and estimate it by sampling for 16 or more requests (§6.1). Under continuous batching, the 16 to 128 requests form a trace window of consecutive requests that arrive at their recorded times. A single engine serves these windows, and the distributed deployment also serves trace windows of 32 to 128 requests. At their recorded rates, the conversation and agent traces average 7–10 requests per step. To reach larger batches, we also replay them at 3× and 10× speed, keeping the request lengths and the burst pattern of the arrivals. Microbenchmark. A synthetic microbenchmark serves 𝑚 requests with input length 𝑝 and output length 𝐷 under static batching, so that every decode step holds exactly 𝑚 requests and the three vary independently. Each input is a different stretch of 𝑝 tokens from the ShareGPT conversations. The grid is 𝑚 ∈ {1, 2, 4, . . . , 256}, 𝑝 ∈ {128, 512, 2048, 8192} with 𝑚𝑝 ≤ 131,072, and 𝐷 ∈ {1, 64, 256}, with two runs for each combination, 180 runs in all. Runs whose KV cache does not fit in GPU memory are skipped. It runs once per model and GPU configuration (GPU model and number of GPUs). Measuring energy. We read the cumulative energy counter of each GPU through NVML [26], which updates at 10 Hz, and sum the energies of all GPUs that run the model. For the busy energy of §2.3, we subtract the idle power over the time outside steps. We measure the idle power of each GPU with the model loaded, before its runs.

0.4 0.3 0.2 0.1 0.0

0

32

64

128

256

(a) Requests in the step (m)

7

12 14

27

72

(b) Model size (B)

Figure 4: (a) Energy of one decode step against the number of requests in it, with fitted lines. (b) Shared energy per GB of weights against model size (log axis). Each dashed line is the average of the points of one GPU.

its terms in §5.1. In the microbenchmark, for the same 𝑚 and 𝑝, a longer output 𝐷 only adds decode steps of 𝑚 requests, so the extra energy per added step is the energy of one decode step. It grows nearly linearly with 𝑚, from 2.66 J with one request to 9.15 J with 256 at input length 512 for Qwen2.5-7B on the H100. The intercept of the fitted line is the shared energy and its slope is what one more request adds. Qwen2.5-14B, the A800 and the PRO 6000 behave the same way (Figure 4a). Calibrated on the microbenchmark alone, the law also predicts the energy of subsets of static batches from the Azure conversation and BurstGPT traces on the H100, with an average error of 0.6–2.8% for Qwen2.5-7B and 0.8–3.8% for Qwen2.514B. The energy of a coalition is therefore set by its steps and the tokens it processes in them. Insight 2. The shared energy is mostly the energy of reading the weights once per step. A decode step of Qwen2.5-7B on the H100 lasts 6.70 ms at the median, and the fitted curve of Figure 5a meets zero tokens at 6.68 ms, so most of a step is the work that it does once for all of its requests, which is reading the weights. If the shared energy is the energy of this read, it grows in proportion to the size of the weights, at a fixed energy per GB on each GPU. As shown in Figure 4b, the shared energy per GB of weights stays nearly the same on each GPU from 7B to 72B, at 0.172–0.212 J

Insights

Insight 1. The energy of a step is a shared energy plus the own energy of each request. The shared energy is paid once per step, and the own energy grows with the tokens of the request (Figure 3a). We call this decomposition the energy law and give 4

Duration of the step (ms)

0.102

Azure code

150

100

0.019

Azure conv.

0.025 0.016

BurstGPT

0.019 0.011

50

time sharing step sharing

2k

4k

6k

8k

Energy law c1, c2, b, κ, a

0.05

0.10

Closed form

step sharing, logged steps

Run log step log, energy

shares

while serving

+

step sharing

Scheduler replay

Sampling

continuous batching

request orders

correction

Per-request energy

scaled to measured energy

Figure 7: Overview of StepSplit.

on the H100, 0.211–0.270 J on the A800 and 0.252–0.282 J on the PRO 6000, including Qwen2.5-72B on four GPUs (two on the PRO 6000). The shared energy is therefore set by the model and the GPU, and a microbenchmark can calibrate it once per model and GPU configuration. Insight 3. The shared energy is paid per step, not per unit of time. Time sharing divides static, idle or shared chip power equally among the tenants or tasks active at each instant [17, 40, 41]. This fits the shared energy only if it grows with the duration of a step, but every step reads the weights once however long it lasts. As shown in Figure 5a, the rest of a step grows with its tokens, so a long step is long because of its own tokens. The idle power drawn over this extra time therefore also belongs to the requests that hold the tokens. In the measured run that Figure 3 sketches, a step that prefills a 7,142-token input lasts 158 ms, which is 23.6× the 6.70 ms of a decode step, so time sharing bills its shared energy as 23.6𝑎 instead of 𝑎. It charges 𝐴 and 𝐵, which only decode, 7.9𝑎 each for the prefill of 𝐶 (step 3 of Figure 3b). Against the exact Shapley values of the 8-request groups under continuous batching, with the same own energies, time sharing has an 𝐿1 error of 0.019–0.112 and step sharing 0.011–0.027 (Figure 5b), with the largest gap on the long inputs of the code and Mooncake traces.

in 8-request batches the shared energy is 0.832 and 0.886 on the two conversation traces and 0.681 on the agent trace, but only 0.183 on the code trace, where long inputs and short outputs make the own energy dominate. As shown in Figure 6b, it also falls as more requests share the steps. In static batches drawn from the Azure conversation trace, the shared energy of a request falls from 70.6 J with 16 requests to 5.4 J with 256, while its own energy stays at 16.9–20.5 J. No single price per token can therefore track the shared energy, and a fair split has to follow the steps of the batch. Insight 5. Under continuous batching and distributed deployment, the steps of a request depend on the other requests. Removing requests shortens the remaining steps, and later arrivals join different ones. On 40 random subsets of each of seven 128request trace windows, the law predicts the energy of a subset with an average error of 0.2–0.7% when given the steps that the subset really takes, but 1.4–5.7% when given the steps of the full run with the other requests removed. In the distributed deployment, 41–50% of the pairs of requests change whether they share a node. The steps of a coalition are therefore not in the log of the full run and have to be reconstructed.

own energy

300

100

250

Energy per request (J)

Energy per request (J)

Online ledger

Offline attribution (CPU)

Figure 5: (a) Duration of each step against the tokens it processes. (b) Error of splitting the shared energy by time or by step.

200 150 100 50 0

coefficients

Scheduler

coefficients

0.15

(b) L1 vs. exact Shapley value

shared energy

vLLM engine

0.027 0

(a) Tokens processed in the step

one normal run

fit

0 0

Serving (GPU)

once per configuration

Microbenchmark

0.112

Mooncake

decode steps: 6.70 ms

Calibration

Azure conv.

BurstGPT Mooncake

(a) Trace, 8 requests

Azure code

5

80

As shown in Figure 7, StepSplit turns one run of the attributed workload into per-request shares. A microbenchmark calibrates the energy law once per model and GPU configuration. While the engine runs, an online ledger computes the closed-form shares from the scheduler log. After a run under continuous batching, offline attribution replays the scheduler on a CPU and adds the resulting correction to the closed-form shares. Both divide the measured energy among the requests in proportion to these shares.

60 40 20 0

16

32

64

128

StepSplit

256

(b) Azure conv., requests in the batch

Figure 6: (a) Energy per request in static 8-request batches of each trace. (b) Energy per request against batch size in static batches from the Azure conv. trace.

5.1

A Closed-Form Shapley Value

Following Insight 1, let a run consist of steps 𝑘 = 1, . . . , 𝐾. Let 𝑀𝑘 ⊆ N be the 𝑚𝑘 requests scheduled in step 𝑘, whether prefilling or decoding. Engines cap the tokens per step, so a long input is prefilled in chunks over several steps alongside the decode tokens of other requests (chunked prefill [1]). We model the energy of step 𝑘 as ∑︁ 𝐸𝑘 = 𝑎 + 𝑒𝑖,𝑘 , (2)

Insight 4. The fraction of shared energy varies widely across traces and batch sizes. Because the shared energy of a step is fixed while the own energy grows with the tokens in it, the fraction of shared energy depends on the lengths of the requests and on how many of them share each step. For Qwen2.5-7B on the H100 it is 0.110–0.898 of the batch energy. As shown in Figure 6a,

𝑖 ∈𝑀𝑘 5

(a) full run, as logged

where the shared energy 𝑎 is the same in every step and 𝑒𝑖,𝑘 depends only on the work of request 𝑖 in the step. A chunk  of 𝜋𝑖,𝑘 input tokens starting at input position 𝑜𝑖,𝑘 costs 𝑐 1 𝜋𝑖,𝑘 +𝑐 2 (𝑜𝑖,𝑘 + 𝜋𝑖,𝑘 ) 2 −  2 , and an output token produced at context length ℓ 𝑜𝑖,𝑘 𝑖,𝑘 costs 𝑏 + 𝜅 ℓ𝑖,𝑘 . Here 𝑐 1 is the computation per input token, 𝑐 2 is the attention among the input tokens, 𝑏 is the computation per output token, 𝜅 is the read of one cached token, and 𝑎 is mostly the read of the weights. Idle power enters each coefficient through the time its work takes. Summed over the steps of a request, these terms give its own energy 𝑒𝑖 = 𝑐 1 𝑝𝑖 + 𝑐 2 𝑝𝑖2 + 𝑏 𝑑𝑖 + 𝜅 ctx𝑖 ,

ctx𝑖 =

𝑑𝑖 ∑︁

(𝑝𝑖 + 𝑗).

C arrives

9 steps

A

A: long prefill

B

B: stalled

1

waits

C (b) without A: logged timeline 9 steps, 3 shared: 9a

B

kept, A gone

C (c) without A: scheduler replayed 12 steps, none shared: 12a 3

B C

2

time

(3) one decode step, 7 ms

𝑗=1

The step that ends the prompt also produces the first output token, so a request produces one token in each of 𝑑𝑖 steps. The prefill terms telescope over chunks, so chunking does not change 𝑒𝑖 . The steps in the log of the full run form its logged timeline. On this timeline, the energy law gives every coalition 𝑆 the energy ∑︁ 𝑣ˆ (𝑆) = 𝑒𝑖 + 𝑎 𝐾 (𝑆), (4)

Figure 8: Why the scheduler is replayed. (a) In the full run, 1 stalls B while C arrives. (b) Read off the long prefill of A ○ the logged timeline, the coalition {𝐵, 𝐶} takes 9 steps, and 3 of them are shared. (c) When replayed, B finishes before C 2 and none of the 12 steps is shared ○. 3 arrives ○,

𝑖 ∈𝑆

steps of continuous batching, Eq. (4) is a coverage game with the same equal-split solution [11, 14].

where 𝐾 (𝑆) = {𝑘 : 𝑀𝑘 ∩ 𝑆 ≠ ∅} is the number of logged steps that hold a member of 𝑆. We call 𝑣ˆ the model game to tell it from the measured game 𝑣 of §2.1. The output length 𝑑𝑖 of each request is fixed (§2.3). Under static batching with every prompt prefilled in the first step, request 𝑖 is therefore scheduled in steps 1 to 𝑑𝑖 whatever else is served. The logged steps that hold a member of 𝑆 are then exactly the steps that 𝑆 goes through when served alone, and step 𝑘 holds 𝑀𝑘 ∩ 𝑆. Under continuous batching, the steps of a coalition differ from the logged ones, and §5.3 reconstructs them.

5.2

Calibration

The microbenchmark of §4.1 calibrates the law without trace data. A configuration that serves only the 8-request groups of §6.2 runs the 96 runs with 𝑚 ≤ 8, which cover the requests per step it serves. A configuration that also serves larger batches runs all 180. For Qwen2.5-7B on the H100, the 180 runs take 7.0 minutes, and the runs of every configuration take 2.5–17.8 minutes. Non-negative least squares fits the five coefficients from these runs, and every experiment of a configuration and its online ledger use them. Cards of the same GPU model differ, and the fitted shared energy of two H100s differs by 3.0–3.7%. To keep this difference out of the validation, we fit the coefficients for each card when the 8-request groups of a configuration ran on several cards. In a tensor-parallel group, the GPUs execute every step together, so the run keeps one step log. Their summed energy follows the same law with its own coefficients, and the shared energy then also covers the communication between the GPUs. Because the measured energy is divided in proportion to the shares, a common scale error in the coefficients cancels and only the ratios 𝑐 1 : 𝑐 2 : 𝑏 : 𝜅 : 𝑎 matter.

Proposition 5.1 ([14, 22]). The Shapley value of request 𝑖 in the model game is exactly ∑︁ 𝑎 𝜑 𝑖 = 𝑒𝑖 + 𝑠𝑖 , 𝑠𝑖 = . (5) 𝑚𝑘 𝑘: 𝑖 ∈𝑀𝑘

This result holds unchanged when the shared energy of a step depends on the number of requests in it, with 𝑎(𝑚𝑘 ) in place of 𝑎. Í Proof. With 𝑎(0) = 0, write the model game as 𝑣ˆ = 𝑣 0 + 𝑘 𝑣𝑘 Í with 𝑣 0 (𝑆) = 𝑖 ∈𝑆 𝑒𝑖 and 𝑣𝑘 (𝑆) = 𝑎(|𝑀𝑘 ∩ 𝑆 |). The Shapley value is linear in the game. In 𝑣 0 every marginal contribution of 𝑖 is 𝑒𝑖 , so its share is 𝑒𝑖 . In 𝑣𝑘 a request outside 𝑀𝑘 never changes the value and receives nothing. Two requests in 𝑀𝑘 change |𝑀𝑘 ∩ 𝑆 | in the same way for every 𝑆 that contains neither. They therefore receive equal shares, which add up to 𝑣𝑘 (N ) = 𝑎(𝑚𝑘 ), so each request in 𝑀𝑘 receives 𝑎(𝑚𝑘 )/𝑚𝑘 . Summing over 𝑘 gives Eq. (5). No monotonicity or concavity of 𝑎(𝑚) is used. □

5.3

Scheduler Replay

As shown in Figure 8, we reconstruct the steps of a coalition under continuous batching by replaying the scheduler on a CPU. The energy law still holds on the steps that the coalition goes through, so the game becomes ∑︁ 𝑣ˆ (𝑆) = 𝑒𝑖 + 𝑎 T (𝑆) , (6)

We call Eq. (5) step sharing. As shown in Figure 3b, a request pays its own energy 𝑒𝑖 and its shared energy 𝑠𝑖 , which is an equal part of the shared energy of every step it is in. The rule replaces the 2𝑁 − 1 runs of Eq. (1) by one pass over the log of the run being Í attributed in 𝑂 ( 𝑘 𝑚𝑘 ) operations. The result is classical, and the new finding is that measured GPU energy has this structure. Under static batching, the shared term of Eq. (4) is 𝑎 max𝑖 ∈𝑆 𝑑𝑖 , which is the airport game [22]. On the logged

𝑖 ∈𝑆

where T (𝑆) are the steps that the scheduler produces for 𝑆. A CPU can replay the vLLM scheduler because it is deterministic given arrival times, input lengths, and output lengths. Each step serves the running requests first within the token budget and 6

5.5

then admits arrived requests in arrival order into the remaining budget. Which requests have arrived by the next step depends on the duration of the current step, which we predict as 𝑡ˆ𝑘 = 𝜏0 +𝜏1 𝑃𝑘 +𝜏2𝑛𝑘 +𝜏3 𝐿𝑘 +𝜏4𝐴𝑘 , linear in the prefill tokens 𝑃𝑘 , decoding requests 𝑛𝑘 , decode context 𝐿𝑘 and prefill attention 𝐴𝑘 of step 𝑘. Fitted on the attributed run alone, this model has a median error of 0.2–4.8% per step. Fitting it on 14 to 30 runs of the same workload instead changes the 𝐿1 error of the shares by at most 0.005, so one run is enough. We reimplement the vLLM scheduler for the replay. For other engines, the scheduler code itself can be driven on a CPU, as GPU-free emulators do [2]. In a distributed deployment, a least-loaded router sends each arriving request to the node with the fewest requests in flight, so removing one request can move later ones to another node. We therefore also replay the router: at each arrival, it picks a node from the replayed state of the nodes. The energy of a coalition is Eq. (6) summed over the nodes, each with the coefficients of its GPUs. Any router that decides deterministically from the state it observes, such as round-robin, can be replayed this way.

5.4

6

6.1

Under continuous batching, the steps change with the coalition, so we compute the Shapley value of the replay game of Eq. (6) numerically. For 8 requests, we enumerate all 255 coalitions. Larger windows need permutation sampling [6], and sampling the replay game directly converges slowly (§6.8). We therefore sample only the correction to the closed form 𝜑𝑖 of Eq. (5). Within one permutation, the marginal contributions of the model game are close to those of the replay game because the two games differ only where the schedule changes. Let Δ𝑖 (𝜎) and log Δ𝑖 (𝜎) be the marginal contributions of request 𝑖 in permutation 𝜎 in the replay game and in the model game.

Setup

All experiments use the setup of §4.1, with asynchronous scheduling disabled so that each step is scheduled after the previous one finishes. Prefix caching is disabled except in §6.7. Measured Shapley reference. We compare every allocation with Shapley values measured on the GPU (§2.1). For 8 requests, we run all 255 subsets of a request group in random order so that thermal drift is not a systematic error, and compute Eq. (1) exactly. Under continuous batching, the requests of a group arrive 0.5 s apart, and a subset keeps the arrival times of its members. A budget of 65,536 tokens per step, or 262,144 for large static batches, prefills every prompt in one step. For 𝑁 ≥ 16, we estimate the Shapley value by permutation sampling [6] with 50–150 antithetic pairs. We run all 𝑁 − 1 proper prefixes of each sampled order and its reverse. We call the two references the exact and the sampled Shapley value. Together they take 529,513 runs, and each run is checked for preemptions and exact output lengths. The busy energy leaves out host stalls, which are the part of a step beyond three times its expected duration. The measurement follows Luo et al. [23]. Metrics. The error of an allocation is its normalized 𝐿1 distance to the reference (§3), with both adding up to the measured energy 𝑣 (N ) of the full run. It is 0 when the allocation matches the reference and 2 when the two give all energy to different requests. For 𝑁 ≥ 16, the sampled Shapley value 𝜑ˆ has a sampling error that depends on the measured permutations, so we compare every allocation with a reference that cancels most of it. On the same permutations, the game 𝑀 of StepSplit gives the estimate 𝜑ˆ 𝑀 . Its Shapley value 𝜑 𝑀 comes in closed form under static batching and from 2,000 CPU permutation pairs under continuous batching (3,000 in the distributed deployment). The measured and the modeled marginal contributions move together, so 𝜑ˆ 𝑀 − 𝜑 𝑀 shows how far the measured permutations are off, and the reference 𝑟 = 𝜑ˆ − (𝜑ˆ 𝑀 −𝜑 𝑀 ) removes it. As in Proposition 5.2, 𝑟 is unbiased whatever 𝑀 is, and a less accurate 𝑀 only makes it noisier. Against 𝑟 , the error of StepSplit is the difference between its game and the measured game on

Proposition 5.2. For 𝑄 permutations 𝜎1, . . . , 𝜎𝑄 , each uniformly distributed, 𝑄

 1 ∑︁  log Δ𝑖 (𝜎𝑞 ) − Δ𝑖 (𝜎𝑞 ) 𝑄 𝑞=1

Evaluation

Our evaluation answers four questions. How close is StepSplit to the exact Shapley value of 8-request batches on three GPU generations (§6.2)? Does it stay close in static batches of up to 256 requests and for up to 128 requests under continuous batching (§6.3, §6.4)? Does it carry over to a distributed deployment (§6.5), to MoE models (§6.6) and to prefix caching (§6.7)? And what do the online ledger and the replay cost (§6.8)?

Sampling the Correction

𝜑˜ 𝑖 = 𝜑𝑖 +

Online Ledger

The online ledger wraps three methods of the vLLM scheduler and needs no change to the engine. After each scheduling decision, the ledger adds to every scheduled request its term 𝑒𝑖,𝑘 of Eq. (2) and its part 𝑎/𝑚𝑘 of the shared energy. A background thread closes an attribution window every second at a counter update. It subtracts the idle power over the parts of the window that no step covers and over host stalls, so only waiting for arrivals is left out. It then divides the remaining energy among the requests in proportion to the parts they accumulated in the window, splitting a step that spans two windows by its overlap with each.

(7)

is an unbiased estimate of the Shapley value of request 𝑖 in the replay game. Proof. By Eq. (1), the Shapley value of a game is the expected marginal contribution of 𝑖 over a uniformly random permutation. The expectation of Δ𝑖 (𝜎𝑞 ) is therefore the Shapley value in the log replay game, and that of Δ𝑖 (𝜎𝑞 ) is 𝜑𝑖 , so 𝜑𝑖 and the mean of log Δ𝑖 (𝜎𝑞 ) cancel in expectation. □ This estimate is a control variate, and its variance comes only from the difference between the two games. On one engine, it therefore needs far fewer permutations than sampling the replay game itself (§6.8). In the distributed deployment, once the routing changes, the model game with the logged routing no longer tracks the replay, and the control variate helps little. The permutations run in parallel, in at most 105 MiB each. 7

the same permutations (paired), plus the sampling error of its CPU estimate, in expectation over the CPU pairs. The reference does not favor StepSplit. For request 𝑖, 𝜑𝑖𝑀 − 𝑟𝑖 = 𝜑ˆ𝑖𝑀 − 𝜑ˆ𝑖 , whose expectation is 𝜑𝑖𝑀 − 𝜑𝑖 , the error of StepSplit against the Shapley value 𝜑𝑖 of the measured game, and the sampling noise of this difference can only raise its 𝐿1 . Noise. Identical runs on the H100 differ in energy by 0.1–4.3%, most for short runs. A marginal contribution is the difference of two runs, so the reference is noisy, and we read every result against a noise level. The reference noise applies to every allocation. For 8 requests, it is the 𝐿1 between two exact Shapley values from measuring all 255 subsets twice,√and an allocation equal to the true value would be expected at 1/ 2 of it. For 𝑁 ≥ 16, it is the mean 𝐿1 between bootstrap resamples of 𝑟 , over the measured and the CPU pairs, and 𝑟 itself. Because the error of StepSplit against 𝑟 is paired, it can also be read against the paired noise, the 𝐿1 that the paired comparison would√︁have if the game of StepSplit matched Í the measured game. It is 2/𝜋 𝑖 se𝑖 /𝑣 (N ), with se𝑖 the standard error of the paired difference of request 𝑖. Baselines. Token, Solo and JCalib are defined in §3. Where Solo has no run of a request alone, it uses the runs in which the request came first in a sampled permutation. JCalib is trained on the Shapley values of other request sets of the same workload, exact or sampled, whichever makes it most accurate.

6.2

groups, StepSplit has a median relative error of 1.1%, and a p95 of 5.6% over the requests above 1% of the energy of their batch. Under continuous batching, these are 1.5% and 8.9%. The low 𝐿1 therefore does not hide large errors on single requests.

6.3

Static Batching with up to 256 Requests Solo

JCalib

StepSplit 0.8

0.6

0.6

L1 vs. reference

L1 vs. reference

Token 0.8

0.4

0.2

0

reference noise

0.4

0.2

0 16

32

64

128

(a) Azure conv., requests (N)

256

16

32

64

128

256

(b) BurstGPT, requests (N)

Figure 9: 𝐿1 against the reference under static batching with up to 256 requests. Figure 9 compares StepSplit and the baselines with the reference 𝑟 of §6.1 for Qwen2.5-7B on the H100. A batch of 𝑁 requests holds the first 𝑁 of 256 requests drawn from the Azure conversation or the BurstGPT trace. Each trace and size has 100–150 antithetic permutation pairs. Because requests finish at different steps, a step holds 7 to 102 requests on average. At 16 and 32 requests, StepSplit is at 0.005–0.011 against a reference noise of 0.004–0.007. From 64 to 256 requests, its error grows to 0.015–0.043, but the reference noise grows with it, to 0.014–0.041. At every size, StepSplit thus agrees with the measured marginal contributions within their noise. The reference takes 2,484 to 51,008 GPU runs per trace and size, while StepSplit uses one run. The baselines stay far from the reference at every size. Token is at 0.134–0.433 and Solo is at 0.153–0.394. JCalib is at 0.126–0.266.

Exact Shapley Values in 8-Request Batches

Table 2 compares StepSplit with the exact Shapley value for every platform and model. The two main models on the H100 have four groups per trace, and the other rows have one or two. Static batching. Under static batching, StepSplit stays close to the reference noise. Here every coalition goes through the logged steps that hold its members (§5.1), so the comparison tests the energy law and Proposition 5.1. Over all rows, StepSplit is at 0.004– 0.023 against reference noise of 0.002–0.027, and averaged over the 124 groups it is at 0.011 against 0.008. Token is at 0.411–0.613, 20× to 147× further than StepSplit. Solo is at 0.072–0.274, 4.0× to 34× further. JCalib is trained on exact Shapley values and is at 0.086– 0.198. StepSplit uses no such values, yet its error is lower than that of JCalib in all 124 groups, and 11× lower in the median group. Continuous batching. For the dense and hybrid models, StepSplit is at 0.005–0.019 against reference noise of 0.003–0.012, and at 0.014–0.033 without replay. It is within 0.003 of what the timelines of the coalitions themselves give, which is what an exact replay would give. Averaged over all 124 groups, StepSplit is at 0.017 against a reference noise of 0.008. Token is at 0.431–0.598, 13× to 113× further than StepSplit. Solo is at 0.089–0.195, 2.2× to 25× further. Even without replay, the error of step sharing is lower than that of JCalib in 122 of the 124 groups, and 5.4× lower in the median group. Multiple GPUs. StepSplit is as accurate with tensor parallelism as on one GPU. Over two and four H100s, it is at 0.009–0.011 under static batching, against 0.008–0.011 for the same groups on one GPU, and at 0.011–0.019 under continuous batching. Qwen2.5-72B on four A800s is at 0.007 and 0.013. Per request. The 𝐿1 is the energy-weighted mean of the perrequest relative error |𝜑ˆ𝑖 − 𝜑𝑖 |/𝜑𝑖 . Over all requests of the static

6.4

Continuous Batching with up to 128 Requests

Setup. We run Qwen2.5-7B on the H100 under continuous batching on windows of four traces. Each window holds the first 𝑁 requests of a 128-request window and has 50 antithetic permutation pairs. To reach larger batches, the Azure conversation, Mooncake and Azure code windows run at 10× the trace rate. The BurstGPT arrivals are compressed 12,000×, which matches the request rate of the Azure conversation windows. Unlike the 8-request groups, requests arrive as in the trace, with 2.8 to 36.4 requests per step. In addition, the default budget of 16,384 tokens per step splits long inputs over several steps while new requests wait. As in the 8-request groups, a subset keeps the arrival times of its members, as if the other requests had never been sent. Without replay, the error grows with the window. As shown in Figure 10, the error without replay on the Azure conversation trace grows with the window, from 0.014 at 16 requests to 0.179 at 128. This error is above the reference noise and has a consistent direction. A long prefill stalls the batch while arrivals accumulate. Afterwards more requests decode together, so the same tokens take fewer steps. The logged timeline therefore over-charges long inputs 8

Table 2: Mean 𝐿1 over request groups against the exact Shapley value of 8 requests. Noise is the reference noise, the 𝐿1 between two measurements of the same exact Shapley values.

GPU

Model

Static batching

Arch. GPUs Groups

Continuous batching

Noise Token Solo JCalib StepSplit Noise Token Solo JCalib StepSplit

H100 (Hopper)

Qwen2.5-7B dense Qwen2.5-7B dense Qwen2.5-14B dense Qwen2.5-14B dense Qwen2.5-14B dense Qwen2.5-72B dense Gemma-4-12B dense Qwen3.8-27B hybrid gpt-oss-20b MoE Qwen3-30B-A3B MoE

1 2 1 2 4 4 1 1 1 1

16 4 16 8 8 4 4 4 4 4

0.009 0.011 0.006 0.006 0.003 0.006 0.004 0.002 0.027 0.006

0.559 0.594 0.551 0.584 0.586 0.567 0.585 0.582 0.438 0.532

0.274 0.139 0.269 0.254 0.254 0.130 0.127 0.128 0.091 0.079

0.175 0.091 0.170 0.152 0.153 0.094 0.097 0.099 0.169 0.086

0.008 0.009 0.012 0.011 0.011 0.009 0.021 0.007 0.015 0.011

0.007 0.004 0.005 0.009 0.006 0.006 0.006 0.003 0.009 0.006

0.516 0.565 0.521 0.552 0.550 0.569 0.598 0.561 0.452 0.523

0.140 0.142 0.155 0.158 0.137 0.166 0.177 0.144 0.118 0.089

0.159 0.151 0.138 0.131 0.141 0.097 0.118 0.092 0.173 0.090

0.011 0.019 0.014 0.018 0.013 0.011 0.019 0.013 0.020 0.040

A800 (Ampere)

Qwen2.5-7B dense Qwen2.5-72B dense Gemma-4-12B dense Qwen3.8-27B hybrid gpt-oss-20b MoE Qwen3-30B-A3B MoE

1 4 1 1 1 1

8 4 4 4 4 4

0.016 0.005 0.003 0.003 0.008 0.015

0.595 0.547 0.539 0.557 0.411 0.525

0.226 0.124 0.127 0.127 0.085 0.081

0.116 0.090 0.092 0.097 0.198 0.087

0.007 0.007 0.015 0.008 0.020 0.011

0.012 0.004 0.006 0.003 0.011 0.004

0.578 0.530 0.531 0.542 0.431 0.506

0.190 0.119 0.139 0.115 0.127 0.098

0.141 0.086 0.090 0.089 0.185 0.081

0.015 0.013 0.018 0.012 0.026 0.029

Qwen2.5-7B dense Gemma-4-12B dense PRO 6000 Qwen3.8-27B hybrid (Blackwell) gpt-oss-20b MoE Qwen3-30B-A3B MoE

1 1 1 1 1

8 4 4 4 4

0.007 0.009 0.002 0.027 0.005

0.613 0.555 0.588 0.463 0.520

0.242 0.134 0.137 0.091 0.072

0.131 0.089 0.105 0.149 0.095

0.008 0.011 0.004 0.023 0.013

0.006 0.005 0.003 0.052 0.013

0.590 0.544 0.567 0.471 0.514

0.195 0.157 0.127 0.174 0.099

0.150 0.091 0.104 0.180 0.080

0.014 0.014 0.005 0.035 0.024

L1 vs. reference

Token

Solo

JCalib

StepSplit, no replay

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

StepSplit

StepSplit removes most of this error. On the Azure conversation trace, StepSplit is at 0.008–0.038, 3.7× to 5.3× lower than without replay from 32 requests on, and on BurstGPT at 0.010–0.040, 2.3× to 5.1× lower. It is 0.015–0.027 on the Mooncake trace and 0.014–0.032 on Azure code. This is 1.3× to 2.0× the paired noise on three traces and 1.3× to 3.8× on Mooncake. The gain comes from the replay, which reproduces the engine. Its full-window step counts are within two steps of the logged ones, and its subset step counts are within 0.7% of those on the timelines of the coalitions, against 0.6–10% for the logged timeline. The baselines are far from the reference in every window. Token is at 0.206–0.717, Solo at 0.206–0.733 and JCalib at 0.089–0.560. Even without replay, StepSplit is lower than all three in every window.

0 16

32

64

128

16

(a) Azure conv., requests (N)

L1 vs. reference

reference noise

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

32

64

128

(b) Mooncake, requests (N)

0.8

6.5

0 16

32

64

128

(c) Azure code, requests (N)

16

32

64

Distributed Deployment

Table 3: 𝐿1 in the distributed deployment. Noise is the reference noise.

128

(d) BurstGPT, 12,000×, requests (N)

Figure 10: 𝐿1 against the reference in trace windows of up to 128 requests.

Trace

and under-charges long outputs. Its size differs across traces. On BurstGPT, at the same request rate, it reaches 0.117 at 64 requests. On the Mooncake trace, it is 0.036–0.066. There, 121 of 128 requests wait in the queue, and removing requests mostly shortens the wait. The Azure code trace has 2.8 to 10.0 requests per step, so removing a request barely moves the others, and the logged timeline is within 0.004 of the replay.

𝑵

Noise Token Solo JCalib StepSplit

32 Azure conv. 64 128

0.020 0.037 0.054

0.431 0.342 0.405

0.319 0.298 0.466

0.300 0.303 0.468

0.020 0.038 0.064

32 64 128

0.019 0.025 0.035

0.632 0.402 0.265

0.294 0.485 0.651

0.140 0.216 0.389

0.029 0.028 0.038

Mooncake

Setup. We use the distributed deployment of §4.1. Its router sends each request at its arrival time to the node with the fewest requests in flight, like the vLLM load balancer. Both nodes use the 9

coefficients of Qwen2.5-72B on four H100s. The windows hold 32, 64 and 128 requests of the Azure conversation and Mooncake traces at 10× their rate. Each window has 50 antithetic permutation pairs. The routing changes with the subset. Removing requests changes which node the router sends the remaining requests to. On 40 random subsets of each window, the energy law predicts the energy of a subset with an average error of 0.17–0.42% when given the steps that the subset really takes. Given the logged steps of the full run, the error is 1.9–6.0%, and given the steps of the replay, it is 0.20–0.63%. StepSplit stays close to the reference noise. StepSplit is at 0.020–0.064 against reference noise of 0.019–0.054 (Table 3). The replay is also robust to near-ties. Where near-simultaneous arrivals and completions flip 0.7% of the routing decisions, StepSplit stays at 0.038 against reference noise of 0.035. Replaying only the schedulers, with every request kept on its node of the full run, gives 0.070–0.178, against 0.074–0.180 without replay, so the router must be replayed as well. None of the baselines comes close. Token is at 0.265–0.632, Solo at 0.294–0.651 and JCalib at 0.140–0.468.

6.6

place of links. The rule needs the routing log of the run and assumes that a request is routed to the same experts in every coalition. Splitting per expert is more accurate than splitting each step equally. Table 4 evaluates the split per expert on the H100. The windows hold 16, 32 and 64 requests of the Azure conversation and Mooncake traces at 10× the trace rate, with 50 permutation pairs each and 3.3 to 28 requests per step. To compare only the two splits, we use the steps that each subset really takes instead of the replay, and fit the law, including 𝑎 0 and 𝑤, on the runs of the window instead of the microbenchmark. In all 12 windows, the split per expert has a lower error than splitting each step equally, at 0.007–0.036 against 0.011–0.065.

6.7

Prefix Caching

Table 5: 𝐿1 with prefix caching. Noise is the reference noise, and First charges a block to the request that computed it. Batching

Group

Noise Token Solo JCalib First StepSplit

Static

Mooncake A Mooncake B System prompt Nested

0.004 0.005 0.008 0.006

1.307 1.368 0.538 0.659

0.197 0.181 0.133 0.169

0.156 0.087 0.112 0.137

0.094 0.132 0.020 0.039

0.005 0.005 0.002 0.003

Continuous

Mooncake A Mooncake B System prompt Nested

0.007 0.007 0.008 0.003

1.240 1.417 0.616 0.712

0.157 0.243 0.327 0.351

0.133 0.143 0.305 0.361

0.091 0.090 0.027 0.029

0.010 0.011 0.004 0.008

Mixture-of-Experts Models

Table 4: 𝐿1 of the split per expert (§6.6). Noise is the paired noise, and Equal splits the shared energy of each step equally among its requests, whatever experts they use. Trace

𝑵

Qwen3-30B-A3B

gpt-oss-20b

A shared prefix should be split equally. With prefix caching, when several requests share a prompt prefix, the engine computes its prefill once for the first of them, and the others reuse it. This prefill is a second shared cost like the weight read, because a block is computed once whenever one of the requests that contain it is present. The Shapley value therefore splits the prefill energy of each block equally among the requests that contain it, and StepSplit adds this split to step sharing. In the replay, a block is cached for a request once a request admitted earlier has computed it. Setup. We enable prefix caching, empty the cache before every run, and measure all 255 subsets of four groups of 8 requests with Qwen2.5-7B on the H100. Mooncake A and B come from the Mooncake trace, which records the prompt blocks of every request. In System prompt, 8 conversation prompts follow a common 1,024token system prompt. In Nested, each prompt is a prefix of the next, as in a multi-turn conversation. The cache saves 5–25% of the group energy. We compare StepSplit with First, which charges each block to the request that computed it, as reading the steps of the run does. Both use coefficients from the microbenchmark, which never saw a cached prefix. JCalib is trained on the other three groups or on the 8-request groups of the same trace, whichever is more accurate, and Token bills cached tokens as if they were computed. With the cache on, StepSplit stays close to the reference noise. As shown in Table 5, StepSplit is at 0.002–0.011 against reference noise of 0.003–0.008. In contrast, First is at 0.020–0.132, Token at 0.538–1.417, Solo at 0.133–0.351 and JCalib at 0.087–0.361. On the Mooncake trace windows of 16 and 32 requests with the cache on, the cached tokens match the prefix tree in every run, and the microbenchmark law with the prefill terms of the tree predicts the energy of a subset with an average error of 0.7–0.9%.

Noise Equal Expert Noise Equal Expert 16 Azure conv. 32 64

0.007 0.011 0.028

0.031 0.060 0.065

0.016 0.022 0.036

0.011 0.013 0.028

0.034 0.053 0.043

0.023 0.016 0.035

16 32 64

0.006 0.009 0.013

0.011 0.018 0.033

0.007 0.015 0.012

0.008 0.020 0.022

0.030 0.036 0.031

0.013 0.032 0.026

Mooncake

The shared energy grows with the requests in a step. A step of a mixture-of-experts (MoE) model reads only the experts its tokens are routed to, and more requests touch more experts. On the H100, from 1 to 8 requests per step, the shared energy grows from 1.3 to 2.4 J for gpt-oss-20b and from 2.5 to 5.9 J for Qwen3-30B-A3B on the two GPUs of its serving windows, but by 4% for the dense Qwen2.5-7B. The shared term becomes a function 𝑎(𝑚), which the microbenchmark measures, and Proposition 5.1 gives each request 𝑎(𝑚𝑘 )/𝑚𝑘 of every step it is in. The MoE rows of Table 2 use this split and are at 0.020–0.040 under continuous batching. The read of each expert should be split among its users. In an MoE step, the weights outside the experts are read for all requests in the step, but an expert is read only for the requests routed to it. Here 𝑎 0 (𝑚𝑘 ) is the shared energy of the weights outside the experts, and 𝑤 is the energy of reading all 𝑋 experts of the MoE layers once. With 𝑅𝑖,𝑘 as the experts that request 𝑖 uses in step 𝑘 and 𝑞𝑘,𝑥 as the number of requests in step 𝑘 routed to expert 𝑥, the Shapley value is ∑︁ h 𝑎 0 (𝑚 ) 𝑤 ∑︁ 1 i 𝑘 𝜑 𝑖 = 𝑒𝑖 + + . (8) 𝑚𝑘 𝑋 𝑥 ∈𝑅 𝑞𝑘,𝑥 𝑘: 𝑖 ∈𝑀𝑘

𝑖,𝑘

Each expert is a game of the form of Proposition 5.1, which is the multicast cost sharing of Feigenbaum et al. [11] with experts in 10

6.8

Online Ledger and Replay Cost

by 18–40%. Because this choice acts on the shared term, a provider can charge the premium of a tensor-parallel deployment to the requests that need its latency. Similarly, the shared energy of an MoE step grows with the experts that its tokens touch (§6.6). An MoE engine that groups requests routed to the same experts would therefore lower the shared energy of their steps. Bills of users and of a continuous service. Because the engine schedules individual requests, the bill of a user is the sum of the shares of the requests from that user. In a step of 𝑚𝑘 requests, a user with 𝑛 of them pays 𝑛 𝑎/𝑚𝑘 , whereas a game with users as players would give every user in the step the same part. In addition, the share of a request depends on the requests it ran with. A provider that needs a price before the run can therefore quote one and use the fair share to settle the bill or to report the energy after the run. Finally, a provider attributes a continuous service window by window. Each window is its own game. Its players are the requests that arrive in it, and the state of the engine at its start, such as requests still running and cached prefixes, is held fixed. The requests of a window therefore pay the energy that they add to this state, and the requests still running pay in the window in which they arrived. The carbon of a request. Providers also report the carbon footprint of each request [10, 36]. The carbon of a step is its energy times the carbon intensity of the grid, which changes far more slowly than a step. Because the Shapley value is linear in the game, the carbon share of a request in a window of constant intensity is its energy share times that intensity, and its footprint is the sum over the windows it spans. The online ledger can therefore report the carbon of each request as it bills its energy, and the replay settles both after the run.

We run the ledger, which wraps the vLLM scheduler without changing the engine, on the H100 with Qwen2.5-7B on two 32-request trace windows at 3× the trace rate. To measure its overhead, we also run it on a 128-request window with the ledger off and on in alternation. Its shares equal the closed form recomputed from the scheduler log and sum to the busy energy of the counter within 0.2%. Attribution takes 47 𝜇s per step, which is 0.7% of the mean step time. The ledger changes completion time, throughput and time per output token by at most 0.3%, which is within run-to-run variation. Against the reference of the two windows, the ledger is at 0.086 and 0.114. The closed form on the logged timeline of the measured full run is at 0.076 and 0.112, so settling every second on the timeline of a separate run adds only 0.010 and 0.002. Because the ledger settles on the logged timeline, it carries the error of the logged timeline (§6.4). A provider can therefore bill while the engine runs and settle the bill after the run by replay, which is at an 𝐿1 of 0.008–0.040 on the trace windows of §6.4.

L1 vs. reference

StepSplit, no replay

direct sampling

StepSplit

1

1

0.3

0.3

paired noise

3,000 pairs 0.1

0.1

2,000 pairs 10 pairs, 14 s

0.03

0.03 1

10

10

2

10

3

(a) One engine, CPU time (s)

10

4

1

10

10

2

10

3

10

4

(b) Two nodes, CPU time (s)

Figure 11: 𝐿1 against the CPU time of the replay on the Azure conv. window of 128 requests.

8

The replay finishes in minutes. On one core of an Intel Xeon 6530P, one replay of a window of 16 to 128 requests takes 0.23– 7.74 ms on average, and the 255 subsets of an 8-request group take 0.02–0.69 s. For larger windows, StepSplit samples permutation pairs (§5.4) and replays every prefix of each, so its cost grows with the number of pairs. As shown in Figure 11, the error of StepSplit falls as more pairs are run, so a provider can spend more CPU time for a more accurate bill. On the Azure conversation window of 128 requests, which takes the longest to replay, StepSplit falls below the logged timeline within 10 pairs, or 14 CPU seconds. Over the trace windows of §6.4, an estimate within an 𝐿1 of 0.010 of its limit takes 1–2,446 pairs, which is 4.3× to 870× fewer than sampling the replay game directly, and at most 56.5 CPU minutes. In the distributed deployment, the 3,000 pairs of Table 3 take 86.8 CPU minutes on the same window, and StepSplit is at 0.064 against 0.058 in the limit. The pairs are independent and run in parallel, so on 32 cores both take under 3 minutes.

7

Related Work

The Shapley value in computer architecture. Choi et al. [7] attribute the writes of a shared flash cache to the consolidated workloads and model the subsets they do not observe. Concorde [24] divides the performance difference between two CPU designs among microarchitectural components, and ShapleyIQ [20] reduces the exponential cost of the Shapley value in a microservice system by decomposition. Like these works, we model the subsets we do not run and decompose the game into one game per step. Energy attribution. Time sharing [17, 40, 41] and Fair-CO2 [16] split shared power or carbon by time rather than by steps. Li et al. [21] show that token-based rules violate the Shapley axioms. Neuralwatt bills each request for the server energy measured while it is in flight, in proportion to its share of the tokens of all requests then in flight [25]. In our measurements on 8-request groups, this rule has an 𝐿1 error of 19.9–36.3%, depending on whether output tokens are weighted as in Token, against 0.8–1.1% for StepSplit. Roy [38] attributes energy offline per step by activity weights, and Yuan et al. [48] approximate the Shapley value online. Roy [37] argues that exact shares of the weight read need a replay of every subset, which our step game avoids. In contrast, StepSplit derives the Shapley value from the measured energy of a step and stays close to Shapley values measured on the GPU for up to 256 requests. Energy models and scheduler simulation. Prior work characterizes inference energy [31, 32], models it for mostly homogeneous

Discussion

Deployment choices set the shared term. Per-request shares make the energy cost of these choices visible. Tensor parallelism over two and four H100s makes a step 1.4× to 2.2× faster, but it raises the shared term by 14–40% and the energy of the same work 11

batches [45], or expresses it as a fixed cost per batch or step plus per-token costs [12, 43]. Attribution needs such a law for arbitrary heterogeneous subsets, and our microbenchmark calibrates one once per model and GPU configuration. Revati [2] runs serving schedulers without GPUs for capacity planning, whereas we replay the scheduler to find the steps of each subset. The cost of an MoE layer depends on the experts that a batch activates [15], and we split it by the routing.

9

[12] Carolina Fortuna, Vid Hanžel, Tim Strnad, and Blaž Bertalanič. 2026. Where Should Agents Live? Energy-Memory Characterization of Agentic AI for the Edge-Cloud Continuum. arXiv:2609.18283 https://arxiv.org/abs/2609.18283 [13] Gemma Team. 2026. Gemma 4 Technical Report. arXiv:2607.02770 https://arxiv. org/abs/2607.02770 [14] Victor Ginsburgh and Israel Zang. 2003. The Museum Pass Game and Its Value. Games and Economic Behavior 43, 2 (2003), 322–325. doi:10.1016/s0899-8256(03) 00013-7 [15] Vima Gupta, Jae Hyung Ju, Kartik Sinha, Ada Gavrilovska, and Anand Padmanabha Iyer. 2024. Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection. arXiv:2411.08982 https://arxiv.org/abs/2411.08982 [16] Leo Han, Jash Kakadia, Benjamin C. Lee, and Udit Gupta. 2025. Fair-CO2: Fair Attribution for Cloud Carbon Emissions. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA). Association for Computing Machinery, New York, NY, USA, 646–663. doi:10.1145/3695053.3731023 [17] Mohammad A. Islam and Shaolei Ren. 2016. A New Perspective on Energy Accounting in Multi-Tenant Data Centers. In USENIX Workshop on Cool Topics on Sustainable Data Centers (CoolDC). USENIX Association, Santa Clara, CA, 6 pages. https://www.usenix.org/conference/cooldc16/workshop-program/ presentation/islam [18] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 [19] Imran Latif, Alex C. Newkirk, Matthew R. Carbone, Arslan Munir, Yuewei Lin, Jonathan Koomey, Xi Yu, and Zhihua Dong. 2025. Single-Node Power Demand During AI Training: Measurements on an 8-GPU NVIDIA H100 System. IEEE Access 13 (2025), 61740–61747. doi:10.1109/ACCESS.2025.3554728 [20] Ye Li, Jian Tan, Bin Wu, Xiao He, and Feifei Li. 2024. ShapleyIQ: Influence Quantification by Shapley Values for Performance Debugging of Microservices. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4 (ASPLOS). Association for Computing Machinery, New York, NY, USA, 287–323. doi:10.1145/3623278. 3624771 [21] Yueying Lisa Li, Leo Han, G. Edward Suh, Christina Delimitrou, Fiodar Kazhamiaka, Esha Choukse, Rodrigo Fonseca, Liangcheng Yu, Jonathan Mace, and Udit Gupta. 2025. Fair, Practical, and Efficient Carbon Accounting for LLM Serving. ACM SIGMETRICS Performance Evaluation Review 53, 2 (2025), 99–103. doi:10.1145/3764944.3764967 [22] Stephen C. Littlechild and Guillermo Owen. 1973. A Simple Expression for the Shapley Value in a Special Case. Management Science 20, 3 (1973), 370–372. doi:10.1287/mnsc.20.3.370 [23] Qi Luo, Kunlin Li, Ziwen Wang, Dongsheng Wang, and Yun Chen. 2026. RequestLevel Energy Attribution for Batched LLM Serving. arXiv:2608.00026 https: //arxiv.org/abs/2608.00026 [24] Arash Nasr-Esfahany, Mohammad Alizadeh, Victor Lee, Hanna Alam, Brett W. Coon, David Culler, Vidushi Dadu, Martin Dixon, Henry M. Levy, Santosh Pandey, Parthasarathy Ranganathan, and Amir Yazdanbakhsh. 2025. Concorde: Fast and Accurate CPU Performance Modeling with Compositional Analytical-ML Fusion. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA). Association for Computing Machinery, New York, NY, USA, 1480–1494. doi:10.1145/3695053.3731037 [25] Neuralwatt. [n. d.]. Energy Methodology. https://docs.neuralwatt.com/energymethodology. No publication date stated; accessed 2026-10-31. [26] NVIDIA Corporation. 2026. NVML API Reference Guide. https://docs.nvidia. com/deploy/nvml-api/. Last updated 2026-09-09; accessed 2026-10-10. [27] OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 https: //arxiv.org/abs/2508.10925 [28] OpenAI. [n. d.]. Pricing. https://developers.openai.com/api/docs/pricing. No publication date stated; accessed 2026-11-03. [29] Felipe Oviedo, Fiodar Kazhamiaka, Esha Choukse, Allen Kim, Amy Luers, Melanie Nakagawa, Ricardo Bianchini, and Juan M. Lavista Ferres. 2026. Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling. Joule 10, 8 (Aug. 2026), 102430. doi:10.1016/j.joule.2026.102430 [30] Mark Patel, Clea Chen, Raphaela Wagner, Suryanshi Goyal, and Yvonne Ferrier. 2026. Frontiers of Compute: The Technologies to Reduce AI Inference Costs. McKinsey & Company. Retrieved November 4, 2026 from https://www.mckinsey.com/industries/semiconductors/our-insights/ frontiers-of-compute-the-technologies-to-reduce-ai-inference-costs [31] Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2024. Characterizing Power Management Opportunities for LLMs in the Cloud. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (ASPLOS). Association for Computing Machinery, New York, NY, USA, 207–222. doi:10.1145/3620666.3651329

Conclusion

Splitting the energy of a batch by tokens is far from the fair split, and the Shapley value has needed a GPU run for every subset of the requests. We showed by measurement that each step of batched inference costs a shared energy, mostly to read the weights, plus the energy that each request adds on its own. Under this structure, the Shapley value has a closed form computed from one run, and a CPU replay of the scheduler and the router supplies the steps when they change with the subset. Against Shapley values measured on the GPU, StepSplit is far closer than splitting by tokens or a regression trained on measured Shapley values, and it also runs online inside vLLM. We will release StepSplit, including the online ledger.

References [1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/ conference/osdi24/presentation/agrawal [2] Amey Agrawal, Mayank Yadav, Sukrit Kumar, Anirudha Agrawal, Garv Ghai, Souradeep Bera, Elton Pinto, Sirish Gambhira, Mohammad Adain, Kasra Sohrab, Chus Antonanzas, and Alexey Tumanov. 2026. Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving. arXiv:2601.00397 https://arxiv.org/abs/ 2601.00397 [3] anon8231489123. 2023. ShareGPT_Vicuna_unfiltered. https://huggingface.co/ datasets/anon8231489123/ShareGPT_Vicuna_unfiltered. A cleaned derivative of ShareGPT conversations; accessed 2026-10-31. [4] Anthropic. [n. d.]. Pricing. https://platform.claude.com/docs/en/about-claude/ pricing. No publication date stated; accessed 2026-11-03. [5] Sergio Aquino-Brítez, Pablo García-Sánchez, Andrés Ortiz, and Diego AquinoBrítez. 2025. Towards an Energy Consumption Index for Deep Learning Models: A Comparative Analysis of Architectures, GPUs, and Measurement Tools. Sensors 25, 3, Article 846 (2025), 30 pages. doi:10.3390/s25030846 [6] Javier Castro, Daniel Gómez, and Juan Tejada. 2009. Polynomial Calculation of the Shapley Value Based on Sampling. Computers & Operations Research 36, 5 (2009), 1726–1730. doi:10.1016/j.cor.2008.04.004 [7] Wonil Choi, Bhuvan Urgaonkar, Mahmut Kandemir, Myoungsoo Jung, and David Evans. 2020. Fair Write Attribution and Allocation for Consolidated Flash Cache. In Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Association for Computing Machinery, New York, NY, USA, 1063–1076. doi:10.1145/3373376.3378502 [8] Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A. Smith, Nicole DeCario, and Will Buchanan. 2022. Measuring the Carbon Intensity of AI in Cloud Instances. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT). Association for Computing Machinery, New York, NY, USA, 1877–1894. doi:10.1145/3531146.3533234 [9] Mian Dong, Tian Lan, and Lin Zhong. 2014. Rethink Energy Accounting with Cooperative Game Theory. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom). Association for Computing Machinery, New York, NY, USA, 531–542. doi:10.1145/2639108.2639128 [10] Cooper Elsworth, Keguo Huang, David Patterson, Ian Schneider, Robert Sedivy, Savannah Goodman, Ben Townsend, Parthasarathy Ranganathan, Jeff Dean, Amin Vahdat, Ben Gomes, and James Manyika. 2025. Measuring the Environmental Impact of Delivering AI at Google Scale. arXiv:2508.15734 https://arxiv.org/abs/2508.15734 [11] Joan Feigenbaum, Christos H. Papadimitriou, and Scott Shenker. 2001. Sharing the Cost of Multicast Transmissions. J. Comput. System Sci. 63, 1 (2001), 21–41. doi:10.1006/jcss.2001.1754 12

and Implementation (OSDI). USENIX Association, Carlsbad, CA, 521–538. https: //www.usenix.org/conference/osdi22/presentation/yu [48] Xianyi Yuan, Hanlong Liao, Kunming Zhang, Deke Guo, and Guoming Tang. 2026. Fine-Grained Energy Accounting in Production LLM Serving. In Proceedings of the 10th Asia-Pacific Workshop on Networking (APNet). Association for Computing Machinery, New York, NY, USA, 37–43. doi:10.1145/3820441.3820447

[32] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Buenos Aires, Argentina, 118–132. doi:10.1109/isca59077.2024.00019 [33] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA, 155–170. https://www.usenix.org/conference/ fast25/presentation/qin [34] Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 Technical Report. arXiv:2412.15115 https://arxiv.org/abs/2412.15115 [35] Qwen Team. 2026. Qwen3.8-27B Model Card. https://huggingface.co/Qwen/ Qwen3.8-27B. Official model repository; accessed 2026-10-31. [36] Samuel Rincé and Adrien Banse. 2025. EcoLogits: Evaluating the Environmental Impacts of Generative AI. Journal of Open Source Software 10, 111 (July 2025), 7471. doi:10.21105/joss.07471 [37] Probir Roy. 2026. Energy Identifiability and Conserved Provenance for Continuously-Batched LLM Serving: Formal Theory and Proofs. Technical report, Zenodo. doi:10.5281/zenodo.21461311 [38] Probir Roy. 2026. Scheduler-Step GPU Energy Attribution for Continuously Batched LLM Serving. Preprint, Zenodo. doi:10.5281/zenodo.21461304 [39] Lloyd S. Shapley. 1953. A Value for 𝑛 -Person Games. In Contributions to the Theory of Games, Volume II, Harold W. Kuhn and Albert W. Tucker (Eds.). Annals of Mathematics Studies, Vol. 28. Princeton University Press, Princeton, NJ, USA, 307–318. doi:10.1515/9781400881970-018 [40] Prateek Sharma and Alexander Fuerst. 2024. Accountable Carbon Footprints and Energy Profiling for Serverless Functions. In Proceedings of the ACM Symposium on Cloud Computing (SoCC). Association for Computing Machinery, New York, NY, USA, 522–541. doi:10.1145/3698038.3698531 [41] Kai Shen, Arrvindh Shriraman, Sandhya Dwarkadas, Xiao Zhang, and Zhuan Chen. 2013. Power Containers: An OS Facility for Fine-Grained Power and Energy Management on Multicore Servers. In Proceedings of the 18th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Association for Computing Machinery, New York, NY, USA, 65–76. doi:10.1145/2451116.2451124 [42] Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in Serving Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Santa Clara, CA, 965–988. https://www.usenix.org/conference/osdi24/presentation/sheng [43] Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton, and John Paul Shen. 2026. Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms. arXiv:2608.28044 https://arxiv.org/abs/2608.28044 Accepted at the IEEE International Symposium on Workload Characterization (IISWC) 2026. [44] Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2025. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD). Association for Computing Machinery, New York, NY, USA, 5831–5841. arXiv:2401.17644 doi:10.1145/3711896.3737413 [45] Grant Wilkins, Srinivasan Keshav, and Richard Mortier. 2024. Offline EnergyOptimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems. ACM SIGEnergy Energy Informatics Review 4, 5 (2024), 113–119. doi:10.1145/3727200.3727217 Presented at the 3rd Workshop on Sustainable Computer Systems (HotCarbon). [46] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv:2505.09388 https://arxiv.org/abs/2505.09388 [47] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and ByungGon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design 13

Related documents

• The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference#1084260
• Test-Time Compute Games#1389183
• Service-Induced Congestion in Memory-Constrained LLM Serving#1403134
• Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling#974358
• Prediction-Assisted Pricing and Admission for LLM APIs with Stochastic Token Consumption#820725