What Sets the Serving Speed of Looped Language Models? Measurements and Optimizations Graded by Exactness

W HAT S ETS THE S ERVING S PEED OF L OOPED L ANGUAGE M ODELS ? M EASUREMENTS AND O PTIMIZATIONS G RADED BY E XACTNESS

arXiv:2611.05150v1 [cs.DC] 3 Nov 2026

Dezhen Lu 1 Runguo Li 2 Zixiang Ni 3 Yuchen Tang 4 Xingquan Kong 5 Jiaxing Yang 6 Jiarui Guo 7 Lingxiang Gao 8

A BSTRACT Looped language models apply one block of layers many times per token, so a serving engine pays every per-layer overhead once per loop; we measure serving for Ouro and Huginn-0125 in an open-source looped-model engine and grade our optimizations from bitwise-identical changes to those that require an accuracy check. At batch 1, fusing all loop submissions leaves Ouro throughput at 0.99×; the 5689 small kernels per token are the real cost, and fusing them behind rounding fences raises throughput 1.11–1.59× while 1014/1014 requests remain identical. An approximate fusion is no faster and misses a one-point GSM8K rule on 87 questions (61 and 62 correct versus 64), so we do not ship it. Huginn’s released defaults avoid dead KV slots, reuse prefix-cache pages, and serve 1.77–3.16× faster than the base engine; Ouro reaches 1.65–2.15× and agent sessions 1.91–3.21×, while Huginn’s noise and prefix-reuse changes stay within one point on all 1319 Huginn test questions; Huginn’s packed prefill, −0.61 points on the same questions, meets that rule only on its point estimate. Position-keyed noise makes CPU cache reuse exact but still leaves 20 of 128 GPU full hits different, while loop fusion delays prefill admission and a 64-graph cache prevents eager fallbacks. Under prefill–decode disaggregation ExactLoop reaches up to 3.3× the base throughput with peer error handling disabled, and 3.1× in two of three runs with it enabled. Its transfer budget, sized to hold one request’s KV in flight, cuts the commit tail from about one second under the earlier budget to 16 ms (base: 2.8 s), removing the polling rounds that the engine’s interval would otherwise impose. 1

I NTRODUCTION

Looped, or recurrent-depth, language models reuse one block of transformer layers several times for each token. Huginn-0125 runs a four-layer block 32 times between a short prelude and coda (Geiping et al., 2025); Ouro runs all of its layers four times and can exit early (Zhu et al., 2025). Depth becomes a run-time knob, which makes these models attractive for reasoning at a fixed parameter count. For a serving engine a looped model looks like a much deeper model whose layers share weights. The natural first optimization is to fuse the loops of a decode step into one GPU submission, which removes host round trips that grow with depth. We tried this on vllm-rlt (InferMatrixLab, 2026), an opensource engine built for looped models (main branch at com1

University of Tübingen, Tübingen, Germany 2 University of Illinois Urbana-Champaign, Urbana, IL, USA 3 Xi’an Jiaotong University, Xi’an, China 4 Zhejiang Sci-Tech University, China 5 Southwest Petroleum University, China 6 Fudan University, Shanghai, China 7 Nankai University, Tianjin, China 8 Swan College, Central South University of Forestry and Technology, Changsha, China. Correspondence to: Dezhen Lu <dezhen.lu@student.unituebingen.de>. Preprint.

mit 66920da), and it did almost nothing at low load. On an RTX 5090, fusing Ouro’s four loops halved the submissions per token (6.0 to 3.0) and left batch-1 throughput at 0.99×; it helped only at higher load (1.09× at 32 requests). The loops are not the problem. The problem is that every per-layer cost is paid once per loop, so small costs that a conventional engine can ignore become large. We found two that matter most. Dead KV slots. The engine keeps a separate KV plane for each recurrence depth, so that a token at depth d attends to keys computed at depth d. Each plane reserves a slot for every layer. In Huginn only the four recurrent layers write deeper planes; the four prelude and coda layers write plane 0 only. Of the 8 × 32 = 256 layer slots a token reserves, 124 are never written. On a GH200 the base engine therefore stops scaling near 65 tokens/s, because only a few dozen requests fit in the KV pool. The same layout breaks prefix caching: the cache publishes a prefix only when every layer of every plane has written it, which never happens for Huginn. Kernel count. At batch 1, an Ouro token costs 5689 GPU kernels on a GH200, most of them small elementwise and reduction kernels from RMSNorm, rotary embedding (RoPE)

What Sets the Serving Speed of Looped Language Models?

and the gated activation, each repeated for 24 layers and four loops. CUDA graphs remove their launch cost but not their minimum run time. Fusing them is the obvious remedy, but fused kernels round differently, and in a recurrent model a one-ulp difference passes through 96 layer applications. Our first fused kernels changed GSM8K answers.

Table 1. The exactness ladder. Each ExactLoop change, the property it preserves, and the evidence. “Identical” counts requests whose generated token sequences match the same configuration without the change. rung change R0

The exactness ladder. An optimization for a deployed model is easier to adopt the less it changes outputs. Table 1 records what each ExactLoop change preserves at two levels: the arithmetic applied to the same inputs and batch, and the tokens observed after deployment. A R 0 change is bitwise identical under the same batch composition. It may change capacity or scheduling, which can in turn expose the batch-size-dependent arithmetic called R 1; an end-to-end comparison then contains both effects. R 1 changes preserve each row’s operations but change the rows that share a batch, so GEMM algorithms can flip bits (He, 2025). R 2 changes target the same sampling distribution with different random numbers. R 3 changes alter arithmetic and need a task-level accuracy check. Rungs 0 and 1 produce the Huginn highconcurrency and Ouro batch-1 gains. Long-prompt and session gains come mainly from rungs 2 and 3, while the PD gain is split between rung-3 packed prefill and a rung-0 transfer budget.

R0

R0 R0 R1 R1

R2

R3

Contributions. R3

• A characterization of a looped-model engine that traces Huginn’s throughput ceiling to dead KV slots, Ouro’s batch-1 latency to kernel count, and Huginn’s zero prefix-cache hit rate to the publication rule and to orderdependent recurrent-state noise (Section 3). • Plane packing, which stores only live layer slots and holds 1.88× more Huginn tokens in the same pool with bitwise-identical outputs under the same batch composition (Section 4.1). • Exact kernel fusion: fused RMSNorm, RoPE and SiLU kernels that are bitwise identical to PyTorch because the row sum keeps PyTorch’s order and rounding fences, inline PTX conversions the compiler cannot see through, pin every rounding point on the tested PyTorch, CUDA and GPU configurations (Section 4.2). • Position-keyed recurrent-state noise, which makes Huginn’s random initial state a function of (seed, position, channel), so a cached prefix sees the same noise as a cold run independent of draw order (Section 4.4). • An evaluation that reports token-level matches alongside throughput and latency, covers two GSM8K check sizes, real prompts under Poisson arrivals and a third model, and documents four negative results: approximate fusion, speculation at high load, eager-mode kernel launch cost, and a KV transfer paced by engine polling (Sections 5 and 6).

evidence

plane packing

Huginn 233/233 identical with unchanged batches; 309/312 when more requests fit (the latter also changes R 1 batch composition) exact kernel set Ouro 169/169, 208/208 and 1014/1014 identical; GSM8K 87 questions, 0 sequences differ PD fixes, transfer budget transfer layout and timing only; PD tests pass which steps replay a graph; graph budget, eviction graph tests pass adaptive drafts 125/128 identical at 32 requests, 28/28 at C≤4 batch 1: GSM8K 0 of 87 difloop fusion fer; at C≥8, Ouro 202/208 identical position-keyed noise CPU: cached = cold; GPU: full hits 108/128, partial hits 116/128; GSM8K +0.53 points (s.e. 0.51, 1319 questions) packed prefill Ouro 37/40, Huginn 36/41 identical; GSM8K: Ouro −0.99, Huginn −0.61 points approximate fusion not shipped: not bitwise, no faster (Section 5.8)

The supplementary appendix lists every GPU batch we ran, the question it asked and its outcome, including the batches that produced no data and the ones that refuted our own explanations. Most of the individual techniques have precedents: paged KV layouts, kernel fusion, counter-based random numbers and transfer budgets. What this paper adds is their measured effect on looped models, and an exactness grading that decides which of them can be shipped without an accuracy check.

2

BACKGROUND

Looped models. Ouro-1.4B has 24 transformer layers that run four times per token; a learned gate can stop a token early (Zhu et al., 2025). We run it at a fixed depth of four. Huginn-0125 embeds a token, runs two prelude layers, initializes a latent state with truncated normal noise, applies a four-layer recurrent block 32 times to the state and the prelude output, and decodes the result with two coda layers (Geiping et al., 2025). Earlier weight-shared models include Universal Transformers (Dehghani et al.,

What Sets the Serving Speed of Looped Language Models?

2019); early-exit models such as CALM (Schuster et al., 2022) and LayerSkip (Elhoushi et al., 2024) also vary depth per token. The engine. vllm-rlt extends paged KV caching (Kwon et al., 2023) and continuous batching (Yu et al., 2022) to looped models and implements runtime ideas from continuous depth batching (Schwethelm et al., 2026). A request moves through PREFILL, PRELUDE, RECURRENT and CODA phases, and the scheduler forms each batch from rows in one phase. In the LAST_EXITED layout every recurrence depth has its own KV plane, a token at depth d attends to plane d, and a row that exits early copies its last plane forward. The engine supports asynchronous scheduling, CUDA graphs for decode, Triton (Tillet et al., 2019) paged attention, a prefix cache, self-speculative decoding with a shallower draft (Zhang et al., 2024), and prefill– decode (PD) disaggregation (Zhong et al., 2024; Patel et al., 2024) that writes KV pages into the decode engine’s pool with NIXL (NVIDIA, 2025). Prefill runs one row at a time by default; a packed, variable-length prefill path exists but is opt-in.

3

P ER -L OOP OVERHEADS , N OT L OOP A RITHMETIC , S ET THE S PEED

We profiled the base engine with random-token burst workloads (all requests arrive at once, greedy decoding, fixed depth) and with multi-turn agent-session traces (Section 5.1). Loop submissions do not limit batch-1 decode. On the RTX 5090 an Ouro token at batch 1 took 16.2 ms. Fusing its loops cut the submissions per token from 6.0 to 3.0 and the per-step host time several-fold, yet the time per output token (TPOT) stayed at 16.3 ms. A kernel profile on the GH200 explains why: 5689 kernels and 13.05 ms of GPU time per token. With graphs the host no longer launches these kernels, but each still runs for a few microseconds. Each RMSNorm is eight kernels in PyTorch (Paszke et al., 2019), four norms per layer. Dead KV slots cap Huginn’s concurrency. For each Huginn token, upstream reserves 5.41 MB of KV, of which 2.79 MB is ever written. On the RTX 5090 the pool held 1328 tokens, two requests of our burst, and throughput stayed at 17.8 tokens/s from 8 to 64 concurrent requests (16.7 at 64). On a GH200 the pool is larger but the effect is the same: the base engine’s throughput stayed near 65 tokens/s from 32 requests on (Table 2). Cache hits follow KV capacity, and Huginn publishes no prefix. Ouro sessions on the GH200 reused 79.5% of prompt tokens from the prefix cache with 32 sessions but

upstream d=0 d=1 d=2 d=31

packed plane 0

. . . prelude/coda

. . . recurrent

planes 1–31, 4 slots each

dead slot

Figure 1. KV layout of one Huginn token page. Upstream reserves 8 × 32 = 256 layer slots, of which 124 are dead. Packing stores the 132 live slots in 17 blocks of eight instead of 32.

24.8% with 64, whose working set exceeds the 98,336-token pool. On the RTX 5090, whose pool holds 24,480 tokens, the hit rate fell from 45.7% to 10.7% to 2.4% for 16, 32 and 64 sessions. Huginn never hit at all: the cache publishes a page only when every layer of every plane has written it, and Huginn’s boundary layers never write planes 1–31. Prefill runs one row at a time. With 2048-token Ouro prompts, packed prefill was 2.42× and 2.00× faster than the per-row default at 8 and 32 requests, with 61% lower time to first token (TTFT) at 32. KV transfer stalls behind prefill. In a one-prefill, onedecode deployment on two GPUs of one GH200 node, the base engine at 32 requests spent a median of 2.85 s per request between the end of prefill compute and the last KV chunk leaving (the commit tail), as long as the prefill compute itself (2.87 s). The transfer ran at 1.44 GB/s.

4

E XACT L OOP : R EMOVING THE OVERHEADS , G RADED BY E XACTNESS

4.1

Packing live slots changes addresses, not values (R 0)

Packing gives each depth plane only the layer slots that it can hold (Fig. 1). A KV page becomes a run of blocks, each with num_layers slots, and slot s of block b is entry b · num_layers + s of a flattened view of the pool. Plane 0 maps all eight layers; planes 1–31 map the four recurrent layers. A Huginn page then needs 17 blocks instead of 32. Attention, KV writes, early-exit plane copies and the asynchronous finalize kernel take an entry stride and a first-layer offset instead of assuming a full plane. The PD transport sends each plane’s live slot range, and the PD coordinator prices requests in blocks per page. Values never change, only their addresses, so outputs are bitwise identical. For models without boundary layers, such as Ouro, the layout is unchanged. These checks assume equal inputs and equal batch composition. When packing admits more requests, the scheduler also changes the batches; we report those token

What Sets the Serving Speed of Looped Language Models?

comparisons separately in Section 5.4. 4.2

Rounding fences make fused kernels bitwise (R 0)

PyTorch computes a bf16 RMSNorm as a chain of eight kernels: cast to fp32, square, mean, add ϵ, rsqrt, multiply, cast to bf16, and multiply by the weight, plus an addition when a residual is fused. A one-kernel Triton version differs from it in two ways. It sums the squares in a different order, and its compiler moves rounding points. We keep both under control. Keep PyTorch’s summation order. The exact RMSNorm sums the squares in PyTorch’s order. Its first form leaves the mean to PyTorch between a Triton square kernel and a fenced Triton tail for rsqrt(x̄ + ϵ) and the scaled output (Triton’s rsqrt lowers to PyTorch’s instruction; x̄ + ϵ is one IEEE addition). PyTorch 2.14’s mean fixes its order by its launch configuration (W lanes per row set by the row count, four strided accumulators per lane, a halving tree); for Ouro’s and Huginn’s widths the second form repeats those additions in one kernel, and shapes PyTorch would split across warps keep three kernels. Fence the roundings. Writing the same casts as PyTorch is not enough. Triton and LLVM keep an fp32 product live across a cast to bf16 and back (.to(bf16).to(fp32)), and contract a · b + c into a fused multiply-add. Our first fused residual epilogue therefore computed round(r + w · n) while PyTorch computes round(r + round(w · n)), a double rounding (Higham, 2002). On 75,776 bf16 elements, 18.4% differed from PyTorch by one ulp, and 0 differed from a single-rounding oracle. A rounding fence is an inline PTX instruction, cvt.rn.bf16.f32 or mul.rn.f32, that performs one IEEE rounding the compiler cannot see through, so it can neither drop nor merge it. The exact RoPE kernel fences each of its two products before adding them. The exact SiLU-and-multiply kernel fences silu(g) before multiplying by the up projection; its exponential comes from the same CUDA math library that PyTorch links. For the residual addition we use PyTorch’s own addition. Fuse only inside graphs. On the GH200’s Grace host a Triton launch from Python costs more host time than the PyTorch launches it replaces. Under CUDA graphs that cost is paid once at capture; in eager prefill it is paid on every call. ExactLoop therefore uses the fused kernels only while a CUDA graph is being captured, which covers decode, and leaves eager code paths on PyTorch. 4.3

Loop fusion changes only which rows share a batch (R 1)

Rows decoding at a fixed depth run all their loops in one submission, which replays one graph for L loops. On the

GH200 this cut Ouro’s submissions per token from 5.9 to 3.0 and Huginn’s from 17.7 to 3.0 at 8 requests. Each row’s arithmetic is unchanged: with one request at a time, GSM8K token sequences matched on all 87 questions. The batches are not unchanged. The base engine staggers rows across loop phases, while loop fusion runs them together, so GEMMs see different batch sizes, cuBLAS may pick different algorithms, and some bf16 outputs change at high load: 202/208 requests matched the base engine at 8 to 64 requests, while two runs of the base engine matched on 208/208. That makes it R 1. A fused step is L loops long, and the scheduler admitted a fixed number of prefill batches per decode step. Prefill was therefore admitted L times less often per unit of decode work, and TTFT grew under load (Section 5.7). ExactLoop admits up to L prefill batches after an L-loop fused step, which restores the base engine’s ratio of prefill to decode work per loop. 4.4

Per-plane coverage makes prefixes reusable; position keys fix the noise order (R 2)

We changed the publication rule so that each plane must cover only the layers that it holds. Huginn prefixes are then published and reused, but a cache hit still changes outputs. Huginn draws the initial latent state of each token from a perrequest random stream, in the order tokens are processed. A request whose first k prompt tokens come from the cache draws k fewer values, so every later token receives different noise than in a cold run. The prefill chunk size changes the draws in the same way. Position-keyed noise makes the noise a function of the request seed, the token position and the channel. A hash in the style of SplitMix64 (Steele et al., 2014), applied in stages with distinct constants, turns (seed, position, channel) into a 24-bit uniform number, and an inverse normal CDF maps it onto the intended truncated normal distribution at 24-bit resolution, as in counter-based generators (Salmon et al., 2011). The target distribution is the same, subject to this discretization and the inverse-CDF implementation’s rounding. 64-bit wrap-around is identical on CPU and GPU. In CPU tests, cached and cold runs, and prefill chunks of 2 and 8 tokens, produced identical tokens. A seeded run sees different samples from upstream’s sequential generator, so the feature is opt-in together with Huginn prefix caching. On the GPU the noise is only part of the story. We ran 128 prompts of 512 tokens one at a time on Huginn-0125, cold and then again with 496 tokens served from the cache; cold runs repeated exactly (128/128). With position-keyed noise, 108/128 cached runs matched the cold run token for token (95% interval 77–90%), and a longer prompt that reused the same prefix matched in 116/128 (84–95%). An earlier eight-prompt check with the sequential generator

What Sets the Serving Speed of Looped Language Models?

gave 6/8 full hits; there, position keys fixed the one prompt that changed at its first token. The remaining divergences come from prefill arithmetic: a cached run prefills only the suffix, so its GEMMs see fewer rows, and changing only the prefill chunk size also changed outputs (220/256 cold runs matched between chunks of 2048 and 128 tokens). On the GPU, prefix reuse therefore behaves like a change in batch composition (R 1) layered on a R 2 change of samples. The noise itself is not negligible at 32 loops: the two generators gave the same tokens for only 177/256 cold runs. Position keying removes one source of cache dependence without making all GPU cache hits token-identical. 4.5

Packed prefill and its accuracy check (R 3)

The engine resolves packed prefill at construction: it turns it on for CUDA, the LAST_EXITED layout, refill scheduling and a backend with a packed path, and an explicit setting always wins. Packed bf16 prefill computes attention with different arithmetic than per-row prefill, so a few greedy sequences change (Table 1). For Huginn, whose boundary layers need their own attention metadata, packed prefill changed the full GSM8K split by −0.61 points (Section 5.8), within the one-point rule only on its point estimate. The packed path stages its inputs in host-mapped pinned memory (UVA) instead of copying them to the GPU. 4.6

The PD fixes change no values (R 0)

The upstream PD coordinator built every model with Ouro’s configuration class, so a real Huginn checkpoint could not start; it now picks the class from the model type the worker reports. With packing, the transport sends only live slots, 0.516 of the full Huginn layout. Descriptors that are contiguous on both sides are coalesced, and an overlap counter now credits transfers issued during a prefill computation. The connector can also create its UCX (Shamis et al., 2015) backend explicitly with an error-handling mode, and the default transfer budget now holds one long prompt’s KV in flight (Section 6).

5

E VALUATION

5.1

Setup

Platforms. We used three machines. A batch-scheduled GH200 system ran single-GPU jobs and the two-GPU PD jobs, with PyTorch 2.14, Triton 3.8 and NIXL 1.4.1. A standalone GH200 host, whose GPU our runs held exclusively, ran the release table, the accuracy gates and the end-to-end and Huginn session comparisons. A rented RTX 5090 ran the first characterization. Every speedup compares arms run on the same machine.

Workloads. In a burst workload, C requests arrive at once, each with a prompt of 512 or 2048 random tokens and a fixed number of greedy output tokens. Session workloads replay multi-turn traces of a coding agent (SemiAnalysis, 2026) whose prompts grow with the conversation, with n concurrent sessions and the prefix cache on. Ouro runs at depth four with asynchronous scheduling and CUDA graphs unless noted; Huginn runs at depth 32 with CUDA graphs and synchronous scheduling, which the engine requires for it. Configurations. Table 2 measures the released defaults (loop fusion with the prefill share, plane packing, exact kernels in graphs and in eager calls over 256 rows, automatic packed prefill) against the base engine, three runs per cell. ExactLoop evolved during the study, so the component tables that follow state which features their arms had; unless they say otherwise, their ExactLoop arm prefilled one row at a time and had no prefill share. Noise. Apart from Table 2 and the arms that say so, cells are single runs. On the standalone host two runs of the base engine at 32 requests differed by a factor of 0.84, and the first repeat of an arm ran 17% below its later repeats (up to 33% for session arms). Where an arm has repeats we drop the first; we treat differences within the rerun spread as ties. Several gains reported below, such as the graphcount and packing ratios, lie within or close to this band and are presented as directional. Token identity compares each request with the request of the same level, repeat and index. Run-to-run variation. The headline cells retain two runs after warm-up, while several component experiments have only one. Host and scheduling variation can therefore obscure small differences. We discuss component effects only when they exceed the observed rerun spread and leave the remaining measurements as directional. Section 5.8 treats the accuracy subset separately because its sample size differs from the performance runs. 5.2

The released defaults are 1.6–3.2× faster than the base engine

Table 2 is the headline result. Figure 2 plots it. With 512token prompts the released defaults serve Ouro 1.65–2.15× and Huginn 1.77–3.16× faster than the base engine, and both time to first token and time per output token fall at every load. With 2048-token prompts, where packed prefill dominates, Ouro runs 1.91–2.04× faster. Huginn’s median TTFT at 64 requests falls from 22.7 to 4.0 s. Packed prefill and batch composition change some greedy sequences: 285/315, 109/120 and 271/315 requests are identical to the base engine. On the batch-scheduled GH200, with Ouro and Huginn on

What Sets the Serving Speed of Looped Language Models? (b) High-load latency 3.16× 3 Huginn Ouro 2K

2 Ouro 512

1.91×

(c) Sessions base

% of base latency

throughput vs. base

(a) Burst throughput 100

TTFT

75

TPOT

62 56

5054

50

2.72

Hug+prefix 2.34

Huginn 45 Ouro 64 17

25

3.21

Ouro 32

1.91

base 1

1

8

32

64

0 Ouro

concurrent requests

512

Ouro

2K

1×

inn

Hug

2×

3×

throughput vs. base

largest burst load

Figure 2. Main serving results for the released defaults. (a) Throughput speedup over the base engine as concurrency increases; the dashed line marks no speedup. (b) TTFT and TPOT at the largest tested burst load, as a percentage of base latency; lower is better. (c) Throughput speedup on multi-turn agent sessions, including Huginn with and without prefix caching. Values are derived from Tables 2 and 3 (standalone GH200) and, for Ouro sessions, Table A5 (batch-scheduled GH200); markers and bars use the mean of the retained runs.

(b) Huginn-0125

(a) Ouro-1.4B latency per token (s)

Table 2. Released defaults against the base engine, standalone GH200, burst workloads with 512- or 2048-token prompts (Ouro: 128 or 64 output tokens, async scheduling and graphs; Huginn: 64 output tokens, 32 loops, sync and graphs), 64 graphs, 0 eager fallbacks. Output tokens/s; three runs per cell, first dropped, mean of 2. All rows come from one job and one build, the paper tree; each ratio is paired within the job. 109.2 657.4 1454.2 1483.4 208.8 246.9 19.9 109.3 206.6 205.3

1.95× 1.93× 2.15× 1.65× 2.04× 1.91× 1.77× 2.26× 2.84× 3.16×

0.19/0.09 0.29/0.14 1.05/0.51 2.30/1.28 1.66/0.81 6.60/3.30 0.30/0.25 1.23/0.43 4.72/2.02 22.66/3.95

16.6/8.5 19.5/10.2 37.1/17.2 50.8/31.7 46.2/22.9 137.9/75.1 85.2/46.9 129.7/58.5 227.9/118.2 270.1/121.0

different builds, the same workloads gave the same token identity but larger Ouro gains (2.0–2.3× and 3.5–3.8×) and Huginn gains of 1.5–2.7×: the speedups depend on the platform. The base engine’s batch-1 Huginn throughput also moved by 17% between two batches, so the batch-1 Huginn ratio is the least certain of these headline numbers. On the batch-scheduled host, in four interleaved rounds per arm, the released defaults served Ouro agent sessions 1.91× and 3.21× faster than the base engine at 32 and 64 sessions, with a median TTFT of 0.15 instead of 1.98 s at 64, and Huginn sessions 2.72× faster with the prefix cache (Table 3); Table A5 takes the Ouro gain apart. 5.3

Real prompts under Poisson arrivals

We replayed ShareGPT and Alpaca prompts with each reference answer’s length, offline and under Poisson arrivals, on the standalone host and the same two trees (Fig. 3). Offline, with 64 requests at once, they were 1.71× (ShareGPT) and 1.29× (Alpaca) faster for Ouro and 2.01× and 1.60× for Huginn; Alpaca’s answers average 58 tokens, so admission

0.2 0.1 0.05 0.02 5

10

15

request rate (req/s) base

model, prompt C base ExactLoop speedup TTFT p50 b/E (s) TPOT p50 b/E (ms) Ouro, 512 1 55.9 Ouro, 512 8 339.9 Ouro, 512 32 677.9 Ouro, 512 64 901.2 Ouro, 2048 8 102.4 Ouro, 2048 32 129.3 Huginn, 512 1 11.3 Huginn, 512 8 48.3 Huginn, 512 32 72.7 Huginn, 512 64 65.0

0.5

0.5

1

1.5

request rate (req/s) ExactLoop

Figure 3. ShareGPT prompts with their reference answer lengths (prompt ≤1024, answer ≤256 tokens), Poisson arrivals, 256 (Ouro) or 80 (Huginn) requests per rate, standalone GH200, last of two runs; median end-to-end latency per output token (log scale; lower is better).

weighs more. Online, Huginn’s latency per token was 2.9– 4.2× lower at every rate, and its ExactLoop arm had not saturated at 1.5 requests/s. Ouro’s base engine saturated first: its latency per token rose 4.3× from 2 to 4 requests/s, while ExactLoop stayed at 0.041 s at 4 (base: 0.054 s at 2) and was 2.3× lower at 6. From 8 requests/s both were saturated at the same throughput (343–386 against 308–390 tokens/s), so that ceiling lies in what ExactLoop leaves unchanged; we did not trace the step mix. Median TTFT was higher with ExactLoop at 1 of 6 Ouro and 2 of 5 Huginn rates. Ouro-2.6B (48 layers, same shapes) ran 2.0–2.3× and 2.2–3.3× faster on the release workloads (272/315 and 67/120 identical) and 1.68× on ShareGPT. 2 of 10 Poisson runs of Ouro’s released defaults aborted, in a cuBLAS GEMM writing to an unmapped address; the cause is open. 5.4

Packing lifts Huginn’s ceiling; loop fusion carries batch 1

Table A1 isolates packing on an earlier tree. The same pool holds 12080 tokens unpacked and 22736 packed. At 8 requests both layouts fit and run alike (1.01×); at 32 and

What Sets the Serving Speed of Looped Language Models? Table 3. Huginn-0125 agent sessions, 16 sessions, at most four turns and 2000-token prompts, 59 requests, standalone GH200, paper tree, 64 graphs. ExactLoop: released defaults with positionkeyed noise; “+ prefix cache” adds Huginn prefix caching. Four interleaved rounds per arm, the first dropped, mean of three.

arm

tok/s TTFT p50 p90 (s) TPOT (s) hit %

base 12.2 ExactLoop 28.6 + prefix cache 33.3

5.52 0.48 0.36

11.94 0.96 0.69

0.64 0.37 0.27

– 0 78

64 requests packing raised throughput 1.21× and 1.28×, and at 64 requests it cut the median TTFT from 10.8 to 5.5 s. 309/312 requests matched: packing changes no value, but when more requests fit, the batches differ, as with loop fusion. In an earlier tree with per-row prefill, where the batches did not change, all 233/233 requests matched. The first GH200 batch measured smaller gains of 1.14× and 1.16× at 32 and 64 requests (ledger item 10). We report the repeated earlier-tree measurement above, and the lower values lie within the rerun spread. At one request capacity does not bind, so the batch-1 gain of Table 2 comes from the other changes. In one batch on that tree, the fused kernels contributed only 1.00×, 1.07× and 1.03× at 1, 8 and 32 requests (full set over kernels disabled, three runs each, first dropped): Huginn’s layers are large and few, so kernel count matters little, and at one request the remaining changes are loop fusion and the host path. Against a base-engine arm from a later batch the full gain was 1.59×, 1.92× and 2.57× on the standalone host (1.77× at one request in Table 2); the base engine’s batch-1 Huginn speed moved by 17% between two batches. Its 32 loops make submissions matter more than for Ouro: at 8 requests the base engine needed 17.7 submissions per output token and ExactLoop 3.0. Packed prefill helps Huginn as much as Ouro: at 32 requests it was 1.33× faster with 512-token prompts and 2.08× with 2048-token prompts, where it cut the median TTFT from 41.9 to 18.1 s (single runs). 5.5

Prefix caching makes Huginn sessions 1.16× faster on top of ExactLoop

Prefix caching with position-keyed noise reused 78% of prompt tokens and raised Huginn session throughput 1.16× over ExactLoop without the cache, which was itself 2.34× the base engine (Table 3); together 2.72×, with the median TTFT cut to 1/15. The base engine ran alike with 16 and 64 graphs (14.0 tokens/s), and no Huginn session run fell back to eager execution. On an earlier tree the prefill share made no measurable difference here. On the batch-scheduled host an earlier tree gave 3.12× with the prefix cache.

5.6

Exact fusion matches approximate throughput

With the three-kernel RMSNorm the exact set cut kernels per token from 5689 to 2885 at batch 1, a 49% reduction, and GPU time per token from 13.05 to 8.91 ms. With the one-kernel RMSNorm it raised throughput 1.55×, 1.59×, 1.51× and 1.11× at 1, 8, 32 and 128 requests, against 0.97–1.30× with the three-kernel norm (Table A2), and 1014/1014 requests matched the unfused arm. The approximate set, whose RMSNorm sums in its own order and which has no fences, was no faster within the host’s spread (1.23–1.67×) and left only 894/1014 requests unchanged. ncu on the three-kernel form shows why kernel count, not kernel work, sets the batch-1 time: its four exact kernels ran 2.4–3.8 µs each and reached at most 0.2% of DRAM throughput. The exact set therefore costs no measurable throughput in this experiment. We discuss its accuracy result in Section 5.8. Table A3 shows why fusion was first limited to graphs: fused kernels in every scope made eager prefill slower than the base engine’s at all three levels, and graph-only fusion made it faster. Its throughput columns are single runs within the rerun spread, so we use only its prefill times. Packed prefill makes eager calls large, though: the exact kernels in eager calls over 256 token rows cut Ouro’s prefill GPU time by 29% and Huginn’s prefill time by 14% at 64 requests and raised throughput up to 1.06× (Ouro, 2048-token prompts) and 1.05× (Huginn). Fusing Huginn’s own activation and rotary embedding added 1.07–1.11×, and the one-kernel RMSNorm, 2.0–4.4× faster per norm in CUDA graphs and bitwise in all 156 GPU test cases, added 1.13–1.17× for Ouro and 1.06–1.08× for Huginn. Running Ouro’s q/k/v and gate/up projections as one GEMM each over concatenated weights (seven GEMMs per layer and loop to four) is bitwise on this cuBLAS for each of rows 1–8192, eagerly and in graph replay (not on an L40S, so only checked GPUs fuse); it cut decode GPU time at one request by 12% and added 1.06–1.12× (512-token prompts) and 1.07–1.08× (2048). No request changed in these arms; Table 2 includes them. 5.7

The prefill share trades time per token for time to first token

Table A5 shows a trade-off. With 64 sessions, ExactLoop without the prefill share and with per-row prefill cut the median TPOT from 0.99 to 0.43 s but raised the median TTFT from 1.95 to 3.44 s, because fused decode steps delayed prefill admission (Section 4.3); we compare against the 64-graph base engine, although its median TTFT was higher than with 16 graphs (1.48 s), which we cannot explain. Packed prefill raised throughput to 2.79× the base engine and brought the median TTFT down to 0.94 s. With 2048-token bursts, packed prefill was 2.42× and 2.00×

What Sets the Serving Speed of Looped Language Models?

faster; with 512-token bursts it was within noise of per-row prefill (0.98× and 1.19× at 8 and 32 requests, against a rerun of 0.98× and 1.10×), and without UVA staging it fell to 0.88× and 0.93× (2 kept runs each). The prefill share helps the burst case, and was ahead in both kept runs. At 32 requests with 512-token prompts (2 kept runs), it raised throughput from 877 to 1001 tokens/s, cut the median TTFT from 1.11 to 0.82 s and the p90 from 2.13 to 1.63 s, and lowered TPOT from 25.9 to 22.8 ms. In 64-session runs it is a trade-off: on top of packed prefill, the share cut the median TTFT from 0.94 to 0.25 s and the p90 from 4.25 to 1.15 s, but raised the median TPOT from 0.14 to 0.32 s and kept 0.88× of the throughput; with 32 sessions it changed little (0.96×). We made the share the default because TTFT is the metric agents wait on, but the choice depends on the service objective. 5.8

Accuracy checks and the approximate-kernel decision

We used the engine’s GSM8K (Cobbe et al., 2021) harness on a fixed 87-question subset, greedy, one request at a time, with a regression limit of one point. On the GH200, the exact kernel set answered 62 questions correctly, as did the unfused engine, and 0 token sequences differed. On the RTX 5090, loop fusion matched the base engine on all sequences and exit depths (64 correct). The approximate RMSNorm alone scored 61 against 64 (−3.45 points, paired standard error 2.56; 29 sequences differed) and the full approximate set 62 (−2.30, standard error 1.62; 23 differed). Both trip our one-point rule. We used that rule to decide whether to ship the kernels, not as a hypothesis test; the 95% intervals, roughly [−5.5, +0.9] and [−8.5, +1.6], include zero. With 87 questions, the subset cannot separate changes of one and two points, so we repeated both decisions on all 1319 test questions on a GH200, batched (32 requests, CUDA graphs, same stops and scorer). The exact and unfused paths then matched on every sequence. The approximate set changed the score by −0.15 points (s.e. 0.55) and the approximate RMSNorm by +0.76 (0.60), within the rule; GPU, batching and reference differ from the subset. The approximate kernels stay unshipped: they are neither bitwise nor faster. Packed prefill changed Ouro by −0.99 points (s.e. 0.56) and Huginn by −0.61 (0.48): within the rule on the point estimate, with 95% intervals reaching −2.1 and −1.5. Batch composition alone moves scores as much: with 16 requests per batch, per-row Ouro answered 816 instead of 824 and packed prefill changed it by −0.08. Batches of 64 replayed the 32-request outputs almost token for token, and Huginn’s batch 16 replayed them exactly (1319 of 1319), so Huginn has one observation and a one-point loss is not excluded. On MATH-500 (496 questions) packed prefill changed Ouro

by +0.40 (1.21) and Huginn by −0.20 points (0.61), and the exact set matched unfused on all 496. Huginn’s R 2 changes were gated with all 87 questions run as one batch on the released defaults. Sequential noise answered 23 correctly, position-keyed noise 22 (−1.15 points, paired standard error 2.00), and position-keyed noise with prefix reuse 21 (−2.30 against sequential noise, standard error 1.62). Both miss the same one-point rule, although their differences are within about one and a half standard errors. The choice of noise generator alone changed 38 of 87 sequences. Since 87 questions cannot separate oneand two-point effects, we repeated the comparison on the full test split of 1319 questions. There sequential noise answered 300 correctly (22.7%), position-keyed noise 307 (+0.53 points, standard error 0.51) and position-keyed noise with prefix reuse 306 (+0.45 against sequential noise, standard error 0.50; −0.08 against position-keyed noise without reuse, which served 293k prompt tokens from the cache). Both full-split variants remain within one point of sequential noise. This result is more convincing than the subset, though it still comes from a fixed rule. As expected after a change of random numbers, most sequences differ: only 572 of 1319 were identical between the two noise generators, and 636 between position-keyed noise with and without reuse. 5.9

Speculation pays at low load; an adaptive budget avoids its high-load cost

The speculative path needs fixed-depth exits and is not available for Huginn. With enough graphs that no step ran eagerly (128, 0 fallbacks), self-speculation with four draft tokens served Ouro 1.52×, 1.37× and 1.13× the throughput of plain decode at 1, 2 and 4 requests, 1.13× at 8, 1.05× at 16 and 0.75× at 32: once the target model saturates the GPU, the draft’s extra loops are cost. Its verification rows also multiply batch shapes; this arm captured 85 graphs, and with 24 it ran most steps at 16 and 32 requests eagerly, which earlier made it look several times slower than here. On the standalone host ExactLoop’s speculative path was 1.52× and 1.31× the base engine’s at 1 and 8. The speculative path also runs a whole token, prelude, loops and coda, in one engine step (151 steps instead of 408 for a 32-request burst). Without drafts (K=0) that alone gave 1.07–1.08× up to 8 requests and 1.01–1.04× at 16 and 32, with 126/128 sequences identical at 32 (R 1). A load-adaptive draft budget combines both: with at most 8 verification rows per round, each of N ready requests drafts min(4, ⌊8/N ⌋ − 1) tokens. This gave 1.52×, 1.37× and 1.26× at 1, 2 and 4 requests, against 1.13× for fixed K=4 at 4, and 1.01× instead of 0.75× at 32 (three runs per cell, first dropped).

What Sets the Serving Speed of Looped Language Models?

5.10

Decode is already zero-copy; copy kernels and noise syncs remain

We counted every host–device copy, blocking synchronization and copy kernel in a window of decode steps (Table A4). The engine’s host side is already close to zero-copy. With asynchronous scheduling and graphs, decode issues no hostto-device memcpy at all, because inputs are staged in hostmapped memory and read by the kernels, and one pinned 8-byte device-to-host copy per token, the sampled token (one batched copy per step at 32 requests). In packed prefill, UVA staging replaces 3.7 host-to-device memcpys per thousand prompt tokens (27 KB) with zero; without UVA staging, packed prefill ran at 0.88–0.93× of per-row prefill (Section 5.7). What remains is movement on the device and blocking reads on the host. At batch 1 the base engine launches 1004 copy and cast kernels per token, from RMSNorm’s casts and RoPE’s concatenations; the exact kernel set removes all but 29 of them. Huginn’s sampler is the other case. PyTorch 2.14’s trunc_normal_ rejection-samples and checks mask.any() after each round, a blocking read, and the engine calls it once per request for every token’s initial state. In a Huginn decode window ExactLoop issued 6 such blocking reads per step, 2.4 per output token, besides 12 host-side scalar conversions. Position-keyed noise computes the same distribution with an inverse CDF in one kernel (Section 4.4); in the same window it issued 0 blocking reads and 0 scalar conversions, and device-to-host copies fell from 6.3 to 0.3 per step. In two paired runs of three repeats each, it gave Huginn 1.04–1.09× the throughput of sequential noise at 32 requests, 1.01–1.04× at 8 and 0.92– 1.00× at one. On the PD path, KV moves from the prefill pool into the decode pool’s final pages with no staging copy, and with packing only live slots move (Section 4.6). 5.11

Packed prefill and a larger in-flight budget raise PD throughput up to 3.3×

Table 4 compares the base engine with ExactLoop on an earlier tree. With 2 GB in flight and UCX error handling off, so that KV moves by CUDA IPC, ExactLoop served 479 tokens/s at 32 requests in all three runs against 143– 147 (3.3×). This arm disables UCX peer error handling, which the base arm does not, so the ratio is not a transportmatched comparison; the error-handling arm below is the conservative one. The median TTFT fell from 11.0 to 2.0 s and the commit tail from 2.8 s to 16 ms. With NIXL’s default error handling, which keeps peer failure detection and runs over TCP, two runs reached 450–457 tokens/s (3.1×); the third reached 205 with TTFT and tail unchanged and a median TPOT of 116 instead of 21 ms, so it lost time in decode, not in the transfer. Averaged over all three runs, that arm is about 2.6× the base; Section 6 traces

Table 4. Ouro-1.4B PD on one GH200 node, base engine against the final ExactLoop tree, 64 output tokens. ExactLoop columns give the in-flight budget (64 MB chunks for 256 MB, 256 MB chunks for 2 GB) and the UCX transport for KV: TCP is NIXL’s default error handling, IPC is CUDA IPC with error handling off. 1P1D: one prefill and one decode GPU, 2048-token prompts, three runs per arm in alternating order; 8192-token prompts and 2P2D: one or two runs. Ranges over runs; times are medians. ExactLoop base 256 MB, IPC 2 GB, TCP 2 GB, IPC 1P1D, 2048-token prompts C=8, tok/s 107–112 C=32, tok/s 143–147 TTFT (s) 10.8–11.1 TPOT (ms) 30–40 tail (s) 2.80–2.85 1P1D, 8192-token prompts C=8, tok/s 14 TTFT (s) 33.7 2P2D, 2048-token prompts C=32, tok/s 236 C=64, tok/s 262

223–228 271–308 3.4–3.8 29–57 0.82–1.38

228–289 205–457 2.0–2.1 21–116 0.02

283–289 479 2.0 18 0.02

71–80 4.5–5.4

89 3.9

79–89 3.8–4.5

446–450 450–487

610 617

610–617 460–751

this signature to the CUDA graph cache. With 8192-token prompts, 6.4 GB of KV per request, ExactLoop reached 79– 89 tokens/s against 13.8 and cut the median TTFT from 33.7 to 3.8–4.5 s; an 8 GB budget, one round per request, gave 86, so at this length the budget hardly matters. With two prefill and two decode GPUs ExactLoop served 610–617 tokens/s at 32 requests against 236 (446–450 with the old budget); at 64 requests its two runs gave 460 and 751, a decode-side spread like the one above. The PD and transfer-connector GPU tests passed on both trees (52 on the base engine, 56 on the PD branch). Two changes produce the gain: packed prefill and the inflight budget. In an earlier single run with the old budget, per-row prefill reached 157.5 tokens/s at 32 requests and packed prefill 294.4, against 129.0 for the base engine; prefill compute fell from 2.69 to 0.64 s. All arms moved the same bytes, 786 KB per prompt token, yet a 64 MB chunk took 44.5 ms behind per-row prefill and 14.5 ms behind packed prefill: the transfer speed followed the prefill step length, which pointed at the host rather than the link. A single-run sweep of the budget (64 MB chunks and 256 MB in flight against 256 MB and 1 GB) separates the two explanations. With per-row prefill, chunks four times larger took four times longer and bandwidth stayed between 1.26 and 1.40 GB/s, so the in-flight budget was not the limit. With packed prefill, the larger budget cut the median TTFT by 48%. UCX’s own log showed that intra-node remote memory access ran as active messages over TCP on the node’s host network interface, and TCP moves data only when the host polls it. That explains why

What Sets the Serving Speed of Looped Language Models?

chunk time grew with chunk size behind per-row prefill, but not the whole commit tail: removing TCP left chunk times almost unchanged. With three runs each on that tree, raising the budget from 256 MB to 2 GB raised throughput 1.7× (271–308 to 479 tokens/s) and cut the tail from 1.0 s to 16 ms; Section 6 explains why.

6

L ESSONS AND N EGATIVE R ESULTS

Approximate fusion: too few questions. The approximate kernels rounded once where PyTorch rounds twice and summed in a different order, changing some elements by an ulp or two. They changed more GSM8K sequences than the exact set and crossed our one-point rule on 87 questions, but not on the full split. Neither sample shows that accuracy fell, or that it did not fall by a point. For looped models, a per-element ulp bound is not enough evidence of harmlessness; the check needs enough questions to resolve the effect of interest. Fused kernels cost host time in eager mode. With fused kernels in every scope, ExactLoop ran at 0.40× the base engine’s throughput in synchronous eager mode at batch 1 (TPOT 118 to 296 ms, single run), and its prefill at 32 requests took 3.08 instead of 2.32 s. A Triton launch from Python costs more than the PyTorch kernels it replaces on this host. Graph-only fusion removed the regression in prefill (Table A3). A full graph cache turns decode eager without warning. The engine captures one recurrent graph per batch size, 16 by default, and runs every later size eagerly. Staggered PD imports and sessions pass through more sizes: with 16 graphs, 3 of 23 PD runs decoded five times slower at 32 requests, and session runs fell back to eager execution hundreds of times. ExactLoop keeps 64 graphs and evicts the least recently used one once full; with 64 graphs no PD or session run fell back. Both changes decide which steps replay a graph, not what they compute (R 0). The KV path chose TCP, and TCP was not the bottleneck. NIXL’s UCX plugin requests peer failure handling, which shared-memory and CUDA IPC transports lack, so UCX fell back to TCP. Removing that requirement moved the KV to CUDA IPC but barely shortened a chunk; the commit tail was set by budget rounds times the engine step, which the larger in-flight budget removes (details in the appendix).

7

R ELATED W ORK

LLM serving. Paged KV caching (Kwon et al., 2023), iteration-level scheduling (Yu et al., 2022), prefix sharing (Zheng et al., 2024), and chunked prefill (Agrawal

et al., 2024) underpin most serving engines. PD disaggregation (Zhong et al., 2024; Patel et al., 2024) and KV-centric designs (Qin et al., 2025) move KV between machines; they assume one KV entry per layer, which looped models multiply by depth. Variable depth and numerics. Weight sharing across depth goes back to Universal Transformers (Dehghani et al., 2019); Huginn (Geiping et al., 2025) and Ouro (Zhu et al., 2025) use it, while CALM (Schuster et al., 2022) and LayerSkip (Elhoushi et al., 2024) also vary compute per token. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) and self-speculation (Zhang et al., 2024) help only while the target model leaves the GPU idle. Batchsize-dependent kernels make outputs depend on server load (He, 2025); double rounding and contraction are classical (Higham, 2002). Looped-model inference. Continuous depth batching (Schwethelm et al., 2026) schedules single loop iterations so that rows at different depths share a batch; the base engine we measure implements its runtime ideas, and ExactLoop builds on it rather than replacing it. WaveFront (Ha & Kim, 2026), LoopSpec (Cho et al., 2026) and LoopAhead (Liu et al., 2026) draft tokens from shallow recurrent states and verify them at full depth; FlashLoop (Yang & Liu, 2026) skips cross-loop updates and quantizes KV residuals, and ResidualQuant (Kim et al., 2026) stores non-final loops’ KV as 2-bit residuals. They report speedups over autoregressive decoding or full-precision KV, not over a tuned engine, and change what is computed: speculation preserves the output distribution by verification, while sparsity and quantization approximate it. ExactLoop’s R 0 and R 1 changes alter at most which rows share a batch, need no accuracy check, and are orthogonal to these methods; we have not measured them in combination. Looped models in RL. A concurrent submission (Anonymous, 2026) checks learner log-probabilities against deferred-loop rollouts on the same engine; we study serving.

8

C ONCLUSION

Looped models repeat per-layer costs at every depth. Huginn wastes KV on dead depth layers; Ouro launches many small kernels. Live-slot packing and fenced fusion remove both without changing same-batch arithmetic. Packed prefill, position-keyed noise, graph management, and a pollingaware transfer budget supply the remaining gains. Under Poisson arrivals of real prompts, ExactLoop moves Ouro’s saturation to a higher rate but not its saturated throughput, and Huginn’s ExactLoop arm had not saturated.

What Sets the Serving Speed of Looped Language Models?

R EFERENCES Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. Anonymous. Exact log-probabilities and fast deferred rollouts for reinforcement learning on looped transformers. Under submission, 2026. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. Cho, S., Cui, L., Kim, S., Han, D., and Han, I. LoopSpec: Pipelined self-speculative decoding for looped transformers. arXiv:2609.17184, 2026. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. In International Conference on Learning Representations, 2019. Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., Aly, A., Chen, B., and Wu, C.-J. LayerSkip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., and Goldstein, T. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025. Ha, H. and Kim, J.-J. WaveFront decoding: Parallelized self-speculative decoding for looped language models. arXiv:2609.23033, 2026. He, H. Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism (blog), 2025. Higham, N. J. Accuracy and Stability of Numerical Algorithms. SIAM, 2nd edition, 2002. InferMatrixLab. vllm-rlt: a serving engine for looped language models. https://github.com/ InferMatrixLab/vllm-rlt, main branch at commit 66920da, 2026.

Kim, H., Lee, J., Cho, S., Han, D., Han, I., and Kim, S. ResidualQuant: KV cache quantization for looped transformers with 2-bit residuals. arXiv:2610.10381, 2026. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, 2023. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 2023. Liu, Z., Xue, N., Bao, R., Xu, Y., Cao, X., Yao, J., and Chen, Z. LoopAhead: Speculating ahead for faster decoding in looped language models. arXiv:2610.21594, 2026. NVIDIA. NIXL: NVIDIA inference xfer library. https: //github.com/ai-dynamo/nixl, 2025. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019. Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st Annual International Symposium on Computer Architecture, 2024. Qin, R., Li, Z., He, W., Cui, J., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies, 2025. Salmon, J. K., Moraes, M. A., Dror, R. O., and Shaw, D. E. Parallel random numbers: As easy as 1, 2, 3. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2011. Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V. Q., Tay, Y., and Metzler, D. Confident adaptive language modeling. In Advances in Neural Information Processing Systems, 2022. Schwethelm, K., Rueckert, D., and Kaissis, G. Depthadaptive inference of looped language models via continuous depth batching. arXiv:2608.09444, 2026. SemiAnalysis. cc-traces-weka-062126: WekaTrace corpus derived from claude code proxy traces. Hugging Face

What Sets the Serving Speed of Looped Language Models?

dataset, https://huggingface.co/datasets/ semianalysisai/cc-traces-weka-062126, 2026. Shamis, P., Venkata, M. G., Lopez, M. G., Baker, M. B., Hernandez, O., Itigin, Y., Dubman, M., Shainer, G., Graham, R. L., Liss, L., Shahar, Y., Potluri, S., Rossetti, D., Becker, D., Poole, D., Lamb, C., Kumar, S., Stunkel, C., Bosilca, G., and Bouteiller, A. UCX: An open source framework for HPC network APIs and beyond. In IEEE 23rd Annual Symposium on High-Performance Interconnects, 2015. Steele, G. L., Lea, D., and Flood, C. H. Fast splittable pseudorandom number generators. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, 2014. Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019. Yang, W. and Liu, S. FlashLoop: Fast and memory-efficient looped transformers via lazy updates. arXiv:2609.29812, 2026. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for transformerbased generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation, 2022. Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. Zhu, R.-J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., Wei, B., Wen, Z., Yin, F., Xing, H., Li, L., Shi, J., Ma, K., Li, S., Kergan, T., Smith, A., Qu, X., Hui, M., Wu, B., Min, Q., Huang, H., Zhou, X., Ye, W., Liu, J., Yang, J., Shi, Y., Lin, C., Zhao, E., Cai, T., Zhang, G., Huang, W., Bengio, Y., and Eshraghian, J. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025.

What Sets the Serving Speed of Looped Language Models?

A

M EASUREMENT P ROVENANCE AND A RTIFACT C ONTENTS

The measurements below come from several frozen revisions and jobs rather than one final tree. This appendix records that provenance and the number of runs behind each group; Table A6 lists the GPU batches in execution order. “Retained runs” counts runs after the warm-up policy described in the evaluation. • Released defaults. Ouro uses the paper tree. Huginn runs on an earlier tree and is checked against the paper tree on 24 requests. Two of three runs per cell are retained. Ratios are paired within each job, but the two models do not share one build. • Plane packing. The earlier-tree packed/unpacked comparison covers 8, 32 and 64 requests, with two of three runs per cell retained. Same-batch identity and the changed-capacity result are reported separately. • Exact kernels. Interleaved rounds compare unfused, exact, and approximate sets, retaining four runs per cell. Exactness is tested on the reported shapes and GPU software stack. The approximate arm is an internal accuracy check, not a full accuracy study. • Agent sessions. Ouro and Huginn multi-turn traces retain three runs per arm after the first round, on a tree measured before the last kernel changes. • Prefix reuse identity. The paired cold/cached check covers 128 Huginn prompts with position-keyed noise (full hits 108/128, partial 116/128) and only eight with the sequential generator. • Accuracy. The first Ouro checks use an 87-question subset; the noise, cache, packed-prefill and repeated kernel checks use the 1319-question full split. Each arm is one deterministic greedy run; packed prefill has a second, independent batch size for Ouro only. Packed prefill and the exact set were also checked on MATH500 (496 questions, on the standalone host, one run per arm; intervals reach −2.0 and −1.4 points); even the GSM8K full split cannot exclude a one-point loss for packed prefill. • PD. The 1P1D comparison uses three runs per arm; 8192-token and 2P2D checks use one or two runs. All tests are on a single node, and the fastest IPC arm disables UCX peer error handling. Artifact contents. The repository includes the LaTeX sources, generated tables, plotted figure data, the modified vllm-rlt snapshot, focused unit and GPU tests, and GSM8K tooling. It omits model weights, datasets, raw profiler traces, benchmark logs, and the historical run harness. The snapshot also has later architecture guards, so it is not the source revision for every result. Users can inspect the packed-KV

mapping, prefix coverage, loop scheduler, PD configuration, and exact kernels, and can run selected tests where the required devices and software are available. They cannot rerun the complete performance campaign from this release.

B

A DDITIONAL L IMITATIONS

The real-prompt replay ran on the standalone host only, with one round per arm and the last of two runs per rate; it shares the release table’s platform but not its workloads. Prompt and answer lengths were capped at 1024 and 256 tokens, which keeps 911 of 2000 ShareGPT requests. 2 of 10 Poisson runs of Ouro’s unmodified released defaults aborted with an illegal memory access, both at 4 requests/s and neither in the same place of an exact replay. A GPU core dump of the second abort shows a Warp MMU fault in a cuBLAS split-K GEMM (nvjet, sm_90) right after a tensor-memory-accelerator store, that is, a GEMM output or workspace address that was no longer mapped; ExactLoop’s Triton kernels were not running. One run each with packed prefill, fused kernels or asynchronous scheduling disabled did not abort, which at this rate is not evidence. We suspect a tensor freed on one stream while another still used it, but the cause remains open. Our pre-registered online criterion, the highest rate whose median latency per token stays within twice the engine’s own lowest-rate value, rated ExactLoop and the base engine equal for both models, so the predicted 1.5-fold gain in sustainable rate was not met; the criterion takes ExactLoop’s lower floor as its reference, and an absolute threshold would have been the better choice. Ouro-2.6B was measured for throughput and token identity only, not accuracy. The released-defaults comparison, packing and kernel component tables, and final PD comparison retain two of three runs per cell; earlier characterization and component tables often have one. At 32 requests, variation on the standalone host overlapped several component differences, which we report as ties. Huginn’s prefill share was measured only at 32 and 64 requests and in one session run. Ouro sessions on the paper tree come from the batch-scheduled host; the release table and the Huginn sessions come from the standalone host, where the Ouro release gains are smaller (ledger rows 48 and 52). Huginn’s GPU prefix-reuse check used 128 prompts, eight for the sequential generator. The first Ouro accuracy checks use 87 questions; the noise, cache, packed-prefill and repeated kernel checks use the full 1319-question split. For packed prefill the 95% intervals still reach −1.5 (Huginn) and −2.1 points (Ouro). Bitwise kernel results apply to the tested shapes, PyTorch/CUDA versions, and GPUs. A capacity change can still alter GEMM batch shapes and therefore the generated tokens.

What Sets the Serving Speed of Looped Language Models?

PD experiments used one node because the engine starts workers locally; a cross-node path would have used TCP on our systems. The fastest IPC result disables UCX peer error handling, leaving failures to the control channel. Session traces contain long, repetitive prompts and may overstate cache hits elsewhere. We evaluate Ouro at depth four and Huginn at depth 32 with fixed decoding depth. We have not measured adaptive exits, other recurrence depths, larger models, or other GPU families. The evaluation has no non-looped baseline of matched quality. It therefore shows how much of the looped-model overhead the engine removes, not whether a looped model is cheaper to serve than a standard model of the same quality. Both looped models are small, the upstream engine release is labeled experimental, and all GPU results come from one GH200 host type and one RTX 5090. Huginn packed prefill is within the one-point rule on its point estimate only (see the accuracy checks in the main text).

C

T HE KV PATH : TCP WAS N OT THE B OTTLENECK

NIXL’s UCX plugin requests peer failure handling by default. Shared-memory and CUDA IPC transports have no peer failure handler, so UCX falls back to TCP for active messages and emulates remote writes with them; removing TCP from the allowed transports made the PD engines fail at startup. With the backend created with error handling disabled, UCX’s log showed CUDA IPC and shared memory and no TCP lane, and the engines also ran with TCP removed. A 64 MB chunk still took 16.9 instead of 20.0 ms, the commit tail stayed near one second, and behind per-row prefill a chunk still took 51.7 ms. Chunks are submitted, and their completion is observed, only between engine steps, under a fixed in-flight budget, so the tail is set by the number of budget rounds times the step length. That is why the larger budget helped and the faster link did not. A budget that holds one request’s KV, 256 MB chunks and 2 GB in flight for 1.6 GB per request, removes the rounds: in two runs each the commit tail fell from 0.96–1.05 s to 17 ms, the median TTFT from 3.38–3.91 to 2.01 s, and throughput rose to 463.4–466.0 tokens/s. A 1 GB budget, two rounds, kept most of the tail. Holding one request’s KV is a property of this load, not a rule: with 8192-token prompts at 8 requests, where one request’s KV exceeds 2 GB, the tail was 41 ms with 2 GB in flight, 43 ms with 8 GB and 0.08–0.27 s with 256 MB (the PD table of the main paper), so what counts is the KV still in flight when prefill compute ends, against the budget. ExactLoop’s default budget is now 2 GB; inflight bytes are not a buffer, since NIXL writes straight from the prefill pool into the decode pool. Disabling error handling has a cost: peer failures then surface only through the control channel.

D

C OMPONENT TABLES

These component measurements support the evaluation section of the main paper. They come from earlier trees than the headline tables, as each caption states. Table A1. Plane packing alone on Huginn-0125 with the released defaults otherwise, 512-token prompts, 64 output tokens, GH200, same KV pool. Throughput in output tokens/s, median TTFT in seconds. Three runs per cell, first dropped, mean of two.

C unpacked packed TTFT unpacked TTFT packed 8 32 64

96.5 144.6 133.0

97.7 174.9 170.9

0.60 2.69 10.76

0.60 2.78 5.48

Table A2. Fused kernels on Ouro-1.4B, otherwise released defaults, 512-token prompts, 128 output tokens, standalone GH200; speedup over all fused kernels off. Two interleaved rounds of three runs, first of each dropped, mean of four. C unfused (tok/s) exact, 3-kernel norm

exact approx.

1 8 32 128

1.55× 1.59× 1.51× 1.11×

54.9 350.2 782.4 1218.8

1.30× 1.24× 1.09× 0.97×

1.58× 1.67× 1.40× 1.23×

Table A3. Fused-kernel scope, Ouro-1.4B, 512-token prompts, GH200, per-row prefill. “graph”: fused kernels only during graph capture; “always”: every scope; “off”: no kernel fusion; “rerun”: a second base-engine arm, showing run-to-run spread. Base in output tokens/s, other columns relative to it; prefill time in seconds for base, graph and always. One kept run per cell.

C base graph always

off rerun

prefill b/g/a

8 351 1.38× 1.21× 0.97× 1.00× 0.44/0.33/0.57 32 749 1.31× 1.12× 0.98× 0.84× 1.61/1.39/2.16 64 927 1.22× 1.09× 1.14× 0.96× 4.16/3.58/4.45

What Sets the Serving Speed of Looped Language Models?

Table A4. Data movement per output token during decode, GH200, Ouro-1.4B, from a profiler window after prefill. H2D/D2H: host– device memcpys (D2H B: bytes); pageable: pageable-memory copies in the window; syncs: blocking stream, event and device synchronizations; copy kern.: device kernels that copy or cast. decode path

engine H2D D2H D2H B pageable syncs copy kern. kernels

Ouro, async, C=1 base 0.00 1.00 Ouro, async, C=1 ExactLoop 0.00 1.00 Ouro, async, C=32 base 0.00 0.06 Ouro, async, C=32 ExactLoop 0.00 0.03 Ouro, spec. K=4, C=8 base 0.03 0.03 Ouro, spec. K=4, C=8 ExactLoop 0.03 0.03 Ouro, sync, C=8 base 0.25 0.75 Ouro, sync, C=8 ExactLoop 0.12 0.12 Ouro, sync static, C=8 base 0.00 0.75 Ouro, sync static, C=8 ExactLoop 0.00 0.12

8 8 8 8 13 13 24 8 24 8

0 0 0 0 0 0 6 8 0 0

2.12 2.06 0.21 0.07 0.03 0.03 1.08 0.28 0.83 0.16

1004.0 29.0 44.2 1.1 85.5 4.4 126.5 3.6 126.0 3.5

5669 2852 249 89 480 249 710 356 710 356

Table A5. Ouro-1.4B agent sessions, six turns, prompts up to 8000 tokens, prefix cache on, batch-scheduled GH200, earlier tree, 64 graphs unless noted. “rows”: per-row prefill; “share”: per-loop prefill share. Throughput in output tokens/s, median TTFT and TPOT in seconds. Three runs per arm, the first dropped as in the burst tables, mean of two. 32 sessions 64 sessions tok/s TTFT TPOT tok/s TTFT TPOT base, 16 graphs base ExactLoop, no share, rows + packed prefill released (+ share)

98 113 135 187 179

0.15 0.15 0.34 0.13 0.11

0.30 72 0.20 78 0.20 91 0.12 218 0.12 192

1.48 1.95 3.44 0.94 0.25

1.08 0.99 0.43 0.14 0.32

What Sets the Serving Speed of Looped Language Models? Table A6. Ledger of GPU batches, in the order run. “5090”: rented RTX 5090; “GH200-B”: batch-scheduled GH200 (one GPU, two for PD); “GH200-S”: standalone GH200 host. Refuted expectations and batches without data are included. # system

question

1 5090

Where does time go in the base engine?

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

outcome

Huginn’s pool holds 1328 tokens and throughput stays near 17.8 tokens/s from 8 requests on; speculative decoding wins only at low load; hit rates follow pool size. 5090 Do fewer submissions speed up decode? Submissions per token 6.0 to 3.0; throughput 0.99× at batch 1 and 1.09× at 32 requests; 160/169 requests identical. 5090 Where does host time go? Per-row KV bookkeeping is visible at 32 requests; a cheaper shared record broke the per-layer safety contract and was withdrawn. 5090 Can PRELUDE be fused into decode? Regression: batches split into small cohorts. A policy fix removed it, the remaining gain was small, and the feature stays opt-in. 5090 Do cache fixes and packed prefill help sessions? Packed prefill was the largest session lever; the hit rate tracked pool size. 5090 How much do fused norm, RoPE and SiLU kernels Large batch-1 gain; greedy outputs changed. give? 5090 Which fused kernel carries the gain? RMSNorm. Both missed the one-point rule: −3.45 and −2.30 points on 87 questions. That subset cannot show 5090 Are approximate kernels harmless on GSM8K? whether accuracy drops. Loop fusion: 0 of 87 sequences differ. 5090 Is loop fusion identical on real text at low load? 41/41 sequences identical; two base runs 41/41. GH200-B Is plane packing bitwise, and how much capacity 233/233 identical with the same batch; 309/312 when the larger capacity changes batch composition. does it add? 1.88× tokens, 1.14× and 1.16× at 32 and 64 requests. Session arms exceeded Huginn’s context: no data. GH200-B Is the fused RMSNorm bitwise? No: its residual epilogue rounded once, and 18.4% of elements differed by one ulp. Fixed with PyTorch’s addition. GH200-B Is the fixed norm bitwise; does packed prefill help All requests identical; packed prefill 2.42× and 2.00× with 2048-token prompts. long prompts? GH200-B Does packed prefill work for Huginn? Yes, up to 2.08×; tokens change, so it needed a gate. GH200-B Can rsqrt move into the tail kernel bitwise? Yes; outputs identical with fewer launches. GH200-B Do fenced RoPE and SiLU stay bitwise? 169/169 identical; 1.14–1.44×. GH200-B Four-GPU end-to-end batch Cancelled before it started, at the user’s request; its arms moved to GH200-S. GH200-S Do the exact kernels and Huginn packed prefill pass Ouro: 0 of 87 sequences differ. Huginn: 20 vs 21 correct of 70. SiLU: 0 mismatching elements. accuracy gates? GH200-S How much faster is ExactLoop end to end? Huginn 1.49–1.83×. Ouro eager mode 0.40×: fused kernels outside graphs cost host time. Session arms timed out: no data. GH200-S Does graph-only fusion fix that? What happens in Prefill regression gone. Loop fusion delayed admission (median TTFT 1.41 to 6.41 s). Huginn prefix sessions? cache 2.12× (single run). GH200-S Is packed prefill slower for short prompts? No; an earlier reading came from a slow first repeat and was retracted. Without UVA staging, 0.88× and 0.93×. GH200-S Does the per-loop prefill share undo the TTFT cost? Bursts: better on every metric. Sessions: TTFT down, time per token up. GH200-S Released defaults, repeats; GPU cache identity; cen- 1.33–1.45× (Ouro), 1.85–2.60× (Huginn) in every repeat. Cache hits match cold runs in 6/8 cases. sus; ncu Decode is zero-copy on the host. No ncu: counters restricted. GH200-B First PD runs No data: a benchmark bug, an upstream coordinator that could not start Huginn, and an overlap counter that missed coalesced transfers. GH200-B How much does ExactLoop help PD? 2.28× at 32 requests; commit tail as long as prefill compute; Huginn packing slower in one run. GH200-B Is the transfer limited by the budget or by the link? Not the budget behind per-row prefill; with packed prefill a larger budget cut TTFT by 48%. UCX used TCP. GH200-B Does KV move faster without TCP? The engines failed at startup: peer error handling needs TCP. GH200-B Same, with error handling off TCP gone, but a chunk took 16.9 instead of 20.0 ms: our TCP explanation was wrong. GH200-B Larger budgets under CUDA IPC; Huginn PD pack- A 2 GB budget, one request’s KV, cut the commit tail to 17 ms and gave 463.4–466.0 tokens/s in both ing repeated runs; now the default. Huginn packing in PD: no slowdown in two repeats. GH200-S Does position-keyed noise cost speed? No: 1.04–1.09× of sequential noise at 32 requests, ties at 1 and 8; it removes Huginn’s blocking noise reads. GH200-S Component effects and a Huginn gate, earlier tree, Exact kernels 1.03–1.50×, 507/507 identical; packing 1.21× and 1.28× at 32 and 64 requests; Huginn with repeats GSM8K 23, 22 and 21 of 87 (sequential, position, position with reuse). Base-engine arm and branch tests: no data (bad flag; no GPU flag). Base engine 143–147 tokens/s, three runs. ExactLoop ran at the old budget (the benchmark’s default GH200-B Final PD with repeats; PD tests; ncu overrode the engine’s); tests ran without the GPU. ncu: 2.4–3.8 µs per exact kernel. GH200-S Same, fixed: base-engine Huginn arm; branch GPU Final tree 1.59–2.57× the base engine on Huginn, 1.59–2.49× without fused kernels. No branch adds tests a GPU test failure: each failure is a missing FlashAttention package or also fails on the base engine. GH200-B Same, fixed: 2 GB budget, repeats, PD GPU tests CUDA IPC 3.3× at 32 requests (479 vs 143–147 tokens/s), TCP 3.1× in two of three runs; 8192-token prompts 79–89 vs 13.8; PD tests pass. GH200-S Huginn gate on the full test split No data at first (the protocol step needs a git checkout). Rerun: 300, 307, 306 of 1319 (sequential, position, with reuse): pass. GH200-S Same, with FlashAttention 3 and 4 FlashAttention tests pass; two failures remain, also on the base engine. GH200-B Adaptive drafts; the slow TCP run K=0 1.06–1.11×; the slow decode recurred once in five runs. GH200-B Speculation crossover; graph counters Budget 8 kept 1.55× at one request; the slow run had 252 eager graph fallbacks. GH200-B Does graph eviction remove the cliff? PD at budget 4: 195–208 to 283 tokens/s; sessions gained, GPU tests pass. GH200-B How many graphs by default? 64: sessions never fell back; six PD runs neither. GH200-B Sessions again: three runs, graph counters Base 16 to 64 graphs 1.15× and 1.10×; released 1.58× and 2.45× the base engine; Huginn 2.89×; earlier single runs superseded. GH200-B Prefill share in sessions, three runs Share cut TTFT 0.94 to 0.25 s at 0.88× throughput; no share with per-row prefill 3.44 s. GH200-B Release table and speculation with enough graphs No fallbacks: Ouro 1.45–1.57× and 2.33–2.54×, Huginn 1.43–2.21×; fixed K=4 0.75× at 32, not 0.30×. GH200-B FlashAttention 3 and 4 instead of Triton FA3 0.25–0.95×, FA4 0.80–1.03× of Triton: no gain. GH200-B Prefill split; exact eager kernels Elementwise: half of Ouro prefill; fused −29%, identical. GH200-B Huginn act/RoPE; one-kernel norm 1.07–1.11×, 1.06–1.18×, bitwise; first session run slow. GH200-B Release table, one-kernel norm tree Ouro 1.59–1.84× and 2.79–2.92×, Huginn 1.56–2.62×. GH200-S Kernel table, one-kernel norm, interleaved One-kernel norm: exact 1.11–1.59×, approx. 1.23–1.67×; 1014/1014 identical. GH200-B Tree before fused projections; sessions interleaved; Ouro 1.75–2.06×, sessions 1.87–3.00×; probe 132/132 bitwise. order probe GH200-B One GEMM for q/k/v and gate/up; paper tree Bitwise; 1.06–1.12× (rule set before the run), adopted: Ouro 2.0–3.8×, sessions 1.91–3.21×. Exact = unfused on all 1319; approx. −0.15 and +0.76; packed: Ouro −0.99 (batch 16: −0.08), GH200-S Do the accuracy decisions hold on the full split? Huginn −0.61; batch 64 replays batch 32. GH200-S Is Huginn’s packed result a sampling accident? How Batch 16 replays batch 32 (1319/1319 identical): one observation. Position-keyed noise, 128 prompts: often do GPU cache hits match? full hits 108/128, partial 116/128. GH200-S Do the release and Huginn-session gains hold with Same token identity; Ouro 1.65–2.15× (512) and 1.91–2.04× (2048), Huginn 1.77–3.16×; sessions one build? 2.72×. Smaller Ouro gains than GH200-B; the release and Huginn-session tables use this run GH200-S Real prompts, Poisson arrivals, a third model? ShareGPT offline Ouro 1.71×, Huginn 2.01×; online latency per token up to 5.6× and 2.9–4.2× lower; Ouro saturates later at the same ceiling; pre-registered sustainable-rate criterion not met. Ouro-2.6B 2.0–3.3×. 2 of 10 Ouro runs aborted (cuBLAS split-K GEMM, MMU fault). GH200-S Does packed prefill hold on a second benchmark? MATH-500, 496 questions: Ouro +0.40 (s.e. 1.21), Huginn −0.20 (0.61); exact = unfused on 496 of 496.

Related documents

• Closing the Loop: Practical Training Recipes for Looped Language Models#1155608
• OPEN-1B: A Fully Auditable Training Run#1123745
• OPEN-1B: A Fully Auditable Training Run#919384
• LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models#1356930
• Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis#925145