Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Zhixu Du1∗ , Yuhan Cheng1 , Hai (Helen) Li1 , Yiran Chen1

arXiv:2611.07211v1 [cs.DC] 4 Nov 2026

1 Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA

Abstract Looped transformers scale inference-time reasoning by repeatedly reusing a shared core, increasing computation without increasing parameter count. Serving prefilling and decoding with a uniform loop budget couples their costs. We present LoopDuet, a phase-dependent depth and training-free serving approach that selects the two budgets jointly while preserving decoding depth at prefilling boundaries. Our vLLM implementation gathers tokens requiring further recurrence into compact batches, turning the selected allocation into executable savings with existing weights and attention kernels. At runtime, a queue-age controller adjusts prefilling depth to retain quality as demand changes. Experiments on the full HotpotQA dataset with one A100-80GB demonstrate two benefits of this flexibility. First, separate phase budgets improve the quality–throughput trade-off: Huginn delivers 19% more throughput than the fastest tested uniform setting meeting the same accuracy criterion, while a higher-throughput operating point gains 6.04 EM points over uniform depth eight at 97% of its throughput. Second, the same execution mechanism retains more quality under load: on two evaluated OLMo arrival traces, adaptive prefilling gains 0.92–1.35 EM points over the most accurate tested fixed-depth controls at the same 100% attainment of a one-second time-to-first-token target. Together, these results establish LoopDuet as a practical way to turn flexible model computation into higher serving capacity and better quality under load.

1.

Introduction

Looped transformers scale reasoning computation by repeatedly applying a shared core, increasing effective depth without increasing parameter count [25, 46]. Shared weights support different inference budgets, offering quality–cost choices at deployment. Exploiting this flexibility requires a depth allocation: a rule that decides how many loops each token receives. LLM serving processes each request in two phases, prefilling and decoding (Figure 1), which provide a natural basis for depth allocation. Prefilling processes prompt tokens together in several chunks, to build the KV cache; the final prompt position produces the first output token. Decoding then generates subsequent tokens one position at a time. A uniform loop budget couples their costs: lowering prefilling depth also lowers decoding depth. For long prompts and short answers, retaining deep decoding therefore incurs repeated computation across the entire prompt. Prior work exploits a common asymmetry between the phases: decoding is more sensitive to reduced computation than prefilling [31]. SwiftKV [52] computes prompts’ later-layer KV cache directly from an earlier layer’s output, skipping some later-layer computation during prefilling while decoding runs the full model; because this changes the input to the later layers’ KV projections, it distills their Q,K, and V projections from the ∗ Correspondence E-mail: zhixu.du@duke.edu

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

One prompt chunk Token 𝑦𝑡

𝑥𝑗 read

Shared core repeat loops

reuse Shared core repeat loops

KV cache write

update

First token 𝑦1 after final chunk

Next token 𝑦𝑡+1

(a) Prefilling

(b) Decoding

next token

next chunk

𝑥𝑖 𝑥𝑖+1 · · ·

Fig. 1: Prefilling advances by chunk; decoding advances by token. Prefilling processes chunk tokens in parallel. Cached context carries across chunks; the final chunk produces the first output token. Both phases reuse the same core weights.

original model. POP [31] applies a similar early exit during prefilling without training, but skips fewer layers since more aggressive pruning degrades quality. SPEED [50] processes prefilling tokens through only the first K layers and omits their deep-layer KV states, reducing both prefilling computation and decode-time KV reads, but requires fine-tuning. Our work applies this asymmetry to looped transformers without training: it reduces prefilling loops rather than physical layers, selects prefilling and decoding depths jointly, and adapts prefilling depth to load during serving. We present LoopDuet, a training-free serving approach that jointly selects prefilling depth 𝑃 and decoding depth 𝑆, with 𝑃 ≤ 𝑆. Joint selection includes both uniform budgets and split budgets that retain deeper decoding. In a split setting, ordinary prefilling positions stop after 𝑃 loops, while decoding positions and scheduled prompt-chunk endpoints receive 𝑆 (Section 3.1). This preserves the final prompt position’s first-token computation while reducing recurrent work over the surrounding context. This allocation saves computation only if the engine stops precessing positions that have finished their P loops. In a serving batch, however, each engine step mixes prompt chunks and decoding tokens from many requests, and after P loops only one query per request still needs recurrence. To remove completed prefilling positions from the subsequent core calls, LoopDuet applies a common loop boundary: after P loops, our vLLM implementation gathers the positions that need further recurrence into a compact batch for the remaining 𝑆 − 𝑃 loops. Prompt context remains available in a shared KV cache, so the compact queries can attend to it using existing attention kernels and weights (Section 3.2). A fixed P either misses first-token latency targets during bursts or gives up quality when capacity is available. To track changing request rates, LoopDuet starts from the jointly selected operating point and uses a queue-age controller that lowers 𝑃 when first-token backlog grows and restores it when capacity permits (Section 3.3). On the full HotpotQA dataset with one A100-SXM4-80GB, Huginn (𝑆, 𝑃) = (32, 16) delivers 19% more throughput than the fastest tested uniform setting meeting the same accuracy criterion. At a higher-throughput point, (14, 8) gains 6.04 exact-match (EM) points over uniform depth eight while retaining 97% of its throughput. This flexibility also retains more quality under load. For OLMo on HotpotQA, joint selection identifies uniform depth four as the fastest tested setting meeting the accuracy criterion. Starting there, the controller keeps 𝑆 = 4 and adjusts 𝑃 ∈ {2, 3, 4}. On the evaluated 60 requests/s and 10/60 requests/s traces, it retains 0.92 and 1.35 more EM points than the most accurate tested fixed prefilling depth with the same 100% attainment of a one-second time-to-first-token (TTFT) target. We make three contributions: • Joint depth selection: We characterize quality–throughput choices by comparing phase-dependent allocations with competitive reduced uniform depths. 2

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers • Batched execution and runtime control: We implement compact recurrence and queue-age adaptation in vLLM, translating depth allocation into serving capacity and quality retention under load. • Role-dependent sensitivity: We characterize depth sensitivity across two model families and three tasks, using recurrent-state measurements to screen candidate prefilling depths for task-quality validation.

2.

Serving Looped Transformers

Depth and memory are separate resources. The models we study consist of a prelude, a recurrent core, and a coda. The prelude computes an input representation 𝑒 𝑖 for token 𝑖. Each core application updates its recurrent state using the previous state, 𝑒 𝑖 , and causal attention context; the coda produces the representation for next-token prediction. Huginn uses 2/4/2 prelude/core/coda layers, while the OLMo retrofits use 4/6/4. Weight reuse controls parameter memory, but each loop incurs attention and feed-forward computation. Recurrent depth can also affect KV storage. A depth-indexed layout stores context for each iteration, whereas terminal or shared layouts reuse a cache across loops, making memory scale with physical layers rather than unrolled depth [25, 33, 48]. Our serving implementation uses one shared KV plane per core layer. Reducing prefilling depth removes computation beyond this cache-sharing benefit. All serving comparisons use the same shared layout. Prefilling work can dominate the arithmetic. Let 𝑎, 𝑏, and 𝑐 denote the numbers of prelude, core, and coda layers, and let 𝑐 𝑘 = 𝑎 + 𝑏𝑘 + 𝑐 be the layer applications for a position receiving 𝑘 loops. For a prompt of length 𝐿 followed by 𝐷 decoding-token evaluations, uniform execution at depth 𝑆 performs (𝐿 + 𝐷) 𝑐 𝑆 layer applications. For OLMo HotpotQA, prompts average 796 tokens and answers about 4.7 tokens, so prompt positions account for over 99% of this work. This concentration motivates reducing prefilling work while retaining decoding depth. Arithmetic savings and wall-time savings differ because prefilling processes many positions in parallel, whereas decoding advances sequentially. Chunked prefilling and continuous batching further couple their latency [2, 35]. The layer model identifies removable computation; phase timing determines its wall-time value. The optimization unit should survive batching. Conditional computation saves hardware work when pertoken selections remove operations from the executed batch. MoE systems illustrate this distinction between per-token selection and aggregate expert traffic [57, 28]. We use a common prefilling-loop boundary, after which the engine gathers endpoints and decoding positions for further recurrence.

3.

LoopDuet Design

LoopDuet allocates recurrent computation by serving phase. We first select prefilling and decoding budgets jointly from their measured quality–throughput trade-off. The execution path then enforces these budgets by gathering positions that require further recurrence, while retaining context from positions that stop earlier. Finally, queue-age control adjusts prefilling depth within an offline-characterized range. Figure 2 connects the allocation rule to this execution path.

3.1.

Separate Prefilling and Decoding Depths

We write (𝑆, 𝑃) for decoding and prefilling core iterations, respectively, with 𝑃 ≤ 𝑆. For an unchunked prompt of length 𝐿, positions 1, . . . , 𝐿 − 1 execute 𝑃 iterations. Position 𝐿 remains part of prefilling but executes 𝑆

3

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

first output

Illustration: S = 4, P = 2

Prefilling

Prelude

skip

4 Core loop

Decoding

Scheduled tokens

HP

Queue age q ↦ P Retain other rows

Core ×P N×d

3

Scatter I Core ×(S − P)

1

1

2

··· L−1

Context KV stops at P

L

Last prompt keeps S

y1

y2

Shared KV cache (one plane per core layer)

Decoding tokens keep S

(a) Separate prefilling and decoding depths

Coda

N×d

Gather I

Read / write

2

H̃

B×d

B×d

I: final query of each request, including prompt-chunk endpoints.

(b) Compact recurrent execution in a mixed batch

Fig. 2: LoopDuet reduces prefilling loops while preserving decoding depth. (a) Ordinary prompt tokens receive 𝑃 loops during prefilling; the final prompt token and decoding tokens retain 𝑆. (b) After 𝑃 loops on 𝑁 scheduled tokens, gather the final query of each of 𝐵 requests, run 𝑆 − 𝑃 loops on this compact tensor, and scatter the updated rows before the coda. Here 𝑑 is the hidden width. Both paths use the same KV planes and existing attention kernels. The diagram shows hidden-state flow; prelude states and attention metadata use the same gather indices. The dashed arrow changes 𝑃 between engine steps. Every prompt-chunk endpoint retains 𝑆, including intermediate endpoints; single-query batches follow the full-depth path.

iterations to produce the first output token. Every subsequent generated position also executes 𝑆. Prelude and coda computation remain present for all positions. Continuing positions need access to the shallower prompt context. In the accuracy harness’s depth-indexed cache, skipped entries are filled from the last computed recurrence at the same physical core layer. In the serving implementation’s shared layout, each recurrence overwrites that layer’s KV plane, leaving each prompt position’s last computed state available to later queries. Decoding can therefore access prompt context without executing the omitted prefilling loops. We evaluate task accuracy under this approximation to full-depth context. Prefilling endpoints retain decoding depth. The final prompt position belongs to prefilling, but its coda output produces the first output token. Reducing its depth would compute that token from a shallower state; for short-answer tasks, the first token often determines the answer. LoopDuet therefore keeps this position at 𝑆 while reducing work over the preceding positions; Section 4.5 quantifies this choice. POP likewise preserves first-token computation in its prefill-specific allocation [31].

3.2.

Compact Execution with Shared Context

To realize these savings, the engine removes completed prefilling positions from later core calls. For a batch of 𝑁 tokens from 𝐵 requests with hidden width 𝑑, the prelude and first 𝑃 core applications operate on the full batch. Tokens within each prompt chunk are processed with causal attention; successive loops update their recurrent states. Let 𝐼 contain each request’s final scheduled query index. Gathering these rows produces a 𝐵 × 𝑑 tensor for the remaining 𝑆 − 𝑃 loops. The corresponding prelude states, positions, and one-query-per-request attention metadata follow the same indices. The gather reduces the active queries while leaving prompt KV entries in the shared cache; continuing positions can therefore attend to the full retained context through existing attention kernels. After recurrence, the engine scatters the updated rows back into the 𝑁 × 𝑑 tensor before the coda. Other rows retain their loop-𝑃 states. With chunked prefilling, every scheduled prompt-chunk endpoint continues to depth 𝑆, including intermediate 4

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

endpoints. This gives each engine step the same final-query metadata rule; only the final prompt endpoint produces the first output token. A batch whose maximum query length is one follows the full-depth path. The supplement provides engine-step pseudocode. Batches containing prefilling tokens run eagerly because the gathered tensor has a variable shape; decodingonly batches use full CUDA graphs. Uniform-depth controls use the same execution modes. At 𝑃 = 𝑆, this configuration is already 1.42× faster for OLMo and 1.22× for Huginn than vLLM’s compiled mode in the recorded controls, with EM differences within 0.06 points. Our depth comparisons therefore use the faster measured configuration.

3.3.

Adapt Prefilling Depth to Queue Age

Static depth selection identifies a budget for a workload, but capacity requirements change with arrival rates. LoopDuet responds by adjusting 𝑃 while retaining the selected decoding depth. Let 𝑞 be the age of the oldest admitted request awaiting its first token, including time in service. Before each engine step, the controller computes   𝑞 − 0.25 𝑧 = clip , 0, 1 , (1) 1.00 − 0.25
𝑃 = round 𝑃max − 𝑧(𝑃max − 𝑃min ) . (2) For an empty waiting set, 𝑞 = 0. Rising queue age lowers 𝑃 toward 𝑃min ; receding backlog restores it toward 𝑃max . Decoding depth remains fixed. For the main OLMo experiment, the accuracy–throughput comparison favors uniform depth four. We keep 𝑆 = 4 and use (𝑃max , 𝑃min ) = (4, 2), with parameters fixed before all evaluated traces. Here 𝑃max is the accuracy-screened starting depth, while 𝑃min sets the quality–capacity trade-off available under pressure. A prompt spanning multiple engine steps may encounter several values of 𝑃. Offline quality measurements establish this operating range; the runtime has no correctness feedback. We therefore evaluate both task quality and TTFT attainment as the controller changes depth. The appendix reports the supplementary 𝑆 = 11 control study separately.

4.

Evaluation

The evaluation of LoopDuet follows the serving decision from joint depth selection (Section 4.2) to compact execution (Section 4.3) and quality retention under load (Section 4.4). Role ablations and recurrent-state measurements then explain depth sensitivity and assess candidate-budget screening (Sections 4.5–4.6).

4.1.

Setup and Comparison Protocol

Models and tasks. We evaluate Huginn-0125 and recurrent OLMo-2 checkpoints trained at 16 and 32 loops [25, 46, 63]. Their widths are 5,280 and 2,048, respectively. The accuracy study uses the complete GSM8K test set, HotpotQA distractor dataset, and ARC-Challenge (ARC) test set [13, 70, 12]. GSM8K uses five demonstrations and greedy generation, with the last numeric answer extracted by the harness. ARC uses five demonstrations and scores the choice labels at the first answer position. HotpotQA requests short-phrase answers. OLMo uses one demonstration and a four-paragraph construction retaining the two supporting paragraphs and two distractors, with distractors removed as needed to meet a 1,000-token prompt cap. Huginn uses two demonstrations and a ten-paragraph construction. These are within-model comparisons under fixed prompt

5

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Exact match (%) ↑

35

(32, 32)

(32, 16)

(20, 20)

(14, 8)

30

Uniform (𝑆, 𝑆)

25

LoopDuet (𝑆, 𝑃)

(8, 8) 1

1.5

2

2.5

3

3.5

Throughput (req/s) ↑

Fig. 3: Separate budgets improve Huginn’s quality–throughput choices. Full-set HotpotQA, one A100-80GB, 𝐶 = 64. LoopDuet (32, 16) gives 1.19× throughput relative to the fastest tested uniform setting meeting the accuracy criterion, (20, 20). LoopDuet (14, 8) gains 6.04 EM points over uniform eight at 97% of its throughput. Lines join measured uniform points as a visual guide.

construction, rather than an accuracy ranking between models. The appendix records context variants and extraction details. Metrics. We report task accuracy, with exact match (EM) and token-level F1 for HotpotQA. All score differences are expressed in percentage points. Joint-depth comparisons use (32, 32) as the baseline; prefilling-depth ablations compare configurations at the same decoding depth 𝑆. Serving. All timings use vLLM 0.14.1, bf16, one exclusively used A100-SXM4-80GB and no prefix caching. Wall time excludes model loading and an eight-request warm-up. Closed-loop runs keep up to 𝐶 requests in flight; open-loop runs admit seeded Poisson or alternating-rate arrivals when due. TTFT starts at scheduled arrival and includes queueing. Time per output token (TPOT) is the interval from first to last token divided by the number of subsequent tokens, evaluated on multi-token responses. Latency distributions describe variation within recorded runs. Evaluation implementation. Serving accuracy and performance are measured in the same vLLM runs. We use a separate accuracy harness for token-role and KV-cache ablations. For recurrent-state analysis, we first generate answers at each checkpoint’s reference depth. We then run the model on the original prompts and recorded answers, keeping the answer tokens fixed and recording hidden states at each loop.

4.2.

Joint Depth Selection Improves the Trade-off

We compare uniform (𝑆, 𝑆) and LoopDuet’s split-depth (𝑆, 𝑃) settings in the same serving engine. Including reduced uniform depths tests whether separate phase budgets improve the quality–throughput choices available to deployment. All evaluated configurations use the complete HotpotQA dataset, 𝐶 = 64, and fixed within-model prompts, initialization, KV layout, precision, and execution mode. Throughput is completed requests divided by wall time. The appendix lists all 11 Huginn and six OLMo configurations with EM and F1. Huginn retains answer quality at lower prefilling cost. Separate budgets benefit Huginn at both higher-quality and higher-throughput operating points (Figure 3). At the higher-quality point, (32, 16) reaches EM 33.69 and F1 44.29 at 1.82 requests/s. The fastest tested uniform setting meeting the same full-depth criterion is (20, 20), with 6

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Table 1: For Huginn, LoopDuet (14, 8) adds 6.04 EM points over uniform depth eight while retaining 97% of its throughput. Comparisons use full-set HotpotQA, 𝐶 = 64. Throughput ratios report measured LoopDuet/uniform performance for configurations whose throughputs differ by at most 10%. Positive quality differences favor LoopDuet. Model

LoopDuet (𝑆, 𝑃)

Uniform

Throughput ratio

ΔEM

ΔF1

Huginn Huginn Huginn Huginn OLMo r32

(14, 8) (24, 12) (32, 16) (32, 16) (11, 4)

(8, 8) (12, 12) (16, 16) (18, 18) (4, 4)

0.973 0.979 0.960 1.073 0.918

+6.04 -0.45 +1.62 +0.97 +0.32

+7.13 -0.47 +1.90 +1.10 +0.49

EM 33.52 at 1.54 requests/s. Relative to that alternative, LoopDuet provides 1.19× throughput and a +0.18-point EM difference. Relative to full (32, 32), it provides 1.87× throughput with a −0.09-point EM change. The benefit also extends to higher-throughput settings. At (14, 8), Huginn reaches 30.33 EM at 3.49 requests/s, compared with 24.29 EM at 3.59 requests/s for uniform (8, 8). Retaining deeper decoding therefore adds 6.04 EM and 7.13 F1 points while preserving 97% of uniform-eight throughput. This point trades quality relative to full depth for capacity, while retaining more quality than uniform reduction at nearby cost. Table 1 compares nearby measured alternatives. Against uniform (18, 18), (32, 16) gives 7.3% higher throughput and 0.97 more EM points; against (16, 16), it gives 1.62 more EM points at 96% of throughput. The results also show why prefilling and decoding depths must be selected jointly. At (24, 12), Huginn gives 31.60 EM at 2.43 requests/s, while uniform (12, 12) gives 32.05 EM at 2.48 requests/s, an EM difference of −0.45. Additional decoding loops therefore benefit specific joint allocations. OLMo selects a uniform starting point. For OLMo HotpotQA, uniform (4, 4) reaches 56.46 requests/s with EM 37.95, a −0.54-point change from (32, 32). Across the six configurations, EM ranges from 37.95 to 38.49; (4, 4) provides the highest throughput. LoopDuet (11, 4) gives 51.85 requests/s and EM 38.27, a +0.32-point difference from uniform four, with F1 50.12 versus 49.63. The preferred point depends on the deployment’s quality objective; under the stated EM criterion, uniform four supplies the largest measured throughput. Both models produce short HotpotQA answers, yet their accuracy–throughput trade-offs favor different depth allocations. Joint selection therefore determines whether a workload benefits from deeper decoding at a given prefilling budget. It expands Huginn’s options through split budgets and selects OLMo’s uniform (4, 4) point as the starting budget for load control (Section 4.4).

4.3.

Prefilling-Loop and Batched Speedups

A layer-pass model bounds the savings. Split execution assigns 𝑐 𝑃 to the first 𝐿 − 1 prompt positions and 𝑐 𝑆 to the final prompt position and all decoding positions. The ratio of uniform to split work is 𝑅layer =

(𝐿 + 𝐷) 𝑐 𝑆 . (𝐿 − 1) 𝑐 𝑃 + (𝐷 + 1) 𝑐 𝑆

(3)

With 𝑀 prompt chunks, replacing 𝐿 − 1 and 𝐷 + 1 in the denominator by 𝐿 − 𝑀 and 𝐷 + 𝑀 accounts for the full-depth chunk endpoints. For long prompts with short answers, the ratio approaches 𝑐 𝑆 /𝑐 𝑃 : 3.57 for OLMo (32, 8), 2.31 for OLMo (11, 4), and 1.94 for Huginn (32, 16). The model omits adapter, vocabulary-head, memory-traffic, and scheduling costs.

7

Median TTFT (s) ↓

Throughput (req/s) ↑

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

30 20 10 0

27.32 9.16

(32, 32)

60 40 20 0

(32, 8)

51.85 24.3

1.82

2 1

0.98

0 (11, 11)

(11, 4)

(32, 32)

(32, 16)

Uniform (𝑆, 𝑆) LoopDuet (𝑆, 𝑃)

4 3 2 1 0

3.33 1.11

(32, 32)

(32, 8)

(a) OLMo r32: 𝑆 = 32

1.5 1 0.5 0

1.25 0.58

(11, 11)

(11, 4)

(b) OLMo r32: 𝑆 = 11

60 40 20 0

51.93 27.78

(32, 32)

(32, 16)

(c) Huginn: 𝑆 = 32

Fig. 4: Removing prefilling loops increases throughput and reduces time to first token. Full-set HotpotQA, 𝐶 = 64, one A100-80GB. Each column compares the same model and decoding depth: OLMo uses four paragraphs and Huginn ten. Top: request throughput. Bottom: median closed-loop TTFT, including scheduling effects. Each axis uses absolute units and its own scale. The throughput ratios are 2.98×, 2.13×, and 1.87×; the corresponding EM changes are −0.07, −0.16, and −0.09 points.

Fixed decoding depth isolates the prefilling savings. To isolate prefilling savings, we hold decoding depth fixed. Figure 4 shows OLMo’s 808.7-second run falling to 271.1 seconds and Huginn’s 7,588-second run to 4,064 seconds. The corresponding EM changes are −0.07, −0.16, and −0.09 points. These fixed-𝑆 ablations isolate prefilling savings; joint depth comparisons guide the choice of decoding depth. For the three 𝐶 = 64 configurations, measured speedups reach 85–96% of the layer-pass ratios, consistent with compact dense execution realizing most of the arithmetic savings. These are within-model ratios; absolute OLMo and Huginn times also reflect different widths and context constructions. The savings persist at serving concurrency. Concurrency and token budgets constrain different aspects of the serving batch. We vary both to test whether the savings persist as the executed grouping changes. Huginn (32, 16) sustains 1.87× throughput across 𝐶 = 32, 64, 128 at the default 8,192-token engine-step budget, relative to matched (32, 32) controls. At 𝐶 = 128, token budgets of 4,096, 8,192, and 16,384 yield speedups of 1.84×, 1.87×, and 1.88×, with EM differences of +0.12, −0.09, and −0.14 points. The reduced-depth runs execute 3,797, 1,902, and 957 engine steps, respectively. This sweep changes the executed batching regime while retaining the prefilling-depth benefit. Median TTFT falls from 126.6 to 68.7 seconds at budget 4,096 and from 103.1 to 54.8 seconds at 16,384. OLMo shows the same consistency across 𝐶 = 32, 64, 128: (11, 4) gives speedups of 2.10×, 2.13×, and 2.09×; (32, 8) gives 2.92×, 2.98×, and 2.98×. Less prefilling work also reduces interference. Prefilling and decoding share the engine schedule, so removing prefilling work can improve latency in both phases. At OLMo (11, 4), median closed-loop TTFT falls from 1.26 to 0.58 seconds and median TPOT from 0.420 to 0.196 seconds. The share with TTFT at most one second rises from 44.2% to 99.7%. Decoding retains the same loops per token; the TPOT reduction accompanies the lighter prefilling workload in the schedule. At 𝐶 = 128, measured request throughput rises from 24.8 to 51.8 requests/s. Huginn’s median TTFT falls from 51.9 to 27.8 seconds. Workload and granularity determine the payoff. The payoff depends on workload composition. HotpotQA’s long prompts and short answers yield the measured throughput gains, whereas GSM8K’s longer responses give 8

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Table 2: One-second TTFT attainment (%) / EM on all HotpotQA examples, OLMo r32, 𝑆 = 4, four-paragraph construction, open-loop 𝐶 = 128. The same queue-control parameters apply to all four traces. LoopDuet attains the one-second target for every request on the first three traces and for 87.5% on the 20/80 burst trace. Bold highlights LoopDuet.

Arrival trace

Fixed (4, 4)

Fixed (4, 3)

Fixed (4, 2)

LoopDuet (queue) (4, 𝑃 ∈ [2, 4])

Poisson 40 req/s Poisson 60 req/s Burst 10/60 req/s Burst 20/80 req/s

100 / 37.92 8.6 / 38.04 46.9 / 38.04 21.3 / 38.20

100 / 36.29 100 / 36.35 100 / 36.42 39.1 / 36.42

100 / 31.83 100 / 31.74 100 / 31.82 100 / 31.91

100 / 37.96 100 / 37.27 100 / 37.77 87.5 / 35.12

wall-time ratios of 0.98–1.01 across four serving pairs despite layer-pass ratios of 1.52–2.71. Amdahl’s law explains this gap:   −1 𝑅time ≈ (1 − 𝑓 ) + 𝑓 /𝑟 + 𝜖 , (4) where 𝑓 is the baseline wall-time fraction spent on the affected work, 𝑟 its acceleration, and 𝜖 the added overhead normalized by baseline time. Longer responses lower 𝑓 , which bounds the gain by 1/(1 − 𝑓 ) regardless of 𝑟. The appendix reports the GSM8K timings. Selection granularity also determines whether savings survive batching. In the FFN-routing study, retaining half the neurons per token gives some accuracy-preserving configurations, yet their union covers 99.80–99.95% of neurons at batch size 16. A common prefilling-loop boundary instead removes later core applications from ordinary prompt positions across the batch. Complete routing and union results appear in the appendix.

4.4.

Queue Control Retains Quality under Load

More quality at the required capacity. Starting from OLMo’s selected (4, 4) point, LoopDuet’s controller keeps 𝑆 = 4 and adjusts 𝑃 ∈ {2, 3, 4}. Fixed prefilling depths at the same 𝑆 provide the controls. All policies process the same dataset and seeded arrivals, with parameters fixed across loads. At Poisson 60 requests/s, the controller retains 0.92 more EM points than the most accurate tested fixed depth attaining the one-second TTFT target for every request. Fixed 𝑃 = 4 gives EM 38.04 at 8.6% attainment; fixed 𝑃 = 3 reaches 100% at EM 36.35. The controller also reaches 100%, with EM 37.27 and median / 99th-percentile TTFT of 0.486 / 0.725 seconds. The same benefit holds under alternating 10/60 requests/s demand. The controller reaches EM 37.77 at 100% attainment, a 1.35-point gain over fixed 𝑃 = 3 at EM 36.42. Its TTFT p99 is 0.682 seconds. The trace spends approximately 87% of wall time at 𝑃 = 4 and 13% at 𝑃 = 3, restoring depth when queue age permits. These are controller-state time fractions, not token-weighted per-request depths. At Poisson 40 requests/s, capacity is sufficient and the controller remains at 𝑃 = 4. Both it and fixed 𝑃 = 4 attain the target for every request, with EM 37.96 and 37.92, respectively. The operating envelope sets the quality–latency trade-off. On the stronger 20/80 burst trace, the controller gives EM 35.12 and 87.5% attainment, with TTFT p99 1.110 seconds. Fixed 𝑃 = 2 reaches 100% at EM 31.91. The controller’s +3.21-point EM difference is therefore a quality–latency trade-off, distinct from the equal-attainment gains above. The appendix examines how the depth range and control signal affect queueing and retained quality.

9

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

TTFT ≤ 1 s (%) ↑

+1.35 EM 100 𝑃=2

𝑃=3

80 Fixed prefilling depth LoopDuet (queue)

60

𝑃=4

40 31

32

33

34

35

36

37

38

39

Exact match (%) ↑

Fig. 5: Adaptation retains more quality at full measured SLO attainment. OLMo at 𝑆 = 4 on the 10/60 requests/s burst trace. LoopDuet’s queue controller gains 1.35 EM points over fixed 𝑃 = 3, with zero one-second TTFT violations. Uniform ( 𝑃, 𝑃) OLMo r32

LoopDuet (𝑆, 𝑃)

−4.85

(11, 4)

+1.06

OLMo r16

−1.44

(16, 4) Huginn

+1.74 −10.99

(14, 8)

+0.68 −12

−8

−4

0

4

Accuracy change vs. (𝑆, 𝑆) (pp) ↑

Fig. 6: Deeper decoding preserves GSM8K accuracy across checkpoints. At the same prefilling depth, LoopDuet gains 5.91, 3.18, and 11.68 accuracy points over uniform reduction for OLMo r32, OLMo r16, and Huginn, respectively. Bars show changes from each checkpoint’s (𝑆, 𝑆) control (zero line); row labels give (𝑆, 𝑃). LoopDuet retains depth 𝑆 for decoding and the final prefilling position.

4.5.

Token Role Explains Depth Sensitivity

The first output token needs full depth. We first reduce only the final prompt position. For Huginn on HotpotQA with 𝑆 = 32 and ordinary prompt positions at 𝑃 = 16, running the final position at 16 instead of 32 loops lowers EM by 1.74 points relative to (32, 16) and by 1.97 points relative to (32, 32). Keeping this position at 𝑆 costs one extra full-depth position per request. Deeper decoding preserves GSM8K accuracy. At the same prefilling depth, retaining deeper decoding and first-token computation improves GSM8K accuracy by 3.18–11.68 points across all three checkpoints (Figure 6). For OLMo r32, (11, 4) changes accuracy by +1.06 points relative to (11, 11), whereas uniform (4, 4) changes it by −4.85 points. Huginn (14, 8) changes accuracy by +0.68 points relative to its depth-14 control while uniform depth eight loses 10.99 points. The benefit depends on the model and task. On HotpotQA, Huginn (14, 8) gains 4.97 EM points over uniform depth eight and remains within 0.72 points of its depth-14 control. The OLMo comparisons change by 0.16–0.20 points at the same prefilling depth (Figure 7). This difference explains why retaining deeper decoding is more valuable for Huginn on this task. Prefilling depth must also match task sensitivity. On ARC, increasing OLMo r32’s prefilling depth from four to eight at 𝑆 = 11 changes the accuracy delta from −1.54 to +0.85 points (Figure 8). Huginn (14, 8) gains 5.63 points over uniform depth eight but loses 4.44 points relative to (14, 14). Deeper decoding improves this 10

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Uniform ( 𝑃, 𝑃)

LoopDuet (𝑆, 𝑃)

OLMo r32

−0.46 −0.26

(11, 4) OLMo r16

−0.46 −0.30

(16, 4) Huginn

−5.69

(14, 8)

−0.72 −8

−4

−6

−2

0

EM change vs. (𝑆, 𝑆) (pp) ↑

Fig. 7: Huginn benefits more from deeper decoding on HotpotQA. LoopDuet (14, 8) gains 4.97 EM points over uniform depth eight and stays within 0.72 points of its depth-14 control. The OLMo gains are 0.20 and 0.16 points. Bars show EM changes from each (𝑆, 𝑆) control (zero line); row labels give (𝑆, 𝑃). Uniform ( 𝑃, 𝑃) OLMo r32

LoopDuet (𝑆, 𝑃)

−1.45 −1.54

(11, 4) OLMo r32

+0.60 +0.85

(11, 8) Huginn

−10.07

(14, 8)

−4.44 −12

−8

−4

0

4

Accuracy change vs. (𝑆, 𝑆) (pp) ↑

Fig. 8: ARC also requires sufficient prefilling depth. At 𝑆 = 11, increasing OLMo prefilling depth from four to eight changes the accuracy delta from −1.54 to +0.85 points. Huginn (14, 8) gains 5.63 points over uniform depth eight, with a −4.44-point change from (14, 14). Bars use the same-𝑆 control as zero; row labels give (𝑆, 𝑃).

allocation, while the remaining accuracy cost motivates task-specific prefilling depth. The serving cache retains the effect. Shared-KV harness comparisons yield deltas of −0.23 and −0.09 for OLMo GSM8K and HotpotQA, and −0.38 and −0.49 for Huginn. The Huginn HotpotQA comparison is (32, 16) versus (32, 32).

4.6.

Recurrent States Prefilling Depths

Role-specific state changes help explain depth sensitivity and identify candidate prefilling depths. Distances are measured separately for ordinary prompt tokens, the final prefilling token, and decoding tokens. Replaying recorded answers through 𝑆ref loops supplies the states at each intermediate loop. For token 𝑖 and a candidate decoding depth 𝑆 ≤ 𝑆ref , define ∥ℎ 𝑃 − ℎ 𝑆 ∥ 2 𝑑𝑖 (𝑃; 𝑆) = 𝑖 𝑆 𝑖 . (5) ∥ℎ𝑖 ∥ 2 Both states come from the same trajectory. Curves to the terminal reference use 𝑆 = 𝑆ref ; lower-𝑆 diagnostic cells use the corresponding pairwise intermediate-loop distances. The available traversals use 𝑆ref = 32 for OLMo r32 and Huginn and 16 for OLMo r16. These are finite-depth reference states, rather than demonstrated 11

90th-percentile distance

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Prefilling Decoding Prefilling threshold .24 Decoding threshold .11

0.6 0.4 0.2 0 2

4

8

6

10

12

Core loop

Fig. 9: State distance has role-dependent quality implications. OLMo r32 GSM8K, depth-32 reference. At loop four, ordinary prompt tokens (prefilling) and generated tokens (decoding) have distances 0.18 and 0.20, but their depth interventions have different accuracy effects. Horizontal lines show thresholds fitted across the evaluated depth configurations.

mathematical fixed points. At four loops, OLMo r32 GSM8K has prefilling-token and decoding-token 90th-percentile distances 0.18 and 0.20. Prefilling reduction to (32, 4) leaves measured accuracy at 48.29, equal to (32, 32). Uniform reduction to (4, 4) gives 42.53 (−5.76 points). Thus comparable distance magnitudes accompany different quality outcomes depending on the role receiving reduced depth. Since uniform reduction changes both prefilling and decoding computation, this is evidence of role sensitivity rather than an isolated causal intervention on decoding states. Across the measured cells, the best separating threshold on the prompt 90th-percentile distance is approximately 0.24. Applying the same procedure to decoding-token distances for uniform-depth cells gives approximately 0.11. These fits associate larger prompt-state deviations with preserved quality under prefilling reduction. The roughly twofold difference describes the evaluated cells and roles; it does not require prompt states always to settle earlier. The supplement reports the crossing-depth comparisons. One reference trace supplies candidate statistics. Let 𝑞 .9 (𝑃; 𝑆) be the 90th percentile of the ordinary-prompt distances for a candidate (𝑆, 𝑃). We propose candidate depths satisfying 𝑞 .9 (𝑃; 𝑆) ≤ 𝜏,

𝜏 ≈ 0.24.

(6)

One replay of recorded answers supplies state statistics for all candidate loop pairs, including lower-𝑆 cells extracted from the same trajectory. We then evaluate the proposed depths on task accuracy. Fitting on two checkpoints and evaluating the third gives thresholds between 0.237 and 0.241, close to the pooled value of 0.24. The distance curves suggest candidate prefilling depths around 3–4 for OLMo and 13–16 for Huginn. The appendix compares the resulting selections with task-quality measurements and alternative selection rules. Task sensitivity matters even at small state distances. On ARC, OLMo (32, 4) and (11, 4) lose 1.88 and 1.54 accuracy points, and Huginn (32, 16) loses 1.45 points, all at distances below 0.18. OLMo GSM8K (32, 8) loses 1.29 points at distance 0.037. Conversely, Huginn GSM8K (14, 8) gains 0.68 points at distance 0.614. These observations suggest that the task relevance of a state change matters alongside its magnitude, motivating task-specific depth selection. The appendix examines these patterns, aggregation choices, and prompt composition.

5.

Related Work

Phase-specific prefilling computation. Prior methods reduce prefilling cost through approximation or architectural separation. SwiftKV distills later-layer prompt cache states [52], while POP skips deep prefilling and 12

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

preserves KV projections and first-token computation [31]. SPEED changes prompt visibility using representation diagnostics [50]. Decode-Branch Transformers separate prefilling/cache and decoding paths [44], and YOCO enables prefilling exit through a shared self-decoder cache [59]. LoopDuet connects joint depth selection on existing recurrent checkpoints to batched execution and queue-driven prefilling budgets. Parameter sharing and recurrent depth. Transformer blocks [62] support depth-wise sharing in ALBERT, iterative refinement in Universal Transformers, and implicit equilibrium in DEQ [37, 19, 5]. Huginn and recurrent retrofits expose repeated computation as an inference budget [25, 46]. Fixed-point shaping supports terminal KV sharing and distilled or shallow teacher prefilling [33]. Other recurrent methods allocate precision (RAMP), change retained context (HLA and shared-memory models), or train across budgets (Elastic Latent Recurrence) [64, 11, 49, 36]. Our focus is phase-level depth allocation with existing weights. Early-exiting and skip layers Adaptive computation selects work at several granularities. ACT and PonderNet learn halting [26, 6]; BranchyNet and SkipNet add exits or block skipping [61, 66]. DeeBERT, FastBERT, and patience-based exits use prediction confidence or consistency [68, 45, 75]. LayerDrop trains removable layers, Mixture-of-Depths learns capacity-constrained token routing, and Mixture-of-Recursions learns token-level recurrence through shared blocks [22, 54, 4]. We select phase-level budgets that remove batched work at a common loop boundary. Variable-depth autoregressive inference must preserve cache consistency and efficient batching. CALM, FREE, SkipDecode, and EE-LLM address these requirements for early exits [55, 3, 14, 10]. Draft & Verify and LayerSkip verify reduced-depth drafts [72, 21]; speculative sampling preserves the target distribution through verification [39, 8]. SpecInfer organizes token trees, Medusa adds prediction heads, and EAGLE variants construct feature-based drafts [47, 7, 41, 40]. Looped-model systems manage heterogeneous exits, batch recurrence steps, or speculate across position and depth [48, 56, 29]. LoopDuet directly uses the retained shallower prompt states during decoding. Scheduling and latency control. Clipper, Clockwork, and AlpaServe optimize batching, predictability, and resource sharing [15, 27, 42]; Orca and FastServe schedule iterations and preemption [71, 67]. SARATHI variants mix prompt chunks with decoding, while DistServe and Splitwise separate phase resources [1, 2, 74, 51]. Parrot uses application dependencies [43], and Apparate adapts exits with accuracy feedback [18]. LoopDuet’s controller adjusts prefilling work using queue age within an offline-characterized quality–capacity range. Conditional computation under batching. MoE architectures select subsets of FFN computation per token [57, 23, 38, 17, 73]. FastMoE and MegaBlocks optimize sparse training; Tutel and DeepSpeed-MoE optimize expert execution and parallelism [30, 24, 34, 53]. Shared weights also create batching opportunities in adapter serving, as developed in LoRA, S-LoRA, and Punica [32, 58, 9]. These systems motivate considering aggregate executed work. Section 6 develops the connection to MoE serving.

6.

Discussion

Recurrence as a third axis of conditional computation. LoopDuet’s phase-dependent depth connects naturally to MoE: physical layer, expert identity, and recurrent iteration form three axes of computation allocation, a 3D MoE abstraction. Conventional MoE routes tokens among experts within a layer, providing the horizontal axis [57, 23]. MoD and layer-skipping methods adapt computation across layers, providing the vertical axis [54, 21]. Recurrence adds a third axis by revisiting shared blocks: states evolve across iterations while parameters remain tied. MoE principles extend to recurrent computation. Several developments in looped models can be understood through MoE principles. Sparse experts expand capacity: routing within shared blocks increases

13

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

their representational capacity, as explored by the Sparse Universal Transformer [60], MoEUT [16], and scaling studies of looped MoE models under matched compute, parameter, and KV budgets [65]. Sparse work requires efficient batching: MoE systems group tokens assigned to the same expert, while continuous depth batching groups tokens at different recurrence steps for execution through a shared core [4, 56]. Prediction enables work to begin earlier: expert prefetching has an analogue in wavefront decoding, which uses an intermediate recurrent state to draft the next token [29]. These connections motivate extending further MoE serving techniques to recurrent models. Phase and batching determine the benefit of sparsity. Prefilling and decoding have different bottlenecks, motivating phase-specific serving policies [74, 51]. For recurrent models, this distinction also determines how much depth each phase should receive. MoE serving further shows that per-token sparsity must reduce aggregate executed work to produce system-level savings. Lynx consolidates low-affinity token-to-expert assignments onto a smaller batch-wide expert set to reduce memory traffic [28], while DeepSpeed-MoE coordinates expert parallelism with inference kernels [53]. Our neuron-union measurements reveal the same aggregation effect within looped blocks: per-token selections cover nearly every neuron across a batch. A common prefilling-depth boundary instead removes later core applications from ordinary prefilling positions. Lynx favors expert reassignment during decoding, while our measurements favor recurrence reduction during prefilling. Joint scheduling of residency and recurrence. MoE serving coordinates computation with parameter movement. SiDA-MoE predicts expert activation ahead of execution and overlaps prediction with inference to manage GPU/host residency [20]. MoE-Infinity uses request-level activation traces for expert caching and prefetching in memory-constrained, single-user serving [69]. Both schedule data movement around active experts, while depth control determines how often a resident block is reused. These decisions interact in a recurrent MoE: loop depth can change routing locality and prefetch opportunities, while expert residency affects the cost of each additional iteration. This coupling motivates joint scheduling of expert residency and per-phase recurrent depth.

7.

Conclusion

LoopDuet jointly selects prefilling and decoding depth to expand serving choices for looped language models. For Huginn, LoopDuet delivers 19% more throughput than the fastest tested uniform configuration meeting our accuracy criterion, and 6.04 more EM points than uniform depth eight at 97% of its throughput. For OLMo HotpotQA, the accuracy–throughput comparison favors uniform depth four; adapting its prefilling budget retains 0.92–1.35 more EM points than the most accurate tested fixed settings achieving the same one-second TTFT attainment on the evaluated 60 requests/s and 10/60 burst traces. Role-separated measurements and batch-compatible execution connect these depth decisions to concrete quality–throughput trade-off.

14

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

AI Usage Disclosure OpenAI Codex and Anthropic Claude Code were used as AI-assisted tools throughout the research and manuscript preparation process. Their contributions included implementing experimental code according to the authors’ instructions, executing experiments, organizing and refactoring code, inspecting experimental artifacts, and assisting with debugging and validation. These tools were also used to support literature searches, identify relevant prior work, and verify bibliographic references. During manuscript preparation, AI assistance was used to draft and revise text, improve clarity and presentation, generate figures and supporting validation scripts, and check the consistency of reported statistics against experimental results. All research ideas, scientific hypotheses, methodological decisions, and experimental designs originated from the authors. The AI tools primarily served to implement the authors’ instructions and assist with technical execution and documentation. The authors reviewed and verified the core implementations, experimental results, and scientific claims, and retained full responsibility for the accuracy, originality, and integrity of the work.

References [1] A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, 2023. URL https://arxiv.org/abs/2308.16369. [2] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, Santa Clara, CA, July 2024. USENIX Association. ISBN 978-1-939133-40-3. URL https://www.usenix.org/conference/osdi 24/presentation/agrawal. [3] S. Bae, J. Ko, H. Song, and S.-Y. Yun. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=eeP1y7zPQ7. [4] S. Bae, Y. Kim, R. Bayat, S. Kim, J. Ha, T. Schuster, A. Fisch, H. Harutyunyan, Z. Ji, A. Courville, and S.-Y. Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=QuqsEIVWIG. [5] S. Bai, J. Z. Kolter, and V. Koltun. Deep equilibrium models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files /paper/2019/file/01386bd6d8e091c2ab4c7c7de644d37b-Paper.pdf. [6] A. Banino, J. Balaguer, and C. Blundell. Pondernet: Learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021. URL https://openreview.net/forum?id=1EuxRTe0WN. [7] T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=PEpbUobfJv. [8] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper. Accelerating large language model decoding with speculative sampling, 2023. URL https://arxiv.org/abs/2302.01318.

15

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

[9] L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy. Punica: Multi-tenant lora serving. In P. Gibbons, G. Pekhimenko, and C. D. Sa, editors, Proceedings of Machine Learning and Systems, volume 6, pages 1–13, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/file/05 4de805fcceb78a201f5e9d53c85908-Paper-Conference.pdf. [10] Y. Chen, X. Pan, Y. Li, B. Ding, and J. Zhou. EE-LLM: Large-scale training and inference of early-exit large language models with 3d parallelism. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=xFk0w9zoV3. [11] Y. Chen, S. Zhang, N. Wang, F. Kang, and R. Jia. Hybrid latent attention for looped language models, 2026. URL https://arxiv.org/abs/2610.07940. [12] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/ 1803.05457. [13] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. [14] L. D. Corro, A. D. Giorno, S. Agarwal, B. Yu, A. Awadallah, and S. Mukherjee. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference, 2023. URL https://arxiv.org/ab s/2307.02628. [15] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica. Clipper: A LowLatency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 613–627, Boston, MA, Mar. 2017. USENIX Association. ISBN 978-1-931971-37-9. URL https://www.usenix.org/conference/nsdi17/technical-sessions/ presentation/crankshaw. [16] R. Csordás, K. Irie, J. Schmidhuber, C. Potts, and C. D. Manning. Moeut: Mixture-of-experts universal transformers. Advances in Neural Information Processing Systems, 37:28589–28614, 2024. [17] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.70. URL https://aclanthology.org/2024.acl-long.70/. [18] Y. Dai, R. Pan, A. Iyer, K. Li, and R. Netravali. Apparate: Rethinking early exits to tame latency-throughput tensions in ml serving. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 607–623, 2024. [19] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HyzdRi R9Y7. [20] Z. Du, S. Li, Y. Wu, X. Jiang, J. Sun, Q. Zheng, Y. Wu, A. Li, H. H. Li, and Y. Chen. Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models. In P. Gibbons, G. Pekhimenko, 16

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

and C. D. Sa, editors, Proceedings of Machine Learning and Systems, volume 6, pages 224–238, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/file/698cfaf72a208aef 2e78bcac55b74328-Paper-Conference.pdf. [21] M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, A. Aly, B. Chen, and C.-J. Wu. LayerSkip: Enabling early exit inference and self-speculative decoding. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622–12642, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.681. URL https://aclanthology.org/2024.acl-long.681/. [22] A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum ?id=SylO2yStDr. [23] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. URL http://jmlr.org/papers/v23/21-0998.html. [24] T. Gale, D. Narayanan, C. Young, and M. Zaharia. Megablocks: Efficient sparse training with mixtureof-experts. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems, volume 5, pages 288–304. Curan, 2023. URL https://proceedings.mlsys.org/paper_files/pa per/2023/file/5a54f79333768effe7e8927bcccffe40-Paper-mlsys2023.pdf. [25] J. Geiping, S. M. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: //openreview.net/forum?id=S3GhJooWIC. [26] A. Graves. Adaptive computation time for recurrent neural networks, 2017. URL https://arxiv.org/ abs/1603.08983. [27] A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace. Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 443–462. USENIX Association, Nov. 2020. ISBN 9781-939133-19-9. URL https://www.usenix.org/conference/osdi20/presentation/gujarati. [28] V. Gupta, J. H. Ju, K. Sinha, A. Gavrilovska, and A. P. Iyer. Lynx: Enabling efficient moe inference through dynamic batch-aware expert selection, 2026. URL https://arxiv.org/abs/2411.08982. [29] H. Ha and J.-J. Kim. Wavefront decoding: Parallelized self-speculative decoding for looped language models, 2026. URL https://arxiv.org/abs/2609.23033. [30] J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, and J. Tang. Fastmoe: A fast mixture-of-expert training system, 2021. URL https://arxiv.org/abs/2103.13262. [31] J. He, Z. Fu, J. Wang, and Q. Li. Pop: Prefill-only pruning for efficient large model inference, 2026. URL https://arxiv.org/abs/2602.03295.

17

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

[32] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. [33] B. Huang, C. Shi, J. Chen, S. Wen, Z. Liu, E. Xing, and X. Ma. Towards looped models done right, part ii: Rethinking at fixed points, 2026. URL https://arxiv.org/abs/2610.06833. [34] C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, H. Chau, P. Cheng, F. Yang, M. Yang, and Y. Xiong. Tutel: Adaptive mixture-of-experts at scale. In D. Song, M. Carbin, and T. Chen, editors, Proceedings of Machine Learning and Systems, volume 5, pages 269–287. Curan, 2023. URL https://proceedings.mlsys.org/paper_files/paper/2023/file/5616d34cf8ff7394 2cfd5aa922842556-Paper-mlsys2023.pdf. [35] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. [36] F. Ladhak, S. Acharya, S. A. Serano, S. Sun, B. Ginsburg, and K. C. Puvvada. One model to loop them all: Elastic latent recurrence for language models, 2026. URL https://arxiv.org/abs/2610.12582. [37] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1eA7AEtvS. [38] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb. [39] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v2 02/leviathan23a.html. [40] Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.422. URL https://aclanthology.org/2024.emnlp-main.422/. [41] Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/f orum?id=1NdN7eXyb4. [42] Z. Li, L. Zheng, Y. Zhong, V. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663–679, Boston, MA, July 2023. USENIX Association. ISBN 978-1-939133-34-2. URL https: //www.usenix.org/conference/osdi23/presentation/li-zhouhan.

18

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

[43] C. Lin, Z. Han, C. Zhang, Y. Yang, F. Yang, C. Chen, and L. Qiu. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 929–945, Santa Clara, CA, July 2024. USENIX Association. ISBN 978-1-939133-40-3. URL https://www.usenix.org/conference/osdi24/presentation/lin-c haofan. [44] L. Liu, M. Wang, and T. Zhao. Decode-branch transformers: Decoupling the primary prefill path from additional decode computation, 2026. URL https://arxiv.org/abs/2608.12385. [45] W. Liu, P. Zhou, Z. Wang, Z. Zhao, H. Deng, and Q. Ju. FastBERT: a self-distilling BERT with adaptive inference time. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6035–6044, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.537. URL https://aclanthology.org/2020.acl-main.537/. [46] S. M. McLeish, A. Li, J. Kirchenbauer, D. S. Kalra, B. R. Bartoldson, B. Kailkhura, A. Schwarzschild, J. Geiping, T. Goldstein, and M. Goldblum. Teaching pretrained language models to think deeper with retrofitted recurrence. In Third Conference on Language Modeling, 2026. URL https://openreview.n et/forum?id=PXVQTHYwgt. [47] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang, X. Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932–949, 2024. [48] A. Mishra, A. P. Masdemont, V. C. Vendrell, J. R. Giralt, A. Behboodi, and F. V. Massoli. Enabling dynamic computation in looped lms, 2026. URL https://arxiv.org/abs/2610.09013. [49] G. Monea, K. Ramji, Y. El-Kurdi, L. A. Lastras, Y. Artzi, N. Godey, and R. F. Astudillo. The surprising effectiveness of shared memory in looped transformers, 2026. URL https://arxiv.org/abs/2610.0 2383. [50] J. Oh, H. Jeon, H. Ji, K. Kong, and J.-Y. Lee. Shallow prefill, deep decoding: Efficient long-context inference via layer-asymmetric kv visibility, 2026. URL https://arxiv.org/abs/2605.06105. [51] P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini. Splitwise: Efficient Generative LLM Inference Using Phase Splitting . In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132, Los Alamitos, CA, USA, July 2024. IEEE Computer Society. doi: 10.1109/ISCA59077.2024.00019. URL https://doi.ieeecomputersociety.org/10 .1109/ISCA59077.2024.00019. [52] A. Qiao, Z. Yao, S. Rajbhandari, and Y. He. SwiftKV: Fast prefill-optimized inference with knowledgepreserving model transformation. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25734– 25753, Suzhou, China, Nov. 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1306. URL https://aclanthology.org/2025.emnlp-main.1306/. [53] S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. DeepSpeedMoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In 19

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html. [54] D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024. URL https://arxiv.or g/abs/2404.02258. [55] T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Q. Tran, Y. Tay, and D. Metzler. Confident adaptive language modeling. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=uLYc4L3C81A. [56] K. Schwethelm, D. Rueckert, and G. Kaissis. Depth-adaptive inference of looped language models via continuous depth batching, 2026. URL https://arxiv.org/abs/2608.09444. [57] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1ckMDqlg. [58] Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica. Slora: Scalable serving of thousands of lora adapters. In P. Gibbons, G. Pekhimenko, and C. D. Sa, editors, Proceedings of Machine Learning and Systems, volume 6, pages 296–311, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/file/906419cd502575b617cc 489a1a696a67-Paper-Conference.pdf. [59] Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei. You only cache once: Decoder-decoder architectures for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=25Ioxw576r. [60] S. Tan, Y. Shen, Z. Chen, A. Courville, and C. Gan. Sparse universal transformer. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 169–179, 2023. [61] S. Teerapittayanon, B. McDanel, and H.-T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pages 2464–2469. IEEE, 2016. [62] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee24 3547dee91fbd053c1c4a845aa-Paper.pdf. [63] E. P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi, N. Dziri, A. Ettinger, M. Guerquin, D. Heineman, H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, J. Morrison, T. Murray, C. Nam, J. Poznanski, V. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg, D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi. 2 OLMo 2 furious (COLM’s version). In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=2ezugTT9kU. 20

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

[64] K. Wan and C. Chen. Ramp: Recurrent adaptive mixed precision for efficient prefill in recurrent-depth language models, 2026. URL https://arxiv.org/abs/2610.13041. [65] S. Wang, G. Zhang, K. Luo, Y. Wu, S. Liu, J. Liu, W. Huang, S. Yan, and J. Li. Smelt: Scaling laws for compute-matched moe looped transformers. arXiv preprint arXiv:2609.01343, 2026. [66] X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018. [67] B. Wu, Y. Zhong, Z. Zhang, S. Liu, F. Liu, Y. Sun, G. Huang, X. Liu, and X. Jin. Fast distributed inference serving for large language models, 2024. URL https://arxiv.org/abs/2305.05920. [68] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.204. URL https: //aclanthology.org/2020.acl-main.204/. [69] L. Xue, Y. Fu, Z. Lu, L. Mai, and M. Marina. Moe-infinity: Efficient moe inference on personal machines with sparsity-aware expert cache, 2025. URL https://arxiv.org/abs/2401.14361. [70] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259/. [71] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-1. URL https://www.usenix.org/conference/osdi22/presentation/yu. [72] J. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11263–11282, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.607. URL https://aclanthology.org/2024.acl-long.607/. [73] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou. MoEfication: Transformer feed-forward layers are mixtures of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 877–890, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.71. URL https: //aclanthology.org/2022.findings-acl.71/. [74] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210, Santa Clara, CA, July 2024. USENIX Association. ISBN 978-1-939133-40-3. URL https://www.usenix.org/conference/osdi24/pre sentation/zhong-yinmin. 21

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

[75] W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei. Bert loses patience: Fast and robust inference with early exit. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18330–18341. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/d4dd111a4fd97339423 8aca5c05bebe3-Paper.pdf.

22

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

A.

Appendix Overview

This appendix develops the execution details behind LoopDuet, then examines how task, context, batching, and load affect depth allocation. Recurrent-state and FFN-routing studies explain the choice of token roles and execution granularity. Section I provides the complete numerical comparisons.

B.

Workload and Implementation Details

Notation and metrics. We write (𝑆, 𝑃) for decoding and prefilling core iterations, with 𝑃 ≤ 𝑆. We report task accuracy, including exact match (EM) and token F1 for HotpotQA; accuracy differences are percentage points. Joint-depth comparisons use (32, 32) as the full-depth baseline. Prefilling-depth ablations compare (𝑆, 𝑃) with (𝑆, 𝑆).† Workloads. GSM8K uses all 1,319 test examples, five demonstrations, greedy decoding, and extraction of the final numeric answer. ARC-Challenge (ARC) uses all 1,172 test examples, five demonstrations, and the highest-scoring choice label at the first answer position. HotpotQA uses all 7,405 distractor-development examples with normalized EM and token F1. HotpotQA prompt construction is fixed within each model’s comparisons. OLMo uses one demonstration and four paragraphs: the two supporting passages and two distractors selected in their original order. Distractors are removed as needed to satisfy the 1,000-token prompt cap. Huginn uses two demonstrations and ten paragraphs. These inputs supply the supporting passages directly. Models and execution. OLMo has 4/6/4 prelude/core/coda layers at width 2,048; Huginn has 2/4/2 at width 5,280. Serving uses vLLM 0.14.1 on one A100-SXM4-80GB with bf16, an 85% GPU-memory budget, and prefix caching disabled. Uniform and split depths both use shared KV caches, zero recurrent-state initialization, eager prefilling batches, and full CUDA graphs for decoding. Zero initialization gives each serving token a fixed initial recurrent state. A separate accuracy harness supports token-role and cache ablations. It uses the checkpoint’s random-state initialization and either shared or depth-indexed KV caches. In a depth-indexed cache, skipped entries inherit the last computed recurrence at the same physical core layer. Comparisons use the same implementation and cache layout for both configurations; serving accuracy and performance come from the same vLLM runs. Parallel tokens, sequential chunks and loops. Each core application processes the token positions in a scheduled prompt chunk together with causal attention. Successive loops update their recurrent states, and successive chunks extend the cached context. The final prompt position produces the first output token. Decoding then advances one generated position per request at a time. Algorithm 1 implements LoopDuet’s compact execution. The first 𝑃 loops process all 𝑁 scheduled tokens. The remaining 𝑆 − 𝑃 loops operate on a gathered 𝐵 × 𝑑 tensor containing each request’s final scheduled query, where 𝐵 is the request count and 𝑑 the hidden width. Every prompt-chunk endpoint continues to depth 𝑆, including intermediate endpoints, as do decoding positions. The final prompt endpoint produces the first output token. † Scores, differences, and ratios are computed from unrounded values and rounded independently for display; subtracting or dividing displayed values can therefore differ in the last digit.

23

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Algorithm 1 LoopDuet execution in one engine step Require: Scheduled tokens 𝑋, positions, KV cache; 𝑆, 𝑃 1: 𝐸 ← Prelude(𝑋); initialize recurrent state 𝐻 2: if batch has multiple-query prefilling work and 𝑃 < 𝑆 then 3: for 𝑘 = 1 to 𝑃 do 4: 𝐻 ← Core(𝐻, 𝐸, KV) 5: end for 6: 𝐼 ← last query position of each scheduled request b ← Gather(𝐻, 𝐼); 𝐸 b ← 𝐸𝐼 7: 𝐻 b 8: 𝑀 ← positions and attention metadata for 𝐼 9: for 𝑘 = 𝑃 + 1 to 𝑆 do b ← Core( 𝐻, b 𝐸, b KV; 𝑀) b 10: 𝐻 11: end for b 12: 𝐻 ← Scatter(𝐻, 𝐼, 𝐻) 13: else 14: for 𝑘 = 1 to 𝑆 do 15: 𝐻 ← Core(𝐻, 𝐸, KV) 16: end for 17: end if 18: return Coda(𝐻)

Prelude states, positions, and attention metadata follow the gathered indices. Scattering restores the updated rows before the coda; other prompt positions retain their loop-𝑃 context in one shared KV plane per core layer. Batches whose maximum query length is one execute all 𝑆 loops. Arrival and control semantics. Open-loop requests are timestamped by scheduled arrival and admitted before engine steps, so TTFT includes admission delay and queueing. Alternating-rate traces use seeded exponential interarrivals with a new rate every 30 seconds. Closed-loop runs replenish completed requests to maintain at most 𝐶 active requests. In open-loop runs, 𝐶 = 128 limits the number of sequences scheduled per engine step. Requests enter the engine when due and can wait in its queue beyond this scheduling cap. LoopDuet’s queue controller observes the oldest first-token waiting age, including prefilling requests already in service. It maps ages from 0.25 to 1 second linearly onto the configured depth range, then rounds to an integer. The main configuration fixes 𝑆 = 4 and selects 𝑃 ∈ {2, 3, 4} before each engine step. A request spanning several chunks can encounter different budgets. The supplementary rate controller estimates demand from arrivals over a trailing window of at most five seconds, plus the number waiting divided by a two-second drain interval. It selects the deepest configured depth with sufficient measured capacity. Controller occupancy measures the fraction of wall time spent at each prefilling depth (Table A10). A request’s executed depth follows the policy in effect when each prompt chunk is scheduled. Timing measurements. We measure end-to-end serving time, excluding model loading, an eight-request warm-up, and diagnostic calibration. Latency percentiles summarize requests within each run. The joint-depth comparison uses one execution per configuration at 𝐶 = 64 and an 8,192-token engine-step budget. Load and batching studies vary the arrival trace, concurrency cap, or token budget.

24

Exact match (%) ↑

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

38.6

(32, 32) (8, 8)

38.4

(11, 4)

38.2 Uniform (𝑆, 𝑆)

38

LoopDuet (𝑆, 𝑃)

(4, 4)

37.8 5

10

15

20

25

30

35

40

45

50

55

60

65

Throughput (req/s) ↑

Fig. A1: Uniform depth four supplies OLMo’s starting budget. Uniform (4, 4) has the highest throughput among the six HotpotQA settings, all of which satisfy the quality criterion relative to (32, 32). Table A6 reports F1 and EM changes.

Depth updates and execution modes. Prefilling depth can be set at startup or changed in a running engine. At fixed (𝑆, 𝑃) = (11, 4) and 40 requests/s, a runtime update in a reused engine yields EM 38.28 and median TTFT 0.15 seconds; startup configuration in a separate process yields EM 38.20 and 0.17 seconds. Both use 𝐶 = 128 with the same prompts, arrival seed, cache, and precision. Eager prefilling with full decoding CUDA graphs is faster than compiled execution for both uniform-depth configurations: 304.7 versus 432.6 seconds for OLMo (11, 11), and 5,639.3 versus 6,877.2 seconds for Huginn (24, 24). Uniform-depth and LoopDuet comparisons use this faster execution path.

C.

Joint Depth Selection and Context Sensitivity

Joint depth selection exposes different allocations for the two models. Across 11 Huginn configurations, LoopDuet’s split budgets improve the accuracy–throughput choices highlighted in the main paper. Across six OLMo configurations, uniform (4, 4) provides the highest throughput: 56.46 requests/s at EM 37.95, 0.54 points below (32, 32) (Figure A1). This setting supplies the starting budget for OLMo’s controller. Selection uses the closed-loop quality–throughput comparison; decoding depth is then fixed across the open-loop arrival traces. The useful prefilling depth also varies by task. In the accuracy harness, OLMo ARC (11, 4) and (32, 4) change accuracy by −1.54 and −1.88 points relative to their same-𝑆 controls. Increasing 𝑃 to eight changes these deltas to +0.85 and 0.00. On Huginn HotpotQA at 𝑆 = 32, 𝑃 = 8 changes EM by −2.63 points, whereas 𝑃 = 16 changes it by −0.23. Huginn ARC remains more sensitive: (32, 16) and (14, 8) change accuracy by −1.45 and −4.44 points. Depth allocation therefore depends on the task as well as the checkpoint. Context length changes the accuracy cost even when runtime gains remain similar. Relative to (24, 24), Huginn (24, 12) achieves approximately 1.85× speedup with two, four, and ten paragraphs, while EM changes by −0.43, −0.45, and −1.92 points. OLMo (11, 4) with two paragraphs achieves 2.11× speedup and a −1.35-point EM change relative to (11, 11). Depth selection therefore uses task accuracy for each prompt construction. The state-distance thresholds in Section G are calibrated on the default prompts.

D.

Serving Gains across Batch Budgets

Layer-pass savings translate into shorter serving times. On HotpotQA, reducing OLMo’s prefilling depth from 11 to four at fixed 𝑆 = 11 lowers wall time from 304.7 to 142.8 seconds and median TPOT from 0.420 to 0.196 seconds (Table A7). The long-prompt limits 𝑐 𝑆 /𝑐 𝑃 are 3.57, 2.31, and 1.94 for OLMo (32, 8), OLMo (11, 4), and Huginn (32, 16), respectively. A finite-workload estimate also accounts for the measured mean prompt length 𝐿¯ and 25

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

¯ Let 𝑎, 𝑏, 𝑐 count prelude, core, and coda layers, so 𝑐 𝑘 = 𝑎 + 𝑏𝑘 + 𝑐. The unchunked generated-output count 𝑂. layer-pass estimate is ¯ 𝑆. b (𝑆, 𝑃) = ( 𝐿¯ − 1)𝑐 𝑃 + 𝑂𝑐 𝑊 The 𝑂¯ full-depth evaluations comprise the final prompt position and the 𝑂¯ − 1 subsequent decoding positions. Dividing the uniform estimate by the LoopDuet estimate gives 3.52, 2.29, and 1.94; measured speedups are 85–96% of these values. Measured runtime also includes work at additional chunk endpoints and non-layer costs. The corresponding GSM8K estimates and timings appear in Table A8. Batch budgets change grouping while preserving the savings. Huginn’s prefilling savings persist across batch sizes. At 𝐶 = 128, (32, 16) reaches 1.84×, 1.87×, and 1.88× speedup over (32, 32) with engine-step token budgets of 4,096, 8,192, and 16,384. At the default 8,192-token budget, speedup remains 1.87× for 𝐶 = 32, 64, 128. For these long prompts, the token budget limits admitted prefilling work before the request-concurrency cap. Table A9 gives the engine-step counts and latencies for the three token budgets. The open-loop Huginn runs at four and eight requests/s are overload stress tests: arrivals exceed the measured service capacities, so TTFT includes accumulated queueing. At Poisson four requests/s, the 4,096and 16,384-token comparisons reach 1.84× and 1.88× speedup, with EM changes of +0.41 and −0.35 points. Huginn (24, 12) versus (24, 24) reaches approximately 1.85× speedup at both four and eight requests/s, with EM changes of −1.82 and −1.78 points. Thus the runtime benefit persists under queueing, while the selected depth determines the associated accuracy cost.

E.

Fixed-Depth Capacity and Queueing

Reducing prefilling work can prevent backlog from accumulating at an otherwise unsustainable arrival rate. For OLMo at 𝑆 = 11 and Poisson 40 requests/s, reducing 𝑃 from 11 to four lowers median TTFT from 57.7 to 0.17 seconds and raises one-second attainment from 0.7% to 100%. EM changes from 38.34 to 38.20. Crossing the required service capacity produces a larger latency improvement than the execution-time ratio alone. At 80 requests/s, both configurations accumulate backlog, but 𝑃 = 4 lowers median TTFT from 104.8 to 24.5 seconds. LoopDuet therefore reduces service work and expands the range of arrival rates supported within a latency target.

F.

Quality Retention under Changing Load

Adaptation preserves depth when capacity permits. Starting from OLMo (4, 4), the controller fixes 𝑆 = 4 and adjusts 𝑃 ∈ {2, 3, 4}. At Poisson 60 requests/s and on the 10/60 burst trace, it attains one-second TTFT for every request at EM 37.27 and 37.77. These scores exceed fixed 𝑃 = 3, the most accurate tested fixed depth with the same attainment, by 0.92 and 1.35 points. At 40 requests/s, available capacity permits 𝑃 = 4 throughout, so the queue controller meets the latency target while retaining the starting depth. Fixed 𝑃 = 4 also attains 100%; the two policies yield EM 37.92 and 37.96, respectively.‡ Higher load makes the quality–latency trade-off explicit. On 20/80 bursts, adaptation gives EM 35.12 at 87.5% attainment, compared with EM 31.91 at 100% for fixed 𝑃 = 2. Retaining deeper computation gains 3.21 EM points while reducing target attainment by 12.5 percentage points. ‡ The policies are executed independently. Fixed 𝑃 = 4 uses 7,652 engine steps with median TTFT 0.087 seconds; the queue policy

uses 5,938 steps with median TTFT 0.112 seconds.

26

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Table A1: Lower 𝑃min increases capacity under heavy load. Rate-based control at 𝑆 = 11, 𝑃max = 11; rates are requests/s, and TTFT columns report one-second attainment.

Arrival trace

𝑃min = 2 TTFT (%) EM

𝑃min = 4 TTFT (%) EM

Poisson 20 Poisson 40 Poisson 60 Burst 10/60 Burst 20/80

100 100 100 99.9 44.6

100 100 4.1 23.7 15.0

38.74 38.30 33.25 34.23 34.33

38.70 38.27 38.11 38.10 38.28

The depth range determines the available response. At 𝑆 = 11, the queue policies use (𝑃min , 𝑃max ) = (4, 11), (2, 11), or (2, 4); the rate policies use (2, 11) or (4, 11). Each variant retains its parameters across its evaluated traces. At 40 requests/s, allowing the queue controller to return to 𝑃max = 11 gives 63.3% attainment with 𝑃min = 4 and 92.3% with 𝑃min = 2. Setting (𝑃min , 𝑃max ) = (2, 4) gives 100% at EM 38.35; fixed (11, 4) also gives 100%, at EM 38.20. This load can support deeper decoding than the main 𝑆 = 4 configuration. The upper prefilling budget therefore affects queueing as well as the amount of computation retained. The rate controller illustrates the role of the lower budget (Table A1). Both 𝑃min settings meet the target at 20 and 40 requests/s; static full depth at 20 requests/s gives 99.96% attainment and EM 38.41. At 60 requests/s, 𝑃min = 2 provides enough capacity for 100% attainment at EM 33.25, while 𝑃min = 4 retains EM 38.11 with 4.1% attainment. Burst arrivals expose the same trade-off between service capacity and quality. On 20/80 bursts, the queue controller with 𝑃 ∈ [2, 4] retains EM 34.27 at 34.1% attainment, while fixed 𝑃 = 2 reaches 72.2% attainment at EM 32.25. At 80 and 120 requests/s, offered load exceeds every tested policy’s measured capacity at its shallowest depth. The queue variants with 𝑃max = 11 spend 97.6–98.8% of wall time at 𝑃min (Table A10), identifying loads that require additional capacity beyond the available depth reduction.

G.

Recurrent-State Diagnostics and Task Sensitivity

Replaying recorded answers exposes intermediate states. We first generate answers at each checkpoint’s reference depth. We then run the model on the original prompts and recorded answers, keeping the answer tokens fixed and recording hidden states at every loop. Reference depths are 32 for OLMo r32 and Huginn, and 16 for OLMo r16. During replay, next-token predictions match the recorded answer tokens at a rate of 95.1–99.6% across the nine checkpoint–task combinations. For token 𝑖, the normalized distance between loops 𝑃 and 𝑆 is 𝑑𝑖 (𝑃; 𝑆) = ∥ℎ𝑖𝑃 − ℎ𝑖𝑆 ∥ 2 /∥ℎ𝑖𝑆 ∥ 2 . One replay provides the intermediate states for every candidate loop pair. For example, OLMo GSM8K gives 90th-percentile prefilling distances 𝑞 .9 (4; 11) = 0.180443 and 𝑞 .9 (4; 32) = 0.182533. Huginn’s states continue changing through loop 32, so its distances describe deviation from this finite reference depth. Role sensitivity extends beyond settling order. At median distance 0.1, Huginn HotpotQA crosses at loop 15 for ordinary prefilling tokens and loop 19 for decoding tokens. On GSM8K, the crossings are 20 and 19. Earlier prefilling-state settling is therefore task dependent. Quality-based thresholds show a larger role difference for mean and upper-tail distances than for the median.

27

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Table A2: Task sensitivity varies at similar state-distance scales. The upper rows have small distances with accuracy losses above one point; the lower rows retain accuracy within one point at larger distances. Distances are prefilling-token 90th percentiles; changes use same-𝑆 controls. Model / task

(𝑆, 𝑃)

Distance

Δ acc.

Huginn / ARC OLMo r32 / ARC OLMo r32 / ARC OLMo r32 / GSM8K

(32, 16) (11, 4) (32, 4) (32, 8)

0.17201 0.11403 0.11518 0.03653

−1.45 −1.54 −1.88 −1.29

Huginn / GSM8K Huginn / GSM8K Huginn / HotpotQA

(32, 8) (14, 8) (14, 8)

0.62001 0.61445 0.46377

−0.91 +0.68 −0.72

Table A3: The fitted threshold remains near 0.24 when each checkpoint is held out in turn. Each fit uses the other two checkpoints. Held-out checkpoint

Threshold

Agreement

OLMo r32 OLMo r16 Huginn

0.240964 0.240964 0.236799

18/21 6/6 13/17

Combined

37/44

The fitted prefilling-to-decoding threshold ratios are 2.22 for the 90th percentile, 2.15 for the mean, and 1.11 for the median, using 44 prefilling-depth configurations and 28 uniform-depth configurations. These fits connect state changes to the quality impact of each depth intervention. Small state changes can affect tasks differently. On ARC, state distances below 0.18 accompany accuracy losses of 1.45–1.88 points (Table A2). OLMo GSM8K (32, 8) likewise loses 1.29 points at distance 0.037, while (32, 4) has zero measured accuracy change at distance 0.183. Conversely, Huginn GSM8K (14, 8) gains 0.68 points at distance 0.614. These observations suggest that the task relevance of a state change matters alongside its magnitude, motivating task-specific depth selection. Threshold transfer and aggregation. The diagnostic evaluation spans eight model–task groups across three checkpoints from two model families. Depth pairs reuse states from the same replay. The pooled 90th-percentile threshold is 0.240964; fitting on two checkpoints and evaluating the third gives thresholds from 0.236799 to 0.240964 (Table A3). We propose depths below the fitted threshold and evaluate their task accuracy. For the binary comparisons in Tables A3–A5 and the complete diagnostic table, the quality criterion combines an accuracy loss of at most one percentage point with a non-significant two-sided exact McNemar comparison at the five-percent level. This paired test compares which examples change correctness against the same-𝑆 control. Agreement is the fraction of configurations for which the distance rule and this task-based criterion give the same decision. The 90th-percentile rule retains its agreement on held-out checkpoints, whereas mean and median aggregation give lower agreement after transfer (Table A4). All three aggregations match the per-model depth lookup in the pooled comparison. The shared state-distance criterion supplies candidate budgets for task-accuracy validation. Calibration can also cover both prefilling-depth and uniform-depth reductions with a common threshold.

28

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Table A4: Upper-tail aggregation retains its agreement across checkpoints. Agreement is measured against the task-accuracy criterion. The first five rules use 44 prefilling-depth configurations; the joint rule adds 28 uniform-depth configurations. Held-out results combine the three checkpoint folds; dashes mark settings without a held-out evaluation. Selection rule

Pooled

Held-out

90th-percentile distance Mean distance Median distance Per-model depth lookup Decoding-derived rule Joint 90th-percentile rule

37/44 37/44 37/44 37/44 35/44 60/72

37/44 36/44 34/44 — — —

Table A5: State distance and task sensitivity jointly inform depth selection. Counts group configurations by the pooled distance threshold and the measured quality criterion. Table A2 shows the task-level accuracy effects.

State distance 𝑞 .9 ≤ 0.240964 𝑞 .9 > 0.240964

Quality criterion Satisfied Not satisfied 18 3

4 19

A rule derived from decoding-token distances provides a parameter-free alternative, with lower agreement on prefilling-depth choices. Table A4 compares these calibration choices. Prompt composition shapes the aggregate distance. Demonstrations account for 90.28% and 91.10% of ordinary prefilling tokens in OLMo r32 and Huginn GSM8K, and approximately 82% on ARC. Aggregate distances therefore largely reflect the demonstration portion of these few-shot prompts.

H.

FFN Routing and Batch Unions

Batch savings from per-token sparsity depend on overlap among selected neurons. To characterize this overlap and its quality cost, we select late-loop FFN neurons from earlier-loop activations and apply masks before the down projection. The measurements cover task accuracy and neuron coverage under dense execution. At 50% retention, accuracy changes by −0.14 points on OLMo HotpotQA, −0.34 on OLMo ARC, and +0.09 on Huginn ARC. GSM8K changes by −2.50 for OLMo and −2.73 for Huginn; Huginn HotpotQA changes by −0.86. A separate frozen-routing comparison uses OLMo r32 at 𝑆 = 11 on all 1,319 GSM8K examples: the first four loops are dense, and later loops retain a fixed per-token subset selected from loop-four activations. Retaining 25% gives 34.42% accuracy versus 47.61% for its dense control, a 13.19-point decrease (Table A11). Static and grouped routes yield losses of 1.8–20.6 points. The quality cost thus depends on the task, retained fraction, and routing granularity. At batch size 16, the union of 50% per-token selections covers 99.80–99.95% of neurons (Table A15). Nearly every neuron remains active for some token in the batch. LoopDuet instead removes later core applications from ordinary prefilling positions at a common boundary, making the reduction explicit in the dense batched execution.

29

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

I.

Complete Comparison Tables

The tables fixed-𝑆 HotpotQA and GSM8K serving comparisons (Tables A7 and A8), batch-budget and controlleroccupancy measurements (Tables A9 and A10), and frozen FFN-retention comparisons (Table A11). The complete serving, accuracy-harness, and state-distance tables follow (Tables A12–A14), with neuron-union coverage in Table A15. Run identifiers encode task (hp, arc, or gsm; no task prefix in accuracy runs denotes GSM8K), depths (s, p), shared KV (kvsh), a shallow final prompt token (ls), and FFN routing (sf). C is the sequence cap, T is the engine-step token budget, and pois and burst identify arrival traces. Open-loop runs use 𝐶 = 128 even when the suffix is omitted; the default token budget is 8,192. hp2 and hp4 indicate two- and four-paragraph context variants. For 𝑆 = 11, ctrl2, ctrl4, and ctrl42 denote queue ranges (𝑃min , 𝑃max ) = (2, 11), (4, 11), (2, 4); rate2 and rate4 use the corresponding lower bounds with 𝑃max = 11.

30

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

Table A7: Full HotpotQA development set, 𝐶 = 64, one A100-80GB. Each uniform-depth baseline uses (𝑆, 𝑆) with the same model, prompt format, cache, and engine path. Fraction is measured speedup divided by the analytic layer-pass ratio, rather than hardware utilization.

Model / HotpotQA context (𝑆, 𝑃) OLMo r32 / 4 paragraphs OLMo r32 / 4 paragraphs Huginn / 10 paragraphs

Wall time (s) EM Uniform → LoopDuet Speedup Uniform → LoopDuet Layer ratio Fraction

(32, 8) (11, 4) (32, 16)

808.7 → 271.1 304.7 → 142.8 7588 → 4064

38.49 → 38.42 38.43 → 38.27 33.79 → 33.69

2.98× 2.13× 1.87×

3.51 2.29 1.94

85% 93% 96%

Table A6: Complete measured joint-depth configurations on HotpotQA at 𝐶 = 64. Holds uses the full (32, 32) serving control. Model

(𝑆, 𝑃)

Req/s

EM

F1

ΔEM

Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32

(32, 32) (24, 24) (20, 20) (18, 18) (32, 16) (16, 16) (14, 14) (24, 12) (12, 12) (14, 8) (8, 8) (32, 32) (11, 11) (32, 8) (8, 8) (11, 4) (4, 4)

0.976 1.313 1.536 1.698 1.822 1.897 2.150 2.431 2.484 3.489 3.587 9.156 24.302 27.315 32.220 51.850 56.464

33.79 33.52 33.52 32.72 33.69 32.07 31.76 31.60 32.05 30.33 24.29 38.49 38.43 38.42 38.37 38.27 37.95

44.42 44.19 44.14 43.19 44.29 42.40 42.03 41.91 42.38 40.35 33.22 50.62 50.53 50.47 50.42 50.12 49.63

+0.00 -0.27 -0.27 -1.07 -0.09 -1.72 -2.03 -2.19 -1.74 -3.46 -9.49 +0.00 -0.05 -0.07 -0.12 -0.22 -0.54

Table A8: GSM8K serving comparisons on all 1,319 test examples at 𝐶 = 64. Wall-time pairs are uniform (𝑆, 𝑆) followed by LoopDuet (𝑆, 𝑃); accuracy changes are percentage points. Layer ratios use the same workload-dependent estimate as Table A7. Model

(𝑆, 𝑃)

Huginn Huginn OLMo r32 OLMo r32

(14, 8) (32, 16) (11, 4) (32, 8)

Wall time (s)

Speedup

Layer ratio

Δ accuracy

151.43 → 152.42 324.94 → 331.10 87.09 → 86.34 225.61 → 223.35

0.99× 0.98× 1.01× 1.01×

1.52 1.71 2.00 2.71

+0.68 +2.12 +0.53 −0.15

31

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A9: Huginn HotpotQA batching sweep at 𝐶 = 128: (32, 16) versus (32, 32). Steps count LoopDuet engine calls; median TTFT pairs are uniform followed by LoopDuet. Accuracy changes are EM percentage points. Token budget

Engine steps

Speedup

ΔEM

Median TTFT (s)

4,096 8,192 16,384

3,797 1,902 957

1.84× 1.87× 1.88×

+0.12 −0.09 −0.14

126.6 → 68.7 116.0 → 62.1 103.1 → 54.8

Table A10: Queue-controller depth allocation over time. Each share is the percentage of total wall time spent at the indicated prefilling depth; time before the first controller update is unassigned. Requests follow the policy active when their chunks are scheduled. 𝑆

(𝑃min , 𝑃max )

Arrival trace (req/s)

4 4 11 11 11 11

(2, 4) (2, 4) (2, 11) (2, 11) (4, 11) (4, 11)

Burst 10/60 Burst 10/60 Poisson 80 Poisson 120 Poisson 80 Poisson 120

Depth 𝑃

Wall time (%)

4 3 2 2 4 4

86.99 12.92 97.62 98.15 98.73 98.82

Table A11: Frozen FFN routing on OLMo r32, 𝑆 = 11, all 1,319 GSM8K examples. Loops 1–4 are dense; loops 5–11 retain the indicated fraction of neurons using a per-token selection frozen from loop four. The dense control uses all neurons throughout. Retained neurons 100% (dense) 50% 25%

Correct answers

Accuracy (%)

Δ accuracy (pp)

628 622 454

47.61 47.16 34.42

+0.00 −0.45 −13.19

Table A12: Serving performance across depth, load, context, batching, and execution-mode settings (133 runs). TTFT is in seconds. Accuracy is EM on HotpotQA and numeric-answer correctness on GSM8K. Run olmo32_s11_ctrl2_burst_C128 olmo32_s11_ctrl2_pois120_C128 olmo32_s11_ctrl2_pois40_C128 olmo32_s11_ctrl2_pois80_C128 olmo32_s11_ctrl42_burst olmo32_s11_ctrl42_burst2080 olmo32_s11_ctrl42_pois40 olmo32_s11_ctrl42_pois60 olmo32_s11_ctrl42_pois80 olmo32_s11_ctrl4_burst2080 olmo32_s11_ctrl4_burst_C128 olmo32_s11_ctrl4_pois120_C128 olmo32_s11_ctrl4_pois40_C128

𝑛

Wall (s)

Acc.

TTFT p50

TTFT p99

≤ 1s (%)

7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405

225.7 98.9 187.3 99.3 225.3 162.5 186.5 125.1 98.9 167.4 227.8 143.9 187.3

36.06 32.10 37.99 32.21 37.25 34.27 38.35 36.93 32.15 38.26 38.16 38.16 38.00

0.979 18.554 0.829 3.120 0.580 1.278 0.150 0.648 2.819 6.886 2.860 41.044 0.952

1.191 36.247 1.087 5.768 0.939 2.455 0.354 0.975 5.456 16.721 5.631 80.898 1.219

57.2 0.3 92.3 0.7 100.0 34.1 100.0 99.6 3.8 10.9 15.9 0.3 63.3

32

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A12: Serving performance across depth, load, context, batching, and execution-mode settings (133 runs). TTFT is in seconds. Accuracy is EM on HotpotQA and numeric-answer correctness on GSM8K. Run olmo32_s11_ctrl4_pois80_C128 olmo32_s11_p11_burst_C128 olmo32_s11_p11_pois120_C128 olmo32_s11_p2_C64 olmo32_s11_p2_burst2080 olmo32_s11_p2_burst_C128 olmo32_s11_p2_pois120_C128 olmo32_s11_p2_pois40_C128 olmo32_s11_p2_pois60 olmo32_s11_p2_pois80_C128 olmo32_s11_p3_C64 olmo32_s11_p3_burst olmo32_s11_p3_burst2080 olmo32_s11_p3_pois60 olmo32_s11_p3_pois80 olmo32_s11_p4_burst2080 olmo32_s11_p4_burst_C128 olmo32_s11_p4_pois120_C128 olmo32_s11_p4_pois60 olmo32_s11_rate2_burst olmo32_s11_rate2_burst2080 olmo32_s11_rate2_pois20 olmo32_s11_rate2_pois40 olmo32_s11_rate2_pois60 olmo32_s11_rate2_pois80 olmo32_s11_rate4_burst olmo32_s11_rate4_burst2080 olmo32_s11_rate4_pois20 olmo32_s11_rate4_pois40 olmo32_s11_rate4_pois60 olmo32_s11_rate4_pois80 repro_olmo32_s11_p4_pois40_C128 olmo32_s4_ctrl42_burst2080_C128 olmo32_s4_ctrl42_burst_C128 olmo32_s4_ctrl42_pois40_C128 olmo32_s4_ctrl42_pois60_C128 olmo32_s4_p2_burst2080_C128 olmo32_s4_p2_burst_C128 olmo32_s4_p2_pois40_C128 olmo32_s4_p2_pois60_C128 olmo32_s4_p3_burst2080_C128 olmo32_s4_p3_burst_C128 olmo32_s4_p3_pois40_C128 olmo32_s4_p3_pois60_C128 olmo32_s4_p4_burst2080_C128 olmo32_s4_p4_burst_C128 olmo32_s4_p4_pois40_C128

𝑛

Wall (s)

Acc.

TTFT p50

TTFT p99

≤ 1s (%)

7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405

143.8 321.7 303.4 98.0 161.9 224.8 98.1 186.4 124.5 98.3 120.0 225.1 164.0 124.8 120.2 166.7 227.1 143.3 143.5 224.8 162.2 372.8 186.5 124.5 98.9 227.3 167.4 372.8 186.5 144.2 144.0 186.5 161.7 225.1 186.4 124.8 161.1 224.5 186.4 124.3 162.6 224.7 186.4 124.5 165.2 225.8 186.4

38.11 38.39 38.39 32.09 32.25 32.19 32.25 32.07 32.22 32.22 36.84 36.85 36.89 36.88 36.80 38.15 38.08 38.20 38.10 34.23 34.33 38.74 38.30 33.25 32.23 38.10 38.28 38.70 38.27 38.11 38.26 38.28 35.12 37.77 37.96 37.27 31.91 31.82 31.83 31.74 36.42 36.42 36.29 36.35 38.20 38.04 37.92

25.376 53.072 120.422 0.270 0.692 0.122 17.617 0.091 0.139 2.213 0.493 0.325 2.934 0.401 13.233 6.070 2.055 40.453 9.191 0.173 1.110 0.170 0.152 0.166 2.217 2.479 6.291 0.174 0.148 9.203 24.762 0.149 0.848 0.427 0.112 0.486 0.128 0.050 0.039 0.073 1.584 0.132 0.049 0.136 4.313 1.045 0.087

50.065 95.322 238.887 0.530 1.796 0.268 35.371 0.196 0.285 4.840 0.658 0.791 8.626 0.912 26.698 15.928 4.785 80.305 18.909 0.917 2.178 0.452 0.359 0.344 4.804 5.328 16.115 0.442 0.364 18.988 49.555 0.352 1.110 0.682 0.291 0.725 0.398 0.133 0.085 0.172 4.754 0.412 0.130 0.367 12.004 1.904 0.218

0.4 4.4 0.3 99.9 72.2 100.0 2.7 100.0 100.0 10.6 99.8 100.0 28.1 99.9 3.5 16.0 29.1 1.2 4.2 99.9 44.6 100.0 100.0 100.0 10.6 23.7 15.0 100.0 100.0 4.1 2.0 100.0 87.5 100.0 100.0 100.0 100.0 100.0 100.0 100.0 39.1 100.0 100.0 100.0 21.3 46.9 100.0

33

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A12: Serving performance across depth, load, context, batching, and execution-mode settings (133 runs). TTFT is in seconds. Accuracy is EM on HotpotQA and numeric-answer correctness on GSM8K. Run olmo32_s4_p4_pois60_C128 gsm_huginn_s14_p14_C64 gsm_huginn_s14_p8_C64 gsm_huginn_s32_p16_C64 gsm_huginn_s32_p32_C64 gsm_olmo32_s11_p11_C64 gsm_olmo32_s11_p4_C64 gsm_olmo32_s32_p32_C64 gsm_olmo32_s32_p8_C64 hp2_huginn_s24_p12_C64 hp2_huginn_s24_p24_C64 hp2_olmo32_s11_p11_C64 hp2_olmo32_s11_p4_C64 hp4_huginn_s24_p12_C64 hp4_huginn_s24_p24_C64 hp_huginn_s12_p12_C64 hp_huginn_s14_p14_C64 hp_huginn_s14_p8_C64 hp_huginn_s16_p16_C64 hp_huginn_s18_p18_C64 hp_huginn_s20_p20_C64 hp_huginn_s24_p12_C128 hp_huginn_s24_p12_C32 hp_huginn_s24_p12_C64 hp_huginn_s24_p12_pois2 hp_huginn_s24_p12_pois4 hp_huginn_s24_p12_pois8 hp_huginn_s24_p24_C128 hp_huginn_s24_p24_C32 hp_huginn_s24_p24_C64 hp_huginn_s24_p24_C64_compiled hp_huginn_s24_p24_pois2 hp_huginn_s24_p24_pois4 hp_huginn_s24_p24_pois8 hp_huginn_s32_p16_C128 hp_huginn_s32_p16_C128_T16384 hp_huginn_s32_p16_C128_T4096 hp_huginn_s32_p16_C32 hp_huginn_s32_p16_C64 hp_huginn_s32_p16_pois4_T16384 hp_huginn_s32_p16_pois4_T4096 hp_huginn_s32_p32_C128 hp_huginn_s32_p32_C128_T16384 hp_huginn_s32_p32_C128_T4096 hp_huginn_s32_p32_C32 hp_huginn_s32_p32_C64 hp_huginn_s32_p32_pois4_T16384

𝑛

Wall (s)

Acc.

TTFT p50

TTFT p99

≤ 1s (%)

7405 1319 1319 1319 1319 1319 1319 1319 1319 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405

131.4 151.4 152.4 331.1 324.9 87.1 86.3 225.6 223.3 1255.3 2319.0 212.9 101.0 1709.8 3164.1 2981.6 3443.8 2122.4 3903.3 4362.2 4821.7 3098.1 3099.1 3046.3 3728.2 3102.4 3102.2 5753.2 5746.1 5639.3 6877.2 5747.7 5743.3 5744.7 4062.7 4006.8 4208.1 4062.2 4064.1 4008.3 4212.2 7588.4 7529.5 7744.7 7584.8 7587.9 7545.5

38.04 20.62 21.30 21.46 19.33 47.08 47.61 47.69 47.54 49.22 49.66 43.50 42.15 45.08 45.52 32.05 31.76 30.33 32.07 32.72 33.52 31.60 31.60 31.60 31.75 31.56 31.61 33.52 33.49 33.52 33.57 33.46 33.38 33.40 33.69 33.71 33.94 33.69 33.69 33.53 34.09 33.79 33.84 33.81 33.79 33.79 33.88

3.239 0.134 0.131 0.289 0.295 0.090 0.088 0.228 0.214 4.817 8.900 0.658 0.313 8.097 14.958 20.402 23.573 14.522 26.707 29.852 32.992 47.371 6.577 20.828 2.336 608.259 1078.914 87.896 12.159 38.595 47.084 987.432 1927.579 2398.588 62.114 54.823 68.721 8.611 27.779 1065.871 1161.981 116.042 103.121 126.551 16.049 51.934 2830.441

7.027 1.031 0.663 1.189 2.058 0.351 0.238 0.800 0.493 6.433 11.876 1.153 0.527 11.268 20.878 22.068 25.499 15.739 28.888 32.299 35.714 50.767 9.816 22.566 7.001 1226.281 2150.136 94.349 18.199 41.776 50.916 1998.943 3842.400 4767.165 66.597 59.283 72.345 12.859 30.117 2126.855 2325.044 124.260 111.358 133.032 24.031 56.187 5632.529

8.6 95.1 100.0 95.1 95.1 100.0 100.0 100.0 100.0 0.0 0.0 97.7 99.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 18.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

34

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A12: Serving performance across depth, load, context, batching, and execution-mode settings (133 runs). TTFT is in seconds. Accuracy is EM on HotpotQA and numeric-answer correctness on GSM8K. 𝑛

Wall (s)

Acc.

TTFT p50

TTFT p99

≤ 1s (%)

7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405

7744.8 2064.6 298.4 316.0 304.7 432.6 745.4 373.0 303.3 303.4 142.9 150.4 142.8 745.4 372.8 186.5 142.9 807.3 857.7 808.7 271.0 294.2 271.1 131.2 131.1 229.8

33.68 24.29 38.38 38.27 38.43 38.49 38.41 38.41 38.34 38.38 38.18 38.26 38.27 38.18 38.24 38.20 38.23 38.47 38.43 38.49 38.39 38.30 38.42 38.06 37.95 38.37

2926.280 14.136 3.705 0.503 1.255 1.779 0.103 0.268 57.718 104.849 1.769 0.238 0.584 0.105 0.129 0.167 24.490 10.027 1.366 3.328 3.354 0.467 1.108 1.625 0.538 0.944

5822.902 15.288 4.126 0.763 1.677 2.385 0.317 0.820 115.667 208.069 1.976 0.334 0.783 0.335 0.312 0.368 49.172 11.159 1.945 4.450 3.747 0.702 1.484 1.813 0.723 1.265

0.0 0.0 0.3 99.8 44.2 0.5 100.0 100.0 0.7 0.4 0.6 100.0 99.7 100.0 100.0 100.0 2.1 0.0 5.5 0.0 0.1 99.7 44.1 0.4 99.7 97.0

Run hp_huginn_s32_p32_pois4_T4096 hp_huginn_s8_p8_C64 hp_olmo32_s11_p11_C128 hp_olmo32_s11_p11_C32 hp_olmo32_s11_p11_C64 hp_olmo32_s11_p11_C64_compiled hp_olmo32_s11_p11_pois10 hp_olmo32_s11_p11_pois20 hp_olmo32_s11_p11_pois40 hp_olmo32_s11_p11_pois80 hp_olmo32_s11_p4_C128 hp_olmo32_s11_p4_C32 hp_olmo32_s11_p4_C64 hp_olmo32_s11_p4_pois10 hp_olmo32_s11_p4_pois20 hp_olmo32_s11_p4_pois40 hp_olmo32_s11_p4_pois80 hp_olmo32_s32_p32_C128 hp_olmo32_s32_p32_C32 hp_olmo32_s32_p32_C64 hp_olmo32_s32_p8_C128 hp_olmo32_s32_p8_C32 hp_olmo32_s32_p8_C64 hp_olmo32_s4_p4_C128 hp_olmo32_s4_p4_C64 hp_olmo32_s8_p8_C64

Table A13: All 169 supplied paired accuracy comparisons; accuracy and deltas in percentage points. Run

Control

arc_huginn_s14_p14 arc_huginn_s14_p14 arc_huginn_s14_p14_sf0.5s8 arc_huginn_s14_p4 arc_huginn_s14_p4 arc_huginn_s14_p8

arc_huginn_s24_p24 arc_huginn_s32_p32 arc_huginn_s14_p14 arc_huginn_s14_p14 arc_huginn_s4_p4 arc_huginn_s14_p14

arc_huginn_s14_p8 arc_huginn_s14_p8_sf0.5s8 arc_huginn_s14_p8_sf0.5s8 arc_huginn_s24_p24 arc_huginn_s32_p16

arc_huginn_s8_p8 arc_huginn_s14_p14 arc_huginn_s14_p8 arc_huginn_s32_p32 arc_huginn_s32_p32

arc_huginn_s32_p4 arc_huginn_s32_p4 arc_huginn_s32_p8 arc_huginn_s32_p8 arc_huginn_s4_p4

arc_huginn_s32_p32 arc_huginn_s4_p4 arc_huginn_s32_p32 arc_huginn_s8_p8 arc_huginn_s14_p14

𝑛

Control acc.

Run acc.

Δ

1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172

43.52 43.60 38.82 38.82 23.46 38.82 28.75 38.82 34.39 43.60 43.60 43.60 23.46 43.60 28.75 38.82

38.82 38.82 38.91 24.66 24.66 34.39 34.39 34.47 34.47 43.52 42.15 24.23 24.23 35.24 35.24 23.46

-4.69 -4.78 +0.09 -14.16 +1.19 -4.44 +5.63 -4.35 +0.09 -0.09 -1.45 -19.37 +0.77 -8.36 +6.48 -15.36

35

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A13: All 169 supplied paired accuracy comparisons; accuracy and deltas in percentage points. Run

Control

arc_huginn_s4_p4 arc_huginn_s4_p4 arc_huginn_s8_p8 arc_huginn_s8_p8 arc_huginn_s8_p8 arc_olmo32_s11_p11

arc_huginn_s24_p24 arc_huginn_s32_p32 arc_huginn_s14_p14 arc_huginn_s24_p24 arc_huginn_s32_p32 arc_olmo32_s16_p16

arc_olmo32_s11_p11 arc_olmo32_s11_p11_sf0.5s4 arc_olmo32_s11_p2 arc_olmo32_s11_p4 arc_olmo32_s11_p4 arc_olmo32_s11_p4_sf0.5s4

arc_olmo32_s32_p32 arc_olmo32_s11_p11 arc_olmo32_s11_p11 arc_olmo32_s11_p11 arc_olmo32_s4_p4 arc_olmo32_s11_p11

arc_olmo32_s11_p4_sf0.5s4 arc_olmo32_s11_p8 arc_olmo32_s11_p8 arc_olmo32_s16_p16 arc_olmo32_s32_p16

arc_olmo32_s11_p4 arc_olmo32_s11_p11 arc_olmo32_s8_p8 arc_olmo32_s32_p32 arc_olmo32_s16_p16

arc_olmo32_s32_p16 arc_olmo32_s32_p2 arc_olmo32_s32_p4 arc_olmo32_s32_p4 arc_olmo32_s32_p8 arc_olmo32_s32_p8

arc_olmo32_s32_p32 arc_olmo32_s32_p32 arc_olmo32_s32_p32 arc_olmo32_s4_p4 arc_olmo32_s32_p32 arc_olmo32_s8_p8

arc_olmo32_s4_p4 arc_olmo32_s4_p4 arc_olmo32_s4_p4 arc_olmo32_s8_p8 arc_olmo32_s8_p8 arc_olmo32_s8_p8

arc_olmo32_s11_p11 arc_olmo32_s16_p16 arc_olmo32_s32_p32 arc_olmo32_s11_p11 arc_olmo32_s16_p16 arc_olmo32_s32_p32

hp_huginn_s14_p14 hp_huginn_s14_p14 hp_huginn_s14_p8 hp_huginn_s14_p8 hp_huginn_s20_p20 hp_huginn_s20_p20

hp_huginn_s24_p24 hp_huginn_s32_p32 hp_huginn_s14_p14 hp_huginn_s8_p8 hp_huginn_s24_p24 hp_huginn_s32_p32

hp_huginn_s24_p12 hp_huginn_s24_p24 hp_huginn_s24_p8 hp_huginn_s24_p8 hp_huginn_s32_p16

hp_huginn_s24_p24 hp_huginn_s32_p32 hp_huginn_s24_p24 hp_huginn_s8_p8 hp_huginn_s32_p32

hp_huginn_s32_p16_kvsh hp_huginn_s32_p16_ls hp_huginn_s32_p16_ls hp_huginn_s32_p16_sf0.5s16 hp_huginn_s32_p16_sf0.5s16 hp_huginn_s32_p32_sf0.5s16

hp_huginn_s32_p32_kvsh hp_huginn_s32_p16 hp_huginn_s32_p32 hp_huginn_s32_p16 hp_huginn_s32_p32 hp_huginn_s32_p32

hp_huginn_s32_p32_sf0.5s16_g32 hp_huginn_s32_p32_sf0.5s16_static

hp_huginn_s32_p32 hp_huginn_s32_p32

𝑛

Control acc.

Run acc.

Δ

1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 1172 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405

43.52 43.60 38.82 43.52 43.60 54.86 54.69 54.18 54.18 54.18 52.73 54.18 52.65 54.18 54.78 54.69 54.86 54.69 54.69 54.69 52.73 54.69 54.78 54.18 54.86 54.69 54.18 54.86 54.69 33.54 33.90 31.48 25.79 33.54 33.90 33.54 33.90 33.54 25.79 33.90 34.02 33.67 33.90 33.67 33.90 33.90 33.90 33.90

23.46 23.46 28.75 28.75 28.75 54.18 54.18 53.84 44.45 52.65 52.65 52.39 52.39 55.03 55.03 54.86 54.52 54.52 44.62 52.82 52.82 54.69 54.69 52.73 52.73 52.73 54.78 54.78 54.78 31.48 31.48 30.76 30.76 33.23 33.23 31.64 33.54 30.97 30.97 33.67 33.53 31.92 31.92 32.96 32.96 33.03 22.40 17.93

-20.05 -20.14 -10.07 -14.76 -14.85 -0.68 -0.51 -0.34 -9.73 -1.54 -0.09 -1.79 -0.26 +0.85 +0.26 +0.17 -0.34 -0.17 -10.07 -1.88 +0.09 +0.00 -0.09 -1.45 -2.13 -1.96 +0.60 -0.09 +0.09 -2.07 -2.42 -0.72 +4.97 -0.31 -0.66 -1.90 -0.35 -2.58 +5.17 -0.23 -0.49 -1.74 -1.97 -0.70 -0.93 -0.86 -11.49 -15.96

36

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A13: All 169 supplied paired accuracy comparisons; accuracy and deltas in percentage points. Run

Control

hp_huginn_s32_p4 hp_huginn_s32_p8 hp_huginn_s32_p8 hp_huginn_s8_p8 hp_huginn_s8_p8 hp_huginn_s8_p8

hp_huginn_s32_p32 hp_huginn_s32_p32 hp_huginn_s8_p8 hp_huginn_s14_p14 hp_huginn_s24_p24 hp_huginn_s32_p32

hp_olmo16_s16_p2 hp_olmo16_s16_p4 hp_olmo16_s16_p4 hp_olmo16_s16_p8 hp_olmo16_s16_p8 hp_olmo16_s4_p4

hp_olmo16_s16_p16 hp_olmo16_s16_p16 hp_olmo16_s4_p4 hp_olmo16_s16_p16 hp_olmo16_s8_p8 hp_olmo16_s16_p16

hp_olmo16_s8_p8 hp_olmo32_s11_p11 hp_olmo32_s11_p11 hp_olmo32_s11_p11_sf0.5s4 hp_olmo32_s11_p11_sf0.5s4_g32

hp_olmo16_s16_p16 hp_olmo32_s16_p16 hp_olmo32_s32_p32 hp_olmo32_s11_p11 hp_olmo32_s11_p11

hp_olmo32_s11_p11_sf0.5s4_static hp_olmo32_s11_p2 hp_olmo32_s11_p4 hp_olmo32_s11_p4 hp_olmo32_s11_p4_kvsh hp_olmo32_s11_p4_ls

hp_olmo32_s11_p11 hp_olmo32_s11_p11 hp_olmo32_s11_p11 hp_olmo32_s4_p4 hp_olmo32_s11_p11_kvsh hp_olmo32_s11_p11

hp_olmo32_s11_p4_ls hp_olmo32_s11_p4_sf0.5s4 hp_olmo32_s11_p4_sf0.5s4 hp_olmo32_s11_p8 hp_olmo32_s11_p8 hp_olmo32_s16_p16

hp_olmo32_s11_p4 hp_olmo32_s11_p11 hp_olmo32_s11_p4 hp_olmo32_s11_p11 hp_olmo32_s8_p8 hp_olmo32_s32_p32

hp_olmo32_s32_p16 hp_olmo32_s32_p16 hp_olmo32_s32_p2 hp_olmo32_s32_p4 hp_olmo32_s32_p4 hp_olmo32_s32_p8

hp_olmo32_s16_p16 hp_olmo32_s32_p32 hp_olmo32_s32_p32 hp_olmo32_s32_p32 hp_olmo32_s4_p4 hp_olmo32_s32_p32

hp_olmo32_s32_p8 hp_olmo32_s4_p4 hp_olmo32_s4_p4 hp_olmo32_s4_p4 hp_olmo32_s8_p8

hp_olmo32_s8_p8 hp_olmo32_s11_p11 hp_olmo32_s16_p16 hp_olmo32_s32_p32 hp_olmo32_s11_p11

hp_olmo32_s8_p8 hp_olmo32_s8_p8 huginn_s14_p14 huginn_s14_p14 huginn_s14_p14_sf0.5s8 huginn_s14_p14_sf0.5s8_g32

hp_olmo32_s16_p16 hp_olmo32_s32_p32 huginn_s24_p24 huginn_s32_p32 huginn_s14_p14 huginn_s14_p14

huginn_s14_p14_sf0.5s8_static huginn_s14_p4

huginn_s14_p14 huginn_s14_p14

𝑛

Control acc.

Run acc.

Δ

7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 7405 1319 1319 1319 1319 1319 1319

33.90 33.90 25.79 31.48 33.54 33.90 38.35 38.35 37.89 38.35 38.34 38.35 38.35 38.42 38.43 38.45 38.45 38.45 38.45 38.45 37.99 38.38 38.45 38.19 38.45 38.19 38.45 38.41 38.43 38.42 38.43 38.43 38.43 37.99 38.43 38.41 38.45 38.42 38.43 38.45 38.42 38.43 20.32 20.47 18.57 18.57 18.57 18.57

16.54 31.26 31.26 25.79 25.79 25.79 34.61 38.06 38.06 38.37 38.37 37.89 38.34 38.45 38.45 38.31 36.04 36.64 32.17 38.19 38.19 38.28 38.00 38.00 38.11 38.11 38.31 38.31 38.42 38.49 38.49 32.19 38.34 38.34 38.37 38.37 37.99 37.99 37.99 38.41 38.41 38.41 18.57 18.57 15.85 7.28 4.47 7.43

-17.35 -2.63 +5.47 -5.69 -7.75 -8.10 -3.74 -0.30 +0.16 +0.01 +0.03 -0.46 -0.01 +0.03 +0.01 -0.14 -2.40 -1.81 -6.28 -0.26 +0.20 -0.09 -0.45 -0.19 -0.34 -0.08 -0.14 -0.09 -0.01 +0.07 +0.05 -6.24 -0.09 +0.35 -0.07 -0.04 -0.46 -0.43 -0.45 -0.04 -0.01 -0.03 -1.74 -1.90 -2.73 -11.30 -14.10 -11.14

37

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A13: All 169 supplied paired accuracy comparisons; accuracy and deltas in percentage points. Run

Control

huginn_s14_p4 huginn_s14_p6 huginn_s14_p8 huginn_s14_p8 huginn_s14_p8_kvsh huginn_s14_p8_ls

huginn_s4_p4 huginn_s14_p14 huginn_s14_p14 huginn_s8_p8 huginn_s14_p14_kvsh huginn_s14_p14

huginn_s14_p8_ls huginn_s14_p8_sf0.5s8 huginn_s14_p8_sf0.5s8 huginn_s24_p24 huginn_s32_p16 huginn_s32_p4

huginn_s14_p8 huginn_s14_p14 huginn_s14_p8 huginn_s32_p32 huginn_s32_p32 huginn_s32_p32

huginn_s32_p4 huginn_s32_p8 huginn_s32_p8 huginn_s4_p4 huginn_s4_p4

huginn_s4_p4 huginn_s32_p32 huginn_s8_p8 huginn_s14_p14 huginn_s24_p24

huginn_s4_p4 huginn_s8_p8 huginn_s8_p8 huginn_s8_p8 olmo16_s16_p2 olmo16_s16_p4

huginn_s32_p32 huginn_s14_p14 huginn_s24_p24 huginn_s32_p32 olmo16_s16_p16 olmo16_s16_p16

olmo16_s16_p4 olmo16_s16_p8 olmo16_s16_p8 olmo16_s4_p4 olmo16_s8_p8 olmo32_s11_p11

olmo16_s4_p4 olmo16_s16_p16 olmo16_s8_p8 olmo16_s16_p16 olmo16_s16_p16 olmo32_s16_p16

olmo32_s11_p11 olmo32_s11_p11_sf0.5s4 olmo32_s11_p11_sf0.5s4_g32 olmo32_s11_p11_sf0.5s4_static olmo32_s11_p2 olmo32_s11_p4

olmo32_s32_p32 olmo32_s11_p11 olmo32_s11_p11 olmo32_s11_p11 olmo32_s11_p11 olmo32_s11_p11

olmo32_s11_p4 olmo32_s11_p4_kvsh olmo32_s11_p4_ls olmo32_s11_p4_ls olmo32_s11_p4_sf0.5s4

olmo32_s4_p4 olmo32_s11_p11_kvsh olmo32_s11_p11 olmo32_s11_p4 olmo32_s11_p11

olmo32_s11_p4_sf0.5s4 olmo32_s11_p8 olmo32_s11_p8 olmo32_s16_p16 olmo32_s32_p16 olmo32_s32_p16

olmo32_s11_p4 olmo32_s11_p11 olmo32_s8_p8 olmo32_s32_p32 olmo32_s16_p16 olmo32_s32_p32

olmo32_s32_p2 olmo32_s32_p4

olmo32_s32_p32 olmo32_s32_p32

𝑛

Control acc.

Run acc.

Δ

1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319 1319

2.05 18.57 18.57 7.58 19.86 18.57 19.26 18.57 19.26 20.47 20.47 20.47 2.05 20.47 7.58 18.57 20.32 20.47 18.57 20.32 20.47 44.43 44.43 42.99 44.43 44.05 44.43 44.43 47.38 48.29 47.38 47.38 47.38 47.38 47.38 42.53 47.76 47.38 48.45 47.38 48.45 47.38 47.01 48.29 47.38 48.29 48.29 48.29

7.43 16.38 19.26 19.26 19.48 17.74 17.74 15.92 15.92 20.32 19.64 8.11 8.11 19.56 19.56 2.05 2.05 2.05 7.58 7.58 7.58 40.71 46.17 46.17 44.35 44.35 42.99 44.05 47.38 47.38 44.88 31.16 26.76 41.77 48.45 48.45 47.54 47.01 47.01 46.55 46.55 47.16 47.16 47.38 47.84 47.84 41.62 48.29

+5.38 -2.20 +0.68 +11.68 -0.38 -0.83 -1.52 -2.65 -3.34 -0.15 -0.83 -12.36 +6.07 -0.91 +11.98 -16.53 -18.27 -18.42 -10.99 -12.74 -12.89 -3.71 +1.74 +3.18 -0.08 +0.30 -1.44 -0.38 +0.00 -0.91 -2.50 -16.22 -20.62 -5.61 +1.06 +5.91 -0.23 -0.38 -1.44 -0.83 -1.90 -0.23 +0.15 -0.91 +0.45 -0.45 -6.67 +0.00

38

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A13: All 169 supplied paired accuracy comparisons; accuracy and deltas in percentage points. Run

Control

olmo32_s32_p4 olmo32_s32_p8 olmo32_s32_p8 olmo32_s4_p4 olmo32_s4_p4 olmo32_s4_p4

olmo32_s4_p4 olmo32_s32_p32 olmo32_s8_p8 olmo32_s11_p11 olmo32_s16_p16 olmo32_s32_p32

olmo32_s8_p8 olmo32_s8_p8 olmo32_s8_p8

olmo32_s11_p11 olmo32_s16_p16 olmo32_s32_p32

𝑛

Control acc.

Run acc.

Δ

1319 1319 1319 1319 1319 1319 1319 1319 1319

42.53 48.29 47.01 47.38 47.38 48.29 47.38 47.38 48.29

48.29 47.01 47.01 42.53 42.53 42.53 47.01 47.01 47.01

+5.76 -1.29 +0.00 -4.85 -4.85 -5.76 -0.38 -0.38 -1.29

Table A14: Prefilling-state distance and task accuracy across 44 depth configurations. Distances compare loops 𝑃 and 𝑆 from the same reference-depth replay. The quality criterion summarizes measured accuracy; the distance threshold identifies candidate depths. Model

Task

𝑆

𝑃

Distance

Δ

Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn Huginn OLMo r16 OLMo r16 OLMo r16 OLMo r16 OLMo r16 OLMo r16 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32

ARC ARC ARC ARC ARC GSM8K GSM8K GSM8K GSM8K GSM8K GSM8K HotpotQA HotpotQA HotpotQA HotpotQA HotpotQA HotpotQA GSM8K GSM8K GSM8K HotpotQA HotpotQA HotpotQA ARC ARC ARC ARC ARC ARC ARC GSM8K GSM8K

14 14 32 32 32 14 14 14 32 32 32 14 24 24 32 32 32 16 16 16 16 16 16 11 11 11 32 32 32 32 11 11

4 8 4 8 16 4 6 8 4 8 16 8 8 12 4 8 16 2 4 8 2 4 8 2 4 8 2 4 8 16 2 4

0.93704 0.50296 0.93917 0.51466 0.17201 1.03466 0.80493 0.61445 1.03494 0.62001 0.22501 0.46377 0.47428 0.25692 0.88394 0.47760 0.14947 0.34119 0.10829 0.01564 0.29106 0.09401 0.01841 0.34402 0.11403 0.01860 0.34437 0.11518 0.02531 0.01096 0.44610 0.18044

-14.16 -4.44 -19.37 -8.36 -1.45 -11.14 -2.20 +0.68 -12.36 -0.91 -0.83 -0.72 -2.58 -1.90 -17.35 -2.63 -0.23 -3.71 +1.74 -0.08 -3.74 -0.30 +0.01 -9.73 -1.54 +0.85 -10.07 -1.88 +0.00 -0.17 -5.61 +1.06

39

Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers Table A14: Prefilling-state distance and task accuracy across 44 depth configurations. Distances compare loops 𝑃 and 𝑆 from the same reference-depth replay. The quality criterion summarizes measured accuracy; the distance threshold identifies candidate depths. Model

Task

𝑆

𝑃

Distance

Δ

OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32 OLMo r32

GSM8K GSM8K GSM8K GSM8K GSM8K HotpotQA HotpotQA HotpotQA HotpotQA HotpotQA HotpotQA HotpotQA

11 32 32 32 32 11 11 11 32 32 32 32

8 2 4 8 16 2 4 8 2 4 8 16

0.02956 0.44747 0.18253 0.03653 0.01190 0.39714 0.16846 0.03244 0.39763 0.17306 0.04271 0.01047

-0.23 -6.67 +0.00 -1.29 -0.45 -6.28 -0.26 -0.14 -6.24 -0.09 -0.07 +0.05

Table A15: At batch size 16, per-token FFN sparsity leaves nearly all neurons active. Neuron-union coverage (percent) for 50% per-token routing. Model

Task

𝑆

Answer tokens

𝐵=4

𝐵 = 16

𝐵 = 64

Groups of 32, 𝐵 = 16

OLMo r32 OLMo r32 Huginn Huginn

GSM8K HotpotQA GSM8K HotpotQA

11 11 14 32

116,390 28,173 148,075 27,952

91.35 90.02 90.50 89.52

99.95 99.85 99.87 99.80

100.00 100.00 100.00 100.00

99.72 98.29 97.61 97.16

40

Related documents

• Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts#1156088
• DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections#1354315
• Learning to Extrapolate in Depth with Looped Transformers#1325352
• Sustainable Processing-in-DRAM Integration and Replacement for Transformer Inference Servers#1402733
• SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture#1051246