Prefix Keys and Values Matter Late: RIPE Caching for Self-Speculative Decoding in Looped Language Models

P REFIX K EYS AND VALUES M ATTER L ATE : RIPE C ACHING FOR S ELF -S PECULATIVE D ECODING IN L OOPED L ANGUAGE M ODELS

arXiv:2610.34436v1 [cs.DC] 25 Oct 2026

Dezhen Lu 1 Runguo Li 2 Zixiang Ni 3

A BSTRACT A looped language model runs one block of layers R times per token, so its key–value (KV) cache holds R planes per recurrent layer, and self-speculative decoding (draft with a few loops, verify with all R) reads them all. We find that prefix keys and values matter late: at equal perturbation size, loops 25–32 are e15.0 (Huginn-0125) and e12.8 (Raven-Llama) times more sensitive than loops 2–9, and the planes approach the last one geometrically, at 0.88 and 0.81 per loop. Whether the deepest plane alone suffices then depends on how ripe the late planes are: on Raven, storing only it is indistinguishable from a bf16-to-fp32 change; on Huginn, 1% of positions carry 74% of its error. We turn this into RIPE, which stores the deepest plane everywhere except a window of recent positions kept at full depth. On Huginn, a 64-position window removes 89% of the tail positions at 11.9% of the full cache’s memory (1024-token context), and any window at least as long as the draft block makes batched verification block-independent by construction. Nine pre-registered GPU batches back these results. Simple models of acceptance, decoding cost and memory, two of them checked on held-out measurements, account for the measured self-speculative speed-ups (2.09× with the full cache, 2.58× with the deepest plane) and decompose the 2.24× fixed-memory throughput gain of one shared plane. 1

I NTRODUCTION

Looped, or recurrent-depth, language models reuse one block of transformer layers many times per token (Geiping et al., 2025; Zhu et al., 2025; McLeish et al., 2025). Because a shallow run of the same weights can serve as a draft, they support self-speculative decoding without a separate draft model: draft with d ≪ R loops, verify with all R (Geiping et al., 2025; Cho et al., 2026; Ha & Kim, 2026; Li et al., 2026; Liu et al., 2026). The weights are small; the KV cache is not. Every loop of every recurrent layer writes its own keys and values, so Huginn-0125 stores 2,788 KB per token, against 169 KB if only one plane were kept. At an 808-token sequence, a 95 GB budget then holds 36 sequences with the full cache and 302 with one plane (Section 3.5). Which planes have to be stored? Huginn’s own code offers caches that keep fewer planes (Geiping et al., 2025), Ouro reports that the last loop’s KV suffices for its model (Zhu et al., 2025), and WaveFront shows that cross-loop sharing raises self-speculative speed (Ha & Kim, 2026). We measured the question directly in nine pre-registered GPU 1

University of Tübingen, Tübingen, Germany 2 University of Illinois Urbana-Champaign, Urbana, IL, USA 3 Xi’an Jiaotong University, Xi’an, China. Correspondence to: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>, Runguo Li <runguol2@illinois.edu>, Zixiang Ni <nzx@stu.xjtu.edu.cn>. Preprint.

batches: decision rules and predictions were frozen before each run, and many predictions failed (Appendix B lists all of them). This paper organizes those measurements with small analytical models (Table 1 lists what each was fitted on and which checks use held-out data) and turns them into a cache design. RIPE in one line: ripe plane for the past, full depth for the present (Figure 1). Contributions. • Where KV precision matters. A sensitivity model (KL grows as a power, exponent 1.82–2.70, of a prefix-KV perturbation) and a ripening model (planes converge geometrically with depth) show that only the late loops need accurate prefix keys and values, and predict when the deepest plane alone costs no more than bf16 rounding (Sections 3.1, 3.2). Truncating finished tokens to shallow planes, our starting plan, is backwards. • RIPE. A cache that keeps the deepest plane everywhere except a recent full-depth window, with its memory cost, a decoding step (Algorithm 1) and a proof that batched verification is block-independent when the window covers the block (Section 4). • Pre-registered evidence on two model families. Nine GPU batches with frozen decision rules, including every failed prediction: the deepest plane’s error tail on Huginn and its removal by a 64-position window, its absence on Raven, and real self-speculative speed-ups

RIPE: Key–Value Caching for Looped Language Models (a) Full KV cache

settled positions: recent window W : deepest (ripe) plane only all planes

(c) One self-speculative step

loop 1 → R

draft ŷ1 , . . . , ŷγ

block γ+1 every loop’s plane for every position: 100%

(b) RIPE(W ): 6.1% of the full cache, plus the window (11.9% in total for W =64 at a 1024-token context)

d ≪ R loops each; cached positions: plane R

verify [x, ŷ1 , . . . , ŷγ ]

one pass, all R loops; loop r reads plane r if t − j ≤ W , else plane R

accept prefix + 1 token

store accepted planes; slide the window

W ≥ γ: the block reads exactly what sequential decoding reads, so batched verification is block-independent in exact arithmetic (Proposition 1).

Figure 1. RIPE at a glance (schematic; grid sizes not to scale). Rows are the prelude (bottom), loops 1, . . . , R and the coda (top) of each prefix position (columns). (a) A full cache keeps every loop’s keys and values. (b) RIPE keeps only the deepest, ripe plane for settled positions and all planes for the W most recent ones; the block being verified lies inside the window. Memory from Eqs. (3) and (10). (c) A self-speculative step. We measured the deepest-plane cache (W = 0) inside the speculative decoder and windows W > 0 by teacher-forced replay (Section 7).

(Section 5). • Performance models and pitfalls. Acceptance, cost, memory and fixed-memory throughput models, two checked on held-out data (Sections 3.3–3.5); per-head position filtering, sinks and a second plane do not help, and cuDNN attention makes bf16 decoding nonreproducible on GH200 (Sections 5.4, 5.5).

2

BACKGROUND

Looped models. A prelude P of npre layers embeds token xt ; a core block B of ncore layers runs R times on a state initialized at random; a coda C of ncoda layers produces logits: (r)

st

(r−1)

= B st

(R)

zt = C st

.

(r)
, P (xt ); C<t ,

r = 1, . . . , R, (1) (2)

At loop r, each core layer emits keys and values for token t; we call loop r’s keys and values over all core layers plane r, written (K (r) , V (r) ). In the full cache, the query of token t at loop r attends to plane r of every prefix position. Huginn-0125 (Geiping et al., 2025) has npre = ncoda = 2, ncore = 4, R = 32, and 55 heads of width 96. Raven-Llama (McLeish et al., 2025) has npre = ncoda = 4, ncore = 6, R = 32, and grouped-query attention with 8 KV heads of width 64. Cache policies. A policy π(r, t, j) ∈ {1, . . . , R} names the plane of prefix position j that token t reads at loop r (Figure 1). The full cache has π = r; the deepest-plane cache has π = R. Huginn’s compress-sS keeps S core slots and overwrites them modulo S; with S = ncore the last writer of every slot is loop R, so it is exactly the deepestplane cache. Huginn’s slot map hard-codes npre = 2; for Raven we use the general map slot = (step−npre ) mod S + npre , checked on CPU against direct substitution. With

b = 2 Hkv dh bytes per layer per token in bf16, an L-token sequence needs Mfull (L) = bL (npre + ncoda + R ncore ),

(3)

Mdeep (L) = bL (npre + ncoda + ncore ), that is, 2,788 vs. 169 KB per token on Huginn (6.1%) and 410 vs. 28.7 KB on Raven (7.0%). Self-speculative decoding. A draft runs γ tokens through d loops each; one verification pass runs the block of γ + 1 tokens through all R loops at once, accepts the longest prefix on which draft and verifier agree (greedy), and adds one token from the verifier. We write τ for tokens per verification. Measurement modes. Teacher-forced prefill replaces attention with an explicit fp32 softmax, records each call’s keys and values, and substitutes any loop’s prefix K/V by another plane; the last 64 positions act as decoded tokens. Generation decodes greedily with the full cache, then replays each trajectory through the arm and through an fp32 full cache, which sets the precision floor. We report KL(reference ∥ arm) per position.

3

M ODELS

All models in this section are post hoc: we built them after the pre-registered batches had run. We label each check by whether the data it uses entered the fit. 3.1

Sensitivity: only the deep loops’ prefix KV matters

Let δr be the relative perturbation of the prefix keys and values that P loop r reads. To first order the logits move ′ by ∆z ≈ r Jr δr . With p = softmax(z) and p = softmax(z + ∆z), KL(p ∥ p′ ) = 12 ∆z ⊤ F (p) ∆z + O(∥∆z∥3 ),

(4)

RIPE: Key–Value Caching for Looped Language Models Table 1. Post hoc models of this paper and how each was checked. “Fitted on” names the data that set its parameters; “Check” says whether the comparison data were held out from the fit. Batch numbers refer to Appendix A. Model

Form

Fitted on

Checked against

Check

Result

KV memory

Eq. (3), (10)

architecture only

held out

≤ 0.005 GB (print precision); exact

Peak memory

cache + o0 + o1 B

batch-6 cells

held out

≤ 0.15%

Sensitivity

region arms (pre-registered) fp32 curve, loops 8–31

theory

g = 2.70 / 1.82 (fp32)

Ripening

log KL = a + b[deep] + g log ∥δ∥ ∆r = cρr ; bf16 adds ∆∞

in sample

max | log err| fp32 0.28 / 0.51, bf16 0.27 / 0.19

Deepest-plane error

Eq. (6)

sensitivity + ripening

partly held out

3.6 × 10−6 vs. 3.2 × 10−6 (factor 2)

Acceptance

τ = (1 − αγ+1 )/(1 − α)

batch-3 simulation

held out

within 6%

Speed-up

Eq. (8)

one factor η, 24 timings

26 WaveFront cells; window extras (b. 8) batch-8 cells, other node, B ≤ 256 second-order theory, g = 2 fp32 curve; bf16 curve with one more parameter deepest-plane cache arm (fp32), not in the fit batch-8 real decoder the same timings, both caches

in sample

2.08/2.61 vs. 2.09/2.58

where F (p) = diag(p)−pp⊤ . Dropping cross terms, KL ≈ P 2 r sr ∥δr ∥ for small perturbations: quadratic, weighted by a per-loop sensitivity sr . We fit the exponent rather than fix it at 2. The pre-registered test of batches 2 and 9 substitutes the prefix K/V of either loops 2–9 (shallow) or loops 25–32 (deep) by plane p ∈ {1, 4, 8, 16, 24, 32}, measures the size ∥δ∥ as the root-sum-square of the relative distances over loops, layers, keys and values, and regresses log KL at the first decoded position on region and log ∥δ∥ (Figure 2a,b). In fp32 the size exponent is 1.82 [1.68, 1.97] on Raven, consistent with the second-order term, and g = 2.70 [2.52, 2.87] on Huginn, which excludes 2: Huginn’s shallow loops follow the quadratic (slope 2.06) but its deep loops are steeper (2.71). The largest substitutions (plane 1) are outside the small-perturbation regime of Eq. (4); Raven’s shallow region sits at the fp32 resolution. At equal size the deep region is e15.0 [e14.2 , e15.8 ] (Huginn, 40 texts) and e12.8 [e12.0 , e13.7 ] (Raven, 40 texts) times more sensitive. In bf16 the coefficients are 6.7 and 3.8; there 23% and 21% of the rows sit at the bf16 KL floor, which caps the measurable gap. The consequence for caching: the loops that need accurate prefix K/V are the deep ones. At equal storage, keeping the deepest plane agrees with the full cache at the first decoded position in 0.990 of texts, whereas a window of the first four loops, the starting plan’s choice, reaches 0.845 (Table 3). 3.2

Ripening: planes converge geometrically

Let ∆r = ∥K (r) − K (R) ∥/∥K (R) ∥ over prefix positions, averaged over layers and texts, and likewise for values. Fig-

ure 2c shows that in fp32, ∆r ≈ cρr from loop 8 on (on Raven the rate drifts upward with r; largest log residual 0.51). Huginn ripens at ρ = 0.88 per loop (half-life 5.2 loops) and Raven at ρ = 0.81 (3.3 loops); values behave the same way (ρ = 0.87 and 0.81). In bf16 the curve flattens at a rounding floor, p (5) ∆bf16 ≈ (cρr )2 + ∆2∞ , r and this one extra parameter reproduces the bf16 curve (∆∞ = 0.0072 for Huginn’s keys, 0.0048 for Raven’s). Over the deep loops 25–31, Huginn’s keys are still 0.0116 away from plane R in bf16, above its floor, while Raven’s are at its floor (0.0047); in fp32 Raven’s are 61 times closer than Huginn’s. Deepest-plane error. Storing only plane R sets δr = ∆r for every r < R. With the sensitivity profile of Section 3.1, only the deep loops count, and the fitted region model gives KLdeep ≈ ea+b S g ,

S 2 = ncore

31 X r=25

2 ∆K + ∆Vr 2 . r

(6) For Huginn in fp32 this predicts 3.6 × 10−6 at the first decoded position from the ripening curve, or 1.7 × 10−6 from the measured perturbation size; the deepest-plane cache, which the fit did not use, measures 3.2×10−6 (median). The agreement is to within a factor of two, not better. Substituting plane R in the shallow loops alone costs only 2.4×10−9 . For Raven the model predicts 1.8 × 10−8 (4.7 × 10−9 from the measured size), 202–366 times less than for Huginn, and the measurement is at the fp32 resolution (median below 10−9 , mean 2.7 × 10−8 ). Remark 1 (ripeness test, a hypothesis). If the deep loops’

RIPE: Key–Value Caching for Looped Language Models (b) Raven-Llama: same asymmetry

(c) planes ripen geometrically 100

100

loops 25–32 loops 2–9

∆K r

KL at first decoded position

(a) Huginn-0125: deep loops dominate 100

10−5

10−5

10−2

10−4 10−1

100

101

perturbation size ∥δ∥

10−3

10−2

10−1

100

perturbation size ∥δ∥

101

Huginn fp32 Raven fp32

bf16 bf16

10

20

30

loop r

Figure 2. Why only late planes matter, and when the deepest plane is enough. (a,b) First-decoded-position KL after substituting the prefix keys and values of loops 2–9 or 25–32 by planes 1–32 (fp32, AlpacaEval; points: arm medians over texts; lines: the pre-registered regression). At equal perturbation size the deep loops cost e12.8 –e15.0 times more. (c) Distance of loop r’s prefix keys to loop R’s: filled fp32, open bf16; solid cρr fitted to fp32, dotted with one bf16 floor (Eq. (5)). Raven’s deep loops (shaded) sit at the bf16 floor; Huginn’s do not.

∆r is at the bf16 floor, deepest-plane storage perturbs those loops about as much as rounding already does, and its cost should be indistinguishable from a precision change; otherwise it should exceed it. Both models fit this reading (Section 5.1), but two models cannot establish it. The test needs one prefill per prompt and no generation. 3.3

Acceptance

If each draft token is accepted with probability α, independently of its position in the block, the tokens per verification follow (Leviathan et al., 2022; Chen et al., 2023) τ (α, γ) =

1 − αγ+1 . 1−α

(7)

Batch 3 simulated greedy self-speculation on 60 GSM8K problems with γ = 4 and recorded acceptance at each draft position given that all earlier ones were accepted. It is flat: across positions it varies by at most 0.06 (full cache) and 0.04 (deepest plane) for d ∈ {2, 4, 8} (Figure 5a), and Eq. (7) with the mean αd matches the simulated τ within 2.6%. Out of sample, the batch-3 values predict τ = 3.40 (full) and 4.33 (deepest plane) at d = γ = 4; batch 8’s real batched-verify decoder measured 3.34 and 4.47 on 24 other GSM8K problems and 3.61 and 4.53 on 12 more. Drafts that read the deepest plane accept α4 = 0.93 instead of 0.81, and their rejection rate falls faster with draft depth: 1 − αd ∝ ρda with ρa = 0.66 against 0.76. 3.4

Cost

We assume that at batch size 1 our eager decoder is bound by kernel launches and weight reads, so that a forward costs in proportion to the layer passes it runs, c(d) = npre + d ncore + ncoda , almost regardless of how many tokens it carries. A sequential token costs c(R) = 132 passes; a block costs γ c(d) + c(R), scaled by one overhead factor η

for everything else. The speed-up over sequential decoding is then τ (αd , γ) c(R)
. S(d, γ) = (8) η γ c(d) + c(R) With η = 1 and the measured block counts, Eq. (8) gives 2.24× and 2.80× against measured 2.09× and 2.58× (12 problems, d = γ = 4). One factor η = 1.07, fitted to all 24 per-problem timings at once, gives 2.08× and 2.61×, with per-problem errors up to 5.4%. Sequential decoding is equally fast with either cache (ratio 1.00), so the gain of the deepest plane is entirely its higher acceptance. Figure 5b evaluates Eq. (8) over γ. This is an extrapolation: the flat-acceptance assumption was tested only up to γ = 4, and only d = γ = 4 was timed. In the model, deepestplane drafts both raise the speed-up and move the best block length up: at d = 4 the full cache peaks at γ = 5 (1.98×), while the deepest plane still rises at γ = 8 (2.87×). 3.5

Memory and fixed-memory throughput

WaveFront’s batched emulator (Ha & Kim, 2026) allocates exactly Eq. (3): over all 26 cells of batches 6 and 8 (sequences of 808 tokens; 0, 1 or 4 shared core slots), the reported cache size matches the formula to its printed precision. Peak reserved memory adds o0 + o1 B, with o0 = 7.2 GB and o1 = 0.18 GB (full) or 0.15 GB (one slot) per sequence, fitted on batch 6. This predicts the 9 batch-8 cells, which ran on another node and up to B = 256, within 0.15% (Figure 3). Under the pre-registered 95 GB budget, the largest batch is Bmax = 36 with the full cache (2.25 GB per sequence) and 302 with one slot. Throughput per configuration follows T (B) = B/(a + bB), a fixed per-step time plus a per-sequence time (largest residual 2% and 6% for the two series below). The full cache with draft-then-verify saturates at T∞ = 449 tokens/s

RIPE: Key–Value Caching for Looped Language Models

measured peak (GB)

100 95 GB budget

50 y = x batch 6 (fit) batch 8 (held out) 0 0

20

40

60

80

100

model peak memory (GB)

Figure 3. Peak reserved memory of WaveFront’s batched emulator against the memory model (Eq. (3) plus o0 + o1 B fitted on batch 6). Batch-8 cells (open) ran later on another node and were not used in the fit. Table 2. KV memory per sequence at L = 1024 from Eqs. (3) and (10) (bf16, MB = 106 bytes), and its fraction of the full cache. Huginn-0125

Raven-Llama

Cache

MB

frac.

MB

frac.

full deepest plane two planes RIPE, W = 16 RIPE, W = 64 RIPE, W = 256

2,855 173 260 215 341 843

100% 6.1% 9.1% 7.5% 11.9% 29.5%

419 29 42 35 54 127

100% 7.0% 10.0% 8.5% 12.8% 30.2%

with half-saturation at B1/2 = 33; one shared slot with WaveFront’s scheduler at 689 tokens/s with B1/2 = 27 (batch 6). The fixed-memory advantage of sharing has two factors (batch 6, Figure 5c). At the full cache’s best batch (B = 32), one shared slot already runs 365 tokens/s against the best full-cache configuration’s 218, 1.67×; with the same scheduler and acceptance rate the shared slot is 1.63× faster than the full cache at that batch, the cache size being the only difference. Memory then lets the shared slot run B = 64, another 1.34×, for 2.24× in all (Section 5.3).

4

RIPE

RIPE (Recent-window Intact, deepest Plane Elsewhere) reads the deepest plane for old positions and the own plane for the W most recent ones: ( r, t − j ≤ W, πRIPE (r, t, j) = (9) R, t − j > W, MRIPE (L) = Mdeep (L)+b ncore (R−1) min(W, L). (10) Our implementation keeps a deepest-plane array for all positions plus a ring buffer of W × (R − 1) core planes; a position leaving the window drops its non-deepest planes. At L = 1024, W = 64 costs 341 MB on Huginn (11.9% of 2,855 MB) and 54 MB on Raven (12.8%), i.e. about 168 MB per Huginn sequence over the deepest-plane cache (Table 2).

Algorithm 1 One RIPE self-speculative step (greedy) 1: Input: cache C (plane R of all positions; planes 1, . . . , R − 1 of the last W ), last token x, draft depth d, block length γ ≤ W 2: for i = 1 to γ do 3: ŷi ← arg max of a d-loop pass on ŷi−1 (ŷ0 = x); cached positions read plane R 4: end for 5: z ← one R-loop pass on [x, ŷ1 , . . . , ŷγ ]; loop r of token t reads plane r of j if t − j ≤ W , else plane R 6: k ← max{i : ŷ1..i = arg max z0..i−1 }; emit ŷ1..k and arg max zk 7: append the emitted tokens’ planes to C; drop planes 1, . . . , R − 1 of positions older than W Proposition 1 (block-independent verification). If W ≥ γ, verifying a block of γ + 1 tokens in one pass under πRIPE computes, in exact arithmetic, the same logits and the same cached keys and values as decoding the tokens one at a time under πRIPE . Proof sketch. πRIPE depends only on r and t − j. A block token t reads earlier block tokens j with t − j ≤ γ ≤ W , hence plane r, which the block pass has computed by the time it runs loop r. A cached position j < t0 with t−j ≤ W also satisfies t0 −j ≤ W , so its plane r is still in the window. All other positions read plane R, which is stored for every position. These are exactly the reads of sequential decoding, so the accepted tokens’ cached values agree as well. The deepest-plane cache (W = 0) violates the premise: a block token cannot read plane R of an earlier block token, which does not exist yet, so batched verification reads plane r there, and outputs depend on where blocks start. Section 5.2 measures how much this matters on Huginn. Drafts and the choice of W . Drafts do not affect correctness and may read any stored plane; reading plane R everywhere gives them the higher acceptance of Section 3.3. The ripeness test suggests the window length: where the deep loops are ripe to bf16 precision (Raven), W = γ suffices for exact verification; where they are not (Huginn), W = 64 removed most of the tail in our data (Section 5.1). We did not run a window larger than zero inside the speculative decoder (Section 7).

5

E VALUATION

Protocol. Each batch had a pre-registration with decision rules and predictions, a frozen file table (SHA-256), onnode CPU self-tests, a separate unscored Nsight Compute pass, and an archive verified on both ends. All runs used one NVIDIA GH200 (96 GB) with PyTorch 2.14 and CUDA

RIPE: Key–Value Caching for Looped Language Models Table 3. Equal-storage caches, first-decoded-position top-1 agreement with the full cache (batch 2, Huginn, AlpacaEval, bf16). Planes: core planes kept per position. Cache deepest plane (compress-s4) two deepest planes (s8) share every 8 loops first 4 loops (shallow window) share every 4 loops first 8 loops (shallow window)

Planes

Agreement

1 2 4 4 8 8

0.990 0.995 0.980 0.845 0.970 0.935

Table 4. RIPE windows on Huginn (batch 8, 48 prompts). Tail: positions with KL> 0.1. Extra: memory over the deepest-plane cache at L = 1024 (Eq. (10)). Cache deepest plane (W = 0) RIPE, W = 16 RIPE, W = 64 RIPE, W = 256 + 4 sinks two planes (s8) fp32 full cache

Tail

Max KL

Flips

Extra MB

9 5 1 0 9 9 1

5.05 0.97 0.13 0.04 5.03 5.24 0.23

35 37 31 10 38 42 32

0 42 168 670 10 87 —

13.0, bf16 and greedy decoding unless stated, and cuDNN attention disabled from batch 7 on (Section 5.5). Prompts are GSM8K 8-shot problems (Cobbe et al., 2021) and AlpacaEval instructions or reference texts (Dubois et al., 2024). Huginn uses HF revision bb6621b6 and Raven revision a5f6f126, both with the official modeling code unchanged. Appendix A lists the batches.

On the problem of Figure 4c, W = 16 already cuts the peak to 0.97 and W = 64 to 0.13. Keeping four attention-sink positions (Xiao et al., 2023) at full depth does nothing, and neither does a second stored plane at similar memory. The 48 reference trajectories were bitwise identical to batch 7’s, on another node.

5.1

5.2

Generation fidelity and the tail

Huginn: mostly a precision change, plus a tail. The preregistered statistic is KL(L0 ∥deepest)/KL(L0 ∥fp32) per bin of generated position with a prompt-paired bootstrap. On 48 confirmatory prompts the label is “exceeds a precision change”, through a single bin (ratio 13.8 [2.28, 21.2]) in which one problem carries 75% of the KL; other bins have ratios 1.6–3.3. Over all positions the median KL is lower for the deepest plane than for fp32 (6.4 × 10−5 vs. 8.2 × 10−5 ) but the mean is 5.7 times higher, because 1% of positions carry 74% of the KL (fp32: 21%; Figure 4a). The 9 positions with KL above 0.1 (fp32: 1) lie in 4 prompts, 5 of them in one GSM8K problem at nearby positions (Figure 4c). Only 2 of them have a reference top-2 probability margin below 0.1, where 8% of all positions lie: near-ties are over-represented (0.7 expected, 2.7×) but account for a minority of the tail. The largest value, 5.05, is one of those two. Raven: indistinguishable from a precision change. On the same 72 prompts, Raven’s deepest plane is “indistinguishable from a precision change” on both prompt sets (bin ratios 1.04–1.51), its largest KL is 0.020 (fp32: 0.014), no position exceeds 0.1, and it flips the top-1 token at 48 positions against 76 for fp32. We had predicted the opposite, a larger effect than on Huginn; the ripening model accounts for the difference after the fact (Section 3.2). RIPE windows remove most of Huginn’s tail. Batch 8 replayed 48 of these prompts through RIPE caches (Table 4, Figure 4b). The pre-registered count of tail positions falls from 9 to 1 at W = 64 (89% fewer; ratio 0.11 [0.00, 0.20], wide because four prompts carry all tail positions), and the largest KL from 5.05 to 0.13, below the fp32 replay’s 0.23.

Self-speculative decoding

Batch 8 ran a real greedy self-speculative decoder on Huginn: depth-4 drafts, four per block, and one 32-loop verification pass per block. With the full cache it accepts 3.34 tokens per verification on GSM8K and is 2.09× faster than sequential decoding; with the deepest plane, 4.47 and 2.58× (Sections 3.3–3.4 model both). With the deepest plane, batched verification depends on where blocks start (Section 4). The pre-registered test compares verification targets of batched and sequential deepestplane decoding at identical contexts, against the same comparison for the full cache, where the two are equal in exact arithmetic (an fp64 CPU test confirms it) and differ only by bf16 numerics. The ratio is 1.37 [1.20, 1.62], below the pre-registered threshold of 2 (label “indistinguishable from numerics”; we had predicted 3–30), and 28 of 32 batched trajectories equal the sequential ones. On Huginn the block dependence is therefore small; Proposition 1 removes it by construction. Using WaveFront’s released code (Ha & Kim, 2026) unchanged, batch 5 reproduced its Huginn numbers on GH200: speed-up 2.64× at acceptance 0.878 without sharing and 4.14× at 0.965 with one shared slot. 5.3

Fixed-memory throughput

Batches 6 and 8 placed WaveFront’s scheduler and draftthen-verify under a 95 GB budget at the measured acceptance rates (Figure 5c). The best configuration without sharing is draft-then-verify at B = 32 (218 tokens/s, batch 6) and B = 34 (253 tokens/s, batch 8); with one shared slot, WaveFront at B = 64 (487) and B = 256 (572), so sharing is 2.24× and 2.26× faster. The two batches ran

(a) deepest plane vs. fp32

(b) Huginn: RIPE windows

10−1

(c) Huginn: one GSM8K problem W =16 W =64

10−1

Huginn deepest Huginn fp32 Raven deepest Raven fp32

10−3

10−5

10−3

10−1

100

W =0 W =16 W =64 W =256 two planes fp32

10−3

10−5

101

10−3

10−3

KL x

fp32 W =0

KL

fraction of positions with KL > x

RIPE: Key–Value Caching for Looped Language Models

10−1

101

10−6

40

50

60

70

generated position

KL x

4 0.8

d=2

2

deepest plane full cache

0.6 1

2

3

draft position in block

4

3

d=4

2.5

8 2

2 1.5

deepest plane full cache

1

full cache (DtV) one shared slot (WaveFront)

600

tokens/s

d=8 8 d=4

(c) fixed memory (95 GB budget)

(b) speed-up model (extrapolated in γ)

(a) acceptance is flat in the chain 1

speed-up over sequential

P (accept | previous accepted)

Figure 4. Error of deepest-plane caches in generation (teacher-forced replay of full-cache trajectories). (a) Survival function of per-position KL, 7,130 Huginn and 7,647 Raven positions on the same 72 prompts. (b) RIPE windows on the 48 Huginn prompts of batch 8. (c) KL along the GSM8K problem that holds 5 of Huginn’s 9 tail positions.

400

200

0 1

2

3

4

5

6

7

8

100

draft tokens per block γ

101

102

batch size B

Figure 5. (a) Acceptance at each draft position given that earlier ones were accepted (batch 3, Huginn, GSM8K); flatness supports Eq. (7). (b) Eq. (8) with η = 1.07 (lighter: d = 2, darker: d = 8; dashed: full cache); markers: measured speed-ups of the real decoder at d = γ = 4 (batch 8). Curves beyond γ = 4 are extrapolations. (c) Throughput of WaveFront’s batched emulator at measured acceptance rates; filled: batch 6, open: batch 8 (another node, reported separately); lines: T (B) = B/(a + bB) fitted to batch 6, drawn up to the memory model’s Bmax at 95 GB (dashed).

on different nodes, where identical anchor cells differed by 2–12%, so they are not pooled. Two pre-registered memory predictions failed (the full cache would run out between B = 24 and 32, and at B = 38); the memory model of Section 3.5 reproduces both cells’ peak memory (B = 38 needs 99.4 GB: above the budget, within the device). On Huginn our deepest-plane cache, which needs the same memory as one shared slot, is not lossless relative to the full cache (Section 5.1); we did not check which plane WaveFront’s shared slot holds. 5.4

What did not help

Per-head position filtering. The starting plan also proposed keeping, per head, only the prefix positions that a shallow loop attends to. Batch 1 found cross-depth attention affinity real but mostly static (sink plus recent tokens). Batch 8 froze a per-head policy from batch 1 and tested it on 64 unseen AlpacaEval texts at a budget of 10% of the prefix: filtering costs 72 times more KL than deepest-plane storage, the two costs add (ratio 0.88), and the per-head policy is 1.88× worse than top-k for every head [1.65, 2.20]. Selecting

at loop 4 instead of 2 cuts the cost to 0.33 of its value, consistent with ripening. Sinks and a second plane. Neither removed any of Huginn’s tail positions (Table 4); a recent window did. 5.5

Reproducibility on GH200

Batch 4 found that the bf16 full-cache reference was not reproducible: a same-process rerun matched bitwise at only 30% of positions, with mean KL 1.6 × 10−4 , about two thirds of the bf16-to-fp32 floor. PyTorch’s deterministic mode did not remove it; batch 6b traced it to cuDNN’s scaled-dot-product attention (cuDNN 9.24). With that backend disabled, reruns are bitwise identical (100% of positions, across processes and nodes) and decoding is not slower; the deepest-plane labels and ratios were unchanged. Separately, WaveFront’s decoder and its autoregressive baseline produce identical texts on 31/40 problems without sharing, with cuDNN disabled or not, and our own full-cache batched and sequential decoders, equal in exact arithmetic, agree on only 20 of 32 trajectories in bf16. Computation shape alone flips bf16 trajectories at this rate, so a “lossless” claim

RIPE: Key–Value Caching for Looped Language Models

for speculative decoding in bf16 needs a stated reference computation.

6

R ELATED WORK

Looped models. Huginn (Geiping et al., 2025), Ouro (Zhu et al., 2025) and retrofitted recurrence (McLeish et al., 2025) train or convert models that iterate a core block. Ouro reports that keeping only the last loop’s KV is nearly lossless on its model; we measure when this holds and why it fails on Huginn. WISE (Wan & Chen, 2026) finds that attention routing stabilizes early in depth and reuses the early loops’ attention support; we find that prefix keys and values matter late and reuse the last loop’s planes. Both exploit convergence across depth, in opposite directions. Self-speculation in looped models. Huginn’s zero-shot selfspeculation (Geiping et al., 2025), LoopSpec’s pipelined branches (Cho et al., 2026), WaveFront’s diagonal schedule with cross-recurrence KV sharing (Ha & Kim, 2026), Mature-Spec’s mature values for shallow drafts (Li et al., 2026) and LoopAhead (Liu et al., 2026). Self-speculation by layer skipping in standard models is Draft&Verify (Zhang et al., 2023); the acceptance model is that of Leviathan et al. (2022) and Chen et al. (2023). KV compression. Cross-loop KV compression and quantization (Neill & Reid, 2026; Kim et al., 2026) and memoryefficient looped inference (Yang & Liu, 2026); for standard transformers, attention sinks and streaming windows (Xiao et al., 2023), quantization (Liu et al., 2024) and paging (Kwon et al., 2023). RIPE’s recent window plays the role of a streaming window along depth rather than along the sequence.

7

L IMITATIONS AND CONCLUSION

Limitations. Two models, one trained recurrent and one retrofitted; the ripening explanation and the ripeness test are post hoc. RIPE with W > 0 was evaluated only by teacher-forced replay on Huginn; we did not run it inside the speculative decoder, nor deepest-plane drafts against RIPE verification. The speed-ups come from a batch-1 eager framework and the throughput frontier from WaveFront’s emulator; Eq. (8) was timed at one (d, γ). The tail statistics rest on four prompts. Greedy decoding, GSM8K and AlpacaEval only; per-head filtering at one budget and one selection rule. Conclusion. In looped language models the prefix keys and values that matter are those read by the last loops, and how fast a model’s planes ripen decides whether the deepest plane alone is enough. Simple models of sensitivity, ripening, acceptance, cost and memory account for what nine pre-registered batches measured; the memory and accep-

tance models predict held-out measurements. RIPE keeps the deepest plane everywhere except a short recent window: at 11.9% of the full cache’s memory on Huginn it removes most of the deepest plane’s error tail, and it keeps batched verification consistent with sequential decoding.

R EFERENCES Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023. Cho, S., Cui, L., Kim, S., Han, D., and Han, I. LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers. arXiv:2609.17184, 2026. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168, 2021. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475, 2024. Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv:2502.05171, 2025. Ha, H. and Kim, J.-J. WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models. arXiv:2609.23033, 2026. Kim, H., Lee, J., Cho, S., Han, D., Han, I., and Kim, S. ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals. arXiv:2610.10381, 2026. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180, 2023. Leviathan, Y., Kalman, M., and Matias, Y. Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192, 2022. Li, G., Su, Z., Yu, H., Jiang, J., Ren, T., Li, Z., et al. Shallow Queries, Mature Values: Depth-Asynchronous SelfSpeculation for Looped Transformers. arXiv:2609.34538, 2026. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv:2402.02750, 2024.

RIPE: Key–Value Caching for Looped Language Models

Liu, Z., Xue, N., Bao, R., Xu, Y., Cao, X., Yao, J., and Chen, Z. LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models. arXiv:2610.21594, 2026. McLeish, S., Li, A., Kirchenbauer, J., Kalra, D. S., Bartoldson, B. R., Kailkhura, B., et al. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. arXiv:2511.07384, 2025. Neill, J. O. and Reid, F. Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers. arXiv:2607.15456, 2026. Wan, K. and Chen, C. Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models. arXiv:2609.27373, 2026. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453, 2023. Yang, W. and Liu, S. FlashLoop: Fast and MemoryEfficient Looped Transformers via Lazy Updates. arXiv:2609.29812, 2026. Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. arXiv:2309.08168, 2023. Zhu, R.-J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., et al. Scaling Latent Reasoning via Looped Language Models. arXiv:2510.25741, 2025.

RIPE: Key–Value Caching for Looped Language Models

A

BATCHES

Each batch had a pre-registration with decision rules and predictions frozen before the run, a frozen file table (SHA-256), on-node CPU self-tests, a separate unscored Nsight Compute pass, and an archive verified at both ends (Table 5). Table 5. Pre-registered batches, one GH200 each.

B

Batch

Question

Pre-registered label

1 2 3 4 5 6, 6b 7 8 9

cross-depth attention affinity which loops’ prefix KV matter (Huginn) deepest plane in generation; acceptance reference asymmetry; confirmation; tail WaveFront baseline reproducibility; fixed-memory frontier batches 3–5 with cuDNN off batched verification; windows; per-head filtering; frontier Raven-Llama: region test; deepest plane vs. fp32

GSM8K trivial; AlpacaEval intermediate deep KV matters exceeds precision change asymmetry negligible; exceeds speed-ups in band cuDNN is the source; shared ≥ 1.5× reproducible; unchanged as numerics; window removes most; per-head worse deep KV matters; indistinguishable

P REDICTION LEDGER

Table 6 lists every pre-registered prediction and its outcome, including the ones that failed. Table 6. Pre-registered predictions and outcomes (all reported, including failures). Batch

Prediction

Outcome

1 1 1

coverage rises with r head-median coverage at r = 4 in 0.6–0.85 shallow top-k above the static set by a margin

2 2 3 3 4 4 4 5 5 6 6 6 6 6 6 6b 7 7 7 8 8 8 8 8 8 8 8 8 8 9 9 9

deep KV matters at matched size compress-s4 first-position agreement ≥ 0.95 deepest-plane cache indistinguishable from a precision change deepest-plane cache raises acceptance reference asymmetry changes the ratio by > 10% confirmatory set “intermediate” bf16 reruns bitwise identical speed-up 2–3× unshared, 3.5–5× shared WaveFront token-lossless (match ≥ 0.99) bf16 rerun noise is real deterministic mode removes it WaveFront still not lossless in deterministic mode speed-up falls with batch size shared ≥ 1.5× best lossless at 95 GB unshared runs out of memory between batch 24 and 32 source is cuDNN attention reruns reproducible with cuDNN off ratios rise to 2.5–4 with cuDNN off WaveFront 40/40 with cuDNN off batched deepest-plane verification ̸= sequential (ratio 3–30) batched verification closer to the full cache tokens per verification 3.0–3.6 (full), + ≥ 0.5 (shared) speed-up 1.5–2.3× (full), 2.0–3.0× (shared) window 64 removes most tail positions two planes remove 20–50% per-head ≈ all top-k filtering costs > 3× sharing; costs add unshared OOM at batch 38 shared/lossless ratio 2.4–2.8 Raven: deep KV matters, deep coefficient ≥ 3 (bf16) Raven: deepest-plane first-position agreement ≥ 0.95 Raven: deepest plane “exceeds”, more than on Huginn

held GSM8K above (0.91), AlpacaEval held direction held; magnitude too small (“trivial” on GSM8K) held (e6.7 bf16, e15.0 fp32) held (0.990) failed (“exceeds”, via the tail) held (3.31→4.33 at d = 4) failed (0.5–5%) failed (“exceeds”, one prompt) failed (30% of positions) held (2.64×, 4.14×) failed (31/40 texts identical) held failed failed by rule (8/8; chance ≈13%) held held (2.24×) failed (32 fits) held held failed (unchanged) failed (31/40, 31/40) failed (1.37, numerics) failed (0.94) held (3.34, 4.47) held (2.09, 2.58) held (89%) failed (none) failed (1.88× worse) held (72, 0.88) failed (fits) failed (2.26) held (3.8) held (0.98) failed (indistinguishable, no tail)

RIPE: Key–Value Caching for Looped Language Models

C

F ITTED PARAMETERS

All fits are produced by one script from the batch summaries and raw per-position records, and every number in this paper is generated from those files; none is typed by hand (Tables 8, 9; Figure 6). Table 7. Ripening fits (keys; values in parentheses). ρ and c from fp32, loops 8–31; ∆∞ fitted to bf16 with ρ, c fixed. ρ half-life (loops) ∆∞ (bf16) deep-loop ∆, bf16 deep-loop ∆, fp32 max | log residual|, fp32

Huginn

Raven

0.88 (0.87) 5.2 (5.1) 0.0072 (0.0115) 0.0116 (0.0192) 0.0097 (0.0162) 0.28

0.81 (0.81) 3.3 (3.3) 0.0048 (0.0101) 0.0047 (0.0098) 1.6 × 10−4 (3.7 × 10−4 ) 0.51

Table 8. Throughput fits T (B) = B/(a + bB) per configuration of WaveFront’s batched emulator (Section 3.5): saturation throughput T∞ = 1/b (tokens/s), half-saturation batch B1/2 = a/b and largest relative residual. Batch-8 series cover a narrow batch range and ran on another node; their parameters are descriptive only. Batch

Slots

Scheduler

α

B range

Cells

T∞

B1/2

Residual

6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 8 8 8 8 8 8 8 8 8 8

0 0 0 0 0 1 1 1 1 1 4 4 4 4 4 0 0 0 0 0 1 1 1 1 1

AR DtV DtV WaveFront WaveFront AR DtV DtV WaveFront WaveFront AR DtV DtV WaveFront WaveFront AR DtV DtV WaveFront WaveFront AR DtV DtV WaveFront WaveFront

— 0.878 0.965 0.878 0.965 — 0.878 0.965 0.878 0.965 — 0.878 0.965 0.878 0.965 — 0.878 0.965 0.878 0.965 — 0.878 0.965 0.878 0.965

1–32 1–32 1–32 1–32 1–32 1–64 1–64 1–64 1–64 1–64 1–64 1–64 1–64 1–64 1–64 32–38 32–38 32–38 32–38 32–38 64–256 64–256 64–256 64–256 64–256

5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 4 4 4 4 4 5 5 5 5 5

394 449 612 244 310 461 494 656 535 689 395 473 539 521 644 372 380 503 219 280 308 443 555 483 605

69 33 34 13 11 83 38 39 30 27 69 35 29 29 24 56 18 18 8 7 28 24 20 18 14

4% 2% 3% 3% 6% 4% 3% 6% 4% 6% 6% 8% 10% 9% 8% 2% 2% 2% 0% 0% 2% 1% 2% 2% 2%

RIPE: Key–Value Caching for Looped Language Models Table 9. Acceptance model check on the batch-3 simulation (γ = 4): mean conditional acceptance αd , its spread across the four draft positions, number of blocks, and tokens per verification predicted by Eq. (7) vs. simulated. Rejection 1 − αd ≈ kρda fits k = 0.62, ρa = 0.76 (full cache) and k = 0.42, ρa = 0.66 (deepest plane). Cache

d

αd

Spread

Blocks

τ model

τ sim.

full cache full cache full cache deepest plane deepest plane deepest plane

2 4 8 2 4 8

0.637 0.805 0.932 0.800 0.928 0.984

0.041 0.062 0.021 0.041 0.008 0.005

2280 1676 1281 1619 1276 1139

2.46 3.40 4.37 3.36 4.33 4.84

2.44 3.31 4.34 3.41 4.33 4.85

∆V r

100

deep loops

10−2

10−4

Huginn fp32 Raven fp32 5

Huginn bf16 Raven bf16 10

15

20

25

30

loop r

Figure 6. Ripening of prefix values, as Figure 2c shows for keys: fp32 (filled), bf16 (open), geometric fit to fp32 over loops 8–31 (solid) and the same fit with one bf16 floor (dotted).

Related documents

• Prefix Keys and Values Matter Late: RIPE Caching for Self-Speculative Decoding in Looped Language Models#1457590
• Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions#430488
• Decision-Aware Suffix Prediction and Reasoning of Business Processes#973771
• Comprehension precedes production of a complex call sequence#405045
• Truncation Samplers Are Nested: Comparing Decoding Rules Is a Measurement Problem#1326820