Shallow Encoding Emits Deep Memories

2026-11-03

Shallow Encoding Emits Deep Memories Mingyuan Ma* , Yangmin Li* , Harris Nover, Zheyu Fu, Po-Han Huang, Alex Settle, Ashwin Nanjappa† NVIDIA

arXiv:2611.06084v1 [cs.LG] 4 Nov 2026

Project page

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers, so every layer keeps a memory of earlier tokens: a key–value cache or a fixed-size recurrent state. Serving them spends more compute and high-bandwidth memory (HBM) on these memories than the output requires. The prefill runs every layer for every prompt token, although only the last token’s prediction is used, and every decode step loads and stores whole recurrent states, which on Qwen3.8-Flash-Next take as many bytes per request as the attention cache of thousands of tokens. We find two properties of trained hybrids that locate this excess. First, writes are easier than reads: writing a deep layer’s memory of a token needs little of the deep computation that predicting the next token needs. Trained copies of the deep layers’ write paths, which we call emitters, therefore write a prompt’s deep memories from the residual stream of a shallower anchor layer, and all prompt tokens but the last run as few as 63% of the layers. Second, recurrent states have a state sink: most of their energy lies in a rank-one term along a fixed direction of each head, and the rest is a low-rank working memory. SEED (Shallow Encoding Emits Deep Memories) builds on both: it makes the prefill shallow in depth and the decode state shallow in representation. Each recurrent state is stored as its sink plus low-rank factors, which the Gated DeltaNet, Kimi Delta Attention and Mamba-2 updates extend exactly; we prove that the open-loop error of the periodic compressions back to low rank is bounded independently of their number. SEED keeps the original weights frozen, compresses no attention cache, and runs all layers for every generated token. On six checkpoints of five production hybrids of 30B to over 300B parameters, it is nearly lossless on eleven official exams. Served with SGLang on NVIDIA GB300 GPUs on an agentic-coding trace, it raises the per-GPU throughput of Qwen3.8-Flash-Next 1.2–1.4×. Neither property was designed into these models.

Keywords: LLM inference, hybrid models, linear attention, recurrent state, prefill, algorithm–system co-design

1. Introduction Hybrid language models keep a few softmax-attention layers and make the other token-mixing layers recurrent (Dao & Gu, 2024; Yang et al., 2025b; Kimi Team et al., 2025), and production models now take this form (Qwen Team, 2025; NVIDIA et al., 2025b; MiniMax et al., 2025). Every layer keeps a memory of earlier tokens. An attention layer keeps a key–value (KV) cache, one key and one value per token. A linear-attention or state-space layer folds each token into a fixed-size recurrent state, one 𝑑𝑘 × 𝑑𝑣 matrix per head. Serving runs in two phases with different bottlenecks (Patel et al., 2024; Zhong et al., 2024; Qin et al., 2025). The prefill processes all 𝑁 prompt tokens together and is compute-bound. The decode generates one token per request per step; each step reads the weights and the memories of every resident request, so it is bound by high-bandwidth memory (HBM) bandwidth and by how many requests fit next to the weights (§2). Both phases spend more compute and HBM on these memories than the output requires. In the prefill, only the last prompt token predicts the first reply token; the other 𝑁 − 1 run all 𝐿 layers of the model only to write their memories, and agentic workloads make these prompts long (DeepSeek-AI et al., 2026). In the decode, every recurrent head loads its whole state from HBM and stores it back at every step, and fixed-size does not mean small: on Qwen3.8-Flash-Next, the states of one request Correspondence: Ashwin Nanjappa† . * Equal contribution: Mingyuan Ma and Yangmin Li co-led the project, and their order was decided at random. © 2026 NVIDIA. All rights reserved.

Shallow Encoding Emits Deep Memories

original model

SEED

outputs discarded

memories

y1

emitters replace layers k..L−1

memories

layer L−1

···

K, V cache

layer L−1

emitter

K, V cache

layer k+1 layer k

··· ···

state S K, V cache

layer k+1 layer k

emitter emitter

K, V cache

ĥ(k) anchor layer k−1

prompt code E → z → D h(k)

layer k−1

···

state S

layer 3 layer 2 layer 1 layer 0

··· ··· ··· ···

K, V cache state S state S state S

x1

x2

· · · xN −1

layer 3 layer 2 layer 1 layer 0

xN

x1

(b) Decode — shallow in representation yt+1

all L layers

x2

shallow state prefix storage

···

shallow state

··· ··· ··· ···

K, V cache

· · · xN −1

y1

xN : all L layers

(a) Prefill — shallow in depth

shallow state shallow state shallow state

xN

yt+1

state S

S dk ×dv

yt original layer, unchanged

attention cache: read cache entries, append one read: ot = S ⊤ qt write: rewrite all of S size: dk dv numbers per head, 16,384 for dk =dv =128 skipped

added by SEED

recurrent state

attention cache, uncompressed: read cache entries, append one shallow state S all L read: ot = (qt⊤ a) v̄ + V (U ⊤ qt ) layers write: update a, decay U , ⊤ v̄ + add a column (exact) sink V⊤ every W steps: compress to rank R a U factors size at rank R: (R+1)d + Rd v k yt numbers per head, 4,224 for R=16 attention cache × output discarded

Figure 1 | SEED keeps a shallow encoding of a hybrid model’s memories in each phase. Left: original model; right: SEED; layer stacks as in Qwen3.8-Flash-Next, with three recurrent layers per attention layer. (a) Prefill, shallow in depth: 𝑥1 ..𝑥𝑁 −1 run layers 0..𝑘−1 only. Emitters, trained copies of the deep layers’ write paths, write every deeper memory from ℎ(𝑘) , the output of anchor layer 𝑘−1, directly or through an optional prompt code. At the end of the prefill, the states left by 𝑥1 ..𝑥𝑁 −1 are compressed to the shallow form of (b); 𝑥𝑁 runs all 𝐿 layers. (b) Decode, shallow in representation: each stored recurrent state is its sink 𝑎¯ 𝑣 ⊤ plus rank-𝑅 ⊤ factors 𝑈 𝑉 . A step updates the sink’s coefficient 𝑎, decays 𝑈 and appends one column to each factor, which keeps the update exact; every 𝑊 steps, the factors are compressed back to rank 𝑅. Every generated token runs all 𝐿 layers, and no attention cache is compressed.

take 113.2 MB, as much as the attention cache of about 4.5K tokens (Table 2). We show that this excess is structural. Trained hybrids have two properties, one per phase, that make their memories far simpler than the computation and storage that serve them. We find both in all five production models we study, Qwen3.8-Flash-Next (Qiu et al., 2026b), Nemotron-3.5Lightning (NVIDIA, 2026c), Kimi-Linear (Kimi Team et al., 2025), GLM-5.3-Flash (Z.ai, 2026; GLM-5-Team et al., 2026) and Nemotron-3.5-Super (NVIDIA, 2026b), whose recurrent layers are Gated DeltaNet, Kimi Delta Attention and Mamba-2. Writes are easier than reads. A layer writes its memory of a token through its write path, which reads only the residual stream that enters the layer. Predicting the next token, a read, needs the attention and feed-forward blocks of every deep layer. This write–read asymmetry is large. We call 𝑘 the anchor depth and layer 𝑘 − 1, at 63–89% of the depth, the anchor layer; its output is the residual stream ℎ(𝑘) (Elhage et al., 2021). Fed into the deep layers’ own write paths without any training, ℎ(𝑘) moves the next-token distribution by a Kullback–Leibler (KL) divergence below 0.1 nats per token. An untrained early exit at the same depth, the output head applied to ℎ(𝑘) (the logit lens; nostalgebraist, 2020), is off by 6.8–11.1 nats per token (Figure 2). We trace the gap to the two roles of a deep layer. At a prompt token, its only lasting output is the memory it writes; the rest of its computation reaches the reply only through the inputs of later writes (§3).

2

Shallow Encoding Emits Deep Memories

Table 1 | SEED in the two serving phases. In each phase, a property of trained hybrids lets SEED keep an encoding of the memories that is shallow along the phase’s bottleneck, compute in the prefill and HBM in the decode, and emit from it what the model reads. Prefill (compute-bound)

Decode (HBM-bound)

Property

writes are easier than reads: a deep layer’s write needs little of the deep computation (§3)

the state sink: a rank-one term along a fixed direction of each head holds most of the states’ energy; the rest is low-rank (§4)

Model reads

caches and states of layers 𝑘..𝐿−1 for each prompt token

read-out and update of each 𝑑𝑘 ×𝑑𝑣 recurrent state

Shallow encoding

in depth: prompt tokens 𝑥1 ..𝑥𝑁 −1 run only layers 0..𝑘−1; their output ℎ(𝑘) stands in for layers 𝑘..𝐿−1

in representation: the shallow state, the sink 𝑎¯ 𝑣 ⊤ plus rank-𝑅 factors 𝑈 , 𝑉

Emitted by

emitters, trained copies of the deep layers’ write paths, write every deep cache and state from ℎ(𝑘)

the factors, which give the read-out and the exact update; compressed to rank 𝑅 every 𝑊 steps

Saves

prompt tokens but the last run as few as 63% of the layers and 71% of the parameter FLOPs

the state is 2.0–3.9× smaller after a compression, 1.3–2.6× on average

States

every stored recurrent state is shallow: the states that 𝑥1 ..𝑥𝑁 −1 leave are compressed to the sink plus rank-𝑅 factors at the end of the prefill, and the decode keeps this form

Unchanged

the weights; no attention cache is compressed or quantized; the last prompt token and every generated token run all 𝐿 layers

Recurrent states have a sink. In the median Gated DeltaNet head of Qwen3.8-Flash-Next, one direction holds 85% of the state’s energy, its squared Frobenius norm. This direction is close to the head’s mean value 𝑣¯. The values’ component along 𝑣¯ carries most of their energy, and the linear recurrence accumulates it into a rank-one term 𝑎¯ 𝑣 ⊤ , the state sink, whose coefficient follows the head’s own recurrence (Lemma 1). Pooled over heads, the sink holds 69–99.7% of the state energy on all five models. Like the attention sink of softmax layers (Xiao et al., 2024), it forms regardless of the prompt, on Qwen3.8-Flash-Next even on random tokens; unlike it, it lies along a direction of the state, not at a position in the sequence. The sink holds the energy but not the prompt. After the head’s normalization, it acts as a bias and as the gain of the rest, and replacing it by its mean over other prompts costs only 0.002 nats per token. What the head has read lies in the orthogonal content, a short working memory: on Qwen3.8-Flash-Next, a median of 6 directions hold 90% of its energy (§4). The model’s own update preserves the form. A compact state is useful only if it survives every update without growing back to 𝑑𝑘 × 𝑑𝑣 . A sink plus low-rank factors does: the form 𝑎¯ 𝑣 ⊤ + 𝑈 𝑉 ⊤ is closed under the Gated DeltaNet, Kimi Delta Attention and Mamba-2 updates, each step updating the sink’s coefficient and appending one column to each factor, an exact rank-one update of the content (Lemma 2). The only approximation is a compression of the factors back to rank 𝑅, once at the end of the prefill and then every 𝑊 steps; it never touches the sink. In the open loop, where the compressed and the exact recurrence see the same inputs, its errors do not compound: the accumulated error stays within the largest single compression error times the head’s decay horizon, however many compressions occur (Corollary 4). In the closed loop, where errors also change later inputs, the measured drift grows only 1.2–1.4× from the first to the last 4K of 16K decode steps while the number of compressions grows 4×, on the four models followed this long (§6.1).

3

Shallow Encoding Emits Deep Memories

SEED. SEED, Shallow Encoding Emits Deep Memories, builds one part on each property (Figure 1, Table 1). In the prefill, its encoding is shallow in depth. The first 𝑁 − 1 prompt tokens run only layers 0..𝑘−1, and emitters, trained copies of the deep layers’ write paths, write every deep attention cache and recurrent state from ℎ(𝑘) . An optional compact prompt code of ℎ(𝑘) , read by the emitters, can stand in for the deep memories in a prefix cache. In the decode, its encoding is shallow in representation. From the end of the prefill on, every recurrent state is stored as its sink plus rank-𝑅 factors instead of a dense 𝑑𝑘 × 𝑑𝑣 matrix, and every step computes its read-out and its exact update from them. Each encoding is shallow along the bottleneck of its phase: the prefill’s saves FLOPs, and the decode’s saves HBM bytes. All prompt tokens but the last run as few as 63% of the layers and 71% of the parameter FLOPs, and the state is 2.0–3.9× smaller after each compression. The original weights stay frozen, no attention cache is compressed, and the last prompt token and every generated token run all 𝐿 layers. Every setting follows from one set of criteria, applied to inexpensive measurements of the original model (§3.1). Evidence. We apply SEED to six checkpoints of five production hybrids of 30B to over 300B parameters: Qwen3.8-Flash-Next in bf16 and NVFP4, Nemotron-3.5-Lightning, Kimi-Linear, GLM5.3-Flash and Nemotron-3.5-Super. Each is compared with its original model question by question, with the same sampling seeds. Under the official graders, SEED is nearly lossless: 63 of the 66 checkpoint–exam cells of eleven exams lie within two standard errors of the original model or above it, and the repetitions on which no design choice rested agree with them (§6). Its answers change about as often as those of a second sample of the original model. The next-token distributions stay close. With the exact prefill, the shallow state drifts 0.002–0.007 nats per token; NVIDIA’s official NVFP4 checkpoints drift 1.9–3.5× as much on the three models that have one (§6.1). Served with SGLang on NVIDIA GB300 GPUs on an agentic-coding trace, Qwen3.8-Flash-Next with SEED delivers 1.2–1.4× the original model’s throughput per GPU, both peak to peak and at an equal per-user rate of 40 tokens/s: shallow prefill relieves the prefill instance in disaggregated serving, and shallow states relieve the recurrent-state pool that limits the batch in aggregated serving (§7). Neither property was designed into these models. Contributions. • Two properties of trained hybrid models: the write–read asymmetry, for recurrent states as well as attention caches, and the state sink, each measured on five production models (§§3–4). • SEED, which turns both into a serving method for frozen checkpoints, with a proof that the Gated DeltaNet, Kimi Delta Attention and Mamba-2 updates preserve its state form exactly and an open-loop error bound that does not grow with the number of compressions (§5). • An evaluation of six checkpoints against their original models on official exams, decode drift and serving throughput: SEED matches the original models’ exam scores, and Qwen3.8-Flash-Next with SEED delivers 1.2–1.4× the throughput per GPU (§§6–7). We will release the code, the component sets and their training data.

2. Setting Cost model. We count the cost of each phase with a roofline model. A prefill chunk of 𝑛 tokens reads the 𝑀w bytes of weights once and spends 𝐹 FLOPs per token, two per parameter the token meets; it is compute-bound once its arithmetic intensity 𝑛𝐹/𝑀w exceeds the GPU’s ridge point, its FLOP rate divided by its HBM bandwidth. 𝐹 omits attention-score FLOPs, which SEED skips 4

Shallow Encoding Emits Deep Memories

in the deep attention layers, and recurrence-scan FLOPs, which it keeps, so on long prompts 𝐹 understates its saving. For a chunk without the last prompt token, SEED lowers both 𝐹 and the weight bytes read. A decode GPU with 𝑀HBM bytes of HBM holds the weights and 𝐵 resident requests; at context length 𝑇 , each occupies 𝑀r (𝑇 ) bytes of caches and states, and each decode step moves 𝑄r (𝑇 ) bytes for it: its recurrent states, read and written in full, and the cache entries the step reads. These are all entries under dense attention; under the sparse attention of Qwen3.8-Flash-Next and GLM-5.3-Flash, they are 2,048 selected tokens per layer plus the indexer keys of all tokens, pooled over blocks of four. A step also reads the weights the batch touches, 𝑀wstep ≤ 𝑀w (on a mixture-of-experts model, only the experts the batch is routed to). Hence 𝑡step ≳

𝑀wstep + 𝐵 𝑄r (𝑇 ) , HBM bandwidth

𝐵 ≤

𝑀HBM − 𝑀w , 𝑀r (𝑇 )

(1)

and the tokens per second per GPU, 𝐵/𝑡step , depend on a request’s memories twice: through capacity, as their size caps the batch, and through bandwidth, as their traffic adds to every step. In both, the attention cache grows with 𝑇 ; the recurrent state, which SEED shrinks, does not. A prefix cache, which keeps the memories of cached prompts for reuse, competes for the same HBM. Models. The five models have 30B to 320B parameters (Appendix C) and differ in their recurrent layers, attention and weight precision. Qwen3.8-Flash-Next (Qiu et al., 2026b) is a 48-layer mixtureof-experts model in which every fourth layer is block-sparse softmax attention and the other three are Gated DeltaNet layers (Yang et al., 2025b) of 48 value heads. Each head keeps an fp32 state 𝑆𝑡 of 𝑑𝑘 × 𝑑𝑣 (key by value dimension, 𝑑𝑘 = 𝑑𝑣 = 128), updated by the gated delta rule ⊤ 𝑆𝑡 = 𝑔𝑡 (𝐼 − 𝛽𝑡 𝜅𝑡 𝜅⊤ 𝑡 ) 𝑆𝑡−1 + 𝛽𝑡 𝜅𝑡 𝑣𝑡 ,

𝑜𝑡 = 𝑆𝑡⊤ 𝑞𝑡 ,

𝑦𝑡 = RMSNorm(𝑜𝑡 ) ⊙ sigmoid(𝜁𝑡 ),

(2)

−1/2

with keys 𝜅𝑡 of unit norm and queries 𝑞𝑡 of norm 𝑑𝑘 in R𝑑𝑘 , values 𝑣𝑡 and read-outs 𝑜𝑡 in R𝑑𝑣 , a write strength 𝛽𝑡 ∈ (0, 1), a decay 𝑔𝑡 ∈ (0, 1), a per-head RMSNorm (Zhang & Sennrich, 2019) and, in this model, a sigmoid output gate with input 𝜁𝑡 . Qwen3.8-Flash-Next’s sparse attention caches 2,112 bytes per token and layer, and its residual stream holds 10,240 numbers per token (four parallel streams). Nemotron-3.5-Lightning (NVIDIA, 2026c) has 52 layers: 23 Mamba-2 layers (Dao & Gu, 2024) of 64 heads with a 64 × 128 state each, 6 attention layers and 23 stateless expert layers. Nemotron-3.5-Super (NVIDIA, 2026b) has 88 layers of the same kinds: 40 Mamba-2 layers of 128 heads with 64 × 128 states, 8 attention layers and 40 expert layers whose 512 routed experts compute in a 1,024-number latent space; we use its language model without the vision encoder. Kimi-Linear (Kimi Team et al., 2025) has 27 layers: 20 Kimi Delta Attention layers of 32 heads, each with a 128 × 128 state under a gated delta rule with per-channel decay, and 7 multi-head latent attention (MLA) layers (DeepSeek-AI et al., 2024) that cache 576 numbers per token. GLM-5.3-Flash (Z.ai, 2026; GLM-5-Team et al., 2026) has 34 Kimi Delta Attention layers (Kimi Team et al., 2025) of 64 heads with 128 × 128 states and 11 DeepSeek sparse attention layers (DeepSeek-AI et al., 2025), each caching a 512-number latent and a pooled indexer key (1,057 bytes per token and layer) and attending to 2,048 selected tokens. Its residual stream is four streams of 4,096 mixed by manifold-constrained hyper-connections (Zhu et al., 2025; Xie et al., 2025). Its expert and dense feed-forward weights and most sparse-attention projections are fp8; its Kimi Delta Attention layers stay bf16. Each checkpoint gets one component set: the emitters, prompt code and state settings that SEED adds to it. We treat NVIDIA’s NVFP4 release of Qwen3.8-Flash-Next, Qwen3.8-Flash-Next-NVFP4, which stores its routed experts in the four-bit NVFP4 format (NVIDIA et al., 2025a), as a separate checkpoint with its own component set. Appendix C gives the remaining dimensions. 5

Shallow Encoding Emits Deep Memories

Table 2 | SEED’s recurrent state is 2.0–3.9× smaller right after a compression, and its prefill runs fewer layers; attention caches stay uncompressed. Bytes of one request (fp32 states, bf16 attention caches; MB = 106 bytes), for the representation, not an engine’s allocation. SEED’s state, the factors plus the explicit sink, is at rank 𝑅 after a compression, 𝑅+𝑊/2 on average (the midpoint of a compression window; resident and traffic rows) and 𝑅+𝑊 at the peak, just before a compression. 4K+4K: a 4,096-token prompt after 4,096 generated tokens; 32K+8K: a 32,768-token prompt after 8,192. Traffic counts every state as read and written in full at every step, SEED’s factors included, although the update of §5.1 rescales 𝑈 , updates the sink’s coefficient and leaves 𝑉 unchanged apart from the appended column. Qwen3.8-Flash-Next and GLM-5.3-Flash read 2048 selected tokens per sparse layer and step, plus the indexer keys of all tokens, pooled over blocks of four (GLM-5.3-Flash: in the model’s own fp8); the NVFP4 Qwen3.8-Flash-Next checkpoint has the same caches and states. Short-convolution windows (0.8–5.0 MB per request, equal in both columns) are not included. FLOPs: 2 per parameter that a token meets; compression per generated token in fp64 and in fp32 GFLOP (Appendix F). Qwen3.8Flash-Next

Nemotron-3.5Lightning

Orig. SEED

Orig.

SEED

State at 𝑅 (MB) mean, 𝑅+𝑊/2 (MB) peak, 𝑅+𝑊 (MB)

113.2 113.2 113.2

29.2 43.4 57.5

48.2 48.2 48.2

Attn. cache / token (B) Resident, 4K+4K (MB) Resident, 32K+8K (MB)

25,344 25,344 321 251 1,151 1,081

KimiLinear

GLM-5.3Flash

Nemotron-3.5Super

Orig. SEED

Orig. SEED

Orig.

SEED

18.8 27.9 36.9

41.9 41.9 41.9

21.3 31.8 42.3

142.6 142.6 142.6

72.4 108.1 143.7

167.8 167.8 167.8

65.5 97.0 128.5

6,144 99 300

6,144 78 280

8,064 108 372

8,064 98 362

11,627 11,627 238 203 619 584

8,192 235 503

8,192 164 433

State traffic / step (MB) Step traffic, 4K+4K (MB) Step traffic, 32K+8K (MB)

226 283 308

87 143 168

96 147 348

56 106 307

84 150 414

64 130 394

285 311 323

216 242 254

336 403 671

194 261 530

Prefill layers run Prefill / prompt token (GFLOP) Compression, fp64 (GFLOP) fp32 (GFLOP)

48 12.1 – –

31 9.1 0.18 0.49

52 5.8 – –

33 4.1 0.11 0.21

27 5.5 – –

19 4.1 0.10 0.16

45 32.2 – –

31 23.5 0.35 0.55

88 23.4 – –

78 21.3 0.40 0.74

Memory per request and accounting rules. Table 2 gives the bytes of one request at the precision of each model’s reference implementation; the recurrent-state bytes do not depend on the context length. Three rules fix what we count. First, the original weights are frozen, and every original component (weights, attention caches, arithmetic) keeps its precision: SEED compresses no attention cache, its prompt code is the only quantized component it adds, and the exams use no speculative decoding. Second, only the emitters and the prompt code are trained, on public text, synthetic retrieval documents, public prompts and the frozen model’s own responses to public prompts, the kinds of data that published distillation recipes use (Bercovich et al., 2025; Wang et al., 2024) (Appendix D). Third, accuracy is measured on official exams, paired with the original model (§6).

3. Shallow in Depth: The Write–Read Asymmetry Writing a prompt token’s deep memories needs far less of the deep computation than predicting the next token. Fed the anchor layer’s output ℎ(𝑘) , at 63–89% of the depth, the deep layers’ own write paths move the next-token distribution by a KL below 0.1 nats per token without any training, while an untrained early exit, the output head applied to ℎ(𝑘) , is off by 6.8–11.1 nats per token. SEED’s prefill rests on this asymmetry; its risk is retrieval, which a copy probe guards against (§3.1).

6

Shallow Encoding Emits Deep Memories

(a) Qwen3.8-Flash-Next KL to the original (nats / token)

(c) at depth k, untrained

(b) Kimi-Linear

k = 31

k = 19

10

Qwen3.8-Flash-Next k = 31

Nemotron-3.5-Lightning k = 33

1

Kimi-Linear k = 19 0.1 GLM-5.3-Flash k = 31 0.01

Nemotron-3.5-Super k = 78

0.001 0

20

40

skipped layer (of 48)

0

10

20

skipped layer (of 27)

layer skipped for the prompt tokens (memory still written) layer skipped for the generated tokens

×141 0.079

11.1

×82 0.093

Qwen3.8-Flash-Next k = 31 Nemotron-3.5-Lightning k = 33

7.7

×208 0.036

7.5 ×78

0.087

6.8

×198 0.034

6.8

0.01 0.1 1 10 KL to the original (nats / token)

deep write paths fed h (k) output head on h (k)

(d) copying, untrained

Kimi-Linear k = 19 GLM-5.3-Flash k = 27 GLM-5.3-Flash k = 31 Nemotron-3.5-Super k = 47 Nemotron-3.5-Super k = 58 Nemotron-3.5-Super k = 69 Nemotron-3.5-Super k = 78 0

0.5

1

copies kept

Figure 2 | Writes are easier than reads in the hybrids we serve. (a, b) Per-token KL from the original model’s next-token distribution when one layer is skipped only for the prompt tokens, its memory still written (green), or only for the generated tokens (black); the deepest layers matter most for the generated tokens because they produce the next-token distribution. Dashed: the anchor depth. (c) KL at the anchor depth without training: the deep layers’ write paths fed ℎ(𝑘) (green) against an untrained early exit, the output head applied to ℎ(𝑘) (black), with the ratio of the two KLs. (d) The risk is retrieval: identifiers copied by the untrained shallow prefill, as a share of the original model’s; hollow: below three quarters. Protocols: Appendix A.

Shallow prefill. A prefill runs every layer for every prompt token but uses only the last token’s output. For the other tokens, the deep layers’ only lasting result is the deep memories: the keys and values that each deep attention layer caches and the update that each deep recurrent layer folds into its state. SEED keeps these memories and replaces the rest of the deep computation. The first 𝑁 − 1 prompt tokens run layers 0..𝑘−1, up to the anchor layer, and skip the full computation of layers 𝑘..𝐿−1, whose memories emitters write from the anchor layer’s output ℎ(𝑘) ; the criteria of Section 3.1 set the anchor depth 𝑘 per model. The last prompt token and every generated token run all 𝐿 layers: the last prompt token reads the emitted memories, writes its own with the original layers and gives the distribution of the first reply token. Every deep memory is a function of a shallow residual stream. Layer ℓ writes the memory (ℓ) of token 𝑡 from the residual stream ℎ𝑡 that enters it. An attention layer projects this input to a key and a value. A linear-attention layer projects it to a key, a value, a write strength and a decay, convolves the key and value over the last few tokens, and folds them into its state. Through the (ℓ) (𝑘) attention and feed-forward blocks of layers 𝑘..ℓ−1, ℎ1..𝑇 is a function of the residual stream ℎ1..𝑇 at any earlier depth 𝑘: (︀ (ℓ) (𝑘) )︀ memℓ (ℎ1..𝑇 ) = memℓ 𝑓𝑘→ℓ (ℎ1..𝑇 ) . (3) The anchor layer’s output thus determines every deeper memory. The question is how much of 𝑓𝑘→ℓ a write needs. Emitters. A write needs very little of 𝑓𝑘→ℓ . An emitter for layer ℓ is an architectural copy of the layer’s write path: its read of the residual stream (on Qwen3.8-Flash-Next and GLM-5.3-Flash, the input-dependent weights that mix the four streams into the layer’s input), its normalization and its key and value projections, plus a sparse-attention layer’s indexer key and a linear-attention layer’s write-strength and decay projections, short convolution and scan. The emitter starts from 7

Shallow Encoding Emits Deep Memories

^ (𝑘) the layer’s own weights but reads ℎ(𝑘) of the same tokens instead of ℎ(ℓ) , or its reconstruction ℎ when a prompt code is used. The untrained gap. Without training, the emitters form the untrained shallow prefill: POP’s training-free fill (He et al., 2026) extended to recurrent layers, and our baseline. Its increase in next-token cross-entropy is the untrained gap: 0.074 nats per token on Qwen3.8-Flash-Next at 𝑘 = 31, and 0.031–0.075 across the five models at the anchor depths we use. Its KL is 0.03–0.09 nats per token on all five models, against 6.8–11.1 for the early exit at the same depth (Figure 2c; Appendix A). This contrast is the write–read asymmetry. We then train the emitters together with the prompt code against the frozen model (§5.2). The asymmetry, layer by layer. Skipping one layer’s output for the prompt tokens, while still writing its memory from the residual stream that enters it, costs less than skipping it for the generated tokens (Figure 2a, b; Figure 9 in Appendix A). In KL, this holds in all 16 sampled layers of Qwen3.8-Flash-Next, 16 of 17 sampled layers of Nemotron-3.5-Lightning, 18 of 22 of Nemotron-3.5-Super and 23 of the 26 layers of Kimi-Linear below its last; in next-token loss, it holds in 34 of the 35 layers of the transformer Qwen3-8B-Base (Yang et al., 2025a) below its last (Figure 10). (Skipping the last layer for the prompt tokens changes no memory.) On these models, all exceptions lie in the first quarter of the layers. On GLM-5.3-Flash, the asymmetry holds in all 7 sampled layers from layer 24 of 45 on, including every sampled layer above its anchor depth, and reverses below, where SEED keeps the original layers. At and above the first emitted layer, the generated-token skip costs 5.6–49 times as much as the prompt-token skip on Qwen3.8-Flash-Next, 5.5–38 on Nemotron-3.5-Lightning, 5.4–96 on Kimi-Linear, 6.5–14 on Nemotron-3.5-Super and 11–19 on GLM-5.3-Flash. Why writes are easier than reads. We attribute the asymmetry to the two roles of a deep layer. At a prompt token, only its write matters, and a skipped layer still writes its own memory, so the skip reaches the reply only through the inputs of later layers’ writes. At a generated token, the deep layers must also produce the next-token distribution, a read that needs the attention and feed-forward blocks of every deep layer. In free-running generation, an error there also feeds back through the sampled tokens, whereas prompt tokens are given. Both reasons confine the shortcut to the prefill. They also explain the two curves of Figure 2a, b in the deepest layers: the generated-token skip rises again because these layers produce the next-token distribution, while the prompt-token skip keeps falling because at a prompt token their output is never read. The writes need not be exact. Beyond the first emitted layer, whose write path reads ℎ(𝑘) itself, the released emitters’ attention caches differ from the original model’s by 2–79% relative squared error per layer and tensor, and their recurrent states, pooled over each layer’s heads, by 20–52% on average over the emitted layers (Tables 6 and 14). Yet the six released sets stay within a KL of 0.004–0.032 nats per token of the original model with exact states, and 0.006–0.039 with the states compressed every 𝑊 steps, as served (above a second run of the original model; Table 6). The later layers tolerate inexact memories, so the emitters are trained mainly on the output KL rather than on a memory-reconstruction loss (§5.2). The prompt code. Since ℎ(𝑘) determines every deep memory, one vector per token suffices to restore a prompt’s deep memories for a prefix cache or a transfer between nodes. The prompt code

8

Shallow Encoding Emits Deep Memories

is a compact, lossy encoding of this vector for storage; a running request keeps the emitted caches and states, which its decode steps read. On Qwen3.8-Flash-Next, ℎ(31) holds 10,240 numbers per token, about twice the bf16 keys and values of its five attention layers above the anchor layer. SEED codes the offset Δ = ℎ(𝑘) − ℎ(0) from the token’s input embedding ℎ(0) , which is looked up from the token id. With an encoder 𝐸, a decoder 𝐷 and the mean offset 𝜇, the code keeps 𝑟 coordinates 𝑧 = 𝐸(Δ − 𝜇) in NVFP4, plus the exact bf16 values, at gap-coded one-byte indices, of the 𝑚 largest entries of the remainder Δ − 𝜇 − 𝐷𝑧. These entries are outliers, and a fixed rank-𝑟 code could capture each only by spending one of its directions on it (Appendix A.1). The emitters do not need ^ (𝑘) , ℎ(0) + 𝜇 + 𝐷𝑧 plus the exact values, in the code, but every set with a code reads the decoded ℎ training and in every exam, so its exams score the memories that a stored code would restore. How large the code must be. Copy fidelity, not the spectrum of the offset, sets the code’s rank. Closed-form codes, the untrained principal-component codes that initialize training, of rank 4,096 and 6,144 fail on Qwen3.8-Flash-Next to copy identifiers that the shallow prefill without a code (uncoded) copies, although rank 4,096 holds 0.853 of the offset’s variance (Table 7, Appendix A.1). Both Qwen3.8-Flash-Next sets store 𝑟 = 8192 coordinates plus 𝑚 = 256 exact values, 5.4 KB per token. The deep cache, the attention cache of the layers above the anchor layer, takes 10.6 KB in bf16 and about as much as the code in fp8. With 𝑚 = 128 exact values each, Nemotron-3.5-Lightning’s code keeps 𝑟 = 2,048 coordinates of a residual stream of 2,688 numbers, Nemotron-3.5-Super’s 𝑟 = 512 of 4,096, and Kimi-Linear’s all 𝑟 = 2,304. The code is trained end to end with the emitters, which learn to read it: on all 5 sets with a code, feeding the released emitters the exact residual ^ (𝑘) raises their KL by at least 13% (Table 6). stream instead of ℎ Where a code does not pay. GLM-5.3-Flash’s residual stream is 16,384 numbers wide, while its four sparse attention layers at and above 𝑘 = 31 cache only 4,228 bytes per token. No probed closed-form code within the bytes of its bf16 deep cache copies identifiers: the best copies at most 2 of 600, where the uncoded shallow prefill copies 530. GLM-5.3-Flash therefore runs without a code, and its emitters read the exact ℎ(31) . 3.1. Choosing the Settings One set of criteria and thresholds fixes the anchor depth, the prompt code and the state rank of every set. It uses only inexpensive measurements of the original model, its untrained shallow prefill and closed-form codes; no exam score enters (exact statements and reasons: Appendix A.1). The main measurement is the untrained gap gap(𝑘) on 48 held-out 4K windows with 256 teacher-forced continuation tokens; it falls towards the deep layers (Figure 11, Appendix A). Each criterion guards against one failure: • Depth. 𝑘 is the shallowest anchor depth whose (i) first emitted layer (index 𝑘) is a softmaxattention layer; (ii) untrained gap is at most 0.075 nats per token, three quarters of a budget of 0.1 nats for the untrained shallow prefill and shallow state together; and (iii) untrained shallow prefill copies at least three quarters of the identifiers that the original model copies on a copy probe, which hides one record with a universally unique identifier (UUID) in 4K–16K tokens of web text (Appendix A.1). Copying needs its own criterion: an identifier is copied only if all its tokens are, so a per-token average loss hardly registers a failed copy. • Code. The code keeps the full offset if that takes no more bytes than even the fp8 deep cache. Otherwise, it uses the smallest closed-form rank that takes no more bytes than the bf16 deep cache, which a stored code replaces, and shows no significant copy loss against the uncoded 9

Shallow Encoding Emits Deep Memories

shallow prefill (exact McNemar test, 𝑝 < 0.10); failing that, there is no code. It keeps 𝑚 = 256 exact values on models whose chat-template tokens reconstruct worse than their other tokens, and 𝑚 = 128 otherwise. • State. The state drift is the per-token KL between the original model with exact states and with shallow states: the prompt-final state compressed once, then again every 𝑊 = 𝑅 steps. Both runs are teacher-forced on the original model’s own replies to 48 held-out prompts, and the KL is averaged over decode steps 961–1,024. The state rank 𝑅 ∈ {8, 16, 32} is the smallest whose state drift is at most a quarter of gap(𝑘), else 𝑅 = 32; the compression period is 𝑊 = 𝑅. Table 7 (Appendix A.1) lists the measurements and resulting settings. The copy criterion is binding on GLM-5.3-Flash: at depth 27 its untrained gap is small (0.045), but its untrained shallow prefill copies only 269 of the original model’s 595 identifiers, so its anchor depth is 31, where it copies 530. Kimi-Linear gets 𝑅 = 32 because its gap is small (0.031), not because its states compress worse.

4. Shallow in Representation: The State Sink Pooled over heads, most of the energy of a hybrid model’s recurrent states lies in one rank-one term per head, along a fixed direction that does not depend on the prompt: the state sink. The rest, the content, is a low-rank working memory. The sink forms because the values share a common component that the linear recurrence accumulates; after the head’s normalization, it acts as a bias and a gain, while what the head has read about the prompt lies mainly in the content. SEED keeps every layer in the decode (§3), so its decode saving must come from how the memories are represented, and this structure supplies it. We analyse it on all 36 × 48 = 1,728 Gated DeltaNet heads of Qwen3.8-Flash-Next, at the end of six held-out web prompts of 3,584 tokens, and score one-time interventions on these states by the loss on the next 512 tokens. One direction carries the state. Let 𝑠⋆ 𝑢⋆ 𝑣⋆⊤ be the top term of the singular value decomposition (SVD) of a head’s state 𝑆𝑇 . It carries a median 85% of ‖𝑆𝑇 ‖2𝐹 , the second term about 5%, and the median share of the top term lies between 71% and 99.8% in every layer (Figure 13, Appendix C). Its output-side vector 𝑣⋆ is nearly the same across unrelated prompts (median | cos | 0.92), on a shuffled prompt (0.90) and on random token ids (0.87; Figure 13c). The attention sink of softmax layers also forms regardless of the tokens’ meaning and on random tokens (Xiao et al., 2024; Gu et al., 2025), but it is a position in the sequence, often the initial tokens, whereas the state sink is a direction of the state. The direction is the state’s component along the mean value. Once the keys, decays and write strengths are fixed, the recurrence (2) is linear in the values, so splitting each value along a fixed direction splits the state in the same way. Lemma 1 (sink decomposition). Let 𝑆𝑡 = Φ𝑡 𝑆𝑡−1 + 𝛽𝑡 𝜅𝑡 𝑣𝑡⊤ with any 𝑑𝑘 × 𝑑𝑘 matrices Φ𝑡 (in (2), Φ𝑡 = 𝑔𝑡 (𝐼 − 𝛽𝑡 𝜅𝑡 𝜅⊤ ¯ ̸= 0 and split each value and the state along it: 𝑡 )). Fix 𝑣 𝑣𝑡 = 𝜔𝑡 𝑣¯ + 𝜀𝑡 with 𝜀⊤ ¯ = 0, 𝑡 𝑣

𝑎𝑡 =

𝑆𝑡 𝑣¯ , ‖¯ 𝑣 ‖2

𝐶𝑡 = 𝑆𝑡 − 𝑎𝑡 𝑣¯⊤ ,

so that 𝐶𝑡 𝑣¯ = 0. Then the sink coefficient 𝑎𝑡 and the content 𝐶𝑡 each follow the head’s own recurrence, 𝑎𝑡 = Φ𝑡 𝑎𝑡−1 + 𝛽𝑡 𝜔𝑡 𝜅𝑡 ,

𝐶𝑡 = Φ𝑡 𝐶𝑡−1 + 𝛽𝑡 𝜅𝑡 𝜀⊤ 𝑡 ,

(4)

10

Shallow Encoding Emits Deep Memories

−0.1

0.01

(b) sink, 94%

0.1

(c) content, 6%

1

key index

aT

(d) the sink's two vectors

0 −1

64

0

64 key index

127

0

64 value index

127

1

v̄

64 value index

127

0

64 value index

127

Frobenius norm, median over heads

0.01

1e−4

2 1 0.5

0.2 0.1

8 16 24 singular index i

layer 0

64 value index

(f) norms vs prompt length

R = 16

1

0

32

16

layer 12

(h) top direction vs v̄

(g) sink's share 1

1

0.75

0.75

0.5 0.25

64 256 1K 3.5K prompt tokens t

layer 24

127

|cos(v⋆, v)| ̄

(e) singular spectra 1

1e−6

0 −1

0

share of the state's energy

127

si/s1, median over heads

−0.001 0 0.001

(a) state, 100%

0

layer 24, head 7

−0.01

layer 46

0

0.5 0.25

0

12 24 layer

46

state (e), sink (f)

0

0

12 24 layer

46

content (e, f)

Figure 3 | A rank-one sink along the head’s mean value holds most of a recurrent state’s energy, and the content left over lies in few directions (Qwen3.8-Flash-Next; the first 3,584 tokens of one of the six held-out web prompts of Appendix C). (a) The 128 × 128 state of head 7 of layer 24, a middle head of the layer by the sink’s share of the state’s energy. (b) Its sink 𝑎𝑇 𝑣¯⊤ , the state’s projection on the head’s mean value 𝑣¯ (estimated on other text). (c) The content 𝑆𝑇 − 𝑎𝑇 𝑣¯⊤ . Titles: shares of ‖𝑆𝑇 ‖2𝐹 , which add to one. (a)–(c) share one colour scale, log(1 + |𝑥|/10−3 ) in the magnitude, nothing clipped. (d) 𝑎𝑇 and 𝑣¯, each divided by its largest magnitude. (e) Singular spectra of the state (solid) and the content (dashed), divided by the head’s top singular value; per-index medians over the layer’s 48 heads. Dotted: the rank SEED keeps. (f) Frobenius norms of the sink (solid) and the content (dashed) against the prompt length; medians over heads. (g) The sink’s share of the state’s energy and (h) | cos | between 𝑣⋆ and 𝑣¯, per head; bars: layer medians.

and their energies add: for every 𝑧 ∈ R𝑑𝑘 , ‖𝑆𝑡 − 𝑧¯ 𝑣 ⊤ ‖2𝐹 = ‖𝑎𝑡 − 𝑧‖2 ‖¯ 𝑣 ‖2 + ‖𝐶𝑡 ‖2𝐹 . In particular, 𝑎𝑡 𝑣¯⊤ is the matrix of the form 𝑧¯ 𝑣 ⊤ closest to 𝑆𝑡 . The identity holds for any direction. We fix 𝑣¯ as the head’s mean value over generic text, independent of the prompt, because that direction carries most of the state (as measured below). We call the projection 𝑎𝑇 𝑣¯⊤ the state sink and 𝐶𝑇 the content. The sink’s coefficient 𝑎𝑇 follows the head’s own recurrence, driven by 𝜔𝑡 , the values’ component along their mean, which averages one over generic text by construction. The component 𝜔𝑡 𝑣¯ carries 60–99% of the values’ energy, E‖𝜔𝑡 𝑣¯‖2 /E‖𝑣𝑡 ‖2 (layer medians; 64–94% from layer 8 on). If every value equals 𝑣¯ and 𝑆0 = 0, then 𝜔𝑡 = 1, 𝐶𝑇 = 0 ∑︀ and 𝑎𝑇 = 𝑡 Φ𝑇 · · · Φ𝑡+1 𝛽𝑡 𝜅𝑡 depends on the keys, decays and write strengths alone; Zhuang et al. (2026) derive this case for one repeated key with constant decay and write strength.

11

Shallow Encoding Emits Deep Memories

The measurements match the decomposition. Over heads, the state’s top singular term has a median (Frobenius) cosine similarity of 0.95 with the sink, and 𝑢⋆ a median | cos | of 1.00 with 𝑎𝑇 (Figure 3 shows one prompt). The sink holds a median 75% of the state’s energy, and more than half in the median head of every layer (53–98%). In the four layers of Figure 3f, the median norms of the sink and the content each stay within 17% of their mean from 256 prompt tokens on, and the sink is the larger in every one. The content lies in few directions: a median of 6 hold 90% of its energy and 17 hold 99%, close to the rank 𝑅 = 16 that SEED keeps on Qwen3.8-Flash-Next. Because the content is the state projected off one direction, 𝐶𝑇 = 𝑆𝑇 (𝐼 − 𝑣^𝑣^⊤ ) with 𝑣^ = 𝑣¯/‖¯ 𝑣 ‖, its spectrum interlaces the state’s, 𝑠𝑖+1 (𝑆𝑇 ) ≤ 𝑠𝑖 (𝐶𝑇 ) ≤ 𝑠𝑖 (𝑆𝑇 ) (Figure 3e). Low effective ranks have been reported for such states (Nazari & Rusch, 2026), and in Qwen3-Next for one of two groups of heads, while the other approaches the rank bound (Sun et al., 2026). The other models. The Mamba-2 heads of Nemotron-3.5-Lightning decompose in the same way along the head’s mean input. Over all 23 × 64 heads on six held-out prompts of 3,584 tokens, the sink holds a median 64% of a head’s energy, and the content reaches 90% of its energy in a median of one direction and 99% in four. The Kimi Delta Attention heads of Kimi-Linear and GLM-5.3-Flash decay each key channel at its own rate, and their values lie less along their mean: 16% and 13% of their energy in the median head, against 81% on Qwen3.8-Flash-Next. In the median head, their sink holds 34% and 24% of the energy, but it concentrates in the heads that hold the most energy and so holds 82% and 86% of the state energy pooled over all heads. Over the five models, the sinks hold 69–99.7% of the pooled state energy (Appendix C gives every model’s median and pooled shares). The content of Kimi-Linear and GLM-5.3-Flash is as low-rank as on Qwen3.8-Flash-Next: pooled over heads and layers by energy, rank 16 leaves 0.065 and 0.057 of it (Qwen3.8-Flash-Next: 0.063), and rank 32, the rank they keep (§3.1), leaves 0.019 and 0.015 (Appendix A.1). After normalization, the sink is a bias and a gain. Every gated-delta head normalizes its read-out before gating it. Write the read-out as 𝑜𝑡 = 𝑏𝑡 + 𝑟𝑡 , with the sink’s part 𝑏𝑡 = (𝑞𝑡⊤ 𝑎𝑡 ) 𝑣¯ and the content’s part 𝑟𝑡 = 𝐶𝑡⊤ 𝑞𝑡 . As 𝑟𝑡 ⊥ 𝑣¯, a unit-gain RMSNorm without its stabilizing constant returns √ 𝑑𝑣 (𝑏𝑡 + 𝑟𝑡 )/(‖𝑏𝑡 ‖2 + ‖𝑟𝑡 ‖2 )1/2 . The sink’s part thus sets a fixed output direction, up to the sign of ⊤ 𝑞𝑡 𝑎𝑡 , and through the normalizer scales the content, √ as the sink logit does in a softmax denominator. When ‖𝑏 ‖ ≫ ‖𝑟 ‖, the normalized read-out is 𝑑𝑣 times this direction√plus the content scaled by 𝑡 𝑡 √ 2 2 𝑑𝑣 /‖𝑏𝑡 ‖, up to an error of at most ‖𝑟𝑡 ‖ /(2‖𝑏𝑡 ‖ ) relative to its norm 𝑑𝑣 (Appendix B, Eq. (9)). In most heads the sink’s part is the larger: over the scored tokens, the ratio of the mean norms of 𝑏𝑡 and 𝑟𝑡 has a median of 2.3 over the heads and exceeds one in 82% of them. The sink holds the energy, the content holds the prompt. Eviction shows that both parts are needed but that the sink’s dependence on the prompt matters little (Appendix C). Removing the sink while keeping the content costs +0.073 nats per token, and keeping only the sink +0.067. Replacing each head’s sink by its mean over the other prompts costs only +0.002, and taking it from a prompt of random tokens +0.017. The sink’s coefficient depends on the prompt, but what the head has read about the prompt lies mainly in the content. The content is a short working memory. The content matters more than these evictions suggest. They act once, on the state at the end of the prefill (before the last prompt token, as in SEED), after which the last prompt token and the 512 scored tokens update it exactly. When the state is instead compressed after every teacher-forced continuation token, keeping only the sink costs +1.66 nats per token over a base of 1.76 (Appendix C). The content is dominated by recent writes. 12

Shallow Encoding Emits Deep Memories

From 𝑆0 = 0, 𝐶𝑇 = 𝑡 Φ𝑇 · · · Φ𝑡+1 𝛽𝑡 𝜅𝑡 𝜀⊤ each damped by the decays 𝑡 is a sum of rank-one writes, (︀ ∏︀𝑇 )︀ ∑︀ applied after it, so its 𝑟 most recent writes approximate it within 𝑡≤𝑇 −𝑟 𝑢=𝑡+1 𝑔𝑢 𝛽𝑡 ‖𝜀𝑡 ‖. This is small in heads whose typical decays are well below one. On Qwen3.8-Flash-Next, with each head’s mean decay, the time scale 1/(1 − mean decay) is about 17 tokens in the median head, at most ten tokens in 41% of the heads, and above 100 tokens in 28%. The error bounds of §5.1 use instead the head’s decay horizon 1/(1 − 𝑔¯), set by its largest decay 𝑔¯, a worst case. ∑︀

What this means for SEED. The sink plus a rank-8 content costs +0.023 nats per token, against +0.025 for a plain rank-8 truncation, whose top term lies close to the sink. Keeping the sink apart also gives the stored form its structure: its direction is shared by all requests, it is never truncated, and it keeps the content orthogonal to 𝑣¯. SEED thus stores a head’s state, a bias and gain plus a low-rank working memory, as the sink plus rank-𝑅 factors (§5.1).

5. SEED SEED follows from the two findings. Every deep memory is a function of a shallow residual stream (§3), so the prefill runs only layers 0..𝑘−1 on all but the last prompt token and emitters write the deeper memories. The recurrent states hold most of their energy, pooled over heads, in a sink along a fixed direction of each head, and the rest, the content, is low-rank (§4), so the decode keeps the sink exactly and the content at low rank. Section 5.1 gives the exact update of this factored state and the bound on its only approximation; Section 5.2 trains the emitters and the prompt code against the frozen model. Each model has four settings (Table 7, Appendix A.1): the anchor depth 𝑘, the prompt code (𝑟 coded coordinates plus 𝑚 exact values; GLM-5.3-Flash has none), the state rank 𝑅 and the compression period 𝑊 . 5.1. The Shallow State In the decode, where bytes are the bottleneck, SEED keeps each recurrent state as its sink plus a rank-𝑅 content, 𝑆 = 𝑎¯ 𝑣 ⊤ + 𝑈 𝑉 ⊤ . The mean value 𝑣¯ ∈ R𝑑𝑣 is a per-head constant of the component set, shared by all requests. The factors 𝑎 ∈ R𝑑𝑘 , 𝑈 ∈ R𝑑𝑘 ×𝑅 and 𝑉 ∈ R𝑑𝑣 ×𝑅 belong to the request, and the columns of 𝑉 are orthogonal to 𝑣¯. An exact update. Each step of the update (2) splits its value as in Lemma 1, 𝜔 = 𝑣 ⊤ 𝑣¯/‖¯ 𝑣 ‖2 and 𝜀 = 𝑣 − 𝜔¯ 𝑣 . It gives the part along 𝑣¯ to the sink’s coefficient and appends the rest as one new content column: 𝑎+ = 𝑔 𝑎 + 𝛽 (𝜔 − 𝑔 𝜅⊤ 𝑎) 𝜅, 𝑈 + = 𝑔𝑈, 𝜅 , [︀

]︀

𝑉 + = 𝑉, 𝛽 (𝜀 − 𝑔𝑉 𝑈 ⊤ 𝜅) . [︀

]︀

(5)

The scalar 𝜔 − 𝑔 𝜅⊤ 𝑎 and the vector 𝜀 − 𝑔𝑉 𝑈 ⊤ 𝜅 are the delta rule’s own prediction error 𝑣 − 𝑔𝑆 ⊤ 𝜅, split along 𝑣¯ and orthogonal to it, so between compressions the factored update is exact. Since 𝜀 and the columns of 𝑉 are orthogonal to 𝑣¯, so is the appended column: the content never gains a component along 𝑣¯, so the split of Lemma 1 is invariant under the update and is never recomputed. Between compressions, the factors thus hold a rank-𝑅 summary and, much like a short attention cache, the most recent writes (keys in 𝑈 , delta-rule corrections in 𝑉 ). The same exact update holds for Mamba-2, with the sink along the head’s mean input, and for the per-channel decay of Kimi-Linear and GLM-5.3-Flash (Lemma 2).

13

Shallow Encoding Emits Deep Memories

Read-out and bytes. At 𝑤 steps after the last compression (1 ≤ 𝑤 ≤ 𝑊 ; a step reads after its update), the read-out 𝑜 = (𝑞 ⊤ 𝑎) 𝑣¯ + 𝑉 (𝑈 ⊤ 𝑞) uses all 𝑅 + 𝑤 content columns and costs (𝑑𝑘 + 𝑑𝑣 )(𝑅 + 𝑤 + 1) multiply-adds instead of 𝑑𝑘 𝑑𝑣 : 2 · 128 (𝑅 + 𝑤 + 1) instead of 1282 on Qwen3.8-Flash-Next. The sink is its own term rather than one more factor column because 𝑣¯ is shared by all requests; it then costs 𝑑𝑘 numbers per request instead of 𝑑𝑘 + 𝑑𝑣 , and no compression spends a rank on it. A head holds (𝑅+1)𝑑𝑘 +𝑅𝑑𝑣 numbers right after a compression and (𝑅+𝑊 +1)𝑑𝑘 +(𝑅+𝑊 )𝑑𝑣 just before the next, against 𝑑𝑘 𝑑𝑣 for a dense head: 16.9 KB, 33.3 KB and 65.5 KB in fp32 on Qwen3.8-Flash-Next (Table 2 gives the bytes per request). Periodic compression and its bound. Every 𝑊 steps, a warm-started subspace iteration truncates the (𝑅 + 𝑊 )-column content back to rank 𝑅. This truncation is the shallow state’s only approximation. It leaves the sink untouched, and the content stays orthogonal to 𝑣¯ because the truncation only projects it within its own column or row space (Lemma 2). In the open loop, the compressed state 𝑆˜𝑡 and the exact recurrence 𝑆𝑡 see the same keys, values, decays and write strengths. If every decay is at most 𝑔¯ < 1 and 𝜖¯ is the largest Frobenius norm of a single compression error, then for every step 𝑡 (Corollary 4, Appendix B) ‖𝑆˜𝑡 − 𝑆𝑡 ‖𝐹 ≤

𝜖¯ 𝜖¯ ≤ . 1 − 𝑔¯ 𝑊 1 − 𝑔¯

Repeated compressions thus do not compound in the open loop: the error stays within one compression error times the head’s decay horizon 1/(1 − 𝑔¯), however many compressions occur. The bound also holds with 𝑔¯ replaced by the largest mean decay over the 𝑊 steps between two compressions, and the sink coefficient stays exact. In the closed loop, a compression error also changes later keys and values; Section 6.1 follows this loop over 16K decode steps. The rule of §3.1 gives 𝑅 = 𝑊 = 16 on both Qwen3.8-Flash-Next sets, Nemotron-3.5-Lightning and Nemotron-3.5-Super and 𝑅 = 𝑊 = 32 on Kimi-Linear and GLM-5.3-Flash (Table 7). The shallow state also composes with 16-bit storage of the factors (§6.1). Serving. SEED runs in SGLang (Zheng et al., 2024) as two opt-in settings, one per part, so an operator enables either or both per workload. With shallow prefill, the first 𝑁 − 1 prompt tokens run layers 0..𝑘−1 and the emitters, and the last prompt token runs all 𝐿 layers. The state left by the first 𝑁 − 1 prompt tokens is compressed once, and 𝑥𝑁 runs as the first decode step. In disaggregated serving, a prefill node ships only the sink coefficients and the factors of each recurrent state to the decode node. Each request counts its own steps from the end of its prefill, and the compressions due in a step are batched over all heads of those requests. The engine’s recurrent-state pool holds each Qwen3.8-Flash-Next state as its sink coefficient and factors; for Nemotron-3.5-Lightning, Nemotron-3.5-Super, Kimi-Linear and GLM-5.3-Flash, the exam engines keep the state dense between compressions and factor it at each compression, which gives the same state in exact arithmetic (Lemma 2). 5.2. Training One recipe trains every component set against the frozen model, which stays the only teacher. We train all new parameters jointly: the emitters and the prompt code’s encoder 𝐸 and decoder 𝐷, 0.11B–0.71B exported parameters per set (Table 10). The main objective is the KL divergence to the frozen checkpoint’s next-token distribution (Figure 4; Appendix D lists the auxiliary terms). The emitters start as copies of the frozen layers’ write paths, 𝐸 and 𝐷 as the top principal directions of the offset’s covariance, and 𝜇 is the mean offset, a fixed buffer. 14

Shallow Encoding Emits Deep Memories

(a) One training step runs the served prefill

original layer, frozen

added by SEED, trained (the only parts with gradients)

recurrent state

teacher: original model, all L layers KL SEED set

E layers 0..k−1

D

prompt code, NVFP4

(b) Stage 1: off-policy distillation 600M tokens fixed text: web, retrieval documents, the frozen model’s replies

emitters

state, rank R

continuation, all L layers

(c) Stage 2: on-policy rounds KL

teacher and set answer the same prompts

forward + reverse KL on the replies

held-out KL check

done

to the teacher another round safeguards

exam sets never used as data

stage-2 prompts 13-gram filtered

original weights unchanged

Figure 4 | One recipe trains every component set. Only SEED’s components are trained, with the frozen model as the only teacher, in a forward pass that runs the prefill as served. Stage 1 distils on fixed text; stage 2 is on-policy, on the replies of the teacher and of the current set to the same prompts. Per-model terms: Appendix D.

The served prefill in every training step. The training forward pass runs the prefill as inference does: the prompt code passes through its storage quantizer (straight-through (Bengio et al., 2013); the exact values are selected without gradient), and every prompt-final state, the state left by the first 𝑁 − 1 prompt tokens, is compressed at the decode rank (on GLM-5.3-Flash, every emitted layer’s). Every component thus learns to write memories that survive their stored form. Training departs from inference only in truncating by an exact eigendecomposition instead of a subspace iteration and in running the continuation without the periodic compression (Appendix D). Two stages. Stage 1 distils the frozen model in 4K and 16K windows of public text (Penedo et al., 2024; Ben Allal et al., 2025), public reasoning and chat corpora, synthetic retrieval documents and the frozen model’s own responses to public prompts; part of the windows are aligned to the start of an answer (Table 12 gives every set’s data). Stage 2 runs on-policy rounds in which the frozen model and the current set answer public prompts. The forward KL applies to the frozen model’s answers, and both the forward and the reverse KL to the set’s own answers; the reverse KL penalizes the set for tokens that it writes and the frozen model would not (Agarwal et al., 2024; Gu et al., 2024; Ko et al., 2025). Stage 2 runs one or two rounds; after each, we measure the held-out KL and the divergence of the set’s own replies (Appendix D). Every model releases its stage-2 set except Nemotron-3.5-Super, whose stage-1 set is released. Model-specific terms. Nemotron-3.5-Lightning, GLM-5.3-Flash and Nemotron-3.5-Super add a term that aligns the emitted prompt-final states with the frozen model’s. Without it, the emitted states of Nemotron-3.5-Lightning and GLM-5.3-Flash are the furthest from the frozen model’s as the layers read them (Table 14, Appendix D); Nemotron-3.5-Super follows Nemotron-3.5-Lightning’s recipe. GLM-5.3-Flash adds a second term for the indexer keys that its emitters write. Convergence and cost. On every set (Figure 5), at least 89% of stage 1’s reduction of the held-out KL happens within its first 150M tokens, a quarter of the stage; afterwards the KL changes slowly. Stage 2 changes the teacher-forced KL on held-out web windows by less than 0.0006 nats per token: it acts on the set’s own on-policy replies, which a teacher-forced KL does not see (Appendix D). Training is a one-time cost per checkpoint and changes no original weight. For the bf16 Qwen3.8-Flash-Next set, stage 1 takes 160 minutes on 16 nodes of four GPUs, and stage 2 another 111 minutes plus the time to serve its generations (Table 13). Appendix D gives the settings,

15

Shallow Encoding Emits Deep Memories

held-out web windows

Qwen3.8-Flash-Next Qwen3.8-Flash-Next bf16 NVFP4 stage 1

stage 2

stage 1

stage 2

answer windows

Nemotron-3.5Lightning

Kimi-Linear

GLM-5.3-Flash

stage 1

stage 1

stage 1

stage 2

stage 2

Nemotron-3.5Super

stage 2

stage 1

stage 2

KL (nats/token)

0.05 0.02 0.01 0.005 0.002 0

500

training tokens (M)

0

500

training tokens (M)

0

500

training tokens (M)

0

500

training tokens (M)

0

500

training tokens (M)

0

500

training tokens (M)

Figure 5 | Every set converges the same way: most of the KL reduction comes early in stage 1. Teacher-forced KL to the frozen model on held-out web and answer windows against training tokens (dashed: start of stage 2; dotted: start of a further round).

the data and the costs (Table 13).

6. Accuracy SEED is nearly lossless on every model. On eleven official exams, 63 of the 66 checkpoint–exam cells of the six component sets lie within two standard errors of the original model or above it (Table 3, Figure 6a). Its shallow state keeps the next-token distributions closer to the original model’s than NVIDIA’s official NVFP4 checkpoints do (Figure 6b), and over 16K decode steps its drift grows far more slowly than the number of compressions. Protocol. Each component set and its original model answer the same questions with the same per-request sampling seeds (on MMLU, all but at most 0.4% of samples). Table 3 gives the results, and Table 16 (Appendix E) adds the stage-1 sets that we examined. Both arms are served by SGLang with their recurrent states in fp32. The throughput measurements of Section 7 use the same component sets and state precision. Exams. Every exam runs with lm-eval (Biderman et al., 2024) 0.4.13 and its official grader, with each vendor’s recommended sampling configuration; GPQA-Diamond and MMLU use the prompt and answer extraction of OpenAI simple-evals (OpenAI, 2024), and GPQA-Diamond also its option permutation. The eleven exams (Mathematical Association of America, 2025; Rein et al., 2024; Hendrycks et al., 2021a;b; Lightman et al., 2024; Zhou et al., 2023; Jain et al., 2025; Chen et al., 2021; Liu et al., 2023; Bai et al., 2025; Hsieh et al., 2024) are GPQA-Diamond, MMLU, AIME 2024+25, MATH-500, IFEval, LiveCodeBench v6 (problems from August 2024 to April 2025), HumanEval+, LongBench v2, and RULER at 32K, 64K and 128K. Repetitions and selection. Every exam runs eight repetitions except LongBench v2 (four), MMLU (one) and each RULER length (one; RULER is greedy), always the first 𝑛 repetitions. No sample is dropped, and no repetition was rerun because of its outcome. Where a model had several candidate sets, the choice between them rested on repetitions 0–3. On the thirty cells of GPQA-Diamond, AIME, MATH-500, IFEval and LiveCodeBench of all six sets, repetitions 4–7, on which no choice rested, agree with the eight-repetition values within about one of their standard errors.

16

Shallow Encoding Emits Deep Memories

Table 3 | SEED matches the original model’s scores across the eleven exams: 63 of the 66 cells lie within two standard errors of the original model or above it. Each component set (on Nemotron-3.5-Super, its stage-1 set; §5.2) is paired with its original model on the same questions and per-request seeds. Rows: original score, the set’s score and their paired difference (mean ± standard error over questions). Eight repetitions per exam except LongBench v2 (four), MMLU and RULER (one each). RULER GPQA Diamond

AIME 2024+25 MATH-500

MMLU

LiveCode LongBench IFEval Bench v6 HumanEval+ v2

32K

64K

128K

Qwen3.8-Flash-Next (bf16) original 91.9 99.4 SEED 91.5 98.8 difference −0.3±0.4 −0.6±0.5

99.3 92.5 91.7 88.5 98.8 92.6 91.1 88.9 −0.5±0.1 +0.1±0.1 −0.6±0.3 +0.4±0.4

93.9 94.8 +0.9±0.4

64.3 93.2 94.2 95.5 65.3 96.4 95.5 95.0 +0.9±0.9 +3.2±0.6 +1.2±0.5 −0.5±0.4

Qwen3.8-Flash-Next-NVFP4 original 92.4 99.6 SEED 91.2 99.6 difference −1.1±0.5 0.0±0.4

99.2 92.3 91.6 88.9 98.9 92.5 91.5 88.8 −0.3±0.2 +0.2±0.2 −0.1±0.3 −0.1±0.4

94.4 94.6 +0.2±0.4

65.3 94.7 94.9 95.3 64.6 96.1 95.4 94.6 −0.7±0.9 +1.4±0.5 +0.5±0.5 −0.7±0.4

Nemotron-3.5-Lightning original 76.3 89.6 SEED 75.0 88.8 difference −1.3±1.0 −0.8±1.1

98.2 88.1 91.5 78.2 98.3 88.0 92.0 77.6 +0.1±0.2 −0.2±0.2 +0.4±0.4 −0.6±0.6

91.9 92.0 +0.1±0.6

47.1 45.7 −1.4±1.1

96.5 96.4 95.9 96.4 96.0 95.6 0.0±0.2 −0.3±0.2 −0.3±0.3

Kimi-Linear original 67.6 67.5 SEED 68.0 66.9 difference +0.4±1.4 −0.6±2.5

93.2 77.7 73.2 44.3 92.4 78.3 71.7 43.4 −0.8±0.5 +0.6±0.4 −1.5±0.6 −0.9±0.8

78.7 78.3 −0.4±0.8

34.7 34.6 −0.1±1.1

95.0 94.0 94.5 95.0 93.9 94.1 0.0±0.1 −0.1±0.2 −0.4±0.2

GLM-5.3-Flash original 91.3 96.0 SEED 91.1 98.1 difference −0.2±0.5 +2.1±1.3

98.7 93.0 89.9 87.8 98.6 92.8 90.7 88.2 −0.1±0.2 −0.1±0.1 +0.8±0.4 +0.4±0.4

94.8 94.7 −0.2±0.3

63.2 96.7 96.3 96.4 64.1 96.5 96.5 96.1 +0.8±0.8 −0.2±0.1 +0.2±0.2 −0.3±0.2

Nemotron-3.5-Super original 89.1 SEED 90.1 difference +1.0±0.6

99.4 99.4 −0.1±0.1

95.0 95.5 +0.5±0.4

59.3 58.1 −1.2±0.8

98.5 98.5 0.0±0.7

92.1 91.7 92.1 92.1 0.0±0.1 +0.5±0.4

86.3 86.3 0.0±0.4

96.8 94.9 96.0 96.7 94.8 95.9 0.0±0.2 −0.1±0.2 −0.1±0.2

Scores and answers. Across the six sets, on unrounded values, 63 of the 66 checkpoint–exam cells (Figure 6a) lie within two standard errors of the original model or above it; Appendix E analyses every cell at least one standard error below. Equal scores can hide answer flips (Dutta et al., 2024), but SEED flips answers about as often as a second sample of the original model does, and its replies are not longer (Appendix E). 6.1. How Close the Next-Token Distributions Are The shallow state stays closer to the original model’s next-token distributions than NVIDIA’s official NVFP4 checkpoints do. Exam scores are coarse, so we compare the distributions directly. The decode drift is the KL divergence from the original model’s next-token distribution, teacher-forced on the original model’s own replies to held-out prompts (48 replies in math, science and code; 1,024 decode steps), measured above the noise floor of a second run of the original model; the perplexity change is the change in perplexity of the same replies (Appendix C; on GLM-5.3-Flash, 24 replies and 256 steps, above its full-rank factored path). The drift is the primary measure, as it sees every token’s full distribution; the perplexity change, given for the exam-tested form and the NVFP4 checkpoints, expresses it in the standard unit of language-model quality. The exams and every drift measurement keep the recurrent states in fp32 except where a lower precision is the variable. Table 4 gives every storage form at its better rounding; bf16 appears only in Table 9, since fp16 matches or beats it on every model.

17

Shallow Encoding Emits Deep Memories

Qwen3.8-Flash-Next bf16 Qwen3.8-Flash-Next NVFP4

Nemotron-3.5-Lightning Kimi-Linear

GLM-5.3-Flash Nemotron-3.5-Super

shallow state (exam form) official NVFP4 checkpoint

(a) every exam and set: the differences cluster at zero

(b) perplexity: shallow state vs official NVFP4

GPQA-Diamond

0.46%

Qwen3.8Flash-Next

AIME 2024+25 MATH-500

1.2% 0.46%

Nemotron-3.5Lightning

MMLU

1.9%

IFEval LiveCodeBench v6

Kimi-Linear

0.10% no NVFP4 release

HumanEval+ 0.21% GLM-5.3Flash release is FP8 (our base)

LongBench v2 RULER 32K RULER 64K

−4

0.55%

Nemotron-3.5Super

RULER 128K −3

−2

−1

0

1

2

3

SEED minus original model (percentage points, ± s.e.)

4

2.1%

0.0

0.5

1.0

1.5

2.0

2.5

perplexity change of the original's replies (%)

Figure 6 | SEED matches the original model’s scores on every model, and its shallow state moves the next-token distribution less than an official NVFP4 checkpoint does. (a) The 66 paired differences of Table 3 (± one standard error; shaded: ±1 point). (b) The perplexity of the original model’s own replies changes by less than 0.6% under the shallow state of the exams, less than under NVIDIA’s official NVFP4 checkpoints (Table 4; GLM-5.3-Flash above its full-rank factored path). Table 4 | The shallow state stays closer to the original model than the official NVFP4 checkpoints do, and it composes with 16-bit storage. Top: decode drift (§6.1) in nats per token; grey: stored state size as a share of the original fp32 state (shallow state: right after a compression). 𝑠 : stochastic rounding (Table 9 gives both roundings). Bottom: change in the perplexity of the same replies. –: no NVFP4 release (Kimi-Linear) or an FP8 release, the checkpoint we use (GLM-5.3-Flash).

original model, dense state fp32 (exams) fp16 fp8 SEED, shallow state fp32 (exams) fp16 fp8 official NVFP4 checkpoint

Qwen3.8Flash-Next

Nemotron-3.5Lightning

KimiLinear

GLM-5.3Flash

Nemotron-3.5Super

0 100% 0.0006 50% 0.0073𝑠 25%

0 100% 0.0004𝑠 50% 0.0487 25%

0 100% 0.0002 50% 0.0048𝑠 25%

0 100% 0.0006 50% 0.0055𝑠 25%

0 100% 0.0002𝑠 50% 0.0745 25%

0.0056 26% 0.0056 13% 0.0067 6%

0.0060 39% 0.0059𝑠 20% 0.0136 10%

0.0017 51% 0.0016𝑠 25% 0.0023 13%

0.0027 51% 0.0021 25% 0.0030 13%

0.0070 39% 0.0069 20% 0.0160 10%

0.0105

0.0209

–

–

0.0190

+0.46% +1.94%

+0.10% –

+0.21% –

+0.55% +2.05%

perplexity of the original model’s replies, change SEED, fp32 (exams) +0.46% official NVFP4 checkpoint +1.16%

The shallow state alone. In fp32, as in the exams, the shallow state drifts 0.002–0.007 nats per token on the five models and changes the perplexity of the original model’s replies by less than 0.6% (here with the exact prefill; §3 measures the shallow prefill). NVIDIA’s official NVFP4 checkpoints of Qwen3.8-Flash-Next, Nemotron-3.5-Lightning and Nemotron-3.5-Super, whose four-bit weights are an accepted serving choice, drift 1.9–3.5 times as much (0.0105 nats per token on Qwen3.8-Flash-Next) and raise the perplexity of the same replies by 1.2–2.1%.

18

Shallow Encoding Emits Deep Memories

Qwen3.8-Flash-Next bf16 Qwen3.8-Flash-Next NVFP4

Nemotron-3.5-Lightning Kimi-Linear

Nemotron-3.5-Super

KL to the original model (nats/token)

dashed: official NVFP4 checkpoint, same colour 0.020 0.015 0.010 0.005 0.000 0

4

8

12

16

decode step (thousands)

Figure 7 | The shallow state’s drift grows far more slowly than its number of compressions. Decode drift of the released state form (fp32) per 1,024 decode steps of long replies of the original model (24 replies on both Qwen3.8-Flash-Next sets and Kimi-Linear, 16 on Nemotron-3.5-Lightning, 12 on Nemotron-3.5-Super; 𝑅 = 𝑊 = 16, 32 on Kimi-Linear); dashed: the official NVFP4 checkpoint of the same model over its first 1,024 steps (Table 4).

Lower precision composes. fp16 factors leave the drift within 0.0006 nats per token of fp32 factors on all five models, so right after a compression the shallow state saves a further 2.0–3.9× on top of a 16-bit state (the newest columns stay in fp32 between compressions). Right after a compression, fp16 factors are about as small as a dense fp8 state or smaller and drift less than it on all five models. Below 16 bits, a dense state must be requantized to fp8 after every token, and on the Mamba-2 models its drift rises to as much as 0.075 nats per token at its better rounding. fp8 factors, quantized only at each compression, drift 1.8–4.7× less on Nemotron-3.5-Lightning, Kimi-Linear, GLM-5.3-Flash and Nemotron-3.5-Super and about as much on Qwen3.8-Flash-Next, at 2.0–3.9× fewer bytes right after a compression. Closed-loop drift over long decodes. Corollary 4 bounds the error of the open loop, in which the compressed and the exact recurrence see the same inputs; in the closed loop of autoregressive decoding, a compression error also changes later keys and values. We therefore follow the released state form, with fp32 states as in the exams, on both Qwen3.8-Flash-Next sets, Kimi-Linear, Nemotron-3.5-Lightning and Nemotron-3.5-Super over 16K decode steps of long replies, 512–1,024 compressions (Figure 7). The drift rises over the first few thousand steps and then grows slowly: from the first to the last 4K steps it grows by 1.2–1.4× while the number of compressions grows 4×. After 16K steps it is still below the drift of the official NVFP4 checkpoints over their first 1,024 steps, on all three models that have one.

7. Serving Performance SEED’s two parts relieve different serving bottlenecks, and together they raise the peak throughput per GPU of served Qwen3.8-Flash-Next to 1.16–1.42× the original model’s (Figure 8, Table 5). Shallow prefill lowers the prefill’s FLOPs. Shallow states shrink the recurrent state, which sets every decode step’s state traffic and caps the batch. Where the savings come from. On Qwen3.8-Flash-Next, shallow prefill runs 31 of the 48 layers plus the emitters on every prompt token but the last (33 of 52 on Nemotron-3.5-Lightning, 19 of 19

Shallow Encoding Emits Deep Memories

27 on Kimi-Linear, 31 of 45 on GLM-5.3-Flash and 78 of 88 on Nemotron-3.5-Super). This lowers the parameter FLOPs per prompt token of Qwen3.8-Flash-Next from 12.1 to 9.1 GFLOP. For the skipped layers, it also removes the prefill-side reads of their weights and attention caches. Right after a compression, shallow states reduce the recurrent state of one request from 113.2 to 29.2 MB on Qwen3.8-Flash-Next, from 48.2 to 18.8 MB on Nemotron-3.5-Lightning, from 41.9 to 21.3 MB on Kimi-Linear, from 142.6 to 72.4 MB on GLM-5.3-Flash and from 167.8 to 65.5 MB on Nemotron-3.5-Super. Across the five models, the state is 2.0–3.9× smaller right after a compression and 1.3–2.6× on average (Table 2). A state is read and written at every decode step. In the byte account, a smaller state therefore lowers the per-step state traffic and raises the number of requests that fit in a byte budget. Attention caches remain uncompressed. On these mixture-of-experts checkpoints, the weights dominate a decode step’s bytes up to batches of several hundred requests (Appendix F). The smaller state therefore helps mainly through capacity. Appendix F gives the byte and FLOP account from the checkpoints’ tensor shapes. Setup. Qwen3.8-Flash-Next is served with SGLang on GB300 GPUs. Aggregated serving uses one instance for both phases: NVFP4 on one GPU (tensor parallelism 1, TP1) or bf16 on two (TP2). Disaggregated bf16 serving uses one two-GPU instance per phase (1P:1D), with 16K-token prefill chunks. Every comparison is paired with the original model on the same hardware. The served component sets are the released Qwen3.8-Flash-Next sets of Section 6 (rank 𝑅 = 16, code 8192+256, state compression every 16 steps). Both arms keep their recurrent states in fp32, the precision of the models’ reference implementations, the default of the serving engine and the precision of the exams; the byte ratio of the shallow and the dense state is the same at any common precision. Workload and metrics. The throughput runs reuse cached prefixes. Each measurement uses the InferenceX (formerly InferenceMAX) (SemiAnalysis, 2026b) workload, an agentic-coding trace of recorded Claude Code sessions (cc-traces-weka-062126-256k (SemiAnalysis, 2026a)). The AIPerf client (NVIDIA, 2026a) replays it in agentic-replay mode for 900 seconds at a fixed concurrency, the number of active sessions. Throughput per GPU counts every output token and every prompt token, whether the prefix cache serves it or the prefill computes it, so a cache hit counts the same in both arms. The P90 end-to-end output rate per user, the rate that 90% of requests reach, is 1/P90(end-to-end latency/output tokens). It includes queueing and the time to first token (TTFT). Ratios at an equal per-user rate interpolate each configuration’s throughput linearly in this rate between adjacent concurrencies. Prefix cache. In SGLang, a hybrid model’s prefix cache keeps a slot for every cached prefix in the same recurrent-state pool that the running requests use (Zheng et al., 2024; Pan et al., 2025). When the pool is full, cached slots are evicted, the prefix-cache hit rate falls and the evicted prefixes must be recomputed by the prefill. Smaller states leave room for more cached prefixes and more running requests. Measurements. Each measurement in Figure 8 is a single 900-second run and includes SEED’s own work: the emitters in the prefill and the state compressions in the decode. Up to 16 sessions, where the configurations serve alike, their throughputs agree within 1.3%. At each concurrency, SEED’s curve takes the best of its three measured configurations (shallow states only, shallow prefill only, both) by throughput; in the prefix-cache panels (c, g, k) it takes the highest prefix-cache hit, and in the TTFT panels (d, h, l) the Pareto frontier of all three, the measurements that no other

20

Shallow Encoding Emits Deep Memories

original model

throughput vs. user rate

100

NVFP4 TP1 aggregated K tok/s/GPU

shallow states only

(a)

80

shallow prefill only

SEED (best measured configuration)

throughput vs. concurrency

100

prefix-cache hit (%)

100

(b)

80

60

60

40

40

20

gain over the original

throughput vs. TTFT (d)

80 80

60 40

60

20

20

(c) 0

0

30

60

(e)

K tok/s/GPU

bf16 TP2 aggregated

50

40

30

30

20

20

10

10

0

0

0

30

60

90

120

(i)

K tok/s/GPU

40

0

50

8

16

24

32

40

48

0

8

16

24

32

40

48

(f)

50

40

50

bf16 1P:1D disaggregated

0

90

50

30

20

20

10

1

2

3

(h)

40

80

30 20

60

10

(g) 0

8

16

24

32

40

48

0

8

16

24

32

40

48

(j)

40

30

0 0.5

0 50

0.5

1

2

(l)

40 80

30 20

60

10

10

(k) 0

0 30 60 90 P90 output rate per user (tok/s)

0

0

16 32 48 concurrent sessions

64

0

16 32 48 concurrent sessions

64

0

1 2 5 10 P90 time to first token (s)

Figure 8 | SEED reaches 1.16–1.42× the original model’s peak throughput per GPU, at 8 to 16 more sessions. Served Qwen3.8-Flash-Next with fp32 recurrent states. Rows: aggregated NVFP4 TP1 (a–d), aggregated bf16 TP2 (e–h), disaggregated bf16 1P:1D (i–l). Columns: throughput per GPU against the P90 output rate per user; throughput and prefix-cache hit against concurrency; throughput against the P90 time to first token (Pareto frontiers). Charcoal: the original model; dashed: shallow states alone and shallow prefill alone; green: SEED, the best of its three configurations at each concurrency (§7); shaded: its gain.

measurement beats in both quantities. This curve lies within 1.1% of both parts together wherever those were measured; the ratios in the text and in Table 5 are those of both parts together. The rate panels (a, e, i) draw every measurement with a P90 per-user rate of at least 10 tokens/s, in order of concurrency; the middle columns draw every measurement up to 48 sessions in aggregated and 64 in disaggregated serving. Throughput. Peak to peak, SEED with both parts reaches 1.20× the original model’s throughput per GPU on NVFP4 TP1, 1.16× on bf16 TP2 and 1.42× on 1P:1D. At an equal per-user rate of 40 tokens/s (25 ms per output token, end to end), it delivers 1.23×, 1.26× and 1.42× the original model’s throughput (Table 5). Over per-user rates of 25 to 50 tokens/s, inside the measured range of both arms in all three deployments, the ratio is 1.16–1.26× in aggregated serving and 1.24–1.49× in 1P:1D. In aggregated serving, the two peaks come at similar per-user rates: 23.7 tokens/s for SEED and 22.8 for the original model on NVFP4 TP1, and 26.2 and 26.1 on bf16 TP2. In 1P:1D, SEED’s peak comes at a far higher per-user rate: 22.8 tokens/s, compared with 9.0 for the original model. Beyond the original model’s peak, the gap widens, because its fp32 states fill the state pool in aggregated serving and its prefill instance saturates in 1P:1D. At 40, 44 and 64 sessions, the original model’s P90 per-user rate has fallen below 4 tokens/s, and SEED serves 2.1×, 1.9× and 3.0× the original model’s throughput.

21

Shallow Encoding Emits Deep Memories

Table 5 | SEED raises the peak throughput per GPU of served Qwen3.8-Flash-Next in every deployment (Figure 8). Peak: each configuration’s highest throughput over its measured concurrencies (tokens/s/GPU), its concurrency and its ratio to the original model’s peak. At 𝐶 ⋆ : the throughput ratio at SEED’s peak concurrency (36, 40 and 64 sessions), where the original model is past its own peak (P90 per-user rate 6.8, 6.7 and 1.2 tokens/s). At 40 tok/s: the throughput ratio at an equal P90 per-user rate of 40 tokens/s. peak

at 𝐶 ⋆

at 40 tok/s

ratio

ratio

aggregated, NVFP4 TP1 original 28 shallow states 32 shallow prefill 28 SEED (both) 36

80,219 – – 92,343 1.15× 1.38× 86,385 1.08× 1.17× 95,989 1.20× 1.50×

– 1.16× 1.06× 1.23×

aggregated, bf16 TP2 original shallow states shallow prefill SEED (both)

32 32 32 40

46,492 – – 49,235 1.06× 1.40× 48,301 1.04× 1.20× 54,047 1.16× 1.57×

– 1.18× 1.11× 1.26×

1P:1D, bf16 original shallow states shallow prefill SEED (both)

48 48 56 64

33,906 – – 35,638 1.05× 1.08× 45,853 1.35× 2.71× 48,141 1.42× 2.96×

– 1.04× 1.45× 1.42×

Configuration

sessions

tok/s/GPU

ratio

Aggregated serving. On GPUs shared by both phases, shallow prefill reduces computation, while shallow states raise request capacity. In aggregated NVFP4 TP1 serving, SEED peaks at 36 sessions, where the original model is past its own peak and serves 6.8 tokens/s per user at P90. At this concurrency, the original model falls to 63.9K tokens/s/GPU, while SEED holds 96.0K. Relative to the original model at the same concurrency, shallow prefill alone raises throughput by 17%, shallow states alone by 38% and both together by 50% (Table 5). At 40 sessions, the original model’s prefix-cache hit rate has fallen to 55% and its throughput to 43.8K, while SEED keeps 85% and 92.8K. At an equal per-user rate of 40 tokens/s, SEED delivers 1.23× the original model’s throughput (89.0K versus 72.3K tokens/s/GPU). In aggregated bf16 TP2 serving, the original model’s throughput collapses once its fp32 states fill the state pool: from 46.5K tokens/s/GPU at 32 sessions to 26.3K at 44. There, its prefix-cache hit rate has fallen to 53%; SEED’s is 80%. SEED peaks at 54.0K at 40 sessions, 1.57× the original model’s 34.4K at that concurrency. At an equal per-user rate of 40 tokens/s, it delivers 1.26× the original model’s throughput (51.2K versus 40.7K). Disaggregated serving. In bf16 1P:1D serving, the prefill instance is the bottleneck, and shallow prefill relieves it. Peak throughput rises from 33.9K tokens/s/GPU for the original model to 45.9K with shallow prefill alone and 48.1K with SEED, that is, 1.35× and 1.42× the original peak. Shallow states alone add 5% (35.6K; Table 5). At equal per-user rates of 25 to 40 tokens/s, SEED delivers 1.42–1.49× the original model’s throughput. At 64 sessions, where the prefill instance is the bottleneck, SEED’s smaller states keep more prefixes cached than shallow prefill alone does: its prefix-cache hit rate is 91.7%, compared with 87.5%. This adds 9% (Figure 8j), and SEED serves 2.96× the throughput of the original model, whose hit rate has fallen to 52.2%. The end-to-end gains come with a lower TTFT, which includes queueing, and a higher prefix-cache hit rate (Figure 8).

22

Shallow Encoding Emits Deep Memories

Low load. At low load, the offered load of the replayed trace largely sets the throughput. In aggregated serving, SEED stays within 2% of the original model’s throughput up to 20 sessions and exceeds it by more than 2% from 24 sessions on. Up to 20 sessions, SEED’s P90 output rate per user is 2–3% above the original model’s on bf16 TP2. On NVFP4 TP1, where shallow prefill shortens the time to first token, it lies between −2% and +9% of the original model’s.

8. Related Work Shallow prefill and layer skipping. YOCO (Sun et al., 2024b) and the causal encoder–decoder of DeepSeek-V4.1-Flash (DeepSeek-AI et al., 2026) build a shallow prompt stack into new architectures trained from scratch. For trained transformers, SwiftKV (Qiao et al., 2025) computes the deep keys and values from an earlier layer’s output after fine-tuning the deep projections, POP (He et al., 2026) fills the skipped layers’ caches with their original projections of the last kept residual, SPEED (Oh et al., 2026) fine-tunes the model to decode without the upper layers’ prompt caches, and KV Prediction (Horton et al., 2024) predicts the caches with a small auxiliary model. ADEPT (Yoo et al., 2026) and CALM (Schuster et al., 2022) exit early per token and fill the skipped layers’ keys and values; KV-Direct (Qasim et al., 2026) recomputes them from a checkpointed residual. For a hybrid model, Puzzle-75B (Bercovich et al., 2026) prefills with a narrower jointly trained variant that writes the attention caches and the Mamba-2 states. Unlike the new architectures, SEED finds the depth asymmetry in trained models; unlike the methods for transformers, it also writes recurrent states, where an inexact write persists for every later token; and unlike SwiftKV, SPEED and Puzzle-75B, it changes no checkpoint weight. Depth pruning and conversion with distillation (Muralidharan et al., 2024; Bercovich et al., 2025; Wang et al., 2024) change the network for every token, and prompt compression (Mu et al., 2023; Ge et al., 2024; Jiang et al., 2023) shortens the prompt itself; SEED keeps every layer and every token in the decode. Cache and state compression. Disaggregated serving ships a prompt’s memories between machines (Zhong et al., 2024; Patel et al., 2024; Qin et al., 2025); cache systems page, stream, store and merge them (Kwon et al., 2023; Zheng et al., 2024; Liu et al., 2024a; 2025; Yao et al., 2025), and Marconi (Pan et al., 2025) brings prefix caching to hybrid models. KV-cache compression (Liu et al., 2024b; Hooper et al., 2024; Li et al., 2024; Zhang et al., 2023) shrinks the output of a full prefill. SEED keeps every attention cache at full precision, so all of these apply to its caches unchanged. For recurrent layers, KVBuffer (Zou & Zhong, 2026), DeltaLog (Lin et al., 2026) and ReplaySSM (Liou et al., 2026) keep the recent writes or layer inputs next to a dense state and fold them in periodically, which cuts the per-step traffic at unchanged outputs but not the state that a request holds; SEED keeps the recent writes next to a rank-𝑅 summary and the sink, so the state itself shrinks. Nazari & Rusch (2026) analyse the rank of trained linear-attention states and prune their key and query projections, Sun et al. (2026) find both low-rank and high-rank heads in Qwen3-Next, DASC (Yu et al., 2026) keeps the long-horizon heads and channels of stored prefix states, and GHOST (Menezes & Kyrillidis, 2026) prunes Mamba-2 state dimensions. SEED keeps every head and factors the active state at run time, around a sink that the update preserves exactly. Sinks. Xiao et al. (2024) identified the attention sink, Gu et al. (2025) traced it in part to the softmax normalization, Sun et al. (2024a) linked it to massive activations, Qiu et al. (2026a) unify attention and residual sinks, and Qiu et al. (2025) remove it with a gate in their models. The state sink is its counterpart in recurrent layers: a direction of the state rather than a position in the sequence, which through the head’s normalization scales the content as a sink logit does through a softmax denominator. Zhuang et al. (2026) call the steady state of a repeated key in gated delta 23

Shallow Encoding Emits Deep Memories

networks a semantic state sink and mitigate it; Lemma 1 contains this case. Meng et al. (2024) build a sink into a state-space model, and Jiang & Zhang (2026) use the name “state sink” for a Δ gate that fires on boundary tokens. Wagh & Singh (2026) find that the final states of a frozen Mamba-130M point in nearly the same direction for unrelated sentences.

9. Limitations and Conclusion Limitations. SEED compresses no attention cache. On hybrids whose memory at long context is dominated by their few attention layers, the shallow state’s share of the saving shrinks; there SEED composes with KV-cache quantization, eviction and offloading, which act on exactly the memories it leaves untouched. The anchor depth is set per checkpoint from untrained probes rather than learned, and lands at 63–89% of the depth on the five models; how shallow a residual stream can determine every deep memory, as a function of architecture and training, is open. The open-loop bound is a worst case set by a head’s largest decay, and the measured closed-loop drift grows far more slowly than it allows (§6.1); a closed-loop theory of repeated compression would explain why. Our serving gains are measured on mixture-of-experts checkpoints, whose weights dominate a decode step’s bytes up to batches of several hundred requests, so the smaller state pays off mainly through capacity (Appendix F); dense (non-mixture-of-experts) hybrids, with smaller weights, would also see the per-step traffic saving. Three steps follow directly: serving the shallow state in fp16, which keeps its drift (§6.1) and halves its bytes again; a prefix cache that stores a prompt’s code in place of its deep caches, up to 22% fewer bytes at bf16 caches (Table 18); and training beyond 16K-token windows with the periodic compression in the loop. Conclusion. Two properties of trained hybrids locate the compute and HBM that serving them spends beyond what the output requires. Writes are easier than reads, so a prompt’s deep memories can be emitted from a shallow anchor layer. A recurrent state is a sink plus a short working memory, a form that the model’s own update preserves exactly. SEED turns both into a serving method for frozen checkpoints: it matches the original model’s scores on six checkpoints of five production models and raises the throughput per GPU of Qwen3.8-Flash-Next 1.2–1.4× on an agentic-coding trace. Neither property was designed into these models; both emerge from training, which leaves a hybrid’s memories far simpler than the computation and storage that serve them.

References Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. Ahmad, W. U., Narenthiran, S., Majumdar, S., Ficek, A., Jain, S., Huang, J., Noroozi, V., and Ginsburg, B. OpenCodeReasoning: Advancing data distillation for competitive coding. In Conference on Language Modeling (COLM), 2025. arXiv:2504.01943. Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3639–3664, 2025. arXiv:2412.15204. Ben Allal, L., Lozhkov, A., Bakouch, E., Martín Blázquez, G., Penedo, G., Tunstall, L., Marafioti, A., Piqueres Lajarín, A., Kydlíček, H., Srivastav, V., et al. SmolLM2: When smol goes big – 24

Shallow Encoding Emits Deep Memories

data-centric training of a fully open small language model. In Conference on Language Modeling (COLM), 2025. arXiv:2502.02737. Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. Bercovich, A., Ronen, T., Abramovich, T., Ailon, N., Assaf, N., Dabbah, M., Galil, I., Geifman, A., Geifman, Y., Golan, I., et al. Puzzle: Distillation-based NAS for inference-optimized LLMs. In International Conference on Machine Learning (ICML), 2025. arXiv:2411.19146. Bercovich, A., Abramovich, T., Afrimi, D., Aharon, S., Ailon, N., Anisimov, V., Argov, O. U., Ashkenazi, M., Asida, T., Assaf, N., et al. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing hybrid MoE LLMs. arXiv preprint arXiv:2607.04371, 2026. Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782, 2024. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024. DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., et al. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. DeepSeek-AI, Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., et al. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. DeepSeek-AI, Xu, A., Li, B., Lin, B., Xue, B., Xian, B., Xu, B., Wu, B., Zhang, B., Deng, B., et al. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. Technical report, DeepSeek, 2026. arXiv:2609.19969. Dutta, A., Krishnan, S., Kwatra, N., and Ramjee, R. Accuracy is not all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2407.09141. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https://transformer-circuits.pub/2021/framework/index.html. Ge, T., Hu, J., Wang, L., Wang, X., Chen, S.-Q., and Wei, F. In-context autoencoder for context compression in a large language model. In International Conference on Learning Representations (ICLR), 2024. arXiv:2307.06945. 25

Shallow Encoding Emits Deep Memories

GLM-5-Team, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., Huang, C., et al. GLM-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Gu, X., Pang, T., Du, C., Liu, Q., Zhang, F., Du, C., Wang, Y., and Lin, M. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations (ICLR), 2025. arXiv:2410.10781. Gu, Y., Dong, L., Wei, F., and Huang, M. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations (ICLR), 2024. Guha, E., Marten, R., Keh, S., Raoof, N., Smyrnis, G., Bansal, H., Nezhurina, M., Mercat, J., Vu, T., Sprague, Z., et al. OpenThoughts: Data recipes for reasoning models. In International Conference on Learning Representations (ICLR), 2026. arXiv:2506.04178. He, J., Fu, Z., Wang, J., and Li, Q. POP: Prefill-only pruning for efficient large model inference. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 5864–5877, 2026. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021a. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021b. arXiv:2103.03874. Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2401.18079. Horton, M., Cao, Q., Sun, C., Jin, Y., Mehta, S., Rastegari, M., and Nabi, M. KV prediction for improved time to first token. arXiv preprint arXiv:2410.08391, 2024. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024. arXiv:2404.06654. Hugging Face. Open R1: A fully open reproduction of DeepSeek-R1. https://github.com/huggingface/open-r1, 2025. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations (ICLR), 2025. arXiv:2403.07974. Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2310.05736. Jiang, Y. and Zhang, B. A circuit, not the circuit: Non-unique causal localisation of the Mamba-2 state sink. arXiv preprint arXiv:2606.00930, 2026. Kimi Team, Zhang, Y., Lin, Z., Yao, X., Hu, J., Meng, F., Liu, C., Men, X., Yang, S., Li, Z., et al. Kimi Linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025.

26

Shallow Encoding Emits Deep Memories

Ko, J., Chen, T., Kim, S., Ding, T., Liang, L., Zharkov, I., and Yun, S.-Y. DistiLLM-2: A contrastive approach boosts the distillation of LLMs. In International Conference on Machine Learning (ICML), 2025. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In ACM Symposium on Operating Systems Principles (SOSP), 2023. Kydlíček, H. Math-Verify: Math verification library. https://github.com/huggingface/math-verify, 2025. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tülu 3: Pushing frontiers in open language model post-training. In Conference on Language Modeling (COLM), 2025. arXiv:2411.15124. Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2404.14469. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050. Lin, J., Sun, J., and Sun, G. DeltaLog: Deferred materialization of recurrent states for linear attention decoding. arXiv preprint arXiv:2608.15533, 2026. Liou, Z.-W., Nover, H., Naik, A., Skliar, A., Shovkun, I., Shahaf, I., and Dao, T. ReplaySSM: Cache SSM inputs, not state. arXiv preprint arXiv:2610.13778, 2026. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. CacheGen: KV cache compression and streaming for fast large language model serving. In ACM SIGCOMM Conference (SIGCOMM), 2024a. arXiv:2310.07240. Liu, Y., Cheng, Y., Yao, J., An, Y., Chen, X., Feng, S., Huang, Y., Shen, S., Zhang, R., Du, K., and Jiang, J. LMCache: An efficient KV cache layer for enterprise-scale LLM inference. arXiv preprint arXiv:2510.09665, 2025. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning (ICML), 2024b. arXiv:2402.02750. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101. Mathematical Association of America. AIME 2024 and 2025 problem sets. https://huggingface.co/datasets/HuggingFaceH4/aime_2024, https://huggingface.co/datasets/math-ai/aime25, 2025. McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947.

27

Shallow Encoding Emits Deep Memories

Menezes, M. and Kyrillidis, A. GHOST: Unmasking phantom states in Mamba2 via grouped hidden-state output-aware selection & truncation. In International Conference on Machine Learning (ICML), 2026. arXiv:2602.11408. Meng, T., Tao, Y., and Yin, W. Enhanced structured state space models via grouped FIR filtering and attention sink mechanisms. arXiv preprint arXiv:2408.00244, 2024. MiniMax, Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., et al. MiniMax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313, 2025. Moshkov, I., Hanley, D., Sorokin, I., Toshniwal, S., Henkel, C., Schifferer, B., Du, W., and Gitman, I. AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset. arXiv preprint arXiv:2504.16891, 2025. Mu, J., Li, X. L., and Goodman, N. Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2304.08467. Muralidharan, S., Turuvekere Sreenivas, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2407.14679. Nazari, P. and Rusch, T. K. On state reduction in linear attention, 2026. arXiv:2602.04852. nostalgebraist. Interpreting GPT: The logit lens. LessWrong, https: //www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, 2020. NVIDIA. OpenScienceReasoning-2. Dataset card, https://huggingface.co/datasets/nvidia/OpenScienceReasoning-2, 2025. NVIDIA. AIPerf. https://github.com/ai-dynamo/aiperf, 2026a. NVIDIA. NVIDIA Nemotron 3.5 Super VL 120B-A12B. Model card, https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, 2026b. NVIDIA. NVIDIA Nemotron 3.5 Lightning 30B-A3B. Model card, https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, 2026c. NVIDIA, Abecassis, F., Agrusa, A., Ahn, D., Alben, J., Alborghetti, S., Andersch, M., Arayandi, S., Bjorlin, A., Blakeman, A., et al. Pretraining large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025a. NVIDIA, Blakeman, A., Basant, A., Khattar, A., Renduchintala, A., Bercovich, A., Ficek, A., Bjorlin, A., Taghibakhshi, A., Deshmukh, A. S., et al. Nemotron-H: A family of accurate and efficient hybrid Mamba-Transformer models. arXiv preprint arXiv:2504.03624, 2025b. Oh, J., Jeon, H., Ji, H., Kong, K., and Lee, J.-Y. Shallow prefill, deep decoding: Efficient long-context inference via layer-asymmetric KV visibility. arXiv preprint arXiv:2605.06105, 2026. OpenAI. simple-evals. https://github.com/openai/simple-evals, 2024. Pan, R., Wang, Z., Jia, Z., Karakus, C., Zancato, L., Dao, T., Wang, Y., and Netravali, R. Marconi: Prefix caching for the era of hybrid LLMs. In Proceedings of Machine Learning and Systems (MLSys), 2025. arXiv:2411.19379. 28

Shallow Encoding Emits Deep Memories

Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient generative LLM inference using phase splitting. In ACM/IEEE International Symposium on Computer Architecture (ISCA), pp. 118–132, 2024. Penedo, G., Kydlíček, H., Ben Allal, L., Lozhkov, A., Mitchell, M., Raffel, C., von Werra, L., and Wolf, T. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, 2024. Pyatkin, V., Malik, S., Graf, V., Ivison, H., Huang, S., Dasigi, P., Lambert, N., and Hajishirzi, H. Generalizing verifiable instruction following. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, 2025. arXiv:2507.02833. Qasim, K. U., Zhang, J., Shaheen, M. K., Alharith, R., and Zhang, H. The residual stream is all you need: On the redundancy of the KV cache in transformer inference, 2026. arXiv:2603.19664. Qiao, A., Yao, Z., Rajbhandari, S., and He, Y. SwiftKV: Fast prefill-optimized inference with knowledge-preserving model transformation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 25734–25753, 2025. arXiv:2410.03960. Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In USENIX Conference on File and Storage Technologies (FAST), pp. 155–170, 2025. Qiu, Z., Wang, Z., Zheng, B., Huang, Z., Wen, K., Yang, S., Men, R., Yu, L., Huang, F., Huang, S., et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2505.06708. Qiu, Z., Huang, Z., Wen, K., Jin, P., Zheng, B., Zhou, Y., Huang, H., Wang, Z., Li, X., Zhang, H., et al. A unified view of attention and residual sinks: Outlier-driven rescaling is essential for large language model training. In Conference on Language Modeling (COLM), 2026a. URL https://openreview.net/forum?id=rwk3hL5lS2. arXiv:2601.22966. Qiu, Z., Wang, Z., Li, X., Li, Y., Xu, Y., Wang, Y., Zhang, H., Men, R., Mao, B., Zhang, C., et al. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026b. Qwen Team. Qwen3-Next: Towards ultimate training & inference efficiency. Qwen blog, https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, 2025. Qwen Team. Qwen3.5-4B-Base and Qwen3.5-9B-Base. Model cards, https://huggingface.co/Qwen/Qwen3.5-4B-Base, https://huggingface.co/Qwen/Qwen3.5-9B-Base, 2026. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling (COLM), 2024. arXiv:2311.12022. Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial Winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 8732–8740, 2020. arXiv:1907.10641. Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V. Q., Tay, Y., and Metzler, D. Confident adaptive language modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 29

Shallow Encoding Emits Deep Memories

SemiAnalysis. cc-traces-weka-062126-256k: WekaTrace corpus derived from SemiAnalysis Claude Code proxy traces. https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126-256k, 2026a. SemiAnalysis. InferenceX (formerly InferenceMAX): Open source inference research platform. https://github.com/SemiAnalysisAI/InferenceX, 2026b. Sun, A., Zhang, H., Zhou, H., Ma, Y., Qin, Y., Su, T., Liu, Y., Ma, Z., Xu, J., Gao, J., Hao, J., and He, R. State rank dynamics in linear attention LLMs, 2026. arXiv:2602.02195. Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. In Conference on Language Modeling (COLM), 2024a. arXiv:2402.17762. Sun, Y., Dong, L., Zhu, Y., Huang, S., Wang, W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024b. arXiv:2405.05254. Wagh, B. and Singh, A. Lost in state space: Probing frozen Mamba representations. arXiv preprint arXiv:2605.00253, 2026. Wang, J., Paliotta, D., May, A., Rush, A. M., and Dao, T. The Mamba in the Llama: Distilling and accelerating hybrid models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2408.15237. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024. Xie, Z., Wei, Y., Cao, H., Zhao, C., Deng, C., Li, J., Dai, D., Gao, H., Chang, J., Yu, K., et al. mHC: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta networks: Improving Mamba2 with delta rule. In International Conference on Learning Representations (ICLR), 2025b. Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q., Du, K., Lu, S., and Jiang, J. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. In European Conference on Computer Systems (EuroSys), pp. 94–109, 2025. arXiv:2405.16444. Yoo, S., Malla, S., Choi, C., Lu, W. D., and Choi, J. H. ADEPT: Adaptive dynamic early-exit process for transformers. arXiv preprint arXiv:2601.03700, 2026. Yu, Y., Sun, P., Tan, J., Zhang, T., Xie, Y., Cai, X., and Liu, Y. DASC: Decay-aware state compression for hybrid linear-attention serving, 2026. arXiv:2608.30386. Z.ai. GLM-4.6V-Flash. Model card, https://huggingface.co/zai-org/GLM-4.6V-Flash, 2025. Z.ai. GLM-5.3-Flash. Model card, https://huggingface.co/zai-org/GLM-5.3-Flash, 2026. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. arXiv:1905.07830. Zhang, B. and Sennrich, R. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1910.07467. 30

Shallow Encoding Emits Deep Memories

Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H2 O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2306.14048. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Zhu, D., Huang, H., Huang, Z., Zeng, Y., Mao, Y., Wu, B., Min, Q., and Zhou, X. Hyper-connections. In International Conference on Learning Representations (ICLR), 2025. Zhuang, Z., Wei, Z., Dai, J., Chen, J., Pan, F., Jiang, P., and Gai, K. SinkRec: Mitigating semantic state sink in long sequence recommendation with memory-conditioned gated delta networks, 2026. arXiv:2606.09888. Zou, L. and Zhong, L. KVBuffer: IO-aware serving for linear attention. arXiv preprint arXiv:2605.19049, 2026.

A. Prefill Depth: Measurements This appendix gives the protocols behind §3. Appendix A.1 turns the measurements into the anchor depth, the code and the state rank. Protocol. Unless stated otherwise, a measurement uses 48 held-out 4K windows of web text, each a prompt of about 3.5K tokens followed by 256 teacher-forced continuation tokens. It is scored by the per-token KL divergence KL(𝑝 ‖ 𝑝^) between the original model’s next-token distribution 𝑝 and the modified model’s 𝑝^, or by the change in next-token cross-entropy. The KLs of the layer skips, the untrained shallow prefill and the trained sets, and the decode drift of Qwen3.8-Flash-Next, Nemotron-3.5-Lightning and Kimi-Linear, sum over the original model’s 1,024 most likely tokens and one bucket for the rest. This is a lower bound on the full KL and close to it: 0.079 against 0.081 for the untrained shallow prefill on Qwen3.8-Flash-Next, 0.093 against 0.094 on Nemotron-3.5-Lightning. The early exit’s KL sums over the full vocabulary. In the untrained shallow prefill, every memory-holding layer at index 𝑘 or above writes its memory through its own untrained write path from the exact ℎ(𝑘) , without a code; its increase in cross-entropy is the untrained gap gap(𝑘) (Figure 11). Phase-specific layer skips. A prompt-side skip removes layer ℓ’s output for the prompt tokens while the layer still writes its memory; a decode-side skip removes it for the continuation tokens, after an exact prompt. The skips sample every third layer of Qwen3.8-Flash-Next, Nemotron-3.5-Lightning and GLM-5.3-Flash, every fourth of Nemotron-3.5-Super and every layer of Kimi-Linear, and score continuation steps 1–255 (GLM-5.3-Flash: 16 windows, steps 1–63). The early exit applies the output head to ℎ(𝑘) on 64 windows (GLM-5.3-Flash: 32). Nemotron-3.5-Super’s engine is not deterministic; its values include a run-to-run floor of 0.005.

31

KL to the original (nats / token)

Shallow Encoding Emits Deep Memories

(a) Nemotron-3.5-Lightning

(b) Nemotron-3.5-Super

k = 33

10

(c) GLM-5.3-Flash k = 78

k = 31

1 0.1 0.01 run-to-run floor 0.001 0

10

20

30

40

50

0

20

skipped layer (of 52)

40

60

80

0

10

skipped layer (of 88)

layer skipped for the prompt tokens (memory still written)

20

30

40

skipped layer (of 45) layer skipped for the generated tokens

Figure 9 | The write–read asymmetry of Figure 2a, b holds on Nemotron-3.5-Lightning, Nemotron-3.5Super and GLM-5.3-Flash. KL from the original model when one layer’s output is skipped only for the prompt tokens, its memory still written (green), or only for the generated tokens (black); sampled layers and windows as in Appendix A. Nemotron-3.5-Super’s values include its engine’s run-to-run floor (grey). Dashed: the anchor depth.

(a) one layer skipped in one phase

(b) direct-answer core

1 0.1 0.01 0.001 0.0001 0

10

20

30

0

skipped layer (Qwen3-8B-Base, 36 layers) 0.01 nats skipped for the prompt (memory still written) skipped for the generated tokens

100

(c) needle recall needle − original (percentage points)

10

core avg (% of original)

held-out ΔNLL (nats / token)

Four smaller checkpoints, one of them a transformer. The asymmetry is not specific to the hybrids we serve (Figure 10). On Qwen3-8B-Base (Yang et al., 2025a), Qwen3.5-4B-Base and Qwen3.5-9B-Base (Qwen Team, 2026) and GLM-4.6V-Flash (Z.ai, 2025) on text, an untrained shallow prefill at 50–75% of the layers keeps the direct-answer core within 2% of the original model and loses mainly needle recall. The core averages GSM8K (Cobbe et al., 2021), HellaSwag (Zellers et al., 2019), ARC-Challenge (Clark et al., 2018) and WinoGrande (Sakaguchi et al., 2020) (without GSM8K for the instruction-tuned GLM-4.6V-Flash); needle recall is the minimum over prompts of 2K, 4K and 8K tokens, 64 per length (256 for Qwen3.5-9B-Base at 16 layers). Training the write paths of Qwen3-8B-Base at 22 of 36 layers on 1B tokens, without a code, recovers its needle loss.

90 80 70 60

0

−3 points

−20 −40 −60 −80

25

50

75

anchor depth (% of layers) Qwen3-8B-Base (36L) Qwen3-8B-Base (36L), trained Qwen3.5-4B-Base (32L)

25

50

75

anchor depth (% of layers) Qwen3.5-9B-Base (32L) GLM-4.6V-Flash, text only (40L)

Figure 10 | Without training, a shallow prefill keeps a model’s direct answers and loses mainly retrieval. (a) Qwen3-8B-Base: held-out NLL change when one layer is skipped only for the prompt tokens (memory still written) or only for the generated tokens; the last layer’s prompt-side change is exactly 0 (marked below the axis). (b, c) Untrained shallow prefill at each anchor depth (hollow, dashed); filled: Qwen3-8B-Base at 22 of 36 layers with trained write paths. (b) Core average, % of the original model. (c) Needle recall minus the original model’s; dotted: −3 points. A checkpoint’s markers share a horizontal shift of at most 1.4 points of depth. Inexact writes. Table 6 gives the KLs and cache errors of §3; Table 14 gives the emitted states’ errors.

A.1. The Selection Rule Every model’s anchor depth, code and state rank follow from one rule, with thresholds shared by all models. This subsection states its criteria (§3.1) exactly, with their reasons; Table 7 lists the 32

Shallow Encoding Emits Deep Memories

Table 6 | Every released set stays close to the original model although its memories are inexact. Teacher-forced KL to the original model (nats per token) above the floor, the KL of a second run of the original model (nonzero where the engine is not deterministic); windows of Appendix A (GLM5.3-Flash: 16 windows, 63 tokens). As served: the emitters read the decoded code and every state is compressed as in the decode. States exact: no compression. Code bypassed: the emitters read the exact ℎ(𝑘) (GLM-5.3-Flash has no code). Cache error: relative squared error of the emitted attention caches at the first emitted layer, whose original write path itself reads ℎ(𝑘) , and beyond it (Nemotron-3.5-Super: no attention layer beyond; its code keeps 512 of 4,096 coordinates). KL above the floor Set Qwen3.8-Flash-Next bf16 Qwen3.8-Flash-Next-NVFP4 Nemotron-3.5-Lightning Kimi-Linear GLM-5.3-Flash Nemotron-3.5-Super

cache error (%)

floor

as served

states exact

code bypassed

first emitted layer

beyond

0.0033 0.0206 0.0018 0.0016 0.0000 0.0047

0.0359 0.0393 0.0145 0.0063 0.0330 0.0099

0.0216 0.0224 0.0100 0.0038 0.0318 0.0062

0.0411 0.0445 0.0269 0.0119 – 0.0132

1.4–2.2 1.7–2.6 0.5–4.0 1.2 0.0–0.8 3.8–38.0

9.0–66.9 10.0–73.5 11.6–28.6 11.5–67.7 2.2–79.5 –

measurements and the resulting choices. The rule. The rule reads only the original model, its untrained shallow prefill (Appendix A) and closed-form codes (the principal-component codes that initialize training), never an exam or a trained set; the NVFP4 Qwen3.8-Flash-Next set reuses the bf16 measurements. The copy probe hides one record, “Record ⟨key⟩ is filed under the identifier ⟨UUID⟩”, at a random depth of 4K to 16K tokens of held-out web text, and counts the items whose identifier the greedy reply copies exactly (300 items on Qwen3.8-Flash-Next, 1,500 on Nemotron-3.5-Lightning, Kimi-Linear and Nemotron-3.5-Super, 600 on GLM-5.3-Flash).

1. Depth. The candidates are the attention-layer boundaries, whose first emitted layer is a softmaxattention layer. 𝑘 is the shallowest measured boundary at which (D1) the untrained gap is at most 0.075 nats per token and (D2) the untrained shallow prefill copies at least three quarters of the identifiers that the original model copies. D1 gives the prefill three quarters of a budget of a tenth of a nat per token (a perplexity rise of about 10%) for the untrained shallow prefill and shallow state together; S lets the state add at most a quarter of the prefill’s gap, so the sum stays within it (0.075 · 5/4 ≈ 0.094). Boundaries below the first measured one (23 on Qwen3.8-Flash-Next, 19 on Nemotron-3.5-Lightning and GLM-5.3-Flash, 11 on Kimi-Linear) are taken to fail D1, as the gap falls with depth at every measured boundary up to 𝑘; on Nemotron-3.5-Super every boundary is measured. ^ 2. Exact values (M). 𝑚 = 256 if the closed-form code’s relative reconstruction error ‖Δ− Δ‖/‖Δ−𝜇‖ at 𝑚 = 128 is larger on chat-template tokens than on other tokens of public chat prompts, else 𝑚 = 128; 𝑚 = 0 without a code. The error is measured at the rank that C1 and C2 select (on GLM-5.3-Flash at 4608, its largest probed rank within the cap), so 𝑟 and 𝑚 are checked jointly. 3. Code. Let 𝑑 be the width of the residual stream. (C3) If the full offset’s code takes no more bytes than even the fp8 deep cache, 𝑟 = 𝑑. Otherwise 𝑟 is the smallest probed rank whose closed-form code (C1) shows no significant copy loss against the uncoded shallow prefill and (C2) takes no more bytes than the bf16 deep cache, which a stored code replaces (Table 18), so a larger code would save nothing. A loss means more identifiers lost than gained with 𝑝 < 0.10 in the exact two-sided McNemar test (McNemar, 1947), a one-sided level of 0.05 against a loss. (C4) If no rank qualifies, there is no code. 4. State (S). 𝑅 is the smallest of 8, 16 and 32 whose state drift is at most gap(𝑘)/4, else 32; 𝑊 = 𝑅. The state drift (§3.1) is the per-token KL(𝑝 ‖ 𝑝˜) from exact states (𝑝) to the shallow form (˜ 𝑝), including the original model’s run-to-run floor. The criteria on the five models. Depth (Figure 11). On GLM-5.3-Flash, D2 rejects 27, whose gap passes D1 (0.045). On Nemotron-3.5-Super, D1 rejects 69 (0.091) and accepts 78 (0.065). Code. C1 sets the ranks of Qwen3.8-Flash-Next (8192, the smallest probed passing rank), Nemotron-3.5-Lightning 33

Shallow Encoding Emits Deep Memories

(2048) and Nemotron-3.5-Super (512). C3 sets Kimi-Linear’s 𝑟 = 𝑑 by bytes alone. On GLM-5.3-Flash every probed rank loses significantly (2048–4608 and 8192); the unprobed ranks up to the cap of C2 (6144) lie between failing ones, and on every probed grid copying grows with the rank, so C4 removes the code. Exact values. The template clause fires on Qwen3.8-Flash-Next (0.180 against 0.153) and on GLM-5.3-Flash (0.488 against 0.384). State (Figure 12). Kimi-Linear’s rank 16 drifts 0.0096 against its budget 0.0077, so it gets rank 32. GLM-5.3-Flash’s budget (0.0087) lies below its floor (0.018), so the fallback gives 32. An ablation set of Kimi-Linear (released code, states trained at rank 8) loses 5.2 ± 2.0 AIME points against the original model when it decodes at rank 16 every 16 steps, and 1.5 ± 2.1 at the criterion’s rank 32 every 32 steps (paired by question and seed, eight repetitions). Table 7 | Inputs and choices of the selection rule (Appendix A.1). The Qwen3.8-Flash-Next column serves its bf16 and NVFP4 sets (probes on bf16). Gap: the untrained gap at the attention-layer boundaries, nats per token. Copies: identifiers copied by the untrained shallow prefill / the original model. Closed form: items lost / gained by a closed-form code of rank 𝑟 against the uncoded shallow prefill (𝑝: exact McNemar). Bytes: nominal per token; in parentheses, over the bf16 / fp8 deep cache. Template / other: relative reconstruction error at 𝑚 = 128 and the selected rank. Full offset: the code of rank 𝑑, over the fp8 deep cache. Drift: decode drift at rank 𝑅, 𝑊 = 𝑅, against gap(𝑘)/4. Qwen3.8Flash-Next

Nemotron-3.5Lightning

KimiLinear

GLM-5.3Flash

Nemotron-3.5Super

depth 𝑘 gap at boundary 23: 0.102 27: 0.100 31: 0.074

19: 0.182 26: 0.127 33: 0.075

11: 0.174 15: 0.078 19: 0.031

58: 0.216 69: 0.091 78: 0.065

copies

31: 290 / 297

33: 1,498 / 1,500

19: 1,500 / 1,497

19: 0.172 23: 0.115 27: 0.045 31: 0.035 27: 269 / 595 31: 530 / 595

4096: 227 / 2 (𝑝 < 0.001) 6144: 102 / 7 (𝑝 < 0.001) 8192: 15 / 7 (𝑝 = 0.134) 10240: 4 / 4 (𝑝 = 1.000) 8192+256: 5,380 (0.509 / 1.019)

1024: 17 / 1 (𝑝 < 0.001) 1536: 12 / 1 (𝑝 = 0.003) 2048: 5 / 1 (𝑝 = 0.219) 2688: 2 / 0 (𝑝 = 0.500) 2048+128: 1,540 (0.752 / 1.504)

1024: 0 / 0 1536: 0 / 0 2048: 0 / 0 2304: 0 / 0

2048–4608: at least 528 / 0 (𝑝 < 0.001) 8192: 144 / 11 (𝑝 < 0.001)

256: 5 / 0 (𝑝 = 0.0625) 512: 3 / 0 (𝑝 = 0.250)

2304+128: 1,684 (0.487 / 0.975)

512+128: 676 (0.660 / 1.320)

full offset 10240+256: 1.237 template / other 0.180 / 0.153 (1.18)

2688+128: 1.855 0.212 / 0.245 (0.86)

2304+128: 0.975 0.075 / 0.078 (0.97)

none; 8192+256 would take 5,380 (1.272 / 2.468) 16384+256: 4.582 0.488 / 0.384 (1.27) at 𝑟 = 4608

state 𝑅, 𝑊 drift, 𝑅 = 8 drift, 𝑅 = 16 drift, 𝑅 = 32 gap(𝑘)/4

0.0209 0.0073 0.0031 0.0184

0.0281 0.0087 0.0029 0.0187

0.0325 0.0096 0.0030 0.0077

0.0865 0.0360 0.0214 0.0087

0.0408 0.0140 0.0054 0.0162

settings 𝑘 𝑟+𝑚 𝑅/𝑊

31 8192+256 16 / 16

33 2048+128 16 / 16

19 2304+128 32 / 32

31 no code 32 / 32

78 512+128 16 / 16

code 𝑟+𝑚 closed form

bytes

78: 1,494 / 1,496

4096+128: 5.258 0.514 / 0.660 (0.78)

Why copying is a separate criterion. The gap, a code’s cost and a closed-form subspace’s variance average over tokens or directions, but a 36-character identifier is copied only if each of its many tokens is. A per-token error rate 𝜂 that an average barely registers compounds to 1 − (1 − 𝜂)𝑛 ≈ 𝑛𝜂 over 𝑛 independent tokens. On Qwen3.8-Flash-Next, the code 4096+512 costs +0.0103 nats per token, under a quarter of the gap, and holds 0.853 of the offset’s variance, yet loses 227 of the 290 identifiers that the uncoded shallow prefill copies. Copying needs the directions that carry a token’s identity, which need not hold much variance. Training keeps the difference: on RULER’s UUID needle tasks, trained sets with the code 4096+512 miscopy far more identifiers than the released sets with 8192+256 (Table 8), and on GLM-5.3-Flash a preliminary trained set with the code 5632+256 at depth 27 copies 124 identifiers, against 528 for a like set without a code (𝑝 = 3 × 10−118 ): training does not repair a 34

Shallow Encoding Emits Deep Memories

untrained gap (nats / token)

(a) Qwen3.8-Flash-Next k = 31 by D1 k = 31

0.12

(b) Nemotron-3.5-Lightning k = 33 by D1 k = 33

0.20

0.00

0.10

0.10

0.10

0.05

0.05

0.05

28

32

k = 78

0.4

0.2 fails D2

0.00 24

(e) Nemotron-3.5-Super k = 78 by D1

k = 31

0.20 0.15

0.04 0.02

k = 19

0.20 0.15

0.15

0.06

(d) GLM-5.3-Flash k = 31 by D2

0.6

0.10 0.08

(c) Kimi-Linear k = 19 by D1

0.00

0.00

36

20 25 30 35 40

anchor depth k

12 15 18 21 24

anchor depth k

0.0 18

anchor depth k

24

30

36

42

anchor depth k

20

40

60

80

anchor depth k

(f) D2: copies kept by the untrained shallow prefill

copies kept

1.00

0.98

1.00

1.00

1.00 1.00 0.89

0.75

Qwen3.8-Flash-Next

threshold (D1 0.075, D2 3/4)

Nemotron-3.5-Lightning

attention boundary

Kimi-Linear

chosen k

GLM-5.3-Flash

other depth

Nemotron-3.5-Super

only boundaries measured passes D1, fails D2

0.45

0.50 0.25

0.11 0.00

0.00 31

33

19

27

31

47

58

69

78

anchor depth k

Figure 11 | The anchor depth is the shallowest boundary that passes both depth clauses. (a)–(e) The untrained gap at the attention-layer boundaries (± standard error, grey lines) and other depths (open markers; GLM-5.3-Flash and Nemotron-3.5-Super: boundaries only); black: the D1 threshold; dashed: the chosen 𝑘. (f) D2: identifiers copied by the untrained shallow prefill over those copied by the original model; threshold 3/4. On GLM-5.3-Flash, 27 passes D1 and fails D2; on Nemotron-3.5-Super, every boundary below 78 fails D1. code that miscopies. Table 8 | RULER’s two UUID tasks on Qwen3.8-Flash-Next: wrong replies out of 100 documents per task and length, niah_single_3 / niah_multikey_3 (the original model: none wrong). The rows with the code 4096+512 are an earlier, unreleased bf16 set after each stage. Set

Code

32K

64K

128K

NVFP4, released bf16, unreleased, stage 1 bf16, unreleased, stage 2 bf16, released

8192+256 4096+512 4096+512 8192+256

1/0 21 / 4 22 / 3 1/0

1/1 18 / 4 12 / 5 1/2

4/5 19 / 11 16 / 15 1/5

Why a budget of a quarter of the gap. D1 assumes that the KL costs of the shallow prefill and of the compressed states add. To second order, they add when the errors are uncorrelated in the Fisher metric of the output distribution. On three preliminary GLM-5.3-Flash sets (4 comparisons), a set’s KL with its state form is 0.85–0.88 of the sum of its KL with exact states and the state form’s KL on the original model. The costs combine sub-additively, so assuming additivity errs on the safe side. The byte account. The deep cache takes 10,560 / 5,280 bytes per token in bf16 / fp8 on Qwen3.8Flash-Next, 2,048 / 1,024 on Nemotron-3.5-Lightning, 3,456 / 1,728 on Kimi-Linear, 4,228 / 2,180 on GLM-5.3-Flash and 1,024 / 512 on Nemotron-3.5-Super. A code of rank 𝑟 (a multiple of 16) with 9 𝑚 exact values takes nominally 16 𝑟 + 3𝑚 + 4 bytes per token: half a byte per NVFP4 coordinate plus an 8-bit scale per sixteen, a bf16 value and a one-byte index per exact value, and one fp32 scale per token. Each index counts the residual-stream positions skipped since the previous exact value;

35

Shallow Encoding Emits Deep Memories

state drift (KL, nats / token)

(a) Qwen3.8-Flash-Next R = 16 by the budget 0.05 0.02 0.01

(b) Nemotron-3.5-Lightning R = 16 by the budget 0.05

0.2

0.02

0.02

0.1

0.01

0.05

0.002

0.001 16

32

0.005 8

state rank R (W = R)

0.01 0.005

0.01

0.001

0.001 8

0.02

0.02 0.002

16

32

state rank R (W = R)

8

16

0.002

32

state rank R (W = R)

(e) Nemotron-3.5-Super R = 16 by the budget

0.05

0.005

0.005

0.002

(d) GLM-5.3-Flash R = 32 (none meets it)

0.05

0.01

0.005

(c) Kimi-Linear R = 32 by the budget

8

16

32

state rank R (W = R)

8

16

32

state rank R (W = R)

share of content energy

(f) content energy beyond rank R 0.1

0.01

Qwen3.8-Flash-Next

drift over steps 1,025–4,096

Nemotron-3.5-Lightning

budget gap(k)/4

Kimi-Linear

floor (see caption)

GLM-5.3-Flash

chosen R

Nemotron-3.5-Super

0.001

8

16

32

64

rank R

Figure 12 | The state clause takes the smallest rank whose state drift fits the budget, else 32. (a)–(e) State drift (± standard error) at 𝑅 = 8, 16, 32 with 𝑊 = 𝑅; open squares: its mean over decode steps 1,025–4,096 of 24 replies, drawn at 1.2𝑅. Black: the budget gap(𝑘)/4. Dotted: the floor, a second run of the original model (on GLM-5.3-Flash, the exact state scanned in 16-token pieces). Star: the chosen 𝑅. (f) Share of the content energy (𝐶 = 𝑆 − 𝑎¯ 𝑣 ⊤ ) beyond rank 𝑅 in prompt-final states of held-out 4K windows, energy-weighted over heads and layers. each gap of 255 or more adds two bytes (on Qwen3.8-Flash-Next at most 78 bytes, 1.5% of the code), which changes no outcome of C2 or C3. The exact values and the drift window. The template clause guards against an observed failure: a Qwen3.8-Flash-Next set on the NVFP4 checkpoint with a full-rank code and 𝑚 = 128 answered 66 of 200 prompts of a RULER multi-key needle task at 8K with an immediate end-of-sequence token, because the code of the answer’s template token collapsed; with 𝑚 = 256, 4 of 800 such prompts did. In the open-loop sum (8), each compression’s error fades with the head’s decays, so recent compressions dominate; the drift is therefore read late, over steps 961–1,024. 𝑊 = 𝑅 is a convention, and the ranks stop at 32 because at 𝑅 = 𝑊 = 32 the 64 factor columns of a 128 × 128 head hold as many numbers as the head.

B. Theory This appendix proves the four facts on which SEED’s shallow state rests. Every recurrent state splits exactly into a sink and a content (Lemma 1). The sink-plus-factors form is closed under the update, so compression is the only approximation (Lemma 2). In the open loop, the compression errors do not accumulate (Proposition 3 and Corollary 4). After the head’s normalization, the sink acts as a bias and a gain (Lemma 5). B.1. One recurrence for three layer types Gated DeltaNet, Kimi Delta Attention and Mamba-2 heads update their states by the same linear recurrence, so we state every result once, for 𝑆𝑡 = Φ𝑡 𝑆𝑡−1 + 𝛽𝑡 𝜅𝑡 𝑣𝑡⊤ ,

Φ𝑡 = (𝐼 − 𝜃𝛽𝑡 𝜅𝑡 𝜅⊤ 𝑡 )Λ𝑡 .

(6)

36

Shallow Encoding Emits Deep Memories

Here 𝑆𝑡 ∈ R𝑑𝑘 ×𝑑𝑣 , the decay Λ𝑡 is diagonal with entries in (0, 1), and 𝜃 ∈ {0, 1} switches the delta rule’s correction on or off. The three layer types set these as follows. • Gated DeltaNet (2): 𝜃 = 1 and Λ𝑡 = 𝑔𝑡 𝐼. • Kimi Delta Attention: 𝜃 = 1 and Λ𝑡 = diag(𝑔𝑡,1 , . . . , 𝑔𝑡,𝑑𝑘 ), one decay per key channel. • Mamba-2: 𝜃 = 0 and Λ𝑡 = 𝛾𝑡 𝐼. Its state is 𝑑𝑥 × 𝑑𝐵 ; in its transpose, the input projection 𝐵𝑡 , the input 𝑥𝑡 and the step size 𝛿𝑡 take the places of 𝜅𝑡 , 𝑣𝑡 and 𝛽𝑡 , and 𝑑𝐵 , 𝑑𝑥 those of 𝑑𝑘 , 𝑑𝑣 . Every step contracts the state. Let 𝜆𝑡 be the largest entry of Λ𝑡 . In the delta rules, ‖𝜅𝑡 ‖ = 1 and 𝛽𝑡 ∈ (0, 1), so 𝐼 − 𝛽𝑡 𝜅𝑡 𝜅⊤ 𝑡 has spectral norm one. Hence, in all three layer types, ‖Φ𝑡 ‖2 ≤ 𝜆𝑡 . The sink direction 𝑣¯ = ̸ 0 is the head’s mean value (for Mamba-2, its mean input 𝑥¯), and 𝜔𝑡 , 𝜀𝑡 , 𝑎𝑡 and 𝐶𝑡 are as in Lemma 1. The shallow state stores a sink coefficient 𝑎 and content factors 𝑈 , 𝑉 , 𝑆 = 𝑎¯ 𝑣⊤ + 𝑈 𝑉 ⊤,

𝑉 ⊤ 𝑣¯ = 0,

and compresses the content 𝑈 𝑉 ⊤ to rank 𝑅 every 𝑊 steps. B.2. The sink decomposition Lemma 1 (Section 4) splits every state exactly into a rank-one sink along 𝑣¯ and a content orthogonal to it. Since both parts follow the head’s own recurrence, SEED can store and update them separately. Proof of Lemma 1. The recurrence is linear in the values, so it suffices to project it onto 𝑣¯. Since 𝑣𝑡⊤ 𝑣¯ = 𝜔𝑡 ‖¯ 𝑣 ‖2 , right-multiplying the recurrence by 𝑣¯/‖¯ 𝑣 ‖2 gives the recurrence for 𝑎𝑡 ; subtracting its ⊤ product with 𝑣¯ from the recurrence for 𝑆𝑡 gives the one for 𝐶𝑡 . For the energies, the two terms of 𝑆𝑡 − 𝑧¯ 𝑣 ⊤ = (𝑎𝑡 − 𝑧) 𝑣¯⊤ + 𝐶𝑡 are orthogonal, because ⟨𝑦¯ 𝑣 ⊤ , 𝐶𝑡 ⟩𝐹 = 𝑦 ⊤ 𝐶𝑡 𝑣¯ = 0 for every 𝑦. Their squared Frobenius norms therefore ⊤ add, and ‖(𝑎𝑡 − 𝑧)¯ 𝑣 ‖𝐹 = ‖𝑎𝑡 − 𝑧‖ ‖¯ 𝑣 ‖. B.3. The factored update is exact The factored form never needs the dense state. Each step updates the sink coefficient and appends one column to each content factor. The next lemma shows that this update is exact for all three layer types and keeps the content orthogonal to 𝑣¯, and that compressing the content leaves the sink untouched. Compression is therefore the shallow state’s only approximation. Lemma 2 (exact factored update). One step of (6) maps a sink plus factors to a sink plus factors with one more column each. Precisely, let 𝑆 = 𝑎¯ 𝑣 ⊤ + 𝑈 𝑉 ⊤ with 𝑉 ⊤ 𝑣¯ = 0, and let 𝑣 = 𝜔¯ 𝑣 + 𝜀 with ⊤ ⊤ + ⊤ + +⊤ 𝜀 𝑣¯ = 0. Then Φ𝑆 + 𝛽𝜅𝑣 = 𝑎 𝑣¯ + 𝑈 𝑉 , where 𝑎+ = Λ𝑎 + 𝛽 (𝜔 − 𝜃 𝜅⊤ Λ𝑎) 𝜅, 𝑈 + = Λ𝑈, 𝜅 , [︀

]︀

𝑉 + = 𝑉, 𝛽 (𝜀 − 𝜃 𝑉 𝑈 ⊤ Λ𝜅) , [︀

]︀

(7)

and 𝑉 +⊤ 𝑣¯ = 0. For 𝜃 = 1 and Λ = 𝑔𝐼, this is the update (5). Moreover, if 𝑄 and 𝑃 have orthonormal columns in the column space and the row space of the content 𝐶 = 𝑈 𝑉 ⊤ , then both compressions 𝐶 ′ = 𝑄𝑄⊤ 𝐶 and 𝐶 ′ = 𝐶𝑃 𝑃 ⊤ satisfy 𝐶 ′ 𝑣¯ = 0, so replacing 𝐶 by 𝐶 ′ leaves the sink coefficient 𝑎 = 𝑆¯ 𝑣 /‖¯ 𝑣 ‖2 unchanged. Proof. The step acts on the sink and the content separately, except for the delta rule’s correction, which reads the content along 𝜅 and writes the result into the new column. Since Λ is symmetric, Φ = Λ − 𝜃𝛽𝜅𝜅⊤ Λ, and substituting 𝑆 and 𝑣 gives Φ𝑆 + 𝛽𝜅𝑣 ⊤ = (Φ𝑎 + 𝛽𝜔𝜅) 𝑣¯⊤ + Φ𝑈 𝑉 ⊤ + 𝛽𝜅𝜀⊤ = 𝑎+ 𝑣¯⊤ + Λ𝑈 𝑉 ⊤ + 𝛽𝜅 (𝜀 − 𝜃 𝑉 𝑈 ⊤ Λ𝜅)⊤ = 𝑎+ 𝑣¯⊤ + 𝑈 + 𝑉 +⊤ . 37

Shallow Encoding Emits Deep Memories

The new column of 𝑉 + combines 𝜀 and the columns of 𝑉 , all orthogonal to 𝑣¯, so 𝑉 +⊤ 𝑣¯ = 0. For the compressions, 𝑄𝑄⊤ 𝐶 𝑣¯ = 0 because 𝐶 𝑣¯ = 0. The same identity makes 𝑣¯ orthogonal to the row space of 𝐶, so 𝑃 ⊤ 𝑣¯ = 0 and 𝐶𝑃 𝑃 ⊤ 𝑣¯ = 0. Both compressions stay in factored form, 𝑄𝑄⊤ 𝐶 = 𝑄 (𝑉 𝑈 ⊤ 𝑄)⊤ ,

𝐶𝑃 𝑃 ⊤ = (𝑈 𝑉 ⊤ 𝑃 ) 𝑃 ⊤ .

B.4. Compression errors do not accumulate In the open loop, the compressed and the exact recurrence see the same keys, values, decays and write strengths, and differ only by the errors that compressions inject. The proposition below bounds the resulting error and shows that the sink coefficient stays exact; its corollary shows that periodic compressions do not compound. Proposition 3 (open-loop error). The error at step 𝑡 is at most the sum of the injected errors, each damped by the decays applied after it. Precisely, let 𝑆˜0 = 𝑆0 + ℰ0 ,

𝑆˜𝑡 = Φ𝑡 𝑆˜𝑡−1 + 𝛽𝑡 𝜅𝑡 𝑣𝑡⊤ + ℰ𝑡

(𝑡 ≥ 1),

with the inputs of (6). Then ‖𝑆˜𝑡 − 𝑆𝑡 ‖𝐹 ≤

𝑡 (︁ ∏︁ 𝑡 ∑︁ 𝑠=0

)︁

𝜆𝑢 ‖ℰ𝑠 ‖𝐹 .

(8)

𝑢=𝑠+1

If, moreover, every ℰ𝑠 𝑣¯ = 0, then 𝑆˜𝑡 𝑣¯ = 𝑆𝑡 𝑣¯ for all 𝑡. Proof. The difference 𝒟𝑡 = 𝑆˜𝑡 − 𝑆𝑡 follows the same recurrence without inputs, driven only by the injected errors: 𝒟𝑡 = Φ𝑡 𝒟𝑡−1 + ℰ𝑡 with 𝒟0 = ℰ0 . Unrolling it gives 𝒟𝑡 =

𝑡 ∑︁

Φ𝑡 · · · Φ𝑠+1 ℰ𝑠 .

𝑠=0

Since ‖Φ𝑢 ℰ‖𝐹 ≤ ‖Φ𝑢 ‖2 ‖ℰ‖𝐹 ≤ 𝜆𝑢 ‖ℰ‖𝐹 , the triangle inequality gives (8). If every ℰ𝑠 𝑣¯ = 0, each term of the sum vanishes on 𝑣¯, so 𝒟𝑡 𝑣¯ = 0. Corollary 4 (periodic compression). If a compression every 𝑊 steps injects an error of norm at most 𝜖¯, the error never exceeds 𝜖¯ times the head’s decay horizon. Precisely, let ‖ℰ𝑠 ‖𝐹 ≤ 𝜖¯, with ℰ𝑠 = 0 unless 𝑊 divides 𝑠, and let the mean of 𝜆𝑢 over each period 𝑢 = 𝑚𝑊 +1, . . . , (𝑚+1)𝑊 be at most 𝑔¯ < 1 (as it is when every 𝜆𝑢 ≤ 𝑔¯). Then, for every 𝑡, ‖𝑆˜𝑡 − 𝑆𝑡 ‖𝐹 ≤

𝜖¯ 𝜖¯ ≤ . 𝑊 1 − 𝑔¯ 1 − 𝑔¯

Proof. Count the compressions before 𝑡 from the most recent, 𝑛 = 0, 1, . . . . The decay product that damps the 𝑛-th one in (8) spans 𝑛 full periods and part of one more. By the inequality of arithmetic and geometric means, each full period contributes at most 𝑔¯ 𝑊 , and the partial period at most one. Hence ∑︁ 𝜖¯ ‖𝑆˜𝑡 − 𝑆𝑡 ‖𝐹 ≤ 𝜖¯ 𝑔¯ 𝑛𝑊 = , 1 − 𝑔¯ 𝑊 𝑛≥0 and 𝑔¯ 𝑊 ≤ 𝑔¯ gives the second bound. B.5. After normalization, the sink is a bias and a gain Gated DeltaNet and Kimi Delta Attention heads normalize their read-out before the output gate. The lemma below shows that the sink then sets a fixed output direction and, through the normalizer, scales the content, as the sink logit does in a softmax denominator.

38

Shallow Encoding Emits Deep Memories

Lemma 5 (the sink as a gain). In a Gated DeltaNet or Kimi Delta Attention head, the read-out splits into a part along 𝑣¯ and a part orthogonal to it, 𝑜𝑡 = 𝑆𝑡⊤ 𝑞𝑡 = 𝑏𝑡 + 𝑟𝑡 ,

𝑏𝑡 = (𝑞𝑡⊤ 𝑎𝑡 ) 𝑣¯,

𝑟𝑡 = 𝐶𝑡⊤ 𝑞𝑡 ,

𝑟𝑡⊤ 𝑣¯ = 0.

An RMSNorm with unit weight and constant 𝜂 ≥ 0 therefore returns √ 𝑑𝑣 (𝑏𝑡 + 𝑟𝑡 ) RMSNorm(𝑜𝑡 ) = (︀ )︀1/2 . ‖𝑏𝑡 ‖2 + ‖𝑟𝑡 ‖2 + 𝑑𝑣 𝜂

(9)

If 𝜂 = 0 and 𝑞𝑡⊤ 𝑎𝑡 ̸= 0, the normalized read-out is a fixed direction plus the content scaled by 1/‖𝑏𝑡 ‖, up to an error quadratic in the ratio 𝜌𝑡 = ‖𝑟𝑡 ‖/‖𝑏𝑡 ‖. With 𝑠 = sign(𝑞𝑡⊤ 𝑎𝑡 ) and 𝑣^ = 𝑣¯/‖¯ 𝑣 ‖, ⃦ (︁(︀ )︁ √︀ 𝜌2 √︀ (︁ )︀1/2 𝑟𝑡 )︁⃦ ⃦ √︀ ⃦ 𝑣+ − 1 ≤ 𝑑𝑣 𝑡 . ⃦RMSNorm(𝑜𝑡 ) − 𝑑𝑣 𝑠^ ⃦ = 𝑑𝑣 1 + 𝜌2𝑡

2

‖𝑏𝑡 ‖

Proof. Since 𝐶𝑡 𝑣¯ = 0, we have 𝑟𝑡⊤ 𝑣¯ = 𝑞𝑡⊤ 𝐶𝑡 𝑣¯ = 0, so ‖𝑜𝑡 ‖2 = ‖𝑏𝑡 ‖2 + ‖𝑟𝑡 ‖2 , which gives (9). With 𝜂 = 0, dividing the numerator and the denominator of (9) by ‖𝑏𝑡 ‖, and using 𝑏𝑡 /‖𝑏𝑡 ‖ = 𝑠^ 𝑣 , gives RMSNorm(𝑜𝑡 ) = The difference

√︀

𝑑𝑣

𝑥 , ‖𝑥‖

𝑥 = 𝑠^ 𝑣+

𝑟𝑡 , ‖𝑏𝑡 ‖

‖𝑥‖ = 1 + 𝜌2𝑡 (︀

)︀1/2

.

√︀ √ √ 𝑑𝑣 𝑥 (1/‖𝑥‖ − 1) thus has norm 𝑑𝑣 (‖𝑥‖ − 1), and 1 + 𝜌2 ≤ 1 + 𝜌2 /2.

C. Experimental Details and State Probes This appendix gives the model dimensions and the protocols of the state measurements, storage precision and closed-loop drift. Model dimensions. Nemotron-3.5-Lightning has 30B parameters (3B active per token), Kimi-Linear 48B (3B), Nemotron-3.5-Super 120B (12B), Qwen3.8-Flash-Next 125B (6B, plus a 51B-parameter n-gram embedding table read by lookup) and GLM-5.3-Flash 320B (18B). Qwen3.8-Flash-Next, GLM-5.3-Flash and Nemotron-3.5-Super run with text input only. Qwen3.8-Flash-Next: hidden size 2560; 512 routed experts (ten active) and one shared; four parallel 2560-dimensional residual streams, mixed and updated at every layer by input-dependent gates. Its sparse-attention layers have 24 query heads and two 256-dimensional key–value (KV) heads, select 2048 tokens per query and keep one indexer key per four tokens, 2112 bytes per token in all. Nemotron-3.5-Lightning: hidden size 2688; attention layers with two 128-dimensional KV heads. Kimi-Linear: hidden size 2304; 256 routed experts (eight active) and one shared. Its MLA layers cache the model’s own latent key–value format: a 512-number latent and a 64-number shared key (the rotary part, which Kimi-Linear leaves unrotated). GLM-5.3-Flash: hidden size 4096 in four residual streams, mixed with input-dependent weights by manifold-constrained hyper-connections (mHC); 288 routed experts (eight active) and one shared, with dense feed-forward layers in the first three layers. Expert, dense feed-forward and most sparse-attention projection weights are fp8 (E4M3, 128 × 128 blocks) with dynamically quantized fp8 activations; the Kimi Delta Attention layers, the indexer and the latent up-projection stay bf16. Each DSA layer has a 32-head indexer whose 128-dimensional keys are pooled over four tokens and cached in fp8 with a 4-byte scale, so the layer caches 1,057 bytes per token: the 1,024-byte bf16 latent plus 33 bytes of indexer key. The fp8 cache figures of Tables 7 and 18 halve the latent only. Nemotron-3.5-Super: hidden size 4096; 512 routed experts (22 active) and one shared, computing in a 1024-dimensional latent space; attention layers with 32 query heads and two 128-dimensional KV heads; Mamba-2 layers with eight groups. We use the language model of its vision-language checkpoint unchanged, without the vision tower and projector. Above the anchor layer, the emitters write the prompt entries of the attention caches of layers 31, 35, 39, 43 and 47 on Qwen3.8-Flash-Next; 33 and 42 on Nemotron-3.5-Lightning; MLA layers 19, 23 and 26 on Kimi-Linear; DSA layers 31, 35, 39 and 43 on GLM-5.3-Flash; and layer 78 on Nemotron-3.5-Super. 39

Shallow Encoding Emits Deep Memories

State probes. States are captured in fp32 at the end of the same six held-out web prompts of 3,584 tokens (Figure 13). Independent random directions in R128 , the baseline of Figure 13c, have a median | cos | of 0.06 (95th percentile 0.18). Figure 3 replays the gated delta rule in fp64 over one prompt without a document boundary. Nemotron-3.5-Lightning uses the held-out stream of its own tokenizer and all 23 × 64 Mamba-2 heads; the content of Figure 13d uses 𝑣¯ estimated on other text, and the other models use the mean input or mean value that their component sets ship. The median and pooled sink shares are 75% and 69% on Qwen3.8-Flash-Next, 64% and 99.6% on Nemotron-3.5-Lightning, 70% and 99.7% on Nemotron-3.5-Super, 34% and 82% on Kimi-Linear, and 24% and 86% on GLM-5.3-Flash. On six windows of each model’s own math and code replies, every share changes by at most 0.05. Eviction and step-wise policies. The eviction experiment acts once, in all 36 layers, on the state after the first 3,583 tokens (SEED also compresses before the last prompt token), with 𝑣¯ estimated on other text; the last prompt token then runs, and the next 512 tokens are scored with exact state updates. The sink’s replacement by its mean over other prompts averages over the other five prompts. The step-wise policies of §4 start from the same state and truncate exactly after every token of a 256-token teacher-forced continuation, with 𝑣¯ the mean value of the other five prompts. Every policy scores the same 1536 tokens. (a) one direction carries most

0.1

0.01

0.001 median over layers

(b) over the heads

(c) same across prompts

median 0.85

500 200 150 100

R = 16

99% median 17

400 300 200

50

(d) low-rank content

prompt A vs B vs random tokens unrelated vectors

600

250

heads

direction 1: 85% of the energy heads (of 1728)

share of the state's energy

1

90% median 6

100

one line per layer (36) 0.0001

0 1

4

8

12

singular direction i

16

0 0.0

0.5

1.0

energy share of direction 1

0.0

0.5

1.0

|cos| of v⋆ between prompts

0

20

40

rank of the content CT

Figure 13 | One direction carries most of the recurrent state, and it is nearly the same on every prompt tested (Qwen3.8-Flash-Next, 36 × 48 heads, six web prompts of 3,584 tokens). (a) Energy share of the state’s 𝑖-th singular direction; grey: one layer (median over heads and prompts); green: median over layers. (b) The top direction’s share over the 1,728 heads. (c) | cos | between a head’s top output directions 𝑣⋆ on two unrelated prompts (green), on a web prompt and a prompt of uniformly random tokens (charcoal outline), and for independent random directions in R128 (grey). (d) Rank holding 90% and 99% of the energy of the content 𝐶𝑇 (Lemma 1) per head, averaged over the prompts; box 25–75%, whiskers 5–95%, bar the median; dashed: Qwen3.8-Flash-Next’s 𝑅 = 16. Storage precision. Decode drift (Tables 4 and 9) is the per-token KL divergence from the original model’s next-token distribution, teacher-forced over the first 1,024 tokens of 48 of its own replies to held-out prompts (16 each of math, science and code). It counts the original model’s 1,024 most likely tokens plus one bucket for the remaining mass, and is reported above a second run of the original model (0.001–0.004 nats per token), paired over replies. On GLM-5.3-Flash, whose reference engine takes about 8 s per decode step, it uses 24 replies over their first 256 tokens and is reported above the full-rank factored path, through which every rounded form runs. Rounding. The original model’s dense state is rounded after every token. The shallow state rounds its factors and sink coefficient at every compression (fp8 with one fp32 scale per vector) and keeps the up to 𝑊 newest columns in fp32 between compressions. Stochastic rounding adds a uniform random integer below the dropped bits to the fp32 bit pattern before truncating, so it is unbiased. Under either rounding, fp16 factors change the shallow state’s drift by at most 0.0006 nats per token.

40

Shallow Encoding Emits Deep Memories

NVFP4 checkpoints. The NVFP4 row of Table 4 decodes the same replies with NVIDIA’s official NVFP4 checkpoints of Qwen3.8-Flash-Next, Nemotron-3.5-Lightning and Nemotron-3.5-Super and their original caches and states, against the bf16 model’s reference pass and above its second run. Our reference path runs them with the served arithmetic of their FP8 and NVFP4 layers and SGLang’s scale conventions. On four held-out 2K-token windows, its per-token loss matches SGLang’s within 0.001 nats on average on Qwen3.8-Flash-Next and Nemotron-3.5-Lightning; on Nemotron-3.5-Super, the NVFP4 checkpoint raises the loss by 0.012 nats in our path and by 0.010 in SGLang.

Table 9 | The shallow state’s drift is the same with fp32 and fp16 factors under either rounding; a dense state’s depends on the format and, on the Mamba-2 models, on the rounding. Decode drift (nats per token above a second run of the original model) of the original model’s dense state, rounded after every token, and of the shallow state, rounded at every compression. The exams use the fp32 shallow state. Last row: size per head and layer in 16 bits, the shallow state’s right after a compression. Qwen3.8Flash-Next

Nemotron-3.5Lightning

KimiLinear

GLM-5.3Flash

Nemotron-3.5Super

format

rounding

dense

shallow

dense

shallow

dense

shallow

dense

shallow

dense

shallow

fp32 fp16 fp16 bf16 bf16 fp8 fp8

– nearest stochastic nearest stochastic nearest stochastic

0 0.0006 0.0006 0.0008 0.0007 0.0088 0.0073

0.0056 0.0056 0.0056 0.0056 0.0056 0.0067 0.0076

0 0.0011 0.0004 0.0057 0.0044 0.0487 0.1752

0.0060 0.0062 0.0059 0.0061 0.0065 0.0136 0.0194

0 0.0002 0.0002 0.0008 0.0004 0.0104 0.0048

0.0017 0.0017 0.0016 0.0017 0.0017 0.0023 0.0027

0 0.0006 0.0010 0.0008 0.0010 0.0070 0.0055

0.0027 0.0021 0.0025 0.0030 0.0024 0.0030 0.0037

0 0.0012 0.0002 0.0066 0.0048 0.0745 0.1740

0.0070 0.0069 0.0070 0.0072 0.0073 0.0160 0.0199

32.8

8.4

16.4

6.4

32.8

16.6

32.8

16.6

16.4

6.4

KB per head, 16 bits

Closed-loop drift. The long replies of §6.1 are the original model’s replies of at least 16,384 tokens (on Nemotron-3.5-Lightning and Nemotron-3.5-Super from their long-reply sets), scored over their first 16,384 decode steps at the released 𝑅 and 𝑊 , per block of 1,024 steps, above a second run of the original model. Only the tokens are teacher-forced: a compression error changes the later keys, values and decays.

D. Training This appendix gives the objectives, data and settings of the two training stages of §5.2; Table 13 gives the run metrics. Both stages. 𝜇 and the sink directions stay fixed. The training forward pass differs from inference in two ways (on GLM-5.3-Flash, three). It truncates the prompt-final states by an exact fp64 eigendecomposition instead of a subspace iteration, and it runs the continuation with the exact recurrence, without the compression every 𝑊 steps. On GLM-5.3-Flash it compresses only the emitted layers’ prompt-final states, whereas inference compresses every Kimi Delta Attention state. Apart from the checkpoint’s own quantization, only the prompt code is quantized (NVFP4, exact values in bf16), in training and in every exam. Both stages use AdamW (Loshchilov & Hutter, 2019) with 𝛽 = (0.9, 0.95), 𝜖 = 10−8 , no weight decay and gradient clipping at 1.0. Stage 1. Stage 1 minimizes the forward next-token KL, the KL of the first 64 continuation positions (the first answer tokens on windows anchored at an answer start) and 0.1× cross-entropy, on 6 × 108 tokens with a 50-step warm-up and cosine decay to 2% of the peak (Tables 10 and 11). Teacher replies supply 59% of the trained tokens on both Qwen3.8-Flash-Next sets, 68% on Nemotron-3.5-Lightning, 37% on Kimi-Linear, 58% on GLM-5.3-Flash and 68% on Nemotron-3.5-Super. Stage 2. A KL on the teacher’s text misses the departures that arise when the components condition on their own tokens. Each stage-2 round therefore pairs the teacher’s and the current set’s replies to public mathematics, chat and output-constraint prompts unseen in stage 1 and to code and science prompts reused from it. The forward KL applies to both arms’ replies, the reverse KL to the set’s own, 41

Shallow Encoding Emits Deep Memories

Table 10 | Hyperparameters of the released sets (Nemotron-3.5-Super: stage 1 only). Component parameters: every exported tensor, including the fixed 𝜇 and sink buffers and rows without gradient (Mamba-2 gate rows: 88M on Nemotron-3.5-Lightning, 134.2M on Nemotron-3.5-Super; indexer query rows: 6.6M on each Qwen3.8-Flash-Next set). Peak LR: stage 1 / stage 2; second stage-2 rounds about 5 × 10−6 . Answer-start windows: the share of the 4K / 16K windows that start at the beginning of an answer. answer-start (%) Component parameters (M)

Set Qwen3.8-Flash-Next bf16 Qwen3.8-Flash-Next-NVFP4 Nemotron-3.5-Lightning Kimi-Linear GLM-5.3-Flash Nemotron-3.5-Super

619.0 619.0 235.8 113.8 708.7 311.3

Peak LR

16K-window steps (%) stage 1

stage 2

Auxiliary alignment

5 × 10−5 / 10−5 5 × 10−5 / 10−5 5 × 10−5 / 10−5 5 × 10−5 / 10−5 5 × 10−5 / 10−5 5 × 10−5 / –

30 30 40 30 30 40

70/60 70/60 70/60 70/60 70/60 –

– – states – states, index keys states

50/30 50/30 50/40 50/30 50/30 50/40

Table 11 | Data recipe of both stages; windows are 4K/16K tokens. † : 13-gram filtered when built against GPQA, MMLU, MATH-500, AIME and IFEval (stage 2 also LiveCodeBench and IFBench). Table 12 gives each set’s shares. Stage

Stream

Sources

1

public text

1

reasoning/chat†

1

teacher replies†

FineWeb-Edu (Penedo et al., 2024), FineMath (Ben Allal et al., 2025); synthetic retrieval facts in web documents; long web documents OpenMathReasoning (Moshkov et al., 2025), OpenScienceReasoning-2 (NVIDIA, 2025), OpenCodeReasoning (Ahmad et al., 2025), OpenR1-Math (Hugging Face, 2025), OpenThoughts3 (Guha et al., 2026) and tulu-3-sft-mixture (Lambert et al., 2025) Frozen-model responses to public code, mathematics, science and chat prompts; long Nemotron-3.5-Lightning and Nemotron-3.5-Super responses

2

paired generation†

2

retained streams

Public prompts (code and science reused from stage 1) answered by the teacher and the current components; forward KL on both, reverse KL on the components’ replies Stage-1 web, retrieval and public-corpus text† and stage-1 teacher replies

Table 12 | The training data of every set. Stage 1: share of the tokens per data group; stage 2: rounds, prompts per round and the domain weights of the paired generations in the first round. † : 13-gram filtered when built against GPQA, MMLU, MATH-500, AIME and IFEval (stage 2 also LiveCodeBench and IFBench). Qwen3.8Qwen3.8Flash-Next Flash-Next Nemotron-3.5- Kimibf16 NVFP4 Lightning Linear

GLM-5.3- Nemotron-3.5Flash Super

stage 1, share of 6 × 108 tokens (%) web text and retrieval documents long web documents reasoning and chat corpora† frozen model’s replies† frozen model’s long replies†

14.1 5.5 21.9 58.5 –

14.1 5.5 21.9 58.5 –

9.9 4.2 17.6 43.3 25.0

15.6 7.8 39.3 37.2 –

15.6 7.8 19.0 57.6 –

9.9 4.2 17.6 43.3 25.0

stage 2 rounds prompts per round† mathematics weight (%) science weight (%) code weight (%) chat weight (%) constraint following weight (%)

2 12,957 26 23 19 14 18

1 12,957 27 23 18 15 17

2 12,965 19 15 14 18 33

1 12,954 25 18 29 11 18

2 12,958 24 18 30 16 12

1 12,957 32 33 11 14 10

42

Shallow Encoding Emits Deep Memories

which get no cross-entropy; the other windows keep the 0.1× cross-entropy. The visible answer and the 20% of positions with the highest teacher entropy count twice (weights normalized per window), and the KL of the first 256 continuation positions is added. Paired replies supply 80% of the tokens and retained stage-1 streams 20% (on Nemotron-3.5-Super, whose round adds 32K-token windows of retained streams, about a third and two thirds), with domain weights set by the measured divergence of the set’s own replies. Each round reads 1.5 × 108 tokens (on Nemotron-3.5-Super 2.1 × 108 , two fifths of them in 32K-token windows) with a short warm-up and a cosine learning-rate schedule. Stage 2 ran two rounds on bf16 Qwen3.8-Flash-Next, Nemotron-3.5-Lightning and GLM-5.3-Flash and one on Qwen3.8-Flash-Next-NVFP4, Kimi-Linear and Nemotron-3.5-Super. Every model releases its stage-2 set except Nemotron-3.5-Super, whose stage-1 set is released. Exam data. No exam is used as training data. Stage-2 prompts and replies and the retained publictext streams pass a 13-gram filter against GPQA, MMLU, MATH-500, AIME, IFEval, LiveCodeBench and IFBench (Pyatkin et al., 2025), and stage 2’s constraint prompts exclude IFEval’s 25 instruction types.

Table 13 | Run metrics of the training stages from their histories (Nemotron-3.5-Super releases its stage-1 set). Nodes have four GPUs; stage 1 starts from the closed-form initialization and each stage-2 round from its predecessor. Held-out: 32 windows of 4K tokens, scored after token 2048 at the run’s training rank. held-out KL steps

tokens

nodes × min

step 0

final

Qwen3.8-Flash-Next bf16 stage 1 881 stage 2, round 1 221 stage 2, round 2 221

600M 151M 151M

16 × 159.7 16 × 58.2 16 × 52.5

0.0315 0.0174 0.0173

0.0176 0.0172 0.0171

Qwen3.8-Flash-Next-NVFP4 stage 1 881 600M stage 2 221 151M

16 × 208.1 16 × 73.7

0.0587 0.0424

0.0423 0.0423

Nemotron-3.5-Lightning stage 1 1635 stage 2, round 1 409 stage 2, round 2 409

600M 150M 150M

8 × 97.6 8 × 25.7 8 × 25.8

0.0615 0.0085 0.0083

0.0085 0.0083 0.0082

Kimi-Linear stage 1 stage 2

1761 441

600M 150M

8 × 57.6 8 × 17.0

0.0176 0.0051

0.0050 0.0049

GLM-5.3-Flash stage 1 stage 2, round 1 stage 2, round 2

881 221 221

600M 151M 151M

16 × 89.5 16 × 24.9 16 × 24.3

0.0145 0.0064 0.0061

0.0064 0.0061 0.0061

Nemotron-3.5-Super stage 1 1635 stage 2 409

600M 214M

8 × 130.5 8 × 35.2

0.0174 0.0063

0.0064 0.0060

Run

Emitted-state alignment. Nemotron-3.5-Lightning, GLM-5.3-Flash and Nemotron-3.5-Super add a term on the emitted prompt-final states: their relative squared error against the frozen model’s states, on Nemotron-3.5-Lightning and Nemotron-3.5-Super weighted by how much the layer later reads of an error. Its weight is set at the start of each run to 0.2 times the answer-window KL over the term’s own value. GLM-5.3-Flash adds a like term for the indexer keys and pooling scores that its emitters write, since the hard top-2048 selection gives them no KL gradient. Table 14 gives the emitted states’ errors against the frozen model’s own states on 48 held-out windows split at mid-window, averaged over the emitted recurrent layers and taken before the stored form (which changes each by at most 0.03); the sets it compares also differ in code or anchor depth. 43

Shallow Encoding Emits Deep Memories

Table 14 | Training moves the emitted prompt-final states towards the frozen model’s: the released sets’ pooled error is 0.20–0.52, against 0.34–1.29 for the untrained shallow prefill. Frobenius: ‖𝑆emit − 𝑆‖2𝐹 /‖𝑆‖2𝐹 , pooled over a layer’s heads (above 1: farther from the frozen model’s state than a zero state). State term: trained with the emitted-state alignment term. Read-out: the error as the layer reads it over the next 64 teacher-forced tokens, relative to the state’s own read-out. Floor: the frozen model’s own state in the stored form. error

floor

Set

state term

Frobenius, read-out, pooled 64 tokens

pooled / read-out

Qwen3.8-Flash-Next bf16 untrained shallow prefill released set

– no

0.615 0.492

0.076 0.021 / 0.001 0.047

Qwen3.8-Flash-Next-NVFP4 untrained shallow prefill released set

– no

0.616 0.520

0.075 0.021 / 0.001 0.055

Nemotron-3.5-Lightning untrained shallow prefill stage-1 set trained with the KL alone (code 1024+128, not released) released set

– no yes

1.289 0.497 0.294

0.334 0.001 / 0.001 0.137 0.100

Kimi-Linear untrained shallow prefill released set

– no

0.466 0.199

0.204 0.004 / 0.002 0.091

GLM-5.3-Flash untrained shallow prefill at 𝑘 = 31 preliminary set at 𝑘 = 27, code 5632+256 preliminary set at 𝑘 = 27, no code preliminary set at 𝑘 = 31, no code (the released configuration) released set (16 windows)

– no no yes yes

0.639 0.576 0.628 0.501 0.399

0.141 0.024 / 0.005 0.110 0.122 0.090 0.074

Nemotron-3.5-Super untrained shallow prefill released set

– yes

0.339 0.261

0.348 0.071

0.000 / 0.000

Served divergence. Each set answers the model’s stage-2 evaluation prompts (public prompts, not exam items) with one seed per prompt, and the frozen model scores every reply token. We report this divergence in 10−3 nats per token above the frozen model’s floor, the divergence of its own replies rescored by prefill. Stage 2 lowers it on Qwen3.8-Flash-Next-NVFP4 from 10.8 to 9.9 (1,013 prompts) and on Kimi-Linear from 2.6 to 2.3 (820 prompts), while it changes the teacher-forced held-out KL of every set it trains by less than 0.0006 nats per token (Table 13).

E. Evaluation Configuration and Results This appendix gives the exam protocol of §6 (Table 15), the answer flips and reply lengths, a reading of every cell at least one standard error below the original model, and the stage-1 sets. Server and sampling. Both phases of a request run in one engine instance without a prefix cache. Qwen3.8-Flash-Next and Qwen3.8-Flash-Next-NVFP4 sample with reasoning effort xhigh, temperature 1.0, top-𝑝 0.95 and top-𝑘 20; Nemotron-3.5-Lightning and Nemotron-3.5-Super with thinking on, temperature 1.0 and top-𝑝 0.95; Kimi-Linear, which has no thinking mode, with temperature 1.0 and top-𝑝 1.0; GLM-5.3-Flash with reasoning effort max, temperature 1.0 and top-𝑝 0.95. RULER runs with thinking off. The generation budget is min(229,376, 262,144 − prompt − 8) tokens, with no forced close. Pairing and significance. Scores and standard errors are paired over questions after averaging each item’s repetitions; counts use unrounded values. No sample is dropped; under process sharding, MMLU questions that occur twice can get different seeds (at most 0.4% of samples). A reply whose thinking never closes scores as no answer; one cut in its visible part is graded as written. If every true difference were zero and the standard errors exact, about 1.5 of the 66 cells would lie more than two standard errors below the original model; 3 do. Some SEED-arm runs that hit a job time limit 44

Shallow Encoding Emits Deep Memories

Table 15 | Task configurations. Sampled exams are paired by question and request seed; RULER is greedy at each length. The official grader gives the result. Task

Items

Repetitions

GPQA-Diamond

198

8

AIME 2024+25 MATH-500

60 500

8 8

MMLU IFEval LiveCodeBench v6

14,042 541 454

1 8 8

164 503

8 4

1,300

1

HumanEval+ LongBench v2 RULER

Grading and configuration simple-evals (OpenAI, 2024) prompt, extraction and option permutation prompt asks for a boxed answer; integer gold answers prompt asks for a boxed answer; last boxed answer, Math-Verify (Kydlíček, 2025) test split, simple-evals prompt and extraction prompt-level strict problems of 2024-08 to 2025-04; official prompt, extraction and public/private tests EvalPlus 0.4.0.dev44 grader on the v0.1.10 data, base and plus tests official zero-shot template and extraction; contexts longer than 120,000 tokens (base tokenizer) are middle-truncated 13 subtasks at 32K/64K/128K; at most 1024 answer tokens

were resumed, which regenerates the in-flight replies; on LiveCodeBench this touches 7.7%, 29.5% and 3.2% of the replies of the bf16 and NVFP4 Qwen3.8-Flash-Next sets and Kimi-Linear. Answer flips and reply lengths. A flip is a question whose correctness differs between two runs (Dutta et al., 2024). The flips use the eight repetitions of AIME 2024, AIME 2025, GPQADiamond, MATH-500, IFEval and LiveCodeBench for the six sets (36 cells). Each cell averages over consecutive pairs (𝑟, 𝑟+1), comparing the set’s repetition 𝑟+1 with the original model’s repetition 𝑟, and the original model’s repetitions 𝑟+1 and 𝑟, so that both comparisons pair different sampling seeds. A set’s answer and the original model’s differ in correctness 7.4% of the time, and two repetitions of the original model 7.3%. Reply lengths pair each reply of a set with the original model’s reply to the same prompt, repetition and seed, over 26 cells (GPQA-Diamond, MMLU, LiveCodeBench and the job that runs AIME, MATH-500 and IFEval together; on Nemotron-3.5-Super also HumanEval+ and LongBench v2) and 171,317 reply pairs. The sets write 0.94 times the original model’s output tokens; 14 cells are shorter by more than two standard errors and 2 longer. Fewer replies reach the generation limit (286 against 338) or end with their thinking unfinished (300 against 353). Cells below the original model. We read the samples of each of the 19 cells at least one standard error below the original model; a regrade applies to both arms, and the official scores remain the result. Grading accounts for part of the difference in five cells (paired, in points): IFEval on bf16 Qwen3.8-Flash-Next, −0.6 ± 0.3 officially and −0.18 ± 0.27 with lm-eval’s language detector run on lower-cased text, since it does not label fully upper-case English as English, and three other checker quirks fixed; MATH-500 on bf16 Qwen3.8-Flash-Next and Qwen3.8-Flash-Next-NVFP4, −0.53 ± 0.15 and −0.30 ± 0.16 officially and −0.33 ± 0.11 and −0.20 ± 0.14 with equivalent notations (on bf16 Qwen3.8-Flash-Next also correct unboxed values) credited; LiveCodeBench on Nemotron-3.5-Lightning, −0.6 ± 0.6 officially and −0.22 ± 0.52 with wall-clock failures counted as passed; and RULER 64K on Nemotron-3.5-Lightning, −0.3 ± 0.2 officially and −0.09 ± 0.18 counting every word of a reference. In the other cells, the official grades stand. Cells above the original model. On RULER, the original Qwen3.8-Flash-Next checkpoints sometimes end their turn right after the answer prefix, an empty reply that the official grader scores as wrong; the SEED sets do so less often, which accounts for their higher RULER scores at 32K and 64K. Stage-1 sets. Table 16 pairs the stage-1 sets of Qwen3.8-Flash-Next-NVFP4, Nemotron-3.5Lightning and Kimi-Linear with the original model on the same questions and seeds; these stage-1 sets did not run HumanEval+ and LongBench v2.

45

Shallow Encoding Emits Deep Memories

Table 16 | Stage 1 against stage 2: of the 27 released − stage-1 differences, only IFEval on Qwen3.8Flash-Next-NVFP4 exceeds two standard errors (on unrounded values). Score, paired: the stage-1 set’s score and its paired difference to the original model, mean ± standard error; released − stage 1: the paired difference between the two sets. Table 3 gives the original model’s and the released sets’ scores. Qwen3.8-Flash-Next-NVFP4 Exam (questions)

stage-1 set score paired

GPQA-Diamond (198) AIME 2024+25 (60) MATH-500 (500) MMLU (14,042) IFEval (541) LiveCodeBench v6 (454) RULER 32K (1,300) RULER 64K (1,300) RULER 128K (1,300)

91.5 99.6 99.1 92.4 90.7 89.6 95.8 95.3 94.4

−0.8 ± 0.6 0.0 ± 0.4 −0.1 ± 0.1 0.0 ± 0.2 −0.9 ± 0.3 +0.6 ± 0.4 +1.1 ± 0.5 +0.4 ± 0.5 −0.9 ± 0.4

Nemotron-3.5-Lightning

Kimi-Linear

released stage-1 set − stage 1 score paired

released stage-1 set − stage 1 score paired

released − stage 1

−0.3 ± 0.6 0.0 ± 0.4 −0.2 ± 0.1 +0.2 ± 0.2 +0.9 ± 0.3 −0.7 ± 0.4 +0.3 ± 0.2 +0.2 ± 0.2 +0.2 ± 0.3

−0.1 ± 0.9 +1.7 ± 1.0 −0.1 ± 0.2 −0.1 ± 0.2 +0.5 ± 0.4 −0.4 ± 0.5 0.0 ± 0.1 0.0 ± 0.2 +0.4 ± 0.2

+0.6 ± 1.3 +0.8 ± 2.6 −0.1 ± 0.4 +0.4 ± 0.4 +0.3 ± 0.6 −0.2 ± 0.6 −0.1 ± 0.1 −0.2 ± 0.1 +0.1 ± 0.2

75.1 87.1 98.4 88.0 91.5 78.0 96.4 96.0 95.2

−1.2 ± 1.1 −2.5 ± 1.4 +0.2 ± 0.2 −0.1 ± 0.2 −0.1 ± 0.4 −0.2 ± 0.5 −0.1 ± 0.2 −0.4 ± 0.2 −0.6 ± 0.3

67.4 66.0 92.5 77.9 71.4 43.7 95.1 94.1 94.0

−0.2 ± 1.2 −1.5 ± 2.4 −0.6 ± 0.4 +0.2 ± 0.4 −1.8 ± 0.6 −0.7 ± 0.8 +0.1 ± 0.1 +0.1 ± 0.2 −0.5 ± 0.2

F. Performance Details This appendix gives the byte and FLOP account behind Table 2 and §7 and the stored forms of a cached prefix. Table 17 | Analytical byte and FLOP account. 𝑏𝑠 and 𝑏𝑐 are the state and cache precisions in bytes; counts use checkpoint tensor shapes. For Mamba-2, use the orientation of Appendix B: 𝑑𝑘 = 𝑑𝐵 = 128 and 𝑑𝑣 = 𝑑𝑥 = 64. Runtime is measured separately. Quantity

Account

Dense state per head Shallow state

𝑏𝑠 𝑑𝑘 𝑑𝑣 𝑏𝑠 [𝑑𝑘 + (𝑑𝑘 + 𝑑𝑣 )(𝑅 + 𝑤)] at 𝑤 steps after a compression, 0 ≤ 𝑤 ≤ 𝑊 : the sink’s coefficient and the factors; the mean direction is shared by all requests and not counted 𝑀r (𝑇 ) (§2): the state bytes plus 𝑇 times the attention-cache bytes per token, plus the short-convolution windows (which Table 2 leaves out) cache reads, state reads/writes and the weights touched by the batch; sparse attention reads selected tokens and indexer keys 𝑛𝑒 [1 − (1 − 𝑛𝑎 /𝑛𝑒 )𝐵 ] per layer for the 𝐵 tokens of a step, with 𝑛𝑒 routed experts of which each token activates 𝑛𝑎 : exact for independent tokens under uniform routing, an upper bound when independent tokens share a skewed routing distribution two FLOPs per parameter used in layers 0..𝑘−1, code and emitters; no per-token output-head term once per 𝑊 decode steps (per generated token in Table 2) and once at prompt completion (a per-request cost, not in Table 2): fp32 products and reconstruction, fp64 orthonormalization and small eigendecomposition n-gram lookup table traffic, page tables, padding, activations, attention scores and recurrence scans

Resident request Decode traffic Expected routed experts

Prompt computation Compression

Excluded terms

Weights against caches and states. With an 8K-token prompt and 1K output tokens, the weights that a decode step reads outweigh the original model’s cache and state traffic up to a batch of 512 requests on bf16 Qwen3.8-Flash-Next, Kimi-Linear, GLM-5.3-Flash and Nemotron-3.5-Super and of 256 on Qwen3.8-Flash-Next-NVFP4 and Nemotron-3.5-Lightning; at the next power of two, 1024 and 512, the caches and states overtake the weights (Table 17). Prefix storage. The original form keeps dense states and every layer’s cache. Form (i) of Table 18 keeps every layer’s cache (the deep ones as the emitters wrote them) and the shallow states, so a reuse needs no computation. Form (ii) keeps the caches and shallow states of layers 0..𝑘−1 and the prompt code; a reuse decodes the code and reruns the emitters. A code alone is not a further form: its reuse 46

Shallow Encoding Emits Deep Memories

would rerun layers 0..𝑘−1. The smaller form per model and cache precision is a byte account, not an implemented cache policy. Table 18 | With bf16 attention caches, form (ii) of a cached prefix takes up to 22% fewer bytes than form (i), and its reuse costs 0.22–1.07 GFLOP per prompt token; with fp8 caches the two differ by at most 3%, except on Nemotron-3.5-Lightning, where (i) is 12% smaller. A cached prefix of 80K tokens in each storage form: bytes per prompt token (the prefix’s states and convolution windows spread over its tokens) at bf16 and fp8 attention caches, and the GFLOP per prompt token that its reuse costs. (i) every layer’s cache, the deep ones as the emitters wrote them, and the shallow states; (ii) the caches and states of layers 0..𝑘−1 and the code, from which the emitters rewrite the deep layers. † The smaller of (i) and (ii) in bytes. The NVFP4 set of Qwen3.8-Flash-Next has the bytes and FLOPs of the bf16 set; GLM-5.3-Flash has no code, so (i) is its only form, and its fp8 figures halve the latent only, its indexer keys being fp8 already. Orig.

(i)

(ii)

Qwen3.8-Flash-Next

bf16 caches (B) fp8 caches (B) reuse (GFLOP)

26,753 14,081 0

25,727 13,055 0

20,420† 13,028† 1.07

Nemotron-3.5-Lightning

bf16 caches (B) fp8 caches (B) reuse (GFLOP)

6,743 3,671 0

6,384 3,312† 0

5,793† 3,745 0.46

Kimi-Linear

bf16 caches (B) fp8 caches (B) reuse (GFLOP)

8,594 4,562 0

8,342 4,310 0

6,500† 4,196† 0.22

GLM-5.3-Flash

bf16 caches (B) fp8 caches (B) reuse (GFLOP)

13,429 7,797 0

12,572† 6,940† 0

– – –

Nemotron-3.5-Super

bf16 caches (B) fp8 caches (B) reuse (GFLOP)

10,270 6,174 0

9,022 4,926† 0

8,591† 5,007 0.62

47

Related documents

• Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs#1215836
• Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs#1194470
• AMU:Admission and Memory Update for Personalized Conversations---Structured Memory with SLM Guided Control#1132177
• Deeper Backpropagation or More Updates? Training Looped Language Models Across Budgets#1489214
• Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference#1061175