StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

S TRATA F LUX * : M EMORY-H IERARCHY-N ATIVE L OW-B IT LLM I NFERENCE Jake Hyun 1 Ziyu Zhao 2 Jae-sun Seo 1 Mohamed S. Abdelfattah 1 Cornell University

2

Carnegie Mellon University

A BSTRACT Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision data can instead stall generation. We introduce StrataFlux, a framework that challenges this tradeoff by recovering accuracy lost to low-bit weight and key–value (KV) cache quantization while retaining quantization’s memory and speed advantages. StrataFlux co-designs refinement representation, selection, and delivery to make host RAM and solid-state drives useful sources of additional precision. Its selective precision streaming reduces transfer volume, overlaps delivery with computation, and reuses fetched precision across decoding steps so generation need not wait for every transfer. We evaluate 2–4-bit weights and 2–3-bit KV-cache on dedicated GPUs and unified-memory platforms. For 2-bit Llama-3.1-8B-Instruct weights, StrataFlux reduces WikiText-2 perplexity from 9.46 to 8.40 while maintaining 95.4% of the quantized base’s throughput on RTX 5090. With 2-bit weights and KV, StrataFlux raises Qwen3-8B’s AIME 2024/2025 accuracy from 11.0% to 47.1% at unchanged GPU memory use. At 128K context, Llama-3.1-8B-Instruct with 3-bit weights and 2-bit KV achieves comparable perplexity and speed to vLLM’s 4-bit AWQ weights and 8-bit KV on RTX 5090, using approximately half the GPU memory. These results show that GPU-resident precision need not set the accuracy ceiling, nor offloading-tier latency dictate decoding speed.

LLM inference involves three coupled objectives: accuracy, latency, and memory use. Model weights encode learned parameters, while the key–value (KV) cache retains keys and values from earlier tokens. Their representations determine both the precision available for computation and the volume of data stored and moved during generation. Quantization and offloading address limited fast-memory capacity differently. Quantization reduces storage and traffic with fewer bits; it can accelerate bandwidth-bound inference but may degrade accuracy (Frantar et al., 2023; Liu et al., 2024). Offloading preserves stored precision by placing data in a larger, slower backing tier, but transfer latency can delay generation (Na et al., 2025). These approaches suggest an apparent choice between accuracy loss and transfer latency. We argue that this is a false choice imposed by separating representation from placement. Considered alone, quantization discards precision that another tier could retain, while offloading brings in that tier only after choosing the representation. Can we use fewer bits in fast memory without sacrificing accuracy or slowing inference? * StrataFlux is a provisional system name and may change in future revisions.

GPU

Quantization

Offloading

StrataFlux

Time

Time

Time

Bitwidth

I NTRODUCTION

Bitwidth

1

Bitwidth

arXiv:2611.08363v1 [cs.LG] 5 Nov 2026

1

SSD

Speed Accuracy

Speed Accuracy

Speed Accuracy

Compress weights and KV

Transfer from slower storage

Selective Precision Streaming

GPU-resident bits

Fetched bits

Figure 1. StrataFlux against quantization and offloading. Column height indicates bitwidth; horizontal flow indicates time. Selective precision streaming recovers accuracy while retaining low-bit inference’s memory and speed advantages.

We address this question with StrataFlux, which employs selective precision streaming to distribute weight and KV precision across tiers (Figure 1). Compact, quantized weights and KV stay in GPU memory as the resident base, providing high-bandwidth access. Larger, cheaper tiers—host memory or solid-state drives (SSDs)—retain additional precision, but their lower bandwidth and higher latency call for selective fetching and flexible scheduling. Recovering accuracy by fetching all additional precision

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

would erode quantization bandwidth savings. Selectivity is viable because protecting a small subset of weights can improve low-bit accuracy (Lin et al., 2024; Kim et al., 2024), while attention can concentrate on a small, query-dependent subset of tokens (Tang et al., 2024; Ribar et al., 2024). StrataFlux exploits this uneven importance to fetch additional precision selectively for the current computation. Retaining quantization’s speed advantage also requires flexible scheduling: selective transfers can still stall generation when their results are needed immediately. For weights, overlapping a fetch only with the weight-matrix operation it refines limits the time available to hide delivery. StrataFlux widens this window by selecting channels earlier or adding their refinement later, allowing transfers and refinement computation to overlap intervening operations. For both weights and KV, it also reuses fetched precision across decoding steps, avoiding repeated transfers and letting data arriving too late for one step benefit later selections. StrataFlux makes LLM inference native to the memory hierarchy, recovering accuracy while retaining the memory and speed advantages of quantization. We contribute: 1. Selective precision streaming. We augment resident low-bit weights and KV with refinement bits from host RAM or SSD, concentrating transfers on channels and tokens that matter most to each computation. 2. Asynchronous precision delivery. By decoupling when precision is fetched from when it is used, StrataFlux hides the latency of fetches without stalling generation. 3. Hierarchy-native inference. By co-designing numerical representation with memory placement, StrataFlux removes the choice between quantization and offloading, enabling fast, accurate, and memory-efficient inference on both dedicated GPU and unified memory platforms.

size constrains which models and contexts fit in fast memory, while the traffic per token can make memory bandwidth the bottleneck for generation speed (Pope et al., 2023; Liu et al., 2024). Quantization reduces these demands by representing values with low-bit codes, such as codebook indices (Han et al., 2016) or scaled and offset integers (Jacob et al., 2018). Post-training quantization (PTQ) compresses weights without retraining the model (Frantar et al., 2023), while KV-cache quantization compresses the keys and values generated during inference (Liu et al., 2024). These savings introduce errors that affect accuracy differently across values. Large-magnitude outliers can be difficult to represent alongside typical values (Dettmers et al., 2022), while saliency describes how strongly representation errors affect computation or loss. PTQ methods address these differences: SmoothQuant rescales activation and weight channels (Xiao et al., 2023), and AWQ protects salient weight channels through scaling (Lin et al., 2024). SqueezeLLM instead combines sensitivity-weighted codebooks with higher-precision storage for selected sensitive weights and outliers (Kim et al., 2024). 2.2

Dynamic Precision Refinement

These methods account for weight sensitivity when constructing the quantized representation, but that representation remains fixed during decoding. Input activations change with each token, shifting which weight channels contribute most to the projection error. Let W be the original floatingc their dequantized approximation, and point weights, W c the weight residual. For input x, each residual R = W −W column contributes xi R:,i to the projection error. Large activation magnitudes amplify these contributions, making |xi | a proxy for which channels to refine: X c )x = Rx = (W − W xi R:,i . (1) i

For 2-bit Llama-3.1-8B-Instruct weights, StrataFlux reduces WikiText-2 perplexity from 9.46 to 8.40 while maintaining 95.4% of the quantized base’s throughput on RTX 5090. With 2-bit weights and KV, StrataFlux raises Qwen3-8B’s AIME 2024/2025 accuracy from 11.0% to 47.1% at unchanged GPU memory use. At 128K context, Llama-3.18B-Instruct with 3-bit weights and 2-bit KV achieves comparable perplexity and speed to vLLM’s 4-bit AWQ weights and 8-bit KV on RTX 5090, using approximately half the GPU memory.

2

BACKGROUND AND M OTIVATION

2.1

Post-Training Quantization

Interactive on-device LLM inference repeatedly reads model weights and the KV cache to generate new tokens. Their

DecDEC (Park et al., 2025) finds that calibration-selected channels often differ from those with the largest activation magnitudes during decoding. At each step, it selects the k largest |xi |, fetches their 4-bit residual columns from host memory, and adds their products with signed xi to the quantized projection’s output. During single-sequence decoding, the memory-bound projection leaves GPU cores available for concurrent refinement without slowing the projection. Fetching awaits the projection’s input, and refinement must finish before its output is consumed to avoid stalls, confining overlap to that projection’s execution window. We now turn to the KV cache, where the tokens receiving the most attention can likewise change between decoding steps. Attention compares a query, a vector derived from the current token’s representation, with cached keys to weigh the cached values. Top-k attention and Quest exploit this

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

2.3

Bandwidth and Latency Constraints (a) Nested refinement

(b) Quality–budget tradeoff KL divergence

10−1 10−2 10−3 4.7 4-bit refined codes

KVarN 2/2 +1/+1

4.9

5.1

(b) KV cache

100

100

75

75

50

50

25

25

0

0

25

50

75

100

Channels refined (%)

0

0

1

2

3

4

5

KV tokens refined (%)

YAQA 2-bit AWQ 3-bit KVarN 2/2 Ranked Random

Figure 3. Ranked versus random selection in Llama-3.1-8BInstruct. (a) With WikiText-2 inputs, weight channels are ranked by input activation magnitude and refined with corresponding columns of R. (b) For PG-19 inputs, KV tokens are ranked by attention weights from resident low-bit keys and restored to FP16. Output mean squared error (MSE) against FP16 is normalized to the error without refinement, then averaged over tokens and Transformer blocks, across projections in (a) and KV heads in (b).

5.3

K+V budget (bits)

Sparse attention +2/+2 +4/+4

(a) Weights

MSE (%)

dependence to restrict attention to selected tokens (Gupta et al., 2021; Tang et al., 2024). SpeCache instead retains a full low-bit cache and substitutes selected 16-bit entries fetched from host memory (Jie et al., 2025). It uses a speculative token to prefetch selected entries for the next decoding step while current-step computation continues. However, fetching complete 16-bit replacements overlooks the opportunity to jointly design refinement encoding and attention computation to reduce transfer and execution costs.

FP16 fetching +8/+8

Figure 2. (a) Nested refinement subdivides GPU-resident quantization bins with fetched bits. (b) KL divergence from BF16 Llama-3.1-8B-Instruct on PG-19 at matched K+V budgets, including resident, fetched, and metadata bits per K/V pair. K/V bitwidth labels give resident bits per key/value element, respectively. All schemes use KVarN 2/2 and select tokens by attention weights from resident low-bit keys (Equation 7). Sparse attention uses selected FP16 entries alone; FP16 fetching and nested refinement retain unselected low-bit entries. FP16 fetching replaces selected entries; nested +b/ + b adds b bits per key/value element.

Sparse attention reduces KV traffic by omitting unselected tokens from attention. SpeCache preserves their contributions through a low-bit cache and fetches complete 16-bit floating-point (FP16) entries for selected tokens. We instead keep every key and value in low bits on the GPU and, for selected tokens, fetch only the additional bits that refine those resident values, adapting the nested quantization principle used for weights in Any-Precision LLM (Park et al., 2024): the fetched bits pick a finer value inside the interval that the resident low-bit value already represents. Figure 2 compares these three representation choices at matched K+V budgets. Nested refinement with 4 or 8 additional bits per selected key/value element yields lower Kullback–Leibler (KL) divergence than both sparse attention and complete FP16 replacement. Encoding determines how much data each refinement requires; selection determines which errors it addresses. Figure 3 isolates selection by adding selected channels’ residual contributions or restoring selected KV entries to FP16. We rank weight channels by input-activation magnitude and KV tokens by attention weights from resident low-bit keys.

Refining only a small top-ranked fraction removes a disproportionate share of output mean squared error. Activationmagnitude-based selection remains effective even on YAQA, whose rounding compensates for quantization errors across input and output channels (Tseng et al., 2026). Compact representations and selective refinement reduce traffic, but do not eliminate access latency. DecDEC’s overlap window shrinks as lower-bit quantization and higher GPU bandwidth accelerate the underlying projection (Park et al., 2025). On DGX Spark, isolated 2 KB random SSD reads have a median latency of 200 µs (709 µs at the 99th percentile)—far longer than the 21 µs average for Llama-3.1-8B-Instruct’s YAQA 2-bit output projection, making long stalls unavoidable with projection-local overlap. Two prior mechanisms suggest a solution. Each Transformer block’s attention and MLP outputs are added to a running activation vector, the residual stream. Deja Vu finds that this vector’s direction changes slowly across neighboring OPT blocks and uses earlier activations to predict which attention heads and MLP neurons to compute later (Liu et al., 2023). KTransformers’ Expert Deferral adds the outputs of CPUcomputed experts in mixture-of-experts (MoE) models to the residual stream one block later, so that their computation overlaps with the GPU’s work (Chen et al., 2025). These mechanisms extend to weight refinement (Section 2.2). Query/key/value and gate/up projections read from the residual stream, allowing earlier activations to predict channels and initiate fetching before those projections execute (fetch lead). Refinement is executed once each projection’s input becomes available. Attention-output and down projections write to the stream, allowing refinement to be deferred to a later stream update (fetch lag). Together, these cases expose overlap opportunities for every atten-

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

ff

+

SiLU(gate) ⊙ up Down +

100

YAQA 2-bit Query/key/value Gate/up 100

50

50

0 0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 100

AWQ 3-bit Query/key/value Gate/up 100

50

50

0 0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0

(c) E ectiveness of Fetch Lag

KL divergence

MSE (%) MSE (%)

Gate/up +

Fetch lag (s = 0.5)

+

Output

Fetch lead (s = 0.5) | Refine

Query/key/value Attention

ff

(b) E ectiveness of Fetch Lead

Transformer block

KL divergence

(a) Fetch lead and lag

s (blocks) s (blocks) 128 channels 256 channels 512 channels

Output

YAQA 2-bit

Down

0.06 0.01 0.04 0.02 0.00 0.00 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0

Output

AWQ 3-bit

Down

0.02 0.04 0.01 0.02 0.00 0.00 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 s (blocks)

Fetch lag

s (blocks) Never added

Figure 4. Fetch lead/lag in Llama-3.1-8B-Instruct on WikiText-2. Shifts s are measured in Transformer blocks; s = 0 means no shift. (a) Gate/up fetch lead and attention-output fetch lag at s = 0.5. The dashed gold path shows early selection; the solid gold input arrow feeds the later refinement step. (b) Projection-output MSE reduction retained by fetch lead, relative to selection at the projection’s input (100%), using errors summed over tokens and blocks. (c) Fetch-lag refinement is applied to all input channels to isolate the effect of delayed addition. Only attention-output or down projections are quantized; KL to FP16 is averaged over predicted tokens across all sequences. (a) Weight channel caching

An independent opportunity arises from repeated selections across consecutive decoding steps. Figure 5 shows that caching only a fraction of all weight channels or KV tokens can yield very high hit rates. Caching refinement bits for recurring selections avoids repeated transfers. Data arriving too late for one step can also serve subsequent selections. We exploit these two benefits to reduce transfers and waits for weight and KV refinement (Section 3.4).

3

S TRATA F LUX : S ELECTIVE P RECISION S TREAMING

StrataFlux’s selective precision streaming expands the precision available to inference beyond what fits in fast memory or can be fetched within a single projection. Figure 6 outlines its weight and KV computation paths and their shared refinement-cache policy. We tailor refinement delivery to both dedicated-GPU and unified-memory platforms.

100

Query/key/value

Output

0

50

Gate/up

Down

50 0

50

100 0

100 0

50

100 0

50

100

Cached weight channels (%) (b) KV token caching

Hit rate (%)

We evaluate the shifts separately: fetch lead tests how well earlier activations predict which channels to refine, while fetch lag tests how delaying refinement addition affects model accuracy. Earlier selection can preserve most of the projection-output mean squared error (MSE) reduction at small shifts (Figure 4(b)); delayed additions continue to reduce KL divergence relative to omitting refinement (Figure 4(c)). The benefit of refinement thus persists beyond the original projection’s execution window. In Section 3.3, StrataFlux combines fetch lead and lag across Transformer blocks to overlap refinement transfers and refinement computation with intervening model computation.

Hit rate (%)

tion and MLP projection, extending refinement beyond the execution window of a single projection (Figure 4(a)).

100 50 0

0

5

1/256

10

Cached KV tokens (%)

15

1/128 1/64 1/32 Random caching (expected)

20

1/16

Figure 5. Reuse across decoding steps in Llama-3.1-8B-Instruct on PG-19. Selections rank (a) weight channels by input activation magnitude and (b) KV tokens by attention weight (Equation 7); legend fractions give the share selected per step. Caches contain the union of selections from the preceding 1, 2, 4, 8, 16, 32, or 64 steps. Axes show the cached share of all entries against hit rate, averaged over decoding steps and Transformer blocks. Dashed lines show expected hit rates under equally likely random caching.

3.1

Distributing Precision Across the Memory Hierarchy

On dedicated-GPU platforms, GPU memory holds quantized weights and a compact KV cache covering all tokens, while the GPU fetches refinement bits directly from pinned host RAM. On unified-memory platforms, the CPU and GPU share RAM holding these compact representations, while SSD stores refinement bits that we load into shared RAM for GPU access (Section 3.4). These relatively inex-

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference StrataFlux: Selective Precision Streaming

GPU

Refinement storage

Refinement ops Base ops

Previous block Q/K/V Q/K/V fetch lead

Refine

+

qᵀ K̂

Down fetch lag (-1)

αᵀ V Select KV

Rescore

Output

+

Miss

Hit

Refine

Hit

Miss

Gate/up

Output fetch lag

Gate/up fetch lead

Hit

Next block Down

+

+

Down fetch lag Retain for later steps

Hit

Miss

Miss Fill

SSD: Background reads of weight and KV refinement bits via CPU threads Base operations

Linear projections

Attention

StrataFlux operations

+

Q/K/V fetch lead (+1)

Hit

RAM cache

Miss

+

Weight refinement: fetch lead

Weight refinement: fetch lag

KV refinement

Figure 6. StrataFlux's selective precision streaming in action. Fetch lead/lag overlaps weight refinement with model computation; KV refinement runs within attention. Labeled lines replace refinement boxes for previous (−1) and next (+1) Transformer blocks. In this unified-memory configuration, the GPU reads the RAM cache directly; misses retain resident low-bit precision while background SSD reads supply later decoding steps. Dedicated GPUs fetch directly from host RAM, without SSD. Some Transformer operations omitted.

pensive backing tiers let StrataFlux use capacity that would otherwise remain underutilized during LLM inference. 3.2 3.2.1

Designing Precision for Selective Refinement

Mk = Ex [zk (x)zk (x)⊤ ], Gk = Ex [xzk (x)⊤ ], and I is the identity matrix. We estimate the expectations on a small RedPajama calibration set (Weber et al., 2024) and use the mean diagonal of Mk as the default λ: Ck∗ (Mk + λI) = RGk + λR.

Weight Refinement

StrataFlux retains the resident weight quantizer’s format and projection kernels. We store quantized weight-refinement columns as refinement bits; C is the corresponding matrix of dequantized values. For selected input-channel indices S(x), we add the selected columns’ contribution to the base quantized projection: X cx + y′ = W xi C:,i . (2) i∈S(x)

Channel selection. Following DecDEC (Park et al., 2025), we partition each projection’s input channels into chunks of 1,024 and select k channels per chunk, targeting the largest |xi |. These indices form S(x); each selected column’s refinement bits are fetched contiguously. Separate per-chunk budgets kqkv , ko , kgu , and kd govern four projection sets: query/key/value, attention-output, gate/up, and down, respectively. We encode refinement values as integers with separate scale factors, giving the refinement kernel a common format independent of the resident weight quantizer. Fitting refinement values. DecDEC quantizes the weight residual R (Section 2.2). StrataFlux uses refinement fitting to adjust refinement values so selected columns can compensate for omitted contributions. Let zk (x) zero all but the k largest-magnitude activations per input chunk. We minimize projection error while penalizing deviations from R with strength λ: Ck∗ = arg min Ex [∥Rx − Czk (x)∥22 ] + λ∥C − R∥2F . (3) C The minimizer of Equation 3 solves Equation 4, where

(4)

We quantize the fitted matrix to 4 bits before inference, following DecDEC’s 4-bit encoding of weight residuals. Neuron-wise alternative. For input x, the quantized MLP combines gate and up outputs into neuron activations h, then computes its unrefined output yq : cgate x, g= W h= ϕ(g) ⊙ u,

cup x, u= W cdown h. yq = W

(5)

Here ϕ is the gate’s activation function, ⊙ is elementwise multiplication, and hj is neuron j’s activation after gating. Channel-wise refinement targets large input activations separately: x for gate/up and h for down. Neuron-wise refinement instead selects the kn neurons with largest |hj | in each chunk of 1,024 and refines their gate/up rows and down columns together. Each selected neuron’s MLP neuron bundle encodes gate/up refinement rows cgj , cuj , a down refinement column cdj , and a 4-bit copy dj of the BF16 down column. For selected neuron indices N (h), we recompute hj from Equation 5 as h′j . As in Equation 2, the resulting output refinement is added to yq : h′j = ϕ(gj + cgj x)(uj + cuj x), X   y ′ = yq + dj (h′j − hj ) + cdj hj .

(6)

j∈N (h)

Neither channel-wise nor neuron-wise MLP refinement consistently offers the better speed–accuracy tradeoff across models, quantizers, platforms, and speed budgets. We therefore retain both options for StrataFlux.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

3.2.2

3.3

KV Refinement

For KV refinement, we tailor nested reconstruction to the resident cache format. Our main KV implementation, StrataFlux-KV, adopts KVarN’s (Muller et al., 2026) quantization format for its balance of accuracy and memory footprint. We co-design its low-bit attention kernels and selective refinement. For adaptations of StrataFlux to other KV-cache formats, see Appendix E. Nested encoding. As illustrated in Figure 2(a), we subdivide each KVarN code’s quantization interval into equal sub-intervals and store their indices as refinement bits in the backing tier. The resident code and fetched index identify a finer reconstruction, independently for keys and values. Token selection. We compute attention weights by applying softmax to scores between the current query and resident low-bit keys. A KV head stores keys and values shared by one or more query heads. Let αh,i denote the resulting attention weight assigned to token i by query head h, and let H(j) be the query heads sharing KV head j. For token i and KV head j, we define the selection score

Extending the Weight Delivery Interval

The runtime uses fetch lead/lag to decouple refinement fetching from its application, hiding latency through overlap with base operations (Section 2.3). Appendix G explains the CUDA stream and event scheduling. Fetch lead. For each query/key/value or gate/up projection, we apply its normalization to a residual-stream activation s Transformer blocks earlier, then select channels and initiate fetching. Once the projection’s input is available, the refinement kernel multiplies fetched columns by corresponding activations and adds the summed products to the quantized projection’s output (Figure 6). Fetch lag. We add attention-output and down refinements to the residual stream s Transformer blocks after their original addition point, overlapping their fetching and computation with intervening projections (Figure 6). Neuron-wise MLP refinement likewise supports fetch lag. Without fetch lead & lag (s = 0) Base ops

sj (i) = max αh,i . h∈H(j)

(7)

This score captures the strongest attention to the token among query heads sharing its keys and values. For each KV head, we select a configurable number of the highest-scoring tokens for refinement. Attention refinement. Combining fetched bits with resident codes reconstructs selected tokens’ higher-precision keys and values (Figure 6). For one query head, residentP scores ℓi and values vi give output o and normalizer Z = i eℓi . Refining tokens S gives scores ℓ′i , values vi′ , and output
P ′ Zo + i∈S eℓi vi′ − eℓi vi ′
. P o = ′ Z + i∈S eℓi − eℓi

QKV

Refine

O

QKV

GU O

D GU

D

With fetch lead & lag (s = 1) Base ops

QKV

O

GU

GU

R

D

QKV

O

R

O (+1)

D (-1)

GU (+2)

QKV (+1)

Refine

GU

D

−30.7%

O

R

GU (+1)

R QKV (+2)

0

25

50

75

100

125

Time (µs) Projection Fetch lead

Attention Fetch lag

Other operations Refinement without lead/lag

(8)

In practice, we group consecutive tokens within each KV head and pack each group’s nested refinement bits contiguously for larger reads. We score token groups using Equation 7, with each query head’s attention weights summed over the group. Because token selection depends on the initial attention scores and the weighted sum depends on refined scores, KV refinement adds a region that cannot overlap the surrounding attention computation. We limit this overhead through contiguous fetches and fusion of value refinement into the weighted sum. Producing KV refinement bits. During prefill, we compute attention on uncompressed keys and values, then encode resident low-bit codes and offload refinement bits. During decoding, we encode completed blocks of new KV entries in the same format.

Figure 7. Nsight Systems kernel traces comparing refinement without fetch lead/lag (s = 0) and with fetch lead/lag (s = 1) on Llama-3.1-8B-Instruct (AWQ 3-bit, RTX 5090, k = 32). QKV/GU/O/D denote query/key/value, gate/up, output, and down projections; R denotes refinement after fetch lead. Refinement for the previous and next blocks are marked with an offset (-1 or +1). Dashed lines show fetch-lead’s early selection; solid lines show activation reads and writes for refinement.

The Nsight Systems kernel traces of our runtime system in Figure 7 confirm the cross-block overlap illustrated in Figure 6. With the projection-local schedule used by DecDEC (s = 0), the main stream waits for refinement after each projection, delaying subsequent operations in the Transformer block’s computation graph. Fetch lead/lag at s = 1 successfully overlaps refinement with computation across neighboring Transformer blocks, reducing these waits and shortening block execution at matched channel count k.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference YAQA 2-bit

9.5

AWQ 3-bit

3.5

9.0

Tuning Weight Refinement Within a Speed Budget

However, the shift needed to hide refinement depends on the quantization method. At k = 32, Figure 8 shows that slower YAQA 2-bit obtains most of its throughput gain at s = 0.5, whereas faster AWQ 3-bit needs s = 1 because its shorter projections leave less time to hide refinement. Furthermore, these sweeps apply the same s to all four projection sets within each method, leaving their individual speed–accuracy tradeoffs unexplored. In Section 3.5, we introduce a tuner that directly explores this search space to select s and k per projection set under a configurable speed budget.

Candidate selection. For each combination of shift distances and channel-wise or neuron-wise MLP refinement, the tuner searches for the largest common channel budget k that meets the slowdown limit in end-to-end decoding. It selects the retained candidate with the lowest KL divergence from the BF16 reference on held-out text, then increases individual channel budgets or the neuron budget while the measured slowdown remains within the limit. We tune each model, weight quantizer, and platform separately. We then fit refinements at the chosen selection budgets, as detailed in Section 3.2.1.

PPL

Figure 8. Perplexity–throughput tradeoffs on Llama-3.1-8BInstruct with k = 32 and a common fetch lead/lag distance s across all four projection sets. PPL uses WikiText-2; throughput uses PG-19 on RTX 5090 with refinement bits in host RAM.

Search space. The weight tuner selects a configuration subject to a user-specified slowdown limit relative to the quantized base. It chooses channel budgets k, fetch lead/lag distances s for each projection set, and whether the MLP uses channel-wise refinement or neuron-wise refinement with budget kn . Following DecDEC (Park et al., 2025), it also tunes the number of GPU thread blocks (groups of cooperating threads) executing refinement. More thread blocks can accelerate refinement but also slow the concurrent quantized projections by competing for GPU resources.

9.0

8.5

8.5

210

225

240

210

240

270

300

Decode throughput (tok/s) Base only

3.4

s=0

s = 0.5

s=1

s = 1.5

s=2

Reusing Precision Across Decoding Steps

For dedicated-GPU platforms, host RAM’s lower access latency than SSD allows refinement bits to be fetched directly without an intermediate cache. For unified-memory systems, however, SSD’s higher access latency motivates caching fetched refinement bits in GPU-accessible RAM. Caching fetched precision. Weight and KV selections recur across consecutive decoding steps (Figure 5). To avoid repeated SSD reads and their latency, we retain fetched refinement bits in a configurably sized RAM cache that the GPU reads directly (Figure 6). Miss policy. Selections can still miss the cache. For weights and KV, the current step can decide to skip missing refinement and use the resident low-bit representation. A CPU loader fetches the bits into the cache for future selections, keeping the transfer useful without making the requesting step wait. Alternatively, the runtime can wait and apply refinement in the current step, trading speed for accuracy. Cache organization. Weight and KV refinement have separate cache budgets. The weight cache stores refinement bits for all four projection sets and uses least-recently-used replacement. KV token groups share capacity across all KV heads and Transformer blocks. Populating the KV refinement cache. During prefill, we write KV refinement bits to SSD and cache copies for the most recently encoded token groups in RAM. During decoding, newly produced refinement bits enter the cache directly, serving subsequent selections without SSD readback.

4

E VALUATION

We implement StrataFlux as an end-to-end system in llama.cpp (llama.cpp contributors, n.d.), with CUDA and Metal backends. We evaluate StrataFlux on Llama-3.18B-Instruct (Meta, 2024a), Llama-3.3-70B-Instruct (Meta, 2024b), Qwen3-8B, Qwen3-32B, Qwen3-30B-A3B (Qwen Team, 2025), Qwen3.5-9B (Qwen Team, 2026), and DeepSeek-V2-Lite (DeepSeek-AI, 2024). Platforms span dedicated GPUs (RTX 5090, RTX 5070 Ti, and RTX PRO 6000) and unified-memory platforms (DGX Spark and MacBook Air M3). Appendix F reports the Mac evaluation. We measure perplexity and KL divergence from BF16 on 2K-token WikiText-2 windows (Merity et al., 2017) and choose PG-19 (Rae et al., 2020) for long-context perplexity. We measure accuracy on RULER (Hsieh et al., 2024), MATH500 (Lightman et al., 2024), and AIME 2024/2025 (Mathematical Association of America, n.d.); AIME averages eight samples per problem. Timings use llama.cpp except for methods with their own runtimes. Appendix A details protocols and implementations. In tables and figures, StrataFlux-W and StrataFlux-KV isolate weight and KV refinement, respectively; StrataFlux without a suffix denotes the full system with both enabled. Unless ablated, DGX Spark uses separate 128 MiB RAM caches for weight and KV refinement. At the small slowdown budgets used in our main results, we find that skipping cache misses gives a better latency–accuracy tradeoff on unified-memory platforms. There, we skip misses and

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

set s = 0: no SSD wait remains for fetch lead/lag to hide. Appendix B evaluates fetch lead/lag with waiting on cache misses at larger budgets. 4.1

Table 2. Weight-only refinement: Qwen3-8B MATH500/AIME accuracy. Platform columns use StrataFlux-W's respective 5% slowdown configurations. 4-bit results appear in Appendix B.

Weight Refinement

StrataFlux-W DGX Spark

Base

We compare StrataFlux-W with DecDEC (Park et al., 2025) on YAQA (Tseng et al., 2026) with QTIP’s format and kernels (Tseng et al., 2024), LNQ with GuidedQuant (GQ) (Kim et al., 2025), AWQ (Lin et al., 2024), and SqueezeLLM (SqLLM) (Kim et al., 2024) bases.

StrataFlux-W RTX 5090

Weights

M500

AIME

M500

AIME

M500

BF16

97.2

71.0

–

–

–

AIME –

YAQA 2-bit YAQA 3-bit LNQ+GQ 3-bit AWQ 3-bit SqLLM 3-bit

77.2 95.0 91.8 91.8 93.0

19.2 66.7 39.4 46.5 47.3

89.4 95.0 93.0 92.6 94.2

40.0 66.0 47.3 51.9 48.3

92.4 95.2 93.2 93.8 93.0

46.7 70.0 49.2 51.0 50.8

BF16

–

7.22

0.000

9.73

0.000

–

9.46 7.61 8.52 9.10 9.36

0.257 0.047 0.157 0.229 0.263

10.95 9.98 10.92 11.88 11.71

0.214 0.052 0.184 0.284 0.255

StrataFlux-W StrataFlux-W StrataFlux-W StrataFlux-W StrataFlux-W

8.94 7.55 8.29 8.64 8.56

0.200 0.041 0.135 0.179 0.174

10.77 9.89 10.70 11.30 11.03

0.175 0.043 0.159 0.222 0.201

DecDEC StrataFlux-W DecDEC StrataFlux-W DecDEC StrataFlux-W DecDEC StrataFlux-W DecDEC StrataFlux-W

9.03 8.40 7.55 7.49 8.37 8.18 8.80 8.38 9.02 8.42

0.211 0.139 0.040 0.033 0.142 0.121 0.195 0.145 0.225 0.156

10.69 10.51 9.89 9.89 10.66 10.47 11.38 10.71 11.36 10.81

0.184 0.132 0.044 0.033 0.170 0.139 0.231 0.167 0.215 0.169

YAQA 2-bit YAQA 3-bit LNQ+GQ 3-bit AWQ 3-bit SqLLM 3-bit

Base

YAQA 2-bit YAQA 3-bit DGX Spark LNQ+GQ 3-bit AWQ 3-bit SqLLM 3-bit YAQA 2-bit YAQA 3-bit RTX 5090

LNQ+GQ 3-bit AWQ 3-bit SqLLM 3-bit

L3.1-8B-I PPL KL

We use a 5% slowdown budget for main weight comparisons because it is sufficient for large accuracy gains (Figure 9(a)). At matched budgets, StrataFlux-W outperforms DecDEC across quantizers (Table 1). These gains extend to reasoning tasks (Table 2) while retaining quantized speed and low GPU memory use (Figure 9). For Qwen3-8B with YAQA 2-bit weights, AIME accuracy rises from 19.2% to 46.7%. Appendix B gives full 2–4-bit, larger-model, and downstream results; Appendix B.7 lists tuned configurations. 4.2

9.50 9.25 9.00 8.75

KV Refinement

KV baselines include KVarN (Muller et al., 2026), KIVI (Liu et al., 2024), KVQuant (Hooper et al., 2024), TurboQuant (Zandieh et al., 2026), and AQUA-KV (Shutova et al., 2025); offloading comparisons include ShadowKV (Sun et al., 2025), ParisKV (Qi et al., 2026), TailorKV (Yao et al., 2025), FreeKV (Liu et al., 2026),

DGX Spark

RTX 5090

9.5

9.2

8.5

8.4 64

8.0

72

Throughput (tok/s) 9.50 9.25 9.00 8.75

240 280

135 150 165

Throughput (tok/s)

Throughput (tok/s) 9.5

9.2

9.0

8.8

8.5

8.4 5.0

4.5

8.0

4.0

4.5

GPU memory (GB)

RTX 5070 Ti

9.0

8.8

56

Qwen3-8B PPL KL

PPL

Weights

Refinement method

PPL

(a) Weight refinement tradeoffs

Table 1. Weight-only refinement: WikiText-2 perplexity and KL divergence with 2–3-bit weights. Platform blocks report StrataFluxW, alongside DecDEC on RTX 5090, tuned to a 5% slowdown budget. DecDEC does not support SSD fetching on DGX Spark. Bold marks the better score within each RTX 5090 pair.

4.0

4.5

GPU memory (GB)

YAQA 2-bit StrataFlux-W

4.0

GPU memory (GB)

AWQ 3-bit DecDEC Base

(b) Matched-budget offloading VRAM PPL 3.9 GB 9.46

YAQA 2-bit

3.9 GB 8.40

StrataFlux-W (5%)

3.9 GB 7.22

245.8 234.4

llama.cpp offloading 5.1 0

100

200

tok/s

Figure 9. Weight-refinement tradeoffs on Llama-3.1-8B-Instruct on WikiText-2. (a) Throughput plots connect configurations tuned for slowdown budgets of 2.5% / 5% / 10% / 20%; memory plots show the configurations tuned for the 5% budget. DGX Spark memory includes the RAM cache. DecDEC does not support SSD fetching on Spark. (b) StrataFlux-W tuned for 5% slowdown on YAQA 2-bit base, and llama.cpp BF16 partial offloading at a matched 3.9 GB GPU memory budget, RTX 5090.

SpeCache (Jie et al., 2025), and KVSwap (Zhang et al., 2026). StrataFlux is configured to select 1/128 (0.78%) of token groups for refinement, adding 4 bits per selected key and value element (+4/+4). Groups contain two consecutive tokens on dedicated GPU platforms and four on unified-memory platforms. For other refinement settings, see Appendix C.6; for resident 2/1 KV, see Appendix C.5. StrataFlux-KV pushes beyond the memory–accuracy and throughput–accuracy frontiers of existing KV quantization and offloading methods, bringing near-BF16 accuracy to 2-bit keys and values without giving up low-bit memory savings or fast decoding (Table 3; Figure 10(a)). Accuracy recovery also extends to alternative attention architectures

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference (a) KV refinement 16.27; 1.1 GB

Table 3. KV-only refinement: PG-19 perplexity/KL and RULER accuracy at 128K, plus MATH500/AIME accuracy, with BF16 weights. Platform blocks report StrataFlux-KV.

16.27

L3.1-8B-I

11.4 10.8

36.02 15.82; 1.3 GB

15.82

36.02

71.0

96.2 96.6 95.8

66.0 69.2 70.6

2/2 StrataFlux-KV 3/2 DGX Spark 3/3

10.47 0.011 75.2 13.32 0.030 71.4 10.34 0.002 77.6 12.98 0.005 74.0 10.33 0.001 77.5 12.93 0.004 74.2

96.0 95.0 96.2

68.3 70.2 69.4

2/2 StrataFlux-KV 3/2 RTX 5090 3/3

10.40 0.003 78.1 12.98 0.007 74.2 10.33 0.001 78.3 12.92 0.002 74.7 10.33 0.001 78.2 12.90 0.001 74.5

95.6 96.2 97.0

69.2 70.0 71.5

13.6

4

2

3

6

9

13.07

11.6

0.2 GB

11.0 10.4 15.10 36.02

36.02 15.10

Aggregate tok/s

L3.1-8B-I

1.3 GB

PPL

Qwen3-8B

12

13.07

PPL

RTX 5090

97.2

10.98 0.067 68.6 14.40 0.130 66.0 10.45 0.014 76.2 13.17 0.026 72.3 10.39 0.008 77.2 13.02 0.016 72.9

Base

6

1.4 GB

14.4 0.1 GB

13.6 12.8 6

4

2

20

KV cache (GB) Quantization

KVarN

KV offloading StrataFlux-KV

2/2

4/2 KIVI

2

ShadowKV

ParisKV

SpeCache

2/2

3/3

3/2

40

60

4 KVQ

2

3

4 TQ

TailorKV

2.5

FreeKV

3.5 AQUA-KV

2

3

18.67

16.27

18.67

L3.1-8B-I

19.48

19.48

PPL

Qwen3-8B

DGX Spark

10.0

15.0 13.5 12

8

6

12

18

18.67

24 18.67

PPL

L3.1-8B-I

0

0 2

4

8

16 32

1

Batch size Base

12.5

16

100

20

KVSwap

15.0

RTX PRO 6000 Max-Q

40

4

(b) Full system (weights + KV) 16.27

DGX Spark

1

Throughput (tok/s)

PPL

RTX 5090

Qwen3-8B R128 M500 AIME

2/2 3/2 3/3

14.4

12.8

KL

BF16 10.32 0.000 77.9 12.89 0.000 74.1

PPL

2

4

8

16

Batch size StrataFlux-KV

Figure 11. StrataFlux’s KV refinement overhead versus batch size on Llama-3.1-8B-Instruct at 128K, using 2/2 KV.

and MoE models (Appendix C, Table A20). On RTX 5090, Llama-3.1-8B-Instruct with 2/2 KV improves from 68.6% to 78.1% RULER accuracy at 128K. For full task and context comparisons, see Appendix C. At batch sizes 1 and 16, refinement reduces throughput by 11.2% and 18.8% on RTX PRO 6000 Max-Q, versus 7.0% and 25.4% on DGX Spark (Figure 11).

13.5

4.3

12.0 10.5 19.48

19.48

PPL

Qwen3-8B

L3.1-8B-I PPL KL R128 PPL

KV

10.2

Qwen3-8B

DGX Spark

PPL

15 14 13

20

15

10

5

30

GPU memory (GB) Quantization

KVarN 2/2 + YAQA

KV offloading

ShadowKV

Runtimes

llama.cpp

StrataFlux

KV 2/2 + YAQA

2-bit

ParisKV

3-bit

4-bit LNQ+GQ

SpeCache

TailorKV

Q4_K_M/q8_0 KV EXL3 2-bit

3-bit

60

90

Throughput (tok/s) 4-bit

FreeKV

4 bpw/KV 4/4 vLLM

4-bit LNQ+GQ

KVSwap AWQ4/FP8 KV QServe

W4A8KV4

4-bit

Figure 10. PG-19 perplexity at 128K: (a) KV refinement with BF16 weights; (b) Weight and KV joint refinement. Memory counts KV in (a), weights+KV in (b), including DGX Spark’s RAM cache. Out-of-range points are shown beside the corresponding borders with their values. Of the evaluated KV offloading systems, only KVSwap supports DGX Spark. The gray boundary marks the baseline frontier. Dashed lines mark BF16 PPL. W/A/KV: weight/activation/cache bits; bpw: bits/weight; KVQ: KVQuant; TQ: TurboQuant.

Full-System Results

We enable both refinement techniques and compare the full StrataFlux system with static weight/KV pairs, vLLM (Kwon et al., 2023), EXL3 (ExLlamaV3 contributors, n.d.), QServe (Lin et al., 2025), llama.cpp, and KV offloading systems. Together, weight and KV refinement let StrataFlux recover long-context and reasoning accuracy with aggressively compressed weights and KV (Table 4), pushing beyond the memory–accuracy and throughput– accuracy frontiers of static quantization pairs, KV offloading systems, and inference runtimes (Figure 10(b)). On RTX 5090, refinement raises Qwen3-8B AIME accuracy from 11.0% to 47.1% with YAQA 2-bit weights and 2/2 KV. Appendix D gives the full configuration comparisons and task results. 4.4

Ablations

Fetch lead/lag and fitting improve accuracy, while the better MLP scheme, channel-wise refinement or neuron bundles,

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Table 4. Joint refinement: PG-19 perplexity/KL and RULER accuracy at 128K, plus MATH500/AIME accuracy. Platform blocks report StrataFlux. L3.1-8B-I Qwen3-8B KV PPL KL R128 PPL KL R128 M500 AIME

Weights BF16

10.32 0.000 77.9 12.89 0.000 74.1 97.2

71.0

YAQA 2-bit 2/2 18.67 0.656 50.4 19.48 0.434 58.8 YAQA 3-bit 2/2 11.75 0.146 67.2 15.15 0.185 63.8 YAQA 4-bit 2/2 11.24 0.090 67.5 14.58 0.143 65.4 LNQ+GQ 4-bit 2/2 11.62 0.129 64.3 14.91 0.167 64.0

68.0 94.6 95.0 95.2

11.0 59.6 62.5 58.3

YAQA 2-bit 2/2 15.15 0.422 60.5 15.98 0.228 65.2 StrataFlux YAQA 3-bit 2/2 10.99 0.064 73.2 13.75 0.068 68.4 DGX Spark YAQA 4-bit 2/2 10.66 0.026 73.5 13.43 0.040 70.3 LNQ+GQ 4-bit 2/2 10.92 0.055 71.1 13.71 0.062 70.6

89.6 95.2 96.2 96.0

40.6 68.8 69.0 65.6

YAQA 2-bit 2/2 13.79 0.312 63.8 14.91 0.165 68.3 StrataFlux YAQA 3-bit 2/2 10.80 0.045 77.1 13.25 0.038 71.2 RTX 5090 YAQA 4-bit 2/2 10.55 0.015 77.2 13.04 0.015 73.5 LNQ+GQ 4-bit 2/2 10.77 0.038 75.2 13.30 0.035 73.8

91.2 95.4 96.4 94.4

47.1 65.6 69.2 61.5

Base

(a) Weight techniques

L3.1-8B-I

10.8

tok/s

10.7 100 + channel-wise

+ fit refinements

+ fetch lead/lag

(c) KV cache

Weights

(d) Weight cache

KV

120 Both

(e) Skip vs wait

PPL

PPL

9.2

9.2

8.8

8.8

10.8 10.6

8.4

10.4

8.4

128 256 512 Cache (MiB)

64

2/2 KV no refinement

100

Neither

128 256 512 Cache (MiB)

YAQA 2-bit

no cache misses

no refinement

25

AWQ 3-bit

50 tok/s

75

no refinement skip misses wait on misses

no cache misses

(f) Storage BF16 reference: 33.24 GB Unified

RAM SSD

3.86 4.43

2.58

6.71 4.30

Dedicated

VRAM RAM

3.86 4.43

2.58

6.44 4.30

0

2 Weights

4 KV

6 RAM cache (W)

5

R ELATED W ORK

Heterogeneous-memory inference systems accommodate weights and KV caches beyond GPU memory. FlexGen (Sheng et al., 2023) schedules compressed tensors across GPU, CPU, and disk; InfiniGen (Lee et al., 2024) and FreeKV (Liu et al., 2026) prefetch selected KV entries from host memory. Quest (Tang et al., 2024) and SparQ (Ribar et al., 2024) reduce KV reads through query-dependent selection. SpeCache (Jie et al., 2025) supplements a full low-bit cache with prefetched 16-bit entries. StrataFlux instead co-designs low-bit attention and selectively fetched refinement bits, using caching and skipped misses to recover accuracy without stalling decoding on backing-tier reads.

tok/s

120

+ neuron bundles

PPL

64

PPL 13.5 13.4 13.3 13.2

10.9

8.0 Base

Qwen3-8B

PPL

+0.18

−0.40

−0.12

8.5

−0.20

9.0

−0.09

9.5

(b) Weight + KV

AWQ 3-bit

−0.19

−0.70

YAQA 2-bit −0.07

PPL

sweeps. With 4-bit weight and KV refinement values, backing-tier storage scales roughly with 4-bit weights and KV, plus metadata and values duplicated in neuron bundles (Figure 12(f)). For other KV bases, see Appendix E.

8.73 8.73 8

GB

For weights, DecDEC (Park et al., 2025) selects refinement channels from current activations and overlaps host-memory fetching and refinement with the corresponding quantized projection. Other inference systems overlap work across Transformer blocks: Deja Vu (Liu et al., 2023) uses earlier activations to predict which attention heads and MLP neurons to compute, while KTransformers (Chen et al., 2025) defers expert outputs to overlap CPU and GPU computation. StrataFlux extends weight-refinement overlap beyond DecDEC’s projection window through early channel selection and deferred addition, and fits refinement values to compensate for unselected channels.

10

RAM cache (KV)

Figure 12. Refinement ablations on Llama-3.1-8B-Instruct, using RTX 5090 in (a,b) and DGX Spark in (c–e). (a) Adding weightrefinement techniques, re-tuned to a 5% slowdown budget. Labels show WikiText-2 PPL changes. The outlined AWQ bar forces neuron bundles. (b) PPL versus throughput for separate and combined weight/KV refinement (also on Qwen3-8B), using PG-19 at 128K, YAQA 3-bit weights, 3/3 KV, and the 5% weight configuration. (c,d) PPL versus RAM-cache capacity with skipped misses: (c) 2/2 KV on PG-19; (d) weight refinement on WikiText-2 at 5% configurations. (e) WikiText-2 PPL versus throughput for skipping and waiting on weight-cache misses with a 128 MiB cache. (f) Storage (GB) by tier: YAQA 2-bit weights and 2/2 KV at 128K.

depends on the quantizer (Figure 12(a)). Weight and KV refinement yield approximately additive accuracy gains and decoding-time overheads, allowing both to run together without negative interference (Figure 12(b)). RAM caches make SSD-backed refinement effective despite skipped misses. Larger caches improve accuracy, while skipping avoids the throughput penalty of waiting (Figure 12(c–e)). Appendix C extends the fetch-budget and cache-capacity

6

C ONCLUSION

StrataFlux recovers accuracy from low-bit weights and KV caches while retaining quantization’s memory and speed advantages. Its selective precision streaming makes host RAM and SSD useful sources of additional precision by coordinating what is fetched, when it arrives, and how it refines computation. Across dedicated GPUs and unifiedmemory platforms, StrataFlux demonstrates that precision distributed across the memory hierarchy can improve accuracy without letting backing-tier transfers dictate decoding speed.

R EFERENCES Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119–3137. Association

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

for Computational Linguistics, 2024. doi: 10.18653/v1/ 2024.acl-long.172. URL https://aclanthology. org/2024.acl-long.172/. Chen, H., Xie, W., Zhang, B., Tang, J., Wang, J., Dong, J., Chen, S., Yuan, Z., Lin, C., Qiu, C., Zhu, Y., Ou, Q., Liao, J., Chen, X., Ai, Z., Wu, Y., and Zhang, M. KTransformers: Unleashing the full potential of CPU/GPU hybrid inference for MoE models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP ’25. Association for Computing Machinery, 2025. doi: 10.1145/3731569.3764843. URL https: //doi.org/10.1145/3731569.3764843. DeepSeek-AI. DeepSeek-V2-Lite. Model card, 2024. URL https://huggingface.co/deepseek-ai/ DeepSeek-V2-Lite. Accessed October 30, 2026. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, volume 35, pp. 30318–30332. Curran Associates, Inc., 2022. doi: 10.52202/068431-2198. ExLlamaV3 contributors. ExLlamaV3: Quantization and inference library. Software repository, n.d. URL https://github.com/turboderp-org/ exllamav3. Accessed October 30, 2026. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=tcbBPnfwxS. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/ 10256836.

Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, volume 37, pp. 1270–1303. Curran Associates, Inc., 2024. doi: 10.52202/079017-0040. URL https://doi.org/ 10.52202/079017-0040. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling, 2024. URL https://openreview.net/forum? id=kIoBbc76Sy. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704–2713, 2018. URL https: //openaccess.thecvf.com/content_cvpr_ 2018/html/Jacob_Quantization_and_ Training_CVPR_2018_paper.html. Jie, S., Tang, Y., Han, K., Deng, Z.-H., and Han, J. SpeCache: Speculative key-value caching for efficient generation of LLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 27917– 27928. PMLR, 2025. URL https://proceedings. mlr.press/v267/jie25a.html. Kim, J., El Halabi, M., Park, W., Schaefer, C. J., Lee, D., Park, Y., Lee, J. W., and Song, H. O. GuidedQuant: Large language model quantization via exploiting end loss guidance. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 30011–30037. PMLR, 2025. URL https://proceedings.mlr.press/ v267/kim25d.html.

Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J. Memory-efficient transformers via top-k attention. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pp. 39– 52. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.sustainlp-1.5. URL https:// aclanthology.org/2021.sustainlp-1.5/.

Kim, S., Hooper, C. R. C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. SqueezeLLM: Dense-and-sparse quantization. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 23901–23923. PMLR, 2024. URL https://proceedings.mlr.press/ v235/kim24f.html.

Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In The Fourth International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1510.00149.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In ACM Symposium on Operating Systems Principles, pp. 611–626,

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

2023. doi: 10.1145/3600006.3613165. URL https: //doi.org/10.1145/3600006.3613165. Lee, W., Lee, J., Seo, J., and Sim, J. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation, pp. 155–172. USENIX Association, 2024. URL https://www.usenix.org/conference/ osdi24/presentation/lee. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In International Conference on Learning Representations, 2024. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pp. 87–100, 2024. Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Proceedings of Machine Learning and Systems, volume 7, 2025. URL https://hanlab.mit.edu/ projects/qserve. Liu, G., Li, C., Ning, Z., Lin, J., Yao, Y., Ke, D., Guo, M., and Zhao, J. FreeKV: Boosting KV cache retrieval for efficient LLM inference. In International Conference on Learning Representations, 2026. URL https:// openreview.net/pdf?id=wXAn7orB1H. Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Ré, C., and Chen, B. Deja Vu: Contextual sparsity for efficient LLMs at inference time. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 22137–22176. PMLR, 2023. URL https://proceedings.mlr.press/ v202/liu23am.html. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32332–32344. PMLR, 2024. URL https://proceedings.mlr.press/ v235/liu24bz.html. llama.cpp contributors. llama.cpp: LLM inference in C/C++. Software repository, n.d. URL https://github. com/ggml-org/llama.cpp. Accessed October 30, 2026.

Mathematical Association of America. AIME 2024 and 2025 problems. Competition problems, reproduced with permission by LIVE by Po-Shen Loh, n.d. URL https://live.poshenloh.com/ past-contests. AIME I and II, 2024 and 2025. Accessed October 30, 2026. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https:// openreview.net/forum?id=Byj72udxe. Meta. Llama-3.1-8B-Instruct. Model card, 2024a. URL https://huggingface.co/meta-llama/ Llama-3.1-8B-Instruct. Accessed October 30, 2026. Meta. Llama-3.3-70B-Instruct. Model card, 2024b. URL https://huggingface.co/meta-llama/ Llama-3.3-70B-Instruct. Accessed October 30, 2026. Muller, L. K., Bich, P., Boretti, C., Chang, H.-M., Zhuang, J., and Cavigelli, L. KVarN: Variance-normalized KVcache quantization mitigates error accumulation in reasoning tasks, 2026. URL https://arxiv.org/abs/ 2606.03458. Na, S., Jeong, G., Ahn, B. H., Jezghani, A., Young, J., Hughes, C. J., Krishna, T., and Kim, H. FlexInfer: Flexible LLM inference with CPU computations. In Proceedings of Machine Learning and Systems, volume 7. MLSys, 2025. Park, Y., Hyun, J., Cho, S., Sim, B., and Lee, J. W. Any-precision LLM: Low-cost deployment of multiple, different-sized LLMs. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 39682– 39701. PMLR, 2024. URL https://proceedings. mlr.press/v235/park24e.html. Park, Y., Hyun, J., Kim, H., and Lee, J. W. DecDEC: A systems approach to advancing low-bit LLM quantization. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25), pp. 803–819. USENIX Association, 2025. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, volume 5, pp. 606–624, 2023. Qi, Y., Chen, X., Jiang, H., Wang, Q., Peng, B., and Palpanas, T. ParisKV: Fast and drift-robust KV-cache retrieval for long-context LLMs. In International Con-

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

ference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 100229–100248, 2026. URL https://proceedings.mlr.press/ v306/qi26b.html. Qwen Team. Qwen3 Model Family. Model repository, 2025. URL https://github.com/QwenLM/ Qwen3. Accessed October 30, 2026. Qwen Team. Qwen3.5-9B. Model card, 2026. URL https: Ac//huggingface.co/Qwen/Qwen3.5-9B. cessed October 30, 2026. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for longrange sequence modelling. In International Conference on Learning Representations, 2020. URL https:// openreview.net/forum?id=SylKikSYDH. Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., and Orr, D. SparQ attention: Bandwidth-efficient LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42558–42583. PMLR, 2024. URL https://proceedings.mlr.press/ v235/ribar24a.html. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094–31116. PMLR, 2023. URL https://proceedings.mlr.press/ v202/sheng23a.html. Shutova, A., Malinovskii, V., Egiazarian, V., Kuznedelev, D., Mazur, D., Surkov, N., Ermakov, I., and Alistarh, D. Cache me if you must: Adaptive key-value quantization for large language models. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 55451–55473, 2025. URL https://proceedings.mlr.press/ v267/shutova25a.html. Sun, H., Chang, L.-W., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y., and Chen, B. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 57355–57373, 2025. URL https://proceedings.mlr.press/ v267/sun25b.html.

Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47901– 47911. PMLR, 2024. URL https://proceedings. mlr.press/v235/tang24l.html. Tseng, A., Sun, Q., Hou, D., and De Sa, C. QTIP: Quantization with trellises and incoherence processing. In Advances in Neural Information Processing Systems, volume 37, pp. 59597–59620. Curran Associates, Inc., 2024. doi: 10.52202/079017-1904. URL https: //doi.org/10.52202/079017-1904. Tseng, A., Sun, Z., and De Sa, C. Modelpreserving adaptive rounding. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 122351–122370. PMLR, 2026. URL https://proceedings.mlr.press/ v306/tseng26b.html. Weber, M., Fu, D. Y., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. RedPajama: an open dataset for training large language models. In Advances in Neural Information Processing Systems, volume 37, pp. 116462–116492. Curran Associates, Inc., 2024. doi: 10.52202/079017-3697. URL https://doi.org/10.52202/079017-3697. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient posttraining quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 38087–38099. PMLR, 2023. URL https://proceedings.mlr.press/ v202/xiao23c.html. Yao, D., Shen, B., Lin, Z., Liu, W., Luan, J., Wang, B., and Wang, W. TailorKV: A hybrid framework for long-context inference via tailored KV cache optimization. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 20340–20359, 2025. doi: 10.18653/v1/2025. findings-acl.1043. URL https://aclanthology. org/2025.findings-acl.1043/. Zandieh, A., Daliri, M., Hadian, M., and Mirrokni, V. TurboQuant: Online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations, pp. 56418–56439, 2026. URL https: //openreview.net/pdf?id=tO3ASKZlok.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

Zhang, H., Xia, C., and Wang, Z. KVSwap: Disk-aware KV cache offloading for long-context on-device inference. In ACM International Conference on Mobile Systems, Applications and Services, pp. 723–736, 2026. doi: 10. 1145/3745756.3809234. URL https://doi.org/ 10.1145/3745756.3809234.

AI U SAGE D ISCLOSURE We used Anthropic’s Claude Code and OpenAI’s Codex to assist with implementation and debugging, evaluation and visualization scripts, figure and table preparation, and manuscript revision. These tools were used iteratively under author direction. The authors determined the research objectives, directed revisions, and selected the material included in the paper, and remain responsible for the implementation, experimental results, and final manuscript.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

A PPENDIX C ONTENTS A Evaluation Setup and Implementation Details

16

B Extended Weight Evaluation

16

B.1 Accuracy and Downstream Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

B.2 Long-Context Weight-Only Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

B.3 Larger Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

B.4 Slowdown-Budget Sweeps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

B.5 YAQA Recovery Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

B.6 Refinement and Cache Ablations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

B.7 Weight Refinement Tuner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C Extended KV Evaluation

24

C.1 Accuracy and Context Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

C.2 Alternative Attention Architectures and MoE Models . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

C.3 KV Representation and Selection Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

C.4 Fetch Budgets, RAM Caches, and Miss Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

C.5 Refinement with 2-bit Keys and 1-bit Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

C.6 KV Refinement Settings Sweep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

D Joint Configurations and Task Results

31

D.1 Configuration Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

D.2 Full Task Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

E Applicability to Other KV Bases

33

F Mac Evaluation

34

G Parallel Weight Refinement with CUDA Streams

34

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

A

E VALUATION S ETUP AND I MPLEMENTATION D ETAILS

Perplexity and KL. Weight evaluation uses 2K-token WikiText-2 windows. For long-context evaluation, we sample 17 PG-19 test books with sufficient length and evaluate 16 windows of 128K tokens from each book. We prefill each window except its final 256 tokens, then score those tokens sequentially with teacher forcing. Context-length sweeps shorten the windows to 8K–128K tokens while keeping their endpoints and scored tokens fixed. KL uses the same datasets, windows, and scored tokens as perplexity. We approximate divergence from BF16 by keeping its 1,024 most probable vocabulary tokens separate and grouping all remaining tokens into one category in both distributions. Evaluation software. Weight evaluation uses the Language Model Evaluation Harness (lm-eval) (Gao et al., 2023). We evaluate LongBench (Bai et al., 2024) using the benchmark’s evaluation scripts. Timing and memory. We implement StrataFlux in llama.cpp, with CUDA 13.4.2 on NVIDIA GPUs and Metal on the Mac. For fair comparisons, we port baselines with author-provided kernels into the same framework. Systems with their own runtimes, including KV offloading systems, vLLM, and ExLlamaV3, are measured in their respective implementations. KV and joint decode timing uses a PG-19 prompt, prefill to C − 641 tokens for context length C, 512 untimed steps, and 128 timed steps. For weight-only comparisons on NVIDIA GPUs, we measure decode throughput at empty context for WikiText-2 and at 128K context for PG19. Memory counts stored representations, metadata, and RAM caches where used; captions specify whether weights, KV, or both are counted. Memory values at 128K context assume 131,072 tokens. KV refinement settings. Main evaluations use resident key/value bitwidths of 2/2, 3/2, and 3/3, with +4/+4 refinement bits and a 1/128 token-group fetch fraction. Groups contain two tokens on dedicated GPUs and four on unifiedmemory platforms. Appendix C.6 varies resident and fetched bitwidths, token-group size, and fetch fraction. The first 128 tokens remain in FP16 unless ablated. A 128-token buffer for the current token block also remains in FP16; resident-memory totals include both allocations. Baseline Kernels Integrated into llama.cpp YAQA. We integrate QTIP’s released decompression and matrix–vector kernels for YAQA’s weight format. SqLLM and LNQ+GQ. SqLLM uses Any-Precision LLM’s quantizer and kernels; LNQ+GQ uses GuidedQuant’s weights with adapted Any-Precision kernels. AWQ 4-bit. We integrate the authors’ released matrix–

vector and matrix–matrix kernels. KVQuant. We add grouped-query attention support to KVQuant’s released quantization and attention kernels. KIVI. We retain KIVI’s released matrix–vector kernels and supply packing and softmax integration in llama.cpp. KVarN. KVarN baselines use the released Triton kernels. DecDEC. We port DecDEC’s correction kernel into llama.cpp and select its configurations with DecDEC’s tuner. Our Baseline Implementations and Adaptations AWQ 3-bit. The authors release a 4-bit matrix–vector kernel but no 3-bit kernel, so we implement a group-size-128 3-bit kernel in llama.cpp. TurboQuant. With no author-released implementation available at integration time, we adapt the community vLLM backend (v0.30.0) to llama.cpp and match the paper’s quantization settings. AQUA-KV. The authors release PyTorch code without CUDA kernels; we adapt it to CUDA, reconstructing cached keys and values during decoding. SpeCache. With no author-released code available, we implement the paper’s low-bit KV cache and selective FP16 fetching in llama.cpp. Systems Evaluated in Their Own Runtimes We measure ShadowKV, FreeKV, ParisKV, TailorKV, and KVSwap in their released runtimes. The vLLM, ExLlamaV3, and QServe comparisons also use their respective runtimes; we add Qwen3-8B support to QServe. StrataFlux Implementation Weights. We adapt DecDEC’s dynamic error-compensation kernel, replacing bucket-based selection with exact radixsort selection and improving zero-copy request batching at low GPU thread-block counts. We add fetch lead/lag scheduling, RAM caches, and refinement delivery logic, and develop an independent tuner inspired by DecDEC’s search procedure. KV. We adopt KVarN’s quantization algorithm and independently develop CUDA and Metal attention kernels with fused refinement operations.

B

E XTENDED W EIGHT E VALUATION

We extend Sections 4.1 and 4.4 across weight bases, model sizes, slowdown budgets, and cache configurations. We also test weight refinement with YAQA’s recovery fine-tuning (Section B.5).

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

B.1

Accuracy and Downstream Tasks

Tables A1 and A2 extend the 2–4-bit PPL/KL comparisons across platforms; Table A4 adds downstream DecDEC comparisons. Table A1. Weight-only refinement: WikiText-2 perplexity/KL with DGX Spark configurations at 5% slowdown budgets. Weights BF16 YAQA 2-bit YAQA 3-bit YAQA 4-bit LNQ+GQ 3-bit LNQ+GQ 4-bit AWQ 3-bit AWQ 4-bit SqLLM 3-bit SqLLM 4-bit

Method Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W

L3.1-8B-I PPL KL 7.22 0.000 9.46 0.257 8.94 0.200 7.61 0.047 7.55 0.041 7.33 0.014 7.31 0.012 8.52 0.157 8.29 0.135 7.48 0.034 7.43 0.031 9.10 0.229 8.64 0.179 7.59 0.049 7.51 0.039 9.36 0.263 8.56 0.174 7.56 0.052 7.48 0.040

Qwen3-8B PPL KL 9.73 0.000 10.95 0.214 10.77 0.175 9.98 0.052 9.89 0.043 9.79 0.016 9.72 0.014 10.92 0.184 10.70 0.159 9.95 0.044 9.87 0.038 11.88 0.284 11.30 0.222 10.01 0.059 9.96 0.047 11.71 0.255 11.03 0.201 9.98 0.062 9.97 0.048

Table A2. Weight-only refinement: WikiText-2 perplexity/KL with configurations tuned on RTX 5090 at 5% slowdown budgets. Weights BF16 YAQA 2-bit YAQA 3-bit YAQA 4-bit LNQ+GQ 3-bit LNQ+GQ 4-bit AWQ 3-bit AWQ 4-bit SqLLM 3-bit SqLLM 4-bit

Method Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W

L3.1-8B-I PPL KL 7.22 0.000 9.46 0.257 9.03 0.211 8.40 0.139 7.61 0.047 7.55 0.040 7.49 0.033 7.33 0.014 7.30 0.011 7.29 0.010 8.52 0.157 8.37 0.142 8.18 0.121 7.48 0.034 7.42 0.031 7.41 0.027 9.10 0.229 8.80 0.195 8.38 0.145 7.59 0.049 7.52 0.042 7.44 0.031 9.36 0.263 9.02 0.225 8.42 0.156 7.56 0.052 7.51 0.044 7.44 0.035

Qwen3-8B PPL KL 9.73 0.000 10.95 0.214 10.69 0.184 10.51 0.132 9.98 0.052 9.89 0.044 9.89 0.033 9.79 0.016 9.75 0.013 9.75 0.010 10.92 0.184 10.66 0.170 10.47 0.139 9.95 0.044 9.89 0.038 9.88 0.034 11.88 0.284 11.38 0.231 10.71 0.167 10.01 0.059 9.92 0.049 9.88 0.038 11.71 0.255 11.36 0.215 10.81 0.169 9.98 0.062 9.86 0.049 9.85 0.038

Table A3. Weight-only refinement on non-Instruct Llama-3.1-8B: WikiText-2 PPL, 2K windows, 5% slowdown budget on RTX 5090. Weights

Base

DecDEC

StrataFlux-W

BF16 LNQ+GQ 3-bit LNQ+GQ 4-bit YAQA 2-bit YAQA 3-bit YAQA 4-bit

6.24 7.49 6.53 8.12 6.62 6.35

– 7.37 6.50 7.82 6.56 6.32

– 7.21 6.47 7.38 6.51 6.31

Table A3 reports the non-Instruct Llama comparison. Table A4. Weight-only refinement on Qwen3-8B: MATH500 and AIME 2024/2025 (avg@8) accuracy at 5% slowdown budgets; StrataFlux-W columns use the configuration tuned on each platform. StrataFlux-W StrataFlux-W Base DecDEC DGX Spark RTX 5090 Weights M500 AIME M500 AIME M500 AIME M500 AIME BF16 97.2 71.0 – – – – – – YAQA 2-bit 77.2 19.2 86.2 30.8 89.4 40.0 92.4 46.7 YAQA 3-bit 95.0 66.7 95.4 66.9 95.0 66.0 95.2 70.0 YAQA 4-bit 95.8 71.5 95.4 67.7 95.6 70.8 97.0 69.8 LNQ+GQ 3-bit 91.8 39.4 93.0 44.4 93.0 47.3 93.2 49.2 LNQ+GQ 4-bit 96.2 64.4 96.4 66.7 95.8 66.0 94.4 67.5 AWQ 3-bit 91.8 46.5 92.8 50.8 92.6 51.9 93.8 51.0 AWQ 4-bit 95.8 64.8 95.2 66.9 96.4 68.5 95.8 65.0 SqLLM 3-bit 93.0 47.3 92.8 49.8 94.2 48.3 93.0 50.8 SqLLM 4-bit 95.2 64.2 96.0 67.1 95.6 66.5 95.2 70.2

B.2

Long-Context Weight-Only Refinement

Table A5 evaluates weight refinement at 128K context with FP16 KV. Table A5. Weight-only refinement: PG-19 PPL/KL and RTX 5090 tok/s at 128K with FP16 KV, on Llama-3.1-8B-Instruct and Qwen38B. StrataFlux-W reuses its empty-context 5% configurations. Weights BF16 YAQA 2-bit YAQA 3-bit YAQA 4-bit LNQ+GQ 3-bit LNQ+GQ 4-bit

Method Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W Base StrataFlux-W

PPL 10.32 15.89 13.73 10.87 10.71 10.51 10.47 13.65 12.96 10.77 10.69

L3.1-8B-I KL tok/s 0.000 47.6 0.473 64.2 0.311 57.3 0.057 62.5 0.042 56.0 0.016 61.4 0.013 60.0 0.303 68.1 0.243 67.3 0.043 65.7 0.037 65.2

PPL 12.89 16.55 14.83 13.38 13.16 13.02 12.95 18.15 17.21 13.28 13.22

Qwen3-8B KL tok/s 0.000 OOM 0.254 58.2 0.162 50.2 0.045 57.0 0.034 47.2 0.014 55.4 0.010 48.6 0.328 61.2 0.283 60.6 0.037 59.1 0.032 58.1

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

B.3

DGX Spark

Larger Models

Table A6 compares weight refinement on Llama-3.3-70BInstruct and Qwen3-32B.

Weights

LNQ+GQ 3-bit

Table A6. Weight-only refinement: WikiText-2 perplexity and KL divergence on Llama-3.3-70B-Instruct and Qwen3-32B at matched 5% slowdown budgets. Throughput uses empty context: Qwen332B on RTX 5090; ∗ Llama-3.3-70B-Instruct on RTX PRO 6000. Weights BF16 YAQA 2-bit

YAQA 3-bit

YAQA 4-bit

LNQ+GQ 3-bit

LNQ+GQ 4-bit

AWQ 3-bit

AWQ 4-bit

SqLLM 3-bit

SqLLM 4-bit

Method Base Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W Base DecDEC StrataFlux-W

PPL 3.86 6.19 5.71 5.52 4.50 4.36 4.33 4.06 4.00 3.99 5.39 5.15 5.05 4.26 4.20 4.17 5.75 5.09 5.04 4.40 4.16 4.16 5.64 5.10 5.07 4.28 4.17 4.17

L3.3-70B-I KL tok/s∗ 0.000 OOM 0.449 43.5 0.373 42.8 0.349 41.9 0.141 36.9 0.111 35.6 0.107 35.3 0.046 32.7 0.035 31.3 0.033 31.4 0.320 46.6 0.282 43.5 0.261 44.9 0.096 36.5 0.082 34.0 0.078 35.1 0.371 35.3 0.260 34.0 0.244 34.1 0.110 37.8 0.069 36.0 0.066 36.1 0.377 46.3 0.272 44.2 0.253 44.5 0.105 36.1 0.079 34.1 0.074 34.9

PPL 7.61 8.30 8.04 8.04 7.70 7.68 7.69 7.62 7.61 7.62 8.15 8.10 7.94 7.72 7.65 7.65 8.56 8.30 8.01 7.84 7.76 7.71 8.68 8.52 8.09 7.82 7.71 7.69

Qwen3-32B KL tok/s 0.000 OOM 0.211 76.2 0.164 71.8 0.153 72.8 0.064 67.7 0.049 62.9 0.043 62.5 0.023 62.9 0.016 58.7 0.015 59.6 0.173 98.0 0.153 92.9 0.132 92.6 0.053 78.2 0.044 74.4 0.039 73.4 0.241 83.8 0.187 81.2 0.146 80.8 0.070 77.6 0.052 73.8 0.043 73.3 0.242 98.1 0.211 94.6 0.148 94.3 0.071 78.1 0.049 74.5 0.043 73.8

AWQ 3-bit

SqLLM 3-bit

YAQA 4-bit

LNQ+GQ 4-bit

AWQ 4-bit

SqLLM 4-bit

Budget / method Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W

PPL 8.52 8.34 8.29 8.31 8.31 9.10 8.75 8.64 8.63 8.64 9.36 8.72 8.56 8.51 8.51 7.33 7.31 7.31 7.31 7.31 7.48 7.45 7.43 7.44 7.44 7.59 7.52 7.51 7.51 7.51 7.56 7.50 7.48 7.47 7.47

L3.1-8B-I KL tok/s 0.157 60.9 0.139 61.4 0.135 59.8 0.135 57.1 0.135 52.8 0.229 63.3 0.190 61.1 0.179 59.2 0.175 56.2 0.177 52.9 0.263 60.8 0.194 60.5 0.174 59.3 0.170 57.4 0.171 53.1 0.014 48.1 0.012 47.5 0.012 46.6 0.012 44.4 0.012 40.9 0.034 49.4 0.031 49.9 0.031 49.0 0.031 46.7 0.031 42.3 0.049 49.5 0.040 48.8 0.039 46.1 0.038 45.4 0.038 42.4 0.052 49.5 0.043 48.1 0.040 48.6 0.039 46.6 0.039 44.0

Qwen3-8B PPL KL 10.92 0.184 10.73 0.160 10.70 0.159 10.64 0.149 10.66 0.154 11.88 0.284 11.33 0.234 11.30 0.222 11.16 0.214 11.12 0.211 11.71 0.255 11.19 0.210 11.03 0.201 11.14 0.187 11.11 0.186 9.79 0.016 9.73 0.014 9.72 0.014 9.71 0.014 9.72 0.014 9.95 0.044 9.88 0.038 9.87 0.038 9.88 0.039 9.86 0.038 10.01 0.059 9.96 0.047 9.96 0.047 9.90 0.046 9.90 0.045 9.98 0.062 9.89 0.050 9.97 0.048 9.87 0.046 9.88 0.046

Table A8. Weight-only refinement on RTX 5090: 2–4-bit bases across slowdown budgets. Columns report WikiText-2 PPL/KL and empty-context decode tok/s. RTX 5090

B.4

Slowdown-Budget Sweeps

Tables A7, A8, and A9 compare weight-refinement accuracy and throughput across slowdown budgets on DGX Spark, RTX 5090, and RTX 5070 Ti. Table A7. Weight-only refinement on DGX Spark: 2–4-bit bases across slowdown budgets. Columns report WikiText-2 PPL/KL and Llama-3.1-8B-Instruct empty-context tok/s.

Weights BF16

YAQA 2-bit

DGX Spark Weights BF16

YAQA 2-bit

YAQA 3-bit

Budget / method Base Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W Base 2.5% / StrataFlux-W 5% / StrataFlux-W 10% / StrataFlux-W 20% / StrataFlux-W

PPL 7.22 9.46 8.96 8.94 8.90 8.92 7.61 7.56 7.55 7.56 7.56

L3.1-8B-I KL 0.000 0.257 0.202 0.200 0.194 0.198 0.047 0.042 0.041 0.041 0.041

tok/s 15.6 72.0 70.9 69.2 65.7 60.0 57.7 57.4 55.8 53.1 49.0

Qwen3-8B PPL KL 9.73 0.000 10.95 0.214 10.81 0.178 10.77 0.175 10.67 0.174 10.69 0.174 9.98 0.052 9.91 0.044 9.89 0.043 9.89 0.045 9.88 0.044

YAQA 3-bit

Budget / method Base Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC 5% / StrataFlux-W 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC 5% / StrataFlux-W 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W

PPL 7.22 9.46 8.90 8.49 9.03 8.40 8.87 8.39 8.78 8.37 7.61 7.57 7.50 7.55 7.49 7.54 7.48 7.52 7.47

L3.1-8B-I KL tok/s 0.000 104.7 0.257 245.8 0.197 239.4 0.149 240.3 0.211 233.2 0.139 234.4 0.193 228.9 0.137 223.9 0.183 218.3 0.134 211.6 0.047 222.5 0.041 215.1 0.034 217.5 0.040 213.3 0.033 212.7 0.038 209.3 0.032 202.6 0.037 197.5 0.030 184.8

Qwen3-8B PPL KL tok/s 9.73 0.000 103.0 10.95 0.214 232.3 10.57 0.172 223.1 10.66 0.144 227.7 10.69 0.184 220.8 10.51 0.132 224.4 10.53 0.170 217.0 10.43 0.123 209.6 10.46 0.163 205.5 10.37 0.120 192.9 9.98 0.052 213.1 9.93 0.046 206.1 9.89 0.036 208.3 9.89 0.044 202.0 9.89 0.033 202.8 9.88 0.042 198.8 9.87 0.032 194.2 9.87 0.041 191.7 9.86 0.031 171.5

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference RTX 5090 Weights

Budget / method Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC LNQ+GQ 3-bit 5% / StrataFlux-W 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC 5% / StrataFlux-W AWQ 3-bit 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC 5% / StrataFlux-W SqLLM 3-bit 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC 5% / StrataFlux-W YAQA 4-bit 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC LNQ+GQ 4-bit 5% / StrataFlux-W 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC 5% / StrataFlux-W AWQ 4-bit 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W Base 2.5% / DecDEC 2.5% / StrataFlux-W 5% / DecDEC SqLLM 4-bit 5% / StrataFlux-W 10% / DecDEC 10% / StrataFlux-W 20% / DecDEC 20% / StrataFlux-W

L3.1-8B-I PPL KL tok/s 8.52 0.157 314.3 8.40 0.146 312.4 8.20 0.122 306.0 8.37 0.142 311.5 8.18 0.121 299.7 8.32 0.138 302.8 8.11 0.112 285.8 8.39 0.147 287.4 8.07 0.105 261.6 9.10 0.229 304.5 8.88 0.204 294.4 8.60 0.170 295.0 8.80 0.195 292.6 8.38 0.145 289.0 8.76 0.189 289.5 8.31 0.140 270.9 8.56 0.166 270.2 8.19 0.120 255.7 9.36 0.263 315.4 9.10 0.233 312.2 8.47 0.162 307.3 9.02 0.225 311.3 8.42 0.156 299.6 8.91 0.213 305.2 8.33 0.143 288.6 8.86 0.209 289.1 8.13 0.121 263.6 7.33 0.014 204.2 7.30 0.012 197.6 7.29 0.010 199.1 7.30 0.011 195.9 7.29 0.010 194.3 7.30 0.011 190.6 7.29 0.010 185.5 7.29 0.010 180.5 7.29 0.009 169.5 7.48 0.034 264.7 7.43 0.031 263.8 7.42 0.028 258.9 7.42 0.031 263.1 7.41 0.027 254.5 7.45 0.032 249.9 7.40 0.027 241.9 7.45 0.032 238.8 7.39 0.025 218.9 7.59 0.049 262.8 7.53 0.043 259.5 7.45 0.033 257.4 7.52 0.042 257.7 7.44 0.031 250.4 7.51 0.041 254.1 7.43 0.029 237.0 7.47 0.036 236.2 7.40 0.028 218.4 7.56 0.052 265.6 7.51 0.044 262.9 7.46 0.037 257.9 7.51 0.044 262.2 7.44 0.035 254.6 7.52 0.044 254.6 7.42 0.032 241.9 7.52 0.044 238.8 7.41 0.029 216.4

Qwen3-8B PPL KL tok/s 10.92 0.184 290.9 10.78 0.187 290.5 10.51 0.144 284.5 10.66 0.170 287.1 10.47 0.139 281.1 10.55 0.159 283.0 10.54 0.130 263.7 10.85 0.181 265.6 10.42 0.117 240.4 11.88 0.284 286.0 11.63 0.265 281.5 11.24 0.208 273.9 11.38 0.231 276.1 10.71 0.167 271.5 11.24 0.220 273.2 10.49 0.143 259.0 11.21 0.218 253.8 10.50 0.144 238.3 11.71 0.255 292.1 11.42 0.234 291.2 10.85 0.174 267.1 11.36 0.215 287.3 10.81 0.169 280.3 11.20 0.201 283.8 10.62 0.145 264.0 11.41 0.216 266.2 10.53 0.132 244.9 9.79 0.016 193.8 9.76 0.015 185.9 9.77 0.011 189.0 9.75 0.013 183.3 9.75 0.010 184.1 9.74 0.013 180.6 9.75 0.010 176.0 9.74 0.012 172.5 9.74 0.010 158.4 9.95 0.044 248.3 9.89 0.038 247.6 9.88 0.036 242.5 9.89 0.038 244.9 9.88 0.034 238.0 9.89 0.041 236.7 9.89 0.033 225.9 9.96 0.043 224.2 9.89 0.031 193.4 10.01 0.059 247.5 9.95 0.055 245.2 9.88 0.040 241.4 9.92 0.049 242.2 9.88 0.038 236.9 9.90 0.047 238.0 9.81 0.034 223.0 9.89 0.045 226.1 9.79 0.031 206.0 9.98 0.062 249.0 9.87 0.050 246.4 9.89 0.042 239.3 9.86 0.049 245.0 9.85 0.038 235.7 9.87 0.050 234.8 9.81 0.038 226.3 9.88 0.051 222.2 9.81 0.033 205.8

Table A9. Weight-only refinement on RTX 5070 Ti: 2–4-bit bases across slowdown budgets. Columns report WikiText-2 PPL/KL and empty-context decode tok/s. RTX 5070 Ti L3.1-8B-I Qwen3-8B Budget / method PPL KL tok/s PPL KL tok/s Base 9.46 0.257 165.9 10.95 0.214 158.7 2.5% / DecDEC 9.06 0.215 155.8 10.58 0.175 146.4 2.5% / StrataFlux-W 8.54 0.153 158.9 10.54 0.136 149.8 5% / DecDEC 8.97 0.205 152.8 10.56 0.173 144.6 5% / StrataFlux-W 8.49 0.147 154.2 10.45 0.129 149.4 YAQA 2-bit 10% / DecDEC 8.89 0.196 149.5 10.54 0.171 141.0 10% / StrataFlux-W 8.37 0.135 148.9 10.36 0.120 143.5 20% / DecDEC 8.61 0.164 139.1 10.37 0.151 134.1 20% / StrataFlux-W 8.30 0.127 135.5 10.34 0.115 130.8 Base 7.61 0.047 149.4 9.98 0.052 145.1 2.5% / DecDEC 7.57 0.041 141.7 9.90 0.044 136.9 2.5% / StrataFlux-W 7.51 0.034 144.5 9.90 0.035 138.6 5% / DecDEC 7.56 0.040 138.4 9.89 0.043 133.8 5% / StrataFlux-W 7.49 0.032 141.0 9.88 0.034 137.6 YAQA 3-bit 10% / DecDEC 7.51 0.035 133.8 9.88 0.042 129.1 10% / StrataFlux-W 7.47 0.030 135.5 9.87 0.031 130.0 20% / DecDEC 7.48 0.032 125.0 9.83 0.037 120.5 20% / StrataFlux-W 7.45 0.028 124.2 9.85 0.030 120.0 Base 8.52 0.157 183.1 10.92 0.184 172.5 2.5% / DecDEC 8.23 0.126 179.4 10.42 0.145 169.4 2.5% / StrataFlux-W 8.22 0.123 176.3 10.48 0.123 167.0 5% / DecDEC 8.33 0.141 176.1 10.69 0.167 165.9 8.03 0.103 170.7 10.51 0.126 163.3 LNQ+GQ 3-bit 5% / StrataFlux-W 10% / DecDEC 8.33 0.141 170.1 10.70 0.171 160.8 10% / StrataFlux-W 8.06 0.103 165.2 10.42 0.117 157.1 20% / DecDEC 8.27 0.133 161.0 10.54 0.152 152.4 20% / StrataFlux-W 8.00 0.098 147.8 10.39 0.109 139.6 Base 9.10 0.229 171.0 11.88 0.284 163.2 2.5% / DecDEC 8.86 0.202 167.2 11.53 0.243 156.9 2.5% / StrataFlux-W 8.52 0.163 166.4 11.14 0.204 156.5 5% / DecDEC 8.60 0.172 162.9 11.07 0.207 154.5 5% / StrataFlux-W 8.35 0.142 161.4 10.69 0.153 154.0 AWQ 3-bit 10% / DecDEC 8.53 0.162 156.5 11.04 0.203 148.6 10% / StrataFlux-W 8.17 0.117 154.2 10.40 0.128 147.4 20% / DecDEC 8.40 0.146 147.7 10.71 0.170 140.4 20% / StrataFlux-W 8.04 0.102 141.7 10.33 0.114 134.0 Base 9.36 0.263 183.1 11.71 0.255 172.6 2.5% / DecDEC 9.09 0.234 179.2 10.90 0.177 169.4 2.5% / StrataFlux-W 8.44 0.158 173.0 11.02 0.179 164.1 5% / DecDEC 8.64 0.183 175.8 11.21 0.192 166.0 5% / StrataFlux-W 8.17 0.129 166.7 10.60 0.138 163.6 SqLLM 3-bit 10% / DecDEC 8.71 0.191 170.3 11.27 0.199 160.2 10% / StrataFlux-W 8.08 0.114 166.3 10.50 0.125 153.0 20% / DecDEC 8.32 0.145 160.9 10.71 0.150 152.5 20% / StrataFlux-W 7.97 0.102 149.0 10.38 0.114 142.9 Base 7.33 0.014 132.2 9.79 0.016 128.0 2.5% / DecDEC 7.30 0.012 126.8 9.75 0.014 120.8 2.5% / StrataFlux-W 7.28 0.009 127.7 9.76 0.011 123.1 5% / DecDEC 7.29 0.010 123.5 9.74 0.013 118.1 YAQA 4-bit 5% / StrataFlux-W 7.29 0.009 124.2 9.75 0.010 120.6 10% / DecDEC 7.28 0.009 118.2 9.73 0.011 113.6 10% / StrataFlux-W 7.28 0.008 119.2 9.75 0.010 115.8 20% / DecDEC 7.28 0.009 110.6 9.73 0.011 106.4 20% / StrataFlux-W 7.28 0.008 109.2 9.75 0.010 106.2 Weights

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference 8.6

PPL

8.4

L3.1-8B-I, RTX 5090 −0.02

−0.12 −0.13

8.2

10.6

YAQA 2-bit

AWQ 3-bit

LNQ+GQ 3-bit

10.2

L3.1-8B-I, RTX 5070 Ti

PPL

8.6

−0.02

−0.12

+0.06

−0.13

8.2

YAQA 2-bit

+0.01

AWQ 3-bit

LNQ+GQ 3-bit

Table A10. Weight-only refinement after YAQA recovery finetuning: WikiText-2 PPL/KL using configurations tuned on RTX 5090. Original and fine-tuned bases are shown for each bitwidth; both DecDEC and StrataFlux-W refine the fine-tuned weights. L3.1-8B-I

Qwen3-8B, RTX 5070 Ti −0.10

10.8

8.4

10.6

Weights

Method

−0.07

YAQA 2-bit

AWQ 3-bit

LNQ+GQ 3-bit

10.2

+0.02

YAQA 2-bit

AWQ 3-bit

LNQ+GQ 3-bit

YAQA 2-bit plain residual values

+ fitted refinement values

Figure A1. Fitted versus unfitted weight refinement on WikiText-2, at the 5% configurations tuned on RTX 5090 and RTX 5070 Ti. Each pair fixes selection counts, schedule, and MLP scheme: gray bars use plain residuals; red bars use fitted refinement. Labels give the PPL change from fitting. RTX 5070 Ti L3.1-8B-I Qwen3-8B Budget / method PPL KL tok/s PPL KL tok/s Base 7.48 0.034 148.3 9.95 0.044 141.3 2.5% / DecDEC 7.43 0.030 144.3 9.89 0.041 139.0 2.5% / StrataFlux-W 7.38 0.024 142.1 9.90 0.032 137.8 5% / DecDEC 7.42 0.029 142.2 9.85 0.039 136.5 7.38 0.024 139.0 9.88 0.031 133.8 LNQ+GQ 4-bit 5% / StrataFlux-W 10% / DecDEC 7.41 0.028 137.5 9.83 0.037 132.4 10% / StrataFlux-W 7.38 0.023 133.4 9.88 0.030 129.5 20% / DecDEC 7.40 0.026 129.0 9.83 0.035 124.5 20% / StrataFlux-W 7.37 0.022 121.7 9.87 0.029 118.7 Base 7.59 0.049 152.4 10.01 0.059 144.8 2.5% / DecDEC 7.47 0.036 148.4 9.88 0.043 140.8 2.5% / StrataFlux-W 7.41 0.029 147.2 9.86 0.035 139.7 5% / DecDEC 7.48 0.037 145.7 9.95 0.048 138.6 5% / StrataFlux-W 7.40 0.026 143.1 9.80 0.032 136.2 AWQ 4-bit 10% / DecDEC 7.44 0.031 140.8 9.83 0.037 133.6 10% / StrataFlux-W 7.38 0.024 136.5 9.77 0.029 130.6 20% / DecDEC 7.40 0.026 132.1 9.80 0.034 126.1 20% / StrataFlux-W 7.36 0.021 126.8 9.84 0.026 119.5 Base 7.56 0.052 148.3 9.98 0.062 141.4 2.5% / DecDEC 7.46 0.037 144.5 9.85 0.048 138.9 2.5% / StrataFlux-W 7.38 0.028 142.5 9.83 0.036 137.3 5% / DecDEC 7.44 0.035 142.2 9.79 0.041 136.4 5% / StrataFlux-W 7.38 0.026 139.4 9.78 0.032 134.3 SqLLM 4-bit 10% / DecDEC 7.41 0.031 137.5 9.77 0.038 132.3 10% / StrataFlux-W 7.37 0.025 134.0 9.79 0.030 129.8 20% / DecDEC 7.39 0.029 129.0 9.75 0.034 124.4 20% / StrataFlux-W 7.36 0.023 123.7 9.79 0.027 118.3

YAQA 3-bit

YAQA Recovery Fine-Tuning

StrataFlux-W is compatible with YAQA recovery finetuning. We use QTIP's RedPajama fine-tuning recipe, but freeze embeddings, final normalization, and the output head during end-to-end fine-tuning; Qwen3-8B’s query/key norms remain frozen during both block-wise and end-toend fine-tuning. This preserves the BF16 refinement target; normalization changes elsewhere are absorbed by rescaling its projection weights. We find that this preserves the gains of fine-tuning while maintaining compatibility with weight refinement (Table A10).

B.6

KL

PPL

KL

9.46 8.45

0.257 0.161

10.95 10.24

0.214 0.162

2.5%

8.32 8.24

0.142 0.133

10.22 10.50

0.146 0.137

DecDEC StrataFlux-W

5%

8.35 8.19

0.149 0.128

10.26 10.40

0.153 0.125

DecDEC StrataFlux-W

10%

8.32 8.18

0.141 0.126

10.21 10.37

0.146 0.120

DecDEC StrataFlux-W

20%

8.28 8.22

0.137 0.126

10.14 10.36

0.140 0.117

7.61 7.50

0.047 0.040

9.98 9.73

0.052 0.046

DecDEC StrataFlux-W

2.5%

7.48 7.48

0.036 0.032

9.75 9.78

0.043 0.035

DecDEC StrataFlux-W

5%

7.48 7.47

0.035 0.032

9.75 9.79

0.041 0.032

DecDEC StrataFlux-W

10%

7.48 7.47

0.035 0.031

9.74 9.77

0.040 0.032

DecDEC StrataFlux-W

20%

7.47 7.45

0.033 0.029

9.73 9.76

0.039 0.031

7.33 7.28

0.014 0.013

9.79 9.71

0.016 0.015

Original base Fine-tuned base

YAQA 4-bit

Qwen3-8B

PPL

DecDEC StrataFlux-W

Original base Fine-tuned base

Weights

B.5

Budget

Original base Fine-tuned base

10.4

8.0 7.8

−0.06

10.4

8.0 7.8

Qwen3-8B, RTX 5090 10.8

DecDEC StrataFlux-W

2.5%

7.27 7.27

0.011 0.010

9.70 9.73

0.014 0.011

DecDEC StrataFlux-W

5%

7.27 7.28

0.011 0.009

9.69 9.74

0.013 0.010

DecDEC StrataFlux-W

10%

7.27 7.27

0.011 0.009

9.69 9.74

0.013 0.010

DecDEC StrataFlux-W

20%

7.26 7.27

0.010 0.009

9.69 9.74

0.012 0.010

Refinement and Cache Ablations

Figure A1 compares fitted and unfitted weight refinement at fixed configurations. Figure A2 compares skipping and waiting on weight-cache misses across slowdown budgets. Tables A11 and A12 give the cache measurements and paired technique ablations. Table A11. Weight-cache capacity and miss policy on DGX Spark: Llama-3.1-8B-Instruct, 5% configuration, YAQA 2-bit and AWQ 3-bit. WikiText-2 PPL/KL and decode tok/s; capacities in MiB. Raw values behind Figure 12(d,e). YAQA 2-bit

AWQ 3-bit

MiB

Miss policy

PPL

KL

tok/s

PPL

KL

tok/s

64 128 256 512 128

skip skip skip skip wait

9.01 8.94 8.88 8.80 8.52

0.207 0.200 0.192 0.183 0.152

68.8 69.2 69.7 69.4 17.9

8.70 8.64 8.59 8.53 8.31

0.186 0.179 0.172 0.166 0.137

57.4 57.6 57.6 57.4 27.7

WikiText-2 PPL

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference 9.4

increment of k for channel-wise gate/up and down refinement. It also considers disabling refinement for projection sets when needed to meet the budget. On DGX Spark, we search skip-policy configurations only at s = 0: skipping cache misses leaves no SSD wait for fetch lead/lag to hide.

Base

9.2

2.5% 40%

9.0

40%

8.8

5%

30%

2.5%

10%

20%

30%

10%

5%

20%

50

55

60

65

70

Decode throughput (tok/s) skip misses

wait on misses

Figure A2. Waiting versus skipping weight-cache misses on Llama3.1-8B-Instruct with YAQA 2-bit weights, DGX Spark, and a 128 MiB cache. Curves plot WikiText-2 PPL against throughput. Points follow slowdown budgets of 2.5%, 5%, 10%, 20%, 30%, and 40%. Measured wait-policy throughput is non-monotonic across budgets. Table A12. Paired weight-refinement ablations on Llama-3.1-8BInstruct, RTX 5090: WikiText-2 PPL/KL and empty-context decode tok/s. Pairs are tuned separately at a 5% slowdown budget, except in the unchanged-settings pair: there, s = 0 disables lead/lag in the final configuration while retaining its k, fitted refinements, MLP scheme, and GPU thread-block counts. YAQA 2-bit

AWQ 3-bit

Comparison

Setting

PPL

KL

tok/s

PPL

KL

tok/s

Fitting, s = 0

Unfitted Fitted

8.76 8.67

0.181 0.170

234.1 230.1

8.70 8.61

0.182 0.171

288.1 287.5

Fitting, tuned s

Unfitted Fitted

8.68 8.59

0.174 0.162

234.9 234.3

8.50 8.38

0.158 0.145

290.2 289.0

MLP scheme

Channel-wise Neuron bundles

8.59 8.40

0.162 0.139

234.3 234.4

8.38 8.56

0.145 0.165

289.0 289.3

Lead/lag, unfitted

s=0 Tuned s

8.76 8.68

0.181 0.174

234.1 234.9

8.70 8.50

0.182 0.158

288.1 290.2

Lead/lag, full method

s=0 Tuned s

8.57 8.40

0.157 0.139

232.2 234.4

8.57 8.38

0.168 0.145

286.8 289.0

Lead/lag, unchanged settings

s=0 Tuned s

8.34 8.40

0.131 0.139

182.2 234.4

8.33 8.38

0.141 0.145

271.1 289.0

B.7

Weight Refinement Tuner

The tuner searches separately for each model, weight quantizer, and platform under a specified slowdown budget relative to the quantized base. It selects channel budgets k, fetch lead/lag distances s, the MLP refinement scheme, and GPU thread-block counts. Resource allocation. We first profile base projections alongside refinement while varying its thread-block count and selection budget. The resulting timing table identifies launch configurations that balance refinement speed against interference with the base projections. Budget-constrained search. For each combination of shift distances and channel-wise or neuron-wise MLP refinement, the tuner searches for the largest common channel budget k that satisfies the slowdown limit in end-to-end decoding. For neuron-wise MLP refinement, it sets kn = round(3k/4): each increment of kn fetches 4/3 as many bytes as each

Accuracy selection and budget allocation. The tuner compares feasible candidates by KL divergence from BF16 on held-out text. Starting from the lowest-KL candidate, it increases individual selection budgets while remeasuring end-to-end latency and retaining increases that satisfy the slowdown limit. These final per-projection budgets need not be equal. Refinement fitting during search. Fitted refinement values depend on the selection budget (Section 3.2.1). We approximate candidate KL during search using refinements pre-fitted at a ladder of power-of-two selection budgets, avoiding a separate fit for each candidate. Once the final per-projection budgets are selected, we fit refinement values at those exact budgets before quantizing them for inference. Tuned configurations. Table A13 lists tuned configurations across slowdown budgets and miss-policy sweeps. Each row displays one configuration. The Weights column gives the quantizer and bitwidth. Four-entry tuples give the query/key/value, attention-output, gate/up, and down configurations for channel-wise refinement. Three-entry tuples give query/key/value, attention-output, and neuron bundles. This order applies to selection counts (k, or kn for neuron bundles) and shift distances s. ntb lists the GPU threadblock count shared by channel-wise kernels, followed by the count for the neuron-bundle kernel when used. Counts k are per 1,024 input channels or neurons, and s is in Transformer blocks. SSD-backed configurations use a 128 MiB cache; miss-policy sweeps identify skipping or waiting. Table A13. Tuned weight-refinement configurations. Model and platform headings identify each configuration block. The main DGX Spark skip-policy search uses only s = 0; skip/wait policy sweeps are listed separately. Llama-3.1-8B-Instruct / DGX Spark Weights Budget k 2.5% 3,3,2 5% 10,14,8 AWQ 3 10% 21,21,21,21 20% 31,31,23 2.5% 5,6,0,5 5% 16,16,14,16 AWQ 4 10% 28,28,0,28 20% 34,33,33,33 2.5% 0,4,3 5% 11,0,8 LNQ+GQ 3 10% 19,19,14 20% 33,32,24

s

ntb

0,0,0 0,0,0 0,0,0,0 0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0 0,0,0 0,0,0 0,0,0

7,4 2,4 3 3,8 3 4 3 6 7,4 6,4 6,8 5,16

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Llama-3.1-8B-Instruct / DGX Spark Weights Budget k LNQ+GQ 4

SqLLM 3

SqLLM 4

YAQA 2

YAQA 3

YAQA 4

2.5% 5% 10% 20% 2.5% 5% 10% 20% 2.5% 5% 10% 20% 2.5% 5% 10% 20% 2.5% 5% 10% 20% 2.5% 5% 10% 20%

8,8,0,8 14,0,9 23,24,16 40,40,29 9,8,6 10,14,9,9 18,18,19,18 28,29,28,27 0,15,15,15 10,9,10,10 21,21,21,21 33,33,33,33 8,0,5 14,12,8 0,26,18 32,33,22 5,5,4 16,16,12 28,28,21 35,35,26 0,6,4 18,18,12 0,32,24 42,43,30

s

ntb

0,0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0

7 5,4 7,8 8,8 6,4 3 4 3 4 6 5 6 1,4 2,4 2,8 3,8 1,8 2,4 3,8 3,4 1,4 2,4 2,8 3,8

Llama-3.1-8B-Instruct / DGX Spark, skip-policy sweep Weights Budget k s 2.5% 6,12,4 0,0,0 5% 0,17,12 0,0,0 10% 24,23,18 0,0,0 YAQA 2 20% 0,37,28 0,0,0 30% 0,40,30 0,0,0 40% 0,46,32 0,0,0

3,4 5,4 2,8 4,8 13,4 13,4

Llama-3.1-8B-Instruct / DGX Spark, wait-policy sweep Weights Budget k s 2.5% 0,3,1 0,1.5,1.5 5% 0,3,1 0,1.5,1.5 10% 4,4,3 2.5,2,2 YAQA 2 20% 10,10,8 1.5,1.5,1.5 30% 40,0,9 2,0,1.5 40% 16,0,11 1.5,0,1.5

2,4 4,4 2,4 3,4 9,4 14,4

Llama-3.1-8B-Instruct / RTX 5070 Ti Weights Budget k 2.5% 3,10,2 5% 18,0,18,18 AWQ 3 10% 37,37,37,37 20% 45,40,39,40 2.5% 28,28,34,36 5% 52,53,49,52 AWQ 4 10% 45,45,45,45 20% 53,55,52,52 2.5% 6,8,3 5% 63,63,33,61 LNQ+GQ 3 10% 29,28,20 20% 60,61,58,60 2.5% 45,42,47,45 5% 48,45,48,49 LNQ+GQ 4 10% 68,59,58,59 20% 64,64,64,63

s

ntb

0,0.5,0 1,0,0,0 1,0,0,0.5 0,0,0,0 0.5,1,0,0.5 1,0.5,0,0.5 0,0.5,0,0 0,0,0,0 0,1,0.5 1,0.5,0,0.5 0.5,0,0.5 1,0.5,0,0.5 1,0.5,0,0 1,0,0,0 1,0,0,0.5 1,0,0,0

3,4 4 6 7 6 24 24 24 4,4 16 16,16 16 16 16 16 24

ntb

ntb

Llama-3.1-8B-Instruct / RTX 5070 Ti Weights Budget k 2.5% 10,10,10,10 5% 32,37,31,30 SqLLM 3 10% 27,27,26,27 20% 43,43,43,43 2.5% 85,46,46,45 5% 48,48,48,48 SqLLM 4 10% 54,53,53,54 20% 59,60,57,58 2.5% 15,23,12 5% 28,28,16 YAQA 2 10% 37,39,25 20% 97,49,35 2.5% 32,32,31,32 5% 66,50,44,44 YAQA 3 10% 62,48,47,48 20% 61,61,60,61 2.5% 59,60,58,58 5% 82,60,58,60 YAQA 4 10% 73,77,70,70 20% 87,87,86,87 Llama-3.1-8B-Instruct / RTX 5090 Weights Budget k 2.5% 8,0,0,8 5% 17,0,17,13 AWQ 3 10% 13,13,13,13 20% 29,28,22,27 2.5% 10,0,10,10 5% 20,0,18,13 AWQ 4 10% 30,39,26,28 20% 53,54,46,46 2.5% 18,0,13,9 5% 17,0,13,13 LNQ+GQ 3 10% 59,65,5 20% 29,29,16 2.5% 25,0,15,11 5% 21,0,21,18 LNQ+GQ 4 10% 23,25,21,22 20% 40,32,31,32 2.5% 17,0,9,9 5% 17,0,13,13 SqLLM 3 10% 10,10,10,10 20% 26,26,20,22 2.5% 16,0,16,9 5% 22,0,21,21 SqLLM 4 10% 19,19,19,19 20% 29,29,28,29 2.5% 16,28,12 5% 39,40,18 YAQA 2 10% 39,39,20 20% 50,51,24 2.5% 29,28,26,26 5% 45,46,20 YAQA 3 10% 51,51,24 20% 53,53,49,49 2.5% 42,42,34,34 5% 38,39,38,38 YAQA 4 10% 50,50,50,50 20% 58,60,57,57

s

ntb

0.5,0.5,0,0 1,0.5,0,0 0,0,0,0 1,0,0,0 1,0.5,0,0 1,0,0,0 1,0,0,0 0,0,0,0 1,1,0.5 1,1,0.5 1,0,0.5 0.5,0,0.5 0.5,0.5,0,0 0,0.5,0,0.5 0,0,0,0 0,0,0,0 1,0,0,0.5 0,1,0,0.5 1,0,0,0.5 0,0,0,0.5

4 16 16 16 16 16 16 24 6,4 6,4 7,16 14,32 5 7 14 16 7 7 23 28

s

ntb

0.5,0,0,0 1,0,0,0 0,1,0,0 1,0,0,0 0.5,0,0,0 0.5,0,0,0 0.5,0.5,0,0.5 0.5,1,1,0.5 0.5,0,0,0 0.5,0,0,0 1,1,0.5 1,0,0.5 0.5,0,0,0 0.5,0,0,0 1,1,0,0 1,0,0,0 0.5,0,0,0 0.5,0,0,0 0.5,0,0,0 1,0,0,0 1,0,0,0 0.5,0,0,0 0.5,0,0,0 0,0,0,0 0,0,0.5 0,0,0.5 0,0,0.5 0.5,0,0.5 1,0,0,0 0,0,0.5 0.5,0,0.5 0,0,0,0.5 0,0.5,0,0 0,0,0,0.5 0.5,0.5,0.5,0.5 0,0,0,0.5

8 16 14 14 16 16 24 24 16 16 24,4 24,16 11 14 16 16 16 16 16 16 11 24 24 38 23,8 23,16 19,32 38,16 56 56,16 38,32 38 56 28 38 38

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Qwen3-8B / DGX Spark Weights Budget k 2.5% 0,8,2,2 5% 0,16,16,16 AWQ 3 10% 25,21,20,21 20% 28,27,28,27 2.5% 10,10,0,10 5% 11,11,0,11 AWQ 4 10% 20,20,20,19 20% 37,37,37,37 2.5% 4,0,3 5% 13,10,7 LNQ+GQ 3 10% 0,24,18 20% 34,34,26 2.5% 0,13,10 5% 0,19,12 LNQ+GQ 4 10% 31,31,23 20% 44,44,32 2.5% 5,4,4,4 5% 8,8,8,8 SqLLM 3 10% 22,23,22,22 20% 31,30,30,32 2.5% 0,9,8,9 5% 11,12,11,11 SqLLM 4 10% 24,24,24,24 20% 38,38,38,38 2.5% 5,7,4 5% 11,13,8 YAQA 2 10% 27,29,19 20% 32,33,24 2.5% 0,9,7 5% 0,17,12 YAQA 3 10% 32,34,22 20% 38,40,27 2.5% 0,21,15 5% 14,14,11 YAQA 4 10% 30,29,21 20% 42,42,32 Qwen3-8B / RTX 5070 Ti Weights Budget k 2.5% 0,6,6,6 5% 22,21,14 AWQ 3 10% 44,44,44,44 20% 54,54,53,55 2.5% 19,19,19,19 5% 35,41,35,41 AWQ 4 10% 61,61,60,61 20% 63,63,62,62 2.5% 0,16,14 5% 15,17,7 LNQ+GQ 3 10% 32,39,22 20% 44,44,32 2.5% 27,28,21 5% 46,46,24 LNQ+GQ 4 10% 44,45,29 20% 54,57,40 2.5% 11,11,8 5% 36,40,24,37 SqLLM 3 10% 42,42,42,42 20% 51,51,50,51 2.5% 38,0,37,37 5% 45,42,45,47 SqLLM 4 10% 44,45,44,45 20% 62,62,61,61

s

ntb

0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0 0,0,0

3 5 3 5 10 11 11 7 7,4 3,4 3,8 5,8 5,4 5,8 5,8 7,8 5 3 4 9 7 5 6 6 2,4 5,4 2,8 3,8 5,4 6,4 4,8 2,8 7,4 3,4 3,8 2,8

s

ntb

0,0.5,0,0.5 1,1,0.5 1,1,0,0.5 1,0,0,0.5 0.5,1,0,0 0.5,1,0,0.5 1,0.5,0,0.5 0,0,1,0 0,1,0.5 0.5,1,0 1,0.5,0.5 0.5,0,0.5 0.5,1,0.5 0.5,0,0.5 0.5,0,0.5 0,0,0.5 0.5,1,0.5 0.5,0.5,0,0.5 1,0.5,1,0 1,0,1,0 1,0,0,0 1,0.5,0,0 0,0,0,0 1,0,0,0

3 4,4 6 6 6 8 24 24 5,16 5,32 16,16 16,32 11,16 16,16 16,16 16,32 5,16 5 16 16 16 16 16 16

Qwen3-8B / RTX 5070 Ti Weights Budget k 2.5% 21,28,14 5% 27,28,18 YAQA 2 10% 67,74,29 20% 52,52,36 2.5% 26,31,17 5% 32,37,24 YAQA 3 10% 72,50,34 20% 56,57,42 2.5% 38,39,22 5% 49,52,35 YAQA 4 10% 57,59,40 20% 67,67,48 Qwen3-8B / RTX 5090 Weights Budget 2.5% 5% AWQ 3 10% 20% 2.5% 5% AWQ 4 10% 20% 2.5% 5% LNQ+GQ 3 10% 20% 2.5% 5% LNQ+GQ 4 10% 20% 2.5% 5% SqLLM 3 10% 20% 2.5% 5% SqLLM 4 10% 20% 2.5% 5% YAQA 2 10% 20% 2.5% 5% YAQA 3 10% 20% 2.5% 5% YAQA 4 10% 20%

s

ntb

1,1,0.5 1,0,0.5 0.5,0.5,0.5 0,0,0.5 0.5,1,0.5 0.5,1,0.5 1,0,0.5 0,0,0.5 1,0,0.5 1,1,0.5 0.5,0,0.5 0,0,0.5

6,4 6,4 6,32 6,32 5,4 6,16 6,32 16,32 7,4 6,32 6,32 16,32

k

s

ntb

2,0,2 17,0,16,10 60,60,32,32 48,48,44,48 14,0,13,5 18,0,16,10 29,29,26,29 51,112,50,51 0,16,12,5 0,11,11,11 13,13,10 27,31,20 16,0,14,14 31,0,26,25 54,54,9 31,31,23 10,0,9,10 11,0,11,11 25,27,25,26 28,28,27,28 16,0,16,16 34,0,23,21 25,25,25,25 37,37,37,37 11,10,8 22,20,16 40,41,22 47,46,26 17,17,13 36,36,23 57,57,25 44,44,33 19,18,14 42,42,26 44,45,28 47,48,36

0,0,0.5 1,0,0,0 1,0.5,1,0.5 1,0.5,1,0.5 0.5,0,0,0 0.5,0,0,0 0.5,0.5,0,0.5 1,0.5,1,0.5 0,0.5,0,0 0,0.5,0,0 0.5,0.5,0.5 0.5,0,0.5 1,0,1,0 1,0,1,0 1,1,0.5 0,0,0.5 1,0,1,0 1,0,0,0 1,0.5,1,0.5 1,0,1,0 0.5,0,0,0 1,0,0,0 1,0.5,0,0.5 1,0,1,0 0,0,0.5 0,0,0.5 0,0,0.5 0,0,0.5 0,0,0.5 0,0,0.5 0.5,0,0.5 0.5,0,0.5 0,0,0.5 0,0,0.5 0,0,0.5 0.5,0,0.5

12,32 16 12 16 16 16 32 32 32 32 32,16 32,16 48 32 48,4 16,16 24 32 24 32 48 32 48 16 7,16 7,16 20,32 32,32 32,8 24,32 32,32 32,32 5,4 8,16 32,32 32,16

Llama-3.3-70B-Instruct / RTX PRO 6000 / 5% slowdown budget Weights k s ntb AWQ 3 25,25,25,25 0,0.5,0,0 8 AWQ 4 30,30,31,29 1,0,0,0 40 LNQ+GQ 3 13,13,10 0.5,0.5,0.5 17,32 LNQ+GQ 4 24,24,24,24 0,0.5,0,0 28 SqLLM 3 16,16,15,15 0,0,0,0 16 SqLLM 4 24,24,24,24 1,0,0,0 28 YAQA 2 32,32,24 0.5,0.5,1 38,8 YAQA 3 42,37,33,34 0,0,0,0.5 32 YAQA 4 42,45,38,42 0,0,0,0.5 32

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

L3.1-8B-I

80 75 70 0.2 GB

65

Qwen3-8B

Table A14. KV-only refinement: GPU-resident KV-cache size and PG-19 PPL/KL at 128K, with RULER accuracy at 8K–128K. ∗ GB counts only KV memory. Baseline sizes are for RTX 5090; StrataFlux-KV headings identify the platform. DGX Spark sizes include its RAM cache. KVQ and TQ denote KVQuant and TurboQuant.

1.3 GB

70

0.1 GB

60 1.4 GB

50 6

5 4 3 KV cache (GB) KVarN

Quantization

Offloading Ours

2 0

15 30 45 60 Throughput (tok/s)

2/2

4/2

KIVI

KVQuant

2

3

4

AQUA-KV

2

3

4

ShadowKV TailorKV StrataFlux-KV

ParisKV

2

4

TurboQuant

2.5

3.5

SpeCache

FreeKV 2/2

3/2

3/3

Figure A3. KV-only refinement: RULER accuracy at 128K (13 tasks, 96 samples each) versus GPU-resident KV-cache size and RTX 5090 throughput at the same context. StrataFlux-KV uses its dedicated-GPU configuration. Gray shading extends below the baseline frontiers; dashed lines mark BF16 accuracy. TailorKV's size is labeled at the KV-cache axis's right edge. Qwen3-32B / RTX 5090 / 5% slowdown budget Weights k s AWQ 3 18,20,18,18 0,0.5,0,0 AWQ 4 19,19,19,17 0,0.5,0,0 LNQ+GQ 3 16,18,17,18 0,0.5,0,0 LNQ+GQ 4 28,26,28,26 1,0,0,0 SqLLM 3 16,18,16,16 0,0.5,0,0 SqLLM 4 27,27,26,27 1,0,0,0 YAQA 2 41,41,15 0.5,0,0.5 YAQA 3 201,32,19 0.5,0,0.5 YAQA 4 27,28,20 0.5,0,0.5

C

ntb

11 12 14 16 14 16 50,20 50,20 25,20

E XTENDED KV E VALUATION

We extend Section 4.2 across KV precisions and context lengths. In configuration labels, K and V give resident bits per key and value element; +4/+4 gives their additional refinement bits. The fetch budget is the fraction of eligible token groups selected. Main host-RAM configurations use two-token groups. Main SSD-backed configurations use four-token groups and skip refinement on RAM-cache misses. C.1

Accuracy and Context Length

Figure A3 compares KV-cache size and decode throughput against RULER accuracy at 128K.

Llama-3.1-8B-Instruct Method GB∗ BF16 17.2 KVarN 2/2 2.6 KVQ 2-bit 2.6 TQ 2.5-bit 2.7 KIVI-2 3.2 KVarN 4/2 3.7 KVQ 3-bit 3.7 TQ 3.5-bit 4.1 KVQ 4-bit 4.8 KIVI-4 5.4 TailorKV-2 0.2 FreeKV 1.3 SpeCache 2.7 ShadowKV 2.8 ParisKV 3.8 StrataFlux-KV / DGX Spark 2/2 2.7 3/2 3.2 3/3 3.8 StrataFlux-KV / RTX 5090 2/2 2.6 3/2 3.1 3/3 3.7 Qwen3-8B Method GB∗ BF16 19.3 KVarN 2/2 2.9 KVQ 2-bit 2.9 TQ 2.5-bit 3.1 KIVI-2 3.6 KVarN 4/2 4.1 KVQ 3-bit 4.2 TQ 3.5-bit 4.6 KVQ 4-bit 5.4 KIVI-4 6.1 TailorKV-2 0.1 FreeKV 1.4 SpeCache 3.0 ShadowKV 3.1 ParisKV 4.3 StrataFlux-KV / DGX Spark 2/2 3.0 3/2 3.6 3/3 4.2 StrataFlux-KV / RTX 5090 2/2 2.9 3/2 3.5 3/3 4.1

PPL 10.32 10.98 11.71 10.98 10.90 10.38 10.56 10.43 10.38 10.33 11.21 11.62 10.68 11.24 13.07

KL 0.000 0.067 0.125 0.055 0.059 0.008 0.024 0.010 0.007 0.002 0.107 0.126 0.035 0.086 0.282

R8 95.1 94.4 93.4 93.6 93.8 94.8 94.8 94.9 95.1 95.3 94.1 93.7 95.0 90.5 93.1

R16 92.9 90.4 88.8 90.8 91.7 92.8 92.2 92.6 93.0 93.1 92.5 87.8 92.6 88.4 90.2

R32 87.1 85.5 84.8 84.4 85.6 87.1 86.9 87.1 87.1 87.0 88.4 80.8 87.1 84.4 85.6

R64 85.8 82.5 82.3 82.9 83.2 85.9 85.3 85.3 85.7 85.8 81.7 75.4 85.5 81.0 78.5

R128 77.9 68.6 69.6 68.8 71.6 77.2 77.0 76.7 77.5 78.0 68.6 63.6 74.0 70.6 68.9

10.47 10.34 10.33

0.011 0.002 0.001

95.2 95.0 95.2

92.7 93.2 93.2

87.5 87.1 87.0

85.1 85.8 85.8

75.2 77.6 77.5

10.40 10.33 10.33

0.003 0.001 0.001

95.1 95.1 95.0

92.9 93.1 93.0

87.1 87.0 86.9

86.0 85.8 85.8

78.1 78.3 78.2

PPL 12.89 14.40 14.40 36.02 13.85 13.04 13.21 13.99 12.96 12.90 13.66 14.68 13.96 15.10 14.11

KL 0.000 0.130 0.167 1.115 0.106 0.014 0.041 0.102 0.011 0.004 0.098 0.184 0.106 0.187 0.154

R8 93.8 93.0 91.2 90.2 92.9 93.8 93.6 94.0 93.8 94.0 93.7 91.1 93.8 89.4 93.3

R16 93.8 91.9 90.2 86.8 92.1 93.6 93.1 93.1 93.6 93.5 93.1 86.4 93.6 87.3 91.6

R32 92.1 89.0 85.1 81.3 88.8 91.9 91.9 92.0 91.9 92.1 90.5 82.2 91.5 82.8 88.5

R64 80.6 72.6 67.7 58.8 73.0 79.7 79.1 78.6 79.9 80.8 78.1 68.7 77.0 74.7 77.4

R128 74.1 66.0 60.4 48.2 67.0 73.9 72.2 73.4 74.0 74.2 71.4 54.2 69.3 70.5 70.1

13.32 12.98 12.93

0.030 0.005 0.004

94.0 93.7 93.8

93.4 93.7 93.8

91.6 92.0 92.2

77.2 79.5 79.8

71.4 74.0 74.2

12.98 12.92 12.90

0.007 0.002 0.001

94.0 93.9 93.9

93.8 93.8 93.8

91.8 92.0 92.1

79.9 80.4 80.3

74.2 74.7 74.5

Tables A14 and A15 compare 128K accuracy, memory, and throughput; Tables A16 and A17 report task accuracy. Tables A18 and A19 give context-length sweeps.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Table A15. KV-only refinement: PG-19 PPL/KL and RTX 5090 throughput at 128K. StrataFlux-KV fetches +4/+4 bits from host RAM for 1/128 of two-token groups. KV configuration BF16 KV KVarN 2/2 KIVI-2 KVQuant 2-bit TurboQuant 2.5-bit AQUA-KV 2-bit KVQuant 3-bit TurboQuant 3.5-bit AQUA-KV 3-bit KVarN 4/2 KIVI-4 KVQuant 4-bit AQUA-KV 4-bit ShadowKV ParisKV TailorKV-2 FreeKV SpeCache StrataFlux-KV 2/2 StrataFlux-KV 3/2 StrataFlux-KV 3/3

PPL 10.32 10.98 10.90 11.71 10.98 10.66 10.56 10.43 10.39 10.38 10.33 10.38 10.32 11.24 13.07 11.21 11.62 10.68 10.40 10.33 10.33

L3.1-8B-I KL tok/s 0.000 47.6 0.067 57.1 0.059 27.4 0.125 13.3 0.055 51.7 0.030 5.1 0.024 12.8 0.010 54.2 0.006 5.0 0.008 52.3 0.002 29.2 0.007 11.8 0.002 5.0 0.086 54.8 0.282 56.0 0.107 18.0 0.126 56.4 0.035 44.3 0.003 66.6 0.001 65.7 0.001 65.1

PPL 12.89 14.40 13.85 14.40 36.02 13.14 13.21 13.99 12.98 13.04 12.90 12.96 12.95 15.10 14.11 13.66 14.68 13.96 12.98 12.92 12.90

Qwen3-8B KL tok/s 0.000 OOM 0.130 53.4 0.106 25.0 0.167 13.2 1.115 48.1 0.029 4.5 0.041 12.7 0.102 50.7 0.017 4.5 0.014 48.8 0.004 26.6 0.011 11.5 0.016 4.5 0.187 50.7 0.154 51.6 0.098 15.0 0.184 30.3 0.106 40.5 0.007 62.7 0.002 61.9 0.001 61.4

Table A16. KV-only refinement: quantized KV task accuracy. R4–R128 give RULER accuracy at 4K–128K; LB32 uses LongBench prompts truncated to 32,128 tokens; M500/AIME report MATH500/AIME accuracy. Platform headings distinguish StrataFlux-KV configurations; baseline accuracies are shared across devices. Llama-3.1-8B-Instruct Configuration R4 BF16 KV 95.7 KVarN 2/2 94.4 KIVI-2 94.7 KVQuant 2-bit 94.8 TurboQuant 2.5-bit 94.5 AQUA-KV 2-bit 95.4 KVQuant 3-bit 95.6 TurboQuant 3.5-bit 95.7 AQUA-KV 3-bit 95.6 KVarN 4/2 95.5 KIVI-4 95.6 KVQuant 4-bit 95.7 AQUA-KV 4-bit 95.7 StrataFlux-KV / DGX Spark 2/2 95.8 3/2 95.7 3/3 95.8 StrataFlux-KV / RTX 5090 2/2 95.9 3/2 95.7 3/3 95.7 Qwen3-8B Configuration BF16 KV KVarN 2/2 KIVI-2 KVQuant 2-bit TurboQuant 2.5-bit AQUA-KV 2-bit

R4 95.2 94.2 94.0 93.5 92.3 94.9

R8 93.8 93.0 92.9 91.2 90.2 93.7

R16 93.8 91.9 92.1 90.2 86.8 93.4

R8 95.1 94.4 93.8 93.4 93.6 94.8 94.8 94.9 95.0 94.8 95.3 95.1 95.1

R16 92.9 90.4 91.7 88.8 90.8 92.8 92.2 92.6 92.8 92.8 93.1 93.0 92.9

R32 87.1 85.5 85.6 84.8 84.4 86.5 86.9 87.1 87.0 87.1 87.0 87.1 87.0

R64 85.8 82.5 83.2 82.3 82.9 85.7 85.3 85.3 85.8 85.9 85.8 85.7 85.8

R128 77.9 68.6 71.6 69.6 68.8 76.6 77.0 76.7 77.7 77.2 78.0 77.5 77.8

LB32 49.7 49.3 49.2 48.8 49.0 49.3 49.5 49.5 49.6 49.6 49.6 49.6 49.5

95.2 95.0 95.2

92.7 93.2 93.2

87.5 87.1 87.0

85.1 85.8 85.8

75.2 77.6 77.5

49.6 49.5 49.6

95.1 95.1 95.0

92.9 93.1 93.0

87.1 87.0 86.9

86.0 85.8 85.8

78.1 78.3 78.2

49.6 49.5 49.5

R32 92.1 89.0 88.8 85.1 81.3 91.9

R64 R128 LB32 M500 AIME 80.6 74.1 49.4 97.2 71.0 72.6 66.0 48.6 96.2 66.0 73.0 67.0 48.8 96.0 64.8 67.7 60.4 48.0 94.4 54.0 58.8 48.2 48.4 79.4 30.0 79.8 72.9 49.2 96.0 70.4

Qwen3-8B Configuration R4 R8 KVQuant 3-bit 94.9 93.6 TurboQuant 3.5-bit 95.1 94.0 AQUA-KV 3-bit 95.3 94.1 KVarN 4/2 95.4 93.8 KIVI-4 95.0 94.0 KVQuant 4-bit 95.3 93.8 AQUA-KV 4-bit 95.2 93.9 StrataFlux-KV / DGX Spark 2/2 95.4 94.0 3/2 95.2 93.7 3/3 95.3 93.8 StrataFlux-KV / RTX 5090 2/2 95.5 94.0 3/2 95.3 93.9 3/3 95.3 93.9

R16 93.1 93.1 93.6 93.6 93.5 93.6 93.6

R32 91.9 92.0 92.2 91.9 92.1 91.9 92.1

R64 R128 LB32 M500 AIME 79.1 72.2 49.2 96.2 70.8 78.6 73.4 49.4 97.0 69.8 80.5 74.5 49.3 97.2 71.9 79.7 73.9 49.3 96.2 71.0 80.8 74.2 49.4 96.0 71.3 79.9 74.0 49.2 96.6 72.1 80.7 74.2 49.3 95.6 70.2

93.4 91.6 77.2 93.7 92.0 79.5 93.8 92.2 79.8

71.4 74.0 74.2

49.0 49.4 49.3

96.0 95.0 96.2

68.3 70.2 69.4

93.8 91.8 79.9 93.8 92.0 80.4 93.8 92.1 80.3

74.2 74.7 74.5

49.3 49.3 49.3

95.6 96.2 97.0

69.2 70.0 71.5

Table A17. KV-only refinement: selective KV task accuracy. R8– R128 give RULER accuracy at 8K–128K; LB32 uses LongBench prompts truncated to 32,128 tokens. M500/AIME report Qwen38B MATH500/AIME accuracy. ShadowKV and TailorKV do not compress KV produced during decoding and are omitted from the M500/AIME comparison. Platform headings distinguish StrataFlux-KV configurations; baseline accuracies are shared across devices. Llama-3.1-8B-Instruct Configuration BF16 KV ShadowKV ParisKV TailorKV-2 TailorKV FreeKV SpeCache KVSwap StrataFlux-KV / DGX Spark 2/2 3/2 3/3 StrataFlux-KV / RTX 5090 2/2 3/2 3/3

R8 95.1 90.5 93.1 94.1 93.6 93.7 95.0 83.9

R16 92.9 88.4 90.2 92.5 91.8 87.8 92.6 78.4

R32 87.1 84.4 85.6 88.4 87.3 80.8 87.1 73.3

R64 85.8 81.0 78.5 81.7 77.8 75.4 85.5 69.0

R128 77.9 70.6 68.9 68.6 65.2 63.6 74.0 49.0

LB32 49.7 49.3 49.6 48.8 48.6 49.6 49.5 48.5

95.2 95.0 95.2

92.7 93.2 93.2

87.5 87.1 87.0

85.1 85.8 85.8

75.2 77.6 77.5

49.6 49.5 49.6

95.1 95.1 95.0

92.9 93.1 93.0

87.1 87.0 86.9

86.0 85.8 85.8

78.1 78.3 78.2

49.6 49.5 49.5

Qwen3-8B Configuration R8 R16 BF16 KV 93.8 93.8 ShadowKV 89.4 87.3 ParisKV 93.3 91.6 TailorKV-2 93.7 93.1 FreeKV 91.1 86.4 SpeCache 93.8 93.6 KVSwap 88.6 84.0 StrataFlux-KV / DGX Spark 2/2 94.0 93.4 3/2 93.7 93.7 3/3 93.8 93.8

R32 92.1 82.8 88.5 90.5 82.2 91.5 78.3

R64 80.6 74.7 77.4 78.1 68.7 77.0 68.6

R128 74.1 70.5 70.1 71.4 54.2 69.3 59.6

LB32 49.4 49.3 49.3 49.1 49.1 49.2 49.1

M500 97.2 – 95.4 – 96.8 96.6 94.0

AIME 71.0 – 64.2 – 63.5 70.2 52.7

91.6 92.0 92.2

77.2 79.5 79.8

71.4 74.0 74.2

49.0 49.4 49.3

96.0 95.0 96.2

68.3 70.2 69.4

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Qwen3-8B Configuration R8 R16 StrataFlux-KV / RTX 5090 2/2 94.0 93.8 3/2 93.9 93.8 3/3 93.9 93.8

R32

R64

R128

LB32

M500

AIME

91.8 92.0 92.1

79.9 80.4 80.3

74.2 74.7 74.5

49.3 49.3 49.3

95.6 96.2 97.0

69.2 70.0 71.5

Configuration

Metric

8K

16K

32K

64K

128K

AQUA-KV 4-bit

PPL KL Spark tok/s 5090 tok/s

10.35 0.002 9.7 44.2

10.31 0.002 6.8 30.4

10.29 0.002 4.4 17.8

10.19 0.002 2.6 9.1

10.32 0.002 1.5 5.0

10.87 0.056 0.12 8.7

10.99 0.072 0.19 8.2

11.10 0.087 0.32 9.2

11.09 0.097 0.59 8.8

16.27 0.477 1.13 8.7

KV offloading / DGX Spark

Table A18. KV-only refinement: PG-19 context sweep on Llama3.1-8B-Instruct. ∗ GB counts only the resident KV representation and, on DGX Spark, its refinement RAM cache; weights and backing-tier storage are excluded. Throughput is tok/s at the column’s context length.

KVSwap

PPL KL GB∗ tok/s

KV offloading / RTX 5090

ShadowKV

PPL KL GB∗ tok/s

10.69 0.042 0.23 88.3

10.66 0.044 0.40 86.9

10.65 0.045 0.74 84.6

10.60 0.048 1.41 67.9

11.24 0.086 2.76 54.8

ParisKV

PPL KL GB∗ tok/s

10.59 0.045 0.32 77.6

10.61 0.060 0.55 75.5

10.62 0.072 1.02 72.9

10.60 0.085 1.96 67.2

13.07 0.282 3.84 56.0

Configuration

Metric

8K

16K

32K

64K

128K

BF16 KV

PPL KL GB∗ Spark tok/s 5090 tok/s

10.34 0.000 1.07 14.9 96.1

10.29 0.000 2.15 14.1 90.1

10.27 0.000 4.30 12.5 80.2

10.17 0.000 8.59 9.9 65.0

10.32 0.000 17.18 7.3 47.6

PPL KL Spark tok/s 5090 tok/s

10.74 0.045 15.3 92.9

10.73 0.049 14.7 88.0

10.74 0.051 13.9 79.7

10.64 0.052 13.2 72.1

10.98 0.067 11.3 57.1

TailorKV-2

PPL KL GB∗ tok/s

10.38 0.018 0.03 36.3

10.36 0.027 0.04 32.3

10.35 0.035 0.05 34.9

10.25 0.040 0.09 23.2

11.21 0.107 0.16 18.0

PPL KL GB∗ tok/s

10.45 0.015 0.33 58.0

10.50 0.026 0.39 56.3

10.58 0.038 0.53 50.8

10.61 0.052 0.79 49.6

11.62 0.126 1.32 56.4

Quantization

KVarN 2/2

KIVI-2

PPL KL Spark tok/s 5090 tok/s

10.82 0.055 14.7 82.1

10.84 0.058 14.0 72.5

10.81 0.059 12.9 58.9

10.69 0.059 11.3 42.7

10.90 0.059 8.8 27.4

FreeKV

11.15 0.084 12.9 60.2

11.11 0.086 11.6 48.8

11.12 0.087 9.5 36.2

11.04 0.089 7.0 22.6

11.71 0.125 3.9 13.3

SpeCache

KVQuant 2-bit

PPL KL Spark tok/s 5090 tok/s

PPL KL GB∗ tok/s

10.45 0.013 0.18 78.8

10.42 0.016 0.35 76.4

10.42 0.018 0.69 67.7

10.34 0.021 1.36 58.1

10.68 0.035 2.70 44.3

TailorKV-1

PPL KL Spark tok/s 5090 tok/s

10.77 0.040 15.0 88.9

10.76 0.042 14.5 85.1

10.78 0.045 13.6 77.2

10.67 0.045 12.4 65.9

10.98 0.055 10.2 51.7

PPL KL

11.46 0.128

11.59 0.149

11.58 0.157

11.60 0.173

13.71 0.322

TurboQuant 2.5-bit

AQUA-KV 2-bit

KVQuant 3-bit

PPL KL Spark tok/s 5090 tok/s PPL KL Spark tok/s 5090 tok/s

10.54 0.022 9.8 44.4 10.52 0.020 12.9 59.5

10.51 0.021 6.9 30.6 10.47 0.021 11.4 48.1

10.54 0.024 4.5 17.9 10.46 0.021 9.1 35.0

10.48 0.030 2.7 9.1 10.35 0.021 6.4 21.9

10.66 0.030 1.5 5.1 10.56 0.024 3.9 12.8

PPL KL Spark tok/s 5090 tok/s

10.42 0.007 15.0 89.1

10.37 0.007 14.5 85.7

10.35 0.008 13.7 78.7

10.26 0.008 12.5 67.9

10.43 0.010 10.4 54.2

AQUA-KV 3-bit

PPL KL Spark tok/s 5090 tok/s

10.38 0.005 9.7 44.4

10.35 0.006 6.8 30.5

10.34 0.006 4.5 17.9

10.24 0.007 2.6 9.1

10.39 0.006 1.5 5.0

KVarN 4/2

PPL KL Spark tok/s 5090 tok/s

10.38 0.007 15.1 91.8

10.34 0.007 14.5 86.2

10.32 0.007 13.5 76.9

10.22 0.007 12.6 68.3

10.38 0.008 10.4 52.3

KIVI-4

PPL KL Spark tok/s 5090 tok/s

10.35 0.002 14.7 83.4

10.30 0.002 14.0 74.4

10.28 0.002 12.9 61.0

10.19 0.002 11.1 44.8

10.33 0.002 8.6 29.2

KVQuant 4-bit

PPL KL Spark tok/s 5090 tok/s

10.38 0.006 12.5 57.3

10.34 0.006 10.8 46.0

10.32 0.006 8.4 33.0

10.22 0.006 5.8 20.3

10.38 0.007 3.3 11.8

TurboQuant 3.5-bit

StrataFlux-KV / DGX Spark

StrataFlux-KV 2/2

PPL KL GB∗ tok/s

10.42 0.007 0.32 14.9

10.39 0.006 0.48 14.7

10.37 0.006 0.80 14.2

10.27 0.006 1.44 13.4

10.47 0.011 2.71 11.4

StrataFlux-KV 3/2

PPL KL GB∗ tok/s

10.35 0.002 0.36 14.8

10.31 0.002 0.55 14.6

10.29 0.002 0.93 14.1

10.19 0.002 1.71 13.1

10.34 0.002 3.25 11.0

StrataFlux-KV 3/3

PPL KL GB∗ tok/s

10.35 0.001 0.39 14.8

10.30 0.001 0.61 14.5

10.28 0.001 1.07 14.0

10.18 0.001 1.97 12.8

10.33 0.001 3.79 10.7

StrataFlux-KV / RTX 5090

StrataFlux-KV 2/2

PPL KL GB∗ tok/s

10.41 0.004 0.19 88.0

10.37 0.004 0.35 85.4

10.35 0.003 0.67 81.8

10.25 0.002 1.30 75.7

10.40 0.003 2.58 66.6

StrataFlux-KV 3/2

PPL KL GB∗ tok/s

10.35 0.001 0.22 87.5

10.30 0.001 0.41 84.9

10.29 0.001 0.80 81.2

10.18 0.001 1.57 75.1

10.33 0.001 3.12 65.7

StrataFlux-KV 3/3

PPL KL GB∗ tok/s

10.35 0.001 0.25 87.5

10.30 0.001 0.48 84.6

10.28 0.001 0.93 80.8

10.18 0.001 1.84 74.4

10.33 0.001 3.65 65.1

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Table A19. KV-only refinement: PG-19 context sweep on Qwen38B. ∗ GB counts only the resident KV representation and, on DGX Spark, its refinement RAM cache; weights and backing-tier storage are excluded. Throughput is tok/s at the column’s context length. Configuration

Metric

8K

16K

32K

64K

128K

BF16 KV

PPL KL GB∗ Spark tok/s 5090 tok/s

13.23 0.000 1.21 14.7 93.8

12.91 0.000 2.42 13.7 87.5

12.64 0.000 4.83 12.1 77.0

12.99 0.000 9.66 9.4 61.4

12.89 0.000 19.33 6.7 OOM

KVarN 2/2

PPL KL Spark tok/s 5090 tok/s

13.77 0.062 14.9 89.9

13.47 0.065 14.4 84.9

13.11 0.059 13.6 76.4

14.44 0.131 12.7 68.5

14.40 0.130 10.7 53.4

KIVI-2

PPL KL Spark tok/s 5090 tok/s

13.56 0.048 14.4 78.6

13.25 0.048 13.7 69.1

13.01 0.051 12.5 55.3

13.88 0.090 10.7 39.4

13.85 0.106 8.3 25.0

KVQuant 2-bit

PPL KL Spark tok/s 5090 tok/s

14.48 0.117 12.8 57.6

14.18 0.124 11.1 47.3

13.97 0.129 8.8 34.7

14.39 0.158 6.3 21.7

14.40 0.167 4.1 13.2

TurboQuant 2.5-bit

PPL KL Spark tok/s 5090 tok/s

20.66 0.582 14.7 86.0

21.20 0.631 14.2 81.9

21.82 0.685 13.2 73.6

33.85 1.047 11.9 62.2

36.02 1.115 9.6 48.1

AQUA-KV 2-bit

PPL KL Spark tok/s 5090 tok/s

13.40 0.012 9.3 41.1

13.11 0.013 6.5 27.9

12.88 0.013 4.1 16.2

13.21 0.028 2.4 8.1

13.14 0.029 1.4 4.5

KVQuant 3-bit

PPL KL Spark tok/s 5090 tok/s

13.47 0.026 12.6 56.6

13.20 0.027 10.8 45.8

12.90 0.028 8.5 33.5

13.29 0.039 6.4 21.3

13.21 0.041 3.9 12.7

TurboQuant 3.5-bit

PPL KL Spark tok/s 5090 tok/s

13.80 0.057 14.7 86.3

13.56 0.067 14.2 82.4

13.12 0.051 13.3 75.4

14.32 0.118 12.0 64.3

13.99 0.102 9.9 50.7

AQUA-KV 3-bit

PPL KL Spark tok/s 5090 tok/s

13.27 0.005 9.3 41.0

12.96 0.006 6.4 27.8

12.69 0.006 4.1 16.1

13.08 0.017 2.4 8.1

12.98 0.017 1.4 4.5

PPL KL Spark tok/s 5090 tok/s

13.32 0.008 14.8 88.8

13.00 0.008 14.2 82.9

12.72 0.009 13.2 73.3

13.15 0.014 12.0 64.7

13.04 0.014 9.8 48.8

PPL KL Spark tok/s 5090 tok/s

13.23 0.002 14.4 80.1

12.92 0.002 13.7 70.8

12.64 0.002 12.5 57.5

13.01 0.004 10.6 41.4

12.90 0.004 8.0 26.6

KVQuant 4-bit

PPL KL Spark tok/s 5090 tok/s

13.30 0.007 12.3 54.5

12.97 0.007 10.3 43.5

12.69 0.007 8.2 31.4

13.04 0.011 5.6 19.3

12.96 0.011 3.4 11.5

AQUA-KV 4-bit

PPL KL Spark tok/s 5090 tok/s

13.24 0.004 9.3 41.0

12.93 0.004 6.4 27.8

12.65 0.004 4.0 16.1

13.05 0.016 2.4 8.1

12.95 0.016 1.3 4.5

Configuration

KVSwap

KIVI-4

8K

16K

32K

64K

128K

PPL KL GB∗ tok/s

14.09 0.074 0.14 7.4

14.10 0.101 0.21 7.7

14.21 0.133 0.36 7.7

15.64 0.259 0.66 7.6

15.82 0.289 1.27 7.5

KV offloading / RTX 5090

ShadowKV

PPL KL GB∗ tok/s

15.78 0.229 0.26 84.9

14.85 0.193 0.45 83.3

13.92 0.145 0.83 81.1

15.65 0.214 1.59 62.5

15.10 0.187 3.11 50.7

ParisKV

PPL KL GB∗ tok/s

13.34 0.050 0.36 70.8

13.10 0.064 0.62 71.6

12.87 0.070 1.15 68.9

14.09 0.140 2.21 62.8

14.11 0.154 4.32 51.6

TailorKV-2

PPL KL GB∗ tok/s

13.09 0.017 0.03 24.5

12.85 0.030 0.03 25.7

12.58 0.037 0.04 21.8

13.62 0.082 0.05 16.7

13.66 0.098 0.07 15.0

FreeKV

PPL KL GB∗ tok/s

13.45 0.021 0.36 47.3

13.34 0.040 0.43 38.9

13.30 0.062 0.58 35.5

14.37 0.146 0.86 31.5

14.68 0.184 1.42 30.3

SpeCache

PPL KL GB∗ tok/s

13.43 0.026 0.21 73.4

13.14 0.030 0.39 70.9

12.94 0.038 0.77 63.0

13.74 0.080 1.53 53.1

13.96 0.106 3.04 40.5

Quantization

KVarN 4/2

Metric

KV offloading / DGX Spark

StrataFlux-KV / DGX Spark

StrataFlux-KV 2/2

PPL KL GB∗ tok/s

13.39 0.011 0.35 14.6

13.09 0.010 0.53 14.4

12.82 0.009 0.88 13.9

13.34 0.028 1.60 13.0

13.32 0.030 3.04 10.9

StrataFlux-KV 3/2

PPL KL GB∗ tok/s

13.28 0.002 0.38 14.6

12.96 0.002 0.60 14.3

12.69 0.002 1.03 13.7

13.07 0.005 1.90 12.6

12.98 0.005 3.64 10.5

StrataFlux-KV 3/3

PPL KL GB∗ tok/s

13.25 0.001 0.42 14.5

12.93 0.001 0.67 14.2

12.66 0.001 1.18 13.7

13.03 0.003 2.20 12.4

12.93 0.004 4.24 10.2

StrataFlux-KV / RTX 5090

StrataFlux-KV 2/2

PPL KL GB∗ tok/s

13.39 0.007 0.21 84.7

13.07 0.005 0.39 82.2

12.80 0.004 0.75 78.5

13.07 0.009 1.47 72.3

12.98 0.007 2.90 62.7

StrataFlux-KV 3/2

PPL KL GB∗ tok/s

13.28 0.002 0.25 84.2

12.96 0.001 0.47 81.6

12.68 0.001 0.90 77.9

13.02 0.002 1.77 71.5

12.92 0.002 3.50 61.9

StrataFlux-KV 3/3

PPL KL GB∗ tok/s

13.25 0.001 0.29 84.1

12.93 0.001 0.54 81.3

12.66 0.001 1.05 77.4

12.99 0.001 2.07 70.9

12.90 0.001 4.11 61.4

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

C.2

Alternative Attention Architectures and MoE Models

Table A20. KV-only refinement: PG-19 perplexity/KL and RULER accuracy at 32K, plus MATH500 accuracy for Qwen3.59B. The comparison covers hybrid linear/full attention (Qwen3.59B) and MoE models with standard attention (Qwen3-30B-A3B) or latent attention (DeepSeek-V2-Lite). Refinement applies only to full attention. For DeepSeek, 2/2 and 3/3 indicate 2- and 3-bit latent precision; 3/2 is inapplicable because the shared latent representation uses one bitwidth. Qwen3.5-9B Qwen3-30B-A3B DeepSeek-V2-Lite K/V Method PPL KL R32 M500 PPL KL R32 PPL KL R32 BF16 9.48 0.000 96.6 85.6 9.29 0.000 94.1 9.12 0.000 67.5 Base 9.71 0.027 96.3 83.0 10.06 0.086 89.9 9.59 0.138 59.9 2/2 StrataFlux-KV 9.54 0.004 96.5 88.0 9.55 0.021 93.6 9.17 0.017 67.2 Base 9.62 0.014 96.5 84.6 9.42 0.016 93.9 3/2 – StrataFlux-KV 9.51 0.003 96.5 86.6 9.35 0.004 93.9 Base 9.50 0.004 96.5 86.6 9.34 0.009 93.9 9.16 0.026 66.5 3/3 StrataFlux-KV 9.48 0.001 96.5 89.6 9.32 0.002 94.0 9.13 0.006 67.2

Table A20 groups the three architectures; Tables A21–A23 give per-model task results. Table A21. KV-only refinement: accuracy on Qwen3.5-9B. R4– R32 are RULER context lengths in thousands of tokens. M500 is MATH500. KVQuant uses C4 calibration and retains 1% outliers. TurboQuant presets mix bitwidths across channels. StrataFlux-KV uses two-token groups, 1/128 selection, and no cache misses on dedicated GPUs; Base disables refinement. Method BF16 KV KIVI KVQuant TurboQuant AQUA-KV Base StrataFlux-KV Base StrataFlux-KV Base StrataFlux-KV

Bits 16/16 2 4 2 3 4 2.5 3.5 2 3 4 2/2 3/2 3/3

R4 97.1 97.2 97.1 96.9 97.1 97.1 97.1 97.1 97.1 97.1 97.1 97.1 97.1 97.1 97.1 97.1 97.1

R8 97.1 97.0 97.1 97.0 97.0 97.1 96.9 97.2 97.0 97.1 97.1 97.0 97.2 97.1 97.2 97.2 97.1

R16 96.6 96.7 96.6 96.6 96.7 96.6 96.5 96.6 96.9 96.7 96.7 96.5 96.6 96.4 96.6 96.6 96.6

R32 96.6 96.4 96.5 96.6 96.7 96.5 96.4 96.5 96.9 96.7 96.7 96.3 96.5 96.5 96.5 96.5 96.5

M500 85.6 81.6 85.2 94.2 86.2 89.0 88.8 86.2 87.4 83.8 84.6 83.0 88.0 84.6 86.6 86.6 89.6

Table A22. KV-only refinement: accuracy on DeepSeek-V2-Lite. R4–R32 are RULER context lengths in thousands of tokens. KVQuant uses C4 calibration and retains 1% outliers. TurboQuant presets mix bitwidths across channels. StrataFlux-KV uses two-token groups, 1/128 selection, and no cache misses on dedicated GPUs; Base disables refinement. StrataFlux-KV bitwidths give latent precision, rather than separate K/V widths. Method BF16 KV KIVI KVQuant TurboQuant AQUA-KV Base StrataFlux-KV Base StrataFlux-KV

Bits 16/16 2 4 2 3 4 2.5 3.5 2 3 4 2 3

R4 91.1 86.7 90.4 86.4 90.2 90.9 84.4 90.9 90.7 91.2 90.9 86.8 90.4 90.6 90.7

R8 85.2 78.6 85.3 75.4 83.1 84.4 77.3 84.3 82.5 84.8 85.1 79.2 84.5 84.1 85.2

R16 77.8 69.8 77.7 65.4 76.2 78.1 69.0 77.0 74.8 77.5 78.4 70.9 78.8 77.5 78.2

R32 67.5 58.5 67.3 53.2 64.0 66.9 52.5 66.0 63.8 65.8 66.7 59.9 67.2 66.5 67.2

Table A23. KV-only refinement: accuracy on Qwen3-30B-A3B. R4–R32 are RULER context lengths in thousands of tokens. KVQuant uses C4 calibration and retains 1% outliers. TurboQuant presets mix bitwidths across channels. StrataFlux-KV uses two-token groups, 1/128 selection, and no cache misses on dedicated GPUs; Base disables refinement. Method BF16 KV KIVI KVQuant TurboQuant AQUA-KV Base StrataFlux-KV Base StrataFlux-KV Base StrataFlux-KV

Bits 16/16 2 4 2 3 4 2.5 3.5 2 3 4 2/2 3/2 3/3

R4 95.6 94.6 95.5 93.9 95.3 95.5 93.0 95.3 95.6 95.6 95.7 93.4 95.6 95.5 95.7 95.6 95.6

R8 95.8 95.0 95.9 93.9 95.3 95.6 91.4 95.7 95.7 95.8 95.8 93.3 95.8 95.6 95.9 95.7 95.8

R16 95.1 93.1 94.8 91.9 94.6 94.8 91.1 94.7 94.8 94.7 94.7 92.0 95.0 94.9 94.8 95.1 94.9

R32 94.1 92.3 94.1 90.7 94.2 94.2 85.2 93.6 94.1 94.1 94.2 89.9 93.6 93.9 93.9 93.9 94.0

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

C.3

L3.1-8B-I

KV Representation and Selection Controls

Tables A24 and A25 vary the retained FP16 tokens and the attention-score aggregation used for token-group selection.

PPL

2/2

Table A24. KV representation and selection controls: Llama-3.18B-Instruct, PG-19 at 32K and 128K. Four-token groups, 1/128 fetch fraction. We retain the first 128 tokens in FP16 following KVarN (Muller et al., 2026) and select by the maximum score over query heads; controls change one setting. Configuration

32K PPL KL 128K PPL KL

2/2

Reference First 128 tokens quantized Mean over query heads

10.36 10.36 10.36

0.004 0.004 0.004

10.42 10.41 10.41

0.003 0.004 0.003

3/2

Reference First 128 tokens quantized Mean over query heads

10.29 10.29 10.29

0.001 0.001 0.001

10.33 10.33 10.33

0.001 0.001 0.001

3/3

Reference First 128 tokens quantized Mean over query heads

10.29 10.29 10.28

0.001 0.001 0.001

10.33 10.33 10.33

0.001 0.001 0.001

Table A25. KV representation and selection controls: Qwen3-8B, PG-19 at 32K and 128K. Four-token groups, 1/128 fetch fraction. Reference shows our configuration, where we retain the first 128 tokens in FP16 following KVarN (Muller et al., 2026) and select token groups using the maximum score over query heads. Each control changes one of these settings. GPU resident KV

Configuration

32K PPL KL 128K PPL KL

2/2

Reference First 128 tokens quantized Mean over query heads

12.81 12.81 12.80

0.006 0.006 0.006

13.01 13.00 13.01

0.009 0.010 0.009

3/2

Reference First 128 tokens quantized Mean over query heads

12.69 12.69 12.68

0.001 0.002 0.001

12.93 12.93 12.93

0.002 0.002 0.002

3/3

Reference First 128 tokens quantized Mean over query heads

12.66 12.66 12.66

0.001 0.001 0.001

12.90 12.90 12.90

0.002 0.002 0.002

C.4

Fetch Budgets, RAM Caches, and Miss Policies L3.1-8B-I

Qwen3-8B 14.50

PG-19 PPL

11.00

14.00

10.75

13.50

10.50

13.00 10.25 65

70

75

60

Decode throughput (tok/s) 2/2

3/2

65

70

75

Decode throughput (tok/s) 3/3

Base

Figure A4. KV fetch-fraction sweep: PG-19 PPL versus RTX 5090 throughput at 128K. Curves identify resident K/V bitwidths; fractions 1/256, 1/128, and 1/64 run from highest to lowest throughput. StrataFlux-KV uses two-token groups and +4/+4 bits. Open markers show the matching bases without refinement.

10.45

10.75

10.40

10.50

10.35

3/3 10.39 10.36 10.33

64 128 256 512

64 128 256 512

64 128 256 512

Qwen3-8B 2/2 14.40

PPL

GPU resident KV

3/2

11.00

3/2

13.20

13.05

13.60

12.96 12.90

12.90

12.80 64 128 256 512

3/3 13.02

64 128 256 512

64 128 256 512

RAM cache (MiB) Skip misses

No cache misses

No refinement

Figure A5. KV RAM-cache capacity by resident bitwidth on DGX Spark: PG-19 PPL at 128K with four-token groups, +4/+4 bits, 1/128 selection, and skipped misses. Dotted lines assume no cache misses; dashed lines show the matching no-refinement bases.

Figure A4 varies the host-RAM fetch fraction on RTX 5090; Figure A5 varies RAM-cache capacity on DGX Spark. Table A31 reports throughput when waiting on cache misses. Table A26 compares SSD-backed configurations; Tables A27–A30 report cache-capacity and no-cache-miss comparisons. Table A26. KV-only refinement: PG-19 PPL/KL and DGX Spark throughput at 128K. StrataFlux-KV uses +4/+4 bits, 1/128 selection, four-token groups, and an SSD backing tier with a 128 MiB RAM cache and skipped misses. Configuration BF16 KV KVarN 2/2 KIVI-2 KVQuant 2-bit TurboQuant 2.5-bit AQUA-KV 2-bit KVQuant 3-bit TurboQuant 3.5-bit AQUA-KV 3-bit KVarN 4/2 KIVI-4 KVQuant 4-bit AQUA-KV 4-bit StrataFlux-KV 2/2 StrataFlux-KV 3/2 StrataFlux-KV 3/3

PPL 10.32 10.98 10.90 11.71 10.98 10.66 10.56 10.43 10.39 10.38 10.33 10.38 10.32 10.47 10.34 10.33

L3.1-8B-I KL tok/s 0.000 7.3 0.067 11.3 0.059 8.8 0.125 3.9 0.055 10.2 0.030 1.5 0.024 3.9 0.010 10.4 0.006 1.5 0.008 10.4 0.002 8.6 0.007 3.3 0.002 1.5 0.011 11.4 0.002 11.0 0.001 10.7

Qwen3-8B PPL KL tok/s 12.89 0.000 6.7 14.40 0.130 10.7 13.85 0.106 8.3 14.40 0.167 4.1 36.02 1.115 9.6 13.14 0.029 1.4 13.21 0.041 3.9 13.99 0.102 9.9 12.98 0.017 1.4 13.04 0.014 9.8 12.90 0.004 8.0 12.96 0.011 3.4 12.95 0.016 1.3 13.32 0.030 10.9 12.98 0.005 10.5 12.93 0.004 10.2

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Table A27. SSD-backed KV refinement on DGX Spark: Llama3.1-8B-Instruct, PG-19 at 128K. PPL, KL, and decode tok/s by resident bits, fetched group fraction, and RAM-cache capacity. The 1/128, 128 MiB configuration repeats Table A26. GPU resident KV 2/2

3/2

3/3

GPU resident KV 2/2

3/2

3/3

Fetch fraction 1/256 1/128 1/64 1/256 1/128 1/64 1/256 1/128 1/64

PPL 10.50 10.48 10.47 10.34 10.34 10.34 10.33 10.33 10.33

64 MiB KL tok/s 0.014 11.5 0.013 11.2 0.013 10.6 0.003 11.1 0.002 10.7 0.002 10.3 0.002 10.8 0.001 10.5 0.002 10.0

Fetch fraction 1/256 1/128 1/64 1/256 1/128 1/64 1/256 1/128 1/64

256 MiB PPL KL tok/s 10.49 0.011 11.5 10.46 0.009 11.4 10.44 0.008 11.1 10.34 0.002 11.1 10.34 0.002 10.9 10.33 0.002 10.6 10.34 0.001 10.8 10.33 0.001 10.6 10.33 0.001 10.3

PPL 10.50 10.47 10.46 10.34 10.34 10.34 10.34 10.33 10.33

128 MiB KL tok/s 0.013 11.5 0.011 11.4 0.010 10.8 0.002 11.1 0.002 11.0 0.002 10.4 0.001 10.8 0.001 10.7 0.001 10.2

512 MiB PPL KL tok/s 10.49 0.010 11.5 10.45 0.007 11.3 10.43 0.006 11.0 10.34 0.002 11.1 10.34 0.002 10.9 10.33 0.001 10.6 10.34 0.001 10.7 10.33 0.001 10.6 10.33 0.001 10.3

Table A28. KV refinement with no cache misses: Llama-3.1-8BInstruct, PG-19 at 128K. PPL/KL by fetched group fraction, using four-token groups.

GPU resident KV 2/2

3/2

3/3

2/2 Fetch fraction 1/256 1/128 1/64

PPL 10.44 10.42 10.39

KL 0.005 0.003 0.003

2/2 Fetch fraction 1/256 1/128 1/64

KL 0.001 0.001 0.001

Table A29. SSD-backed KV refinement on DGX Spark: Qwen38B, PG-19 at 128K. PPL, KL, and decode tok/s by resident bits, fetched group fraction, and RAM-cache capacity. The 1/128, 128 MiB configuration repeats Table A26. GPU resident KV 2/2

3/2

3/3

Fetch fraction 1/256 1/128 1/64 1/256 1/128 1/64 1/256 1/128 1/64

PPL 13.41 13.36 14.28 12.99 12.97 13.14 12.94 12.93 13.00

64 MiB KL tok/s 0.038 11.1 0.036 10.6 0.119 10.0 0.007 10.6 0.006 10.1 0.022 9.7 0.005 10.3 0.004 9.8 0.013 9.4

PPL 13.37 13.32 13.28 12.99 12.98 12.97 12.94 12.93 12.93

128 MiB KL tok/s 0.033 11.1 0.030 10.9 0.028 10.3 0.006 10.7 0.005 10.5 0.005 9.9 0.004 10.3 0.004 10.2 0.004 9.6

256 MiB KL tok/s 0.030 11.0 0.026 10.9 0.023 10.5 0.005 10.6 0.005 10.4 0.004 10.1 0.004 10.3 0.003 10.1 0.003 9.8

PPL 13.28 13.23 13.20 12.97 12.96 12.96 12.93 12.93 12.93

512 MiB KL tok/s 0.026 11.0 0.021 10.9 0.018 10.4 0.004 10.6 0.004 10.4 0.003 10.0 0.003 10.2 0.003 10.0 0.003 9.7

PPL 13.05 13.01 12.97

KL 0.013 0.009 0.006

GPU resident KV 3/2 PPL KL 12.93 0.002 12.93 0.002 12.92 0.002

3/3 PPL 12.90 12.90 12.90

KL 0.002 0.002 0.001

Table A31. KV-only refinement: DGX Spark throughput (tok/s) when waiting on cache misses at 128K, with four-token groups. Columns give RAM-cache capacities (MiB). Matching PPL/KL appears in Tables A28 and A30. GPU resident KV

3/3 PPL 10.34 10.33 10.33

PPL 13.34 13.30 13.25 12.98 12.97 12.96 12.94 12.93 12.93

Table A30. KV refinement with no cache misses: Qwen3-8B, PG19 at 128K. PPL/KL by fetched group fraction, using four-token groups.

2/2 GPU resident KV 3/2 PPL KL 10.34 0.001 10.33 0.001 10.33 0.001

Fetch fraction 1/256 1/128 1/64 1/256 1/128 1/64 1/256 1/128 1/64

3/2

3/3

C.5

Fetch fraction 1/256 1/128 1/64 1/256 1/128 1/64 1/256 1/128 1/64

L3.1-8B-I 64 128 256 8.6 8.8 9.1 6.8 8.1 8.7 4.0 5.8 7.5 8.2 8.6 9.0 7.0 7.9 8.4 3.9 6.0 7.5 8.2 8.6 8.9 6.9 7.8 8.2 4.0 5.8 7.2

512 9.2 9.2 8.3 9.0 8.8 8.0 8.9 8.6 7.9

Qwen3-8B 64 128 256 512 7.8 8.5 9.3 8.5 6.5 7.7 8.6 8.4 2.9 5.5 7.1 8.1 7.8 8.4 8.9 8.3 6.5 7.3 8.0 8.2 3.1 5.5 6.9 8.2 7.8 8.2 8.4 8.6 6.2 7.1 7.9 8.5 3.0 5.4 7.0 7.4

Refinement with 2-bit Keys and 1-bit Values

Table A32 compares Base and StrataFlux-KV with 2-bit keys and 1-bit values. Table A32. KV refinement with resident 2/1 bits at 128K: PG-19 PPL/KL, RULER accuracy, Qwen3-8B MATH500 accuracy, RTX 5090 decode tok/s at 128K, and memory. ∗ KV memory only. StrataFlux-KV uses +0/+1 bits, two-token groups, and a 1/16 fetch fraction. In our Llama-3.1-8B-Instruct 32K sweep, +0/+1 yields the lowest PPL/KL among tested fetched-bitwidth pairs at matched payload. L3.1-8B-I Qwen3-8B Method PPL KL R128 tok/s GB∗ PPL KL R128 M500 tok/s GB∗ BF16 10.32 0.000 77.9 47.6 17.2 12.89 0.000 74.1 97.2 OOM 19.3 KVarN 2/1 31.29 1.180 40.7 77.8 2.0 112.6 2.318 18.0 28.6 74.2 2.3 StrataFlux-KV 2/1 11.65 0.135 64.1 63.5 2.0 15.16 0.181 60.4 93.4 59.8 2.3

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

KV Refinement Settings Sweep

L3.1-8B-I

D

J OINT C ONFIGURATIONS AND TASK R ESULTS

D.1

Configuration Comparisons

PPL 10.32 13.79 13.76 13.74 10.80 10.74 10.72 12.98 12.95 12.95

Qwen3-8B

KL GB tok/s† PPL KL GB tok/s† 0.000 33.2 47.6 12.89 0.000 35.7 OOM 0.312 6.4 106.6 14.91 0.165 7.1 97.6 0.313 7.0 104.2 14.88 0.162 7.7 95.4 0.311 7.5 102.1 14.84 0.161 8.3 93.3 0.045 7.3 102.0 13.25 0.038 8.0 91.7 0.044 7.8 100.3 13.20 0.034 8.6 90.3 0.042 8.4 98.3 13.17 0.034 9.2 88.2 0.241 7.3 117.9 17.46 0.292 8.0 105.1 0.242 7.9 115.8 17.29 0.286 8.6 103.3 0.242 8.4 113.1 17.27 0.285 9.2 100.6

0.06

0.02

0.04

0.06

10.32 0.02

0.04

0.06

0.02

0.04

0.06

13.05 12.90 12.75 0.00

0.02

0.04

0.06

12.75 12.70 12.65 0.00

0.02

0.04

0.06

0.04

0.06

0.04

0.06

0.04

0.06

Resident 3/2, 4-token groups

10.36

PPL L3.1-8B-I Configuration BF16 StrataFlux / YAQA 2-bit / 2/2 StrataFlux / YAQA 2-bit / 3/2 StrataFlux / YAQA 2-bit / 3/3 StrataFlux / YAQA 3-bit / 2/2 StrataFlux / YAQA 3-bit / 3/2 StrataFlux / YAQA 3-bit / 3/3 StrataFlux / LNQ+GQ 3-bit / 2/2 StrataFlux / LNQ+GQ 3-bit / 3/2 StrataFlux / LNQ+GQ 3-bit / 3/3

10.65 10.50 10.35 0.00

10.28 0.00

10.32 10.28 0.00

0.02

0.04

0.06

12.75 12.70 12.65 0.00

0.02

Resident 3/3, 2-token groups

PPL

10.32 10.30 10.28 0.00

0.02

0.04

0.06

12.72 12.70 12.68 12.65 0.00

0.02

Resident 3/3, 4-token groups

10.32

PPL

Qwen3-8B

0.04

10.36

Table A33. Joint configurations at 128K: PG-19 PPL/KL, GPU memory (GB), and decode tok/s. ∗ DGX Spark; † RTX 5090. StrataFlux uses RTX 5090's 5% weight configurations. GB counts stored weights and KV. W/A/KV: weight/activation/cache bits; bpw: bits/weight; FP8: 8-bit floating point. L3.1-8B-I

0.02

13.05 12.90 12.75 0.00

Resident 3/2, 2-token groups

Tables A33, A34, and A35 compare inference-system configurations, static weight/KV pairs, and the four weight/KV refinement combinations.

Configuration PPL KL GB tok/s∗tok/s† PPL KL GB tok/s∗tok/s† BF16 10.32 0.000 33.2 7.3 47.6 12.89 0.000 35.7 6.7 OOM vLLM AWQ4 + FP8 KV 11.03 0.074 14.3 16.8 106.8 13.49 0.078 15.8 15.3 96.6 EXL3 3.0 bpw + 4/4 KV 11.39 0.107 8.9 25.6 99.6 14.00 0.125 9.8 23.6 91.3 EXL3 4.0 bpw + 4/4 KV 10.78 0.051 9.8 23.6 101.7 13.73 0.086 10.6 21.9 92.9 EXL3 4.0 bpw + 2/2 KV 36.26 1.364 7.6 25.7 104.7 46.43 1.394 8.2 24.0 95.6 llama.cpp Q4 K M + q8 0 KV 10.59 0.027 14.0 12.7 63.1 13.17 0.027 15.3 11.5 57.2 llama.cpp Q4 K M + q4 0 KV 10.88 0.062 9.8 14.4 64.6 13.94 0.110 10.5 13.0 58.7 llama.cpp Q3 K M + q8 0 KV 11.13 0.073 13.1 13.0 64.2 13.75 0.092 14.4 11.9 58.1 llama.cpp Q3 K M + q4 0 KV 11.46 0.109 8.9 15.0 65.9 14.64 0.172 9.6 13.5 59.6 QServe W4A8KV4 11.34 0.105 10.3 11.2 42.6 14.67 0.163 11.2 9.9 37.2

10.65 10.50 10.35 0.00

Resident 2/2, 4-token groups

PPL

These comparisons include the four configurations evaluated in Section 4.3. Table A35 reports all four weight/KV refinement combinations at each resident format; Figure 12 shows the YAQA 3-bit, 3/3 comparison.

Qwen3-8B

Resident 2/2, 2-token groups

PPL

Figure A6 varies fetched key/value bitwidths and the selected token-group fraction for two- and four-token groups. +4/+4 performs well across evaluated fetch budgets and resident precisions, supporting its use in our main comparisons.

PPL

C.6

10.30 10.28 0.00

0.02

0.04

0.06

12.72 12.70 12.68 12.65 0.00

0.02

Fetched refinement payload (bytes per cached K/V element pair) Base +1/+0 +2/+0 +3/+0 +4/+0

+0/+1 +1/+1 +2/+1 +3/+1 +4/+1

+0/+2 +1/+2 +2/+2 +3/+2 +4/+2

+0/+3 +1/+3 +2/+3 +3/+3 +4/+3

+0/+4 +1/+4 +2/+4 +3/+4 +4/+4

Figure A6. KV refinement payload versus PG-19 PPL at 32K: left, Llama-3.1-8B-Instruct; right, Qwen3-8B. Rows compare resident 2/2, 3/2, and 3/3 with two- and four-token groups. Curves vary fetched key/value bits independently from 0 to 4, excluding 0/0; zero leaves that component unrefined. Color identifies fetched key bits; darker shades indicate more fetched value bits. Each curve shows all registered selected-group fractions from 1/256 to 1/16. Payload bytes per cached K/V element pair equal fetched key plus value bits, multiplied by the selected fraction and divided by eight. Dashed lines mark unrefined bases.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference Table A34. Joint refinement: static weight/KV pairs at 128K: PG19 PPL/KL, GPU memory (GB), and RTX 5090 tokens/s. GB counts stored weights and KV. Neither weight nor KV refinement is enabled. Weight base KV configuration PPL BF16

16/16

Weights

StrataFlux L3.1-8B-I Qwen3-8B K/V Weight KV PPL KL GB tok/s PPL KL GB tok/s

YAQA 3-bit

3/2

YAQA 3-bit

3/3

YAQA 4-bit

2/2

YAQA 4-bit

3/2

YAQA 4-bit

3/3

L3.1-8B-I Qwen3-8B KL GB tok/s PPL KL GB tok/s

10.32 0.000 33.2 47.6 12.89 0.000 35.7 OOM

YAQA w2

KIVI-2 18.55 0.645 KIVI-4 15.97 0.477 KVQuant 2bit-1% 19.79 0.708 KVQuant 3bit-1% 16.64 0.521 KVQuant 4bit-1% 16.08 0.486 KVarN 2/2 18.67 0.656 KVarN 4/2 16.18 0.494

7.1 9.2 6.5 7.6 8.6 6.4 7.5

32.9 35.2 13.7 13.2 12.1 84.7 74.7

18.84 0.403 7.9 16.59 0.257 10.3 18.92 0.414 7.2 17.09 0.290 8.4 16.63 0.262 9.6 19.48 0.434 7.1 16.83 0.271 8.3

29.4 31.5 14.2 13.5 12.2 76.6 67.5

YAQA w3

KIVI-2 11.62 0.130 8.0 KIVI-4 10.90 0.059 10.1 KVQuant 2bit-1% 12.41 0.192 7.4 KVQuant 3bit-1% 11.17 0.084 8.4 KVQuant 4bit-1% 10.95 0.064 9.5 KVarN 2/2 11.75 0.146 7.3 KVarN 4/2 10.98 0.068 8.4

32.4 34.6 14.2 13.6 12.5 81.7 72.4

14.60 0.155 8.7 13.42 0.049 11.2 14.98 0.196 8.1 13.68 0.081 9.3 13.46 0.054 10.5 15.15 0.185 8.0 13.59 0.059 9.2

29.1 31.1 14.3 13.6 12.3 74.4 65.7

YAQA w4

KIVI-2 11.17 0.079 8.8 KIVI-4 10.54 0.018 11.0 KVQuant 2bit-1% 11.88 0.140 8.2 KVQuant 3bit-1% 10.77 0.041 9.3 KVQuant 4bit-1% 10.58 0.023 10.4 KVarN 2/2 11.24 0.090 8.2 KVarN 4/2 10.60 0.025 9.3

31.9 34.1 14.2 13.7 12.4 79.0 70.3

14.01 0.118 9.6 13.04 0.017 12.0 14.51 0.174 8.9 13.33 0.051 10.1 13.08 0.023 11.3 14.58 0.143 8.9 13.18 0.026 10.1

28.6 30.6 14.2 13.5 12.2 71.8 63.7

KIVI-2 15.26 0.427 8.0 KIVI-4 13.70 0.306 10.1 KVQuant 2bit-1% 16.52 0.506 7.4 LNQ+GQ w3 KVQuant 3bit-1% 14.18 0.342 8.5 KVQuant 4bit-1% 13.81 0.315 9.5 KVarN 2/2 15.39 0.442 7.3 KVarN 4/2 13.80 0.317 8.4

33.7 36.2 14.4 13.8 12.6 90.9 79.9

20.27 0.452 8.8 18.20 0.331 11.2 20.93 0.491 8.1 18.78 0.366 9.3 18.34 0.339 10.5 20.76 0.473 8.0 18.38 0.341 9.2

29.9 32.3 14.0 13.4 12.2 81.6 71.6

KIVI-2 11.52 0.117 8.9 KIVI-4 10.79 0.046 11.0 KVQuant 2bit-1% 12.34 0.180 8.3 LNQ+GQ w4 KVQuant 3bit-1% 11.04 0.068 9.3 KVQuant 4bit-1% 10.84 0.050 10.4 KVarN 2/2 11.62 0.129 8.2 KVarN 4/2 10.84 0.053 9.3

33.0 35.5 14.5 13.9 12.6 86.1 76.1

14.40 0.143 9.6 13.30 0.040 12.1 14.83 0.189 9.0 13.60 0.073 10.2 13.35 0.046 11.4 14.91 0.167 8.9 13.46 0.050 10.1

29.5 31.7 14.4 13.7 12.4 77.8 68.7

LNQ+GQ 3-bit 2/2

LNQ+GQ 3-bit 3/2

LNQ+GQ 3-bit 3/3

LNQ+GQ 4-bit 2/2

LNQ+GQ 4-bit 3/2

LNQ+GQ 4-bit 3/3

Table A35. Weight/KV component combinations at 128K: PG-19 PPL/KL, GPU memory (GB), and RTX 5090 tok/s. Each four-row group fixes resident formats and uses the 5% weight configuration. All groups use the same CUDA attention implementation. GB counts weights and KV. StrataFlux L3.1-8B-I Qwen3-8B K/V Weight KV PPL KL GB tok/s PPL KL GB tok/s – – 18.67 0.656 137.5 19.48 0.434 125.7 ✓ – 15.57 0.458 133.5 17.02 0.311 123.3 6.4 7.1 YAQA 2-bit 2/2 – ✓ 15.95 0.472 109.2 16.66 0.255 99.2 ✓ ✓ 13.79 0.312 106.6 14.91 0.165 97.6 – – 16.42 0.511 134.6 17.06 0.287 123.0 ✓ – 14.15 0.348 130.0 15.23 0.189 120.0 YAQA 2-bit 3/2 7.0 7.7 – ✓ 15.91 0.473 107.2 16.62 0.255 97.4 ✓ ✓ 13.76 0.313 104.2 14.88 0.162 95.4 – – 16.21 0.496 132.5 16.85 0.274 120.8 ✓ – 13.94 0.329 128.1 15.05 0.179 118.3 YAQA 2-bit 3/3 7.5 8.3 – ✓ 15.91 0.472 105.2 16.56 0.253 95.4 ✓ ✓ 13.74 0.311 102.1 14.84 0.161 93.3 – – 11.75 0.146 130.7 15.15 0.185 120.2 ✓ – 11.52 0.124 126.6 14.83 0.167 114.6 YAQA 3-bit 2/2 7.3 8.0 – ✓ 10.96 0.058 105.2 13.48 0.048 95.8 ✓ ✓ 10.80 0.045 102.0 13.25 0.038 91.7 Weights

D.2

– ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓ – ✓

– – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓ – – ✓ ✓

11.06 0.076 128.1 13.71 0.071 117.8 10.89 0.060 123.5 13.46 0.058 112.2 7.8 8.6 10.90 0.058 103.0 13.43 0.046 93.6 10.74 0.044 100.3 13.20 0.034 90.3 10.98 0.067 125.8 13.56 0.061 115.9 10.81 0.051 122.0 13.32 0.049 110.7 8.4 9.2 10.89 0.057 100.8 13.39 0.045 92.0 10.72 0.042 98.3 13.17 0.034 88.2 11.24 0.090 123.9 14.58 0.143 113.8 11.17 0.085 119.8 14.47 0.137 111.1 8.2 8.9 10.60 0.018 100.3 13.10 0.018 91.2 10.55 0.015 97.5 13.04 0.015 90.1 10.66 0.031 121.6 13.31 0.038 111.4 10.61 0.027 117.1 13.23 0.034 108.3 8.7 9.5 10.53 0.017 98.3 13.05 0.014 89.6 10.49 0.014 95.9 12.98 0.011 87.5 10.59 0.025 119.7 13.16 0.028 110.0 10.55 0.021 115.7 13.09 0.025 107.0 9.3 10.1 10.53 0.017 96.6 13.03 0.014 87.9 10.48 0.013 94.6 12.96 0.011 85.8 15.39 0.442 152.2 20.76 0.473 137.6 14.48 0.372 149.4 19.61 0.423 134.6 7.3 8.0 13.68 0.301 119.2 18.42 0.336 107.0 12.98 0.241 117.9 17.46 0.292 105.1 13.96 0.330 150.0 18.61 0.354 135.6 13.23 0.269 147.2 17.63 0.308 132.1 7.9 8.6 13.64 0.302 117.0 18.23 0.330 104.8 12.95 0.242 115.8 17.29 0.286 103.3 13.83 0.318 147.1 18.41 0.343 133.1 13.11 0.258 144.4 17.46 0.298 130.1 8.4 9.2 13.64 0.301 114.1 18.20 0.329 102.6 12.95 0.242 113.1 17.27 0.285 100.6 11.62 0.129 140.2 14.91 0.167 127.6 11.52 0.120 137.7 14.83 0.161 124.6 8.2 8.9 10.84 0.044 111.7 13.36 0.040 101.2 10.77 0.038 110.4 13.30 0.035 99.5 10.90 0.060 138.4 13.59 0.062 125.8 10.82 0.054 135.6 13.53 0.056 122.9 8.8 9.5 10.78 0.044 110.0 13.31 0.038 99.5 10.71 0.038 109.2 13.25 0.032 97.7 10.85 0.053 136.1 13.44 0.052 123.7 10.77 0.046 133.6 13.38 0.047 120.8 9.3 10.1 10.78 0.043 107.7 13.29 0.037 97.4 10.71 0.037 106.8 13.23 0.032 95.5

Full Task Results

Tables A36, A37, and A38 report RULER, LongBench, MATH500, and AIME results for the evaluated full-system configurations. Table A36. Joint refinement at 128K on Llama-3.1-8B-Instruct and Qwen3-8B: RULER/MATH500/AIME accuracy, throughput, and GPU memory. GB includes weights, KV, and DGX Spark's RAM caches. StrataFlux combines the 5% weight configuration with 2/2 KV. DGX Spark L3.1-8B-I Qwen3-8B Configuration R128 tok/s GB R128 M500 AIME tok/s GB BF16 77.9 7.3 33.2 74.1 97.2 71.0 6.7 35.7 StrataFlux (LNQ+GQ 4-bit) 71.1 22.1 8.5 70.6 96.0 65.6 20.8 9.2 StrataFlux (YAQA 4-bit) 73.5 21.4 8.4 70.3 96.2 69.0 20.3 9.1 StrataFlux (YAQA 3-bit) 73.2 23.3 7.6 68.4 95.2 68.8 21.9 8.3 StrataFlux (YAQA 2-bit) 60.5 25.6 6.7 65.2 89.6 40.6 23.9 7.4

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference RTX 5090 L3.1-8B-I Qwen3-8B Configuration R128 tok/s GB R128 M500 AIME tok/s GB BF16 77.9 47.6 33.2 74.1 97.2 71.0 OOM 35.7 LNQ+GQ 4-bit / 2/2 64.3 86.1 8.2 64.0 95.2 58.3 77.8 8.9 StrataFlux (LNQ+GQ 4-bit) 75.2 110.4 8.2 73.8 94.4 61.5 99.5 8.9 YAQA 4-bit / 2/2 67.5 79.0 8.2 65.4 95.0 62.5 71.8 8.9 StrataFlux (YAQA 4-bit) 77.2 97.5 8.2 73.5 96.4 69.2 90.1 8.9 YAQA 3-bit / 2/2 67.2 81.7 7.3 63.8 94.6 59.6 74.4 8.0 StrataFlux (YAQA 3-bit) 77.1 102.0 7.3 71.2 95.4 65.6 91.7 8.0 YAQA 2-bit / 2/2 50.4 84.7 6.4 58.8 68.0 11.0 76.6 7.1 StrataFlux (YAQA 2-bit) 63.8 106.6 6.4 68.3 91.2 47.1 97.6 7.1 llama.cpp Q4 K M + q8 0 75.3 63.1 14.0 72.0 95.6 71.9 57.2 15.3 EXL3 4 bpw + 4/4 75.5 101.7 9.8 71.6 94.4 66.9 92.9 10.6 vLLM AWQ4 + FP8 KV 74.7 106.8 14.3 70.0 95.4 68.3 96.6 15.8

Table A37. Joint refinement: DGX Spark task grid. R4–R128 give RULER accuracy at 4K–128K; LB/M500 report LongBench/MATH500 accuracy. LongBench prompts are truncated to 130,432 tokens here, versus 32,128 in the KV-only comparisons. Throughput uses 128K context. Each pair uses the same weight base and resident 2/2 KV; Base disables both refinements. Configurations match the main full-system table. DGX Spark / Llama-3.1-8B-Instruct Weights Method R4 R8 BF16 95.7 95.1 llama.cpp Q4 K M + q8 0 95.5 95.0 EXL3 4 bpw + 4/4 95.3 94.7 vLLM AWQ4 + FP8 KV 95.1 94.1 Base 93.7 92.2 LNQ+GQ 4-bit StrataFlux 95.4 94.2 Base 94.2 93.6 YAQA 4-bit StrataFlux 95.9 94.9 Base 93.8 92.8 YAQA 3-bit StrataFlux 95.0 94.7 Base 91.6 90.5 YAQA 2-bit StrataFlux 93.9 93.2

Table A38. Joint refinement: RTX 5090 task grid. R4–R128 give RULER accuracy at 4K–128K; LB/M500 report LongBench/MATH500 accuracy. LongBench prompts are truncated to 130,432 tokens here, versus 32,128 in the KV-only comparisons. Throughput uses 128K context. Each pair uses the same weight base and resident 2/2 KV; Base disables both refinements. Configurations match the main full-system table. RTX 5090 / Llama-3.1-8B-Instruct Weights Method R4 R8 BF16 95.7 95.1 llama.cpp Q4 K M + q8 0 95.5 95.0 EXL3 4 bpw + 4/4 95.3 94.7 vLLM AWQ4 + FP8 KV 95.1 94.1 Base 93.7 92.2 LNQ+GQ 4-bit StrataFlux 95.4 94.3 Base 94.2 93.6 YAQA 4-bit StrataFlux 95.9 94.7 Base 93.8 92.8 YAQA 3-bit StrataFlux 94.9 94.7 Base 91.6 90.5 YAQA 2-bit StrataFlux 94.1 92.9 RTX 5090 / Qwen3-8B Weights Method

R16 92.9 93.2 92.6 92.1 89.9 92.2 91.2 93.1 90.4 92.6 87.1 90.9

R32 87.1 87.8 88.7 87.6 84.0 86.1 86.0 87.3 85.7 86.9 80.4 85.5

R64 R128 LB tok/s 85.8 77.9 49.6 7.3 85.3 75.3 49.1 12.7 84.9 75.5 48.1 23.6 83.2 74.7 48.4 16.8 79.8 64.3 47.9 21.3 84.0 71.1 48.3 22.1 80.3 67.5 48.7 21.2 84.5 73.5 49.0 21.4 80.6 67.2 46.8 23.0 84.0 73.2 47.0 23.3 73.0 50.4 46.2 24.8 80.0 60.5 46.5 25.6

R4

BF16

95.2 93.8 93.8 92.1 80.6 74.1 49.4 97.2

71.0

6.7

llama.cpp Q4 K M + q8 0 95.0 93.9 93.4 91.9 79.5 72.0 49.2 95.6 EXL3 4 bpw + 4/4 94.8 94.2 93.7 90.8 78.5 71.6 48.1 94.4 vLLM AWQ4 + FP8 KV 94.8 93.9 93.0 91.0 78.8 70.0 48.0 95.4

71.9 66.9 68.3

11.5 21.9 15.3

LNQ+GQ 4-bit

Base 94.0 92.9 91.3 87.3 72.4 64.0 48.1 95.2 StrataFlux 95.5 94.2 93.8 90.8 78.8 70.6 48.4 96.0

58.3 65.6

19.5 20.8

YAQA 4-bit

Base 94.5 93.7 92.1 87.9 71.1 65.4 48.4 95.0 StrataFlux 95.5 94.5 93.2 91.3 76.4 70.3 49.0 96.2

62.5 69.0

19.5 20.3

YAQA 3-bit

Base 94.4 93.4 92.2 86.6 71.1 63.8 48.1 94.6 StrataFlux 95.3 94.6 93.7 91.4 76.0 68.4 48.9 95.2

59.6 68.8

20.9 21.9

YAQA 2-bit

Base 92.9 90.9 87.2 77.8 63.8 58.8 43.3 68.0 StrataFlux 94.4 93.0 90.8 83.6 70.2 65.2 44.1 89.6

11.0 40.6

22.5 23.9

R8 R16 R32 R64 R128 LB M500 AIME tok/s

R32 87.1 87.8 88.7 87.6 84.0 86.2 86.0 87.4 85.7 86.4 80.4 84.2

R64 R128 LB tok/s 85.8 77.9 49.6 47.6 85.3 75.3 49.1 63.1 84.9 75.5 48.1 101.7 83.2 74.7 48.4 106.8 79.8 64.3 47.9 86.1 84.7 75.2 48.4 110.4 80.3 67.5 48.7 79.0 85.3 77.2 49.1 97.5 80.6 67.2 46.8 81.7 83.8 77.1 47.0 102.0 73.0 50.4 46.2 84.7 81.4 63.8 46.5 106.6

R8 R16 R32 R64 R128 LB M500 AIME tok/s

llama.cpp Q4 K M + q8 0 95.0 93.9 93.4 91.9 79.5 72.0 49.2 95.6 EXL3 4 bpw + 4/4 94.8 94.2 93.7 90.8 78.5 71.6 48.1 94.4 vLLM AWQ4 + FP8 KV 94.8 93.9 93.0 91.0 78.8 70.0 48.0 95.4

71.9 66.9 68.3

57.2 92.9 96.6

LNQ+GQ 4-bit

Base 94.0 92.9 91.3 87.3 72.4 64.0 48.1 95.2 StrataFlux 95.4 94.0 93.8 91.5 81.0 73.8 48.7 94.4

58.3 61.5

77.8 99.5

YAQA 4-bit

Base 94.5 93.7 92.1 87.9 71.1 65.4 48.4 95.0 StrataFlux 95.5 94.5 93.4 91.1 79.5 73.5 49.0 96.4

62.5 69.2

71.8 90.1

YAQA 3-bit

Base 94.4 93.4 92.2 86.6 71.1 63.8 48.1 94.6 StrataFlux 95.4 94.4 94.0 91.3 78.9 71.2 48.8 95.4

59.6 65.6

74.4 91.7

YAQA 2-bit

Base 92.9 90.9 87.2 77.8 63.8 58.8 43.3 68.0 StrataFlux 94.3 92.7 91.2 85.6 72.3 68.3 44.5 91.2

11.0 47.1

76.6 97.6

E DGX Spark / Qwen3-8B Weights Method

R4

R16 92.9 93.2 92.6 92.1 89.9 91.7 91.2 93.5 90.4 93.0 87.1 91.4

A PPLICABILITY TO OTHER KV BASES

The main KV implementation adopts KVarN’s quantization format. Other bases require a refinement encoding and decoder that combine their resident codes with separately stored refinement bits. The selection of token groups and the full-sequence attention update remain the same, but the refinement levels must match the base’s quantization bins— the value ranges mapped to each resident code—and its coordinate transformations and metadata. Nonuniform quantization bins. Our KVQuant-style adaptation starts from sensitivity-weighted nonuniform codebooks (Hooper et al., 2024) and fits finer levels within each bin using sensitivity-weighted calibration samples. Our TurboQuant adaptation uses its scalar quantizer optimized for mean squared error in the rotated, normalized coordinate system (Zandieh et al., 2026); each bin is refined using levels derived from the base’s coordinate distribution restricted to that bin. Both keep the resident code as the identifier of the coarse bin and store the finer-level index separately.

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

Evaluation scope. Table A39 compares Base and StrataFlux-KV reconstruction accuracy for the KVQuant and TurboQuant adaptations. Table A39. KV-only refinement: nested refinements for other resident formats, PG-19 at 128K. Base and StrataFlux-KV rows report PPL/KL side by side. Refinement uses +4/+4 bits, two-token groups, and a 1/128 fetch fraction. Resident base KVQuant 2-bit + 1% outliers KVQuant 3-bit + 1% outliers KVQuant 4-bit + 1% outliers TurboQuant 2.5-bit TurboQuant 3.5-bit

F

Method Base StrataFlux-KV Base StrataFlux-KV Base StrataFlux-KV Base StrataFlux-KV Base StrataFlux-KV

L3.1-8B-I PPL KL 11.71 0.125 10.46 0.012 10.56 0.024 10.35 0.003 10.38 0.007 10.33 0.001 10.98 0.055 10.47 0.005 10.43 0.010 10.34 0.001

Qwen3-8B PPL KL 14.40 0.167 13.07 0.021 13.21 0.041 12.92 0.005 12.96 0.011 12.89 0.002 36.02 1.115 14.03 0.091 13.99 0.102 12.89 0.004

M AC E VALUATION

Table A40 compares weight-only, KV-only, and joint refinement on a MacBook Air M3 (16 GB, Metal), using Llama-3.1-8B-Instruct with llama.cpp's Q3 K M weights and KVarN 2/2 KV. Table A40. MacBook Air M3 weight/KV refinement: WikiText-2 PPL/KL on one sequentially scored 16K-token window. Throughput averages 30 paired 32-token blocks at the listed initial contexts. Configuration Base StrataFlux-W StrataFlux-KV StrataFlux

StrataFlux Weight KV – – ✓ – – ✓ ✓ ✓

PPL 6.71 6.66 6.44 6.38

KL 0.126 0.118 0.081 0.073

Throughput (tok/s) Empty context 16K context 14.1 10.6 15.1 11.3 13.8 9.9 14.4 10.5

Weight refinement selects k = 2 channels per 1,024 in all four projection sets. Both refinement components use separate 128 MiB SSD-backed RAM caches; KV refinement uses the main unified-memory settings.

G

PARALLEL W EIGHT R EFINEMENT WITH CUDA S TREAMS

Separate CUDA streams do not guarantee parallel execution. If the base projection starts first, its large grid can occupy all streaming multiprocessors (SMs), leaving refinement queued until an SM becomes available. Merely assigning refinement to another stream does not prevent this serialization. DecDEC’s projection-local overlap. DecDEC (Park et al., 2025) keeps refinement on the current stream and places the base projection on another stream behind an event wait.

The intent is for faster same-stream scheduling to start refinement before the cross-stream base projection, including during CUDA Graph replay. Refinement’s small grid leaves SMs idle, allowing the scheduler to place the projection’s thread blocks on those SMs while refinement runs. The event is recorded before refinement, so the cross-stream dependency does not wait for refinement to finish. A second event joins the projection back into the current stream before the output is consumed; that stream’s own ordering also ensures refinement is complete. Both kernels add to the output atomically. The snippets show the stream/event structure captured in the CUDA Graph, with kernel arguments and event/stream allocation omitted. cudaEventRecord(fork, current); refine<<<small_grid,threads,0,current>>>(...); cudaStreamWaitEvent(other, fork, 0); base_gemv(..., other); cudaEventRecord(done, other); cudaStreamWaitEvent(current, done, 0);

StrataFlux’s fork-and-continue scheduling. We continue the graph on another stream while the fetch-lead or fetch-lag kernel remains in flight on the previous execution stream. Each outstanding operation j has its own events and continuation stream. The graph waits for its completion only where its result is needed, allowing projection sets with different shift distances s to overlap independently. For fetch lead, when the projection’s input activations become available, we fork again: the refinement branch waits for the earlier fetch and applies refinement using those activations, while the base projection proceeds on another stream. The branches join before the projection output is consumed. Fetch lag instead joins before the deferred residual update is consumed. Contributions use atomic additions to initialized accumulators. cudaEventRecord(fork[j], current); fetch_or_refine<<<small_grid,threads,0,current>>>(...); cudaEventRecord(done[j], current); cudaStreamWaitEvent(next[j], fork[j], 0); current = next[j]; // Where j’s result is consumed: cudaStreamWaitEvent(current, done[j], 0);

Related documents

• A Target-Centric Survey of Quantization-Aware Training#798303
• A Target-Centric Survey of Quantization-Aware Training#796890
• TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference#1157034
• PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference#373478
• HCCL: Collective Communication for Meta Training and Inference Accelerators#423914