Scaled Outer Product: A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models
arXiv:2605.14929v1 [cs.LG] 14 May 2026
Earl Killian May 15, 2026
Abstract Scaled Outer Product (SOP) is a post-training quantization methodology for large language model weights, designed to deliver near-lossless fidelity at 4.5–6 bits per weight on hardware with per-layer LUT decode. The methodology combines per-layer search of fixed and dynamic codebook pairs selected by a per-block selection bit, signed per-block scales, activation-weighted cosine selection, and multiple-choice knapsack promotion of sensitive layers with outlier and sparse-residual correction. Fixed codebooks include NF4, BOF4, Split87, and SH4; per-layer optimized codebooks (DD4) are hosted in LUT SRAM. A new hardware-efficient LUT output format (HIF) is proposed to improve performance, energy, and cost. Across six open model families, the recommended FP6 operating point (E2M3sUE4M4, 6.5 bpw) achieves lower weight reconstruction error than the conventional per-layer-POT FP8 baseline (E4M3, 8.0 bpw) at 1.5 bpw lower storage cost, demonstrating that block-scaled small atoms with carefully chosen scale precision can replace conventionally-deployed FP8. Full evaluation across the 4.5–6 bpw range, including layer promotion and sparse residual correction, is reported in a companion paper.
1
Overview
The methodology rests on seven interlocking ideas, developed in the sections that follow: 1. flexible block scaling, parameterized by block size (g ≥ 1, typically g ∈ {8, 16, 32, 64}) and scale-format width (8 to 16 bits, signed or unsigned), with optional reuse of scale-factor bits as codebook-selection metadata (§7); 2. an activation-weighted cosine similarity (ACos) that better predicts downstream perplexity than per-element error in the regimes where it matters for allocation; 3. per-input-channel importance weights (channel norms) derived from a small calibration corpus; 4. a compact alphabet of fixed-ROM atoms (Gaussian-quantile NF4, sinh-grid SH4, Lloyditerated BOF4) and the data-adaptive DD4, with sign-negated variants of each atom available for the pair search. The block size g=16 makes the codebook widths n ∈ {4, 5, 6} all pack to byte-aligned payloads per-block (2n bytes for codes plus the scale word). n=4 is the default; n=5 a 32-entry extension; 5. a per-layer pair search that selects, for each layer, the best two-codebook decomposition from this alphabet, with a one-bit per-block selector indicating which codebook reconstructs each block;
1
6. two complementary post-quantization corrections, both managed by the allocator: per-block outlier extraction (OPQ) that pulls high-magnitude weights out of the codebook path and stores them exactly, and an activation-weighted sparse residual (Wr) for the dominant remaining reconstruction errors; and 7. a multiple-choice knapsack allocator that, given a global BPW budget and per-layer promotion profiles (the ACos attainable at each candidate higher-precision promotion format), chooses for each layer which corrections to apply and which format to promote to. A defining feature of the methodology is that the geometry of the codebook, atom selection, partition split, promotion target, and budget allocation are decided per-model and per-layer from the observed weight and activation statistics. No global rule is imposed; the methodology produces a different set of per-layer codebook shapes for every model and every budget. Recent precision-aware scaling laws [Kumar et al., 2024] indicate that the regime of relevance for post-training quantization is shifting: as model training increasingly incorporates low-precision arithmetic, the weight distributions PTQ encounters will themselves reflect low-precision-training dynamics rather than being purely BF16-trained. Methodology that flexibly adapts to per-layer distributional features — rather than committing to a single atom family or scale format — is positioned to absorb this shift; the methodology presented here is designed for that flexibility.
2
Block Scaling
For an LLM linear layer Y = XW ⊤ with activations X ∈ RT ×K and weights W ∈ RM ×K , the block scaling represents X and W in fewer average bits per value by sharing a factor per-block scale across g consecutive elements along the contracted dimension K. With a per-block scale of s bits and a per-element format of b bits, the average is s/g + b bpw. For example, block-scaling along K in groups of g=16 with a 12-bit floating-point scale (e.g. E5M6) and per-element E2M3 (FP6) yields (12 + 16·6)/16 = 6.75 bpw, with a lower KL divergence than direct quantization of FP8 at a lower cost in our evaluation [Killian, 2026a]. Block scaling in this family of formats has been studied at the standards level by the OCP MX specification [Darvish Rouhani et al., 2023] and at the block-size and scale-precision tradeoff axes by Fasoli et al. [Fasoli et al., 2026]. Define quantized tensors QX ∈ QT ×K and QW ∈ QM ×K in target format q, and block scale tensors sX ∈ RT ×K/g and sW ∈ RM ×K/g . The block-scaling reconstruction is X[t, k] ≈ sX [t, ⌊k/g⌋] · QX [t, k],
W [m, k] ≈ sW [m, ⌊k/g⌋] · QW [m, k].
Substituting into the GEMM and grouping the inner sum by K-blocks: K/g−1
Y [t, m] ≈
X b=0
sX [t, b] sW [m, b]
g−1 X
QX [t, b·g + r] QW [m, b·g + r].
r=0
The inner sum over r is a g-step dot product in the quantized domain — no scale factor enters — and the outer sum accumulates K/g scaled block contributions. In matrix form, K/g−1
Y ≈
X
sX [:, b] ⊗ sW [:, b] ⊙ QX [:, Gb ] QW [:, Gb ]⊤ ,
b=0
where Gb = {b·g, . . . , (b + 1)g − 1} indexes the K-block, ⊗ is the rank-1 outer product (a T ×1 column times a 1×M row gives a T ×M scale tile), ⊙ is the Hadamard product, and juxtaposition 2
is the standard matrix product. Each K-block contributes a T ×M matrix product of quantized slices, scaled element-wise by the rank-1 outer product of its per-block scales. This is the form the SOP back-end implements directly (§3). Letting q denote the target format with most-positive value t+ and most-negative value t− , two scaling rules are common. Absmax scaling sets the scale so that no value in the block saturates: maxr X[t, b·g + r] maxr −X[t, b·g + r] abs sX [t, b] = max . , t+ −t− The scale is non-negative. When the format has equal-magnitude endpoints (t+ = −t− ) this reduces to maxr |xr |/t+ . If the format is additionally closed under negation, the sign bit of the scale carries no information and can be freed to serve as a block metadata bit. Equal endpoints alone do not imply closed under negation: the NF4 codebook of QLoRA [Dettmers et al., 2023] spans equal endpoints but contains seven negative entries against eight positive, so negating an interior codeword does not generally produce another codeword. For such formats, allowing negative scale factors gives the reconstruction two effectively-different codebooks per block, at the cost of dedicating the sign bit to that selection. Argmax scaling preserves the magnitude of the dominant excursion by mapping it to the format’s larger-magnitude endpoint. Letting r∗ = arg maxr |X[t, b·g + r]|, x∗ = X[t, b·g + r∗ ], and t∗ = arg max(|t+ |, |t− |) (i.e., t∗ = t+ if |t+ | ≥ |t− | and t∗ = t− otherwise), ∗ ∗ sarg X [t, b] = x /t .
The scale’s sign equals the sign of x∗ when t∗ is positive and the opposite when t∗ is negative; either way, x∗ reconstructs to t∗ exactly and the full asymmetric range of the codebook is used. An asymmetric (not closed under negation) or paired codebook reads the scale sign at the matrixunit boundary and applies the polarity flip, preserving the dynamic-range advantage that absmax discards. In either case, X[t, k] QX [t, k] = roundq , sX [t, ⌊k/g⌋] and analogously for sW and QW . SOP supports both rules; the choice is part of the per-layer format specification. When the scale format provides no spare bit for metadata (e.g., a bytealigned unsigned 8-bit scale), argmax requires sacrificing the metabit; SOP’s 12-bit scale format provides both a signed scale and a separate metabit in a single field (§7).
3
Scaled Outer Product
The block-scaled GEMM of §2 is mapped onto the matrix unit as the sum of the scaled rank-1 outer products. For each output tile Y [T , M] of size Tr × Mr (where T and M index a contiguous block of tokens and output features respectively), the matrix unit iterates: for b = 0, . . . , K/g − 1 t ← 0 ∈ RTr ×Mr for r = 0, . . . , g − 1 t += QX [T , b·g + r] ⊗ QW [M, b·g + r] Y [T , M] += t ⊙ sX [T , b] ⊗ sW [M, b]
(K-block loop) (zero block accumulator) (in-block dot-product) (quantized rank-1 outer product) (rank-1 scaled accumulate)
The inner loop accumulates g rank-1 outer products in the quantized domain into a tile-sized partial result t, without the inclusion of the scale factor. At each K-block boundary, t is multiplied 3
element-wise by a rank-1 scale tile constructed from the column of sX at tokens T and the column of sW at output features M, then added to the running output. This is the entire SOP micro-kernel. It reduces block-scaled GEMM to two primitives: rank-1 quantized outer product (executed g times per K-block) and rank-1 Hadamard scale-and-accumulate (executed once per K-block). The Per-block scales are loaded once per K-block — a factor of g less bandwidth than quantized weights and activations — so the scale-load overhead remains small even at small g. The SOP back-end implements this micro-kernel with a 128 × 128 matrix unit using Fully Complementary SRAM (FC-SRAM), a memory-in-compute fabric [Doluca and Riordan, 2021]. Integration with standard high-performance GEMM cache-blocking is detailed in the SOP hardware specification [Killian, 2026b]. FC-SRAM is not required to implement SOP, but hosting the 256 small LUTs (e.g., 32 × 8) in other memory technology (foundry 6T SRAM, register files, or flip-flop arrays) results in a significant increase in gate count and area. The same matrix-form equation also exposes the data-reuse structure that makes high-throughput implementation possible. The rank-1 outer product is the matrix unit’s primitive: at each Kelement step, Tr activation values multiplied with Mr weight values produce Tr · Mr MAC outputs. Each loaded QX value participates in Mr MAC operations and each loaded QW value in Tr . At the SOP matrix-unit dimensions Tr = Mr = 128, each operand load amortizes over 128 MACs — the bandwidth-amplification factor that lets the matrix unit run at high throughput without saturating memory. Per-block scales follow the same pattern: a Tr +Mr -element scale vector is loaded per K-block, and the Tr × Mr scale tile is constructed in-place as a rank-1 outer product applied to Tr · Mr · g MAC outputs. At Tr = Mr = 128, g = 16, bscale = 12 bits, and bop = 4 bits, the per-K-block bandwidth split is 12·256 = 3072 scale bits against 4·16·256 = 16,384 operand bits, so scales account for 3072/19,456 ≈ 15.8% of operand traffic. Per scale value, reuse over Tr · g or Mr · g MACs is a factor of g higher than per-operand reuse, which is what makes the 12-bit scale affordable. Two equivalent reformulations of block-scaled GEMM differ in their hardware implications. The pre-scaling view dequantizes weights element-wise, W [m, k] ≈ sW [m, ⌊k/g⌋] · QW [m, k], then runs a standard GEMM on the dequantized values; this is the natural reading for software-only PTQ. The post-scaling view, expressed by the matrix-form equation above, contracts in the quantized domain and applies scales as a rank-1 tile per K-block; this is what enables the SOP backend to use a low-precision MAC array and a narrow scale-application datapath. Both compute the same Y ; only the second yields a hardware datapath whose MAC-array width matches the quantization width rather than the dequantization width.
4
Channel Norms
For each linear layer, a per-input-channel importance vector c ∈ Rdin is computed by passing a small calibration corpus through the unquantized model and recording the RMS magnitude of activations entering that layer: q cj = Ex∼Dcalib x2j , j = 1, . . . , din . Channel norms are layer-specific and (for practical purposes) stable across reasonable choices of calibration text. The computation is a one-time, model- and calibration-keyed precomputation; no gradient information is required and no labeled data is used. The construction is closely related to the activation-magnitude saliency used by AWQ [Lin et al., 2024], though SOP uses channel norms as importance weights inside a fidelity metric rather than as a basis for scaling weights to protect salient channels.
4
At quantization time, c is tiled across the output dimension and reshaped to per-block weights, yielding the importance vectors cb used in ACos (§5) and Wr (§9).
5
Activation-Weighted Cosine Similarity
b Let a linear layer have weight matrix W ∈ Rdout ×din partitioned into blocks {wb }N b=1 of size g (typically g=16), and let cb be the layer’s per-block channel-norm vector (§4). Define the cb weighted inner product and norm by q X cb,i ai bi , ∥a∥cb = ⟨a, a⟩cb . ⟨a, b⟩cb =
i
The activation-weighted cosine similarity for the layer’s quantized reconstruction Ŵ is then N
b 1 X ⟨wb , ŵb ⟩cb ACos(W, Ŵ ) = ∈ [−1, 1]. Nb ∥wb ∥cb ∥ŵb ∥cb
b=1
For 4-bit pair quantization, ACos values typically fall in [0.997, 0.999]; we report differences in parts per million (ppm) of the gap to unity. ACos was chosen because per-element MSE / SQNR overweight low-impact channels, biasing PTQ allocations toward errors visible in the residual norm but invisible in the model’s output distribution. The empirical advantage is regime- dependent: in the promotion regime — where the allocator picks among layers and formats with substantially different fidelity — ACos correlates more tightly with downstream KL than MSE/SQNR; in the bare regime, all four metrics correlate similarly with KL across the model families we have studied. Quantitative correlation results are reported in a companion paper [Killian, 2026a].
6
Primary Formats
The methodology employs three orthogonal format axes. Codebook bit-width (n). The default is n=4 (16-entry per-layer LUT), giving a base quantization of approximately 4.5 BPW once the per-block scale is amortized over g=16 weights. An n=5 alternative (32-entry LUT, ∼5.5 BPW base) is supported and under investigation; n=3 remains an open problem because no 3-bit codebook geometry has thus far reached useful fidelity without a preprocessing step such as a Hadamard rotation [Ashkboos et al., 2024] or an output-preserving outlier projection. LUT value format. SOP-native deployment uses the HIF7 grid (an 80-value quantization tailored to efficient computation: 80 distinct codepoints) for weights and HIF8 (96 values) for activations. GPU deployment uses standard floating-point grids (E4M3 or E8M7). At n=4 the choice between HIF7 and the 63-value E2M3 grid is empirically nearly indistinguishable in pairing-ACos, so E2M3 is typically used as a publication-friendly substitute. SOP also supports odd-width LUT grids (3, 4, 5, and 6 bits), enabling PTQ at intermediate precisions. Per-block scale format. In addition to the common E8M7 and E4M3/UE5M3 block scale factors, SOP supports 12-bit scale factors with both a sign bit and a metadata bit in a single field. The scale-format axis is rich enough to deserve its own treatment, given in §7. 5
LUT-scale coupling. The LUT value format and the per-block scale format are not independent design parameters: they jointly determine the representable weight range and the precision allocation across the per-block dynamic range. A weight wi = s · LUT[ki ] has its representable range determined by the product of the scale’s range and the LUT’s range; in exponent terms, the weight’s dynamic range is approximately scale.emin + LUT.emin to scale.emax + LUT.emax. Maximizing usable weight range therefore requires maximizing the joint range, not the LUT range or the scale range in isolation. The naive strategy — normalize the LUT to occupy [−LFMT.tmax, +LFMT.tmax] — fails because it leaves the LUT’s smallest non-zero magnitude well above the LFMT’s smallest representable normal value, wasting the LFMT’s denormal-adjacent precision. The methodology instead pins the LUT’s smallest non-zero magnitude to the LFMT’s smallest normal value: given an LFMT ExMy (bias 2x−1 − 1) and a unit-normalized LUT with smallest non-zero magnitude ℓmin , the LUT x−1 is multiplied by 2(2−2 )−exp(ℓmin ) , placing ℓmin at the LFMT’s smallest normal exponent. The unit normalization itself targets |LUT| ≤ 2 − 2−y (the largest representable below 2 at exponent 0), so the LUT spans from the LFMT’s smallest normal to just below 2, leaving exponent 1 and above as headroom that the per-block scale reaches into. Once the LUT is normalized this way, the per-block scale is computed against max(LUT), not against LFMT.tmax: for absmax-scaled blocks, s = |w|max / max(LUT). LUT-format hosting capacity. Different LUT atoms have different intrinsic dynamic ranges max(LUT)/ min|x|>0 |LUT|, and the LFMT’s own normal-range max/min ratio bounds which atoms can be hosted without resorting to subnormal values. Atoms whose dynamic range exceeds the LFMT’s normal-range capacity can sometimes be hosted via subnormal extension at a quality cost from subnormal rounding noise. Table 1: LUT-format hosting capacity for the methodology’s atom alphabet.
†
LUT atom
ratio
E2M3 (7.5)
HIF7 (120)
E4M3 (∞† )
NF4 Split87 SH4 MPO2
12.6 18.3 26.8 64.0
✓ * * ×
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
E4M3’s normal max/min ratio exceeds any atom in the methodology’s alphabet. Asterisk (*) indicates atoms
hosted at E2M3 only via subnormal range. The complete table including E3M3, HIF8, and the n=5 atoms appears in Appendix B.
Split87 and MPO2 are as specified in [Egiazarian et al., 2026]. SH4 is built from the sinh-based companding function [Gersho and Gray, 1992] g = sinh(α · t) + c for t uniformly spaced in [−1, 1], with α = 1.50 and c = −0.02.1 HIF7/HIF8 hardware rationale. The HIF7 and HIF8 grids are designed jointly with the SOP datapath, which computes weight-activation products via shifted product accumulation rather than a fully general FP unit. Each grid value decomposes as a 5-bit two’s complement coefficient and 1
Application of this construction to LLM weight quantization was communicated to the author by Dan Alistarh (personal communication, May 2026). Alistarh subsequently identified α = 1.43 as optimal across bit-widths in his independent analysis; we use α = 1.50 throughout this paper and leave a comparison to future work.
6
a per-element power-of-two shift amount; the integer accumulator update for a weight-activation pair (w, a) at output position (i, j) is iacc[i, j] += (wtc5[i] · atc5[j]) ≪ (wsa[i] + asa[j]), where wtc5 and atc5 are the 5-bit two’s complement weight and activation coefficients, and wsa ∈ {0, 1, 2, 3} (2-bit) and asa ∈ {0, 1, 2, 3, 4} are the per-element shift amounts. The shift sum is therefore 0–7, three levels of logic. Activation shift values 5–7 are available via an RTL option for extended-range deployments, extending the shifter at every output position from three to four levels of logic. The asymmetric shift budget between weights (0–3) and activations (0–4) reflects an architectural reality: activation outliers, produced at runtime by the model’s own computation, have no software-side escape valve, while weight outliers can be addressed by methodology mechanisms (§8, §9); Appendix A discusses the asymmetry and its hardware cost in detail. The resulting grid sizes are 80 values for HIF7 and 96 for HIF8, with normal-range max/min ratios of 120 and 240 respectively. These ranges are sized to host the methodology’s full atom alphabet (NF, BOF, SH, DD, MPO2 at both n=4 and n=5) without subnormal usage on the weight side, and to absorb activation outliers without the methodology’s outlier-handling apparatus on the activation side. Empirical HIF7 fidelity. The HIF7 grid hosts the methodology’s atom alphabet without subnormal usage and exposes the per-block scale format as the remaining precision axis. 2 reports mean per-element reconstruction MSE for HIF7 across six model families and seven candidate scale formats spanning 8-, 10-, and 12-bit containers, with E8M7 (BF16) as a 16-bit reference floor. All measurements use block size g=16 with absmax-derived per-block scales quantized into the indicated scale format under the per-layer Flayer integer-shift search of §7, and weights snapped to the HIF7 grid in the scaled domain. The MSE is unweighted (no calibration data required) and averaged across all linear-layer weight tensors per model. Table 2: HIF7 reconstruction MSE across six model families and seven scale formats. Units of 10−7 , block size g=16, Flayer search active. 8-bit
†
10-bit
12-bit
16-bit
Model
UE4M4 E4M3 UE4M6 E4M5 UE5M7 E5M6 E4M7 E8M7
Gemma-3-1B SmolLM3-3B Llama-3.2-3B Qwen3.5-4B Mistral-7B† Qwen3-8B
3.27 1.74 2.04 0.65 0.05 3.83
3.92 2.08 2.43 0.77 0.06 4.57
2.95 1.58 1.85 0.59 0.05 3.46
3.02 1.61 1.89 0.60 0.05 3.54
2.94 1.57 1.84 0.59 0.05 3.45
2.95 1.58 1.85 0.59 0.05 3.46
2.94 1.57 1.84 0.59 0.05 3.45
2.94 1.57 1.84 0.59 0.05 3.45
Mistral-7B-v0.3 exhibits MSE approximately 30–70× lower than the other models across every scale format. Two
compounding empirical signatures attend the anomaly: a higher mean Flayer shift (k∗ ≈7.3 vs. 4.3–5.7 for the others), and a lower per-layer max-to-mean MSE ratio (3.4× vs. 4.6–6.4× at HIF7 for the remaining models). The combination is consistent with systematically smaller and more uniform per-block weight magnitudes than the other BF16-released models studied here, a tensor-statistical property rather than a format artifact. The scale-format ordering is identical to the other models; only the magnitude floor is lower.
Two patterns hold across all six model families. First, unsigned scales weakly dominate signed 7
scales at narrow widths, with the margin diminishing as width increases. At 8-bit width, UE4M4 beats E4M3 by 17–20%; at 10-bit, UE4M6 beats E4M5 by 2–7%; at 12-bit, the two families are indistinguishable. The opportunity cost of the sacrificed mantissa bit is highest at the tightest budget and vanishes once mantissa headroom is generous. The 12-bit column further disentangles two effects that the narrower widths conflate: the sign-bit-waste comparison (UE5M7 vs. E4M7, both y=7) is a tie to three significant figures, while the mantissa-for-exponent trade (E4M7 vs. E5M6, both signed) costs ∼0.4% per dropped mantissa bit. Second, scale precision saturates by 12 bits. In every model row, the 12-bit scale formats and the 16-bit E8M7 reference agree to three significant figures, and the residual MSE is the HIF7 gridquantization floor itself. Further scale precision does not reduce it. This is the empirical basis for the 12-bit S1E5M5 recommendation in §7. HIF7–E2M3 substitutability. The grid-substitutability claim made earlier in this section admits direct measurement. At the recommended UE4M4 scale, E2M3 reconstruction MSE exceeds HIF7’s by 3.8–4.1% across all six model families: Table 3: HIF7–E2M3 substitutability at the recommended UE4M4 scale. Model
HIF7sUE4M4 (8.5 bpw)
E2M3sUE4M4 (6.5 bpw)
Penalty
Gemma-3-1B SmolLM3-3B Llama-3.2-3B Qwen3.5-4B Mistral-7B Qwen3-8B
3.27 × 10−7 1.74 × 10−7 2.04 × 10−7 6.49 × 10−8 5.16 × 10−9 3.83 × 10−7
3.41 × 10−7 1.81 × 10−7 2.11 × 10−7 6.74 × 10−8 5.36 × 10−9 3.98 × 10−7
4.1% 3.9% 3.8% 3.8% 3.8% 4.0%
The penalty is remarkably uniform across model families (3.8–4.1%), and considerably smaller than uniform quantization theory would predict for the codebook-size reduction 80 → 63. Uniform 1/N 2 scaling would imply a (80/63)2 − 1 ≈ 61% MSE penalty; the observed ∼4% is ∼15× smaller. The reason is structural: HIF7’s 17 codepoints absent from E2M3 are positioned to support the shift-add datapath rather than to minimize MSE on Gaussian-distributed weights. E2M3 captures the MSE-relevant subset of HIF7 efficiently, supporting its use as a publication-friendly substitute throughout the methodology literature without compromising the empirical conclusions drawn from it. Block-scaled FP6 versus POT-scaled FP8. The recommended SOP operating point at the FP6 tier is the E2M3 atom hosted in the HIF7 LUT with per-block UE4M4 scaling, written E2M3sUE4M4 (6.5 bpw). The conventional FP8 deployment baseline is E4M3ˆ0sUE8M0: bare E4M3 weights with a single power-of-two scale per layer (8.0 bpw). Table 4 reports size-weighted weight MSE for both across six models.
8
Table 4: Block-scaled FP6 versus per-layer-POT FP8 across six models. Model Gemma-3-1B SmolLM3-3B Llama-3.2-3B Qwen3.5-4B Mistral-7B Qwen3-8B
E2M3sUE4M4 (6.5 bpw, per-block)
E4M3ˆ0sUE8M0 (8.0 bpw, per-layer POT)
3.41 × 10−7 1.81 × 10−7 2.11 × 10−7 6.74 × 10−8 5.36 × 10−9 3.98 × 10−7
4.40 × 10−7 2.35 × 10−7 2.72 × 10−7 8.69 × 10−8 6.89 × 10−9 5.13 × 10−7
The narrower weight format paired with a per-block mantissa-bearing scale outperforms the wider weight format paired with a per-layer power-of-two scale, at 1.5 bpw lower storage. The methodology’s choice of block-scaled FP6 over layer-POT FP8 is therefore not a precision compromise: it is a Pareto-optimal operating point along the (storage, fidelity) frontier. Per-layer maxMSE distributions confirm the comparison is honest across layer types: max-to-mean ratios under E4M3ˆ0sUE8M0 match those under per-block mantissa-bearing scale formats (UE4M6, UE5M7) on every model, indicating that the Flayer search of §7 has absorbed the per-layer dynamic-range variation that would otherwise have made per-layer POT scaling fragile on outlier-heavy attention projections. HIF7 packing. HIF7 and HIF8 are primarily LUT output formats, but it is possible to store and compute on them directly. The HIF7 grid contains 80 distinct values, requiring log2 80 ≈ 6.32 bits of information per weight. Current SOP hardware stores them in 8-bit containers, costing 1.68 bpw relative to the algorithmic lower bound. A 7-bit packing (128-value container, 80 used) would reduce the overhead to 0.68 bpw and align with half-byte memory boundaries. Throughout this paper HIF7 storage cost is reported at the deployed 8-bit container size.
7
Scale Format and Metabit Allocation
The per-block scale format carries two affordances beyond the scalar magnitude: a sign bit that enables per-block polarity flip on asymmetric atoms, and one or more metabits used by the pair search (§10) to record FMTa/FMTb selection or other per-block metadata. At narrow scale widths these affordances compete; at 12 bits they coexist. The SwExMy notation. A scale word is denoted SwExMy in a b-bit container, with w ∈ {0, 1} sign bits, x exponent bits, y mantissa bits, and m = b−w−x−y per-block metabits. Conventionally we write UExMy for S0ExMy (unsigned) and bare ExMy for S1ExMy (signed). Containers are byteor half-byte-aligned: b ∈ {8, 12, 16}. Existing support for 5-bit weight formats could host scales as well; §6 shows it approaches b = 12 quality at intermediate bpw cost. Bit placement. Sign and metabits are placed at the ends of the word, with the magnitude (exponent followed by mantissa) occupying the contiguous middle. When w = 1, bit b−1 holds the sign; all m metabits sit at the LSB end (bits m−1 down to 0). When w = 0 and m ≥ 1, bit b−1 is repurposed as the first metabit, and any remaining m−1 metabits sit at the LSB end (bits m−2 down to 0). Mantissa LSBs displaced by metabits are wired to zero in the magnitude path; the AND-mask cost in hardware is negligible.
9
Worked examples. Format
b
w
x
y
m
Layout (MSB→LSB)
E4M3 UE4M3 E5M6 S1E5M5 S0E6M5 S1E5M4 S0E5M5
8 8 12 12 12 12 12
1 0 1 1 0 1 0
4 4 5 5 6 5 5
3 3 6 5 5 4 5
0 1 0 1 1 2 2
s eeee mmm µ eeee mmm s eeeee mmmmmm s eeeee mmmmm µ µ eeeeee mmmmm s eeeee mmmm µµ µ eeeee mmmmm µ
Per-layer scale shift (Flayer ). The chosen scale format’s representable exponent range is fixed, but the distribution of per-block scale magnitudes scp = |w|max,p /LUTmax varies per layer and is generally not centered in the format’s normal range. A narrow-exponent scale format applied directly would either clamp high-magnitude scales (loss of dynamic range) or push low-magnitude scales into subnormal territory (loss of precision), depending on where the layer’s distribution falls relative to the format’s normal-range bracket. The Flayer search resolves this by introducing a ∗ per-layer integer shift k ∗ ∈ Z applied to all block scales before quantization: scp 7→ scp · 2k , with k ∗ chosen to maximize the fraction of per-block scales landing in the scale format’s normal range. ∗ At dequantization, the shift is undone by multiplying the recovered scale by 2−k ; k ∗ is stored once per layer at negligible bpw cost. The shift is exact for scale formats closed under multiplication by powers of two — which includes all UExMy and ExMy formats with x ≥ 1, since the shift is absorbed by the exponent field with no mantissa roundoff. For in-capacity cells (those whose scale magnitude fits in the format’s normal range without shift), the search returns k ∗ = 0 and is a no-op. The shift is load-bearing on narrow-exponent scale formats applied to high-dynamic-range layers, where it recovers precision that would otherwise be lost to clamping or subnormal storage. Empirically across the six model families profiled in §6, mean k ∗ ranges from 4.3 to 7.3 for 4-exponent-bit scale formats (E4· / UE4·) and is approximately zero for 5-exponent-bit formats (E5· / UE5·), reflecting that the wider exponent range of E5 absorbs the scale distribution natively while E4 requires the shift to do so. This is the mechanism that allows the 4-exponent-bit scale formats in the table of §6 to track the 5-exponent-bit formats closely at 10- and 12-bit widths. Why 12 bits matters. At an 8-bit scale budget, sign and metabit are mutually exclusive: E4M3 provides the sign bit (and thus per-block polarity flip on asymmetric atoms) but no metabit, while UE4M3 provides one metabit (FMTa/FMTb selection) but no sign. The choice forecloses one of the two affordances. At 12 bits the affordances coexist. S1E5M5 provides both a sign and a metabit on top of an FP11-precision magnitude, suitable for dual-codebook deployment with sign-flip enabled. S0E6M5 trades the sign for slightly wider exponent range in the metabit-only setting. S1E5M4 provides two metabits at the cost of one mantissa bit, suitable for configurations that require richer per-block metadata. The 0.75 bpw extra at g=16 over the 8-bit alternative buys the joint sign + metabit affordance plus mantissa headroom. Tradeoffs. Prior work on microscaling format limits [Fasoli et al., 2026] investigated whether finer per-block scales improve quality along the block-size axis; our finding here addresses the orthogonal scale-precision axis at fixed block size, and concludes that 12-bit signed scales saturate the available headroom. Mantissa-LSB stealing has a sharp cost at 10-bit containers and a mild cost at 12-bit. 10
Across the models we have profiled, dropping from E5M6 (y=6) to S1E5M5 (y=5) is essentially neutral, while dropping to y=4 at the FP10 width costs 4–12% in pair-mean ACos [Killian, 2026a]. The sign-bit-as-metabit trade — µ at bit b−1 when w=0 — is essentially free at any width because the mantissa is unaffected. Sign-flip availability is tied to the deployed scale format, not the pair-search configuration. A pair trained with EM sign-flip enabled but deployed under a metabit-only unsigned scale (e.g., UE4M3) loses the sign-flip affordance at inference; the methodology enforces consistency between training and deployment to keep reported fidelity faithful to the deployed datapath. The recommended SOP-native scale format is S1E5M5 in a 12-bit container: signed scale (for asymmetric-atom polarity flip) and one metabit (for FMTa/FMTb selection) in a single field, with five-bit mantissa precision sufficient for typical block-magnitude resolution. GPU deployments lacking 12-bit storage typically use UE4M3 (no sign, one metabit in 8 bits), foregoing the ∼25 ppm pair-mean ACos benefit that signed scale confers on asymmetric atoms in our evaluation [Killian, 2026a].
8
Outlier Per-Quantum Extraction (OPQ)
Block weight distributions are roughly Gaussian with a small number of high-magnitude outliers that any low-bit codebook either represents poorly or covers at the cost of resolution in the mass of the distribution. Outlier extraction has a long history in LLM quantization, beginning with the LLM.int8() observation that 6.7B-parameter models develop emergent activation outliers that disproportionately affect downstream loss [Dettmers et al., 2022]; SpQR’s combination of per-tensor outlier storage with sparse quantized representation [Dettmers et al., 2024]; and SqueezeLLM’s explicit dense-and-sparse decomposition [Kim et al., 2024]. OPQ adapts the idea to the per-block weight setting: positions where |wi |/σb exceeds a threshold m are stored as exact BF16 values, and the corresponding entries in the codebook input are zeroed. The codebook fits the residual, which concentrates more tightly around its representable range. The threshold is derived from a target quantile q ∈ (0, 1) via FM (m) = (2Φ(m) − 1)M = q, where Φ is the standard normal CDF and M is the block’s max-order count (typically M = g). Default q=0.92 at block size g=16 gives m ≈ 2.7 and roughly 0.3% mean outlier rate across the models we have profiled. Outliers are encoded as a 16-bit element-index paired with a BF16 value, giving 32 bits per outlier. At a 0.3% mean outlier rate the cost is 32 · 0.003 ≈ 0.10 bpw; higher quantiles or layers with denser outlier geometry push toward 0.20 bpw. The outlier stream P is applied at inference as a sparse contribution to the matrix-multiply output: Youtlier [t, m] = (m,k)∈O vm,k · X[t, k], where O indexes the outlier positions and vm,k are the stored values. This is a sparse axpy pass over Y in software; the SOP backend integrates it lock-step with the matrix-unit issue traversal. OPQ is orthogonal to codebook choice and to format promotion; its bpw cost appears as a separate budget line in the MCKP allocator (§12).
9
Sparse Residual Correction (Wr)
After codebook quantization produces a reconstruction Ŵ , the element-wise residual E = W − Ŵ is computed, and the top-σ entries by activation-weighted magnitude |Em,k | · ck are stored as a sparse correction (the channel norms ck from §4 are reused; values are quantized to E3M4). The activation weighting matters: ranking by raw magnitude would spend the budget on errors invisible in the model’s output, while the activation-weighted metric targets errors that actually propagate. 11
The construction is structurally similar to SpQR’s sparse outliers, applied to the codebook residual rather than the original weight tensor and ranked by activation salience rather than magnitude alone [Dettmers et al., 2024]. Each stored entry occupies a 16-bit element-index plus an 8-bit E3M4 value, plus headers — approximately 32 bits per entry amortized. At sparsity σ the cost is 32 · σ bpw. The default σ = 0.001 (denoted .Wr0.1 for 0.1% sparsity) costs ∼0.032 bpw with typical fidelity gains of 5–15 ppm of pair-mean ACos at 4-bit codebooks; a lower-knee variant .Wr0.06 (0.06% sparsity) costs ∼0.020 bpw with a small fidelity regression. Wr composes additively with OPQ: OPQ removes high-magnitude outliers before codebook fitting, so the residual on which Wr operates is dominated by medium-magnitude codebook errors that complement rather than overlap with OPQ’s contribution. Like OPQ, Wr’s cost is an independent budget line in the MCKP allocator.
10
Per-Layer Pair Search
Each linear layer’s weights are quantized via a pair of n-bit codebooks (La , Lb ) supplemented by one bit of per-block metadata indicating which codebook reconstructs that block. The metadata bit lives in the scale word per §7. For each layer, the pair is chosen by a search over an alphabet of candidate codebooks (the “atoms”) and a single partition parameter p: 1. Construct the FMTa codebook La , either by ROM lookup (fixed-codebook atoms) or, for dataadaptive atoms, by histogram-DP cluster selection on the deployed LUT value format’s grid points. The grid-constrained construction couples the optimization objective to the deployed reconstruction objective, eliminating the snap-displacement cost incurred by free-space Lloyd iteration with a final snap-to-grid step. 2. Score every block against La under the per-block scale and rank by per-block ACos. The top p% of blocks form the FMTa pool. 3. The remaining (100 − p)% form the FMTb fitting pool; Lb is constructed from this residual subset (ROM or fit as appropriate, with the same grid-constrained construction for dataadaptive atoms). 4. A finishing reassignment evaluates each block under both LUTs (and, when the scale format is signed, under both signs of the per-block scale) and picks the better of the candidates per block. Pair-search design axes. The pair-search procedure above has four design choices that are independent in principle, though the literature commonly conflates them: (i) the metric used to rank blocks for the FMTa/FMTb partition in step 2 (ACos or unweighted block MSE); (ii) the metric used in the per-block finishing reassignment in step 4 (block MSE or ACos); (iii) the weighting scheme in the data-adaptive atoms’ Lloyd updates that produce La and Lb (uniform or channel-norms-weighted); and (iv) the residual-pool definition itself (p% above-median by the partition metric, top-k, or quantile threshold). The methodology supports independent selection along each axis on a per-layer basis. Cross-axis empirical comparisons across model families and deployment scale formats appear in [Killian, 2026a]. The codebook alphabet at n=4 centers on four atoms: • NF4 [Dettmers et al., 2023]: Gaussian-quantile inverse-CDF grid; fixed ROM. • BOF4 [Blumenberg et al., 2025]: block-optimum- fitted grid from Monte-Carlo Lloyd iteration on the per-block-max distribution; fixed ROM. 12
• Split87 [Egiazarian et al., 2026]: another fitted grid learned by coordinate descent on the blockwise MSE objective over the training pool of tensors then snapped to the LUT format. • SH4: closed-form sinh grid g(α, c) = sinh(α u) + c, normalized; fixed ROM. • DD4 [Egiazarian et al., 2026, Killian, 2026a]: data-derived DP-clustered grid trained per layer; the learned secondary in the NF4—DD4 pair (equivalent to PO2(NF4) in the multi-grid framework of [Egiazarian et al., 2026]). The pair search considers each atom and its sign-negated form as separate candidates. The corresponding n=5 alphabet extends each atom to 32 entries and adds an FP5 ROM (E2M2). The pair-search mechanism — one bit of per-block metadata selecting among two pre-prepared formats — has structural antecedents in Cook et al.’s adaptive block-scaled data types (IF4, perblock FP4-versus-INT4 selection [Cook et al., 2026]) and Four Over Six (FO6, NVFP4 with per-block scale-6/scale-4 selection [Cook et al., 2025]). Both fix a two-format alphabet globally and select per block by an MSE criterion. Concurrent work by Egiazarian et al. [Egiazarian et al., 2026] formalizes the power-of-twogrids (PO2) problem and studies several learned and structured per-block grid-selection schemes, including MPO2, PO2(NF4), PO2(Split87), and SFP4. That work also defines the PO2 algorithm for residual grid learning: given a fixed or learnable primary grid, the algorithm constructs a second grid from high-error residual blocks, alternates block assignment with Lloyd updates, and snaps the resulting code points to a target format such as FP8 E4M3. Grid Games is complementary to SOP: it studies the statistical and hardware implications of choosing among multiple grids within a block, whereas SOP treats fixed and learned grids as candidate atoms inside a broader per-layer allocation framework, where an atom may be paired, sign-flipped, promoted, or replaced on a layer-by-layer basis according to an activation-weighted objective and the target scale/LUT format. The SFP4 construction in Grid Games is primarily aimed at compatibility with existing E2M1/NVFP4 Tensor Core datapaths, while the more general learned-grid constructions require codebook translation or lookup support. Results from [Egiazarian et al., 2026] also illustrate a useful caution for SOP-style search: the fully learned MPO2 pair achieves strong distribution-level MSE, but has worse modellevel KL than the more prior-structured PO2(Split87), which the authors attribute to overfitting model-specific distributions. SOP can adopt any fixed codebook as a candidate in its evaluation and residual-training flow, so including Split87 and SH4 alongside NF4 and BOF4 is a natural extension. As fixed-ROM atoms, Split87 and SH4 admit the same pair-search treatment as NF4 or BOF4: each can pair with adaptive FMTb atoms such as DD4 or with other fixed atoms, carry a per-block sign metabit, and be promoted on a per-layer basis.2 The pair-search infrastructure does not privilege any particular atom’s provenance. Whether a codebook was derived from a closed-form prior, a Lloyd fit on a per-tensor distribution, or pooled training across model collections, the per-layer search treats it uniformly as an FMTa or FMTb candidate. The choice among competing atoms is decided at the deployed scale format and LUT format, rather than by a fixed global ordering. For constructing an FMTb data-adaptive atom in step 3 of the pair search, the PO2 algorithm of Egiazarian et al. [Egiazarian et al., 2026] is the most relevant published construction. A critical empirical finding is that the winning pair varies between layers and depends on both the LUT value format and the scale format used at deployment. Across the model families we have profiled, the per-layer optimal pair concentrates on a handful of structural patterns — typically a fixed-ROM FMTa (NF4, BOF4, Split87, or SH4, or a sign-negated variant of these) paired with an adaptive or fixed FMTb (DD4 or SH4) — but the specific winner is a function of the layer’s 2
Split87—DD4 and SH4—DD4 in our notation are PO2(Split87) and PO2(SH4) in [Egiazarian et al., 2026].
13
weight distribution and the deployment target’s LUT and scale formats. The absence of a globally optimal pair is itself an architectural advantage: the SOP backend supports per-layer codebooks in SRAM, so the methodology can capitalize on per-layer diversity that a globally tuned scheme cannot.
11
Promotion Profiles
For each layer ℓ, the promotion profile is a function ρℓ : Fpromo → [0, 1],
ρℓ (f ) = ACos Wℓ , Qf (Wℓ ) ,
where Fpromo is the set of higher-precision promotion targets (selected FP8 / FP10 / FP12 grids) and Qf denotes block-scaled quantization to format f . The profile records the per-layer ACos at the base pair format ρℓ (base) as well as at every promotion candidate. Profiles are precomputed once per (model, calibration) pair and form the input to the budget allocator. A small set of |Fpromo | promotion targets (typically four to eight) is enough to span the cost-fidelity Pareto frontier at practical BPW budgets.
12
Multiple-Choice Knapsack Allocation
Given a global BPW budget B, a base pair format with cost bpwbase , and the per-layer promotion profiles {ρℓ }, the allocator chooses for each layer both a format (base or one element of Fpromo ) and a correction setting (any subset of OPQ and Wr), maximizing parameter-weighted P ACos across all layers subject to the budget. Let nℓ be the parameter count of layer ℓ, N = ℓ nℓ the model’s total parameter count, and Fℓ = {base} ∪ Fpromo × {none, OPQ, Wr, OPQ+Wr} the per-layer candidate set. Write bf and ρℓ (f ) for the cost and ACos of candidate f at layer ℓ. Introducing per-(layer, candidate) indicators xℓ,f ∈ {0, 1}, the allocation problem is the multiplechoice 0/1 knapsack XX max nℓ xℓ,f ρℓ (f ) x
s.t.
ℓ f ∈Fℓ
XX
nℓ xℓ,f bf ≤ N B,
ℓ f ∈Fℓ
X
xℓ,f = 1
for each layer ℓ,
f ∈Fℓ
xℓ,f ∈ {0, 1}. Both objective and budget constraint are weighted by the layer’s parameter count nℓ , so wide layers contribute proportionally to both fidelity gain and bpw cost. Because the number of layers is at most a few hundred and |Fℓ | is small, the problem is solved exactly in seconds via dynamic programming. Hessian-aware mixed-precision allocation in the same spirit appears in HAWQ-V3 [Yao et al., 2021] and BRECQ [Li et al., 2021]; SOP’s MCKP differs in using ACos (activation-salience-weighted, gradient-free) rather than Hessian or block-reconstruction surrogates as the per-layer fidelity score.
14
13
End-to-End Pipeline
The full quantization pipeline for a model is: 1. Calibrate. Compute per-layer channel norms from a small text corpus (§4). 2. Pair-search. For each layer, identify the best (La , p, Lb ) tuple from the n-bit codebook alphabet (§10). If OPQ is enabled for the layer, it preprocesses the weight matrix before pair search (§8); if Wr is enabled, it fits residuals after the chosen pair quantizes (§9). 3. Profile. Measure each layer’s ACos at every candidate promotion format (§11). 4. Allocate. Solve the multiple-choice knapsack (§12) to pick a promotion target (or the base pair) per layer subject to the global BPW budget. 5. Materialize. Construct each layer’s per-pair codebooks (or its promoted-format weights) and assemble the quantized model. The pipeline is parametric in (n, B, LUT format, scale format). Each combination produces a different per-layer codebook configuration, capturing one of the methodology’s central claims: that no single codebook geometry is universally optimal, and that exploiting per-layer diversity — enabled by SOP’s SRAM-resident per-layer codebooks and the per-block metabit (§7) — yields fidelity gains unavailable to globally tuned PTQ methods.
14
Conclusion
The Scaled Outer Product methodology combines per-layer pair-searched codebooks with per-block scaled atoms for low-bit weight quantization. Across six open model families, the recommended FP6 operating point (E2M3sUE4M4, 6.5 bpw) matches or beats per-layer-POT FP8 (E4M3ˆ0sUE8M0, 8.0 bpw) on weight-MSE at 1.5 bpw lower storage, demonstrating that block-scaled small atoms with carefully chosen scale precision can replace conventionally-deployed FP8 in practice. Full evaluation of the methodology across the 4.5–6 bpw range, including layer-promotion via MCKP allocation and sparse residual correction, is reported in a companion paper [Killian, 2026a]; the underlying hardware substrate is specified in the SOP architecture specification [Killian, 2026b].
References Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Patrick Blumenberg, Thomas Graave, and Tim Fingscheidt. Improving block-wise LLM quantization by 4-bit block-wise optimal float (BOF4): Analysis and variations. arXiv preprint arXiv:2505.06653, 2025. Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, and Song Han. Four over six: More accurate NVFP4 quantization with adaptive block scaling. arXiv preprint arXiv:2512.02010, 2025. Jack Cook, Hyemin S. Lee, Kathryn Le, Junxian Guo, Giovanni Traverso, Anantha P. Chandrakasan, and Song Han. Adaptive block-scaled data types (IF4). arXiv preprint arXiv:2603.28765, 2026.
15
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Dusan Stosic, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In International Conference on Learning Representations (ICLR), 2024. Sinan Doluca and Thomas J. Riordan. Ultra-low supply-voltage static random-access memory (SRAM) with 8-transistor cell with P and N pass gates to same bit lines. U.S. Patent No. 11,170,844 B1, assigned to Aril Computer Corp., 2021. Filed Jul. 7, 2020; granted Nov. 9, 2021. Vage Egiazarian, Erik Schultheis, Andrei Panferov, Earl Killian, Torsten Hoefler, and Dan Alistarh. Grid games: The power of multiple grids for quantizing large language models. arXiv preprint arXiv:2605.12327, 2026. Andrea Fasoli, Monodeep Kar, Chi-Chun Liu, Swagath Venkataramani, Viji Srinivasan, Leland Chang, and Naigang Wang. Is finer better? The limits of microscaling formats in large language models. arXiv preprint arXiv:2601.19026, 2026. Allen Gersho and Robert M. Gray. Vector Quantization and Signal Compression, volume 159 of Kluwer International Series in Engineering and Computer Science. Kluwer Academic Publishers, Boston, MA, 1992. ISBN 978-0-7923-9181-4. Earl Killian. DCD: Dual codebook decode for hardware-aware LLM quantization. In preparation; arXiv preprint forthcoming, 2026a. Earl Killian. Scaled outer product (SOP): Architecture specification. In preparation; provisional patent application filed May 2026, 2026b. Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. SqueezeLLM: Dense-and-sparse quantization. In International Conference on Machine Learning (ICML), 2024. Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:2411.04330, 2024.
16
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), 2024. Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael W. Mahoney, and Kurt Keutzer. HAWQ-V3: Dyadic neural network quantization. arXiv preprint arXiv:2011.10680, 2021.
A
HIF7/HIF8 shift-budget asymmetry rationale
The asymmetric shift budget between weights (wsa ∈ {0, 1, 2, 3}) and activations (asa ∈ {0, 1, 2, 3, 4}) reflects a fundamental asymmetry in outlier handling between the two operand classes. Activation outliers have no software-side escape valve. Activations are produced at runtime by the model’s own computation; their distribution depends on input and on the quantization state of upstream layers. No offline preprocessing can shift mass out of activation outliers because their values are not known at deployment time. The hardware must allocate sufficient dynamic range to absorb them directly. Weight outliers have multiple methodology escape valves. Weight outliers can be addressed by complementary methodology mechanisms operating before quantization is committed: OPQ (§8) extracts outlier values into a sparse side channel, sparse residual correction (§9) restores the high-error residual after LUT decode, and external preprocessing such as Hadamard rotation [Ashkboos et al., 2024] redistributes weight magnitudes across rows before quantization. Each of these reduces the required dynamic range in the LUT itself, so the per-element shift budget for weights can be narrower without quality cost. Hardware cost of the extended activation range. The activation shift values 5–7 are available via an RTL option for extended-range deployments, at a hardware cost of widening the shifter at every output position in the matrix unit. A 128 × 128 array contains 16,384 shifters; extending each from three levels of logic to four imposes a real area and timing cost across the array. The base configuration (shift sum 0–7, three levels of logic) is the methodology’s reference deployment; the extended configuration is reserved for workloads with extreme activation outlier behavior beyond what the base range absorbs.
B
Complete LUT-format hosting capacity
The body of §6 presents a four-atom × three-container summary of LUT-format hosting capacity. Table 5 extends the comparison to the six-atom alphabet (adding NF5 and SH5) and the fivecontainer range (adding E3M3 and HIF8) used in the methodology’s pair-search experiments.
17
Table 5: Complete LUT-format hosting capacity. LUT atom
ratio
E2M3 (7.5 / 60∗ )
HIF7 (120)
E3M3 (60)
HIF8 (240)
E4M3 (∞† )
NF4 Split87 SH4 NF5 MPO2 SH5
12.6 18.3 26.8 25.2 64.0 75.7
✓ * * * × ×
✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ × ×
✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓
∗
E2M3 normal max/min is 7.5; subnormal extension reaches 60 with attendant quality cost. † E4M3’s normal max/min ratio exceeds the table’s range; all atoms fit comfortably with headroom. Asterisk (*) indicates atoms hosted at E2M3 only via subnormal range. The methodology’s pair search measures each candidate atom at the deployed (LFMT, SFMT) combination, so atoms whose dynamic range exceeds the LFMT’s capacity register the resulting saturation losses in per-block reconstruction quality and lose to better-suited atoms in the per-layer search.
C
HIF7 packing and grid extension
The HIF7 grid contains 80 distinct values, requiring log2 80 ≈ 6.32 bits of information per weight. Current SOP hardware stores them in 8-bit containers (§6), costing 1.68 bpw relative to the algorithmic lower bound. This headroom admits two complementary uses, addressing different deployment priorities. Compressed packing. A 7-bit packing (128-value container, 80 used) would reduce the storage overhead to 0.68 bpw and align with half-byte memory boundaries. Sub-7-bit packings recover additional headroom at increasing decode-logic complexity. These directions reduce DRAM bandwidth and on-die SRAM footprint without changing the LUT alphabet or the matrix-unit datapath. Extended grid via tc6. Alternatively, the 8-bit container size can be retained and the headroom spent on grid refinement. 256 codepoints accommodate a tc6 signed coefficient (64 levels) with the existing shift budgets, yielding a HIF8 grid whose alphabet covers the methodology’s atoms at finer per-octave resolution. The alternative direction — extending the per-element shift range (currently wsa ∈ {0, 1, 2, 3}, asa ∈ {0, 1, 2, 3, 4}, §A) — spends the same codepoints on dynamic range rather than precision. The choice between tc6 and an extended shift range is settled by the same principle as the unsigned-versus-signed scale finding in §7: when per-block scale infrastructure covers dynamic range, additional bits buy more by refining within-block precision than by extending dynamic range. At fixed 8-bit weight width and recommended UE4M6 per-block scale, the mantissa-rich E2M5 weight format reduces reconstruction MSE by approximately 14× over the deployed-standard E4M3 on Llama-3.2-3B; E3M4 reduces it by approximately 4×. By the same logic, the methodologically preferred direction for an extended HIF grid is tc6 with unchanged shift budgets, not an extendedshift HIF.
18
Table 6: FP8-class weight formats on Llama-3.2-3B at two scale strategies.∗ KL (×10−3 )
KL rel.†
Config
UE4M6 (8.625 bpw)
ˆ0sUE8M0 (8.0 bpw)
UE4M6
ˆ0sUE8M0
E2M5 E3M4 E4M3
1.04 1.56 3.78
—‡ 2.32 5.68
0.18 0.28 0.67
—‡ 0.41 1.00
E2M3¶
4.36
—
0.77
—
∗
Calibration dataset c4, evaluation dataset wikitext2, 50K tokens. Relative to the industry-standard E4M3ˆ0sUE8M0 at 8.0 bpw. Reported bpw is logical throughout: atom bits plus scale bits divided by block size g=16. The SOP v19 hardware pads unsigned scales narrower than 12 bits into a 12-bit scale-word, raising deployed bpw by up to 0.125 above logical for scales below 12 bits; UE4M6 (10 logical bits) at g=16 is 8.625 logical and 8.75 deployed. ‡ E2M5’s format max (3.94) is too narrow to host every layer’s worst weight under any single integer shift, forcing systematic truncation under per-layer POT scaling. Per-block scaling is required to use this atom; the analogous boundary on the E2-class atoms appeared earlier in the methodology (§6). § All layers including lm head are quantized at the same atom and scale format in this experiment; no per-layer promotion is applied. This isolates the atom-and-scale effects from the methodology’s promotion infrastructure (§11, §12), which in deployment would lift lm head to a higher-precision format as standard practice. †
¶
E2M3 is an FP6 atom; at UE4M6 scale this is 6.75 deployed / 6.625 logical bpw, not the 8.625/8.0 bpw of the
FP8-class rows above. Per-layer POT is inadmissible for E2-class atoms (§6).
The KL improvements decompose multiplicatively into independent atom and scale effects. Reading down each KL column, moving from E4M3 to E3M4 (the mantissa-precision axis) reduces KL by approximately 2.4× at either scale strategy. Reading across each KL row, moving from ˆ0sUE8M0 to UE4M6 (the scale-precision axis) reduces KL by approximately 1.5× at either atom. The joint 3.6× reduction from the industry-standard E4M3ˆ0sUE8M0 to E3M4sUE4M6 is the product of the per-axis effects to within measurement precision, indicating that mantissa precision and per-block scale resolution are independent levers on PTQ quality and can be optimized separately. E2M5 reaches the lowest KL of the three atoms at UE4M6 — 5.5× below the deployed-standard operating point — but its narrow format max excludes the per-layer POT cell of the grid. The per-block scale does structural work in the E2M5 column, not just marginal quality work: without it the atom cannot host the model’s weight distribution at all. The E3M4 column is the interpretively cleanest of the three: both scale strategies are viable, and the 1.5× ratio between them is the honest cost of foregoing per-block scale resolution at this weight tier. The MSE-to-KL compression varies by format: E2M5’s 14× MSE advantage compresses to 3.6× in KL, while E3M4’s 4× MSE advantage compresses to 2.4×. The disparity reflects that the perblock scale absorbs dynamic-range differences (E3M4’s contribution) more readily than within-block precision differences (E2M5’s contribution); the surviving KL gap is the unabsorbed within-block precision benefit.3 The hardware cost of moving to tc6 is the multiplier area: a 6 × 6 → 12 multiplier replaces the 3 The PTQ result prompts an obvious question about training: if mantissa-rich E2M5 dominates the standard E4M3 for inference at fixed bpw, should training adopt E2M5 as well? This paper does not address that question. Training imposes constraints that inference does not: gradient precision, activation handling under heavier tails, and optimizer-state stability across many update steps. The PTQ finding is suggestive but not dispositive; whether the E2M5-over-E4M3 advantage transfers to training is an empirical question for follow-up work in that setting.
19
current 5 × 5 → 10, an approximately 44% area increase per multiplier, with proportional growth in accumulator and shifter widths. The resulting grid would host the methodology’s atom alphabet at finer per-octave resolution while preserving the shift-add datapath’s structural advantages over a fully general multiplier. Whether the precision gain justifies the area cost is a deployment-context decision that the methodology does not fix; this appendix documents the design space.
D
Per-layer reconstruction distributions
The body of §6, Table 4, advocates block-scaled E2M3 over per-layer-POT E4M3 at 1.5 bpw lower storage. The comparison rests on the assertion that E4M3ˆ0sUE8M0 is a fair 8.0-bpw comparator: collapsing an entire layer’s dynamic range onto a single integer shift, combined with the Flayer search of §7, should reproduce the same per-layer MSE distribution as the mantissa-bearing perblock scales of §7. This appendix tests that assertion directly, reporting the per-layer maximum MSE divided by the per-layer mean MSE for each of the six model families across the three E4M3 scale formats. Table 7: Per-layer max-to-mean MSE ratio for E4M3 weight quantization under three scale formats. All values are dimensionless ratios; lower values indicate a tighter per-layer MSE distribution. Block size g=16, Flayer search active. Model Gemma-3-1B SmolLM3-3B Llama-3.2-3B Qwen3.5-4B Mistral-7B† Qwen3-8B
E4M3ˆ0sUE8M0 (8.0 bpw, per-layer POT)
E4M3sUE4M6 (8.625 bpw, per-block)
E4M3sUE5M7 (8.75 bpw, per-block)
3.6 4.4 4.0 4.4 2.9 1.6
3.6 4.4 3.7 4.4 2.9 1.6
3.6 4.4 3.7 4.4 2.9 1.6
The three E4M3 scale formats produce essentially indistinguishable max-to-mean ratios within each model row. The largest within-model spread is Llama-3.2-3B at 7.3% (4.0 vs. 3.7); the other five families agree to under 2%. No model exhibits the multiplicative distributional asymmetry that would mark per-layer POT scaling as a fundamentally different operating regime from per-block mantissa-bearing scaling at this weight tier. The body claim therefore holds: Flayer has absorbed the per-layer dynamic-range variation that would otherwise have made per-layer POT scaling fragile on outlier-heavy attention projections. Mistral-7B’s consistently lower ratio (here 2.9× at E4M3, versus the 3.4× reported at HIF7 in the footnote of Table 2) is the same architectural anomaly: the magnitude floor moves with weight format, but the property of a tighter-than-average per-layer distribution is consistent across formats. Qwen3-8B exhibits an even tighter distribution at 1.6×: no single linear layer’s reconstruction MSE exceeds the model-wide mean by more than 60%. The property is independent of this appendix’s comparator choice and applies uniformly to every E4M3 configuration measured here.
E
Format string grammar (brief )
Format strings used throughout this paper follow the schema: WFMT0 |WFMT1 |...[^N][sSFMT][+PFMT][.OPQσ][.Wrρ] 20
where each bracketed component is optional. WFMT (weight format). The per-block LUT alphabet. Multiple WFMTs separated by | indicate per-block selection via a metadata bit, e.g. NF4—DD4 chooses between the fixed NF4 codebook (LUT0) and a per-layer-trained DD4 codebook (LUT1) per block. Single-WFMT configs omit the pipe. Atoms used in this paper: E2M3 (FP6), E4M3 (FP8), NF4 / NF5 (NormalFloat), DD4 / DD5 (per-layer Dynamic Decode), Split87, SH4 / SH5, MPO2, BOF4. ˆN (block size). The per-block scope. ˆ16 is default and may be elided. ˆ0 indicates layermax scope (one scale per layer). Block sizes 8 and 32 are admissible but not used in this paper’s experiments. sSFMT (scale format). The per-block scale format in container size w+x+y bits. UExMy is unsigned (no sign bit); ExMy is signed (sign bit may double as a metabit for atom-pair selection or per-block polarity). UE8M0 is power-of-two only; mantissa-bearing variants (e.g. UE4M3, UE4M6, UE5M3) carry fractional precision. +PFMT (promotion). The methodology promotes worst-quantized layers. A fixed target +PFMT promotes selected layers to PFMT (common values: E2M3sUE4M3 at 6.5 bpw, E4M3ˆ at 8.0 bpw layer-max, E5M6ˆ at 12.0 bpw, E8M7 at 16.0 bpw BF16). Alternatively, +knapN invokes the MCKP allocator (§12) at a budget of +N/10 bpw above base, jointly selecting layers and PFMTs; an optional /PFMT/ suffix list restricts the candidate set. .OPQσ (outlier per-quantum extraction). σ (§8).
Sparse weight-outlier extraction at sigma-threshold
.Wrρ (sparse residual correction). Activation-weighted sparse residual correction at sparsity ρ percent (§9). Bare .Wr defaults to ρ = 0.1%. Examples. • E2M3sUE4M4 — FP6 weight, per-block (g=16) unsigned UE4M4 scale. 6.5 bpw. • E4M3ˆ0sUE8M0 — FP8 weight, layer-max power-of-two scale. 8.0 bpw. • NF4—DD4sUE4M3+knap0.10.Wr — NF4/DD4 pair, per-block UE4M3 scale, knapsack promotion at +0.10 bpw budget, default-sparsity Wr correction. ∼ 4.7 bpw effective.
21