Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús

arXiv:2608.20953v1 [cs.CL] 21 Aug 2026

Multiverse Computing {bakbergen.ryskulov, iker.garcia, david.montero, david.jansen, ali.hashemi, jezabel.garcia, antonio.tiene, roman.orus}@multiversecomputing.com Abstract

QAH MECHANISM

COMPRESSION PIPELINE

Original Model

Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted QuantizationAware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B→60B→MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4× less weight memory and half the teacher’s parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7× faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyperparameter search.

Model (Hypernova-60B) — https://huggingface.co/ MultiverseComputingCAI/Hypernova-60B-2605

1

Introduction

The memory and compute cost of large language models (LLMs) has made low-precision, reduced-size deployment a practical necessity rather than an option. A now-common production pattern combines two compression steps in sequence: structural compression, which reduces a model to a fraction of its original parameter count by modifying its architecture, and 4-bit quantization, which further cuts memory and inference cost. Applied together, these steps deliver large efficiency gains, but at a measurable cost in reasoning, mathematical problem-solving, code generation, and long-context ability. For models that have already been through multi-stage post-training (supervised fine-tuning, RLHF/RLAIF, model merging, agentic-behavior tuning), this degradation compounds across stages and must be recovered before the model can be shipped. We refer to this recovery step as healing.

uncompressed · full size capability multi-stage post-training

teacher logits frozen · no gradient · precomputed offline

compress & quantize

Compressed Student KL Divergence

↓ significant capability loss compressed & quantized · 2× smaller

student logits MXFP4 · fake quantizers

KL(pt ∥ ps )

updates student weights

Healed ↑ capability recovered

compressed & quantized

Figure 1: QAH overview. Structural compression followed by 4-bit quantization sharply reduces capability. QAH heals the compressed, quantized student by distilling from the original uncompressed model (frozen teacher, dashed arrow), rather than re-fitting hard labels or distilling from the recovered full-precision checkpoint.

The default healing recipe is quantization-aware training (QAT) (Jacob et al. 2018; Liu et al. 2023), which inserts fake-quantizers into the forward pass and continues training under a task (cross-entropy) loss. In our deployment setting we found QAT to be both expensive and operationally fragile: it converges slowly, and if training is allowed to continue past its peak it collapses, so shipping a QAT checkpoint safely requires careful early stopping against a held-out signal. Recent work on 4-bit inference reports the same fragility and proposes quantization-aware distillation (QAD), which replaces the task loss with a KL objective against a frozen full-precision copy of the model (NVIDIA 2026). QAD is a strong recipe when compression is quantization-only, because a full-precision counterpart of the quantized model exists by construction. Under structural compression that assumption breaks. The natural QAD teacher is the structurally compressed bfloat16 checkpoint that precedes quantization—but that checkpoint is not an independently trained model. It is a distillation-

recovered approximation of the original, and it already carries the capacity loss from compression. Distilling the 4-bit student from it anchors the student to a degraded target and caps its accuracy at the recovered checkpoint’s ceiling. This left us with a concrete deployment question: how should we heal a model that has been both structurally compressed and 4-bit quantized? Our answer, and the recipe this paper documents, is Quantization-Aware Healing (QAH). QAH heals the compressed, quantized student by distilling from the original, uncompressed model—a strictly stronger teacher—rather than from the recovered checkpoint (Figure 1). The teacher and student no longer share an architecture; the student is supervised only through the teacher’s output distribution, which is architecture-agnostic. Concretely, on a two-stage pipeline that compresses a GPT-OSS 120B model to 60B and then re-quantizes it to MXFP4, the QAH student matches or exceeds its own bfloat16 source on 7 of 9 benchmarks, and reaches the original 120B model on LiveCodeBench—while using roughly 4× less weight memory and running at half the teacher’s parameter count. The practical reading is that the quantization stage is not a lossy postprocessing step to be minimized, but a second opportunity to apply teacher supervision that the bfloat16 checkpoint never received. The recipe described here was used to produce Hypernova-60B, an open-weight model released by Multiverse Computing under Apache 2.0; the public release incorporates further training beyond the pipeline evaluated in this paper, so the numbers we report are our own measurements of the pipeline rather than the released checkpoint’s published figures. This is an experience-and-recipe paper rather than a claim of a new algorithmic primitive: QAH combines wellunderstood ingredients (knowledge distillation and fakequantized training) in a way that fits the realities of a compress-then-quantize production pipeline. Our contribution is in identifying where the standard recipe fails in that pipeline, in a recipe that works reliably without a hyperparameter search, and in the operational lessons that made it deployable. Contributions. • We identify why the standard QAD recipe is a poor fit once structural compression precedes quantization: the only available full-precision teacher is a recovered checkpoint that caps the student’s accuracy. We describe QAH, which distills from the original uncompressed model instead (§3). • On a GPT-OSS 120B→60B→MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4× lower weight memory, and reaches the 120B model on LiveCodeBench (§5). • We report the deployment lessons that made QAH usable in practice: it converges about 7× faster than a matched QAT baseline and does not require hand-tuned early stopping, and we document a large, reproducible quality gap between distributed-training backends that practitioners should be aware of (§6).

2

Background and Related Work

Low-precision formats. Microscaling formats (MX, MXFP4, MXFP6, NVFP4) replace per-tensor exponent overhead with block-wise scaling factors, giving aggressive memory and compute reductions while preserving most of the bfloat16 forward pass (Rouhani et al. 2023; Micikevicius et al. 2022). MXFP4 quantizes weights only and is the format behind the public GPT-OSS release (OpenAI 2025); NVFP4 quantizes weights and activations and is used by Nemotron 3 Nano (NVIDIA 2026). Post-training quantization (PTQ) methods—GPTQ (Frantar et al. 2023), AWQ (Lin et al. 2024), SmoothQuant (Xiao et al. 2023)—suffice at moderate compression ratios but leave a non-negligible 4bit accuracy gap on reasoning and coding for models above roughly 10B parameters, which is the regime we operate in. Quantization-aware training. QAT inserts straightthrough-estimator fake-quantizers into the forward pass so weights can adapt to the quantized representation (Jacob et al. 2018; Krishnamoorthi 2018). For LLMs, LLM-QAT (Liu et al. 2023) closes much of the PTQ gap for moderate-scale dense models, and LR-QAT (Bondarenko, Chiaro, and Nagel 2024) reduces its memory overhead. All of these optimize a task loss, which forces the student to re-acquire behavior already present in the original model—a re-traversal that is doubly costly when the student has also been structurally modified. We use QAT as our baseline (§3.3). Knowledge distillation. Distillation (Hinton, Vinyals, and Dean 2015) trains a student to match a teacher’s output distribution rather than rediscover it from labels; for LLMs, MiniLLM (Gu et al. 2024) and DistiLLM (Ko et al. 2024) show it outperforms supervised fine-tuning on reasoning and instruction following. Crucially, distillation does not require teacher and student to share an architecture—crossarchitecture distillation is standard, from BERT compression (DistilBERT (Sanh et al. 2019), TinyBERT (Jiao et al. 2020)) to vision-language pruning (Wang et al. 2023). QAH relies on exactly this property, since its teacher (the uncompressed model) and student (the compressed, quantized model) differ in architecture. The idea of combining distillation with quantization itself is not new (Polino, Pascanu, and Alistarh 2018); our contribution is the choice of teacher in the compress-then-quantize setting. Structural compression. A complementary family of methods reduces parameter count by modifying architecture: structured pruning of heads, neurons, or layers (Ma, Fang, and Wang 2023; Xia, Zhong, and Chen 2022), outlieraware sparsity (Yin et al. 2023; McGowan et al. 2024), embedding-dimension slicing (SliceGPT (Ashkboos et al. 2024)), low-rank decomposition (SVD-LLM (Wang et al. 2024a), ASVD (Yuan et al. 2023)), and tensor-network methods (Novikov et al. 2015; Xu, Xu, and Mandic 2023; Tomut et al. 2025). The common property is that the compressed model is never independently trained at full precision: its weights are derived from the original via a compression operator, and any full-precision version of it is itself a distillation-recovered approximation. This is precisely why a QAD teacher taken from that checkpoint is suboptimal, and

why healing under structural compression calls for a different teacher—the original pre-compression model. Quantization-aware distillation. QAD, formalized for NVFP4 by NVIDIA (2026), starts from the original bfloat16 model as a frozen teacher and trains a fake-quantized student to match its logits via KL divergence. The reported advantages over QAT are no re-traversal of post-training, robustness to data coverage, and single-stage recovery to near-full-precision accuracy. Standard QAD presupposes architectural identity between teacher and student, which holds when compression is quantization-only but not when structural compression comes first. QAH targets that compoundcompression regime by distilling directly from the original model, making no assumption about shared layer structure. To our knowledge, healing for models that have undergone both structural compression and low-precision quantization has not been studied.

3 3.1

Method

The Compound-Compression Regime

Let MHP be the original, uncompressed model, which has already completed multi-stage post-training (SFT, RLHF/RLAIF, and any downstream behavior tuning). We call the pipeline of structural compression followed by lowprecision quantization the compound-compression regime. In it, MHP undergoes three modifications before deployment. Structural compression. A compression operator S reduces the parameter count of MHP by modifying its architecture—removing attention heads, feed-forward neurons, or transformer layers, or reducing embedding dimensionality. The output is a compressed intermediate MC = S(MHP ), 2× smaller in our experiments. MC is never trained at full precision: its weights are derived from MHP by the operator, and no full-precision checkpoint of the compressed architecture is trained from scratch. Full-precision recovery. Before quantization, MC is recovered in bfloat16 to a checkpoint MBF16 by distilling from MHP via KL divergence. This adapts the compressed weights to recover capability lost to compression. It is not independent training: the recovered checkpoint is tethered throughout to MHP ’s output distribution and inherits the capacity ceiling of the compressed architecture. Quantization. A quantization operator Q casts MBF16 to MXFP4, yielding MFP4 = Q(MBF16 ). The full pipeline MFP4 = Q(R(S(MHP ))), with R the recovery operator, defines the regime. Quantization adds a further accuracy gap on top of the compression loss. Healing goal. Healing returns M′FP4 = H(MFP4 ) such that (i) M′FP4 approaches MHP on a target evaluation suite, (ii) M′FP4 remains a valid MXFP4 deployment artifact, and (iii) healing does not regress a broader sanity suite. The defining feature of healing, as opposed to generic fine-tuning, is that MHP exists and is accessible: the target behavior is observable on a frozen teacher, so the student need not rediscover it from labels.

3.2

Quantization-Aware Healing

QAH replaces the recovered checkpoint MBF16 as teacher with the original model MHP . Teacher and student now have different architectures: MHP is full-size and uncompressed, while the student MFP4 is structurally compressed. Notably, the advantage comes from the teacher being uncompressed, not from a difference in precision: MHP is the original GPTOSS release, whose weights are already largely MXFP4 (§4), so QAH distills a compressed 4-bit student from an uncompressed teacher of comparable precision rather than from a higher-precision one. The student is supervised only through the teacher’s output distribution. The QAH loss is a teacher– student KL divergence on output logits: h 
LQAH = Ex∼D KL softmax MHP (x) / τ
i softmax MFP4 (x; Qθ ) / τ , (1) where Qθ are straight-through-estimator MXFP4 fakequantizers inserted into the student’s forward pass, τ is the distillation temperature (τ = 1 throughout), and D is a healing corpus. The KL uses the standard next-token shift and is averaged over masked positions. Following NVIDIA (2026), teacher logits are precomputed offline from the released (MXFP4) Hugging Face checkpoint of MHP , once per example, and truncated to the top-k logits (k = 100) to bound storage. The student sees no hard labels—only the teacher’s distribution.

3.3

Quantization-Aware Training Baseline

QAT inserts the same fake-quantizers Qθ but optimizes a task loss: 
 (2) LQAT = E(x,y)∼D CE y, MFP4 (x; Qθ ) , with (x, y) input–label pairs and CE next-token crossentropy. Unlike QAH, QAT does not condition on MHP at training time; it must re-acquire, from labels, behavior that MHP already encodes.

3.4

Making QAH Fit the Memory Budget

Healing at 16k–32k context within a fixed GPU-memory envelope is the main engineering obstacle. A direct implementation of Eq. 1 materializes the full [B, L, V ] student log-softmax and its autograd graph, whose peak memory O(BLV ) becomes prohibitive at L = 32k with a vocabulary V of order 2×105 . We instead adopt the offline top-K logits and fused chunked-KL implementation of Ryskulov et al. (2026): the teacher’s top-k logits are cached once rather than recomputed each step, and the KL loss and its gradient are accumulated block-by-block along the sequence, so peak intermediate memory becomes linear in sequence length rather than proportional to vocabulary size while remaining bitidentical to the dense loss. This is what lets QAH train at 32k context on the same hardware as short-context QAT; we refer the reader to that work for the derivation and kernel details. We additionally apply a two-phase masking schedule—an attention-mask-only stability window, then assistant-focused masking for the bulk of training—which further reduces effective sequence length; loss-mask-only training consistently underperformed.

4

Experimental Setup

Models and pipeline. We evaluate GPT-OSS 120B and 20B (OpenAI 2025), both released with their MoE weights (the large majority of parameters) in MXFP4 and remaining tensors in BF16; we refer to these released checkpoints as MXFP4 throughout. Our pipeline has two stages. First we compress each model—120B to 60B and 20B to 9B—using a tensor-network compression operator (Tomut et al. 2024; Bercovich et al. 2024; Jansen et al. 2026), then recover each compressed checkpoint by continued bfloat16 training with KL distillation from the corresponding uncompressed GPTOSS teacher. Second, we re-quantize the recovered 60B and 9B checkpoints to MXFP4, again distilling from the uncompressed teacher (QAH) or, for the baseline, training under cross-entropy (QAT). Data. Both stages train on a mixture of NVIDIA Nemotron (NVIDIA 2025; Yang et al. 2026) and SmolTalk 2 (Bakouch et al. 2025) data, covering generaldomain, science, coding, mathematics, safety, and reasoning tasks. Training. All runs use 8 nodes of NVIDIA H200 GPUs with FSDP2 for parameter sharding. Both stages optimize a pure KL objective against the uncompressed teacher (QAT excepted), with assistant-focused loss masking. For the MXFP4 QAH stage we use sequence length 32k, global batch size 64, learning rate 5×10−6 , and 400 steps. Quantizationsensitive submodules (embeddings, layer norms, selected attention components) are frozen to limit destructive drift. Evaluation. We evaluate on nine benchmarks: MMLUPro (Wang et al. 2024b) (general knowledge), GPQA Diamond (Rein et al. 2023) (science), AIME 2025 (Zhang and Math-AI 2025) (mathematics), IFBench (Pyatkin et al. 2026) (instruction following), SciCode (Tian et al. 2024) (scientific coding), LiveCodeBench (Jain et al. 2025) (coding), τ 2 bench (Barres et al. 2025) (agentic tool use), Aider (Gauthier 2024) (agentic coding), and AA-LCR (Team 2025) (longcontext reasoning).

AIME 2025 (76.3 vs. 70.7), +2.7 on Aider (40.9 vs. 38.2), and +2.3 on τ 2 -bench (61.7 vs. 59.4). This is a direct consequence of the QAH recipe rather than of quantization per se: the quantization stage performs a second pass of KL distillation against the original 120B distribution, so the quantized student receives teacher supervision that the bfloat16 checkpoint never saw. The deployment-relevant takeaway is that in a distillationbased healing pipeline, the quantization step is not a lossy postprocessing step to be minimized but a place to add capability. We are deliberate about what this does and does not show: it does not establish that 4-bit representations are inherently superior; it shows that, given the extra distillation pass QAH performs at quantization time, the deployable 4bit artifact is at least as capable as its bfloat16 source at a fraction of the memory (§7). Despite being half the teacher’s parameter count and quantized to 4 bits, the QAH student retains most of the teacher’s capability. On LiveCodeBench it slightly exceeds the 120B model (66.5 vs. 66.0; a difference well within run-to-run noise, so we read it as matching the teacher), and on GPQA Diamond it closes to within 1.6 points (67.4 vs. 69.0). MMLU-Pro and IFBench show gaps of 4.2 and 3.4 points. The largest residual gap is on AA-LCR (−7.3), an extreme long-context benchmark where capacity lost to compression is hardest to recover. Efficiency. These accuracy results come with real serving wins. At 4-bit precision the QAH model uses roughly 4× less weight memory than the bfloat16 student, and at half the teacher’s parameter count it roughly halves compute per token, enabling deployment on substantially smaller hardware. For model families released in bfloat16 rather than 4-bit (e.g., Qwen (Qwen Team 2026)), the combined parameter and precision reduction would amount to roughly 8× less compute per token. QAH therefore inverts the usual low-bit trade-off: instead of trading accuracy for efficiency, it delivers a model that is at once cheaper to serve, lighter in memory, and at least as strong as its bfloat16 counterpart.

5.2

5 5.1

Results

Quantization as a Second Training Opportunity

Figure 2 compares three checkpoints across nine benchmarks: the original gpt-oss-120B teacher, the 60B bfloat16 student recovered from compression, and the same 60B student re-quantized to MXFP4 under QAH. Naively quantizing the recovered bfloat16 checkpoint to MXFP4 without healing incurs a substantial accuracy drop that QAH must recover; as the figure shows, it more than recovers it on most benchmarks. The central practical finding is that the 4-bit checkpoint is not a degraded copy of the 16-bit one—it is generally better. The MXFP4 60B student matches or beats its bfloat16 source on 7 of 9 benchmarks; the two exceptions, MMLU-Pro and SciCode, lose by only 0.2 and 1.4 points. Gains on the rest are substantial: +7.4 on AA-LCR (42.7 vs. 35.3), +5.6 on

QAH vs. QAT: Cost and Stability

For a practitioner, the choice between healing recipes turns less on peak accuracy than on how much it costs to reach that peak and how safe the resulting checkpoint is to ship. Figure 3 compares QAH and QAT when quantizing GPT-OSS 9B to MXFP4, plotting average performance on MMLU-Pro, LiveCodeBench, and GPQA Diamond against training steps. This 20B→9B configuration was produced and validated for an internal client deployment (details withheld), giving us a second end-to-end setting—alongside the released 60B model—in which the QAH recipe was applied in practice. The two recipes reach a comparable peak (54.9 for QAH vs. 54.6 for QAT), but differ sharply on the two axes that matter for deployment. First, cost: QAH peaks in about 100 steps versus roughly 700 for QAT—about 7× fewer steps; the same pattern holds at 60B, where QAH peaks in roughly 400 steps. Second, stability: QAH stays within about two points of its peak through all 1200 steps, whereas QAT collapses once past its peak, losing nearly 19 points by step 1200

GPT-OSS-120B MXFP4 78.0

80.0

74.0 73.8

70.7

76.3

69.0

65.7 67.4

63.3

GPT-OSS-60B BF16

GPT-OSS-60B MXFP4

66.0 65.5 66.5

58.4 59.9

68.4

59.4 61.7 45.3

37.5 35.6

MMLU-Pro

AIME25

GPQA-D

IFBench

34.2

SciCode

LiveCode Bench

2-Bench

50.0

42.7

38.2 40.9

35.3

Aider

AA-LCR

Figure 2: Benchmark performance of three checkpoints in our pipeline: the original gpt-oss-120B (grey), the compressed-andrecovered gpt-oss-60B in bfloat16 (hatched blue), and the same 60B model re-quantized to MXFP4 under QAH (red). The 120B model is the frozen teacher for both the recovery and the quantization stages. The 4-bit QAH student matches or beats its own bfloat16 source on 7 of 9 benchmarks.

Avg QAH

Average score

56 54.9

54.6

Avg QAT

52 48 44 40 36

100

200

300

400

500

600

700

Step

800

900

1000 1100 1200

Figure 3: Average performance of QAH and QAT on MMLU-Pro, LiveCodeBench, and GPQA Diamond as training progresses, quantizing GPT-OSS 9B to MXFP4. QAH peaks at 54.9 in roughly 100 steps and stays stable through 1200 steps. QAT reaches a comparable peak (54.6) only at step 700, then collapses, losing nearly 19 points by step 1200. (from 54.6 at step 700 to roughly 36). Similar QAT-style instability has been reported in concurrent work on NVFP4 distillation (NVIDIA 2026). We attribute the asymmetry to the loss. KL distillation against a frozen teacher pins the student to the teacher’s distribution and gives it no incentive to drift once matched; the cross-entropy objective keeps pushing the quantized student toward hard labels and eventually erodes unrelated capabilities inherited from the original model. Operationally this is the difference between a recipe that can be run to convergence and shipped, and one that requires hand-tuned early stopping against a held-out signal to avoid deploying a collapsed model. In a production setting the second is a standing operational liability, and it is a large part of why we adopted QAH.

6

Deployment Lessons

Beyond the recipe itself, three lessons shaped whether healing worked in practice. We report them because they cost us time to learn and are, we believe, transferable.

The distributed backend is a hyperparameter. FSDP2 (Zhao et al. 2023) and DeepSpeed ZeRO3 (Rajbhandari et al. 2020; Rasley et al. 2020) are usually treated as throughput-equivalent, interchangeable backends for sharded data parallelism, and in principle the choice should not affect optimization quality. Empirically we found a large and consistent gap. Table 1 summarizes the best QAT run under each backend on the 120B student, drawn from a sweep of eleven configurations spanning context lengths {2, 4, 8}k, batch sizes {128, 256}, {4, 8} nodes, and learning rates from 10−7 to 5×10−4 (full sweep in Appendix A). No DeepSpeed configuration reaches the released 120B MXFP4 baseline on GPQA Diamond, plateauing at 65.15 across the entire sweep, while FSDP2 reaches 73.74. The gap is largest on reasoning-heavy GPQA Diamond (−8.6 points) and persists on MMLU-Pro and AIME 2025. We suspect an interaction between DeepSpeed’s mixed-precision communication path and the MXFP4 weight encoding and leave the precise diagnosis to future work; the immediate, actionable lesson is that for MXFP4 distillation workloads

Backend

GPQA:D

MMLU-P

AIME25

LCB

pute cost. We do not evaluate other families (Llama, Qwen, Mistral) or formats (NVFP4, INT4, FP8). DeepSpeed 65.15 69.45 76.67 66.79 Proprietary compression operator. The operator proFSDP2 73.74 70.95 80.00 66.42 ducing the 60B and 9B students is proprietary. QAH is agTarget (120B MXFP4) 69.0 78.0 80.0 66.0 nostic to the compression method—it needs only logit-level access to the uncompressed teacher—but we have not verTable 1: Best QAT run under each distributed backend on the ified that the gains transfer to other structural-compression 120B student. The DeepSpeed row is the strongest of eight approaches such as layer pruning, SliceGPT, or low-rank deconfigurations (8k context, lr 5×10−6 , bs 256, 8 nodes, 5000 composition. steps); FSDP2 is the strongest of three (2k context, lr 10−5 , bs 128, 4 nodes, 2200 steps). Across the full DeepSpeed 8 Conclusion sweep no configuration exceeds 65.15 on GPQA Diamond. We described Quantization-Aware Healing, the recipe we Bold marks the per-column best among QAT runs. adopted for recovering LLMs that have been both structurally compressed and 4-bit quantized. QAH heals by distilling the compressed, quantized student from the original uncomthe backend must be treated as a tuned hyperparameter, and pressed model rather than from a recovered full-precision we default to FSDP2. checkpoint. On a GPT-OSS 120B→60B→MXFP4 pipeline Freeze quantization-sensitive submodules. In our sweep, the resulting 4-bit model matches or beats its own bfloat16 unfreezing layer norms and embedding projections during source on 7 of 9 benchmarks and reaches the 120B model on the quantization stage—especially at higher learning rates— LiveCodeBench, at roughly 4× lower weight memory and produced checkpoints worse than the unhealed MXFP4 half the parameter count. Against a matched QAT baseline model. Freezing embeddings, layer norms, and selected atit converges about 7× faster and, unlike QAT, does not coltention components was necessary for stable healing under lapse under continued training—so it can be shipped without both QAT and QAH. hand-tuned early stopping. Alongside the recipe we reported the operational lessons that made it work, including a large Long-context healing is memory-bound, but tractable. and reproducible quality gap between distributed-training Healing at 16k–32k context is the binding constraint on the backends, and we were explicit about the comparison we did memory envelope. The chunked-KL implementation of §3.4 not run. For practitioners, the practical message is that in is what made 32k-context QAH fit the same hardware as a distillation-based healing pipeline the quantization step is short-context QAT: it reduces peak intermediate memory not a cost to be minimized but an additional opportunity for from O(BLV ) to O(BcV ) while remaining bit-identical teacher supervision, yielding a model that is simultaneously to the dense loss, so the long-context capability that comcheaper to serve, lighter in memory, and at least as accurate pression damages most can actually be healed rather than as its full-precision counterpart. The recipe is not merely a skipped. research prototype: it produced the open-weight Hypernova60B release and a separate model built for an internal client 7 Limitations and Open Questions deployment, both healed with the pipeline described here. We are explicit about scope, in the spirit of reporting what did not get tested alongside what did. References The direct QAD baseline is missing. Our central argument is that distilling from the original model beats distilling Ashkboos, S.; Croci, M. L.; do Nascimento, M. G.; Hoefrom the recovered bfloat16 checkpoint (standard QAD). We fler, T.; and Hensman, J. 2024. SliceGPT: Compress Large motivate this from the quantization literature—a quantizedLanguage Models by Deleting Rows and Columns. In Proand-recovered checkpoint is by construction no stronger than ceedings of ICLR. the model it approximates—but we did not run the head-toBakouch, E.; Ben Allal, L.; Lozhkov, A.; Tazi, N.; Tunhead experiment (QAH vs. QAD-from-MBF16 at matched stall, L.; Patiño, C. M.; Beeching, E.; Roucher, A.; Reedi, configuration). It is the single most valuable experiment to A. J.; Gallouédec, Q.; Rasul, K.; Habib, N.; Fourrier, C.; Kyadd, and until it is run our claim of a recovered-teacher ceiling dlicek, H.; Penedo, G.; Larcher, H.; Morlon, M.; Srivastav, should be read as well-motivated but not directly measured. V.; Lochner, J.; Nguyen, X.-S.; Raffel, C.; von Werra, L.; and Single run, small-n benchmarks. Every number is a Wolf, T. 2025. SmolLM3: smol, multilingual, long-context single run with no seed variance or confidence intervals, reasoner. https://huggingface.co/blog/smollm3. and some benchmarks are small (AIME 2025 has 30 probBarres, V.; Dong, H.; Ray, S.; Si, X.; and Narasimhan, lems), so individual deltas should be read as indicative rather K. 2025. tau2-Bench: Evaluating Conversational Agents than significant. Where two systems are within a point or in a Dual-Control Environment. arXiv preprint two—most notably the LiveCodeBench comparison to the arXiv:2506.07982. teacher—we read the result as a match, not a win. Bercovich, A.; Ronen, T.; Abramovich, T.; Ailon, N.; Assaf, One model family, one format, one data mixN.; Dabbah, M.; Galil, I.; Geifman, A.; Geifman, Y.; Golan, ture. All experiments use GPT-OSS MoE transformers I.; et al. 2024. Puzzle: Distillation-based nas for inference(120B→60B and 20B→9B) quantized to MXFP4 with a sinoptimized llms. arXiv preprint arXiv:2411.19146. gle Nemotron + SmolTalk mixture, owing to the high com-

Bondarenko, Y.; Chiaro, R. D.; and Nagel, M. 2024. LowRank Quantization-Aware Training for LLMs. arXiv preprint arXiv:2406.06385. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. In Proceedings of ICLR. Gauthier, P. 2024. Aider LLM Leaderboards: Polyglot Coding Benchmark. https://aider.chat/docs/leaderboards/\# polyglot-leaderboard. Accessed: 2024. Gu, Y.; Dong, L.; Wei, F.; and Huang, M. 2024. MiniLLM: Knowledge Distillation of Large Language Models. In Proceedings of ICLR. Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531. Jacob, B.; Kligys, S.; Chen, B.; Zhu, M.; Tang, M.; Howard, A.; Adam, H.; and Kalenichenko, D. 2018. Quantization and Training of Neural Networks for Efficient Integer-ArithmeticOnly Inference. In Proceedings of CVPR. Jain, N.; Han, K.; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations. Jansen, D.; Rausch, R.; Hashemi, A.; Montero, D.; and Orús, R. 2026. Block Removal for Large Language Models through Constrained Binary Optimization. arXiv preprint arXiv:2602.00161. Jiao, X.; Yin, Y.; Shang, L.; Jiang, X.; Chen, X.; Li, L.; Wang, F.; and Liu, Q. 2020. TinyBERT: Distilling BERT for Natural Language Understanding. In Findings of EMNLP. Ko, J.; Kim, S.; Chen, T.; and Yun, S.-Y. 2024. DistiLLM: Towards Streamlined Distillation for Large Language Models. In Proceedings of ICML. Krishnamoorthi, R. 2018. Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper. arXiv preprint arXiv:1806.08342. Lin, J.; Tang, J.; Tang, H.; Yang, S.; Chen, W.-M.; Wang, W.C.; Xiao, G.; Dang, X.; Gan, C.; and Han, S. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. In Proceedings of MLSys. Liu, Z.; Oguz, B.; Zhao, C.; Chang, E.; Stock, P.; Mehdad, Y.; Shi, Y.; Krishnamoorthi, R.; and Chandra, V. 2023. LLMQAT: Data-Free Quantization Aware Training for Large Language Models. arXiv preprint arXiv:2305.17888. Ma, X.; Fang, G.; and Wang, X. 2023. LLM-Pruner: On the Structural Pruning of Large Language Models. In Proceedings of NeurIPS. McGowan, J.; Lai, W. S.; Chen, W.; Aldridge, H.; Clarke, J.; Garcia, J.; Xia, R.; Liang, Y.; Hennequin, G.; and Bernacchia, A. 2024. Efficient Model Compression Techniques with FishLeg. arXiv preprint arXiv:2412.02328. Micikevicius, P.; Stosic, D.; Burgess, N.; Cornea, M.; Dubey, P.; Grisenthwaite, R.; Ha, S.; Heinecke, A.; Judd, P.; Kamalu, J.; Mellempudi, N.; Oberman, S.; Shoeybi, M.; Siu, M.; and

Wu, H. 2022. FP8 Formats for Deep Learning. arXiv preprint arXiv:2209.05433. Novikov, A.; Podoprikhin, D.; Osokin, A.; and Vetrov, D. 2015. Tensorizing Neural Networks. In Proceedings of NeurIPS. NVIDIA. 2025. Nemotron 3 Nano: Open, Efficient Mixtureof-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. Technical report. NVIDIA. 2026. Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery. arXiv preprint arXiv:2601.20088. OpenAI. 2025. GPT-OSS: Open-weight Models Model Card. arXiv preprint arXiv:2508.10925. Polino, A.; Pascanu, R.; and Alistarh, D. 2018. Model Compression via Distillation and Quantization. In Proceedings of ICLR. Pyatkin, V.; Malik, S.; Graf, V.; Ivison, H.; Huang, S.; Dasigi, P.; Lambert, N.; and Hajishirzi, H. 2026. Generalizing verifiable instruction following. Advances in Neural Information Processing Systems, 38. Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. Rajbhandari, S.; Rasley, J.; Ruwase, O.; and He, Y. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, 1–16. IEEE. Rasley, J.; Rajbhandari, S.; Ruwase, O.; and He, Y. 2020. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of KDD. Rein, D.; Hou, B. L.; Stickland, A. C.; Petty, J.; Pang, R. Y.; Dirani, J.; Michael, J.; and Bowman, S. R. 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint arXiv:2311.12022. Rouhani, B. D.; Zhao, R.; More, A.; Hall, M.; Khodamoradi, A.; Deng, S.; Choudhary, D.; Cornea, M.; Dellinger, E.; Denolf, K.; Dusan, S.; Elango, V.; Golub, M.; Heinecke, A.; James-Roxby, P.; Jani, D.; Kaul, G.; Liu, C.-C.; Melnick, L.; Mesmakhosroshahi, M.; Rodriguez, A.; Schulte, M.; Shafipour, R.; Shao, L.; Siu, M.; Dubey, P.; Micikevicius, P.; Naumov, M.; Verrilli, C.; Wittig, R.; Burger, D.; and Chung, E. 2023. Microscaling Data Formats for Deep Learning. arXiv preprint arXiv:2310.10537. Ryskulov, B.; García-Ferrero, I.; Montero, D.; Jansen, D.; Hashemi, A.; Garcia, J. R.; Tiene, A.; and Orús, R. 2026. Offline Top-K Logits and a Fused Chunked KL Loss. arXiv:2608.03796. Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108. Team, A. A. 2025. Artificial Analysis Long Context Reasoning Benchmark(LCR). Tian, M.; Gao, L.; Zhang, S. D.; Chen, X.; Fan, C.; Guo, X.; Haas, R.; Ji, P.; Krongchon, K.; Li, Y.; et al. 2024.

Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37: 30624–30650. Tomut, A.; Jahromi, S. S.; Sarkar, A.; Kurt, U.; Singh, S.; Ishtiaq, F.; Muñoz, C.; Bajaj, P. S.; Elborady, A.; del Bimbo, G.; Alizadeh, M.; Montero, D.; Martin-Ramiro, P.; Ibrahim, M.; Alaoui, O. T.; Malcolm, J.; Mugel, S.; and Orus, R. 2025. CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks. In Proceedings of the 33rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN). Tomut, A.; Jahromi, S. S.; Sarkar, A.; Kurt, U.; Singh, S.; Ishtiaq, F.; Muñoz, C.; Bajaj, P. S.; Elborady, A.; del Bimbo, G.; et al. 2024. Compactifai: extreme compression of large language models using quantum-inspired tensor networks. arXiv preprint arXiv:2401.14109, 12. Wang, T.; Zhou, W.; Zeng, Y.; and Zhang, X. 2023. EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning. In Findings of ACL. Wang, X.; Zheng, Y.; Wan, Z.; and Zhang, M. 2024a. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression. arXiv preprint arXiv:2403.07378. Verify author list against arXiv before camera-ready. Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Hu, S.; Zhu, A.; Li, B.; Liang, Y.; Ding, M.; Chen, B.; Yang, J.; and Chen, W. 2024b. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In Proceedings of NeurIPS. Xia, M.; Zhong, Z.; and Chen, D. 2022. Structured Pruning Learns Compact and Accurate Models. In Proceedings of ACL. Xiao, G.; Lin, J.; Seznec, M.; Wu, H.; Demouth, J.; and Han, S. 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of ICML. Xu, M.; Xu, Y. L.; and Mandic, D. P. 2023. TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition. arXiv preprint arXiv:2307.00526. Yang, Z.; Liu, Z.; Chen, Y.; Dai, W.; Wang, B.; Lin, S.-C.; Lee, C.; Chen, Y.; Jiang, D.; He, J.; Pi, R.; Lam, G.; Lee, N.; Bukharin, A.; Shoeybi, M.; Catanzaro, B.; and Ping, W. 2026. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. Yin, L.; Wu, Y.; Zhang, Z.; Hsieh, C.-Y.; Wang, Y.; Jia, Y.; Pechenizkiy, M.; Liang, Y.; Wang, Z.; and Liu, S. 2023. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. arXiv preprint arXiv:2310.05175. Yuan, Z.; Shang, Y.; Song, Y.; Wu, Q.; Yan, Y.; and Sun, G. 2023. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. arXiv preprint arXiv:2312.05821.

Run Backend R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

SL

FSDP2 2k DeepSpeed 4k FSDP2 2k DeepSpeed 4k DeepSpeed 8k DeepSpeed 8k DeepSpeed 8k DeepSpeed 8kf DeepSpeed 8kf DeepSpeed 8kf FSDP2 2k

Target (120B MXFP4)

lr −5

10 10−5 10−5 10−5 5×10−6 5×10−6 5×10−4 10−7 5×10−7 10−6 10−5

GPQA:D MMLU-P

LCB

69.19 65.15 65.66 65.15 65.15 63.64 59.09 65.15 58.59 64.14 73.74

70.86 69.15 70.28 69.39 69.45 68.89 66.86 69.22 69.11 69.25 70.95

61.94 66.42 64.93 63.81 66.79 63.43 56.72 67.54 64.18 65.30 66.42

72.73

79.47

68.66

Table 2: QAT sweep (R1–R11) over the 120B student; best checkpoint score per metric. R11 repeats R1 with ∼2× more steps. 8kf denotes 8k context with a position-id fix. R11 is the sole run reaching the MXFP4 baseline on GPQA Diamond and is the QAT reference arm in the main text.

Zhang, Y.; and Math-AI, T. 2025. American Invitational Mathematics Examination (AIME) 2025. Zhao, Y.; Gu, A.; Varma, R.; Luo, L.; Huang, C.-C.; Xu, M.; Wright, L.; Shojanazeri, H.; Ott, M.; Shleifer, S.; Desmaison, A.; Balioglu, C.; Damania, P.; Nguyen, B.; Chauhan, G.; Hao, Y.; Mathews, A.; and Li, S. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. In Proceedings of VLDB.

A

QAT Training-Systems Sweep

Table 2 reports the eleven-run QAT sweep on the 120B student that underlies the backend lesson in §6. Every run starts from the same recovered checkpoint with the same frozensubmodule set, Nemotron + SmolTalk corpus, gradient clipping at 1.0, and 100-step warmup. The variables probed are the distributed backend, effective sequence length, learning rate, batch size, cluster size, and layer-norm freezing strategy. Only the FSDP2 configuration (R11) reaches the released 120B MXFP4 baseline on GPQA Diamond; every DeepSpeed configuration plateaus 3.5–14.2 points below it across a four-order-of-magnitude learning-rate range. R11 is the QAT arm used in the matched comparison in the main text.

Related documents

• Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs#662140
• Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs#661090
• A Nuclear-Norm Lower Bound for Dithered Scalar Quantization of Matrix Products#972692
• Privacy-Preserving LLMs Routing#31211
• Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration#147073