When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs Lucky Verma Independent Researcher [email protected]
Pratik Yadav University of Maryland, Baltimore County [email protected]
Abstract
vocabulary. This signal is logged for denoising/remasking diagnostics, but its meaning under shorthorizon PEFT is unclear. We test the transfer directly: can top-1 collapse serve as a PEFT stability warning, and if not, what family-local monitor is more useful for inspection? The transfer fails. Across three DLM model families spanning 816 DLM PEFT configurations (LLaDA-family, four cohorts, n=671 + Dream-7B boundary cohort n=100 + MDLM-OWT 130M boundary cohort n=45), the warning fires in 816/816 (100%) configurations, while actual training collapse, logged by the same training loop’s collapsed flag, occurs in 0/816 (0%) at the 200-step horizon. The diagnostic has zero precision. Matched AR controls on Pythia {410M, 1B, 2.8B, 6.9B} and Qwen3.5-9B (360 audited configurations; App. C) also show 0/360 actual collapses, so the result does not indicate a generic masked-CE collapse phenomenon. The warning fails to transfer into the tested DLM-LoRA PEFT setting. The failure has a measured explanation. Across the same 671 configurations, top-1 token frequency is 0.83 ± 0.13 at training step 0; every configuration is already above 0.5, the median configuration crosses 0.95 within 4 optimizer steps, and the legacy fire-step is stability-agnostic (Mann– Whitney U : p=0.20, n.s.; Fig. 3). A parameterside check at the worst rank-amplification corner gives the complementary measurement: per-token CE gradients are only modestly concentrated (Gini 0.29, largest evaluated token-position CE-gradient share 1.5%), while LoRA-parameter gradients are concentrated (Gini 0.46, one matrix carries 63.0% of gradient mass; App. D). Top-1 tracks token-side pre-equilibrium concentration; max gradient norm samples the parameter-side routing that separates stable from unstable runs. We evaluate max gradient norm as a familylocal triage signal with Mann–Whitney U tests
arXiv:2606.24119v1 [cs.LG] 23 Jun 2026
Discrete diffusion language model (DLM) finetuning inherits inexpensive diagnostics from denoising-time confidence monitors, but their PEFT-training meaning is untested. We test top-1 argmax concentration as a collapse warning. Across 816 LoRA/PEFT configurations from three DLM families, the warning fires for every configuration while logs record 0/816 actual collapses at the 200-step horizon, giving zero precision. The cause is pre-equilibrium saturation: top-1 concentration is already high before optimization and quickly becomes insensitive to final training stability. We then evaluate max LoRA gradient norm, a parameterside signal that samples gradient routing rather than token concentration. On a pooled held-out LLaDA-family split, a train-optimized threshold identifies top-decile final-loss configurations with precision 0.68 and F1 =0.79, above the all-positive top-1 baseline even at the lower split-bootstrap confidence bound. Autoregressive controls and cross-family threshold failures bound the result to short-horizon DLMLoRA inspection rather than a universal collapse detector. Workflow: drop top-1 as a PEFT alarm, log max-gradient early in training, and calibrate thresholds per DLM family before routing runs for inspection.
1
Introduction
Discrete diffusion language models (DLMs) (Nie et al., 2025; Sahoo et al., 2024; Ye et al., 2025) reconstruct fully masked sequences through iterative denoising, using bidirectional context rather than left-to-right prediction. As DLM checkpoints and fine-tuning recipes spread (Zhang et al., 2024, 2026; Wu et al., 2026; Kuiper et al., 2025; Yang et al., 2026), practitioners need low-cost monitors for short-run LoRA training. A tempting candidate is already exposed by the audited DLM runners: the top-1 collapse rate, which measures whether argmax predictions concentrate on a small token 1
A Detector transfer fails 671/671
LLaDA family n = 671
B Family-calibrated gradient separation
C Held-out triage precision 1.0
3.23x
LLaDA full
0/671
0/100
MDLM-OWT n = 45
0/45
LLaDA method (source scale) 1.48x
Critical grid 45/45
2.11x
AR 410M 1.03x
AR 1B
DLM warning n/a 0/360
AR controls n = 360 0
50
Rate (%)
1
2
0.68
0.6 0.4 0.2
1.39x
AR 6.9B 100
0.8
362x
Precision
100/100
Dream-7B n = 100
0.148
0.0 10
100
1000
Unstable / stable max-gradient ratio
Top-1 baseline
Max-gradient
Figure 1: The transferred top-1 warning has zero precision, while max-gradient gives a LLaDA-family triage signal. (A) Across the 816 DLM PEFT configurations, the top-1 warning fires in every configuration and observed collapse is 0/816; AR controls have 0/360 collapses and no top-1 warning by definition. (B) Stable-vs-unstable max-gradient effect sizes are large in the LLaDA-family DLM cohorts (3.23×, 362× on the source-scale methodcomparison set, and 1.48×; Mann–Whitney U with Bonferroni m=6 and bootstrap CIs), while AR controls are smaller or non-portable. (C) On a fixed held-out LLaDA-family split (n=671), max-gradient precision is 0.68 with split-bootstrap 95% CI [0.500, 0.947], compared with the all-positive top-1 baseline ceiling 0.148 (recall 0.94, F1 =0.79).
DLM-LoRA triage workflow. The audited workflow is three steps: drop top-1 as a PEFT alarm at this horizon, log max-gradient by step ∼ 25, and calibrate thresholds per DLM family before routing high-gradient configurations to inspection or separately validated follow-up sweeps. Three findings follow from existing data: top-1 is not a PEFT warning at this horizon, max-gradient is a family-local inspection trigger inside LLaDAfamily runs, and mask ratio should be tuned per model rather than exported as a single operating window. Mask ratio is the strongest tested low-cost covariate in the mask-ratio holdout probes; maxgradient is the supported early inspection signal while preserving mask-ratio design as a separate tuning axis.
and Bonferroni correction across six analyzable families (m=6). On LLaDA2.0-mini (n=144), unstable configurations have 3.23× higher median max-gradient norm than stable configurations (pBonf =2.7×10−7 , bootstrap 95% CI [2.76, 3.97]); on the method-comparison set (n=395), the ratio is 362× (stable median 99.3 vs. unstable median 35,960.4 in the source scale; pBonf =5 × 10−21 , CI [202, 779]). The key check is held-out performance. On a fixed 80/20 split of the 671-configuration LLaDA-family corpus, a threshold selected on training configurations predicts top-decile finalloss configurations on held-out configurations with precision 0.68, recall 0.94, and F1 =0.79, versus 0.13 precision for the all-positive top-1 baseline on this fixed split. A separate split-bootstrap gives 95% CI [0.500, 0.947], disjoint from the splitbootstrap baseline ceiling 0.148; each bootstrap replicate resamples configurations, redraws the train/test split, and reselects the threshold on train; even the lower CI bound exceeds 3× the baseline, and the supported use is inspection and routing rather than a high-precision gate (Limitations). Separately, a B=200 random-split step-k sweep shows max-gradient precision stabilizing from step ∼ 25 onward, while loss-at-step-k is non-monotonic: loss is stronger at step 11 for extreme high-loss configurations but trails max-gradient at steps 25– 100 (App. B.1). Cross-family thresholds do not transfer; calibration is per family, not a global constant.
Contributions. 1. An 816-configuration refutation: top-1 fires in 816/816 DLM PEFT configurations while 0/816 actual collapses occur (§4.1). 2. A two-level saturation characterization showing why the warning fails, with token-side pre-equilibrium saturation and parameter-side gradient routing evidence (§4.5). 3. A family-calibrated max-gradient triage protocol with held-out precision 0.68 (CI [0.500, 0.947]) on the pooled LLaDA-family corpus (§4.1). 4. Thirteen falsification probes and matched AR controls that bound the claim to short-horizon DLM-LoRA PEFT (App. D). 2
Manuscript values are source-mapped through local run manifests, claim-bearing aggregates, and verification summaries; public paper source, reference scripts, and the sanitized aggregate result artifacts that back the tables and figures are released at GitHub repository (result artifacts).
2
Background
2.1
Discrete Diffusion Language Models
masked-CE serve as diagnostic controls; Dream, MDLM-OWT, and LLaDA-MoE runs act as boundary cohorts. The primary DLM setup is: • LLaDA-8B-Instruct (Nie et al., 2025): 8B parameter masked diffusion LM. Mask token ID: 126336. Architecture: LLaDAModel (custom, non-HF-standard). • LLaDA2.0-mini (Bie et al., 2025): 15.93B MoE masked diffusion LM. Mask token ID: 156895. This model provides the 60configuration rank×mask surface and the 2×2 task-performance factorial.
Discrete diffusion language models (DLMs) train by adding discrete noise to token sequences (masking tokens at rate ρ) and learning to reconstruct the original tokens from the noisy input. At inference, DLMs iteratively denoise a fully masked sequence over T steps, using bidirectional attention at each step (Nie et al., 2025; Sahoo et al., 2024). We write the masked-diffusion training objective in the permasked-token form used by our implementation: " # 1 X L(θ) = −Et,xt log pθ (xi0 | xt ) , |Mt | i∈Mt (1) where Mt = {i : xit = [MASK]} and xi0 is the clean token at position i. This differs fundamentally from AR next-token prediction. The density of gradient signal scales with ρ and the prediction entropy grows with the number of tokens jointly predicted, which together drive the rank–mask interaction we characterize in Sec. 4.1
3
Methodology
3.1
Correct Training Objective
• Dream-7B (Ye et al., 2025): 7B parameter masked diffusion LM. Loaded via AutoModel. Requires boolean attention mask. LoRA configuration. All primary DLM runs adapt attention projections only (q_proj, k_proj, v_proj, o_proj). The LLaDA2.0-mini surface uses ranks {4, 8, 16, 32, 64} and 12 mask ratios spanning ρ ∈ [0.05, 0.95]; the task-performance factorial uses ranks {4, 64} and masks {0.40, 0.90} for 3 seeds per configuration, and the operating-cell method comparison (App. E) uses n=10 seeds per method at the learning rate selected by the α-sweep. The older LLaDA-Instruct pilot uses the coarser 5 × 4 grid ρ ∈ {0.3, 0.5, 0.7, 0.9}, and the Dream-7B pilot uses a learning-rate-resolved rank×mask grid described in Appendix C. Training. Short pilot runs use 30–40 steps for API validation; the LLaDA2.0-mini surface uses 200 steps at lr= 10−4 (reported as an observedprefix diagnostic because the legacy top-1 detector early-stops all 60 traces at step 11). The 2 × 2 factorial uses 1000 steps on a 152-example hand-written arithmetic corpus with 20 held-out prompts and is reported as masked-CE convergence evidence; generation-quality evaluation is separated from this diagnostic claim. Batch size 4, AdamW, gradient norms recorded pre-clipping (0.5 threshold). LLaDA2.0-mini runs on an H100 NVL (96GB) workstation; pilots on CHIP HPC (UMBC) NVIDIA L40S (48GB). Implementation: HuggingFace transformers (Wolf et al., 2020) + PEFT (Mangrulkar et al., 2022). Gradient norms are global ℓ2 norms over trainable LoRA parameters. Family-canonical hyperparameters are used across model families (LLaDA lr= 10−4 eff. batch
Standard HuggingFace PEFT training assumes a model-internal supervised loss, but LLaDA-style DLM forward passes return logits only because the caller defines the masking distribution. Following Sahoo et al. (2024), we mask tokens externally and use Eq. 1 with loss computed only over masked positions. Appendix A gives the drop-in API fixes needed to reproduce this objective. 3.2
Experimental Setup
Models. We evaluate LoRA fine-tuning in three roles. LLaDA-family DLMs provide the primary top-1 refutation and max-gradient separation; Pythia/Qwen causal models under matched 1
We encountered five silent-failure modes in the standard HuggingFace + PEFT stack when running LoRA on LLaDA/Dream (loss API returning None, generation API kvcache assertion, target-module auto-detection, Dream model-class loader, Dream attention-mask dtype). Drop-in fixes appear in Appendix A; public release artifacts are linked in Appendix A.
3
64; Dream-default lr= 2×10−6 ; MDLM-OWT lr= 10−4 batch 1); the 816/816 fire-rate identity is an empirical aggregate under these familyspecific settings, not a hyperparameter-invariance proof (App. C).
50% of predicted argmax tokens concentrate on a single token within a short observation window. This makes it a plausible but unvalidated short-run PEFT stability monitor; the Dream and MDLM boundary cohorts use harmonized logging fields for the same test.
Scope note on AR baselines. Two AR controls play distinct roles. Training-stack sanity: a Mistral-7B LoRA baseline under standard nexttoken cross-entropy (Appendix A) verifies that the implementation itself is not the instability source. Masked-CE control: Pythia-1B (Biderman et al., 2023) on the same 5 × 12 grid (180 runs, n=3 seeds, §4.3) tests the loss-vs-architecture confound by holding the loss fixed while varying architecture and pretraining; Qwen3.5-9B (Qwen Team, Alibaba, 2026) adds a larger matched control in §4.3.
4
Denominator and result. We aggregate across three DLM model families totalling 816 configurations: LLaDA family (n=671; four cohorts from 2–3 model checkpoints), a Dream-7B dense boundary cohort (n=100; App. C), and an MDLM-OWT 130M dense boundary cohort (n=45; App. C). The cohorts use harmonized logging fields for top1_warning_detected and post-hoc collapsed; MDLM measures top-1 from the training-time masked-input forward pass, while LLaDA-family runs use the corresponding runner proxy (App. C). The top-1 collapse warning fires in 816/816 (100%) configurations; actual collapse occurs in 0/816 (0%). The diagnostic has zero precision at this horizon. In PEFT finetuning at ≤200 steps, the warning fires on a preequilibrium artifact of LoRA updates rather than on divergence dynamics. AR controls (0/360 collapses across the audited main and extended-mask grids; §4.3) show that the tested masked-CE controls do not produce an analogous collapse pattern. Source-level provenance is recorded in Appendix A.
Experiments and Results
The experiments answer a diagnostic question, not a method-comparison question: can a low-cost monitor identify DLM-LoRA configurations that should be inspected before the late training loss is known? We first test the transferred top-1 warning, then evaluate max gradient norm under the same held-out label, and finally use the LLaDA2.0-mini rank×mask surface, AR controls, token/gradient measurements, and task probe to mark the boundary of the claim. Preliminary LLaDA-8B-Instruct experiments across a 5 × 4 rank×mask grid motivated the denser LLaDA2.0-mini study but are omitted from the main body.
Max gradient norm separates stable from unstable configurations. We report the maximum LoRA gradient ℓ2 norm over the training trajectory (pre-clipping at the standard 0.5 threshold) as the triage signal. Within DLM family, the median maxgradient ratio between unstable (top-decile finalloss) and stable (sub-median final-loss) configurations is 3.23× on LLaDA2.0-mini full surface (n=144, Mann–Whitney U , pBonf =2.7 × 10−7 , m=6, bootstrap 95% CI [2.76, 3.97]); 362× on the LLaDA method-comparison set (n=395; stable median 99.3, unstable median 35,960.4 in the source scale; pBonf =5 × 10−21 , CI [202, 779]); and 1.48× (pBonf =0.036) on the 10-seed critical expansion (n=120, compressed dynamic range). AR controls show smaller, inconsistent separation (Table 2; verification summaries in App. A), supporting family calibration rather than a global threshold.
Evaluation object and baselines. The object under evaluation is the training monitor. The main baselines are therefore diagnostic: the transferred top-1 warning, max-gradient-up-to-step-k, loss-atstep-k, mask-ratio covariates, and matched AR masked-CE controls. PEFT variants enter as boundary and method-comparison cohorts, but the claimbearing question stays fixed: whether an early DLM-LoRA monitor can route top-decile final-loss configurations to inspection better than the transferred top-1 warning under family-specific thresholds. Table 1 summarizes the action-facing verdict. 4.1
Top-1 Has Zero Precision; Max Gradient Norm Provides Calibrated Triage
The audited LLaDA-family runner exposes a top-1-frequency collapse heuristic: it emits top1_warning_detected when more than
Held-out precision check. A fixed 80/20 split over the full 671-configuration LLaDA-family cor4
Step-k monitor timing
pus (ntrain =536, ntest =135), including the methodcomparison cohort (n=395), with the max-gradient threshold F1 -optimized on train predicts top-decile final-loss on test with precision 0.68, recall 0.94, and F1 =0.79.2 A separate B=1000 splitbootstrap, which resamples configurations, redraws the train/test split, and reselects the threshold on train in each replicate, gives 95% precision CI [0.500, 0.947], disjoint from the always-positive baseline ceiling 0.148. This is a roughly 5× precision lift over the fixed-split baseline; even the lower confidence bound is more than 3× the baseline ceiling, but the absolute precision remains moderate. The supported use is therefore inspection and routing rather than an automatic decision rule; the pooled evaluation is appropriate because the primary grid alone is underpowered for threshold calibration (Limitations). A late-vs-early gradientratio rule and its conjunction with the threshold are less precise (App. A); the next paragraph reports a separate B=200 random-split timing sweep.
1.0
max-gradient
train loss
top-1
Held-out precision
final-loss label
inspection window
0.8 0.6 0.4 0.2 all-positive ceiling
0.0 5
10
11
25
50
100
200
Training checkpoint k
Figure 2: Step-k precision separates stable inspection from final-loss hindsight. On the 671-configuration LLaDA-family corpus, max-gradient precision stabilizes through the step-25–100 window where loss-atstep-k is least reliable; the step-200 loss point is the final-loss label. Colored ribbons are bootstrap 95% CIs; the gray band marks the inspection window; the dotted line is the all-positive precision ceiling. The top-1 line (blue) exceeds this ceiling at k≥10 because random splits can correlate top-1 with loss by chance; the zeroprecision result is on the fixed full-corpus split (0/816 collapses).
Early-warning timing: stable inspection before late loss settles. The practical case for max gradient norm is timing, not absolute precision or compute saving. We sweep three predictors — max-gradient-up-to-step-k, loss-at-step-k, and max top-1 token-frequency-up-to-step-k — across k ∈ {5, 10, 11, 25, 50, 100, 200} on the same 671configuration corpus with B=200 random 80/20 splits and F1 -optimized thresholds on train (Fig. 2; Tab. 3). Max-gradient precision stabilizes at 0.73– 0.75 from step 25 onward. Loss-at-step-k is nonmonotonic: it spikes to 0.79 at step 11, dips to 0.50–0.65 at steps 50–100, and becomes tautological at step 200 because loss is then the label. A practitioner reading only loss at step 11 would observe higher single-point precision (0.79) but loses reliable signal for any inspection triggered between steps 25 and 100; max gradient norm accumulates in the same training logs without requiring an additional forward pass and remains predictive across the full step-25–100 window. Max top-1-up-tostep-k never exceeds 0.27, consistent with the preequilibrium-artifact framing.
norm at every training step and inspect it by step ∼ 25. Calibrate high-gradient thresholds per family, e.g., absolute max-grad values above 50–100 in the LLaDA2.0-mini logging scale or above a locally calibrated high quantile. These thresholds are inspection triggers, not prospectively validated cross-family cutoffs or compute-saving policies. Appendix A gives the logging fields needed to implement this protocol. 4.2
A U-Shaped Gradient Instability Profile Across Mask Ratio
We extend the analysis to LLaDA2.0-mini (Bie et al., 2025) (inclusionAI/LLaDA2.0-mini, 15.93B MoE, mask token ID 156895), a more recent and larger masked diffusion model. We configure 60 unique rank×mask combinations (n=144 total runs including multi-seed replications): ranks {4, 8, 16, 32, 64} × 12 mask ratios spanning ρ ∈ [0.05, 0.95] (raw grid in Appendix Table 4), with a 200-step budget at lr = 10−4 . The legacy top-1 collapse detector early-stops all 60 raw traces at step 11, so this surface should be read as an observedprefix short-run diagnostic rather than a completed 200-step trajectory.
DLM-LoRA triage workflow. The operational recipe is deliberately narrow. Drop the top-1 collapse warning as a PEFT early-warning signal at ≤ 1K step horizons. Log max LoRA gradient
U-shaped instability profile. Unlike the nonmonotone rank-optimum flip observed in LLaDAInstruct (§4), LLaDA2.0-mini reveals a U-shaped
2
The top-decile final-loss label is defined on the full corpus; by chance the test-split unstable fraction is 13.3% (18/135) vs. 9.3% in train.
5
Table 1: Actionable monitor verdict. The deliverable is not a new PEFT method; it is a claim-matched triage protocol for DLM-LoRA training monitors at the tested horizons. Decision
• Low-mid operating region (mask ∈ [0.30, 0.40]): in the direct one-seed observed-prefix grid, these configurations have low gradient norms across all five ranks, with ρ=0.45 supported only by a narrower r=64 boundary run. In the 3-seed completed grid, the lowest mean gradient norms shift toward low-mid masks and the r=64 values at ρ ∈ {0.30, 0.40} are noisy; we therefore base the practical recommendation on convergence and held-out CE evidence rather than on a replicated global gradient minimum. Practical recommendation: avoid ρ > 0.70 for LLaDA2.0-mini LoRA at lr=10−4 in the tested setup; treat ρ = 0.30–0.40 as a conservative low-mid default, not a global optimum.
Evidence-backed use
Top-1 fires in 816/816 DLM configurations with 0/816 actual collapse; not a PEFT collapse detector at this horizon. Fixed-split max-gradient precision 0.68, Use triage recall 0.94; B=200 random-split sweep stable by step 25; inspection trigger only. Keep baseline Loss-at-k precision reaches 0.79 at step 11 but falls to 0.50–0.65 at steps 50– 100; step 200 is the label by construction. DLM-LoRA AR masked-CE controls have 0/360 only collapse and smaller or inconsistent separation, so the warning failure is scoped to the tested DLM-LoRA monitor transfer. No global cutoff Cross-family thresholds do not transfer; high-gradient values are inspection triggers only after per-family calibration. Drop warning
LLaDA2 (Bie et al., 2025) independently reports high gradient variance at extreme masking during pre-training and clips their noise-schedule coefficient within [αmin , αmax ], a bandwidth that maps to our operating window. Our LoRA characterization adds the rank dimension and shows amplification concentrates on the high-mask arm.
gradient instability profile across mask ratio (companion to Figure 1 panel B). Because top-1 fires uniformly, this surface explains where maxgradient triage from §4.1 becomes useful. The highmask arm corresponds to high-mask gradient amplification: in this observed-prefix grid, fine-tuning LoRA on a DLM at ρ > 0.70 produces gradient magnitudes up to 6.0× larger than the operatingwindow maximum (34.8 vs. 5.8), in proportion to LoRA rank. The left arm is sparse-signal variance. We use these two names throughout: Two instability mechanisms:
Replication correction. The 1-seed observedprefix 12.9× high-mask amplification contracts to 1.19× under the 3-seed full-grid replication (Table 4), with high-mask std as large as 60. We therefore use the replicated surface as the canonical magnitude estimate and keep symbolic-regression descriptors as appendix-only exploratory summaries, not decision rules.
• Left arm (mask < 0.15): Sparse supervision, only 5–15% of tokens are masked per sequence. The per-batch gradient estimate has high variance (few prediction targets, noisy signal). Gradient norm spikes reach 7.7–23.6 across ranks.
Standard AR PEFT does not traverse this surface. Standard AR LoRA fine-tuning uses dense next-token supervision over all non-first positions, has no mask-ratio dimension, and therefore cannot exhibit a rank–mask interaction in its standard recipe. The natural follow-up question is whether the U-shape we observe on LLaDA2.0-mini is a property of DLM bidirectional architecture or of the masked-CE objective itself. We answer that with a paired AR control in the next subsection.
• Right arm (mask > 0.70): High-mask gradient amplification, predicting 70–95% of tokens simultaneously produces a high-entropy prediction task with large loss and gradient magnitudes. Rank amplifies this arm directionally: in the 1-seed surface (Table 4) r=4 at ρ=0.95 reaches 2.7 and r=64 reaches 34.8 (12.9×); in the 3-seed replication (Table 4) the same configurations give 34.5±9.7 and 41.2±17.5 respectively (1.19×, n=3 seeds), so we keep the high-mask asymmetry as a directional finding and the 3-seed values as the canonical magnitude.
4.3 AR Baseline Control (summary; full grids in App. C) To isolate the masked-CE objective from DLM bidirectional attention, we ran matched randommask cross-entropy controls on Pythia-1B, Pythia {410M, 2.8B, 6.9B}, and Qwen3.5-9B, with 360 audited configurations across main grids and 6
figuration crosses 0.95 within 4 optimizer steps. The legacy detector samples at step 11 and fires in every configuration because the threshold (0.5) is below the corpus-wide initialization distribution. Stable configurations (sub-median final loss) and unstable configurations (top-decile final loss) have indistinguishable median fire-step (11 vs 11; Mann–Whitney U two-sided p=0.20, nstable =336, nunstable =68; remaining 267 mid-band configurations are excluded from this stability contrast). The complementary saturation-step diagnostic (first step where top-1 crosses 0.95) is significant in the opposite direction: unstable configurations saturate faster (median 1.0) than stable configurations (median 4.0), p=4.7×10−5 (nstable =178, nunstable =68; conditioned on configurations that crossed 0.95 by step 200). A signal that saturates before training, fires faster on unstable runs, and is uniform across the corpus cannot discriminate stability; it measures a pre-equilibrium argmax-concentration artifact of LoRA’s small-magnitude initialization plus a few masked-CE updates against an alreadyconfident pre-trained DLM. Figure 3 reports the aggregate timing evidence.
extended-mask supplements. At matched configuration (r=64, ρ=0.40), the max-grad-norm magnitude is 2.0×–2.5× smaller on AR than on LLaDA2.0-mini (16.61 on Pythia-1B vs 33.4 on LLaDA2.0-mini); the high-vs-mid ratio at r=64 is 1.20× on Pythia-1B AR control vs 2.54× on LLaDA2.0-mini, and Qwen3.5-9B shows a crossfamily mid-mask peak rather than the U-shape. AR controls report 0/360 actual collapses, supporting a DLM-family-scoped interpretation rather than a masked-CE-generic one. The grid, denominator, and cross-architecture detail live in App. C; we keep here only the body-essential conclusion: masked-CE alone is not sufficient to reproduce the DLM-family rank-amp magnitude, so the maxgradient triage signal in §4.1 is calibrated against the DLM family it serves. 4.4
DLM Scale-Architecture Boundary (summary; full grids in App. C)
The LLaDA2.0-mini operating window survives unevenly across DLM scales and architectures. The loss-side high-mask disadvantage replicates on Dream-7B (7B dense, lr-calibrated; App. C) and LLaDA2.1-mini (4-configuration transfer; App. C), but the rank-amplification direction is mixed on MDLM-OWT-130M (n=3 replication; the earlier single-seed lr-modulation pattern does not survive replication) and softens on LLaDA-MoE-A1B (1.4B small-MoE: gradient-side amplification 1.89– 2.47×, loss-side flat; App. C). We claim DLMfamily scope rather than an architecture-general window, with per-model lr calibration required; the full scale-boundary table and per-model lr discussion are reported in App. C. 4.5
Level 2 (parameter-side): rank-amp is optimization routing, not token routing. The token-side concentration above coexists with a near-uniform per-position information density, so the warning signal must measure something other than tokendistribution concentration. At the worst rankamplification corner (r=64, ρ=0.95, LLaDA2.0mini, n=3 seeds, last-10 steps; App. D), the pertoken cross-entropy-gradient distribution has Gini 0.287 ± 0.056 and the largest evaluated token position contributes only 1.54% ± 0.17% of total CE-gradient mass (uniform baseline 0.8%). In the same runs, the LoRA-parameter gradient distribution has Gini 0.463 ± 0.031 and a single LoRA matrix carries 63.0% ± 3.6% of total parameterside gradient mass. At this high-mask corner, rankamplification is therefore an optimization-routing phenomenon: the masked-CE signal arrives spread across token positions but is funnelled through a small subset of high-rank LoRA adapters in the late trajectory. Max gradient norm samples that late-trajectory routing, supporting why it carries discriminative information that top-1 does not.
Why Top-1 Fires in Every DLM Configuration: A Two-Level Characterization
The 816/816 fire vs 0/816 collapse asymmetry reflects a structural mismatch between what the metric measures and what training stability requires. We characterize it with two corpus-wide measurements that decouple token-space concentration from parameter-space gradient routing. Level 1 (token-side): top-1 is saturated before training. Across all 671 LLaDA-family configurations, the top-1 token frequency at training step 0 has mean 0.83 and standard deviation 0.13; 100% of configurations are already above 0.5 at step 0, and 65% are already above 0.8. The median con-
What this characterization predicts. The useful monitor should depend on late-trajectory parameter dynamics, not early token confidence: max-gradient fits this pattern inside the calibrated 7
is not significant (paired t p=0.40), and Table 6 shows no Bonferroni-corrected rank-64 advantage.
LLaDA-family split, while the always-positive top1 warning does not. The pathology is scoped to LoRA-on-pretrained-DLM regimes; AR controls (App. C) and the DLM scale-boundary check (App. C) support this boundary. Huang and Mirzasoleiman (2026) studies masked-diffusion signal/noise decomposition in a different generalization regime. The full pre-equilibrium trajectory and timing breakdown are shown in Fig. 3.
5
Related Work
LoRA/PEFT work introduces low-rank and quantized adapters (Hu et al., 2022; Dettmers et al., 2023; Liu et al., 2024) plus rank-allocation and optimizer-side variants (Zhao et al., 2024; Zhang et al., 2025; Chang et al., 2025; Park et al., 2025), but these works study generic or AR adaptation regimes rather than DLM mask-ratio monitor transfer. DLM work studies objectives and decoding (Sahoo et al., 2024; Nie et al., 2025; Ye et al., 2025), scaling and surveys (Bie et al., 2025; Li et al., 2025a), mask-agnostic fine-tuning (Piskorz et al., 2025), and recent systems or adapters including noise-aware LoRA (Kuiper et al., 2025; Yang et al., 2026; Xu et al., 2025; Wang et al., 2026); these improve dLLM adaptation or inference but do not test whether top-1 collapse warnings transfer into supervised LoRA fine-tuning. We use the term collapse for training-time top-1 argmax saturation, distinct from the representational layer collapse reported in fully-trained DLMs (Conzelmann et al., 2026); auditing warning-signal precision under matched false-positive control has precedent outside language modeling (Mullett, 2026). Like Schaeffer et al. (2023), we show that a familiar metric changes meaning outside its calibration regime; App. E gives the fuller taxonomy.
Why no single-axis intervention prevents saturation. The empirical 816/816 identity is consistent with a masked-CE convergence argument: if fitting increases expected top-1 mass before optimization settles, then convergence-preserving single-axis interventions should preserve the legacy fire event. Two probes show the boundary. A loss-level entropy bonus on MDLM-OWT with λ ∈ {0.5, 1.0, 2.0, 5.0, 10.0} does not reduce top-1 mass at this horizon (−0.008 at λ=0.5, +0.051 at λ=10; App. D); canonical PiSSA improves MDLM-OWT final loss (−0.43 at 200 steps; 1.82 → 1.24, paired delta −0.57 at step 1000) without changing the fire identity (App. D). App. D reports all thirteen probes; the bound is explanatory scaffolding, not a load-bearing theorem. Scope refinements. The low-mid operating region does not define an architecture-general optimum: some final-loss probes prefer lower masks, while the convergence and held-out CE probes mainly support avoiding high-mask regimes in LLaDA-family settings (App. A). The worst rankamplification corner shifts from ρ=0.95 (12.9×, one seed) to ρ=0.90 (84.7±60.4, three seeds); high-mask capacity effects and LLaDA2.1-mini transfer remain underpowered. We therefore state a scoped diagnostic, not an architecture-general recipe.
6
Conclusion
Top-1 fires in 816/816 configurations while observed collapse is 0/816 across three DLM families because the token-side signal saturates before training stability is observable. Max gradient norm instead gives a family-local inspection signal: precision 0.68 on the pooled LLaDA-family split and stable step-25–100 behavior. The scoped recommendation is to drop top-1 as a PEFT collapse warning, log max-gradient for inspection, and recalibrate mask ratio per model before reusing inference-time confidence monitors as training alarms.
Task-performance sanity check. A small indomain masked-CE convergence probe checks whether the gradient surface predicts downstream loss reduction, not generation accuracy. LLaDA2.0mini is trained for 1000 steps on 152 hand-written arithmetic examples, crossing rank {4, 64} with mask ratio {0.40, 0.90} and evaluating masked-CE on 20 disjoint prompts (App. C). Finding. Table 5 matches the surface ordering: operating-window configurations (ρ=0.40) reach lower final and holdout losses than high-mask configurations (ρ=0.90). The within-window rank gap
Limitations Budget and seeds. The primary 60-configuration rank×mask grid uses n=3 seeds at 200 training steps, expanded to n=10 at twelve critical configurations. Power analysis (App. C) places adequate detection of 2× ratios at n ≥ 30 for high-mask 8
configurations, so rank-amplification magnitudes are directional estimates; the top-1 refutation and max-gradient triage claims rest on the larger audited denominators. Architecture and adapter scope. The maxgradient precision claim is calibrated on the pooled LLaDA-family corpus (n=671); the primary rank×mask grid alone (n=264) contains too few unstable configurations to calibrate a held-out threshold reliably, so the pooled evaluation is the appropriate unit. The zero-precision top-1 denominator additionally includes Dream-7B and MDLMOWT-130M boundary cohorts across four LLaDAfamily cohorts (2–3 model checkpoints). Adapters are placed on attention projections (q,k,v,o); MLP, embedding, and LM-head LoRA placement, quantization-mask interaction (Zhang et al., 2026; Wu et al., 2026), and fully matched dense LLaDA8B replication are follow-up axes. Task and use scope. The task probe is an in-domain masked-CE convergence check rather than an accuracy-grade generation benchmark. Max-gradient is therefore presented as the tested LLaDA-family alternative to top-1 for early inspection, while coupled (ρ, r, family) intervention design and generation-quality gains remain separate claims for future work. Low-mid mask ratios are a conservative LLaDA-family default in the tested setup, not a global optimum; per-architecture validation is required. Diagnostic horizon. The 816/816 zeroprecision result is bounded to short-run PEFT diagnostics at the tested horizon. We test the inherited legacy warning threshold (> 50% argmax concentration at step 11); recalibrated thresholds or alternative top-1-derived statistics could behave differently and remain unvalidated. Separate 2000-step sidecars on Dream-7B (27/27 fire, 0/27 collapse) and LLaDA2.0-mini MoE (9/9 fire, 0/9 collapse) are consistent with this warning-failure pattern, but remain outside the 816-configuration headline denominator. The result should not be read as a claim about full fine-tuning, DLM pretraining from scratch, or budgets beyond these bounded sidecars.
NC-SA-4.0); the 152-example instruction corpus is hand-written, no PII, no scraped third-party content. Backbone weights are publicly released (LLaDA-family per model cards; LLaDA-MoE7B-A1B per Zhu et al., 2025; Pythia Biderman et al., 2023 + Qwen3.5-9B Qwen Team, Alibaba, 2026 under Apache 2.0). Aggregate compute is ∼ 119 kg CO2 eq total, estimated from reported GPU-hours and US grid-intensity context (Electricity Maps, 2024). The max-gradient triage protocol operates only on training diagnostics and produces no model outputs; we do not anticipate disproportionate or novel harms beyond those already present in supervised LoRA fine-tuning. AI assistants were used for coding support, layout repair, audit checklists, and prose editing; all claims, numbers, and experimental results were author-verified against local run artifacts. Public artifacts include the arXiv source, reference logging scripts, and sanitized aggregate result JSON/CSV files backing the tables and figures (GitHub repository; result artifacts). The public artifacts intentionally exclude raw per-run prompts/completions, W&B metadata, local paths, checkpoints, and adapter weights.
Ethical considerations
Mark Chen and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
References Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In ICML. Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, Chengxi Li, Chongxuan Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Xiaocheng Lu, Yuxin Ma, and 12 others. 2025. LLaDA2.0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745. Yupeng Chang, Chenlu Guo, Yi Chang, and Yuan Wu. 2025. LoRA-MGPO: Mitigating double descent in low-rank adaptation via momentum-guided perturbation optimization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 648–659. Association for Computational Linguistics.
All training and evaluation data are publicly released English-language benchmarks under permissive licenses (GSM8K (Cobbe et al., 2021), HumanEval (Chen et al., 2021), MMLU (Hendrycks et al., 2021): MIT; MetaMathQA-5K: CC-BY-
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
9
2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. PiSSA: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems.
Alexander Conzelmann, Albert Catalan-Tatjer, and Shiwei Liu. 2026. Layer collapse in diffusion language models. arXiv preprint arXiv:2605.06366.
David Mullett. 2026. Benchmarking recursive-collapse warning claims under matched false-positive control. arXiv preprint arXiv:2606.00329.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. arXiv preprint arXiv:2502.09992.
Electricity Maps. 2024. Electricity map: Live CO2 emissions of electricity consumption. https:// app.electricitymaps.com. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR).
JuneYoung Park, Minjae Kang, Seongbae Lee, Haegang Lee, Seongwan Kim, and Jaeho Lee. 2025. Riemannian optimization for LoRA on the stiefel manifold. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 20971–20985. Association for Computational Linguistics.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).
Julianna Piskorz, Cristina Pinneri, Alvaro Correia, Motasem Alfarra, Risheek Garrepalli, and Christos Louizos. 2025. Masks can be distracting: On context comprehension in diffusion language models. arXiv preprint arXiv:2511.21338.
Jianhao Huang and Baharan Mirzasoleiman. 2026. Tuning the implicit regularizer of masked diffusion language models: Enhancing generalization via insights from k-Parity. arXiv preprint arXiv:2601.22450.
Qwen Team, Alibaba. 2026. Qwen3.5-9B model card. Hugging Face model card.
Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, and Eunhyeok Park. 2025. GraLoRA: Granular low-rank adaptation for parameter-efficient finetuning. In Advances in Neural Information Processing Systems.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. 2024. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems (NeurIPS).
Damjan Kalajdzievski. 2023. A rank stabilization scaling factor for fine-tuning with LoRA. arXiv preprint arXiv:2312.03732.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? In NeurIPS.
Ruurd Jan Anthonius Kuiper, Lars de Groot, Bram van Es, Maarten van Smeden, and Ayoub Bagheri. 2025. LAD: LoRA-adapted diffusion. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
Shuaidi Wang, Zhan Zhuang, Ruping HUANG, and Yu Zhang. 2026. NaRA: Noise-aware LoRA for parameter-efficient fine-tuning of diffusion LLMs. arXiv preprint arXiv:2605.29716.
Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. 2025a. A survey on diffusion language models. arXiv preprint arXiv:2508.10875.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics.
Zhizhong Li, Sina Sajadmanesh, Jingtao Li, and Lingjuan Lyu. 2025b. StelLA: Subspace learning in low-rank adaptation using stiefel manifold. In Advances in Neural Information Processing Systems. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weightdecomposed low-rank adaptation. arXiv preprint arXiv:2402.09353.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. 2026. Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding. In International Conference on Learning Representations (ICLR).
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/ huggingface/peft. Software library.
10
Guowei Xu, Wenxin Xu, Jiawang Zhao, and Kaisheng Ma. 2025. GIFT: Guided importance-aware finetuning for diffusion language models. arXiv preprint arXiv:2509.20863. Jingyi Yang, Yuxian Jiang, Xuhao Hu, Shuang Cheng, Biqing Qi, and Jing Shao. 2026. Dare: Diffusion large language models alignment and reinforcement executor. arXiv preprint arXiv:2604.04215. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Proceedings of the 41st International Conference on Machine Learning, pages 57755–57775. Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In ICLR. Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, and Hao Xu. 2025. Sensitivity-LoRA: Lowload sensitivity-based fine-tuning for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 13185– 13199. Association for Computational Linguistics. Tianao Zhang, Zhiteng Li, Xianglong Yan, Haotong Qin, Yong Guo, and Yulun Zhang. 2026. QuantdLLM: Post-training extreme low-bit quantization for diffusion large language models. In International Conference on Learning Representations (ICLR). Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. GaLore: Memory-efficient LLM training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang, Xudong Wang, Lanning Wei, Hongrui Guo, Jiaqi Hu, Wentao Ye, Tieyuan Chen, Chenchen Li, Chengfu Tang, Haibo Feng, Jun Hu, Jun Zhou, and 7 others. 2025. LLaDA-MoE: A sparse MoE diffusion language model. arXiv preprint arXiv:2509.24389.
11
A Top-1 is saturated before the warning sample
Reproducibility and Source Trace
Top-1 token frequency
A
Reproducibility scope. The arXiv source package contains the manuscript source, bibliography, and rendered figures. The public artifact release contains paper source, reference scripts, and the sanitized aggregate result JSON/CSV files that back the tables and figures at GitHub repository (result artifacts). The manuscript values are sourcemapped through local run manifests and claimbearing aggregates rather than copied from tracker prose. The release excludes raw prompts/completions, W&B metadata, local paths, checkpoints, and adapter weights.
1.0
0.95 saturation
0.8
All n=671 Stable n=336 Unstable n=68
step 0: 0.83 ± 0.13
0.6
step 11 sample
0
25
0.5 warning
50
75
100
125
150
175
Training step
Median step
B Timing does not separate stable from unstable runs
10
p = 0.20 11 (no split) 11
p = 4.7 × 10−5 (opposite direction) 4 1
0 Legacy fire step
Compute and setup. The primary LLaDAfamily experiments use H100 NVL-class GPUs; Dream, preliminary LLaDA, and some AR controls use L40S-class GPUs. The paper accounts for approximately 396 GPU-hours across reported experiment groups and estimates ≈ 119 kg CO2 eq. Models are used through HuggingFace transformers and PEFT with explicit masked cross-entropy on masked positions. API pitfalls needed for reproduction are: DLM forward passes may not return a supervised loss, generate() is not the training-time denoising loop, target modules must be explicit, Dream-7B loads through AutoModel, and Dream attention masks must be boolean.
0.95 saturation step
Figure 3: Top-1 collapse is a pre-equilibrium artifact. (A) Across 671 LLaDA-family configurations, top-1 mass starts high and crosses the legacy threshold before the detector samples. (B) The legacy fire-step has no stable/unstable split, while the stricter 0.95 saturation step points in the opposite direction: unstable configurations saturate faster.
Stable and unstable configurations have the same median legacy fire-step (11 vs. 11, Mann–Whitney p=0.20). The stricter 0.95 crossing points in the wrong direction for a collapse detector: unstable configurations saturate faster. B.1
Step-k Precision Sweep
Table 2: Top-1 warning and max-gradient summary. The top-1 warning fires in every audited DLM-family configuration while actual collapse is zero at the tested horizon. Max-gradient separation is family-local, not a global threshold.
Table 3: Step-k held-out precision sweep. Median precision over B=200 random 80/20 splits on 671 LLaDAfamily configurations. Max-gradient is stable from step 25 onward; step-200 loss is the label by construction. k
max-grad
loss-at-k
max top-1
Cohort
Top-1 / coll.
Max-grad evidence
DLM LLaDA2.0-mini-full (n=144) DLM LLaDA2.0-mini-crit12 (n=120) DLM LLaDA-method-comp (n=395)
144/144; 0/144
5 10 11 25 50 100 200
0.22 [0.09, 0.36] 0.69 [0.45, 0.88] 0.71 [0.50, 0.92] 0.73 [0.53, 0.92] 0.75 [0.53, 0.93] 0.74 [0.53, 0.92] 0.74 [0.53, 0.92]
0.13 [0.08, 0.18] 0.69 [0.50, 0.91] 0.79 [0.57, 0.94] 0.56 [0.36, 0.88] 0.64 [0.40, 0.87] 0.50 [0.26, 0.82] 1.00 [0.86, 1.00]
0.14 [0.06, 0.24] 0.19 [0.11, 0.31] 0.23 [0.13, 0.35] 0.26 [0.17, 0.38] 0.26 [0.17, 0.38] 0.25 [0.17, 0.38] 0.26 [0.17, 0.38]
DLM LLaDA2.1-mini (n=12) DLM Dream-7B boundary (n=100) DLM MDLM-OWT-130M boundary (n=45)
12/12; 0/12 100/100; 0/100
3.23× [2.76, 3.97]; 2.7×10−7 1.48× [1.12, 1.76]; 0.036 362× [202, 779]; 5×10−21 (source scale) small n boundary cohort
45/45; 0/45
boundary cohort
C
Surfaces, Controls, and Task Probe
AR Pythia/Qwen masked-CE controls (n=360)
–; 0/360
smaller or inconsistent
B
120/120; 0/120 395/395; 0/395
Scale boundary. The LLaDA2.0-mini low-mid mask recommendation is not architecture-general. Loss-side high-mask disadvantage replicates on Dream-7B after learning-rate calibration and in a small LLaDA2.1-mini transfer check, but rankamplification direction is mixed on MDLM-OWT130M and softens on LLaDA-MoE-A1B. This is why the body states a DLM-family diagnostic and requires per-model calibration.
Top-1 Saturation and Step-k Precision Sweep
Timing. Top-1 token frequency is 0.83 ± 0.13 at step 0 on the LLaDA-family corpus. All configurations are already above 0.5 at step 0, and the median crosses 0.95 within four optimizer steps. 12
Table 4: LLaDA-family surface summaries. The table preserves the source-traced values used in the body; sanitized per-configuration grids are released in the public artifact. Evidence slice
Source-traced contrast
mass). LoRA-parameter gradients are much more concentrated (Gini 0.463 ± 0.031; one LoRA matrix carries 63.0% ± 3.6% of gradient mass). This supports the body interpretation that max-gradient samples parameter-side routing while top-1 samples token-side pre-equilibrium concentration.
Reading
1-seed r=64 surface
max-grad ρ=0.30/0.40: 4.6/5.8; ρ=0.90/0.95: 15.0/34.8 r4 replicated rows mean max-grad (n10/n3): ρ=0.40: 16.4±1.3; ρ=0.90/0.95: 63.3±60.0/34.5±9.7 r64 replicated rows mean max-grad (n10/n3): ρ=0.40: 33.4±11.0; ρ=0.90/0.95: 84.7±60.4/41.2±17.5 3-seed rank ratio r4→r64 ratio ρ=0.40: 2.04×; ρ=0.90/0.95: 1.34/1.19× 10-seed critical configs ρ=0.90 vs 0.40: max-grad ratios 3.87/1.69/2.53× and final-loss deltas +0.17/ + 0.17/ + 0.18 for ranks 4/16/64 LLaDA2.1 transfer ρ=0.90 vs 0.40: max-grad ratios 1.76/3.52× and final-loss deltas +1.11/ + 0.99 for ranks 4/64
tail noisy
Table 7: Single-axis boundary audit. No tested singleaxis intervention prevents the fire-rate identity. The paper therefore remains diagnostic rather than a prospective controller paper.
elevated corrected desc.
scoped
Table 5: In-domain convergence probe. Values are mean ± std over 3 seeds; lower held-out CE is better. Regime
Rank
stable stable high mask high mask
4 64 4 64
ρ Max∥∇∥ Holdout CE 0.40 0.40 0.90 0.90
18.0 21.0 73.9 96.4
0.43 ± 0.08 0.42 ± 0.10 0.57 ± 0.06 0.48 ± 0.04
n
r4 CE
r64 CE
Fire?
Activation timing: gating window and learning-rate trigger Magnitude: learning-rate warm-up N ∈ {10, 20, 50} steps Init amplitude: LoRA-B perturbation Init direction: spectral-init only (no weight subtraction) Spectral-init with weight subtraction (Meng et al., 2024) Adapter/optimizer geometry
no timing shift
no
no timing shift
no
no timing shift
no
shifts 11→33, but with first-update overshoot improves loss while preserving step-0 identity Low-rank group bottleneck (G=4) (Jung et al., 2025) and Stiefel projection do not remove fire λ ∈ {0.5, 1, 2, 5, 10} does not reduce fire normalized thresholds remain family-specific
no
Portability checks
no no
no –
Definitions and non-portability. The logged top-1 warning is an argmax mode-frequency statistic, not mean maximum probability: for runner input zt , evaluated positions It , and at (i) = arg maxu∈V pθt (u | zt , i), it uses Sbt = maxv |{i ∈ It : at (i) = v}|/|It | and St = E[Sbt ]. For LLaDA-family runs, zt is the clean-batch proxy and It all positions; for MDLM-OWT, zt is the masked training input and It masked positions. Crossing a fixed threshold can therefore indicate pre-equilibrium argmax concentration rather than divergence. The corresponding max-gradient sketch is only a family-local scale heuristic:
∆ pBonf
GSM8K-test 1319 2.66±2.30 1.92±0.32 +0.74 HumanEval 164 1.10±0.03 1.64±0.31 −0.54 MMLU-subset 250 1.10±0.04 1.27±0.11 −0.17
1.00 1.00 1.00
AR control. Pythia and Qwen masked-CE controls show 0/360 actual collapses and smaller or inconsistent max-gradient separation. The denominator is Pythia-1B main 5×12×3 (180 configurations), Pythia-410M and Pythia-6.9B matched grids (45 each), plus five 18-configuration Pythia/Qwen sweep or extended-mask blocks. At matched (r=64, ρ=0.40), Pythia-1B max-gradient is 16.61 versus 33.4 on LLaDA2.0-mini, and Qwen3.5-9B shows a mid-mask peak rather than the LLaDA-family U-shape.
D
Observed outcome
Loss-level entropy bonus
Table 6: Operating-window multi-benchmark masked-CE check. The ρ=0.40 rank contrast is not significant after correction. These null results bound the low-mid-mask recommendation to the DLM-LoRA training diagnostic and do not support a downstream generation-quality claim. Benchmark
Axis and probe
GT := max ∥∇θLoRA L(θt )∥2 , 0≤t≤T (2) αL p GT ≤ Cfam T log T σfam (ρ, r, V ), r where Cfam absorbs model/data constants, αL is the LoRA scaling factor, r is LoRA rank, V is the output vocabulary, and σfam denotes the empirical gradient-scale term induced by mask ratio, rank, and family. We do not assert a universal closedform bound for σfam . The sketch is scaffolding, not the basis for the claim: empirically, cross-family normalization reduces raw scale variance but loses portable precision because correlations sign-flip by family, especially on the small MDLM-OWT cohort.
Mechanism and Boundary Audit
Gradient concentration. At the worst rankamplification corner (r=64, ρ=0.95, LLaDA2.0mini, n=3 seeds, last 10 steps), per-token CE gradients are only modestly concentrated (Gini 0.287 ± 0.056; the largest evaluated token position contributes 1.54% ± 0.17% of CE-gradient 13
E Method Comparison and Related Work Method references. Named probes follow PiSSA, GraLoRA, StelLA, rsLoRA, Yu-DARE, and NaRA (Meng et al., 2024; Jung et al., 2025; Li et al., 2025b; Kalajdzievski, 2023; Yu et al., 2024; Wang et al., 2026). Table 8: Operating-cell method comparison. Sourcemapped masked-CE summary. Protocol
Claim-facing conclusion
Default learning rate
rsLoRA is higher CE on all three benches; Yu-DARE trends similarly with high seed variance, so we treat this as learning-rate mismatch. rsLoRA remains higher CE (+3.7–4.2%); NaRA is lower (−1.0–4.7%), with only MMLU Bonferroni-significant. This is learning-rate-dependent, not a method-quality claim. Exact match is 0/20, so generation quality is excluded from paper claims.
Best learning rate (n=10) GSM8K gen. check
Related work taxonomy. LoRA-family and ARside PEFT stability work assume dense next-token supervision and do not expose a mask-ratio axis. DLM work covers objectives, scaling, decoding, masking schedules, train-inference mismatch, and systems that use fixed LoRA-like adapters, but we are not aware of prior work that tests top-1 warning precision as a DLM-LoRA PEFT monitor with matched AR masked-CE controls. The closest genre is metric refutation: a familiar diagnostic changes meaning outside its calibration regime. Scope. Claim-bearing denominators are 816 DLM PEFT configurations, 671 LLaDA-family configurations, and 360 AR masked-CE controls; longer horizons, generation quality, full fine-tuning, and coupled controllers remain future work.
14