Preprint
A DA F LASH : A DAPTIVE S PECULATIVE D ECODING VIA O N -P OLICY D ISTILLED D IFFUSION D RAFTERS Yu-Yang Qian1,2,* Hao-Cong Wu1,2,* Chen Chen3 Jiacheng Sun3 Zhenhua Dong3 Peng Zhao1,2,† Zhi-Hua Zhou1,2 1 State Key Laboratory for Novel Software Technology, Nanjing University 2 School of Artificial Intelligence, Nanjing University 3 Huawei Foundation Model Dept
arXiv:2607.19223v1 [cs.LG] 21 Jul 2026
A BSTRACT Speculative decoding, in which a lightweight draft model first generates a draft sequence that is then verified in parallel by the target model, has become a prevalent paradigm for accelerating large language model inference. Recent work such as DFlash further boosts drafting efficiency by leveraging diffusion drafters, whose parallel denoising mechanism enables draft generation in a single forward pass. In this work, we uncover a central pitfall of diffusion drafters: bidirectional attention is a double-edged sword. On one hand, it endows the model with parallel generation and global contextual modeling capabilities; on the other hand, this inherent global dependency introduces high variance at both the domain-level and the token-level: acceptance rates fluctuate substantially across different domains, and draft token quality also varies heterogeneously at different token positions. To tackle this issue, we propose A DA F LASH framework, comprising two components: (i) an on-policy distillation (OPD) algorithm with reverse-KL divergence tailored for diffusion drafters, bringing stable convergence and effectively reducing domain-level variance; and (ii) an adaptive length head that dynamically adjusts the candidate sequence length on the fly, substantially lowering the verification cost of the target model and handling token-level variance. Experiments demonstrate that A DA F LASH consistently improves speedup rate during deployment, with especially significant gains in high-concurrency scenarios, achieving up to approximately 66% higher throughput than previous state-of-the-art methods.
1
I NTRODUCTION
Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks (Brown et al., 2020; Radford et al., 2021; Ouyang et al., 2022). However, this progress comes at a price of growing computational cost: emerging techniques, including chain-of-thought (Wei et al., 2022), multi-step reasoning (Guo et al., 2025), and long-horizon agentic tasks (Xi et al., 2025), substantially increase the burden of inference. To reduce the inference latency, speculative decoding (Leviathan et al., 2023; Chen et al., 2023) has emerged as a widely adopted paradigm, in which a lightweight draft model rapidly generates a draft sequence that is then verified in parallel by a larger target model, achieving lossless acceleration (Li et al., 2024b; Fu et al., 2024; Hu et al., 2025). Beyond standard autoregressive (AR) drafters, growing interest in the community has led several recent works (Pan et al., 2026; Li et al., 2026) to explore diffusion language models (dLLMs) (Sahoo et al., 2024; Nie et al., 2025; Ye et al., 2025) as non-autoregressive drafters, leveraging their bidirectional attention and mask-and-denoise mechanism to generate multiple draft tokens simultaneously. Most recently, DFlash (Chen et al., 2026) introduced the first one-step dLLM drafter: given a prefix, it initializes k masked positions and generates all draft tokens through a single forward pass. This makes the per-token drafting overhead roughly k times smaller than that of AR drafters, yielding both lower drafting cost and higher acceptance rates. In this work, we uncover a central pitfall that the bidirectional attention inherent in the diffusion drafters is a double-edged sword: while it enables global contextual modeling and one-pass parallel *
Equal Contribution.
†
Corresponding Author: Peng Zhao <[email protected]>.
1
Preprint
generation, the resulting global dependency introduces high variance in draft quality. As illustrated in Figure 1, this variance manifests at two levels: (i) domain-level variance: the acceptance rate fluctuates significantly across task domains, much more so than that of AR drafters; and (ii) tokenlevel variance: the per-position acceptance probability varies substantially within a single draft sequence. Consequently, a static speculative system with a fixed diffusion drafter and a constant candidate length suffers from degraded acceptance rates on out-of-distribution domains, and wastes target model computation on tokens unlikely to be accepted. To address these challenges, we propose A DA F LASH, an online adaptive framework for diffusionbased speculative decoding with two components: (i) an on-policy distillation (OPD) algorithm with reverse-KL and entry-wise divergence clipping, tailored specifically to diffusion drafters, which delivers stable convergence and effectively reduces domain-level variance; and (ii) an adaptive length head that dynamically adjusts the draft sequence length, substantially lowering the target model’s verification cost with only a marginal decrease in acceptance length. We further introduce infrastructure improvements, including an asynchronous training–inference pipeline and adaptive request scheduling, that enable efficient online adaptation within a serving engine. Experiments across eight benchmarks and three different foundation models, including both dense and mixture-of-experts architectures, validate the effectiveness of our A DA F LASH. Both components yield significant performance gains, and our method consistently outperforms previous SOTA speculative decoding methods, including EAGLE-3, multi-token prediction, and DFlash, achieving up to a 5.3× speedup over standard autoregressive decoding. The advantage is especially remarkable at high concurrency, where prior methods degrade below the standard AR baseline, while our A DA F LASH sustains acceleration with up to 66% higher throughput than previous SOTAs. Organization. Section 2 introduces the background and our key observation on high variance issues in diffusion drafters. Section 3 describes our A DA F LASH framework. Section 4 evaluates the effectiveness and efficiency of A DA F LASH. Section 5 concludes the paper. Due to page limits, related work is deferred to Appendix A and additional experiments are deferred to Appendix B.
2
BACKGROUND AND K EY O BSERVATION
This section introduces the background on speculative decoding and diffusion-based drafters, and then presents our key observation regarding the high variance issues of the diffusion drafters. 2.1
S PECULATIVE D ECODING AND D IFFUSION D RAFTERS
Speculative Decoding. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) accelerates inference of a large target model pv by leveraging a smaller draft model qw . The draft model rapidly generates k draft tokens, i.e., a draft sequence, which is then verified in parallel by the target model. Specifically, each draft token xi is accepted with probability min{1, pv (xi | x<i )/qw (xi | x<i )}; all tokens up to the first rejected one are retained, and a corrected token is resampled from the residual distribution at the rejection point. This sampling procedure guarantees that the output distribution is identical to that of the target model, enabling lossless acceleration. Following Leviathan k+1 et al. (2023), the expected number of output tokens per step is E[τ ] = 1−Acc 1−Acc , where Acc ≜ Ex∼qw [min{1, pv (x | x<i )/qw (x | x<i )}] is acceptance rate. The acceleration rate is then given by E[τ ] η= , ρk + 1 where k is the draft sequence’s length (aka, candidate length), and ρ ≪ 1 denotes the ratio of the draft model’s inference time to that of the target model. This expression shows that the speedup is governed by the inference time ratio ρ and the acceptance rate. Diffusion Language Models as Drafters. Existing speculative decoding methods predominantly employ AR drafters that sequentially generate draft tokens, requiring k sequential forward passes: for i = 1, . . . , k : xi ∼ qw (· | x, x1 , . . . , xi−1 ). Recently, several works (Pan et al., 2026; Li et al., 2026) have explored diffusion language models (dLLMs) (Sahoo et al., 2024; Nie et al., 2025; Ye et al., 2025) as non-autoregressive drafters. Unlike 2
Preprint