Official Review · NEURIPS 2024

This paper presents a method for language modeling using simple masked discrete diffusion models. The authors show that a simplified objective combined with optimized training achieves performance improvements over previous diffusion language models. The paper reports state-of-the-art results for diffusion models on standard language modeling benchmarks, approaching the performance of autoregressive models. 1. The simplified diffusion objective is effective in stabilizing training and is well supported by detailed derivations. 2. Section 3 provides clear steps to build a connection between MLM and Diffusion LM 3. The empirical performance is superior to other diffusion language models. 1. I believe the paper should be structured more towards introducing the simplified objective rather than framing the architecture as MDLM since the main idea of bridging absorbing state discrete diffusion with MLM has been introduced in DiffusionBert and D3MM. 2. The comparative results are not fair comparisons. More specifically the use of a more advanced backbone (DiT) and a low-variance sampler weakens the claim that the performance and stability are sourced from the proposed method. 1. Other than the simplified objective and new backbone and sampler, what’s the difference between your method and absorbing state D3MM or more specifically DiffusionBert? 2. Is MLM pretraining required for your method?