Official Review · ICLR 2024

This paper studies the scalability of discrete diffusion language models on leveraging pretrained knowledge, model scaling and zero to few shot generalizations. The authors first draw an elaborate introduction on diffusion language models. Starting from off-the-shelf pretrained language models, the authors further fine-tune them with diffusion objective following RDM, with either task-specific tuning (MT, Summ) or instruction fine-tuning (FLAN). The authors conduct comprehensive experiments to explore the performance on both down-steam tasks and generalizations of discrete diffusion models varying their scale. - This paper studies an important topic on the scalability of diffusion language models, and goes through extensive experimental explorations, which makes a valuable contribution to developing better diffusion language models. - This paper poses comprehensive experimental investigations on scaling, tuning and generalizations of discrete diffusion language models, and compares them against the T5 family. - The author also explores emergent capabilities of autoregressive (L)LMs like in-context learning and reasoning, on their adapted large diffusion language models. - The paper is well written, clear to follow. - The main method in this paper is similar to existing works: - The idea of leveraging pretrained masked models as a knowledge base and starting point for tuning discrete diffusion language models (named as "diffusive adaption") is introduced in [1]. - The main methods in this paper (tuning pretrained MLMs with diffusion objective) is also mainly adhering to the work RDM-Diffusion [2]. - The "prompt-response" format (partially diffusion on target tokens) essentially adopts the setting of DiffuSeq [3]. - While referring BERT-like diffusion language models as 'decoder-only' models, (GPT-like) AR decoder models (which is currently most popular) is not compared or discussed throughout this study. Besides, I believe that authors need further clarifications on their definitions of 'decoder' models, since we do not generally refer BERT-like models as decoder-only models, which might cause some confusions. - The paper does not fully explore the configurations of Diffusion LMs, such as the number of diffusion steps, the noise level, and the length predictor. It would be valuable to explore how these factors affect the quality and diversity of the generated texts, especially when the inference latency due to the demand on multiple diffusion steps would limit the scaling of diffusion models. For example, would larger models remedy the demand for more steps & generalize better to predicted lengths or opposite? [1] He, Z., Sun, T., Wang, K., Huang, X., & Qiu, X. (2022). Diffusion Bert: Improving generative masked language models with diffusion models. *ArXiv preprint arXiv:2211.15029*. [2] Zheng, L., Yuan, J., Yu, L., & Kong, L. (2023). A reparametrized discrete diffusion model for text generation. *ArXiv preprint arXiv:2302.05737*.