Official Review · ICLR 2024

This paper proposes to pre-train a masked language model, and fine-tune this model as a generative discrete absorbing diffusion model for translation, summarization, and instruction-following tasks. Experiments are conducted using the XLM-RoBERTa family of MLMs (86M, 304M, 2.8B, 9.7B variants) finetuned as generative models on IWSLT14 and WMT14 (translation) Gigaword-10K (summarization) and Flan 2022 (instruction-following) datasets. This paper contains a diverse set of experiments using finetuned MLMs as diffusion models, which complements previous work using this methods (e.g., DiffusionBERT). The experiments are described well, and appear to be well-executed. The results show consistent task performance improvements using the larger-scale models, which is consistent with broader observations of the importance of scale in the ML community (although perhaps unsurprising at this point). There does not seem to be a significant methodological contribution. As far as I can tell, the methodological approach is the same as previous work adapting pre-trained MLM's for discrete diffusions [1]; the difference here is the use of the XLM-RoBERTa family of models, vs. previous results using BERT. The paper is clear that its contributions are about scale, not methodology, but it's a little frustrating that so much of the paper (up until Section 4) is dedicated to summarizing the contributions of previous work in the field. This is not the first work to study the scaling behavior of diffusion models for text. For example, multiple diffusion models are trained in [2] together with a scaling law for these models and comparisons to scaling laws for baseline autoregressive models. Because this paper's central contribution is an empirical study of scaling behavior of text diffusions, a more thorough discussion of previous work on scaling text diffusions would be helpful. [1] DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, Xipeng Qiu [2] Likelihood-Based Diffusion Language Models Ishaan Gulrajani, Tatsunori B. Hashimoto I am curious how the approach described in this paper compares to direct MCMC sampling from an MLM. e.g., the approach described in [3]. To be clear, I am quite sympathetic to the idea of finetuning to directly supervise generation rather than inference-time MCMC; but I do think some discussion of this alternative line of work and connections/tradeoffs vs. the proposed approach would be enlightening and strengthen the discussion in Section 3.2. "There exist large discrepancies in architecture and data engineering between our foundation models, XLM-R (Conneau et al., 2019), which were built years ago, and the state-of-the-art large language models like LLaMA (Touvron et al., 2023a;b)." I do not work at a scaling lab, so my insight into these questions is limited, but one explanation I have heard for the lack of recent developments in the MLM space is th