Official Review · NEURIPS 2023

This paper introduces a latent diffusion model for natural language generation, utilizing a continuous diffusion framework. The authors leverage pretrained encoder-decoder models, such as BART and Flan-T5, to establish a continuous feature space. They propose a Perceiver Resampler as a compression module to handle variable-length and high-dimensional challenges, allowing effective learning of the diffusion process and parameterized denoising. The generative process includes a reconstruction module to map the low-dimensional latent space back to the feature space and is supported in unconditional, class-conditional, and seq2seq settings. The proposed approach demonstrates its effectiveness through extensive experiments. 1. The utilization of pretrained encoder-decoder models to establish a feature space is a clever and efficient approach, distinguishing it from previous work that relied on less-capable embedding "encoder-decoder" for latent diffusion. 2. The paper conducts extensive experiments in various settings, including unconditional, class-conditional, and seq-conditional text generation. The evaluation metrics cover aspects of quality, diversity, and other relevant factors. 1. Evaluation concerns arise from the lack of a multilingual setting in the benchmark datasets investigated. The benchmark datasets investigated in the paper were various, which was good, but lacked of multilingual setting, making the evaluation not fully convincing. Several previous diffusion-based text generation studies experimented on machine translation, e.g, CDCD [2], Difformer [2], and DINOISER [3] studied on WMT14 English-to-German dataset, which is most commonly-used setting for non-autoregressive text generation. Can the proposed LD4LG model also be applied in a multilingual setting? 2. The paper lacks a comprehensive literature review, hence comparison and discussion on its contributions with relevant studies. In the experiments, only diffusion-lm and diffuseq were compared. Additionally, the related work section primarily cites papers from 2022, while significant progress has been made in text diffusion research this year. What are the advantages of LD4LG compared to other continuous diffusion models, such as CDCD, Difformer, and DINOISER? How does it compare to discrete diffusion models like DiffusionBERT, reparameterized discrete diffusion model (RDM), and DiffusER? --- [1] Continuous diffusion for categorical data [2] Difformer: Empowering Diffusion Model on Embedding Space for Text Generation [3] DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises [4] DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models [5] A Reparameterized Discrete Diffusion Model for Text Generation [6] DiffusER: Discrete Diffusion via Edit-based Reconstruction See weaknesses. I am happy to raise my rating upon the authors' responses to my concerns.