Official Review · NEURIPS 2023

The paper proposes a new setting (or sub-domain) of text-to-image generation (T2I), namely abstract-to-intricate T2I. To tackle the new setting, the authors propose a method (Salad) to enrich the scene graphs parsed from the text prompts based on the discrete diffusion models. The enriched scene graphs are used as guidance to generate complicated scenes that align better with the initial concise prompts. They report quantitative results and analysis experiments to show the effectiveness of the two-stage system in multiple metrics. - The paper addresses a practical and important problem. Users of the T2I system, in reality, have to write unreasonably long text prompts to generate images with plausible scenes/styles/semantics. Abstract-to-intricate T2I could benefit the users with a more faithful and efficient generation process. - The scene graph hallucination stage achieved with the discrete diffusion model is interesting. Enriching the text prompt in the scene graph space seems like a more controllable and stable process that can preserve the faithfulness of relations and attributes. - The authors conduct an extensive analysis of the components of the system to demonstrate its effectiveness. - Task definition. While I understand the concept of abstract-to-intricate, I think the work lacks a more rigorous definition of the task. It seems unclear why places and progressive verbs are included and other nouns or verbs are discarded in the COCO-A2I. I tried the failure prompt in Fig. 1 on Stable Diffusion, and I got good results most of the time. So what makes these prompts an A2I problem instead of a faithfulness problem of all T2I models? - My major concern is about the experimental setup. It seems that the Salad system is using the pre-trained Stable Diffusion as the image generator compared to other methods like Frido and LDM-G. The comparison could be unfair as all these baselines are trained with much fewer images and inherently have higher FID scores on MSCOCO. In addition, the authors use Frido for text-based enrichment approaches. - The writing needs improvement. There are multiple typos throughout the paper and some inaccurate sentences. For instance, line 133 "attribute (o)", line 226 "baselines: stable diffusioninspired by [6].", line 235-236 "For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization.", line 238 "UNet" (UNet in Stable Diffusion?). - What is the SGH module's output and the SIS step's input? Fig. 2 seems like you are inputting the scene graph matrix into the UNet, while Sec. 3.2 and Fig. 3 show that you convert the graphs into prompts and feed them to the attention layers. - Did you fine-tune the Stable Diffusion model by saying "For the SIS module, we load the parameters of Stable Diffusion3 (v1.4) as the initialization."? - Why do all models achieve such high CLIP scores? Is the CLIP score the cosine similarity between image-text pairs or the CLIP R-precision metric?