Official Review · NEURIPS 2023

This paper studies how to use a pre-trained text-to-image generative diffusion model to do discriminative tasks like image and text matching. Building upon previous works [Li et al. 2023, Clark and Jaini 2023], this paper introduces two technical contributions: 1) Unconditional normalization largely improves image retrieval; 2) Tuning the diffusion model with hard negatives on better-quality data like MSCOCO. The paper also introduces a benchmark to evaluate different diffusion models by image and text matching performance, and specifically studies the biases of two versions of Stable Diffusion model. 1. The two technical contributions (unconditional normalization & tuning on MS COCO) are effective and well ablated. 2. The introduced benchmark could be useful for the community to evaluate the prompt-following ability and biases new diffusion models. 3. The paper studied bias in state-of-the-models, which should draw more attention to the community. 1. The generative-for-discriminative methods typically take many feedforwards of the diffusion model to evaluate a single sample. Although the paper briefly mentions its slow runtime at the end of the Appendix, I think the computational cost is an important factor of the proposed algorithm and should be rigorously presented and studied in the main text, especially when the paper proposes a benchmark - A clear study on the computational cost would be important to others who plan to evaluate on this benchmark. 2. The GDBench covers several hard image-and-text matching datasets. However, I think results on other standard discriminative datasets would be helpful for the audience to understand the advantage of the proposed algorithm and compare to previous works. For example, how does HardNeg-DiffusionITM work on ImageNet compare to Diffusion Classifier [Li et al. 2023] / DiffusionITM (Ours) in Table 2(b)? I understand ImageNet is a 1000-way classification and would be a long time to run for this kind of methods (therefore studying runtime is important as mentioned above), but Diffusion Classifier [Li et al. 2023] has already reported their performance on ImageNet and an apple-to-apple comparison would be very helpful. Same applied to the CLEVR dataset. Another question is how tuning on MSCOCO works for image-text matching on MSCOCO? 3. I am not sure what is the use case to introduce MSCOCO tuning into Diffusion Classifier. It indeed improves the performance of image and text matching compare to the plain one, but all these methods are too slow (or not good enough) to be practically applied. On the other hand, tuning the weights of diffusion models prevents it from serving as a tool for evaluating diffusion models. Then in what case do we need MSCOCO tuning? 1. I am not sure how much MSCOCO HardNeg is clearly better than MSCOCO NoNeg - To me, MSCOCO NoNeg seems to be even better than HardNeg quantitatively, or at least on par with in Table 2. How do the authors land with HardNeg as the default setti