Official Review · ICLR 2024

In this paper, the authors proposed DiffusionExplorer which uses diffusion models to generate hard test examples to probe foundation models. Multiple models are evaluated against DiffusionExplorer from CNN models, ViT models, to VLM models and showed great challenges. 1. The idea of leveraging diffusion models to generate hard examples is very interesting and the authors also illustrated that some generated images are very confusing for foundation models even thought they are quite easy for human beings. 2. The authors conducted extensive experiments and analysis that showed the examples generated by DiffusionExplorer are indeed quite challenge for a wide spectrum of models. 3. Write is good and easy to follow. 1. The images generated by DiffusionExplorer seems quite uncommon and sometime does not make sense in real life. Even though they are quite challenging for foundation models, I am not sure whether these example are a good representation of the performance in real scenarios. 2. The proposed approach seems heavily depends on the performance of Stable Diffusion. As the performance of the diffusion models getting better and better, the images generated by DiffusionExplorer might be less and less challenging. 1. How valuable are these hard examples given that they are unlike to appear in real applications?