Official Review · NEURIPS 2024

This paper introduces DIFFUSIONHOI, a new HOI detector leveraging text-to-image diffusion models. Unlike previous one-stage or two-stage models, diffusion models excel at discerning mid/low-level visual concepts as generative models and possess strong compositionality to handle novel concepts expressed in text inputs. To steer the focus of diffusion models from instance generation to the relationships between humans and objects, this paper exploits textual inversion and devises a human-object relation inversion strategy grounded in the disentanglement of HOI. Furthermore, to transfer extensive knowledge from large-scale diffusion models to assist in recognizing interactions, the paper leverages both text-prompted image generation and conditioned feature extraction capabilities of diffusion models. By embracing text-to-image diffusion models and facilitating relation-driven image generation and prompting, this method demonstrates superior performance. This paper is well-written and easy to follow. It proposes a new diffusion-based solution, DiffusionHOI, for the human-object interaction (HOI) task. Unlike traditional one-stage and two-stage methods, DiffusionHOI benefits from controllable image generation and HOI knowledge transfer from diffusion models. Without making design changes to the HOI decoder, this method achieves superior performance compared to previous state-of-the-art models by deriving HOI-relevant features to assist in HOI detection. 1. I am curious about the number of parameters in the VQGAN used as the backbone of DiffusionHOI. According to Table 1, the authors use ViT-L as their backbone, which appears to be significantly larger compared to previous works, especially those using the Res50 network. Could the authors provide more results using a smaller backbone, such as ViT-B, to offer a clearer and fairer comparison between DiffusionHOI and previous state-of-the-art models? 2. Another concern is that the improvement primarily stems from the additional knowledge provided by the powerful diffusion model. Could this be considered unfair when comparing it to previous works that do not have this significant benefit? In this case, how does this approach differ from simply adding more data for training? As I am familiar with HOI but not as familiar with diffusion models, could the authors provide more details about the data scale and parameter scale of the diffusion model used in this paper? Additionally, I look forward to feedback from other reviewers on this matter. Please refer to weakness.