Official Review · ICLR 2024

Transformer can be viewed as an attention-based graph neural networks over fully connected graph and thus suffer from over-smoothing problem, that the representations induced by deep layers converge to distinguishable values and further results in performance degradation. Inspired by graph representation learning, the paper introduces high-order neighbor information aggregation in Transformer’s attention layer to relieve over-smoothing. Experimental study demonstrated that the proposed method could improve the performance of Transformer over a variety tasks. The paper is well-written and easy to follow. 1.Transformer can be viewed a special case of attention based GNNs. Therefore, introducing graph-filter to enhance Transformer’s attention is promising and effective. 2.Experiments over variant tasks (CV, NLP and Graph, etc.) demonstrated the proposed method can significantly improve the performance of Transformers. 1.The bottleneck of standard Transformer in practice is its high time/space complexity O(N^2), the proposed method introduce high-order neighbor information aggregation introduces more computation/memory overheads, even with approximate computation. 1.How to get the pre-trained large language models integrated with GFSA? Training from scratch or initializing with pre-trained language models and fine-tuning with added GFSA? 2.In section 4.6, the paper talked about the overhead of training time. It is better to add the comparison of inference time. 3.In Eq. (6), do we have any constraints over the coefficients w0, w1, wk? For example w0, w1, wk > 0? In standard Transformer, there is a residual connection in self-attention layer. What is the difference or connection between w0 and this residual connection? Are these learnable coefficients added to all layers? If yes, are they shared across different layers? Similar questions for multi-head attention, are they shared across different heads? Missed references: 1) Graph Attention Networks, ICLR, 2017 2) Diffusion Improves Graph Learning, NIPS, 2019 3) Multi-hop Attention Graph Neural Networks, IJCAI 2021