Official Review · NEURIPS 2023

This paper re-interpret the self-attention mechanism in transformer architectures as a non-parametric kernel density estimator (KDE). Leveraging existing robustness results on classical KDE, the authors develop new variants of transformers; empirical results on a variety of datasets demonstrate how these variants are resistant to adversarial attacks and data contamination. This paper provides a simple yet powerful re-interpretation of the self-attention mechanism through the lens of KDE. This re-interpretation allows to leverage results from robust KDE to design better transformer architectures. Kernel methods are notably amenable to theoretical analysis; therefore, links between transformers and kernels as the one proposed in this paper can help sheding lights on justify the empirical successess of transformer architectures. The paper is in general well-written and its structure is easy to follow. Links between Transformers and kernel methods have been proposed in prior literature, see for example [1]. Although an extensive literature review is proposed by the authors, some important references such as [1] are omitted. **References** [1] Tsai, Yao-Hung Hubert, et al. "Transformer dissection: a unified understanding of transformer's attention via the lens of kernel." arXiv preprint arXiv:1908.11775 (2019). In my view, the paper makes a strong contribution. Could the authors please address my questions below. - In line 93 when the kernel $k_{\sigma}$ is introduced, it should be specified what set does this kernel act on and the fact that the considered class of kernels is translation invariant $\tilde k_\sigma(x,y) = k_\sigma(x-y)$. For example, equation (2) is ambiguous because the kernels used in the joint density and marginal density estimations are different (indexed on different spaces). - In line 95-96, it would be good to add an expression to define the mentioned isotropic Gaussian kernel. - In lines 186-187 it is mentioned that the contruction of attention weight factors necessitate iterative algorithms to calculate the set of weights when computing self-attention at each layer; however, if (sometimes) the weight estimation boils down to a finite dimensional quadratic optimisation problem with linear constraints such as equation (7), can't the weights be determined by simply solving a linear system of equations? - Kernel regression often involves regularisation; this setup is not considered in the paper. Can the authors comment on the potential role of regularisation?