Official Review · ICLR 2025

This paper introduces a novel attention mechanism by which transformers are encouraged to explicitly represent information about relations between elements of a sequence or set of tokens. This relational information is encoded separately from sensory information, or information about the specific entities that enter into a relation. This new dual attention transformer architecture is evaluated on a diverse series of modalities and tasks, demonstrating enhanced performance in language modeling, a simple visual classification task, and a toy relational reasoning dataset. The proposed mechanism is novel, yet is a natural extension of the attention mechanisms within a standard transformer. The idea of explicitly representing relations between entities is an important one that merits further investigation. Empirically, the DAT appears to be more data efficient than the standard transformer, which is especially important in the case of language modeling. While the work is interesting and the idea of building in inductive biases toward representing relational information is potentially useful, there are several framing and experimental issues that should be addressed. First, the claim that standard attention mechanisms only represent sensory information is empirically false. The authors themselves cite several works describing how attention in language models often captures syntactic information, which is inherently relational. Furthermore, recent work has explicitly found that finetuned ViTs represent sensory information in early layers, but represent abstract relational information in their later layers [1]. Finally, “register tokens” – tokens that encode little information about their local image patch, but instead represent global information about an image – have been discovered in large pretrained ViTs [2]. Their very existence challenges the notion that attention mechanisms are only transmitting “sensory information". At very least, the sensory information is not necessarily tied to the pixels and token embeddings that the activation vector appears to correspond to. The proposed method has at least two separate important components: the representation of relational information, and the tying of key and query matrices. It appears very important to test a variant of the standard transformer subject to tied key and query matrices. For example, Figure 8 shows that removing this symmetry condition deteriorates the benefits of relational attention quite a bit, especially in low-data regimes. This should also be done for the experiments presented in section 4.3. Similarly, when using position-relative symbol assignment, a control that modifies the standard transformer with relative positional embeddings should also be included. This is especially important because prior work has demonstrated that these positional embeddings can aid transformers on synthetic generalization tasks [3]. In section 4.4, the authors suggest that relational attention might