Official Review · ICLR 2025

This work focuses on the performance of Transformer architecture on the Long Range Arena benchmark (LRA). Earlier works have shown that simple state-space / recurrent architectures can achieve better performance than Transformer architectures on LRA benchmark, while transformers are still the preferred choice in real-world sequential dependency modelling. Although Transformers can achieve on-par performance on LRA using a pre-training phase with a denoising objective, this leads to additional computational cost and risk of representation collapse. This works shows that Transformers can achieve similar performance on LRA benchmark without an additional pretraining stage with the help of better data augmentation strategies and rotary positional embeddings. This work argues that the tasks in LRA have a positional bias and LRA benchmark should be interpreted with caution. - Incorporating data augmentation during transformer training helps avoid the pre-training stage and still achieves comparable results as state-of-the-art models for LRA benchmark. - Ablations on positional embeddings show that transformers improve in performance with better positional embeddings. - This work points out flaws in the popular LRA benchmark and urges to use this benchmark with caution. - LRA has been known to have local positional bias in the literature (see R1, R2). R2 already incorporates some form of positional embeddings in the transformer architecture helps achieve better performance on LRA benchmark. - Data augmentation would help improve performance of any model on this benchmark - Proposed techniques in this work are marginal since pre-training in R2 already shows the objective which is used in this paper along with positional embeddings. Similarly, data augmentation techniques are well known in the literature to improve model performance. -------- - [R1] ON THE LONG RANGE ABILITIES OF TRANSFORMERS : https://arxiv.org/pdf/2311.16620 - [R2] Never Train from Scratch: FAIR COMPARISON OF LONGSEQUENCE MODELS REQUIRES DATA-DRIVEN PRIORS : https://arxiv.org/pdf/2310.02980 - This work mentions the risk of representation collapse in the two stage training, do you have any concrete evidence that such a collapse happens in your experiments? - Do you know why gMLP does not improve on text-retrieval task with your setup in Sec. 4.2? - Does the same conclusion hold true for other long range dependency benchmarks such as SCROLLS? - Can you provide an estimate of resources required for training on Pathfinder vs Path-X tasks? - Can you compare the performance of MEGA, S4, S5 when these models are trained with data augmentation strategies mentioned in this work? - In Figure 2, why is only gMLP trained with the proposed data augmentation strategies? Why weren't these techniques applied on S4/S5?