Official Review · ICLR 2024

The paper proposed a method for fine-tuning linearized transformers, called Tangent Attention Fine-Tuning (TAFT). The paper claims that TAFT can achieve comparable performance to non-linear fine-tuning on various downstream tasks while enjoying the benefits of linearity such as compositionality, parallel training, machine unlearning, and differential privacy. The paper also introduces Tangent Transformers, which are linearized versions of pre-trained transformer models, and shows how to compute their Jacobian-Vector products efficiently in a single forward pass. The paper demonstrates the advantages of TAFT and Tangent Transformers in several experiments using vision transformer models. To summarize, the paper's main contributions are: - A novel method for fine-tuning linearized transformers that is computationally efficient and competitive with non-linear fine-tuning. - A theoretical analysis of the benefits of linearity for model composition, parallel training, machine unlearning, and differential privacy. - An empirical evaluation of TAFT and Tangent Transformers on various downstream visual classification tasks using vision transformer models. The paper proposes a novel method for fine-tuning linearized transformers, called Tangent Attention Fine-Tuning (TAFT), which is computationally efficient and competitive with non-linear fine-tuning. The paper also introduces Tangent Transformers, which are linearized versions of pre-trained transformer models, and shows how to compute their Jacobian-Vector products efficiently in a single forward pass. The paper demonstrates the advantages of TAFT and Tangent Transformers on several experiments using vision transformer models. In terms of originality, the paper introduces a new method for fine-tuning linearized transformers that is computationally efficient and competitive with non-linear fine-tuning. The paper also introduces Tangent Transformers, which are linearized versions of pre-trained transformer models, and shows how to compute their Jacobian-Vector products efficiently in a single forward pass. These contributions are novel and have not been explored before. In terms of quality, the paper provides a theoretical analysis of the benefits of linearity for model composition, parallel training, machine unlearning, and differential privacy. The paper also provides an empirical evaluation of TAFT and Tangent Transformers on various downstream visual classification tasks using vision transformer models. The experiments are well-designed and the results are presented clearly. In terms of clarity, the paper is well-written and easy to follow. The authors provide clear explanations of their methods and results. The paper also includes visualizations that help to illustrate the concepts presented. In terms of significance, the paper’s contributions have important implications for the field of machine learning. The proposed method for fine-tuning linearized transformers is computationally