Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning

The success of transformers is attributed to several factors, including their capacity to express formal languages over variable-length inputs, and ability of the attention mechanism to focus on relevant input features. However, there is limited understanding of the training dynamics of shallow tran