Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization

We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer. Our theoretical analysis establishes that sta