How Vision Transformers Represent Position: A Mechanistic Comparison of Absolute and Rotary Positional Encodings

Positional embeddings encode the spatial position of each image patch in a Vision Transformer (ViT), allowing the model to distinguish different spatial arrangements of visual content. However, we know little about how the choice of embedding shapes a vision transformer's internal representation. In