A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability

A wide range of methods have been proposed for interpreting language models, delivering important insights into their inner workings. However, different methods and their resulting insights stand in relative isolation: what could the underlying structure of language models be, such that they give ri