Official Review · NEURIPS 2023

_Background_: RASP is a tensor processing language which provides a language to hand-write transformers. Tracr presents a framework to compile RASP programs to a transformer architecture automatically. The authors of this project are interested in distilling a transformer architecture to an equivalent program in RASP. This is challenging because RASP is not a sound language: the behavior of many trained transformers is equivalent to that of a single RASP program. The authors go around this problem by constraining the transformer architecture in strategic places so that there is a deterministic mapping from a transformer to an equivalent program in RASP. The authors present a model distillation procedure to construct such reduced transformers (called Transformer Programs) automatically. The authors test the Transformer Programs on three tasks: 1) in-context learning, 2) recovering handwritten RASP programs, and 3) simple real-world NLP tasks. They observe mixed results: transformer programs can fully solve (1), they cannot generalize to larger sequences for (2), and they are better than non-sequential baselines on (3) but worse than generic transformers. The authors also show the utility and interpretability of the distilled programs by analyzing the programs using off-the-shelf debuggers. Originality * RASP programs are inherently unsound: one RASP program is equivalent to many trained transformers. Solving this problem by constraining the transformer to elicit a bijective mapping is a simple yet elegant idea. Significance * This paper shows a simple method to distill transformers into interpretable programs. This work opens the door for work in program synthesis, static analysis, and model checking to be used for sequential processing tasks. * Many downstream tasks rely on transformers to processing temporal sequences of data into labels. While the transformer architecture presented is constrained, an interpretable transformer can be very useful for tasks where high degree of interpretability and accuracy are required. Clarity and Quality * The framework is well motivated, and the writing is clear and precise. __Weaknesses__ I have three main concerns (paraphrased from the questions section below): * I’m concerned that the constraints mentioned here violate properties[1] that the mechanistic interpretability community have identified as being intrinsic to the utility of transformers. * Poor robustness: Model distillation has been observed to produce models that are more robust to outliers while being slightly less accurate[2]. However, in experiments it seems that the distilled models are neither robust to outliers nor as accurate as baselines, which makes it harder to use this as a drop-in replacement. * Lack of Reusability: An interesting property of the RASP programs (and RASP distilled transformers) is that many tasks can be efficiently solved by precomputing useful primitives. The algorithm presented here does not seem to model thi