Benchmarking Intelligence Better

Benchmarks record what a system does, but performance underdetermines method. Recent defences of LLM cognition revive a broadly Turing-style confidence that enough impressive behaviour settles questions of intelligence. I argue that this still isn't so. Comparative cognition has to work hard to the gap between performance and capacity. But because LLMs are very different from humans and other animals, closing that gap is complicated. My suggestion is that we take advantange of the fact that we can "look inside" LLMs. The marks worth wanting in an intelligent system are properties of inspectable methods rather than of outputs, and mechanistic interpretability now lets us test for them. Working through the indirect object identification circuit in GPT-2, I argue that its method meets a generality and flexibility condition (athought it does something other than what the task's name advertises). To proceed this way is to run what I call a deep benchmark. It blocks the mimic, and it sharpens what a system is intelligent at rather than only whether it is in any way.