LLMs don't know anything: reply to Yildirim and Paul

In their recent Opinion in TiCS, Yildirim and Paul propose that large language models (LLMs) have ‘instrumental knowledge’ and possibly the kind of ‘worldly’ knowledge that humans do. They suggest that the production of appropriate outputs by LLMs is evidence that LLMs infer ‘task structure’ that may reflect ‘causal abstractions of... entities and processes in the real world.' While we agree that LLMs are impressive and potentially interesting for cognitive science, we resist this project on two grounds. First, it casts LLMs as agents rather than as models. Second, it suggests that causal understanding could be acquired from the capacity for mere prediction.