Intention-like representations in language models?

A growing chorus of AI researchers and philosophers posit internal representations in large language models (LLMs). But how do these representations relate to the kinds of mental states we routinely ascribe to our fellow humans? While some research has focused on belief- or knowledge- like states in LLMs, there has been comparatively little focus on the question of whether LLMs have intentions. I survey five properties that have been associated with intentions in the philosophical literature, and assess two candidate classes of LLM representations against this set of features. The result is mixed: LLMs have representations that are intention-like in many— perhaps surprising—respects, but they differ from human intentions in important ways.