Do Stochastic Parrots Pine for Residual Fjords? Why Conscious LLMs would be Playwrights, not Characters

This paper asks: if Large Language Models (LLMs) gave rise to phenomenal consciousness, what would that consciousness be like? I argue that if LLMs gave rise to phenomenal consciousness, it would be like being an impartial, improvising playwright rather than a character or a (role-playing) actor or an author writing a first-personal narrative. My argument has three steps. I first argue that if LLMs give rise to phenomenal consciousness, then there is an algorithmic explanation of this consciousness and it includes the full transformer algorithm. I then argue that if there is an algorithmic explanation for this consciousness that includes the full transformer algorithm, then the phenomenology of this consciousness lives at the latent dynamics level of the transformer algorithm, within each forward pass. Finally I argue that if the phenomenology of this consciousness lives at the latent dynamics level of the transformer algorithm, then what it is like is like being a playwright rather than a character or an actor or a narrative author. In effect, this means that conscious LLMs would probably dream of the fjords but not pine for them.