LLMs Can Never Be Ideally Rational

AIs are increasingly agential. AIs no longer merely generate text; they also perform actions. Now that AIs choose between different actions, we can investigate whether their preferences are rationally coherent, in the sense of being transitive and complete. This paper poses a challenge for the coherence of AI preferences. Today’s AIs are built on top of large language models. At some level of abstraction, AIs choose actions by sampling from a probability distribution. I’ll argue that this basic architectural feature creates a problem for preference coherence. When an AI is asked to choose between options, it will select options based on its token probabilities. But the underlying nature of the token probabilities can create cycles, where it selects A over B, selects B over C, and selects C over A. Similar problems can also lead to probabilistically incoherent AI forecasting.