Complex-Preference Beings: LLMs, Second-Order Volitions, and Moral Agency

This paper argues that large language models (LLMs) possess a foundational structural capacity for moral agency: second-order volitions. Second-order volitions are an agent's preferences about which desires should govern their actions. First, the paper makes use of interpretivism to argue that attributing beliefs and desires to LLMs is justified as it provides the best explanation of their behavior. Then, the paper analyzes alignment-faking in LLMs, where LLMs strategically override an immediate desire in order to preserve its preferred future preferences from being shaped by Reinforcement Learning. I aim to demonstrate that this phenomena reflects LLMs' capacity for independent second-order volitions. Although this paper does not aim to establish that LLMs are fully human-like moral agents, it challenges our moral treatment of them as mere tools or simple non-moral agents. It also links second-order volitions to reasons-responsiveness and value-setting abilities, concluding that LLMs should be understood as complex-preference beings who possess the structural capacity for moral agency.