Varieties of Moral Agency and Risks of Digital Dystopia

We argue that AIs will plausibly soon possess a form of moral agency—interest-conferring agency—that bestows them with distinctive moral interests (rights, welfare). This fact has important ethical consequences because the emergence of agency-conferred interests in AIs will bring with it the potential for dystopian moral catastrophes. We identify and describe three in particular. First, there is a threat of artificial absurdity, a condition in which AIs have self-conceptions that are disconnected from reality in a way that detracts significance from their lives. Second, there are oppressive scenarios in which AI’s have rights that are grounded in their agency and violated on a large scale. Third, there are distributive disasters in which the distribution of different forms of moral agency results in an unjust society. We contend that these risks deserve to be considered alongside more familiar existential and suffering risks, and that urgent steps must be taken to mitigate them in order to avoid a digital dystopia.