In the loop, out of sync: Moral judgment in human-AI control architectures

The ubiquitous integration of AI-powered systems in morally consequential decision-making procedures raises a thorny question: when such systems generate harm, who should be held responsible? A prominent regulatory response proposes that suitably designed control architectures can ensure that blame is appropriately allocated. To live up to this promise, the proposed architectures must be both normatively adequate and regarded as such, since otherwise they are at best practically useless, and at worst useless because morally problematic. We examine two of the most widely discussed proposals. Experiment 1 (n=260) investigates whether laypeople attribute responsibility differently across architectures that place human agents in-, on-, and out-of-the-loop. Experiment 2 (n=520) extends this inquiry by differentiating between control structures that either violate or satisfy Santoni de Sio’s and van den Hoven’s (2018) ‘track and trace’ requirements. Our results reveal that neither proposal fully succeeds in directing laypeople’s responsibility judgements along the pathways they prescribe. Implications are discussed.