Why a Large Language Model cannot explain itself: Explanation, Justification and Artificial Intelligence in Legal and Administrative Decision-making

Judges and administrative decision-makers are expected to explain themselves, giving reasons for their decisions. While the emergence of large language models has prompted renewed interest in the use of AI tools in judicial and administrative settings, existing scholarship identifies the ‘black box’ nature of machine learning tools as impeding transparency and reason-giving. This paper considers the status of the reasoning generated by LLMs in the context of duties of judicial and administrative reason-giving and the problem of explainability. It explains how the move from predictive or discriminative to generative artificial intelligence reshapes the problem of explainability and the limited explanatory value of LLM reasoning. It argues that, notwithstanding the analogous problem of establishing the causal character of human justificatory reasoning, human-generated reasons can be distinguished from machine-generated reasons by their role in a practice of accountability. The fundamental architecture of LLMs precludes their participation in that practice. In consequence, LLMs are incapable of giving reasons in the sense required to support judicial and administrative decision-making.