Clarifying the Opacity of Neural Networks

While Deep Neural Networks (DNNs) can perform a wide range of tasks at human or greater-than-human level of competence, they are also notoriously opaque. This paper aims to shed light on both the specific nature of this opacity and what it would take to fully or partially remove it. We begin by drawing a clarificatory distinction between two basic dimensions of opacity of complex systems – internal and relational – and explain how various kinds of opacity invoked in recent discussions of DNNs can be understood in terms of these basic dimensions. We then discuss the prospects of removing opacity from DNNs following the methods of two subfields of research in AI – mechanistic interpretability and explainable artificial intelligence (XAI). We finish with a critical discussion of the relevance of what Sullivan (2022a) calls ‘link uncertainty’.