The Relations Between Pedagogical and Scientific Explanations of Algorithms: Case Studies from the French Administration

The opacity of some recent Machine Learning (ML) techniques have raised fundamental questions on their explainability, and created a whole domain dedicated to Explainable Artificial Intelligence (XAI). However, most of the literature has been dedicated to explainability as a scientific problem dealt with typical methods of computer science, from statistics to UX. In this paper, we focus on explainability as a pedagogical problem emerging from the interaction between lay users and complex technological systems. We defend an empirical methodology based on field work, which should go beyond the in-vitro analysis of UX to examine in-vivo problems emerging in the field. Our methodology is also comparative, as it chooses to steer away from the almost exclusive focus on ML to compare its challenges with those faced by more vintage algorithms. Finally, it is also philosophical, as we defend the relevance of the philosophical literature to define the epistemic desiderata of a good explanation. This study was conducted in collaboration with Etalab, a Task Force of the French Prime Minister in charge of Open Data & Open Government Policies, dealing in particular with the enforcement of the right to an explanation. In order to illustrate and refine our methodology before going up to scale, we conduct a preliminary work of case studies on the main different types of algorithms used by the French administration: computation, matching algorithms and ML. We study the merits and drawbacks of a recent approach to explanation, which we baptize input-output black box reasoning or BBR for short. We begin by presenting a conceptual framework including the distinctions necessary to a study of pedagogical explainability. We proceed to algorithmic case studies, and draw model-specific and model-agnostic lessons and conjectures.