Explainable AI, Idealized Models and Understanding

[Accepted to the 2026 Philosophy of Science Association meeting, presumptively to appear in the conference proceedings edition of Philosophy of Science] Deep-learning-based AI systems are notoriously opaque. Our tools for coping with this opacity include “explainable AI” (XAI) methods such as LIME, which aims to explain a given AI output by providing a linear model intended to locally approximate the underlying model's true decision boundary. Yet it remains controversial whether LIME models and other XAI methods can provide users with genuine understanding. One recent suggestion—defended in various forms by Fleisher (2022) and Knoks & Raleigh (2026)—is that LIME explanations function much like idealized models in science, which can successfully confer understanding in spite of their limited accuracy and lack of realistic causal detail. I argue that the analogy is a poor one. XAI explanations and idealized models are dissimilar in ways that matter. First, idealized scientific models are useful only when we have enough ancillary knowledge to tell which features of their predictions to take seriously and which to ignore; we’re rarely in this position with respect to complex AI systems. Second, we often need a “translation key” to go from the outputs of an idealized model—which might be false or nonsensical if taken at face value—to correct predictions about the target phenomenon. This also requires knowledge, experience and background theory which we typically lack in XAI settings. Third, it’s often thought that idealized models generate understanding by providing “how-possibly explanations”, which show how an event might have occurred. But how-possibly explanations are rarely useful in XAI contexts. Knoks and Raleigh attempt to shore up XAI's epistemic credentials by suggesting that it's primarily experts rather than laypeople who are in a position to gain understanding from XAI techniques. I argue that this strategy too is overly optimistic. I conclude by considering the epistemic prospects for the growing number of domain-specific variants of LIME.