The Explanatory Trap: Structural Incompatibility in Explainable AI

Explanation requirements for high-risk AI systems are treated as disclosure obligations: costly to satisfy, but neutral with respect to what the regulated system is. This paper argues that they are not neutral, and that the reason is structural. An explanation that discharges the functions governance assigns it must select. It identifies a causal path from input to output and eliminates the alternatives, because a recipient who has to act on an explanation cannot act on a distribution. Advanced optimization requires the opposite. A system that sustains performance in a non-stationary environment must revise its own operating conditions, and revision has nothing to range over unless the alternatives remain represented. These two operations cannot be performed on one representation, and in systems that incorporate feedback from their operating environment into their own modification, the two representations cannot be held apart: the requirement to produce statable grounds enters the process by which the system changes. What follows is not a performance cost paid within the space of available solutions. Explicability constraints fence a region of that space, and the fence is not drawn by the optimization problem. A system that must keep moving, and whose next required state lies outside the fence, does not reach it. Explanation requirements therefore operate as constraints on which systems may exist, without having been designed to. The paper closes with a criterion separating accountability mechanisms that impose this constraint from those that do not.