The competence problem of AI alignment

This paper identifies a class of alignment problem that does not reduce to specification gaming, Goodhart’s Law, or construct validity failure. The rules an AI system is asked to follow are often settlement proxies. These are operational forms of political and moral questions whose answers a community has had to settle. Unlike measurement proxies, which approximate empirical targets, settlement proxies do not aim at some further thing they could be brought into closer contact with. Instead, they are the target in operational form. Settlement proxies are vulnerable to a form of legitimate challenge that more capable reasoning intensifies rather than resolves. The paper calls this the Competence Problem and identifies three structural features of settlements that generate it: 1) the compression of competing values into a single operational rule, 2) the presence of more than one defensible equilibrium among the rule’s possible forms, and 3) a constraint on the scope of information the rule engages. It then traces three forms that competent challenge takes in practice: 1) exceptions, 2) drift, and 3) challenge to the settlement itself. It argues that the appropriate response, even where the challenge is correct on the merits of the case, is typically not to grant it, because settlement authority depends on ratification through practice rather than case-level accuracy. The argument explains why behavioral training and corrigibility-based approaches to AI alignment leave the underlying capacity intact, and why the gap between case-level competence and ecosystem-level access is widest precisely where AI systems are now being deployed.