Measuring language model welfare based on verbal report: An analogical abductive approach

If some language models become welfare subjects, how could we find out what welfare states they are in? We develop an analogical-abductive approach for measuring language model welfare. This approach adapts paradigms used to measure human or non-human animal welfare, for instance verbal reports or non-verbal choice behavior (analogy). Then, one systematically searches for clusters of such indicators in language models. This search for clusters contributes to the cross-validation of welfare measures and motivates explanations in terms of a welfare state that underlies multiple measures (abduction). Additionally, we argue that measures of welfare based on verbal report may already be applicable to current language models because they have substantial introspective abilities, semantic competence, and requisite inclinations for accurate reporting. We further discuss a detailed empirical case study that exemplifies the feasibility and fruitfulness of the analogical-abductive approach and reply to the objection that language model behavior is always better explained in terms of statistical pattern completion or memorization, rather than welfare. While language model welfare measurement undoubtedly faces some remaining theoretical as well as many detailed practical challenges, we conclude that there is a strong case that the analogical abductive approach offers a viable path forward.