LLM measurements that agree with human coders overall can still distort specific effects

Large language models (LLMs) are increasingly adopted in the social sciences as surrogates for human coders. Standard practice validates them by their correspondence with human measurements of the same stimuli using statistics like correlation. However, overall correspondence can conceal biases that