Agreement Without Accuracy: Six Systematic Defects an LLM Annotation Study Could Not See in Its Own Reliability Statistics

Researchers who use large language models as annotators are commonly asked to demonstrate reliability, and test–retest agreement under a fixed seed is the cheapest evidence. We report a study in which it was systematically misleading. Annotating 1,902 student UML class specifications under a fixed s