When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk

[2608.23780] When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk Skip to main content Search Submit Donate Log in Computer Science > Computation and Language arXiv:2608.23780 (cs) [Submitted on 24 Aug 2026 ( v1 ), last revised 26 Aug 2026 (this version, v2)] Title: When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk Authors: Liliana Santos-Deonizio , James Malamut , Ramón Antonio Martínez , Dorottya Demszky View PDF HTML (experimental) Abstract: LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences. Comments: under review Subjects: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) Cite as: arXiv:2608.23780 [cs.CL] (or arXiv:2608.23780v2 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2608.23780 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Dorottya Demszky [ view email ] [v1] Mon, 24 Aug 2026 19:25:12 UTC (8,167 KB) [v2] Wed, 26 Aug 2026 18:36:32 UTC (8,167 KB) Full-text links: Access Paper: View PDF HTML (experimental) TeX Source view license Additional Features Audio Summary Current browse context: cs.CL < prev | next > new | recent | 2026-08 Change to browse by: cs cs.AI cs.HC References & Citations SciX Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from (opens in new tab) (opens in new tab) (opens in new tab)

Related documents

• Data Analysis in Educational Communication: How Audiovisual Stimuli Influence Retention and Comprehension in University Students#287648
• Color Me Confounded: A Critical Analysis of Media Comparisons on ChatGPT in Education#331618
• Adopt $\neq$ Adapt: Longitudinal Analyses of LLM Conversations in the Wild#800043
• Student Talk Matters: Student Voices and Perspectives on Classroom Talk in English and French Classes in England, France and Norway#268797
• A Video Chat Intervention for Enhancing Self-efficacy for Lifestyle Physical Activity#826810