Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing

[2509.21514] Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Machine Learning arXiv:2509.21514 (cs) [Submitted on 25 Sep 2025 ( v1 ), last revised 15 Aug 2026 (this version, v4)] Title: Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing Authors: Joshua Mitton , Prarthana Bhattacharyya , Ralph Abboud , Simon Woodhead View a PDF of the paper titled Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing, by Joshua Mitton and 3 other authors View PDF HTML (experimental) Abstract: Research on Knowledge Tracing (KT) models traditionally focuses on improving predictive accuracy. However, responsible real-world deployment requires models to know when to defer uncertain predictions to a human teacher. We introduce an intrinsic selective prediction layer for existing KT models using Monte Carlo Dropout (MC-Dropout) to quantify uncertainty. We evaluate this approach across three architectures (DKT, SAKT, and AKT) using the Eedi mathematics dataset. Abstaining on the 20\% most uncertain predictions lifts accuracy by 2.3 to 3.0 percentage points, AUC by 1.9 to 2.4 percentage points and F1 by 1.4 to 4.3 percentage points without any retraining. This abstention strategy is highly targeted: the deferred set exhibits 1.45 to 1.60 times the error rate of the kept set. Furthermore, this targeting holds within every question-difficulty quartile and remains fair across student-ability levels. Importantly, MC-Dropout variance gives roughly five times the AUC lift of a calibrated two-parameter logistic (2PL) Item Response Theory (IRT) baseline as a selective-prediction signal. A variance decomposition of the model's epistemic uncertainty (BALD) reveals that the entire classical psychometric stack, comprising question difficulty, student ability, IRT-style outcome ambiguity, and historical curriculum coverage, explains less than 4\% of the signal under linear modeling and at most 23\% even with a non-linear regressor. This leaves 77\% to 90\% as architecture-specific epistemic content that MC-Dropout surfaces and simpler proxies cannot recover. Selective prediction with model-native epistemic uncertainty is therefore a necessary component of responsible KT deployment, complementary to subgroup-fairness audits and downstream classroom evaluation rather than a substitute for them. Comments: Published as Spotlight paper at IRAISE 2026. Link to published version: this https URL . 10 pages, 7 figures. Joshua Mitton and Prarthana Bhattacharyya contributed equally to this paper Subjects: Machine Learning (cs.LG) ; Computation and Language (cs.CL) Cite as: arXiv:2509.21514 [cs.LG] (or arXiv:2509.21514v4 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2509.21514 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Prarthana Bhattacharyya [ view email ] [v1] Thu, 25 Sep 2025 20:06:02 UTC (799 KB) [v2] Mon, 2 Feb 2026 18:21:09 UTC (552 KB) [v3] Thu, 30 Apr 2026 21:45:14 UTC (883 KB) [v4] Sat, 15 Aug 2026 13:06:27 UTC (883 KB) Full-text links: Access Paper: View a PDF of the paper titled Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing, by Joshua Mitton and 3 other authors View PDF HTML (experimental) TeX Source view license Current browse context: cs.LG < prev | next > new | recent | 2025-09 Change to browse by: cs cs.CL References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) IArxiv recommender toggle IArxiv Recommender ( What is IArxiv? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Inductive Graph-based Knowledge Tracing#464508
• Reasoning Beyond Prediction: From Data-Driven to Causal Software Engineering#319725
• Critique-Guided Distillation for Robust Reasoning via Refinement#1050754
• Hakken: Predicting future discoveries to fill the gaps in today's knowledge#927780
• TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stopping#677580