Use of Large Language Models for Rapid Quantitative Feedback in Case-Based Learning: A Pilot Study.

Use of Large Language Models for Rapid Quantitative Feedback in Case-Based Learning: A Pilot Study - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Med Sci Educ . 2025 Feb 28;35(3):1169–1171. doi: 10.1007/s40670-025-02343-6 Use of Large Language Models for Rapid Quantitative Feedback in Case-Based Learning: A Pilot Study Carolyn Qian 1 , Christina Gao 2 , Sang-O Park 1 , Haelynn Gim 1 , Kelly Hou 2 , Benjamin Cook 2 , Jasmin Le 2 , Brandon Stretton 2 , John Maddison 3 , Liam McCoy 4 , Rudy Goh 2 , Matthew Arnold 2 , Haatem Reda 1 , Tamara Kaplan 1 , Galina Gheihman 1 , Stephen Bacchi 1, 2, 3, 5, 6, ✉ Author information Article notes Copyright and License information 1 Harvard Medical School, Harvard University, Boston, MA 02138 USA 2 Adelaide Medical School, The University of Adelaide, Adelaide, SA 5005 Australia 3 Neurology Department, Lyell McEwin Hospital, Haydown Road, Elizabeth Vale, SA 5112 Australia 4 Massachusetts Institute of Technology, Boston, MA 02139 USA 5 Massachusetts General Hospital, Boston, MA 02138 USA 6 Flinders University, Bedford Park, SA 5042 Australia ✉ Corresponding author. Accepted 2025 Feb 17; Collection date 2025 Jun. © The Author(s) under exclusive licence to International Association of Medical Science Educators 2025. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. PMC Copyright notice PMCID: PMC12228618  PMID: 40625928 Abstract Large language models (LLMs) may be able to deliver interactive case-based content and score student interactions with such cases. In this study, GPT-4o demonstrated a high correlation with expert scorers in the evaluation of medical students’ interactions with cases. A difference between LLM scores and expert scorers was corrected through calibration. Supplementary Information The online version contains supplementary material available at 10.1007/s40670-025-02343-6. Keywords: Artificial intelligence, Machine learning, Natural language processing, Medical education, Clinical reasoning Large language models (LLMs) have many possible touted benefits for medical education, including enhancing clinical reasoning practice through case-based learning, as defined previously [ 1 ]. Scoring written content is another possible use of the technology. The development of such approaches requires robust data to support their use, prior to any widespread implementation. This pilot study aimed to examine the delivery and scoring of case-based learning by an LLM, as compared to a panel of faculty educators. The LLM used in this study was GPT-4o from OpenAI [ 2 ]. A custom online platform was developed to enable question-and-answer exchanges with cases, including the collection of history, examination, and investigation results (see Fig. 1 ). Five cases with neurological presenting concerns were delivered through this platform (meningitis, radial nerve palsy following intoxication, giant cell arteritis, Meniere’s disease, and myasthenia gravis). Four student investigators evaluated these cases through the online platform, using free-text questions to gather information with the aim of arriving at a diagnosis and plan. Fig. 1. Open in a new tab Online interface through which interactive cases were delivered Scoring criteria for each of the cases were determined by consensus by four independent expert scorers (three neurology attendings and one medical education specialist). The maximum score in each case ranged from five to eight key points (see Supplementary Information 1 ). Student interactions with the LLM-based cases were recorded through the online platform and then independently scored by each expert. The LLM was then provided with the same scoring criteria and asked to ascertain how many of the key points each student elicited in the individual interactions. The score on each case was reflected as a percentage of the total of the key points that were elicited. The scores of the four experts were compared to that of the LLM using several approaches. Intraclass correlation (ICC) and pairwise correlation were employed to evaluate for case-based agreement. Repeated-measures ANOVA and post hoc comparisons were used to evaluate for differences in scores, and Bland–Altman statistics were used to characterize these differences. In the event of an offset between LLM and expert scorers, a repeat analysis applying calibration was planned to optimize fit. The mean score for the experts was 70.3% (standard deviation 21.2%, median 75%, interquartile range 57.1 to 85.7%, minimum 25%, and maximum 100%). The mean score for the LLM was 82.2% (standard deviation 16.1%, median 82.8%, interquartile range 69.2 to 100%, minimum 69.2%, and maximum 100%). The ANOVA and post hoc comparisons demonstrated a significant difference between LLM scores and expert scores (P < 0.01). Bland–Altman analysis demonstrated that the LLM scores were approximately 12 percentage points higher than the experts on average. The LLM showed high correlation with the expert reviewers (Pearson’s R was 0.89, 0.90, 0.85, and 0.67) and high ICC (single random raters 0.78 and singled fixed raters 0.82). A 10 percentage point calibration was applied to the LLM scores. Following this correction, there were no statistically significant differences between the LLM and expert scores on ANOVA and post hoc comparisons. As expected, the Bland–Altman analysis demonstrated significantly less difference (see Supplementary Information 2 ). In this pilot study, LLM-supported case-based learning was delivered through an online portal and scoring of these interactions by an LLM was well correlated with scoring by a panel of experts. There was a significant difference between the LLM and expert scores, by approximately 12 percentage points. Following calibration, this difference resolved. These findings support the feasibility of delivering LLM-based cases with real-time LLM-based scoring for students. This approach may have significant implications for the delivery of case-based learning content with real-time feedback in medical curricula. Interactive sessions to develop clinical reasoning skills have traditionally relied on patient actors; actor availability and costs may limit access. Repeat practice for students is seldom possible. Scoring of written content is also a resource-intensive process. Our pilot offers a supplement to these traditional approaches. Students can interact with the case on their own schedule and as many times as desired, promoting mastery-based learning [ 3 ]. Faculty can focus on developing cases and scoring systems, reaching a greater number of learners and freeing time to focus on individual struggling learners. This pilot study is limited. Repeat evaluation in larger cohorts would be valuable. Further research is required to understand the impact of interacting with the LLM and receiving LLM-based feedback on student learning outcomes. LLM scoring of LLM-based case interactions correlates well with expert scorers. Prompt modification may ameliorate the difference in scores, although using a calibration method was successful in this study. Developing and improving LLM-based scoring systems in partnership with faculty evaluators may help realize the potential for LLM scoring while mitigating its limitations. Supplementary Information Below is the link to the electronic supplementary material. ESM 1 (103.6KB, docx) (DOCX 104 KB) Funding Stephen Bacchi is supported by a Fulbright Scholarship. Declarations Conflict of Interest The authors declare no competing interests. Footnotes Carolyn Qian and Christina Gao are co-first authors and contributed equally. Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References 1. Stretton B, Kovoor J, Arnold M, Bacchi S. ChatGPT-based learning: generative artificial intelligence in medical education. Med Sci Educ. 2024;34:215–7. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. OpenAI. GPT4o. 2024. 3. Mikhaeil-Demo Y, Barsuk JH, Culler GW, Bega D, Salzman DH, Cohen ER, Templer JW, Gerard EE. Use of a simulation-based mastery learning curriculum for neurology residents to improve the identification and management of status epilepticus. Epilepsy Behav. 2020;111:107247. [ DOI ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials ESM 1 (103.6KB, docx) (DOCX 104 KB) Articles from Medical Science Educator are provided here courtesy of Springer ACTIONS View on publisher site PDF (484.8 KB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• A survey of large language model use in a hospital, research, and teaching campus#19953
• Working with large language models : : impacts on performance, efficiency and perceived responsibility#1106612
• On Epistemic Diversity in Large Language Models#971500
• Large Language Models: A Structured Taxonomy and Review of Challenges, Limitations, Solutions, and Future Directions#285464
• Large Language Models Are Reasoning Teachers#16659