Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

[2604.14892] Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning? Skip to main content Search Submit Donate Log in Computer Science > Machine Learning arXiv:2604.14892 (cs) [Submitted on 16 Apr 2026 ( v1 ), last revised 27 Aug 2026 (this version, v4)] Title: Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning? Authors: Amy Rouillard , Sitwala Mundia , Linda Camara , Ziyaad Dangor , Michael Cameron Gramanie , Ismail Kalla , Shabir A. Madhi , Kajal Morar , Marlvin T. Ncube , Haroon Saloojee , Bruce A. Bassett View PDF HTML (experimental) Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators. Here, we evaluate an LLM Jury, composed of three frontier AI models, for scoring 3334 diagnoses on 300 real-world low- and middle-income country (LMIC) hospital cases. Both LLM- and clinician-generated diagnoses are scored against expert panel diagnoses across four dimensions: diagnosis, differential diagnosis, clinical reasoning, and negative treatment risk. The LLM Jury scores are compared with expert and independent re-scoring panel scores to assess error metrics, inter-rater agreement, severe-risk errors, and the effect of post hoc calibration using isotonic regression. In our data, we find that: (i) the uncalibrated LLM Jury scores preserve ordinal agreement with the expert clinician panel scores, but are systematically lower; (ii) the probability of severe-risk errors is lower for the LLM Jury than the human expert re-score panels; (iii) the LLM Jury combined with LLM diagnoses can be used to identify diagnoses at high risk of error, enabling targeted expert review and improved panel efficiency; (iv) the calibrated LLM Jury scores and rankings of diagnosing agents show excellent agreement with those of the primary expert panels; (v) LLM Jury models show no self-preference bias, they did not score diagnoses generated by their own underlying model or models from the same vendor more (or less) favourably than those generated by other models. Together, these results provide evidence that a calibrated LLM Jury is a trustworthy and reliable proxy for expert clinician evaluation in medical AI benchmarking. Confirming these findings in other clinical settings is an important direction for future work. Comments: Extended version of a paper that appeared in Artificial Intelligence in Medicine 2026 Subjects: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI) Cite as: arXiv:2604.14892 [cs.LG] (or arXiv:2604.14892v4 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2604.14892 Focus to learn more arXiv-issued DOI via DataCite Related DOI : https://doi.org/10.1007/978-3-032-30710-1_44 Focus to learn more DOI(s) linking to related resources Submission history From: Amy Rouillard Dr [ view email ] [v1] Thu, 16 Apr 2026 11:32:09 UTC (413 KB) [v2] Fri, 17 Apr 2026 09:12:13 UTC (413 KB) [v3] Fri, 12 Jun 2026 09:57:39 UTC (399 KB) [v4] Thu, 27 Aug 2026 09:09:40 UTC (666 KB) Full-text links: Access Paper: View PDF HTML (experimental) TeX Source view license Current browse context: cs.LG < prev | next > new | recent | 2026-04 Change to browse by: cs cs.AI References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) IArxiv recommender toggle IArxiv Recommender ( What is IArxiv? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?#674373
• Toward Better Assessment of LLMs' Performance in Clinical Error Detection#625214
• Toward Better Assessment of LLMs' Performance in Clinical Error Detection#622941
• Toward Better Assessment of LLMs' Performance in Clinical Error Detection#459665
• Artificial Intelligence and Multidisciplinary Oncology Decision-Making: A Systematic Review of LLM Concordance with MDT Recommendations and Implications for Clinical Reasoning Education.#371372