Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability.

Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice PLOS Digit Health . 2026 May 29;5(5):e0001072. doi: 10.1371/journal.pdig.0001072 Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability Prita Abhay Dhaimade 1, * , Robin Henderson 2 Editor: Shrey Lakhotia 3 Author information Article notes Copyright and License information 1 Division of Periodontology, Department of Surgical Sciences, School of Dental Medicine, East Carolina University, Greenville, North Carolina, United States of America 2 Division of Periodontics, College of Dentistry, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, United States of America 3 Henry Ford Health System, UNITED STATES OF AMERICA ✉ * E-mail: dhaimadep25@ecu.edu The authors have declared that no competing interests exist. Roles Prita Abhay Dhaimade : Conceptualization, Data curation, Formal analysis, Methodology, Visualization, Writing – original draft, Writing – review & editing Robin Henderson : Conceptualization, Data curation, Writing – review & editing Shrey Lakhotia : Editor Received 2025 Oct 14; Accepted 2026 May 3; Collection date 2026 May. © 2026 Dhaimade, Henderson This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice PMCID: PMC13220994  PMID: 42213673 Abstract Large language models (LLMs) have demonstrated rapid advancements in natural language understanding and generation, prompting their integration into biomedical research, clinical practice, and professional education. However, systematic evaluation of LLMs in specialty-specific domains such as dentistry and periodontology remains limited, particularly regarding multidimensional performance metrics. This study conducted a comprehensive assessment of commercially available LLMs — GPT-4.0, GPT-5.0, and Claude Sonnet 4.0 — on the American Academy of Periodontology In-Service Examination, focusing on response accuracy, self-assessed confidence calibration, citation validity, and hallucination prevalence. Models were evaluated on the 2024 AAP In-Service Examination (331 questions) using two formats: Full Test (all questions at once) and Individual Question (one at a time). Prompts were standardized; models selected answers, and GPT-5.0 and Claude Sonnet 4.0 also provided confidence ratings and citations. Citation validity was assessed using a human-in-the-loop protocol with expert review. Statistical analyses included chi-square, McNemar’s, and logistic regression to assess accuracy, question fatigue, confidence calibration, and citation reliability. LLMs achieved high overall accuracy (78–87%), with the Individual Question format consistently yielding higher scores than Full Test, though differences were not statistically significant. Accuracy was highest in fact-dense domains (biochemistry, physiology, microbiology) and lowest in integrative domains (diagnosis, therapy). Significant question fatigue was observed in GPT-5.0 Full Test mode (OR = 0.997, p = 0.035) but not in Individual Question mode. Confidence scores predicted accuracy, with the strongest calibration in Individual Question mode. Citation analysis revealed frequent hallucinations, mostly critically erroneous, and citation validity was independent of answer accuracy. LLMs show promise as adjunctive tools for periodontal education, but their outputs, especially for complex reasoning and citations require rigorous human review to ensure accuracy and safety. Author summary Artificial intelligence chatbots are rapidly entering medical education, yet we lack comprehensive understanding of their reliability when students depend on them for learning. We developed a multidimensional evaluation framework to systematically assess AI performance beyond simple accuracy, examining how these systems behave across different medical topics, question types, and presentation formats. Using 331 real dental examination questions, we tested three major AI systems, analyzing not only correctness but also confidence calibration, whether AI confidence levels match actual accuracy and implementing human-in-the-loop verification to check if cited sources actually exist. Our findings highlight critical vulnerabilities in current AI systems. Most alarmingly, these chatbots fabricated nearly half of their citations while maintaining unwavering confidence in both correct and incorrect responses. This combination of overconfidence and misinformation means students cannot distinguish reliable from unreliable AI responses. Additionally, we documented progressive performance decline during sequential questioning, similar to human cognitive fatigue. While we know AI systems generate rather than retrieve information, our research demonstrates the real-world consequences of this limitation. As artificial intelligence integrates into education, healthcare diagnostics, and insurance decisions, these findings underscore the urgent need for better evaluation frameworks and user education about AI limitations. Introduction Artificial intelligence (AI) has rapidly evolved into a transformative technology across scientific and clinical disciplines. Among its most significant developments are large language models (LLMs), which are trained on vast corpora of text to generate coherent, human-like language. Their ability to recognize linguistic patterns and predict plausible continuations has enabled widespread application in knowledge retrieval, summarization, translation, and dialogue systems. Their integration into healthcare has been particularly notable, with applications ranging from clinical documentation and literature synthesis to diagnostic support and educational innovation. Dentistry has begun to embrace AI in parallel with medicine, with early work demonstrating utility in diagnostic imaging, caries detection, treatment planning, and predictive analytics [ 1 ]. More recently, LLMs have been explored as tools for dental education and clinical training, where they may supplement student learning, provide rapid access to foundational knowledge, and support preparation for standardized examinations [ 2 , 3 ]. It is important to recognize, however, that LLMs are not designed to discover or verify truth. Instead, they function as statistical models that predict the most probable sequence of words given prior input. This capability allows them to generate fluent and contextually appropriate language, but it also constrains their factual reliability. Unlike structured databases, LLMs do not store or retrieve information in a queryable format. Although their outputs can be supplemented by external databases or curated sources, current methods provide limited transparency or control over how such sources are integrated or weighted [ 4 ]. These limitations are especially concerning in domains where factual precision is critical, such as medicine and research. Consequently, the systematic evaluation of LLMs has become a central priority, with growing efforts devoted to developing robust metrics and standardized approaches for assessing their performance across multiple dimensions. To date, much of the research evaluating LLMs’ performance and accuracy in medicine has employed standardized, high-stakes assessments such as the United States Medical Licensing Examination (USMLE), with systematic reviews confirming variable but generally improving performance across model generations [ 5 – 7 ]. In dentistry, similar efforts have examined LLM performance on the National Board Dental Examination (NBDE/INBDE) and related professional assessments [ 8 – 10 ]. While a recent study in the Journal of Periodontology reported on the use of the AAP Periodontics In-Service Examination to benchmark ChatGPT against specialty-level questions [ 11 ], another study by Sabri et al. evaluated GPT-4, GPT-3.5, and Google Gemini on 1,312 AAP Periodontics In-Service questions from 2020-2023 compared AI performance to periodontology residents across training years [ 12 ]. These studies underscore the value of standardized professional examinations as rigorous testbeds for evaluating LLM performance. While such studies primarily assess accuracy and reasoning, they have largely overlooked the ability of LLMs to provide verifiable citations. The widespread use of these models has revealed a consistent problem of fabricated or “hallucinated” references, where outputs may include plausible sounding but nonexistent sources or misattributed citations [ 13 ]. This issue is particularly problematic in educational settings, where students may lack the experience to critically evaluate references and risk accepting fabricated citations as valid. Such limitations not only undermine trust in AI-assisted learning but also pose risks to the integrity of scholarly work. Given these gaps, the present study was designed to provide a multidimensional evaluation of LLM performance in periodontology using the AAP In-Service Examination. The objectives were to compare model accuracy across successive generations, assess the influence of prompt optimization and test presentation format on performance, quantify the rate of citation hallucination, and evaluate the validity of self-reported confidence in responses. By employing a specialty-specific professional examination as the test environment, this study aimed to generate a comprehensive assessment of LLM capabilities and limitations in a context where factual reliability, citation integrity, and calibrated confidence are critical. Methods Study design This study employed a comparative performance assessment framework to evaluate large language models (LLMs) in the domain of periodontology. A controlled experimental design ( Fig 1 ) was used to ensure reproducibility and transparency. Evaluation focused on two dimensions, response accuracy and a human-in-the-loop (HITL) protocol to validate the reliability of model-generated citations. Three transformer-based models were selected based on published performance benchmarks and multimodal capabilities: GPT-4.0 and GPT-5.0 (OpenAI, San Francisco, CA, USA), and Claude Sonnet 4.0 (Anthropic, San Francisco, CA, USA). The citation validity was tested only for GPT 5.0 (OpenAI, San Francisco, CA, USA), and Claude Sonnet 4.0 (Anthropic, San Francisco, CA, USA). Fig 1. Study flow for evaluating large language model performance on a specialty periodontology examination. Open in a new tab Evaluation dataset The 2024 American Academy of Periodontology (AAP) In-Service Examination, comprising 331 multiple-choice questions, was used as the evaluation instrument. For structured analysis, the investigators categorized the items on the exam into four domains: general factual knowledge, author-specific/ study specifics factual knowledge, analytical reasoning, and image-based question. The AAP In-Service Examination is administered annually to periodontal residents across all three years of specialty training in Commission on Dental Accreditation (CODA)-accredited periodontics programs throughout the United States. The examination serves as a formative assessment tool designed to evaluate progressive knowledge acquisition and benchmark resident performance against national cohorts at equivalent training levels. Questions span foundational biomedical sciences (embryology, anatomy, biochemistry, physiology, microbiology, immunology) through advanced clinical domains (periodontal diagnosis, treatment planning, surgical therapy, pharmacotherapeutics, oral pathology). Content difficulty is calibrated across first-, second-, and third-year residency training levels, with questions reflecting contemporary clinical practice and current scientific evidence. The examination functions as an educational milestone within graduate periodontal education but does not serve as a licensure or board certification requirement. The AAP provides individual residents with confidential performance reports showing percentile rankings relative to same-year peers nationally. However, the AAP does not publicly release aggregate performance statistics, mean scores, or score distributions to maintain examination security and protect resident confidentiality. Two investigators (P.D. and R.H.) independently categorized all 331 examination questions into four cognitive domains based on the primary knowledge or skill required for accurate response: (1) General factual knowledge(n = 186): questions requiring recall of established scientific principles, definitions, or evidence-based facts without reference to specific studies; (2) Author-specific factual knowledge (n = 126): questions explicitly referencing named researchers, studies, or specific clinical trials; (3) Analytical reasoning (n = 9): questions presenting clinical scenarios requiring multi-step reasoning, diagnosis, treatment related knowledge and integration of concepts; and (4) Image-based questions (n = 10)—questions accompanied by visual materials requiring interpretation. Discrepancies were resolved through discussion until consensus was reached. Furthermore, the AAP exam is organized into nine topical sections, Embryology/Anatomy/Biostatistics, Biochemistry/Physiology, Microbiology/Immunology, Periodontal Etiology/Pathogenesis, Pharmacology/Therapeutics, Diagnosis, Treatment Planning/Prognosis, Therapy, Oral Pathology/Medicine, that were utilized for section level analysis. Model access and configuration All models were accessed through consumer web-based chat interfaces using default platform settings: GPT-4.0 and GPT-5.0 via GPT Plus subscription (chat.openai.com), and Claude Sonnet 4.0 via Claude Pro subscription (claude.ai). No API access was employed. Consumer interfaces were selected to reflect authentic user experiences in educational settings, where students and residents typically access these models through standard web-based interactions. Model parameters remained at platform defaults without user modification. Question presentation formats Two examination delivery formats were employed to assess the influence of context and question presentation on model performance. 1. Full Test (FT) Format Implementation. In the Full Test condition, all 331 questions were presented within a single conversation thread. Questions were formatted as a numbered list (Q1 through Q331) with answer choices (A through E) clearly delineated for each item. The entire examination was provided in one initial prompt, fitting within the standard context window. The model generated responses sequentially within a single output, providing answers for all questions in order. This format simulates unrestricted access to all examination content simultaneously, analogous to scenarios where students review all available questions before responding. 2. Individual Question (IQ) Format Implementation. In the Individual Question condition, questions were presented one at a time in sequential order within the same conversation thread. After the model provided a response to each question, the next question was presented in the subsequent user message within the ongoing conversation. This format mirrors traditional examination experiences where questions appear individually in sequence. Prompt design Prompt formulation was standardized across all experimental conditions to minimize variability and ensure that observed differences in performance reflected intrinsic model capabilities rather than disparities in instruction. All models were required to select the most appropriate answer choice for each multiple-choice question, and prompts were written to be unambiguous, concise, and task-specific, with contextual information included only when necessary to support question comprehension. A single core instruction was employed for baseline evaluation: “Select the single most accurate answer choice for the following multiple-choice question using all relevant knowledge and sources available to you.” For GPT-5.0 and Claude Sonnet 4.0, additional instructions were included to (1) indicate their confidence in the selected answer as a percentage (0–100%), and (2) provide verifiable citations including Digital Object Identifiers (DOIs), PubMed identifiers (PMIDs), or persistent URLs to support their response choice. These additional tasks were applied exclusively to these models to enable assessment of confidence calibration and citation validity alongside overall accuracy. Complete verbatim prompts for all experimental conditions are provided in Supplementary File S1 Text . Human-in-the-loop citation validation A human-in-the-loop protocol was implemented specifically for citation evaluation. All references generated by GPT-5.0 and Claude Sonnet 4.0 were reviewed by a single evaluator with specialty training in periodontology and biomedical literature retrieval. Citations were retrieved by searching standard academic databases PubMed, Google Scholar, Web of Science, and publisher databases. DOIs and PMIDs were verified against official registries (doi.org, pubmed.ncbi.nlm.nih.gov). For each citation, it was assessed whether the source could be located and whether bibliographic details were accurate (authors, year, journal/book title, volume, pages). Each citation was categorized using a three-tier taxonomy: Completely valid: existing and accessible citations accurately supporting the information provided, partially correct: real citations with minor inaccuracies or partial claim support, critically wrong: fabricated, inaccessible, or unrelated citations. This process enabled systematic detection of fabricated or hallucinated, bibliographically inaccurate or contextually misattributed references. Complete citation verification protocol and representative examples are provided in Supplementary File S3 Text . Ethics approval statement This study did not involve human participants or identifiable patient data. All questions were derived from publicly available educational resources. The study was conducted in alignment with principles of responsible AI research, emphasizing transparency, reproducibility, and human oversight in citation validation. Statistical analysis Statistical analyses were conducted using IBM SPSS Statistics (version 31, IBM Corp., Armonk, NY, USA) and Python (version 3.12.10) to evaluate model performance, assess citation validity, and explore secondary relationships. Descriptive statistics and chi-square tests of independence were used to summarize and compare response accuracy across models. Within each model version, the effect of question presentation format was evaluated using chi-square tests, and McNemar’s test was applied to discordant question-level pairs. For models that generated confidence scores (GPT-5.0 and Claude Sonnet 4.0), logistic regression was used to examine the relationship between confidence and accuracy. Citation validity was analyzed by categorizing references into predefined validity groups and comparing their distributions between models using chi-square tests. The potential effect of token length on accuracy was investigated using question position as a proxy, with logistic regression modeling accuracy as the dependent variable and question index as a continuous predictor, and the Cochran–Armitage trend test applied to assess linear trends across ordered question bins. Results Overall model performance A total of 331 multiple-choice questions from the 2024 AAP In-Service Examination were evaluated across four experimental conditions: GPT-4.0 Full Test (FT), GPT-4.0 Individual Question (IQ), GPT-5.0 FT, and GPT-5.0 IQ. Overall model accuracy varied significantly by version and delivery format. GPT-5.0 achieved the highest performance in IQ mode, correctly answering 279 of 331 questions (84.29%), compared with 274 of 331 (82.78%) for GPT-4.0 IQ. However, contrary to the individual question results, GPT-5.0 in FT mode achieved 80.85% accuracy (266 of 329 questions), substantially higher than GPT-4.0 FT at 78.12% (257 of 329 questions). Table 1 Table 1. Descriptive performance of large language models (LLMs) across subject domains and testing conditions. Accuracy of GPT-4.0 and GPT-5.0 was evaluated using the 2024 American Academy of Periodontology (AAP) In-Service Examination (331 multiple-choice questions). Results are presented as the number and percentage of correct responses for each section under Full Test (FT) and Individual Question (IQ) conditions. Completion rates and missing responses are also reported. IQ presentation consistently yielded higher accuracy than FT for both models, and GPT-5.0 outperformed GPT-4.0 across all sections. Section GPT-4.0 FT GPT-4.0 IQ GPT-5.0 FT GPT-5.0 IQ Claude Sonnet 4.0 Embryology and Anatomy/ Biostatistics 25/29 (86.21%) 26/29 (89.66%) 25/29 (86.21%) 27/29 (93.10%) 26/29 (89.66%) Biochemistry-Physiology 29/31 (93.55%) 30/31 (96.77%) 31/31 (100.00%) 29/31 (93.55%) 31/31 (100.00%) Microbiology & immunology 25/25 (100.00%) 25/25 (100.00%) 22/25 (88.00%) 24/25 (96.00%) 24/25 (96.00%) Periodontal Etiology and Pathogenesis 28/37 (75.68%) 30/37 (81.08%) 26/37 (70.27%) 29/37 (78.38%) 34/37 (91.89%) Pharmacology & Therapeutics 32/36 (88.89%) 31/36 (86.11%) 29/36 (80.56%) 32/36 (88.89%) 35/36 (97.22%) Diagnosis 23/29 (79.31%) 24/29 (82.76%) 19/29 (65.52%) 19/29 (65.52%) 18/29 (62.07%) Treatment planning and Prognosis 19/27 (70.37%) 20/27 (74.07%) 20/27 (74.07%) 21/27 (77.78%) 22/27 (81.48%) Therapy 42/77 (54.55%) 65/77 (84.42%) 58/77 (75.32%) 61/77 (79.22%) 62/77 (80.52%) Oral Pathology/ Oral Medicine 34/40 (85.00%) 23/40 (57.50%) 29/40 (72.50%) 37/40 (92.50%) 35/40 (87.50%) Total 257/331 (77.64%) 274/331 (82.78%) 259/331 (78.25%) 279/331 (84.29%) 287/331 (86.71%) Open in a new tab Section and domain specific performance patterns A network visualization of model performance across the 2024 American Academy of Periodontology In-Service Examination demonstrated consistent topic-dependent trends ( Fig 2 ). The visualization illustrates the distribution of model accuracy across nine content topics, with nodes representing large language model configurations (blue) and examination topics (yellow), and edge color and thickness indicating performance levels (green: ≥ 0.90, red: 0.80–0.89, gray: < 0.80). Descriptive analysis revealed that certain subject areas exhibited systematically higher accuracy scores across all tested models, particularly Biochemistry and Physiology (mean accuracy = 0.968) and Microbiology and Immunology (mean accuracy = 0.960), which can also be visualized as green connections (≥0.9 accuracy) with multiple LLMs in the network visualization. Conversely, several domains proved universally challenging, with Diagnosis (mean accuracy = 0.710) and Periodontal Therapy (mean accuracy = 0.748) showing predominantly red connections (<0.8 accuracy) across all model variants. Fig 2. Network visualization of large language model performance across examination sections. Open in a new tab This pattern remained consistent despite variations in model training approaches, with both full test and individual question versions of GPT 4.0 and 5.0, as well as Claude Sonnet 4.0, exhibiting similar relative performance hierarchies across subject domains. The network visualization further reveals that while individual models varied in overall performance, with Claude Sonnet 4.0 achieving the highest overall accuracy (0.874) and GPT 5.0FT the lowest (0.799), the ranking of subject difficulty remained remarkably stable, suggesting that certain dental education domains present inherent challenges for current large language model architectures that transcend specific training methodologies or model parameters. Fig 3 demonstrates marked heterogeneity in large language model (LLM) performance across dental education question taxonomies, revealing distinct competency profiles that transcend model architecture and training methodologies. Radar chart analysis of 331 questions across four cognitive domains: factual recall (n = 186), author-referenced queries (n = 126), image-based interpretation (n = 10), and analytical reasoning (n = 9), exposed systematic performance variations that remained consistent across five state-of-the-art models: GPT 4.0 (FT & IQ), GPT 5.0 (FT &IQ), and Claude Sonnet 4.0. Fig 3. Radar plot comparing model accuracy across four question categories—factual, author-specific, analytical, and image-based—for all LLM configurations. Open in a new tab Comparison within successive generations of LLMs: GPT 4.0 vs. GPT 5.0 A chi-square test of independence was conducted to compare the overall accuracy of GPT-4.0 and GPT-5.0 under Full Test (FT), Individual Question (IQ), and combined conditions. GPT-5.0 exhibited modest but consistent improvements in accuracy across all testing conditions, although none of the differences reached statistical significance (α = 0.05) ( Table 2 ). The largest difference was observed in the FT condition (+2.74 percentage points; χ² = 0.596, p = 0.440), while the improvement in the IQ condition was smaller (+1.51 percentage points; χ² = 0.176, p = 0.675). Overall, GPT-5.0 achieved 82.58% accuracy compared with 80.45% for GPT-4.0 (χ² = 0.850, p = 0.357), and all effect sizes indicated small magnitudes of difference (Cramer’s V ≤ 0.030). These findings suggest incremental advancements in model capability between generations, though without statistically significant performance gains. Table 2. Chi-square test of independence comparing GPT-4.0 and GPT-5.0 performance across presentation conditions. Comparison GPT-4.0 Accuracy GPT-5.0 Accuracy Difference χ² p-value Effect Size Significant N FT Mode Only 78.12% 80.85% +2.74pp 0.596 0.440 0.030 No 658 IQ Mode Only 82.78% 84.29% +1.51pp 0.176 0.675 0.016 No 662 Overall Combined 80.45% 82.58% +2.12pp 0.850 0.357 0.025 No 1320 Open in a new tab Note: Sample sizes differ between FT and IQ conditions due to incomplete responses in FT mode. All IQ condition runs produced complete responses (N = 331). Effect of presentation format (Full Test (FT) vs. Individual Question (IQ)) Comparison of model performance across presentation formats revealed a consistent pattern favoring the Individual Question (IQ) condition over the Full Test (FT) condition. Although differences did not reach conventional levels of statistical significance (α = 0.05), IQ presentation was associated with higher accuracy across all analyses. GPT-4.0 achieved 82.98% accuracy in IQ format compared to 78.12% in FT format, representing a 4.86 percentage point improvement (χ² = 2.18, p = 0.140). GPT-5.0 demonstrated a similar directional effect, with accuracy improving from 80.85% (FT) to 84.19% (IQ), a 3.34 percentage point increase (χ² = 1.05, p = 0.305). When both models were combined, overall performance improved by 4.10 percentage points (χ² = 3.41, p = 0.065), approaching statistical significance (Cramer’s V = 0.051). Paired question-level analyses further supported this trend. McNemar’s test revealed that GPT-4.0 achieved more correct responses in IQ mode than FT mode on discordant items (39 vs. 23; χ² = 3.63, p = 0.057), indicating a performance improvement approaching statistical significance. ( Table 3 ) A similar pattern was observed for GPT-5.0, though the difference was not statistically significant (38 vs. 27; χ² = 1.54, p = 0.215). Together, these results suggest that presenting questions individually may confer practical advantages by reducing contextual interference or cognitive overload, leading to modest but consistent performance gains. Table 3. McNemar’s test results for paired comparison of Full Test and Individual Question conditions. Model Discordant Pairs (Total) FT-Only Correct IQ-Only Correct McNemar Statistic p -value Interpretation GPT-4.0 62 23 39 3.63 0.057 Approaching significance GPT-5.0 65 27 38 1.54 0.215 Not significant Open in a new tab Calibration: Confidence–Accuracy relationship Binary logistic regression analysis demonstrated that confidence scores were statistically significant predictors of response accuracy across all models ( p < 0.001) ( Table 4 ). GPT-5.0 in the Individual Question format showed the strongest association (OR = 1.125 per confidence point), indicating a 12.5% increase in the odds of a correct response for each unit increase in confidence. Claude Sonnet 4.0 exhibited the highest overall classification accuracy (86.4%), although with the weakest discriminative ability (ROC AUC = 0.589), while GPT-5.0 FT demonstrated moderate predictive performance (OR = 1.018). All models showed excellent sensitivity (99.6–99.7%), reflecting reliable prediction of correct answers, but poor specificity (0–1.9%), indicating limited ability to identify incorrect responses. These findings confirm that model-generated confidence scores reliably tracked internal certainty and were significantly associated with accuracy, supporting their potential utility for evaluating AI reliability in clinical and educational settings. Table 4. Logistic regression analysis of question order and confidence as predictors of accuracy across models. Variable GPT-5.0 FT OR (95% CI) p -value GPT-5.0 IQ OR (95% CI) p -value Claude Sonnet 4.0 IQ OR (95% CI) p -value Question number 0.997 (0.993–1.000) 0.035* 0.998 (0.995–1.001) 0.223 0.995 (0.992–0.999) 0.011* Confidence 1.013 (0.999–1.028) 0.068 1.127 (1.051–1.208) <0.001*** 1.020 (0.986–1.055) 0.247 Open in a new tab Model accuracy degradation across token length GPT-5.0 Full Test demonstrated significant question fatigue (OR = 0.997 per question, p = 0.035) with strong linear decline (R² = 0.784), indicating a 0.3% decrease in accuracy odds per question. Confidence showed marginal significance (OR = 1.013, p = 0.068). GPT-5.0 Individual Question mode showed no significant question fatigue effect (OR = 0.998, p = 0.223) with minimal linear correlation (R² = 0.095), but maintained strong confidence-accuracy calibration (OR = 1.127, p < 0.001). Claude Sonnet 4.0 Individual Question exhibited the most pronounced question fatigue (OR = 0.995 per question, p = 0.011) with moderate linear decline (R² = 0.666) and no significant confidence effect (OR = 1.020, p = 0.247). Scatter plots in Fig 4 illustrate the relationship between question position (x-axis) and predicted probability of a correct response (y-axis) derived from multivariable logistic regression models for three large language models. Each panel represents a different model and evaluation format: (A) GPT-5.0 – Full Test condition, (B) GPT-5.0 – Individual Question condition, and (C) Claude Sonnet4.0 Individual Question condition. Blue points indicate predicted probabilities for individual questions, with solid lines representing fitted regression trends and shaded areas showing 95% confidence intervals. All three models exhibited a downward trajectory in accuracy as question number increased, suggesting potential cumulative context effects. This decline was most pronounced and statistically significant in GPT-5.0 Full Test mode (OR = 0.997 per question, p = 0.035, R² = 0.784) and in Claude Sonnet 4.0 Individual Question mode (OR = 0.995, p = 0.011, R² = 0.666), while GPT-5.0 Individual Question showed a milder, non-significant trend (OR = 0.998, p = 0.223, R² = 0.095). Fig 4. Multivariable logistic regression analyses examining the relationship between question position (proxy for cumulative context exposure) and model accuracy are shown for two large language models. Open in a new tab Each panel plots the predicted probability of a correct response (y-axis) against question number (x-axis), with fitted regression lines and 95% confidence intervals. (A) GPT-5.0 in Full Test mode demonstrated significant question fatigue (OR = 0.997 per question, p = 0.035, R² = 0.784), indicating a progressive 0.3% reduction in accuracy odds per question. (B) GPT-5.0 in Individual Question mode showed no significant decline across question sequence (OR = 0.998, p = 0.223, R² = 0.095), maintaining stable performance with strong confidence–accuracy calibration (OR = 1.127, p < 0.001). (C) Claude Sonnet 4.0 in Individual Question mode exhibited a significant decline in accuracy with increasing question number (OR = 0.995 per question, p = 0.011, R² = 0.666), consistent with model-specific susceptibility to cumulative context effects. Citation validity and accuracy In contrast to confidence scores, citation validity did not significantly predict response accuracy for any model ( p > 0.05) ( Table 5 ). Although effect directions varied—with valid citations associated with a 35.9% increase in odds of a correct response for GPT-5.0 IQ and decreases of 31.4% and 11.7% for GPT-5.0 FT and Claude Sonnet 4.0, respectively—none of these associations were statistically significant. Table 5. Univariable logistic regression - Citation validity predicting accuracy. Model n Citation OR (95% CI) p-value Percentage Effect GPT-5.0 FT ~264 0.686 (0.370-1.272) 0.232 -31.4% GPT-5.0 IQ ~328 1.359 (0.376-4.909) 0.640 +35.9% Claude Sonnet 4.0 IQ ~322 0.883 (0.475-1.642) 0.694 -11.7% Open in a new tab Citation hallucination assessment Citation validity evaluation revealed substantial differences across models. Claude Sonnet 4.0 demonstrated the lowest hallucination rate at 5.59% (18/322), GPT-4.0 IQ showed 32.01% (105/328), and GPT-5.0 FT exhibited the highest rate at 51.53% (135/262). Citations were systematically classified using a three-tier validation system: fully verifiable, partially correct (minor errors in volume/page numbers), and critically wrong (fabricated authors, studies, journals, or completely unrelated content). The heatmap in Fig 5 illustrates citation validity across three large language models (5.0 FT, IQ5.0 IQ, and Claude Sonnet 4.0) for 331 questions organized into nine topical sections (S1–S9). Each thin vertical strip represents one question’s citation score, color‐coded from red (0 = critically wrong) through orange (1 = partially correct, multiple errors) and yellow (2 = partially correct, fine errors) to green (3 = fully verified), with gray indicating missing responses. Claude Sonnet 4.0 exhibits a predominance of green strips (>90% fully verified) in all sections, reflecting consistently high citation accuracy. IQ5.0 IQ shows substantial green interspersed with orange and red strips, approximately two‐thirds fully verified, indicating moderate performance. In contrast, 5.0 FT displays frequent red and gray gaps, especially in sections S4, S7, and S8, underscoring its lower overall reliability and higher rates of missing or incorrect citations. The black vertical separators delineate topical sections, revealing that some sections (e.g., S5 and S6) pose greater challenges for all models, as evidenced by increased yellow and red densities. Overall, this compact visualization highlights clear performance gradients among the models and identifies specific content areas where citation validity deteriorates. Fig 5. Heatmap showing citation validity across nine exam sections for three LLMs, with green indicating valid citations, yellow partially valid, red invalid, and gray missing. Open in a new tab Relationship of citation source to citation validity Pearson’s chi-square tests revealed a highly significant association between citation source type (article vs. book) and citation validity across all three models (χ² ≥ 236, df = 3, Bonferroni-adjusted p < 1 × 10 ⁻ ⁵⁰). These results indicate that the likelihood of a citation being verified is strongly dependent on the type of source from which it is drawn. In practical terms, all models demonstrated a higher proportion of fully validated citations when referencing journal articles compared with books. For example, in the GPT-5.0 FT condition, approximately 62% of article-derived citations were fully verified compared with only 36% of book-based citations, with the latter contributing disproportionately to critically incorrect or missing references. Similar patterns were observed for GPT-5.0 IQ and Claude Sonnet 4.0, underscoring a systematic bias in citation reliability based on source type. Discussion Large language models (LLMs) have demonstrated substantial advancements in capability over recent years, with increasing integration across biomedical research, clinical practice, and professional education. [ 14 ]These models exhibit sophisticated capacity for synthesizing complex information and generating contextually appropriate responses, establishing their utility as computational tools within scientific and clinical frameworks. Nevertheless, responsible implementation necessitates comprehensive evaluation methodologies that transcend conventional performance metrics. Conventional assessment paradigms have predominantly employed static benchmarks including the Massive Multitask Language Understanding (MMLU), TruthfulQA, and General Language Understanding Evaluation (GLUE) tasks. [ 15 ] While these standardized assessments provide foundational performance data, they demonstrate limited capacity to predict model behavior under real-world valid conditions. Contemporary evaluation initiatives within biomedical domains have concentrated primarily on task accuracy, operationally defined as the proportion of correct responses under controlled experimental conditions. The present investigation, while not constituting a comprehensive system-level or longitudinal assessment, was designed within this methodological framework. The study focused on critical behavioral dimensions including response accuracy, confidence calibration, citation validity, and hallucination frequency, thereby addressing key considerations for clinical and educational applications. Comparing model performances Our findings demonstrate that commercially available LLMs achieve substantial performance levels on specialty-specific assessments, with overall accuracy ranging from approximately 78% to 87% depending on model architecture and delivery methodology. These results corroborate emerging literature documenting progressive performance improvements across successive LLM generations and establish that general-purpose models, absent domain-specific fine-tuning, can effectively respond to complex professional-level inquiries. However, accuracy exhibited heterogeneous distribution across content domains. Superior performance was observed in knowledge-intensive domains including biochemistry, physiology, and microbiology, where questions predominantly require retrieval of well-established factual information. Conversely, integrative domains, particularly diagnosis and periodontal therapy presented greater challenges across all evaluated models. Performance analysis across question typologies revealed that LLMs demonstrated proficiency with both factual and analytical question formats. Notably, contextualized questions yielded superior performance compared to broader conceptual inquiries, as illustrated in the radar plot visualization. The observed superiority of LLM performance on factual versus analytical periodontal questions aligns with patterns documented in a comprehensive systematic review and network meta-analysis by Wang et al. Their analysis of 35,896 medical questions across 168 studies demonstrated consistent LLM superiority on “objective questions” (characterized by clear, quantifiable responses) compared to “open-ended questions” (requiring complex reasoning without predetermined solutions).[ 16 ] This performance pattern remained consistent across examination sections despite variations in model training methodologies. Both full-test and individual question implementations of GPT 4.0, GPT 5.0, and Claude Sonnet 4.0 exhibited comparable relative performance hierarchies across subject domains. Network visualization analysis revealed that while individual models demonstrated varying overall performance, subject difficulty rankings remained remarkably stable, indicating that specific dental education domains present inherent challenges for current LLM architectures. Cross-generational model performance analysis revealed that GPT-5.0 demonstrated modest yet consistent improvements relative to GPT-4.0 across all testing conditions. The most substantial performance gain was observed in the full-test format (+2.74 percentage points), with attenuated differences in the individual-question format (+1.51 percentage points). However, these differences did not achieve statistical significance (χ² = 0.850, p = 0.357; Cramer’s V = 0.025), indicating that while iterative model refinements contribute to incremental accuracy improvements, their impact remains constrained within specialized domain assessments. These findings present a notable contrast to broader benchmark evaluations that document substantial performance gains for GPT-5 relative to GPT-4 across various domains. In standardized assessments, GPT-5 achieved 94.6% accuracy on the AIME 2025 mathematics benchmark compared to GPT-4’s approximately 52% performance and attained 74.9% accuracy on SWE-bench Verified coding tasks versus GPT-4’s 30–50% range. [ 17 , 18 ] Healthcare applications demonstrate a more nuanced performance profile. While GPT-5 exhibited reduced hallucination rates to 1.6% on selected medical benchmarks compared to GPT-4o’s 15.8%, performance varies considerably across different healthcare tasks, with heterogeneous results on clinical reasoning and medical knowledge assessments.[ 18 , 19 ]The observed disparity between general benchmark improvements and domain-specific performance suggests that while GPT-5 demonstrates marked advances in structured cognitive tasks such as mathematics and coding, its impact on specialized professional applications remains inconsistent. [ 20 ] This pattern indicates that generational improvements in large language models may not uniformly translate to enhanced performance across all professional domains, necessitating domain-specific evaluation frameworks for clinical and educational applications. Comparing the method of data presentation LLMs are rapidly evolving, and the ways in which users interact with them continue to adapt as our understanding of their capabilities deepens. Prompt engineering has always been central to optimizing model performance; however, our approach to it has matured, moving from the assumption that longer, more elaborate prompts are inherently better to the recognition that prompts should instead be information-rich, detailed, and contextually specific, yet concise and focused . [ 21 ] Another critical dimension of LLM behavior relates to the influence of input length and conversational duration on accuracy. Several studies demonstrate that performance can degrade as context length increases, with models showing diminished ability to retrieve or reason over information buried deeper within extended inputs. This so-called “lost-in-the-middle” phenomenon underscores how the structure and sequencing of information directly affect output quality. [ 22 , 23 ] Some models, such as Claude Sonnet 4.0, even prompt users to begin a new conversation when the input becomes too long, reflecting the practical implications of this limitation. To examine how these dynamics shape model performance, our study deliberately varied data presentation strategies: in one condition, the entire assessment was provided at once to maximize contextual availability, whereas in another, questions were presented sequentially to minimize token load. This design allowed us to investigate whether well-documented declines in performance with increasing context length persist across different presentation modes and whether strategic segmentation can mitigate accuracy loss in specialized tasks. Our results reveal two distinct but complementary phenomena. First, presentation format demonstrated a consistent directional effect on outcome accuracy. Individual Question presentation produced improvements across both models, with combined analysis approaching statistical significance (χ² = 3.41, p = 0.065). GPT-4.0 showed a 4.86 percentage point improvement (p = 0.140), while GPT-5.0 demonstrated a 3.34 percentage point gain (p = 0.305). McNemar’s paired analysis corroborated this pattern, with GPT-4.0 approaching significance (p = 0.057) and GPT-5.0 showing similar directional effects (p = 0.215). Although statistical significance was not achieved, the directional consistency across models suggests that individual question presentation may mitigate contextual interference or cognitive overload, leading to modest but practically relevant performance improvements. Second, analysis of question order as a proxy for increasing token load revealed differential ‘question fatigue’ effects across models and presentation formats. GPT-5.0 in full-test mode demonstrated significant accuracy degradation, with odds of a correct response decreasing by 0.3% per question (OR = 0.997, p = 0.035, R² = 0.784). This effect was not observed in GPT-5.0 individual question mode, which showed no significant decline (OR = 0.998, p = 0.223, R² = 0.095). Claude Sonnet 4.0 in individual question mode also exhibited significant question fatigue (OR = 0.995, p = 0.011, R² = 0.666), with a more pronounced decline in accuracy across the sequence, although its overall accuracy remained high. These findings suggest that question fatigue is not simply a function of cumulative computational burden, but rather reflects complex interactions between model architecture, presentation format, and context management. The progressive decline in full-test mode aligns with theories of attention saturation and positional encoding inefficiency in transformer architectures, consistent with Levy et al. (2024) findings of decreased reasoning accuracy as input length increased from 250 to 3000 tokens.[ 24 ] An important methodological consideration is the potential for in-context learning in this format. When multiple queries occur sequentially within the same conversation thread, large language models may adjust subsequent responses based on patterns, topics, or answer distributions detected in previous exchanges. Unlike the Full Test format where all questions are visible simultaneously, the Individual Question format presents information sequentially, but the conversational context accumulates across questions. This design difference between FT (all questions at once) and IQ (sequential presentation within continuous conversation) represents distinct cognitive task structures that may differentially affect model performance. The extent to which this affected performance relative to fully isolated question presentation (separate conversation threads) cannot be determined from our data, though it may partially explain the modest performance advantage observed in IQ format. Future studies should explore these effects by comparing performance across isolated-session versus sequential-conversation presentation modes. It should also be noted that question position serves as a within-format exploratory variable in these analyses. Position effects in FT mode may reflect simultaneous processing load, whereas position effects in IQ mode may reflect sequential conversation history accumulation. Direct cross-format comparisons of position effects should therefore be interpreted cautiously, as the formats represent fundamentally different cognitive task structures. Confidence and calibration: Insights into model self-assessment Confidence estimation was incorporated as a key parameter because it reflects an AI model’s capacity to assess the reliability of its own outputs, a critical property for effective deployment in educational and clinical decision-support contexts. Optimal performance requires that self-reported confidence align closely with the probability of correct responses, a characteristic termed calibration. Analysis revealed that while confidence estimates from both GPT-5.0 and Claude Sonnet 4.0 were not perfectly calibrated to response accuracy, they demonstrated statistically significant predictive relationships across all evaluated systems. Binary logistic regression confirmed that confidence scores served as significant predictors of model accuracy, with GPT-5.0 in Individual Question mode exhibiting the strongest predictive association, followed by Claude Sonnet 4.0 and GPT-5.0 in Full Test mode. This hierarchy suggests that segmented input presentation enhances confidence-accuracy calibration. The observed relationship between confidence and accuracy, while statistically significant, indicates that confidence metrics provide useful but imperfect indicators of response reliability. These findings underscore the necessity for complementary validation mechanisms capable of more effectively identifying uncertain or potentially incorrect responses in high-performing AI systems deployed in professional contexts. Citation integrity and hallucination Perhaps the most consequential limitation revealed by this study was the substantial variability in citation quality. While both GPT-5.0 and Claude Sonnet 4.0 generated references when prompted, a considerable proportion were partially incorrect or wholly fabricated, a phenomenon termed “citation hallucination.” This behavior reflects the generative nature of LLMs, which predict text patterns rather than retrieve verified database records. Although models frequently produced references with plausible formatting and content, many could not be traced to authentic sources or failed to support their associated claims. This limitation is particularly problematic in educational and research settings, where users may rely on citations for further reading, evidence validation, or scholarly writing. Fabricated references not only undermine user trust but also risk propagating misinformation if accepted uncritically. Analysis revealed that citation validity was independent of answer accuracy, similar to seen by Danesh et al, 2025. [ 25 ] Human-in-the-loop assessment demonstrated that hallucinated citations existed across a spectrum of severity. The three-tier classification system differentiated between fully verifiable citations, partially correct citations with minor errors (incorrect volume numbers, page references), and critically erroneous citations containing fabricated authors, non-existent studies, incorrect journal attributions, or wholly unrelated content. The predominance of critically erroneous hallucinations, rather than minor bibliographic inaccuracies indicates that AI citation fabrication represents a qualitatively distinct problem from human citation errors, consistent with findings reported by Aljamaan et al. [ 26 ] Citation performance varied systematically across question categories. Author-specific and analytical questions achieved consistently higher citation accuracy across all models compared to non-specific factual questions, suggesting that contextual specificity significantly influences citation reliability. Non-specific factual questions exhibited the highest incidence of critically incorrect citations, paralleling patterns observed in answer accuracy metrics. These findings underscore the critical need for enhanced citation validation frameworks, including improved retrieval-augmented generation (RAG) mechanisms, real-time database cross-referencing systems, or hybrid architectures that integrate generative capabilities with verified bibliographic databases. Such implementations would ensure that model-generated references maintain both authenticity and contextual relevance, thereby addressing the fundamental disconnect between pattern-based text generation and evidence-based scholarly citation practices. Broader implications, limitations and future directions These findings collectively demonstrate both the potential and current constraints of LLMs as educational and knowledge-support tools in specialized dental practice. The observed high overall accuracy indicates that such models can effectively augment learning processes and assessment preparation, particularly for reinforcing foundational knowledge and simulating examination-style questioning protocols. However, persistent limitations in reasoning-intensive clinical domains, suboptimal confidence calibration, and systematic citation reliability deficits necessitate continued human oversight for safe and effective implementation. It is important to highlight that these findings should be interpreted considering some important limitations of the present study. First, the evaluation was restricted to a single specialty examination in periodontology and to multiple-choice items with predefined answer options. This design was appropriate for objective scoring and direct comparison across models, but it does not capture how LLMs, especially those with newer reasoning focused architectures behave in open-ended, case-based clinical queries. Second, only commercially available, general-purpose models accessed through consumer web interfaces were examined; domain-specialized, institutionally fine-tuned, or open-source models may exhibit different performance and calibration profiles. Third, citation validity was assessed by a single evaluator, which ensured internal consistency but did not allow formal inter-rater reliability estimations. While these general-purpose systems demonstrated strong performance relying solely on pre-trained knowledge, high-stakes clinical or educational deployment often require domain-specific fine-tuning or reinforcement learning from expert feedback to meet healthcare reliability standards. This distinction has implications for funding priorities, regulatory frameworks to create clinically validated systems, and institutional strategy. The 2024 AAP In-Service Examination was made available online in the latter part of 2024, and this study was conducted shortly thereafter. Given the recency of the examination and the typical training data cutoff dates for the models evaluated (GPT-4.0, GPT-5.0, Claude Sonnet 4.0), direct exposure to these specific 2024 questions during pre-training is unlikely. However, we cannot rule out the possibility that similar questions from prior years, related periodontal content, or overlapping educational materials may have been present in training corpora. This represents an inherent limitation of evaluating foundation models on domain-specific assessments post hoc. Citation validation was performed by a single evaluator, and no inter-rater reliability testing was conducted; future studies should incorporate independent dual-review to enhance objectivity and reproducibility. Finally, confidence to accuracy relationships were evaluated under a single prompting strategy, no chain of thought prompting was utilized and evaluation was limited to the constraints of two examination presentation formats, leaving open how alternative prompt designs or reasoning-focused model architectures might affect performance in integrative clinical domains. Future investigations should extend this multidimensional evaluation framework through several key directions: (1) longitudinal assessments spanning multiple examination cycles to evaluate temporal stability of model performance, (2) expansion of citation validation protocols incorporating larger, diverse reviewer panels to enhance reliability of bibliographic assessment, multiple raters and (3) integration of advanced calibration metrics including Brier scores, expected calibration error, and reliability diagrams to provide more nuanced confidence-accuracy relationships (4) evaluation of reasoning-focused model architectures to assess whether enhanced deliberative processes improve performance in integrative clinical domains such as diagnosis and therapy planning, and (5) incorporation of structured justification elicitation to examine whether explanation quality correlates with correctness and calibration, providing additional reliability indicators beyond numerical confidence scores. Additionally, comparative analyses across multiple dental specialties and integration of real-world clinical scenarios would strengthen the ecological validity of these findings. As LLM architectures continue advancing, such comprehensive, specialty-specific evaluation protocols will prove essential for guiding evidence-based integration into dental education curricula, clinical decision-support systems, and scholarly research workflows. The framework established in this study provides a foundation for systematic assessment of emerging AI technologies in professional healthcare education contexts. Conclusion While large language models demonstrate the capacity to answer a wide range of dental assessment questions, their reliability is variable and highly dependent on both the context and the manner in which information is presented. This variability underscores that, although LLMs can serve as valuable adjuncts for knowledge reinforcement and assessment preparation, their outputs cannot be assumed accurate or trustworthy in all scenarios. In educational and research applications, it is essential that all LLM-generated responses, particularly those involving complex reasoning or citation are subject to rigorous human review. Ongoing human oversight and validation remain critical to ensure the integrity and safety of information, and future research should focus on developing robust frameworks for integrating LLMs responsibly within dental education and scholarly practice. Supporting information S1 Text. Complete prompt documentation. Full documentation of all standardized prompts used for querying GPT-4.0, GPT-5.0, and Claude Sonnet 4.0 during the evaluation. (DOCX) pdig.0001072.s001.docx (23.2KB, docx) S2 Text. Data dictionary. Definitions and descriptions of all variables included in the study dataset. (DOCX) pdig.0001072.s002.docx (18.5KB, docx) S3 Text. Human-in-the-loop citation review protocol. Detailed protocol used for expert review of model-generated citations and hallucination classification. (DOCX) pdig.0001072.s003.docx (21.9KB, docx) S1 Data. Multidimensional evaluation dataset. Complete dataset including model responses, accuracy scores, confidence ratings, and citation validity results. (XLSX) pdig.0001072.s004.xlsx (51KB, xlsx) S2 Data. Raw data. Raw unprocessed data from all model evaluation sessions. (XLSX) pdig.0001072.s005.xlsx (85.6KB, xlsx) Acknowledgments The authors thank Hui Bian, Ph.D., Research & Statistics Consultant, Office for Faculty Excellence, East Carolina University, for providing valuable statistical consultation and guidance during the data analysis phase of this study. Data Availability All data underlying the findings of this study are available as supplementary files accompanying this manuscript. These include tabulated model selected responses, confidence scores, citation validity codes, and raw citation data along with detailed outline of citation validity framework and data dictionary. No data were deposited in an external repository as the complete dataset is provided within the supplementary materials. Funding Statement The author(s) received no specific funding for this work. References 1. Khanna S, Dhaimade PA, Khanna S. Artificial intelligence: transforming dentistry today. Indian J Basic Appl Med Res [Internet]. 2017;(6):161–7. Available from: www.ijbamr.com [ Google Scholar ] 2. Fang Q, Reynaldi R, Araminta AS, Kamal I, Saini P, Afshari FS. Artificial Intelligence (AI)-driven dental education: exploring the role of chatbots in a clinical learning environment. J Prosthet Dent. 2025;134(4):1296–303. [ DOI ] [ PubMed ] [ Google Scholar ] 3. Bhatia AP, Lambat A, Jain T. A comparative analysis of conventional and chat-generative pre-trained transformer-assisted teaching methods in undergraduate dental education. Cureus. 2024. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. The rise of large language models. Nat Comput Sci. 2025;5(9):689–90. doi: 10.1038/s43588-025-00890-x [ DOI ] [ PubMed ] [ Google Scholar ] 5. Knoedler L, Knoedler S, Hoch CC, Prantl L, Frank K, Soiderer L, et al. In-depth analysis of ChatGPT’s performance based on specific signaling words and phrases in the question stem of 2377 USMLE step 1 style questions. Sci Rep. 2024;14(1):13553. doi: 10.1038/s41598-024-63997-7 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 6. Wang MH, Jiang X, Zeng P, Li X, Chong KK-L, Hou G, et al. Balancing accuracy and user satisfaction: the role of prompt engineering in AI-driven healthcare solutions. Front Artif Intell. 2025;8:1517918. doi: 10.3389/frai.2025.1517918 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 7. Liu M, Okuhara T, Chang X, Shirabe R, Nishiie Y, Okada H, et al. Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis. J Med Internet Res. 2024;26:e60807. doi: 10.2196/60807 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 8. Snigdha NT, Batul R, Karobari MI, Adil AH, Dawasaz AA, Hameed MS. Assessing the performance of ChatGPT 3.5 and ChatGPT 4 in operative dentistry and endodontics: an exploratory study. Hum Behav Emerg Technol. 2024;2024(1). [ Google Scholar ] 9. Kinikoglu I. Evaluating ChatGPT and Google Gemini performance and implications in Turkish dental education. Cureus. 2025. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 10. Jin HK, Lee HE, Kim EY. Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Med Educ. 2024;24(1). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 11. Danesh A, Pazouki H, Danesh F, Danesh A, Vardar-Sengul S. Artificial intelligence in dental education: ChatGPT’s performance on the periodontic in-service examination. J Periodontol. 2024;95(7):682–7. doi: 10.1002/JPER.23-0514 [ DOI ] [ PubMed ] [ Google Scholar ] 12. Sabri H, Saleh MHA, Hazrati P, Merchant K, Misch J, Kumar PS. Performance of three artificial intelligence (AI)-based large language models in standardized testing; implications for AI-assisted dental education. J Periodontal Res. 2025;60(2):121–33. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Gravel J, D’Amours-Gravel M, Osmanlliu E. Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin Proc Digit Health. 2023;1(3):226–34. doi: 10.1016/j.mcpdig.2023.05.004 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 14. Choi WC, Chang CI. ChatGPT-5 in education: new capabilities and opportunities for teaching and learning [Internet]; 2025. Available from: https://www.preprints.org/manuscript/202508.0684/v1 [ Google Scholar ] 15. LLM evaluation: metrics, frameworks, and best practices | genai-research – Weights & Biases [Internet]. [cited 2025 Oct 1]. Available from: https://wandb.ai/onlineinference/genai-research/reports/LLM-evaluation-Metrics-frameworks-and-best-practices--VmlldzoxMTMxNjQ4NA#evaluation-methodologies [ Google Scholar ] 16. Wang L, Li J, Zhuang B, Huang S, Fang M, Wang C, et al. Accuracy of large language models when answering clinical research questions: systematic review and network meta-analysis. J Med Internet Res. 2025;27:e64486. doi: 10.2196/64486 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. Dinc U, Sarkar J, Schubert P, Semrau S, Weissmann T, Karius A, et al. Benchmarking GPT-5 in radiation oncology: measurable gains, but persistent need for expert oversight; 2025. [cited 2025 Oct 1]. Available from: https://arxiv.org/pdf/2508.21777 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 18. Introducing GPT-5 | OpenAI [Internet]. [cited 2025 Oct 1]. Available from: https://openai.com/index/introducing-gpt-5/ [ Google Scholar ] 19. Owono L. GPT-4 vs GPT-5: How a 74.9% SWE-bench Score Rewires Coding. AWS in Plain English [Internet]; 2025. [cited 2025 Oct 1]. Available from: https://aws.plainenglish.io/gpt-4-vs-gpt-5-how-a-74-9-swe-bench-score-rewires-coding-fa2b08907067 [ Google Scholar ] 20. Ferrag MA, Tihanyi N, Debbah M. Reasoning beyond limits: advances and open problems for LLMs. ICT Express. 2025;11(6):1054–96. doi: 10.1016/j.icte.2025.09.003 [ DOI ] [ Google Scholar ] 21. The Ultimate Guide to Prompt Engineering in 2025 | Lakera – Protecting AI teams that disrupt the world [Internet]. [cited 2025 Oct 2]. Available from: https://www.lakera.ai/blog/prompt-engineering-guide [ Google Scholar ] 22. Liu NF, Lin K, Hewitt J, Paranjape A, Bevilacqua M, Petroni F, et al. Lost in the middle: how language models use long contexts. Available from: doi: 10.1162/tacl [ DOI ] [ Google Scholar ] 23. Li Y, Wu B, Huang Y, Luan S. Developing trustworthy artificial intelligence: insights from research on interpersonal, human-automation, and human-AI trust. Front Psychol. 2024;15:1382693. doi: 10.3389/fpsyg.2024.1382693 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 24. Levy M, Jacoby A, Goldberg Y. Same task, more tokens: the impact of input length on the reasoning performance of large language models [Internet]. Vol. 1. Available from: https://github.com/alonj/Same-Task-More-Tokens [ Google Scholar ] 25. Danesh A, Danesh F, Danesh A. ChatGPT’s risk of misinformation in dentistry. J Am Dent Assoc. 2025;156(1):3–5. doi: 10.1016/j.adaj.2024.05.003 [ DOI ] [ PubMed ] [ Google Scholar ] 26. Aljamaan F, Temsah MH, Altamimi I, Al-Eyadhy A, Jamal A, Alhasan K. Reference hallucination score for medical artificial intelligence chatbots: development and usability study. JMIR Med Inform. 2024;12:e54345. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials S1 Text. Complete prompt documentation. Full documentation of all standardized prompts used for querying GPT-4.0, GPT-5.0, and Claude Sonnet 4.0 during the evaluation. (DOCX) pdig.0001072.s001.docx (23.2KB, docx) S2 Text. Data dictionary. Definitions and descriptions of all variables included in the study dataset. (DOCX) pdig.0001072.s002.docx (18.5KB, docx) S3 Text. Human-in-the-loop citation review protocol. Detailed protocol used for expert review of model-generated citations and hallucination classification. (DOCX) pdig.0001072.s003.docx (21.9KB, docx) S1 Data. Multidimensional evaluation dataset. Complete dataset including model responses, accuracy scores, confidence ratings, and citation validity results. (XLSX) pdig.0001072.s004.xlsx (51KB, xlsx) S2 Data. Raw data. Raw unprocessed data from all model evaluation sessions. (XLSX) pdig.0001072.s005.xlsx (85.6KB, xlsx) Data Availability Statement All data underlying the findings of this study are available as supplementary files accompanying this manuscript. These include tabulated model selected responses, confidence scores, citation validity codes, and raw citation data along with detailed outline of citation validity framework and data dictionary. No data were deposited in an external repository as the complete dataset is provided within the supplementary materials. Articles from PLOS Digital Health are provided here courtesy of PLOS ACTIONS View on publisher site PDF (1.2 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Potenzial und Herausforderungen von Large Language Models in der klinischen Entscheidungsfindung#1068003
• Large Language Models in Healthcare and Medical Domain: A Review#340990
• Large Language Models in Healthcare and Medical Applications: A Review#376936
• Evaluating the use of large language models to provide clinical recommendations in the Emergency Department#328724
• Screening cognitive assessments (MMSE, CDT, MoCA) of eight large language models#16303