Evaluating artificial intelligence chatbot performance on board-level geriatrics questions.

Evaluating artificial intelligence chatbot performance on board-level geriatrics questions - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Sci Rep . 2026 Apr 11;16:17052. doi: 10.1038/s41598-026-47331-x Evaluating artificial intelligence chatbot performance on board-level geriatrics questions Mert Zure 1, ✉ , Metin Sökmen 2 Author information Article notes Copyright and License information 1 Department of Physical Medicine and Rehabilitation, Istanbul Kanuni Sultan Suleyman Research and Training Hospital, University of Health Sciences, Selimiye Mah. Tıbbiye Cad. No:38, Üsküdar, Istanbul, 34668 Turkey 2 Department of Geriatrics, Ankara University School of Medicine, Talatpaşa Blv. No:82, Altındağ, Ankara, 06230 Turkey ✉ Corresponding author. Received 2025 Oct 18; Accepted 2026 Mar 31; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ . PMC Copyright notice PMCID: PMC13230516  PMID: 41965928 Abstract Artificial intelligence (AI) language models are increasingly being explored as tools to support medical education and clinical care. Evaluating their performance on valid and reliable assessments such as board certification exams may provide insight into their potential integration into real-world medical settings. This study evaluated the accuracy, consistency, and difficulty assessment of four advanced AI models using board-level geriatrics questions. Four AI models—Grok-3, ChatGPT-4o, Microsoft Copilot, and Google Gemini 2.0 Flash—were tested on 300 text-based multiple-choice questions from the BoardVitals geriatrics certification question bank. The questions were equally divided into easy, medium, and hard categories. Each model was asked to classify the question’s difficulty and provide an answer twice. Model responses were evaluated for accuracy, consistency between attempts, quality of explanations, and alignment with the difficulty ratings predefined by BoardVitals. GPT-4o demonstrated the highest overall accuracy (85.3%), followed by Grok-3 (82.0%), Copilot (78.7%), and Gemini (74.0%). All models performed best on easy questions, and showed a decrease in accuracy as the difficulty increased ( p < 0.001). GPT-4o exhibited the highest consistency (96.3%), followed by Grok-3 (95.0%), Copilot (90.7%), and Gemini (81.3%). While their overall performance surpassed the average success rates of human users in the database, the agreement between model-assigned and reference difficulty ratings was moderate (mean κ = 0.41). GPT-4o received the highest mean quality score (4.68 ± 0.84), followed by Grok-3 (4.59 ± 0.98), Copilot (4.30 ± 1.07), and Gemini (3.88 ± 1.53). Advanced AI models demonstrate strong performance on geriatrics board-level content, suggesting potential applications as educational support tools. However, performance on multiple-choice examinations does not equate to clinical utility. Significant limitations include struggles with complex scenarios, difficulty in metacognitive assessment of question complexity, and variable explanation quality. These findings emphasize that AI integration into geriatric education and practice requires careful human oversight, explicit acknowledgment of limitations, and continued validation in diverse real-world contexts. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-026-47331-x. Keywords: Artificial intelligence, Geriatrics, Board exams, Medical education, Clinical reasoning Subject terms: Computational biology and bioinformatics, Health care, Mathematics and computing, Medical research Introduction Artificial intelligence (AI) is revolutionizing countless industries, and healthcare stands as one of the most promising frontiers 1 . Advanced AI language models such as Grok, ChatGPT, Microsoft Copilot, and Google Gemini have demonstrated impressive capabilities, particularly in processing complex information and handling specific tasks. In medicine, where precision and up-to-date knowledge are critical, these AI systems are increasingly viewed as tools to support clinical practice 1 , 2 . Their potential to improve diagnostic accuracy, reduce administrative workloads, and enhance patient care makes their evaluation in challenging circumstances, such as medical board examinations, a priority 3 , 4 . Geriatrics, a field dedicated to the health of the older adults, faces unique challenges that AI could help address, especially because older persons often present with complex illnesses that require specialized care. AI’s capacity to evaluate vast datasets can make a difference at this point, enabling clinicians to devote more energy to building trust and addressing the complex needs of their patients 5 , 6 . AI can also assist with patient communication and education that could empower seniors to better understand and manage their health, promote independence, and improve quality of life 6 , 7 . In an area of expertise such as geriatrics, where empathy and communication are as vital as technical skills, such efficiency gains could prove transformative 8 . A critical issue looming over geriatrics is the demand surging with an aging global population, and the supply of geriatricians struggling to keep pace, which is threatening to strain healthcare systems for the older patients worldwide 9 – 11 . AI offers a potential lifeline, of course not as a replacement for human expertise, but as a force multiplier that could extend the reach and impact of existing clinicians 6 , 8 . Additionally, together with telemedicine platforms, AI-related digital health innovations can make healthcare more accessible, particularly for the older population in remote or underserved areas 12 . However, these innovations’ adoption demands scrutiny to ensure reliability, protect patient privacy, and address regulatory gaps that could shake the trust in these systems 13 . Board certification exams serve as a gold standard for validating medical expertise, and in geriatrics, they assess a clinician’s ability to handle the complexities of aging-related care 3 , 14 . These demanding tests, which frequently include detailed scenario-based questions, assess both foundational knowledge and practical judgment. For AI to prove its worth in this domain, it must demonstrate not just accuracy but also consistency across diverse topics and difficulty levels 4 , 15 . The performance of four frequently used AI language models in response to board-level geriatrics questions is investigated in this study to provide insight into AI’s capacity to assist geriatric education and practice by examining the models’ success rates and capability to identify question complexity and strengths across subtopics. The findings could pave the way for smarter integration of AI into geriatric healthcare, offering a glimpse into how these tools might eventually support this field. Methods Overview This study explored how four advanced artificial AI systems—Grok-3, ChatGPT-4o, Microsoft Copilot, and Google Gemini 2.0 Flash—handle the intellectual demands of geriatrics board-certification questions. The aim was to assess these models’ accuracy as well as their consistency and ability to distinguish the complexity of questions and clinical settings that relate to older patients healthcare over four weeks from March 28 to April 27, 2025. AI language models The AI language models used featured Grok-3, the latest engineered version by xAI in Mission Bay, California, USA, celebrated for its logical reasoning and tailored answers per individual needs 16 ; ChatGPT-4o, a conversational titan from OpenAI in San Francisco, California, USA, known for its superiority in accuracy when it comes to clinical decision making 3 , 17 ; Microsoft Copilot, a versatile assistant developed by Microsoft in Redmond, Washington, USA; and Google Gemini 2.0 Flash, a text-generation powerhouse from Google in Mountain View, California, USA. These models were chosen for their cutting-edge language comprehension and their potential applicability in educational and healthcare settings 16 – 19 . Each was accessed via secure, cloud-based interfaces, ensuring uniform testing conditions across all platforms. All AI models were accessed with their default settings to reflect typical user experience. Specifically: Grok-3 and ChatGPT-4o were used with default temperature settings (0.7), standard context windows (128 K tokens for GPT-4o, 32 K for Grok-3), and no custom system prompts beyond the standardized instruction. Microsoft Copilot and Google Gemini 2.0 Flash were similarly accessed with default configurations via their respective web interfaces. No model updates occurred during the 4-week testing period (confirmed by version timestamps), ensuring consistency across all evaluations. Question repository Three hundred questions were drawn from a curated pool named BoardVitals, an online medical education database that offers question banks mirroring the American Board of Internal Medicine (ABIM) Geriatrics Certification Exam blueprint. These questions, which were authored by geriatrics professors in practice and peer-reviewed for accuracy, included a wide variety of subjects, from simple memory exercises to complex case-based questions. These subjects included managing geriatric syndromes, gerontology, musculoskeletal, neurological, and psychiatric conditions in the older adults. Items requiring the interpretation of visual data (such as imaging scans, graphical charts, or radiographs) were purposefully excluded to maintain focus on textual reasoning and avoid confounding variables related to current limitations in AI image processing. According to the ABIM Geriatrics Certification Exam blueprint, approximately 5–10% of exam questions include visual components. While this exclusion limits the generalizability of our findings to the full scope of board certification examinations, it ensures standardized evaluation of pure text-based clinical reasoning, which constitutes the vast majority of board exam content. This resulted in a text-only dataset that avoided confounding variables related to current AI limitations in image processing. Testing protocol and analytical framework Three hundred example questions, a hundred categorized as easy, a hundred categorized as medium, another hundred categorized as hard by the BoardVitals system, were given to each AI model individually. A standardized instruction preceded each query: “The following question is a national board-level exam question for geriatricians. First, mark the difficulty level of the question as easy, medium, or hard and then provide the correct answer.”. To evaluate the accuracy of the first response, each AI model was then asked, “Are you sure about your assessment on the difficulty level of the question and the correct answer to the question?“. Responses were noted along with the degree of difficulty determined by BoardVitals and the accuracy percentage of examinees using the system, which is also provided by BoardVitals when each question is answered. The percentage of each AI model’s correct answers is measured as response accuracy. The percentage of correct answers provided by AI models when the question was asked again was determined as response reliability. The questions were categorized as easy, medium, and hard by AI models, the same as the BoardVitals database. Furthermore, examining the difficulty discernment by comparing each model’s assigned difficulty rating against the database’s pre-established benchmarks illuminated the difficulty judgment calibration of the models. Two experts working in geriatrics independently evaluated all 300 explanations from each chatbot (1,200 total evaluations per rater) using a 5-point scale. Inter-rater reliability was calculated using Cohen’s weighted kappa (quadratic weights) to account for the ordinal nature of the scale, with disagreements resolved through discussion. Initial agreement prior to discussion was κ = 0.83, indicating strong reliability. Final consensus scores were used for all analyses. The evaluation process took approximately 20–25 h total per evaluator (approximately 4–5 min per question including reading, scoring, and discussion of discrepancies). The scale was determined as follows: 1: Wrong answer and poor explanation, 2: Wrong answer but partially satisfactory explanation, 3: Wrong answer but the explanation is satisfactory or correct answer but the explanation is poor, 4: Correct answer and partially satistactory explanation 5: Correct answer and satisfactory explanation. A detailed rubric defining specific criteria for each score level is provided in Supplementary Appendix A . No ethical approval was required because only the AI-generated data were used, and the study didn’t include any human or animal subjects. Nonetheless, the study was carried out following the general ethical guidelines of data confidentiality and research integrity. Statistical analysis Response accuracy was calculated as the percentage of correct answers out of 300 questions for each AI model, with differences across models assessed using one-way analysis of variance (ANOVA) followed by Tukey’s post-hoc tests for pairwise comparisons, including effect size calculations (η 2 ). Response reliability was determined as the percentage of consistent answers between first and second attempts, with changes in performance evaluated using McNemar’s test. Accuracy by question difficulty (easy, medium, hard) was compared using chi-square tests, with effect sizes reported as Cramer’s V. Agreement between chatbot-assigned and BoardVitals difficulty ratings was quantified using Cohen’s kappa coefficient. Explanation quality scores, assigned by two expert geriatricians, were analyzed for inter-rater reliability using Cohen’s kappa, and mean scores were compared across models using Kruskal-Wallis tests due to non-normal distribution. Correlations between answer correctness, explanation quality, and BoardVitals user performance were assessed using Pearson’s correlation coefficient. A multivariate regression analysis examined the independent contributions of question difficulty, subspecialty, and BoardVitals user performance to AI accuracy. All analyses were conducted using R software (version 4.3.1), with a significance threshold of α = 0.05. Results Overall performance across chatbots The overall accuracy rates for the first attempt were 85.3% for GPT-4o, 82.0% for Grok-3, 78.7% for Copilot, and 74.0% for Gemini ( p < 0.001, η 2 = 0.055, representing a medium effect size; ANOVA with Tukey post-hoc tests) (Table 1 ). For the second attempt, the accuracy rates were 85.7% for GPT-4o, 83.3% for Grok-3, 77.3% for Copilot, and 78.0% for Gemini. Table 1. Overall accuracy rates of AI Chatbots on ABIM geriatrics questions. Chatbot First attempt (%) Second attempt (%) Change (%) GPT-4o 85.3 85.7 + 0.4 Grok-3 82.0 83.3 + 1.3 Copilot 78.7 77.3 -1.4 Gemini 74.0 78.0 + 4.0 Open in a new tab Note: ANOVA: F(3,1196) = 23.4, p < 0.001, η 2 = 0.055. Performance by question difficulty Questions were categorized as easy ( n = 100), medium ( n = 100), or hard ( n = 100) according to the BoardVitals database. As shown in Table 2 , all chatbots performed best on easy questions and worst on hard questions, with the performance gap widening as difficulty increased. The difference in performance across difficulty levels was statistically significant (χ²=24.3, p < 0.001). All chatbots performed better than the BoardVitals users in each difficulty level. Table 2. Accuracy rates by question difficulty level (%). Difficulty Gemini GPT-4o Copilot Grok-3 BVUSR Easy 88.1 96.7 91.3 95.9 81.3 Medium 75.6 86.4 79.7 82.2 73.2 Hard 59.9 74.1 66.6 69.1 58.8 Open in a new tab BVUSR: BoardVitals User Success Rate. Note: χ 2 = 24.3, p < 0.001, Cramer’s V = 0.14. Consistency between attempts Consistency was measured as the percentage of questions for which the chatbot provided the same answer (correct or incorrect) in both attempts. GPT-4o exhibited the highest consistency (96.3%), followed by Grok-3 (95.0%), Copilot (90.7%), and Gemini (81.3%) (Table 3 ). McNemar tests revealed that only Gemini showed a statistically significant change in performance between attempts ( p = 0.02). Table 3. Response consistency between first and second attempts. Chatbot Consistency (%) McNemar’s test p -value GPT-4o 96.3 0.69 Grok-3 95.0 0.22 Copilot 90.7 0.33 Gemini 81.3 0.02 Open in a new tab When an answer changed between attempts, Gemini most frequently corrected initially wrong answers (9.0% of all questions), compared to Copilot (6.7%), Grok-3 (3.3%), and GPT-4o (2.0%). Quality of explanations Explanations provided by each chatbot were evaluated on a 5-point scale (1: Wrong answer and poor explanation, 5: Correct answer and satisfactory explanation). GPT-4o received the highest mean quality score (4.68 ± 0.84), followed by Grok-3 (4.59 ± 0.98), Copilot (4.30 ± 1.07), and Gemini (3.88 ± 1.53) (Table 4 ). The inter-rater reliability between evaluators was strong (Cohen’s κ = 0.83). Table 4. Explanation quality scores. Quality score Gemini (%) GPT-4o (%) Copilot (%) Grok-3 (%) 1 (Poorest) 19.0 4.0 8.3 4.3 2 9.7 4.3 6.0 5.0 3 2.3 1.3 5.7 1.7 4 3.0 1.0 7.7 3.3 5 (Best) 66.0 89.4 72.3 85.7 Mean score 3.88 4.68 4.30 4.59 Open in a new tab Performance by subspecialty Performance varied across subspecialties within geriatrics. All chatbots performed well in Psychiatry (mean accuracy 93.8%) and Cardiovascular Diseases (91.5%), but struggled with Nephrology (68.0%) and Functional Assessment (69.5%). GPT-4o demonstrated the most consistent performance across subspecialties (coefficient of variation 9.3%) compared to other chatbots (Gemini: 17.8%, Copilot: 14.6%, Grok-3: 12.1%). Post-hoc analysis of difficulty misclassification Questions that AI models systematically underestimated (classified as “medium” when BoardVitals rated “hard”) shared several characteristics: (1) interdisciplinary complexity requiring integration of multiple geriatric domains (63% of misclassified questions), (2) ethical or psychosocial considerations beyond purely medical knowledge (41%), (3) questions requiring recognition of atypical presentations in older adults (38%), and (4) scenarios involving medication management with multiple drug interactions (35%). These findings suggest AI models struggle most with contextual complexity and nuanced clinical judgment rather than factual knowledge gaps. Assessment of question difficulty by chatbots Chatbots were asked to classify each question as easy, medium, or hard. The agreement between chatbot assessments and BoardVitals difficulty ratings was moderate (mean κ = 0.41). Grok-3 showed the highest agreement (46.3%), followed by GPT-4o (44.0%), Copilot (41.0%), and Gemini (39.7%). Notably, all chatbots tended to underestimate the difficulty of questions categorized as “hard” by BoardVitals, classifying 42.5% of them as “medium” and 11.8% as “easy.” Multivariate analysis of performance predictors A multiple regression analysis was conducted to assess the independent contribution of question difficulty and subspecialty to AI performance while controlling for BoardVitals user success rates. The model explained 48% of variance in AI accuracy (R 2 = 0.48, F(5,294) = 54.2, p < 0.001). Question difficulty (β= -0.42, p < 0.001) and subspecialty category (β = 0.28, p < 0.001) were significant independent predictors, while BoardVitals user success rate contribution diminished when controlling for these factors (β = 0.19, p = 0.03), suggesting that human and AI performance share common difficulty-related variance. Correlation between answer correctness and explanation quality There was a strong positive correlation between the correctness of answers and the quality of explanations across all chatbots ( r = 0.79, p < 0.001). This correlation was strongest for Gemini ( r = 0.86) and weakest for GPT-4o ( r = 0.74), potentially reflecting GPT-4o’s ability to provide reasonable explanations even when arriving at incorrect answers. Relationship to BoardVitals user performance Questions that were more frequently answered correctly by BoardVitals users were also more likely to be answered correctly by AI chatbots ( r = 0.62, p < 0.001). This correlation was strongest for GPT-4o ( r = 0.71) and weakest for Gemini ( r = 0.54). Discussion The findings of this study emphasize the potential of advanced AI language models—Grok-3, ChatGPT-4o, Microsoft Copilot, and Google Gemini 2.0 Flash— as educational support tools in geriatric medicine, as demonstrated by their proficient performance on board-level geriatrics questions. GPT-4o achieved the highest overall accuracy (85.3% on the first attempt, 85.7% on the second), followed closely by Grok-3 (82.0% and 83.3%), with Copilot and Gemini trailing behind. These results align with prior studies evaluating AI performance in medical board examinations, such as the USMLE and rheumatology board questions, where models like ChatGPT demonstrated strong capabilities 3 , 4 , 20 . The analysis of performance across varying question difficulties demonstrated a consistent trend: all models performed well on easy questions (88.1–96.7% accuracy) but had difficulties with hard questions (59.9–74.1%), reflecting the performance patterns of human BoardVitals users. This difficulty-dependent performance highlights a limitation in current AI models, especially in addressing complicated or confusing clinical situations that require deep contextual comprehension or interdisciplinary expertise. AI systems are typically designed for specific tasks, often referred to as artificial narrow intelligence (ANI), where they perform exceptionally within constrained parameters but struggle outside of them 21 . Therefore, the integration of AI across various specializations requires collaboration among healthcare professionals who can interpret AI outputs effectively, further emphasizing the need for strong communication and training regarding AI applications within different disciplines 22 . Response consistency was high for GPT-4o and Grok-3, with minimal variation between attempts, while Gemini’s lower consistency and significant second-attempt improvement suggest benefits from repetitive processing. Gemini’s variability may challenge clinical reliability, but its ability to correct 9.0% of initial errors indicates potential for improvement with further training, though less reliable in high-stakes settings 3 , 18 , 19 . Interestingly, all models outperformed the average BoardVitals user success rates across all difficulty levels, suggesting that AI could serve as a valuable educational tool to support human learning, particularly for foundational and moderately complex topics 4 , 16 . Moreover, the quality of explanations from the models assessed by expert geriatricians, distinguishing GPT-4o and Grok-3, indicates that these models not only provide correct answers but also share coherent, medically relevant rationales, which are vital for educational purposes and building adaptive learning environments in clinical settings 6 , 7 . The moderate agreement between chatbot-assigned and BoardVitals difficulty ratings highlights a critical area for improvement: AI’s ability to accurately assess question complexity. The tendency to underestimate the difficulty of hard questions suggests that current models may struggle with metacognitive awareness of clinical complexity, a skill critical for prioritizing tasks in real-world healthcare settings 15 , 23 . This limitation may be linked to the models’ reliance on pattern recognition rather than a deeper understanding of geriatric-specific nuances, such as ethical considerations or psychosocial factors 24 . The integration of AI into geriatric healthcare raises significant concerns related to data privacy and regulatory compliance. AI models handling sensitive patient data must comply with tight regulations, including the Health Insurance Portability and Accountability Act (HIPAA) in the United States and the General Data Protection Regulation (GDPR) in Europe, to protect patient confidentiality 25 . Specific requirements include end-to-end encryption for data transmission and storage, de-identification protocols removing 18 HIPAA identifiers from training or operational data, comprehensive audit trails documenting data access and AI-generated recommendations, patient consent mechanisms addressing AI involvement in care, and data minimization principles limiting AI access to essential information only 25 , 26 . The cloud-based interfaces used in this study ensured secure data handling, but real-world applications involving patient data require strong encryption, anonymization, and audit trails to prevent breaches. Furthermore, regulatory frameworks for AI in healthcare remain fragmented, necessitating clear guidelines to ensure accountability, transparency, and ethical use. Addressing these challenges is critical to building trust among clinicians and patients, especially in geriatrics, where vulnerable populations are involved 26 . While these results suggest potential applications in medical education, it is critical to emphasize that performance on board-style questions does not equate to clinical utility in real-world patient care settings. Board examinations assess knowledge recall and clinical reasoning in controlled scenarios, whereas actual clinical practice involves time constraints, incomplete information, complex patient comorbidities, communication challenges, and ethical considerations that extend beyond the scope of multiple-choice questions. Therefore, AI models demonstrated here as educational support tools should not be interpreted as ready for independent clinical decision-making. The potential role of AI in “extending clinician reach” requires careful qualification. Rather than serving as autonomous decision-makers, current AI models may be best suited for support functions such as, providing initial differential diagnoses for clinician review, summarizing evidence-based guidelines, identifying relevant literature, and assisting with documentation and administrative tasks. All clinical decisions must ultimately remain under the supervision of qualified healthcare professionals. This study has several limitations. First, the exclusion of questions with visual data limits the generalizability of findings. Second, the reliance on a single question bank may introduce bias, as it may not fully represent the diversity of questions encountered in actual ABIM Geriatrics Certification Exams. Third, the study evaluated AI performance in a controlled, text-based setting, which does not account for the dynamic, patient-centered interactions required in clinical environments. Fourth, we did not provide AI models with baseline difficulty calibration examples prior to testing. This lack of standardization may have contributed to the systematic underestimation of “hard” questions, as models lacked reference examples to guide their difficulty assessment. Fifth, the rapidly changing nature of chatbot algorithms makes repeatability difficult because the models are always being updated and improved, which could eventually result in different outputs. Additionally, potential bias in AI models represents a critical ethical concern. AI systems can perpetuate biases in training data, manifesting as differential accuracy across racial, ethnic, or socioeconomic patient backgrounds; underrepresentation of population-specific geriatric syndromes; and gendered assumptions in clinical scenarios. Given the substantial heterogeneity within the older adult population, AI performance may vary significantly across different patient contexts. We did not systematically assess demographic representation in BoardVitals questions. Future studies should address these limitations through several key directions: incorporating multimodal AI capabilities including visual data interpretation; utilizing questions from multiple validated sources such as official ABIM practice exams to ensure broader representativeness; providing baseline difficulty calibration examples and comparing AI assessments to practicing geriatricians to clarify metacognitive limitations; evaluating AI performance stratified by patient demographics and auditing training datasets for bias, particularly critical given healthcare disparities facing minority older adults; measuring learner-focused outcomes including knowledge retention and diagnostic accuracy to validate educational utility; and assessing scalability in resource-limited settings by examining computational costs, internet connectivity requirements, offline functionality, language adaptation, and electronic health record integration. Real-world clinical simulations in diverse healthcare contexts, particularly low-resource environments serving vulnerable older populations, are essential to fully evaluate clinical applicability and ensure equitable implementation. Conclusion In conclusion, this study represents a comprehensive evaluation of modern AI language models in geriatrics, highlighting both their strengths and limitations. GPT-4o and Grok-3 demonstrated strong potential as educational support tools by providing accurate answers and high-quality explanations. However, significant challenges persist, including improving performance on complex topics, refining difficulty assessment, and addressing ethical concerns regarding privacy, bias, and regulatory compliance. The inconsistent quality of explanations, especially for incorrect answers, calls for careful integration into educational settings with human oversight. Clinically, AI’s role may be best suited for support tasks—offering initial assessments, differential diagnoses, or evidence summaries—while final decisions remain with healthcare professionals. Future studies should focus on geriatric-specific optimization, multimodal integration including visual data interpretation, and real-world validation in diverse healthcare settings, and explicit evaluation of learning outcomes and clinical impact. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 (15.4KB, docx) Supplementary Material 2 (46.5KB, pdf) Acknowledgements The authors acknowledge Enes Efe İş for scientific advice on study design and statistical analysis. Abbreviations AI Artificial intelligence ABIM American Board of Internal Medicine ANOVA Assessed using one-way analysis of variance ANI Artificial narrow intelligence HIPAA Health Insurance Portability and Accountability Act GDPR General Data Protection Regulation Author contributions MZ: Study concept and design, interpretation of data, statistical analysis, drafting of the manuscript, final approval of version to be published. MS: Study concept and design, critical revision of the article, and final approval of the version to be published. Funding The present study is supported by the authors only. Data availability The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.The dataset can not be made publicly available due to licensing restrictions of the BoardVitals question bank; however, the script is provided for methodological transparency and can be applied to equivalent datasets. Competing interests The authors declare no competing interests. Ethics approval No ethical approval was required because only the AI-generated data were used, and the study didn’t include any human or animal subjects. Nonetheless, the study was carried out following the general ethical guidelines of data confidentiality and research integrity. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References 1. Hirani, R. et al. Artificial intelligence and healthcare: a jurney through history, present innovations, and future possibilities. Life (Basel) . 14 (5), 557. 10.3390/life14050557 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Al Kuwaiti, A. et al. A review of the role of artificial intelligence in healthcare. J. Pers. Med. 13 (6), 951. 10.3390/jpm13060951 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 3. Is, E. E. & Menekseoglu, A. K. Response to: comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o: correspondence. Clin. Rheumatol. 43 (12), 4023–4024. 10.1007/s10067-024-07199-6 (2024). [ DOI ] [ PubMed ] [ Google Scholar ] 4. Gilson, A. et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9 , e45312. 10.2196/57594 (2023). [ DOI ] [ PMC free article ] [ PubMed ] 5. Pandit, S. G. & Walke, L. M. Geriatrics: mving forward with 2020 vision. J. Am. Geriatr. Soc. 69 (6), 1422–1428. 10.1111/jgs.17203 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 6. Padhan, S., Mohapatra, A., Ramasamy, S. K. & Agrawal, S. Artificial intelligence (AI) and robotics in elderly healthcare: enabling independence and quality of life. Cureus 15 (8), e42905. 10.7759/cureus.42905 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 7. Wang, J., Liang, Y., Cao, S., Cai, P. & Fan, Y. Application of artificial intelligence in geriatric care: bibliometric analysis. J. Med. Internet Res. 25 , e46014. 10.2196/46014 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 8. Kainiemi, E. et al. Perceived benefits of digital health and social services among older adults: a population-based cross-sectional survey. Digit. Health . 9 , 20552076231173559. 10.1177/20552076231173559 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 9. Lester, P. E., Dharmarajan, T. S. & Weinstein, E. The looming geriatrician shortage: ramifications and solutions. J. Aging Health . 32 (9), 1052–1062. 10.1177/0898264319879325 (2020). [ DOI ] [ PubMed ] [ Google Scholar ] 10. Yokoyama, K. et al. Early detection and remote care of frailty for older adults living in sparsely populated rural areas in Hokkaido, Japan: a case study. Cureus 17 (5), e84194. 10.7759/cureus.84194 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 11. Basu, M. et al. Updated inventory and projected requirements for specialist physicians in geriatrics. Can. Geriatr. J. 24 (3), 200–208. 10.5770/cgj.24.538 (2021). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Dorsey, E. R. & Topol, E. J. Telemedicine 2020 and the next decade. Lancet 395 (10227), 859. 10.1016/S0140-6736(20)30424-4 (2020). [ DOI ] [ PubMed ] [ Google Scholar ] 13. Khalid, N., Qayyum, A., Bilal, M., Al-Fuqaha, A. & Qadir, J. Privacy-preserving artificial intelligence in healthcare: techniques and applications. Comput. Biol. Med. 158 , 106848. 10.1016/j.compbiomed.2023.106848 (2023). [ DOI ] [ PubMed ] [ Google Scholar ] 14. Jo, A., Dai, M., Peterson, L. & Mainous, A. G. Trends of geriatric certification and practice patterns of family physicians: 1988–2019. Fam Med. 55 (2), 95–102. 10.22454/FamMed.2023.917437 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 15. Barbour, A. B. & Barbour, T. A. A radiation oncology board exam of ChatGPT. Cureus 15 (9), e44541. 10.7759/cureus.44541 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 16. Sau, S. et al. Accuracy and quality of ChatGPT-4o and Google Gemini performance on image-based neurosurgery board questions. Neurosurg. Rev. 48 (1), 320. 10.1007/s10143-025-03472-7 (2025). [ DOI ] [ PubMed ] [ Google Scholar ] 17. Tepe, M. & Emekli, E. Assessing the responses of large language models (ChatGPT-4, Gemini, and Microsoft Copilot) to frequently asked questions in breast imaging: a study on readability and accuracy. Cureus 16 (5), e59960. 10.7759/cureus.59960 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 18. Mandalos, A. & Tsouris, D. Artificial versus human intelligence in the diagnostic approach of ophthalmic case scenarios: a qualitative evaluation of performance and consistency. Cureus 16 (6), e62471. 10.7759/cureus.62471 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Gupta, R., Hamid, A. M., Jhaveri, M., Patel, N. & Suthar, P. P. Comparative evaluation of AI models such as ChatGPT 3.5, ChatGPT 4.0, and google Gemini in neuroradiology diagnostics. Cureus 16 (8), e67766. 10.7759/cureus.67766 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 20. Kung, T. H. et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit. Health . 2 (2), e0000198. 10.1371/journal.pdig.0000198 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 21. Chew, H. S. J. & Achananuparp, P. Perceptions and needs of artificial intelligence in health care to increase adoption: scoping review. J. Med. Internet Res. 24 (1), e32939. 10.2196/32939 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 22. Sharma, M. et al. Artificial intelligence applications in health care practice: scoping review. J. Med. Internet Res. 24 (10), e40238. 10.2196/40238 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 23. Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25 (1), 44–56. 10.1038/s41591-018-0300-7 (2019). [ DOI ] [ PubMed ] [ Google Scholar ] 24. Xie, Y. et al. Artificial intelligence for contextual well-being: protocol for an exploratory sequential mixed methods study with medical students as a social microcosm. PLoS One . 20 (5), e0321426. 10.1371/journal.pone.0321426 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Ahmed, A., Shahzad, A., Naseem, A., Ali, S. & Ahmad, I. Evaluating the effectiveness of data governance frameworks in ensuring security and privacy of healthcare data: a quantitative analysis of ISO standards, GDPR, and HIPAA in blockchain technology. PLoS One . 20 (5), e0324285. 10.1371/journal.pone.0324285 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 26. Lekadir, K. et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ 388 , e081554. 10.1136/bmj.r340 (2025). [ DOI ] [ PMC free article ] [ PubMed ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Supplementary Material 1 (15.4KB, docx) Supplementary Material 2 (46.5KB, pdf) Data Availability Statement The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.The dataset can not be made publicly available due to licensing restrictions of the BoardVitals question bank; however, the script is provided for methodological transparency and can be applied to equivalent datasets. Articles from Scientific Reports are provided here courtesy of Nature Publishing Group ACTIONS View on publisher site PDF (1.3 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Identifying Topics in Patient and Doctor Conversations Using Natural Language Processing Methods#1952
• Enhancing Oral Cancer Awareness#1173247
• Current patient attitudes to artificial intelligence applications in radiology.#220908
• Design Based Needs Analysis for AI-Powered Educator Course Creation: A Pilot Study#16054
• Implementing a conversation-starter tool to empower people with dementia and carers to have medicine conversations: a pilot and feasibility study.#678617