Using Generative AI to Appraise the Quality of Medical Education Research Studies: Agreement Between AI-Generated and Human MERSQI Scores.

Using Generative AI to Appraise the Quality of Medical Education Research Studies: Agreement Between AI‐Generated and Human MERSQI Scores - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice AEM Educ Train . 2026 May 14;10(3):e70189. doi: 10.1002/aet2.70189 Using Generative AI to Appraise the Quality of Medical Education Research Studies: Agreement Between AI‐Generated and Human MERSQI Scores Mohammad Aldalou 1 , William S Brooks 1 , Emily Rush 2, 3 , Bernard Landry‐Wegener 4 , Corey Bills 5 , David A Cook 6 , John Swope 7 , Adam B Wilson 3, ✉ Author information Article notes Copyright and License information 1 Department of Medical Education, Marnix E. Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, Alabama, USA 2 Office of Institutional Effectiveness, Academic Affairs, Rush University, Chicago, Illinois, USA 3 Department of Anatomy and Cell Biology, Rush Medical College, Rush University, Chicago, Illinois, USA 4 Department of Internal Medicine, Johns Hopkins School of Medicine, Baltimore, Maryland, USA 5 Department of Emergency Medicine, Anschutz School of Medicine, University of Colorado, Aurora, Colorado, USA 6 Department of General Internal Medicine, Mayo Clinic, Rochester, Minnesota, USA 7 Curricu.me, Londonderry, New Hampshire, USA * Correspondence: Adam B. Wilson ( adam_wilson@rush.edu ) ✉ Corresponding author. Revised 2026 Apr 30; Received 2026 Feb 3; Accepted 2026 May 6; Collection date 2026 Jun. © 2026 The Author(s). AEM Education and Training published by Wiley Periodicals LLC on behalf of Society for Academic Emergency Medicine. This is an open access article under the terms of the http://creativecommons.org/licenses/by/4.0/ License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. PMC Copyright notice PMCID: PMC13176095  PMID: 42146734 ABSTRACT Objectives The increasing volume of medical education research necessitates efficient, reliable, and scalable methods for conducting quality appraisals. The Medical Education Research Study Quality Instrument (MERSQI) is a widely used tool, although its manual scoring process remains resource‐intensive. This study evaluated how well large language models (LLMs) appraise medical education research using the MERSQI tool in comparison with human judges. Methods Three LLMs (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro) assigned MERSQI domain scores to 1423 medical education research articles. The authors compared AI‐generated scores with human‐generated scores using intraclass correlation coefficients (ICCs) across the six MERSQI domains. They evaluated the agreement between AI‐ and human‐generated MERSQI composite scores using Bland–Altman plots. Results Domain‐level ICC values ranged from fair (0.24) to near perfect (0.81), with the lowest agreement observed in the ‘sampling,’ ‘validity evidence,’ and ‘data analysis’ domains. No single LLM consistently outperformed the others across all domains. Composite score agreement with human ratings was substantial and similar across LLMs (ICC range: 0.65–0.69). GPT‐5 produced slightly lower composite scores than humans, while Claude Sonnet 4 and Gemini 2.5 Pro produced higher scores, with Gemini showing the largest deviation. The Bland–Altman plots for Gemini 2.5 Pro suggested proportional bias, indicating its agreement with human scores varied across the range of study quality. Conclusions These LLMs demonstrated substantial agreement with human raters for MERSQI composite scores, but domain‐level agreement varied. Systematic differences in scoring patterns highlight the need for human oversight and additional calibrations before integrating LLMs into systematic review appraisal workflows. Keywords: AI‐assisted research, artificial intelligence, medical education, MERSQI 1. Introduction Rigorous quality appraisals are an important aspect of systematic reviews, as they help identify low‐quality studies and justify the validity of synthesized conclusions [ 1 , 2 ]. Quality appraisals systematically examine the methodological quality and trustworthiness of empirical research by assessing its relevance to a specific review question or context and verifying whether a study's design, sampling, data collection, analysis, and reporting are appropriate and transparent [ 3 ]. Often conducted in duplicate, quality appraisals of empirical studies are labor‐intensive as they demand considerable human capital, time, and expertise to meticulously evaluate each study against predefined quality criteria [ 4 ]. As the volume of research literature expands, and with it the number and size of literature reviews, such demands could hinder the timely reporting of synthesized evidence or dissuade investigators from conducting quality appraisals altogether. These burgeoning constraints, coupled with incidental human discrepancies, errors, and oversights, underscore the need for more efficient, consistent, and feasible methodologies. Artificial intelligence (AI) has shown promise in automating complex tasks and supporting large‐scale data processing. Over the past decade, researchers have increasingly explored the use of machine learning to automate quality appraisal processes across various domains, including evidence‐based medicine and education. Existing tools to appraise research quality include RobotReviewer for randomized controlled trials [ 5 ] and SciScore for life sciences research [ 6 ]. Similar AI methods have also been used to predict expert ratings in national research appraisals such as the United Kingdom's Research Excellence Framework [ 5 , 7 ]. Advancements in both domain‐agnostic and specialized GenAI tools may offer greater rigor for quality appraisals by increasing objectivity and consistency in fields where the volume of research output overwhelms traditional quality appraisal workflows [ 1 ]. While early applications are encouraging, the use of GenAI for appraising research quality remains in its infancy and requires further validation [ 8 , 9 ]. A body of literature in clinical medicine suggests that GenAI has the potential to critically appraise study methods; however, evidence for this specific use case is limited [ 10 , 11 , 12 ]. For example, only a small proportion of software tools for supporting the automation of systematic review processes have been tested for risk of bias assessment, and most were applied to specific study designs, such as randomized controlled trials [ 10 ]. Not all tools were compared against human evaluators, and they generally needed further validation using larger datasets across a wider range of topics [ 10 ]. In a related review of software systems aimed at enhancing the efficiency of systematic reviews, Affengruber et al. found that RobotReviewer was the only tool evaluated as a critical appraisal system, highlighting a lack of performance data for other critical appraisal systems [ 11 ]. These findings underscore a gap in validating newer GenAI tools for appraising research quality beyond clinical trials. This gap is particularly relevant to medical education research, where study designs are often heterogeneous (e.g., qualitative studies, quasi‐experimental studies, and curriculum evaluations) and constituent disciplines may compound variability further. For example, Emergency medicine education includes research on simulation‐based procedural training, point‐of‐care ultrasound curricula, residency assessment, and team‐based clinical decision‐making, often with variable outcome measures and inconsistent reporting standards [ 13 , 14 , 15 ]. Systematic reviews synthesizing this scholarship must navigate this complexity, as outcomes may influence training program designs, accreditation standards, clinical workflows, and the quality of patient care delivered by graduates [ 16 ]. This inherent variability makes medical education both a useful test case and a demanding setting for investigating whether newer GenAI systems can apply established appraisal criteria across broad, methodologically diverse literature. The Medical Education Research Study Quality Instrument (MERSQI) provides an established mechanism for assessing the methodological quality of empirical quantitative studies. Prior studies have reported good to excellent interrater reliability for MERSQI scores and have shown associations between higher scores and markers of scholarly quality, including research funding and editorial acceptance decisions [ 17 , 18 , 19 , 20 , 21 ]. This study examined how well MERSQI scores derived from three commercial LLMs (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro) agreed with human‐generated scores at the composite and individual domain score levels and whether systematic scoring differences existed between AI and human raters. Understanding GenAI's ability to apply the MERSQI tool may support its use as a second coder or to automate certain research tasks under human supervision. This, in turn, may enhance the consistency, scalability, and routine implementation of quality appraisals in medical education research syntheses. 2. Methods 2.1. Medical Education Research Study Quality Instrument The Medical Education Research Study Quality Instrument (MERSQI) is a tool for appraising the methodological quality of medical education research studies. It evaluates six domains: study design, sampling, type of data, validity evidence, data analysis, and outcomes. Each domain consists of specific criteria, with scores ranging from 0 to 3 or 1 to 3, depending on the category. A composite MERSQI score is derived by summing the scores across the domains, with a maximum score of 18. In practice, it is recommended and more meaningful to report and interpret MERSQI domain scores over composite scores to better gauge a study's methodological strengths and limitations. Higher domain or composite scores indicate stronger methodological quality in medical education research [ 17 ]. For terms and technologies referenced throughout this manuscript, see Table 1 . TABLE 1. Key terms and technologies. Term Description AI Artificial intelligence; computational systems designed to perform tasks that typically require human cognition, such as pattern recognition, prediction, classification, reasoning, or language generation. GenAI Generative artificial intelligence; AI systems that generate new content, such as text, data, code, images, or summaries, in response to user prompts. LLM Large language model; a type of generative AI trained on large text corpora to process, interpret, and generate human language. GPT‐5 A large language model developed by OpenAI and evaluated in this study as one of the commercial LLMs used to generate MERSQI scores. Claude Sonnet 4 A large language model developed by Anthropic and evaluated in this study as one of the commercial LLMs used to generate MERSQI scores. Gemini 2.5 Pro A large language model developed by Google and evaluated in this study as one of the commercial LLMs used to generate MERSQI scores. MERSQI Medical Education Research Study Quality Instrument; a six‐domain instrument used to appraise the methodological quality of empirical quantitative medical education research. API Application programming interface; a software interface that enables programmatic interaction with digital systems, including AI platforms. JSON JavaScript Object Notation; a structured, machine‐readable format commonly used to organize and return data from software systems or AI outputs. Open in a new tab 2.2. Data Collection and Eligibility Criteria This study compares the accuracy of AI‐generated MERSQI scores to human‐generated scores from a convenience sample of previously published systematic reviews. The dataset collected MERSQI scores from original published studies, or if data were unpublished, directly from the original authors. Both approaches, detailed below, were undertaken in parallel until the target sample size of at least 1000 MERSQI‐scored studies was achieved. According to a power analysis (conducted in R [ 22 ] using the “ICC.Sample.Size” package), to detect an intraclass correlation coefficient (ICC) of 0.20 (slight agreement) versus 0.30 (fair agreement) at a power of 0.90, a sample size of at least 923 MERSQI‐scored studies was required ( k = 2 ratings, 2‐sided, alpha = 0.05). 2.2.1. Approach 1—Unpublished Datasets A search was conducted through MedEdMentor using the keyword “MERSQI” to identify review articles where authors applied the full MERSQI tool to appraise at least 20 education‐focused studies published in clinical or medical education journals. For reviews that did not publish the individual study‐level MERSQI scores, we contacted the corresponding authors, asking them to provide the study‐level scores. 2.2.2. Approach 2—Published Datasets We identified additional datasets through PubMed using the keyword “MERSQI” to find reviews that published their study‐level MERSQI data (e.g., in the main text or as supplemental appendices). Starting with the most recently published reviews, we examined each published study and selected for inclusion those with publicly available data until the target sample size was achieved. 2.2.3. Eligibility Criteria We included studies that addressed any medical education topic. No date or year restrictions were applied. Works in related healthcare disciplines/journals (e.g., dental education, nursing) were excluded. We included only reviews with study‐level human‐generated MERSQI scores across all six domains that used original (unmodified) MERSQI scoring definitions. If an original research study was included in more than one review, scores from the first published review were used. The present study did not require ethical approval as it did not involve human subjects. 2.3. LLMs and Prompting Procedure Three commercial LLMs were selected for evaluation: GPT‐5 (OpenAI, San Francisco, CA; openai.com ), Claude Sonnet 4 (Anthropic, San Francisco, CA; anthropic.com ), and Gemini 2.5 Pro (Google DeepMind, Mountain View, CA; deepmind.google.com ). These represented the most advanced available model from each of the three most widely used commercial AI platforms at the time of this study. Each was prompted to appraise the methodological quality of medical education research studies, with human‐generated MERSQI scores serving as the reference standard. We obtained the full text PDF of each included study and converted them to machine‐readable text using PyMuPDF, pdfplumber, or OCR (pytesseract), in that order, until acceptable text was returned. We then used Jupyter Notebook and application programming interfaces (APIs) to upload each study text to each LLM. For GPT‐5, the reasoning effort was set to “minimal,” the verbosity to “low,” and all other settings were left at their default values. For Claude Sonnet 4 and Gemini 2.5 Pro, the temperature was set to 0.5 (i.e., moderately low randomness) to ensure more focused and consistent outputs. Other configurations were left at their default values. We crafted a prompt that requested structured JavaScript Object Notation (JSON) outputs, including a one‐sentence article summary, MERSQI domain scores, justifications, and other metadata. The full prompting protocol and JSON schema are available as supplemental material accompanying the online article (Data S1 ). We used the same zero‐shot prompt for all three LLMs to establish their baseline performance without prompt optimization or refinements. Results were aggregated into a pandas DataFrame, where domain‐level scores were summed to obtain a total composite score and then exported to timestamped Excel files for further processing. This automated approach enabled efficient, reproducible, and scalable quality appraisal of a large corpus of medical education research articles. 2.4. Statistical Analysis The primary outcome was the degree of agreement between each LLM and human‐generated MERSQI scores across composite and individual domain score levels. The secondary outcome was the presence and magnitude of systematic scoring differences between AI and human raters. All statistical analyses were conducted in SPSS (version 29, IBM Corp., Armonk, NY). Domain‐ and composite‐level agreement between each LLM and human raters was assessed using a two‐way random effects intraclass correlation coefficient (ICC [ 1 , 2 ]) for absolute agreement. Because no externally validated “true” MERSQI score exists independently of human judgment, human ratings served as the reference standard, and AI‐human agreement is interpreted throughout as a proxy measure of AI scoring accuracy. Agreement was interpreted using criteria by Landis and Koch with fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect (> 0.80) agreement [ 23 ]. Systematic differences in mean MERSQI scores between each LLM and human raters were examined using repeated measures ANOVA. Effect sizes were measured using partial eta‐squared ( η 2 ) and were interpreted as small (0.01–0.05), medium (0.06–0.13), or large (≥ 0.14) effects [ 24 ]. Alpha was set at 0.05, and outcomes are reported as means ( μ ) and standard deviations (SD). Bland–Altman plots were used to visually assess the agreement between AI‐generated and human MERSQI composite scores. A linear regression of the score differences on the means tested for the presence of proportional bias (i.e., whether the measurement differences varied with score averages) [ 25 ]. A significant result ( p < 0.050) for proportional bias indicates that the methods being compared do not agree equally across the range of values. 3. Results 3.1. Data Collection Outcomes and Summary of Study Features 3.1.1. Data From Unpublished Datasets The MedEdMentor search yielded 25 review articles that met the inclusion criteria. The corresponding authors of these 25 articles were invited to share their article‐level MERSQI data underlying their published reviews. Three corresponding authors responded with their unpublished datasets, contributing 1334 individually MERSQI‐scored articles in total. 3.1.2. Data From Published Datasets The PubMed search returned 174 unique records, which underwent screenings and full‐text reviews in the sequence of the returned records. Of these, three published reviews, comprising a total of 89 MERSQI‐scored studies, were added to the dataset. Further efforts to acquire additional published datasets were discontinued upon receiving one large dataset of 1000+ MERSQI‐scored studies from one corresponding author [ 26 , 27 ]. The final dataset for analysis consisted of 1423 MERSQI‐scored articles, compiled by combining the unpublished and published datasets. The MERSQI‐scored studies were published between 1969 and 2021. Human MERSQI scores were generated by research teams from contributing systematic reviews. Scores were derived either by a single individual (3.0%; 43/1423) or through consensus after blinded independent reviews by two reviewers (97.0%; 1380/1423). However, detailed characteristics of individual raters, including their level of formal MERSQI training and years of experience, were not reported across the contributing reviews and could not be extracted or presented. The sources of the MERSQI‐appraised studies and score derivation information for all 1423 articles are presented as supplemental material accompanying the online article (Table S1 ). 3.2. Domain Score Agreement and Analysis Inter‐rater agreement between the LLMs and human raters was conducted across 1423 articles for each of the six MERSQI domains and composite scores. Domain‐level ICC values ranged from fair (ICC = 0.24; 95% CI = −0.02 to 0.44) to near‐perfect agreement (ICC = 0.81; 95% CI = 0.76 to 0.84; Figure 1 ). No single LLM outperformed the others across all domains (Figure 1 ). ICC values were generally the lowest for the sampling, validity evidence, and data analysis domains. Figure 2 presents AI versus human mean domain scores. On average, humans tended to assign higher scores to the sampling domain ( p < 0.001; η 2 ≥ 0.14, large effect) and lower scores to the validity evidence domain ( p < 0.001; η 2 ≥ 0.61, large effect) compared to the LLMs (Figure 2 ). Effect sizes for differences between human and LLM scores among the remaining four domains were medium to small ( η 2 ≤ 0.08). FIGURE 1. Open in a new tab Intraclass correlation coefficient (ICC) values comparing agreement between the large language models (LLM) and human raters across the Medical Education Research Study Quality Instrument (MERSQI) composite and domain scores. Lines indicate GPT‐5 (green circles, solid), Claude Sonnet 4 (orange triangels, dotted), and Gemini 2.5 Pro (blue squares, dashed). Higher ICC values reflect stronger agreement between the LLM model and human scoring. FIGURE 2. Open in a new tab Mean Medical Education Research Study Quality Instrument (MERSQI) domain scores for the large language models (LLMs) and human raters. Lines represent GPT‐5 (green circles, solid), Claude Sonnet 4 (orange triangles, dotted), Gemini 2.5 Pro (blues squares, dashed), and human raters (gray diamonds, dashed). 3.3. Composite Score Agreement and Analysis Composite score agreement with human values was substantial and similar across the three LLMs (ICC for GPT‐5 = 0.69 [95% CI = 0.66 to 0.72]; Claude Sonnet 4 = 0.68 [95% CI = 0.60 to 0.74]; Gemini 2.5 Pro = 0.65 [95% CI = 0.35 to 0.79]; Figure 1 ). Bland–Altman plots for composite scores are presented in Figure 3A–C . The negative mean below 0.00 for GPT‐5 (Figure 3A ) indicates this model ( μ GPT‐5 = 11.65, SD = 2.02) returned modestly lower ( p = 0.002; η 2 = 0.01, small effect) composite scores than human raters ( μ Human = 11.79, SD = 2.00). Claude Sonnet 4 (Figure 3B ) and Gemini 2.5 Pro (Figure 3C ) returned slightly higher ( p < 0.001) composite scores ( μ Claude Sonnet 4 = 12.32, SD = 2.02, η 2 = 0.12, medium effect; μ Gemini 2.5 Pro = 12.72, SD = 1.75, η 2 = 0.31, large effect) than humans, overall. The statistical test for proportional bias was nonsignificant for GPT‐5 ( p = 0.566) and Claude Sonnet 4 ( p = 0.653). FIGURE 3. Open in a new tab (A–C) Bland–Altman plots comparing composite Medical Education Research Study Quality Instrument (MERSQI) scores assigned by each large language model (LLM) and human raters. The x ‐axis shows the mean of LLMs and human scores. The y ‐axis shows the score difference (LLM model minus human). The solid red line represents the mean difference, the gray dashed lines indicate the 95% limits of agreement, and the black dotted line represents a trend line. A downward‐sloping trend line indicates that the LLM assigned higher scores than human raters at lower mean MERSQI values and lower scores at higher mean values, consistent with proportional bias. (A) GPT‐5 versus human raters: mean difference = −0.14; proportional bias test p = 0.566. (B) Claude Sonnet 4 versus human raters: mean difference = +0.53; proportional bias test p = 0.653. (C). Gemini 2.5 Pro versus human raters: mean difference = +0.93; proportional bias test p < 0.001. The trend line for Gemini 2.5 Pro (Figure 3C ) had a downward slope. The model assigned higher MERSQI scores than humans, mainly among studies with lower human MERSQI scores. This difference decreased as human MERSQI scores increased. The statistical test for proportional bias was significant for Gemini 2.5 Pro ( p < 0.001), indicating substantial variation in agreement with human‐generated composite scores across the range of values. 4. Discussion We evaluated the use of LLMs to appraise the quality of medical education research studies using the MERSQI tool. Among the three LLMs evaluated (GPT‐5, Claude Sonnet 4, and Gemini 2.5 Pro), no single model outperformed another across all six MERSQI domains. The highest score agreement was observed among all three LLMs and humans when evaluating the domains of study design and type of data. The LLMs were less accurate in appraising the domains of sampling, validity evidence, and data analysis. All three LLMs showed substantial and similar agreement with human composite scores. GPT‐5 tended to assign slightly lower scores than humans, whereas Claude Sonnet 4 and Gemini 2.5 Pro assigned higher composite scores. Notably, Gemini 2.5 Pro's scoring varied depending on study quality, whereby it rated lower‐quality studies more favorably and higher‐quality studies less favorably than humans, suggesting that Gemini 2.5 Pro may be more lenient when uncertain. 4.1. Comparisons With Prior Works Within the broader context of using LLMs to support research and scholarly work, studies have demonstrated varying degrees of success, with some showing significant time savings and others highlighting the need for careful validation of AI‐driven decisions [ 28 , 29 , 30 ]. Consistent with our findings, RobotReviewer, a critical appraisal tool for classifying the risk of bias in clinical trials, has also demonstrated moderate to fair agreement with human raters (Cohen's κ ≤ 0.60), with large variability between domains [ 31 , 32 ]. Likewise, evaluations of earlier ChatGPT models applying the Cochrane Risk of Bias 2.0 framework yielded weak results, with fair overall agreement (Cohen's κ = 0.24) and domain‐level agreement ranging from fair to slight [ 33 , 34 ]. 4.2. Practical Implications Our findings suggest that current LLMs cannot independently perform accurate quality appraisals across all domains of interest and are best suited as assistive or pre‐screening aids under human supervision. Our collective findings also underscore the value in using domain‐level MERSQI scores over composite scores in isolation. The use of AI‐generated composite scores alone may mask the effects of compensatory scoring, whereby overinflated scores in one domain may cancel out underinflated scores in another domain. Hence, composite MERSQI scores from a given LLM may closely approximate human composite scores, despite discrepancies at the domain level. It is probable that iterative prompt revisions could lead to improved appraisals. This is an area of ongoing investigation by our study team. For example, it may be helpful to revise the prompt to more explicitly address domains with low agreement. Such revisions could include examples and nonexamples, more detailed instructions or scoring rubrics, and chain‐of‐thought reasoning that decomposes tasks into sub‐tasks (e.g., to analyze validity using a separate prompt). A prompting protocol might also require the LLM to analyze the same text two or three times. While this approach would increase the overall cost, this investment may be justifiable if performance substantially improves. While our findings suggest that fully automated independent appraisals of medical education research quality by current LLMs cannot fully replace human appraisals, there may be practical use cases that leverage human‐AI collaborations to maximize research efficiencies. For example, LLMs could serve as a second reviewer for domains where stronger alignment with human raters was observed (e.g., study design, type of data), thereby offsetting the time commitment required of a second human rater to review the domains where LLMs underperformed. While not ideal, this division of labor reduces the overall cost and time required to complete quality appraisals [ 35 , 36 , 37 ]. The value of such an arrangement is amplified for clinician‐educators whose protected research time is limited. In emergency medicine, where systematic reviews are commonly performed by small academic teams balancing clinical, educational, and scholarly responsibilities, a partial reduction in duplicate‐review burden could meaningfully expand what these teams are able to synthesize and how often [ 38 ]. 4.3. Limitations Several limitations should be acknowledged. The present findings are unlikely to generalize to other LLMs (e.g., Perplexity, Llama) or future GenAI models. Further investigation is needed to understand the mechanisms underlying the discrepancies between humans and LLMs, and to test whether fine‐tuning or targeted calibration of LLMs could improve domain‐level accuracy [ 39 , 40 ]. Medical education research encompasses a diverse range of reporting standards and study designs, including qualitative studies, quasi‐experimental research, and curriculum evaluations, among others. Herein, studies were not classified by design type and, therefore, it is unknown whether LLM scoring varies across different study designs. It is unclear whether LLMs' abilities to analyze multimodal inputs (e.g., figures, diagrams, tables) would have improved or diminished appraisal accuracy in this study. Visual elements often encapsulate details that are not fully conveyed in the text itself. As such, an LLM's incorrect interpretation of visual elements could systematically disadvantage studies that rely heavily on visual data [ 41 , 42 ]. Although nearly all studies (97.0%; 1380/1423) included in this dataset were independently appraised by two human reviewers, variability in judgment or operational definitions may have influenced the scoring outcomes. Broader validation against larger expert panels or crowdsourced groups (e.g., > 1000 raters) would provide stronger benchmarks for evaluating human‐AI agreement [ 43 , 44 , 45 , 46 , 47 ]. Similarly, exploring intra‐model variability using a test–retest design is warranted. 4.4. Future Directions Though the MERSQI tool is widely used in health professions education, the literature would benefit from investigating LLM outcomes using other appraisal tools (e.g., GRADE tool). Furthermore, given the MERSQI tool is best suited for quantitative research, future studies could extend this work by evaluating the application of LLMs to appraise qualitative research using tools such as Popay et al.'s framework [ 48 ]. GenAI's evolving ability to execute multimodal document parsing also warrants separate investigations of accuracy and reliability. Lastly, the present study's design did not permit a critical investigation of diagnostic accuracy (i.e., sensitivity, specificity, and area under the ROC curve) related to rejecting or accepting manuscripts for publication based on their AI‐generated MERSQI scores. Investigating this information in future works could have implications for medical education journals exploring AI‐assisted screening mechanisms for managing manuscript quality control. 5. Conclusions No single LLM (GPT‐5, Claude Sonnet 4, or Gemini 2.5 Pro) consistently outperformed the others across all six MERSQI domains. The LLMs showed moderate or fair agreement with human raters in three to four domains, indicating that human oversight is still essential for most aspects of quality appraisal. These findings encourage the medical education research community to thoughtfully consider the delicate balance between leveraging LLMs as research assistants and maintaining the essential role of human expertise. Author Contributions David A. Cook: conceptualization, investigation, writing – original draft, data curation, methodology. Adam B. Wilson: conceptualization, writing – original draft, writing – review and editing, investigation, methodology, supervision, formal analysis, visualization. Emily Rush: conceptualization, investigation, writing – original draft, writing – review and editing. Bernard Landry‐Wegener: conceptualization, data curation, investigation, writing – review and editing. John Swope: software, data curation, writing – review and editing. Mohammad Aldalou: conceptualization, investigation, writing – original draft, methodology, writing – review and editing, data curation, resources. William S. Brooks: conceptualization, investigation, writing – original draft, writing – review and editing, project administration. Corey Bills: conceptualization, investigation, writing – review and editing, data curation. Funding The authors have nothing to report. Conflicts of Interest The authors declare no conflicts of interest. Supporting information Table S1: Database. AET2-10-e70189-s002.xlsx (180.7KB, xlsx) Data S1: Methods supplement: prompt text. AET2-10-e70189-s003.pdf (149KB, pdf) Data S2: Supplementary index. AET2-10-e70189-s001.pdf (64.2KB, pdf) Acknowledgments The authors have nothing to report. Aldalou M., Brooks W. S., Rush E., et al., “Using Generative AI to Appraise the Quality of Medical Education Research Studies: Agreement Between AI‐Generated and Human MERSQI Scores,” AEM Education and Training 10, no. 3 (2026): e70189, 10.1002/aet2.70189. Supervising Editor: Daniel Egan Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request. References 1. Shaheen N., Shaheen A., Ramadan A., et al., “Appraising Systematic Reviews: A Comprehensive Guide to Ensuring Validity and Reliability,” Frontiers in Research Metrics and Analytics 8 (2023): 1268045. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Shea B. J., Reeves B. C., Wells G., et al., “AMSTAR 2: A Critical Appraisal Tool for Systematic Reviews That Include Randomised or Non‐Randomised Studies of Healthcare Interventions, or Both,” BMJ 358 (2017): j4008. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 3. Al‐Jundi A., “Critical Appraisal of Clinical Research,” Journal of Clinical and Diagnostic Research 11 (2017): JE01–JE05. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Flores‐Cohaila J. A., Garcia‐Portocarrero P., Saldaña‐Amaya D. A., Miranda‐Chavez B., and Copaja‐Corzo C., “Leveraging Evaluation of Quality on Medical Education Research With ChatGPT,” Medical Teacher 47, no. 3 (2025): 401–403. [ DOI ] [ PubMed ] [ Google Scholar ] 5. Marshall I. J., Kuiper J., and Wallace B. C., “RobotReviewer: Evaluation of a System for Automatically Assessing Bias in Clinical Trials,” Journal of the American Medical Informatics Association 23, no. 1 (2016): 193–201. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 6. Roelandse M., Ozyurt I. B., Evanko D., and Bandrowski A., “Assessing the Effectiveness of SciScore in Supporting the Reproducibility of Scientific Research,” Science Editor 46 (2023): 46–52. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 7. Šuster S., Baldwin T., Lau J. H., et al., “Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study,” Journal of Medical Internet Research 25 (2023): e35568. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 8. Mahuli S. A., Rai A., Mahuli A. V., and Kumar A., “Application ChatGPT in Conducting Systematic Reviews and Meta‐Analyses,” British Dental Journal 235, no. 2 (2023): 90–92. [ DOI ] [ PubMed ] [ Google Scholar ] 9. Ge L., Agrawal R., Singer M., et al., “Leveraging Artificial Intelligence to Enhance Systematic Reviews in Health Research: Advanced Tools and Challenges,” Systematic Reviews 13, no. 1 (2024): 269. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 10. Khalil H., Ameen D., and Zarnegar A., “Tools to Support the Automation of Systematic Reviews: A Scoping Review,” Journal of Clinical Epidemiology 144 (2022): 22–42. [ DOI ] [ PubMed ] [ Google Scholar ] 11. Affengruber L., van der Maten M. M., Spiero I., et al., “An Exploration of Available Methods and Tools to Improve the Efficiency of Systematic Review Production: A Scoping Review,” BMC Medical Research Methodology 24, no. 1 (2024): 210. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Motzfeldt Jensen M., Brix Danielsen M., Riis J., et al., “ChatGPT‐4o Can Serve as the Second Rater for Data Extraction in Systematic Reviews,” PLoS One 20, no. 1 (2025): e0313401. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Hildreth A. F., Maggio L. A., Iteen A., Wojahn A. L., Cook D. A., and Battista A., “Technology‐Enhanced Simulation in Emergency Medicine: Updated Systematic Review and Meta‐Analysis 1991–2021,” AEM Education and Training 7, no. 2 (2023): e10848. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 14. Colmers‐Gray I. N., Walsh K., and Chan T. M., “Assessment of Emergency Medicine Residents: A Systematic Review,” Canadian Medical Education Journal 8, no. 1 (2017): e106–e122. [ PMC free article ] [ PubMed ] [ Google Scholar ] 15. Weile J., Nebsbjerg M. A., Ovesen S. H., Paltved C., and Ingeman M. L., “Simulation‐Based Team Training in Time‐Critical Clinical Presentations in Emergency Medicine and Critical Care: A Review of the Literature,” Advances in Simulation 6, no. 1 (2021): 3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 16. van der Leeuw R. M., Lombarts K. M., Arah O. A., and Heineman M. J., “A Systematic Review of the Effects of Residency Training on Patient Outcomes,” BMC Medicine 10, no. 1 (2012): 65. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. Reed D. A., Cook D. A., Beckman T. J., Levine R. B., Kern D. E., and Wright S. M., “Association Between Funding and Quality of Published Medical Education Research,” JAMA 298, no. 9 (2007): 1002. [ DOI ] [ PubMed ] [ Google Scholar ] 18. Stephenson C. R., Vaa B. E., Wang A. T., et al., “Conference Presentation to Publication: A Retrospective Study Evaluating Quality of Abstracts and Journal Articles in Medical Education Research,” BMC Medical Education 17, no. 1 (2017): 193. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Reed D. A., Beckman T. J., Wright S. M., Levine R. B., Kern D. E., and Cook D. A., “Predictive Validity Evidence for Medical Education Research Study Quality Instrument Scores: Quality of Submissions to JGIM's Medical Education Special Issue,” Journal of General Internal Medicine 23, no. 7 (2008): 903–907. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 20. Smith R. P. and Learman L. A., “A Plea for MERSQI,” Obstetrics & Gynecology 130, no. 4 (2017): 686–690. [ DOI ] [ PubMed ] [ Google Scholar ] 21. Gorbanev I., Agudelo‐Londoño S., González R. A., et al., “A Systematic Review of Serious Games in Medical Education: Quality of Evidence and Pedagogical Strategy,” Medical Education Online 23, no. 1 (2018): 1438718. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 22. R Core Team , R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2020). [ Google Scholar ] 23. Landis J. R. and Koch G. G., “The Measurement of Observer Agreement for Categorical Data,” Biometrics 33, no. 1 (1977): 159. [ PubMed ] [ Google Scholar ] 24. Cohen J., Statistical Power Analysis for the Behavioral Sciences (Routledge, 2013). [ Google Scholar ] 25. Kaur P. and Stoltzfus J., “Bland–Altman Plot: A Brief Overview,” International Journal of Academic Medicine 3, no. 1 (2017): 110. [ Google Scholar ] 26. Cook D. A., Brydges R., Zendejas B., Hamstra S. J., and Hatala R., “Technology‐Enhanced Simulation to Assess Health Professionals,” Academic Medicine 88, no. 6 (2013): 872–883. [ DOI ] [ PubMed ] [ Google Scholar ] 27. Cook D. A., Hatala R., Brydges R., et al., “Technology‐Enhanced Simulation for Health Professions Education,” JAMA 306, no. 9 (2011): 978–988. [ DOI ] [ PubMed ] [ Google Scholar ] 28. Wang S., Scells H., Zhuang S., Potthast M., Koopman B., and Zuccon G., “Zero‐Shot Generative Large Language Models for Systematic Review Screening Automation” (2024), 403–420. 29. Marshall I. J. and Wallace B. C., “Toward Systematic Review Automation: A Practical Guide to Using Machine Learning Tools in Research Synthesis,” Systematic Reviews 8, no. 1 (2019): 163. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 30. Rekowska D., Gulser S. S., Santpurkar N., et al., “OP26 Artificial Intelligence for Literature Screening and Selection: Does the Evidence Support Its Use in Systematic Literature Reviews?,” International Journal of Technology Assessment in Health Care 40, no. S1 (2024): S13. [ Google Scholar ] 31. Hirt J., Meichlinger J., Schumacher P., and Mueller G., “Agreement in Risk of Bias Assessment Between RobotReviewer and Human Reviewers: An Evaluation Study on Randomised Controlled Trials in Nursing‐Related Cochrane Reviews,” Journal of Nursing Scholarship 53, no. 2 (2021): 246–254. [ DOI ] [ PubMed ] [ Google Scholar ] 32. Tian Y., Yang X., Doi S. A., et al., “Towards the Automatic Risk of Bias Assessment on Randomized Controlled Trials: A Comparison of RobotReviewer and Humans,” Research Synthesis Methods 15, no. 6 (2024): 1111–1119. [ DOI ] [ PubMed ] [ Google Scholar ] 33. Pitre T., Jassal T., Talukdar J. R., Shahab M., Ling M., and Zeraatkar D., “ChatGPT for Assessing Risk of Bias of Randomized Trials Using the RoB 2.0 Tool: A Methods Study” (2023), medRxiv 2023.11.19.23298727, http://medrxiv.org/content/early/2023/11/22/2023.11.19.23298727.abstract . 34. Kuitunen I., Ponkilainen V. T., Liukkonen R., et al., “Evaluating the Performance of ChatGPT‐4o in Risk of Bias Assessments,” Journal of Evidence‐Based Medicine 17, no. 4 (2024): 700–702. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 35. Lai H., Liu J., Bai C., et al., “Language Models for Data Extraction and Risk of Bias Assessment in Complementary Medicine,” npj Digital Medicine 8, no. 1 (2025): 74. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 36. Helms Andersen T., Marcussen T. M., Termannsen A. D., Lawaetz T. W. H., and Nørgaard O., “Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers,” Cochrane Evidence Synthesis and Methods 3, no. 4 (2025): e70036. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 37. Cichewicz A., Kadambi A., Lavoie L., Mittal L., Pierre V., and Raorane R., “MSR41 Application of Artificial Intelligence as a Decision Support Tool for Abstract Screening: Implications for Time and Cost Savings” (2023). 38. Bentley S., Stapleton S. N., Moschella P. C., et al., “Barriers and Solutions to Advancing Emergency Medicine Simulation–Based Research: A Call to Action,” AEM Education and Training 4, no. S1 (2020): S130–S139. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 39. Yu Y., Rong G., Shen H., et al., “Fine‐Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review,” ACM Transactions on Software Engineering and Methodology 34, no. 1 (2025): 1–26. [ Google Scholar ] 40. Yang H., Zhang Y., Xu J., Lu H., Heng P.‐A., and Lam W., “Unveiling the Generalization Power of Fine‐Tuned Large Language Models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (Association for Computational Linguistics, 2024), 884–899. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 41. Pang C., Cao Y., Yang C., and Luo P., “Uncovering Limitations of Large Language Models in Information Seeking From Tables,” in Findings of the Association for Computational Linguistics ACL 2024 (Association for Computational Linguistics, 2024), 1388–1409. [ Google Scholar ] 42. Botsis T., Fairman J. E., Moran M. B., and Anagnostou V., “Visual Storytelling Enhances Knowledge Dissemination in Biomedical Science,” Journal of Biomedical Informatics 107 (2020): 103458. [ DOI ] [ PubMed ] [ Google Scholar ] 43. Cocos A., Qian T., Callison‐Burch C., and Masino A. J., “Crowd Control: Effectively Utilizing Unscreened Crowd Workers for Biomedical Data Annotation,” Journal of Biomedical Informatics 69 (2017): 86–92. [ DOI ] [ PubMed ] [ Google Scholar ] 44. Heim E., Roß T., Seitel A., et al., “Large‐Scale Medical Image Annotation With Crowd‐Powered Algorithms,” Journal of Medical Imaging 5, no. 3 (2018): 34002. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 45. Mak R. H., Endres M. G., Paik J. H., et al., “Use of Crowd Innovation to Develop an Artificial Intelligence–Based Solution for Radiation Therapy Targeting,” JAMA Oncology 5, no. 5 (2019): 654–661. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 46. Abououf M., Otrok H., Mizouni R., Singh S., and Damiani E., “How Artificial Intelligence and Mobile Crowd Sourcing Are Inextricably Intertwined,” IEEE Network 35, no. 3 (2021): 252–258. [ Google Scholar ] 47. Chatterjee S., Mukhopadhyay A., and Bhattacharyya M., “A Review of Judgment Analysis Algorithms for Crowdsourced Opinions,” IEEE Transactions on Knowledge and Data Engineering 32, no. 7 (2020): 1234–1248. [ Google Scholar ] 48. Popay J., Rogers A., and Williams G., “Rationale and Standards for the Systematic Review of Qualitative Literature in Health Services Research,” Qualitative Health Research 8, no. 3 (1998): 341–351. [ DOI ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Table S1: Database. AET2-10-e70189-s002.xlsx (180.7KB, xlsx) Data S1: Methods supplement: prompt text. AET2-10-e70189-s003.pdf (149KB, pdf) Data S2: Supplementary index. AET2-10-e70189-s001.pdf (64.2KB, pdf) Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request. Articles from AEM Education and Training are provided here courtesy of Wiley ACTIONS View on publisher site PDF (1.2 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Expanding the Roles of Medical Educators in the Generative Artificial Intelligence Era: A Qualitative Study.#364667
• Generative artificial intelligence–assisted medical self-care and associated factors among undergraduate students at Arsi University, South-Eastern Ethiopia.#1091344
• All Quiet on the AI Front: The Future of Undergraduate Math Research, Viewed from Duluth#1076271
• Characterizing large language model generative artificial intelligence variability in the production of objective structured clinical examination stations#651099
• Impactos da inteligência artificial na esfera acadêmica: análise comparativa entre os Guias e Marcos Referenciais da UNESCO na Educação, o Plano Brasileiro de IA e Normativas Universitárias Brasileiras#277596