Artificial Intelligence for Teaching Case Curation: Evaluating Model Performance on Imaging Report Discrepancies.

Artificial Intelligence for Teaching Case Curation: Evaluating Model Performance on Imaging Report Discrepancies - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Acad Radiol . Author manuscript; available in PMC: 2026 Jun 1. Published in final edited form as: Acad Radiol. 2025 Mar 9;32(6):3139–3146. doi: 10.1016/j.acra.2025.02.011 Artificial Intelligence for Teaching Case Curation: Evaluating Model Performance on Imaging Report Discrepancies Michael Bartley 1, * , Zachary Huemann 1, * , Junjie Hu 2 , Xin Tie 1 , Andrew B Ross 1 , Tabassum Kennedy 1 , Joshua D Warner 1 , Tyler Bradshaw 1 , Edward M Lawrence 1 Author information Article notes Copyright and License information 1 Department of Radiology, UW-Madison School of Medicine & Public Health 2 Department of Computer Science, UW-Madison * Co first authors ✉ Corresponding author: Edward M Lawrence, MD/PhD, Department of Radiology, University of Wisconsin School of Medicine & Public Health, E3/311 Clinical Science Center, 600 Highland Ave., Madison, WI USA 53792-3252, elawrence@uwhealth.org , fax: 608-263-0876 Issue date 2025 Jun. PMC Copyright notice PMCID: PMC12129674  NIHMSID: NIHMS2064500  PMID: 40058990 The publisher's version of this article is available at Acad Radiol Abstract Rationale/Objectives: Assess the feasibility of using a large language model (LLM) to identify valuable radiology teaching cases through report discrepancy detection. Materials and Methods: Retrospective study included after-hours head CT and musculoskeletal radiograph exams from January 2017-December 2021. Discrepancy level between trainee’s preliminary interpretation and final attending report was annotated on a 5-point scale. RadBERT, a LLM pre-trained on a vast corpus of radiology text, was fine-tuned for discrepancy detection. For comparison and to ensure the robustness of the approach, Mixstral 8×7B, Mistral 7B, and Llama 2 was also evaluated. The model’s performance in detecting discrepancies was evaluated using a randomly selected hold-out test set. A subset of discrepant cases identified by the LLM were compared to a random case set by recording clinical parameters, discrepant pathology, and evaluating possible educational value. F1 statistic was used for model comparison. Pearson’s chi-squared test was employed for to assess discrepancy prevalence and score between groups (significance set at p<0.05). Results: The fine-tuned LLM model achieved an overall accuracy of 90.5% with a specificity of 95.5% and a sensitivity of 66.3% for discrepancy detection. The model sensitivity significantly improved with higher discrepancy scores, 49% (34/70) for score 2 versus 67% (47/62) for score 3, and 81% (35/43) for score 4/5 (p<0.05 compared to score 2). LLM-curated set showed a significant increase in the prevalence of all discrepancies and major discrepancies (scores 4 or 5) compared to a random case set (P<0.05 for both). Evaluation of the clinical characteristics from both the random and discrepant case sets demonstrated a broad mix of pathologies and discrepancy types. Conclusion: A LLM can detect trainee report discrepancies, including both higher and lower-scoring discrepancies, and may improve case set curation for resident education as well as serve as a trainee oversight tool. Introduction: Case-based learning is a vital part of radiology-based education, whether experienced in a radiology reading room, incorporated into didactic teaching experiences, or combined in curated teaching sets. [ 1 – 3 ] Current radiology training does not guarantee equivalent clinical exposure, and multiple factors affect the distribution of cases experienced by an individual learner. It is particularly important that trainees be exposed to challenging cases which have a higher likelihood of misinterpretation. [ 4 , 5 ] This includes ‘discrepant’ cases with discrepancies between the trainee’s initial preliminary interpretation and final attending read. [ 6 , 7 ] Educators can manually collect these discrepant cases for use as training cases; however, this often requires a significant time investment and may be susceptible to the trainer’s biases. [ 8 ] Artificial intelligence (AI) has the potential to assist with the identification and curation of these cases. [ 9 ] Fully implemented, this would allow for rapid assembly of a diverse collection of teaching cases and could allow for evaluation of the patterns of errors for individual trainees. [ 10 ] Moreover, it could facilitate a comprehensive analysis of the types of exams that produce the most trainee discrepancies and may help inform education programs. Ultimately, developing an AI model capable of identifying discrepant and educationally valuable cases could provide both more uniform case exposure and a more individualized approach toward case-based education for medical learners of all levels. The purpose of this study was to assess the feasibility of using a large language model (LLM) to identify difficult teaching cases for radiology education by detecting discrepancies between preliminary and final reports in a complete data set of head CT and musculoskeletal radiograph cases. Methods: Study cohort This retrospective study was approved by the Institutional Review Board at our institution and informed consent was waived. The picture archiving and communication system (PACS) was searched resulting in a full data set of 10,335 on-call non-contrast head CT scans and 16,686 musculoskeletal radiographs that were completed in the emergency department between the hours of 2100 and 0700 from January 1, 2017, through December 31, 2021. These two study types were chosen as two commonly obtained imaging studies in the emergency department with two different modalities. Non-contrast head CT, instead of contrast-enhanced head CT, was used given its much greater prevalence and to maintain consistency. There was no challenge or limitation to the use of contrast-enhanced head CT for the LLM model. Archived reports from these studies were then gathered, including both the initial preliminary report saved by the trainee at the time of initial review and the final report signed by the attending radiology. A subset of 1000 cases (500 of each type) was randomly selected for evaluation while the remaining cases were used for model development. An overall study flowchart is shown in figure 1 . Figure 1. Open in a new tab A flow diagraph showing the steps in data labeling and model training is shown. The initial randomly selected data was used to train the model the first time, we then inferenced over all the data and selected the 500 most discrepant cases based on the model scores (*). These labeled cases were used to train the model the second time and this process was repeated. Report review and discrepancy grading Reports were analyzed by one board-certified radiologist (E.M.L.) for discrepancies between the preliminary and final reports. An additional subspecialist in either neuroradiology (T.K.) or musculoskeletal radiology (A.B.R.) assisted in defining the discrepancy levels for common pathological conditions as well as for consensus review of cases with borderline or difficult to quantify discrepancies. Report discrepancies were primarily evaluated based on changes to the impression section (i.e. the study conclusion, diagnostic hypothesis, or actionable pathology) as this was felt to be the most clinically relevant to providers for direction of medical management. The impression statement format varied between providers, often listed in numbers or blocks of free text, and no reformatting was utilized for model training. In rare cases, when necessary for discrepancy scoring by the board certified radiologists, the body (findings) of the report was utilized to clarify the intended meaning of the report; however, this was not used for model training. Reviewers were blinded to the identity of both the trainee and supervising radiologist. Each paired preliminary and final radiology report was assigned a discrepancy score from 1 to 5, defined as follows: 1 = textual change without change to interpretation, 2 = minor discrepancy, unlikely to change management, 3 = discrepancy that may alter non-emergent management, 4 = major discrepancy with possible management change within 12 hours, 5 = critical discrepancy. Discrepancies with a score of either 4 or 5 were considered ‘major discrepancies’ in our analysis. Examples of scores 2–5 for both head CT and musculoskeletal radiographs are provided in Table 1 . In cases where multiple preliminary reports were placed by the trainee, each report was graded independently regarding discrepancy. For example, if an initial preliminary report (preliminary report #1) listed ‘no acute osseous abnormality’ but the trainee changed their report at a later time (preliminary report #2) to correctly describe an acute fracture, the first report but not the second report would be labeled as discrepant. Including both preliminary reports was useful for model training as it gave the model example reports that were both discrepant and non-discrepant for the same study. These cases might also offer educational value because they contained features that initial confused or misled the trainee. From a practical standpoint the inclusion of both reports might also be justified because while the last preliminary report would reflect the revised judgement of the trainee, early reports might have also had an effect on patient management and clinical decision-making, especially if there was increased time between the two preliminary reports. Table 1. Example diagnoses and report changes for varying discrepancy score level. MSK radiographs Head CT 2 - Change equivocal to definite fracture - Add presumed benign osseous lesion - Add possible sinusitis - Query normal pressure hydrocephalus 3 - Additional fracture - Superficial foreign bodies in wound - Outpatient work-up (e.g. MRI) recommended - Additional small intracranial hemorrhage 4 - Change from no acute to acute fracture - Elbow joint effusion (i.e. occult fracture) - Small subdural or subarachnoid hematoma - Non-displaced occipital fracture 5 - Missed femoral neck fracture - Missed or misdiagnosed epidural hematoma Open in a new tab Model training and case selection Out of the full data set of 10,335 on-call non-contrast head CT scans and 16,686 MSK radiographs, an initial training set of 1000 randomly selected cases from each type (both head CT and MSK radiograph) was used for fine-tuning a pretrained LLM. Due to the low frequency of discrepancies, particularly major discrepancies, the model underwent a second stage of fine-tuning where additional training data was selected from the remaining total case collection based on cases that were predicted by the LLM to be most discrepant. This consisted of 250 studies for each type (500 total) obtained consecutively after further model training between sets. This active learning approach resulted in a higher number of discrepant cases available for model training. Final training included all datasets from both study types. The model’s performance was then evaluated on a hold-out test set comprising 1000 cases randomly selected from the cohort, with 500 of each type. Figure 1 details the study design, including the multiple training sets. RadBERT was used as the LLM, which is pretrained on a large corpus of radiology text [ 11 ]. The preliminary report and final report were tokenized, concatenated and, fed to RadBERT as input. The output embeddings were fed to a classifier that was trained using binary cross entropy to predict discrepant (scores 2–5) and non-discrepant reports. Fine-tuning with RadBERT took 22 minutes on a single Nvidia V100 GPU. Following model tuning and analysis, the LLM results were presented to the study authors as an excel spreadsheet with model results reported for each imaging study. The results were not reported on the radiology information system or electronic medical record, given the retrospective and experimental nature of the current work. Comparative analysis between LLM-curated and uncurated case sets To better understand the educational utility of the identified discrepant cases and to assess for potential model bias, the initial randomly selected case set and one of the LLM-curated sets were manually evaluated and compared regarding discrepancy frequency, case pathology, and other pertinent clinical parameters. The radiology reports and electronic medical records was reviewed by one author (M.B.). Discrepancy type was defined as a false-negative (FN) discrepancy if pathology was missed by the trainee and subsequently included in the attending final report versus false-positive (FP) discrepancy if possible pathology was reported by the trainee that was deemed absent on final review by the attending. Case pathology was recorded, representing the specific pathology of the discrepant report. In complex cases with multiple pathologies, only the pathology associated with the discrepancy was recorded. If more than one discrepant pathology was present, the pathology with the highest discrepancy score or the greatest clinical importance was recorded. For example, if the trainee missed a subarachnoid hemorrhage and dental caries the subarachnoid hemorrhage was considered the primary pathology. Additional clinical parameters that were considered included the location of the patient at the time of discrepancy reporting and associated management changes. Location of patient at time of discrepancy reporting was broken down into three categories: (1) emergency department, (2) admitted to hospital, or (3) discharged and was determined by correlating the patient’s location per the patient record with timestamp on the final report. Consideration for management changes was primarily based on the supervising radiologist’s final report. If recommendations were made, the patient record was investigated to determine if those recommendations were completed. Management changes were tracked to the specified time for follow-up listed in the radiology report or associated provider’s discharge summary or a maximum of 6 months past the report date if no time was specified. Generative model testing We compared the fine-tuned RadBERT model to other open-sourced generative LLMs including Mixstral 8×7B, Mistral 7B, and Llama2. To adapt these models to our task, we employed the in-context learning technique [ 12 ]. The models were evaluated with 0, 1, 3, 5, 10, 25, 50, 100 examples given in the prompt. The prompt template used can be found in appendix 1 . Statistical analysis To quantify the performance of the fine-tuned LLM, accuracy and F1 score were evaluated as well as sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Results were also broken down according to discrepancy scores. The performance of different LLMs was compared using the F1 score. F1 score is a commonly used metric to evaluate machine learning model accuracy, combining both precision and recall into a single metric. Pearson’s chi-squared test was applied to determine statistical significance of discrepancy prevalence and score between groups, with P< 0.05 considered significant. Results: Quantitative results: The randomly selected test set contained 175 discrepant cases (16.9%) and 859 nondiscrepant cases, including 70/175 (40%) with score 2, 62/175 (35.4%) with score 3, and 43/175 (24.6%) with score 4 or 5 as illustrated in Figure 1 . The fine-tuned LLM attained an overall accuracy of 90.5% and an F1 score of 0.70 with a sensitivity of 66.3%, specificity of 95.5%, PPV of 74.8%, and NPV of 93.3% ( Table 2 ). Additionally, the model showed a statistically significant increase (P<0.005) in sensitivity for higher-level discrepancies (score 3 or score 4/5) compared to minor discrepancies (score 2). Sensitivity of 48.5% (34/70) for score 2, 75.8% (47/62) for score 3, and 81.3% (35/43) for scores 4/5. Table 2. Accuracy results for the fine-tuned LLM according to radiology study type. MSK radiographs Head CT Combined Sensitivity 0.69 0.64 0.66 Specificity 0.99 0.94 0.97 Accuracy 0.93 0.88 0.91 PPV 0.87 0.67 0.75 NPV 0.94 0.93 0.93 Open in a new tab Generative model results comparison: All generative LLMs included in this study (Mixtral-8×7B, Mistral, and Llama2) demonstrated inferior performance compared to the task-specific fine-tuned LLM, as evidenced by their F1 scores ( Figure 2 ). The best performing generative model was Mixstral 8×7b with an F1 score of 0.60, followed by Mistral with an F1 score of 0.53, and llama2 with an F1 score of 0.43. Figure 2. Open in a new tab Changes of F1 scores for the generative large language models with varying numbers of examples in their prompt. All generative models under-performed the task specific fine-tuned RadBERT indicated by the dotted line (F1 = 0.70). Comparing the uncurated and LLM-curated case sets The percentage of discrepancies (scores 2–5) and major discrepancies (scores 4, 5) increased significantly for the LLM-curated set compared to the uncurated set. Of the 1,102 combined studies (both head CT and MSK radiograph), there were 946 discrepancies (85.8%) in the LLM-curated discrepant set compared to 218 out of 2096 (10.4%) for the uncurated set. Major discrepancies (scores 4 or 5) increased from 2.1% (45/2096) in the uncurated set to to 23.1% (255/1002) in the LLM-curated set ( Figure 3 ). The LLM-curated case set showed an overall similar distribution of discrepant pathologies compared to the uncurated case set for both head CT and MSK radiograph ( Table 3 ). Illustrative examples from the head CT and MSK radiograph sets are shown in Figures 4 and 5 respectively. Figure 3. Open in a new tab Bar graph comparing the distribution of discrepancy scores between the uncurated and LLM-curated sets demonstrates the increased frequency of both total and major discrepancies for the curated set. Table 3. Distribution of case pathologies for both head CT and MSK radiographs from the LLM-curated and uncurated sets Head CT Case Pathol ogy LLM-curated (n, %) Uncurated (n, %) ENT abnormaity 28 (14%) 11 (10%) Dental abnormality 26 (13%) 10 (9%) Mass 25 (12%) 14 (13%) Cerebral abnormality 25 (12%) 20 (18%) Vascular abnormality 22 (11%) 15 (14%) Intracranial Hemorrhage 21 (10%) 11 (10%) Fracture 14 (7%) 11 (10%) Additional hemorrhage 9 (4%) 8 (7%) Foreign body 5 (2%) 2 (2%) No intracranial abnormality 31 (15%) 6 (5%) Total 206 111 MSK Radiographs Case Pathology LLM-curated (n, %) Uncurated (n, %) Fracture 115 (48%) 31 (34%) Tendon/ligament injury 32 (13%) 10 (11%) Additional fracture 30 (13%) 18 (20%) Osseous abnormality 29 (12%) 12 (13%) Foreign body 8 (3%) 2 (2%) Infection 4 (2%) 3 (3%) Joint effusion 2 (1%) 0 (0%) Mass 2 (1%) 3 (3%) Dislocation 1 (0.5%) 6 (7%) No Fracture/abnormality 17 (7%) 5 (6%) Total 239 91 Open in a new tab Figure 4. Illustrative example of an after hours CT head with discrepancy score 4. Open in a new tab 37-year old patient presented after bicycle versus car collision. Axial (A) and coronal (B) CT images of the head demonstrates a small area of hyperdense thickening (yellow oval in A, arrow in B) along the midline falx, consistent with a small subdural hematoma. This finding was missed by the overnight resident and was not included in the preliminary report text (C) but was documented in the final attending report (D). A follow up head CT (not shown) obtained 8 hours later demonstrated no change in the finding and no intervention was required. Figure 5. Illustrative example of an after hours MSK radiograph with discrepancy score 4. Open in a new tab 31-year old patient presented after a fall with knee pain. Small avulsion fracture of the lateral tibial plateau was noted by the overnight resident on knee radiograph; however, the significance as a Segond-type fracture, including risk for underlying ACL injury, was not expressed until the final report. The curated case set exhibited a higher prevalence of FP discrepancies - case where an abnormality was called on an initial preliminary report that was subsequently removed on the final report - particularly for head CT, with 17% in the LLM-curated set compared to 6% in the uncurated set. For MSK radiographs the FP rate 8% in the LLM-curated set compared to 5% in the uncurated set. Furthermore, the LLM-curated case set showed a slight increase in the percentage of patients located in the emergency room at the time of discrepancy reporting (38% for the LLM-curated group compared to 21% for the uncurated group) with a resultant decrease in the percentage of admitted patients (LLM-curated vs. uncurated: 31% vs. 45%). The percentage of discharged patients remained similar between the two groups (LLM-curated vs. uncurated: 31% vs. 34%). Lastly, there was an increase in the proportion of patients who experienced a preliminary management change in the combined LLM-curated case sets (61.9% (273/441)) compared to the uncurated case sets (49.5% (100/202)). Discussion: This study sought to evaluate a fine-tuned LLM for discrepancy detection in after-hours head CT and MSK radiographs, with the goal of identifying and curating valuable cases for radiology education. We found that the model demonstrated a promising degree of accuracy, particularly regarding specificity and NPV. Moreover, analysis of the LLM-curated case set highlighted the potential of LLMs to quickly and automatically identify a rich set of difficult teaching case, without introducing biases in the types of pathologies represented. We found that the RadBERT model fine-tuned for detecting discrepancies surpassed the zero-shot performance of larger, open-source, decoder-only LLMs. While the trend in language modeling is towards large, chatbot-style, decoder-only LLMs, studies have shown that smaller LLMs fine-tuned for specific tasks can still outperform general-purpose chatbot LLMs. [ 13 ] We evaluated various approaches for optimizing the prompts for these generative LLMs, and while some techniques resulted in improved performance, fine-tuning still produced the best results. Discrepancies were rare in the randomly selected case set, especially major discrepancies (4%). Manually identifying these discrepant cases for teaching case curation would be time-consuming. This prevalence is similar to prior studies, which ranged from 1% to 4% [ 14 – 17 ]. In contrast, the LLM-curated case set had a much higher prevalence of both total and major discrepancies, which could be very useful for case-based radiology teaching. The ability to curate both higher and lower-scoring discrepancies can be useful as they might have different educational purposes. For example lower-scoring discrepancies can certainly offer education value and are more likely to occur versus the rarity of a higher-scoring discrepancy; however, from a practical perspective, including from the perspective of the ordering provider, there is greater clinical importance for higher-scoring discrepancies especially for training programs where the trainee continues to have independent interpretations occur overnight (such as the study institution). Having an automated tool that can identify a group of cases for educator review could save valuable time and might allow for collection of a broader range of representative cases and discrepancies. Prior literature has proposed the idea of integrating AI into education practices, including case curation [ 18 ]; however, to our knowledge this study is among the first to actually test the feasibility in practice. In addition to improving efficiency, utilizing a fine-tuned LLM for case identification ensures that the curated set includes real-life cases with pathologies that posed significant challenges to the interpreting trainees. Future work might look to fine-tune the model to focus even more on the higher-scoring discrepancies, which might be useful as a tool for monitoring and evaluating overnight performance, as well as to focus on lower-scoring discrepancies, which might be useful as a tool to educate residents regarding more common interpretative and reporting errors. As an educator it would be important to consider the potential risk of having a discrepancy evaluation or reporting system considered as punitive or reproachful, which is more likely if higher-scoring discrepancies are emphasized, and an optimal system might try to emphasize both. We assessed whether the LLM-curated cases exhibited any bias toward a specific pathologic subtype. Our analysis revealed that the fine-tuned LLM collected discrepant cases with a distribution of general case pathology and type similar to those in the uncurated set. There was a trend toward the LLM identifying an increased percentage of FP discrepancies, particularly for head CT. this could be viewed as a strength of the model, as the tendency to ‘over-call’ equivocal or confusing findings on call is a common challenge for trainees, especially at the junior level. Having more examples of these ‘over-calls’, or FP discrepancies, could be useful for radiology education. Finally, we observed an increase in the percentage of patients in the LLM-curated group who were located in the ED at the time of discrepancy reporting and who had management changes as a result of the discrepancy. This may be attributed to the increased prevalence of cases with major discrepancies in the LLM-curated group, further supporting the model’s ability to detect cases of high teaching value. Mellnick et al. [ 17 ] previously evaluated patient location at the time of discrepancy reporting and found 23.2% of patients were discharged at the time of discrepancy reporting, not that dissimilar to those found in the current study (31 and 34% for the random and curated sets). Discrepancy detection algorithms could have applications beyond teaching case curation. For example, they may help monitor the progress of trainees by automatically evaluating the frequency of different types of discrepancy between their reports and the attendings’ reports. This could help educators tailor their instructions to the types of cases the trainees struggle with. Additionally, this type of algorithm could lay the groundwork for a more advanced multimodal triage algorithm that could predict the likelihood of a discrepancy based on the images and preliminary report – this is a promising avenue of future research. Other benefits might also include uses for meeting quality assurance targets, collecting data for risk management or staffing adjustments, and automated discrepancy notification. Prior work has shown promise for chatbot LLMs regarding their ability to synthesize and summarize scientific papers [ 20 ]. However, this is a different scientific aim then what was evaluated in the current work. Instead, the authors assessed the model’s ability to detect differences and discrepancies between two reports on the same imaging study. Future work might investigate the ability of AI to summarize the discrepancies present between the two reports, which might allow for even greater efficiency for teaching case curation and trainee oversight. The study had several limitations. First, the study focused on discrepancies on after hours ED studies at a single institution. Future work will include studies from multiple institutions and non-ED studies, to develop a more broadly applicable model. Second, we did not evaluate discrepancies for exams that were read during working hours. These exams might have different characteristics than those read after hours. Third, the model focused on two study types (head CT and MSK radiographs), and future research could extend our method to other study types. Lastly, we did not investigate the most advanced commercial chatbot LLMs due to data ownership and privacy concerns. In addition, these LLMs cannot be fine-tuned, and are not easily deployable within a teaching program. In contrast, open-sourced LLMs can be deployed on premise with greater institutional control and oversight. In conclusion, this study demonstrated that a fine-tuned LLM can be utilized to detect trainee report discrepancies with promising accuracy and may be useful for improved resident education, case set curation, and trainee oversight. Supplementary Material 1 NIHMS2064500-supplement-1.docx (14.1KB, docx) Acknowledgments “Research reported in this publication was supported by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number R01EB033782” Footnotes Declaration of interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. References: 1. Fromke EJ, Jordan SG, Awan OA. Case-based Learning: Its Importance in Medical Student Education. Acad Radiol. 2022. Aug;29(8):1284–1286. [ DOI ] [ PubMed ] [ Google Scholar ] 2. Sugi MD, Kennedy TA, Shah V, Hartung MP. Bridging the gap: interactive, case-based learning in radiology education. Abdom Radiol (NY). 2021. Dec;46(12):5503–5508. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 3. Chorney ET, Lewis PJ. Integrating a radiology curriculum into clinical clerkships using case oriented radiology education. J Am Coll Radiol. 2011. Jan;8(1):58–64, 64.e1–4. [ DOI ] [ PubMed ] [ Google Scholar ] 4. Poot JD, Chetlen AL. A Simulation Screening Mammography Module Created for Instruction and Assessment: Radiology Residents vs National Benchmarks. Acad Radiol. 2016. Nov;23(11):1454–1462. [ DOI ] [ PubMed ] [ Google Scholar ] 5. Chen PH, Roth H, Galperin-Aizenberg M, Ruutiainen AT, Gefter W, Cook TS. Improving Abnormality Detection on Chest Radiography Using Game-Like Reinforcement Mechanics. Acad Radiol. 2017. Nov;24(11):1428–1435. [ DOI ] [ PubMed ] [ Google Scholar ] 6. Meyer RE, Nickerson JP, Burbank HN, Alsofrom GF, Linnell GJ, Filippi CG. Discrepancy rates of on-call radiology residents’ interpretations of CT angiography studies of the neck and circle of Willis. AJR Am J Roentgenol. 2009. Aug;193(2):527–32. [ DOI ] [ PubMed ] [ Google Scholar ] 7. Filippi CG, Schneider B, Burbank HN, Alsofrom GF, Linnell G, Ratkovits B. Discrepancy rates of radiology resident interpretations of on-call neuroradiology MR imaging studies. Radiology. 2008. Dec;249(3):972–9. [ DOI ] [ PubMed ] [ Google Scholar ] 8. Wildenberg JC, Chen PH, Scanlon MH, Cook TS. Attending Radiologist Variability and Its Effect on Radiology Resident Discrepancy Rates. Acad Radiol. 2017. Jun;24(6):694–699. Doi: 10.1016/j.acra.2016.12.004. Epub 2017 Jan 24. [ DOI ] [ PubMed ] [ Google Scholar ] 9. Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol. 2019. Nov;92(1103):20190389. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 10. Pusic M, Pecaric M, Boutis K. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Acad Med. 2011. Jun;86(6):731–6. [ DOI ] [ PubMed ] [ Google Scholar ] 11. Yan A, McAuley J, Lu X, Du J, Chang EY, Gentili A, Hsu CN. RadBERT: Adapting Transformer-based Language Models to Radiology. Radiol Artif Intell. 2022. Jun 15;4(4):e210258. doi: 10.1148/ryai.210258. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, et al. Language models are few-shot learners. arXiv preprint 2020. arXiv:2005.14165, 1. [ Google Scholar ] 13. Tie X, Shin M, Pirasteh A et al. Personalized Impression Generation for PET Reports Using Large Language Models. J Digit Imaging. Inform. med. 2024; 37:471–488. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 14. Weinberg BD, Richter MD, Champine JG, Morriss MC, Browning T. Radiology resident preliminary reporting in an independent call environment: multiyear assessment of volume, timeliness, and accuracy. J Am Coll Radiol. 2015. Jan;12(1):95–100. [ DOI ] [ PubMed ] [ Google Scholar ] 15. Cooper VF, Goodhartz LA, Nemcek AA Jr, Ryu RK. Radiology resident interpretations of on-call imaging studies: the incidence of major discrepancies. Acad Radiol. 2008. Sep;15(9):1198–204. [ DOI ] [ PubMed ] [ Google Scholar ] 16. Wysoki MG, Nassar CJ, Koenigsberg RA, Novelline RA, Faro SH, Faerber EN. Head trauma: CT scan interpretation by radiology residents versus staff radiologists. Radiology. 1998. Jul;208(1):125–8. [ DOI ] [ PubMed ] [ Google Scholar ] 17. Mellnick V, Raptis C, McWilliams S, Picus D, Wahl R. On-Call Radiology Resident Discrepancies: Categorization by Patient Location and Severity. J Am Coll Radiol. 2016. Oct;13(10):1233–1238. doi: 10.1016/j.jacr.2016.04.020. Epub 2016 Jun 17. [ DOI ] [ PubMed ] [ Google Scholar ] 18. Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol. 2019. Nov;92(1103):20190389. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, et al. The Llama 3 Herd of Models. arXiv preprint 2024. arXiv:2407.21783. [ Google Scholar ] 20. Infante A, Guerri G, Del Ciello A, Gullì C, Chiloiro S, De Vizio S, Ruscelli L, Merlino B, Natale L, Sala E, Gaudino S. Would ChatPDF be advantageous for expediting the interpretation of imaging and clinical articles in PDF format? J Med Artif Intell 2025;8:1. [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials 1 NIHMS2064500-supplement-1.docx (14.1KB, docx) ACTIONS View on publisher site PDF (1.1 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Artificial Intelligence/Machine Learning in Nuclear Medicine and Hybrid Imaging#10826
• From Artificial Intelligence to Robotics: How to Navigate Technological Innovation in Radiology With the Gartner Hype Cycle.#242315
• Extended reality and web-based imaging in case-based anatomy education: a two-year study of knowledge retention on anomalies and variations.#1097804
• The Image Characteristics of Adult Appendix on Computed Tomography#845262
• Student Experiences in Artificial Intelligence-Supported Learning Environments: A Nuanced Mixed-Methods Case Study#281761