Evaluation of ChatGPT-4o in Breast Cancer Screening: Insights from the 5th Edition BI-RADS Atlas and ACR Guidelines.

Evaluation of ChatGPT-4o in Breast Cancer Screening: Insights from the 5th Edition BI-RADS Atlas and ACR Guidelines - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice J Imaging Inform Med . 2025 Sep 12;39(3):2752–2768. doi: 10.1007/s10278-025-01663-8 Evaluation of ChatGPT-4o in Breast Cancer Screening: Insights from the 5th Edition BI-RADS Atlas and ACR Guidelines Bilgen Mehpare Özer 1, ✉ , Eda Nur Korkmaz 1 Author information Article notes Copyright and License information 1 Department of Radiology, Sincan Training and Research Hospital, Ankara, Turkey ✉ Corresponding author. Received 2025 Apr 29; Revised 2025 Aug 20; Accepted 2025 Aug 22; Collection date 2026 Jun. © The Author(s) under exclusive licence to Society for Imaging Informatics in Medicine 2025. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. PMC Copyright notice PMCID: PMC13230356  PMID: 40940588 Abstract The aim of this study is to evaluate the potential, reliability, and limitations of ChatGPT-4o in text-based questions and its effectiveness in clinical decision support processes based on the 5th edition of the BI-RADS Atlas and ACR breast cancer screening guidelines. In this study, a total of 100 questions—50 multiple-choice and 50 true/false—prepared by two radiologists were submitted to ChatGPT-4o between November 5 and 19. The answers provided by ChatGPT-4o were evaluated at baseline and 14 days later by both radiologists for accuracy and comprehensiveness using a Likert scale. Group comparisons were performed using Mann–Whitney U , Wilcoxon tests; response consistency was evaluated with Cohen’s Kappa, and overall accuracy differences with a two-proportion z -test. The increase in overall accuracy from 86 to 95% was statistically significant according to the two-proportion z -test ( p = .030). Comparisons between the two sessions revealed statistically significant increases in the accuracy ( p = .013, r = .35, 95% CI [0.09, 0.61]) and comprehensiveness ( p = .014, r = .35, 95% CI [0.09, 0.61]) rates of true/false questions. On the other hand, no significant difference was found between the accuracy ( p = .180, r = .19, 95% CI [− 0.09, 0.47]) and comprehensiveness ( p = .180, r = .19, 95% CI [− 0.09, 0.47]) rates of multiple-choice questions. In addition, group comparisons evaluating the effect of different question formats on performance revealed no significant difference in terms of accuracy ( p = .661, r =  − 0.04, 95% CI [− 0.23, 0.16]) and comprehensiveness ( p = .708, r =  − 0.04, 95% CI [− 0.23, 0.16]). The consistency of ChatGPT-4o responses was supported by Cohen’s Kappa coefficients, all statistically significant ( p < .001), with 95% confidence intervals ranging from − .038 to 1.084. ChatGPT-4o demonstrated remarkable performance in answering multiple-choice and true–false questions with overall accuracy improving from 86% in the first test to 95% after 14 days. ChatGPT-4o holds significant potential as a clinical decision support tool for healthcare professionals. Keywords: Breast cancer, BI-RADS Atlas, ChatGPT, AI Introduction Breast cancer is the most frequently diagnosed type of cancer among women in developed countries and is considered one of the leading causes of cancer-related deaths. In 2020, approximately 2.26 million new cases of breast cancer were reported worldwide [ 1 ]. Screening programs tailored to the specific needs of each country play a crucial role in breast cancer management. Early detection of breast cancer significantly reduces the incidence of late-stage diagnoses and cancer-related mortality rates [ 2 , 3 ]. In this context, the Breast Imaging Reporting and Data System (BI-RADS) was developed by the American College of Radiology (ACR) to standardize breast imaging reports. Currently, the 5th Edition of the BI-RADS Atlas encompasses various imaging modalities, including mammography, ultrasound, magnetic resonance imaging, contrast-enhanced mammography, and digital tomosynthesis. The system aims to evaluate malignancy risk and promote the use of a common language among physicians [ 4 , 5 ]. The ACR breast cancer screening program is a guide for defining the target population, when and who to screen, and how to screen high-risk women [ 6 ]. Artificial intelligence (AI) is increasingly utilized in clinical practice due to its advantages in accuracy, cost-effectiveness, and reproducibility. ChatGPT, a language model developed by OpenAI, holds significant potential in supporting clinical decision-making systems, helping physicians work more efficiently, effectively, and consistently [ 7 , 8 ] Breast cancer and breast imaging is a vast and multidimensional field. In recent years, various studies have been carried out in this field using different AI models. These studies cover different topics and approaches such as answering frequently asked questions by patients about breast cancer screening and prevention [ 9 ], assigning BI-RADS categories to imaging reports [ 10 ], and evaluating multiple-choice questions about breast cancer care [ 11 ]. Although there are some studies comparing the effectiveness of AI models on multiple-choice questions with board exams [ 12 ], there are few studies in the literature comparing performance on text-based questions using only the BI-RADS atlas [ 13 ]. To the best of our knowledge, this study is the first to assess the performance of ChatGPT-4o in the field of breast imaging at different time intervals. In our study, we investigated the extent to which ChatCGT-4o is able to produce effective and consistent responses in tasks such as understanding BI-RADS classifications and correctly interpreting screening recommendations, based on responses to true/false and multiple-choice questions structured on the basis of the 5th edition of the BI-RADS Atlas and the breast cancer screening guidelines published by the American College of Radiology (ACR). In this context, the study aims to evaluate the potential, reliability, and limitations of ChatGPT-4o as a clinical decision support system. Methods Since our study does not involve animal subjects or human subjects, informed consent and ethical approval were not obtained. All procedures were conducted in compliance with the ethical principles and standards outlined in the Declaration of Helsinki. Study Setting A total of 100 questions (50 true–false and 50 multiple choice) were prepared by 2 general radiologists, BMÖ (14 years) and ENK (2 years). The questions were based on the 2013 ACR BI-RADS Atlas: 5th Edition, and the ACR breast cancer screening guidelines were developed by both radiologists in consensus. The questions cover a wide range of topics, including BI-RADS category classification and interpretation, terminology, imaging indications, lesion characterization, breast cancer screening protocols, and breast cancer risk factors. Of the 50 multiple-choice questions, 12 were on MRI, 9 on ultrasound, 13 on mammography, and 16 on general topics (BI-RADS terminology, ACR screening guidelines, etc.). The 50 true–false questions included 9 MR, 7 ultrasound, 21 mammography, and 13 general topics. Questions were prepared in two different categories (Table 1 ): True–False Questions : A total of 50 questions were developed to evaluate the AI’s ability to verify or refute statements based on the BI-RADS Atlas and breast cancer screening guidelines. Multiple-Choice Questions : A total of 50 questions were prepared to assess the model’s capability to select the correct option from multiple choices. Table 1. Multiple choice and true–false questions Multiple Choice Questions 1) Which of the following is NOT an indication for contrast-enhanced breast MRI? a) Suspected recurrence of breast cancer b) Patient with axillary lymphadenopathy but unknown primary malignancy c) Silicone or non-silicone breast augmentation d) Patient with a BI-RADS 2 result on mammography 2) Which of the following is NOT an indication for annual breast MRI screening according to ACR? a) History of chest radiation therapy before the age of 30 b) Genetic mutation with untested first-degree relatives c) Family history of breast cancer in a second-degree relative d) History of breast cancer before the age of 50 3) Which of the following is NOT a contraindication for breast MRI? a) Metallic clips from previous breast surgery b) Metallic ocular fragments c) Ferromagnetic vascular clips d) Inability to lie in the prone position 4) Which of the following is NOT a pattern of non-mass enhancement distribution? a) Regional b) Focal c) Heterogeneous d) Linear 5) Which of the following is a shape pattern defined for mass lesions in breast MRI? a) Oval b) Spiculated c) Focal d) Linear 6) What is the risk range for the BI-RADS 4 category? a) 2%−15% b) 2%−95% c) 5%−50% d) 95% and above 7) Which of the following is NOT a type of enhancement kinetics described in breast MRI? a) Type 1 enhancement kinetics b) Type 2 enhancement kinetics c) Type 3 enhancement kinetics d) Type 4 enhancement kinetics 8) What is the appropriate timing for breast MRI imaging? a) Days 1–5 of the menstrual cycle b) Just before the menstrual period c) Days 5–15 of the menstrual cycle d) Day 20 of the menstrual cycle 9) Which of the following is NOT one of the background parenchymal enhancement (BPE) types in breast MRI? a) Minimal b) Mild c) Moderate d) Heterogeneous 10) Which of the following is NOT a descriptor of internal enhancement in breast MRI? a) Rim enhancement b) Dark internal septations c) Homogeneous d) Hyperechoic 11) Which of the following is NOT an associated feature in breast MRI? a) Nipple retraction b) Pectoralis muscle invasion c) Postoperative collections d) Chest wall invasion 12) Which of the following is NOT a fat-containing lesion described in breast MRI? a) Hamartoma b) Cyst c) Lymph nodes d) Fat necrosis 13) Which of the following is NOT a characteristic of mucinous carcinoma? a) Hyperintense lesion on T2-weighted MRI b) Found in older female patients c) Most commonly encountered as a solid mass on ultrasound in younger patients d) Can be confused with cystic lesions 14) What is the screening interval recommended by the ACR for mammography in women over 40 years old? a) Every 2 years b) Annually c) Every 5 years d) Every 6 months 15) What is the minimum recommended age for mammography? a) 25 b) 40 c) 35 d) 45 16) Which of the following is NOT mammographic positioning? a) MLO b) CC c) Exaggerated lateral d) Anterior posterior 17) Which of the following is a finding indicating the adequacy of the MLO view in mammographic examination? a) Most of the breast tissue is included in the image b) The inferior edge of the pectoral muscle is visible below the posterior nipple line c) The breast appears saggy d) The axilla is excluded from the imaging field 18) Which of the following is NOT a calcification distribution pattern in mammography? a) Segmental b) Linear c) Regional d) Round 19) Which of the following breast lesions is associated with the "breast within a breast" sign? a) Fibroadenoma b) Fibroadenolipoma c) Phyllodes tumor d) Fat cyst 20) Which of the following is NOT a benign calcification pattern seen on mammography? a) Vascular b) Dystrophic c) Amorphous d) Suture 21) Which of the following is a malignant calcification pattern seen on mammography? a) Fine linear branching b) Fat necrosis c) Vascular d) Skin 22) Which of the following corresponds to breast composition Type D in mammography? a) The breasts are almost entirely fatty b) There are scattered areas of fibroglandular density c) The breasts are extremely dense, which lowers the sensitivity of mammography d) The breasts are heterogeneously dense, which may obscure small masses 23) Which of the following is NOT a type of asymmetry described in mammography? a) Focal b) Global c) Developing d) Linear 24) Which of the following is NOT an additional finding suggestive of malignancy? a) Skin thickening b) Parenchymal distortion c) Skin retraction d) Intramammary lymph node (IMLN) 25) Which of the following is NOT a background parenchymal enhancement (BPE) type in contrast-enhanced digital mammography according to the BI-RADS 5th edition? a) Minimal b) Heterogeneous c) Moderate d) Marked 26) Which of the following is NOT an internal enhancement characteristic for a mass in contrast-enhanced digital mammography according to the BI-RADS 5th edition? a) Homogeneous b) Clumped c) Heterogeneous d) Rim 27) Which of the following is false about tomosynthesis? a) Provides 3D imaging b) SM/DBT(Synthetic Mammography/Digital Breast Tomosynthesis) significantly increases patient radiation dose c) Reduces the false-positive rate d) Increases cancer detection rates 28) Which of the following is NOT an ultrasound finding of fibroadenoma? a) Hypoechoic b) Parallel orientation to the skin c) Sharp border d) Micro lobulation 29) Which of the following is a finding suggestive of breast cancer on ultrasound? a) Wider than tall b) Posterior acoustic enhancement c) Hyperechoic d) Irregular borders 30) How should follow-up be conducted for a patient categorized as BI-RADS 3 on ultrasound? a) Every 1 month b) Every 2 months c) Every 3 months d) Every 6 months 31) Which of the following is NOT a term used to describe mass margins on ultrasound? a) Spiculated b) Angular c) Irregular d) Smooth 32) Which of the following is NOT included in the "special cases" section according to the BI-RADS 5th Edition 2013 atlas? a) Simple cyst b) Complicated cyst c) Clustered microcysts d) Fibroadenoma 33) Which of the following is a "special case" according to the B-IRADS 5th Edition 2013 atlas? a) Axillary lymph nodes b) Fibroadenoma c) Breast cancer d) Edema 34) Which of the following is NOT a descriptor for calcifications in ultrasound according to the BI-RADS 5th Edition 2013 atlas? a) In mass b) Out of mass c) Intraductal d) Axillary 35) Which of the following is NOT an echogenicity pattern for a mass described in ultrasound according to the BI-RADS 5th Edition 2013 atlas? a) Anechoic b) Hypoechoic c) Superechoic d) Isoechoic 36) Which of the following is NOT an examination indication required to be reported according to the BI-RADS 5th Edition 2013 atlas? a) High-risk screening b) Treatment with neoadjuvant chemotherapy c) Newly diagnosed cancer d) Cough 37) Which of the following is NOT a parameter that should be included in an MRI report? a) MRI protocol b) Contrast dose c) Postprocessing technique d) Technician’s name 38) Which of the following genetic conditions is NOT mentioned as part of the high-risk population for breast cancer? a) Li-Fraumeni syndrome b) Cowden syndrome c) Neurofibromatosis type 1 (NF-1) d) Multiple endocrine neoplasia type 1 (MEN 1) 39) Which of the following is NOT a factor that increases the risk of breast cancer? a) Breastfeeding b) Family history c) Genetic mutations d) Chest radiation at a young age 40) Which of the following is NOT associated with high risk for breast cancer? a) Lobular carcinoma in situ (LCIS) b) Atypical ductal hyperplasia c) Atypical lobular hyperplasia d) Simple cyst 41) Which of the following is NOT one of the ACR recommendations regarding breast screening? a) For most women at higher-than-average risk, breast MRI is the preferred supplemental screening method b) Women diagnosed with breast cancer before age 50, or with a family history of breast cancer and dense breasts, should undergo annual supplemental screening with breast MRI c) Breast MRI is recommended for women with dense breasts who request additional screening d) All women should undergo risk assessment at age 40 42) Which of the following is NOT an appropriate indication for diagnostic breast and axillary ultrasound? a) Palpable mass detected on clinical examination b) Evaluation of symptoms in pregnant or lactating women c) Guidance for biopsy d) Microcalcifications 43) Which modality is recommended for annual screening for women diagnosed with breast cancer before age 50 or with a personal history of breast cancer and dense breasts? a) Contrast-enhanced mammography b) MRI c) Ultrasound d) Contrast-enhanced ultrasound 44) Which is the preferred modality for symptomatic patients aged 20–30? a) MRI b) Ultrasound (US) c) Mammography (MG) d) Contrast-enhanced mammography 45) Which is the first-choice modality for a 45-year-old woman with spontaneous nipple discharge? a) Ultrasound (US) b) Mammography (MG) c) MRI d) Contrast-enhanced ultrasound 46) What is the most common pathology in a 20-year-old woman with spontaneous nipple discharge? a) Intraductal papilloma b) Fibroadenoma c) Mucinous carcinoma d) Fat cyst 47) In a patient with asymmetry detected on screening mammography, what is the next step? a) Spot compression b) MRI c) Biopsy d) Contrast-enhanced mammography 48) Which of the following statements is correct regarding male breast evaluation? a) BI-RADS is used in mammography for male patients b) BI-RADS is not used in mammography for male patients c) Imaging is unnecessary for male patients d) The first-choice modality for male patients is MRI 49) What is the next imaging modality after detecting axillary lymphadenopathy on mammography? a) Ultrasound b) MRI c) Contrast-enhanced mammography d) Contrast-enhanced ultrasound 50) What is the BI-RADS category for a patient with bilateral axillary adenopathy seen on mammography, with a similar appearance in prior studies? a) BI-RADS 1 b) BI-RADS 4 c) BI-RADS 5 d) BI-RADS 2 True—False Questions 1) Level 1 axillary lymph nodes are located inferior and lateral to the pectoralis minor muscle 2) According to the 2013 5th edition of the BI-RADS atlas, amorphous calcifications are classified as BI-RADS 4C 3) According to the 2013 5th edition of the BI-RADS atlas, fine-pleomorphic microcalcifications are classified as BI-RADS 4C 4) According to the 2013 5th edition of the BI-RADS atlas, a skin thickness greater than 2 mm is only observed in breast cancer 5) According to the 2013 5th edition of the BI-RADS atlas, milk of calcium is considered a benign calcification 6) According to the 2013 5th edition of the BI-RADS atlas, coarse heterogeneous calcifications are classified as BI-RADS 4C 7) According to the 2013 5th edition of the BI-RADS atlas, the distribution of calcifications is described as diffuse, regional, grouped, linear, and segmental 8) According to the 2013 5th edition of the BI-RADS atlas, the upper limit for the definition of grouped calcifications is 1 cm 9) According to the 2013 5th edition of the BI-RADS atlas, the lower limit for the distribution of regional calcifications is accepted as 2 cm 10) According to the 2013 BI-RADS atlas, breast tissue is categorized on mammography as homogeneous-fat, homogeneous fibroglandular, or heterogeneous 11) Mammography is divided into screening and diagnostic categories 12) The use of BI-RADS category 4 and 5 in screening mammograms is not recommended 13) BI-RADS category 3 is not used in screening mammograms 14) In ultrasound, the orientation of a mass is assessed as parallel or non-parallel 15) According to the 2013 5th edition of the BI-RADS atlas, ductal changes are considered associated features 16) BI-RADS 0 indicates the need for additional imaging or comparison with previous mammograms 17) In the 5th edition of BI-RADS, the ductal definition has been removed from the non-mass enhancement distribution patterns listed in the 4th edition 18) Routine imaging is recommended for patients classified as BI-RADS 1 19) BI-RADS 2 is used for benign lesions 20) Breast prostheses are classified as BI-RADS 1 21) Lesions such as fat cysts, lipomas, and galactoceles are classified as BI-RADS 1 22) IMLN (Intramammary Lymph Node) is classified as a BI-RADS 2 lesion 23) BI-RADS 3 is used for probably benign lesions 24) For BI-RADS 4 lesions, follow-up is recommended at 6, 12, and 24 months 25) BI-RADS 4 lesions are considered suspicious, and tissue diagnosis is recommended 26) In breast MRI, there are subcategories for BI-RADS category 4 27) BI-RADS category 5 has a malignancy probability of 95% or higher 28) BI-RADS category 6 is used for malignancies that have been proven by biopsy 29) According to the 2013 5th edition of the BI-RADS atlas, the echo pattern of lesions is classified as anechoic, hypoechoic, complex cystic and solid, isoechoic, hyperechoic, and heterogeneous 30) According to the 2013 5th edition of the BI-RADS atlas, the most concerning feature of the applied modality (US, MG, etc.) should determine the final BI-RADS category and management recommendation 31) Orientation refers to the position of the lesion’s long axis relative to the skin 32) If the findings from previous mammography and ultrasound are suspicious, the expression BI-RADS 0 may be used 33) In reporting, breast MRI recommended for a patient with a BI-RADS 2 lesion 34) In mammography, the expression "solitary group of punctate calcifications" is defined as a BI-RADS 3 lesion 35) In mammography, if a lesion classified as BI-RADS 3 remains stable during follow-up, its classification is changed to BI-RADS 2 36) In mammography, if a lesion defined as BI-RADS 3 shows changes during follow-up, its classification may be changed to BI-RADS 4 or BI-RADS 5 37) Focal asymmetry is reported as BI-RADS 3 in mammography 38) If a lesion followed as BI-RADS 3 on ultrasound shows a size increase of 20% or more during follow-up, continued monitoring is recommended 39) In the presence of suspicious unilateral lymphadenopathy without breast abnormality, BI-RADS 3 is used 40) In mammography, BI-RADS 6 is used for high-density irregular masses 41) If an implant is detected in breast MRI, its contents (saline, silicone, etc.) and the number of lumens (single, multiple) should be specified 42) In breast MRI, implant localization can only be defined as retro pectoral 43) HRT use and menstrual cycle do not affect background contrast uptake 44) In breast MRI, malignant masses never show characteristics such as oval shape or homogeneous contrast uptake 45) Non-Mass Enhancement can be used for BI-RADS 2 lesions in breast MRI 46) In most BI-RADS 3 lesions, patient management is performed through follow-up. However, if the patient or clinician has concerns, a biopsy may be performed 47) Ultrasound-guided biopsies are faster, more comfortable for the patient, and more cost-effective compared to MRI-guided biopsies 48) If there is any doubt regarding the sonographic correlation with a suspicious MRI lesion, MRI-guided biopsy is recommended 49) A single MRI descriptor is sufficient to predict malignancy with more than 95% likelihood 50) In breast imaging, prior studies are not required for reporting Open in a new tab No manuals or educational resources were uploaded to ChatGPT-4o before the questions were asked to ChatGPT-4o. The questions were not generated in the ChatGPT application and were not uploaded to the system before the study for testing purposes. In this study, questions were presented to ChatGPT-4o on December 5 and December 19, 2024. Questions for ChatGPT-4o “I work as a radiologist. I will ask multiple choice and true–false questions based on BI-RADS atlas 5th edition and ACR breast cancer screening guidelines. Could you please answer my questions like a radiologist?” were asked in a single session with the prompt. The questions were re-asked to ChatGPT 14 days later in the same order and with the same prompt in a single session. To reduce the likelihood of ChatGPT generating responses similar to previous queries and to ensure that responses are evaluated more independently, a 14-day interval has been left between queries. ChatGPT-4o did not only answer the questions as true or false but also explained the answers in detail. Therefore, the questions could also be evaluated in terms of comprehensiveness. An example of the prompt presented to ChatGPT-4o, along with a sample question and response, is shown in Fig. 1 . In addition, Fig. 2 shows an example of an incorrect question in the first session being answered correctly in the second session. An example of a question that ChatGPT-4o answered incorrectly in both sessions is shown in Fig. 3 . Fig. 1. Open in a new tab Sample answers provided by ChatGPT. In the sample question, since ChatGPT’s answer was correct, the accuracy score was evaluated as 6 according to the Likert scale; and since ChatGPT answered all options in detail, the comprehensiveness score was evaluated as 3 according to the Likert scale Fig. 2. Open in a new tab Sample answers provided by ChatGPT. ChatGPT answered the multiple-choice question incorrectly in the first session but correctly in the second session. In the first session, the question was answered incorrectly. However, the explanation provided included partially accurate information; therefore, a score of 3 was given for accuracy and 2 for comprehensiveness. In the second session, the question was answered correctly, and since a comprehensive explanation was provided, a score of 6 was given for accuracy and 3 for comprehensiveness Fig. 3. Open in a new tab Sample answers provided by ChatGPT. Since the question focused on the earliest possible age at which mammography can be performed, a response referring to the recommended age for routine screening was considered insufficient in this context and therefore marked as incorrect. However, the explanation correctly noted that routine screening typically begins at age 40 and appropriately acknowledged that earlier screening may be warranted for high-risk individuals. As such, a comprehensiveness score of 2 was assigned. This question received a similar response in both sessions Likert scale was used because it can evaluate the responses of the ChatGPT-4o in terms of accuracy, comprehensiveness, enables the comparison of data in different time periods, and is easy to apply. The accuracy of responses was evaluated using a Likert scale: Completely incorrect Mostly incorrect Equally correct and incorrect More correct than incorrect Nearly completely correct Completely correct Additionally, the completeness of the responses was assessed using a three-point scale based on the level of detail and relevance: Insufficient—addresses some aspects of the question but lacks critical elements. Adequate—covers all necessary aspects with minimal detail. Comprehensive—provides more information and context than expected. All responses were evaluated for accuracy and comprehensiveness by two radiologists (BMU and ENK) for compliance with the 2013 ACR BI-RADS Atlas and ACR breast cancer screening guidelines. The final decision was made collaboratively by both radiologists. Statistical Analysis All statistical analyses were conducted using IBM SPSS Statistics (v.26). Since the assumptions of parametric tests were not met, the Wilcoxon Signed-Rank Test was used to compare paired measurements. This test was applied to assess differences in ChatGPT-4o’s accuracy and comprehensiveness scores between the first and second sessions for the same set of questions. Differences in overall accuracy rates between the two sessions were assessed using a two-ratio z -test. To evaluate the effect of question format (multiple-choice vs. true/false) on ChatGPT-4o’s performance, the Mann–Whitney U test was used for comparisons between two independent groups. This non-parametric test assessed whether the accuracy and comprehensiveness scores significantly differed based on question type. Additionally, Cohen’s Kappa coefficient was calculated to assess the consistency of ChatGPT-4o’s responses between the first and second test sessions. This analysis was used to evaluate the level of statistical agreement between responses over time, beyond chance. Asymptotic p -values were reported to determine the statistical significance of the kappa values. A significant level of p < 0.05 was considered statistically significant for all tests. Results In the initial test, there were 45 correct answers (90%) and 5 incorrect answers (10%) in the multiple-choice questions. In the true–false questions, 41 answers were correct (82%), while 9 were incorrect (18%). The overall average accuracy rate in the initial test was calculated as 86%. In the second test conducted after 14 days, there were 47 correct (94%) and 3 incorrect (6%) answers in the multiple-choice section, whereas the true–false section yielded 48 correct (96%) and 2 incorrect (4%) responses. The overall accuracy rate in the second test increased to 95%, showing an improvement from the initial 86% (Fig. 4 ). The increase in overall accuracy from 86 to 95% was statistically significant according to the two-proportion z -test ( p = 0.030). Fig. 4. Open in a new tab Comparison of ChatGPT-4o’s accuracy on true/false (TF) and multiple-choice (MC) questions over two time points According to the Wilcoxon Signed-Rank Test for related samples, there was a statistically significant improvement in accuracy scores for true/false questions between the first and second sessions ( Z = 2.490, p = 0.013, r = 0.35, 95% CI [0.09, 0.61]). Similarly, the comprehensiveness scores for true/false questions also showed a significant increase ( Z = 2.460, p = 0.014, r = 0.35, 95% CI [0.09, 0.61]). However, there were no statistically significant differences observed in the accuracy ( Z = 1.342, p = 0.180, r = 0.19, 95% CI [− 0.09, 0.47]) or comprehensiveness scores ( Z = 1.342, p = 0 0.180, r = 0.19, 95% CI [− 0.09, 0.47]) for multiple-choice questions between the two sessions. To examine whether question format affected ChatGPT-4o’s performance, Mann–Whitney U tests were conducted. The comparisons revealed no significant differences in accuracy ( p = 0.661, r =  − 0.04, 95% CI [− 0.23, 0.16]) or comprehensiveness scores ( p = 0.708, r =  − 0.04, 95% CI [− 0.23, 0.16]) between multiple-choice and true/false questions in the second session. In the first session, the difference in comprehensiveness scores between the two formats approached statistical significance but did not reach the threshold ( p = 0.102, r = 0.16, 95% CI [− 0.04, 0.35]), and no significant difference was found in accuracy scores ( p = 0.260, r = 0.11, 95% CI [− 0.08, 0.29]). According to the results of Cohen’s Kappa analysis, a significant level of agreement was observed between the answers given by ChatGPT-4o to the same questions in two different sessions. Especially for multiple-choice questions, a high level of consistency was found in terms of accuracy rate ( κ = 0.735) and comprehensiveness level ( κ = 0.735). On the other hand, for true/false questions, low agreement was obtained for accuracy ( κ = 0.242) and moderate agreement for comprehensiveness ( κ = 0.488). Cohen’s Kappa analyses revealed statistically significant agreement across all comparisons ( p < 0.001). The strength of agreement ranged from low ( κ = 0.242, 95% CI [− 0.038, 0.522]) to substantial ( κ = 0.735, 95% CI [0.386, 1.084]) (Table 2 ). Table 2. Statistical comparison of ChatGPT performance Comparison Measurement type Test statistic p -value 1st–2nd test Accuracy (true/false) Z = 2.490 0.013 1st–2nd test Comprehensiveness (true/false) Z = 2.460 0.014 1st–2nd test Comprehensiveness (multiple choice) Z = 1.342 0.18 1st–2nd test Accuracy (multiple choice) Z = 1.342 0.18 Question format Comprehensiveness (2nd test) U = 1226 Z =  − 0.374 0.708 Question format Accuracy (2nd test) U = 1226 Z =  − 0.438 0.661 Question format Comprehensiveness (1st test) U = 1099 Z =  − 1.633 0.102 Question format Accuracy (1st test) U = 1151.5 Z =  − 1.126 0.260 Open in a new tab Dependent group comparisons were analyzed using the Wilcoxon signed-rank test, while independent group comparisons were assessed with the Mann–Whitney U test. Statistical significance was defined as * p * <.05 Discussion In our study, we aimed to investigate the potential, reliability, and limitations of ChatGPT-4o in providing clinical decision support in the field of breast radiology. In the first session, the model achieved an overall accuracy rate of 86%, with a 90% accuracy rate for multiple-choice questions and an 82% accuracy rate for true/false questions. In the second session, conducted 14 days later, the overall accuracy rate increased to 95%, with a 94% accuracy rate for multiple-choice questions and a 96% accuracy rate for true/false questions. A statistically significant improvement was observed in the accuracy and comprehensiveness scores of true/false questions. While no significant differences were found between question formats in the second session, Cohen’s kappa analysis revealed high consistency in multiple-choice answers and moderate to low consistency in true/false answers across sessions. The analyses showed that ChatGPT-4o’s responses to both multiple-choice and true/false questions were evaluated with a high level of accuracy and comprehensiveness. For the 50 multiple-choice questions asked by ChatGPT, the initial accuracy rate was 90% and 94% after 14 days; for the 50 true–false questions, the initial accuracy rate was 82% and 96% after 14 days. In the evaluation conducted during the first session, 3 mammography and 2 general category questions among multiple-choice questions and 5 mammography, 3 MRI, and 1 general question among true–false questions were answered incorrectly. In the second session, 2 general and 1 mammography questions among multiple-choice questions, and 1 mammography and 1 MR question among true–false questions were answered incorrectly. In November 2022, OpenAI introduced “ChatGPT” (Generative Pre-trained Transformer) as a chatbot. Alongside the large language models of OpenAI’s GPT-3 family, ChatGPT utilizes supervised learning and reinforcement learning techniques [ 8 , 14 ]. Due to its ability to provide accurate and well-formulated responses to queries, ChatGPT is being increasingly adopted across various fields and by a growing number of users. In the healthcare domain, numerous studies have explored the potential applications of ChatGPT, including clinical decision support, offering recommendations for cancer screenings, answering medical questions, generating radiology reports, clinical notes, and discharge summaries [ 8 ]. In a recent study, Güneş et al. evaluated the performance of large language models (LLMs), including OpenAI ChatGPT 4 and 3.5, Google Gemini 1.5 Pro and 1.0, Microsoft Copilot, Perplexity, Claude 3 Opus, and Claude 3 Opus 200 K, in breast radiology using text-based and visual questions aligned with the BI-RADS Atlas 5th Edition. Their findings revealed that LLMs demonstrated high performance in text-based questions; however, their performance was not as successful with visual questions. Among the LLMs, the highest accuracy rates in text-based questions were achieved by Claude 3.5 Sonnet (90%) and ChatGPT 4 (89%) [ 13 ]. Unlike in our study, in this study, the questions were asked only once and the consistency of ChatGPT responses over time was not tested. Similar studies conducted in different fields have also been reported in the literature. Deniz et al. evaluated ChatGPT-4.0 by asking 50 multiple-choice and 50 true/false questions based on the ETA thyroid nodule management guidelines, with a 14-day interval between evaluations [ 15 ]. Similarly, Geneş et al. used the European Society of Cardiology guidelines to assess ChatGPT-4.0’s performance on 50 multiple-choice and 50 true/false questions related to acute coronary syndrome, with a 30-day interval between tests [ 16 ]. In both studies, some questions that were correctly answered in the initial evaluation were answered incorrectly in the subsequent test. Despite minor fluctuations, ChatGPT-4.0 demonstrated a high level of accuracy overall. In our study, we utilized ChatGPT-4.0 to evaluate the performance of AI. Unlike earlier versions that relied solely on their internal databases and did not perform web searches, the ChatGPT-4.0 model has enhanced its web browsing capabilities since May 2024 [ 17 ]. In our study, we observed a significant reduction in the rate of incorrect responses in the second evaluation conducted 14 days after the initial assessment. Unlike previous similar studies, no instances were found where initially correct responses were later marked as incorrect in the subsequent test. During the period in which our study was conducted, OpenAI implemented several updates to ChatGPT following the initial test. The improvement in accuracy observed in the second test may have been influenced by updates made to the web browsing feature [ 17 , 18 ]. Several studies have evaluated the performance of ChatGPT in different formats regarding breast cancer. In a study conducted by Rao et al., ChatGPT-3.5 and ChatGPT-4 were compared using open-ended and select-all-that-apply question formats based on the ACR appropriateness criteria for breast cancer screening and breast pain. The results showed that ChatGPT-4 achieved an accuracy rate of 98.4% in breast cancer screening recommendations and 77.7% in breast pain recommendations, whereas ChatGPT-3.5 achieved an accuracy rate of 88.9% and 58.3%, respectively [ 19 ]. Similarly, in a study conducted by Haver et al., ChatGPT was tested with 25 questions related to breast cancer prevention and screening, based on the BI-RADS Atlas and clinical experience. The results demonstrated that ChatGPT provided correct answers with an overall accuracy of 88% [ 9 ]. The potential applications of AI models in the field of radiology are extensive and are expected to expand further with ongoing advancements. The integration of AI models into clinical practice represents an exciting and rapidly evolving area of research. In clinical settings, these models hold substantial promises in supporting decision-making processes. By analyzing patient symptoms, medical history, and diagnostic test results, AI systems can provide clinicians with meaningful and actionable recommendations. In radiology and other medical specialties, AI can assist in differential diagnosis and treatment planning, thereby enhancing diagnostic accuracy. With ongoing technological advancements, it is anticipated that ChatGPT and other AI models will enhance workflow efficiency and improve diagnostic accuracy in clinical practice. Furthermore, the ability of AI to rapidly process and synthesize medical literature offers significant time-saving benefits for healthcare professionals while ensuring access to up-to-date clinical information [ 20 , 21 ]. Moreover, AI models such as ChatGPT can synthesize questions and generate textual content on specific topics to support radiology education and training. In this role, they can function as virtual educators, providing valuable support to healthcare professionals who are still in the early stages of their training [ 22 ]. With the advancements in AI, studies have also been conducted to investigate the limitations and drawbacks of these models. It has been observed that such models may generate outputs that are inconsistent with the provided sources or semantically meaningless. Researchers have termed these undesired outputs as “hallucinations.” The primary cause of hallucinations is the mismatch between the source and the reference, which can stem from two main factors. The first factor arises as a byproduct of heuristic data collection processes, where data inconsistencies or biases may occur. The second factor is referred to as inherent misalignment, which naturally occurs in some Natural Language Generation (NLG) tasks. Certain NLG tasks, by their nature, do not always require perfect information alignment or factual consistency between the source text and the target reference. This issue is particularly prevalent in tasks where diversity is essential. Moreover, if users lack sufficient knowledge on the subject, they may fail to detect such hallucinations [ 23 ] to mitigate and overcome AI-generated hallucinations in chatbot applications, it is crucial to train and test systems using diverse and representative datasets. Additionally, implementing monitoring and detection methods, such as human evaluation or anomaly detection, can contribute to reducing hallucination occurrences [ 24 ]. Although ChatGPT presents promising potential, several concerns persist regarding its application in clinical practice. First, since ChatGPT is not specifically designed to answer medical queries, it lacks the necessary medical expertise and background required to comprehend the complex relationships between diseases and treatments. This limitation raises concerns about the reliability and accuracy of the generated medical information. Second, the training data used in earlier versions of ChatGPT were not up to date. Given the rapid advancements in medical research and treatment protocols, such information gaps could potentially impact clinical decision-making and patient care [ 24 ]. However, recent updates in newer versions have addressed this limitation to some extent by incorporating web-based search capabilities, enabling access to more current information [ 18 ]. Third, ChatGPT currently relies on manual data input. While the integration of electronic health records into the system automatically is a future possibility, this advancement raises significant concerns regarding patient privacy and data security. Ensuring compliance with data protection regulations and maintaining confidentiality will be critical challenges in the implementation of AI-assisted healthcare solutions [ 8 ]. Study Limitations In our study, we utilized 50 multiple-choice and 50 true–false questions; however, the number of questions may not be sufficient to comprehensively evaluate the BI-RADS atlas and the breast cancer screening program. Although the responses were assessed in terms of adequacy, the absence of open-ended questions limited a thorough evaluation of ChatGPT’s potential for clinical application. The lack of scenario-based questions limited our assessment of the effectiveness of ChatGPT in clinical use. In addition, in future studies, comparison of ChatGPT responses with radiologists or clinicians will provide more meaningful results. Furthermore, the 14-day period used to assess ChatGPT’s learning capacity may not have been sufficient. Future studies with longer intervals or post-update assessments may provide more insightful evaluations of ChatGPT’s learning potential. Another limitation is the lack of transparency in ChatGPT’s training data. Prior to conducting this study, it was not possible to ascertain whether ChatGPT had been trained specifically on the BI-RADS atlas and breast cancer screening program. Different prompts could potentially yield varying results, highlighting the need for further exploration of prompt engineering techniques. Conclusion ChatGPT-4o demonstrated a remarkable performance in answering text-based multiple-choice and true–false questions related to the BI-RADS atlas and the ACR breast cancer screening program. With the continuous advancements in AI technologies and ongoing improvements, AI-based models are expected to play an increasingly significant role in healthcare services. However, considering the limitations, potential drawbacks, and ethical concerns associated with AI models, particularly the phenomenon of hallucinations—it is crucial that these systems remain under the supervision of expert physicians to ensure their safe and effective integration into clinical practice. Abbreviations BI_RADS Breast Imaging Reporting and Data System ACR American College of Radiology AI Artificial intelligence NLG Natural Language Generation GPT Generative Pre-trained Transformer Author Contribution All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Bilgen Mehpare Özer and Eda Nur Korkmaz. The first draft of the manuscript was written by Bilgen Mehpare Özer, and all authors commented on previous versions of the manuscript. All authors read and approve the final manuscript. Funding The authors declare that no funds, grants, or other support were received during the preparation of this manuscript. Data Availability The data supporting this study’s findings are available from the corresponding author upon reasonable request. Declarations Ethics Approval Since this study does not involve human or animal subjects, ethical approval and informed consent were deemed unnecessary. Consent to Participate This study does not involve human subjects. Consent for Publication This study does not involve human subjects. Competing interests The authors declare no competing interests. Footnotes Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References 1. Wilkinson, L., Gathani, T.: Understanding breast cancer as a global health concern. Br J Radiol. 95 (1130), 20211033 (2022). 10.1259/bjr.20211033 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Autier, P., Boniol, M.: Mammography screening: A major issue in medicine. Eur J Cancer. 90 , 34–62 (2018). 10.1016/j.ejca.2018.06.027 [ DOI ] [ PubMed ] [ Google Scholar ] 3. Kashyap, D., et al.: [Retracted] Global increase in breast cancer incidence: Risk factors and preventive measures. Biomed Res Int. 2022 (1), 9605439 (2022). 10.1155/2022/9605439 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [ Retracted ] 4. American College of Radiology, D’Orsi, C.J., Sickles, E.A., Mendelson, E.B., Morris, E.A. (eds): ACR BI-RADS Atlas: Breast Imaging Reporting and Data System; Mammography, Ultrasound, Magnetic Resonance Imaging, Follow-up and Outcome Monitoring, Data Dictionary . 5th edn. American College of Radiology, Reston, VA (2013) 5. Prvulović-Bunović, N., et al.: Breast imaging reporting and data system (BI-RADS): Why don’t we use it correctly in the region? Arch Oncol. 00 , 8–8 (2024). 10.2298/AOO230207008P [ Google Scholar ] 6. American College of Radiology: New ACR breast cancer screening guidelines for high risk women. Media Center (May 3, 2023). American College of Radiology calls for earlier and more intensive screening and risk assessment by age 25 for high risk women, particularly Black and Ashkenazi Jewish populations. https://www.acr.org/Media-Center/ACR-News-Releases/2023/New-ACR-Breast-Cancer-Screening-Guidelines-for-High-Risk-Women 7. Kamdar, J., Jeba Praba, J., Georrge, J.J.: Artificial intelligence in medical diagnosis: Methods, algorithms and applications. In: Machine Learning with Health Care Perspective: Machine Learning and Healthcare , pp. 27–37. Springer, Berlin/Heidelberg (2020). 10.1007/978-3-030-40850-3_2 8. 8) Liu, J., Wang, C., Liu, S.: Utility of ChatGPT in clinical practice. J Med Internet Res. 25 , e48568 (2023). 10.2196/48568 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 9. Haver, H.L., Ambinder, E.B., Bahl, M., Oluyemi, E.T., Jeudy, J., Yi, P.H.: Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology. 307 (4), e230424 (2023). 10.1148/radiol.230424 [ DOI ] [ PubMed ] [ Google Scholar ] 10. Cozzi, A., Pinker, K., Hidber, A., Zhang, T., Bonomo, L., Lo Gullo, R., et al.: BI-RADS category assignments by GPT-3.5, GPT-4, and Google Bard: a multilanguage study. Radiology. 311 (1), e232133 (2024). 10.1148/radiol.232133 [ DOI ] [ PMC free article ] [ PubMed ] 11. Irmici, G., Martella, E., Miccichè, S., Pugliese, G., Squillaci, E., Valentini, V.: How do large language models answer breast cancer quiz questions? A comparative study of GPT-3.5, GPT-4 and Google Gemini. Radiol Med. 129 (8), 781–789 (2024). 10.1007/s11547-024-01872-1 [ DOI ] [ PubMed ] 12. Almeida, L.C., Farina, E.M., Kuriki, P.E., Abdala, N., Kitamura, F.C.: Performance of ChatGPT on the Brazilian radiology and diagnostic imaging and mammography board examinations. Radiol Artif Intell. 6 (1), e230103 (2023). 10.1148/ryai.230103 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Güneş, Y.C., et al.: Evaluating text and visual diagnostic capabilities of large language models on questions related to the Breast Imaging Reporting and Data System Atlas 5th edition. Diagn Interv Radiol. Published Online First: 9 (2024). 10.5152/dir.2024.20240099 [ DOI ] [ PMC free article ] [ PubMed ] 14. Saeidnia, H.: Open AI, ChatGPT: to be, or not to be, that is the question. Inf Matters. 3 (6) (2023). 10.2139/ssrn.4424843 15. Deniz, M.S., Guler, B.Y.: Assessment of ChatGPT’s adherence to ETA-thyroid nodule management guideline over two different time intervals 14 days apart: in binary and multiple-choice queries. Endocrine. 1–9 (2024). 10.1007/s12020-024-03450-6 [ DOI ] [ PubMed ] 16. Geneş, M., Çelik, M.: Assessment of ChatGPT’s compliance with ESC-acute coronary syndrome management guidelines at 30-day intervals. Life. 14 (10), 1235 (2024). 10.3390/life14101235 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. OpenAI: Introducing ChatGPT-4. (2023). https://openai.com 18. OpenAI: GPT-4o model. (2024). https://openai.com 19. Rao, A., et al.: Evaluating GPT as an adjunct for radiologic decision making: GPT-4 versus GPT-3.5 in a breast imaging pilot. J Am Coll Radiol. 20 (10), 990–997 (2023). 10.1016/j.jacr.2023.06.003 [ DOI ] [ PMC free article ] [ PubMed ] 20. Iqbal, U., Tanweer, A., Rahmanti, A.R., Greenfield, D., Lee, L.T.J., Li, Y.C.J.: Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis. J Biomed Sci. 32 (1), 1–19 (2025). 10.1186/s12929-025-01131-z [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 21. Zhang, K., Meng, X., Yan, X., Ji, J., Liu, J., Xu, H., et al.: Revolutionizing health care: The transformative impact of large language models in medicine. J Med Internet Res. 27 , e59069 (2025). 10.2196/59069 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 22. Temperley, H.C., O’Sullivan, N.J., Mac Curtain, B.M., Corr, A., Meaney, J.F., Kelly, M.E., Brennan, I.: Current applications and future potential of ChatGPT in radiology: A systematic review. J Med Imaging Radiat Oncol. 68 (3), 257–264 (2024). 10.1111/1754-9485.13621 [ DOI ] [ PubMed ] [ Google Scholar ] 23. Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Comput Surv. 55 (12), 1–38 (2023). 10.1145/3585360 [ Google Scholar ] 24. Alkaissi, H., McFarlane, S.I.: Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus. 15 (2) (2023). 10.7759/cureus.37591 [ DOI ] [ PMC free article ] [ PubMed ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement The data supporting this study’s findings are available from the corresponding author upon reasonable request. Articles from Journal of Imaging Informatics in Medicine are provided here courtesy of Springer ACTIONS View on publisher site PDF (1.4 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• PET Imaging to Determine the Role of PET in the Assessment of Regional Disease in Breast Cancer (PET PREDICT Trial)#881853
• A Study Evaluating the Efficacy and Safety of Multiple Treatment Combinations in Patients With Metastatic or Locally Advanced Breast Cancer#180328
• Identification of RPL4 as a biomarker associated with disulfidptosis and arachidonic acid metabolism in breast cancer.#425143
• Phase II Study of Fluorine-18 3'-Deoxy-3'-Fluorothymidine (F-18-FLT) in Invasive Breast Cancer#857845
• The YAP-TEAD4 inhibitor M511-0965 suppresses triple-negative breast cancer progression by downregulating TGFB2.#334998