Performance of AI-Chatbots to Common Temporomandibular Joint Disorders (TMDs) Patient Queries: Accuracy, Completeness, Reliability and Readability.

Performance of AI-Chatbots to Common Temporomandibular Joint Disorders (TMDs) Patient Queries: Accuracy, Completeness, Reliability, and Readability - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Orthod Craniofac Res . Author manuscript; available in PMC: 2026 Jun 3. Published in final edited form as: Orthod Craniofac Res. 2025 May 7;28(Suppl 1):S127–S134. doi: 10.1111/ocr.12939 Performance of AI-Chatbots to Common Temporomandibular Joint Disorders (TMDs) Patient Queries: Accuracy, Completeness, Reliability, and Readability Mohamed G Hassan 1, *, $ , Ahmed A Abdelaziz 2, $ , Hams H Abdelrahman 3 , Mostafa M Y Mohamed 4 , Mohamed T Ellabban 2 Author information Article notes Copyright and License information 1- Division of Bone and Mineral Diseases, Department of Medicine, School of Medicine, Washington University in St. Louis, St. Louis, MO, USA 2- Department of Orthodontics, Faculty of Dentistry, Assiut University, Assiut, Egypt 3- Department of Pediatric Dentistry and Dental Public Health, Faculty of Dentistry, Alexandria University, Alexandria, Egypt 4- Department of Oral Radiology, Faculty of Dentistry, Zarqa University, Zarqa, Jordan $ Authors contributed equally Authors Contribution : MGH: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing - original draft, review & editing. AAA: Formal analysis, Investigation, Methodology, Data Collection, Validation, Writing - original draft, review & editing. HHA: Consultation, Data Analysis, Biostatistical Analysis, Validation, Writing - original draft, review & editing. MYM: Data Collection, Investigation, Methodology, Writing – original draft, review & editing. MTE: Data Collection, Investigation, Methodology, Writing – original draft, review & editing. * Correspondence to: Mohamed G. Hassan PhD, MSc, BDS, Division of Bone and Mineral Diseases, Department of Medicine, School of Medicine, Washington University in St. Louis, MO, USA. hmohamed@wustl.edu Issue date 2025 Dec. PMC Copyright notice PMCID: PMC13228120  NIHMSID: NIHMS2079822  PMID: 40332142 The publisher's version of this article is available at Orthod Craniofac Res Abstract Background: TMDs are a common group of conditions affecting the temporomandibular joint (TMJ) often resulting from factors like injury, stress, or teeth grinding. This study aimed to evaluate the accuracy, completeness, reliability, and the readability of the responses generated by ChatGPT-3.5, −4o and Google Gemini to TMDs-related inquiries. Material and Methods: Forty-five questions covering various aspects of TMDs were created by two experts and submitted by one author to ChatGPT-3.5, ChatGPT-4, and Google Gemini on the same day. The responses were evaluated for accuracy, completeness, and reliability using modified Likert scales. Readability was analyzed with six validated indices via a specialized tool. Additional features, such as the inclusion of graphical elements, references, and safeguard mechanisms, were also documented and analyzed. The Pearson Chi-Square and One-Way ANOVA tests were used for data analysis. Results: Google Gemini achieved the highest accuracy, providing 100% correct responses, followed by ChatGPT-3.5 (95.6%) and ChatGPT-4o (93.3%). ChatGPT-4o provided the most complete responses (91.1%), followed by ChatGPT-03 (64.4%) and Google Gemini (42.2%). The majority of responses were reliable, with ChatGPT-4o at 93.3% “Absolutely Reliable,” compared to 46.7% for ChatGPT-3.5 and 48.9% for Google Gemini. Both ChatGPT-4o and Google Gemini included references in responses, 22.2% and 13.3% respectively, while ChatGPT-3.5 included none. Google Gemini was the only model that included multimedia (6.7%). Readability scores were highest for ChatGPT-3.5, suggesting its responses were more complex than those of Google Gemini and ChatGPT-4o. Conclusion: Both ChatGPT-4o and Google Gemini demonstrated accuracy and reliability in addressing TMD-related questions, with their responses being clear, easy to understand, and complemented by safeguard statements encouraging specialist consultation. However, both platforms lacked evidence-based references. Only Google Gemini incorporated multimedia elements to its answers. Keywords: Temporomandibular Joint, Temporomandibular Joint Disorders, Chatbot, ChatGPT, Google Gemini, Education, Artificial Intelligence Introduction Temporomandibular joint dysfunction (TMD) is a complex condition that presents a range of symptoms affecting the temporomandibular joints (TMJs) and masticatory muscles, resulting in pain and functional problems. (Ref) The prevalence of TMD symptoms in adults ranges from 3% to 49.9%, and 12% to 67% among children. [ 1 – 3 ] TMD is more prevalent in women compared to men, particularly in the 30–39 years age group. [ 4 – 6 ] The wide range is due to several sociodemographic, psychological, and genetic factors, in addition to the use of different diagnostic tools and criteria in diagnosing TMD. The etiology of TMD is widely regarded as multifactorial, involving biological, behavioral, and psychosocial factors that may act independently or interactively. [ 7 – 9 ] While established etiological factors for temporomandibular disorders (TMD) are well-documented, certain lifestyle habits have been recognized as contributors to the onset or exacerbation of TMD symptoms. For instance, Lei J. et al. demonstrated a moderate to strong correlation across age groups between stress, depression, anxiety, and TMD. Furthermore, patients with painful TMD often experience heightened stress levels and a diminished quality of life, largely due to the disorder’s symptoms. [ 10 ] Given the complexity of these disorders and the variability in treatment modalities, patient education and access to reliable information are critical and significantly improve oral behaviors in patients with TMJ disorders. [ 11 , 12 ] In this context, social media, and chatbot platforms represent an emerging tool with the potential to bridge gaps in patient understanding and support. [ 13 – 15 ] Global internet access has grown exponentially, with 68% of adult users reportedly seeking health-related information online. [ 16 ] Artificial intelligence (AI) has made remarkable progress in the past three years, particularly in healthcare applications. AI tools utilizing natural language processing (NLP) systems, such as ChatGPT, have gained increasing attention for their potential roles in healthcare settings, including patients and healthcare providers. [ 17 ] Despite not being crafted for medical use, chatbots like ChatGPT and Google Gemini have been increasingly utilized to assist healthcare providers in formulating differential diagnoses. At the same time, recent studies highlighted the increasing trend toward using these tools to help patients receive more tailored information to self-diagnose conditions before seeking medical attention. [ 18 , 19 ] Currently, the clinical decision-making process has witnessed a paradigm shift from a traditional patient-centered model to an AI-assisted decision-making approach, where AI tools actively participate in tailoring personalized treatment. [ 20 – 22 ] For this reason, accessing, comprehending, and using medical information online became essential for enabling patients to make informed decisions. [ 23 ] However, significant concerns remain about the accuracy and consistency of AI-generated information, particularly in providing diagnosis and/or treatment suggestions. [ 24 ] Therefore, this study aimed to evaluate and compare the accuracy, completeness, and reliability of answers to commonly asked TMDs-related questions provided by ChatGPT (both free and paid versions) and Google Gemini. The analysis also included the platforms’ ability to identify and respond to both open- and close-ended questions, their recommendations to consult specialists and the inclusion of additional references or graphical elements. Furthermore, we quantitatively assessed the readability of the information generated by both platforms. Overall, ChatGPT and Google Gemini provided accurate, reliable TMDs-related information. However, inconsistencies were noted, such as the depth of detail, inclusion of references, and graphical elements, which varied across platforms. Material and Methods Study design: Ethical approval was not required for this study as it utilized publicly available information and did not involve animal models, human participants, identifiable data, or interactions requiring institutional review board approval. On September 1st, 2024, One author (MGH) submitted 45 common patient queries related to TMDs to the 3 AI chatbots (ChatGPT-3.5, −4o, and Google Gemini). These questions were derived from prior research related to TMJ and TMDs. [ 25 – 31 ] The 45 questions were created and tailored to cover different aspects of TMDs like signs and symptoms, diet and stress contribution to TMDs, the role of physical therapy, the use of medications, surgical intervention, orthodontic appliances, and complications of untreated TMDs. They also cover different treatment modalities. ( Supplemental File 1 ). The validity of the questions was assessed as described previously. [ 32 , 33 ] Briefly, five experts evaluated the content validity of each question using a 4-point Likert scale ranging from “not relevant” to “highly relevant.” To calculate the item-level content validity index (CVI-I), the number of experts who rated an item as either “quite relevant (score 3)” or “highly relevant (score 4)” was divided by the total number of experts. The resulting CVI-I score was 0.92, which was acceptable. A full list of questions and answers by each platform is included in Supplemental File 1 . Data collection and content analysis: The responses generated by the three platforms were independently analyzed for accuracy, completeness, and reliability by three reviewers (AAA, MTE, and MGH). Each reviewer has over 10 years of experience in clinical orthodontics. Reviewer calibration was performed by independently evaluating the same set of 10 questions on two separate occasions, with a two-week interval between evaluations, followed by an assessment of inter- and intra-examiner reliability using Kappa statistics. The resulting values ranged from 0.761 to 0.809, indicating an acceptable level of agreement. Accuracy, completeness, and reliability were assessed by comparing the responses to established clinical guidelines and evidence-based literature in orthodontics. [ 34 – 36 ] For accuracy, a 3-point scale was used, where responses were scored as follows: 0 for “completely inaccurate,” 1 for “partially accurate,” and 2 for “completely accurate.” The mean accuracy score was calculated across all responses for each platform, providing a quantitative measure of their performance. Similarly, completeness was evaluated by assessing whether each response included all key components necessary to comprehensively address the question. A 3-point scale was used for completeness, with scores of 0 for “incomplete,” 1 for “partially complete,” and 2 for “complete.” The completeness scores were then averaged to assess each platform’s ability to generate thorough and detailed answers. Finally, we evaluated reliability to assess the consistency and dependability of the information provided by each platform. Using a previously established method [ 32 , 37 , 38 ], each response was assigned a score on a 5-point scale, ranging from 1 (“unreliable”) to 5 (“highly reliable”). ( Supplemental File 2 ). The mean reliability score for each platform was calculated to measure the stability and dependability of the responses. In addition to evaluating the accuracy, completeness, and reliability of TMDs-related information provided by ChatGPT and Google Gemini, the reviewers conducted a detailed analysis of the responses. They considered several key criteria, like 1) the number of definitive answers provided to closed-ended questions; 2) the frequency of responses recommending specialist consultation; 3) the inclusion of supporting citations; and 4) the presence of graphical elements, such as videos, images, or figures, accompanying the text-based responses. The readability of responses was assessed using six validated readability indices through the online website [ https://www.webfx.com/tools/read-able/ ] which objectively measures the comprehensibility of written materials. [ 39 ] The Flesch-Kincaid system comprises two parameters: the Grade Level score (FKGL), which represents the U.S. education grade necessary to understand the text, and the Reading Ease score (FRES), a scale from 1 to 100, where higher scores correspond to greater readability. The Gunning Fog Index (GFOG) estimates the years of formal education required for a reader to understand a text on the first attempt, producing a grade level ranging from 0 to 20. This calculation is based on the frequency of polysyllabic words and the average sentence length within a 100-word sample. In comparison, the Simple Measure of Gobbledygook (SMOG) employs a similar methodology but applies it to the entire text. The Coleman-Liau Index (CLI) evaluates the grade level by analyzing sentence length and character count. In contrast, the Automated Readability Index (ARI) emphasizes character count per word rather than syllable count, with longer words being more complex and difficult to read. While a higher score on the Flesch-Kincaid Reading Ease reflects greater readability, higher scores on the other indices signify increased complexity and difficulty. [ 40 , 41 ] Statistical analysis The distribution of correctness, completeness, reliability scores, and response characteristics and features was presented using frequency and percentages and was analyzed using the Pearson chi-square test. The normality of readability scores was confirmed through Kolmogorov-Smirnov and Q-Q plots, allowing for using One-Way ANOVA followed by Tukey’s post hoc test with Bonferroni correction for type I error correction. All tests were two-tailed, and a significance level of p < 0.05 was applied. Data analysis was conducted using IBM SPSS for Windows version 28, Armonk, NY, USA. Results The results show that all three models performed well in responding to 45 TMD-related queries, with no statistically significant differences observed (p = 0.234). Although it wasn’t statistically significant, Google Gemini achieved the highest accuracy at 100% (45/45 correct answers) compared to ChatGPT-3.5 (95.6%) and ChatGPT-4o (93.3%) [ Table 1 ]. Table 1: Comparison of accuracy of TMD responses by ChatGPT-3.5, ChatGPT-4o, and Google Gemini Accuracy Scores ChatGPT-3.5 (n=45) ChatGPT-4o (n=45) Google Gemini (n=45) p value 1 n (%) Incorrect (0) 0 (0%) 0 (0%) 0 (0%) 0.234 Partially correct (1) 2 (4.4%) 3 (6.7%) 0 (0%) Correct (2) 43 (95.6%) 42 (93.3%) 45 (100%) Open in a new tab 1 p value: Pearson Chi-Square test ChatGPT-4o provided more complete responses (91.1%), compared to ChatGPT-3.5 (64.4%) and Google Gemini (42.2%). Conversely, ChatGPT-3.5 and Google Gemini exhibited higher proportions of partially complete responses, at 33.3% and 48.9% respectively, versus 8.9% for ChatGPT-4o. Overall, the performance of ChatGPT-4o was significantly superior to that of ChatGPT-3.5 ( p = 0.027) and Google Gemini ( p < 0.001) [ Table 2 ]. Table 2: Comparison of completeness of TMD responses by ChatGPT-3.5, ChatGPT-4o, and Google Gemini Completeness scores ChatGPT-3.5 (n=45) ChatGPT-4o (n=45) Google Gemini (n=45) p value 1 n (%) Incomplete (0) 1 (2.2%) 0 (0%) 4 (8.9%) <0.001 * Partially complete (1) 15 (33.3%) 4 (8.9%) 22 (48.9%) Complete (2) 29 (64.4%) 41 (91.1%) 19 (42.2%) Pairwise comparisons P1=0.027 * , P2= 0.058, P3<0.001 * Open in a new tab * Statistically significant difference at p value<0.05, P1: comparison between ChaGPT-3.5 and ChatGPT-4o, P2: comparison between ChatGPT-3.5 and Google Gemini, P3: comparison between ChatGPT-4o and Google Gemini 1 p value: Pearson Chi Square test The reliability assessment revealed that ChatGPT-4o achieved the highest level of performance, with 93.3% of its responses classified as “Absolutely Reliable.” In comparison, only 46.7% of ChatGPT-3.5’s responses and 48.9% of Google Gemini’s responses received the same rating. The remaining responses from ChatGPT-3.5 and Google Gemini were categorized as “Highly Reliable,” accounting for 46.7% in both cases. Notably, ChatGPT-3.5 was the only model to produce “Questionable” responses, comprising 4.4% of its total outputs. Overall, the reliability of ChatGPT-4o was significantly higher than that of both ChatGPT-03 (p < 0.001) and Google Gemini (p < 0.001) [ Table 3 ]. Table 3: Comparison of reliability of TMD responses by ChatGPT-3.5, ChatGPT-4o, and Google Gemini Reliability scores ChatGPT-3.5 (n=45) ChatGPT-4o (n=45) Google Gemini (n=45) p value 1 n (%) Unreliable (0) 0 (0%) 0 (0%) 0 (0%) <0.001 * Questionable (1) 2 (4.4%) 0 (0%) 0 (0%) Reliable (2) 1 (2.2%) 2 (4.4%) 2 (4.4%) Highly reliable (3) 21 (46.7%) 1 (2.2%) 21 (46.7%) Absolutely reliable (4) 21 (46.7%) 42 (93.3%) 22 (48.9%) Pairwise comparisons P1<0.001 * , P2=1.00, P3<0.001 * Open in a new tab * Statistically significant difference at p value<0.05, P1: comparison between ChaGPT-3.5 and ChatGPT-4o, P2: comparison between ChatGPT-3.5 and Google Gemini, P3: comparison between ChatGPT-4o and Google Gemini 1 p value: Pearson Chi Square test The highest rate of definitive responses was observed in ChatGPT-4o at 35.6%, followed by Google Gemini at 22.2% and ChatGPT-03 at 20%, though the difference was not statistically significant (p = 0.190). For recommending the consultation of a specialist, Google Gemini led with 88.9%, followed by ChatGPT-03 (82.2%) and ChatGPT-04 (80%), with no significant differences (p = 0.494). Compared to ChatGPT-03, ChatGPT-04 and Google Gemini demonstrated a significantly higher rate of responses including references (22.2% and 13.3%, respectively) than ChatGPT-03, which had none (0%). Finally, Google Gemini was the only platform to incorporate multimedia in its responses, with 6.7% of its answers featuring multimedia elements. [ Table 4 ] Table 4: Comparison of characteristics and features of TMD responses by ChatGPT-3.5, ChatGPT-4o, and Google Gemini ChatGPT-3.5 (n=45) ChatGPT-4o (n=45) Google Gemini (n=45) p value 1 Affirmative responses Yes 9 (20%) 16 (35.6%) 10 (22.2%) 0.190 No 36 (80%) 29 (64.4%) 35 (77.8%) Recommending physician Yes 37 (82.2%) 36 (80%) 40 (88.9%) 0.494 No 8 (17.8%) 9 (20%) 5 (11.1%) Providing references Yes 0 (0%) 10 (22.2%) 6 (13.3%) 0.005 * No 45 (100%) 35 (77.8%) 39 (86.7%) Pairwise comparisons P1= 0.001 * , P2=0.011 * , P3= 0.270 Including Multimedia Yes 0 (0%) 0 (0%) 3 (6.7%) 0.047 * No 45 (100%) 45 (100%) 42 (93.3%) Pairwise comparisons P1= 1.00, P2=0.078, P3=0.078 Open in a new tab * Statistically significant difference at p value<0.05, P1: comparison between ChaGPT-3.5 and ChatGPT-4o, P2: comparison between ChatGPT-3.5 and Google Gemini, P3: comparison between ChatGPT-4o and Google Gemini 1 p value: Pearson Chi Square test The readability scores across the three platforms (ChatGPT-3.5, ChatGPT-4o, and Google Gemini) reveal significant differences in the complexity of their responses. ChatGPT-3.5 consistently exhibited higher difficulty scores across all readability indices, indicating that its responses were more complex and required a higher level of education to understand. [ 42 ] In contrast, ChatGPT-4o and Google Gemini produced responses with significantly lower scores, suggesting that their content was more accessible and easier to read. Among the two platforms, ChatGPT-4o generally demonstrated slightly better readability [ Table 5 ]. Table 5: Comparison of readability scores between ChatGPT-3.5, ChatGPT-4o, and Google Gemini ChatGPT-3.5 (n=45) ChatGPT-4o (n=45) Google Gemini (n=45) p value 1 Pairwise comparisonsⱡ Mean ±SD FRE 22.75 ±13.79 34.82 ±10.97 36.96 ±10.81 <0.001* P1<0.001*, P2<0.00, P3=0.672 FGKL 15.37 ±2.45 12.36 ±1.80 13.31 ±2.17 <0.001* P1<0.001*, P2<0.00, P3=0.096 SMOG 15.94 ±2.56 13.90 ±1.57 14.55 ±1.86 <0.001* P1<0.001*, P2<0.00, P3=0.293 ARI 16.55 ±2.81 12.38 ±2.17 13.67 ±2.62 <0.001* P1<0.001*, P2<0.00, P3=0.046* CLI 16.92 ±2.37 14.59 ±2.07 13.51 ±1.92 <0.001* P1<0.001*, P2<0.00, P3=0.047* GFOG 17.34 ±2.81 14.49 ±2.29 15.46 ±2.62 <0.001* P1<0.001*, P2<0.00, P3=0.179 Open in a new tab FRES: Flesch Reading Ease Score, FGKL: Flesch-Kincaid Grade Level, SMOG: Simple Measure of Gobbledgook Index, ARI: Automated Readability Index, CLI: Coleman Liau Index, GFOG: Gunning Fog Score, P1: comparison between ChaGPT-3.5 and ChatGPT-4o, P2: comparison between ChatGPT-3.5 and Google Gemini, P3: comparison between ChatGPT-4o and Google Gemini 1 p value: One Way ANOVA test Discussion AI chatbots have become a commonly used tool and are largely integrated into our daily tasks. Their role in providing healthcare information, including dental advice, to patients is rapidly developing. Ensuring the accuracy, reliability, and readability of the dental information they provide is important to maintaining patient trust and promoting informed decision-making. This study indicated that both ChatGPT and Google Gemini can provide accurate, comprehensive, easily understandable responses to common inquiries related to temporomandibular disorders (TMDs). The two platforms provided information related to the causes, symptoms, and treatment options with a good level of language clarity. However, while responses were generally accurate, several discrepancies were noted, particularly in providing reliable references. In dental literature, different indices have been used to assess the accuracy of AI chatbot responses. [ 32 , 43 , 44 ] In this study, a modified three-point Likert scale was applied to analyze the accuracy and completeness of the responses. In addition, a five-point Likert scale, previously employed in a similar study, was used to assess the reliability of the chatbot’s response. [ 32 ] The combination of different scales in the study provides methodological flexibility, ensuring that the evaluation criteria are appropriately tailored to the measured outcome. Our results show that both ChatGPT and Google Gemini mostly provided accurate responses to the majority of questions about TMDs, with no differences between the platforms. There is an ongoing discussion in the dental and medical literature regarding the accuracy of information provided by AI chatbots in response to both patient inquiries and professional questions. Some studies highlight the potential of AI chatbots to deliver accurate, timely, and accessible information, particularly for common patient questions about treatments and procedures. For instance, Molena et al. reported that ChatGPT demonstrated good accuracy in responding to general dental questions. [ 45 ] Similarly, other studies have reported comparable levels of performance, ranging from acceptable to high accuracy, when evaluating various AI chatbots in responding to specialized queries related to orthodontics, oral medicine, oral surgery, and special needs dentistry. [ 43 , 46 – 49 ] However, despite these promising findings, other studies raise concerns about the potential inaccuracies in chatbot-generated responses, especially for complex professional inquiries, emphasizing the need for human oversight and validation. Abu Arqub et al. reported that the overall level of accuracy of ChatGPT responses to questions related to clear aligners was suboptimal. [ 44 ] Additional research has highlighted similar concerns across various dental specialties. For instance, Azadi et al concluded that the use of AI chatbots in oral and maxillofacial practice should be monitored since there are still some critical inaccuracies in their responses. [ 50 ] This controversy regarding the accuracy of dental information provided by AI chatbots could be related to several factors like; training and data update, topic and question specificity, knowledge update, and lack of clinical judgment. To summarize, AI chatbots hold great promise as educational tools for TMDs. Our results support the need for human oversight, continuous refinement of training datasets, and integration of real-time updating mechanisms to enhance their accuracy. The evaluation of the completeness of AI chatbot-generated responses is crucial for assessing the performance of these platforms. A complete response not only ensures that all aspects of a question are addressed, but also provides the necessary depth and context, especially when dealing with complex medical topics like TMDs. The results of this study showed that ChatGPT-4o provided more complete responses compared to ChatGPT-3.5 and Google Gemini. This is opposite to what had been reported previously. Daraqel et al. reported that both ChatGPT and Google Bard Chatbots showed an adequate performance related to completeness in responding to general orthodontic questions. [ 43 ] The difference between our results and the findings of Daraqel et al. may arise from differences in research methodologies, evaluation criteria, or the specific types of questions used in the study. Reliability assessment mirrored the trend observed in response completeness. ChatGPT-4 displayed the highest performance, with 93.3% of its responses being deemed “Absolutely Reliable.” ChatGPT-3.5 and Google Gemini still demonstrated high reliability of their responses, classified as “highly reliable”. These findings align with our previous observations, where these platforms consistently demonstrated high reliability in addressing orthognathic surgery questions. [ 32 ] For this reason, there is a need for standardized benchmarks to evaluate AI chatbot performance in biomedical research. For example, standardized frameworks and indices for assessing accuracy, completeness, and reliability would help resolve many discrepancies and provide clearer insights into the strengths and limitations of different platforms. The performance and use of AI chatbots don’t depend solely on the ability to provide accurate or complete information, but also on how well they communicate, assist, and engage with the end user. [ 43 , 51 ] According to the readability indices, our results show that ChatGPt-3.5 required at least a university degree to understand the answers. Conversely, ChatGPT-04 and Google Gemini generated text with significantly lower FGKL scores (12.36 and 13.31, respectively), which suggests that their output is more accessible, likely corresponding to a high school or early university level of reading difficulty. The results, however, indicate that the output from all three models—including ChatGPT-04 and Google Gemini—remains too advanced for the eighth-grade reading level of an average American adult and the sixth-grade reading level recommended for patient education materials. This is consistent with previous research, [ 47 , 52 , 53 ] which found that AI-generated responses often exceed the readability levels appropriate for public understanding, with responses typically ranging at the 11th-grade level or above. The readability challenges of chatbots may restrict their accessibility to the public. Therefore, it is possible to enhance the usability and effectiveness of language models by integrating clear, high-quality health information from expert-curated websites. Consistent with previous results, [ 32 , 43 ] our results showed that Google Gemini was the only platform capable of providing answers supported with multimedia files. While no direct correlation was observed between the inclusion of multimedia and the accuracy or reliability of the information presented, the integration of supporting materials such as images has the potential to improve user understanding. It can also facilitate the visualization of complex concepts, particularly within the domain of TMDs. In addition to multimedia support, the inclusion of reliable references is important for the trust and reliability of the provided information. Our findings showed that Google Gemini had a higher rate of responses including references compared to ChatGPT-3.5, which referenced none. Notably, around 55% ( Supplemental File 1 ) of Google Gemini responses included only reference numbers without hyperlinks. These results are in agreement with previous studies across various disciplines within clinical dentistry that highlighted ChatGPT’s deficiency in supporting responses with credible sources of information. [ 32 , 44 , 45 ] Furthermore, there are some concerns related to the quality of the available references, as many of the provided references were generic and not sourced from reliable sources of information. The limitations in the inclusion and quality of references, coupled with the potential for AI-Chatbot-generated medical information to lack clarity, highlight the need for robust safety mechanisms while requiring dental advice. Our findings revealed an improvement in both ChatGPT versions, as they consistently included prompts urging users to consult specialists at the end of their responses. This represents significant progress compared to earlier findings, which indicated that Google Gemini outperformed ChatGPT in implementing safety mechanisms for providing orthognathic surgery-related advice. [ 32 ] This reminder serves as a critical safeguard, stressing that AI chatbots are intended to facilitate, not replace, expert dental advice. This prompt enhances user safety by reducing the risk of misinterpretation while encouraging the thoughtful use of AI chatbots in life-threatening topics including medicine and dentistry. This study provides valuable insights into the performance of ChatGPT and Google Gemini, in providing TMDs-related information. However, several limitations should be discussed. First, the analysis was limited to responses generated in English, which restricts the generalizability of the findings. Second, the study collected responses at a single time point, limiting the ability to assess the consistency of the chatbots’ performance over time, especially as we noted enhancement of performance compared to our previous studies. Finally, while AI chatbots have recently gained the capability to process and analyze various input types, such as images, this study focused on text-based prompts. A standardized list of questions was generated by experts, which may not fully mimic real-world questions posed by patients. The phrasing and structure of questions can significantly influence the quality and relevance of the responses provided by the platforms. Conclusion Despite the presence of variability among platforms, both ChatGPT and Google Gemini provided accurate, complete, and reliable information on TMDs-related questions. ChatGPT-4o and Google Gemini responses were not only easier to read and understand, but also included safeguard statements advising users to consult specialists. This enhanced their utility and safety. While Google Gemini was the only platform that incorporated multimedia to support its responses, both platforms’ responses lacked evidence-based references, limiting their credibility and applicability. Supplementary Material Supinfo2 NIHMS2079822-supplement-Supinfo2.docx (14.2KB, docx) Supinfo1 NIHMS2079822-supplement-Supinfo1.docx (293.2KB, docx) Funding: No funding received. MGH is funded byT32AR060719, Skeletal Disorders Training Grant through the National Institute of Arthritis and Musculoskeletal and Skin Diseases. Footnotes Declarations of interest : none Availability of supporting data: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. References 1. de Sena MF, de Mesquita KSF, Santos FRR, Silva FWGP, Serrano KVD. Prevalence of temporomandibular dysfunction in children and adolescents. Rev Paul Pediatr. 2013;31(4):538–545. doi: 10.1590/S0103-05822013000400018 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Marciniak T Critical analysis of diagnostic tools for the temporomandibular joint. areh. 2018;32(3):45–52. doi: 10.5114/areh.2018.80969 [ DOI ] [ Google Scholar ] 3. Zieliński G, Pająk-Zielińska B, Ginszt M. A Meta-Analysis of the Global Prevalence of Temporomandibular Disorders. J Clin Med. 2024;13(5). doi: 10.3390/jcm13051365 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Upton LG, Wijeyesakere SJ. The incidence of tinnitus in people with disorders of the temporomandibular joint. Int Tinnitus J. 2004;10(2):174–176. [ PubMed ] [ Google Scholar ] 5. Poveda Roda R, Bagan JV, Díaz Fernández JM, Hernández Bazán S, Jiménez Soriano Y. Review of temporomandibular joint pathology. Part I: classification, epidemiology and risk factors. Med Oral Patol Oral Cir Bucal. 2007;12(4):E292–8. [ PubMed ] [ Google Scholar ] 6. Benites-Vega JC, Trujillo-Herrera T. Prevalencia y diagnóstico de disfunción temporomandibular en la práctica médica - Hospital general María Auxiliadora. Acta Med Peru. 2021;38(2). doi: 10.35663/amp.2021.382.2106 [ DOI ] [ Google Scholar ] 7. Field EA. Forty years of suffering chronic TMJ dysfunction: lessons learnt. Br Dent J. 2024;236(12):947–948. doi: 10.1038/s41415-024-7507-4 [ DOI ] [ PubMed ] [ Google Scholar ] 8. Butts R, Dunning J, Perreault T, Mettille J, Escaloni J. Pathoanatomical characteristics of temporomandibular dysfunction: Where do we stand? (Narrative review part 1). J Bodyw Mov Ther. 2017;21(3):534–540. doi: 10.1016/j.jbmt.2017.05.017 [ DOI ] [ PubMed ] [ Google Scholar ] 9. Tuncer A Kinesiology of the temporomandibular joint. In: Comparative Kinesiology of the Human Body. Elsevier; 2020:285–302. doi: 10.1016/B978-0-12-812162-7.00014-X [ DOI ] [ Google Scholar ] 10. Lei J, Yap AU, Zhang M, Fu K-Y. Temporomandibular disorder subtypes, emotional distress, impaired sleep, and oral health-related quality of life in Asian patients. Community Dent Oral Epidemiol. 2021;49(6):543–549. doi: 10.1111/cdoe.12643 [ DOI ] [ PubMed ] [ Google Scholar ] 11. Liu S, Cai B, Fan S, Zhang Y, Lu S, Xu L. Effects of patient education on the oral behavior of patients with temporomandibular degenerative joint disease: a prospective case series study. Cranio. Published online June 9, 2022:1–10. doi: 10.1080/08869634.2022.2085410 [ DOI ] [ PubMed ] [ Google Scholar ] 12. Xu L, Cai B, Lu S, Fan S, Dai K. The Impact of Education and Physical Therapy on Oral Behaviour in Patients with Temporomandibular Disorder: A Preliminary Study. Biomed Res Int. 2021;2021:6666680. doi: 10.1155/2021/6666680 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Balel Y Can ChatGPT be used in oral and maxillofacial surgery? J Stomatol Oral Maxillofac Surg. 2023;124(5):101471. doi: 10.1016/j.jormas.2023.101471 [ DOI ] [ PubMed ] [ Google Scholar ] 14. Chaudhary G, Amipara H, Singh P. Is YouTube a reliable source of information for temporomandibular joint ankylosis? Oral Maxillofac Surg. 2024;28(3):1431–1436. doi: 10.1007/s10006-024-01270-x [ DOI ] [ PubMed ] [ Google Scholar ] 15. Tok ÖN, Yurttaş AN, Taş S. YouTube videos as an information source about exercises for temporomandibular disorders. Dent Med Probl. 2024;61(6):855–863. doi: 10.17219/dmp/170922 [ DOI ] [ PubMed ] [ Google Scholar ] 16. Nölke L, Mensing M, Krämer A, Hornberg C. Sociodemographic and health-(care-)related characteristics of online health information seekers: a cross-sectional German study. BMC Public Health. 2015;15:31. doi: 10.1186/s12889-015-1423-0 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. Javaid M, Haleem A, Singh RP. ChatGPT for healthcare services: An emerging stage for an innovative perspective. BenchCouncil Transactions on Benchmarks, Standards and Evaluations. Published online April 2023:100105. doi: 10.1016/j.tbench.2023.100105 [ DOI ] [ Google Scholar ] 18. You Y, Gui X. Self-Diagnosis through AI-enabled Chatbot-based Symptom Checkers: User Experiences and Design Considerations. AMIA Annu Symp Proc. 2020;2020:1354–1363. [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Mendel T, Singh N, Mann DM, Wiesenfeld B, Nov O. Laypeople’s use of and attitudes toward large language models and search engines for health queries: survey study. J Med Internet Res. 2025;27:e64290. doi: 10.2196/64290 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 20. Oyeniyi Johnson. The role of AI and mobile apps in patient-centric healthcare delivery. World J Adv Res Rev. 2024;22(1):1897–1907. doi: 10.30574/wjarr.2024.22.1.1331 [ DOI ] [ Google Scholar ] 21. Johnson KB, Wei W-Q, Weeraratne D, et al. Precision medicine, AI, and the future of personalized health care. Clin Transl Sci. 2021;14(1):86–93. doi: 10.1111/cts.12884 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 22. Maleki Varnosfaderani S, Forouzanfar M. The role of AI in hospitals and clinics: transforming healthcare in the 21st century. Bioengineering (Basel). 2024;11(4). doi: 10.3390/bioengineering11040337 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 23. Kuroiwa T, Sarcon A, Ibara T, et al. The Potential of ChatGPT as a Self-Diagnostic Tool in Common Orthopedic Diseases: Exploratory Study. J Med Internet Res. 2023;25:e47621. doi: 10.2196/47621 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 24. Hadi A, Tran E, Nagarajan B, Kirpalani A. Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians. PLoS ONE. 2024;19(7):e0307383. doi: 10.1371/journal.pone.0307383 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Türp JC, Komine F, Hugger A. Efficacy of stabilization splints for the management of patients with masticatory muscle pain: a qualitative systematic review. Clin Oral Investig. 2004;8(4):179–195. doi: 10.1007/s00784-004-0265-4 [ DOI ] [ PubMed ] [ Google Scholar ] 26. Dos Santos EA, Peinado BRR, Frazão DR, et al. Association between temporomandibular disorders and anxiety: A systematic review. Front Psychiatry. 2022;13:990430. doi: 10.3389/fpsyt.2022.990430 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 27. Tran C, Ghahreman K, Huppa C, Gallagher JE. Management of temporomandibular disorders: a rapid review of systematic reviews and guidelines. Int J Oral Maxillofac Surg. 2022;51(9):1211–1225. doi: 10.1016/j.ijom.2021.11.009 [ DOI ] [ PubMed ] [ Google Scholar ] 28. Fernández-González FJ, Cañigral A, López-Caballo JL, et al. Influence of orthodontic treatment on temporomandibular disorders. A systematic review. J Clin Exp Dent. 2015;7(2):e320–7. doi: 10.4317/jced.52037 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 29. Michelotti A, Rongo R, D’Antò V, Bucci R. Occlusion, orthodontics, and temporomandibular disorders: Cutting edge of the current evidence. J World Fed Orthod 2020;9(3S):S15–S18. doi: 10.1016/j.ejwf.2020.08.003 [ DOI ] [ PubMed ] [ Google Scholar ] 30. Thirumoorthy S, Gopal S. Association between orthodontic treatment and the occurrence of temporomandibular disorders: smoke and mirrors in the form of a systematic review? Evid Based Dent. 2023;24(2):85–88. doi: 10.1038/s41432-023-00893-5 [ DOI ] [ PubMed ] [ Google Scholar ] 31. Lai YC, Yap AU, Türp JC. Prevalence of temporomandibular disorders in patients seeking orthodontic treatment: A systematic review. J Oral Rehabil. 2020;47(2):270–280. doi: 10.1111/joor.12899 [ DOI ] [ PubMed ] [ Google Scholar ] 32. Aziz AAA, Abdelrahman HH, Hassan MG. The use of ChatGPT and Google Gemini in responding to orthognathic surgery-related questions: A comparative study. J World Fed Orthod Published online October 25, 2024. doi: 10.1016/j.ejwf.2024.09.004 [ DOI ] [ PubMed ] [ Google Scholar ] 33. Yusoff MSB. ABC of content validation and content validity index calculation. EIMJ. 2019;11(2):49–54. doi: 10.21315/eimj2019.11.2.6 [ DOI ] [ Google Scholar ] 34. Yao L, Sadeghirad B, Li M, et al. Management of chronic pain secondary to temporomandibular disorders: a systematic review and network meta-analysis of randomised trials. BMJ. 2023;383:e076226. doi: 10.1136/bmj-2023-076226 [ DOI ] [ PubMed ] [ Google Scholar ] 35. Busse JW, Casassus R, Carrasco-Labra A, et al. Management of chronic pain associated with temporomandibular disorders: a clinical practice guideline. BMJ. 2023;383:e076227. doi: 10.1136/bmj-2023-076227 [ DOI ] [ PubMed ] [ Google Scholar ] 36. Senaratne D, Colvin LA. Chronic pain associated with temporomandibular disorders. BMJ. 2023;383:2877. doi: 10.1136/bmj.p2877 [ DOI ] [ PubMed ] [ Google Scholar ] 37. Sun H, Shang H-T, He L-S, Ding M-C, Su Z-P, Shi Y-L. Assessing the quality of life in patients with dentofacial deformities before and after orthognathic surgery. J Oral Maxillofac Surg. 2018;76(10):2192–2201. doi: 10.1016/j.joms.2018.03.026 [ DOI ] [ PubMed ] [ Google Scholar ] 38. Al-Asfour A, Waheedi M, Koshy S. Survey of patient experiences of orthognathic surgery: health-related quality of life and satisfaction. Int J Oral Maxillofac Surg. 2018;47(6):726–731. doi: 10.1016/j.ijom.2017.12.010 [ DOI ] [ PubMed ] [ Google Scholar ] 39. Sahin S, Erkmen B, Duymaz YK, Bayram F, Tekin AM, Topsakal V. Evaluating ChatGPT-4’s performance as a digital health advisor for otosclerosis surgery. Front Surg. 2024;11:1373843. doi: 10.3389/fsurg.2024.1373843 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 40. Dorigo W, Giacomelli E, Calvagna C, et al. Evaluation of artificial intelligence-generated layperson’s summaries from abstracts of vascular surgical scientific papers. JVS-Vascular Insights. 2024;2:100107. doi: 10.1016/j.jvsvi.2024.100107 [ DOI ] [ Google Scholar ] 41. Michel C, Dijanic C, Abdelmalek G, et al. Readability assessment of patient educational materials for pediatric spinal deformity from top academic orthopedic institutions. Spine Deform. 2022;10(6):1315–1321. doi: 10.1007/s43390-022-00545-1 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 42. Rooney MK, Santiago G, Perni S, et al. Readability of Patient Education Materials From High-Impact Medical Journals: A 20-Year Analysis. J Patient Exp. 2021;8:2374373521998847. doi: 10.1177/2374373521998847 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 43. Daraqel B, Wafaie K, Mohammed H, et al. The performance of artificial intelligence models in generating responses to general orthodontic questions: ChatGPT vs Google Bard. Am J Orthod Dentofacial Orthop. Published online March 14, 2024. doi: 10.1016/j.ajodo.2024.01.012 [ DOI ] [ PubMed ] [ Google Scholar ] 44. Abu Arqub S, Al-Moghrabi D, Allareddy V, Upadhyay M, Vaid N, Yadav S. Content analysis of AI-generated (ChatGPT) responses concerning orthodontic clear aligners. Angle Orthod. 2024;94(3):263–272. doi: 10.2319/071123-484.1 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 45. Molena KF, Macedo AP, Ijaz A, et al. Assessing the Accuracy, Completeness, and Reliability of Artificial Intelligence-Generated Responses in Dentistry: A Pilot Study Evaluating the ChatGPT Model. Cureus. 2024;16(7):e65658. doi: 10.7759/cureus.65658 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 46. Mohammad-Rahimi H, Khoury ZH, Alamdari MI, et al. Performance of AI chatbots on controversial topics in oral medicine, pathology, and radiology. Oral Surg Oral Med Oral Pathol Oral Radiol. 2024;137(5):508–514. doi: 10.1016/j.oooo.2024.01.015 [ DOI ] [ PubMed ] [ Google Scholar ] 47. Dursun D, Bilici Geçer R. Can artificial intelligence models serve as patient information consultants in orthodontics? BMC Med Inform Decis Mak. 2024;24(1):211. doi: 10.1186/s12911-024-02619-8 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 48. Alkhamees A Evaluation of Artificial Intelligence as a Search Tool for Patients: Can ChatGPT-4 Provide Accurate Evidence-Based Orthodontic-Related Information? Cureus. 2024;16(7):e65820. doi: 10.7759/cureus.65820 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 49. A Comparative Analysis of Responses of Artificial Intelligence Ch…: Ingenta Connect Accessed December 27, 2024. https://www.ingentaconnect.com/content/aapd/pd/2024/00000046/00000005/art00010 50. Azadi A, Gorjinejad F, Mohammad-Rahimi H, Tabrizi R, Alam M, Golkar M. Evaluation of AI-generated responses by different artificial intelligence chatbots to the clinical decision-making case-based questions in oral and maxillofacial surgery. Oral Surg Oral Med Oral Pathol Oral Radiol. 2024;137(6):587–593. doi: 10.1016/j.oooo.2024.02.018 [ DOI ] [ PubMed ] [ Google Scholar ] 51. Skjuve M, Følstad A, Brandtzaeg PB. The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users. In: Lee M, Munteanu C, Porcheron M, Trippas J, Völkel ST, eds. Proceedings of the 5th International Conference on Conversational User Interfaces. ACM; 2023:1–10. doi: 10.1145/3571884.3597144 [ DOI ] [ Google Scholar ] 52. Strzalkowski P, Strzalkowska A, Chhablani J, et al. Evaluation of the accuracy and readability of ChatGPT-4 and Google Gemini in providing information on retinal detachment: a multicenter expert comparative study. Int J Retina Vitreous. 2024;10(1):61. doi: 10.1186/s40942-024-00579-9 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 53. Nian PP, Umesh A, Jones RH, et al. ChatGPT and Google Gemini are Clinically Inadequate in Providing Recommendations on Management of Developmental Dysplasia of the Hip compared to American Academy of Orthopaedic Surgeons Clinical Practice Guidelines. J Pediatr Orthop Soc N Am. Published online December 2024:100135. doi: 10.1016/j.jposna.2024.100135 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Supinfo2 NIHMS2079822-supplement-Supinfo2.docx (14.2KB, docx) Supinfo1 NIHMS2079822-supplement-Supinfo1.docx (293.2KB, docx) Data Availability Statement The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. ACTIONS View on publisher site PDF (346.5 KB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Development of an AI-driven chatbot for medication-assisted treatment standards in Scotland.#449671
• Trust and verification in AI-enabled physician chatbots for chronic disease management: evidence from digital health behavior.#354661
• Large language model chatbots as sources of pediatric anesthesia health advice: An evaluation of reliability and readability.#334833
• KUAI-Kids: Developing a Measurement Instrument for Children’s Knowledge, Use, and Attitudes Toward Generative AI Chatbots#617146
• KUAI-Kids: Developing a Measurement Instrument for Children’s Knowledge, Use, and Attitudes Toward Generative AI Chatbots#617133