Enhancing trustworthiness of Arabic online health information quality evaluation using an enhanced BERT architecture with PCA and ICA feature weighting.

Enhancing trustworthiness of Arabic online health information quality evaluation using an enhanced BERT architecture with PCA and ICA feature weighting - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Sci Rep . 2026 Mar 6;16:12434. doi: 10.1038/s41598-026-43158-8 Search in PMC Search in PubMed View in NLM Catalog Add to search Enhancing trustworthiness of Arabic online health information quality evaluation using an enhanced BERT architecture with PCA and ICA feature weighting Yousef Baqraf Yousef Baqraf 1 School of Computer Sciences, Universiti Sains Malaysia, 11800 USM Penang Penang, Malaysia 3 Faculty of Computer Science & Engineering, Al-Ahgaff University, 50512 Hadhramout Al Mukalla, Yemen Find articles by Yousef Baqraf 1, 3, ✉ , Pantea Keikhosrokiani Pantea Keikhosrokiani 2 Faculty of Information Technology and Electrical Engineering and Faculty of Medicine, University of Oulu, PL 8000, 90014 Oulun yliopisto Oulu, Finland Find articles by Pantea Keikhosrokiani 2 , Yu-N Cheah Yu-N Cheah 1 School of Computer Sciences, Universiti Sains Malaysia, 11800 USM Penang Penang, Malaysia Find articles by Yu-N Cheah 1 Author information Article notes Copyright and License information 1 School of Computer Sciences, Universiti Sains Malaysia, 11800 USM Penang Penang, Malaysia 2 Faculty of Information Technology and Electrical Engineering and Faculty of Medicine, University of Oulu, PL 8000, 90014 Oulun yliopisto Oulu, Finland 3 Faculty of Computer Science & Engineering, Al-Ahgaff University, 50512 Hadhramout Al Mukalla, Yemen ✉ Corresponding author. Received 2025 Jun 4; Accepted 2026 Mar 2; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ . PMC Copyright notice PMCID: PMC13083897  PMID: 41792455 Abstract Despite the exponential increase in the availability of online health information, its quality remains questionable, presenting a significant challenge to address. This study addresses this issue by using artificial intelligence techniques, such as deep learning, to evaluate the quality of health information and to mimic human-level evaluation capabilities. The key methodologies used in the study included an enhanced version of Arabic BERT for medical data, feature extraction techniques incorporating Principal Component Analysis (PCA) and Independent Component Analysis (ICA), and modified loss functions using information entropy to improve the model’s certainty and calibration during document classification. The results of the study were encouraging: the proposed PCA-based model achieved higher accuracy than the competing models and reached 94.7% on the dataset used, comparable to reported human-level performance. Finally, these findings may contribute to improving the reliability of online health information in Arabic contexts and provide a foundation for future efforts aimed at supporting healthcare decision-making. The methodologies and results presented here offer policymakers and researchers valuable tools to assess and ensure the trustworthiness of online health information. Keywords: Deep learning, Online health information, Quality assessment, Arabic online health information, Trustworthiness, Calibrating AI systems, Arab countries Subject terms: Health care, Public health Introduction Over the past decade, advancements in artificial intelligence (AI) have prompted researchers to acknowledge the urgent need for developing intelligent systems that are not only highly capable but also safe and secure 1 – 4 . However, AI Safety is still in its early stages, with researchers actively working to identify its key challenges and limitations 5 . Concurrently, online health information has grown exponentially, and concerns about misleading content and its impact on public health have increased 6 , 7 . Furthermore, individuals increasingly rely on the Internet to access information across various aspects of their lives, including health conditions 8 – 10 . Consequently, ensuring AI safety is particularly critical in domains such as online health information, where the reliability of information directly affects public health 6 . The Internet has emerged as the central source of health information for patients, caregivers, physicians, and the general public 11 – 13 . Despite this dependence, studies show that the public and patients typically view online health information as trustworthy 14 and consider it a potential alternative to in-person medical care 15 . Nonetheless, a recent systematic review by 11 underscores the poor quality of online health information, highlighting the need for further investigation. For online health information to be practical, it needs to be clear, reliable, and relevant to the patient’s symptoms to suggest suitable actions. However, search engine results frequently include untrustworthy, incorrect, or poor-quality content 16 – 18 . Given the substantial reliance on online health information, ensuring its quality presents a critical challenge, especially in light of the limitations inherent in current evaluation methods. According to studies in a recent systematic review 19 , the proposed models 20 – 28 have three significant limitations: limited generalizability, restricted coverage of quality dimensions, and an oversimplification of the complexity involved in evaluating health information quality. This oversimplification is obvious in neglecting uncertainty and calibration metrics, which are fundamental for developing safe models for assessing health information quality. First, regarding generalizability, models are designed to assess health information in a specific language, classifying them as unimodal language models. Seven research papers in the literature review focused on English, as shown in references 21 – 24 , 26 – 28 . In contrast, only one study concentrated on Arabic 20 , and another examined Dutch 25 . Secondly, the quality dimensions covered, the definition of health information quality (HIQ), and the quality criteria used in these papers lack universal acceptance, with various studies employing different interpretations and evaluating HIQ through diverse criteria 11 , 29 . The assessed papers reveal considerable limitations and inconsistencies in the strategies used to evaluate the quality of health information. While some studies use a single quality criterion 21 , others, such as study 22 , apply a broad set of 119 different criteria, the highest number used among all the studies in the systematic review. None of the studies in the systematic review examined whether the criteria they used reflected actual human ethical values. Such an examination is essential to ensure that the models align with ethical standards, as there is a significant risk of AI misalignment 5 , which can lead to undesirable outcomes. Furthermore, value alignment is the most investigated approach for achieving safety in AI 5 . Consequently, the included studies do not discuss the risk of omitting quality criteria or specify which aspects of health information quality might be affected. Finally, existing solutions in the literature often oversimplify the complexity of evaluating health information quality by neglecting the examination of uncertainty and calibration metrics and failing to propose techniques to improve them. Since automated evaluation of health information quality is not a low-risk, high-reward approach, it entails significant risks, may lead to negative consequences, and requires careful investigation 5 . To address these challenges, this paper presents a comprehensive approach to evaluating the quality of Arabic online health information. First, we reassess the relevance of existing quality assessment tools to ensure they genuinely reflect ethical human values, using the recently proposed quality dimensions by 30 . Second, we have collected and annotated relevant datasets that capture the complexity of health information quality, which are used to train and test our proposed models. More details about the dataset are provided in 31 . Third, the study enhances the Arabic BERT architecture by incorporating feature extraction techniques, such as Principal Component Analysis (PCA) and Independent Component Analysis (ICA), alongside modified loss functions that leverage information entropy. As a result, the study proposes four different architectures: one for PCA, one for ICA, one for the modified entropy function, and the last, the enhanced BERT model trained on Arabic medical textual data, which will be discussed next. These enhancements are designed to improve model certainty and calibration, therefore overcoming the limitations of previous studies and achieving human-level performance in assessing health information quality. Finally, this study examines the impact of enhancing BERT with Arabic medical textual data on model performance, explicitly focusing on model certainty and calibration. It aims to explore how this enhancement improves the contextualized representations of medical language in Arabic and how these improvements affect the accuracy and effectiveness of text classification, particularly regarding certainty and calibration for both high- and low-quality classes. Refer to Fig. 1 for a more pictorial representation of different techniques’ effects. Fig. 1. Open in a new tab Effects of ICA, PCA, and information entropy on document separation based on quality. Theoretical framework The theoretical framework of this paper is grounded in recombination theory, which declares that most new knowledge arises through the combination or reconfiguration of existing knowledge components 32 – 34 . The study by 34 demonstrated that scientific advancement is significantly driven by papers introducing novel ideas by integrating diverse references from distant domains. Similarly 33 , conducted a comprehensive analysis of 17.9 million research papers across various scientific disciplines, revealing a consistent pattern: influential breakthroughs typically build upon established knowledge combinations while incorporating unexpected and unconventional elements. While the potential for beneficial combinations is continuously growing, recognizing them is becoming increasingly challenging due to cognitive and social limitations 35 , as cited in 36 . Recent advances in deep learning illustrate this principle, often relying on creative recombination of established methods 36 . For instance, specialized architectures such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been cleverly developed to address image processing challenges in the case of CNNs and to handle signal processing tasks in the case of RNNs, building on the foundation of a multi-layer perceptron. The study 37 employed a hybrid Convolutional Neural Network and Deep Belief Network model, optimized using a combination of Particle Swarm Optimization (PSO) and Al-Biruni Earth Radius (BER) algorithms (CNN-DBN-PSOBER), for oral cancer detection. In contrast, the study 38 utilized a Recurrent Neural Network optimized with the Dynamic Fitness Al-Biruni Earth Radius algorithm (RNN-DFBER) for wind power forecasting. Both studies integrate established neural network architectures with advanced optimization techniques to achieve superior performance. Recombination has been crucial in advancing deep learning, resulting in remarkable achievements across various domains. These include high-accuracy image processing 39 , transformative applications in gaming and generative models 40 – 42 , and object identification tasks 43 . Moreover, deep learning has expanded its impact into areas such as drug development and genetics 44 , 45 . Beyond performance improvements, deep learning holds the potential to reshape scientific progress itself by altering how researchers operate and enabling the integration of ideas from seemingly unrelated fields 46 – 48 . Accordingly, this paper draws on recombination theory to inform its Objective, which aims to leverage the potential of combining existing approaches to generate novel and insightful solutions. The proposed models in our study include several approaches aimed at selecting the most relevant features to reduce uncertainty in the feature space. BERT embeddings capture high-dimensional and context-rich representations, but they often introduce redundancy and correlations among features 49 . Our approach incorporates techniques such as PCA and ICA to strengthen the BERT architecture and address these limitations. The primary objective of these models is to achieve dimensionality reduction and source separation within the feature space generated by Arabic BERT. Additional details are provided in the related sections for each method used. Methodology The method proposed in this study for predicting health information quality contains two main parts. The first part involves three stages of model training: initially, web pages are processed using a fine-tuned Arabic BERT model trained on medical data to obtain vector representations for each webpage. These vector representations then undergo dimensionality reduction through Independent Component Analysis (ICA) or Principal Component Analysis (PCA) to extract the most critical data components for classification. Finally, the reduced data is passed through the proposed pooling layer for classification. The second part contains the second and third proposed models. The first model uses a modified categorical cross-entropy loss function with an entropy-penalized term as its objective. The second is the enhanced Arabic BERT model trained on medical data. This part consists of two stages: first, web pages are processed using the Arabic BERT model to obtain vector representations for each webpage; second, these vector representations are directly fed into the proposed pooling layer for classification. By integrating these two parts, the study aims to improve the certainty and calibration of health information quality prediction. The following sections will discuss more information about each part and the dataset’s collection and creation. For more information about the effect of the proposed approach on the distribution and use of online health information, refer to Fig. 2 . The figure illustrates how the proposed framework integrates ethical human values 30 with an enhanced Arabic BERT architecture to deliver more reliable online health information to healthcare professionals, patients, the general public, and smart health devices. It highlights findings from the systematic review 19 indicating that much of the health information available on the Internet, through platforms such as Google, Bing, and Yahoo, is unreliable, inaccurate, and of low quality. In the middle layer, a proposed benchmark dataset with ranking estimation factors 31 is introduced as part of a control layer designed to assess and filter content. The figure also demonstrates how perception prediction is managed through machine learning 50 , showing how the proposed methods in this paper collectively ensure quality control and trustworthy health information delivery. Fig. 2. Open in a new tab The proposed method for ensuring quality health information distribution and use. Finally, as the last validation step, we used Uniform Manifold Approximation and Projection (UMAP) 51 – 53 . This non-linear method preserves both local and global data structures of the dataset. This is used to confirm that the components derived from PCA and ICA preserved meaningful association beyond a simple linear relationship. Data collection Six Arabic medical terms were selected for this study, along with their corresponding English translations: hypertension, stroke, heart attack, blood pressure treatment, stroke treatment, and angina pectoris treatment. The total number of terms used was 11, including the treatment search terms, as shown in Fig. 3 . These terms were chosen based on their prevalence, as indicated by Google Trends data. Data were collected using two search engines, Google and Bing, from which the first 100 search results were retrieved for each term. Fig. 3. Open in a new tab Shows the step-by-step process of identifying, screening, and selecting the web pages included in this study. The web page searches were performed on January 15, 2024, utilizing the Google Chrome browser. All preliminary data, including browsing history, caches, and cookies, were cleared to minimize potential bias before conducting the searches. From an initial collection of 1,200 identified web pages, 503 met the inclusion criteria and were subsequently included in the analysis. Inclusion and exclusion criteria for the web pages The following criteria determine whether retrieved web pages are included or excluded: Health information must be written in Arabic. Content should address emergency medical conditions (such as heart disease, stroke, and hypertension) and be presented in text format. Excluded formats include audio, video, books, and scientific articles. Sources must not originate from social forums, social media platforms, or news agencies in news form. The quality assessment We adopted the steps and quality dimensions outlined in 30 to assess the quality of a web page’s content. These steps include all the necessary statistics for analyzing the text, as demonstrated in Fig. 4 . Some studies in the literature review focus on metadata features, while others assess the textual content and quality score. To this end, we examine all available statistical features to gain further insight into the quality dimensions used, as referenced in the paper 30 . By this method, we tried to increase the signal-to-noise ratio. The overall summary of constructing a dataset for Arabic online health information is as follows: This evaluation aims to assess the quality of webpages providing information on emergency conditions such as heart disease, stroke, and hypertension. Our task is to code a random sample of 100 webpages from a list of 503, obtained from Google and Bing, using specific search strings and criteria outlined in the ‘Data collection’ section. The reviewers evaluated the content against relevant quality criteria, assigning a code of 1 if a criterion was met and 0 if it was not For treatment criteria, we followed the method proposed by 54 . A score of 5 was assigned for detailed treatment information, 2–4 for unclear or incomplete information, and 1 when no description was provided. After each researcher had completed coding the 100 webpages, both reviewers’ coding was reviewed, and the inter-rater reliability was 0.94. The disagreements were resolved through consensus. Subsequently, one reviewer coded the remaining webpages. Fig. 4. Open in a new tab Statistical features of text for quality prediction. From an ethical perspective, we assess health information across 15 quality dimensions, containing source quality, content trustworthiness, and treatment quality dimensions. Source quality dimensions include authorship, currency, advertising policy, and website type, ensuring that information is attributed to credible authors, remains up to date, and is clearly distinguished from commercial influence. Content trustworthiness is evaluated through attribution, complementarity, justifiability, evidence-based medicine, and emergency guidance. Treatment quality focuses on treatment descriptions and is assessed using six sub-criteria: treatment works (mechanism of action), benefits, risks, impact on quality of life, consequences of no treatment, and areas of uncertainty. The dimensions are scored using a binary scheme (0 = no information provided; 1 = information provided), whereas treatment descriptions are rated on a 5-point Likert scale, with higher scores reflecting greater detail. This structured scoring system offers a transparent framework for evaluating, together, the trustworthiness and ethical quality of health information. Additional information on these quality dimensions can be found in 30 , and the associated dataset is described in 31 . Quality score aggregation and labeling procedure Each webpage was assessed based on three groups of quality criteria: source quality, content trustworthiness, and treatment quality 30 . All source quality and content trustworthiness criteria were scored using a binary scale, where a value of 1 means the criterion was met and 0 means it was not. Each of these criteria was given equal weight. Treatment quality was assessed separately across six sub-criteria: Treatment Works, benefits, risks, impact on quality of life, consequences of no treatment, and areas of uncertainty. Each sub-criterion was evaluated using a 5-point scale, where a score of 1 signifies that no description was provided, and a score of 5 represents a thorough and detailed explanation. Additional information about the dataset can be found in 31 . The raw quality score for each webpage was calculated as the total of all binary source and content trustworthiness scores, plus the total of the six treatment sub-criteria scores. No differential weighting was used; all binary criteria had equal weight (1), while treatment criteria contributed based on their assigned 1–5 scores. Because the dataset has a wide range of quality scores, the raw total quality score was first log-transformed to reduce the effect of extreme values. The scores were then rescaled using min-max normalization to create a final normalized quality score between 0 and 1. A fixed global threshold of 0.5 was then applied to assign binary class labels. Webpages with normalized scores of 0.5 or above were classified as high quality, while those with scores below 0.5 were classified as low quality. Dataset characteristics and preprocessing The dataset consists of 369 web pages (73.36%) classified as very low quality, while 134 (26.64%) were classified as high quality. This distribution is noticeably imbalanced, with most low-quality pages compared to a smaller yet significant subset of high-quality pages. Such an imbalance affects the training of predictive models. As a result, we used back-translation as an oversampling technique on the training set using the Deep Translator Python library, which has been shown to generate high-quality data 55 , among other techniques discussed later in this section. We used AraBART 56 to summarise the web page content to a maximum of 512 tokens, corresponding to the BERT model’s token limit. Furthermore, the AraBERT preprocessor 57 was used to clean the dataset by performing essential preprocessing steps, including removing diacritics (Tashkeel), standardizing Arabic characters to a uniform format, and eliminating special symbols and punctuation. This process ensured the Arabic text was consistently formatted for subsequent input into the enhanced Arabic BERT models. Additional details about the dataset are presented in Table 1 . Table 1. Overview of the dataset categorized by health topics. Topic Number of samples Number of sent Number of words Number of tokens Heart Disease 113 5,114 124,907 157,353 Blood Pressure 112 5,370 122,722 146,448 Strokes 96 4,936 116,114 142,372 Heart Attack Treatment 75 4,213 99,482 125,857 Blood Pressure Treatment 57 2,536 63,954 76,832 Strokes Treatment 50 2,920 67,577 83,170 Open in a new tab To perform data augmentation in the training data, the dataset was split into 70% for training and 30% for testing. Accordingly, the training set consisted of 256 low-quality and 96 high-quality web pages, while the test set comprised 113 low-quality and 38 high-quality web pages. Following recent recommendations on bias reduction and fairness in machine learning and assessment practices 58 , 59 , we applied oversampling with three augmentation strategies: paraphrasing 60 , back-translation 61 using the Deep Translator Python library, and a combined approach that included random word insertion, sentence shuffling, synonym insertion, and synonym replacement 62 . After augmentation, the training dataset consisted of 512 samples, balanced equally between 256 high-quality and 256 low-quality web pages. Figure 5 illustrates the point at which our techniques are applied. Fig. 5. Open in a new tab Illustration of where oversampling techniques such as paraphrasing, randaugment, and back-translation are applied within the text processing pipeline. Finally, the dataset was divided into training and test sets using two techniques: a train-test split with 70% for training and 30% for testing, and k-fold cross-validation with or , depending on the model architecture. To prevent data leakage, the augmentation pipeline was applied exclusively to the training data, ensuring that each original sample and its augmented versions remained within the same fold during cross-validation. For more detailed information, refer to Fig. 6 . Fig. 6. Open in a new tab Illustration of k-fold cross-validation, where k is set to 5 or 10 based on the model complexity. Models This study developed five distinct models using various feature-extraction and weighting techniques. First, we discuss the vector representation step for all documents. Second, we describe the two models that employ ICA and PCA feature weighting, including the UMAP model, which is used to validate the results of both ICA and PCA. Third, we discuss the model employing a modified categorical cross-entropy loss function with an entropy-penalized term to enhance model performance. Finally, we present an enhanced Arabic BERT model trained on medical data, which leverages specialized training to improve performance in medical text classification. Furthermore, the study’s experimental setup employs PyTorch 63 to implement the neural network models. The experimental setup varies depending on the proposed modeling approach. The following sections discuss the hyperparameters applied, including the optimizer, learning rate, batch size, and regularization techniques. In our experiments, we selected the model parameters based on empirical tuning and performance evaluations. Vector representation of the health documents Before discussing the proposed models, we generated the vector representations of the health documents using Arabic BERT 64 , as illustrated in Fig. 7 . Each page content in the set includes source quality criteria, treatment quality criteria, and trustworthiness criteria as proposed in a recent article 30 . Fig. 7. Open in a new tab The transformation of web pages to vector representation. The main objective is to obtain a vector representation of each webpage feature by passing through Arabic BERT models using the previous components. In a formal context, when considering an input web page , Arabic BERT creates a vector representation for each input, denoted as Where , where represents the number of tokens per webpage, which equals 512 the maximum number of tokens, and represents the number of features for each token as shown in the matrices 1 which equal 768 in Arabic BERT represented as . The transformation of text into vector representations captures semantic relationships and contextual information related to the quality dimensions, reflecting the ethical human values targeted by this study. 1 ICA-PCA proposed models with UMAP validation BERT embeddings are designed to transform high-dimensional, context-rich representations into more discriminative and compact features for classification. They often produce embeddings with significant redundancy and correlation among features 49 . Our proposed solution leverages PCA and ICA, among other techniques, to enhance the BERT architecture designed to address these issues. The primary objective of these models is dimension reduction and source separation in the feature space generated by Arabic BERT in the previous step, as shown in Fig. 8 . This step determines a weighted combination of the matrix features generated by Arabic BERT for each document. Fig. 8. Open in a new tab Transformation of web page vector representations to an ICA-PCA vector space. The goal is to find the linear weighted combination that produces a reduced matrix while retaining the maximum amount of relevant information. The first method is PCA 65 , which determines the directions along which the most variance is present by considering that the features with the highest variance are the most relevant, using the second statistical moment. The PCA equation 3 , which is the solution of the objective function 2 , can be understood as expressing the equality between the left side, which involves the product of principal components (W) and eigenvalues ( ), and the right side, where the covariance matrix (C) is multiplied by the principal components. PCA aims to discover a weight vector so that, upon projecting the data onto it, the resulting principal component can account for the maximum possible variance while keeping its length equal to one (i.e., a unit vector). This is crucial because maximizing the variance explained by the projection guarantees that the principal component effectively captures the most significant information within the health information dataset. 2 3 The second method for reweighting features produced by Arabic BERT involves implementing the FastICA 66 . The primary objective of the ICA algorithm is to identify distinctive semantic components that capture meaningful information about the documents in the embedding space. This algorithm rotates the axes to minimize the amount of shared information. It effectively rotates the embedding axes so that the components are as independent as possible by minimizing mutual information. In contrast, PCA removes linear dependencies among features, while preserving shared information more than ICA does. The pooling layer for both PCA and ICA (See Figs. 9 and 10 ), which is the final layer before the classification layer, contains several components designed to leverage the knowledge generated in the preceding steps. Fig. 9. Open in a new tab Detailed workflow for transforming BERT’s learned representations for classification tasks. Fig. 10. Open in a new tab Transforming learned representations of BERT features for classification. First, we apply mean pooling to the token embeddings, using the attention mask to ignore padded tokens. This produces a single 768-dimensional vector representing the 512-token sequence of health-related web pages. Second, after computing the meaning-pooling representation for all samples, we apply PCA or ICA to obtain reduced vector representations of the web pages. This step learns a transformation that either captures the directions of most significant variance (in the case of PCA) or identifies statistically independent components (in the case of ICA). Third, after dimensionality reduction, the feature vectors pass through a fully connected feed-forward layer followed by a batch normalization layer, which normalizes the activations and reduces internal covariance shift during training. Finally, the enhanced feature vectors are passed through a Leaky ReLU activation function and fed into the classification layer to predict the quality of the health information. The Arabic BERT architecture used in these models and the proposed Information Entropy model (discussed in the next section) is based on the base version of Arabic-BERT 64 . The configuration includes 12 transformer encoder blocks, each with 12 attention heads and a single feed-forward network for classification. The model supports a maximum input length of 512 tokens and has 768 hidden dimensions, resulting in approximately 110 million parameters. To train the PCA-proposed model effectively, the AdamW optimizer was employed with a learning rate of , a batch size of 16, and 25 epochs. The learning rate value was selected according to the grid search range . To prevent overfitting and ensure the Arabic-BERT model generalizes effectively to the test set, L1 regularization was applied with a coefficient of , selected from the grid search range . A moderate L1 regularization coefficient was chosen to encourage sparsity in the feature space and improve model stability and generalization. Furthermore, an early stopping technique was employed with a patience of 5,a minimum improvement delta of 0, and automatic restoration of the best weights. The PCA-proposed model also uses 5-fold stratified cross-validation to train and test the model while calculating various metrics, including Train Accuracy, Test Accuracy, Test Loss, and Test Entropy. In each fold, the data is split into 80% for training and 20% for testing. The model’s performance is evaluated based on the average metric scores obtained across the testing sets in the last epoch of all five folds. For more detailed information about Cross-Validation, refer to Fig. 6 . Finally, the number of features (PCA components) and the number of hidden layers were determined using a grid search within a predefined parameter space. The components were tested with values of 100, 200, 300, 400, and 500, while the number of layers was tested at 1, 2, and 3. For more detailed information, refer to Fig. 11 Fig. 11. Open in a new tab Layer-wise analysis of loss and accuracy relative to features (PCA_components) count. For training the ICA-proposed model, the AdamW optimizer was selected with a learning rate of , within the grid search range , along with a batch size of 4 and 50 epochs. To prevent overfitting and enhance the generalization of the ICA-proposed model to the test set, L1 regularization was applied with a coefficient of , selected from the range . A relatively stronger L1 regularization was used to suppress noisy or redundant ICA components and improve feature robustness. Additionally, weight decay regularization was employed with a coefficient of , chosen from the same range, instead of applying the early stopping technique. The hyperparameters for both models were determined through experimental evaluation. Due to the small batch size required for optimal ICA component optimization and the sensitivity of the FastICA-based feature weighting process, which resulted in instability, including inconsistent convergence and high performance variance across folds, ICA was omitted from the final k-fold comparison to maintain methodological reliability and to prevent presenting unreliable results that could be considered an exploratory model. The number of features (ICA components) and the number of hidden layers were determined using a grid search within a predefined parameter space. The components were tested with 100, 200, 300, and 400 values, while the number of layers was tested with 1, 2, and 3. For more detailed information, refer to Fig. 12 . Fig. 12. Open in a new tab Layer-wise analysis of loss and accuracy relative to features(ICA_components) count. Finally, to validate the embedding representation of the health-related web pages obtained from PCA and ICA, we additionally applied the Uniform Manifold Approximation and Projection (UMAP) algorithm 51 – 53 , a non-linear dimensionality reduction method designed to preserve both local and global structures of the dataset. UMAP operates by constructing a graph that captures the relationships within the high-dimensional data and then identifies a low-dimensional embedding in which the graph structure closely approximates that of the original high-dimensional space 53 . This is accomplished by optimizing a cross-entropy objective function (as shown in Equation 4 ) between the fuzzy topological representations of the data in both spaces. 4 The UMAP proposed model integrates the Uniform Manifold Approximation and Projection (UMAP) as a non-parametric dimensionality reduction technique applied to the contextual embeddings generated by the Arabic BERT model. UMAP was used to project the high-dimensional embedding space into a compact latent representation while preserving local and global semantic structures. The reduced UMAP features were then fed into a downstream classification model trained using the AdamW optimizer with a learning rate of , selected from the grid search range , a batch size of 16, and 50 epochs. To prevent overfitting, L1 regularization and weight decay were applied with coefficients of , both chosen from the range . Early stopping was employed with a patience of 5, a minimum improvement delta of 0, and automatic restoration of the best-performing weights. Information entropy proposed model The study employed a modified categorical cross-entropy Loss with an entropy-penalized term as the objective function. Combining these terms primarily encourages the model to make more accurate predictions, increase its confidence, and eliminate features that contribute to increased model uncertainty. This approach ensures reliable certainty and calibration in health information quality prediction tasks. The model incorporates information entropy as a learning parameter to enhance its certainty by including it as an extra term in the loss function ( 5 ). The entropy term in Equation ( 5 ) is computed for each sample and then averaged across the batch, ensuring that the regularization operates at the batch level during training. During the forward pass, if certain features negatively impact the model’s certainty, their influence is minimized by reducing their corresponding weights toward zero, as shown in Fig. 13 . The figure shows a negative relationship, illustrating how weights diminish as entropy approaches 1. This process enables the model to optimize the feature selection and weighting to maximize certainty by reducing reliance on high-entropy Features. More precisely, all trainable parameters in the encoder are updated during the backward pass, including the two main components: multi-head self-attention and feedforward neural networks, as shown in Fig. 14 . Fig. 13. Open in a new tab Impact of integrating entropy on model weight distribution. Fig. 14. Open in a new tab Adjust the weights based on the information entropy. 5 To implement this model, the training process utilised the AdamW optimiser with a learning rate of , and a batch size of 16 was used for training over 3 epochs. To prevent overfitting and enhance the generalisation of the Arabic-BERT model to the test set, weight decay regularisation was used with a lambda value of . To further optimize the performance of the proposed model, a grid search over a predefined parameter space, including the values and , was conducted to determine the best values influencing the entropy term ( 5 ). The Entropy-proposed and base models were also tested using 10-fold stratified cross-validation while calculating various metrics, including Train Accuracy, Test Accuracy, Test Loss, and Test Entropy. In each fold, the data is split into 90% for training and 10% for testing. The model’s performance is evaluated based on the average metric scores obtained across the testing sets in the last epoch of all ten folds. For more detailed information about Cross-Validation, refer to Fig. 6 . Contextual representations in Arabic medical text The literature review 19 identified one major problem related to word representation. The studies used word embeddings trained on general or scientific medical text, which could lead to poor performance 67 , 68 . The primary purpose of this proposed model is to check the effect of enhanced word representations, including different health information distributions, Medical books and Medical web pages from the internet, on model certainty and calibration. The process of building this proposed model consists of five components. First, word collection is performed to identify suitable keywords for creating the dataset. Second, the data undergoes a cleaning and normalisation process. Third, the dataset is prepared for training and testing the model. Fourth, the architecture of the proposed model is developed. Finally, the pretraining techniques are applied. First, Keyword variation is vital in ensuring accurate and representative health information in Arabic and producing unbiased pre-trained deep learning models 69 . The study employed a systematic data collection method through surveys, academic literature, and advanced language models(ChatGPT) 70 . ChatGPT produced the highest number of keywords compared to the survey and the literature, as shown in Fig. 15 . More details about the collected keywords are as follows: The study conducted a survey to gather keywords from more than 30 medical experts across 15 different fields. The study gathered keywords from the literature of seven different studies 71 – 77 . As several studies have explored the application and benefits of artificial intelligence and large language models in public health 78 – 80 . ChatGPT can be a valuable resource for individuals and societies seeking to make informed decisions about their health 79 , 80 , as cited in 78 .We requested ChatGPT, a language processing tool, to act as an Arabic medical expert and provide relevant keywords for 30 medical fields to facilitate online searching. The ChatGPT prompt we used: You act as a health expert in Arabic. Can you provide guidance on 30 different medical fields and recommend 10 appropriate keywords for online health information searches? Following the collection of the keywords, a medical expert assessed them for suitability and implemented the necessary adjustments. Supplementary File A contains a sample of the collected keywords, while Supplementary File B provides the questionnaire used in their collection. Fig. 15. Open in a new tab Number of collected keywords with source. Second, this study used several data cleaning and normalisation tools to convert raw data into a format suitable for model pre-training. The WebBootCaT Tool 81 Part of Sketch Engine 82 corpora collection tools were used to eliminate irrelevant content, including HTML tags, web scripts, and advertisements. Furthermore, for text normalisation and removing the special characters, we used the AraBERT preprocessor 57 . Third, the dataset statistics: the final corpus contains 5,112,553 sentences and 119,377,101 words. The corpus was divided into 95 text files, with sentences separated by lines. The dataset was split into training and testing sets, with 80% allocated for training and 20% for testing. Fourth, the model architecture for the MArBERT proposed model, which stands for Medical Arabic BERT, is based on the medium version of Arabic-BERT 64 . The configuration includes 8 transformer encoder blocks, each with 8 attention heads and a single feed-forward network. The model supports a maximum input length of 512 tokens and has 512 hidden dimensions, resulting in approximately 42 million parameters. Figure 16 provides additional information regarding the Architecture. Fig. 16. Open in a new tab The MArBERT model architecture. Finally, to ensure proper pre-training of MArBERT, it is important to follow the original steps of Arabic BERT. First, for MARBERT pretraining, we utilised Masked Language Modelling (MLM) and Next Sentence Prediction (NSP). The MLM technique helps models learn word relationships in the corpus, where 15% of the input tokens are chosen for replacement. The strategy to replace the masked token is as follows: 80% is replaced with the [MASK] token, 10% with a random token, and 10% with the original token. Additionally, we incorporated Next Sentence Prediction (NSP) to enable the language model to learn relationships between sentences, which are not used in the base model. To implement this model, the training process utilised the AdamW optimiser with a learning rate of , and a batch size of 16 was used for training over 15 epochs. To prevent overfitting and enhance the generalisation of the Arabic-BERT model to the test set, weight decay regularisation was used with a lambda value of . Furthermore, an early stopping technique was employed with the following parameters: patience of 5, a minimum improvement delta of , and the restoration of the best weights. Model performance evaluation This study used several metrics to measure model performance, including Accuracy, Recall, Precision, and F1-score.Each of these metrics provides insights into a specific aspect of model performance, and together, they offer a comprehensive evaluation of the model. Additionally, this study introduces three metrics that have not been previously used in related research. The first is the Matthews Correlation Coefficient, which is particularly effective for providing a balanced measurement in binary classification, especially when the dataset is imbalanced, a common scenario in health information quality assessment. The second metric is Expected Calibration Error (ECE) 83 , a widely used measure in machine learning for assessing the calibration of predictive models. Calibration refers to the degree to which a model’s predicted probabilities align with the observed likelihood of events. Expected Calibration Error (ECE) was computed for binary classification using equal-width binning of predicted probabilities into 10 bins over the interval [0, 1]. For each bin, we computed the absolute difference between the mean predicted probability (confidence) and the empirical frequency of the target class. The final ECE was obtained as a weighted average of these differences, where weights correspond to the proportion of samples in each bin, ignoring empty bins. We report class-conditional ECE separately for high-quality and low-quality classes. For each class, calibration was computed using predicted probabilities for that class, treating it as the positive class. Specifically, the same ECE formulation was applied to each class by adjusting the target labels accordingly. Model probabilities were obtained directly from the sigmoid output without additional post-hoc calibration. ECE is defined as: 6 where represents the set of samples in bin m , denotes the number of samples in bin m , n is the total number of samples, m denotes the bin index, is the empirical accuracy of bin m , and is the mean predicted probability (confidence) in bin m . The Entropy metric quantifies the model’s certainty. When entropy is close to zero, the model is confident in its predictions; however, higher entropy values indicate greater uncertainty, with a maximum of one signifying complete uncertainty. Finally, the dataset was encoded with 0 representing the low-quality class and 1 representing the high-quality class. In line with the standard convention, the high-quality class (label 1) is treated as the positive class when computing evaluation metrics such as precision, recall, and F1-score. This ensures that precision and recall reflect the model’s ability to correctly distinguish high-quality health information from low-quality content. Results In this section, the experimental results are presented. First, the results of the data augmentation techniques are discussed. Second, the effect of the feature extraction techniques on the feature space is examined. Finally, the performance of the proposed models is compared using various metrics, including certainty and calibration measures. Data augmentation techniques results This section presents the results of the data augmentation techniques. It evaluates the model’s ability to distinguish between two categories, high-quality and low-quality classes, using the base version of Arabic-BERT 64 , which is described in detail in the Methods section. We examined three augmentation methods: Paraphrasing, RandAugment, and Back-Translation, evaluating their effectiveness using metrics such as accuracy, F1 score, precision, and recall. Unlike other sampling methods, such as SMOTE 84 and ADASYN 85 , which are applied to word embeddings rather than raw text, this approach provides only limited linguistic variation. The back-translation technique delivered the best results, achieving 87% accuracy and an F1 score of 84%. Paraphrasing was nearly as effective, with an accuracy of 0.86% and an F1 score of 82%. RandAugment had the lowest performance, achieving an accuracy of 84% and an F1 score of 79%. For further discussion, the model’s performance with back-translated data showed a relatively small discrepancy of 11.32% (90.27% vs. 78.95%). In contrast, paraphrased data resulted in a larger gap of 13.6% (89.3% vs. 75.7%). Nevertheless, performance across both paraphrasing and back-translation techniques remained relatively balanced across all evaluation metrics. The model’s performance with RandAugment varied notably across quality classes. It achieved 87.8% accuracy in the low-quality class, whereas in the high-quality class, the accuracy dropped to 71.2%, resulting in a 16.6% point gap. In conclusion, back-translation data augmentation shows the most stable results, achieving the highest accuracy and F1 score while reducing differences between high- and low-quality classes, making it the most suitable choice for training all the models used in this study. Nevertheless, the discrepancy between low- and high-quality classes cannot be fully resolved by data augmentation alone; therefore, we propose complementary approaches, such as ICA, PCA, and an entropy model, to address this issue. Impact of ICA, PCA, and UMAP on data space The implementation of ICA, PCA, and UMAP enhanced the data space representation, making it more effective for the proposed models to learn. Consequently, the PCA model showed the strongest performance among the proposed models, achieving outcomes that were comparable to, or slightly higher than, the average human performance of 94.3% reported in this systematic review 19 , as well as the human performance estimated for this dataset 31 , as further detailed in the following section. As shown in Fig. 17 , the PCA feature space helps to form two distinct clusters. However, these clusters do not correspond to the original classes (high-quality and low-quality health information). The primary purpose of feature extraction techniques like PCA is to separate data based on maximum variance, while ICA aims to reduce shared information among features. The ICA produces a similar effect; however, some overlap remains between the two classes, as shown in Fig. 18 . Fig. 17. Open in a new tab Effect of PCA on data space representation. Fig. 18. Open in a new tab Effect of ICA on data space representation. UMAP, in contrast, is employed as a nonlinear dimensionality reduction method that preserves both local and global data structures, serving as a validation step for the linear approach. The results obtained with UMAP were highly similar to those of PCA, showing clear separation between the two classes, as illustrated in Fig. 19 . However, the UMAP reduction appeared to introduce three distinct clusters, rather than the clear two-class structure observed with ICA and PCA. Fig. 19. Open in a new tab Effect of UMAP on data space representation. In conclusion, the results demonstrated the effectiveness of PCA and ICA in reducing dimensionality and enhancing class separation. Although a significant improvement in separation was observed, some overlap remained, particularly with ICA. This remaining overlap is expected, considering the substantial overlap of features in the original feature space as shown in Fig. 20 . Fig. 20. Open in a new tab Representation of class clusters in the data space. Performance evaluation of proposed models We compared the performance of the five trained models (Base Model, Entropy Model, ICA Model, PCA Model, and Medical Enhanced Model) using the train-test split or k-fold cross-validation, evaluated with various performance metrics (Table 2 , Fig. 21 ). Table 2. K_Fold performance metrics for the proposed models. Model Train Acc Train SD Test Acc Test SD Loss Loss SD Entropy Entropy SD Class0 Acc Class0 SD Class1 Acc Class1 SD Base_Model 91 ±.01 86 ±.06 0.36 ±.11 0.34 ±.03 83.17 ±9.74 88.60 ±4.99 Entropy_Model 93 ±.01 84 ±.05 0.37 ±.08 0.38 ±.03 76.52 ±8.09 91.77 ±5.79 PCA_Model 94.8 ±.90 94.7 ±2.51 0.18 ±.00 0.32 ±.04 95.29 ±3.19 94.16 ±3.20 Medical_Enhanced_Model 95 ±.01 86 ±.05 0.32 ±.05 0.36 ±.01 81.22 ±9.03 91.43 ±3.56 Open in a new tab Fig. 21. Open in a new tab Comparison of proposed models’ performance. Overall, as shown in Fig. 21 , the PCA_Model achieves the highest scores across all metrics, except for recall. This indicates a slight weakness in identifying positive samples, which, in our case, correspond to high-quality health information, as defined in the Methods section. Nevertheless, the PCA_Model achieves the highest score (91) and MCC (82), demonstrating that the model provides balanced performance across both classes, as further supported by the results in Table 2 . Regarding the Recall metric, the Entropy_Model achieves the highest score of 89, demonstrating its exceptional performance in identifying positive samples and showing that the model does not have a basis in predicting the negative class very often. The Medical Enhanced Model follows closely, achieving a recall value of 87. Finally, the ICA Model achieved the lowest recall, indicating that it is the weakest of the proposed models at identifying positive samples. Almost all models outperformed the base model in terms of , accuracy, and MCC. The PCA_Model achieved the highest scores, with an score of 91, an accuracy of 93, and an MCC of 82, followed closely by the ICA model and the Entropy_Model, with a slight margin of difference. Finally, regarding the k-fold cross-validation results presented in Table 2 . The PCA model demonstrates consistently high performance across all measured metrics, while other models show competitive results in specific individual metrics. Interestingly, the base model performs slightly better on a few metrics than the proposed models. Nevertheless, the proposed model demonstrates distinct advantages over the base model in overall performance. Assessment of model uncertainty and calibration To evaluate the uncertainty and calibration of the models, this study employs five key metrics: ECE_High (Expected Calibration Error for high-quality information),ECE_Low (Expected Calibration Error for low-quality information), Test_Entropy, AC_Correct_Predictions (average confidence for correct predictions), and AC_Incorrect_Predictions (average confidence for incorrect predictions), as illustrated in Fig. 22 . Fig. 22. Open in a new tab Comparison of the performance of proposed models in terms of uncertainty and calibration. Regarding ECE_High, the ICA_Model achieves the highest Expected Calibration Error with a score of 0.80, followed closely by the Medical_Enhanced_Model with a score of 0.77 and the PCA_Model with a score of 0.71. Interestingly, the Base Model achieves the lowest Calibration Error of 0.14, followed closely by the Entropy_Model, with a score of 0.19. For ECE_Low, the Base_Model achieves the highest Expected Calibration Error (ECE) score of 0.20, followed closely by the Entropy_Model and the Medical Enhanced Model, each with a score of 0.18. The ICA_Model and PCA_Model achieved the lowest Expected Calibration Error, with scores of 0.05 and 0.12, respectively. Overall, the results indicate that calibration performance varied across model architectures. The calibration improvement was class-dependent: PCA and ICA performed better for low-quality predictions, and Base performed better for high-quality ones; however, the Entropy-based model achieved balanced performance across both classes, with ECE_High = 0.19 and ECE_Low = 0.18, indicating its robustness in maintaining consistent calibration. The results of the entropy metric showed variation across the different models, with the ICA_Model and PCA_Model obtaining the lowest entropy scores of 0.12 and 0.19, respectively. These results are noticeable in the probability distributions for class predictions, as shown in Figs. 25 and 26 . In contrast, the Base_Model and Entropy_Model achieved the highest score of 0.35, as apparent in the probability distributions for class predictions 23 . The Medical_Enhanced_Model demonstrated moderate performance, scoring 0.29. As illustrated in Fig. 24 , the Entropy_Model produces smoother probability transitions and fewer extreme (0 or 1) predictions than other models, particularly around the class decision boundary. This indicates more stable prediction behavior and reduced abrupt confidence changes for both high- and low-quality samples. Fig. 25. Open in a new tab The probability distribution of the ICA model for the two classes. Fig. 26. Open in a new tab The probability distribution of the PCA model for the two classes. Fig. 23. Open in a new tab The predicted probabilities of the Entropy Model for both classes show smoother transitions and fewer extreme predictions compared with other models. Fig. 24. Open in a new tab The probability distribution of the medical enhanced model for the two classes. In the Figs. 23 , 24 , 25 , and 26 , the Y-axis represents the probability of assigning a sample to the high-quality (blue) or low-quality (orange) class. At the same time, the X-axis (Sample Index) groups individual samples into high-quality and low-quality classes based on their actual labels. The test dataset contains 30% of the total web page samples, including 38 high-quality web pages shown in the first part of the Figure and 113 low-quality web pages in the second part. The model will output two probabilities for each webpage, corresponding to high-quality and low-quality classifications. The ideal model with zero entropy would assign a probability of 1 to all 38 high-quality web pages presented in the first part of the Figs. 23 , 24 , 25 , and 26 , and a probability of 0 to the remaining 133 web pages. Conversely, in the second part, the model would assign a probability of 1 to the 133 low-quality web pages and a probability of 0 to the 38 high-quality web pages. The results indicate that these models still produce reliable results despite some uncertainty in identifying high-quality content. The results for model confidence showed that both the ICA_Model and PCA_Model achieved the highest Average confidence in correct predictions, but also exhibited the highest confidence in incorrect predictions, indicating a tendency toward overconfidence in their predictions. The Base_Model and Entropy_Model achieved similar scores in average confidence for correct predictions. However, the Entropy_Model had the lowest average confidence for incorrect predictions, making it a less overconfident model. Finally, the Medical_Enhanced_Model demonstrated the best performance among all the models, achieving an average confidence of 0.91 for correct predictions and 0.69 for incorrect predictions. This indicates a clear separation in confidence between correct and incorrect predictions, as illustrated in Fig. 27 . Fig. 27. Open in a new tab Confidence distribution histograms for correct and incorrect Prediction. Per-class performance comparison of proposed models Figure 28 presents a comparative analysis of the classification performance of the five proposed models: Base_Model, Entropy_Model, ICA_Model, PCA_Model, and Medical_Enhanced_Model, on both low-quality and high-quality health information classes, using the training and testing splits. For the cross-validation performance, refer to Table 2 . The results indicate that ICA_Model achieved the highest accuracy of 97.3% for the low-quality class, followed by PCA_Model 96.46%, while the Base_Model showed the lowest accuracy of 90.27%. On the other hand, for the high-quality class, the Entropy_Model demonstrated the best performance with an accuracy of 89.4%, outperforming the other proposed models, whereas ICA_Model had the lowest accuracy of 76.3%. These findings suggest that feature weighting techniques, such as entropy, ICA, and PCA, have varying contributions to the prediction task, underscoring the trade-offs in accuracy between low and high-quality health information detection. Fig. 28. Open in a new tab Evaluating proposed model performance on a per-class basis. The Entropy_Model provided the most balanced performance across both low-quality and high-quality classes. It achieved 92.04% accuracy for the low-quality class and 89.4% accuracy for the high-quality class, showing relatively high scores for both classes without extreme variations. In contrast, the ICA_Model, while excelling in low-quality class 97.3%, performed the worst in high-quality class 76.3%, indicating an imbalanced performance. In summary, the analysis highlights that some of the proposed models showed unbalanced performance in classifying low and high quality information, except for Entropy_Model and Medical_Enhanced_Model, which achieved the most balanced performance. Therefore, these models are preferable for applications that require consistent classification accuracy for both low and high quality health information classes. Error analysis The error rates resulting from the per-class accuracy analysis, as shown in Fig. 28 , indicate that most misclassifications occurred in the high-quality health information class, with error rates ranging from 10.6% to 23.7%. In contrast, the low-quality health information class consistently showed error rates below 10% across all models. The prediction probability distributions in Figs. 23 , 24 , 25 , and 26 further confirmed this imbalance. While predictions for low-quality webpages were consistently confident, with probabilities clustering near 1.0, predictions for high-quality webpages showed little variability, with many errors occurring near the decision threshold of 0.5. A smaller subset of overconfident errors, caused by assigning confidence levels greater than 0.9 to the wrong class, highlights the importance of verifying outputs with human guidance in real-world applications.S uch misclassifications can have practical implications, as incorrect labeling of high-quality content may discourage users from trusting reliable medical advice. Conversely, overconfident predictions on low-quality webpages could mislead consumers into accepting inaccurate or unsafe health information, emphasizing the societal importance of accurate and well-calibrated AI systems in health contexts. Non-linear embedding results with UMAP The proposed UMAP model achieved an accuracy of 81% with an F1-score of 0.77, indicating balanced performance across classes. Calibration analysis revealed moderate reliability, with lower calibration error for low-quality health information predictions (ECE = 0.20) than for high-quality health information predictions (ECE = 0.46). The 5-fold cross-validation confirmed the model stability, with an average training accuracy of 75.65% (std = 1.54) and test accuracy of 75.0% (std = 3.91). Moreover, the average confidence for correct predictions (0.75) exceeded that for incorrect predictions (0.68), suggesting that the model is generally well-calibrated. Discussion Contribution summary This paper proposes a solution to enhance the certainty and calibration of deep learning models for predicting the quality of health information. Specifically, it investigates five distinct architectures: an enhanced medical version of Arabic BERT, two models incorporating feature extraction techniques, including PCA, ICA, and UMAP for validation, and a model with modified loss functions integrating information entropy. The results demonstrated that these approaches improved the certainty and calibration of health information evaluation, surpassing human performance. Comparative evaluation with prior studies All cross-paper comparisons are presented for contextual understanding. Since prior studies do not provide the prediction-level outputs required to compute confidence intervals or p-values, the reported differences should be interpreted as descriptive, non-inferential trends rather than statistically significant improvements. Our study presents findings that contradict previously reported results. Specifically, we found that paraphrasing does not yield the best performance 31 , 86 , 87 . In contrast, back-translation provides better results 88 with these models’ architecture and dataset under the chosen hyperparameters. Nonetheless, both augmentation methods yield very close results. RandAugment produced the most suboptimal performance, which may be attributed to syntactic or semantic distortions, likely arising from its limited compatibility with Arabic linguistic structures 87 . Recent machine learning systems are designed to automate tasks traditionally performed by humans, and integrating human-level performance into evaluating health information quality offers two crucial advantages 19 , 89 . First, labelled data can be readily obtained from health professionals. Second, error analysis benefits from the intuition and cognitive skills of health specialists. Human-level error can serve as a proxy for the Bayes error, providing an estimate of the optimal error and performance rates 19 , 89 . For example 24 , reports a manual accuracy of 94%, implying an optimal error rate of 6% and an optimal model performance of 94% for that dataset. In other words, when a model’s error approaches the Bayes error, it indicates performance near the theoretical optimum for the given dataset. Conversely, if the model’s error substantially exceeds the Bayes error, this signals the need for further model improvements. In the context of previous studies 22 , our proposed PCA-based model achieved an accuracy of 94.7%, which is in line with the average human performance level of 94.3% reported in the systematic review 22 and on the same dataset 31 . However, it should be emphasized that this comparison is not perfectly one-to-one, since the human performance values reported in the systematic review were pooled from multiple datasets and assessment frameworks. Therefore, while our results indicate that the proposed model achieves human-comparable accuracy, a prospective evaluation on the same datasets with expert human raters is recommended to establish a fair and direct comparison. One possible explanation for this result is that the PCA feature space enables the model to clearly distinguish between high and low quality health information. PCA model achieved the best performance, with an accuracy of 94.7% and an F1 score of 0.91. It demonstrated strong calibration for low-quality health information cases but showed slightly higher error in high-quality predictions. In cross-validation, PCA maintained strong performance across both classes, as presented in Table 2 . ICA model performed similarly, with an accuracy of 0.92 and an F1 score of 0.89, showing excellent calibration for low-quality cases but slight underperformance in high-quality cases. UMAP model achieved 0.81% accuracy and an F1 score of 0.77%. While its overall accuracy was lower than PCA and ICA, it provided balanced precision-recall trade-offs, with an ECE of 0.20 for low-quality cases and 0.46 for high-quality cases, making it a useful nonlinear alternative despite lower F1 performance. Our study confirms that no prior research aligns closely with our methodology, apart from similarities in dataset language and the application of deep learning approaches 20 . Notably, our best-performing model surpasses previous work, which reported an accuracy of 79% and an F1 score of 77%, by achieving 94.7% accuracy and a 91% F1 score, with respective improvements of 17.7% and 14% on the training and test splits. The study 25 proposed an automated classification of webpages about early childhood vaccination, reporting F1 scores of 0.54–0.86.54.86 for unreliable information and 0.82–0.91.82.91 for reliable information, with human performance at 96%. Another study 24 , which evaluated the quality of online health information using hierarchical encoder attention-based neural networks (HEABERT), achieved average F1-macro scores of 0.75(HEABERT), 0.74 with BioBERT, and 0.69 with Random Forest, with overall accuracies ranging from 81% to 86%, comparable to human performance of 94%. Finally, the study 26 evaluating online health information for older adults regarding Alzheimer’s disease reported an F1 score of 90% and human performance of 93%. Our method surpasses previous approaches, with the best models achieving 94.7% accuracy and an F1 score of 91%. In distinguishing between reliable and unreliable health information, the top-performing model achieved 95.29% accuracy for unreliable information and 94.16% for reliable information. These results represent improvements of approximately 9.29% in accuracy over the best unreliable classification and 3.16% over the best reliable classification reported in 25 , 17% in F1 over BioBERT in 24 , and 1% in F1 over 26 , while also exceeding human-level performance, an achievement not previously reported for this task. In contrast to previous methods 19 , our method incorporates quality dimensions that reflect actual human values, as demonstrated in this study 30 . In addition, this investigation introduces, for the first time, an analysis of various certainty and calibration metrics, as shown in Fig. 22 . Our results indicate that the proposed models demonstrate improved performance over the base model, with noticeable gains in certainty and calibration metrics. According to 19 , using various quality criteria is crucial for addressing the complex aspects of health information quality 90 . Acknowledging that each quality criterion provides unique advantages 19 . Therefore, this study adopts approximately 15 quality dimensions proposed by 30 , excluding Privacy, Transparency, and Financial Disclosure, to comprehensively evaluate the quality of health information. The source quality dimensions focus on assessing the trustworthiness of information providers, whereas the content quality criteria examine the accuracy and credibility of the information itself. Contrary to the study by 24 , our work leverages deep learning to explore the complex knowledge landscape of health information quality, generate and test hypotheses across the 15 proposed dimensions, and identify valuable combinations 90 . Unlike the previous study 24 , which constrained models to learn only specific dimensions, our approach includes all quality dimensions as input text without forcing the model to predict them. This allows the deep learning model to autonomously discover meaningful combinations and interactions, leveraging its full expressive capacity. None of the previous studies in the literature evaluate the proposed model’s performance on a per-class basis, which makes direct comparison with prior work difficult. Nonetheless, the results in Fig. 28 show that the proposed models outperform the base model. In terms of error analysis, most errors occurred in the high-quality health information cases, with error rates ranging from 10% to 23.7%. In contrast, low-quality cases were rarely misclassified, with error rates below 10% in the training and test split. The prediction probability distribution plots further indicate that errors clustered around the 0.5 decision threshold, with only a few overconfident misclassifications above 0.9. These findings suggest that probability scores can support uncertainty identification in a human-based review. To our knowledge, no previous studies have examined these aspects of the models, further limiting the establishment of a direct benchmark for comparison. Nonetheless, our best-performing PCA model (Table 2 ) achieves comparable results for both low and high-quality health information, with only a 1.13% difference: 94.16% accuracy for high-quality data and 95.29% accuracy for low-quality data. Ethical and AI responsibility considerations Contrary to the Existing solutions, which rely on limited textual representation features and their associated weighting, such as TFIDF 20 , 25 , bag-of-words 25 , or basic word embeddings 23 , 24 , 27 , which fail to capture the syntactic and semantic complexity of health information. Our proposed solution uses Principal Component Analysis (PCA) and Independent Component Analysis (ICA), among other techniques, to enhance the BERT architecture. This approach yields a more informative feature space, improving the model’s calibration and predictive certainty, as seen in the results section. While AI models offer opportunities to enhance the assessment of health information quality, their use also raises ethical concerns, particularly regarding bias, misinformation risks, and overreliance on automation. In this study, the models exhibit a systematic bias toward predicting low-quality rather than high-quality health information, primarily due to data imbalance. This finding is consistent with previous studies 91 – 94 . Moreover, in certain cases, the proposed models produce misclassifications with high confidence, as illustrated in Fig. 26 , which can be regarded as a form of misinformation–a phenomenon similarly reported in other works 95 – 99 . Automation remains one of the most valuable features of AI in healthcare, delivering efficiency gains across diverse contexts. Nonetheless, while our study and recent work demonstrate that large language models are approaching expert-level performance in specific medical domains, such as ophthalmology 100 , 101 , the extensive literature supports viewing these tools as augmenting rather than replacing healthcare professionals 96 , 102 – 104 . Implications and future research The findings of this study have several significant implications. First, it demonstrated the impact of feature weighting techniques, such as PCA and ICA, on model certainty and calibration. Consequently, future research could explore alternative feature weighting methods for different aspects of health information quality dimensions to comprehensively enhance model certainty and calibration. Second, the study highlights the importance of improving model certainty and calibration, particularly in the context of health information quality, where the risks associated with deployment significantly outweigh the potential benefits of deep learning. Future research should focus on developing more robust safety mechanisms to effectively implement deep learning models in healthcare applications and externally validating these models using diverse datasets to enhance their generalizability and applicability. In light of the class-dependent calibration observed in this study, future work could explore class-wise temperature scaling and other adaptive calibration approaches to achieve more balanced and reliable probabilistic predictions across classes. Third, the proposed models hold potential for real-world integration in several ways. First, deployment as a browser plug-in could provide users with real-time quality assessments of online health information, helping individuals identify unreliable sources during everyday browsing. Second, embedding these models into search engines as ranking mechanisms would prioritize credible, authoritative health information, thereby reducing exposure to misleading or low-quality health information. Finally, incorporation into conversational agents or health chatbots could enable the delivery of trustworthy recommendations and the identification of suspicious content during user interactions. Finally, the findings of this study highlight the need for introducing a new layer to regulate health information quality, particularly for human users and AI-powered devices, as shown in Fig. 2 . Future research should focus on enhancing this layer by introducing new elements or improving the existing ones. Limitations The study has three main limitations, the most significant being the imbalance in the dataset, which is heavily skewed toward low-quality web pages, as observed in prior research 11 , 31 , 75 , 105 – 111 , which suggested that most of the health information provided online is of low quality. This imbalance may limit the model’s generalization ability across different quality classes and online contexts. Although the proposed models demonstrate robust performance within the evaluated dataset, further external validation is necessary to confirm their stability and generalizability. Future research should therefore examine the models using data collected from different time periods, alternative search queries, and additional countries or domain types to strengthen the external validity of the findings. The second limitation relates to the assumptions of PCA and ICA. PCA assumes that variance corresponds to relevance, but this assumption might not hold for all types of health information, as high variance can sometimes represent noise rather than meaningful information. ICA assumes that the underlying signals in the text are independent and non-Gaussian. It attempts to reweight the features by reducing the influence of Gaussian features in the data, as shown in Figure 29 , which corresponds to reweighted features of the original features, which initially showed Gaussian-like distributions, as shown in Fig. 30 . However, this assumption might also not hold for health information. Fig. 29. Open in a new tab Histograms for the first Four ICA components. Fig. 30. Open in a new tab Histograms for the first four main features. The third limitation is reducing the document size to 512 tokens to meet Arabic BERT requirements, which may result in the loss of some information. Future research could address this limitation using alternative models like Longformer, which supports longer input sequences. Summary and conclusion In this article, we investigated the effects of various deep-learning models, including an enhanced Arabic BERT to improve the vector representation of medical text, feature extraction techniques (PCA and ICA), and modified loss functions that incorporate information entropy to improve model certainty and calibration during document classification. Various effective learning methods enable intelligent machines to reduce uncertainty within the knowledge space. In this work, we apply two methods influencing the feature space: Independent Component Analysis (ICA) and Principal Component Analysis (PCA). Additionally, we incorporate information entropy as a learning parameter in the cost function to eliminate features that contribute to increased model uncertainty. This study used around 15 quality dimensions from the literature, excluding Privacy, Transparency, and Financial Disclosure. It thus addresses key aspects of health information quality, including source credibility, treatment description, and content trustworthiness, contributing to more reliable health information distribution. The study’s findings demonstrated promising results, with the PCA-based proposed model outperforming all other models, although the alternatives also showed competitive performance. These results suggest the potential to improve the regulation of health information quality and to contribute to safer interactions between human users and AI-powered technologies. However, further validation, including case studies and real-world testing in contexts such as telemedicine, IoT devices, and wearable technologies, is necessary before practical deployment can be realized. Supplementary Information Below is the link to the electronic supplementary material. Supplementary material 1. (311.8KB, pdf) Supplementary material 2. (123.4KB, pdf) Author contributions Yousef Baqraf, Pantea Keikhosrokiani, and Yu-N Cheah contributed equally to this work. Data availability The data used in this study are available upon request from the first and corresponding authors. Declarations Competing interests The authors declare no competing interests. Ethics statement This research involved manual coding of web pages and a voluntary questionnaire completed by medical professionals in their professional capacities to identify suitable search terms for web pages. No personal or sensitive information was collected, and all survey responses were anonymized. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-026-43158-8. References 1. Yampolskiy, R.V.: Artificial Intelligence Safety and Security. CRC Press, ??? (2018). 2. Armstrong, S.: Introduction to the technological singularity. The Technological Singularity: Managing the Journey, 1–8 (2017). 3. Charisi, V., et al.: Towards moral autonomous systems. arXiv preprint arXiv:1703.04741 (2017). 4. Brundage, M. et al.: The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228 (2018). 5. Yampolskiy, R.V.: AI: Unexplainable, Unpredictable, Uncontrollable. CRC Press, ??? (2024). 6. Shakeri Hossein Abad, Z. et al. Digital public health surveillance: A systematic scoping review. Npj. Digit. Med. 4 (1), 41. (2021). [ DOI ] [ PMC free article ] [ PubMed ] 7. Swire-Thompson, B. et al. Public health and online misinformation: Challenges and recommendations. Annu. Rev. Public Health 41 (1), 433–451 (2020a). [ DOI ] [ PubMed ] [ Google Scholar ] 8. Blumenthal, D. Doctors in a wired world: Can professionalism survive connectivity?. Milbank Q. 80 (3), 525–546 (2002). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 9. Shortliffe, E.H., et al.: Networking health: Prescriptions for the internet. Networking Health (2000). 10. Swire-Thompson, B. et al. Public health and online misinformation: Challenges and recommendations. Annu. Rev. Public Health 41 (1), 433–451 (2020b). [ DOI ] [ PubMed ] [ Google Scholar ] 11. Daraz, L. et al. Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet. J. Gen. Intern. Med. 34 , 1884–1891 (2019). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Ghezzi, P., Chumber, S. & Brabazon, T.: Educating medical students to evaluate the quality of health information on the web. The Philosophy of Information Quality, 183–199 (2014). 13. Allam, A., Schulz, P. J. & Krauthammer, M. Toward automated assessment of health web page quality using the discern instrument. J. Am. Med. Inform. Assoc. 24 (3), 481–487 (2017). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 14. Mead, N., Varnam, R., Rogers, A. & Roland, M. What predicts patients’ interest in the internet as a health resource in primary care in England?. J. Health Serv. Res. Policy 8 (1), 33–39 (2003). [ DOI ] [ PubMed ] [ Google Scholar ] 15. Popovac, M. & Roomaney, R. Measuring online health-seeking behaviour: Construction and initial validation of a new scale. Br. J. Health Psychol. 27 (3), 756–776 (2022). [ DOI ] [ PubMed ] [ Google Scholar ] 16. Rieh, S. Y. Judgment of information quality and cognitive authority in the web. J. Am. Soc. Inf. Sci. Technol. 53 (2), 145–161 (2002). [ Google Scholar ] 17. Eysenbach, G. & Köhler, C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 324 (7337), 573–577 (2002a). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 18. Eysenbach, G. & Köhler, C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 324 (7337), 573–577 (2002b). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Baqraf, Y. K. A., Keikhosrokiani, P. & Al-Rawashdeh, M. Evaluating online health information quality using machine learning and deep learning: A systematic literature review. Digit. Health 9 , 20552076231212296 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 20. Alasmari, A., Alhothali, A. & Allinjawi, A. Hybrid machine learning approach for Arabic medical web page credibility assessment. Health Informatics J. 28 (1), 14604582211070998 (2022). [ DOI ] [ PubMed ] [ Google Scholar ] 21. Al-Jefri, M.M., Evans, R., Ghezzi, P. & Uchyigit, G.: Using machine learning for automatic identification of evidence-based health information on the web. In: Proceedings of the 2017 International Conference on Digital Health, 67–174 (2017). 22. Di Sotto, S. & Viviani, M. Health misinformation detection in the social web: An overview and a data science approach. Int. J. Environ. Res. Public Health. 19 (4), 2173 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 23. Feng, J., Zou, L., Ye, O. & Han, J. Web2vec: Phishing webpage detection method based on multidimensional features driven by deep learning. IEEE Access 8 , 221214–221224 (2020). [ Google Scholar ] 24. Kinkead, L., Allam, A. & Krauthammer, M. Autodiscern: Rating the quality of online health information with hierarchical encoder attention-based neural networks. BMC Med. Inform. Decis. Mak. 20 (1), 1–13 (2020). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Meppelink, C. S. et al. Reliable or not? An automated classification of webpages about early childhood vaccination using supervised machine learning. Patient Educ. Couns. 104 (6), 1460–1466 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 26. Robillard, J.M., Alhothali, A., Varma, S. & Hoey, J.: Intelligent and affectively aligned evaluation of online health information for older adults. In: AAAI Workshops (2017). 27. Upadhyay, R., Pasi, G. & Viviani, M.: Health misinformation detection in web content: a structural-, content-based, and context-aware approach based on web2vec. In: Proceedings of the Conference on Information Technology for Social Good, 19–24 (2021). 28. Upadhyay, R., Pasi, G. & Viviani, M. Vec4cred: A model for health misinformation detection in web pages. Multim. Tools Appl. 82 (4), 5271–5290 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 29. Zhang, Y., Sun, Y. & Xie, B. Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results. J. Assoc. Inf. Sci. Technol. 66 (10), 2071–2084 (2015). [ Google Scholar ] 30. Baqraf, Y. & Keikhosrokiani, P.: Health information quality assessment using artificial intelligence: Quality dimensions from healthcare professionals’ perspective. In: International Conference of Reliable Information and Communication Technology , 1–14 (Springer, 2023). 31. Baqraf, Y., Keikhosrokiani, P., Cheah, Y.-N., Alomarji, A. & Baqraf, F.: Artificial intelligence for automating the establishment of an arabic benchmark dataset for enhancing health information quality assessment. Discover Artificial Intelligence (2025). 32. Weitzman, M. L. Recombinant growth. Q. J. Econ. 113 (2), 331–360 (1998). [ Google Scholar ] 33. Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and scientific impact. Science 342 (6157), 468–472 (2013). [ DOI ] [ PubMed ] [ Google Scholar ] 34. Wang, J., Veugelers, R. & Stephan, P. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Res. Policy 46 (8), 1416–1436 (2017). [ Google Scholar ] 35. Fleming, L. Recombinant uncertainty in technological search. Manage. Sci. 47 (1), 117–132 (2001). [ Google Scholar ] 36. Bianchini, S., Müller, M. & Pelletier, P.: Deep learning in science. arXiv preprint arXiv:2009.01575 (2020). 37. Myriam, H. et al. Advanced meta-heuristic algorithm based on particle swarm and Al-Biruni earth radius optimization methods for oral cancer detection. IEEE Access 11 , 23681–23700 (2023). [ Google Scholar ] 38. Karim, F. K., Khafaga, D. S., Eid, M. M., Towfek, S. & Alkahtani, H. K. A novel bio-inspired optimization algorithm design for wind power engineering applications time-series forecasting. Biomimetics 8 (3), 321 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 39. Gatys, L.A., Ecker, A.S. & Bethge, M.: A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 (2015). 40. Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. nature 529 (7587), 484–489 (2016). [ DOI ] [ PubMed ] 41. Goodfellow, I.,et al.: Generative adversarial nets. Advances in neural information processing systems 27 , (2014). 42. Goodfellow, I. et al. Generative adversarial networks. Commun. ACM 63 (11), 139–144 (2020). [ Google Scholar ] 43. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 , (2015). [ DOI ] [ PubMed ] 44. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. nature 521 (7553), 436–444 (2015). [ DOI ] [ PubMed ] [ Google Scholar ] 45. Ramsundar, B., Eastman, P., Walters, P. & Pande, V.: Deep Learning for the Life Sciences: Applying Deep Learning to Genomics, Microscopy, Drug Discovery, and More. “O’Reilly Media, Inc”., ??? (2019). 46. Cockburn, I.M., Henderson, R. & Stern, S.: The impact of artificial intelligence on innovation. Working Paper 24449, National Bureau of Economic Research (2018). DOI: 10.3386/w24449. http://www.nber.org/papers/w24449 . 47. Agrawal, A., McHale, J. & Oettl, A.: Finding needles in haystacks: Artificial intelligence and recombinant growth. In: The Economics of Artificial Intelligence: An Agenda, 149–174. University of Chicago Press, ??? (2018). 48. Furman, J. L. & Teodoridis, F. Automation, research technology, and researchers’ trajectories: Evidence from computer science and electrical engineering. Organ. Sci. 31 (2), 330–354 (2020). [ Google Scholar ] 49. Dalvi, F., Sajjad, H., Durrani, N. & Belinkov, Y.: Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010 (2020). 50. Baqraf, Y. & Keikhosrokiani, P.: The prediction of health information quality perception using machine learning and deep learning techniques. In: 2023 11th International Conference on Information and Communication Technology (ICoICT), 104–109 (IEEE, 2023). 51. McInnes, L., Healy, J. & Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018). 52. Chavooshi, M. & Mamonov, A.V.: Autoencoded umap-enhanced clustering for unsupervised learning. arXiv preprint arXiv:2501.07729 (2025). 53. Ghojogh, B., Ghodsi, A., Karray, F. & Crowley, M.: Uniform manifold approximation and projection (umap) and its variants: tutorial and survey. arXiv preprint arXiv:2109.02508 (2021). 54. Charnock, D., Shepperd, S., Needham, G. & Gann, R. Discern: An instrument for judging the quality of written consumer health information on treatment choices. J. Epidemiol. Community Health 53 (2), 105–111 (1999). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 55. Sugiyama, A. & Yoshinaga, N.: Data augmentation using back-translation for context-aware neural machine translation. In: Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), 35–44 (2019). 56. Eddine, M.K., Tomeh, N., Habash, N., Roux, J.L., & Vazirgiannis, M.: Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization. arXiv preprint arXiv:2203.10945 (2022). 57. Antoun, W., Baly, F. & Hajj, H.: Arabert: Transformer-based model for arabic language understanding. In: LREC 2020 Workshop Language Resources and Evaluation Conference 9 , 11–16 (2020). 58. Khasawneh, M.A.S. & Khasawneh, Y.J.A.: Achieving assessment equity and fairness: Identifying and eliminating bias in assessment tools and practices (2023). 59. Mavrogiorgos, K., Kiourtis, A., Mavrogiorgou, A., Menychtas, A. & Kyriazis, D. Bias in machine learning: A literature review. Appl. Sci. 14 (19), 8860 (2024). [ Google Scholar ] 60. Eddine, M.K., Tomeh, N., Habash, N., Roux, J.L. & Vazirgiannis, M.: Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization. arXiv preprint arXiv:2203.10945 (2022). 61. Ciolino, M., Noever, D. & Kalin, J.: Back translation survey for improving text augmentation. arXiv preprint arXiv:2102.09708 (2021). 62. Ma, E.: NLP Augmentation. https://github.com/makcedward/nlpaug (2019). 63. Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019). 64. Safaya, A., Abdullatif, M. & Yuret, D.: KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online) 2054–2059 (2020). https://www.aclweb.org/anthology/2020.semeval-1.271 . 65. Abdi, H. & Williams, L. J. Principal component analysis. WIREs Comput. Stat. 2 (4), 433–459 (2010). [ Google Scholar ] 66. Langlois, D., Chartier, S. & Gosselin, D. An introduction to independent component analysis: Infomax and FastICA algorithms. Tut. Quant. Methods Psychol. 6 (1), 31–38 (2010). [ Google Scholar ] 67. Lee, J. et al. Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36 (4), 1234–1240 (2020). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 68. Yunianto, I., Permanasari, A.E. & Widyawan, W.: Domain-specific contextualized embedding: A systematic literature review. In: 2020 12th International Conference on Information Technology and Electrical Engineering (ICITEE), 162–167 (IEEE, 2020). 69. Sarker, I. H. Deep learning: A comprehensive overview on techniques, taxonomy, applications and research directions. SN Comput. Sci. 2 (6), 420 (2021). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 70. Brown, T. et al. Language models are few-shot learners. Advances in neural information processing systems 33 , 1877–1901 (2020). [ Google Scholar ] 71. Jasem, Z., AlMeraj, Z. & Alhuwail, D. Evaluating breast cancer websites targeting Arabic speakers: Empirical investigation of popularity, availability, accessibility, readability, and quality. BMC Med. Inform. Decis. Mak. 22 (1), 126 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 72. Albalawi, Y., Buckley, J. & Nikolov, N. S. Investigating the impact of pre-processing techniques and pre-trained word embeddings in detecting Arabic health information on social media. J. Big Data 8 (1), 95 (2021). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 73. Alkhateeb, J. M. & Alhadidi, M. S. Information about epilepsy on the internet: An exploratory study of Arabic websites. Epilepsy Behav. 78 , 288–290 (2018). [ DOI ] [ PubMed ] [ Google Scholar ] 74. Elshahidi, M. H. Arabic websites relevant to patient information about spinal cord injury: Characteristics and quality assessment using the DISCERN tool. Med. J. Babylon 15 (2), 183 (2018). [ Google Scholar ] 75. Halboub, E., Al-Ak’hali, M. S., Al-Mekhlafi, H. M. & Alhajj, M. N. Quality and readability of web-based Arabic health information on COVID-19: An infodemiological study. BMC Public Health 21 (1), 1–7 (2021). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 76. Alhajj, M. N. et al. Quality and readability of web-based Arabic health information on denture hygiene: An infodemiology study. J. Contemp. Dent. Pract. 21 (9), 956–960 (2020). [ PubMed ] [ Google Scholar ] 77. Alakhali, M. S. Quality assessment of information on oral cancer provided at Arabic speaking websites. Asian Pac. J. Cancer Prev. 21 (4), 961 (2020). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 78. Biswas, S. S. Role of ChatGPT in public health. Ann. Biomed. Eng. 51 (5), 868–869 (2023). [ DOI ] [ PubMed ] [ Google Scholar ] 79. Baclic, O. et al. Artificial intelligence in public health: Challenges and opportunities for public health made possible by advances in natural language processing. Canada Communicable Disease Report 46 (6), 161 (2020). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 80. Panch, T., Pearson-Stuttard, J., Greaves, F. & Atun, R. Artificial intelligence: Opportunities and risks for public health. Lancet Digit. Health 1 (1), 13–14 (2019). [ DOI ] [ PubMed ] [ Google Scholar ] 81. Baroni, M., Kilgarriff, A., Pomikálek, J. & Rychlỳ, P.: Webbootcat. instant domain-specific corpora to support human translators. In: Proceedings of the 11th Annual Conference of the European Association for Machine Translation (2006). 82. Kilgarriff, A. et al. The sketch engine: ten years on. Lexicography 1 (1), 7–36 (2014). [ Google Scholar ] 83. Pavlovic, M.: Understanding model calibration-a gentle introduction and visual exploration of calibration and the expected calibration error (ece). arXiv preprint arXiv:2501.19047 (2025). 84. Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 16 , 321–357 (2002). [ Google Scholar ] 85. He, H., Bai, Y., Garcia, E.A. & Li, S.: Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In: 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), 1322–1328 (Ieee, 2008). 86. Refai, D., Abo-Soud, S. & Abdel-Rahman, M. Data augmentation using transformers and similarity measures for improving Arabic text classification. IEEE. Access. 10.1109/access.2023.3336311 (2023). [ Google Scholar ] 87. ElSabagh, A. A., Azab, S. S. & Hefny, H. A. A comprehensive survey on Arabic text augmentation: Approaches, challenges, and applications. Neural. Comput. Appl. 10.1007/s00521-025-11020-z (2025). [ Google Scholar ] 88. Ozolins, U., Hale, S., Cheng, X., Hyatt, A. & Schofield, P. Translation and back-translation methodology in health research-a critique. Expert Rev. Pharmacoecon. Outcomes Res. 20 (1), 69–77 (2020). [ DOI ] [ PubMed ] [ Google Scholar ] 89. Ng, A.: Machine Learning Yearning: Technical Strategy for Ai Engineers in the Era of Deep Learning, (2019). 90. Bianchini, S., Müller, M., Pelletier, P.: Deep learning in science. arXiv preprint arXiv:2009.01575 (2020). 91. Canca, C.: Did you find it on the internet? ethical complexities of search engine rankings. Perspectives on Digital Humanism 135 (2022). 92. Loeb, S. et al. Dissemination of misinformative and biased information about prostate cancer on YouTube. Eur. Urol. 75 (4), 564–567 (2019). [ DOI ] [ PubMed ] [ Google Scholar ] 93. Loeb, S. et al. Quality of bladder cancer information on YouTube. Eur. Urol. 79 (1), 56–59 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 94. Betschart, P. et al. Information on surgical treatment of benign prostatic hyperplasia on YouTube is highly biased and misleading. BJU Int. 125 (4), 595–601 (2020). [ DOI ] [ PubMed ] [ Google Scholar ] 95. Biswas, S., Logan, N. S., Davies, L. N., Sheppard, A. L. & Wolffsohn, J. S. Assessing the utility of ChatGPT as an artificial intelligence-based large language model for information to answer questions on myopia. Ophthalmic Physiol. Opt. 43 (6), 1562–1570 (2023). [ DOI ] [ PubMed ] [ Google Scholar ] 96. Høj, S. et al. Evaluating the scientific reliability of ChatGPT as a source of information on asthma. J. Allergy Clin. Immunol. Glob. 3 (4), 100330 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 97. Tang, L. et al. Evaluating large language models on medical evidence summarization. Npj Digit. Med. 6 (1), 158 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 98. Cocci, A. et al. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer Prostatic Dis. 27 (1), 103–108 (2024). [ DOI ] [ PubMed ] [ Google Scholar ] 99. Faisal, S. et al. Evaluating the comprehension and accuracy of ChatGPT’s responses to diabetes-related questions in Urdu compared to English. Digit. Health. 10 , 20552076241289730 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 100. Thirunavukarasu, A. J. et al. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study. PLOS Digit. Health. 3 (4), 0000341 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 101. Delsoz, M. et al. The use of ChatGPT to assist in diagnosing glaucoma based on clinical case reports. Ophthalmol. Ther. 12 (6), 3121–3132 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 102. Samaan, J. S. et al. Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obes. Surg. 33 (6), 1790–1796 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 103. Chang, L.-C. et al. Evaluation of the quality and readability of ChatGPT responses to frequently asked questions about myopia in Traditional Chinese language. Digit. Health 10 , 20552076241277020 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 104. Ong, Q. C. et al. Advancing health coaching: A comparative study of large language model and health coaches. Artif. Intell. Med. 157 , 103004 (2024). [ DOI ] [ PubMed ] [ Google Scholar ] 105. Al-Ak’hali, M. S., Fageeh, H. N., Halboub, E., Alhajj, M. N. & Ariffin, Z. Quality and readability of web-based Arabic health information on periodontal disease. BMC Med. Inform. Decis. Mak. 21 , 1–8 (2021). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 106. Shazil Jamal, M. et al. Evaluation of the quality of information online for arthroscopic Bankart repair. Ann. R. Coll. Surg. Engl. 105 (5), 394–399 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 107. Batar, N., Kermen, S., Sevdin, S., Yıldız, N. & Güçlü, D. Assessment of the quality and reliability of information on nutrition after bariatric surgery on YouTube. Obes. Surg. 30 (12), 4905–4910 (2020). [ DOI ] [ PubMed ] [ Google Scholar ] 108. Corcelles, R., Daigle, C. R., Talamas, H. R., Brethauer, S. A. & Schauer, P. R. Assessment of the quality of internet information on sleeve gastrectomy. Surg. Obes. Relat. Dis. 11 (3), 539–544 (2015). [ DOI ] [ PubMed ] [ Google Scholar ] 109. Koball, A. M. et al. Content and accuracy of nutrition-related posts in bariatric surgery Facebook support groups. Surg. Obes. Relat. Dis. 14 (12), 1897–1902 (2018). [ DOI ] [ PubMed ] [ Google Scholar ] 110. Palma, A. F. et al. Quality of information for women seeking breast augmentation in the Internet. J. Plast. Surg. Hand Surg. 50 (5), 262–271 (2016). [ DOI ] [ PubMed ] [ Google Scholar ] 111. Kloosterboer, A., Yannuzzi, N. A., Patel, N. A., Kuriyan, A. E. & Sridhar, J. Assessment of the quality, content, and readability of freely available online information for patients regarding diabetic retinopathy. JAMA Ophthalmol. 137 (11), 1240–1245 (2019). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Supplementary material 1. (311.8KB, pdf) Supplementary material 2. (123.4KB, pdf) Data Availability Statement The data used in this study are available upon request from the first and corresponding authors. Articles from Scientific Reports are provided here courtesy of Nature Publishing Group ACTIONS View on publisher site PDF (7.0 MB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Relapse prediction and individualized treatment-effect modeling in relapsing multiple sclerosis: a systematic review.#1016102