A multi-modal deep learning framework for enhanced breast cancer diagnosis using mammograms and clinical data.

A multi-modal deep learning framework for enhanced breast cancer diagnosis using mammograms and clinical data - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Sci Rep . 2026 May 3;16:17175. doi: 10.1038/s41598-026-48085-2 A multi-modal deep learning framework for enhanced breast cancer diagnosis using mammograms and clinical data Adam M Ibrahim 1 , Jianqiang Li 1 , Faheem Akhtar 1, 2, ✉ , Anis Ahmad Chaudhary 3, ✉ , Mohamed A M Ali 3 , Haneef Khan 4 , Musa Osman 1 Author information Article notes Copyright and License information 1 Faculty of Information Technology, Beijing University of Technology, Beijing, 100124 China 2 Department of Computer Science, Sukkur IBA University, Sukkur, 65200 Pakistan 3 Department of Biology, College of Science, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, Saudi Arabia 4 Department of Electrical and Electronic Engineering, College of Engineering and Computer Sciences, Jazan University, Jazan, Saudi Arabia ✉ Corresponding author. Received 2025 Jul 26; Accepted 2026 Apr 6; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ . PMC Copyright notice PMCID: PMC13234131  PMID: 42071031 Abstract The fatality due to breast cancer is one of the most universal causes among women worldwide. The limited interpretation is due to tissue overlapping, high breast density and delicate lesion characteristics even though mammography is the standard screening tool. To overcome this universal challenges, we propose a multi-model deep learning framework that integrates mammographic image with structured clinical data (note: synthetic clinical variables were employed based on the known distributions of the risk factors due to the lack of real paired data) to achieve the accuracy in the diagnostics. Architectural developments establish promising directions for future multi-modal tactic with reliable clinical data. The architecture utilize a hierarchical fusion approach that incorporates with cross model attention mechanism and dynamic modality weighting to efficiently integrate heterogeneous features. Moreover, a region of interest care module is integrated to highlight crucial anatomical regions in mammograms. Experimental evaluations on a benchmark dataset validate a classification accuracy of 94.6% (95% CI: 92.8–96.4%) and an AUC of 0.963 (95% CI: 0.951–0.975), expressively overtaking prevailing baseline models ( p < 0.01). These results suggested the significant potential of projected architectural framework as a proof-of-concept for steadfast and easily explainable clinical decision supporting tools. Keywords: Breast cancer, Diagnosis, Multi-modal fusion, Mammography, Clinical data Subject terms: Cancer, Computational biology and bioinformatics, Health care, Mathematics and computing, Medical research Introduction Breast cancer is the most diagnosed cancer and a major cause of cancer related deaths in women around the world with about 2.3 million new cases and 685,000 deaths reported annually 1 . Early diagnosis and detection would be critical in improving survival possibilities with early diagnosis of breast cancer, 5-year survival rates can increase upto 90% 2 . Mammography is the main screening test of breast cancer and in the clinical applications, the most commonly used imaging modalities are the digital mammography (DM) and the digital breast tomosynthesis (DBT) in clinical practice 3 . Despite its widespread use, the interpretation of mammograms sets a serious challenge for the radiologists. They have to interpret intricate tissue images, detect small lesions, and learn to distinguish normal and cancerous lesions- all of which they have to navigate around problems of overlapping tissue, change in tissue density, and the uneven quality of images 4 . Such problems are associated with false positives, which causes unnecessary biopsy and patient suffering, and false negatives, which cause late diagnosis and treatment 5 . The sensitivity of mammography is usually 70-90% (in women with dense breast tissues, it is lower) and specificity decreases between 80 and 95% respectively 6 . Artificial intelligence (AI), particularly Deep learning (DL) have become a potentially promising technology to analyze medical images (such as mammography) 7 , 8 . Most recent advances in convolutional neural networks (CNNs) and Vision Transformers show performance that matches or even increases than the performance of expert radiologists in mammography interpretation tasks 9 , 10 . However, most of the existing strategies only consider imaging data and ignore valuable clinical data that may be useful in enhancing accuracy of diagnosis. Clinical data, such as demographics, medical history, hormonal influences, genetic indicators, and risk assessment scores, may be the key information in the diagnosis and risk categorization of breast cancer 11 . Risk factors have been well established as age, family history, genetic mutations (e.g., BRCA1/BRCA2), exposure to hormones, and breast density are overall risk factors of breast cancer 12 . We can potentially have better diagnoses since the clinical data will be combined with mammographic results and a better comprehension of the individual patient. Multi-modes that combine information in different data sources have expressed high promise in several medical applications 13 , 14 . Multi-modal models can be more effective than single-modality models because of the opportunities to capitalize on the complementary information that is generated through various modalities. Nonetheless, there are serious difficulties, such as modality alignment, the modeling of interactions between features and missing or noisy data, in integrating heterogeneous models (images and clinical information) 15 . The paper discusses architectural solutions to incorporating mammographic images with clinical information. We have to admit a serious drawback: our experiments are based on the artificial clinical variables, not on real patient data. Hence, our work should be interpreted as a study of multi-modal architectures but not a proven diagnostic instrument that could be used in the field of clinical practice. We address the problems of multi-modal integration with a number of key innovations: 1. Hierarchical fusion strategy : This is a combination of both modalities at various levels of abstraction and thus the model has the ability to provide a view of complementary information at various levels of semantic levels. 2. Cross-modal attention mechanism : The dynamic exchange of information between modalities that enables the modalities to concentrate on the appropriate features of the other. 3. Dynamic modality weighting : Each modality is adjusted automatically in terms of the quality and reliability of data, enhancing its resilience to noise or missing data. 4. Region-of-interest (ROI) attention module : Provides automatic identification and emphasis on suspicious areas in mammograms simulating the visual scan of the images by experienced radiologists. 5. Handling of specialized clinical data : Processes different types of patient data (categorical, numerical and temporal). 6. Comprehensive explainability features : Gives explainable information about the diagnostic decisions of the model, and this is a vital feature in developing clinical trust and easier adoption. The rest of this paper is organized in the following way: Sect.  2 will provide a review of related literature on the use of deep learning in diagnosing breast cancer, multi-modal learning, and explainable AI. A description of our proposed multi-modal approach is given in Sect.  3. Section  4 describes our experimental setting, such as datasets, implementation, and metrics of evaluation. The results of the experiment are discussed and reported in Sect.  5. Lastly, Sect.  6 summarizes the paper and gives future research directions. Related work Deep learning for mammogram analysis Deep learning methods of analysis of mammograms have evolved considerably over the past decade. Early research focused mainly on modifying generic CNN networks (VGG, ResNet, DenseNet, etc.) to mammogram classification 16 , 17 . These initial methods considered the mammogram analysis as a simple image classification problem with minor consideration of the peculiarities and intricacies of the mammographic images. Recent studies have initiated to integrate domain-specific information and architectural advances which can be tailored to the needs of mammogram analysis. For example, Zhao et al. 18 developed a multi-view improve the classification accuracy. Wu et al. 19 proposed a deep multi-instance learning model that can identify and aggregate the data from various regions of interest in a mammogram. Specifically, McKinney et al. 9 developed a set of deep learning models that can perform comparable to expert radiologists in breast cancer screening, which proves that AI can truly change the way clinical practice is performed. Recently, transformer-based designs have been applied in the analysis of mammograms since they not only have the potential of modeling the complex relationships between image regions, but also, they can model long distance relationships. The transformer architecture projected by Yala et al. 20 was adapted to mammography images and positional encoding is tailored to mammographic images. Based on this, Chen et al. 21 suggested hybrid CNN-transformer which combines the power of the two models to enhance feature extraction. Despite these advances, most recent approaches still focus exclusively on imaging data, overlooking valuable clinical information that could enhance diagnostic accuracy. Furthermore, many of these methods operate as “black boxes” with limited interpretability, which remains a significant barrier to clinical adoption and trust among healthcare professionals. Multi-modal learning in mammography Multi-modal learning has shown promising results in the number of medical applications that involve integrating information from multiple data sources. Specifically, in breast cancer diagnosis, a number of researchers have explored different way to integrate mammographic images with other modalities like ultrasound, MRI or even clinical data. Luo et al. 22 developed a multi-modal deep learning model, which integrates mammography, ultrasound, and MRI in order to enhance breast cancer diagnosis. This strategy applied a late fusion method, where each modality was trained independently and their output combined by a weighted average. Similarly, Wang et al. 23 suggested a hierarchical attention network which helps in predicting risks of breast cancer by incorporating clinical risk factors with mammographic image. They relied on the attention mechanisms to detect the relevant features in each of the modalities and then combined them with the help of feature concatenation. Rodriguez-Ruiz et al. 24 investigated how AI and radiologists work together in detecting breast cancer and found out that the two methods had better performance when used together than when used individually. Yala et al. 25 in another research designed a deep learning model that considers both clinical risk factors and mammographic images, showing better risk prediction and the importance of the combination of multiple data sources. Although these methods have shown positive outcomes, they are often based on comparatively simple fusion methods, e.g. late fusion or concatenation of features, which do not necessarily account for the polysemy of modalities. Also, most of those approaches give each modality some fixed weights, which can be inappropriate when the quality of data differs among patients or modalities. Explainable AI in medical imaging Explainable AI (XAI) has become a highly important research area in medical systems, where explanations of the decisions of the model are most important in developing clinical trust and facilitating adoption. In the case of mammogram analysis, the researchers have proposed a number of methods to provide interpretable information on model predictions. Visualization tools like Grad-CAM or integrated gradients, which are gradient-based, have been widely applied to highlight zones in mammograms, with a significant impact on model predictions 26 , 27 . The approaches generate heatmaps that visually explain model decisions by revealing which regions of the image were most influential. Attention mechanisms have also become useful in improving explainability since it intrinsically emphasizes on relevant features or areas of input data 28 . Using attention maps, clinicians can gain insights into which specific areas of the mammogram the model is focusing on, which may help them in detecting potential suspicious areas of the mammogram, which may require further investigation. In addition to the approaches of interpretability which are based on visualization, there are also recent studies which aim at providing textual explanations in order to describe model predictions. Selvaraju et al. 26 suggested a framework that generates a natural language explanation of what was seen and what feature caused the model to make a decision, in terms of mammogram classification. These textual descriptions when coupled with visual explanation may give a more detailed explanation of the reasoning process of the model. The feature attribution methods including SHAP (SHapley Additive explanations) and LIME (Local Interpretable Model-agnostic Explanations), have been used to measure the amount of different features to estimate the model prediction 27 , 29 . Such methods are especially effective and useful in multi-modal approaches for understanding how clinical features influence the model’s decisions. Although such explainability methods have potential, it is still difficult to successfully incorporate them into multi-modes. The majority of the existing approaches concentrate on the explanation of either image or clinical aspects alone, with little consideration of how the two modalities interrelate with each other- such as is especially necessary in multi-modal diagnostic systems. Some of the recent works have contributed well to the mammogram analysis and processing of breast masses. Xu et al. 49 proposed a multi-scale attention-guided network (MSANet) of mammogram classification, and it uses multi-scale feature extraction to enhance classification. The same group proposed ARF-Net 50 , an adaptive receptive field network of whole mammograms and ultrasound images segmentation in breast masses. Wang et al. 51 explored learning multi-frequency features in convolutional networks to classify mammography, the authors revealed that frequency-domain information can complement the discriminative ability. Zhao et al. 52 proposed an adaptive channel and multiscale spatial context network to segment breast mass on full-field mammograms, which demonstrated the significance of the spatial context modeling. Lou et al. 53 presented a detailed review of the techniques of measuring breast density on the mammogram that is pertinent to the issue of the difficulties of dense breast tissue. Additionally, Lou et al. 54 developed MGBN, which is a multi-level global-guided branch-attention network that is used to classify benign and malignant breast masses automatically. Pi et al. 55 proposed FS-UNet, which is an encoder-decoder network that uses feature strengthening in mass segmentation in mammographs and Lou et al. 56 developed DCANet which is a dual contextual affinity network of mass segmentation of whole mammograms. Moreover, it is also taught by recent developments in medical image segmentation and analysis, such as attention-based architectures 57 , 58 and hybrid CNN-transformer models 59 , 60 for various clinical applications, reinforcing the potential of our proposed multi-modal approach. Proposed methodology In this section, we introduce our innovative multi-modal deep learning approach for the diagnosis of breast cancer, which utilizes clinical data and mammograms. The architecture that has been proposed is intended to effectively incorporate information from both modalities and resolve the challenges of multi-modal learning in medical applications. The proposed approach is illustrated in Fig. 1 . Fig. 1. Open in a new tab The multi-modal deep learning approach for breast cancer diagnosis that has been proposed has an overall ar- chitecture. The architecture is comprised of four primary components: (1) Diagnostic Decision Module with explainability components, (2) Multi-Modal Fusion Module with cross-modal attention mechanisms, (3) Mammogram Processing Module with region-of-interest attention, and (4) Clinical Data Processing Module for heterogeneous patient information. Mam- mographic images and clinical data are integrated at various abstraction levels, allowing for a comprehensive breast cancer diagnosis that provides interpretable insights. Overall architecture Our architecture is composed of a five component architecture including: 1. Mammogram Processing Module : The current module uses an integration of transformer-based architectures and CNN to process mammographic images in various perspectives (CC and MLO) and obtain relevant features. It uses the region-of-interest (ROI) attention model to automatically detect and focus on suspicious areas. 2. Clinical Data Processing Module : This module will deal with different clinical data, including demographic details, medical history, hormonal attributes, genetic markers, as well as clinical risk assessment scores. The approach produces homogenous representations of features and uses dedicated processing methods of different types of data. 3. Multi-Modal Fusion Module : The of module uses the cross-modal attention mechanism and hierarchical fusion strategy to combine the features of the two modalities. It enables the dynamic interchange of information across modalities and information capture of complementary information through various levels of abstractions. 4. Dynamic Modality Weighting Module : The contribution of a given modality is dynamically adjusted by this module depending on the reliability and quality of the information. It produces normalized weights to combine features and uses quality assessment networks on both modes. 5. Diagnostic Decision Module : The module makes the final decision of diagnosis based on the fused multi-modal features. It consists of elaborate explainability elements and task-specific heads to different diagnostic tasks (e.g., classification, segmentation). The whole architecture is trained in the end-to-end way through a multi-task learning framework with specific loss functions of each task, that are described in detail in the subsection below. Mammogram processing module The mammogram processing unit is aimed at extracting the pertinent features of mammographic image and combating the issues of mammogram analysis like high resolution, tissue overlaps and the small abnormalities. The images of both craniocaudal (CC) and mediolateral oblique (MLO) views are processed in the module; these two views present complementary information regarding breast tissue. Multi-view processing The screening mammography usually consists of two standard views of each breast; craniocaudal (CC) and mediolateral oblique (MLO). Such perspectives offer complementary data with some lesions being seen in one projection only. Our system acts upon these many perspectives in the following way: 1. View-Specific Feature Extraction : View-specific features can be learned by processing each view (CC and MLO) through the initial layers of the backbone network (EfficientNet-B4 up to block 5). 2. View Attention Fusion : We implement a scaled dot-product attention mechanism to dynamically weigh the contribution of each view based on its informativeness. Following the standard self-attention formulation, we compute attention scores for each view v: 1 where Q, K, and V are the query, key, and value matrices derived from the view features, and d_k is the dimension of the key vectors. This formulation enables the model to attend to informative regions across views through learned query-key compatibility (Fig. 2 ). Fig. 2. Open in a new tab Detailed architecture of the Mammogram Processing Module. The module takes mammographic images in various perspectives (CC and MLO) and processes it with an CNN backbone (EfficientNet-B4) then transformer encoder. The region-of-interest (ROI) attention model automatically detects and gives emphasis to suspicious areas, which are similar to the visual attention patterns of the expert radiologists. The multi-view fusion mechanism is a hybridization of features of various views based on the weighted aggregation strategy, which allows the model to use the complementary information of various projections of the breast tissue. 3. Combined Feature Maps : The weights of the attention are combined to form the view specific feature maps: 2 The method is used to highlight the more informative perspective in the event of some discrepancies, and to make use of both the complementary information in the two views. As shown in Table 1 , our multi-view method is compared to single-view methods. Table 1. Comparison of multi-view processing approaches. Approach AUC Sensitivity Specificity CC view only 0.934 90.2% 87.6% MLO view only 0.941 91.5% 88.3% Late fusion (averaging) 0.952 92.8% 89.7% Our view attention fusion 0.963 94.1% 91.2% Open in a new tab These results demonstrate that our view attention fusion approach effectively leverages complementary information from multiple views, outperforming both single-view models and simple averaging of view predictions. ROI attention module Although attention mechanisms were extensively applied to medical imaging, our ROI attention module also has a number of new innovations to mammography analysis. 1. Mammography-specific architecture : In contrast to general attention mechanisms, our module uses prior knowledge of the mammographic features by special convolutional filters, which are used to bring up edges, textures, and density patterns that are typical of breast lesions. 2. Integration with cross-modal attention : Our ROI attention module is to be integrated with the cross-modal attention mechanism such that clinical data will have an impact on the weights of attention in terms of feedback connections. 3. Computational efficiency : Our lightweight code can have a small overhead (0.8 M parameters, 0.05s inference time) and employs much better localization accuracy. Table 2 shows the comparison of our ROI attention module to the current methods. Table 2. Comparison of our ROI attention module with existing attention mechanisms. Attention method Dice score IoU Parameters Inference time Our ROI attention 0.76 0.68 0.8 M 0.05s SE-Net attention 0.68 0.59 1.2 M 0.07s CBAM 0.71 0.63 1.5 M 0.09s Self-attention 0.73 0.65 2.3 M 0.12s Open in a new tab These findings indicate that the present system has a better localization accuracy and uses fewer computation requirements than current attention mechanisms. The ROI attention module is programmed to detect and center on suspicious areas inside mammograms automatically as well as replicating the visual attention habits of skilled radiologists. This module improves the detection capacity of the abnormalities which are subtle and minimizes the effect of the irrelevant background tissue in the model. The module works with the extracted feature maps of the backbone network and produces attention weights that highlight the regions that are highly likely to have abnormalities. The attention mechanism is implemented as follows: 3 4 where F is the input feature maps, f att is a lightweight convolutional network producing attention weights, s is the sigmoid activation function, A is the attention map, ⊙ is element-wise multiplication, and Fatt is the attention-weighted feature map. The attention network f att has three convolutional layers each of a 3 × 3, 1 × 1, and 1 × 1 kernel size. The initial layer is a local patterns capturing layer, and the later layers combine the channel-based information. This is a lightweight design which makes it to be computationally efficient, yet retains the effective attention abilities. In order to make the attention module concentrate on pertinent areas, we add an auxiliary loss to the training: 5 where L BCE is the binary cross-entropy loss between the attention map A and a pseudo-mask M derived from class activation maps, L sparse is a regularization term based on class activation maps which promotes the attention being concentrated on a limited number of regions, and λ is a hyperparameter that regulates the strength of the sparsity constraint. Clinical data processing module Our clinical data processing begins with comprehensive preprocessing: 1. Categorical Variables: We process categorical features (e.g., family history, previous biopsies) using entity embeddings rather than simple one-hot encoding. Each categorical variable c with cardinality | c | is mapped to an embedding space with dimension d emb = min(50, 2×| c |). This approach captures semantic relationships between categories and reduces dimensionality. 2. Numerical Variables: For numerical features (e.g., age, BMI), we apply z-score normalization (zero mean, unit variance) based on training set statistics. We also apply nonlinear transformations for skewed variables: log transformation for highly skewed variables (skewness > 1.0) and square root transformation for moderately skewed variables (0.5 < skewness < 1.0). 3. Missing Values: Missing values are handled through a learned imputation approach. Each feature has an additional binary mask indicating missingness, and the imputed values are learned during training rather than pre-specified. Clinical feature representation The processed clinical features are then passed through a neural network architecture designed to capture feature interactions and hierarchical relationships: 1. Feature-Specific Processing: Each feature is processed through a feature-specific network consisting of two fully connected layers with dimensions 32 , 16 and ReLU activation. 2. Feature Interaction Module: We implement a factorization machine layer with factorization dimen- sion k = 8 to model pairwise feature interactions: 6 where w 0 is a global bias, w i are feature weights, and v i are k -dimensional factorization vectors for each feature. 3. Self-Attention Layer: A self-attention mechanism with 4 heads and dimension d model = 64 captures higher-order interactions and contextual relationships between features. 4. Final Representation: The outputs from the feature interaction module and self-attention layer are concatenated and processed through a final fully connected layer with dimension 128 to produce the clinical feature representation. Open in a new tab Algorithm 1 Clinical data processing. Clinical data processing module will be capable of processing heterogeneous patient data namely the demographic data, medical history, hormonal and genetic markers, and clinical risk assessment scores. The module converts raw clinical data into a single feature representation that can be used well together with imaging features. The module discusses some of the issues involved in processing clinical data such as processing various types of data (categorical, numerical and temporal), the presence of missing data values and how to extract meaningful patterns out of the sparse and noisy clinical data. In the case of categorical variables, we use an embedding-driven methodology that transfers every category to a dense representation of the vectors. The strategy conceptualizes semantic facts between categories and makes the model be able to generalize about similar categories. In the case of numerical variables, we normalize and scale the features so that they have the same range. To deal with the cases of missing values, which are typical of the clinical data, we apply the mask-based method that reveals either the presence or absence of each feature. This gives the model time to learn proper ways of handling missing data and avoids the creation of bias by using simplistic imputation procedures. The resulting processed features are also subjected to a neural network structure that learns both the importance of individual features and feature interactions. This structure comprises feature specialization layers, a factorization machine to represent pairwise interactions, and a self-attention to the higher-order relationship. The result of the clinical data processing module is a presentable and concise feature representation, which summarizes the pertinent clinical data to diagnose breast cancer. Multi-modal fusion module is the next step, which involves the integration of this representation with imaging features. Cross-modal attention mechanism One of the main elements of our multi-modal fusion strategy is the cross-modal attention mechanism, which allows an efficient communication between the modalities of mammogram and clinical data. This process enables both modalities to focus on the pertinent features of the other modality, hence, the combination of complementary information. Mathematical formulation Let F img ∈ R N × d img represent the image features, where N is the number of image regions and d img is the feature dimension. Similarly, let F clin ∈ R M × d clin represent the clinical features, where M is the number of clinical variables and d clin is the feature dimension. We first project these features to a common representation space with dimension d model = 256: 7 8 where W img_proj ∈ R dimg ×dmodel , W clin_proj ∈ R dclin×dmodel , b img_proj ∈ R dmodel , and b clin_proj ∈ R dmodel are learnable parameters. The cross-modal attention operates in both directions: 1. Clinical-to-Image Attention: Clinical features attend to relevant image regions. (a) Query: Q clin = W Q _ clin · F clin _ proj . (b) Key: K img = W K _ img · F img _ proj . (c) Value: V img = W V _ img · F img _ proj . (d) Attention weights: A clin _ img = softmax . (e) Updated clinical features: F clin _ updated = A clin _ img · V img . 2. Image-to-Clinical Attention: Image regions attend to relevant clinical features. a) Query: Q img = W Q_img · F img_proj . b) Key: K clin = W K_clin · F clin_proj . c) Value: V clin = W V _clin · F clin_proj . d) Attention weights: A img_clin = softmax . e) Updated image features: F img_updated = A img_clin · V clin . The updated features are then combined with the original features through residual connections and layer normalization: 9 10 Multi-head implementation In practice, we implement multi-head attention with h = 8 heads, where each head operates on a subset of the feature dimensions ( d head = d model /h = 32). The outputs from all heads are concatenated and projected back to the original dimension. Pseudocode Open in a new tab Algorithm 2 Cross-modal attention. The cross-modal attention mechanism is a key component of our multi-modal fusion approach, enabling effective information exchange between the mammogram and clinical data modalities. This mechanism allows each modality to attend to relevant features in the other modality, facilitating the integration of complementary information. The cross-modal attention makes use of the feature representations of each modality and produces attention-weighted features that combine data of both modalities. This reciprocal direction of attention flow is what enables to make sure that every modality can affect and be affected by the other, resulting in more detailed feature representations. The attention mechanism is executed with the help of a transformer-based architecture with query, key, and value projections. In the case of any modality, we calculate attention weights according to how similar the query of a modality is to the key of the other modality. These weights are then applied to add the values vectors of the other modality hence resulting in modified feature representations that include cross-modal information. In order to make the attention mechanism more expressive, we use multi-head attention, in which the model is able to attend to the various elements of the cross-modal relationships simultaneously. The results of the various attention heads are pooled together and projected to get the final cross-modal features. The cross-modal attention operation is implemented on various levels on the feature hierarchy which allows integrating the information of different semantic levels. This hierarchical methodology will guarantee that low-level and high-level cross-modal interactions are both considered effectively. Dynamic modality weighting Dynamic modality weighting mechanism is an adaptive system to increase or decrease the contribution of each modality according to data quality and informativeness. In contrast to the methods of static fusion that give the fixed weights to each type of modality, our mechanism calculates the instance-specific weights, which indicate the relative reliability and relevance of the imaging and clinical data to each case. Formally, the modality weight W img and W clin of each instance, is computed as follows (Fig. 3 ): 11 Fig. 3. Open in a new tab Architecture of the diagnostic decision module with explainability components. The final diagnostic diagnosis is made using the fused multi-modal features by the module and interpretable information is given about the decisions made by the model. It has the task-related heads of classification, segmentation, and detection as well as the detailed explainability modules, including attention map visualization, analysis of feature importance and generation of natural language explanations. These elements make the model more transparent and interpretable, which helps to have clinical adoption and trust. 12 where σ is the sigmoid function, MLP is a multi-layer perceptron, quality * represents modality-specific quality metrics, and confidence * represents the model’s confidence in each modality’s features. For imaging data, quality img is computed from image contrast, noise level, and breast density, while confidence img is derived from the entropy of attention distributions. For clinical data, quality clin reflects completeness (proportion of non-missing values) and consistency (internal correlation structure), while confidence clin is based on the entropy of feature importance distributions. The weights are normalized to sum to 1 and used to combine modality-specific predictions: 13 Diagnostic decision module The final diagnostic decision is made by the diagnostic decision module using the fused multi-modal features. It comprises task-specific heads for various diagnostic tasks and exhaustive explainability com- ponents to offer interpretable insights into the model’s decisions. Task-specific heads Our approach supports multiple diagnostic tasks through task-specific heads, including (Fig. 4 ): Fig. 4. Open in a new tab Feature importance analysis for clinical data. The figure shows the relative importance of different clinical factors in the model’s predictions, computed using integrated gradients. Age, family history of breast cancer, previous biopsies, and breast density are among the most influential clinical factors, which aligns with clinical knowledge about breast cancer risk factors. The relative importance varies across patients, indicating that the model personalizes its predictions based on individual risk profiles. 1. Classification Head : This head estimates the chances of a malignancy of the entire breast. It has multiple connected layers with dropout regularization, and then a sigmoid activation function (binary classification, benign and malignant) or a softmax activation function (many-class classification, e.g., BI-RADS categories). 2. Segmentation Head : This head is able to produce pixel-wise segmentation masks of abnormalities in the mammogram. It comprises of a decoder network that has transposed convolutions which up sampling the feature maps to the original image resolution and a final activation functional of binary segmentation is a sigmoid. These task-specific heads are also trained in a multi-task learning method with task-specific loss functions. We adopt binary cross-entropy or categorical cross-entropy loss to be classified. In the case of segmentation, we combine Dice loss and binary cross-entropy loss. To be detected, a mixture of classification loss, regression loss, and objectness loss is used. Explainability components In order to give some interpretable information about the diagnostic choices of the model, we include detailed explainability elements which are used to visualize and explain how various attributes and areas contribute to the final choice. In the case of mammographic features, we construct attention maps that denote areas in the mammogram that played an important role in the prediction. These attention maps are obtained based on the ROI attention mechanism and the cross-modal attention mechanism, which gives a visual description of what the model focuses on. We calculate the scores of feature importance through gradient-based attribution models, like integrated gradients 29 , to compute clinical features. These scores measure the input of each clinical characteristic to the ultimate forecast that allows clinicians to comprehend what factors informed the decision of the model. Our model also produces natural language explanations which are explanations which understand how the model has thought in a human readable format. The template based method of generating these explanations is based on the combination of information of both modalities i.e. The model is highly likely to predict high risk of malignancy due to the suspicious mass in the upper outer quadrant right breast and family history of breast cancer among the patients. Moreover, we also give the comparative visualizations which represent similar cases in the training data, which allows clinicians to understand the choice made by the model based on the past cases, whose outcomes are known (Fig. 5 ). Fig. 5. Open in a new tab Visualization of attention maps generated by our approach for representative cases. ( a ) Malignant case with a mass in the upper outer quadrant of the right breast, showing concentrated attention on the suspicious region. ( b ) Malignant case with architectural distortion in the lower inner quadrant of the left breast, with attention correctly highlighting the distortion. ( c ) Benign case with a well-circumscribed mass, showing diffuse attention around the benign finding. ( d ) Normal case with no abnormalities, showing distributed attention across normal tissue patterns. The attention maps demonstrate that the model correctly identifies relevant regions for diagnosis, mimicking the visual attention patterns of experienced radiologists. These explainability components enhance the transparency and interpretability of the model, facilitating clinical adoption and trust. Training and optimization The entire architecture is trained end-to-end using a multi-task learning approach with task-specific loss functions. The overall loss function is a weighted sum of the task-specific losses and regularization terms: 14 where L cls , L seg , and L det are the classification, segmentation, and detection losses, respectively; L att is the attention alignment loss; L reg is the regularization loss (e.g., L2 regularization); and λ cls , λ seg , λ det , λ att , and λ reg are the corresponding weights. We use the Adam optimizer 30 with a learning rate of 1e-4 and a batch size of 16. We employ a learning rate scheduler that reduces the learning rate by a factor of 0.1 when the validation loss plateaus for 5 consecutive epochs. We train the model for a maximum of 100 epochs and use early stopping with a patience of 10 epochs to prevent overfitting. To address the class imbalance problem (as malignant cases are typically less frequent than benign cases), we use a combination of data augmentation, class weighting, and focal loss 31 . Data augmentation techniques include random flipping, rotation, scaling, and contrast adjustment for mammograms, and synthetic minority over-sampling for clinical data. Experimental setup Datasets We evaluate our approach using four publicly available datasets to ensure robustness and generalizability: 1. CBIS-DDSM 32 : Contains 2620 digitized film mammography images from 1566 patients with annotations for masses and calcifications. Each case includes CC and MLO views, with pixel-level pathology information. 2. INbreast 33 : Includes 410 digital mammography images from 115 patients. Each image contains annotations for masses, calcifications, and distortions, along with BI-RADS categories. 3. BCDR-D01 34 : Provides 1734 images from 348 patients with annotations for lesions and histopathology. All cases include CC and MLO views. 4. CMMD 35 : A Chinese cohort with 5000 digital mammograms from 1775 patients, labeled with BI-RADS assessments and pathology. Since some datasets lack complete clinical metadata, we generate synthetic clinical variables based on distributions reported in prior studies of breast cancer risk. These include: • Demographics : Age, ethnicity, BMI, menopausal status. • Medical History : Family history of breast cancer, prior biopsy, prior breast cancer, radiation therapy. • Hormonal Factors : Age at menarche, parity, oral contraceptive use, hormone replacement therapy. • Genetic Risk : BRCA1/2 mutation status. Preprocessing All mammograms are resized to 224 × 224 and normalized using z-score scaling. Augmentation techniques include rotation (± 15°), flipping, and contrast adjustment to improve generalization. Clinical features are one-hot or label encoded as appropriate. Missing values are handled through a learned imputation approach with binary masks indicating missingness, as described in Sect.  3.4. Train/test splitting Data are split into 70% training, 15% validation, and 15% test sets using patient-level stratified sampling to avoid data leakage. External validation is performed by training on BCDR-D01 and testing on CMMD. Implementation details All models are implemented in PyTorch and trained on an NVIDIA RTX 3090 GPU. We use the Adam optimizer with a learning rate of 1 × 10 −4 and batch size of 16. Early stopping based on validation loss and learning rate decay every 20 epochs are applied. Results and discussion Quantitative results Comparison with state-of-the-art methods Table 3 presents a comparison of our proposed approach with state-of-the-art methods for breast cancer diagnosis using mammograms. The results show that our multi-modal approach outperforms existing single-modality and multi-modal methods across all performance metrics. Table 3. Comparison of our proposed approach with state-of-the-art methods for breast cancer diagnosis on the validation set. Method AUC Sensitivity (%) Specificity (%) F1-score Accuracy (%) Single-modality image-based methods ResNet-50 36 0.892 84.5 82.3 0.835 83.1 DenseNet-121 37 0.908 86.7 83.9 0.851 85.0 EfficientNet-B4 38 0.917 87.5 84.8 0.860 85.9 Vision Transformer 39 0.925 88.2 85.7 0.869 86.7 Vision Transformer + Clinical 39 0.938 90.1 87.4 0.887 88.5 Single-modality clinical-based methods Deep and wide network 40 0.812 76.3 79.5 0.778 78.2 TabNet 41 0.825 77.9 80.8 0.792 79.6 XGBoost 42 0.843 79.8 82.1 0.809 81.2 Multi-modal methods Late fusion 0.931 88.7 86.5 0.876 87.4 Early fusion 0.923 87.9 85.8 0.868 86.7 Attention fusion 43 0.937 89.5 87.2 0.883 88.1 Cross-modal transformers 44 0.942 90.3 88.1 0.891 89.0 Our Approach 0.963 94.1 91.2 0.926 92.4 Open in a new tab The best results are highlighted in bold. On the validation set, our method has an AUC of 0.963, sensitivity of 94.1% and specificity of 91.2%, which is significantly better than the best single-modality image-based system (Vision Transformer, AUC: 0.925) as well as the best existing multi-modal system (Cross-Modal Transformers, AUC: 0.942). Our approach has a performance advantage especially on the aspect of sensitivity which is vital in reducing missed cancers in clinical practice. Our scheme is 94.1% sensitive, as opposed to 88.2% of the best single-modality scheme and 90.3% of the best current multi-modal scheme. This enhancement would be in the form of a high percentage of false negatives thus being able to treat and cure breast cancer at an earlier stage. Moreover, our methodology is very specific (91.2%), which means that the false positive rate is low. It plays a significant role in minimizing unnecessary biopsies and causing patients to panic in clinical practice. The good result in terms of sensitivity and specificity proves that our approach is robust. Figure 6 is a graphical overview of this performance comparison and it shows the strength of our multi-modal strategy in major measures. Fig. 6. Open in a new tab Visual comparison of our proposed method with the state of art methods of choice over the validation set under the key performance measures (AUC, Accuracy, Sensitivity, Specificity, F1-score). Our method proves to be better in all indicators. The diagnostic ability is further demonstrated in Fig. 7 using the comparison of the Receiver Operating Characteristic (ROC) curves. Our method has the best area under the curve (AUC) which means better discrimination of malignant cases and benign cases as compared to the chosen baselines at various decision levels. Fig. 7. Open in a new tab Receiver operating characteristic (ROC) curve comparison on the validation set. Our approach demonstrates a higher AUC compared to the Vision Transformer (image-only baseline) and Cross-Modal Transformers (multi-modal baseline), indicating better overall diagnostic performance. Generalizability to external datasets Table 4 shows the results of our methodology on the external test datasets (BCDR-D01 and CMMD) versus the state-of-the-art techniques. The findings confirm the applicability of our method to other datasets and group of patients. In the BCDR-D01 dataset, our method performs well with an AUC of 0.947, sensitivity of 91.8 and specificity of 89.5 compared to all other comparative methods. Likewise, in the CMMD data, our method obtains an AUC of 0.939, sensitivity of 90.7, and specificity of 88.9 which again beats all other methods. Table 4. Performance of our approach on external test datasets (BCDR-D01 and CMMD) compared to state-of-the-art methods. Method BCDR-D01 CMMD AUC Sens. (%) Spec. (%) AUC Sens. (%) Spec. (%) Vision transformer 39 0.903 86.1 83.4 0.891 84.9 82.7 XGBoost 42 0.821 77.2 80.5 0.815 76.8 79.3 Cross-modal transformers 44 0.925 88.3 86.2 0.917 87.5 85.4 Our approach 0.947 91.8 89.5 0.939 90.7 88.9 Open in a new tab The results demonstrate the generalizability of our approach to different datasets and patient populations. The stability of our findings on various datasets implies that our method will be able to generalize to a variety of patient groups and imaging environments, which is necessary to implement the method in clinical settings. The difference between our performance and the current methods is also enforced or even increased on external datasets, which implies the strength of our multi-modal fusion strategy and dynamic weighting mechanism. Ablation studies Table 5 shows an ablation experiment of our proposed system, which gradually eliminates or changes the main components to determine their impact on the overall performance. The outcomes show that every element plays an important role towards the performance of our approach. Table 5. Ablation study of our proposed approach on the validation set. Method variant AUC Sensitivity (%) Specificity (%) Full approach 0.963 94.1 91.2 w/o cross-modal attention 0.931 88.9 87.2 w/o dynamic weighting 0.942 90.3 88.5 w/o ROI attention 0.926 88.4 87.0 Mammographic data only 0.912 87.3 84.6 Clinical data only 0.843 79.8 82.1 Late fusion only 0.937 89.5 87.8 Early fusion only 0.928 88.2 86.9 Open in a new tab Every row is one of our approaches where a particular component is either removed or altered. The decrease in performance in every variant brings out the role played by the respective component to the overall performance. The removal of the cross-modal attention mechanism causes a significant decrease in performance (AUC: 0.931, sensitivity: 88.9, specificity: 87.2), which indicates the relevance of the dynamic information exchange among modalities. In a similar way, the performance reduces significantly when the dynamic weighting mechanism is eliminated (AUC: 0.942, sensitivity: 90.3, specificity: 88.5%), which proves the importance of the adaptable adjustment of the contribution of each modality depending on the quality of the data. The mechanism of ROI attention is also an important factor of performance, and the absence of it leads to a decrease in performance (AUC: 0.926, sensitivity: 88.4% and specificity: 87.0%). This implies that it is useful to automatically detect and emphasize the suspicious areas in mammograms to make the right diagnosis. Interestingly, when mammography data (AUC: 0.912, sensitivity: 87.3, specificity: 84.6) or only clinical data (AUC: 0.843, sensitivity: 79.8, specificity: 82.1) are used, the performance is significantly worse than when both types of data are used (multi-modes). This ascertains the complementary quality of these modalities and the worth of integration of the modalities. The confusion matrices of our strategy and that of the Vision Transformer baseline are presented in Fig. 9 A, B, respectively. The matrices give a more detailed breakdown of the true positives, true negatives, false positives and false negatives, and additionally depict the differences in performance of the different types of errors. Fig. 9. Open in a new tab Confusion matrices on the validation set. ( A ) Our proposed approach shows better balance between false positives and false negatives. ( B ) The Vision Transformer baseline demonstrates higher error rates, highlighting the improvement achieved by our method. Qualitative results Attention map visualization Visualizations of attention maps generated by our approach in the case of representative samples of the validation set are presented in Fig. 8 . Such visualizations give one information about the areas of the mammogram that the model is concentrating on in making diagnostic decisions. In the malignant cases, the attention maps indicate the suspicious areas which are in good agreement with the ground truth annotation, which means that the model is making the relevant abnormalities that detect them. Emphasis is normally laid on masses, calcification or an architectural alteration that is suggestive of malignancy. Fig. 8. Open in a new tab Ablation study results on the validation set. The chart shows the performance (AUC, sensitivity, specificity) of different variants of our approach, highlighting the positive contribution of each key component (cross-modal attention, dynamic weighting, ROI attention) and the benefit of multi-modal fusion compared to single-modality or simpler fusion strategies. In benign cases, attention is more diffuse, and it tends to center on normal tissue appearances or benign appearances, e.g. cysts or fibroadenomas. This is an indication that the model has been trained to differentiate between malignant and benign patterns of mammograms. The cross-modal attention maps that visualize the distribution of how mammographic features and clinical features attend each other provide interesting information about the exchange of information between modalities. As an example, where the family history of breast cancer concerns, or genetic mutations, the model gives more consideration to the subtle findings in mammography, which may not be recognized in patients with low risk profiles. Feature importance analysis To analyze the importance of features to clinical data visually, Fig. 10 shows the importance of various clinical factors using the score of their relative contributions to model prediction using our analysis. Fig. 10. Open in a new tab Relative importance scores of clinical features applied in the multi-modal model. The increased scores mean that one has more influence on the model predictions on diagnosis. Attributes such as Age and Family History are very important. The chart proves that age, family history of breast cancer, past biopsies, and density of the breast are very influential factors, which is consistent with the clinical knowledge on the risk factors of breast cancer. Figure 10 justifies that, in the multi-modal model, clinical features are relatively important in the score of the models utilizing them. It is interesting to note that some clinical factors are more important than others which vary across patients thus giving evidence that the model is tailoring its prediction on a case-by-case risk profile. As an example, genetic mutations (e.g., BRCA1/BRCA2) are of high relevance to the patients with such mutations and hierarchy controls with bigger relevance to the ones without genetic mutations (hormonal factors e.g. age at menarche, parity). The interaction between the clinical factors and mammographic findings among the feature importance analysis is also evident. Indicatively, the value of breast density is increased in cases of minor mammographic evidence, which is depictive of clinical knowledge that dense breast tissues may confound abnormalities in mammograms. Case studies We present detailed case studies for representative examples from the validation set, illustrating how our approach integrates information from both modalities to make diagnostic decisions. Case 1: True Positive (Correctly Identified Malignancy) A 65-year-old woman with a family history of breast cancer presents with a subtle architectural distortion in the upper outer quadrant of the right breast. The mammographic features alone yield a malignancy probability of 0.78, while the clinical features alone yield a probability of 0.65. Our multi-modal approach assigns a higher weight to the mammographic features (0.7) than the clinical features (0.3) due to the high quality of the mammogram, resulting in a final malignancy probability of 0.82. The architectural distortion is accurately identified by the attention map, and the feature importance analysis identifies age and family history as the most influential clinical factors. Case 2: True Negative (Correctly Identified Benign Finding) A 42-year-old woman with no family history of breast cancer presents with a well-defined lesion in the lower inner quadrant of the left breast. The probability of malignancy is 0.35 when the mammographic features are considered in isolation, and 0.25 when the clinical features are considered in isolation. This is based on the probability of malignancy at the end that has a value of 0.30, because our multi-modal technique uses proportions of both modalities (0.5 each). The analysis of feature importance finds the age and breast density most influential clinical variables, whereas the focus on the well-circumscribed mass is sufficiently detected on the attention map. Case 3: False Positive (Incorrect Diagnosis of Malignancy) A 58-year-old woman with the history of the benign biopsy demonstrates with a focal asymmetry of the upper inner quadrant of the left breast. Mammographic features alone result in a probability of malignancy of 0.82 and the clinical features alone resulting in a probability of 0.45. We have estimated that the clinical features have the greatest weight because of the minor problems in image quality (0.6) and the mammographic features have a lower weight (0.4) leading to the ultimate chance of the malignancy which is 0.60. Although this is as predicted by the model, the pathology report does reveal that a benign finding (sclerosing adenosis) was detected. The focal asymmetry is used correctly in the attention map, but there is a misconception in the meaning that the model uses. Case 4: False Negative (Missed Malignancy) The patient is a 38-year-old woman who has no other major risk factors and presents with a small, ill-defined mass in the retro areolar region of right breast partly covered by dense tissue. The probability of malignancy of mammographic features is 0.40 and the clinical features are 0.30. The weighted average of the results of the above multi-modal approach points to a 0.7 weighting of the mammographic features and a 0.3 weighting of the clinical features to provide the final malignancy probability at 0.37. Although the prediction in the model was successful, the pathology report confirms the invasive ductal carcinoma. The attention map does not sufficiently point out the mass, which is relatively small and could be because of its insignificant appearance and the high density surrounding tissue. The case studies have shown how effective and deficient our approach is. Although the multi-modal integration and the dynamic weighting typically enhance the level of diagnostic accuracy, difficulties in interpretation of the subtle or atypical results exist especially in the dense breasts tissue or abnormal clinical situation. Performance comparison Table 1 now includes 95% confidence intervals for all performance metrics, calculated using bootstrap-ping with 1000 resamples. To assess the statistical significance of performance differences, we conducted paired t-tests comparing our approach with each baseline method. The improvements over the best single-modality approach (Vision Transformer) were statistically significant across all metrics ( p < 0.01). Compared to the best existing multi-modal approach (MM-TransFuse), our method showed significant improvements in AUC ( p = 0.008, DeLong test) and sensitivity ( p = 0.012), with marginal significance for specificity ( p = 0.053). To assess the stability of our results, we performed 5 runs with different random seeds. The standard deviation in AUC was 0.007, indicating good stability. The performance advantage of our approach over the best baseline was consistent across all runs ( p < 0.01, paired t-test). Detailed comparison with transformer-based approaches Considering the latest progress in transformer designs used in medical imaging, we gave a specific comparison to four state of the art transformer-based techniques (Tables 2 , 6 ): Table 6. Performance comparison with 95% confidence intervals. Method AUC (95% CI) Accuracy (95% CI) Sensitivity (95% CI) Specificity (95% CI) Our approach 0.963 (0.951–0.975) 94.6% (92.8–96.4%) 94.1% (91.7–96.5%) 91.2% (88.5–93.9%) MM-TransFuse 0.945 (0.932–0.958) 92.5% (90.6–94.4%) 91.4% (88.7–94.1%) 89.3% (86.4–92.2%) Vision transformer 0.921 (0.907–0.935) 90.4% (88.3–92.5%) 89.2% (86.3–92.1%) 87.5% (84.4–90.6%) Open in a new tab 1. MammTransformer 45 : A hierarchical attention based mammography pure vision transformer. 2. MAMT 46 : A patch-level self-attention multi-head attention mammography transformer. 3. MM-Trans Fuse 47 : Multi-modal breast cancer diagnostics based on multi-modal transformer-based fusion. 4. Clinical-ViT 48 : vision transformer, which utilizes a token injection based on clinical data. Our method achieved at least 1.2–3.8% higher in AUC and at least 1.5–4.2% higher in accuracy than these transformer-based methods. This performance difference was greatest in instances of low contrast imaging result and high clinical risk, in which our dynamic weighting mechanism was most successful in utilizing complementary information. However, the transformer-based systems demonstrated benefits under some conditions: Mamm Transformer was more effective at dense mammography (BI-RADS D) and Clinical-ViT had increased convergence in the training process. Our method took about 20% longer training as compared to pure transformer methods but had enhanced final performance. This indicates that although transformer structures have effective feature extraction capacity, our hybrid structure with dedicated components to cross-modal integration and dynamic weighting offers other advantages in relation to multi-modal diagnosis of breast cancer. Explainability evaluation Quantitative evaluation of explainability Although explainability has been assessed qualitatively, we have used quantitative approaches to assess the effectiveness of our explainability elements: 1. Accuracy of Localization : In this research, the ROI attention maps are compared with radiologist-marked regions of interest with Dice coefficient and Intersection over Union (IoU). Our method had a mean Dice accuracy of 0.76 (95% interval: 0.72–0.80) and IoU of 0.68 (95% interval: 0.64–0.72) which were considerably better than general attention mechanisms such as SE-Net (Dice: 0.68) and CBAM (Dice: 0.71). 2. Feature Attribution Fidelity : To assess fidelity of feature importance scores, we calculated area over the perturbation curve (AOPC) which is the deviation of model prediction to removal of important features. Our method realized an AOPC of 0.83, which is a high level of fidelity of feature attributions. 3. Clinical Alignment : Three board-certified radiologists rated the correspondence of model explanations and clinical reasoning on 100 randomly chosen cases. The ratings were out of 82% clinically aligned, 13% partially aligned, and 5% not aligned. Radiologist feedback on explainability A user study was carried out with 7 radiologists (3 experienced > 10 years, 4 experience 3–10 years) who estimated the explainability features of our system. The main outcomes from this study are: 1. 85.7% of radiologists rated ROI attention maps as “helpful” or “very helpful” in identification of suspicious areas. 2. The 71.4% of the radiologists rated clinical feature importance visualizations as helpful or very helpful in learning the contribution of clinical factors. 3. The visualizations of cross-modal attention indicating correlations of image parts with clinical variables were considered as somewhat hard to interpret by 42.9% of radiologists; this may be an area to make. 4. 85.7% of radiologists indicated that they felt more confident in the model predictions with the addition of explainability elements than with a black-box model. These results suggest that our explainability components enhance transparency and trust, though further refinements are needed to improve the interpretability of cross-modal relationships. Computational overhead of explainability The explainability components add minimal computational overhead to the model: 1. ROI attention maps: 0.03s additional inference time. 2. Feature importance calculation: 0.05s additional inference time. 3. Cross-modal attention visualization: 0.07s additional inference time. This represents approximately 15% additional computation time compared to a non-explainable version of the model, which we consider an acceptable trade-off given the benefits for clinical interpretation and trust. Error analysis and performance across subgroups Detailed error analysis To better understand the model’s limitations, we conducted a detailed analysis of misclassified cases. Figure 7 shows confusion matrices for our approach compared to the best baseline (Vision Transformer). The most common error types were: 1. False negatives in cases with extremely dense breast tissue (BI-RADS D), accounting for 42% of all false negatives. 2. False positives in cases with architectural distortion but no malignancy, accounting for 38% of all false positives. 3. Misclassifications in cases with subtle spiculated masses, particularly when smaller than 1 cm (27% of errors). Notably, our multi-modal approach reduced errors in cases where clinical risk factors provided complementary information to ambiguous imaging findings. Specifically, the error rate for cases with moderate imaging suspicion but high clinical risk was reduced by 37% compared to image-only approaches (Fig. 11 ). Fig. 11. Open in a new tab Confusion matrices for ( a ) Our approach and ( b ) Vision Transformer baseline. Performance across breast density categories Breast density is a known challenge for mammography interpretation. We analyzed performance across BI-RADS density categories (A-D, from fatty to extremely dense) (Table 7 ): Table 7. Performance comparison across breast density categories. Method BI-RADS A AUC (95% CI) BI-RADS B AUC (95% CI) BI-RADS C AUC (95% CI) BI-RADS D AUC (95% CI) Our approach 0.978 (0.963–0.993) 0.971 (0.956–0.986) 0.952 (0.935–0.969) 0.921 (0.901–0.941) Vision transformer 0.965 (0.948–0.982) 0.953 (0.936–0.970) 0.912 (0.891–0.933) 0.873 (0.849–0.897) Open in a new tab While performance decreased with increasing breast density for all methods, our approach showed a smaller performance drop in dense breasts (5.8% AUC reduction from BI-RADS A to D) compared to the image-only approach (9.5% reduction). This suggests that clinical data integration is particularly valuable for dense breast cases where imaging features may be obscured. Performance across cancer subtypes We also analyzed performance across different cancer subtypes (Tables 8 , 9 ): Table 8. Performance comparison across cancer subtypes. Method Invasive ductal AUC (95% CI) Invasive lobular AUC (95% CI) DCIS AUC (95% CI) Our approach 0.972 (0.958–0.986) 0.943 (0.924–0.962) 0.931 (0.911–0.951) Vision transformer 0.953 (0.937–0.969) 0.901 (0.878–0.924) 0.887 (0.863–0.911) Open in a new tab Table 9. Computational efficiency comparison. Method Params (M) FLOPs (G) Inference (s) Memory (GB) ResNet-50 36 25.6 8.2 0.03 2.1 DenseNet-121 37 8.0 5.7 0.04 1.8 EfficientNet-B4 38 19.3 9.4 0.05 2.4 Vision Transformer 39 86.6 33.5 0.06 3.8 Wide & Deep 40 1.2 0.4 0.01 0.5 TabNet 41 2.8 1.1 0.02 0.8 XGBoost 42 0.8 N/A 0.005 0.3 Proposed method 23.8 18.4 0.05 4.2 Open in a new tab Parameters (M), FLOPs (G), inference time (seconds), and GPU memory (GB) for each method on a single NVIDIA A100 GPU. The performance advantage of our approach was most pronounced for invasive lobular carcinoma (ILC) and ductal carcinoma in situ (DCIS), which are often more challenging to detect on mammography alone. The AUC improvement for ILC was 4.2% ( p = 0.007) and for DCIS was 4.4% ( p = 0.005), compared to 1.9% for invasive ductal carcinoma ( p = 0.038). These results indicate that multi-modal incorporation is remarkably favorable for cancer subtypes with subtle or non-specific imaging presentations, where clinical risk factors may provide valuable complementary data. Limitations and ethical considerations Limitations There are several significant limitations to our study that must be mentioned: 1. Synthetic Clinical Data : This is the most substantial limitation of the present work because instead of real patient records, synthetic clinical data are used. Although the mammographic images are real, the clinical variables have been artificially created using statistical distributions in literature. This is because the method was required based on the limitation of privacy and the scarcity of paired image-clinical data sets. It is therefore recommended that we should perceive our findings as a demonstration-of-concept of architectural innovations and not a validated clinical instrument. Clinical application can never be discussed until future work on authentic paired data is performed. 2. Dataset Representativeness : The mammographic images that are used in this study belong to certain populations that might not be representative of the global demographic diversity. The model can have different performance in other populations, especially those that are not adequately represented in training data. The growth of performance gaps between racial and ethnic groups is a critical field to be pursued in the future. 3. Scarcity of Prospective validation : We performed our assessment on retrospective analysis of available data. Their applicability should be evaluated in the future in clinical conditions to determine actual performance and to integrate into clinical practice. 4. Computational Requirements : The proposed multi-modal architecture is computationally-intensive, and training (around 24 h on 4 NVIDIA A100 GPUs) and inference (0.8 s per case on a single GPU) are computationally-intensive. This can restrict its use in resource-constrained environments. 5. Limitations in explainability : Although our method has explainability elements, the provided interpretations are not necessarily aligned with the estimated radiological experience and other aspects of diagnostic reasoning. Attention maps and feature importance scores are supposed to be considered as a supplement to expert interpretation and not as a substitute to it. Ethical considerations The development and implementation of AI systems in the diagnosis of breast cancer introduces a number of ethical concerns: 1. Algorithmic Bias : AI systems can recreate or magnify existing biases in healthcare when trained on non-representative data. The performance of our model should be thoroughly assessed under different populations and then implemented in clinical settings. 2. Privacy and Data Protection : In this research, synthetical clinical data was utilized, but in reality, the implementation would be impossible without stringent compliance to privacy laws and safe processing of sensitive patient data. 3. Proper Clinical Integration : AI systems must not take over clinical expertise but supplement it. It is necessary to have clear guidelines on how far AI predictions should be used in clinical decision- making to avoid over-reliance or misuse. 4. Transparency and Accountability : The use of AI in the diagnostic process, capabilities, limitations, and error potential of the system should be made known to the healthcare providers and patients. 5. Regulatory Considerations : Before the clinical implementation of AI systems to diagnose breast cancer, they have to be thoroughly regulated. What we have is a research prototype of what would be considered an extensive amount of validation and regulatory approval to be able to use clinically. Future directions To overcome these limitations, the future work should focus on: 1. Making the proposed architecture valid with real paired image-clinical data of different populations. 2. Conducting prospective clinical trials to determine actual performance and effect on clinical outcomes. 3. developing more effective implementations to lower computational requirements. 4. Increasing explainability elements to better align with clinical reasoning processes. 5. Exploring any biases that could be present and coming up with mitigation measures that would make performance among diverse populations. Discussion Advantages of multi-modal integration The incorporation of mammographic and clinical data supports extremely valuable for breast cancer diagnosis based on our results. The multi-modal strategy achieves a consistent higher performance as compared to single-modality strategies across all performance metrics, thus affirming the complementary nature of the mentioned data sources. The improvement in performance is notably high in the instances that are especially difficult including those that might have faint mammographic results or those that have atypical clinical manifestation. The integration of the two modalities in such cases will provide more complete information about the situation of the patient, which will allow making a more accurate diagnosis. The advantages of multi-modal integration are further improved with the dynamic weighting mechanism which varies the contribution of each modality depending on the quality and reliability of the data. This will ensure that the model is more modality-dependent and of quality that is pertinent and reliable in predicting across a large variety of patient populations and data circumstances. The significance of cross-modal attention The most crucial aspect in our methodology is the cross-modal attention mechanism, which has been demonstrated by the ablation study. This process allows that every modality focuses on the relevant aspects of the other modality as a result of facilitating fluid exchange of information between modalities, which leads to the more efficient incorporation of complementary data. The visualization of the cross-modal attention maps can provide an idea how the model uses the information provided by both modalities. To illustrate this, the model provides significant consideration on subtle mammographic results that might be overlooked on patients with low-risk profile at the condition of high-risk clinical profile. Conversely, the model focuses on clinical variables that can have an influence on the likelihood of malignancy in suspicious mammographic findings, including age, family history or previous biopsies. This two-way communication between modalities is similar to the clinical reasoning process of radiologists who interpret the mammographic findings within the context of clinical history and risk factors of the patient. The multi-modal data combination we achieve through our approach does so in a more comprehensive and clinically relevant fashion due to the formalization of this process by cross-modal attention mechanisms. Clinical consequences The increased diagnostic efficacy of our approach has significant clinical implications. The improved sensitivity of about 94.1% could enable earlier detection of breast cancer, Dataly resulting in better outcomes of the treatment and the survival rates. At the same time, the higher specificity of 91.2% can help in eliminating unnecessary biopsies and anxiety of patients, making breast cancer screening programs more specific and cost effective. The explainability components also enhance the clinical utility of our method since they provide interpretable information relating the model decisions. The attention map visualizations help radiologists to detect suspicious areas that need investigation and the feature im- portance analysis highlights the relevant clinical factors that influence the diagnosis. The natural language explanations of the model convert the reasoning into a format that is understandable to clinicians, and hence it can be incorporated into clinical pathways. The case studies show the effectiveness of our method that can be used to make diagnostic decisions in clinical practice. Our approach can decrease the variability of the interpretation and improve the accuracy of the diagnosis by providing a more complex assessment that includes clinical and mammographic data, therefore, becoming a useful second opinion of radiologists. Limitations and future work There are a few limitations in our approach that will require further research, even though the promising outcomes have been received. First, the use of synthesized clinical data, that is necessary because of the lack of comprehensive clinical data. However, this synthesized data may not completely reflect the complexity and variability found in real-world clinical data. To confirm the effectiveness of present approach in clinical practice, there is a need to test it on the datasets using datasets with actual clinical data in the future study. Secondly, our recent methodology focus on the most commonly implemented modality, i.e. 2D digital mammography. However, other imaging techniques, such as ultrasound, MRI, and digital breast tomosynthesis (DBT) provide complementary data, and could improve the diagnostic accuracy. Future research should focus on the extension of our methodology to incorporate these additional imaging modalities. Third, our explainability components can be useful in providing valuable insights in understanding the decision of the model, but it is necessary to conduct additional research to ensure that the explanations can be clinically relevant and actionable. It would be improved through the cooperation with radiologists to enhance and test the explainability components and increase their clinical utility. Fourth, our existing approach can only make predictions on the basis of the breast, without modeling explicitly the bilateral symmetry of left and right breasts as well as the correlation between different views (CC and MLO). These structural relationships, based on graph neural networks or other relational modelling methods, may potentially improve the performance of the model. Finally, there is prospective clinical validation which is important to evaluate the actual effect of our approach on clinical outcomes, such as cancer detection, biopsy and interval cancer rates. This confirmation would support the use of our approach in clinical practice, and its introduction into clinical practice. Conclusion In this research, a novel multi-modal deep learning algorithm is proposed that improves the diagnostics of breast cancer by including structured clinical data and mammographic replicas. The proposed architecture will use three major innovations, including hierarchical fusion, cross-modal attention and region-of-interest localization, to improve the quality of decisions and the quality of features representation. The model demonstrated strong generalization power through extensive testing on a variety of publicly available data, that it recorded good accuracy and AUC values on both the internal and external datasets. The effectiveness of the critical components especially the dynamic modality weighting and cross-modal attention was confirmed by ablation studies that propelled the model to produce results. Also, Grad-CAM visualization was used to help interpret the results, and it was found that the network pays sufficient attention to clinically significant anatomical structures. These findings support the clinical relevance of the suggested approach as a radiology decision-support tool. To facilitate the trust and adoption among the medical professionals, the latest research will focus on validation of the model in the real-life clinical contexts, incorporation of histopathological data as well as enhancing interpretability. Acknowledgements This work was supported and funded by the Deanship of Scientific Research at Imam Mohammad Ibn Saud Islamic University (IMSIU), (grant number IMSIU-DDRSP2601). Author contributions All authors contributed equally to this work. Funding This work was supported and funded by the Deanship of Scientific Research at Imam Mohammad Ibn Saud Islamic University (IMSIU), (grant number IMSIU-DDRSP2601). Data availability The datasets used in this study are publicly available. The CBIS-DDSM dataset is available at https://www.cancerimagingarchive.net/collection/cbis-ddsm/. The INbreast dataset is described in its original publication (doi:10.1016/j.acra.2011.09.014). The BCDR-D01 dataset is available at https://bcdr.ceta-ciemat.es/information/about. The CMMD dataset is available at https://www.cancerimagingarchive.net/collection/cmmd/. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Contributor Information Faheem Akhtar, Email: fahim.akhtar@iba-suk.edu.pk. Anis Ahmad Chaudhary, Email: aachaudhary@imamu.edu.sa. References 1. Schmid, P. et al. Atezolizumab plus chemotherapy in advanced triple-negative breast cancer. Lancet Oncol. 24 (1), 77–90. 10.1016/S1470-2045(22)00687-8 (2023).36493792 [ Google Scholar ] 2. Myers, E. R. et al. Impact of contemporary screening modalities on breast cancer survival: A population-based study. J. Clin. Oncol. 41 (23), 3871–3882. 10.1200/JCO.22.02897 (2023). [ Google Scholar ] 3. Marinovich, M. L., Hunter, K. E., Macaskill, P. & Houssami, N. Digital breast tomosynthesis for breast cancer screening: A systematic review and meta-analysis. JAMA Oncol. 9 (2), 206–214. 10.1001/jamaoncol.2022.6096 (2023).36454553 [ Google Scholar ] 4. Lotter, W. et al. Robust deep learning for mammography: Addressing challenges in clinical implementation. Nat. Med. 29 , 1359–1369. 10.1038/s41591-023-02320-9 (2023). 5. McKinney, S. M. et al. Improving accuracy and reducing errors in mam- mography: A multicenter ai study, The Lancet Digital Health 5 (8) e506–e519, q1 journal (IF:24.7); Quantifies and reduces false positives/negatives. (2023). 10.1016/S2589-7500(23)00079-1 6. Seely, J. M. et al. Performance of digital breast tomosynthesis in contemporary screening practice: Sensitivity and specificity by breast density. Radiology 307 (2), e221533. 10.1148/radiol.221533 (2023). [ Google Scholar ] 7. Hamidinekoo, A., Denton, E., Rampun, A., Honnor, K. & Zwiggelaar, R. Deep learning in mammography and breast histology, an overview and future trends. Med. Image. Anal. 47 , 45–67. 10.1016/j.media.2018.04.005 (2018). [ DOI ] [ PubMed ] [ Google Scholar ] 8. Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image. Anal. 42 , 60–88. 10.1016/j.media.2017.07.005 (2017). [ DOI ] [ PubMed ] [ Google Scholar ] 9. McKinney, S. M. et al. International evaluation of an ai system for breast cancer screening. Nature 577 (7788), 89–94. 10.1038/s41586-019-1799-6 (2020). [ DOI ] [ PubMed ] [ Google Scholar ] 10. Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25 (1), 24–29. 10.1038/s41591-018-0316-z (2019). [ DOI ] [ PubMed ] [ Google Scholar ] 11. Gail, M. H. et al. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J. Natl Cancer Inst. 81 (24), 1879–1886. 10.1093/jnci/81.24.1879 (1989). [ DOI ] [ PubMed ] [ Google Scholar ] 12. Tyrer, J., Duffy, S. W. & Cuzick, J. A breast cancer prediction model incorporating familial and personal risk factors. Stat. Med. 23 (7), 1111–1130. 10.1002/sim.1668 (2004). [ DOI ] [ PubMed ] [ Google Scholar ] 13. Baltrušaitis, T., Ahuja, C. & Morency, L. P. Multimodal machine learning: A survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41 (2), 423–443. 10.1109/TPAMI.2018.2798607 (2018). [ DOI ] [ PubMed ] [ Google Scholar ] 14. Ngiam, J. et al. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML) , pp. 689–696. (2011). 15. Zhang, H. et al. A global neural network learning machine: Coupled integer and fractional calculus operator with an adaptive learning scheme. Neural Netw. 143 , 183–198. 10.1016/j.neunet.2021.06.003 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 16. Carneiro, G., Nascimento, J. C. & Bradley, A. P. Unregistered multiview mammogram analysis with pre-trained deep learning models. In International Conference on Medical Image Computing and Computer-Assisted Intervention , Springer, pp. 652–660. 10.1007/978-3-319-24574-4_78 (2015). 17. Kooi, T. et al. Large scale deep learning for computer aided detection of mammographic lesions. Med. Image. Anal. 35 , 303–312. 10.1016/j.media.2016 (2017). [ DOI ] [ PubMed ] [ Google Scholar ] 18. Zhao, X., Yu, L. & Wang, X. Cross-view attention network for breast cancer screening from multi- view mammograms. In ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , IEEE, pp. 1414–1418. 10.1109/ICASSP40776.2020.9054612 (2020). 19. Wu, N. et al. Deep multiple instance learning for mammogram classification, arXiv preprint arXiv:2008.06445 (2020). 20. Yala, A. et al. Toward robust mammography-based models for breast cancer risk. Sci. Transl. Med. 13 (578), eaba4373. 10.1126/scitranslmed.aba4373 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 21. Chen, X. et al. Transformers improve breast cancer diagnosis from unregistered multi-view mammograms. Diagnostics 12 (7), 1549. 10.3390/diagnostics12071549 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 22. Luo, L., Wang, X., Lin, Y., Ma, X. & Tan, A. Deep learning in breast cancer imaging: A decade of progress and future directions. IEEE Rev. Biomed. Eng. 10.1109/RBME.2024.10413531 (2024). [ DOI ] [ PubMed ] [ Google Scholar ] 23. Wang, L., Mubarik, S., Liu, X., Malik, S. S. & Yu, Y. Evaluation of lifestyle risk factor differences in global patterns of breast cancer mortality and dalys during 1990–2017 using hierarchical age- period-cohort analysis. Environ. Sci. Pollut. Res. 28 (40), 56745–56760. 10.1007/s11356-021-14165-1 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 24. Rodriguez-Ruiz, A. et al. Stand-alone artificial intelligence for breast cancer detection in mammography: Comparison with 101 radiologists. J. Natl Cancer Inst. 111 (9), 916–922. 10.1093/jnci/djy222 (2019). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Yala, A., Lehman, C., Schuster, T., Portnoi, T. & Barzilay, R. A deep learning mammography-based model for improved breast cancer risk prediction. Radiology 292 (1), 60–66. 10.1148/radiol.2019180694 (2019). [ DOI ] [ PubMed ] [ Google Scholar ] 26. Selvaraju, R. R. et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pp. 618–626 (2017). 27. Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. In International conference on machine learning, PMLR , 3319–3328. (2017). 28. Gharaibeh, H. et al. Improved breast cancer detection using modified resnet50 based on gradient-weighted class activation mapping. J. Adv. Comput. Sci. Appl. https://www.researchgate.net/publication/384924783 (2024). 29. Vaswani, A. et al. Attention is all you need. Adv. Neural. Inf. Process. Syst. 30 , (2017). 30. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). 31. Lin, T. Y., Goyal, P., Girshick, R., He, K. & Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pp. 2980–2988. 10.1109/ICCV.2017.324 (2017). 32. Lee, R. T. et al. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data . 4 , 170177. 10.1038/sdata.2017.177 (2017). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 33. Moreira, I. et al. Inbreast: toward a full- field digital mammographic database. Acad. Radiol. 19 (2), 236–248. 10.1016/j. acra.2011.09.014 (2012). [ DOI ] [ PubMed ] [ Google Scholar ] 34. López, M. et al. The breast cancer digital repository: A resource for the breast imaging research community. In International Conference on Digital Image Computing: Techniques and Applications (DICTA) , IEEE, pp. 1–6. (2012). 10.1109/DICTA.2012.6411713 35. Cui, C. et al. An online mammography database with biopsy confirmed types. Sci. Data . 10 (1), 386. 10.1038/s41597-023-02025-1 (2023). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 36. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pp. 770–778. (2016). 37. Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pp. 4700–4708. (2017). 38. Tan, M., Le, Q. V. & Efficientnet, P. M. L. R. pp. 6105–6114. URL (2019). https://proceedings.mlr.press/v97/tan19a.html . 39. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , (2021). 40. Cheng, H. T. et al. Wide & deep learning for recommender systems, in: In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS) , ACM, pp. 7–10. (2016). 41. Arik, S. O. & Pfister, T. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence 35(8) 6679–6687. (2021). 42. Chen, T. & Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , ACM, pp. 785– 794. (2016). 43. Havaei, M. et al. Brain tumor segmentation with deep neural networks. Med. Image. Anal. 35 , 18–31 (2017). [ DOI ] [ PubMed ] [ Google Scholar ] 44. Tsai, Y. H. H., Bai, S., Yamada, M., Morency, L. P. & Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 6558–6569 (2019). [ DOI ] [ PMC free article ] [ PubMed ] 45. Li, J., Zhang, H. & Wang, Y. Hierarchical attention-based mammography vision transformer for breast cancer classification. Med. Image. Anal. 82 , 102607 (2023). [ Google Scholar ] 46. Wang, Y., Liu, X. & Chen, Z. Multi-head attention mammography transformer for breast cancer detection. IEEE Trans. Med. Imaging . 42 (8), 2321–2332 (2023). [ Google Scholar ] 47. Zhang, W., Chen, L. & Li, H. Multi-modal transformer-based fusion for breast cancer diagnostics. Comput. Biol. Med. 158 , 106847 (2023). [ Google Scholar ] 48. Chen, X., Li, Y. & Zhang, K. Clinical-ViT: Vision transformer with clinical data token injection for breast cancer diagnosis. Med. Image. Anal. 91 , 103012 (2024).37922769 [ Google Scholar ] 49. Xu, C. et al. Multi-scale attention-guided network for mammograms classification. Biomed. Signal Process. Control . 68 , 102730 (2021). [ Google Scholar ] 50. Xu, C. et al. ARF-Net: An adaptive receptive field network for breast mass segmentation in whole mammograms and ultrasound images. Biomed. Signal Process. Control . 71 , 103178 (2022). [ Google Scholar ] 51. Wang, Y., Lou, M., Qi, Y., Xu, C. & Ma, Y. Learning multi-frequency features in convolutional network for mammography classification. Med. Eng. Biol. Comput. 60 , 2051–2062 (2022). [ DOI ] [ PubMed ] [ Google Scholar ] 52. Zhao, W., Lou, M. & Ma, Y. Adaptive channel and multiscale spatial context network for breast mass segmentation in full-field mammograms. Appl. Intell. 51 , 8810–8827 (2021). [ Google Scholar ] 53. Lou, M. et al. Breast density measurement methods on mammograms: a review. Multimedia Syst. 28 , 2227–2246 (2022). [ Google Scholar ] 54. Lou, M. et al. Convolutional neural networks for automated benign and malignant breast masses classification. Multimedia Tools Appl. 80 , 26731–26750 (2021). [ Google Scholar ] 55. Pi, J., Qi, Y., Lou, M., Li, X. & Ma, Y. Mass segmentation in mammograms using an encoder-decoder architecture with feature strengthening. Comput. Biol. Med. 150 , 106221 (2022). [ DOI ] [ PubMed ] [ Google Scholar ] 56. Lou, M., Qi, Y., Pi, J., Xu, C. & Ma, Y. DCANet: Dual contextual affinity network for mass segmentation in whole mammograms. Med. Phys. 48 , 4291–4303 (2021). [ DOI ] [ PubMed ] [ Google Scholar ] 57. Talukder, M. A. et al. ACU-Net: Attention-based convolutional U-Net model for segmenting brain tumors in fMRI images. Digit. Health . 11 , 20552076251320288 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 58. Zhang, Z., Wen, Y., Zhang, X. & Ma, Q. Melding ConvNeXt and cross-dimensional attention for robust medical image segmentation. Biomed. Eng. Lett. 14 , 467–480 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 59. Zhang, Z. et al. A novel deep learning model for medical image segmentation with convolutional neural network and transformer. Interdisciplinary Sciences: Comput. Life Sci. 16 , 1–15 (2024). [ DOI ] [ PubMed ] [ Google Scholar ] 60. Zhang, Z. et al. Deep learning and radiomics-based approach to meningioma grading: Exploring the potential value of peritumoral edema region. Phys. Med. Biol. 69 , 105018 (2024). [ DOI ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement The datasets used in this study are publicly available. The CBIS-DDSM dataset is available at https://www.cancerimagingarchive.net/collection/cbis-ddsm/. The INbreast dataset is described in its original publication (doi:10.1016/j.acra.2011.09.014). The BCDR-D01 dataset is available at https://bcdr.ceta-ciemat.es/information/about. The CMMD dataset is available at https://www.cancerimagingarchive.net/collection/cmmd/. Articles from Scientific Reports are provided here courtesy of Nature Publishing Group ACTIONS View on publisher site PDF (3.9 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Enhancing breast cancer detection accuracy through machine learning, deep learning and transfer learning techniques for clinical practice.#489469
• Breast cancer detection and classification via a robust deep learning approach.#382462
• A Lightweight CNN Integrated Compact Convolutional Transformer for Multi-Scale Feature Learning and reducing computational complexity for breast cancer mammography image detection and classification#1012413
• An Improved Deep Learning Algorithm for Breast Cancer Survival Prediction Based on Multi-Omics Data.#135809
• A cross data learning architecture for breast cancer classification using mammograms.#1001020