A New Method of Modeling the Multi-stage Decision-Making Process of CRT Using Machine Learning with Uncertainty Quantification.

A New Method of Modeling the Multi-stage Decision-Making Process of CRT Using Machine Learning with Uncertainty Quantification - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice J Imaging Inform Med . 2025 Sep 19;39(3):2290–2308. doi: 10.1007/s10278-025-01666-5 A New Method of Modeling the Multi-stage Decision-Making Process of CRT Using Machine Learning with Uncertainty Quantification Kristoffer Larsen 1 , Chen Zhao 2 , Zhuo He 2 , Joyce Keyak 3 , Qiuying Sha 1 , Diana Paez 4 , Xinwei Zhang 5 , Guang-Uei Hung 6 , Jiangang Zou 5, ✉ , Amalia Peix 7, ✉ , Weihua Zhou 8, 9, ✉ Author information Article notes Copyright and License information 1 Department of Mathematical Sciences, Michigan Technological University, Houghton, MI USA 2 Department of Computer Science, Kennesaw State University, Marietta, GA USA 3 Department of Radiological Sciences, Department of Biomedical Engineering, and, Department of Mechanical and Aerospace Engineering , University of California, Irvine, CA USA 4 Nuclear Medicine and Diagnostic Imaging Section, Division of Human Health, Department of Nuclear Sciences and Applications, International Atomic Energy Agency, Vienna, Austria 5 Department of Cardiology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, Jiangsu China 6 Department of Nuclear Medicine, Chang Bing Show Chwan Memorial Hospital, Changhua, Taiwan 7 Nuclear Medicine Department, Institute of Cardiology, La Habana, Cuba 8 Department of Applied Computing, Michigan Technological University, Houghton, MI USA 9 Center for Biocomputing and Digital Health, Institute of Computing and Cybersystems, and Health Research Institute, Michigan Technological University, Houghton, MI USA ✉ Corresponding author. Received 2025 Feb 17; Revised 2025 Jul 29; Accepted 2025 Aug 26; Collection date 2026 Jun. © The Author(s) under exclusive licence to Society for Imaging Informatics in Medicine 2025. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. PMC Copyright notice PMCID: PMC13230379  PMID: 40973911 Previous version available: This article is based on a previously available preprint posted on ArXiv on April 28, 2024: " A new method of modeling the multi-stage decision-making process of CRT using machine learning with uncertainty quantification ". Abstract Current machine learning-based (ML) models usually attempt to utilize all available patient data to predict patient outcomes while ignoring the associated cost and time for data acquisition. The purpose of this study is to create a multi-stage ML model to predict cardiac resynchronization therapy (CRT) response for heart failure (HF) patients. This model exploits uncertainty quantification to recommend additional collection of single-photon emission computed tomography myocardial perfusion imaging (SPECT MPI) variables if baseline clinical variables and features from electrocardiogram (ECG) are not sufficient. Two hundred eighteen patients who underwent rest-gated SPECT MPI were enrolled in this study. CRT response was defined as an increase in left ventricular ejection fraction (LVEF) > 5% at a 6 ± 1 month follow-up. A multi-stage ML model was created by combining two ensemble models: Ensemble 1 was trained with clinical variables and ECG; Ensemble 2 included Ensemble 1 plus SPECT MPI features. Uncertainty quantification from Ensemble 1 allowed for multi-stage decision-making to determine if the acquisition of SPECT data for a patient is necessary. The performance of the multi-stage model was compared with that of Ensemble models 1 and 2. The response rate for CRT was 55.5% ( n = 121) with overall male gender 61.0% ( n = 133), an average age of 62.0 ± 11.8, and LVEF of 27.7 ± 11.0. The multi-stage model performed similarly to Ensemble 2 (which utilized the additional SPECT data) with AUC of 0.75 vs. 0.77, accuracy of 0.71 vs. 0.69, sensitivity of 0.70 vs. 0.72, and specificity 0.72 vs. 0.65, respectively. However, the multi-stage model only required SPECT MPI data for 52.7% of the patients across all folds. By using rule-based logic stemming from uncertainty quantification, the multi-stage model was able to reduce the need for additional SPECT MPI data acquisition without significantly sacrificing performance. Keywords: CRT, Machine learning, Multi-stage, SPECT MPI, Uncertainty quantification Introduction Approximately 6.2 million adults in the USA have heart failure (HF) [ 1 ], with an estimated treatment cost of $43.6 billion in 2020 [ 2 ]. Cardiac resynchronization therapy (CRT) is a standard treatment for HF that coordinates the functions of the left and right ventricles at an average cost of > $60,000 per patient [ 3 ]. In the USA alone, the number of CRT procedures is > 60 , 000 per year [ 3 ]. Despite the prevalence of CRT, a significant proportion (30–40%) of patients who receive CRT do not respond to the treatment because the criteria for selecting patients for treatment have not been fully optimized, among other factors, such as the complexity of HF [ 4 ]. Guidelines recommend CRT for patients with left ventricular ejection fraction (LVEF) ≤ 35%, sinus rhythm, left bundle branch block (LBBB) with QRS duration 150 ms, and New York Heart Association (NYHA) class II, III, or ambulatory IV symptoms on guideline-directed medical therapy (GDMT) [ 5 ]. Selection of patients based on electrical dyssynchrony, such as measured from electrocardiogram (ECG), is not sufficient; mechanical dyssynchrony, such as measured from SPECT MPI, has shown importance for CRT despite not being used in current guidelines [ 6 , 7 ]. However, SPECT MPI has associated costs and radiation exposure. Therefore, CRT patient selection may be improved by utilizing personalized prediction methods, which can incorporate SPECT MPI. Machine learning (ML) encompasses a group of techniques which empower computers to learn tasks without explicit commands by drawing upon data to automatically discover meaningful patterns which can generate predictions at an individual level [ 8 ]. Previous studies have utilized ML to predict CRT outcomes using a variety of different features. Tokodi et al. [ 9 ] applied ML to predict all-cause mortality of CRT patients; Feeny et al. [ 10 ] and Kalscheur et al. [ 11 ] applied ML to predict CRT response using clinical features and QRS morphology [ 12 ]. Recent efforts to predict CRT outcomes using ML span a wide spectrum of modeling strategies. Traditional ML models, such as logistic regression and SVMs, have been employed with full-data reliance on clinical and imaging features, including work by Howell et al. and Manohar et al., who used combinations of clinical, ECG, echocardiographic, and CT-derived parameters for outcome prediction [ 13 , 14 ]. Deep learning-based prognostic models have emerged using complex input representations, such as Chang et al.’s use of synthetic echocardiographic strain traces to train convolutional neural networks with strong predictive performance [ 15 ] and Larsen et al.’s fusion of SPECT MPI images with clinical and ECG features [ 16 ]. Other studies focus on feature selection and dimensionality reduction, such as Nejadeh et al., who proposed methods to identify the most informative feature subset for downstream prediction tasks [ 17 ]. The integration of multimodal imaging, including MRI, CT, and SPECT with clinical and ECG data, has also shown promise, as seen in works by Khamzin et al. and Haque et al. [ 18 , 19 ]. To address clinical workflow realities, sequential decision models have gained attention, where information is acquired in stages, considering cost and diagnostic value; for instance, Khamzin et al. applied model-derived features from simulated electrophysiology to augment initial data [ 18 ]; however, these models are limited in that they rely on data from a single time point and incorporate only a narrow subset of features derived from various medical assessments. For a more accurate and clinically relevant modeling of CRT response within a medical decision-support framework, it is important to account for both the sequential nature in which diagnostic information is obtained and the associated costs of acquiring each test. In this study, we aim to improve prediction of CRT response by accurately modeling the practical multi-staged decision-making process of healthcare professionals using a multi-stage ML model guided by uncertainty quantification. This approach differs from previous studies as the results of all medical tests for all patients are not always necessary for prognostic predictions. Therefore, SPECT MPI acquisition can be omitted for patients whose data from the guidelines has low uncertainty for a decision. Methods Patient Population This study draws from two multicenter clinical trials and a single center patient population for post hoc analysis. “Value of intraventricular synchronism assessment by gated SPECT myocardial perfusion imaging in the management of HF patients submitted to cardiac resynchronization therapy” (IAEA VISION-CRT, n = 199) [ 20 ] is a non-randomized, multinational prospective cohort study. “SPECT Guided LV Lead Placement for Incremental Benefits to CRT Efficacy” (GUIDE-CRT, n = 137) [ 21 ] is a randomized, prospective cohort study. Additionally, patients from Taichung Veterans General Hospital were included (Taiwan-CRT, n = 31). After excluding patients lost-to-follow-up ( n = 133), patients who died before follow-up ( n = 1), and patients with any missing features (ECG QRSd, PCI, MI, HTN, ACEI, or SPECT Concordance, n = 15), 218 patients remained for analysis. The endpoint was CRT response and was defined as an absolute increase in LVEF > 5% measured from the pre-implantation baseline to the 6 ± 1 month follow-up. Ethical approval was obtained from all relevant institutional review committees at each participating site, and all study participants provided written informed consent prior to enrollment. Gated SPECT MPI Acquisition and Quantification For VISION-CRT, the details of gated SPECT MPI acquisition can be found in our previous work [ 16 , 20 ]. For GUIDE-CRT, gated SPECT MPI was performed by SPECT systems with low-energy, general-purpose collimators within 7 days before CRT implantation [ 14 ]. The gated SPECT scan was performed 60–90 min after injection with 25–30 mCi of 99mTc-MIBI at rest. Dual-head or triple-head camera systems were used for image acquisition following resting gated SPECT MPI protocol. Gated images were acquired with a photopeak window of the 99mTc set as a 20% energy window centered over 140 keV using the electrocardiographic R-R interval of 8 frames per cardiac cycle with a 50% beat acceptance window. All images were reconstructed using a system equipped with iterative reconstruction algorithms with at least a 64 × 64 matrix and a 1.0 zoom factor. For Taiwan-CRT, gated SPECT MPI was performed by dual-head SPECT/CT camera (Symbia T2, Siemens Healthcare) [ 22 ]. The gated SPECT scan was performed 60 min after injection with 20–30 mCi of 99mTc-MIBI at rest. Gated images were acquired using step-and-shoot acquisition with 25 s per stop and 32 stops over the 180° with a 64 × 64 matrix with 6.4 mm per pixel using the electrocardiographic R-R interval of 8 frames per cardiac cycle. Images were reconstructed using standard ordered subsets expectation maximization. Despite these variations in initial acquisition parameters across the different trials, a crucial step was taken to ensure post-acquisition data consistency: all SPECT image processing for the entire combined dataset was performed centrally and consistently by the same experienced operator. This processing was executed exclusively using Emory Cardiac Toolbox (ECTb4, Atlanta, GA), which was implemented to quantify short-axis and planar projection images for automated measurement of left ventricular (LV) function and LV mechanical dyssynchrony (LVMD). This standardized approach in image reconstruction and quantitative analysis aimed to mitigate potential biases that could arise from using disparate software platforms or varied operator techniques. Clinical Data Patient data included pre-CRT implantation baseline clinical features, ECG, and resting gated SPECT MPI variables. A total of 44 variables were extracted from these tests—baseline clinical features: age, gender, ethnicity, NYHA class, smoking history, presence of coronary artery disease (CAD), coronary artery bypass graft surgery (CABG), diabetes (DM), hypertension (HTN), myocardial infarction (MI), percutaneous coronary intervention (PCI), and current medical treatment including patient prescribed medication; ECG features: QRS duration and presence of LBBB; SPECT variables: myocardial mass, stroke volume, left ventricular scar percentage, LVEF, left ventricular end-systolic volume (ESV), end systolic eccentricity (ESE), end diastolic eccentricity (EDE), shape index (ESSI and EDSI), wall thickening (WT), summed rest score (SRS), both systolic and diastolic phase peak (PP), phase standard deviation (PSD), phase kurtosis (PK), phase skew (PS), and phase bandwidth (PBW). Additionally, the concordance between CRT LV lead position used and the optimal LV lead position identified by ECTb4 as the latest contracting viable site was recorded as a binary variable [ 23 , 24 ]. Uncertainty Quantification Despite the impressive performance of ML across multiple domains, a significant hindrance to widespread implementation is the lack of uncertainty quantification causing a lack of transparency for clinicians [ 25 ]. Current methods in uncertainty quantification allow for interpretable measurements to monitor skeptical predictions. Kwon et al. applied uncertainty quantification via Bayesian neural networks for ischemic stroke lesion segmentation [ 26 ]. Linmans et al. utilized ensemble uncertainty quantification for tumor detection [ 27 ]. In the present study, pseudo-bootstrapped ensembles were employed. Ensembles are a technique which combines multiple base ML models to form a more informed prediction; using aggregation across these base model outputs allows for predictions as if the ensemble were a single standard ML model. Elastic-net logistic regression was selected as the base learner model for the ensembles. During training of the ensembles, two tunable hyperparameters were learned: (1) the fraction of training samples to use to train the models during random sampling and (2) the number of base models to use to form the ensemble. During inference, instead of predicting a single probability for CRT response, the ensemble provided multiple probabilities which depended on the number of base models. These probabilities were averaged to form an aggregated probability value, and the standard deviation was derived to estimate the uncertainty. It follows that a higher standard deviation indicates greater uncertainty of the sample, while a relatively lower standard deviation indicates less uncertainty of the sample [ 28 , 29 ]. The prediction pathway for a sample starts with generating the array of predictions from Ensemble 1 using clinical and ECG data; from here, it is possible that a patient’s CRT response predictions have low uncertainty; if so, then the predictions from Ensemble 1 are taken as final; if not, then additional SPECT MPI data is acquired and joined onto the Stage 1 data generating the Ensemble 2 prediction array, which is aggregated to create the final CRT response prediction. Multi-staged Modeling In clinical practice, CRT follows a multi-staged decision-making process that can be generally separated into two stages: (1) an initial stage containing basic clinical variables and ECG features (stage 1) and (2) a second stage containing basic clinical variables, ECG features, and variables derived from gated SPECT MPI (stage 2). Considering the medical cost and associated time for the procedure, gated SPECT MPI could be considered for patients with high uncertainty at the first stage. We propose to use sequential trustworthy modeling via uncertainty quantification to reduce clinical costs and improve efficiency by integrating basic clinical variables, ECG features, and gated SPECT MPI features into a multi-staged model (Fig. 1 ). Two key parameters are the standard deviation threshold and the midway threshold, which control the tolerable level of uncertainty in a sample before SPECT acquisition becomes necessary. If the level of uncertainty is relatively small, the output from Ensemble 1 is used. Fig. 1. Open in a new tab Multi-staged decision-making process The overall data modeling pipeline is shown in Fig. 2 . A nested cross-validation structure was used where the outer training fold (train (9/10): n = 196, test (1/10): n = 22) was initially processed before two separate validation sets were sliced off (remaining train: n = 156, validation 1: n = 20, validation 2: n = 20). The outer training fold was then split again into an inner fivefold stratified shuffle split cross-validation (train (9/10): n = 176, test (1/10): n = 20), for hyperparameter tuning then training on the whole outer training fold with the optimal hyperparameter configuration. This is repeated for both ensembles’ models. Afterward, one of the validation sets is used for training multiple multi-staged models each with different scaling hyperparameters which is then evaluated on the second validation set. The highest performing multi-stage model is then evaluated on the outer test fold for the final performance metrics. Fig. 2. Open in a new tab Data modeling pipeline The data were pre-processed using centering/scaling and spatial sign, a technique to reduce the effect of outliers by projecting the data onto a multidimensional sphere. Feature selection was performed using Recursive Feature Elimination (RFE), a method of iteratively removing unimportant features to find the optimal set of features. From the original training fold, two validation sets were sliced to tune the multi-staged model’s hyperparameters: standard deviation threshold and midway thresholds. These thresholds form the gated rule-based logic for determining if a patient needs to move from stage 1 into stage 2. The remaining training data were used in the inner cross-validation to tune the hyperparameters of the ensemble models, such as the number of base models in the ensemble, the percentage of samples to randomly select for each base model, and the respective base model’s hyperparameters. These procedures are repeated twice for both stage 1 data (clinical and ECG features) and stage 2 data (clinical, ECG, and SPECT MPI features), thereby obtaining two ensemble models, ensemble 1 from stage 1 data and ensemble 2 from stage 2 data. The ML models used Scikit-learn package version 1.2.2 [ 30 ]. Hyperparameter tuning was performed using Keras-Tuner; the elastic-net logistic regression base model used in the ensembles had its L1 and L2 ratios tuned over a large grid. After the two ensemble models were trained separately, the first validation set was used to tune the multi-stage model’s hyperparameter thresholds; the metric for the hyperparameter optimization was a scaled weighted area under the (receiver operating characteristic) curve (AUC). The original AUC of the model was constrained using a scaled weight function (Fig. 3 ) to optimize a balance between predictions kept from Ensemble 1 and those cascading into Ensemble 2. First, the fraction of samples that did not require the stage 2 data were inputted into the scaling function, which was then used to weight the AUC. Augmenting the AUC in this way constrains the multi-stage model to prevent only using Ensemble 2. Second, the hyperparameter tuning process was repeated for different scaling parameters in a range between [0.5, 9]. Afterwards, the second validation set was used to evaluate the learned thresholds from the different scaling parameters to select the best multi-staged model with the highest normal AUC. Finally, the best multi-stage model was evaluated using the outer test set. Fig. 3. Open in a new tab Scaled weighted AUC function For comparison to the multi-stage model, both Ensemble 1 and 2 performances were compared. Additionally, a new set of performance metrics was compared based on the 2012 ACCF/AHA/HRS guideline recommendations for CRT admission. CRT responders were predicted if the patient belonged to Class I or Class IIa CRT recommendations [ 31 ]. Considering the growing attention and demonstrated capabilities of neural networks in clinical prediction tasks, a multilayer perceptron (MLP) model is also incorporated as a base learner model with corresponding ensemble models and a multi-stage model. Additionally, for robustness purposes, a simulation study is conducted to understand the effects of sample size on the performance of the model. The simulation involved random sampling with replacement of percentages of the training fold within the tenfold outer cross-validation; this fraction of training samples ranges from [0.1, 1.0] in increments of 0.10, resulting in 10 different samples that are then repeated three times. Ultimately, three times repeated cross-validation is used. Afterwards, performance metrics of the model are aggregated to calculate means and standard deviations, which show how the model performs under different perturbations to the training data size. Random seeds were fixed during the start of the experiment; moreover, random seeds were set during data splitting and preprocessing along with each model’s initial configuration for reproducibility. Statistical Analysis Performance metrics to evaluate the predictive ability of the different models included AUC, accuracy, sensitivity, and specificity. Feature importance was generated using permutation tests. For baseline analysis, categorical variables were expressed in counts and percentages with p -values generated via the chi-square test of independence, while continuous variables were expressed as mean ± standard deviation with p -values generated via two-sample t -tests. DeLong AUC test was used to generate confidence intervals and p -values. McNemar’s test was used to compare sensitivity and specificity of the models for p -values and confidence intervals. Additionally, the features used for the Ensemble 1 model are considered for a multivariate analysis using logistic regression to determine statistical significance of features; the analysis is similarly repeated for Ensemble 2. Results In total, 218 patients underwent CRT implantation, which included an initial gated SPECT MPI and clinical assessment (Table 1 ). During a 6±1 month follow-up, all the assessments were repeated. Overall, a total of 121 out of the 218 (55.5%) patients responded to CRT, which was defined as an absolute increase in LVEF >5% from baseline to follow-up. Average performance metrics across the outer validation folds are presented in Table 2 for each of the models. Table 1. Baseline characteristics of the enrolled patients Variable Overall ( n = 218) Response ( n = 121, 55.5%) Non-response ( n = 97, 44.5%) p -value Clinical Age (yrs.) 62.0 ± 11.8 60.5 ± 12.1 63.2 ± 11.5 0.098 Male 133 (61.0%) 63 (64.9%) 70 (57.9%) 0.353 Race African 15 (6.9%) 9 (9.3%) 6 (5.0%) 0.326 Asian 76 (34.9) 21 (21.6%) 55 (45.5%) < 0.0001* Caucasian 21 (9.6%) 11 (11.3%) 10 (8.3%) 0.593 Hispanic 76 (34.9%) 45 (46.4%) 31 (25.6%) 0.002* Indian 30 (13.7%) 11 (11.3%) 19 (15.7%) 0.465 Smoking 41 (18.8%) 18(18.6%) 23 (19.0%) 1.0 DM 53 (24.3%) 27 (27.8%) 26 (21.5%) 0.354 HTN 117 (53.7%) 57 (58.8%) 60 (49.6%) 0.225 MI 34 (15.6%) 23 (23.7%) 11 (9.1%) 0.006* CAD 68 (31.2%) 41 (42.3%) 27 (22.3%) 0.003* CABG 3 (1.4%) 1 (1.0%) 2 (1.7%) 1.0 PCI 12 (5.5%) 7 (7.2%) 5 (4.1%) 0.488 NYHA II 59 (27.1%) 21 (21.6%) 38 (31.4%) 0.145 III 127 (58.3%) 65 (67.0%) 62 (51.2%) 0.027* IV 32 (14.7%) 11 (11.3%) 21 (17.4%) 0.292 ACEI/ARB 179 (82.1%) 73 (75.3%) 106 (87.6%) 0.029* Imaging ECG QRSd 158.6 ± 27.2 157.8 ± 27.3 159.2 ± 27.1 0.718 LBBB 214 (98.17%) 96 (99.0%) 118 (97.5%) 0.776 SPECT SRS 18.2 ± 12.2 21.6 ± 12.5 15.5 ± 11.3 < 0.0001* ESV 192.6 ± 108.0 207.1 ± 118.7 181.0 ± 97.5 0.082 LVEF 27.7 ± 11.0 28.8 ± 11.1 26.9 ± 10.9 0.207 Mass 215.2 ± 58.4 224.3 ± 61.6 207.9 ± 54.9 0.042* Stroke volume 62.9 ± 23.2 71.0 ± 24.1 56.4 ± 20.1 < 0.0001* WT% 22.0 ± 16.9 24.9 ± 16.6 19.8 ± 16.9 0.026* WT Sum 11.2 ± 8.7 12.7 ± 8.5 10.1 ± 8.6 0.027* Concordance 49 (22.5%) 25 (25.8%) 24 (19.8%) 0.379 Scar % 22.7 ± 14.0 27.0 ± 16.0 19.2 ± 11.1 < 0.0001* Diastolic PBW 169.4 ± 80.5 171.6 ± 78.9 167.7 ± 82.0 0.725 PK 8.4 ± 7.3 8.8 ± 7.8 8.1 ± 6.8 0.456 PS 2.5 ± 0.8 2.5 ± 0.9 2.5 ± 0.8 0.819 PP 221.0 ± 0.8 220.6 ± 34.2 221.4 ± 43.7 0.886 PSD 52.3 ± 20.7 52.7 ± 20.0 52.0 ± 21.4 0.803 Systolic PBW 158.3 ± 77.5 159.8 ± 77.3 157.1 ± 77.9 0.798 PK 132.5 ± 8.1 9.2 ± 9.3 7.5 ± 6.9 0.134 PP 132.5 ± 37.2 131.0 ± 32.8 133.6 ± 40.4 0.599 PSD 50.0 ± 20.6 50.5 ± 20.8 49.7 ± 20.5 0.758 EDE 0.6 ± 0.2 0.5 ± 0.2 0.6 ± 0.1 0.002* EDSI 0.8 ± 0.1 0.8 ± 0.1 0.7 ± 0.1 0.003* EDV 255.5 ± 118.0 278.2 ± 129.4 237.4 ± 105.1 0.013* ESE 0.6 ± 0.2 0.6 ± 0.2 0.6 ± 0.1 < 0.0001* ESSI 0.8 ± 0.1 0.8 ± 0.1 0.8 ± 0.1 0.001* Open in a new tab Data are expressed as mean ± standard deviation or count (percentage) * p -value < 0.05 DM diabetes mellitus, HTN hypertension, MI myocardial infarction, CAD coronary artery disease, CABG coronary artery bypass graft surgery, PCI percutaneous coronary intervention, NYHA New York Heart Association, ACEI/ARB angiotensin-converting-enzyme inhibitors/angiotensin receptor blockers, SRS summed rest score, ESV end-systolic volume, WT wall thickening, Scar % percentage non-viable LV, PBW phase bandwidth, PK phase kurtosis, PS phase skew, PP phase peak, PSD phase standard deviation, EDE end-diastolic eccentricity, EDSI end-diastolic shape index, EDV end-diastolic volume, ESE end-systolic eccentricity, ESSI end-systolic shape index Table 2. Performance of ensemble models and multi-stage model Model AUC Accuracy Sensitivity Specificity ENET Multi-stage 0.75 (0.10) [0.67, 0.80] 0.71 (0.11) 0.70 (0.13) [0.61, 0.77] 0.72 (0.20) [0.63, 0.81] ENET Ensemble 1 0.70 (0.08) [0.60, 0.74] 0.64 (0.07) 0.61 (0.09) [0.52, 0.70] 0.67 (0.13) [0.58, 0.76] ENET Ensemble 2 0.77 (0.08) [0.70, 0.83] 0.69 (0.11) 0.72 (0.14) [0.64, 0.80] 0.65 (0.15) [0.55, 0.74] MLP Multi-stage 0.70 (0.10) [0.61, 0.76] 0.64 (0.08) 0.62 (0.19) [0.53, 0.71] 0.67 (0.24) [0.57, 0.76] MLP Ensemble 1 0.67 (0.10) [0.57, 0.72] 0.64 (0.08) 0.68 (0.23) [0.49, 0.69] 0.60 (0.24) [0.49, 0.69] MLP Ensemble 2 0.74 (0.09) [0.66, 0.79] 0.65 (0.09) 0.67 (0.14) [0.58, 0.75] 0.63 (0.13) [0.53, 0.73] Guideline N/A 0.53 0.75 (n/a) [0.68, 0.82] 0.26 (n/a) [0.17, 0.34] Open in a new tab Performance metrics are presented as mean (standard deviation). Bold represents best performance values. 95% confidence intervals ([]) are additionally presented for AUC (DeLong) and sensitivity/specificity (McNemar) For predicting CRT response, ENET Ensemble 2 shows the highest AUC of 0.77, with the ENET multi-stage model trailing behind with 0.75. Both these models outperformed ENET Ensemble 1, AUC of 0.70. The ENET multi-stage model exhibited the highest accuracy of 0.71, outperforming both ENET Ensemble 1, with 0.64 and ENET Ensemble 2 with 0.69. Moreover, the accuracy standard deviation of the multi-stage model was the same as ENET Ensemble 2, 0.11, but was outperformed by ENET Ensemble 1, at 0.07. For sensitivity, ENET Ensemble 2 displayed a performance of 0.72, slightly better than that of the ENET multi-stage model, 0.70; however, the standard deviation was slightly higher, 0.14 vs. 0.13, respectively. Both models outperformed ENET Ensemble 1, which had a sensitivity of 0.61. For specificity, the ENET multi-stage model moderately outperformed both ENET Ensembles 1 and 2, with values of 0.72, 0.67, and 0.65, respectively. However, the standard deviation for specificity of the multi-stage models, 0.20, was greater than that of ENET Ensemble 1 and 2, 0.13 and 0.15, respectively. The corresponding AUC plots for the ENET models are shown in Fig. 4 . Fig. 4. Open in a new tab AUC plots for ENET ensemble models and multi-stage model The performance of the ENET multi-stage model with respect to AUC and sensitivity was only slightly inferior to ENET Ensemble 2 but outperformed in terms of accuracy and specificity. Of note, the ENET multi-stage model only required 52.7% (25th percentile: 5.7%, 50th percentile: 50.0%, 75th percentile: 77.3%) of the patients to acquire the additional SPECT MPI modality. Moreover, when compared to the guidelines model, the ENET multi-stage model shows impressive accuracy, 0.71 vs. 0.53, respectively. The guideline performance is higher in terms of sensitivity, 0.75 vs. 0.70, when compared to the ENET multi-stage model, but this is outweighed by the sizeable boost in specificity by the ENET multi-stage model, 0.72 vs. 0.26, respectively. All models generated trended towards improvement over the guideline model. Statistical testing on AUC, sensitivity, and specificity is presented in Table 3 . Significant differences in AUC between the ENET multi-stage model and ENET Ensemble 1 model were detected with a p -value of 0.043. All ENET models had significant differences in specificity compared to the Guideline model with p -values < 0.001, < 0.001, and < 0.001, respectively. Table 3. Statistical testing p -values for different metrics ENET Multi-stage ENET Ensemble 1 ENET Ensemble 2 MLP Multi-stage MLP Ensemble 1 MLP Ensemble 2 Guideline ENET Multi-stage ENET Ensemble 1 AUC: 0.043 Sens: 0.018 Spec: 0.275 ENET Ensemble 2 AUC: 0.239 Sens: 0.4674 Spec: 0.0.052 AUC: 0.023Sens: 0.028 Spec: 0.731 MLP Multi-stage AUC: 0.083 Sens: 0.020 Spec: 0.317 AUC: 0.721 Sens: 0.827 Spec: 1.000 AUC: 0.010 Sens: 0.011 Spec: 0.715 MLP Ensemble 1 AUC: 0.005 Sens: 0.655 Spec: 0.016 AUC: 0.374 Sens: 0.074 Spec: 0.046 AUC: 0.002 Sens: 0.398 Spec: 0.354 AUC: 0.160 Sens: 0.052 Spec: 0.593 MLP Ensemble 2 AUC: 0.687 Sens: 0.467 Spec: 0.061 AUC: 0.165 Sens: 0.209 Spec: 0.465 AUC: 0.078 Sens: 0.058 Spec: 0.593 AUC: 0.123 Sens: 0.133 Spec: 0.371 AUC: 0.030 Sens: 0.209 Spec: 0.465 Guideline AUC: N/A Sens: 0.317 Spec: < 0.001 AUC: N/A Sens: 0.0.027 Spec: < 0.001 AUC: N/A Sens: 0.555 Spec: < 0.001 AUC: N/A Sens: 0.035 Spec: < 0.001 AUC: N/A Sens: 0.224 Spec: < 0.001 AUC: N/A Sens: 0.166 Spec: < 0.001 Open in a new tab The p -values follow the order: DeLong AUC, McNemar’s sensitivity, McNemar’s specificity The MLP multi-stage model had AUC 0.70, accuracy 0.64, sensitivity 0.62, and specificity 0.67. MLP Ensemble 1 had AUC 0.68, accuracy 0.64, sensitivity 0.68, and specificity 0.60, while MLP Ensemble 2 had AUC 0.74, accuracy 0.65, sensitivity 0.67, and specificity 0.63. This pattern of performance between the MLP Ensemble models and the MLP multi-stage model follows similarly compared to the ENET models. The importance of each variable for Ensembles 1 is presented in Fig. 5 and for Ensemble 2 in Fig. 6 . The feature importance is averaged within the ensemble of each fold, then the absolute value is aggregated across folds to provide a measure of how much the model relies on the specific feature. Additionally, fold averaged p -values are presented. No significant features were detected. Ensemble 1 makes Heavy use of race parameters, age, CAD, and NYHA class 3. Ensemble 2 Heavily uses LVEF, race parameters, ESSI, ESE, and EDV. The most important variable in Ensemble 2 was LVEF. Hyperparameters for the multi-stage model are presented in Table 4 . Fig. 5. Open in a new tab Feature importance of stage 1 ensemble models Fig. 6. Open in a new tab Feature importance of stage 2 ensemble models Table 4. ENET multi-stage model hyperparameters Outer CV fold Std threshold Midway threshold Scaling weight Ensemble 1 Ensemble 2 0 0.01 0.10 0.5 25 (0.95) 37 (0.70) 1 0.15 0.04 8 25 (0.95) 34 (0.95) 2 0.06 0.06 9 25 (0.70) 37 (0.70) 3 0.20 0.02 0.5 34 (0.70) 25 (0.70) 4 0.19 0.02 1 25 (0.95) 25 (0.95) 5 0.20 0.03 0.5 28 (0.84) 40 (0.95) 6 0.15 0.06 0.5 49 (0.71) 40 (0.95) 7 0.20 0.02 0.5 25 (0.70) 34 (0.85) 8 0.01 0.10 0.5 25 (0.86) 49 (0.86) 9 0.01 0.10 0.5 25 (0.95) 37 (0.85) Open in a new tab Hyperparameters for the ENET based ensemble models are presented as: number of models (fraction of samples randomly selected for each individual model) The results of the multivariate analysis on the Ensemble 1 and Ensemble 2 features are shown in Table 5 . Table 5. Multivariate logistic regression significance Variable Stage 1 Stage 2 Clinical Age (yrs.) 0.002 0.0267 Male 0.619 0.358 Race African 0.089 0.022 Asian 0.001 0.027 Caucasian 0.388 0.550 Hispanic 0.047 0.004 Indian 0.209 0.041 Smoking 0.867 0.453 DM 0.156 0.020 HTN 0.927 0.288 MI 0.397 0.441 CAD 0.019 0.026 CABG 0.131 0.032 PCI 0.097 0.005 NYHA II 0.873 0.923 III 0.142 0.038 IV 0.714 0.640 ACEI/ARB 0.013 0.002 Imaging ECG QRSd 0.156 0.045 LBBB 0.623 0.704 SPECT SRS n/a n/a ESV n/a n/a LVEF n/a 0.002 Mass n/a n/a Stroke volume n/a 0.642 WT % n/a 0.313 WT Sum n/a 0.320 Concordance n/a n/a Scar % n/a 0.027 Diastolic PBW n/a 0.104 PK n/a 0.697 PS n/a n/a PP n/a 0.415 PSD n/a 0.178 Systolic PBW n/a 0.414 PK n/a 0.871 PP n/a 0.388 PSD n/a n/a EDE n/a n/a EDSI n/a n/a EDV n/a 0.047 ESE n/a 0.965 ESSI n/a 0.262 Open in a new tab Additionally, Fig. 7 displays the results of a simulation study conducted to understand the effects of sample size on the performance of the model. The colored lines display the average performance, while the ribbon displays the standard deviation. The AUC has a sharp increase after increasing the percent of training samples beyond 20%, but then plateaus with a slightly decreasing amount of variation as the fraction of training data increases. This trend is similar for accuracy and specificity, while sensitivity increases moderately until 40% of the training data is used, and the variation decreases correspondingly. The plateau of performance from the sample size indicates that the model is receiving decreasing benefits as the training sample size increases. Therefore, it is likely that increasing the sample size will not drastically improve performance; however, it should be noted that the test sets within the cross-validation folds are kept unchanged. Hence, developing new data features is paramount to increasing the model capabilities for predicting CRT response rather than sample size, which also remains important. Fig. 7. Open in a new tab Sample size simulations for multi-stage model (mean with standard deviation) Discussion In this study, we developed a multi-staged-based ML ensemble model which utilized uncertainty quantification and internal logic to predict CRT response using different stages of data. This work demonstrates the potential for a more patient-centric and cost-effective approach to CRT evaluation. By prioritizing the use of less invasive data initially and only proceeding to more expensive modalities when necessary, this approach can improve patient care by reducing the burden of testing and potentially optimizing the selection of patients who are most likely to benefit from CRT, ultimately leading to more efficient resource allocation within the healthcare system. The data for this study was constructed from 218 patients from the VISION-CRT, Taiwan-CRT, and GUIDE-CRT clinical trials. The multi-stage model’s performance, which includes the internal logic rules for cascading patients to the next stage, was compared with the performance of two composite ensemble models. Ensemble 1 contained only baseline clinical and ECG; the multi-stage model always utilized Ensemble 1, and only utilized Ensemble 2 with the additional features derived from gated SPECT MPI if Ensemble 1's CRT response prediction had uncertainty greater than the uncertainty threshold or midway threshold. The uncertainty was below the uncertainty threshold or midway threshold. The ENET Ensemble 2 model achieved the highest AUC and sensitivity, while the ENET multi-stage model achieved the highest accuracy and specificity. Considering that the ENET Ensemble 2 model always had the additional gated SPECT MPI modality while the ENET multi-stage model only used this modality on average 52.7% of the time shows modest potential. Although these models still show moderate accuracy, utilizing higher optimized and non-linear models has the potential for improved performance. Overall, the ENET multi-stage model showed improved performance over the MLP version across all metrics; however, this difference in performance is likely stemming from differences in sample efficiency considering the low sample size of this analysis. Nevertheless, neural network-based models are more flexible with their input data and could be trained with rich non-structured data directly, such as SPECT MPI polar maps to enhance performance. The value of uncertainty quantification is highlighted in comparison between the Ensemble 2 and multi-stage model; patients with lower uncertainty during stage 1 can be considered more trustworthy. For the ensemble 1 model, when restricting the sample with high confidence, i.e., samples not requiring additional data acquisition, we observe AUC 0.76, accuracy 0.77, sensitivity 0.58, and specificity 0.82. In comparison with the ensemble 1 model applied to all samples, the performance on the subset of data is improved, leading to decreased selective risk; however, due to the small sample size of this analysis, it is possible that out-of-distribution samples, which would be harder to predict, may no longer be evaluated. In comparison, Tokodi et al. developed the SEMMELWEIS-CRT score, a single-staged ML risk stratification system for predicting 1- to 5-year all-cause mortality in CRT patients, which achieved AUCs ranged from 0.768 to 0.803 across time points [ 9 ]. While useful for long-term mortality predictions, it does not address short-term CRT response decision-making, which is the focus of our study. Feeny et al. employed a single-staged naïve Bayes classifier ML model to predict CRT response [ 10 ]; like the Ensemble 1 model, clinical variables and ECG parameters were considered for modeling, but this does not effectively leverage potentially acquired myocardial perfusion data, such as from SPECT MPI. Though a similar AUC of 0.70 is observed in both the Ensemble 1 model and the naïve Bayes model. DL models have the capability to utilize higher-dimensional data natively, such as directly from imaging modalities. Larsen et al. construct a DL model drawing from clinical, ECG, SPECT MPI-derived parameters, and SPECT MPI polar map images to predict CRT response, which results in an AUC of 0.83 higher than the performance of the Ensemble 1 or multi-staged model [ 16 ]. However, it should be considered that the multi-staged modeling framework is agnostic to the choice of base mode; therefore, it is possible to construct the Ensemble 2 model to include SPECT MPI polar map images to improve performance. Many studies have applied ML to predict CRT response [ 9 ]–[ 19 ]; however, these studies have implicitly assumed that all modalities of data (biological variables, baseline characteristics, ECG, and gated SPECT MPI) will be available for the ML models at inference. In contrast, the multistage model is able to form patient predictions in a more flexible manner which takes into account the cost of acquiring new data; moreover, through this modeling procedure, not only can we generate feature importance like standard ML algorithms, the frequency of data use at each stage can be calculated, providing more nuanced information regarding the overall importance of different tests towards the ultimate goal. Both mechanical and electrical dyssynchrony influence the outcome of CRT response; however, they are not often found concurrently in a given patient [ 32 ]. The creation of CRT arose to remedy mechanical dyssynchrony with the end goal of improving cardiac function [ 33 ]–[ 34 ]. However, when generating the patient selection criteria for CRT, since electrical and mechanical dyssynchrony are correlated in addition to the relative ease of measuring electrical dyssynchrony over mechanical dyssynchrony, it was presumed that only electrical dyssynchrony would be necessary. While QRS duration has proven important for predicting CRT response, in many specific instances, it has shown otherwise [ 35 ]. Moreover, continual research into CRT response has highlighted that electrical and mechanical dyssynchrony are not equivalent nor interchangeable [ 36 , 37 ]. Historically, this false equivalence partially can explain the high rate of CRT non-response [ 37 ]. Nevertheless, our experiments reveal that while electrical dyssynchrony is not perfect, in many cases, it is sufficient to indicate the need for CRT; furthermore, information pertaining to mechanical dyssynchrony is not always imperative when predicting CRT response. Hence, we showed that a multi-stage model that does not always require gated SPECT MPI data can perform about as well as a model which always utilizes SPECT MPI. Interestingly, neither LBBB nor QRS duration made it to the top 10 important variables in the Ensemble 2 or even Ensemble 1 which further emphasizes the need for rigorous clinical investigation into the power of electrical dyssynchrony. While LBBB morphology has been traditionally associated with improved CRT outcomes, recent studies indicate that not all patients with LBBB respond favorably to CRT [ 38 – 40 ]. Sassone et al. noted in their study among 243 patients undergoing CRT found that although LBBB was present in 70%, only 60% of LBBB patients experienced CRT response [ 41 ]. A “U-shaped” distribution of QRS duration was observed, with non-responders clustered between 120 and 130 ms and above 180 ms. This suggests that both excessively short and long QRS durations may be less predictive of CRT response in LBBB patients. Furthermore, the VISION-CRT trial, which exclusively included patients with LBBB, revealed that despite the presence of LBBB, a significant proportion of patients did not exhibit the expected clinical benefits from CRT. This underscores the complexity of CRT response and the limitations of relying solely on QRS morphology as a selection criterion in favor of multivariate considerations for both mechanical and electrical dyssynchrony measures. CRT response is a difficult task to predict due to the complex origins of HF [ 42 ]. However, by modeling the prediction task as a sequential diagnostic process, the need for additional patient data (i.e., additional medical tests) can be incorporated into the ML models via gated rule-based logic. Using uncertainty quantification, further data can be acquired when necessary. Moreover, the associated cost and wasted time of additional tests can be circumvented for HF patients. The generalizability and representativeness of the multistage was strengthened by the data sources: the VISION-CRT and GUIDE-CRT multicenter, multinational trials. Spanning 10 centers across 8 different countries including Brazil, Chile, Colombia, Cuba, India, Mexico, Pakistan, and Spain, the VISION-CRT trial provided a significant spread of HF patients worldwide. Moreover, GUIDE-CRT encompassed 19 centers across China. Building off previous applications of ML to aid in CRT decision-making, we develop a methodology to predict CRT response by sequentially escalating data acquisition depending on uncertainty: starting with basic clinical and ECG data, then into more costly SPECT MPI. Moreover, the multi-stage model achieves comparable predictive performance against the full data model, demonstrating a reduction in data acquisition burden. The methodology is flexible for a variety of multi-stage clinical workflows involving the successive collection of medical tests. Ultimately, the novelty of this method lies in its multi-stage clinical decision support framework, which adaptively incorporates additional medical test data as necessary based on predictive uncertainty, thus enabling refinement of personalized decisions; augmenting conventional ML models to consider sequential aspects of increasing data dimensionality. Limitations and Future Work The small number of patients ( n = 218) enrolled in this study increases the potential for variability in the analysis and may limit the statistical power to detect certain associations. Additionally, the limited representation of ethnic groups in the dataset may affect the generalizability of our findings to more diverse populations. Future studies with larger and more ethnically diverse samples are needed to validate and extend these observations. Our study acknowledges limitations concerning data heterogeneity. As detailed in the Methods, the contributing clinical trials (VISION-CRT, Taiwan-CRT, and GUIDE-CRT) employed different SPECT acquisition protocols, which could influence cardiac function metrics like phase SD and bandwidth. While formal adjustment for these retrospective differences was not feasible, all SPECT image processing was consistently performed by a single operator using Emory Cardiac Toolbox (ECTb4). ECTb4’s phase analysis, a key data component, is well-established for its robust performance, demonstrating strong temporal resolution [ 43 ], high reproducibility and repeatability [ 44 , 45 ], and reliability even with lower count data [ 46 ]. This processing standardization helps mitigate potential influences from varying software or operator approaches. For future work, encoding the different acquisition protocols as a predictor itself may prove useful in modeling, specifically with differences in SPECT MPI derived parameters. The small number of patients (218) enrolled in this study also has the potential to increase the variability of the analysis. This analysis considers ECG and SPECT-MPI due to their widespread usage; however, other alternative imaging features could be considered either in replacement of SPECT-MPI or inserted as intermediate stages, such as from Speckle tracking echocardiography for strain features, PET-MPI for quantitative perfusion features in addition to those provided by SPECT-MPI, or Cardiac Magnetic Resonance Imaging for tissue characterization. Ultimately, the multi-staged modeling framework can be adapted to any sequential set of available imaging modalities. Algorithmic improvements to the methodology include utilizing reinforcement [ 47 ], a paradigm of ML that works with sequential staged data that does not need explicit labels; in doing so, the loosely defined definition of CRT response can be subverted. From a clinical perspective, utilizing high-dimensional data, such as the gated SPECT MPI 4D scans themselves in stage 2, along with the inclusion of further stages, such as genetic testing, could improve CRT response [ 48 ]. New Knowledge Gained In this study, we employed uncertainty quantification to navigate a cascading multi-staged decision-making process through ML. Using a two-staged approach, when basic clinical and ECG data are not sufficient, gated SPECT MPI can be introduced. Conclusions The proposed multi-staged method using ML to predict CRT response combining multi-staged uncertainty quantifying ensembles represents a step towards accurately modeling the complex decision-making process faced by clinicians. The results of this approach highlight the great potential of modeling tasks in a sequential fashion, which provides more informative clinical interpretation for patient care. Acknowledgements The authors would like to thank the International Atomic Energy Agency (IAEA) for providing access to the data of the multicenter trial: “Value of intraventricular synchronism assessment by gated-SPECT myocardial perfusion imaging in the management of heart failure patients submitted to cardiac resynchronization therapy” (IAEA VISION-CRT), Coordinated Research Protocol E1.30.34. The authors would also like to thank the Department of Cardiology, the First Affiliated Hospital Nanjing Medical University for providing access to the data of the multicenter trial: “SPECT Guided LV Lead Placement for Incremental Benefits to CRT Efficacy” (GUIDE-CRT). Author Contribution Kristoffer Larsen • Conception design • Analysis and interpretation of data • Drafting of the manuscript and revising it critically for important intellectual content • Final approval of the submitted manuscript Chen Zhao • Conception design • Revising the manuscript for important intellectual content Zhuo He • Conception design • Revising the manuscript for important intellectual content Joyce Keyak • Revising the manuscript for important intellectual content Qiuying Sha • Analysis and interpretation of data • Revising the manuscript for important intellectual content Diana Paez • Active involvement in collecting data and participation in data analysis • Revising the manuscript for important intellectual content Xinwei Zhang • Active involvement in collecting data • Revising the manuscript for important intellectual content Guang-Uei Hung. • Active involvement in collecting data • Revising the manuscript for important intellectual content Jiangang Zou • Active involvement in collecting data • Revising the manuscript for important intellectual content Amalia Peix • Active involvement in collecting data • Revising the manuscript for important intellectual content Weihua Zhou • Conception design • Analysis and interpretation of data • Revising the manuscript for important intellectual content. • Final approval of the submitted manuscript Funding This research was supported by a Michigan Technological University Research Excellence Fund Research Seed grant (PI: Weihua Zhou), a research seed grant from Michigan Technological University Health Research Institute (PI: Weihua Zhou), two grants from the National Institutes of Health, USA (1R15HL172198, PI: Weihua Zhou; 1R15HL173852, PI: Qiuying Sha), and a Michigan Technological University Health Research Institute Graduate Fellowship (PI: Kristoffer Larsen). This work was also supported by the National Natural Science Foundation of China (82070521), the Clinical Competence Improvement Project of Jiangsu Province Hospital (JSPH-MA-2020–3), and the Project on New Technology of Jiangsu Province (JX233C202103). Data Availability The patient data that supports the findings of this study is available from the corresponding author, Weihua Zhou, upon reasonable request. The computer source code for this study is available at https://github.com/MIILab-MTU/CRT_ MultiStageML_Uncertainty. Declarations Ethics Approval The studies were approved by an institutional review committee. Consent to Participate Informed consent was obtained from all individual participants included in the studies. Consent for Publication Consent to publication was obtained from all individual participants included in the studies. Competing Interests The authors declare no competing interests. Footnotes Kristoffer Larsen is the first author. Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Contributor Information Jiangang Zou, Email: jgzou@njmu.edu.cn. Amalia Peix, Email: atpeix@gmail.com. Weihua Zhou, Email: whzhou@mtu.edu. References 1. S. S. Virani et al., “Heart disease and stroke statistics—2020 update: a report from the American Heart Association”. Circulation, 141:139–596, 2020. [ DOI ] [ PubMed ] 2. M. Urbich et al., “A Systematic Review of Medical Costs Associated with Heart Failure in the USA (2014–2020)”. Pharmacoeconomics, 38:1219–1236, 2020. [ DOI ] [ PMC free article ] [ PubMed ] 3. A. Auricchio et al., “Long-term clinical effect of hemodynamically optimized cardiac resynchronization therapy in patients with heart failure and ventricular conduction delay”. JACC, 39:2026–2033, 2002. [ DOI ] [ PubMed ] 4. A. Auricchio and F. W. Prinzen, “Non-Responders to Cardiac Resynchronization Therapy–The Magnitude of the Problem and the Issues–”. Circ J, 75:521–527, 2011. [ DOI ] [ PubMed ] 5. E. Donal et al., “New multiparametric analysis of cardiac dyssynchrony: machine learning and prediction of response to CRT”. JACC: Cardiovasc. Imaging, 12:1887–1888, 2019. [ DOI ] [ PubMed ] 6. W. Zhou and E. V. Garcia, “Nuclear Image-Guided Approaches for Cardiac Resynchronization Therapy (CRT)”. Curr Cardiol Rep, 18:1–11, 2016. [ DOI ] [ PMC free article ] [ PubMed ] 7. Z. He, E. V. Garcia, and W. Zhou, “Nuclear Image-Guided Methods for Cardiac Resynchronization Therapy”. J Nucl Cardiol, 10.1007/978-3-030-62195-7_25, 2021. 8. C. Janiesch, P. Zschech, and K. Heinrich, “Machine learning and deep learning”. Electronic Markets, 31:685–695, 2021. 9. M. Tokodi et al., “Machine Learning-Based Mortality Prediction of Patients Undergoing Cardiac Resynchronization Therapy: The SEMMELWEIS-CRT Score”. Eur Heart J, 41:1747–1756, 2020. [ DOI ] [ PMC free article ] [ PubMed ] 10. A. K. Feeny et al., “Machine Learning Prediction of Response to Cardiac Resynchronization Therapy: Improvement Versus Current Guidelines.” Circ Arrhythm Electrophysiol, 10.1161/CIRCEP.119.007316, 2019. [ DOI ] [ PMC free article ] [ PubMed ] 11. M. M. Kalscheur et al., “Machine learning algorithm predicts cardiac resynchronization therapy outcomes: lessons from the COMPANION trial”. Circ Arrhythm Electrophysiol , 10.1161/CIRCEP.117.005499, 2018. [ DOI ] [ PMC free article ] [ PubMed ] 12. F. de A. Fernandes et al., “A machine learning method integrating ECG and gated SPECT for cardiac resynchronization therapy decision support”. Eur J Nucl Med Mol Imaging, 50:3022–3033, 2023. [ DOI ] [ PMC free article ] [ PubMed ] 13. Howell et al. , “Using Machine-Learning for Prediction of the Response to Cardiac Resynchronization Therapy: the SMART-AV Study”. JACC Clin Electrophysiol, 7: 1505-1515, 2021. [ DOI ] [ PMC free article ] [ PubMed ] 14. Manohar et al. , “Prediction of Cardiac Resynchronization Therapy Response Using a Lead Placement Score Derived From 4-Dimensional Computed Tomography”. Circ Cardiovasc Imaging, 10.1161/circimaging.122.014165. [ DOI ] [ PMC free article ] [ PubMed ] 15. Chang et al. , “Deep Learning Significantly Boosts CRT Response Prediction Using Synthetic Longitudinal Strain Data: Training on Synthetic Data and Testing on Real Patients”. Biomed. J., 10.1016/j.bj.2024.100803. [ DOI ] [ PMC free article ] [ PubMed ] 16. Larsen et al. , “A New Method Using Deep Learning to Preidct the Response to Cardiac Resynchronization Therapy”. J Imaging Inform Med, 10.1007/s10278-024-01380-8. [ DOI ] [ PMC free article ] [ PubMed ] 17. Nejadeh et al. , “Predicting the response to cardiac resynchronization therapy (CRT) using the deep learning approach”. Biocybern. Biomed Eng., 41: 758-778, 2021. 18. Khamzin et al. , “Machine Learning Prediction of Cardiac Resynchronization Therapy Response From Combination of Clinical and Model-Driven Data”. Front Physio., 10.3389/fphys.2021.753282. [ DOI ] [ PMC free article ] [ PubMed ] 19. Haque et al. , “Interpretable machine learning predicts cardiac resynchronization therapy responses from personalized biochemical and biomechanical features”. BMC Med. Inform. Decis. Mak., 10.1186/s12911-022-02015-0. [ DOI ] [ PMC free article ] [ PubMed ] 20. A. Peix et al., “Value of intraventricular dyssynchrony assessment by gated-SPECT myocardial perfusion imaging in the management of heart failure patients undergoing cardiac resynchronization therapy (VISION-CRT)”. J Nucl Cardiol, 28:55–64, 2021. [ DOI ] [ PMC free article ] [ PubMed ] 21. J. Zou et al., “SPECT-guided LV lead placement for incremental CRT efficacy: validated by a prospective, randomized, controlled study”. JACC: Cardiovasc. Imaging, 12:2580–2583, 2019. [ DOI ] [ PubMed ] 22. G.-U. Hung et al., “Left-ventricular dyssynchrony in viable myocardium by myocardial perfusion SPECT is predictive of mechanical response to CRT”. Ann Nucl Med, 35:947–954, 2021. [ DOI ] [ PMC free article ] [ PubMed ] 23. M. J. Boogers et al., “Optimal left ventricular lead position assessed with phase analysis on gated myocardial perfusion SPECT”. Eur J Nucl Med Mol Imaging, 38:230–238, 2011. [ DOI ] [ PMC free article ] [ PubMed ] 24. W. Zhou et al., “Development and validation of an automatic method to detect the latest contracting viable left ventricular segments to assist guide CRT therapy from gated SPECT myocardial perfusion imaging”. J Nucl Cardiol, 25: 1948–1957, 2018. [ DOI ] [ PMC free article ] [ PubMed ] 25. E. Begoli, T. Bhattacharya, and D. Kusnezov, “The need for uncertainty quantification in machine-assisted medical decision making”. Nature Machine Intelligence, 1:20–23, 2019. 26. Y. Kwon, J.-H. Won, B. J. Kim, and M. C. Paik, “Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation”. Computational Statistics Data Analysis, 10.1016/j.csda.2019.106816, 2020. 27. J. Linmans, S. Elfwing, J. van der Laak, and G. Litjens, “Predictive uncertainty estimation for out-of-distribution detection in digital pathology”. Medical Image Analysis, 10.1016/j.media.2022.102655, 2022. [ DOI ] [ PubMed ] 28. E. Heid, C. J. McGill, F. H. Vermeire, and W. H. Green, “Characterizing uncertainty in machine learning for chemistry”. Journal of Chemical Information and Modeling, 63:4012–4029, 2023. [ DOI ] [ PMC free article ] [ PubMed ] 29. D. Morgan and R. Jacobs, “Opportunities and challenges for machine learning in materials science”. Annual Review of Materials Research, 50:71–103, 2020. 30. Pedregosa et al. , “Scikit-learn: Machine Learning in Python”. JMLR, 12: 2825-2830, 2011. 31. C. M. Tracy, “2012 ACCF/AHA/HRS Focused Update of the 2008 Guidelines for Device-Based Therapy of Cardiac Rhythm Abnormalities: A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines”. JACC, 60: 2604–2605, 2012. [ DOI ] [ PubMed ] 32. M. Fudim and S. Borges-Neto, “A troubled marriage: When electrical and mechanical dyssynchrony don’t go along”. J Nucl Cardiol, 26: 1240–1242, 2019. [ DOI ] [ PubMed ] 33. I. Lozano et al., “Impact of biventricular pacing on mortality in a randomized crossover study of patients with heart failure and ventricular arrhythmias”. Pacing and Clinical Electrophysiology, 23:1711–1712, 2000. [ DOI ] [ PubMed ] 34. C.-M. Yu et al., “Tissue Doppler velocity is superior to displacement and strain mapping in predicting left ventricular reverse remodelling response after cardiac resynchronisation therapy”. Heart, 92: 1452–1456, 2006. [ DOI ] [ PMC free article ] [ PubMed ] 35. T. Abraham et al., “Imaging cardiac resynchronization therapy”. JACC: Cardiovasc. Imaging, 2:486–497, 2009. [ DOI ] [ PubMed ] 36. W. AlJaroudi, J. Chen, W. A. Jaber, S. G. Lloyd, M. D. Cerqueira, and T. Marwick, “Nonechocardiographic imaging in evaluation for cardiac resynchronization therapy”. Circ Cardiovascr Imaging, 4:334–343, 2011. [ DOI ] [ PMC free article ] [ PubMed ] 37. P. Carita et al., “Non-responders to cardiac resynchronization therapy:` Insights from multimodality imaging and electrocardiography. A brief review”. Int J Cardiol Cardiovasc Dis, 225:402–407, 2016. [ DOI ] [ PubMed ] 38. Linde et al., “Long-term benefits of biventricular pacing in congestive heart failure: results from the Multisite Stimulation in cardiomyopathy (MUSTIC) study”. JACC, 40: 111-118, 2002. [ DOI ] [ PubMed ] 39. Moss et al., “Cardiac-Resynchronization Therapy for the Prevention of Heart-Failure Events”. NEJM, 361: 1329-1338, 2009. [ DOI ] [ PubMed ] 40. Gold et al., “Effect of QRS Duration and Morphology on Cardiac Resynchronization Therapy Outcomes in Mild Heart Failure: Results from the Resynchronization Reverses Remodeling in Systolic Left Ventricular Dysfunction (REVERSE) study”. Circ, 126: 822-829, 2012. [ DOI ] [ PubMed ] 41. Sassone et al., “Relation of QRS duration to response to cardiac resynchronization therapy”. Am J Cardiol, 115: 214-219, 2014. [ DOI ] [ PubMed ] 42. S. Y. Naqvi, A. Jawaid, I. Goldenberg, and V. Kutyifa, “Non-response to cardiac resynchronization therapy”. Current heart failure reports, 15:315–321, 2018. [ DOI ] [ PubMed ] 43. Chen J, Faber TL, Cooke CD, Garcia EV. Temporal resolution of multi-harmonic phase analysis of ECGgated myocardial perfusion SPECT studies. J Nucl Cardiol 2008;15:383–91. [PMID: 18513645] [PMCID: PMC2992837]. [ DOI ] [ PMC free article ] [ PubMed ] 44. Trimble MA, Velazquez EJ, Adams GL, Honeycutt EF, Pagnanelli RA, Barnhart HX, Chen J, Iskandrian AE, Garcia EV, Borges-Neto S. Repeatability and reproducibility of phase analysis of gated SPECT myocardial perfusion imaging used to quantify cardiac dyssynchrony. Nucl Med Commun 2008;29:374–81. [PMID: 18317303] [PMCID: PMC3048057]. [ DOI ] [ PMC free article ] [ PubMed ] 45. Lin X, Xu H, Zhao X, Folks RD, Faber TL, Garcia EV, Chen J. Repeatability of left ventricular dyssynchrony and function parameters in serial gated myocardial perfusion SPECT studies. J Nucl Cardiol 2010;17:811–6. [PMID: 20440590] [PMCID: PMC2992839]. [ DOI ] [ PMC free article ] [ PubMed ] 46. Cheung A, Zhou Y, Faber TL, Garcia EV, Zhu L, Chen J. The performance of phase analysis of gated SPECT myocardial perfusion imaging in the presence of perfusion defects: A simulation study. J Nucl Cardiol 2012;19:500–6 [PMID: 22203443] [PMCID: PMC3731539]. [ DOI ] [ PMC free article ] [ PubMed ] 47. Liu et al., “Reinforcement Learning for Clinical Decision Support in Critical Care: Comprehensive Review”. JMIR, 22. 2020. [ DOI ] [ PMC free article ] [ PubMed ] 48. Ferro et al., “Impact of DCM-Causing Genetic Background on Long-Term Response to Cardiac Resynchronization Therapy”. JACC: Clinical Electrophysiology, 10: 1455–1464, 2024. [ DOI ] [ PMC free article ] [ PubMed ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement The patient data that supports the findings of this study is available from the corresponding author, Weihua Zhou, upon reasonable request. The computer source code for this study is available at https://github.com/MIILab-MTU/CRT_ MultiStageML_Uncertainty. Articles from Journal of Imaging Informatics in Medicine are provided here courtesy of Springer ACTIONS View on publisher site PDF (2.3 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Distributionally robust decision-making under ambiguity: case study of water environmental management#1062932
• Fuzzy, Intuitionistic and Neutrosophic Set Theories and Their Applications in Decision Analysis#719912
• Valuation and Selection Processes in Human Decision Making under Uncertainty#354132
• Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review#925703
• Subjective Risk Decomposition: A New View for Uncertainty Quantification#1012273