Do machine learning models add value in assessing operative stress?

Do Machine Learning Models Add Value in Assessing Operative Stress? - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Surgery . Author manuscript; available in PMC: 2026 Apr 16. Published in final edited form as: Surgery. 2026 Mar 17;194:110141. doi: 10.1016/j.surg.2026.110141 Search in PMC Search in PubMed View in NLM Catalog Add to search Do Machine Learning Models Add Value in Assessing Operative Stress? Michael A Jacobs Michael A Jacobs , MS 1 Center for Healthcare Evaluation, Research, and Promotion, VA Pittsburgh Healthcare System, Pittsburgh, PA Find articles by Michael A Jacobs 1 , Paula K Shireman Paula K Shireman , MD, MBA, MS 2 Departments of Medical Physiology and Primary Care & Rural Medicine, College of Medicine, Texas A&M University, Bryan, TX Find articles by Paula K Shireman 2 , Jonathan C Silverstein Jonathan C Silverstein , MD, MS 3 Department of Biomedical Informatics, University of Pittsburgh School of Medicine, Pittsburgh, PA Find articles by Jonathan C Silverstein 3 , Daniel E Hall Daniel E Hall , MD, MDiv, MHSc 1 Center for Healthcare Evaluation, Research, and Promotion, VA Pittsburgh Healthcare System, Pittsburgh, PA 4 Department of Surgery, University of Pittsburgh School of Medicine, Pittsburgh, PA 5 Geriatric Research Education and Clinical Center, VA Pittsburgh Healthcare System, Pittsburgh, PA 6 Wolff Center at University of Pittsburgh Medical Center, Pittsburgh, PA Find articles by Daniel E Hall 1, 4, 5, 6 Author information Article notes Copyright and License information 1 Center for Healthcare Evaluation, Research, and Promotion, VA Pittsburgh Healthcare System, Pittsburgh, PA 2 Departments of Medical Physiology and Primary Care & Rural Medicine, College of Medicine, Texas A&M University, Bryan, TX 3 Department of Biomedical Informatics, University of Pittsburgh School of Medicine, Pittsburgh, PA 4 Department of Surgery, University of Pittsburgh School of Medicine, Pittsburgh, PA 5 Geriatric Research Education and Clinical Center, VA Pittsburgh Healthcare System, Pittsburgh, PA 6 Wolff Center at University of Pittsburgh Medical Center, Pittsburgh, PA Author Contributions: Mr. Jacobs and Dr. Hall had full access to all study data and take responsibility for the accuracy of the data analysis. Concept and design: Jacobs and Hall. Acquisition, analysis, or interpretation of data: All authors. Drafting of the manuscript: Jacobs and Hall. Critical revision of the manuscript for important intellectual content: All authors. Statistical analysis: Jacobs. Supervision: Hall. ✉ Corresponding Author: Daniel E. Hall, UPMC Presbyterian, Suite F1264, 200 Lothrop Street, Pittsburgh, PA 15213, hallde@upmc.edu Issue date 2026 Jun. Keywords: gradient boosting, machine learning, surgical risk prediction, surgical risk calculator PMC Copyright notice PMCID: PMC13082668  NIHMSID: NIHMS2158356  PMID: 41844446 The publisher's version of this article is available at Surgery Introduction The American College of Surgeons risk calculator 1 relies on the “principal” Current Procedural Terminology (CPT) code to account for procedure-related heterogeneity, including physiologic stress. Relying on the principal CPT creates two potential issues: 1) the most stressful procedure may not always be coded as principal, and 2) different procedures performed together may cumulatively increase stress. To evaluate the effects of relying on the principal CPT code, Cohen et al. 2 use the CatBoost 3 machine learning (ML) algorithm to predict surgical outcomes using 1) the principal CPT code, 2) the most stressful two CPT codes, and 3) all CPT codes available. While CatBoost has advantages over other ML methods when working with categorical data 3 , it lacks the interpretability of simpler models. Other methods to account for operative stress, like the expanded Operative Stress Score (OSS) 4 , may provide similar predictive performance while being easier to implement, interpret, and explain. This study examines whether CatBoost demonstrates clinically useful predictive improvement over the OSS. Methods This study was determined exempt by the VA Pittsburgh Healthcare System’s Institutional Review Board and reported according to STROBE guidelines. We used 2020 National Surgical Quality Improvement Program Participant Use File data. Multivariable logistic regression models predicted two outcomes: 30-day mortality and Clavien-Dindo IV complications (CDIV). 4 As in prior work, 4 parsimonious models with four independent variables accounting for patient-, presentation- and procedure-related risk were used to approximate the full models used by ACS NSQIP. Patient-related risks were modeled with the Risk Analysis Index (RAI). 5 Presentation acuity was modeled with the presence of any Preoperative Acute Serious Condition (PASC) 4 and case status (elective, urgent, or emergent). Procedure-related heterogeneity was modeled with either CatBoost-derived logit scores or the OSS. The OSS quantifies surgery-related physiologic stress by ranking each CPT code between 1 (very low) and 5 (very high), and accounts for multiple concurrent procedures by assigning the maximum score from available CPT codes. 4 While this does not account for cumulative effects of multiple procedures, it reduces the importance of which CPT is coded as principal. In cases where the principal CPT lacked an assigned OSS, the case was either (a) excluded or (b) scored with the data’s most frequent OSS value: 3. We excluded cases missing RAI variables. We examined models using the areas under the Receiver Operating Characteristic and Precision-Recall curves. Finally, we visually examined CatBoost-derived logit scores for mortality at each OSS level to look for outliers (particularly high-risk outliers in OSS 1-2) and determine what procedures these cases represent. Analyses were performed using R 4.3.1 (R Project for Statistical Computing). Confidence intervals for Precision-Recall curves were estimated using the binomial technique recommended by Boyd, Eng and Page (2013). 6 Results After excluding 8,268 cases for missing RAI variables, the final cohort included 894,700 cases ( Table 1 ) with 96.4% having a principal CPT code scorable by the OSS. Multiple CPT codes were reported in 34.8% of cases. Cases with multiple CPT codes had higher frailty, OSS, PASC rates, mortality, and CDIV. Table 1. Descriptive Statistics for the cohort Overall one CPT code multiple CPT codes p-value a N (%) 894,700 583,344 (65.2) 311,356 (34.8) Age mean [SD] 56.6 [17.3] 56.2 [18.1] 57.5 [15.7] <.001 Sex <.001 Female 513,338 (57.4) 330,735 (56.7) 182,603 (58.6) Male 381,362 (42.6) 252,609 (43.3) 128,753 (41.4) Race <.001 Asian 33,244 (3.7) 22,906 (3.9) 10,338 (3.3) Black 90,226 (10.1) 56,069 (9.6) 34,157 (11.0) Native American 4,960 (0.6) 3,166 (0.5) 1,794 (0.6) Pacific Islander 3,042 (0.3) 2,063 (0.4) 979 (0.3) White 569,738 (63.7) 365,050 (62.6) 204,688 (65.7) Unknown 193,490 (21.6) 134,090 (23.0) 59,400 (19.1) Ethnicity <.001 Hispanic 82,116 (9.2) 55,157 (9.5) 26,959 (8.7) Non-Hispanic 643,675 (71.9) 409,226 (70.2) 234,449 (73.5) Unknown 168,909 (18.9) 118,961 (20.4) 49,948 (16.0) RAI mean [SD] 18.1 [8.4] 17.8 [8.6] 18.6 [8.0] <.001 PASC 21,631 (2.4) 12,026 (2.1) 9,605 (3.1) <.001 Case status <.001 Elective 713,824 (79.8) 445,915 (76.4) 267,909 (86.0) Urgent 84,274 (9.4) 63,375 (10.9) 20,899 (6.7) Emergent 96,602 (10.8) 74,054 (12.7) 22,548 (7.2) OSS <.001 1 27,339 (3.1) 19,479 (3.3) 7,860 (2.5) 2 372,552 (41.6) 267,664 (45.9) 104,888 (33.7) 3 395,296 (44.2) 249,034 (42.7) 146,262 (47.0) 4 59,146 (6.6) 24,135 (4.1) 35,011 (11.2) 5 8,365 (0.9) 3,280 (0.6) 5,085 (1.6) Not Scored 32,002 (3.6) 19,752 (3.4) 12,250 (3.9) 30-day mortality 9,689 (1.1) 5,690 (1.0) 3,999 (1.3) <.001 CDIV complications 22,186 (2.5) 11,319 (1.9) 10,867 (3.5) <.001 Open in a new tab %, percent; CDIV, Clavien-Dindo 4; CPT, Current Procedural Terminology; N, number; OSS, Operative Stress Score; PASC, Preoperative Acute Serious Conditions; RAI, Risk Analysis Index; SD, standard deviation a Continuous variables were compared using Mann-Whitney U tests and categorical variables were compared using chi-squared tests. Compared to OSS models, CatBoost models had nominally larger c-statistics and areas under the Precision-Recall curve with similar confidence intervals ( Table 2 ). This pattern was observed in the overall sample as well as the portion of the sample with multiple CPTs. Table 2. Model Statistics (all models include RAI, PASC, and case status) Overall Sample N=894,700 Multiple CPT Surgeries N=311,356 30-day mortality ROC AUC [CI] PR AUC [CI] N ROC AUC [CI] PR AUC [CI] N OSS (excluding cases with unscored principal CPT code) 0.921 [0.919, 0.924] 0.183 [0.175, 0.191] 862,698 0.913 [0.908, 0.917] 0.205 [0.192, 0.218] 299,106 OSS (assigning 3 to cases with unscored principal CPT code) 0.921 [0.918, 0.923] 0.182 [0.174, 0.190] 894,700 0.910 [0.906, 0.915] 0.202 [0.190, 0.214] 311,356 CatBoost 0.933 [0.931, 0.935] 0.219 [0.211, 0.227] 894,700 0.930 [0.927, 0.934] 0.280 [0.266, 0.294] 311,356 CDIV complications ROC AUC [CI] PR AUC N ROC AUC [CI] PR AUC N OSS (excluding cases with unscored principal CPT code) 0.859 [0.857, 0.862] 0.281 [0.275, 0.287] 862,698 0.860 [0.857, 0.864] 0.329 [0.320, 0.338] 299,106 OSS (assigning 3 to cases with unscored principal CPT code) 0.858 [0.855, 0.861] 0.277 [0.271, 0.283] 894,700 0.857 [0.853, 0.861] 0.322 [0.313, 0.331] 311,356 CatBoost 0.878 [0.875, 0.880] 0.334 [0.328, 0.340] 894,700 0.883 [0.880, 0.887] 0.397 [0.388, 0.406] 311,356 Open in a new tab CDIV, Clavien-Dindo 4 complications; CI, confidence interval; N, number; OSS, Operative Stress Score; PASC, Preoperative Acute Serious Conditions; PR AUC, Precision-Recall Area Under the Curve; RAI, Risk Analysis Index; ROC AUC; Receiver Operating Characteristic Area Under the Curve. In the samples excluding unscored principal CPT codes, there were 9364 and 3791 mortalities and 21,181 and 10,206 CDIV complications in the overall and multiple CPT samples; these were used to calculate the reported confidence intervals. Examining CatBoost-derived logit scores within each OSS level, we observed multiple outliers, including: 1) a cystourethroscopy with fulguration of a 2-5 cm bladder tumor with a 46.2% predicted 30-day mortality risk, 2) a femoropopliteal embolectomy with decompression fasciotomy with a 93.2% predicted 30-day mortality risk, and 3) an orchiectomy with scrotal exploration with a 45.7% predicted 30-day mortality risk. These outliers represent 3 unique, sparsely sampled combinations of CPT codes. Overall the sample included 3182 cystourethroscopies, 506 embolectomies and 483 orchiectomies in 244, 356, and 167 unique combinations, respectively. Discussion While CatBoost models demonstrated nominally increased predictive performance over the OSS, the improvement is small with uncertain clinical significance. There are no published guidelines for interpreting the clinical significance of changes in these statistics, and interpretation likely depends on the context for misclassification. Furthermore, CatBoost-derived logit scores are both difficult to interpret and effectively concealed in the “black box” model, sacrificing the intuitive and surgeon-defined scale of the OSS. We worry that reflexive application of ML techniques “chase” marginal improvements of uncertain clinical significance at the substantial cost of “black box” methods that defy bedside interpretation or explanation. While ML is useful for nonlinear effects, interaction effects, and datasets with too many variables to feasibly explore, we prefer simpler, explainable models when they make predictions similar to ML techniques. And as demonstrated here, interpretable models often make predictions of similar performance. We observed several cases where CatBoost predicted unrealistically high mortality risk. This is likely due to small sample size of the unique CPT combination, and similar sparsity is likely present among other combinations. While no model makes perfect predictions, explainable models are often easier to regularize, preventing outlier, wildly inaccurate predictions. They are also easier to double-check when implausible predictions occur. Many surgeons may not be aware that the NSQIP risk calculator has transitioned from logistic regression-based models to ML models 1 , and so are prone to aberrant predictions when specific variable combinations have too few examples in the data with critical implications for patient counselling and surgical decision making. Surgeons are accountable for their decisions, even when guided by black box models, so these models should be able to convey why a patient is considered high-risk. Model performance should be balanced with interpretability, and simpler models may have advantages over ML that should not be overlooked. When black box models provide clinically meaningful improvements, tools to explain such models, like SHAP values and local additive explanations, should be routinely reported. Neither these metrics nor the intrinsic limitations of ML techniques are transparently disclosed in the NSQIP risk calculator. They should be. Funding/Support: The authors disclose grant funding from the National Institutes of Health, Veterans Health Administration Office of Research and Development, and Breast Cancer Research Foundation outside the scope of this work. Conflict of Interest/Disclosures: Dr. Hall reported support from the U.S. Department of Veterans Affairs, the Veterans Health Administration, the Office of Research and Development with grants I01HX003322, I01HX003215, I01HX003095, I01CX002150, HX003201-01 and grant 1U01TR002393 from the National Institutes of Health outside the scope of this work. No funding source had any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The opinions expressed here are those of the authors and do not necessarily reflect the position of the United States government. Data Sharing Statement: The NSQIP PUF data used in this study is available to NSQIP-participating hospitals upon request from the American College of Surgeons. References 1. Liu Y, Ko CY, Hall BL, Cohen ME. American College of Surgeons NSQIP Risk Calculator Accuracy Using a Machine Learning Algorithm Compared with Regression. J Am Coll Surg. 2023;236(5):1024–1030. doi: 10.1097/XCS.0000000000000556 [ DOI ] [ PubMed ] [ Google Scholar ] 2. Cohen ME, Liu Y, Hall BL, Ko CY. ACS NSQIP Surgical Risk Calculator Accuracy When Operative Risk is Represented by the Principal CPT ® code Versus Many Codes. Ann Surg. Published online February 7, 2025. doi: 10.1097/SLA.0000000000006661 [ DOI ] [ Google Scholar ] 3. Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A. CatBoost: unbiased boosting with categorical features. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc.; 2018:6639–6649. https://dl.acm.org/doi/abs/10.5555/3327757.3327770 [ Google Scholar ] 4. Yan Q, Kim J, Hall DE, et al. Association of Frailty and the Expanded Operative Stress Score with Preoperative Acute Serious Conditions, Complications and Mortality in Males Compared to Females: A Retrospective Observational Study. Ann Surg. 2023;277(2):e294–e304. doi: 10.1097/SLA.0000000000005027 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 5. Hall DE, Jacobs CA, Reitz KM, et al. Frailty Screening Using the Risk Analysis Index: A User Guide. Jt Comm J Qual Patient Saf. 2025;51(3):178–191. doi: 10.1016/j.jcjq.2024.12.005 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 6. Boyd K, Eng KH, Page CD. Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals. In: Blockeel H, Kersting K, Nijssen S, Železný F (eds) Machine Learning and Knowledge Discovery in Databases. Springer Berlin; Heidelberg; 2013:451–46. [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement The NSQIP PUF data used in this study is available to NSQIP-participating hospitals upon request from the American College of Surgeons. ACTIONS View on publisher site PDF (77.2 KB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• The Cutting Edge: A Systematic Review of Artificial Intelligence and Machine Learning in Predicting Esophagectomy Outcomes.#678321
• Interpretable machine learning for postoperative nausea and vomiting prediction in elderly orthopedic patients: a comparative study.#288738
• Machine Learning-Based Prediction of Postoperative Pain After Pediatric Ambulatory Surgery#880142
• Comparative Analysis of Logistic Regression and Machine Learning Algorithms for Predicting De Novo Stress Urinary Incontinence or Symptomatic Worsening Following Pelvic Floor Reconstruction: A Multicenter Study.#368079
• Validation of a Prediction Model for Inadequate Bowel Preparation#918566