Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice JNCI Cancer Spectr . 2026 Feb 27;10(2):pkag018. doi: 10.1093/jncics/pkag018 Search in PMC Search in PubMed View in NLM Catalog Add to search A process evaluation of early palliative care implementation in the National Cancer Institute Community Oncology Research Program Lisa Zubkoff Lisa Zubkoff , PhD 1 Department of Medicine, Division of General Internal Medicine and Population Science, University of Alabama at Birmingham, Birmingham, AL, United States 2 Birmingham/Atlanta VA Geriatric Research Education and Clinical Center, Birmingham VA Health Care System, Birmingham, AL, United States Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Writing - original draft, Writing - review & editing Find articles by Lisa Zubkoff 1, 2, ✉ , Elizabeth Carpenter-Song Elizabeth Carpenter-Song , PhD 3 Department of Anthropology, Dartmouth College, Hanover, NH, United States Conceptualization, Formal analysis, Investigation, Methodology, Writing - original draft, Writing - review & editing Find articles by Elizabeth Carpenter-Song 3 , Kathleen Doyle Lyons Kathleen Doyle Lyons , ScD 4 Department of Occupational Therapy, Massachusetts General Hospital Institute of Health Professions, Boston, MA, United States Conceptualization, Investigation, Writing - original draft, Writing - review & editing Find articles by Kathleen Doyle Lyons 4 , J Nicholas Odom J Nicholas Odom , PhD, RN 5 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States 6 Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States Conceptualization, Funding acquisition, Investigation, Writing - review & editing Find articles by J Nicholas Odom 5, 6 , Sarah Khalidi Sarah Khalidi , MPH 7 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States Conceptualization, Formal analysis, Investigation, Writing - original draft, Writing - review & editing Find articles by Sarah Khalidi 7 , Andres Azuero Andres Azuero , PhD 8 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States Conceptualization, Formal analysis, Investigation, Writing - review & editing Find articles by Andres Azuero 8 , Marie Flannery Marie Flannery , PhD, RN 9 School of Nursing, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States Conceptualization, Funding acquisition, Investigation, Writing - original draft, Writing - review & editing Find articles by Marie Flannery 9 , Richard Taylor Richard Taylor , PhD, DNP, CRNP, APN-BC 10 Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States Conceptualization, Investigation, Writing - review & editing Find articles by Richard Taylor 10 , Jaimie Richards Jaimie Richards , PhD 11 Department of Medicine, Division of General Internal Medicine and Population Science, University of Alabama at Birmingham, Birmingham, AL, United States Conceptualization, Investigation, Project administration, Writing - review & editing Find articles by Jaimie Richards 11 , Karen Mustian Karen Mustian , PhD, MPH 12 Department of Surgery, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States Conceptualization, Investigation, Writing - review & editing Find articles by Karen Mustian 12 , Megan Wells Megan Wells , MPH 13 Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States Conceptualization, Investigation, Writing - review & editing Find articles by Megan Wells 13 , Judy Hancock Judy Hancock , MHA, CCRP 14 Cancer Research for the Ozarks NCORP, Springfield, MO, United States Conceptualization, Investigation, Writing - review & editing Find articles by Judy Hancock 14 , Mark Wojtowicz Mark Wojtowicz , MS, MBA 15 Oncology Research and Innovation, Geisinger Cancer Institute NCORP, Danville, PA, United States Conceptualization, Investigation, Writing - review & editing Find articles by Mark Wojtowicz 15 , Matthias Weiss Matthias Weiss , MD 16 Wisconsin NCORP at ThedaCare, Appleton, WI, United States Conceptualization, Investigation, Writing - review & editing Find articles by Matthias Weiss 16 , Supriya Mohile Supriya Mohile , MD, MS 17 Department of Medicine, Division of Hematology/Oncology, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States Conceptualization, Funding acquisition, Investigation, Methodology, Writing - original draft, Writing - review & editing Find articles by Supriya Mohile 17, 2 , Marie Anne Bakitas Marie Anne Bakitas , DNSc, RN 18 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States 19 Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States Conceptualization, Funding acquisition, Investigation, Methodology, Writing - original draft, Writing - review & editing Find articles by Marie Anne Bakitas 18, 19, 2 Author information Article notes Copyright and License information 1 Department of Medicine, Division of General Internal Medicine and Population Science, University of Alabama at Birmingham, Birmingham, AL, United States 2 Birmingham/Atlanta VA Geriatric Research Education and Clinical Center, Birmingham VA Health Care System, Birmingham, AL, United States 3 Department of Anthropology, Dartmouth College, Hanover, NH, United States 4 Department of Occupational Therapy, Massachusetts General Hospital Institute of Health Professions, Boston, MA, United States 5 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States 6 Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States 7 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States 8 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States 9 School of Nursing, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States 10 Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States 11 Department of Medicine, Division of General Internal Medicine and Population Science, University of Alabama at Birmingham, Birmingham, AL, United States 12 Department of Surgery, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States 13 Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States 14 Cancer Research for the Ozarks NCORP, Springfield, MO, United States 15 Oncology Research and Innovation, Geisinger Cancer Institute NCORP, Danville, PA, United States 16 Wisconsin NCORP at ThedaCare, Appleton, WI, United States 17 Department of Medicine, Division of Hematology/Oncology, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States 18 School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States 19 Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States ✉ Corresponding author: Lisa Zubkoff, PhD, Division of General Internal Medicine and Population Science, Department of Medicine, 710 19th Street South, ALGEN- 694, Birmingham, AL 35294, United States ( [email protected] ). 2 Author Contributions: S. Mohile and M.A. Bakitas are co-senior authors. Roles Lisa Zubkoff : PhD , Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Writing - original draft, Writing - review & editing Elizabeth Carpenter-Song : PhD , Conceptualization, Formal analysis, Investigation, Methodology, Writing - original draft, Writing - review & editing Kathleen Doyle Lyons : ScD , Conceptualization, Investigation, Writing - original draft, Writing - review & editing J Nicholas Odom : PhD, RN , Conceptualization, Funding acquisition, Investigation, Writing - review & editing Sarah Khalidi : MPH , Conceptualization, Formal analysis, Investigation, Writing - original draft, Writing - review & editing Andres Azuero : PhD , Conceptualization, Formal analysis, Investigation, Writing - review & editing Marie Flannery : PhD, RN , Conceptualization, Funding acquisition, Investigation, Writing - original draft, Writing - review & editing Richard Taylor : PhD, DNP, CRNP, APN-BC , Conceptualization, Investigation, Writing - review & editing Jaimie Richards : PhD , Conceptualization, Investigation, Project administration, Writing - review & editing Karen Mustian : PhD, MPH , Conceptualization, Investigation, Writing - review & editing Megan Wells : MPH , Conceptualization, Investigation, Writing - review & editing Judy Hancock : MHA, CCRP , Conceptualization, Investigation, Writing - review & editing Mark Wojtowicz : MS, MBA , Conceptualization, Investigation, Writing - review & editing Matthias Weiss : MD , Conceptualization, Investigation, Writing - review & editing Supriya Mohile : MD, MS , Conceptualization, Funding acquisition, Investigation, Methodology, Writing - original draft, Writing - review & editing Marie Anne Bakitas : DNSc, RN , Conceptualization, Funding acquisition, Investigation, Methodology, Writing - original draft, Writing - review & editing Received 2025 Jul 29; Revised 2025 Dec 10; Accepted 2026 Feb 6; Collection date 2026 Apr. © The Author(s) 2026. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence ( https://creativecommons.org/licenses/by-nc-nd/4.0/ ), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact [email protected] for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact [email protected]. PMC Copyright notice PMCID: PMC13076938 PMID: 41761561 Abstract Background Care delivery trials to integrate early palliative care in community oncology practices are needed to enhance guideline-concordant care; however, little is known about how to successfully implement these trials within national research networks. Methods A process evaluation was conducted to identify and address practice and patient recruitment challenges within a nationwide cluster randomized trial testing 2 implementation strategies (virtual learning collaborative vs technical assistance) for integrating the Educate, Nurture, Advise, Before Life Ends (ENABLE) early palliative care program in the National Cancer Institute Community Oncology Research Program. To examine implementation barriers, we collected quantitative practice and clinician characteristics via surveys and conducted qualitative interviews with clinicians and staff. Interview data were coded and summarized using content analyses to identify barriers to implementing study procedures and the ENABLE program. Results Surveys and interviews were completed by 33 clinicians, 23 ENABLE coaches, and 19 coordinators in 9 practice clusters between April 2021 and June 2022. Although 78% of practice clusters identified availability of some palliative care services, none routinely referred all newly diagnosed advanced cancer patients to these services as recommended by guidelines. Key research and clinical ENABLE implementation barriers included limited staffing during and after COVID-19, low physician buy-in, belief that ENABLE overlapped with existing palliative services, and participant burden. In response, several trial modifications were made to enhance flexibility with study procedures and the clinical ENABLE intervention. Conclusions Implementation trials within cancer clinical trial networks must employ pragmatic and iterative study procedures to accommodate clinical practice workflows. Trial Registration ClinicalTrials.gov Identifier: NCT04062552 . Introduction Evidence-based early palliative care is recommended for individuals with advanced cancer to improve quality of life (QOL). 1 According to the American Society of Clinical Oncology, clinicians should refer patients with advanced cancer to specialized interdisciplinary palliative care teams early in the course of the disease, alongside standard cancer treatment. 2 Over a decade ago, the Educate, Nurture, Advise, Before Life Ends (ENABLE) model of early concurrent oncology palliative care demonstrated improved QOL for patients with advanced cancer 3-5 and their family caregivers, 6 , 7 but like many cancer treatment advances, widespread implementation has not occurred. Aspiring to close the gap between what we know and what we do, 8 , 9 implementation research focuses on changing individual and organizational culture, processes, and practices to integrate evidence-based advances into standard care. 10 , 11 Two implementation strategies are virtual learning collaboratives and technical assistance, which support capacity building and scale-up of evidence-based practices. 12 Building on prior work and guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance framework, 13-17 we designed a hybrid type III effectiveness-implementation cluster randomized trial to compare virtual learning collaborative and technical assistance approaches for implementing the evidence-based ENABLE early palliative care model for patients with advanced cancer in the National Cancer Institute (NCI) Community Oncology Research Program (NCORP) network. The University of Rochester Cancer Center (URCC) 18110CD ( NCT04062552 ) was activated in October 2019, just prior to the COVID-19 pandemic. The study design proposed 3 cohorts of 6-18 practices (clusters), each with a recruitment goal of 16-30 patients; within each cohort, half of the practices (clusters) would be randomized to implement ENABLE using a virtual learning collaborative approach and the other half using a technical assistance approach. 12 In this report, we describe a process evaluation of the first cohort of enrolled practices to characterize practices and staff participants, identify barriers to conducting implementation trial study procedures and the ENABLE model, and present study changes that improved trial feasibility. We also provide recommendations for conducting future implementation trials within cancer research networks. Future reports will present the implementation cluster randomized trial’s main and secondary outcomes. Methods In this process evaluation 18 we examined challenges in conducting a phase II hybrid type III effectiveness-implementation cluster randomized trial, by evaluating our first study cohort’s ( n = 9) practice and staff characteristics, patient enrollment, and barriers to implementing study procedures from April 2021 to June 2022. Setting Practice clusters were recruited from the URCC NCORP network. NCORP is an “NCI-approved national network that brings cancer control clinical trials and care delivery studies to people in their own communities.” 19 The URCC NCORP research base serves as the study coordinating center. NCORP community affiliates receive NCI funding to conduct study procedures. Clinical intervention The evidence-based ENABLE intervention for patients with advanced cancer and their family caregivers has been described. 3-5 Briefly, ENABLE includes a comprehensive palliative care assessment, 6 weekly nurse-led telehealth coaching sessions that cover essential palliative care topics, and monthly follow-up (see Figure 1 ). 20 A parallel ENABLE intervention for family caregivers consists of 3 weekly sessions and monthly follow-up with a nurse coach. The study team adapted the patient and caregiver facing materials for NCORP practices and developed ENABLE training modules for the clinical nursing staff who would deliver the intervention as nurse coaches. Nurse coaches were required to be registered nurses or advanced practice providers with a minimum of 2 years’ experience in oncology and/or palliative care. The online training comprised of interactive learning modules and recorded role play sessions to develop confidence with delivering ENABLE content. Training took an average of 13.4 hours to complete; 16.1 continuing education units were offered for nurse coaches who completed training ( Supplementary Methods, Table S1 ). Figure 1. Open in a new tab Components of the Educate, Nurture, Advise, Before Life Ends program. Adapted from National Consensus Project. 20 Abbreviation: PCP = primary care physician. Implementation strategies: virtual learning collaborative and technical assistance NCORP practice clusters were randomized to 1 of 2 strategies to implement ENABLE over 15 months: virtual learning collaborative 21-27 or technical assistance. 28-30 Research and clinical staff members at practice clusters randomly assigned to the virtual learning collaborative were required to attend a monthly group-based learning session led by study team members (K.D.L., J.N.O.). The virtual learning collaborative structure included a “report out” from each practice regarding study implementation, a facilitator-led discussion about ENABLE delivery, study procedures, quality improvement strategies that could aid implementation, and a practice cluster-led discussion about successes or challenges faced. The virtual learning collaborative followed an “all teach, all learn” philosophy and facilitators summarized and disseminated a document of the “group wisdom.” 24 , 26 Staff members at practice clusters randomly assigned to the technical assistance were required to attend an individual monthly consultation call with an ENABLE expert (R.T.) to discuss practice clusters' questions about ENABLE recruitment and delivery. In the technical assistance consultations, the leader offered expert advice based on prior ENABLE and palliative care research but avoided discussion of other practices. Data collection Practice and staff characteristics The URCC NCORP Research Base collected practice-level descriptors (eg, number of new cancer cases annually, presence of palliative care specialist, types of palliative care services provided) and research and clinical staff members’ self-reported baseline characteristics (eg, sex, years in practice) at study registration or enrollment via REDCap. Qualitative interviews To explore ENABLE implementation, a senior medical anthropologist with experience in implementation research (E.C.S.) conducted semistructured interviews with practice staff at month 12 of the 15-month intervention. At each practice, the nurse coach and study coordinator participated in the interview; additional staff members who were interested also participated. Interviews ranged from 30 to 60 minutes and were audio-recorded and transcribed. Transcripts were coded using Atlas.ti (version 24.1.1). Analysis began with an immersive review of the transcripts to gain familiarity with the dataset, followed by inductive open-coding using a focused analytic approach centered on implementation barriers. Coding was conducted by the lead qualitative analyst who developed structural codes based on the interview questions, and case memos distilled preliminary themes. 31 To enhance the credibility of the analysis, the lead analyst presented preliminary themes to study team members for feedback (ie, peer scrutiny) in debriefing sessions. 32 Team discussions informed further refinement of key themes. Results This process evaluation provides a descriptive summary of the first cohort and rich qualitative themes that explain practice and patient recruitment patterns and implementation barriers in an ongoing, nationwide implementation parent trial. Building on these themes, we present adaptations made to study procedures and the clinical ENABLE intervention to improve study feasibility. Practice and staff participants, practice characteristics, and enrollment Nine practice clusters were recruited between October 2019 and January 2021, and the 15-month virtual learning collaborative and technical assistance activities occurred from April 2021 to June 2022 ( Figure 2 ). All 9 practice clusters, which were mostly urban (88.9%), cared for individuals with advanced solid tumors. Practice clusters saw between 500 and 3000 new cancer patients annually; on average, 81.9% of patients served were White (median = 93%). No practice clusters routinely provided guideline-recommended early palliative care to all newly diagnosed advanced cancer patients 2 ( Table 1 ); however, 78% identified that some type of palliative care service was available to their patients. Figure 2. Open in a new tab ENABLE cohort 1 timeline. a Reflects the date of the first and last patient enrolled on the study. Practices were given a 15-month enrollment period. Abbreviations: TA = technical assistance; VLC = virtual learning collaborative. Table 1. Practice cluster characteristics Descriptive characteristics of active clusters ( n = 9) a No. (%) Median (IQR) Minimum Mean Maximum Estimated No. of new cancer cases per year ≥ 18 y of age - 2000 (850-2200) 500 1714 3000 Percentage of race or ethnicity of patients served American Indian or Alaska Native - 0.05 (0-0.07) 0 0.3 1.9 Asian - 0.1 (0.1-0.8) 0 5.5 47.4 Black or African American - 1.5 (0.5-13) 0.5 8.8 38.3 Hispanic or Latino - 1.0 (0.09-1.4) 0 1.0 3 Native Hawaiian or Other Pacific Islander - 0.07 (0.0-0.2) 0 1.0 7.8 White - 93.0 (80.0-95.0) 32.2 82 99 More than one race - 0.0 (0.0-0.0) 0 1.2 9.2 Other or unknown - 0.2 (0.0-0.8) 0 0.9 5.2 Presence of palliative care specialist Yes 5 (55.6) No 4 (44.4) Currently offer palliative care services on site Yes 7 (77.8) No 2 (22.2) Routinely provide early palliative care (functioning interprofessional team for each patient) No 9 (100.0) Type of palliative care services provided No current palliative care services 2 (22.2) Inpatient palliative care consultation service 6 (66.7) Home-based palliative care program (as part of hospice agency) 5 (55.6) Outpatient clinic 4 (44.4) Inpatient palliative care unit 3 (33.3) Telemedicine program 3 (33.3) Clinic practice: stand-alone; co-located; embedded 3 (33.3) Inpatient hospice beds per contractual relationship with a hospice agency 2 (22.2) Other b 6 (66.7) Rurality c Yes 1 (11.1) No 8 (88.9) Open in a new tab Abbreviation: IQR = interquartile range. a Source: Practice Cluster Interest Form (self-reported). b Other category includes home-based palliative care program (not part of hospice agency); screening (trigger) criteria to identify patients with palliative care needs; consultation service in a nursing home or subacute facility; community practice: facility visits, nursing home, assisted living, long-term acute care, home visit/home-based program; independent physician practice. c Based on Practice Cluster Rural-Urban Commuting Area codes (completed by University of Rochester Cancer Center). Across practice clusters, 91 physicians, nurses, and coordinators completed demographic information: 54% female, 55% White, and 20% Asian race ( Table 2 ). Of these 91 participants, 52% were physicians, 27% were ENABLE coaches, and 21% were research staff. Most physicians were medical oncologists (51%) who had been in practice for an average (SD) of 14.68 (11.1) years. Most ENABLE coaches (54%) were RNs, over half of whom had oncology or palliative certification. Table 2. Practice cluster personnel characteristics 2a. Provider, nurse coach, coordinator characteristics ( n = 91) a No. (%) Role Oncology physician and provider 47 (51.6) ENABLE coach (clinician) 24 (26.4) ENABLE coach (nonclinician) 1 (1.1) Coordinator 19 (20.9) Sex Male 19 (21.0) Female 49 (54.0) Unknown or not reported 23 (25.0) Ethnicity Non-Hispanic 69 (76.0) Unknown 22 (24.0) Race American Indian or Alaskan Native 1 (1.1) Asian 18 (20.0) White 50 (55.0) More than 1 race b 1 (1.1) Unknown or not reported 21 (23.0) 2b. Oncology physician ( n = 45) c No. (%) Median (IQR) Minimum Mean Maximum Years been in practice since oncology fellowship, median (IQR) - 13 (6-24) 0 14.68 39 Total No. of patients seen in typical workday, median (IQR) - 15 (10-20) 7 15.83 40 No. of days per week patients seen, median (IQR) - 4 (4-5) 2 4.2 5 Certified Medical oncologist 23 (51.1) Radiation oncologist 4 (8.9) Missing 18 (40.0) Board certified in oncology Yes 26 (58.0) No 3 (6.7) Missing 16 (36.0) Board certified in palliative care Yes 3 (6.7) No 26 (58.0) Missing 16 (36.0) 2c. ENABLE coaches (clinician) ( n =24) No. (%) Highest clinical designation Registered nurse 13 (54.0) Advanced practice nurse (nurse practitioner or clinical nurse specialist) 6 (25.0) Missing 5 (21.0) Highest academic degree Associate Degree in Nursing 2 (8.3) Bachelor of Science in Nursing 9 (38.0) Master of Science in Nursing 6 (25.0) Doctor of Philosophy 1 (4.2) Doctor of Nursing Practice 1 (4.2) Missing 5 (21.0) Certification None 7 (29.0) Palliative care 5 (21.0) Oncology 5 (21.0) Other 2 (8.3) Missing 5 (21.0) Open in a new tab Abbreviations: ENABLE = Educate, Nurture, Advise, Before Life Ends; IQR = interquartile range. a Source: Physician ENABLE coaches and coordinator demographics. b Mark all that apply. c Excludes provider role. The 9 practice clusters each enrolled 0-19 patients and 0-14 caregivers ( Table 3 ). Whereas 3 (33%) practice clusters met the patient accrual goal of 16-30 patients, 4 (44%) did not meet the minimum accrual goal, and 2 (22%) did not accrue. Qualitative interviews with staff ( n = 28) revealed several key barriers. Table 3. Patient and caregiver accrual among participating practice clusters Practice cluster Patients enrolled a Caregivers enrolled a 7 19 1 1 18 14 3 16 12 4 7 1 8 4 0 6 2 1 5 1 0 2 0 0 9 0 0 Total enrolled 67 29 Open in a new tab a All patient accruals occurred between July 6, 2021, and August 22, 2022. Caregiver accruals occurred between July 8, 2021, and August 31, 2022. Qualitative interview themes about study barriers Theme 1: study procedures were viewed as complex in a context of limited staff and time Across practice clusters, there were common challenges related to limited staffing, time constraints, and the COVID-19 pandemic. Participants described limited “bandwidth” and existing “employee shortages” that were, in some cases, exacerbated by the need to pull staff to the frontlines of COVID-19 care during surges. Research coordinators were often recruiting patients for multiple studies concurrently; not having a dedicated staff member to recruit for this study posed a challenge. The coordinators [at our site] are just amazing research clinicians. They are the two who are out there recruiting for every study that [our site] has. They’re amazing at what they do. Their time is limited. There’s not one person that’s devoted to trying to identify patients for this study. (Practice 102) Some participants emphasized their perception that study-related activities were unfamiliar, complex, and hard to follow compared with other study protocols. The protocol is just too much. It’s too wordy. It’s too redundant. Things are not easily accessible. It tells you to go here or there. It’s directing you to do things that REDCap. It just isn’t laid out as well as it could be… . It was a huge challenge to read the protocol and figure out kind of how we were gonna handle things. There are multiple tables in the protocol. It is a very confusing protocol, and I’ve been in research for 10 years, so that took a lot of time. (Practice 108) Theme 2: ENABLE implementation faced limited leadership buy-in and perceived overlap with existing services Participants reported valuing the ENABLE implementation because of its holistic, individualized, and problem-solving orientation and as a meaningful augmentation of symptom-focused treatment. It [ENABLE] addresses issues that are not so much disease management, but life management as you’re in this—entering this new phase of your life… . I mean you don’t ever tell your life story with your provider. You don’t really have an opportunity to do that, so [ENABLE] provides an environment to do that or to help with problem-solving at home ‘cause so much is about provider visits or—and rightfully so, discussing about the disease and disease management and symptom management, not so much as the emotional and the family management of the disease as well. (Practice 104) However, in some lower enrollment practice clusters, participants described limited physician buy-in generally for nontherapeutic trials such as ENABLE. For example, “Our physician buy-in for this study, after we opened it, has been very low.” (Practice 109) Some participants felt that a key organizational barrier in their practice clusters was the availability of a preexisting palliative care program. Well, I think one of the biggest problems was that we had something so similar to it. It was almost a disservice. It wasn’t a necessary thing for the patient because they had other resources. For us, it really wouldn’t be benefitting the patient at all because they have something almost exactly like it already in place as a resource. (Practice 109) Other participants reported that patients were frequently “overwhelmed” in the context of an advanced cancer diagnosis. In such cases, they felt that some participants viewed study participation as burdensome rather than potentially beneficial. The ones that declined for me, they just were overwhelmed. They just didn’t—it was just another layer of, “I can’t handle this situation right now.” They’re just like it was the easiest thing to not focus on. (Practice 104) Responses to challenges In response to practice-reported recruitment challenges, the study team solicited input from NCI and URCC NCORP Research Base leadership. There was consensus that pragmatic modifications to the study protocol were needed to improve study accrual while also flexing study processes to accommodate the realities of implementing early palliative care in the COVID-19 era. 33 , 34 First, we modified the primary aim so that the goal was to gather preliminary data to inform future trial design for implementation of early palliative care. Second, we switched from a cohort design to a more practical “rolling admission” approach where practice clusters were randomized when they were ready to begin study activities. Third, we reduced the number of practice clusters and the patient recruitment goal for each practice cluster. Fourth, we removed the ENABLE caregiver component so practice clusters could focus on delivering program components only to patients. Other changes added flexibility to how practice clusters implemented ENABLE. We reduced the nurse coach training requirements to start study activities and allowed them to complete subsequent trainings over time. We broadened staff eligibility to act as ENABLE coaches: in addition to registered nurses or advanced practice providers, physicians (eg, medical and radiation oncologists, including oncology trainees) and other professionals (eg, social workers, chaplains) with appropriate credentials and experience could be trained to deliver ENABLE. We added flexibility for practice clusters to implement fewer ENABLE program elements at their discretion: they were required to offer a palliative care assessment to all patients but could then choose to offer none or any of the 6 sessions to meet patient needs. The modifications (summarized in Table 4 ) led to increased practice and participant accrual after the amendment was activated on June 22, 2023 ( Figure 2 ). Table 4. Response to challenges Challenge Cohort 1 requirements Rolling admission requirements Study aims Primary aim: Compare the effectiveness of VLC vs TA on ENABLE program uptake, as measured by the proportion of patients who complete the ENABLE program, defined as having a palliative care assessment and completing the 6 ENABLE sessions. Secondary aim: Compare the effectiveness of VLC vs TA on NCORP practices’ overall ENABLE program implementation, as measured by the General Organizational Index. Exploratory aim 1: Compare NCORP practices randomized to VLC or TA on patient and caregiver QOL and mood outcomes. Exploratory aim 2: Determine the relationship among ENABLE program uptake, overall ENABLE program implementation, and patients’ QOL and mood across the 2 strategies. Primary aim: Gather preliminary data on the effectiveness of VLC vs TA on ENABLE program uptake, defined as the proportion of patients who complete a palliative care assessment and at least 4 ENABLE sessions. Secondary aims: Evaluate the preliminary effectiveness of VLC or TA on patient QOL and mood outcomes. Evaluate the relationship between ENABLE program uptake and patients’ QOL and mood across the 2 strategies. Exploratory aims: Evaluate the effectiveness of VLC vs TA on participating institutions’ overall ENABLE program implementation, as measured by the General Organizational Index. Evaluate the relationship between overall ENABLE program implementation and patients’ QOL and mood across the 2 strategies. Study design Original study design: Three cohorts where practice clusters start the study arms (VLC or TA intervention) at the same time Revised study design: Rolling admission approach where practice clusters are randomized and enter the study arms (VLC or TA intervention) when ready to begin study activities Practice cluster and participant recruitment Original sample sizes: Practice clusters: 48-58 (3 cohorts, 16 practices each with flexibility to recruit additional practices, up to 20% of 48, in cohorts 2 and 3) Practice cluster goal enrollment: 15 patients per practice cluster Total target sample size: 720-1440 -Patients: 720 (360 patients per strategy) -Caregivers: 720 (each patient may identify a caregiver) Revised sample sizes: Practice clusters: 16 with the option to recruit up to 4 more practice clusters per arm (total up to 24 practices) Practice cluster goal enrollment: 13 patients per practice cluster Total target sample size: 208 -Patients: 208 -Caregivers: not enrolling Training requirement Original ENABLE training requirements: Coordinators: Study platforms and procedures training: 2 required modules (1-2 hours) and 4 optional modules Nurse coaches: ENABLE training: 2 required modules (19-20 hours) and 1 optional module Study platforms and procedures training; see coordinators Physicians: No training required Revised ENABLE training requirements: Coordinators: Study platforms and procedures training: 2 required modules (1 to 2 hours) and 4 optional modules Nurse coaches: ENABLE training: 2 required modules (3-4 hours) and 9 optional modules if deliver ENABLE sessions Study platforms and procedures training; see coordinators Physicians: No training required unless serving as a nurse coach Clinical intervention Original ENABLE delivery requirements: Required: Palliative care assessment, 6 weekly sessions, 6 monthly follow-up calls Revised ENABLE delivery requirements: Required: Palliative care assessment Optional: At least 4 weekly sessions, 4-6 monthly follow-up calls Open in a new tab Abbreviations: ENABLE = Educate, Nurture, Advise, Before Life Ends; QOL = quality of life; TA = technical assistance; VLC = virtual learning collaborative. Discussion Despite practices’ best intentions and commitment to participating in this implementation trial testing 2 strategies to integrate the ENABLE model of guideline-concordant early palliative care for patients with advanced cancer, 2 numerous challenges disrupted the ability to carry out the trial as originally designed. Hence, after completing cohort 1, we made pragmatic trial adaptations (which may be encountered in hybrid effectiveness-implementation clinical trials 33 , 34 ) to change the trial design, study procedures, ENABLE training requirements, and components implemented. Importantly, this was the first implementation trial conducted within the URCC NCORP, using a study design and procedures that were unfamiliar to community oncology practices who were more accustomed to “therapeutic” and cancer control studies. This trial also required clinical resources. The demands of the COVID-19 pandemic relative to staffing and clinic visit procedures created implementation barriers, even though practices expressed excitement about ENABLE implementation. Implementing a new clinical practice requires significant resources, even under ideal conditions. Among participating practices, few staff had prior experience for delivering ENABLE. Although the training was self-paced and on-line, 35 , 36 it was impractical for nurses to complete during a busy clinic. The overwhelming staff demands to provide clinical care during the COVID-19 pandemic undermined efforts to implement a new clinical service, as most health-care facilities experienced clinical staffing shortages and role reassignments. Benefits for clinical staff members included learning how to deliver a new guideline-concordant care service and continuing education units. These potential benefits were not enough to incentivize clinical staff who faced increased workload during a resource-limited time. We thus reduced the required training to be consistent with other NCORP studies. Despite challenges, most practices supported the study goal to provide early palliative care to their patients. Although the study benefited from the collaborative infrastructure of the NCORP research network, new study procedures needed to be developed for this implementation study as the first of its kind in the network including methods for data collection and implementation processes. 37 , 38 NCORP was developed to conduct discrete time-limited, research studies, and practices may not have capacity or infrastructure to make long-term commitments to create new services within the context of a research protocol. In this study, we asked staff to change their clinical practice, using either clinical or research staff, with a goal of sustained services. Although practice clusters had flexibility in how they operationalized ENABLE, some did not have capacity to integrate clinical staff members in a new evidence-based practice. In another study conducted within NCORP, Watabayashi et al. 39 compared guideline informed intervention of primary prophylactic colony-stimulating factor standing orders with usual care and identified implementation challenges (eg, institutional buy-in, variation in local infrastructure and systems, need for sufficient effort). To translate findings into practice, the team used the Practical, Robust, Implementation, and Sustainability Model tool to propose activities to facilitate successful implementation of an intervention in future research. 40 , 41 Based on our lessons learned, we confirm and expand on the recommendations proposed by Watabayashi et al. 39 for implementation trials. First, we recommend working closely with consortium leadership to discuss appropriateness of study objectives and feasibility thresholds for potential participants. Second, key practice constituents—leadership, managers, day-to-day project staff, related departments—should be involved in the design of the study procedures and clinical service implementation procedures. Clinical and research teams should collaboratively identify contingency plans for institutional changes that may impact capacity and engagement. Third, research teams should plan and collaborate early with clinicians delivering the intervention. In our experience, although most practices were supportive of offering early palliative care, the combination of training for and delivering a new service and completing research study activities during the COVID-19 pandemic proved extraordinarily challenging. A strength of our process evaluation is that we sought and used feedback from all practices, regardless of their degree of success with enrollment. However, only 31% ( n = 28) of the 91 staff described in the practice characteristics survey engaged in the semistructured interviews. As such, the barriers noted in the interviews may reflect the opinions of only a subset of clinicians and staff. It is also possible that those opinions may be subject to recall bias and social desirability bias. We attempted to minimize that latter threat to validity by having the interviews conducted by a co-investigator who introduced herself as a member of the study team but was not one of the developers of the ENABLE study and did not work for the NCORP Research Base. The interviewer expressed a genuine interest in how to improve study procedures and probed for success stories and stumbling blocks. We feel these procedures helped minimize social desirability bias; however, we cannot be certain that we fully eliminated this threat, as participants could have provided answers they felt more comfortable stating. Regarding recall bias, the virtual learning collaborative and technical assistance facilitators noted similarity between challenges reported during technical assistance and virtual learning collaborative sessions and the themes generated in the analysis of the semistructured data, which helped us conclude that recall bias did not generate a large threat to validity in this process evaluation. Although lessons learned from this process evaluation are important contributions to the literature on implementation challenges and adaptations, we acknowledge that this manuscript does not include the analysis comparing the effects of the 2 study arms—virtual learning collaborative and technical assistance—on primary and secondary outcomes. In conclusion, when considering the adoption and implementation process of a new clinical practice, research teams should use a pragmatic and iterative approach that allows for ongoing adjustments as part of the study design. Supplementary Material pkag018_Supplementary_Data pkag018_supplementary_data.docx (34.4KB, docx) Acknowledgments The research reported in this manuscript uses resources of University of Alabama at Birmingham, Dartmouth College, Massachusetts General Hospital Institute for Health Professions, University of Rochester, University of Rochester Cancer Center, and University of Rochester Cancer Center NCORP Research Base. The funder did not play a role in the design of the study; the collection, analysis, or interpretation of the data; the writing of the manuscript; or the decision to submit the manuscript for publication, and the authors have ultimate authority over study activities. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute. This protocol (URCC18110CD) is approved by the National Cancer Institute Central Institutional Review board (IRB), acting as the single IRB of record for all study practices. The protocol is registered at clinicaltrials.gov ( NCT04062552 ). Provenance and peer review: Not commissioned; peer reviewed for ethical and funding approval prior to submission. Contributor Information Lisa Zubkoff, Department of Medicine, Division of General Internal Medicine and Population Science, University of Alabama at Birmingham, Birmingham, AL, United States; Birmingham/Atlanta VA Geriatric Research Education and Clinical Center, Birmingham VA Health Care System, Birmingham, AL, United States. Elizabeth Carpenter-Song, Department of Anthropology, Dartmouth College, Hanover, NH, United States. Kathleen Doyle Lyons, Department of Occupational Therapy, Massachusetts General Hospital Institute of Health Professions, Boston, MA, United States. J Nicholas Odom, School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States; Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States. Sarah Khalidi, School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States. Andres Azuero, School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States. Marie Flannery, School of Nursing, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States. Richard Taylor, Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States. Jaimie Richards, Department of Medicine, Division of General Internal Medicine and Population Science, University of Alabama at Birmingham, Birmingham, AL, United States. Karen Mustian, Department of Surgery, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States. Megan Wells, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States. Judy Hancock, Cancer Research for the Ozarks NCORP, Springfield, MO, United States. Mark Wojtowicz, Oncology Research and Innovation, Geisinger Cancer Institute NCORP, Danville, PA, United States. Matthias Weiss, Wisconsin NCORP at ThedaCare, Appleton, WI, United States. Supriya Mohile, Department of Medicine, Division of Hematology/Oncology, Wilmot Cancer Institute, University of Rochester Cancer Center, National Cancer Institute Community Oncology Research Program (NCORP) Research Base, Rochester, NY, United States. Marie Anne Bakitas, School of Nursing, University of Alabama at Birmingham, Birmingham, AL, United States; Department of Medicine, Division of Gerontology, Geriatrics and Palliative Care, Center for Palliative and Supportive Care, University of Alabama at Birmingham, Birmingham, AL, United States. Author contributions Lisa Zubkoff (Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Writing—original draft, Writing—review & editing), Elizabeth Carpenter-Song (Conceptualization, Formal analysis, Investigation, Methodology, Writing—original draft, Writing—review & editing), Kathleen Doyle Lyons (Conceptualization, Investigation, Writing—original draft, Writing—review & editing), J. Nicholas Odom (Conceptualization, Funding acquisition, Investigation, Writing—review & editing), Sarah Khalidi (Conceptualization, Formal analysis, Investigation, Writing—original draft, Writing—review & editing), Andres Azuero (Conceptualization, Formal analysis, Investigation, Writing—review & editing), Marie Flannery (Conceptualization, Funding acquisition, Investigation, Writing—original draft, Writing—review & editing), Richard Taylor (Conceptualization, Investigation, Writing—review & editing), Jaimie Richards (Conceptualization, Investigation, Project administration, Writing—review & editing), Karen Mustian (Conceptualization, Investigation, Writing—review & editing), Megan Wells (Conceptualization, Investigation, Writing—review & editing), Judy Hancock (Conceptualization, Investigation, Writing—review & editing), Mark Wojtowicz (Conceptualization, Investigation, Writing—review & editing), Matthias Weiss (Conceptualization, Investigation, Writing—review & editing), Supriya Mohile (Conceptualization, Funding acquisition, Investigation, Methodology, Writing—original draft, Writing—review & editing), and Marie Bakitas (Conceptualization, Funding acquisition, Investigation, Methodology, Writing—original draft, Writing—review & editing) Supplementary material Supplementary material is available at JNCI Cancer Spectrum online. Funding This work was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA229197. The URCC NCORP Research Base is supported by the National Cancer Institute of the National Institutes of Health under award number UG1CA189961. Cancer Research for the Ozarks NCORP is supported by the National Cancer Institute of the National Institutes of Health under award number UG1CA189822. Geisinger Cancer Institute NCORP is supported by the National Cancer Institute of the National Institutes of Health under award number UG1CA189847. Wisconsin NCORP is supported by the National Cancer Institute of the National Institutes of Health under award number UG1CA189956. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Conflicts of interest The authors declare that they have no conflicts of interest. Data availability Data that support exploratory outcome results for URCC NCORP Research Base clinical trials adhere to the NCI NCTN/NCORP Data Archive requirements. Per these requirements, data that support the findings of this study are not publicly available. Data requests can be made and must be approved through the URCC NCORP Research Base by contacting the Data Sharing Committe directing ([email protected]; [email protected]; [email protected]). All data access will require a formal data use agreement. References 1. Palliative Care in Cancer. National Cancer Institute. 2021. Accessed February 21, 2025. https://www.cancer.gov/about-cancer/advanced-cancer/care-choices/palliative-care-fact-sheet 2. Sanders JJ, Temin S, Ghoshal A, et al. Palliative care for patients with cancer: ASCO guideline update. J Clin Oncol. 2024;42:2336-2357. 10.1200/JCO.24.00542 [ DOI ] [ PubMed ] [ Google Scholar ] 3. Bakitas M, Lyons KD, Hegel MT, et al. Effects of a palliative care intervention on clinical outcomes in patients with advanced cancer: the Project ENABLE II randomized controlled trial. JAMA. 2009;302:741-749. 10.1001/jama.2009.1198 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Bakitas MA, Dionne-Odom JN, Ejem DB, et al. Effect of an early palliative care telehealth intervention vs usual care on patients with heart failure: the ENABLE CHF-PC randomized clinical trial. JAMA Intern Med. 2020;180:1203-1213. 10.1001/jamainternmed.2020.2861 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 5. Bakitas MA, Tosteson TD, Li Z, et al. Early versus delayed initiation of concurrent palliative oncology care: patient outcomes in the ENABLE III randomized controlled trial. J Clin Oncol. 2015;33:1438-1445. 10.1200/JCO.2014.58.6362 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 6. Dionne-Odom JN, Azuero A, Lyons KD, et al. Family caregiver depressive symptom and grief outcomes from the ENABLE III randomized controlled trial. J Pain Symptom Manage. 2016;52:378-385. 10.1016/j.jpainsymman.2016.03.014 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 7. Dionne-Odom JN, Hull JG, Martin MY, et al. Associations between advanced cancer patients’ survival and family caregiver presence and burden. Cancer Med. 2016;5:853-862. 10.1002/cam4.653 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 8. Chambers DA, Pintello D, Juliano-Bult D. Capacity-building and training opportunities for implementation science in mental health. Psychiatry Res. 2020;283:112511. 10.1016/j.psychres.2019.112511 [ DOI ] [ PubMed ] [ Google Scholar ] 9. Ducharme LJ, Wiley TRA, Zur JB, Vizcaino-Riveros JA, Martin L. A review of implementation research funded by the National Institute on Drug Abuse, 2007-2023: progress and opportunities. J Subst Use Addict Treat. 2024;167:209489. 10.1016/j.josat.2024.209489 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 10. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50:217-226. 10.1097/MLR.0b013e3182408812 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 11. Eccles MP, Mittman BS. Welcome to implementation science. Implementation Sci. 2006;1:1. 10.1186/1748-5908-1-1 [ DOI ] [ Google Scholar ] 12. Zubkoff L, Lyons KD, Dionne-Odom JN, et al. A cluster randomized controlled trial comparing Virtual Learning Collaborative and Technical Assistance strategies to implement an early palliative care program for patients with advanced cancer and their caregivers: a study protocol. Implement Sci. 2021;16:25. 10.1186/s13012-021-01086-3 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Zubkoff L, Dionne-Odom JN, Pisu M, et al. Developing a “toolkit” to measure implementation of concurrent palliative care in rural community cancer centers. Palliat Support Care. 2018;16:60-72. 10.1017/S1478951517000323 [ DOI ] [ PubMed ] [ Google Scholar ] 14. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. 2013;103:e38-46-e46. 10.2105/AJPH.2013.301299 [ DOI ] [ Google Scholar ] 15. Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64. 10.3389/fpubh.2019.00064 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 16. Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns. 2001;44:119-127. 10.1016/s0738-3991(00)00186-5 [ DOI ] [ PubMed ] [ Google Scholar ] 17. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322-1327. 10.2105/ajph.89.9.1322 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 18. Skivington K, Matthews L, Simpson SA, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061. 10.1136/bmj.n2061 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. About NCORP. U.S. Department of Health and Human Services. Accessed February 9, 2025. https://ncorp.cancer.gov/about/ 20. National Consensus Project for Quality Palliative Care. Clinical practice guidelines for quality palliative care. Kans Nurse. 2004;79:16-20. [ Google Scholar ] 21. Gotham HJ, Paris M Jr, Hoge MA. Learning collaboratives: a strategy for quality improvement and implementation in behavioral health. J Behav Health Serv Res. 2023;50:263-278. 10.1007/s11414-022-09826-z [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 22. Hulscher ME, Schouten LM, Grol RP, Buchan H. Determinants of success of quality improvement collaboratives: what does the literature show? BMJ Qual Saf. 2013;22:19-31. 10.1136/bmjqs-2011-000651 [ DOI ] [ Google Scholar ] 23. Lowther HJ, Harrison J, Hill JE, et al. The effectiveness of quality improvement collaboratives in improving stroke care and the facilitators and barriers to their implementation: a systematic review. Implement Sci. 2021;16:95. 10.1186/s13012-021-01162-8 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 24. Nadeem E, Olin SS, Hill LC, Hoagwood KE, Horwitz SM. Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91:354-394. 10.1111/milq.12016 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 25. Schouten LM, Grol RP, Hulscher ME. Factors influencing success in quality-improvement collaboratives: development and psychometric testing of an instrument. Implement Sci. 2010;5:84. 10.1186/1748-5908-5-84 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 26. Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ. 2008;336:1491-1494. 10.1136/bmj.39570.749884.BE [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 27. Zamboni K, Baker U, Tyagi M, Schellenberg J, Hill Z, Hanson C. How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review. Implement Sci. 2020;15:27. 10.1186/s13012-020-0978-z [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 28. Leeman J, Calancie L, Hartman MA, et al. What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective?: a systematic review. Implement Sci. 2015;10:80. 10.1186/s13012-015-0272-7 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 29. Scott VC, Jillani Z, Malpert A, Kolodny-Goetz J, Wandersman A. A scoping review of the evaluation and effectiveness of technical assistance. Implement Sci Commun. 2022;3:70. 10.1186/s43058-022-00314-1 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 30. Viglione C, Stadnick NA, Birenbaum B, et al. A systematic review of dissemination and implementation science capacity building programs around the globe. Implement Sci Commun. 2023;4:34. 10.1186/s43058-023-00405-7 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 31. Padgett D. Qualitative & Mixed Methods in Public Health . Sage Publications; 2012. Ch. 8, Data Analysis & Interpretation. 32. Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Education for Information. 2004;22:63-75. 10.3233/EFI-2004-22201 [ DOI ] [ Google Scholar ] 33. Boulton R, Semkina A, Jones F, Sevdalis N. Expanding the pragmatic lens in implementation science: why stakeholder perspectives matter. Implement Sci Commun. 2025;6:48. 10.1186/s43058-025-00730-z [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 34. Zatzick D, Palinkas L, Chambers DA, et al. Integrating pragmatic and implementation science randomized clinical trial approaches: a PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) analysis. Trials. 2023;24:288. 10.1186/s13063-023-07313-0 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 35. McDarby M, Miller M, Rosa WE, Buller H, Ferrell BR. Multidisciplinary oncology clinicians’ experiences delivering spiritual care to patients with cancer and their care partners. Support Care Cancer. 2024;32:586. 10.1007/s00520-024-08773-z [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 36. Miller M, Rosa WE, Buller H, McDarby M, Ferrell BR. “I often just don’t know what to say!”: variations in multidisciplinary palliative care clinicians’ confidence and needs related to spiritual care. Am J Hosp Palliat Care. 2025;42:560-570. 10.1177/10499091241265108 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 37. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. 10.1186/1748-5908-4-50 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 38. Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17:75. 10.1186/s13012-022-01245-0 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 39. Watabayashi KK, Bell-Brown A, Kreizenbeck K, et al. Successes and challenges of implementing a cancer care delivery intervention in community oncology practices: lessons learned from SWOG S1415CD. BMC Health Serv Res. 2022;22:432. 10.1186/s12913-022-07835-4 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 40. Trinkley KE, Glasgow RE, D’Mello S, Fort MP, Ford B, Rabin BA. The iPRISM webtool: an interactive tool to pragmatically guide the iterative use of the Practical, Robust Implementation and Sustainability Model in public health and clinical settings. Implement Sci Commun. 2023;4:116. 10.1186/s43058-023-00494-4 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 41. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34:228-243. 10.1016/s1553-7250(08)34030-6 [ DOI ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials pkag018_Supplementary_Data pkag018_supplementary_data.docx (34.4KB, docx) Data Availability Statement Data that support exploratory outcome results for URCC NCORP Research Base clinical trials adhere to the NCI NCTN/NCORP Data Archive requirements. Per these requirements, data that support the findings of this study are not publicly available. Data requests can be made and must be approved through the URCC NCORP Research Base by contacting the Data Sharing Committe directing ([email protected]; [email protected]; [email protected]). All data access will require a formal data use agreement. Articles from JNCI Cancer Spectrum are provided here courtesy of Oxford University Press ACTIONS View on publisher site PDF (757.3 KB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top