Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care in Improving Survival among Patients with Metastatic Cancer—The PRO-TECT Trial

Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care in Improving Survival among Patients with Metastatic Cancer—The PRO-TECT Trial - NCBI Bookshelf An official website of the United States government Here's how you know The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site. The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely. Log in Show account info Close Account Logged in as: username Dashboard Publications Account settings Log out Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation Bookshelf Search database Books All Databases Assembly Biocollections BioProject BioSample Books ClinVar Conserved Domains dbVar Gene Genome GEO DataSets GEO Profiles GTR Identical Protein Groups MedGen MeSH NLM Catalog Nucleotide OMIM PMC Protein Protein Clusters Protein Family Models PubChem BioAssay PubChem Compound PubChem Substance PubMed SNP SRA Structure Taxonomy ToolKit ToolKitAll ToolKitBookgh Search term Search Browse Titles Advanced Help Disclaimer NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health. Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care in Improving Survival among Patients with Metastatic Cancer—The PRO-TECT Trial Ethan Basch , MD, MSc, Deborah Schrag , MD, MPH, Sydney Henson , BS, Jennifer Jansen , MPH, Brenda Ginos , MS, Angela M. Stover , PhD, Philip Carr , MPH, Patricia A. Spears , BS, Mattias Jonsson , BA, Allison M. Deal , MS, Antonia V. Bennett , PhD, Gita Thanarajasingam , MD, Lauren J. Rogak , MA, Bryce B. Reeve , PhD, Claire Snyder , PhD, Deborah Bruner , PhD, David Cella , PhD, Lisa A. Kottschade , MSN, Jane Perlmutter , PhD, Cindy Geoghegan , MA, Cleo A. Samuel-Ryals , PhD, Barbara Given , PhD, Gina L. Mazza , PhD, Robert Miller , MD, Jon F. Strasser , MD, Dylan M. Zylla , MD, Anna Rapperport , MA, Anna Weiss , MD, Victoria S. Blinder , MD, and Amylou C. Dueck , PhD. Author Information and Affiliations Authors Ethan Basch , MD, MSc, 1 Deborah Schrag , MD, MPH, 2 Sydney Henson , BS, 1 Jennifer Jansen , MPH, 1 Brenda Ginos , MS, 3 Angela M. Stover , PhD, 1 Philip Carr , MPH, 1 Patricia A. Spears , BS, 1 Mattias Jonsson , BA, 1 Allison M. Deal , MS, 1 Antonia V. Bennett , PhD, 1 Gita Thanarajasingam , MD, 4 Lauren J. Rogak , MA, 2 Bryce B. Reeve , PhD, 5 Claire Snyder , PhD, 6 Deborah Bruner , PhD, 7 David Cella , PhD, 8 Lisa A. Kottschade , MSN, 9 Jane Perlmutter , PhD, 10 Cindy Geoghegan , MA, 11 Cleo A. Samuel-Ryals , PhD, 1 Barbara Given , PhD, 12 Gina L. Mazza , PhD, 3 Robert Miller , MD, 13 Jon F. Strasser , MD, 14 Dylan M. Zylla , MD, 15 Anna Rapperport , MA, 16 Anna Weiss , MD, 16,17 Victoria S. Blinder , MD, 2 and Amylou C. Dueck , PhD 3 . Affiliations 1 University of North Carolina, Lineberger Comprehensive Cancer Center, Chapel Hill 2 Memorial Sloan Kettering Cancer Center, New York, New York 3 Mayo Clinic, Scottsdale, Arizona 4 Mayo Clinic, Division of Hematology, Rochester, Minnesota 5 Duke Cancer Institute, Duke University School of Medicine, Durham, North Carolina 6 Johns Hopkins Schools of Medicine and Public Health, Baltimore, Maryland 7 Emory University, Atlanta, Georgia 8 Northwestern University, Feinberg School of Medicine, Chicago, Illinois 9 Mayo Clinic, Department of Medical Oncology, Rochester, Minnesota 10 Patient Representative, Gemini Group, Ann Arbor, Michigan 11 Patient Representative, Patient and Partners, Madison, Connecticut 12 Michigan State University, College of Nursing, East Lansing 13 American Society of Clinical Oncology, Alexandria, Virginia 14 Christiana Care Health Services, Wilmington, Delaware 15 The Cancer Research Center, HealthPartners/Park Nicollet, Minneapolis, Minnesota 16 Alliance for Clinical Trials in Oncology, Boston, Massachusetts 17 Brigham and Women's Hospital, Boston, Massachusetts Washington (DC): Patient-Centered Outcomes Research Institute (PCORI) ; 2024 Jan . Copyright and Permissions Copyright © 2024. The Alliance for Clinical Trials In Oncology Foundation. All Rights Reserved. This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/ Structured Abstract Background: Patients commonly have symptoms during cancer treatment that can go undetected. Symptom monitoring with electronic patient-reported outcome (ePRO) surveys may detect symptoms early and prompt clinicians to intervene, thereby improving outcomes. Objective: To evaluate whether ePRO symptom monitoring during cancer treatment improves patient survival, physical functioning, health-related quality of life (HRQOL), symptom control, and emergency department (ED) and hospital visits and to elicit feedback from patients, nurses, physicians, and external stakeholders on ePRO implementation barriers, facilitators, utility, value, and dissemination strategies. Methods: The Patient-Reported Outcomes to Enhance Cancer Treatment (PRO-TECT) trial (Alliance AFT-39; NCT03249090 ) was a cluster-randomized trial conducted in 52 US community oncology practices randomly assigned 1:1 to electronic symptom monitoring with patient-reported outcome (PRO) surveys (intervention arm practices) or to usual care (control arm practices). Between October 30, 2017, and March 16, 2020, 1191 adults receiving treatment for metastatic cancer were enrolled, with last follow-up on March 17, 2021. In the ePRO intervention arm, patients (n = 593) were asked to complete a weekly survey via an internet-based or automated telephone system for up to 1 year, including questions about common symptoms, performance status, falls, and financial burden. Severe or worsening symptoms triggered real-time alert notification emails to care team nurses. Longitudinal symptom reports were available at visits. In the control arm, patients (n = 598) received usual care. Both arms received access to standardized symptom management guidelines. The primary outcome of overall survival was assessed at 24-month follow-up based on data from the US National Death Index. Prespecified secondary outcomes included physical function, symptom control, and HRQOL at 3 months, measured by the European Organisation for Research and Treatment of Cancer QLQ-C30 instrument, and ED and hospital use based on electronic health record (EHR) data. Patient feedback about the utility and acceptability of and their satisfaction with the intervention was collected via surveys after 3 months of participation and at study completion. Nurse and physician feedback about barriers, facilitators, strategies, and implementation recommendations was collected after 6 months of experience with the ePRO intervention. Final trial results were shared with external stakeholder representatives of patient advocacy organizations and salient professional institutions (health systems, regulatory agencies, payers, academia) to elicit feedback on the value of the findings, their potential impact, and recommendations for future dissemination and implementation. Results: At 52 practices, 1191 patients were enrolled to participate (mean age, 62.2 years; 694 [58.3%] women; 16.6% Black or African American; 26.6% rural). All patients were included in the analysis, which found no significant differences for overall survival between arms (hazard ratio, 0.99; P = .86). Patients in the ePRO intervention arm experienced significant benefits compared with usual care in physical functioning (odds ratio [OR] for clinically meaningful benefits, 1.35 [95% CI, 1.08-1.70]; P = .009), HRQOL (OR, 1.41 [95% CI, 1.10-1.81]; P = .006), and symptom burden (OR, 1.50 [95% CI, 1.15-1.95]; P = .003). A significant difference in mean number of ED visits also favored the intervention (1.02 for ePRO vs 1.30 for control; P < .001). Findings were similar for hospital visits (0.96 vs 1.12; P = .05) and for a combined metric including ED and hospital visits (1.48 vs 1.81; P = .006). Surveys of patients in the intervention arm found that 463 of 496 (93.3%) thought the ePRO software was easy to use, 471 of 496 (95.0%) thought questions were easy to understand, 359 of 495 (72.5%) experienced improved discussions with their care team, 381 of 494 (77.1%) felt more in control of their own care, and 443 of 496 (89.3%) would recommend ePRO to future patients. In the survey of nurses across practices, 44 of 56 (78.6%) indicated ePRO information was helpful for EHR documentation, and 47 of 56 (83.9%) thought ePRO information improved the quality of discussions with patients. Although 44 of 58 (75.9%) nurses found alerts helpful, 28 of 57 (49.1%) thought that there were “too many” alerts. Among 39 surveyed oncologists, 34 of 39 (87.2%) noted that they reviewed ePRO information for clinical care, and 31 of 34 (91.2%) found the information useful. Semistructured interviews with nurses and physicians echoed these findings, with nurses suggesting there should be dedicated nursing time integrated into job descriptions to field and respond to ePRO alerts in future implementations. Nurses and physicians recommended integration of ePRO software with the EHR system, which was not done in this trial. All 23 external stakeholders thought the trial findings provided compelling evidence of benefits that will affect clinical practice. The stakeholders all expressed that although there was no survival difference between arms, the benefits of the secondary patient-centered outcomes were considered sufficiently compelling as a rationale for the value of the intervention. The stakeholders expressed that evidence of implementation across entire practices is a needed next step toward successful dissemination of this intervention. Conclusions: Symptom monitoring with ePROs during treatment for metastatic cancer did not affect the primary outcome of survival in this trial but conferred statistically significant and clinically meaningful benefits on the secondary patient-centered outcomes of physical function, HRQOL, and symptom control and reduced ED visits and hospitalizations. Feedback from participating patients, nurses, and physicians indicates high levels of acceptance, satisfaction, and endorsement of the intervention, with recommendations to protect some nursing effort in future implementations and to integrate the ePRO software with EHR systems. External stakeholder feedback indicates a need for future evaluation of wider implementation including a larger number of patients per practice, with greater EHR integration, a focus on improving the experiences of nurses with the intervention, and an eye toward generating data to support reimbursement models. In terms of initial dissemination, results have been published in JAMA , were used as the basis for recommendations in a major international clinical practice guideline, and have supported the rationale for ePRO integration in a new US payment model for oncology from the Centers for Medicare & Medicaid Services, with plans for additional publications and webinars. Limitations: The effects of the intervention may have been reduced by conduct of the trial across numerous community practices with diverse procedures for symptom management, by being conducted partially during the COVID-19 pandemic when site personnel were pulled away from care coordination roles on which this intervention relied, by lack of integration of the ePRO software with practice EHR systems, by lack of protected time for nurses to address alert notifications, and by baseline imbalances between arms due to the cluster-randomized design (eg, intervention arm patients overall were receiving later lines of therapy and had less access to palliative care support). It is reassuring that despite these limitations, there were clinically meaningful benefits to the intervention. Background Overview Symptoms are common among patients receiving treatment for advanced cancers and are a major cause of distress, functional disability, and emergency department (ED) and hospital use, 1-4 but their presence or severity go undetected and unaddressed by clinicians up to half of the time. 5-7 There is substantial and growing national interest in integrating electronic patient-reported outcomes (ePROs) into routine oncology practice as a system-level approach to improve detection and management of symptoms. 8 , 9 General Implementation Approach The general approach to implementing ePROs in oncology practice has been established through 2 decades of method work, demonstration projects, and small clinical trials, with subsequent best practice publications. 10-12 As shown in Figure 1 , the approach typically involves a software system through which patients can self-report symptoms on a regular basis via an electronic survey by using a computer, smart device, or automated telephone system. Severe or worsening symptoms trigger alert notifications to the care team (generally to a nurse or navigator), enabling the team to react and manage concerning symptoms in near real time. Reports of longitudinal symptom trajectories can be viewed by the nurse or oncologist at clinic visits to guide discussions and care. Figure 1 Information Flow With Integration of Electronic Patient-Reported Outcomes for Symptom Monitoring in Routine Oncology Care. Prior Research Prior single-center, smaller, and population-based studies have reported improved outcomes for physical function, symptom control, health-related quality of life (HRQOL), and ED and hospital visits when such ePRO systems are used for symptom monitoring in patients receiving cancer treatment. 13-16 Survival benefits have also been reported, specifically in the single-center Symptom Tracking and Reporting (STAR) randomized controlled trial (RCT), 17 which was conducted previously by the current investigators among patients with solid tumor malignancies at Memorial Sloan Kettering Cancer Center, as well as in a French controlled trial among patients with lung cancer 18 and in a large population-based study in Ontario, Canada. 19 In 2017, the investigators on this trial presented the single-center STAR study 17 as a plenary at a major international cancer meeting, the American Society of Clinical Oncology (ASCO) Annual Meeting, followed by publications in high-tier journals, showing clinical benefits of the intervention. The field of oncology had been focused largely on drugs as interventions with clinical benefits, and the demonstration of clinical benefits of a digital health intervention was paradigm shifting. Software companies emerged offering ePRO solutions in oncology, practices began developing or licensing systems, and payers such as the Centers for Medicare & Medicaid Services (CMS) looked to include ePRO symptom monitoring as a potential care enhancement in value-based payment models. 11 To provide evidence of effectiveness and optimal implementation strategies from a large controlled clinical trial in the real-world setting of community oncology practices, where most patients receive treatment for cancer, the Patient-Reported Outcomes to Enhance Cancer Treatment (PRO-TECT) trial was developed. Aims Aim 1: Determine whether systematic integration of electronic patient-reported symptoms into cancer care delivery improves meaningful patient-centered outcomes, including – 1a: physical functioning; – 1b: HRQOL; – 1c : survival; – 1d: ED and hospital visits; – 1e: symptom burden; and – 1f: patient experiences. Aim 2: Elicit perspectives about benefit-burden tradeoffs for integrating PROs into clinical workflow from various stakeholders, including – 2a: patient participants in the trial; – 2b: clinicians and staff conducting the trial at participating sites; and – 2c: representatives of patient and professional organizations. Aim 3: Identify barriers to, facilitators of, and strategies used by practices to integrate PROs into the clinical workflow. Gaps in Existing Evidence The aims of the PRO-TECT trial were developed to address gaps in existing evidence, as detailed above and in prior review publications, 11-13 specifically whether it is beneficial and feasible to monitor symptoms via ePROs across a wide variety of community oncology practices in the United States. Because the prior evidence had been generated from smaller studies or narrower populations, definitive evidence was not available regarding broad integration of ePROs into cancer care. Potential Impact Numerous US community oncology practices, health systems, and payers have considered including ePROs in value-based cancer care delivery models or have started using ePRO systems in the absence of clear implementation standards. This trial was envisioned to guide these efforts. Clinical practice guidelines for ePROs in oncology are being developed by oncology professional organizations and are in need of controlled trial evidence. There has not previously been a multicenter RCT of ePROs for symptom monitoring in community oncology despite high national interest in this topic. Participation of Patients and Other Stakeholders This section describes the overall approach to engagement of patients and other stakeholders in this research. However, it is worth noting that engagement was integral to every step in the research, from idea formulation to design, conduct, evaluation, reporting, and dissemination. There were patient investigators on the team representing a wide spectrum of cancer types and stages and treatment types, and there was extensive iterative work in patient populations through qualitative work and surveys at multiple points. We consulted regularly with and presented interval findings quarterly to the Patient Advocate Committee of the Alliance for Clinical Trials in Oncology (the Alliance) and to the Nursing and Community Oncology committees of the Alliance. We administered feedback surveys to patients, nurses, and physicians across participating research sites and conducted extensive semistructured interviews with patients, nurses, staff, and physicians at sites to assess barriers and facilitators. We also interviewed national stakeholders regarding potential impact of the trial findings. These various activities are detailed throughout this report in various sections as applicable and are described generally in this section. In addition, we reflect on the value of engagement overall in this work and areas where engagement approaches could be further optimized in future efforts. Patients and other stakeholders including nurses and oncologists were extensively involved in the design, conduct, analysis, and reporting of this trial. From the outset, the investigative team emphasized a collaborative model in which each stakeholder provided valuable perspectives, experience, and expertise that enriched the research culture and processes. As listed in Table 1 , multistakeholder participation on the core investigative team—including patients, nurses, oncologists, and methodologists—fostered co-learning and partnership. This engagement approach enabled each stakeholder's perspective to be equally valued, engendered trust, allowed sharing of design and data components with transparency, and promoted the respectful exchange of ideas. Our engagement approach was built on many years of prior multistakeholder participation in research projects in the National Cancer Institute (NCI) cooperative groups (see https://tinyurl.com/NCIpatients ), including collaborations with many of the patient participants in this trial. Table 1 Stakeholders on the Investigative Team and Study Advisory Board. The investigative team initially included 5 patient investigators, and sadly 1 passed away during the study. The 4 remaining patient investigators worked closely with the principal investigator and other investigators throughout the trial, reviewing all materials and approaches, joining scheduled meetings and ad hoc discussions, and weighing in on the framing and reporting of findings. The patient investigators received payment for their time equal to other consultants or investigators on the trial, both because their time is valued and because their contributions are highly meaningful, just as scientific or other professional input is meaningful. All patient investigators and consulted members of the Alliance Patient Advocate Committee were experienced advocates with prior experience advising cancer research through the NCI cooperative group mechanism. Broader input from nonadvocate patients was also considered valuable and was extensively sought from participants in the trial through surveys and interviews, as described in aims 2 and 3. A roster of stakeholders including patients is shown in Table 1 , with details of the areas of expertise and training brought by each that were salient to this trial. We made abundant use of the input of all of these experts throughout this research. Based on early feedback from patient investigators, it was decided to provide modest payments to patients for their time and effort completing outcomes questionnaires. All training materials were reviewed by patient and clinician stakeholders, with substantial changes made because of this engagement. A major decision during the study was to change the hierarchy of outcomes based on emerging data from other research, and feedback from the advisory board and patient investigators was pivotal in our decision-making. A decision to reallocate funds and expand the sample size and alter the recruitment strategy to optimize accrual of Black and African American patients was also made with substantial stakeholder input. Our analytical results presentation and papers were discussed and presented with our patient and other stakeholder participants at every step, yielding more meaningful and patient-relevant products. The ultimate impact of a highly interdisciplinary investigative team including patients was an environment in which salient information within each person's realm of knowledge and experience could be brought forth quickly and be integrated. Moreover, this approach provided a rich forum for an exchange of ideas that ultimately educated all members on alternative perspectives. It is our casual observation that all too often in clinical research, for example in drug development trials, decisions are made by a small group in a rarefied environment without perspectives representative of the ultimate end users of the research products. That said, interdisciplinary collaboration takes additional work to schedule meetings and allow time for honest exchange, or some participants will become marginalized. It is a responsibility of the principal investigators and lead research program coordinators to ensure that the discussion and engagement and a supportive environment are maintained and occur at appropriate touchpoint moments during the research. In our study, we achieved this through several vehicles. First, we were able to have frequent informal interactions with various stakeholders to provide input when needed. Second, we scheduled regular interdisciplinary video calls to review progress and elicit input. Third, we provided progress reports at the NCI Alliance Cooperative Group meetings to larger assemblies of patient representatives, nurse representatives, community oncologists, and outcome researchers for ongoing input. We found that the sum of these activities sometimes provided surprising input that required openness on our part to respond and adapt. There were no negative consequences to engaging in depth with patients and other stakeholders. The time that it took for discussion or presentation universally yielded greater confidence in our approaches and better products ultimately. Because PRO-TECT addressed a high-impact and pressing current question in oncology, it was conducted in partnership with 3 national patient organizations (Research Advocacy Network, Patients and Partners, and Cancer Information and Support Network) and the 2 major US oncology organizations (ASCO and the American Cancer Society [ACS]). The trial was conducted through the Alliance, a national research consortium that conducts multicenter trials at practice sites nationwide and through which ongoing advising was provided by 3 key committees made up of a broader group of patient advocates, oncology nurses, and community oncologists. The statistical center for this trial was also nested in the Alliance (based at the Mayo Clinic). These partnerships are shown in Figure 2 . Figure 2 Organizational Partnerships. Methods Study Overview PRO-TECT was a multicenter cluster-randomized trial designed to evaluate the effectiveness and implementation of electronic symptom monitoring with ePRO surveys during treatment for metastatic cancer. The study schema is shown in Figure 3 , and details of the design are described below. The trial was registered as ClinicalTrials.gov ID NCT03249090 . Figure 3 Study Schema for the Patient-Reported Outcomes to Enhance Cancer Treatment Trial. Study Setting and Recruitment of Patients Community oncology practices in the US national network of the Alliance ( www.allianceforclinicaltrialsinoncology.org ) participated in this trial. Practices were asked to consecutively approach and enroll up to 50 adults with metastatic cancer of any type receiving treatment with chemotherapy, targeted oral therapy, or immunotherapy if they understood English, Spanish, or Mandarin. Patients with indolent lymphoma or acute leukemia and those receiving hormonal monotherapy were excluded. Race and ethnicity information was obtained from patients through fixed categories to determine whether the effects of the intervention differed by race or ethnicity. The protocol and consent form were approved by central and local IRBs. Sites were reimbursed $1500 per enrolled patient participant to cover the costs of research and administrative processes related to patient enrollment, follow-up questionnaires, chart reviews, and data submissions. No funds from this trial were used to support the efforts of any clinical personnel. The $1500 amount was determined by the trial sponsor, the Alliance, based on prior QOL trial budgets and was based on a pretrial survey of practice sites in the Alliance network to determine the reasonable reimbursement for the anticipated research process work by personnel that would be needed for this trial. Practices were also separately reimbursed for any incurred IRB or other regulatory filing fees associated with this trial. To identify, approach, and recruit patients, practice site research managers were trained to work with clinical nurses and oncologists at their practice sites to identify eligible patients. To identify potentially eligible patients, the protocol specified that practice personnel could review clinical documentation such as patient charts in the electronic health record (EHR) or clinical schedules or ask clinical staff about potentially eligible patients through a limited waiver of authorization from the IRB. Potential patient participants could be approached and invited to be in the study by site research personnel or by designated clinical staff. It was specified in the protocol and site trainings that eligible patients were to be approached consecutively, except when purposive enrollment for specific target populations was directed for sites by the University of North Carolina (UNC) central study team, as described below. Cluster Randomization In this cluster-randomized trial, participating practices were randomly assigned 1:1 either to the ePRO intervention arm or to control. Practices were stratified by rural vs urban designation from US census criteria and randomly assigned via permuted blocks with block sizes of 2 or 4. Randomization lists were computer-generated based on random numbers and concealed until sites were randomly assigned. All analyses were at the patient level, comparing outcomes between patients treated at intervention sites and those at control arm sites. A cluster-randomized design was used in this trial because of a concern that patient-level randomization within practices could lead to carryover effects of the intervention in control arm patients. There was concern that if health care professionals saw some patients benefiting from intensified symptom monitoring, they might intensify their surveillance of control arm patients. As noted in later sections, after conducting this trial, the investigators ultimately thought that a cluster-randomized approach was not ideal because there was an imbalance between sites in level of illness (measured by number of lines of prior cancer therapy), access to palliative care services, and timing of accrual of patients (more rapid accrual in control arm with less exposure time of this arm to the COVID-19 pandemic) that may have reduced the measured impact of the intervention. Intervention and Control All participating practices, regardless of randomization arm, received online access to standardized educational materials for managing symptoms, including patient-level and clinician-level versions, as shown in Figure 4 and Figure 5 . These pathways were made available to practices, and to participating nurses in particular, in order to provide standardized, evidence-based information for them to refer to when addressing patients' symptoms. Figure 4 Example Patient-Level Educational Materials for Home Symptom Self-Management. Figure 5 Example Clinician-Level Educational Materials for Symptom Management. Practices randomly assigned to PROs were additionally provided access to an electronic survey system based at UNC's PRO Core facility. 20 Figure 6 shows the interface options of this system for patients, with access to complete surveys either by web, handheld device, or automated telephone system (with a voice speaking the survey questions which patients answered via numerical pushbutton responses). The approach used in this trial of offering multiple modes of patient interfaces was intended to meet people where they were in terms of computer experience, access, and preferences. Surveys were available in English, Spanish, or Mandarin, and patients self-selected the language for completion at registration or could toggle to a different language at each instance of survey completion. Figure 6 Patient Interface Options for Patients Participating in the Patient-Reported Outcomes to Enhance Cancer Treatment Trial (Intervention Arm). This electronic survey system was developed based on prior publications. 10 , 21 , 22 The system included a survey with questions from the NCI Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) validated item library, 23 , 24 selected based on prevalence across various types of advanced cancers and prior publications. 2 The survey included questions that elicited information about pain, nausea, vomiting, constipation, diarrhea, dyspnea, insomnia, and depression, as well as questions about oral intake (eating and drinking), performance status (patient-reported ECOG-ACRIN Cancer Research Group [ECOG-ACRIN] criteria), 25 falls, and financial challenges. 26 Table 2 shows the survey questions and response options as well as the criteria for triggering real-time alert notifications to the care team. Table 2 Patient-Reported Outcomes to Enhance Cancer Treatment Weekly Electronic Patient-Reported Outcome Survey. Training was provided to all practice sites by the central research team at UNC through an interactive 1- to 2-hour dedicated videoconferencing session during startup meetings. This session was attended by the practice's nurse champion, participating clinical nurses, participating administrative staff, site principal investigator, and lead site study coordinators. Attendance was optional for physicians. For both intervention and control practices, training involved a PowerPoint and question-and-answer session focused on study procedures, enrollment processes, forms, database training, and standard symptom management pathways. For intervention practice sites only, training also included lessons on the ePRO symptom monitoring software, a review of how alert notifications and longitudinal symptom reports function, how to customize the system to a practice, how to configure and respond to reminders, the roles and responsibilities of participating health care professionals, how to train patients in use of the ePRO system, and additional required forms. Training refreshers were offered periodically and provided for any new personnel at practice sites, although a “train the trainer” model was adopted in which local site personnel were equipped to train new personnel and successors and could reach out for support to the central team as needed. Patients at practices randomly assigned to the ePRO intervention were asked to complete surveys weekly for 1 year or until they discontinued all cancer treatment. On a day of the week and time of day selected by each patient, an email or automated call prompted survey completion. If the patient did not complete the survey after 24 hours, they received a reminder prompt. If they did not complete the survey after 72 hours, they received a backup call from a clinic staff member reminding them to self-report or offering to administer the survey verbally. Family members, caregivers, or staff were permitted to assist patients completing surveys. The weekly timing of ePRO survey administration was based on prior studies that form the basis for this trial in which weekly symptom survey administration was found to be optimal for catching clinical issues and avoiding meaningful memory degradation of intervening events 24 as well as on patient and nursing stakeholder input for this trial that weekly was the preferred time frame (which was subsequently confirmed in feedback questionnaires in this trial 21 ). As noted above, whenever a PRO-CTCAE score reached a prespecified level of magnitude or worsening compared with the prior survey (see Table 2 ), the patient received an email with a link to patient-level educational materials about their self-management of that symptom (see Figure 4 ). In addition, an email-based alert notification was sent to a designated administrative staff member at the patient's oncology practice who routed it to a clinical nurse responsible for that patient; notifications included ePRO scores and a link to clinician-level educational materials for managing the alert symptoms, which were similar to the symptom pathways provided to practices at baseline to inform and standardize approaches to symptom management based on best available evidence (see Figure 5 ). Alerts were also sent for ECOG-ACRIN scores greater than 2, ECOG-ACRIN scores that worsened by 2 or more points, falls, and financial distress scores greater than 2 (“quite a bit” or “very much”). Reports showing the trajectory of PROs could be visualized on screen or printed for clinicians ( Figure 7 ). These reports were generated by the PRO Core ePRO software and could be viewed on screens, printed, or scanned into the local EHR system. Practices were encouraged to provide reports for care teams to visualize at the time of follow-up clinic visits either on screen or printed, depending on the workflow of the particular practice or clinic. No specific instructions were given on how these should be viewed or used because workflow approaches were variable. Figure 7 Example Clinician Report Showing Longitudinal Trajectory of Patient-Reported Outcomes (for Visualizing or Printed at Clinic Visit). Responses to the alerts by nurses or oncologists were not specified by the study and were determined by the clinicians. Message triage and symptom management procedures are highly variable between and within oncology practices, and the roles of nurses and other personnel within these workflows similarly vary widely. Therefore, no specific instructions were given on how symptom management should be conducted except that it was specified that ultimately (1) a clinical nurse in the practice should review alert notifications and determine whether a patient should be contacted and what actions or escalations should be taken; (2) an administrative staff member should monitor and document the nursing contact, actions, and escalations in response to alert notifications; and (3) a physician or nurse on the care team should review longitudinal symptom reports at each patient visit. Individual practices could determine which personnel should fill these roles for a given patient in their practice. Clinician actions in response to alerts were documented in a standardized form. The electronic system was not integrated into EHR systems because this would not be feasible across so many sites. Patients in both study arms received $75 for their time and effort related to completing a research demographics form and the QLQ-C30 at enrollment and again at 3 months. Aim 1: Determine Whether Integrating Electronic Patient-Reported Outcomes in Cancer Care Improves Patient-Centered Outcomes Methods Study Outcomes: Primary The initial study protocol, approved July 27, 2017, included co-primary outcomes of overall survival and physical function at 3 months. On December 5, 2020, before data analysis, an amendment to the protocol changed the primary outcome to overall survival alone to increase statistical power for this outcome. It was determined that there was more than adequate power for the physical function outcome. At that time, physical function was changed to a key secondary end point, as described below. Dates of death for analysis of this outcome were obtained by linking individual patient records to the National Death Index at the US Centers for Disease Control and Prevention ( https://www.cdc.gov/nchs/ndi/index.htm ). Additionally, participating practices abstracted dates of death from EHRs through month 24 of each patient's enrollment. Survival was selected as a primary outcome for several reasons. Survival outcomes are the currency of oncology research because of the centrality of drug trials in this field. Therefore, to capture the wide attention of the oncology community, survival is a desirable end point. Medical journals similarly prioritize survival outcomes in evaluating manuscripts. Therefore, to optimize potential impact, survival was selected. In the prior large single-center trial on which this trial was based (the STAR study), overall survival significantly increased with a PRO symptom monitoring intervention. Mechanistically, there is prior evidence suggesting that PRO symptom monitoring improves symptom control, thereby improving physical mobility and lengthening tolerability of cancer treatment, both of which are associated with lengthened survival times. That said, a robust group of patient-centered secondary outcomes more proximate to the intervention was also included, which ultimately may be more meaningful to the daily experience of patients with incurable metastatic cancers, as described below. The time points for assessment were based on the STAR study, patient stakeholder input, population characteristics, and logistical constraints of the funding mechanism. Specifically, in the prior single-center trial, significant benefits in HRQOL were seen at 3 and 6 months and in ED admissions at 1 year. 14 The primary time point of 3 months for the HRQOL analyses in the PRO-TECT trial was selected based on review of these data and particularly on patient and nursing stakeholder input noting that this was a meaningful time point and on statistical team input that more patients were projected to be alive and able to provide HRQOL scores at 3 months rather than at later time points, given the level of illness of these patients with metastatic cancers. Multiple additional time points were also included to provide supporting data on trajectories of scores over time, up to 1 year. A cutoff after 1 year for the QOL assessments was selected because it was projected that many patients would have died at this point and because the budget of the funding mechanism would not have been sufficient to support practices administering further surveys beyond 1 year. Study Outcomes: Secondary Key secondary outcomes in this trial were physical function (after a transition from being a co-primary outcome, as noted above), symptom control, HRQOL, and ED visits. Data for physical function, symptom control, and HRQOL were obtained from patients during clinic visits at baseline and months 1, 3, 6, 9, and 12 of participation via the European Organisation for Research and Treatment of Cancer (EORTC) QLQ-C30 questionnaire. 27 The QLQ-C30 is a widely used 30-item questionnaire with excellent measurement properties. 28 Physical function was assessed via 5 items that generated a single score (range, 0-100; symptom burden or control was assessed as a composite of 8 QLQ-C30 symptom scale scores), 29 and overall HRQOL was assessed by combining function and symptom scale scores. 25 , 30 Each outcome was evaluated on a 0 to 100 scale (higher scores are better). Data for ED visits were obtained from data abstracted at each practice from EHRs. In the ePRO arm only, feedback was solicited from participants about the acceptability of and their satisfaction with the ePRO system after 3 and 12 months of participation, respectively. 10 The timing of data collection for these outcomes in aim 1, as well as baseline demographics, other study forms, and data to support outcomes in aims 2 and 3, are shown in the study calendar ( Table 3 and Table 4 ). Table 3 Study Calendar for Data Collection in Control Arm Only. Table 4 Study Calendar for Data Collection in Intervention Arm Only. Heterogeneity of Effect Analysis Preplanned subgroup analyses for the purpose of considering heterogeneity of effect (HTE) in PROs included age; sex; race; ethnicity; education level; employment status; marital status; prior computer, internet, or email experience; rural or urban location; cancer type; time since metastatic diagnosis; and line of cancer therapy. The rationale for this analysis is that in the previously published single-center RCT on which this trial was based, variables included in the multivariable model included age, sex, race, education level, level of prior computer use, and primary cancer type. 17 The analysis of overall survival included these variables. In addition, patient investigators for the PRO-TECT trial thought that these covariates were meaningful to include in analyses. A generalized linear mixed model was fit within each level of the subgroup variable, and the effect of the ePRO intervention was tested similarly to the primary analysis. Next, a generalized linear mixed model was fit including all patients and included fixed effects for arm, time point, and the given subgrouping variable as well as pairwise interactions between arm and visit, arm and subgroup variable, and visit and subgroup variable. Higher-order interactions found to be statistically insignificant were removed from the model. A random practice intercept term was included to account for clustering by practice. Repeated observations by patient were modeled via compound symmetric correlation structure over time. The interaction test P value was based on the type 3 test of the interaction effect between arm and the given subgroup variable. A statistically significant interaction test was used to identify HTE for a given subgroup (ie, covariate). For overall survival, preplanned subgroup analyses included sex, line of therapy, and cancer type. Sample Size Calculations and Study Power The initial protocol included a sample size of 1000, which was increased to 1200 in an IRB amendment on January 17, 2019, when additional funding became available to increase representation of patients who are Black or African American. This sample yielded 90% power for the primary outcome of overall survival, when physical function was still a co-primary outcome, for a hazard ratio (HR) of 0.76 (based on prior research 10 ) from a 2-sided α = .05/2 log-rank test with 576 observed events, with an intracluster correlation coefficient of 0.001 (based on prior trials), assuming the dropout of 150 patients in the first 2.5 years. When physical function became a key secondary end point on December 5, 2020, 1200 patients at 50 to 55 sites provided more than 90% power to detect a 0.37 SD difference between randomization groups based on prior research, 10 assuming an intracluster correlation coefficient of 0.055 and assuming that 85% of patients provided data at 3 months. Time Frame for the Study Participating patients remained enrolled in the trial for 12 months of intervention or control observation or until voluntary withdrawal, discontinuation of all cancer treatment, transfer to hospice, loss to follow-up, or death. Subsequently, practices abstracted chart data for each patient up to 24 months after enrollment for survival. Analytical and Statistical Approaches Statistical analyses were conducted by the Biostatistics Core Facility at the Mayo Clinic. All study data were maintained in highly secure encrypted data files at Mayo and UNC. Although the patients and health care professionals in this trial were aware of study arm allocations due to the cluster-randomized design, the investigative team and statistical team were masked to study arm allocation in data cleaning and analyses. The primary analysis of overall survival used a Cox regression model with cancer diagnosis-related covariates (eg, line of systemic cancer treatment, days since first diagnosis to metastatic cancer, days since metastatic cancer to study enrollment) and a random effect to account for site clustering. All deaths were included in the analysis, and patients without observed deaths were censored on the last date known alive, based on all available study forms and National Death Index administrative database data. For physical function, symptom control, and HRQOL, mean changes from baseline were compared between arms at each visit through a linear combination of parameters from a generalized linear mixed model. Each model included all available data from all time points from all patients according to their randomization arm. Fixed effects included study arm, time point, cancer type, and arm-by-visit interaction. A random practice intercept term was included to account for clustering by practice. In a responder analysis for physical function, symptom control, and HRQOL, patients who completed the QLQ-C30 at baseline and each follow-up time point were categorized as improved on each outcome if their score increased by 5 points or more from baseline, worse if their score decreased by 5 points or more, and otherwise as stable. A 5-point change was selected based on prior research determining that a 5-point change is clinically meaningful to patients based on an established scale for assessing the meaningfulness of score changes to respondents. 31 In a sensitivity analysis, results were repeated with a 10-point change used to define improvement or worsening to provide a wider range to ensure inclusion of clinically meaningful change scores. The proportion of patients with improvement, stability, or worsening was compared between arms at each time point via a cumulative logistic regression model with fixed effects for arm and cancer type and a random practice intercept term to account for clustering by practice. The primary time point for mean comparisons and responder analyses was prespecified as month 3, with supplementary information provided from mean comparisons and responder analyses at months 1, 6, 9, and 12. Preplanned subgroup analysis of physical function used the same mixed model approach described above to compare mean changes from baseline at month 3 between groups and within subgroups. An additional mixed model was performed for interaction testing. Time to first ED visit, time to first hospital visit, and time to first visit considering both ED and hospital visits were analyzed via Fine-Gray competing risk regression with death as a competing event and with adjustment for line of therapy, time from diagnosis to metastasis, time from diagnosis to treatment, and time from metastasis to enrollment, with stratification by site to account for site clustering, during a 1-year follow-up period. The numbers of ED and hospital visits per patient were tabulated for each arm as 0, 1, 2, 3, and 4 or more, and mean numbers of ED and hospital visits per patient were tabulated per arm with CIs and compared between arms via a mixed model. Each model included all available data from all patients according to their randomization arm. Fixed effects included randomization arm and cancer diagnosis-related covariates (line of systemic cancer treatment, days since first diagnosis to metastatic cancer, days since first diagnosis to treatment, and days since metastatic cancer to study enrollment). A random practice intercept term was included to account for clustering by practice. Statistical testing was 2-sided, with P < .05 considered statistically significant, and was carried out in SAS version 9.4 (SAS Institute, Inc). Patient and clinician feedback questionnaire items were analyzed descriptively. Missing Data Repeated observations by patients were modeled through compound symmetric correlation structure over time. Generalized linear mixed modeling produces unbiased estimates under an assumption of missing at random data, including handling patients dropping out due to death or other causes. 32 Time-to-event analyses censored patients at the last available data point if an event had not previously occurred. Changes to the Original Study Protocol A copy of the study protocols and statistical analysis plan are included in the appendices. As noted above, in the initial study protocol, approved July 27, 2017, the sample size was 1000 patient participants. This was increased to 1200 in a protocol amendment on January 17, 2019, to increase the representation of participants who were Black or African American, and a purposive enrollment approach was used to increase representation in the study population (described below). Additionally, physical function was initially a co-primary end point but was transitioned to a secondary end point in a protocol amendment on December 5, 2020, to increase statistical power for the survival outcome and because it was determined that there was more than adequate power for the physical function outcome. All changes were discussed, reviewed, and approved by PCORI. Enrollment was discontinued on March 23, 2020, with 1191 of 1200 (99.3%) planned enrollees, because practice site personnel were no longer able to recruit patients during the COVID-19 pandemic. Results Practices and Patients This trial included 52 community oncology practice sites across 25 states that were randomly assigned 1:1 to ePRO or control. Of these, 14 (26.9%) were designated as rural (7 per arm) ( Table 5 ). Each practice site was independent in that no staff member worked at more than 1 participating practice. Table 5 Roster of Participating Community Oncology Practices and Characteristics. As shown in the patient flow diagram ( Figure 8 ), between October 30, 2017, and March 16, 2020, 1444 patients were approached and 1191 were enrolled (593 ePRO, 598 control). At 3 months, the primary time point for secondary outcome analyses, 1098 of 1191 (92.2%) patients remained in the study and 1066 of 1098 (97.1%) patients completed the QLQ-C30. At 12 months, when the intervention period was completed, 683 of 1191 (57.3%) patients remained in the study, with discontinuation predominantly due to death, hospice, or permanent discontinuation of cancer treatment in this population of people with metastatic cancers (425/508 [83.7%]). All patients were included in the overall survival analysis. Dropout rates were greater in the ePRO arm (41/593 [6.9%]) vs control (7/598 [1.2%]) and were attributed to survey fatigue by 9 of 593 (1.5%) in ePRO and 3 of 598 (0.5%) in control. Figure 8 Patient-Reported Outcomes to Enhance Cancer Treatment Trial Flow Diagram (CONSORT). Baseline characteristics of the patient participants are shown in Table 6 . The median age of the patients was 63 years (range, 28-93 years), with 694 (58.3%) female, 925 (79.5%) White, 317 (26.6%) rural, 468 (39.4%) with a high school education or less, and 201 (16.9%) who never used the internet. Patients had a wide variety of cancer types. Table 6 Patient Characteristics (N = 1191) . Intervention Adherence Engagement and ePRO intervention adherence (completion rates) were high among participating patients and health care professionals. Among ePRO arm patients, 20 565 of 22 486 (91.5%) expected weekly ePRO surveys were completed without substantial reduction in ePRO completion rates over time ( Figure 9 ). Backup reminder calls by staff for patients who did not complete the electronic surveys when prompted were necessary for 3120 of 22 486 (13.9%) of the ePRO surveys. A total of 18 973 (84.4%) surveys were completed by patients themselves, 846 (3.8%) by caregivers, and 606 (2.7%) by staff. Figure 9 Proportion of Patients Completing Expected Electronic Patient-Reported Outcome Surveys. Among 20 565 completed surveys, there were 6979 (33.9%) alert notifications to nurses, most commonly for pain, diminished performance status, and diarrhea ( Table 7 ). Among these notifications, 4122 of 6979 (59.1%) resulted in immediate nursing actions, including telephone discussions with the patient or caregiver (3097 [75.1%]), advice for self-management at home (944 [22.9%]), and supportive medications prescribed or modified (868 [21.1%]). Table 7 Individual Symptom Alerts Contained in the Alert Notification Messages Triggered by the Weekly Survey System to the Care Team . Outcomes Summary There was no statistically significant difference between study arms in the primary outcome of overall survival. There were statistically significant and clinically meaningful benefits in favor of the ePRO arm for the prespecified secondary outcomes of physical function, HRQOL, symptom control, and ED visits and hospitalizations. Overall survival As shown in Figure 10 and in Appendix G , for overall survival through 24 months, the HR was 0.99 ( P = .86). The unadjusted Kaplan-Meier estimated survival rate at 24 months was 42.0% (95% CI, 38.2%-46.2%) for the ePRO intervention arm and 43.5% (95% CI, 39.7%-47.6%) for the standard care arm. A sensitivity analysis was conducted with the 12-month follow-up time point used as a cutoff to evaluate for potential impact of the COVID-19 pandemic on outcomes, with similar findings, with an HR of 1.03 ( P = .72). In an adjusted Cox regression model of overall survival, results comparing arms remained consistent, and results were consistent when we evaluated between-arm differences in overall survival within subgroups of interest, including cancer type, sex, and line of therapy (see supplementary table in Appendix G ). Figure 10 Unadjusted Overall Survival Analysis. Physical function, symptom control, and HRQOL Patients in the ePRO intervention arm experienced statistically significant and clinically meaningful benefits in physical functioning, HRQOL, and symptom burden compared with usual care (results published in JAMA ). 34 As shown in Figure 11 , at the prespecified 3-month time point, 13.8% more patients experienced clinically meaningful benefits (ie, at least 5-point change) in physical functioning (odds ratio [OR], 1.35 [95% CI, 1.08-1.70]; P = .009). Similarly, 16.1% more patients experienced improvements in symptom control (OR, 1.50 [95% CI, 1.15-1.95]; P = .003). Finally, 13.4% more experienced benefits in HRQOL (OR, 1.41 [95% CI, 1.10-1.81]; P = .006). As described above in the “ Methods ” section, this analysis was based on a 5-point change in QLQ-C30 scores on the 0 to 100 scale, which is considered a clinically meaningful score change. Results were similar in a sensitivity analysis with a higher threshold for score changes of 10 points, as shown in Figure 12 . These significant benefits in the proportion of patients with clinically meaningful changes persisted through the later time point of month 9; however, there was no longer a statistically significant difference by month 12. The loss of statistical significance by month 12 may be related to the intervention having the greatest impact earlier in treatment trajectory, or it could be related to attrition and lower sample size at that point as 508 of 1191 (42.7%) patients were no longer in the study for evaluation at month 12, largely because of deaths or hospice or discontinuation of all cancer treatment. Figure 11 Proportion of Patients Experiencing Improved, Stable, or Worsened Outcomes at 3 Months (N = 1191). Figure 12 Sensitivity Analysis: Proportion of Patients Experiencing Clinically Meaningful Improved, Stable, or Worsened Outcomes, at 3 Months. Mean differences in these outcomes were also superior for the ePRO arm compared with usual care. At 3 months, mean change from baseline was significantly better for the ePRO vs the control arm for physical function (mean difference, 2.47 [95% CI, 0.41-4.53]; P = .02), symptom control (mean difference, 2.56 [95% CI, 0.95-4.17]; P = .002 [ePRO, 77.67-80.03; control, 76.75-76.55]), and HRQOL (mean difference, 2.43 [95% CI, 0.90-3.96]; P = .002 [ePRO, 78.11-80.03; control, 77.00-76.50]) ( Figure 13 ). Mean differences remained statistically significant at months 6 and 9 but not at month 12. As noted earlier, at 12-month follow-up, 508 of 1191 (42.7%) patients were no longer in the study for evaluation. To provide additional detail, these data are shown using the observed mean scores between arms without comparison to baseline, with similar patterns observed between arms ( Figure 14 ). Figure 13 Mean Changes From Baseline at Each Assessment Time Point for (a) Physical Function, (b) Symptom Control, and (c) Health-Related Quality of Life (N = 1191). Figure 14 Mean Changes at Each Assessment Time Point Without Comparison With Baseline Values for (a) Physical Function, (b) Symptom Control, and (c) Health-Related Quality of Life (N = 1191). HTE analysis In a prespecified subgroup analysis for the purpose of assessing HTE, no interaction tests reached significance, as shown in Figure 15 . Figure 15 Preplanned Heterogeneity of Effect Analysis of Physical Function. ED visits and hospitalizations The difference in mean number of ED visits was statistically significant between randomizations arms (1.02 for the ePRO arm vs 1.30 for the control arm; P < .001). Differences were similar for the mean number of hospital visits (0.96 vs 1.12; P = .05) and for a combined metric including both ED and hospital visits (1.48 vs 1.81; P = .006). An improvement (increase) in time to first ED visit was observed in favor of the ePRO intervention arm, with an HR of 0.84 ( P = .03). The unadjusted cumulative incidence of an ED visit at 12 months was 48.7% (95% CI, 44.7%-53.0%) for the ePRO intervention arm and 54.8% (95% CI, 50.9%-59.0%) for the standard care arm. For hospitalizations, although there was a similar HR of 0.90, the P value did not reach significance ( P = .20). The proportion of patients with 0, 1, 2, 3, and 4 or more ED visits or hospitalizations was 252 of 593 (42.5%), 129 of 593 (21.8%), 101 of 593 (17.0%), 49 of 593 (8.3%), and 62 of 593 (10.5%), respectively, in the ePRO intervention arm and 218 of 598 (36.5%), 150 of 598 (25.1%), 83 of 598 (13.9%), 65 of 598 (10.9%), and 82 of 598 (13.7%), respectively, in the standard care arm. Symptom burden In an analysis of individual symptom trajectories, significant changes from baseline in mean differences in favor of the ePRO arm were observed for fatigue at all time points and at multiple time points for nausea, insomnia, appetite loss, and diarrhea but not for pain, dyspnea, or constipation ( Figure 16 ). To provide additional detail, these data are shown using the observed mean scores between arms without comparison to baseline, with similar patterns observed ( Figure 17 ). Figure 16 Mean Changes From Baseline at Each Visit for Symptom Scales. Figure 17 Mean Changes at Each Visit for Symptom Scales, Without Comparison to Baseline. Patient experiences Based on a survey administered to patients after 3 months of participation and again at off-study, among 496 patient participants who used the ePRO electronic system intervention, 359 of 495 (72.5%) thought the process improved discussions with their care team at 3 months, which increased to 188 of 244 (77.0%) at off-study; 345 of 493 (70.0%) stated their doctor or nurse used the symptom information they reported at 3 months, which increased to 196 of 243 (80.7%) at off-study; and 381 of 494 (77.1%) noted that self-reporting symptoms weekly from home made them feel more in control of their own care at 3 months, which increased to 205 of 244 (84.0%) at off-study. In terms of overall perceived value of the system, at 3 months, 443 of 496 (89.3%) strongly or somewhat agreed they would recommend the system to other patients (223 of 244 [91.4%] at off-study). Additional survey items administered to patients are described in aim 2, and the results of qualitative interviews with nurses and physicians are reported in aim 3. Discussion The results of this trial did not find an impact on the primary outcome of overall survival but found benefits in the prespecified secondary outcomes of physical functioning, HRQOL, symptom burden, and ED and hospital admissions. Symptoms are common and debilitating for patients with cancer 1 , 2 and are often unmeasured and unrecognized between visits. 5-7 Identifying symptoms early via PROs and alerting clinicians to their presence facilitated interventions to prevent subsequent symptom worsening or complications. 3 This trial showed improved PROs in response to this intervention. Although the impact on patient-centered outcomes and ED and hospital use mirrors the results of the single-center study on which this trial was modeled and subsequent randomized studies and population research, the overall survival benefit in that prior study and subsequent studies was not reproduced. There are several potential reasons for this. First, this trial was conducted across numerous community practices, where it was not possible to adhere to intervention procedures as meticulously as in the prior study, particularly during the COVID-19 pandemic. Second, a substantial portion of the trial was conducted during the pandemic, when clinical staff were pulled away from navigation and care coordination services, which are necessary components of this complex intervention, possibly blunting the intervention effect. Indeed, the magnitude of the benefit on patient-centered outcomes was lower than in the prior study, which may have limited the impact on survival. We did test for the impact of COVID-19 on the findings, and reassuringly, there was not a measurable significant impact. However, the impact of staff diversion, stress felt by personnel, delayed treatments, and reduced follow-up care may have affected QOL and survival outcomes in both arms, blunting the potential differences in outcomes. Third, the results may have been affected by baseline imbalances between arms due to the cluster-randomized design, as shown in Table 6 . For example, intervention arm patients overall were receiving later lines of therapy (third and fourth line) more frequently at baseline compared with control, suggesting that intervention patients may have been more affected by the cumulative adverse effects of multiple lines of prior treatment, and they might have been receiving less efficacious therapies during trial participation. Additionally, intervention arm patients had less access to palliative care support (8.6% vs 15.7%), which could have adversely affected their outcomes (palliative care has been shown to lengthen survival in patients with advanced cancers). Finally, in the intervening period since the prior study, there have been innovations in cancer treatments and an increase in the deployment of navigators and care coordinators in oncology practices, which may have improved symptom management writ large, thereby lessening the overall impact of ePRO monitoring. Nonetheless, the impact seen in this trial on patient-centered outcomes provides strong evidence for clinically meaningful benefits of ePRO monitoring for patients receiving systemic cancer treatment. Aim 2: Elicit Perspectives About Benefit Burden Tradeoffs for Integrating Patient-Reported Outcomes Into Clinical Workflow To elicit perspectives about integrating ePROs into clinical workflow from various stakeholders, surveys were conducted among patients, nurses, and physicians via the ePRO system. Interviews were also conducted with representatives of patient and professional organizations. Methods Patient Perspectives To elicit patient perspectives, we administered feedback surveys after 3 months of participation and when patients completed study participation (ie, off-study). The survey assessed patient comprehension by asking whether ePRO questions were easy to understand; general usability by asking whether the digital ePRO system was easy to use; and meaningfulness or relevance by asking whether the ePRO questions were relevant to them. In addition, as described above, the survey asked about communication or actionability, clinical utility, and self-efficacy by asking whether they thought the process improved discussions with their care team, whether their doctor or nurse used the symptom information, and whether the system made them feel more in control of their own care, respectively. For overall perceived value of the system, they were asked whether they would recommend it to other patients. Nurse and Physician Perspectives After at least 6 months of experience using the system, we elicited nurse input on clinical utility by asking whether ePRO information was helpful for clinical documentation in the EHR, improved quality of discussions with patients, increased efficiency of discussions with patients, and was useful for patient care. For quality and value of care, nurses were asked whether they thought that using the system improved quality of care for patients and whether they would recommend it to other clinics, respectively. We elicited physician (oncologist) input on clinical utility by asking how often ePRO information was used to guide discussions with patients and to make treatment decisions. Patient Advocate and Professional Stakeholder Perspectives Systematic feedback from key stakeholders was elicited as a research method and also as a form of broader engagement. Through a partnership with the ACS, a group of 23 patient representatives and professional stakeholders were identified and individually interviewed by a professional interviewer from the ACS anonymously. Patient representatives in this group included leaders from patient organizations including the Research Advocacy Network, Patients as Partners, and the Cancer Information & Support Network. Professional stakeholders in this group included individuals from the national oncology professional organizations, payers, EHR vendors, federal agencies, and cancer centers in US health systems. These people were selected based on prior knowledge or involvement in the field of ePRO research or implementation to ensure that they would have sufficient background to contextualize the findings of the PRO-TECT trial. These interviews were conducted after the completion of data analysis for the PRO-TECT trial, and the results of the trial were shared with these people at the start of each interview in order to elicit feedback about the meaning and relevance of the results; about potential barriers to, facilitators of, and strategies for implementing ePROs in oncology clinical practice; about the potential value of ePROs based on this trial (ie, whether the cost and effort are worth the measured benefits); and about future directions for follow-up work. For these interviews, a semistructured guide was developed by the ACS and was reviewed by the study investigators. Telephone interviews lasting 30 to 45 minutes followed this guide. All interviews were digitally recorded and were transcribed anonymously by a third-party transcription service. Interview transcripts were analyzed in Dedoose (version 9.0.17, 2021), under standard consensus coding guidelines. Statistical Analyses For the survey data, the primary analysis entailed descriptive statistics of patient responses in the 3-month feedback questionnaire, including medians and ranges for continuous variables and frequencies and relative frequencies for categorical variables. In an unplanned exploratory analysis, we considered how patient responses on the 3-month feedback questionnaire and other patient variables were related to patient completion of weekly surveys. Completion of patient self-reports was tabulated for each patient as the number of completed weekly ePRO questionnaires divided by the number of questionnaires that were expected to be completed. For example, if a patient was expected to complete 26 weekly surveys and actually completed 13 of them, then the patient's completion rate was 50% (ie, 13/26). Therefore, the outcome being modeled was a value between 0% and 100% for each patient. Pairwise relationships between patient survey items with patient baseline characteristics (patient-selected survey mode, age, sex, ethnicity, race, urban vs rural location of clinic, educational attainment, employment status, marital status), baseline technical experience (prior internet use, prior email use, prior computer use), baseline financial status (difficulty paying bills), and completion of self-reporting during trial participation were described via Spearman correlations and χ 2 tests. P values for Spearman correlations were evaluated graphically via a choropleth map (ie, “heat map”). Multivariable linear regression with forward selection (variable entry until statistical significance P < .10) was used to evaluate the relationships of patient characteristics and the rate of completion of self-reports with patient survey items. P < .05 was considered statistically significant. Results Patient Perspectives By the cutoff date of May 4, 2020, for this analysis there were 497 patients in the PRO-TECT trial intervention arm who completed at least 3 months of participation. Among these, 496 of 497 (99.8%) completed the 3-month feedback survey. In addition, by the cutoff date, 257 patients had finished participation in the study, of whom 245 of 257 (95.3%) completed the off-study feedback survey. Reasons for missed off-study surveys were deaths (6 [2.3%]) and preferring not to complete it (6 [2.3%]). Results of the 496 patient surveys at 3 months are shown in Figure 18 . Comprehension of the ePRO questions was high, with 471 of 496 (95.0%) patients strongly or somewhat agreeing that questions were easy to understand at 3 months (and 230/242 [95.0%] at off-study). General usability of the digital ePRO system was also high, with 463 of 496 (93.3%) strongly or somewhat agreeing that the system was easy to use at 3 months (227/243 [93.4%] at off-study). A meaningfulness or relevance question was added to the survey partway through the trial and therefore was administered to fewer patients, with 350 of 383 (91.4%) strongly or somewhat agreeing that the ePRO questions were relevant to them at 3 months (214/237 [90.3%] at off-study). Figure 18 Patient Feedback Survey Responses at 3 Months. Regarding communication or actionability, clinical utility, and self-efficacy, perceived benefits increased over time after more experience with the system. Among patients, 359 of 495 (72.5%) thought the process improved discussions with their care team at 3 months, which increased to 188 of 244 (77.0%) at off-study; 345 of 493 (70.0%) stated their doctor or nurse used the symptom information they reported at 3 months, which increased to 196 of 243 (80.7%) at off-study; and 381 of 494 (77.1%) noted that self-reporting symptoms weekly from home made them feel more in control of their own care at 3 months, which increased to 205 of 244 (84.0%) at off-study. In terms of overall perceived value of the system, at 3 months 443 of 496 (89.3%) strongly or somewhat agreed they would recommend the system to other patients (223/244 [91.4%] at off-study). The survey also assessed practical and process aspects of the system configuration. When asked about the frequency of self-reporting symptoms, 357 of 378 (94.4%) patients responded that weekly was “just about right” (5 [1.3%] said “not often enough” and 17 [4.5%] said “too often”). The length of time to finish the weekly surveys was thought by 474 of 496 (95.6%) to be reasonable (20 [4.0%] said “neither agree nor disagree” and 2 [0.4%] said “disagree”). When asked whether they would like the ePRO symptom questions themselves to be changed from time to time (eg, phrasing, formatting, or appearance), 52 of 383 (13.6%) strongly agreed, 117 (30.5%) somewhat agreed, 163 (42.6%) neither agreed nor disagreed, 20 (5.2%) somewhat disagreed, and 31 (8.1%) strongly disagreed. Regarding the educational materials about home symptom management triggered for severe or worsening symptoms, 302 of 489 (61.8%) thought the materials were useful. Analysis of Covariates Table 8 shows that virtually all patient feedback survey questions were significantly positively correlated with completion of weekly ePRO self-reports in both univariate and multivariable analyses, demonstrating an association between higher rates of ePRO completion and higher perceived comprehension, general usability, meaningfulness, actionability, clinical utility, and self-efficacy related to the system. The rate of completion of ePRO weekly reports overall was high, with 91.5% of expected weekly reports completed on time across all patients (85% directly by patients, 4% by caregivers, and 3% by staff on behalf of patients). Nonetheless, there has been variation in individual patients' compliance with weekly reporting, ranging from 17% to 100%. Table 8 Correlations Between Patient Survey Questions With Variables . Both comprehension (“questions were easy to understand”) and general usability (“system was easy to use”) were significantly correlated with prior experience using connected technologies, specifically with experience using email or the internet ( Table 8 ). However, significant correlation was not seen with simply using a computer, tablet, or smartphone previously. There was a negative correlation of prior internet use with both meaningfulness or relevance (“questions were relevant to me”) and communication or actionability (“improved discussions with my doctor/nurse”), suggesting that the system may offer particular benefits for patients lacking prior experience with connected technologies. Patient-selected survey mode was significantly related to comprehension, general usability, and value in favor of web over automated telephone, whereas clinical utility favored automated telephone. It is unclear in this analysis whether these associations were due to factors leading patients to select 1 mode or another or to inherent experience using those modes. Table 8 also shows relationships between patient survey responses and other patient baseline characteristics. Patient self-identification of race as being White was significantly correlated in univariate and multivariable analyses with perception that the ePRO system improved communication or actionability, clinical utility, self-efficacy, and value but not comprehension, usability, or meaningfulness or relevance. Higher educational attainment was significantly positively correlated with usability and was negatively correlated with meaningfulness or relevance, communication or actionability, and self-efficacy, suggesting that the added benefit of the ePRO system may be greater for patients with less education. Statistical significance of these negative correlations was retained in the multivariable model. Although many of the correlations are statistically significant, indicating the correlation is nonzero, many of the values are just above a value of 0.1, which is considered in the low range (ie, between 0.1 and 0.3). Nurse and Physician Perspectives Surveys were completed by a total of 57 clinical nurses from 23 (88.5%) of the 26 participating community practices. Surveys were not administered at the 3 remaining practices because they had enrolled 2 or fewer patients and thus had limited experience with the PRO-TECT system and workflow. Multiple nurse surveys were completed at practices that included more than 1 site of service. Physician surveys were completed by 39 oncologists from 22 practices (84.6%) as 1 additional practice had no physicians agree to take the survey. Figure 19 shows results of the nurse survey. In terms of clinical utility, 44 of 56 (78.6%) nurses indicated that patient self-reported information was helpful for documentation in EHRs, 47 (83.9%) noted the information improved the quality of their discussions with patients, and 47 (83.9%) found that the information increased the efficiency of their discussions with patients. In an additional clinical utility question posed as “yes/no” (not shown in Figure 19 ), 42 of 56 (75.0%) nurses thought that, overall, patient self-reported information was useful for patient care. In terms of quality and value of care, 39 of 56 (69.6%) nurses thought that the system improved quality of care for patients, 33 (58.9%) would like to use it in the future, and 36 (64.3%) would recommend it to other clinics. Figure 19 Nurse Feedback Survey Responses. With alerts being a potential dissatisfier for health care professionals, nurses were asked about the number of ePRO alerts received, with 26 of 55 (47.3%) stating “too many,” 19 (34.5%) “just right,” 1 (1.8%) “too few,” and 9 (16.4%) “unsure.” When asked whether they found alerts to be helpful, 40 of 56 (71.4%) agreed, 7 (12.5%) neither agreed nor disagreed, and 9 (16.1%) disagreed. The preferred mechanism for receiving alerts was by email (38/55 [69.1%]), followed by EHR message (11/55 [20.0%]), although the ePRO system in this trial was not integrated with EHRs. When asked how much time it took to review each ePRO alert, 29 of 56 (52%) nurses noted less than 5 minutes, 20 of 56 (36%) noted 5 to 10 minutes, and 5 of 56 (9%) noted 11 to 20 minutes (1 noted >20 minutes and 1 “not applicable”). Nurses were also asked how long it took to review full symptom reports at clinic visits, with 22 of 56 (39%) stating less than 5 minutes, 15 of 56 (27%) stating 5 to 10 minutes, and 3 of 56 (5%) stating 11 to 20 minutes (1 noted >20 minutes and 15 “not applicable”). Regarding the symptom management pathways accompanying alerts (see Figure 5 ), 28 of 56 (50.0%) nurses stated they had used the pathways, among whom 23 of 28 (82.1%) agreed the pathways were useful (2 [7.1%] neither agreed nor disagreed, 3 [10.7%] disagreed). These pathways were made available to nurses to in order to provide standardized, evidence-based information for them to refer to when addressing patient-reported symptoms. Feedback was elicited to determine whether provision of pathways was considered helpful by nurses. Although only half of nurses made use of the pathways, most of those who used them found them to be helpful. Reasons for underuse might be availability of local guidelines, limited interest, or difficulties accessing the information. This is an area for future potential research given the level of interest and unrealized potential of pathways because this trial was not designed for a comprehensive assessment of symptom pathway use by nurses. Among physicians (oncologists), 34 of 39 (87.2%) reported that they had actively reviewed patient self-reported information from the study during clinical care, generally through discussions with nurses, and 31 of 34 (91.2%) stated they found the information to be useful. In terms of clinical utility, when asked whether they used the patient-reported information to guide discussions with patients, 11 of 34 (32.4%) stated “often,” 11 of 34 (32.4%) “sometimes,” and 10 of 34 (29.4%) “rarely.” Similarly, when asked whether they used the patient-reported information to make treatment decisions, 7 of 34 (20.6%) stated “often,” 15 of 34 (44.1%) “sometimes,” and 9 of 34 (26.5%) “rarely.” When asked what the ideal approach would be for them to receive patient-reported information at clinic visits, 52% wanted computerized information in the EHR, 26% wanted it to be printed out, 20% wanted to have a tablet computer brought to them with the information, and 2% wanted to be told the information directly by patients. Stakeholder Perspectives From Patient and Professional Organizations As described above, through a partnership with the ACS, individual, anonymous, semistructured interviews were conducted with 23 key stakeholders including 6 patient advocate organization national leaders, 12 health system or professional organization national leaders, and 5 PRO method national leaders. For these interviews, a semistructured interview guide was developed by the long-standing expert Connected Health for Applications & Interventions (CHAI) Core Qualitative Research Facility at the UNC in collaboration with the investigative team and patient partners and with oversight by Dr Tenbroeck Smith of the ACS. The interview guide included questions on the potential impact of the study results on oncology practice and key stakeholder groups. Using this guide, a trained qualitative research team member with experience facilitating health-related conversations with patients and health care professionals conducted in-depth interviews over the telephone. Before each interview, a summary of the results of the PRO-TECT trial was shared with the interviewee. Interviews lasted approximately 30 to 60 minutes and were digitally recorded. The interview recordings were transcribed verbatim by a professional transcription service. Interview transcripts were imported into Dedoose, a qualitative research software management tool, to facilitate analysis. The investigative team and CHAI Core qualitative research team members collaborated to develop a codebook based on the research questions and notes taken during data collection. The initial codebook was pilot-tested by independently coding several transcripts, which led to fine-tuning concept definitions and revising decision rules. The final version of the codebook was then applied to the remaining interview transcripts. Consensus coding guidelines were followed. The CHAI Core qualitative research team addressed intercoder reliability heuristically by following an established consensus coding strategy that allows refinement of the codebook and decision-making process throughout the analytic process. Iterative consensus meetings between coders and with input from the study investigators were held to compare coding and interpretation, and any discrepancy was identified and reconciled through discussion and consensus. Subsequently, the researchers generated reports for each code, which were used to identify emergent themes and subthemes, summarize findings, and select illustrative quotes. This heuristic approach provides a comprehensive approach to ensure intercoder reliability as described previously. A coding-based, directed thematic approach was used to analyze the study's results. Emerging themes and discrepancies were captured and reconciled by discussion and consensus, with input from the study investigators. Once the coded transcripts were reconciled, code reports were generated for each code and narrative summaries were written. These code summaries included a narrative description of the themes and subthemes that emerged related to each code and illustrative quotes highlighting each theme. The investigative team and Dr Smith reviewed these findings in the context of each thematic area to derive conclusions about barriers, strategies, and facilitators, as described below. Overall impressions of the trial findings by the key stakeholder interviewees were positive. The majority of interviewees (18/23) stated that they would recommend implementing ePROs in community oncology practices now based on this evidence, but 4 interviewees were not ready to make this recommendation until further evidence of the feasibility of implementation across entire practices could be provided (ie, in this trial, practices could enroll up to 50 patients, and therefore this trial did not provide evidence about feasibility when hundreds or thousands of patients across a practice are enrolled in an ePRO program). Interviewees reported the value of the intervention for earlier identification of chemotherapy side effects and improved patient health status, including physical functioning, symptom control, and QOL. There was enthusiasm for the measured impact on QOL and ED visits, and the magnitude of benefit was thought to be meaningful for patient care. There was no concern reported about the lack of survival benefits as the benefits on patient-centered outcomes were thought to be a sufficient rationale for the value of the intervention regardless of effects on survival. It was reported that it is reassuring that there is not a decrease in survival related to the intervention because hypothetically patients might be treated with less intensive cancer therapy by their oncologists if their symptoms and side effects were better detected with the intervention. Some stakeholders raised concerns about the finding that nurses thought the ePRO intervention was too time- or effort-intensive, with suggestions for future implementations to ensure adequate protected time for nurses to respond to patient ePRO alert notifications. Interviewees were asked for their input on barriers, facilitators, and strategies across the health system level, health care professional level, and patient level, and their input is summarized in Table 9 . Table 9 Barriers, Strategies, and Facilitators to Implementing ePROs in Oncology Clinical Practice, Identified by Patient and Professional Key Stakeholders. When asked whether the benefits of the PRO-TECT trial intervention were worth the patient and staff effort needed to implement the intervention, a range of perspectives was provided. Most clinicians thought the benefits outweighed the effort expended, citing supporting results, although 2 interviewees voiced concerns about the burden that fell on the clinical team, especially nurses, in the absence of protected time for fielding notifications amid competing responsibilities. All interviewees indicated that addressing logistical issues through planning for implementation up front would enable benefits clearly to outweigh the burden. As a next step, interviewees recommended that providing additional evidence from wide implementation in practices would be valuable to fully understand the feasibility and standard procedures for implementation. Discussion Perspectives from a diverse group of adult patients completing PRO questionnaires digitally in community oncology practices indicate high levels of comprehension, ease of use, perceived meaningfulness, relevance, communication, actionability, clinical utility, improvement of self-efficacy, and value. Perspectives from nurses and oncologists caring for these patients demonstrate favorable perceptions of clinical utility and impact on quality and value of care, with clear feedback that protected time for nurses is necessary for successful future implementations (notably, this trial's budget did not support any nursing effort because it was not known at the time of trial design that this would emerge as a potential barrier to implementation). Perspectives about the trial results from patient advocates and professional stakeholders show substantial enthusiasm about the findings and a recommendation to demonstrate the feasibility of implementation through a future project in which practices enroll many patients into an ePRO symptom monitoring program. The concept of clinical utility in the context of digital health technologies refers to its relevance and usefulness in care delivery. Feedback on the PRO-TECT ePRO software system, and the PRO-CTCAE questions within it, demonstrates that, from patient and health care professional perspectives, these tools have utility in the conduct of routine cancer care. Demonstrating this on a digital health platform is a key step in understanding its role in care and the likelihood of ultimate successful widespread implementation. 33 Several strategies and facilitators were suggested, including clear training and education programs when onboarding an ePRO system in a practice, EHR system integration, and protecting sufficient nurse time for this work. The results of analyses for this aim support the clinical utility and value of integrating ePROs into routine cancer care from the perspectives of patient study participants, nurses, physicians, patient advocates, and professional stakeholders, with helpful recommendations for future implementation efforts. Overall, the findings in this study suggest that the design and functionalities of the PRO-TECT system and PRO-CTCAE questionnaire are perceived as valuable tools to improve delivery of quality cancer care by both patients and clinicians. Aim 3: Identify Barriers, Facilitators, and Strategies Used by Practices to Integrate Patient-Reported Outcomes Into Clinical Workflow In this qualitative aim, interviews were conducted with participating nurses and physicians about their experiences integrating ePROs into daily workflow in order to gain an understanding on the ground of barriers, facilitators, and strategies that could inform future implementation efforts. These interviews are complementary with the interviews of patient advocates and professional stakeholders that were part of aim 2. They differ from the aim 2 interviews in that the interviews for aim 3 were among people who had participated in the PRO-TECT trial and had direct experience with the intervention. Methods A semistructured interview guide was developed in collaboration with the investigative team and the long-standing expert CHAI Core Qualitative Research Facility at the UNC in order to elicit insight and descriptive detail from nurses and physicians about their experiences using and integrating the ePRO system in this trial in clinical workflow (see Interview Guide in Appendix B ). The interview guide included questions about what impact integrating the ePRO system had on clinical workflow on a daily basis and, if implementation was met with barriers or challenges, how practices handled them. Respondents were asked to gauge their and their colleagues' impressions of committing time to integrating the ePRO system and which facilitators and strategies were effective in implementation. Respondents were also asked to discuss the pros and cons of the effort and time it took to integrate the ePRO system vs the benefits the system provided. Using this guide, we conducted in-depth interviews with oncology nurses and physicians (oncologists) at participating practice sites where at least 5 patients had been enrolled in the trial. Practices were asked to provide access to at least 2 nurses and 1 physician at their site who had used the ePRO system for at least 6 months. Interviews were conducted over the telephone or in person by a trained qualitative research team member with experience facilitating health-related conversations with patients and health care professionals. Interviews lasted approximately 15 to 45 minutes and were digitally recorded. The interview recordings were then transcribed verbatim by a professional transcription service. Interview transcripts were imported into Dedoose, a qualitative research software management tool, to facilitate analysis. The investigative team and CHAI Core qualitative research team members collaborated to develop a codebook based on the research questions and notes taken during data collection. We pilot-tested the initial codebook by independently coding several transcripts, which led to fine-tuning concept definitions and revising decision rules. The final version of the codebook was then applied to the remaining interview transcripts. Consensus coding guidelines were followed. The CHAI Core qualitative research team addressed intercoder reliability heuristically by following an established consensus coding strategy that allows refinement of the codebook and decision-making process throughout the analytic process. 34 Initial codebooks were developed for each stakeholder group (eg, patient, oncologist, nurse, clinical research associate) and were piloted by 4 experienced, master's-level and PhD-level CHAI Core researchers trained in qualitative methods who independently coded interview transcripts. Iterative consensus meetings between coders and with input from the study investigators were held to compare coding and interpretation, and any discrepancy was identified and reconciled through discussion and consensus. This process continued until replicability of coding occurred across coding teams, at which point the codes, definitions, and coding decisions were finalized. The final version of the codebooks were then applied (by 1 researcher per transcript 34 ) to the remaining transcripts. Subsequently, the researchers generated reports for each code, which were used to identify emergent themes and subthemes, summarize findings, and select illustrative quotes. This heuristic approach provided a comprehensive approach to ensure intercoder reliability as described previously. A coding-based, directed thematic approach was used to analyze the study's results. 34 Emerging themes and discrepancies were captured and reconciled by discussion and consensus, with input from the study investigators. Once the coded transcripts were reconciled, code reports were generated for each code, and narrative summaries were written. These code summaries included a narrative description of the themes and subthemes that emerged related to each code and illustrative quotes highlighting each theme. The investigative team reviewed these findings in the context of each thematic area to derive conclusions about barriers, strategies, and facilitators, as described below. Results Between August 20, 2018, and October 26, 2020, interviews were conducted with 82 oncology care professionals (51 nurses and 31 physicians [oncologists]) across 25 of the 26 participating PRO-TECT trial intervention sites (1 site only accrued 2 patients on study, so it was excluded due to limited experience with PROs). In general, the findings echoed the results of the survey in aim 2, namely that satisfaction with remote symptom monitoring and with the ePRO approach used in this trial overall was high. Most health care professionals thought the essential components and functionality of the system worked well and were useful and meaningful. Two main themes emerged from analysis of the interview coding regarding ePRO use: positive impact on the quality of cancer care delivery and barriers to integration with clinical workflow. Subthemes included communication with the care team, increased awareness of symptoms, and increased ability for patients to self-manage symptoms. These are described below with illustrative quotations. In addition, for further details and more supporting quotations, the full, highly detailed qualitative report is provided in the appendices. Theme 1: Impact on the Quality of Cancer Care Delivery Health care professionals expressed that ePROs allowed generation of clinically relevant patient symptom information in near real-time and on a regular basis. Example quotation: I think I was—from a clinician, I was able to better address symptoms with patients. I was able to objectively follow their longitudinal symptoms. I think their quality of life was better. I don't know if I prevented any hospitalizations and such. I think I possibly did, with some people having reporting [sic] diarrhea. We got them in for fluids and labs and so on. Or feeling short of breath, and we addressed that right away. —(oncologist) The electronic platform was thought to allow for receiving important symptom information and was a strong channel for patient-oncology care team communication and relationship building. Direct communication with the care team was a notable benefit. I would say, for most part, it did improve or made the life of the physician better. Improved communication between the patient as well as the physician. I think, this would be probably the wave of the future where, instead of the interaction between the physician and the patient occurring only at the times when the visits are occurring, this gives them a tool to communicate to the practice without using a phone or by any electronic medium, kind of like how the cardiologist [sic] have for their pace-maker program, and where the pulmonologist [sic] have for their CPAP program for Sleep Apnea. Their oncologist might have similar kind of a program where the symptoms are constantly monitored, and if this were to beep, then maybe in the [EHRs] of the future, it would actually tag us saying that if this so-and-so entered in an alarming symptom profile, we need to call them, rather than our patients trying to catch hold of us, we are now being alerted in advance, so to speak. —(oncologist) It was noted that patients expressed appreciation for follow-up calls about their symptoms from the cancer care team. In several instances, examples of how the ePRO influenced or guided clinical care (eg, providing a new medication, test, or other intervention) were given: We can get folks in and give them fluids and whatever we need to do sooner…. Sometimes a liter or two of fluids and electrolytes can keep people home for longer. —(oncologist) We had one patient who was on an aromatase inhibitor, and she kept having issues with insomnia. I talked to [the doctor], and [they] had me tell her how to switch the time of day that she was taking it, and I haven't really had to call her since. —(nurse) In addition, health care professionals' awareness of symptoms was affected by ePRO use. Some health care professionals thought ePRO use lowered barriers to reporting between visits and lowered underreporting of symptoms, particularly for patients who did not use other forms of digital communication: Having this program that kinda checks on them every week is really helpful for the patient population where they're not so active on their patient portal. —(oncologist) Health care professionals reported they were better able to monitor and help patients manage their symptoms using ePROs, particularly in the context of other testing or result that may overshadow management of symptoms. This led to increased satisfaction for some health care professionals in their care delivery and generated a more complete picture of the patients' health. Also, it improved my satisfaction because I felt that I was able to keep track of their symptoms. … It's helped me keep track of symptoms that not necessarily—there's so much data with the oncology patients with their symptoms, their blood tests, and imaging studies and so on, so this is a nice way to characterize their symptoms, which we don't always do. —(oncologist) Health care professionals also thought that awareness of symptoms may have helped some patients be more self-aware and self-manage recurrent problems more effectively: She learned the bowel regimen that worked for her, whereas in the past I think that she would be prescribed something. She would take it. It would cause diarrhea, so she'd stop it. Then she'd get constipated. She would just swing violently back and forth between constipation and diarrhea. —(nurse) In summary, the ePRO monitoring system tested in this trial was seen by most health care professionals as a valuable tool to aid them in their care of an ill and highly symptomatic population with metastatic cancer. Oncology professionals expressed that they care about their patients and want to know when their patients are in distress. The software overall was received well, and the training and ongoing technical and implementation support in the trial were seen as appropriate and useful. Theme 2: Barriers to ePRO Use Health care professionals thought that the ePRO system should be integrated into the EHR system. In this trial, the ePRO system could not practically be integrated into EHR systems across many different practices and systems. Health care professionals clearly thought this was a shortcoming that led to challenges accessing information and integrating that information with their usual informatics workflow, which occurs largely in the EHRs. Other than the integration into the [EHR] I think is the only—that's challenging in and of itself. I'd love to see it integrated into my workflow. —(nurse) Many nurses thought that they needed a protected effort to respond to alerts. Piling this additional work on top of existing duties was described as a dissatisfier and potential barrier to other work. A lot. A lot. If all patients were using this system, it would take—we have quite a few patients that we see at this office. If every single one did a weekly report, that's a lot of things to take care of. I feel like, for our staffing here, we're not a huge facility, so we don't quite have the staff to do that for all patients. —(nurse) It was thought that efforts to reduce the number of alerts should be undertaken. Also, asking patients for additional contextual details for severe or worsening symptoms was seen as potentially useful by nurses as a functionality for future system iterations. I would worry about that as far as—especially if some of the questions would trigger an alert when it really maybe didn't mean to. That would add to workflow as far emails go. … If every person had—a lot of emails were coming to us directly as far as alerts go, that would be overwhelming. I would be afraid that that would be overwhelming. —(nurse) Discussion Interviews with 82 nurses and physicians who used the ePRO system in this trial across 25 states provided rich insights into the positive elements and barriers related to integrating PROs in clinical oncology workflow. Overall, interview findings suggest that the intervention was well received by patients, health care professionals, and staff in terms of feasibility and acceptability and was generally perceived to improve communication and symptom awareness. That said, there were concerns that the intervention can potentially create more work or complicate workflow processes. Respondents postulated that implementation challenges might be mitigated through EHR integration in the future. Based on the interviews, facilitators and strategies for future system design and implementation were identified. Facilitators that were identified included the following: Identification of a nurse champion from the start to oversee and assist with local implementation and education Clear procedures for communication of symptom information tailored to the specific workflow approaches at a given practice Offering access to the ePRO symptom monitoring system for all patients receiving cancer treatment, not just selected patients with particular eligibility criteria (eg, based on type of cancer, stage, or treatment type) Adding the financial toxicity information collected from patients into the longitudinal reports Buying out time for nurses who address alerts so they can prioritize reviewing and responding to alerts and reports Strategies that were identified include the following: Ongoing engagement with the local clinical team by the central implementation team to monitor uptake and compliance and address technical and logistical challenges as they arise Assurance of integration with local information technology systems, with a flexible approach to meet health care professionals where they are These facilitators and strategies are being incorporated into ongoing program development efforts toward future implementations. In addition, many small technical suggestions from these interviews to improve the ePRO system are already being integrated into the design and functionality of the system. Of particular interest in this trial was feedback at all levels in aims 2 and 3 regarding the importance of early engagement of patients, health care professionals, staff, and leadership in practices when implementing an ePRO system, including providing education on the value of ePROs, facilitating leadership buy-in and supportive messaging, and using a quality improvement approach for rollout and sustainability. Also, in the future, the importance of limiting alert notifications to those that are clinically actionable and emergent was expressed as was the need for informatics approaches that direct the right information to the right personnel at the right times. Ensuring that new staff receive training was stressed because of the frequent turnover of care delivery personnel. Finally, developing ePRO systems that enable customization at the practice, clinic, and individual health care professional levels was thought to be potentially valuable for future systems. There were limitations to the qualitative analyses. Although surveys and interviews were conducted with a large number of participants, there still may have been some self-selectivity in participation that could bias findings. The areas of connected health technology and telehealth are rapidly evolving, and not all lessons learned may continuously apply over time. Future analyses can focus even more on eliciting input from patients in underserved communities or with lower educational attainment, lower health literacy, a rural location, or lower economic status. More broadly, these findings support the value of a mixed-method approach to understanding the implementation of novel care enhancements. A combination of quantitative data on the use of the intervention; surveys from patients, staff, and health care professionals; and semistructured interviews of these users plus external stakeholders provides a rich picture of the user experience, perceptions of value, and future directions. This approach can serve as an exemplar for future studies of care enhancement interventional research. A success in this trial was the elicitation of wide input from a large number of stakeholders. Discussion Dissemination and Impact of Results The findings from the PRO-TECT trial have been disseminated through multiple high-impact channels (including 2 plenary sessions at international meetings and a JAMA publication 35 ), and have already had substantial impact in the field of symptom monitoring and value-based care in oncology and been circulated back to participating practices and patients and to patient advocacy organizations. PRO-TECT findings are already central in a published international guideline from the European Society for Medical Oncology 11 and supported the evidentiary rationale for inclusion of PROs in the new cancer care value-based payment model from the CMS. 9 Dissemination has occurred via presentations at national and international conferences; in small meetings with representatives of the NCI, CMS, and FDA; in publications; and through direct communications with participating practices and patients. Results of analyses related to patient engagement with technology have been presented in the plenary session of the International Society for Quality of Life Research, 36 and results of the secondary QOL findings have been presented in the online plenary series of the ASCO. 37 An oral presentation of the overall survival results was presented at the annual meeting for the European Society for Medical Oncology in October 2023. In addition, the findings have been reported multiple times throughout the trial in various committees of the NCI cooperative group, the Alliance, specifically in the Patient Advocate Committee, Health Outcomes Committee, Oncology Nursing Committee, and Community Oncology (Physician) Committee. The findings have been presented to representatives of the NCI and FDA through the NCI Moonshot initiative U01 consortium, focused on the PRO-CTCAE. Results have been presented to the CMS in discussions of the planning of the Enhanced Oncology Model, the cancer care payment model that is intended to include ePRO symptom monitoring as a component, partially based on these findings and prior foundational work by the authors leading up to this trial. These high-visibility publications and implementation channels (ie, high-tier journals, presentations, payment models, and international guidelines) speak to the substantial impact these findings have already had and their continued potential. Specifically, these findings provide evidence supporting the wide rollout of ePROs for symptom monitoring as a component of high-quality cancer care programs and medical home models that are increasingly proliferated by payers and health systems. The rapid integration of these results in emerging payment model rationales and guidelines speaks to their salience at this moment, when there is high interest in care models that engage patients, identify patients at risk of complications, and improve coordination for complex chronic illnesses. Moreover, ePROs complement other care enhancements of growing interest in oncology care support services, including patient navigation, advanced care planning, prospective outreach to at-risk patients, and telemedicine. Based on this early high interest, we anticipate that results of the PRO-TECT trial will continue to be disseminated through derivative channels and inform the expanding implementation of ePROs in oncology practices and inclusion of ePROs in additional payment models from private payers domestically and from health authorities internationally. Dissemination of findings to patient partners has been extensive, including quarterly scheduled stakeholder meetings to review all elements of findings, biannual presentations to the Alliance Patient Advocate Committee (as well as the Alliance Nursing Committee and Alliance Community Oncology Committee), patient partner substantive participation in the preparation of all manuscripts and meeting presentations, and preparation of public webinars with patient partners in which results are described. Patient partners have been deeply involved in every step of this project, with deep awareness and understanding of its processes, findings, and dissemination approaches. Results were sent to all participating practices, and the practices disseminated these results directly to participating patients. Specifically, a letter was composed to patients in plain language summarizing the results, which were attached to key publications, and these materials were directed to contact personnel at each practice to share with patients. There were no challenges to this process, although we did not systematically collect information on the success or perception of this process, which could be a focus of future work. Other dissemination activities are planned. For example, findings will be disseminated via patient investigators through their various affiliated patient advocacy organizations—including Patients & Partners, Gemini Group, and Research Advocacy Network—to circulate results to their LISTSERVs and membership. Summary of Results PRO-TECT is the largest controlled clinical trial that has been conducted to assess the use of ePROs for remote symptom monitoring in oncology, and it was designed to address rising interest among payers and health systems in remote symptom monitoring as an approach to improving outcomes and quality of cancer care delivery. 8-10 More than 50 practices across half the states in the United States were involved, an approach intended to provide a real-world view on how practices integrate ePRO monitoring into existing processes. The effectiveness analyses in aim 1 did not find a statistically significant benefit of ePRO symptom monitoring on overall survival but did find significant benefits on the patient-centered outcomes of physical functioning, HRQOL, and symptom burden as well as a reduction in ED and hospital admissions. Physical function was initially a primary outcome in the trial at the time of design and initiation, then was shifted to a key secondary outcome to optimize statistical power. These findings demonstrate that ePRO symptom monitoring improves the patient experience and clinical status, as well as health care use, during cancer treatment. The implementation outcomes in aims 2 and 3 focused on feasibility, barriers, strategies, and facilitators. Patients expressed high levels of acceptance and satisfaction with ePRO use, with most wanting to continue using the intervention, stating that the intervention increased their engagement with their own care and with their care teams and that they would recommend it to other patients. Nurses and physicians similarly thought that the system improved engagement and communication, but they noted that dedicated time for nurses will be necessary in future implementations to ensure that the work of responding to symptom notifications can be done without overburdening nurse workload. Nurses and physicians also noted that in the future, ideally any ePRO system would be integrated into the practice's EHR system to enable seamless flow of ePRO data within clinical processes. Potential approaches for integrating ePROs into EHRs include automated identification or enrollment of eligible patients, triggering surveys through a patient portal, storing ePRO data in the EHR system, visualizing individual or aggregated ePRO results along with other clinical data in tabular or graphic format (ie, similar to laboratory values), flagging abnormal results, triggering alert notifications within individual or shared health care professional in-baskets, monitoring compliance with ePRO reporting at the individual or aggregate level, importing ePRO results into clinical notes by using smart phrases, tracking nursing responses to alert notifications, and aggregating symptom values to support quality care programs (eg, pain management initiatives). External patient advocates and professional stakeholders who were shown the results of the trial expressed high levels of enthusiasm for the findings, noting that this effectiveness evidence supports wider implementation of ePROs in oncology practice. However, they also thought that additional evidence related to implementation was needed, specifically evidence from implementation in practices that enroll larger numbers of patients (more patients per practice rather than the limit of 50 per practice, as in this trial). They thought that although this trial provided sufficient evidence about the optimal implementation approach, demonstrating that this approach can be implemented widely across practices is a next step. Like the nurses and physicians at participating practices, the external stakeholders also noted that integration of the ePRO system into the EHR system of practices will be important in any future implementation evaluation. Potential to Affect Health Care Decision-Making This trial provides the necessary evidentiary groundwork for ePROs to be added to value-based cancer care delivery models and clinical practice guidelines in oncology and to be integrated into practices. The ePRO intervention was demonstrated in this trial to drive health care professionals' actions that address patients' symptoms and functional impairments. Several elements of care decision-making can be positively affected by this intervention. At its heart, this is a communication intervention. Symptom monitoring is not in itself an innovation; in oncology and broadly in care delivery, patients are asked to report their symptoms at multiple touchpoints as a part of routine care. However, because there are inefficiencies in existing mechanisms for symptom monitoring, many symptoms are missed, so there is an opportunity for enhanced communication approaches via connected health technologies to improve symptom detection, as elucidated in this trial. Interventions such as the one in this trial therefore can inform decisions by health care professionals related to symptom management. They can also facilitate the reorientation of goals of care more toward QOL. More broadly, aggregated symptom data across clinic populations can enable quality initiatives around symptom control (eg, if an excess of poorly controlled pain or other symptoms are flagged by the ePRO monitoring program). Lessons Learned In terms of trial design, a lesson learned is that a clustered randomization approach would not be used again by the investigators for a similar study in the future. Practices were aware of their study arm allocations and may have enrolled differentially despite the directive to enroll consecutively screened patients. Patients in the control arm appear to have been healthier, with a higher proportion of patients who were more recently diagnosed with fewer prior lines of cancer treatment and greater access to palliative supportive care services, which are known to improve outcomes ( Table 6 ). The control arm sites also enrolled patients faster and completed accrual several months before the intervention arm. As a result, intervention arm patients spent more total person-time of follow-up within the time frame of the COVID-19 pandemic. In our analyses, we controlled or stratified for these variables, and the trial results were unchanged, which is reassuring. However, it may not be possible to fully account for such variables in analyses, which may have contributed to the negative survival finding, because these factors both favor outcomes in the control arm. For a future trial, patient-level random assignment would be preferred for such an intervention where selectivity of enrollees can play a part. There may be a risk of health care professional behavioral change affecting the care of control patients favorably when patient-level random assignment is used, but this may be worth the tradeoff. In terms of trial conduct, the value of a strong central research administrative team to monitor practice site procedures and compliance in real time, with frequent communication to sites, was found to be highly beneficial for study procedure compliance. In terms of practice procedures, a lesson learned is that the intervention adds effort for nurses to field alert notifications generated by the ePRO system about concerning patient symptoms. Nurses need to review the alert notifications, then reach out to patients when indicated. Survey and interview feedback was received that nurses need a small amount of protected time for these activities, so future implementation efforts will need to consider this. In terms of outcomes, an important lesson was that survival data at community practices are inconsistently reliable. Unlike hospital-based or academic practices, community practices often do not participate in state registries, and they do not follow their patients for dates of death if the patients die at home or in hospice or document dates of death in the EHR. In this trial, data directly from practices on dates of death therefore could not be considered acceptable for the primary outcome analysis of overall survival; dates of death needed to be obtained from the Centers for Disease Control and Prevention National Death Index. Requesting these data from the National Death Index required sufficient patient identifying information, including partial Social Security numbers (last 4 digits) for patient participants, which had to be cleared by local IRBs and collected, and which introduced a modest delay in analysis. In future studies involving community practices, partial Social Security numbers will be included in consent forms up front. In terms of study population, a purposive enrollment approach was used in this trial to try to optimize the proportion of enrolled patients who identified as Black or African American, Asian, or Hispanic or Latin. We collected baseline information about the proportion of patients at sites based on race and ethnicity. The central research team was in constant communication with research personnel at sites, encouraging them to approach patients in these groups, and when a site accrued about half their estimated allotment, they were instructed for a period of several months only to enroll patents in these groups at least up to demographic proportions at their sites. This practice was discontinued when the timeline of the trial became threatened because this approach slowed accrual rates. Nonetheless, this approach was successful for the accrual of Black or African American patients, with almost 17% representation in this trial, which is well above that of most oncology trials. 38 However, this approach was less successful for accrual of Asian patients (less than 2%) and Hispanic or Latin patients (about 5%). Therefore, future efforts will need to include additional strategies, such as purposive inclusion of sites with more representative patient populations; up-front training of site personnel on inclusive accrual; and partnership with local community outreach offices and patient, family, and caregiver centers. 39 In addition, our attempts to have a more representative sample were hampered by the timeline imposed by the funding mechanism. Because we had to complete the trial on timeline, we ultimately had to abandon our purposive accrual approach because it takes much more time to enroll a diverse population at most sites; it requires working closely with local staff, identifying patients, and continued reinforcement. This is an inherently slower process that may not always be feasible on compressed timelines. Therefore, future research may benefit from building a substantial buffer into the timeline if diverse accrual is a priority. In terms of implementation and dissemination, a limitation is that each practice could enroll no more than 50 patients. Although this allowed generalizability because of the large number of participating practices, it limited the ability to demonstrate the feasibility of implementation across practices including more patients. A future demonstration of implementation in larger numbers of patients within practices would provide this evidence. In summary, conduct of this trial demonstrated the challenges of testing care redesign interventions across multiple sites where there is limited operational control on the ground, and the perils of cluster randomization for these kinds of interventions that may lead to unevenness between sites (eg, pace of accrual, approach to patients). In retrospect, perhaps we should not have changed the primary end point of the trial. Indeed, the more proximate end points to the intervention of HRQOL or symptom control may have better been retained, but this is all through the lens of hindsight. The primary end point was changed because it is very difficult to convince the medical establishment of the value of patient-centered interventions based solely on patient-centered end points. Overall survival is the currency of oncology research because of the focus on drug trials, and the prior single-center trial on which this project was based did show a survival benefit. That said, during the time period of this trial, the culture in oncology and more broadly in care delivery has continued to evolve and become more open to QOL end points (probably related to a rise in consumer-connected health innovations and a more patient-centered focus at the FDA). If we were to redesign the trial today, about a decade after the initial trial was initially designed, we would probably conduct a patient-level RCT in a smaller number of sites with a patient-centered outcome such as symptom control or physical function. Generalizability This trial was designed to be highly generalizable to patients receiving cancer treatment for advanced or metastatic disease. Enrollment was open to patients with almost every cancer type, receiving any systemic treatment. The trial was conducted in community practices, where most patients in the United States receive treatment. The results are expected to be viewed as broadly applicable to patients receiving cancer treatment. Study Limitations As noted above in the discussion of aim 1, although this trial showed improvements in its prespecified secondary patient-centered outcomes, overall survival was not affected. This finding is inconsistent with the single-center trial on which this trial was based, which found benefits in both QOL and survival. 14 , 17 There are several potential reasons for this discrepancy, which are noted in the discussion of aim 1, including conduct across many practices with diverse procedures for symptom management; conduct in community practices; and conduct partially during the COVID-19 pandemic, when site personnel were pulled away from care coordination roles on which this intervention relied. Additionally, the results may have been affected by baseline imbalances between arms because of the cluster-randomized design: Intervention arm patients overall were receiving later lines of therapy (third and fourth line) compared with control arm patients, suggesting that intervention patients may have been more affected by the cumulative adverse effects of multiple lines of prior treatment, and they might have been receiving less efficacious therapies during trial participation. Additionally, intervention arm patients had less access to palliative care support, which could have adversely affected their outcomes; palliative care has been shown to lengthen survival in patients with advanced cancers. Nonetheless, the impact seen in this trial on patient-centered outcomes provides compelling evidence for clinically meaningful benefits of ePRO monitoring for patients receiving systemic cancer treatment. Another limitation is in the aim 2 stakeholder interviews. Although these were helpful for eliciting impressions from knowledgeable people when shown the results of the trial, this was not a broad qualitative evaluation across stakeholders with less prior knowledge of ePROs, which could be conducted in the future. As noted above, the ePRO software in this trial was not interfaced with EHR systems at practices. This is because at the time of design of the trial, no feasible technologies were available to interface the software with EHR systems across practices. Feedback from participating nurses and physicians and external stakeholders emphasized the importance of future integration with EHR systems to lessen the load on health care professionals of accessing multiple information systems for patient care. In general, we allowed practices to have a flexible workflow for integrating the intervention within their clinical processes because we recognized that there are many different approaches and personnel deployments for symptom management. We aimed to layer the intervention on top of established local workflows. That said, we did specify that the alert notifications should be reviewed and addressed by clinical nurses. It is possible that nonclinical personnel or artificial intelligence applications might be able to serve as frontline reviewers of notifications with an escalation algorithm, and there are models for nonclinical lay patient navigators that have been tested for care coordination. However, nonclinical personnel or current artificial intelligence capacity may be limited in their ability to address the highly clinical nature of symptom management. This question was beyond the scope of this trial, and future studies could assess the use of lay navigators or artificial intelligence as a part of PRO symptom monitoring interventions. Future Research The following are areas for future research to extend the findings of this trial toward enabling broad acceptance and dissemination of this intervention: Implementation evaluation. As noted above, a systematic, prospective evaluation of implementation is warranted, in which a larger number of patients per practice is enrolled. Because only 50 patients were accrued per practice site in this trial, an evaluation in which practices are able to enroll more patients will provide necessary evidence related to implementation feasibility and lay a path for wider uptake. Toward such an evaluation, the PRO-TECT trial findings provide a clear approach for implementing the intervention in terms of workflow, personnel roles, and informatics. Optimization of alert notifications algorithms. As described in the results of aim 1, in this trial alert notifications were triggered to nurses by more than a third of patient ePRO surveys, whereas only about half of these were thought by nurses to warrant immediate outreach to patients. Therefore, there is an opportunity to optimize alert notifications, either by determining up front whether some of these alerts need not be triggered or whether additional information could be elicited from patients to help nurses decide when action is warranted. Feedback from nurses in the PRO-TECT trial revealed that nurses feel burdened by the alert notifications, yet they also believe the information is valuable and desirable. Machine learning techniques could be used to evaluate whether patient-level variables related to cancer, treatment, specific symptoms, or other characteristics might predict whether an alert is deemed actionable by nurses, which could delineate which alert notifications are clinically helpful or not. A dedicated prospective study to optimize alert notifications by testing various approaches could help identify approaches to reduce the burden of notification on nurses so that nurses can focus their time on the most concerning issues for patients. Financial toxicity ePRO screening. Partway through this trial, a screening question for financial burden was added to the patient ePRO survey. This was outside the scope of the funded aims and was added in response to growing interest in assisting patients with financial navigation during cancer care. Preliminary analysis suggests that adding this screening question was associated with reduced financial toxicity experienced by patients. A dedicated study to assess the benefits of ePRO financial burden monitoring, with an assessment of how practices use this screening information, could be highly impactful in oncology, where financial toxicity is common. Conclusions Symptom monitoring with ePROs during treatment for metastatic cancer did not affect overall survival in the PRO-TECT trial and did confer statistically significant or clinically meaningful benefits on the patient-centered outcomes of physical function, HRQOL, and physical function while also reducing ED visits and hospitalizations. The effects of the intervention may have been reduced due to its conduct partially during the COVID-19 pandemic; the cluster-randomized design, which may have introduced systematic differences between arms; and implementation across more than 50 practices in 25 US states, with diverse approaches to symptom management and care coordination. As noted by external patient and professional stakeholders who reviewed the results, the finding of clinical and utilization benefits despite these limitations provides compelling evidence in a real-world setting. Feedback from participating patients, nurses, administrative staff, and physicians indicates high levels of acceptance, satisfaction, and endorsement of the intervention, with encouragement to integrate the ePRO software with EHR systems in the future and reluctance expressed by some nurses about future feasibility of the intervention unless time can be protected for them to address patient-reported alert notifications. Feedback from professional stakeholders indicates a need for future evaluation of wider implementation in practices, including a larger number of patients per practice. Results from this trial have been published, 34 have been used as the basis for recommendations in a major international clinical practice guideline, 11 have supported the rationale for integrating ePROs in a new US payment model for oncology from the CMS, 9 and are expected to be more impactful with continued dissemination. References 1. Cleeland CS, Zhao F, Chang VT, et al. The symptom burden of cancer: evidence for a core set of cancer-related and treatment-related symptoms from the Eastern Cooperative Oncology Group Symptom Outcomes and Practice Patterns study. Cancer. 2013;119(24):4333-4340. doi:10.1002/cncr.28376 [ PMC free article : PMC3860266 ] [ PubMed : 24114037 ] [ CrossRef ] 2. Reilly CM, Bruner DW, Mitchell SA, et al. A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment. Support Care Cancer. 2013;21(6):1525-1550. doi:10.1007/s00520-012-1688-0 [ PMC free article : PMC4299699 ] [ PubMed : 23314601 ] [ CrossRef ] 3. Panattoni L, Fedorenko C, Greenwood-Hickman MA, et al. Characterizing potentially preventable cancer- and chronic disease-related emergency department use in the year after treatment initiation: a regional study. J Oncol Pract. 2018;14(3):e176-e185. doi:10.1200/JOP.2017.028191 [ PubMed : 29452549 ] [ CrossRef ] 4. Henry DH, Viswanathan HN, Elkin EP, Traina S, Wade S, Cella D. Symptoms and treatment burden associated with cancer treatment: results from a cross-sectional national survey in the U.S. Support Care Cancer. 2008;16(7):791-801. doi:10.1007/s00520-007-0380-2 [ PubMed : 18204940 ] [ CrossRef ] 5. Di Maio M, Gallo C, Leighl NB, et al. Symptomatic toxicities experienced during anticancer treatment: agreement between patient and physician reporting in three randomized trials. J Clin Oncol. 2015;33(8):910-915. doi:10.1200/JCO.2014.57.9334 [ PubMed : 25624439 ] [ CrossRef ] 6. Laugsand EA, Sprangers MAG, Bjordal K, et al. Health care providers underestimate symptom intensities of cancer patients: a multicenter European study. Health Qual Life Outcomes. 2010;8:104. doi:10.1186/1477-7525-8-104 [ PMC free article : PMC2949821 ] [ PubMed : 20858248 ] [ CrossRef ] 7. Basch E, Iasonos A, McDonough T, et al. Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: results of a questionnaire-based study. Lancet Oncol. 2006;7(11):903-909. doi:10.1016/S1470-2045(06)70910-X [ PubMed : 17081915 ] [ CrossRef ] 8. Schmidt T, Valuck T, Perkins B, et al. Improving patient-reported measures in oncology: a payer call to action. J Manag Care Spec Pharm. 2021;27(1):118-126. doi:10.18553/jmcp.2020.20313 [ PMC free article : PMC10394208 ] [ PubMed : 33103618 ] [ CrossRef ] 9. Centers for Medicare & Medicaid Services (CMS). Enhancing oncology model. Accessed October 1, 2022. https://innovation ​.cms ​.gov/innovation-models ​/enhancing-oncology-model 10. Basch E, Barbera L, Kerrigan CL, Velikova G. Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book. 2018;38:122-134. doi:10.1200/EDBK_200383 [ PubMed : 30231381 ] [ CrossRef ] 11. Di Maio M, Basch E, Denis F, et al; ESMO Guidelines Committee. The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO clinical practice guideline. Ann Oncol. 2022;33(9):878-892. doi:10.1016/j.annonc.2022.04.007 [ PubMed : 35462007 ] [ CrossRef ] 12. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21(8):1305-1314. doi:10.1007/s11136-011-0054-x [ PubMed : 22048932 ] [ CrossRef ] 13. Kotronoulas G, Kearney N, Maguire R, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32(14):1480-1501. doi:10.1200/JCO.2013.53.5948 [ PubMed : 24711559 ] [ CrossRef ] 14. Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol. 2016;34:557-565. doi:10.1200/JCO.2015.63.0830 [ PMC free article : PMC4872028 ] [ PubMed : 26644527 ] [ CrossRef ] 15. Barbera L, Sutradhar R, Seow H, et al. Impact of standardized Edmonton Symptom Assessment System use on emergency department visits and hospitalization: results of a population-based retrospective matched cohort analysis. JCO Oncol Pract. 2020;16(9):e958-e965. doi:10.1200/jop.19.00660 [ PubMed : 32463762 ] [ CrossRef ] 16. Mir O, Ferrua M, Fourcade A, et al. Digital remote monitoring plus usual care versus usual care in patients treated with oral anticancer agents: the randomized phase 3 CAPRI trial. Nat Med. 2022;28(6):1224-1231. doi:10.1038/s41591-022-01788-1 [ PubMed : 35469070 ] [ CrossRef ] 17. Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA. 2017;318(2):197-198. doi:10.1001/jama.2017.7156 [ PMC free article : PMC5817466 ] [ PubMed : 28586821 ] [ CrossRef ] 18. Denis F, Basch E, Septans AL, et al. Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA. 2019;321(3):306-307. doi:10.1001/jama.2018.18085 [ PMC free article : PMC6439676 ] [ PubMed : 30667494 ] [ CrossRef ] 19. Barbera L, Sutradhar R, Seow H, et al. The impact of routine Edmonton Symptom Assessment System (ESAS) use on overall survival in cancer patients: results of a population-based retrospective matched cohort analysis. Cancer Med. 2020;9(19):7107-7115. doi:10.1002/cam4.3374 [ PMC free article : PMC7541161 ] [ PubMed : 32794634 ] [ CrossRef ] 20. Patient-Reported Outcomes Core of the University of North Carolina. Accessed July 20, 2021. https://pro ​.unc.edu/ 21. Basch E, Stover AM, Schrag D, et al. Clinical utility and user perceptions of a digital system for electronic patient-reported symptom monitoring during routine cancer care: findings from the PRO-TECT trial. JCO Clin Cancer Inform. 2020;4:947-957. doi:10.1200/CCI.20.00081 [ PMC free article : PMC7768331 ] [ PubMed : 33112661 ] [ CrossRef ] 22. Schoen MW, Basch E, Hudson LL, et al. Software for administering the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events: usability study. JMIR Hum Factors. 2018;5(3):e10070. doi:10.2196/10070 [ PMC free article : PMC6066634 ] [ PubMed : 30012546 ] [ CrossRef ] 23. National Cancer Institute. Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events. Accessed July 20, 2020. https: ​//healthcaredelivery ​.cancer.gov/pro-ctcae 24. Dueck AC, Mendoza TR, Mitchell SA, et al; National Cancer Institute PRO-CTCAE Study Group. Validity and reliability of the US National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JAMA Oncol. 2015;1(8):1051-1059. doi:10.1001/jamaoncol.2015.2639 [ PMC free article : PMC4857599 ] [ PubMed : 26270597 ] [ CrossRef ] 25. Al-Rashdan A, Sutradhar R, Nazeri-Rad N, Yao C, Barbera L. Comparing the ability of physician-reported versus patient-reported performance status to predict survival in a population-based cohort of newly diagnosed cancer patients. Clin Oncol (R Coll Radiol). 2021;33(7):476-482. doi:10.1016/j.clon.2021.01.008 [ PubMed : 33549463 ] [ CrossRef ] 26. de Souza JA, Yap BJ, Wroblewski K, et al. Measuring financial toxicity as a clinically relevant patient-reported outcome: the validation of the COmprehensive Score for financial Toxicity (COST). Cancer. 2017;123(3):476-484. doi:10.1002/cncr.30369 [ PMC free article : PMC5298039 ] [ PubMed : 27716900 ] [ CrossRef ] 27. Aaronson NK, Ahmedzai S, Bergman B, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. 1993;85(5):365-376. doi:10.1093/jnci/85.5.365 [ PubMed : 8433390 ] [ CrossRef ] 28. Fayers P, Bottomley A; EORTC Quality of Life Group; Quality of Life Unit. Quality of life research within the EORTC-the EORTC QLQ-C30. European Organisation for Research and Treatment of Cancer. Eur J Cancer. 2002;38(suppl 4):S125-133. doi:10.1016/s0959-8049(01)00448-8 [ PubMed : 11858978 ] [ CrossRef ] 29. Gundy CM, Fayers PM, Groenvold M, et al. Comparing higher order models for the EORTC QLQ-C30. Qual Life Res. 2012;21(9):1607-1617. doi:10.1007/s11136-011-0082-6 [ PMC free article : PMC3472059 ] [ PubMed : 22187352 ] [ CrossRef ] 30. Fayers PM, Aaronson NK, Bjordal K, Groenvold M, Curran D, Bottomley A, on behalf of the EORTC Quality of Life Group. The EORTC QLQ-C30 Scoring Manual. 3rd ed. European Organisation for Research and Treatment of Cancer; 2001. 31. Osoba D, Rodrigues G, Myles J, Zee B, Pater J. Interpreting the significance of changes in health-related quality-of-life scores. J Clin Oncol. 1998;16(1):139-144. doi:10.1200/JCO.1998.16.1.139 [ PubMed : 9440735 ] [ CrossRef ] 32. Coens C, Pe M, Dueck AC, et al; Setting International Standards in Analyzing Patient-Reported Outcomes and Quality of Life Endpoints Data Consortium. International standards for the analysis of quality-of-life and patient-reported outcome endpoints in cancer randomised controlled trials: recommendations of the SISAQOL Consortium. Lancet Oncol. 2020;21(2):e83-e96. doi:10.1016/S1470-2045(19)30790-9 [ PubMed : 32007209 ] [ CrossRef ] 33. Stover AM, Haverman L, van Oers H, Greenhalgh J, Potter CM, on behalf of the ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Workgroup. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res. 2020;30(11):3015-3033. doi:10.1007/s11136-020-02564-9 [ PMC free article : PMC8528754 ] [ PubMed : 32651805 ] [ CrossRef ] 34. Morse JM. “Perfectly healthy, but dead”: the myth of inter-rater reliability. Qual Health Res. 1997;7(4):445-447. doi:10.1177/104973239700700401 [ CrossRef ] 35. Basch E, Schrag D, Henson S, et al. Effect of electronic symptom monitoring on patient-reported outcomes among patients with metastatic cancer: a randomized clinical trial. JAMA. 2022;327(24):2413-2422. doi:10.1001/jama.2022.9265 [ PMC free article : PMC9168923 ] [ PubMed : 35661856 ] [ CrossRef ] 36. Stover A, Dueck AC, Basch E. Demographic and symptom differences in PRO-TECT trial (AFT-39) cancer patients electing to complete weekly home patient-reported outcome measures (PROMs) via an automated phone call vs. email: implications for implementing PROs into routine care. Plenary Session, International Society for Quality of Life Research; October 21, 2019. 37. Basch E, Schrag D, Henson S, et al. ASCO Online Plenary Series: digital symptom monitoring with patient-reported outcomes in community oncology practices: a U.S. national cluster randomized trial. J Clin Oncol. 2021;39(36 suppl):349527-349527. 38. Awidi M, Al Hadidi S. Participation of Black Americans in cancer clinical trials: current challenges and proposed solutions. JCO Oncol Pract. 2021;17(5):265-271. doi:10.1200/OP.21.00001 [ PMC free article : PMC8258017 ] [ PubMed : 33974816 ] [ CrossRef ] 39. Guerra CE, Fleury ME, Byatt LP, Lian T, Pierce L. Strategies to advance equity in cancer clinical trials. Am Soc Clin Oncol Educ Book. 2022;42:1-11. doi:10.1200/EDBK_350565 [ PubMed : 35687825 ] [ CrossRef ] Related Publications Journal Publications •. Basch E, Stover AM, Schrag D, et al. Clinical utility and user perceptions of a digital system for electronic patient-reported symptom monitoring during routine cancer care: findings From the PRO-TECT trial. JCO Clin Cancer Inform. 2020;4:947-957. doi:10.1200/CCI.20.00081 [ PMC free article : PMC7768331 ] [ PubMed : 33112661 ] [ CrossRef ] •. Basch E, Schrag D, Henson S, et al. Effect of electronic symptom monitoring on patient-reported outcomes among patients with metastatic cancer: a randomized clinical trial. JAMA. 2022;327(24):2413-2422. doi:10.1001/jama.2022.9265 [ PMC free article : PMC9168923 ] [ PubMed : 35661856 ] [ CrossRef ] •. Di Maio M, Basch E, Denis F, et al; ESMO Guidelines Committee. The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO clinical practice guideline. Ann Oncol. 2022;33(9):878-892. doi:10.1016/j.annonc.2022.04.007 [ PubMed : 35462007 ] [ CrossRef ] International Conference Presentations •. Stover AM. Demographic and system differences in PRO-TECT trial (AFT-39) cancer patients electing to complete weekly home patient-reported outcomes measures (PROMs) via an automated call vs email: implications for implementing PROs into routine care. Paper presented at: ISOQOL Annual Meeting, Plenary Session, Oral Podium Presentation; October 2019; San Diego, CA. •. Basch E. Digital symptom monitoring with patient-reported outcomes in community oncology practices: a US national cluster randomized trial. Paper presented at: ASCO Plenary Series, Online Oral Presentation; November 16, 2021. •. Basch E. Digital symptom monitoring with patient-reported outcomes in community oncology practices: a US national cluster randomized trial. Paper presented at: ASCO Annual Meeting, Plenary Series Update, Oral Podium Presentation; June 5, 2022; Chicago, IL. •. Blinder VS. A randomized controlled trial of routine financial toxicity screening via electronic patient-reported outcomes (AFT-39). Paper presented at: ASCO Quality Care Symposium, Oral Podium Presentation; September 30, 2022; Chicago, IL. •. Henson S, Stover AM, Jansen J, Basch E, Teal R. NCI's Alliance for Clinical Trials Biannual Meeting. Oncology Nurses Committee, Patient Advocates Committee, Community Oncology Committee, participating sites, Chicago, IL or virtual; May 11, 2017; November 2, 2017; May 9, 2018; November 2, 2018; May 10, 2019, November 8, 2019; May 14, 2020; May 13, 2021. •. Basch E. Remote symptom monitoring with electronic patient-reported outcomes (ePROs) during treatment for metastatic cancer: overall survival results from the PRO-TECT trial (Alliance AFT-39). Paper presented at: European Society for Medical Oncology (ESMO) Annual Meeting; October 20-24, 2023; Madrid, Spain. Acknowledgments The investigators express sincere appreciation to Neeraj Arora, PhD, for his invaluable input, guidance, and support throughout this project. The investigators also acknowledge the patients and caregivers who participated in this trial and the health care professionals and staff across participating practices who gave their time to this work even amid the challenges of delivering cancer care during the pandemic. Data Sharing Plan Data from this trial are available upon request from the trial sponsor, the Alliance for Clinical Trials in Oncology. Investigators may submit a request for data from this trial by completing an Alliance Data Sharing Request Form, available at https://www.allianceforclinicaltrialsinoncology.org/main/cmsfile?cmsPath=/Public/Datasharing/files/Alliance-DataRequestForm-11112020.docx , and then by submitting the competed form to gro.NTCNecnailla@stpecnoc . Once received, the request will be forwarded to the Alliance Statistics and Data Management Center (SDMC) and Cancer Care Delivery Research Committee. The SDMC will confirm the availability of the requested data. Once the SDMC confirms availability and the Cancer Care Delivery Research Committee approves the request, the investigator will be sent a Data Release. Once the Data Release is received from the requesting investigator, the SDMC will be notified that the requested data may be released. Research reported in this report was funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (IHS-1511-33392). Further information available at: https://www.pcori.org/research-results/2016/comparing-effectiveness-electronic-symptom-monitoring-versus-usual-care-improving-survival-among-patients-metastatic-cancer-pro-tect-trial#project_information Appendices Appendix A. Weekly PRO Survey (PRO-TECT) (PDF, 228K) Appendix B. Qualitative Interview Guides for Nurses and Physicians (PDF, 302K) Appendix C. PRO-TECT Statistical Analysis Plan (PDF, 648K) Appendix D. AFT-39 Protocol (PDF, 644K) Table 1. Timeline for Control Sites Only (PDF, 215K) Table 2. Timeline for Intervention Sites Only (PDF, 168K) Appendix E. PRO-TECT Protocol (PDF, 673K) Table 1. Timeline for Control Sites Only (PDF, 271K) Table 2. Timeline for Intervention Sites Only (PDF, 332K) Appendix F. PRO-TECT Provider Interviews (PDF, 1.6M) Table 1.0. ePRO System Benefit: Relevant, Real-Time and Regular Patient Symptom Information (PDF, 670K) Figure 1.0. Reasons for Patients Underreporting or Delaying Reporting Their Symptoms (PDF, 748K) Table 1.1. ePRO System Benefit: Provides Patient-Centered Platform to Relay Symptoms and Reduce Symptom Under-Reporting (PDF, 840K) Table 1.2. ePRO System Benefit: Facilitates Stronger Connection Between Patient and Oncology Care Team (PDF, 663K) Table 1.3. ePRO System Benefit: Supports Patient-Provider Communication and Relationship-Building (PDF, 653K) Table 1.4. Negative Aspects of the ePRO System (PDF, 658K) Table 2.0. Perceptions on How other Team Members Felt about Patient Alert Emails (PDF, 651K) Table 2.1. Aspects of Alert Emails That Led Nurses to Consider them Clinically Meaningful (PDF, 666K) Table 2.2. Times When Alert Emails Were Deemed Clinically Unmeaningful (PDF, 652K) Table 2.3. Challenges Associated with Patient Symptom Alert Emails (PDF, 829K) Table 3.0. Different Approaches Taken to Deliver Patient Report to Providers (PDF, 826K) Table 3.1. Perceptions of How other Team Members Felt about the Patient Symptom Report (PDF, 727K) Table 4.0. Nurse Perceptions of Staff Willingness to Commit Time and Effort to the ePRO System (PDF, 793K) Table 4.1. Preparation Strategies for Integrating the ePRO System into Clinical Workflow (PDF, 652K) Table 4.2. Reasons Benefits Outweigh Effort or Could Outweigh Effort Put into the ePRO System (PDF, 669K) Table 5.0. Communicating Expectations to Staff: Investment, Value, and Patient Selection (PDF, 831K) Table 5.1. Communicating Expectations to Patients about Using the ePRO System (PDF, 816K) Table 5.2. Suggestions for Improving the Patient Symptom Report (PDF, 666K) Table 5.3. Suggestions for Reducing Workflow Challenges and Ways to Track and Document ePRO Information (PDF, 674K) Appendix G. Supplemental Tables (PDF, 281K) Supplemental Table G1. Maximum Likelihood Estimates: Multivariable Cox Regression Analysis of Overall Survival (PDF, 123K) Supplemental Table G2. Solutions to Fixed Effects in Mixed Models of Physical Function, Symptom Control, and Health-Related Quality of Life (PDF, 193K) Original Project Title: Electronic Patient Reporting of Symptoms during Outpatient Cancer Treatment: A US National Randomized Controlled Trial PCORI ID: HIS-1511-33392 ClinicalTrials.gov ID: NCT03249090 Suggested citation: Basch E, Schrag D, Henson S, et al. (2024). Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care in Improving Survival among Patients with Metastatic Cancer—The PRO-TECT Trial . Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/01.2024.IHS.151133392 Disclaimer The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee. Copyright © 2024. The Alliance for Clinical Trials In Oncology Foundation. All Rights Reserved. This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/ Bookshelf ID: NBK621483 PMID: 41915775 DOI: 10.25302/01.2024.IHS.151133392 Share Views PubReader Print View Cite this Page Basch E, Schrag D, Henson S, et al. Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care in Improving Survival among Patients with Metastatic Cancer—The PRO-TECT Trial [Internet]. Washington (DC): Patient-Centered Outcomes Research Institute (PCORI); 2024 Jan. doi: 10.25302/01.2024.IHS.151133392 PDF version of this title (8.1M) In this Page Background Participation of Patients and Other Stakeholders Methods Aim 1: Determine Whether Integrating Electronic Patient-Reported Outcomes in Cancer Care Improves Patient-Centered Outcomes Aim 2: Elicit Perspectives About Benefit Burden Tradeoffs for Integrating Patient-Reported Outcomes Into Clinical Workflow Aim 3: Identify Barriers, Facilitators, and Strategies Used by Practices to Integrate Patient-Reported Outcomes Into Clinical Workflow Discussion Conclusions References Related Publications Acknowledgments Appendices Other titles in this collection PCORI Final Research Reports Related information NLM Catalog Related NLM Catalog Entries PMC PubMed Central citations PubMed Links to PubMed Recent Activity Clear Turn Off Turn On Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care i... Comparing the Effectiveness of Electronic Symptom Monitoring versus Usual Care in Improving Survival among Patients with Metastatic Cancer—The PRO-TECT Trial Your browsing activity is empty. Activity recording is turned off. Turn recording back on See more... Follow NCBI Twitter Facebook LinkedIn GitHub NCBI Insights Blog Connect with NLM Twitter Facebook Youtube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov