Testing New Ways to Measure How Patients Rate Quality of Life

Testing New Ways to Measure How Patients Rate Quality of Life - NCBI Bookshelf An official website of the United States government Here's how you know The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site. The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely. Log in Show account info Close Account Logged in as: username Dashboard Publications Account settings Log out Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation Bookshelf Search database Books All Databases Assembly Biocollections BioProject BioSample Books ClinVar Conserved Domains dbVar Gene Genome GEO DataSets GEO Profiles GTR Identical Protein Groups MedGen MeSH NLM Catalog Nucleotide OMIM PMC Protein Protein Clusters Protein Family Models PubChem BioAssay PubChem Compound PubChem Substance PubMed SNP SRA Structure Taxonomy ToolKit ToolKitAll ToolKitBookgh Search term Search Browse Titles Advanced Help Disclaimer NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health. Testing New Ways to Measure How Patients Rate Quality of Life Bruce D. Rapkin , PhD, Carolyn E. Schwartz , ScD, Iliana Garcia , MPH, Brieyona C. Reaves , MPH, CCRP, Leah Goldstein , LMSW, Jie Zhang , MPH, Tom Atkinson , PhD, Yuelin Li , PhD, Wes Michael , MBA, Alex Sankin , MD, Mark Schoenberg , MD, Marisa Cortese , PhD, FNP-BC, Una Hopkins , DNP, Seth Lerner , MD, and Bernard Bochner , MD. Author Information and Affiliations Authors Bruce D. Rapkin , PhD, 1 Carolyn E. Schwartz , ScD, 2,3 Iliana Garcia , MPH, 1 Brieyona C. Reaves , MPH, CCRP, 1 Leah Goldstein , LMSW, 4 Jie Zhang , MPH, 2 Tom Atkinson , PhD, 5 Yuelin Li , PhD, 5 Wes Michael , MBA, 6 Alex Sankin , MD, 7 Mark Schoenberg , MD, 7 Marisa Cortese , PhD, FNP-BC, 8 Una Hopkins , DNP, 8 Seth Lerner , MD, 8 and Bernard Bochner , MD 4 . Affiliations 1 Department of Epidemiology and Population Health, Division of Community Collaboration & Implementation Science, Albert Einstein College of Medicine, Bronx, New York 2 DeltaQuest Foundation Inc, Concord, Massachusetts 3 Departments of Medicine and Orthopaedic Surgery, Tufts University Medical School, Boston, Massachusetts 4 Department of Urology, Memorial Sloan Kettering Cancer Center, New York, New York 5 Department of Psychiatry and Behavioral Sciences, Memorial Sloan Kettering Cancer Center, New York, New York 6 Rare Patient Voice LLC, Towson, Maryland 7 Department of Urology, Montefiore Medical Center, Bronx, New York 8 White Plains Hospital Center for Cancer Care, White Plains, New York Washington (DC): Patient-Centered Outcomes Research Institute (PCORI) ; 2020 Mar . Copyright and Permissions Copyright © 2020. Albert Einstein College of Medicine Yeshiva University. All Rights Reserved. This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/ Structured Abstract Background: Health-related quality of life (QOL) assessments are intended to capture the patient's perspective on health and well-being. Even if not articulated, differences in the ways people think about their health and well-being are inherent in all patient-reported outcomes measures. The impact of changes in health status and treatment on QOL can be attenuated, amplified, or obscured by these differences. In 2004, we developed the QOL Appraisal Model and related measures to describe differences in individuals' criteria for evaluating their own QOL. Although challenging to administer, first-generation appraisal measures have been successful in accounting for individual differences and intra-individual response shifts in QOL. More user-friendly appraisal measures that can be readily administered alongside standard patient-reported outcomes are needed and would serve to advance patient-centered research and practice. Objectives: To develop practical measures of patients' cognitive appraisal of QOL, our study aims were to (1) leverage existing data to identify important aspects of appraisal; (2) derive concise appraisal measures in conjunction with stakeholders who use QOL assessments for research, practice, or policy evaluation; and (3) test these measures in a sample of patients with bladder cancer. In this report, we summarize previously published reports describing the creation of new appraisal measures and then present detailed findings regarding their performance and utility in a population affected by bladder cancer. Methods: Through reanalysis of earlier data sets, stakeholder input, and 2 online surveys, we developed 2 new measures, the 75-item QOL Appraisal Profile version 2 (QOLAPv2) and the 23-item Brief Appraisal Inventory (BAI). To validate and directly compare these measures, we conducted a study of patients treated for bladder cancer at 3 local centers. Along with appraisal, the study included well-established measures of global and disease-specific QOL, stable antecedents (demographics, social determinants of health), and time-varying catalysts (recent life experiences and present mood state), as dictated by the QOL Appraisal Model. We completed the survey on 2 occasions to examine changes in health and life events. Results: We gathered appraisal and QOL data from 110 patients with bladder cancer by using the QOLAPv2 and the BAI. Examination of appraisal item distributions and zero-order correlations with QOL, antecedents, and catalysts indicated the need for further multivariate analyses. Our analyses examined how these 2 measures were related to each other and how they accounted for individual differences in QOL. We used principal components analyses to derive dimensions of appraisal from QOLAPv2 and BAI. Although we expected these 2 measures to converge, assessment of appraisal proved to be more complicated than anticipated. We found lower-than-expected overlap between these 2 instruments, although the ability to estimate QOLAPv2 components was strengthened when we included second-order interactions among BAI components. Both new appraisal instruments accounted for individual differences related to antecedents as well as response shifts due to catalysts across a range of QOL outcomes. Adding QOLAPv2 to predictive models led to an average 37% increment in variance explained over BAI alone in the R 2 statistic, demonstrating the value of the longer measure. Conclusions: Study findings support the validity and applicability of the QOLAPv2 and the BAI, warranting further use of these instruments by stakeholders concerned with cognitive and motivational differences in patients' responses to illness and treatment. These measures provide significant value to stakeholders by enabling them to systematically recognize and address patients' personal criteria and standards for QOL in research and practice. Limitations: Because of our in-depth focus on a single disease (ie, bladder cancer) and the relatively small sample, this study must be considered a preliminary step in the validation of the QOLAPv2 and BAI. Comparing quantitative appraisal measures with cognitive think-aloud interviews is also warranted. Further research using these measures is needed to understand whether lower-than-expected redundancy between appraisal as measured by the BAI vs the QOLAPv2 indicates inconsistency in what the 2 instruments explain, or whether the BAI captures emergent aspects of appraisal that the QOLAPv2 cannot detect. Background Health-related quality of life (QOL) encompasses an array of patient-reported outcomes (PROs), each reflecting patients' subjective evaluation of an aspect of their health and well-being. Although QOL instruments have been developed using increasingly sophisticated methodologies, almost all are based on standard items. Such evaluative measures, including specific health-related PRO domains, more global QOL, and measures of satisfaction, provide no information about individuals' frames of reference and standards of comparison for answering questions about their health and well-being; neither do they address the salience of items to particular individuals at a particular time. Accurate interpretation of self-reported QOL depends on understanding the actual criteria that patients use to appraise their own situation and how these criteria underlie and shape responses to standard measures. Understanding the criteria that people use to respond to evaluative questions about QOL is fundamental to patient-centeredness. In this report, we discuss efforts to develop and validate instruments able to capture the particular criteria that individuals use when they report on their own QOL. This study builds on 15 years of work based on a model of QOL appraisal that posits parameters for understanding the perspectives and assumptions that people bring to an assessment situation. Our model focuses on individuals' frames of reference (eg, goals they want to achieve, responsibilities they want to let go); ways of recalling experiences (eg, the most recent, the most upsetting); standards of comparison (eg, other patients, one's ideal health); and relative emphasis in reconciling discrepant experiences (eg, positive versus negative, self-focused vs other-focused). We have conducted studies in populations affected by different chronic conditions and injuries using a combined qualitative and quantitative instrument designed to assess each parameter of appraisal. In this earlier research, we have consistently found marked differences in the appraisal parameters that help to explain variability in people's responses to emergent symptoms, surgical complications, and stressful life events. However, the first-generation appraisal measure is complicated to administer and score. The original QOL Appraisal Profile (QOLAP) included 10 different open-ended questions that could take 15 minutes or more for a trained interviewer to collect. Verbatim responses had to be recorded and coded. The qualitative codes needed to describe these responses varied somewhat from population to population, with a minimum of 35 different codes required. Scoring involved data reduction based on the prevalence of different combinations of codes. This level of effort and complexity is simply not practical for most health outcomes studies. This PCORI project has appreciably advanced this area of research through the development of appraisal measures that are easier to administer and score in any research or clinical context where QOL and other related PROs are measured. Aims of this project were as follows: Leverage existing data from earlier studies to identify empirically the aspects of appraisal needed to explain differences in PROs within and across populations Derive concise, new measures of appraisal by examining empirical findings with stakeholders who use QOL assessment for research, assessment of clinical practice, or policy evaluation Test the reliability, validity, and ease of administration of concise appraisal measures in a new sample of patients with bladder cancer This report summarizes published work to accomplish aims 1 and 2. Detailed analysis and discussion of aim 3, reported here for the first time, constitutes the bulk of this report. Challenges to Assessing and Interpreting Patient-Reported Outcomes Our focus on differences in patients' appraisal stems from a critique of studies of QOL and PROs. Most constructs of interest in patient-centered outcomes research are anchored in individuals' subjective evaluation, with no external point of reference. Such constructs include pain and symptoms, emotional well-being, functional status, role performance, adequacy of social support, and satisfaction with treatment as well as overall ratings of health. Over the past several decades, many generic, domain-, and disease-specific QOL instruments have been developed. These assessment tools are almost exclusively “nomothetic,” meaning they are based on the assumption that standard items are directly comparable across individuals and over time. Whether global or disease-specific, nomothetic items and scales convey little information about what different individuals' ratings actually mean. For example, when individuals rate their overall health as “good,” we cannot determine whether this means “good for a person with my diagnosis,” “good in light of my previously diminished expectations,” or “good compared with my ideal health.” We also cannot know whether their ratings refer to specific symptoms, recent news about their treatment, or an overall sense of well-being. Even ratings of specific areas of functioning such as difficulty climbing stairs are ambiguous without information about respondents' particular expectations and standards of comparison regarding their abilities. Further, symptoms and functioning can vary, so individuals' summary ratings regarding a given period of time may reflect their worst experiences, their most recent, or some subjective average. Of course, instructing people to base their responses on circumstances or comparators might be possible, but these constraints may limit the ability to use the measure in different settings or populations. Many different subjective criteria might affect responses to most QOL items. Instructions to constrain even 3 or 4 of these criteria would be onerous and potentially off-putting to respondents. Consider the kind of instructions that would be involved: “Rate how you felt about your physical functioning this week. Please average over your best, worst, and usual days. Only take into account regular activities that you normally do or want to do. Make your ratings of each item by comparing yourself to what you think is typical for most people at your age.” Such complex instructions run the risk of distorting the person's experience of QOL if, for example, he or she tries to maintain a positive focus or makes comparisons with his or her own progress. Investigator-imposed constraints reduce the patient-centeredness of measures by requiring individuals to base their responses on situations and comparisons that may not be personally salient or meaningful. The inherent and unavoidable ambiguities in QOL measurement have led to paradoxical and counterintuitive findings across different populations and health conditions. For instance, people with chronic illnesses often report QOL equal or superior to that for healthier people. 1-7 Discrepancies commonly arise between clinical measures of health and patients' own ratings. 8-10 Well-being measures do not consistently distinguish known patient groups, are often only weakly related to objective criteria, and show little convergence across measurement perspectives. All these inconsistent findings can be explained by underlying differences in the ways that people appraise their health and well-being. Rationale for Assessing Criteria That Individuals Use to Appraise Quality of Life QOL assessment is intended to capture patients' evaluations of their health and well-being. There is no inherently right way for individuals to think about QOL questions, and efforts to constrain how individuals think may distort their answers. Even so, how individuals formulate responses does have considerable implications for outcomes research. Differences in the ways that research participants think about QOL can obscure the impact of changes in health status and the benefits or harms of treatment. The comparative effectiveness of treatments may depend on individuals' subjective criteria for evaluating the outcomes they experience. For example, understanding how a surgical intervention for cancer affects ratings of role functioning depends on the roles that individuals are considering when making their ratings. Outcomes assessment becomes further complicated because individuals can change the criteria and standards they use to rate their health. Such changes, referred to as “response shifts,” often arise because of experiences with illness and treatment. For instance, criteria for rating difficulty in climbing stairs may change as individuals accommodate to advancing illness or anticipate recovery of stamina and mobility. Research typically neglects differences in the criteria that individuals consider when rating their QOL, in effect relegating those differences to “measurement error.” Analyses that compare or predict QOL scale scores among individuals over time or across treatment conditions contain no information about individual differences in the meaning or use of scales. Even QOL instruments developed using sophisticated psychometric techniques such as item response theory (IRT) and computerized adaptive testing (CAT) do not solve this problem. Consider role performance and satisfaction as an example. Although items such as “I enjoy my job,” “I love my job,” and “I could not imagine doing any other work” might follow a consistent item-difficulty gradient, responses to these items provide no information about the criteria individuals use to appraise their role. In this regard, IRT depends largely on semantic relationships among the items. Individuals' answers to this set of items will likely be consistent, regardless of whether their job satisfaction is related to travel perks, being part of a team, or having a laid-back office environment. To be sure, an estimate of a latent job satisfaction score from these items is likely to represent accurately and reliably the ways individuals would respond to similar job satisfaction items at that time. Yet, even with high internal consistency and desirable scaling properties, 2 people who give identical responses to these items do not necessarily experience their jobs in the same way. This is important because the ways in which changes in health status affect job satisfaction depend mainly on the specific criteria that individuals use to evaluate their work roles. Physical disability may interfere with work involving travel more than it does a job at a laid-back office. These criteria are not measured in QOL assessment including item sets developed using IRT. It follows that observed correlations of job satisfaction with observable catalysts like changes in health status will necessarily depend on the extent to which a sample includes people applying particular criteria. If most people in a sample rate their job satisfaction according to opportunities to travel, then negative correlation with physical disability will likely be greater than if the majority are more concerned about having a laid-back job. Research samples generally comprise unknown, heterogeneous mixtures of people applying these and many other criteria to appraise job satisfaction and other evaluative constructs. This heterogeneity leads to relatively weak and inconsistent associations between objective catalysts and evaluative measures such as job satisfaction or QOL because the same health change relates to people's experiences in different ways. Even if linear associations between health change catalysts and QOL are statistically significant, aggregate effects substantially less than R 2 = 1.0 mean that the experiences of many people in a sample are not well described by the linear association. The challenge is finding the right variables to segment heterogeneous samples to determine how similar influences yield discrepant responses and how similar responses may arise for different reasons. This argument is not a critique of QOL instruments themselves. One can reasonably assume that the evaluative ratings people provide on PRO items accurately reflect their feelings and experiences. However, these instruments are not intended to provide insight into the psychological basis for QOL ratings. Additional information is needed about the ways that people arrive at ratings of QOL and related constructs, to better understand how those ratings reflect individual differences and intra-individual changes in health. Theoretical Foundation: The QOL Appraisal Model To account for these interindividual differences and intra-individual changes in the self-evaluation of QOL, Rapkin and Schwartz 11 developed the QOL Appraisal Model. This model posits that any construct involving subjective evaluation involves 4 aspects of appraisal: (1) the goals, priorities, and concerns that comprise individuals' frame of reference 12 ; (2) the ways that individuals sample experiences within their frame of reference that they deem pertinent to QOL 13-17 ; (3) the standards of comparison they consider in evaluating these experiences, including past history, 18 , 19 perceptions of salient others, social norms, 20 , 21 and personal ideals; and (4) the ways that they formulate summary judgments or emphasize the salience of different experiences based on some combinatory algorithm. 22 This 4-parameter appraisal model has proven useful in understanding individual differences and longitudinal changes in health-related QOL. 23 This QOL Appraisal Model was the basis for the development of the QOL Appraisal Profile (QOLAP), designed to describe the criteria that individuals consider when asked to evaluate their QOL. This first-generation instrument was broadly descriptive to explore the nature and influence of QOL appraisal. The QOLAP was designed to be administered after completion of standard QOL instruments, to elicit individuals' reflections on the ways that they answered preceding questions. This instrument includes 2 qualitative sections that assess individuals' definition of QOL and the personal goals that matter most for their current QOL. We developed 2 sets of Likert-scale items to measure the kinds of experiences and standards of comparison that individuals considered in making their ratings. We also used a set of semantic differential questions to understand the kinds of experiences that individuals emphasized in making their ratings. The QOL Appraisal Model, depicted in Figure 1 , established a basic analytic framework for evaluating the role of appraisal in patient-reported outcomes research. The model can be adapted to look at individual differences as well as intra-individual changes in QOL. The dependent variable in Figure 1 represents change in a QOL construct. Standard determinants of change in QOL include “antecedents” such as background demographic, cultural, and socioeconomic status influences. “Catalysts” include health status changes, treatments, and other life events. Catalysts may influence “mechanisms” expected to influence change in QOL, such as social support, self-management, or coping skills. Antecedents, catalysts, and mechanisms constitute a Standard Model for explaining QOL. As noted in Figure 1 , partitioning variance in QOL that can be explained by the Standard Model is useful. Residual variance (bottom panel of the QOL dependent variable box) represents observed change in QOL that is discrepant from change predicted by the Standard Model (top panel of the QOL dependent variable box). Figure 1 Analytic framework for the QOL Appraisal Model. As Figure 1 depicts, additional residual variation in QOL change may be explained by taking cognitive appraisal into account. Appraisal represents the criteria an individual applies to rating QOL. Antecedents, catalysts, and mechanisms can influence processes of QOL appraisal in ways that attenuate or amplify their effects on QOL. Health-related events can markedly change the ways people think about their QOL, and those changes will be reflected in the way they respond to QOL measures. Changes in appraisal may arise because of coping mechanisms, increased understanding of a diagnosis, avoidant thinking, habituation to stressful circumstances, socialization to an illness-related identity, or self-blame owing to the nature of illness. Response shift is an emergent construct in the QOL Appraisal Model, defined as a discrepancy from expected change in QOL that can be associated with a change in appraisal. The model distinguishes direct response shift, when residual QOL change is correlated with change in appraisal as a main effect, from moderated response shift, when change in appraisal modifies or interacts with responses to catalysts or mechanisms. This model of appraisal and response shift has been demonstrated in a wide variety of studies (described in the next section); we applied it to guide all analyses in this project. Briefly, analyses based on the QOL Appraisal Model proceed as follows: QOL change scores are the dependent variable in multiple linear regression (or the equivalent). Analyses employ a hierarchical order of entry. After controlling for the QOL pre-score, exogenous antecedent variables are entered followed by changes in health state, life events, and other catalysts. Variables describing mechanistic influences on QOL, such as social support or treatment indicators, are distinct from appraisal,. The model is trimmed at each step to reduce the number of predictors in the equation. This order of entry accounts for variance in QOL change that the Standard Model can explain. One would examine effects related to appraisal by testing the predictive value of baseline measures of appraisal, change in appraisal, and interactions between Standard Model predictors and appraisal measures. Response shift is indicated by a change in appraisal that explains residual variance in QOL, either directly or in interaction with catalysts, antecedents, or mechanisms. Total effects attributable to appraisal (without considering Standard Model effects) are also examined, to determine overlap between variance explained by appraisal alone and variance explained by the Standard Model. Analysts should consider multicollinearity in interpreting effects, to understand how different aspects of appraisal influence QOL scores. Findings From Preliminary Studies Based on the QOL Appraisal Model In earlier studies using the original QOLAP, Rapkin, Schwartz, and colleagues 11 found that assessing appraisal clarifies individual differences in response to illness and treatment; this helps to explain discrepancies in individual change. For example, in a sample of 394 people living with HIV/AIDS, Li and Rapkin 24 used classification and regression tree analysis to examine how changes in the Medical Outcomes Study Short Form (SF-36) mental functioning over 6 months were related to changes in appraisal. Measures of “expected change” were calculated based on patient demographics, health changes, and intervening life events. Positive and negative discrepancies from estimates of QOL change (residuals) were associated with several distinct patterns of change in appraisal, such as changes in emotional focus (eg, “focusing on negative experiences and feelings”) and subjective norms (“comparing self with others”). This ability to explain residual change in QOL that remains after standard determinants are controlled using changes in cognitive appraisal represents direct evidence of response shifts. Similarly, in a study of patients with multiple sclerosis, Li and Schwartz 25 found that 20% of patients demonstrated changes in appraisal that accounted for variance in change in QOL measured using the PROMIS (Patient-Reported Outcomes Measurement Information System) General Health measure, the Neuro-QOL Cognitive Function and Positive Affect & Well-Being short forms, and the Ryff Environmental Mastery measure. This study also found that response shifts depended on disease status. As hypothesized, response shifts were most prevalent in a “progressive disease” cohort, somewhat less in a “relapsing” cohort, and relatively rare in a “stable disease” cohort. Classification trees differed qualitatively across cohorts, suggesting recalibration, reprioritization, and reconceptualization of criteria for appraising QOL. In the same sample of patients with multiple sclerosis, Schwartz and colleagues 26 found that individuals' historical and current activities to build cognitive reserve were associated with differences in QOL appraisal. Although item response patterns differed slightly between historical and current active measures of reserve, they generally reflected a tendency for high-reserve individuals to emphasize the positive and to focus on aspects of life that are more controllable, rather than idealized standards of comparison or goals that are unlikely to be attainable. Schwartz, Finkelstein, and colleagues examined evidence for different aspects of response shift in evaluation of PROs for spinal surgery, measured using the SF-36 and the Life Satisfaction Questionnaire-11 (LISAT-11). 27 , 28 They noted that response shifts caused patients to use the same functional outcome report measure differently pretreatment and posttreatment. For example, patients who improved in terms of leg and back pain effectively “moved the goalpost” for rating QOL after surgery, and placed a greater premium on physical functioning. Conversely, patients for whom surgery had no effect reprioritized their ratings of pain to include more social and emotional dimensions. In a separate analysis using the retrospective pretest, these investigators also found that implicit theories of change (ie, differences in patients' conceptualization of the meaning of change) accounted for greater bias in evaluating surgical outcomes than did mere recalibration of scales. Morganstern and colleagues 29 interviewed 50 patients with bladder cancer before surgery. Analyses included content coding of QOLAP personal goal statements, examining the relationship of goal attainment to content, and associating goal-based measures with the European Organization for Research and Treatment of Cancer (EORTC) Quality of Life 30-Item Questionnaire (QLQ-C30) scales. Associations among measures of goal content and goal attainment provided evidence for construct validity. Progress toward goal attainment correlated positively with relationship- and role-based activity goals, but negatively with health concerns. Goals also differed according to age, gender, employment, and marital status, lending further support for construct validity. QLQ-C30 functioning and symptom scales correlated with goal content but not with progress toward goal attainment. This lack of correlation seems counterintuitive, because one might expect that poorer functioning would impede progress toward goals. However, findings suggested that patients' goals may be determined or constrained according to their levels of functioning, and they thus rated their progress relative to those goals. Alternately, progress toward goals may reflect a unique aspect of QOL that is untapped by more standard disease-specific scales like the QLQ-C30. In a later study with an expanded sample of 215 patients with bladder cancer, Anderson and colleagues 30 explored this issue further by examining the contribution that idiographic (or self-defined) measures of QOL derived from the QOLAP made to global ratings of QOL. Idiographic measures included perceived distance from attaining personally defined goals and difficulty associated with goal attainment. Standard measures of role functioning and future perspective explained 15.7% of the variability in preoperative global QOL. Including goal attainment and activity, difficulty explained an additional 12.0% of the variance. Smaller gains were seen on measures of global health, life satisfaction, mental health, and activity, suggesting that idiographic measures capture aspects of QOL distinct from health and functional status defined by nomothetic scales. These results and similar findings in other studies using the QOLAP led us to consider the need for a new generation of appraisal measurement. We were particularly concerned about replacing qualitative questions that were complex to administer and score and about reducing the overall length of the measure. We thought that reexamining these major studies as well as other work on appraisal and response shift would allow us to derive more parsimonious approaches to the assessment of QOL appraisal. That is the basis for the present study. Background on Bladder Cancer Later in this report, we provide an in-depth evaluation and comparison of the properties of new appraisal measures in a prospective study of patients with bladder cancer. We chose to focus on bladder cancer because the disease and its treatment present many challenges to QOL. Each stage of illness has several treatment options, each of which can affect functioning and well-being in different ways. As such, QOL may vary markedly, depending on individuals' experience of disease and treatment-related symptoms and their appraisal of those symptoms. Bladder cancer is the fifth most commonly diagnosed cancer, with an estimated 74 000 new cases in 2015 in the United States alone. It is the second most common genitourinary malignancy; it is the fourth most common malignancy in men and the ninth in women. 31 Depending on prognosis, treatment follows various pathways. Patients identified at an early stage, before cancer has invaded muscular tissue surrounding the bladder, may have the option of active surveillance. This involves repeated trans-urethral resection of bladder tumor (TURBT), in which a cystoscope is inserted through the urethra, the bladder is examined internally, and any local tumor growth is removed. Many patients also receive local treatment with Bacillus Calmette-Guérin (BCG), a tuberculosis vaccine that stimulates immune response in the bladder. TURBT, which may be repeated on an ongoing basis at 3-month intervals, has been associated with discomfort and pain as well as anxiety associated with disease progression. 32 Despite the need for multiple invasive procedures, TURBT allows people to retain their bladder and preserve normal functioning for as long as possible prior to disease progression. High-grade aggressive non-muscle-invasive bladder cancer and muscle-invasive bladder cancer are most often treated with radical cystectomy (ie, removal of the bladder) and concomitant reconstructive surgery to create a urinary diversion. Some patients with less advanced disease also opt to have radical cystectomy to avoid the burden of ongoing surveillance and reduce the possibility of disease progression. Despite survival benefits, radical cystectomy has major potential to negatively affect urinary, bowel, and sexual function, as well as body image. Treatment may also have broader ramifications for these patients' QOL, including social and role functioning, intimacy, self-esteem, and psychological distress. 33 Difficulties that patients encounter are often determined by the type of urinary diversion they have. Patients treated with radical surgery undergo 1 of 3 types of urinary diversions: ileal conduit (urostomy), neobladder, or continent cutaneous urinary diversion (continent reservoir). The decision of the type of urinary diversion depends on surgeons' experience and patients' performance status and comorbidities; thus, accommodating patients' preferences may or may not be possible. Of the 3 main options for urinary diversion, the simplest is ileal conduit. In this procedure, surgeons connect the ureters to an isolated section of distal ileum and construct a skin stoma. A patient who has had an ileal conduit will need to use an appliance (stoma bag) for the rest of his or her life. Although this procedure may have marked impact on body image and daily activities, it has the lowest rate of early and late complications. As such, it is usually the best option for older and sicker patients. The second type of urinary diversion is the orthotopic neobladder. In this approach, surgeons reconstruct a pouch from a section of small bowel and connect it to the urethra. Patients who have a neobladder must learn to void by straining. Approximately 20% of these patients will have long-term incontinence. The third type of urinary diversion is the continent cutaneous urinary diversion. In this technique, surgeons construct a pouch from a piece of colon and connect a small stoma to the skin, usually at the umbilicus. Patients who have had a continent cutaneous urinary diversion must catheterize to empty this pouch every 3 to 6 hours for the rest of their lives. Patients with an orthotopic neobladder or continent cutaneous urinary diversion have more postoperative difficulties and higher complication rates than those with an ileal conduit, but these procedures may help to preserve body image. All 3 different treatment options potentially impinge on everyday activities to differing extents. Because continent cutaneous diversions do not involve a permanent external pouch, patients and physicians may assume that this approach will have better QOL outcomes than the ileal conduit. However, neobladders and continent cutaneous diversions have disadvantages as well. These reconstructions are technically more challenging and more time consuming to perform. Postoperatively, patients leave the hospital with indwelling catheters. Once the catheters are removed, patients must undergo a period of education in the techniques required to care properly for the reservoir or neobladder. Patients are also at a higher risk for diarrhea, vitamin B12 malabsorption, and metabolic disorders. It may take up to 2 years for patients to regain adequate urinary control with a neobladder, so incontinence is an ongoing concern. Nevertheless, the QOL advantages of continent urinary diversions are believed to outweigh the potential disadvantages for patients undergoing radical cystectomy. 34 This particular condition poses many different issues for measuring QOL that our appraisal instruments are well suited to enhance. Stakeholder Engagement Types and Numbers of Stakeholders Involved Throughout this project, we worked with a panel of 10 stakeholders representing developers, participants, and consumers of QOL research. This collaboration enabled us to ensure that the content, format, and procedures of the new appraisal measures were as concise and portable as possible without sacrificing key information. Stakeholders included individuals who have worked with study investigators in various capacities. Panel members are listed below, with further detail in Appendix A : Joanne Buzaglo, PhD , Vice President for Education and Research, Cancer Support Community, Philadelphia, Pennsylvania Joel Finkelstein, MD , Orthopaedic Surgeon, Sunnybrook Health Sciences Centre and University of Toronto, Ontario, Canada Mitch Golant, PhD , Senior Consultant, Strategic Initiatives, Washington, DC Nicole Hollingsworth, EdD, Assistant Vice President, Community and Population Health, Montefiore Medical Center, Bronx, New York Wendy Kahalas, MA , Evaluation Specialist, New York State Department of Health AIDS Institute, Bronx, New York Maureen E. Lyon, PhD , Child and Adolescent Psychologist, Children's National Medical Center, Washington, DC Sherry Schachter, PhD, RN , Director of Bereavement Services, Calvary Hospital, Bronx, New York Mirjam Sprangers, PhD , Professor, Medical Psychology, University of Amsterdam, The Netherlands Timothy Vollmer, MD , Director, Neurosciences Clinical Research, University of Colorado, Denver Elisa Weiss, PhD , Vice President, Patient Access and Outcomes, Leukemia & Lymphoma Society, Rye Brook, New York Methods Used to Identify and Recruit Stakeholder Partners We decided that the most productive step was to bring together people who could work well with one another and who were familiar with assessment of QOL appraisal. We wanted members who were likely to stay involved throughout the project so that they would become increasingly familiar with challenges in the assessment of appraisal. We also wanted stakeholders who would be able to disseminate measures and champion appraisal assessment in their professional roles and affiliations. Several of the stakeholders (and investigators) are themselves cancer patients, survivors, or caregivers or are people living with other chronic, debilitating conditions. Stakeholders had integral roles to play in this project. We wanted and needed to hear what stakeholders had to say about differences in how individuals appraise their QOL as well as the ways these appraisals change over time. We wanted to approach this from several vantage points: clinicians who must ask about QOL in their daily practice; health policymakers who need to understand the wide impact of programs; researchers who have struggled to make sense of complex QOL data; and, of course, patients and survivors who will have particular insights about ways that people interpret QOL questions in different contexts. Methods, Modes, and Intensity of Engagement Phases of Stakeholder Participation To ensure meaningful involvement of stakeholders in this study, we introduced the panel to problems in QOL research and the rationale and methods for appraisal assessment. Rather than jump directly into interpretation of psychometric analyses, we first worked with stakeholders to develop and clarify their own perspectives on QOL appraisal. These early phases lay the foundation for deeper, more focused critical analysis of new appraisal measures. Throughout the project, we conducted 2-hour conference calls with stakeholders every quarter (see Appendix B for the Quarterly Agenda for Stakeholder Panel calls). Stakeholder Impact on the Relevance of the Research Question We worked with the core panel of 10 stakeholders during all stages of psychometric analyses and measurement development. The panel focused on ensuring that the content, format, and procedures were as concise and portable as possible without sacrificing key information. Through quarterly conference calls, this panel had input in writing and adapting items and specifying response formats and instructions. Working with the core stakeholder panel, we presented 4 early concepts for the new appraisal measure that were ultimately consolidated into a preliminary measure and submitted for review by a broader group of stakeholders. Broadening the Stakeholder Base Via an Online Survey To gain further input on measurement development, we asked panel members to identify additional stakeholders to review the draft appraisal measure and provide feedback via an online survey distributed using Survey Monkey. In addition to feedback about the measure, the survey also asked participants about their experiences with QOL research, their need to know about QOL appraisal in their clinical/advocacy work, the pros and cons of existing QOL measures, and whether and how they could accommodate more in-depth QOL appraisal measures. Stakeholder input on participant recruitment To identify online survey participants, core stakeholder panel members drew on their professional organizations (eg, Oncology Nursing Society), research societies (eg, International Society for QOL Research), public health agencies (eg, the Public Health Association of NYC), and patient participants in QOL studies (through the Bronx Community Cancer Coalition). Respondents received $100 for completing the survey. A total of 30 respondents completed the survey. We then coded responses and analyzed them for presentation to the core stakeholder panel. Stakeholder input on study rigor and quality The 30 survey respondents represented a diverse range of clinical, research, and public health professionals who worked with diverse patient populations, including, but not limited to, oncology, pediatrics, HIV/AIDS, and multiple sclerosis. Nearly half of the respondents (48%) indicated that they use QOL measures to assess intervention outcomes. Of those respondents, 39% use QOL measures for needs assessments; 13% use them for comparing groups. Most respondents (75%) indicated that they use QOL measures for repeated assessments; 25% of the respondents use them only at baseline. A majority of the respondents (63%) endorsed the idea of “QOL appraisal” as very important to their work. Many noted that they would be interested in using a brief appraisal measure, once it was validated. We also received useful feedback regarding the length of the instrument, clarity of instructions, literacy level, and anticipated challenges with specific patient populations (eg, adolescents and palliative care). Several respondents also indicated the importance of including religion or spirituality as part of the appraisal. The core stakeholder panel discussed this concept extensively when reviewing survey responses. Stakeholder Impact on Instrument Development Based on feedback from the online survey, the new appraisal measure included 5 sections that addressed the 4 parameters of the Rapkin and Schwartz QOL Appraisal Model. Our original plan, as detailed in our application to PCORI, was to develop 1 new instrument with features similar to those of the original QOLAP. However, the concerns about length expressed by some online survey respondents led us to develop and test a second, briefer instrument as well. The Methods section gives details about the development of both new appraisal instruments. Stakeholder Adoption of Research Evidence Into Practice In addition to providing input on the development of the new measures, Drs. Finkelstein, Hollingsworth, Vollmer, and Weiss adopted the use of 1 or more of the new appraisal instruments into their work. Drs. Buzaglo, Golant, Lyon, and Schachter have discussed ways to incorporate appraisal into their own projects. Stakeholders' firsthand experience with the measures provided useful perspectives on what measurement of appraisal brings to QOL research, clinical practice, and policy. Stakeholders were particularly helpful in conceptualizing the format and clarifying instructions. Several stakeholder panel members, particularly the physicians, asked that we not abandon the unique qualitative assessment of our earlier appraisal interview. They liked having the opportunity to understand patients' concerns in their own words. Based on this feedback, we decided that abandoning the original QOLAP instrument was not necessary and that having several validated approaches to assess appraisal in different research and clinical contexts was most desirable. Rather than foreclose on a single tool, this project allowed us to compare a briefer option for the assessment of QOL appraisal with one that is more in-depth and descriptive, introduced in our Methods section. Methods Overview of Project Aims This project focused on developing measures of QOL appraisal, as a way of improving the validity and precision of QOL outcomes assessment. We sought to create practical measures for clinical and research purposes. Project aims were as follows: Leverage existing data from earlier studies to identify empirically the aspects of appraisal needed to explain differences in PROs within and across populations Derive concise, new measures of appraisal by examining empirical findings with stakeholders who use QOL assessment for research, assessment of clinical practice, or policy evaluation Test the reliability, validity, and ease of administration of concise appraisal measures in a new sample of patients with bladder cancer To address aim 3, the main focus of this report is a multisite study of patients with bladder cancer. Findings for aims 1 and 2 have been reported in 6 publications; these describe analyses that informed item content and format of the new appraisal measures and yield psychometric data obtained from online surveys. 35-40 Work related to aims 1 and 2 provided the basis for the new measures of appraisal used in aim 3. We summarize these 2 aims below. Aim 1: Development of the QOLAP Version 2 Creating Appraisal Items Based on Findings From Earlier Qualitative Research The goal of Aim 1 was to leverage existing data from earlier studies to identify empirically the aspects of appraisal needed to explain differences in PROs within and across populations. Building on stakeholder input and the earlier published work noted previously, the first aim of this project was to take a deeper look at our existing data to determine the best ways to summarize appraisal. We were guided by several major questions in this effort: Which aspects of appraisal are most important to assess? How can the number of questions be reduced or combined to reduce the time involved? Is there unnecessary redundancy among sections of the QOLAP? What are the best item-formats to elicit appraisal? We carried out analyses to address these questions using our earlier data sets; these are documented in Appendix C and in a series of papers that highlight steps that we took to more fully explore the QOLAP data. 30 , 36-42 Here, we draw on this work to summarize the steps taken to develop the QOLAP version 2 (QOLAPv2), based on these results. At the outset, we anticipated that we would need to replace the 2 qualitative sections of the QOLAP used to assess frame of reference: Definition of QOL and Personal Goals. In different studies using the QOLAP, we continually employed and adapted the same qualitative coding system to identify themes associated with QOL definition and personal goal attainment. As such, examining the prevalence and combination of codes pertaining to QOL definitions was relatively straightforward. In general, 11 to 14 thematic codes could account for patients' QOL definitions across different studies. We used language drawn from patients' responses to the QOLAP to write items based on these codes. Rather than necessitating a verbatim response, the new Definition of QOL section of the QOLAPv2 enabled patients to indicate how well their own definition of QOL agreed or disagreed with 18 different phrases corresponding to their own understanding of the meaning of QOL. Examples of phrases included the following: “Perfect health,” “Looking back on my legacy,” and “Having nothing to regret.” Thematic content elicited by the original QOLAP question about personal goals was more complicated to distill into items than we had anticipated. Responses referred to motivational themes and life domains as well as to specific relationships and/or situations. We wanted to capture aspects of goals that moderated the impact of catalysts on QOL. Thus, rather than basing items on the more common goal statements, we focused on the goals of some individuals, but not others, to ensure that that measure was sensitive to these differences. To develop a representative set of items, we wrote “goal statements” that reflected respondents' answers on the QOLAP. Goal statements were cross-classified according to 5 motivational themes (achievement, problem-solving, maintenance, acceptance, and disengagement) and 5 life domains (health, people, roles, self, and living situation). Based on this grid, we selected (with stakeholder inputs) 25 goal statements to represent the types of personal goals that may or may not be relevant to individuals' appraisal of QOL. Items were rated on a 5-point scale indicating how well each statement reflected respondents' own goals and concerns. We also included a “not applicable” option so that people could indicate goal statements that did not pertain to them. Examples of these items included the following: “There is a specific problem or challenge related to my finances, my living situation, or other practical matters that I want to resolve”; “I want to be able to participate fully in important upcoming events”; “There is much more I want to accomplish at my job”; and “I want to stay in my current home for as long as possible.” Sections pertaining to Sampling of Experiences (14 Likert-scale items) and Standards of Comparison (9 Likert-scale items) were retained from the original QOLAP. Sampling of Experiences items ask individuals about the kinds of situations they recalled in responding to QOL questions and address several theoretical considerations that can enter into memory: valence/mood, primacy/recency, cueing, and the social demand characteristics of the interview. For example, we asked, “Did you find yourself thinking about the worst possible moments?,” “Did you try to remember everything relevant over the past 3 months?,” and “Did you try to give your first reaction to the questions?” Standards of Comparison items focused on points of reference considered in evaluating QOL, including “Most people your age,” “Your ideal or dream of perfect health,” and “A time in your life before you had this chronic condition.” The final items probe the ways that individuals reconciled, combined, or emphasized discrepant experiences when they formulated responses to QOL questions. On the original QOLAP, we used a series of semantic differential items; however, this item format was not as informative as we had anticipated because a linear differential (eg, “Things that were completed vs unfinished concerns”) did not work consistently; some people considered both poles equally, whereas others considered neither pole. We decided to rewrite these as Likert-scale items, consistent with the other sections of the new measure. This revised section consists of 9 items describing the QOL areas that individuals emphasized in rating QOL items, including the following: “There are so many ups and downs that it is hard to know what to rate”; “You are more focused on things you have to do than on accomplishments”; and “You have been able to keep up your mood, despite problems.” Examining Psychometric Properties of the QOLAPv2 In fall 2015, our team had the opportunity to administer the QOLAPv2 to more than 4000 patients recruited from the participant panels provided by Rare Patient Voice (RPV) LLC. RPV LLC helps pharmaceutical companies and other industries assemble large cohorts of patients with different chronic illnesses for drug trials. The leadership of RPV concluded that many of their patients would be interested in answering questions about QOL and appraisal. Analysis of this ample RPV data set allowed us to evaluate the psychometric characteristics of the new QOLAPv2 more extensively than would otherwise have been possible. We obtained data on QOL appraisal and other measures from patients with multiple different diagnoses—far more varied than our previous appraisal studies. A total of 4173 members of RPV panels (out of approximately 31 000) responded to our survey, representing a broad range of chronic diseases (about 350 distinct diagnoses). This sample reported having multiple coexisting conditions, including, but not limited to, back pain (56.97%), cancer (51.93%), depression (50.78%), insomnia (44.86%), arthritis (42.15%), asthma (18.47%), and diabetes (11.33%). The average age of participants was 48, with a mean income of approximately $50 000. One limitation of the RPV sample obtained was the age and sex characteristics. The vast majority of this sample was white (91%) and female (85%), consistent with RPV's current membership base. Despite restrictions in demographic composition, we decided to examine psychometric properties of the QOLAPv2 as a prelude to subsequent validation in a more diverse sample, in terms of age and sex (aim 3). Structure of the QOLAPv2 First-order principal components analysis (PCA) yielded 29 components across the 5 sections of the QOLAPv2 measure. We explored associations among different sections by conducting a second-order PCA. These second-order components represent patterns of appraisal, pulling together first-order components from across the 5 different sections of the QOLAPv2. Patterns were meaningful and reflected important aspects of QOL appraisal. Twelve second-order components pulled together different aspects of people's frames of reference, recall of experiences, standards of comparison, and sampling of experiences in ways that were coherent and interpretable. These components suggested clear themes in the different ways that people evaluate their health and well-being. They included the following: Wellness focus : Maintain a calm and healthy lifestyle; learn self-acceptance; keep up health activities and health care; focus on improvements; get used to the way things are; remain positive and balanced; do not think about the worst moments. Health worries : Focus on what doctors say; high frequency of social comparison Recent challenges : Recall recent relevant health episodes and challenges; learn to accept others; let go of self-expectations; make multiple types comparisons in rating QOL. Spiritual focus : Emphasis on faith and generativity Relationship focus : Interest in romance; improved relationships; self-acceptance Maintain roles : Concern about personal accomplishments; Maintain community and work roles, address family responsibilities and problems, and deemphasize self-acceptance; concern about regrets Independence : Maintain independence; resolve problems; remain in one's home; have no regrets; resolve recent money problems and other negative circumstances; keep active; maintain participation in multiple roles. Reduce responsibilities : Let go of responsibilities for a house or for other people; let go of self-expectations; spend time with family; do not think about the QOL issues raised by the survey. Pursue dreams : Pursue dreams and goals; change living situation; do not base QOL ratings on comparisons to others one's own age. Anticipate decline : Prepare loved ones and living situation for declines in health; focus on ups and downs in health; focus on what the doctor said to expect. Be worry-free : Compare oneself to others without health limits and not to others with the same condition; solve money living situation, and practical problems, rather than accept people and roles as they are; let go of self-expectations. Lightness of being : Be spontaneous; do not complain about health; focus on health rather than illness; concern about how one is seen by others Variance in Physical Functioning Explained Using the QOLAPv2 Based on these results, we decided to examine associations of second-order components with different measures of QOL. By way of example, a series of regression analyses based on the QOL Appraisal Model ( Figure 1 ) revealed that demographic antecedents and comorbidity catalysts explained 29% of the variance in physical functioning measured by the PROMIS Global-10. Conversely, personality factors explained 6% of the variance. In contrast, second-order appraisal components alone explained 35% of the variance. When we entered Standard Model demographics and comorbidities first into these models, appraisal accounted for an 11% increment in the R 2 statistic (ie, 46% of total variance was explained). These associations of QOLAPv2 second-order components with measures of QOL, as well as further analyses examining the association of appraisal and personality measures, led us to consider the possibility that appraisal might best be understood in terms of more integrated experiences or patterns of thinking represented by the second-order components rather than discrete parameters represented by first-order components. This set the stage for the next phase of this project. Aim 2: Development of the Brief Appraisal Inventory Derivation of New Items Describing Observed Patterns of Appraisal Aim 2 was to derive concise, new measures of appraisal by examining empirical findings with stakeholders who use QOL assessment for research, assessment of clinical practice, or policy evaluation. Based on stakeholders' discussions regarding the patterns of appraisal evident in the second-order PCA, we were able to create a more concise, 23-item instrument called the Brief Appraisal Inventory (BAI). This new inventory uses a measurement strategy that differs in important ways from the QOLAPv2. Specifically, the QOLAPv2 maintained the molecular (ie, individual or small components) distinctions among the 4 main parameters of appraisal in the QOL Appraisal Model. We wrote the BAI items to reflect molar (ie, larger or overarching components) and emergent patterns of appraisal observed in the QOLAPv2 second-order components analysis, as well as findings of our earlier analyses on the “essence of appraisal.” 37 The BAI asks respondents to consider the extent to which each pattern of appraisal reflected what was on their mind when they responded to the QOL measures they had just completed. Initial drafts of BAI items were reviewed with stakeholders and subjected to cognitive interviews with 5 respondents before initial psychometric testing. Rationale for Assessing Appraisal in Terms of Patterns of Ways of Thinking The BAI items reflect combinations of appraisal parameters that captured the most variance across the QOLAPv2 domains. In writing the BAI items, we drew on much of the language of the source items so that statements were as close as possible to the original combinations in meaning. When combinations of variables loading on the QOLAPv2 second-order components were potentially too complex or ambiguous, we broke these out into 2 separate items. The resulting BAI items were written to reflect things that an individual might be thinking about during an interview. The 23 items may be understood as sampling broadly from the universe of thoughts that individuals may be having as they complete a QOL questionnaire. This led us to put a caption at the top of the measure asking, “What's been on your mind?” This approach used natural language that does not require individuals to drill down introspectively to somewhat more abstract questions about their cognitive experience removed from context. Nonetheless, we recognized that this sample of items could not capture all the detailed self-reflection represented on the lengthier parent measure (QOLAPv2). The question was whether the BAI could account for large enough proportions of variance attributable to appraisal and response shift in meaningful ways, sufficient to warrant its inclusion in future research. We sought to examine the tradeoffs between these 2 approaches in aim 3 analysis. Psychometric Properties of the BAI We administered the newly developed BAI to a second cohort recruited from the RPV online panels. As with the original RPV sample, members of RPV's online panel of approximately 31 000 patients with chronic disease received an email invitation to participate in this study. We used this data set to conduct a psychometric analysis of the 23-item BAI. A total of 592 volunteers completed the BAI along with other measures (mean age 43.8, SD = 18.5; 79% women). In terms of item characteristics, a missing item analysis revealed that 95.1% of respondents completed 19 or more items of the 23-item BAI. The remaining 4.9% of respondents who completed 18 or fewer items were omitted from the analysis. The final sample included 563 respondents who responded to 19 or more items. Participants dropped from the analysis because of missing BAI data tended to be men. 35 The BAI stem asks respondents to consider the list of 23 items in order to indicate “How much was each question on your mind as you completed the survey today?” Responses are based on a 5-point Likert scale (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always). Because we retained only those cases with 4 or fewer missing items, any sporadic missing items were assumed to be “not applicable” and were recoded as “never on my mind.” Frequency analysis revealed that respondents used the full range of the response options on every item. Of these 23 items, 52% of responses had a negative skew toward the “always” response item; 48% had a positive skew toward the “never” response item. The mean response for each individual item ranged from 2.0 to 3.7, with an overall mean of 2.9 for all items. Structure of the BAI and Relationship of Appraisal Components to QOL The 23 BAI items demonstrated many highly significant inter-item correlations ( p < 0.001), prompting us to try to reduce this measure further. PCA with Varimax rotation yielded 5 components that explained 59.7% of the total variance. Item communalities ranged from 46% to 76%. We characterized the 5 components as follows: (1) health worries; (2) interpersonal and independence concerns; (3) accomplishing goals and problem-solving; (4) calm, peaceful, and active; and (5) spiritual growth and altruism. We examined the correlations of the 5 BAI components with the demographics of the second RPV sample, comorbidities, health-related QOL, and NEO Personality Inventory (NEO-PI) 43 with scores for openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism (see Rapkin et al 35 for details on measures used in this study). It should be noted that although the NEO-PI scale names reflect traditional understanding of personality traits, the items themselves offer a straightforward way to examine self-concept. Health worries had the largest number and magnitude of correlations with the other constructs, particularly with worse physical and emotional functioning. People who endorsed health worries tended to be female, younger, and sicker, with low past social support, low conscientiousness (eg, likely to describe oneself as lazy and not thorough), and higher neuroticism (eg, likely to describe oneself as nervous and not relaxed). Interpersonal and independence concerns were correlated with physical functioning. People high on this component also tended to be sicker and have lower levels of education. Accomplishing goals and problem-solving was not associated with QOL subscales. People endorsing this appraisal pattern tended to be male, younger, less likely to have cancer, and more likely to endorse depression. They also reported lower current social support and registered as less conscientious on the NEO-PI. The calm, peaceful, and active component was correlated with better physical functioning. People endorsing this appraisal pattern had fewer comorbidities, were less likely to endorse back pain, had higher levels of perseverance and current social support, and were more agreeable (eg, more likely to describe oneself as trusting and not finding fault with others), as well as conscientious and less neurotic. Spiritual growth and altruism was unrelated to the QOL subscales. People endorsing this appraisal pattern tended to be female, with lower levels of education and higher perseverance scores. Aim 3: Testing Concise Appraisal Measures in a New Bladder Cancer Sample Aim 3 of this study was to test the reliability, validity, and ease of administration of the newly derived, concise appraisal measures in a new sample of patients with bladder cancer. The present study focuses on comparing and validating the QOLAPv2 and the BAI in a prospective study of diverse patients treated for bladder cancer. We conducted a series of analyses focused on construct validity to examine whether these measures demonstrate anticipated demographic differences in appraisal. In accord with hypotheses based on the QOL Appraisal Model, we also tested whether these measures function as mediators and moderators of the impact of changes in clinical status and other stressful events on QOL. We explored the association of appraisal assessment with patients' mood states to examine hypotheses regarding the role of mood in the appraisal process. In all these analyses, we examined the extent to which the BAI could serve as a stand-in for the longer QOLAPv2. In the following section, we describe the results of this prospective study as well as details of our hypotheses. Study Design This prospective study collected data from a diverse sample of bladder cancer survivors recruited from 3 sites: Montefiore Medical Center in the Bronx, New York (Montefiore); Memorial Sloan Kettering Cancer Center on the Upper East Side of Manhattan (MSKCC); and White Plains Hospital in suburban Westchester County immediately north of the Bronx. Data were collected at 2 time points separated by 3-month intervals. The study protocol is in Appendix D . Self-report instruments and the cognitive interview script are in Appendix E . Study sample inclusion criteria included being diagnosed with bladder cancer and having received any cancer-related treatment, including active surveillance, anytime between December 31, 2012, and June 30, 2017. Participants had to speak English or Spanish, be 18 years of age or older, and be able to provide informed consent. Exclusion criteria included not having follow-up care at the recruiting site or having metastatic disease at diagnosis (or both). Study Outcomes Research staff administered the following self-report measures and clinical information in addition to the new appraisal. Quality of life Two measures used were from the EORTC questionnaires. The generic EORTC QLQ-C30 core measure evaluated 6 major domains (physical, role, emotion, social, cognition, and global QOL); 3 symptom scales (fatigue, pain, emesis); and 6 single items (financial impact, dyspnea, sleep disturbance, appetite, diarrhea, and constipation). 44 The bladder cancer-specific module, the EORTC Quality of Life Questionnaire Muscle-Invasive Bladder Cancer Module (QLQ-BLM-30), assesses symptoms related to urination, bowel functioning, and sexual functioning. It also assesses problems with managing a urostomy, catheter manipulation, and body image. 45 (These 2 instruments took respondents about 10 minutes to complete.) Mood state The Brief Mood Introspection Scale (BMIS) is a 16-adjective mood scale designed to measure pleasant-unpleasant and aroused-calm dimensions of mood state. 46 The scale can also be scored to identify positive-tired and negative-calm mood. Each item is 1 of 2 adjectives selected from each of 8 mood states: (1) happy (happy, lively); (2) loving (loving, caring); (3) calm (calm, content); (4) energetic (active, peppy); (5) fearful/anxious (jittery, nervous); (6) angry (grouchy, fed up); (7) tired (tired, drowsy); and (8) sad (sad, gloomy). Participants indicate how well each adjective describes their current mood on a 4-point scale. Cronbach α reliabilities range from 0.76 to 0.83. (The BMIS took approximately 5 minutes to complete.) Stressful events and experiences The Inventory of Recent Life Experiences for Cancer Patients (IRLE-C) is a 30-item measure designed to measure cancer-related hassles. 47 The IRLE-C distinguishes stressors from their mediators, reactions, and consequences, making it particularly useful for testing models relating stress to adjustment in cancer populations. Items are rated on a 4-point Likert scale of intensity over the past month (1 = “not at all part of my life” to 4 = “very much part of my life”). (The IRLE-C took respondents about 5 minutes to complete.) Demographic characteristics Demographics were assessed at the first time point only and included sex, race, ethnicity, marital status, education, employment, and household composition. (This section took respondents about 5 minutes to complete.) Development of the Cognitive Interview Appraisal measures are intended to describe how individuals respond to multiple QOL items. As such, we attempted to validate the 2 new measures against a “think-aloud” approach that asks individuals to share their internal dialogue as they consider and rate representative QOL items. Cognitive interviewing of this sort would not be a practical way to assess appraisal in most large-scale studies that include PROs. It would, however, be very useful to compare responses from the QOLAPv2 and BAI with analysis of themes elicited by a qualitative, cognitive assessment. Therefore, in addition to the 2 new quantitative appraisal measures (QOLAPv2 and BAI), we also developed a third qualitative assessment of appraisal, the Cognitive Assessment of Select QOL Questions. This third measure extends earlier work by Bloem and colleagues 48-50 to examine whether patients spontaneously refer to parameters of the Rapkin and Schwartz QOL Appraisal Model 11 when they think aloud about aspects of QOL. We further structured the assessment approach of Bloem and colleagues to ensure that we could distinguish how patients appraise different aspects of QOL. This interview includes 8 items representing different domains of general functioning and disease-specific QOL. Following each item, patients were asked 5 probes to talk about their appraisal process for that particular item. Patients' verbatim responses were coded for subsequent analysis to provide a “gold standard” measure of appraisal. Unfortunately, problems with reliability of coding precluded analysis of this measure for this report, as we detail below. Data Reduction Strategies We used principal PCAs to reduce the number of IRLE-C and BMIS variables in the analysis. This empirical approach to data reduction ensured that summary variables accurately reflected relationships in our sample. For both measures, sample-specific components were highly similar to components in the scales in reference samples. The analysis of the 30-item IRLE-C suggested a 6-component solution explaining 63% of the variance. After examining different approaches to rotation of these 6 components, we determined an oblique “Promax” solution would be best. This solution demonstrated the best “simple structure,” with the majority of items represented on only 1 component. IRLE-C components included the following: Impact on family (α = 0.89) Physical experiences and enjoyment (α = 0.84) Difficulties with activities of daily living (α = 0.82) Difficulty discussing needs with network (α = 0.78) Lack of information (α = 0.74) Role and time difficulties (α = 0.74) The PCA analysis of the 16 BMIS items yielded 4 component scores, accounting for 63.5% of variance in mood. In this instance, we retained the orthogonal Varimax solution: Sad, anxious, fed up Happy, energized, content Loving, caring Fatigued Psychometric Properties of New Appraisal Measures in the Bladder Cancer Sample We conducted item-level analysis of the QOLAPv2 and BAI to examine response patterns, distributions, and extent of missing data. We determined whether items that were skipped or were marked not applicable were consistent with patients' circumstances (eg, people who are unmarried, have no children, or are unemployed should indicate that certain items are not applicable). We created QOLAPv2 and BAI appraisal scores for the new sample based on results of new, sample-specific PCA with Varimax rotation. The Results section gives the details of these analyses. Criterion validity As noted, the Cognitive Assessment of Select QOL Questions was intended to provide a criterion assessment of appraisal for validating the 2 new quantitative measures. We developed a coding system for this interview using standard ethnographic methods, examining a subset of responses to develop themes, applying them to new data, and determining whether additional themes were required. Themes were scored as either present or absent. Three raters independently coded 100% of the data. At different junctures during coding, we examined inter-rater agreement (κ coefficients) to determine reliability. Whenever the coders disagreed, the final coding for every response was automatically based on the majority opinion (ie, 2 of 3 agreed), although the dissenting rater could ask for adjudication to argue for an alternative decision. All decisions, discussions, and rules were recorded in a common coding manual used by all raters ( Appendix F ). Unfortunately, analyses of inter-rater assessments revealed low inter-rater agreement in coding (eg, grand mean κ = 0.46; unadjusted percent agreement = 0.95; percent positive agreement = 0.32). Accordingly, we deemed these data not to be sufficiently reliable to use as a criterion for evaluating the QOLAPv2 and BAI. We consequently relied solely on extensive redundancy analyses, described below. Construct validity We examined the construct validity of the QOLAPv2 and the BAI by testing hypothesized convergent and discriminative relationships at the item level; this approach was similar to our earlier appraisal studies. We then derived composite scores (using PCA) to enable hypothesis testing at the score level. We decided to examine sample-specific components rather than impose scale scores derived from the RPV sample, to account more accurately for dimensions of appraisal in this sample. Analyses using these composite scores focused on evaluating redundancy between the 2 measures and testing response shift hypotheses. Hypotheses We examined expected demographic differences in appraisal associated with socioeconomic status (SES), age, and sex. In our sample, SES and ethnic/minority status are highly correlated with the site of care (the Montefiore sample is predominately lower SES, the MSKCC sample is predominately higher SES, and the White Plains sample is intermediate SES). We attempted to determine whether demographic differences fully explained the QOL differences between the sites. For example, some patients with upper SES may be treated at Montefiore and some with lower SES may be treated at MSKCC. Our ability to explore these variables depended on the demographic overlap of the patient samples gathered at each site. Specific hypotheses included the following: Compared with patients who have higher SES, patients who have lower SES will do the following: Focus less on new achievement-oriented goals Emphasize the need to solve specific problems relating to their circumstances Report greater need to learn to accept situations that they cannot change Express the desire to disengage from roles and obligations Make more negative (self-critical) comparisons Indicate more financial, work, and living situation concerns Focus more on spiritual goals than on leisure or personal growth Compared with older adults, relatively younger patients will do the following: Express greater desire to return to work roles and less desire to disengage Express more concern about independence and burdening family Express less concern about leaving a legacy Compared with men, women will do the following: Indicate greater salience of concerns related to interpersonal relationships Indicate greater need to provide support and care for others Redundancy Analysis One major analytic question is how well the 23 BAI items accounted for variability in appraisal captured by the 75-item QOLAPv2. We examined this in several ways. First, we conducted canonical correlation analyses of the BAI and QOLAPv2 items. We expected that the BAI would explain an average of approximately 50% of the total variance of individual QOLAPv2 items. Second, we implemented a cluster analysis of the BAI items and then used the resulting clusters in a series of regression models predicting each QOLAPv2 component. These regressions included as predictors the main effects of BAI items, all possible 2-way interactions, and clusters. These regression models provided a more grounded and empirical way of evaluating redundancy between the BAI and the QOLAPv2. Response Shift Hypotheses Consistent with our theory of response shift as outlined in the QOL Appraisal Model, 11 we expected that aspects of appraisal would statistically interact with the impact of worsening illness on health. These relationships appeared in previous studies that used the original version of the appraisal measure, 24-30 , 39 and we hypothesized that both the QOLAPv2 and BAI would behave in a similar manner. Appraisal measures are explicitly intended to account for differences in patients' QOL ratings in response to similar changes in health states, so this test was critical for the new measures. Specific response shift hypotheses included the following: Frames of reference: Greater desire to carry out physical activities, remain independent, or return to roles will amplify the negative impact of worsened health states and stressful events on QOL. Alternatively, appraisal associated with personal growth, acceptance, or disengagement will attenuate the impact of worsening health. Sampling experiences: Recall of negative experiences will amplify the impact of worsening health on QOL ratings. Standards of comparison: Self-enhancing comparisons (comparisons with others with illness) will amplify positive and attenuate negative changes. Self-critical or normative comparisons (with normative age expectations or with times prior to illness) will have the opposite effect. Combinatory algorithms: Greater salience of negative events, recent events, impact on others, and unfinished priorities will amplify negative health changes and stressful life events. Mood State and Discriminant Validity of the Appraisal Measure An additional aspect of validity examined the role of patient mood state on appraisal and QOL. Although we expect current mood to have some influence on QOL ratings, we hypothesize that associations between QOL and appraisal measures will not be attributable to mood state at the time of assessment. This hypothesis is based on the QOL Appraisal Model, which emphasizes cognitive processes involving some degree of recall and reflection. If QOL is highly correlated with current mood and not independently correlated with appraisal, this would indicate that responses to QOL items are based more on respondents' present-state reactions than on any reflection. Alternatively, observed correlations between QOL and appraisal may be spurious, fully determined by each domain's correlations with mood. Such a finding would emerge if answers to appraisal items represented individuals' explanations for their reactions to the QOL items, rather than a description of an evaluative process. If appraisal represents post-hoc justification rather than reflection, then it would necessitate a new theory of QOL response shift that is separate from cognitive appraisal and focuses on changes in mood states. To address these possibilities, we repeated analyses examining the relationships among QOL and appraisal measures, including mood measures as covariates. These steps allowed us to rule out the possibility that patients' mood at time of assessment explained observed relationships between QOL and appraisal. These analyses provided information about the discriminative validity of the new appraisal measures. Statistical Considerations and Power The aims of this study involve validation of new measures of QOL appraisal. Accordingly, validation analyses considered differences in appraisal between known groups (eg, by SES, age, and sex). To detect a medium difference (effect size of 0.48, as specified by the G*Power Program 51 , 52 ) in an appraisal factor between Montefiore and MSKCC patients at p < 0.05 and power = 0.80, we would require a sample size of 110 patients. Analyses also examined performance of these new measures in accounting for variance in QOL after taking into account demographic differences, health status changes, and other life events. Using estimates from our earlier analyses of BAI characteristics, we estimated that 5 orthogonal appraisal components would combine to account for approximately 7.5% of the variance in global QOL, after controlling for 7 prior demographic and health status differences (total R 2 = 30%). This translates to a small effect size of 0.15. 52 A total of 110 patients would be required to detect an increment of R 2 attributable to appraisal of this magnitude at power = 0.80 and p < 0.05. Thus, with our sample of 110, we had adequate power to detect meaningful effects. Power analyses were conducted using G*Power 3.1.9.2. 51 Conduct of the Study A study team member systematically screened the medical records of all patients with bladder cancer treated at each site for the limited purposes of identifying eligible participants and contacting the primary surgeon or treating physician to request assistance in recruiting patients into the study. A study team member contacted treating physicians with the list of all eligible patients identified through medical chart review; the physicians' tasks were to review each case, ensure eligibility, and obtain each patient's approval for the study team to contact them. After we received permission to contact, we sent an opt-out recruitment letter to the approved patients noting our collaboration with their treating physician. If the treating physician was no longer at the medical facility or could not be contacted, then we sent the letter to the identified participants on behalf of the Chair of the Department of Urology at Montefiore or MSKCC. A study team member then contacted eligible patients who did not opt out within 2 weeks of the date noted in the recruitment letter. If the participant was interested, then the study team member contacted him or her by telephone or in clinic, to explain the study further and obtain verbal informed consent. During the initial conversation, the research team member explained the study to the patient and asked for additional information to confirm eligibility and interest in enrollment. If the patient was not interested, then the research staff discarded all information collected during the initial conversation and medical records review, except for any information that must be maintained for screening log purposes. Patients were informed that their decision to participate was voluntary and would not affect their care in any way. They were also told that they would be compensated $25 per interview for their participation in this study. Results Study Sample Characteristics Table 1 shows the descriptive statistics overall and by study site: Montefiore in the Bronx, MSKCC in Manhattan, and White Plains Hospital in suburban Westchester County. The study sample included 110 people with bladder cancer, of whom 70% were male and 30% were female. This sex distribution is consistent with the diagnosis of bladder cancer in the population. 53 The mean age was 70.7 years (SD = 11.2), and study participants reported an average duration of living in their current residence of 22.6 years (SD = 14.5). To assess race more accurately, participants could select any terms that applied to them. Seventy-two percent of the sample endorsed white, 17% endorsed black, and 16% selected the category multiracial. For ethnicity, 18% identified as Hispanic. Seventy-eight percent of the sample endorsed living with someone else for the past 3 months, and 51% had a bachelor's degree or higher. The majority of the sample was not working for pay, with 8% indicating that they were unable to work because of disability, 31% describing themselves as retired, and 19% reporting current work as a volunteer. Samples drawn from each site differed on race and education; MSKCC patients were more likely to be white and more highly educated than the Montefiore patients, with White Plains patients intermediate. Given the few cases from White Plains, we examined site differences associated only with Montefiore (contrast coded as 1) and MSKCC (contrast coded as −1) throughout the remainder of this report, with White Plains cases included as intermediate (contrast coded as 0). Table 1 Demographic Characteristics of Study Sample (N = 110). Descriptive Statistics for the QOLAPv2 and BAI Tables 2 and 3 show item content and descriptive statistics for the QOLAPv2 and BAI of the study sample. The QOLAPv2 had 17 skewed item distributions, or approximately 20% of the items; the BAI had 1 skewed item distribution, or approximately 4% of the items. As noted previously, for subsequent analysis, in each section of the QOLAPv2 and BAI we only retained cases with more than 75% nonmissing items. The missing BAI data in those with less than 25% missing data were handled by mean substitution in subsequent PCAs. Because analyses of appraisal were based on principal components, we did not conduct multiple imputation for the individual items. Table 2 QOLAPv2 Descriptive Statistics. Table 3 BAI Descriptive Statistics. Construct Validity: Item-level Analyses Tables 4 and 5 show correlation coefficients among the QOLAPv2 and BAI items by site and by demographic characteristics. We discuss these tables in tandem, to compare findings between these 2 measures. Again, because of the small number of patients from White Plains in this sample, we focus on differences between Montefiore and MSKCC. Table 4 Correlations Between QOLAPv2 Items and Sample Characteristics. Table 5 Correlations Between BAI Items and Sample Characteristics. Site Differences in Item Endorsement Sites differed markedly in item endorsement in both the magnitude and direction of associations. On the QOLAPv2, Montefiore patients—generally representing lower SES and racial/ethnic minorities—were more likely to define QOL as being in perfect health, feeling calm and peaceful, and having nothing to regret. In contrast to MSKCC patients, Montefiore patients tended to deemphasize the negative things that happened and indicate that other people did not determine their QOL. Rather, their goals related to improving their health by themselves and by getting help from doctors, growing as a spiritual or religious person, increasing their contributions to the community, and learning to live with their physical limitations. They tended to refer to experiences that emphasized the positive to avoid complaining, but also wanted their responses to QOL measures to communicate the seriousness of their situation. The Montefiore sample did not appear to favor a particular standard of comparison. On the BAI, Montefiore patients were more likely to endorse thinking about achieving a calmer, more peaceful, and healthier lifestyle, growing spiritually, preparing their family for the ups and downs of their health condition, settling conflicts with people in their life, and things that come to mind only when completing a survey. In contrast, MSKCC patients showed the opposite pattern on these BAI items, and they were also less likely to endorse a focus on solving problems with the health care system or on finding romance. These findings are consistent with some but not all of our SES-related hypotheses. The supported hypotheses posited that patients with lower SES would focus more on specific problems related to their circumstances and on spiritual growth. Age Differences in Item Endorsement On the QOLAPv2, younger patients were more likely than older patients to define QOL in terms of accomplishments at their job and finding romance and less likely to define QOL in terms of living up to the teachings of their faith, feeling calm and peaceful, and mobility. In terms of goals, younger people were more likely to endorse problems with their living situation, keeping up at work or school, finances, job-related accomplishments, balance between chores and leisure, and getting out of a rut; they were less likely to endorse goals related to staying in their current home. Younger people tended to sample recent experiences and symptom flare-ups, and they were more likely to compare themselves to the kind of life they are really working toward. On the BAI, younger patients were more likely than older patients to endorse thinking about being free of money problems, increasing their volunteer work in the community, having dreams and goals that are different from those of other people their own age, and accomplishing new goals at work. These findings are consistent with some but not all of our age-related hypotheses. The supported hypotheses posited that younger patients would emphasize work roles and independence. Sex Differences in Item Endorsement On the QOLAPv2, women tended to endorse defining QOL in terms of their legacy and living up to the teachings of their faith, but not to define QOL in terms of finding romance. Women tended to endorse goals related to wanting their loved ones to be prepared for the ups and downs of their health condition, whereas men endorsed goals related to improving their relationship with an important person in their life. There were no sex-related patterns of emphasis, experience sampling, or standards of comparison. On the BAI, women tended to endorse solving problems with the health care system, growing spiritually, staying active and productive, spending time with their family before their health worsened, and trying not to complain about their health condition. In contrast, men tended to endorse having dreams and goals that were different from those of most people their age. These findings are consistent with some but not all of our sex-related hypotheses. The supported hypotheses posited that women would emphasize interpersonal relationships, particularly providing support and care for others. Other Demographic Differences in Appraisal Tables 4 and 5 show correlations of QOLAPv2 and BAI items with race, ethnicity, living situation, education, and employment status. Our hypotheses did not directly address these demographic characteristics. However, we observed notable differences by race, ethnicity, education, and employment. These patterns were somewhat similar but not identical to the aforementioned site differences. Living with others was not associated with item-endorsement differences, but stability of living situation (ie, years living in current residence) did exhibit more associations. Construct Validity: Summary Score Structure of the QOLAPv2 and the BAI We conducted a series of PCAs of each of the 5 sections of the QOLAPv2 and the BAI. These analyses paralleled the earlier analysis on the RPV LLC samples, described for aims 1 and 2. However, we decided to examine sample-specific dimensions of appraisal rather than rely on the earlier study to dictate scoring. PCA results in the bladder cancer sample are shown in Tables 6a-6f , including rotated structure coefficients (with major loadings highlighted in green), percentage of variance explained, and interpretation of each component (in the table footnotes). PCAs in the bladder cancer-specific sample were similar but not identical to results in the larger and more heterogeneous RPV population. For example, we retained 7 components in the PCA of the BAI items in the bladder cancer sample, as contrasted with 5 components in the RPV sample. Table 6a Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 1. Definition of QOL. Table 6b Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 2. Combinatory Algorithm—Emphasis. Table 6c Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 3. Personal Goals. Table 6d Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 4. Sampling of Experiences. Table 6e Principal Components Analysis of the QOLAPv2 and the BAI: QOLAPv2 Section 5. Standards of Comparison. Table 6f Principal Components Analysis of the QOLAPv2 and the BAI: BAI. Although 5 of the components were largely equivalent, 2 additional components emerged in the bladder cancer analysis: Considering how you compare to others for this survey and Comparing yourself to social norms . In the more heterogeneous RPV data, items involving normative comparisons were associated with items related to health and independence concerns. In the bladder cancer data, patients had more similar health concerns, which allowed more nuanced differences in consideration of norms to emerge. Redundancy Analysis Results Canonical Correlations We used canonical correlation analyses to examine how well the BAI components captured information from each of the 5 QOLAPv2 sections. Statistically significant canonical correlations were evident in each of these comparisons as detailed in Table 7 . However, shared variance (canonical redundancies) reflecting the ability of the BAI to account for each QOLAPv2 section was relatively low, ranging from 9% to 18%. Table 7 Canonical Redundancy Analysis of the BAI With QOLAPv2 Sections. We then calculated new weighted composites of the QOLAPv2 components, based on the statistically significant canonical correlations from each of the 5 section-specific analyses; doing so enabled us to examine the association of the combined QOLAPv2 with the BAI. This analysis yielded 3 canonical correlations (rc c = 0.89, 0.81, and 0.74, respectively; Wilks' lambda = 0.01, 0.042, and 0.12, respectively; p < 0.001, 0.001, and 0.015, respectively). However, even with the best predictors from each section of the QOLAPv2, the entire set of BAI measures demonstrated low shared variance, or canonical redundancy, with components of the longer measure, explaining only 17% of variance. Although canonical correlations were statistically significant, 17% of shared variance was substantially lower than the level of redundancy we expected between these 2 measures, particularly because BAI was derived from analyses of the QOLAPv2. We decided to conduct further analysis to see whether we could better understand possible relationships between these related measures (QOLAPv2 and BAI). Recall that the BAI was not constructed to have a 1-to-1 correspondence with each section of the QOLAPv2. Rather, the briefer measure was based on a second-order PCA of the QOLAPv2 to capture the most salient patterns of appraisal. Each of the 12 second-order components combined questions about frames of reference, sampling of experiences, standards of comparison, and combinatory algorithms. To ascertain whether we needed to take these complex patterns into account to observe overlap between the QOLAPv2 and BAI, we conducted a series of analyses to account for potential 2-way and higher-level interactions among BAI components. Cluster Analysis to Identify Empirical Patterns of Appraisal To explore all possible combinations of the BAI components, we would have had to test an inordinate number of 3-way and higher-order interaction terms. To simplify this analysis, we identified prevalent combinations of the 7 BAI components using k-means cluster analysis (data not shown). Six clusters emerged with n = 10 to 29 individuals per cluster, defined by the following patterns of high and low scores on the BAI components: Preparing for worsening health; not considering social norms (n = 13) Staying positive; preparing for worsening health; pursuing new goals vs trying not to complain (n = 10) Staying positive; concerned about improving relationships; concerned about how your survey responses compare with others' responses; not concerned about being active and independent (n = 12) Being active and independent; concerned about social norms; trying not to complain; not concerned about improving relationships; not concerned about staying positive (n = 14) Staying positive; not concerned about social norms; not concerned about preparing for worsening health; not concerned about how your survey responses compare with others' responses (n= 17) Not concerned about staying positive; not concerned about preparing for worsening health (n = 29) Additionally, 2 outlier patients did not fit any of these patterns. Regression Analysis to Test Whether Interactions Among BAI Components Improved Redundancy The subsequent series of regressions predicted baseline QOLAPv2 component scores with the BAI canonical correlation scores (main effects), all possible 2-way interactions, and binary codes representing each individual's cluster membership ( Table 8 ). These regression models revealed that the BAI and the QOLAPv2 had an overall average redundancy of 14% based on main effects only. When 2-way interactions were taken into account, they added 12% overall redundancy. The clusters added an additional 2% overall redundancy. Altogether, the main effects, interactions, and clusters had an overall average redundancy of 27.8%. This indicates that the level of redundancy between the 2 measures is higher than can be explained by main effects alone. These findings makes sense, given that the BAI items were written to describe second-order components from the QOLAPv2. Each of the second-order components captures information that arises from several different QOLAPv2 domains. As such, combining these components helps to explain appreciably more variance shared between these 2 appraisal measures. Table 8 Prediction of QOLAPv2 Components Using Main Effects, 2-Way Interactions, and Higher-Order Cluster Patterns Based on BAI Components. Notable relationships exist between each section of the QOLAPv2 and the BAI. However, we had expected that the BAI would explain approximately 50% of the variance of the QOLAPv2. As such, we conclude that the 2 measures are not sufficiently redundant with each other. If the measures were redundant it would suggest that the shorter instrument could be used as a substitute for the longer; however, that is not the case. Rather, analyses suggest that appraisal assessment that distinguishes general appraisal processes from content (QOLAPv2) is distinct from appraisal assessment focused on items that address appraisal processes in the context of different content domains (BAI). For example, if a study were focused on differences in cognitive processes such as recalling life events or making social comparisons, the QOLAPv2 would be the better choice. Alternatively, if the study only needed to make broad distinctions about current concerns, the parsimonious BAI would be a better alternative. In the following section, we examine whether and how well the 2 appraisal measures account for individual differences in QOL related to experiences of changing health and life events. Given our concern in understanding response shifts in patient-reported outcomes, such analyses are key to evaluating these measures. Response Shift Hypotheses Unmeasured individual differences in the criteria that people use to appraise QOL can mask or distort the impact of health changes. For this reason, we directly compared the performance of the BAI with that of the QOLAPv2 in terms of their respective abilities to explain individual differences in QOL and PROs, as predicted by the Rapkin and Schwartz QOL Appraisal Model. This model parses variance of change in QOL that is explained by appraisal alone and variance that is explained by appraisal in combination with other influences. These influences can be antecedents (eg, stable characteristics of the individual) and catalysts (eg, health state changes or other life events). We conducted the analyses as cross-sectional and longitudinal hierarchical multiple regressions (separated out below). These models allowed us to examine directly the ability of appraisal measures to account for variance in QOL that the Standard Model could not explain. 11 For reference, zero-order correlations of the appraisal component scores with the QLQ-C30 well-being, functioning, and symptom scales and Bladder Module (BLM) bladder cancer-specific outcomes are presented in Tables 9 and 10 , respectively. Table 9 Correlations of the QOLAPv2 and the BAI Principal Components With the QLQ-C30. Table 10 Correlations of the QOLAPv2 and the BAI Principal Components With the BLM-30 Scales. We examined 2 classes of predictors expected to influence and interact with appraisal variables: (1) antecedents: demographic or social determinants of health; and (2) catalysts: life experiences and mood state. Note that the Rapkin and Schwartz QOL Appraisal Model also posits that antecedents and catalysts will influence appraisal measures. These relationships are summarized in Table 11 . In this table, we show the overall relationship of each antecedent and catalyst domain with each appraisal component. We also provide the proportion of the total effect for each domain that remains after prior domains are controlled in the model. Table 11 Variance of BAI and QOLAPv2 Components by Antecedents (demographics and treatment site), Catalysts (life experiences), and Mechanisms (mood) at Time 1. Associations of antecedents and catalysts with appraisal components illustrate distinctions among different aspects of appraisal. For example, most of the BAI components are associated primarily with baseline demographic differences. However, life experiences and mood were associated with concerns about preparing for worsening health. Additionally, patients at Montefiore expressed greater concerns about improving relationships and resolving conflicts. Even more nuanced patterns appeared in analyses involving the QOLAPv2 components. For example, stressful life experiences were strongly associated with QOL definition components involving accomplishments at work and family problems. Mood state was associated with definitions of QOL related to being peaceful and worry-free as well as finding love and romance. Demographic antecedents associated with racial and ethnic differences were associated with the view that QOL involved leaving a legacy and adhering to one's faith. Both stressful experiences and mood state were associated with the emphasis that respondents said they placed on experiences involving recent demands when rating their QOL. These and other relationships in Table 11 are consistent with findings from our earlier studies that differences in appraisal mediate associations of antecedents and catalysts with QOL. Cross-sectional Antecedent Models In Table 12 , we show prediction using antecedents, a set of appraisal measures, and their interactions. Each column presents results using different measures of appraisal averaged across 20 PROs. Note that the antecedent measures do not change from analysis to analysis, so the average variance explained is always the same, as shown in the top row of results in Table 12 . The first column of results in this table (labeled BAI) shows average effects in analyses when we used the 5 BAI component scores to describe appraisal. These cross-sectional regression analyses revealed that the full model using the BAI explained an average of 0.292 of PRO variance at baseline. In this instance, antecedents alone accounted for 0.073 of the variance, BAI components added another 0.054 of the variance, and interactions between BAI components and antecedent variables explained an additional 0.164. Moving down the first column of results shows the proportion of total R 2 due to antecedents (30.04%), appraisal (17.33%), and their interactions (52.63%), respectively. Finally, we show the average proportion of the antecedent effect remaining if appraisal measures are controlled first. This provides an indication of the extent to which antecedent effects are direct (62.70%) rather than mediated by appraisal (100%−62.70% = 37.30%). Table 12 Average Variance of 20 PRO Scores at Time 1 Explained by the BAI and QOLAPv2 in Combination With Antecedents (demographics and treatment site). The second column of results in Table 12 (labeled QOLAPv2 Average) shows components of variance explained across the 20 QOL measures, averaging over the 5 QOLAPv2 domains. The remaining columns present components of variance involving each of the QOLAPv2 domains. Full models using each QOLAPv2 domain explained between 0.165 and 0.233 of the variance at baseline (average = 0.19). On average, antecedents alone explained 0.073; main effects for QOLAPv2-assessed appraisal explained between 0.010 and 0.043 (average = 0.026); and appraisal-by-antecedent interactions added from 0.046 to 0.129 (average = 0.091) explained variance to the model. After controlling for appraisal variables, between 72.27% and 99.14% (average = 89.21%) of the variance explained by antecedents remains. The QOLAPv2 domains mediate between 0.86% and 27.73% of the PRO variance associated with antecedents. This compares with the BAI, which mediates 37.3% of the PRO variance associated with antecedents. Sampling of Experiences and Personal Goals are the dominant mediators of antecedent effects on PROs. Cross-sectional Catalyst Models Table 13 is structured similarly to Table 12 . In Table 13 , we show average prediction of 20 PROs using catalysts, a set of appraisal measures, and their interactions. Cross-sectional regression analyses revealed that the full model using the BAI (first column of results) explained an average of 0.353 of PRO variance at baseline. In this instance, catalysts alone accounted for 0.246 of the variance, BAI components added another 0.026 of the variance, and interactions between BAI components and catalysts explained an additional 0.081. Moving down the first column of results shows the proportion of total R 2 due to catalysts (69.76%), appraisal (7.37%), and their interactions (22.87%), respectively. Finally, we show the average proportion of the catalyst effect remaining if appraisal measures are controlled first. This provides an indication of the extent to which catalysts effects are direct (76.45%) rather than mediated by appraisal (100%−76.45% = 23.55%). Table 13 Average Variance of 20 PRO Scores at Time 1 Explained by the BAI and QOLAPv2 in Combination With Catalysts (life experiences) and Mechanisms (moods). The second column of results in Table 13 (labeled QOLAPv2 Average) again shows components of variance explained across the 20 QOL measures, averaging over the 5 QOLAPv2 domains. The remaining columns again present components of variance involving each of the QOLAPv2 domains. Full models using each QOLAPv2 domain explained between 0.294 and 0.345 of the variance at baseline (average = 0.323). On average, catalysts alone explained 0.246; main effects for QOLAPv2-assessed appraisal explained between 0.006 and 0.016 (average = 0.011); and appraisal-by-catalyst interactions added from 0.043 to 0.088 (average = 0.066) explained variance to the model. After controlling for appraisal variables, between 80.51% and 97.44% (average = 89.22%) of the variance explained by catalysts remains. The QOLAPv2 domains mediate between 2.56% and 19.49% of the PRO variance associated with catalysts. This compares with the BAI, which mediates 23.55% of the PRO variance associated with catalysts. Combinatory Algorithm, Personal Goals, and Sampling of Experiences are the dominant mediators of catalyst effects on PROs. In contrast, full models using each section of the QOLAPv2-assessed appraisal explained approximately 32% of the variance at baseline. Of that, catalysts explained 25% alone; main effects for QOLAPv2-assessed appraisal added another 1%; and appraisal-by-catalyst interactions added an average of 7% explained variance to the model. By itself, without taking catalysts into account at all, QOLAPv2-assessed appraisal explained 4% of the variance. Longitudinal Antecedent Models Longitudinal regression analyses examined the interaction of baseline demographics and changes in BAI- vs QOALPv2-assessed appraisal across 20 different PRO outcomes ( Table 14 ). Table 14 Average Variance of Change in 20 PRO Scores Explained by BAI and QOLAPv2 in Combination With Antecedents (demographics and treatment sites). To partition variance appropriately, we conducted hierarchical linear regression analyses, entering baseline PRO, antecedents (measured at baseline only), baseline appraisal, baseline appraisal by antecedent interaction terms, change in appraisal, and, finally, change in appraisal by antecedent interaction terms. In these analyses, baseline PRO explained an average of 0.189 of the variance in PRO change scores. Antecedents alone explained an additional 0.016 of the variance, after adjusting for baseline. As in Tables 12 and 13 , average variance explained by baseline PRO and antecedents prior to entering appraisal measures was identical. The first column of results in Table 14 summarizes BAI-assessed appraisal. In this analysis, the full model explained an average of 0.404 of the variance in PRO change. Of that, BAI-assessed appraisal at baseline added 0.016. Baseline appraisal-by-demographic interactions added an additional 0.09 explained variance. Change in BAI-assessed appraisal (eg, direct response shift) explained 0.01 additional variance in PRO change. Antecedent-by-change-in-appraisal interactions (eg, moderated response shift) explained an additional 0.083 of the variance. Moving down the first column of results, we examine proportion of total R 2 attributed to response shift. The direct response shift effect (change in appraisal) represented 2.44% of R 2 . An additional 20.48% of the total regression was due to moderated response shift (change in appraisal by antecedent interactions). As with Tables 12 and 13 , the second column of results presents average effects across the 5 QOLAPv2 domains of appraisal and the third through seventh columns summarize results for each of the domains. To examine the overall ability of the QOLAPv2 to capture response shift effects, the final column of results in Table 14 shows the maximum variance explained by allowing predictors from all 5 QOLAPv2 domains to enter the regression model. Baseline QOLAPv2 domains accounted for between 0.003 and 0.022, with an average of 0.09. Each section of the QOLAPv2 on its own accounted for about the same amount of variance in QOL as the BAI. The only section of the QOLAPv2 that was markedly superior to the BAI was the Personal Goals section, which accounted for 7% more variance associated with appraisal by Time 1 antecedent interaction. To examine the overall contribution of the QOLAPv2, we combined individual sections by selecting those predictors that entered into the model from each section-specific analysis for each of the 20 QOL outcomes, and then averaged the variance explained across these analyses for each segment of the model. The largest contributions were evident in the interactions between antecedents and Time 1 appraisal and between antecedents and change in appraisal. This result indicates that the different sections of the QOLAPv2 are making unique contributions to explanations of QOL. Consistent with the Rapkin and Schwartz QOL Appraisal Model, this finding suggests that each aspect of appraisal included in the QOLAPv2 plays a distinct role. For example, degree of attention given to recent health changes is separate from the standards of comparison used to evaluate those changes. Note that for both the BAI and the best QOLAPv2 models, similar proportions of total R 2 explained were due to response shift (43% versus 42%, respectively), but overall R 2 explained was much greater for the model using the full QOLAPv2. The longer measure appears to benefit by maintaining distinctions among different aspects of appraisal rather than combining these aspects into more complex global statements. Table 15 further documents the explanatory advantage of the QOLAPv2 over the BAI in accounting for change in QOL. It shows the increment in variance associated with appraisal contributed by each section of the QOLAPv2 over the BAI (ie, if the BAI explains 10% of change in outcomes and a QOLAPv2 section adds another 5%, then that would represent a 50% increment). Table 15a shows the contribution to R 2 by section as well as overall. Tables 15b and 15c explore how these contributions differed, depending on the type of QOL outcome under consideration. Table 15b looks at the proportion of increment in variance added by each section of the QOLAPv2. Table 15c examines the relative information gained per additional item, given the length of each QOLAPv2 section and the additional variance explained by that domain. Table 15 Proportion of Variance Added by QOLAPv2 Components Over BAI Components in Prediction of 20 PRO Scales, in Combination With Antecedents. For example, in the first column in Table 15a , the appraisal domain Definition of QOL contributed an average of 0.080 to variance explained over the BAI, mainly attributable to Time 1 QOL definition by antecedent interactions (0.056). The appraisal domain Personal Goals in the third column contributed an average of 0.112 to explanation of QOL scales. This effect was largely due to the interaction between antecedents and baseline Personal Goals (0.056) as well as change in Personal Goals. In both instances, the QOLAPv2 provided more information than the BAI about differences in the ways that people's gender, age, or racial/ethnic background influence the way they appraise QOL items. Table 15b unpacks the effects in Table 15a by examining whether the QOLAPv2 is more sensitive than the BAI to change in specific QOL domains. For example, Definition of QOL as measured by the QOLAPv2 added more to explanation of change in body image (54.02%) compared with functional scales (6.85%). Personal Goals added substantially to prediction of all QOL domains, particularly sexual functioning. Conversely, QOLAPv2 Sampling of Experiences added only modestly to the prediction of change in most QOL domains beyond the BAI, with the exception of financial burden (61.54%). These findings suggest how tradeoffs involved in using the BAI may have different implications for studying different QOL outcomes. Table 15c takes the analysis in Table 15b one step further, by looking at the average information gained by each QOLAPv2 item. For example, although the 25-item Personal Goals section adds 30.24% to the overall explanation of change in QOL as shown in Table 15b , this translates to only 1.21% per item. Alternatively, the 9 Standards of Comparison items contribute an average of 2.14% per item. These data suggest that optimal explanation of change in QOL domains will require drilling down to understand which appraisal items are most germane. Longitudinal Catalyst Models A final set of longitudinal regression analyses examined the interaction of life events and changes in mood (catalysts) combined with changes in BAI- vs QOLAPv2-assessed appraisal across 20 different QOL outcomes ( Table 16 ). The top portion of the table (“Average of effects in full model”) provides variance explained by each effect in the model out of total variance. The bottom portion (“Partitioning of effects for interpretation” summarizes effects out of R 2 explained. Thus, these different portions of the table distinguish effects calculated using different denominators. These analyses partition variance into more segments than the analyses reported in Table 14 involving antecedents because both catalysts and appraisal are time-changing predictors. Table 16 Average Variance of Change in 20 PRO Scores Explained by the BAI and QOLAPv2 in Combination With Change in Catalysts and Mechanisms. Models involving the BAI and catalysts explained an average of 58.59% of the variance of change in 20 QOL outcomes (section A). Again, 18.9% of the variance was explained by initial QOL score (section A). Time 1 catalysts (ie, baseline cancer-related life events and mood states) explained less than 0.3% of change in QOL, after adjusting for Time 1 QOL (section A). Although Time 1 BAI explained only 1.71% of the average variance in QOL, interactions between Time 1 BAI and Time 1 catalysts explained an average of 7.54% (section B). After controlling for these Time 1 effects, changes in mood or life events accounted for 13.82% of QOL variance (section A). Change in appraisal alone was unrelated to change in QOL, indicating no direct response shift (section B). However, there were relatively large indirect response shifts involving interactions of change in BAI with Time 1 catalysts (9.48%) and change in catalysts (6.87%; section B). As with the analysis of antecedents, results of QOLAPv2 sections were similar to the BAI. The analysis combining best predictions across QOLAPv2 sections again explained substantially more variance than the BAI interaction (75.16%; section A). Interaction effects related to Time 1 catalysts were particularly pronounced. These analyses suggest that life events that occurred preceding Time 1 appear to have a lingering impact on change in QOL depending on individuals' appraisal of QOL (section A). The influence of Time 1 catalysts is affected by both initial appraisal (24.31%) and change in appraisal (11.44%), suggesting that in this 3-month time frame, events prior to the Time 1 interview are as or more influential than events occurring between Times 1 and 2 (section B). Table 17 also summarizes the relative contribution of the QOLAPv2 over the BAI. Increments are attributable to the interactions involving Time 1 catalysts, especially related to Personal Goals, and to a lesser extent, Definition of QOL and Standards of Comparison. The incremental importance of different sections of the QOLAPv2 in explaining the impact of catalysts varied according to types of QOL outcomes. For example, Standards of Comparison items added most to prediction of general cancer symptoms. Sexual functioning was strongly associated with Personal Goals as well as with Standards of Comparison. Urinary functioning was influenced by the particular experiences that individuals sampled and emphasized. Financial burden seemed subject to influences across most of the QOLAPv2 sections. Table 17 Proportion of Incremental Variance in PRO Scales Explained by QOLAPv2 Components Over BAI Components. Discussion Measurement Development Aims The 3 aims of this study were to draw on existing data to determine aspects of QOL appraisal that could be incorporated into a practical measure, to develop a concise instrument, and to test this instrument in a sample of patients with bladder cancer. Work to complete the first 2 study aims, summarized in this report, allowed us to create 2 different instruments, the QOLAPv2 and the BAI. The study of 110 patients with bladder cancer, reported in depth in this report, provided a head-to-head comparison of these 2 new appraisal measures. Both instruments, which were distinct in terms of length and approach, were developed based on empirical findings generated over the past decade and honed in collaboration with our stakeholder panel. The QOLAPv2 measured process and content separately (eg, content specific to frame of reference, experience sampling, standards of comparison, combinatory algorithm). By contrast, the BAI combined content and process based on patterns identified in earlier research. 37 Both measures generally behave in expected ways and have demonstrated some degree of construct validity, including demographic differences, associations with life events, relative independence from mood state, and ability to account for QOL response shifts. 35 , 36 Thus, they provide more practical appraisal measurement than the original QOLAP, 11 which required extensive coding. Enhancing Patient-Centeredness in Patient-Reported Outcomes Research In this 3-month longitudinal study, we found that both measures were able to explain variance in baseline QOL and QOL change over time, and both captured response shift effects. In general, the QOLAPv2 was better at both. The BAI did, however, perform well and could be an acceptable alternative as a practical measure of appraisal in studies where using the QOLAPv2 would not be feasible. The QOLAPv2 overall does appreciably better than the BAI alone in predicting QOL change and capturing response shift effects. As intended, these appraisal measures provide a textured, individual-centered understanding of personal criteria that enter into individuals' ratings of QOL, which standard PRO measures do not capture. These individual criteria are intrinsic to QOL assessment. For that reason, these tools will be useful for amplifying the patient's voice in patient-centered outcomes research and comparative effectiveness research. The primary impetus for this study was to develop methods that could be used in tandem with any kind of evaluative PRO assessment. Appraisal processes apply to any situation in which patients are asked to subjectively evaluate aspects of their health and well-being. We know that people differ in the criteria they use to make these evaluations and that those criteria can change over time as a result of changing health and other catalysts (eg, response shift). Appraisal assessment is intended to describe explicitly those criteria so they can be understood and factored into our interpretation of PROs. In the analyses presented here, we examined whether and to what extent the influence of both antecedents and catalysts were moderated by individual differences associated with appraisal. Appraisal scores might also be used to adjust mean differences in QOL scores associated with appraisal (eg, rating of social role performance given the individual's desire to return to work or community activities). However, such adjustments should be carefully considered. In early response shift research, the presumptive understanding of these differences was that they were a source of “bias” that needed to be adjusted psychometrically. Our appraisal perspective introduced the possibility that these differences were not attributable to bias in measurement, but rather reflected meaningful differences in how people thought. Thus, rather than “adjusting” for response shift to correct estimates of QOL change, our methods enable researchers to identify subgroups of patients who think about their health in different ways. Understanding differences in the personal criteria underlying self-report of PROs is as fundamental to the use and interpretation of QOL measures as knowing whether distance is reported in miles or kilometers or whether an A grade on a report card is for remedial algebra or AP calculus. Taking these differences in meaning into account is especially important in intervention research because aggregate treatment effects may mask important heterogeneity of effects related to differences in appraisal. Individuals' outcomes necessarily depend on the ways that treatments and adverse effects influence what matters to them personally. The benefits of a mutual supportive intervention may be helpful for most people, but not for those who are exclusively focused on family ties or, separately, among those who do not want to identify with others facing a similar health condition. Understanding how people with different perspectives on QOL experience health care interventions is central to supporting decisions about patient-centered care and comparative effectiveness. Appraisal measures may be particularly valuable in clinical contexts, facilitating communication about ways that patients are thinking about and reacting to changes in their health status. With increasing evidence of the validity and feasibility of appraisal measures, we would ultimately recommend that appraisal assessment be considered for use in any situation in which PROs are being examined. We tried to ensure patient-centeredness and relevance to end users of QOL measures by including a diverse stakeholder panel. At the time we formed the stakeholder panel, we planned to involve people with familiarity and expertise as clinicians, policymakers, or researchers using QOL measures. It is the case that 4 of our stakeholders and at least 2 of the investigators have been treated for cancer or other serious chronic illnesses. However, these individuals were not specifically selected for this project to serve a dual role. The nature of the stakeholder panel discussions tended to be technical—involving psychometric methods, measurement theory, and models of QOL. We tried to directly incorporate the patient perspective into this study by drawing on our multiple studies over the past decade using the original QOLAP, which asked patients questions about what matters to them. In addition, our earlier research with patients who have bladder cancer included a section that asked them to comment on the relevance, value, and burden of appraisal assessment. Although some people found that measure to be lengthy, they also found questions about appraisal to be highly relevant. This was part of the impetus for developing briefer measures. We will continue to solicit lay input on these measures as we use them in the field. Another open question concerning patient-centeredness involves the implications of using briefer quantitative appraisal measures. We must acknowledge the tradeoff in this regard: Most patients like the opportunity to talk about their experiences afforded by the original QOLAP qualitative interview. Many say they found the interview helpful as a way of reflecting on their priorities. Although the purely quantitative QOLAPv2 and BAI ask about the same content, they are necessarily less engaging than the original QOLAP interview. When time and resources permit, we would recommend adding some opportunity for patients to reflect on the specific concerns and experiences that entered into their self-evaluation of QOL. Giving patients the opportunity to tell their story will aid in the interpretation of results and improve our ability to give voice to patients' perspectives. The value of practical measures to assess appraisal is the potential to assess this information in a routine and structured manner. Patients' perspectives could be obtained by lay health workers, removing the burden from physicians and other allied health professionals. Working With Appraisal Measures This study has dealt primarily with methodological challenges in assessing appraisal in a way that is parsimonious and feasible. The question remains: How can and should appraisal methods be incorporated into research on patient-reported outcomes? On the practical front, incorporating a 1-page measure like the BAI into most studies would be relatively easy, particularly if it could be administered online or by computer-assisted interview. The greater practical matters involve scoring and analysis of the measure, which we will address in turn. PRO researchers are highly familiar with evaluative scales that tend to converge into fewer dimensions in ways that are relatively consistent across measures and dictated by semantics and valence (eg, physical, emotional). Appraisal items are not constrained in the same way as evaluative items. The QOLAPv2 and especially the BAI were purposively developed to be broad-bandwidth instruments, meaning that items sample many issues and considerations that may describe how individuals have been thinking about QOL. For the sake of brevity, measures do not include multiple items measuring the salience of each issue or concern. As such, appraisal items are not expected to converge or cohere in the same ways that evaluative scales do. In some ways, they are more akin to behavioral inventories to assess food consumption or social activity. An item that asks about frequency of going to the movies may or may not converge with frequency of having friends over for dinner. Whether these items do or do not converge is not solely dictated by item content. Rather, it involves the context of these behaviors. Similarly, reduction of appraisal measures will be influenced by population considerations. Appraisal items may or may not converge, depending on sample composition and such influences as the extent of chronic illness, social class, age norms, and gender roles. For example, in a sample where all people are living with an active chronic disease, there may be no association between concerns about health state and those about patient-provider communications. In another sample including individuals with heterogeneous health states, concerns about health and patient-provider relationships may be highly correlated. Of course, the former reflects a restriction of range in health states, but in this case such restriction is clearly warranted and ecologically valid. Although we have so far relied on sample-specific reduction of the 2 new appraisal measures, we will likely be in a better position to derive context-specific scoring algorithms, taking into account meaningful differences in the composition of samples, after examining the BAI and QOLAPv2 in a wider range of populations. We encourage investigators wishing to make use of these appraisal measures to conduct and share their own PCAs or the equivalent. We have also considered but have not yet explored the idea of using individual BAI items to test specific hypotheses regarding such factors as concerns about impact of illness on the family, return to work, or spiritual goals. Regardless of how measures are selected and scored, analysis of appraisal in PRO studies poses a separate set of challenges. In the Background section of this report, we provided a general analytic framework based on the QOL Appraisal Model, using multiple regression analysis. The logic of partitioning variance due to Standard Model domains (antecedents, catalysts, and mechanisms) from variance due to appraisal is particularly suited to isolating and examining response shift effects. This same model has been applied to observational studies, including the analyses presented in this report, by testing interaction effects involving between-appraisal measures and changes in health status or the occurrence of life events. Individual differences in intervention effects attributable to appraisal have been modeled in a similar manner. Given our roots in the response shift literature, we focused primarily on examining whether and to what extent individual differences in cognition add to the explanation of PROs. However, the QOL Appraisal Model suggests alternative analytic approaches that could be readily incorporated into both observational and interventional research. For example, one useful step may be to start an analysis by examining individual differences in baseline appraisal, hypothesized a priori to moderate intervention effects or the impact of events. Considering how effects of antecedents and catalysts might be mediated by appraisal may also be informative. As our demographic analyses suggest, differences in group culture or norms might be reflected in the ways that people appraise QOL. Potential Barriers to the Uptake of Study Results Despite the substantial body of evidence based on rigorous research to support QOL appraisal assessment, clear barriers exist to the uptake and appropriate use of the new appraisal measures. Three are particularly important: Researcher skepticism: Some PRO researchers may be skeptical about whether individual differences in appraisal matter. In both population-cohort and interventional studies, such differences have long been treated as error variance that can be ignored using statistical approaches such as randomization or covariate adjustment. Pressure to shorten PRO measures: In many quarters, the pressure to shorten PRO measures is strong. Adding appraisal measures may be seen as counter to this goal of parsimony. Preference for prespecified over individualized measures: Among developers of legacy or even computerized adaptive PRO measures, there may be a reluctance to embrace measures that highlight individual differences when their own measures appear to offer a common solution for patient populations. Perhaps most fundamentally, acknowledgment of response shift may call into question the interpretation of a considerable volume of research addressing QOL. For example, findings suggesting few differences related to disease states or intervention conditions may need to be reexamined in light of changes in appraisal. We believe that taking response shift into account represents a paradigm shift in QOL assessment; we see this as a core implication of what it means to be patient-centered. Work on PROs predates the emerging emphasis on patient-centeredness in both care and research. Ironically, most PRO measures are not designed to elicit the patients' perspective on what matters for their quality of life. Current and prior research on the assessment of appraisal helps to bridge and unify these areas. Even if the new measures are easy to administer, they still require analytic methods that may be unfamiliar to end users. Users of PRO measures are most familiar with simple additive scoring algorithms. However, we have found that appraisal assessment generally requires empirical item reduction (eg, PCA). This approach yields summary scores that are specific to a given study sample, rather than applying to all relevant contexts and patient populations. For example, in the heterogeneous RPV sample discussed for aim 1 second-order components, the fact that presence of health worries and normative comparisons about health co-vary makes sense, because individuals with greater health worries would also be the ones thinking about health-relevant comparisons. In the aim 3 bladder cancer sample, health worries were likely more pervasive. This “restriction of range” likely reduced the correlation between worries and norms items, so they loaded on separate components. We do not interpret this to be a statistical artifact; rather, the focus on a specific group allowed differences to emerge in how patients with bladder cancer referenced social norms in evaluating their QOL. More generally, we need more experience using the BAI and QOLAPv2 in larger, more diverse samples, including healthy populations, to make recommendations about the best way to summarize and reduce the length of these instruments in different situations. Items that are highly relevant to people dealing with chronic illness may not be applicable to healthier segments of the population. Thus, the connections among different aspects of appraisal domains likely depend on the mix of health states represented in a particular sample. Similar issues about the age, sex, gender roles, and socioeconomic composition of samples may arise. We do not want to settle prematurely on a single way of scoring these appraisal measures because of the risk of missing important patterns specific to health states. With further research, we may uncover similarities across patient populations based on prognosis, degree of functional limitation, or type of limitations. Although this kind of contingent scoring 54 is unfamiliar, it may become a necessary and desirable feature of all patient-centered outcomes research. Subpopulation Considerations As with all patient-centered methods, the study of appraisal raises the expectation that the QOL impact of illness and treatment will vary as a function of individuals' priorities, concerns, and values, which in turn arise from personal history and current circumstances. If these contextual considerations truly matter, then they have marked implications for research using appraisal measures and other patient-centered methodologies. Of course, one remedy would be to gather data on much larger samples to permit analyses of subgroups defined by their patterns of appraisal and response shift. This may be highly impractical, however, because of cost as well as the availability of patients with all but the most prevalent health conditions. An alternative may be the use of meta-analytic methods to establish effects related to appraisal across multiple samples. For example, the Directors of Symptom Science Centers supported by the National Institute of Nursing Research recently offered recommendations for development of big data sets for symptom science. 55 In consultation with Rapkin, these directors recommended including appraisal measures along with standard PRO assessment in electronic medical records and other data sources that might be linked to examine the impact of illness and treatment on symptoms. 55 Wide-scale dissemination and coordination of research using patient-centered tools such as the QOLAPv2 and the BAI could be facilitated through PCORI's PCOR-NET initiative and through cooperative clinical trials groups, such as the National Cancer Institute's Community Oncology Research Program (NCORP). One concern during the development of the QOLAPv2 and the BAI was our ability to administer the measures in populations that were diverse in terms of gender, age, race, ethnicity, and socioeconomic status. Our original appraisal assessment included qualitative methods that allowed individuals' unique concerns and ways of thinking to emerge. We tried to capture these differences in developing the QOLAPv2 item pool. We were aware that some items would not be as relevant to some segments of the population as to other segments (such as travel for leisure or neighborhood safety). The present study intentionally gathered data from cancer centers serving markedly different patient populations. As expected, site differences were explained largely by antecedents related to social determinants of health. Study Limitations The work done in this project represents a major milestone: the development of 2 new appraisal measures and the opportunity to compare them empirically in a single sample. Nonetheless, the study limitations must be acknowledged. First, the study sample for aim 3 was relatively small and focused on a single disease (ie, bladder cancer). Second, to establish relationships between these measures and examine several sets of antecedents and catalysts over time, we had to do several analyses. We attempted to manage the potential for overinterpreting chance relationships in various ways, including use of multivariate significance tests, data reduction, and combining and reporting findings aggregated across 20 distinct QOL subscales. Third, the follow-up period of 3 months is relatively short, particularly for detecting changes in life experiences and clinical health (catalysts). For this reason, response shift effects that we detected ought to be considered a conservative estimate. Following patients over longer periods of time would overcome this limitation, allowing us to see more change in catalysts and, therefore, more potential for response shift effects. Fourth are concerns about the generalizability of findings, particularly in a small sample. For summarizing appraisal and response shift effects, we reported average regression across multiple dependent variables. Results that pertain to specific outcomes may vary. We will break out aggregate findings to report in future papers focusing on specific PROs. Of course, the most valuable context for interpreting these results will be in the context of further psychometric and validation analysis in new samples. As we have noted, the magnitudes and directions of appraisal and response shift findings reported here conform to hypotheses that were themselves based on earlier research findings. Forthcoming studies in new, very heterogeneous samples are now under way, as described below. In several years, we will be in a position to conduct a meta-analysis of appraisal and response shift differences, including meta-regression examining how study and sample characteristics influence relevant effect sizes. Meta-analyses will also be useful to determine whether certain PRO instruments are more or less subject to appraisal effects. For example, in our original multiple sclerosis data set, we found that Neuro-QOL measures developed using item response theory to minimize differential item functioning (DIF) were still subject to individual differences in appraisal, although less so than the PROMIS Global-10 or the Ryff functioning measures that were created based on classical test theory, without regard to DIF. Finally, we faced difficulty in scoring the cognitive interview data, which forced us not to use these data for establishing criterion validity. We believe that the problem with inter-rater reliability could be attributed mainly to the complexity of the coding system that we developed (see Appendix F for the coding manual). We had numerous specific codes derived directly from the QOLAPv2; many codes appeared with very low frequency, which might have made it difficult for raters to gain sufficient familiarity with them. Despite the problems with coding, the cognitive interview data are of high quality. Our plan is to devise a simpler coding system to examine the relationship of this criterion measure to the BAI and QOLAPv2. We attempted to ensure that our analyses were hypothesis-driven and conservative in several ways: We summarized measures of appraisal and catalysts using orthogonal PCA, reducing the number of tests required and limiting multicollinearity. We tested appraisal effects only after removing variance explained by all other potential predictors. We kept only those predictors with significance of 0.01 or lower for regression models. We focused on explaining variance to understand properties of our measures. Future use of these new appraisal measures will enable us to see whether and how these findings generalize to other settings and other patient populations. Future Research This study is best understood as part of an extensive program of research that began a decade before PCORI funding and will continue afterward. Following are 5 specific areas that warrant further attention. Although our team is already undertaking some of this research, we hope and expect that dissemination of these new appraisal instruments will encourage other research groups to get involved. As in past years, we presented our work at the 2018 International Society for Quality of Life (ISOQOL) Research meeting in Dublin, including 1 plenary, 1 symposium, 2 posters, and a special presentation to the Response Shift Special Interest Group. In addition, work on QOL appraisal was discussed in a recent article on mentorship opportunities, written by Dr. Schwartz in the winter 2018 edition of the ISOQOL newsletter. 56 Conducting Further Work on the Measurement of Appraisal Study findings show that the BAI does not fully capture QOLAPv2 information. Analysis suggests that measures-of-appraisal processes (eg, in QOLAPv2 sections 2, 4, and 5) supplement the content-oriented items on the BAI. In subsequent research, we will administer the BAI in conjunction with a subset of items from the QOLAPv2 to determine the relative value of adding these QOLAPv2 items. Reducing the number of BAI items further, based on empirical findings, may also be possible. Ideally, we would like to derive a brief measure that retains the theoretical structure with a focus on cognitive appraisal processes as well as relevant content. Other investigators may be inspired to examine appraisal processes using methods of their own design. Alternative ways of eliciting individuals' relevant experiences could be a valuable starting point, perhaps using approaches such as momentary ecological assessment, photo-voice, or expressive writing. One very important step would be to determine the convergence among markedly different approaches to examine how people think about QOL. It is possible that different methods do not converge. Rather, they may shed light on different ways people understand and evaluate QOL. Implementing Appraisal Assessment as a Standard in QOL Research We are working with members of our stakeholder panel and others to test the BAI in several new populations: Danish patients with myeloma A population-based sample of residents in Bronx, New York A study, in conjunction with the NCORP, of financial toxicity experienced by patients with leukemia and myeloma A sample of hemophilia caregivers in North America A sample of patients with hemophilia and their caregivers in 6 European countries A longitudinal QOL study of patients who have undergone hip replacement in Toronto, Ontario A longitudinal study of chronically ill patients and their caregivers sampled from the RPV website that we worked with in our earlier steps to develop appraisal instruments We are particularly interested in testing new appraisal measures in the context of comparative effectiveness intervention trials. We believe that investigators can use QOL appraisal to specify hypotheses regarding how different interventions would affect different individuals who think about QOL in different ways. In this sense, appraisal assessment can play a central role in comparative effectiveness studies by helping match individuals to interventions based on their personal criteria for QOL. As a basis for informing future practice, intervention studies should determine whether outcomes are contingent on baseline appraisal. Researchers may want to test whether matching people to interventions based on appraisal is more informative than using preference measures, particularly in situations like bladder cancer, when patients facing cystectomy do not have prior personal experience of living with one or another reconstruction. Using Appraisal as an Outcome As we have gained experience using appraisal assessment, we have become aware of instances in which the goals of interventions involve modifying people's ways of thinking about their health and life circumstances. Such interventions include efforts to support making medical decisions, planning end-of-life care, raising awareness of health risk behaviors, and adapting to irreversible loss of function. The baseline assessment could help to target cognitive-behavioral interventions. We might expect such interventions to modify people's frame of reference, help them find more reasonable standards of comparison, or shift emphasis to more controllable aspects of their situation. Applying these measures as intervention research outcomes would be worthwhile. Incorporating Appraisal Assessment Into Clinical Care Recently, health care delivery systems across the United States and Canada have encouraged providers to identify and address each patient's specific priorities through a program titled “What Matters to You?” 53 This and other patient-centered programs recognize the importance of factoring individuals' priorities and circumstances into health care delivery. QOL appraisal assessment could be readily incorporated into such programs. In our own work, we have developed and pilot-tested interventions to facilitate advance care planning and to support adherence to daily radiation therapy. More generally, use of appraisal assessment can facilitate patient-provider communication by helping clarify patients' perspectives and assumptions. A very interesting study could examine how well providers (or family caregivers) understand individuals' priorities for QOL and their ways of expressing their concerns. Accurate recognition of patients' appraisals of QOL may serve as a gauge of clinician empathy. Developing QOL Theory Development of theory related to QOL appraisal has been impeded by challenges in measurement. Before development of the QOLAPv2 and the BAI, description of appraisal depended on an interview that was challenging to administer and code. Although our work has shown that measures of appraisal consistently help explain the impacts of illness and treatment on PROs, many questions remain about the way appraisal processes come into play. For example, how do different aspects of appraisal affect different QOL domains such as social function or experience of pain? How rapidly does appraisal change and how stable or fluid are these changes in response to health changes? How do social processes such as social support, patient-provider communication, and communication via multiple channels influence appraisal? How aware are people of their own ways of appraising QOL and are these subject to personal control? How do specific assessment settings and formats influence people's QOL appraisal? With the introduction of more portable and user-friendly measures of appraisal, these and other questions will be much easier to address. Conclusions The measurement of QOL appraisal provides direct assessment of the psychological criteria that people use when rating patient-reported outcome measures, relevant to all manner of outcomes research. This study provides evidence supporting the internal consistency, validity, and usefulness of both the QOLAPv2 and the BAI as measures of QOL appraisal. Both measures were able to account for significant variance in individual differences and intra-individual change across a wide spectrum of patient-reported outcomes. Both instruments demonstrated expected relationships with demographic measures of social determinants of health, recent life experiences, mood state, and health-related QOL. Less than expected convergence between the 2 measures appeared to be due to differences in how items were written. Convergence increased when patterns of items were considered. As expected, the lengthier QOLAPv2 accounted for greater variance in QOL outcomes than the BAI. Despite the superior psychometric performance of the longer QOLAPv2 and limited redundancy, the BAI still functioned reasonably well as a measure of appraisal. At this writing, we would recommend that most studies interested in including a measure of appraisal use the BAI because of its brevity and relative ease in scoring. We will continue to evaluate both measures in other contexts. In particular, we will examine cognitive interviews to see whether those shed light on the comparative validity of these instruments. This report provides an overview of results of numerous regression analyses based on a small sample of patients with bladder cancer. Further research will strengthen our ability to extrapolate from the present data set. Reproducibility of findings in independent samples using different measures of appraisal would provide additional support for validity of the BAI and QOLAPv2. Study findings support further use of these measures by stakeholders concerned with cognitive and motivational differences in patients' responses to illness and treatment. The BAI and QOLAPv2 will be especially useful for research stakeholders concerned with accurately understanding how interventions affect QOL. Such appraisal measures offer a systematic way for stakeholders to design and target interventions that take patients' current concerns and perspectives into account. References 1. Andrykowski MA, Brady MJ, Hunt JW. Positive psychosocial adjustment in potential bone marrow transplant recipients: cancer as a psycholsocial transition. Psycho-Oncology. 1993;2:261-276. 2. Breetvelt IS, Van Dam FS. Underreporting by cancer patients: the case of response-shift. Soc Sci Med. 1991;32(9):981-987. [ PubMed : 2047902 ] 3. Bach JR, Campagnolo DI, Hoeman S. Life satisfaction of individuals with Duchenne muscular dystrophy using long-term mechanical ventilatory support. Am J Phys Med Rehabil. 1991;70(3):129-135. [ PubMed : 2039614 ] 4. Cassileth BR, Lusk EJ, Tenaglia AN. A psychological comparison of patients with malignant melanoma and other dermatologic disorders. Am Acad Dermatol. 1982;7:742-746. [ PubMed : 7174911 ] 5. Groenvold M, Fayers PM, Sprangers MA, et al. Anxiety and depression in breast cancer patients at low risk of recurrence compared with the general population: a valid comparison? J Clin Epidemiol. 1999;52(6):523-530. [ PubMed : 10408991 ] 6. Stensman R. Severely mobility-disabled people assess the quality of their lives. Scand J Rehabil Med. 1985;17:87-89. [ PubMed : 3161178 ] 7. Albrecht GL, Devlieger PJ. The disability paradox: high quality of life against all odds. Soc Sci Med. 1999;48(8):977-988. [ PubMed : 10390038 ] 8. Kagawa-Singer M. Redefining health: living with cancer. Soc Sci Med. 1993;37:295-304. [ PubMed : 8356479 ] 9. Padilla GV, Mishel MH, Grant MM. Uncertainty, appraisal and quality of life. J Patient Rep Outcomes. 1992;1(3):155-165. [ PubMed : 1301125 ] 10. Wilson IB. Clinical understanding and clinical implications of response shift. Soc Sci Med. 1999;48(11):1577-1588. [ PubMed : 10400258 ] 11. Rapkin BD, Schwartz CE. Toward a theoretical model of quality-of-life appraisal: implications of findings from studies of response shift. Health Qual Life Outcomes. 2004;2(1):14. [ PMC free article : PMC408464 ] [ PubMed : 15023229 ] 12. Jobe JB. Cognitive psychology and self-reports: models and methods. J Patient Rep Outcomes. 2003;12:219-227. [ PubMed : 12769134 ] 13. Rabiau M, Knauper B, Miquelon P. The eternal quest for optimal balance between maximizing pleasure and minimizing harm: the compensatory health beliefs model. Br J Health Psychol. 2006;11:139-153. [ PubMed : 16480560 ] 14. Alfano CM, Day JM, Katz ML, et al. Exercise and dietary change after diagnosis and cancer-related symptoms in long-term survivors of breast cancer: CALGB 79804. Psycho-Oncology. 2009;18(2):128-133. [ PMC free article : PMC3934632 ] [ PubMed : 18536022 ] 15. Cohen S, Frank E, Doyle WJ, Skoner DP, Rabin BS, Gwaltney JM Jr. Types of stressors that increase susceptibility to the common cold in healthy adults. Health Psychol. 1998;17(3):214-223. [ PubMed : 9619470 ] 16. Mezuk B, Rafferty JA, Kershaw KN, et al. Reconsidering the role of social disadvantage in physical and mental health: stressful life events, health behaviors, race, and depression. Am J Epidemiol. 2010;172(11):1238-1249. [ PMC free article : PMC3025628 ] [ PubMed : 20884682 ] 17. Suls J, Fletcher B. Self-attention, life stress, and illness: a prospective study. Psychosom Med. 1985;47(5):469-481. [ PubMed : 4059480 ] 18. Allison PJ, Locker D, Feine JS. Quality of life: a dynamic construct. Soc Sci Med. 1997;45(2):221-230. [ PubMed : 9225410 ] 19. Bernhard J, Hurny C, Maibach R, Herrmann R, Laffer U. Quality of life as subjective experience: reframing of perception in patients with colon cancer undergoing radical resection with or without adjuvant chemotherapy. Swiss Group for Clinical Cancer Research (SAKK). Ann Oncol. 1999;10(7):775-782. [ PubMed : 10470423 ] 20. Hoeymans N, Feskens EJ, Kromhout D, van den Bos GA. Ageing and the relationship between functional status and self-rated health in elderly men. Soc Sci Med. 1997;45(10):1527-1536. [ PubMed : 9351142 ] 21. Stanton AL, Danoff-Burg S, Cameron CL, Snider PR, Kirk SB. Social comparison and adjustment to breast cancer: an experimental examination of upward affiliation and downward evaluation. Health Psychol. 1999;18(2):151-158. [ PubMed : 10194050 ] 22. Rabiau M, Knauper B, Miquelon P. The eternal quest for optimal balance between maximizing pleasure and minimizing harm: the compensatory health beliefs model. Br J Health Psychol. 2006;11(pt1):139-153. [ PubMed : 16480560 ] 23. Wyrwich KW, Tardino VMS. Understanding global transition assessments. J Patient Rep Outcomes. 2006;15(6):995-1004. [ PubMed : 16900280 ] 24. Li Y, Rapkin B. Classification and regression tree uncovered hierarchy of psychosocial determinants underlying quality-of-life response shift in HIV/AIDS. J Clin Epidemiol. 2009;62(11):1138-1147. [ PMC free article : PMC2754600 ] [ PubMed : 19595576 ] 25. Li Y, Schwartz CE. Data mining for response shift patterns in multiple sclerosis patients using recursive partitioning tree analysis. Qual Life Res. 2011;20(10):1543-1553. [ PubMed : 21909992 ] 26. Schwartz CE, Snook E, Quaranto B, Benedict RH, Vollmer T. Cognitive reserve and patient-reported outcomes in multiple sclerosis. Mult Scler. 2013;19(1):87-105. [ PubMed : 22546847 ] 27. Schwartz CE, Finkelstein JA. Understanding inconsistencies in patient-reported outcomes after spine treatment: response shift phenomena. Spine J. 2009;9(12):1039-1045. [ PubMed : 19574107 ] 28. Schwartz CE, Sajobi TT, Lix LM, Quaranto BR, Finkelstein JA. Changing values, changing outcomes: the influence of reprioritization response shift on outcome assessment after spine surgery. Qual Life Res. 2013;22(9):2255-2264. [ PubMed : 23519975 ] 29. Morganstern BA, Bochner B, Dalbagni G, Shabsigh A, Rapkin B. The psychological context of quality of life: a psychometric analysis of a novel idiographic measure of bladder cancer patients' personal goals and concerns prior to surgery. Health Qual Life Outcomes. 2011;9:10. [ PMC free article : PMC3045868 ] [ PubMed : 21324146 ] 30. Anderson CB, Rapkin B, Reaves BC, et al. Idiographic quality of life assessment before radical cystectomy. Psychooncology. 2017;26(2):206-213. [ PMC free article : PMC5540244 ] [ PubMed : 26620583 ] 31. Surveillance, Epidemiology, and End Results Program (SEER). Cancer stat facts: bladder cancer. NIH National Cancer Institute website. Accessed February 27, 2020. http://seer ​.cancer.gov ​/statfacts/html/urinb.html 32. Van Der Aa MN, Steyerberg EW, Sen EF, et al. Patients' perceived burden of cystoscopic and urinary surveillance of bladder cancer: a randomized comparison. BJU Int. 2008;101(9):1106-1110. [ PubMed : 17888042 ] 33. Zietman A, Skinner E. Quality of life after radical treatment for invasive bladder cancer. Semin Radiat Oncol. 2005;15(1):55-59. [ PubMed : 15662608 ] 34. Porter MP, Wei JT, Penson DF. Quality of life issues in bladder cancer patients following cystectomy and urinary diversion. Urol Clin North Am. 2005;32(2):207-216. [ PubMed : 15862618 ] 35. Rapkin BD, Garcia I, Michael W, Zhang J, Schwartz CE. Development of a practical outcome measure to account for individual differences and temporal changes in quality-of-life appraisal: the Brief Appraisal Inventory. J Patient Rep Outcomes. 2018;27:823-833. [ PubMed : 29127597 ] 36. Rapkin BD, Garcia I, Michael W, Zhang J, Schwartz CE. Distinguishing appraisal and personality influences on quality of life in chronic illness: introducing the Quality-of-Life Appraisal Profile version 2. J Patient Rep Outcomes. 2017;26:2815-2829. [ PubMed : 28593531 ] 37. Rapkin BD, Schwartz CE. Distilling the essence of appraisal: a mixed methods study of people with multiple sclerosis. Qual Life Res. 2016;25(4):793-805. [ PubMed : 26342930 ] 38. Schwartz CE, Finkelstein JA, Rapkin BD. Appraisal assessment in patient-reported outcome research: methods for uncovering the personal context and meaning of quality of life. J Patient Rep Outcomes. 2017;26(26):545-554. [ PubMed : 27988907 ] 39. Schwartz CE, Li J, Rapkin BD. Refining a web-based goal assessment interview: item reduction based on reliability and predictive validity. J Patient Rep Outcomes. 2016;25(9):2201-2212. [ PubMed : 26961007 ] 40. Schwartz CE, Powell VE, Rapkin BD. When global rating of change contradicts observed change: examining appraisal processes underlying paradoxical responses over time. J Patient Rep Outcomes. 2017;26:847-857. [ PubMed : 27722830 ] 41. Rapkin BD, Garcia I, Michael W, Zhang J, Schwartz CE. Development of a practical outcome measure to account for individual differences and temporal changes in quality-of-life appraisal: the Brief Appraisal Inventory. J Patient Rep Outcomes. 2017;In press. [ PubMed : 29127597 ] 42. Schwartz CE, Zhang J, Michael W, Eton DT, Rapkin BD. Reserve-building activities attenuate treatment burden in chronic illness: the mediating role of appraisal and social support. Health Psychol Open. 2018;5(1):2055102918773440. doi:10.1177/2055102918773440 [ PMC free article : PMC5954584 ] [ PubMed : 29785278 ] [ CrossRef ] 43. Rammstedt B, John OP. Measuring personality in one minute or less: a 10-item short version of the Big Five Inventory in English and German. J Res Pers. 2007;41:203-212. 44. Aaronson NK, Ahmedzai S, Bergman B, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. 1993;85(5):365-376. [ PubMed : 8433390 ] 45. Sprangers M, Cull A, Groenvold M, Bjordal K, Blazeby J, Aaronson NK. The European Organization for Research and Treatment of Cancer approach to developing questionnaire modules: an update and overview. J Patient Rep Outcomes. 1998;7(4):291-300. [ PubMed : 9610213 ] 46. Mayer JD, Gaschke YN. The experience and meta-experience of mood. J Pers Soc Psychol. 1988;55(1):102. [ PubMed : 3418484 ] 47. Fillion L, Kohn P, Gagnon P, Wijk MV, Cunningham A. The Inventory of Recent Life Experiences for Cancer Patients (IRLE-C): a decontaminated measure of cancer-based hassles. Psychol Health. 2001;16(4):443-459. 48. Bloem EF, van Zuuren FJ, Koeneman MA, et al. Clarifying quality of life assessment: do theoretical models capture the underlying cognitive processes? J Patient Rep Outcomes. 2008;17:1093-1102. [ PubMed : 18704756 ] 49. Taminiau-Bloem E, van Zuuren F, Visser M, et al. Opening the black box of cancer patients' quality-of-life change assessments: a qualitative study examining the cognitive processes underlying responses to transition items. Psychol Health. 2011;26(11):1414-1428. [ PubMed : 21736499 ] 50. Taminiau-Bloem EF, van Zuuren FJ, Koeneman MA, et al. A “short walk” is longer before radiotherapy than afterwards: a qualitative study questioning the baseline and follow-up design. Health Qual Life Outcomes. 2010;8(1):1. [ PMC free article : PMC2915972 ] [ PubMed : 20637086 ] 51. G*Power [computer program]. Version 3.1.9.2. Dusseldorf, Germany: Heinrich Heine University, 1992-2014. 52. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Erlbaum; 1988. 53. American Cancer Society. Cancer Facts & Figures 2017. Accessed November 29, 2017. https://www ​.cancer.org ​/content/dam/cancer-org ​/research/cancer-facts-and-statistics ​/annual-cancer-facts-and-figures ​/2017/cancer-facts-and-figures-2017.pdf 54. Schwartz CE, Rapkin BD. Reconsidering the psychometrics of quality of life assessment in light of response shift and appraisal. Health Qual Life Outcomes. 2004;2:16. [ PMC free article : PMC408465 ] [ PubMed : 15038830 ] 55. Redeker NS, Anderson R, Bakken S, et al. Advancing symptom science through use of common data elements. J Nurs Scholarsh. 2015;47(5):379-388. [ PMC free article : PMC4618317 ] [ PubMed : 26250061 ] 56. Schwartz CE. Unexpected benefits. Quality of Life Quarterly. 2018;24(1):9-10. Accessed February 27, 2020. https://www ​.isoqol.org ​/wp-content/uploads ​/2019/09/FinalNewsletter_March18-v24i1 ​.pdf Related Publications Rapkin BD, Schwartz CE. Distilling the essence of appraisal: a mixed-methods study of people with multiple sclerosis. J Patient Rep Outcomes. 2016;25(4):793-805, 807-810. [ PubMed : 26342930 ] Schwartz CE, Li J, Rapkin BD. Refining a web-based goal assessment interview: item reduction based on reliability and predictive validity. J Patient Rep Outcomes. 2016;25(9):2201-2212.1-12. [ PubMed : 26961007 ] Schwartz CE, Powell VE, Rapkin BD. When global rating of change contradicts observed change: examining appraisal processes underlying paradoxical responses over time. J Patient Rep Outcomes. 2017;26:847-857. [ PubMed : 27722830 ] Schwartz CE, Finkelstein JA, Rapkin BD. Appraisal assessment in PRO research: methods for uncovering the personal context and meaning of quality of life. J Patient Rep Outcomes. 2017;26:545-554. [ PubMed : 27988907 ] Rapkin BD, Garcia I, Michael W, Zhang, J, Schwartz CE. Distinguishing appraisal and personality influences on quality of life in chronic illness: introducing the Quality-of-Life Appraisal Profile version 2. J Patient Rep Outcomes. 2017;26:2815-2829. [ PubMed : 28593531 ] Schwartz CE, Michael W, Rapkin BD. Resilience to health challenges is related to different ways of thinking: mediators of quality of life in a heterogeneous rare-disease cohort. J Patient Rep Outcomes. 2017;26:3075-3088. [ PubMed : 28660463 ] Rapkin BD, Garcia I, Michael W, Schwartz CE. Development of a practical outcome measure to account for individual differences in quality-of-life appraisal: the Brief Appraisal Inventory. J Patient Rep Outcomes. 2017;27:823-833. [ PubMed : 29127597 ] Schwartz CE, Zhang J, Michael W, Eton DT, Rapkin BD. Reserve-building activities attenuate treatment burden in chronic illness: the mediating role of appraisal and social support. Health Psychol Open. 2018;5(1):2055102918773440. doi:10.1177/2055102918773440 [ PMC free article : PMC5954584 ] [ PubMed : 29785278 ] [ CrossRef ] Acknowledgments We are grateful to the members of our stakeholder panel: Joanne Buzaglo, PhD, Vice President for Education and Research, Cancer Support Community, Philadelphia, Pennsylvania; Joel Finkelstein, MD, Orthopaedic Surgeon, Sunnybrook Health Sciences Centre and University of Toronto, Canada; Mitch Golant, PhD, Senior Consultant, Strategic Initiatives, Washington, DC; Nicole Hollingsworth, EdD, Assistant Vice President, Community and Population Health, Montefiore Medical Center, Bronx, New York; Wendy Kahalas, MA, Evaluation Specialist, New York State Department of Health AIDS Institute, Bronx, New York; Maureen E. Lyon, PhD, Child and Adolescent Psychologist, Children's National Medical Center, Washington, DC; Sherry Schachter, PhD, RN, Director of Bereavement Services, Calvary Hospital, Bronx, New York; Mirjam Sprangers, PhD, Professor, Medical Psychology, University of Amsterdam, The Netherlands; Timothy Vollmer, MD, Director, Neurosciences Clinical Research, University of Colorado, Denver; Elisa Weiss, PhD, Vice President, Patient Access and Outcomes, Leukemia & Lymphoma Society, Rye Brook, New York. We thank Victoria Powell, MPH, for assistance with data management in early stages of the project. We would like to acknowledge the International Society for Quality of Life (ISOQOL) and the ISOQOL Response Shift Special Interest Group for their interest in and support of our work on quality-of-life appraisal. Finally, we express our gratitude to the people who participated in the many appraisal studies over the past 2 decades, which enabled development of the iterations of the appraisal measures and enhanced our understanding of the importance of appraisal in patient-recorded outcomes research. Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#ME-1306-00781) Further information available at: https://www.pcori.org/research-results/2013/testing-new-ways-measure-how-patients-rate-quality-life Appendices A. Stakeholder Panel for this Project (PDF, 831K) B. Quarterly Agenda for Stakeholder Panel (PDF, 827K) C. Recent Manuscripts and Presentations based on Original Quality of Life Appraisal Profile Data Analyses (PDF, 865K) D. Bladder Cancer Study Protocol (PDF, 1.2M) Table 1. % Residual Change in Global QOL, 3 Months Post-Cystectomy, due to Patients' Baseline Appraisal and Predictor × Appraisal Interactions (PDF, 869K) Table 2. Recent Manuscripts based on Original QOL-AP Data Analyses, used to Develop QOLAP2 (PDF, 831K) Table 3. Recent Presentations based on Original QOL-AP Data Analyses (PDF, 855K) Table 4. Results of First Order Principal Components Analysis within Domain of Appraisal (PDF, 825K) Table 5. Patterns of Appraisal Evident in Second Order Components Analysis (PDF, 833K) Table 6. Correlations among Orthogonal Appraisal Principal Components & PROMIS 10 Measures (PDF, 840K) Table 7. Regression Analysis to examine how much appraisal and appraisal by specific PROMIS items added to the predictions of the more global PROMIS items (PDF, 833K) Table 8. Regression Analysis to examine how much appraisal and appraisal by demographic and comorbidity measures added to the prediction of the global PROMIS items (PDF, 837K) Table 9. Description of Measures Included in the Validation Protocol (PDF, 871K) Table 10. Database Search Parameters to Identify Eligible Participants (PDF, 833K) E. Bladder Cancer Study Instruments (PDF, 1.7M) F. Coding Manual for Cognitive Interviews (PDF, 1.1M) Institution Receiving the PCORI Award: Department of Epidemiology and Population Health, Albert Einstein College of Medicine Original Project Title: Development of Practical Outcome Measures to Account for Individual Differences and Temporal Changes in Quality of Life Appraisal PCORI Award Number/Project ID: ME-1306-00781 Suggested citation: Rapkin BD, Schwartz CE, Garcia I, et al. (2020). Testing New Ways to Measure How Patients Rate Quality of Life . Washington, DC: Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/03.2020.ME.130600781 Disclaimer The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee. Copyright © 2020. Albert Einstein College of Medicine Yeshiva University. All Rights Reserved. This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/ Bookshelf ID: NBK621134 PMID: 41779883 DOI: 10.25302/03.2020.ME.130600781 Share Views PubReader Print View Cite this Page Rapkin BD, Schwartz CE, Garcia I, et al. Testing New Ways to Measure How Patients Rate Quality of Life [Internet]. Washington (DC): Patient-Centered Outcomes Research Institute (PCORI); 2020 Mar. doi: 10.25302/03.2020.ME.130600781 PDF version of this title (6.7M) In this Page Background Stakeholder Engagement Methods Results Discussion Conclusions References Related Publications Acknowledgments Appendices Other titles in this collection PCORI Final Research Reports Related information NLM Catalog Related NLM Catalog Entries PMC PubMed Central citations PubMed Links to PubMed Recent Activity Clear Turn Off Turn On Testing New Ways to Measure How Patients Rate Quality of Life Testing New Ways to Measure How Patients Rate Quality of Life Your browsing activity is empty. Activity recording is turned off. Turn recording back on See more... Follow NCBI Twitter Facebook LinkedIn GitHub NCBI Insights Blog Connect with NLM Twitter Facebook Youtube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov