Health Care Delivery
Original Contribution
Validation of a Simple Screening Tool for Identifying Unmet Palliative Care Needs in Patients With Cancer By Paul A. Glare, MD, and Kimberly Chow, ANP-BC Memorial Sloan Kettering Cancer Center, New York, NY
(NCCN) palliative care (PC) guideline recommends PC screening for all patients and provides criteria for identifying those in need of referral to a PC specialist. This two-step process has not yet been validated as accurately identifying patients in need. The aim of this study was to validate a simplified method for screening and referral based on the existing guideline criteria.
Results: Content validity of the tool was high, with eight items rated as essential. Patients who were closer to death had significantly higher scores, indicating its construct validity. Scores were also higher in patients who were identified as needing a consult and in those who had worse pain and other symptoms, indicating its criterion validity. Using a score of ⱖ 5 as the trigger, approximately one third of hospitalized patients in the previous study would have been referred to a PC specialist, twice as many as occurred when the attending oncologist relied on his or her clinical judgment.
Methods: An 11-item screening tool was created, with scores
Conclusion: The tool seems to be a valid method for identify-
from 0 to 14. Content validity was assessed by a panel of local PC experts. Construct and criterion validities were evaluated using data obtained from a previous study of guideline-based screening and referral.
ing patients with cancer with complex PC needs who would benefit from a PC consult. Reliability testing, external validation, and demonstration of the utility of the tool as a decision aid all await confirmation.
Abstract Purpose: The National Comprehensive Cancer Network
Introduction Now that most cancer centers and other hospitals have a palliative care (PC) program,1 questions arise regarding which patients should be seen in these programs and when. The benefits of early integration of PC with standard oncologic care have been demonstrated.2 Although it may be theoretically optimal for PC specialists to take on all aspects of PC, some core elements of PC—as is the case in any medical discipline—should be routine aspects of care delivered by any practitioner.3 These include aligning treatment with a patient’s goals and basic symptom management. Other skills are more complex and take years of training to learn and apply, such as negotiating a difficult family meeting, addressing veiled existential distress, and managing refractory symptoms. Multiple barriers to referral to a specialist PC program exist, including oncologist nonawareness of the program, what the program can do, when to refer, and oncologist unwillingness to refer.4 To help overcome these barriers, the National Comprehensive Cancer Network (NCCN) recommends that all patients with cancer be screened for PC needs at their initial visit and rescreened at appropriate intervals and as clinically indicated. Consultation or collaboration with PC specialists is recommended for patients with more complex problems.5 Although the NCCN PC guideline recommends screening, there have been few studies evaluating the logistics of screening oncology patients on a large scale. This may not be surprising, given that the screening advocated in the guideline involves a comprehensive evaluation of the patient and family across six domains, including: one, pain and other symptoms; two, psychological distress; three, comorbid physical and psychosocial conditions; four, treatment options; five, patient or family concerns about disease Copyright © 2014 by American Society of Clinical Oncology
course and decision making; and six, prognosis. To add to the complexity of screening, the NCCN posits an additional checklist of 24 physical or psychosocial problems for determining which patients have more complex cases and should be considered for consultation or collaboration with a specialist PC program. To evaluate the feasibility of implementing the screening and referral components of the guideline, testing took place in both hospitalized and ambulatory patients in one medical oncology service at Memorial Sloan Kettering Cancer Center (MSKCC; New York, NY).6,7 To facilitate the complex screening process advocated by the guideline, two pencil-and-paper tools were developed, one for the seven initial screening items and the other for the supplementary list of 24 referral criteria. Although these projects demonstrated the feasibility of screening and the prevalence of PC problems in both the inpatient and outpatient settings, questions arose regarding the validity and reliability of the screening tools that had been created. In particular, there seemed to be redundancy in some of the items on the two lists. Simplifying screening to a one-step validated process seems to be desirable before further work is done in this area. The purpose of this article is to describe a one-step screening tool derived from the two-step guideline, validated against the data obtained from the previously published study of inpatients at MSKCC.7
Methods Patient Population A total of 194 patients who were hospitalized in the MSKCC GI Oncology Service during the winter of 2010 to 2011 were screened for PC needs.7 The population of screened patients
J A N U A R Y 2015
•
jop.ascopubs.org
e81
Glare and Chow
Table 1. Screening Tool, Scoring System, and CVRs for Each Tool Item Criterion
Points*
CVR
Locally advanced or metastatic cancer
2
⫺0.15
Functional status of patient (ie, ECOG score)
0-4
1.00
Any serious complication of cancer associated with survival ⬍ 12 months†
1
1.00
Any serious comorbidity
1
0.43
Any other condition complicating care‡
1
⫺0.42
Uncontrolled symptoms
1
1.00
Moderate-severe distress
1
0.43
Patient/family concerns regarding decision making
1
0.14
Additional criteria
Team needs assistance with decision making
1
1.00
Patient/family requests PC consult
1
1.00
Prolonged length of stay
1
0.70
Total
0-14
Abbreviations: COPD, chronic obstructive pulmonary disease; CVR, content validity ratio; ECOG, Eastern Cooperative Oncology Group; KPS, Karnofsky performance score; NCCN, National Comprehensive Cancer Network; PC, palliative care; PCCB, Palliative Care Center of the Bluegrass (Lexington, KY). * Scoring system recommended by PCCB; cut point for consult recommended by PCCB was score ⬎ or ⬍ 3. † Examples given in NCCN guideline include: ECOG of 3 or KPS of 50, hypercalcemia, brain or cerebrospinal fluid metastasis, delirium, superior vena cava syndrome, spinal cord compression, cachexia, malignant effusions, bilirubin ⬎ 2.5 mg/dL, and creatinine ⬎ 3 mg/dL. ‡ NCCN guideline does not specify these conditions. PCCB suggests: liver disease, moderate or end-stage renal disease, moderate or advanced cardiac disease, moderate or advanced COPD, stroke with loss of ⱖ 50% of function, other life-limiting illnesses, or other conditions complicating care.
was younger than usual, with only one third age ⬎ 65 years. More than 90% had advanced disease, and ⬎ 60% were still pursuing active treatment. Admissions were mostly for the management of the complications of advanced cancer or treatment-related toxicities; almost none were for administration of chemotherapy. A waiver to use existing patient data was previously obtained from the MSKCC Institutional Review Board to analyze and disseminate the results of the quality improvement project for which they were collected. A separate waiver was not sought to reuse the same data for the purposes of validating the tool used in the project.
Determination of Tool Validity The screening tool is summarized in Table 1. It comprises 11 items covering five clinical dimensions, which define the need for PC in patients with cancer: extent of disease, performance status (PS), prognosis, comorbidities (with two subparts), and PC-specific problems. Three versions of validity of the tool were evaluated: content validity, construct validity, and the two forms of criterion validity.8 Internal consistency of the individual items was also calculated, but other aspects of reliability (test-retest reliability and inter-rater reliability) could not be evaluated from this data. All calculations were performed in Excel (Microsoft; Redmond, WA) using the data analysis add-in function. e82
JOURNAL
OF
ONCOLOGY PRACTICE
•
Content validity evaluates if a scale or other measurement incorporates the various domains of the phenomenon it purports to measure. In this case, the screening tool would have content validity if its items were considered to be relevant for capturing the kind of problems that are within the scope of practice of a specialist PC program. Although this may be done as a subjective exercise, we attempted to quantify content validity by calculating the content validity ratio (CVR), being the extent of agreement among content experts on how essential a particular item in the tool is, using the following categories: essential to be included; useful, but not essential to be included; or not necessary to be included. The CVR score for an item can range from ⫺1 to ⫹1; a positive value for an item in the CVR accords it at least some content validity. The mean CVR across items may be used as an indicator of overall content validity. Seven physicians and nurse practitioners from our 13-member interdisciplinary team (IDT), all certified in palliative care, returned completed surveys, which rated the essentiality of the items in the tool and the NCCN guideline screening items and referral criteria. Construct validity evaluates the extent to which a measurement corresponds to theoretic concepts concerning the phenomenon under study. PS falls and symptom burden rises as death approaches in patients with advanced cancer,9 so the screening tool score ought to be higher in patients with a shorter survival, if the tool has construct validity. Analysis of variance was used to compare the scores on the screening tool for patients who survived ⬍ 3, 3 to 6, and ⬎ 6 months. Internal consistency, which assesses the consistency of results across the items of a tool, was evaluated as another marker of construct validity using Cronbach’s alpha. Criterion validity evaluates the extent to which the measurement correlates with an external criterion or gold standard for the phenomenon under concern. This is a challenging concept for health care services that are not disease based, such as PC. In this case, symptom scores and consultation requests were used to test for concurrent validity and predictive validity, which together satisfy criterion validity. Concurrent validity was tested by comparing patient self-reports for pain intensity and symptom distress, using the condensed Memorial Symptom Assessment Schedule (CMSAS),10 with the identification of uncontrolled symptoms and moderate-severe distress by the screening tool. Predictive validity was evaluated by testing if the total score on the tool was higher in patients identified as needing consultation—whether based on the clinical judgment of the on-service oncologist or triggered by meeting ⱖ one NCCN referral criteria.
Results Content Validity The items in the screening tool and the referral criteria came directly from a national panel of experts convened by the NCCN and so would be expected to have strong content validity, although no data are available for how this list was developed or if a CVR was calculated for the NCCN panelists. The
V O L . 11, I S S U E 1
Copyright © 2014 by American Society of Clinical Oncology
Simple Screening Tool for Identifying Unmet Palliative Care Needs
Average Score
6 5 4 3 2 1 0
≤ 90 days
91-180 days
> 180 days
Figure 1. Construct validity: higher tool score associated with shorter survival.
CVR for the individual items in the screening tool is listed in Table 1. Nine of the 11 items in the tool were rated as essential by the survey respondents; two items had negative CVRs (ie, useful but nonessential); no member of the IDT rated any of the items as unnecessary for inclusion. When the NCCN guideline lists were evaluated, each of the screening items was rated as essential by a majority of the IDT members, but only eight of the 24 items on referral criteria were rated essential. There was a large degree of overlap between the eight essential referral criteria and the screening items. Nine items on referral criteria were considered unnecessary by at least one member.
Construct Validity The average score on the tool was significantly higher in patients who survived ⬍ 6 months (5.0 v 3.2; t ⫽ 6.1131; df ⫽ 228; P ⬍ .001). Furthermore, as shown in Figure 1, the average screening tool scores were also significantly different across the three survival groups (ⱕ 90, 91 to 180, and ⬎ 180 days), being 5.3, 3.9, and 3.3 points, respectively (SS ⫽ 203.93; df ⫽ 2; MS ⫽ 101.97; P ⬍ .001). The internal consistency of the tool was high, with the extent of agreement in scores across the tool exceeding 80% ( ⫽ 12; E variance ⫽ 1.32; variance ⫽ 5.083; Cronbach’s alpha ⫽ .808), further supporting that the items in the tool are addressing a unified construct.
Criterion Validity Concurrent validity. Complete pain scores and CMSAS scores were available for 122 admissions, including 48 of 100 admissions of patients who were identified as having uncontrolled symptoms on screening and 74 of 129 who did not. Although a lot of data were missing, those screening positive for uncontrolled symptoms had a higher average pain intensity score (⫾ standard deviation [SD]; 4.75 ⫾ 0.54 v 3.74 ⫾ 0.17; t ⫽ 2.08; df ⫽ 136; P ⫽ .04) and a higher physical symptom distress score on the CMSAS (2.00 ⫾ 0.13 v 1.45 ⫾ 0.11; t ⫽ 3.29; df ⫽ 120; P ⫽ .001). However, the average psychological symptom score (⫾ SD) was no higher in patients who screened positive for being distressed than those who were not (1.71 ⫾ 0.36 v 1.60 ⫾ 0.1; P ⫽ .7). Predictive validity. The score on the tool was higher in those patients who were identified as in need of a consult by both Copyright © 2014 by American Society of Clinical Oncology
methods (ie, oncologist judgment and NCCN referral criteria). The score (⫾ SD) was 5.5 ⫾ 0.7 versus 3.6 ⫾ 0.2 (P ⫽ .01) when oncologists used their clinical acumen and 4.9 ⫾ 0.3 versus 3.5 ⫾ 0.2 (P ⫽ .001) for the NCCN referral criteria. Using a score of 5 on the tool as the cut point for triggering a consult, a consult would have been recommended for 34% of screened patients. A score of ⱖ 5 had a high positive predictive value (PPV; 80%) that a patient would meet ⱖ one of the NCCN referral criteria but only a moderate negative predictive value (NPV; 44%), meaning that almost half the patients meeting the NCCN criteria would be missed. On the basis the oncologists’ decision to call a consult, a score of 5 had a PPV of only 25%, suggesting that the clinicians were missing many of the patients who needed a consult. However, the NPV in this setting was 89%, indicating that if the score was ⬍ 5, it was unlikely a consult would be needed.
Discussion This evaluation indicates that this one-step, scored screening tool based on the NCCN PC guideline is a valid method for identifying patients with cancer who might benefit from a PC consult. At a time when workforce issues in both oncology and palliative care are becoming more pressing,3 this is an important tool for busy oncologists to have at their disposal. Validation was performed from multiple perspectives. A majority of a panel of local PC experts concurred that most of the items in the tool were essential to be included. Patients who were closer to death had higher scores on the tool. Patients who had more pain and other symptoms had higher scores, although there was no association between psychological symptoms and screening positive for distress. Scores were higher on the tool in patients for whom a consult was requested, whereas a low score on the tool had a high NPV for consultation. Using a cut point of 5, approximately one third of patients would be referred to a PC specialist—twice as many as currently occurs when oncologists rely on their subjective judgment. The implementation study indicated that these consults were called upstream, occurring at an earlier time and when patients were less distressed.7 Earlier integration of PC with oncology care allows more time for the benefits of PC to be achieved, compared with a PC consult being called in reaction to a pain crisis or a goals-of-care emergency at the time of hospitalization. Other PC screening tools have been developed,4,11-13 including the tool from which this one was adapted,4 but few have undergone validation or other kinds of evaluation. An exception is the NEST (Needs of a Social Nature, Existential Concerns, Symptoms, and Therapeutic Interaction) 13⫹ tool. Although it facilitated greater documentation of illness-related needs than routine clinical assessment, it was less clear that using the NEST 13⫹ produced changes in clinician response, such as calling more PC consults.12 A study presented at the 2013 American Society of Clinical Oncology Quality of Care Symposium14 described a similar screening tool to that reported here, further supporting the content validity. Widespread implementation of a validated screening tool could have significant implications for the PC workforce, which
J A N U A R Y 2015
•
jop.ascopubs.org
e83
Glare and Chow
is already strained (Appendix, online only). Extrapolating data from Ontario, Canada, where all ambulatory patients attending cancer centers have their PS and symptom burden scored at each visit,15 to the United States (1.5 million new cancer diagnoses per year), approximately one in eight new visits (187,500 patients) will result in a score ⱖ 5 (ie, positive screening). Approximately half of these patients would be expected to die within the next 6 months.9 If these patients are seen in clinic monthly,2 screening would result in an extra 1.5 million physician-patient encounters per year. Assuming a full-time equivalent (FTE) PC specialist may see 15 patients per day, approximately 400 additional FTE PC specialists would be needed to be employed for this purpose alone, aggravating the existing specialist PC physician FTE shortage of 8,000. This strengthens the case for a division of labor between generalists and PC specialists.3 Aside from workforce issues, a potential clinical limitation of setting the cut point at 5 is that all patients with metastatic cancer and an Eastern Cooperative Oncology (ECOG) PS of 3 would screen positive. However, it is important to emphasize that this is a screening tool, not an automatic consult trigger, so it does not mean that all patients with an ECOG PS of 3 must have a consult. Rather, it should prompt the oncologist to consider a PC consult if he or she has not already done so and discuss this with the patient and family. In the Japanese study of outpatient screening, only 50% of patients who screened positive proceeded to PC consultation.13 Nevertheless, as a group, patients with metastatic cancer and an ECOG PS of 3 have a survival typically measured in a small number of months16 and increasing symptom burden and distress,9 so it is likely that a PC specialist would be able to assist in the care of many of these patients.13 If oversensitivity remains a concern, another option would be to borrow from cardiology and divide the items into major (ie, extent of disease, PS, complications, comorbidities) and minor criteria (ie, physical symptoms, distress, goals of care), as is done for rheumatic fever,17 and develop rules for referral.17 Beyond PC to the broader issue of tools for the systematic needs assessment of patients with cancer, a review published in 2000 identified nine questionnaires for the assessment of patients’ needs and six for the assessment of needs of family members. Most of the instruments were carefully constructed, with satisfactory validity and reliability.18 However, the need for care was often confounded by satisfaction with care and the problems experienced by patients. Only one questionnaire for patients specifically addressed the need for help; none for family members was so specific. Data on the feasibility of questionnaires for use in regular care were scarce. Issues frequently omitted were spiritual issues, the personal needs of family members, and the continuity of care. It also seemed that most instruments had been constructed for research purposes and had not been tested for use in practical care. Seven years later, another review identified 15 tools that were appraised for psychometric properties, such as validity, reliability, responsiveness to change, and
e84
JOURNAL
OF
ONCOLOGY PRACTICE
•
feasibility, including acceptability to patients.19 Each tool met some of these criteria, but none was found to be complete for all of them. Again, data on how feasible they were to use in practice were scarce, and it was concluded that most were insufficiently tested to recommend for use in routine care. Now that our tool has been validated, its other psychometric properties need to be tested. The previous study of its use in hospitalized patients indicated that although it was quick, easy to use, and effective, there were concerns regarding its acceptability by nurses and oncologists and regarding the accuracy and reliability of the scoring, because many of the items in the tool are subjective or vaguely defined. These issues need further evaluation; they could potentially be resolved by developing an electronic version of the tool that self-populates with clinical data from the electronic health record, supplemented with patient and family self-reports of symptoms, distress, and other concerns. This screening tool could also be adapted for use in special areas, such as the intensive care unit, by replacing the items related to cancer and its complications and comorbidities with commonly used physiology scores and prognostic tools paired with a prolonged length of stay without clinical progress, although revalidation would be required. In conclusion, this simple 11-item screening tool has been shown to be a valid method for screening for PC problems in patients with cancer hospitalized at a comprehensive cancer center. Consistent with the concept of screening in health care, its scoring system provides clinicians with a means to identify which of their large volume of patients should be considered for PC referral. Using a score of ⱖ 5 as the trigger, 34% of patients who were screened would be recommended to receive a consult, with a high PPV that they would meet ⱖ one of the NCCN referral criteria. Now that it has been shown to be valid and feasible to use in routine caregiving, the impact of screening with the tool on quality of care needs to be established. Acknowledgment We thank our physician and nurse practitioner colleagues who participated anonymously in evaluating the content validity of the tool. Authors’ Disclosures of Potential Conflicts of Interest The authors indicated no potential conflicts of interest.
Author Contributions Conception and design: Paul A. Glare Collection and assembly of data: Paul A. Glare Data analysis and interpretation: All authors Manuscript writing: All authors Final approval of manuscript: All authors Corresponding author: Paul A. Glare, MD, Palliative Medicine Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York NY 10065; e-mail: [email protected].
DOI: 10.1200/JOP.2014.001487; published online ahead of print at jop.ascopubs.org on November 12, 2014.
V O L . 11, I S S U E 1
Copyright © 2014 by American Society of Clinical Oncology
Simple Screening Tool for Identifying Unmet Palliative Care Needs
References 1. Hui D, Elsayem A, De la Cruz M, et al: Availability and integration of palliative care at US cancer centers. JAMA 303:1054-1061, 2010 2. Temel JS, Greer JA, Muzikansky A, et al: Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 363:733-742, 2010 3. Quill TE, Abernethy AP: Generalist plus specialist palliative care: Creating a more sustainable model. N Engl J Med 368:1173-1175, 2013 4. Meier DE, Sieger CE: A Guide to Building a Hospital Based Palliative Care Program. New York, NY, Center to Advance Palliative Care, 2004 5. Levy MH, Back A, Benedetti C, et al: NCCN clinical practice guidelines in oncology: Palliative care. J Natl Compr Canc Netw 7:436-473, 2009 6. Glare PA, Semple D, Stabler SM, et al: Palliative care in the outpatient oncology setting: Evaluation of a practical set of referral criteria. J Oncol Pract 7:366-370, 2011 7. Glare P, Plakovic K, Schloms A, et al: Study using the NCCN guidelines for palliative care to screen patients for palliative care needs and referral to palliative care specialists. J Natl Compr Canc Netw 11:1087-1096, 2013 8. Last JM: A Dictionary of Epidemiology (ed 4). Oxford, United Kingdom, Oxford University Press, 2001 9. Seow H, Barbera L, Sutradhar R, et al: Trajectory of performance status and symptom scores for patients with cancer during the last 6 months of life. J Clin Oncol 29:1151-1158, 2011 10. Chang VT, Hwang SS, Kasimis B, et al: Shorter symptom assessment instruments: The Condensed Memorial Symptom Assessment Scale (CMSAS). Cancer Invest 22:526-536, 2004
Copyright © 2014 by American Society of Clinical Oncology
11. Weissman DE, Meier DE: Identifying patients in need of a palliative care assessment in the hospital setting: A consensus report from the Center to Advance Palliative Care. J Palliat Med 14:17-23, 2011 12. Scandrett KG, Reitschuler-Cross EB, Nelson L, et al: Feasibility and effectiveness of the NEST13⫹ as a screening tool for advanced illness care needs. J Palliat Med 13:161-169, 2010 13. Morita T, Fujimoto K, Namba M, et al: Palliative care needs of cancer outpatients receiving chemotherapy: An audit of a clinical screening project. Support Care Cancer 16:101-107, 2008 14. Adelson K, Paris J, Smith C, et al: Standardized criteria for required palliative care consultation on the solid tumor oncology service. Presented at the American Society of Clinical Oncology Quality Care Symposium, November 1, 2013 (abstr 37) 15. Barbera L, Seow H, Howell D, et al: Symptom burden and performance status in a population-based cohort of ambulatory cancer patients. Cancer 116: 5767-5776, 2010 16. Miller RJ: Predicting survival in the advanced cancer patient. Henry Ford Hosp Med J 39:81-84, 1991 17. Ferrieri P: Proceedings of the Jones criteria workshop. Circulation 106:25212523, 2002 18. Osse BH, Vernooij-Dassen MJ, de Vree BP, et al: Assessment of the need for palliative care as perceived by individual cancer patients and their families: A review of instruments for improving patient participation in palliative care. Cancer 88:900-911, 2000 19. Richardson A, Medina J, Brown V, et al: Patients’ needs assessment in cancer care: A review of assessment tools. Support Care Cancer 15:1125-1144, 2007
J A N U A R Y 2015
•
jop.ascopubs.org
e85
Copyright of Journal of Oncology Practice is the property of American Society of Clinical Oncology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Original Contribution
Validation of a Simple Screening Tool for Identifying Unmet Palliative Care Needs in Patients With Cancer By Paul A. Glare, MD, and Kimberly Chow, ANP-BC Memorial Sloan Kettering Cancer Center, New York, NY
(NCCN) palliative care (PC) guideline recommends PC screening for all patients and provides criteria for identifying those in need of referral to a PC specialist. This two-step process has not yet been validated as accurately identifying patients in need. The aim of this study was to validate a simplified method for screening and referral based on the existing guideline criteria.
Results: Content validity of the tool was high, with eight items rated as essential. Patients who were closer to death had significantly higher scores, indicating its construct validity. Scores were also higher in patients who were identified as needing a consult and in those who had worse pain and other symptoms, indicating its criterion validity. Using a score of ⱖ 5 as the trigger, approximately one third of hospitalized patients in the previous study would have been referred to a PC specialist, twice as many as occurred when the attending oncologist relied on his or her clinical judgment.
Methods: An 11-item screening tool was created, with scores
Conclusion: The tool seems to be a valid method for identify-
from 0 to 14. Content validity was assessed by a panel of local PC experts. Construct and criterion validities were evaluated using data obtained from a previous study of guideline-based screening and referral.
ing patients with cancer with complex PC needs who would benefit from a PC consult. Reliability testing, external validation, and demonstration of the utility of the tool as a decision aid all await confirmation.
Abstract Purpose: The National Comprehensive Cancer Network
Introduction Now that most cancer centers and other hospitals have a palliative care (PC) program,1 questions arise regarding which patients should be seen in these programs and when. The benefits of early integration of PC with standard oncologic care have been demonstrated.2 Although it may be theoretically optimal for PC specialists to take on all aspects of PC, some core elements of PC—as is the case in any medical discipline—should be routine aspects of care delivered by any practitioner.3 These include aligning treatment with a patient’s goals and basic symptom management. Other skills are more complex and take years of training to learn and apply, such as negotiating a difficult family meeting, addressing veiled existential distress, and managing refractory symptoms. Multiple barriers to referral to a specialist PC program exist, including oncologist nonawareness of the program, what the program can do, when to refer, and oncologist unwillingness to refer.4 To help overcome these barriers, the National Comprehensive Cancer Network (NCCN) recommends that all patients with cancer be screened for PC needs at their initial visit and rescreened at appropriate intervals and as clinically indicated. Consultation or collaboration with PC specialists is recommended for patients with more complex problems.5 Although the NCCN PC guideline recommends screening, there have been few studies evaluating the logistics of screening oncology patients on a large scale. This may not be surprising, given that the screening advocated in the guideline involves a comprehensive evaluation of the patient and family across six domains, including: one, pain and other symptoms; two, psychological distress; three, comorbid physical and psychosocial conditions; four, treatment options; five, patient or family concerns about disease Copyright © 2014 by American Society of Clinical Oncology
course and decision making; and six, prognosis. To add to the complexity of screening, the NCCN posits an additional checklist of 24 physical or psychosocial problems for determining which patients have more complex cases and should be considered for consultation or collaboration with a specialist PC program. To evaluate the feasibility of implementing the screening and referral components of the guideline, testing took place in both hospitalized and ambulatory patients in one medical oncology service at Memorial Sloan Kettering Cancer Center (MSKCC; New York, NY).6,7 To facilitate the complex screening process advocated by the guideline, two pencil-and-paper tools were developed, one for the seven initial screening items and the other for the supplementary list of 24 referral criteria. Although these projects demonstrated the feasibility of screening and the prevalence of PC problems in both the inpatient and outpatient settings, questions arose regarding the validity and reliability of the screening tools that had been created. In particular, there seemed to be redundancy in some of the items on the two lists. Simplifying screening to a one-step validated process seems to be desirable before further work is done in this area. The purpose of this article is to describe a one-step screening tool derived from the two-step guideline, validated against the data obtained from the previously published study of inpatients at MSKCC.7
Methods Patient Population A total of 194 patients who were hospitalized in the MSKCC GI Oncology Service during the winter of 2010 to 2011 were screened for PC needs.7 The population of screened patients
J A N U A R Y 2015
•
jop.ascopubs.org
e81
Glare and Chow
Table 1. Screening Tool, Scoring System, and CVRs for Each Tool Item Criterion
Points*
CVR
Locally advanced or metastatic cancer
2
⫺0.15
Functional status of patient (ie, ECOG score)
0-4
1.00
Any serious complication of cancer associated with survival ⬍ 12 months†
1
1.00
Any serious comorbidity
1
0.43
Any other condition complicating care‡
1
⫺0.42
Uncontrolled symptoms
1
1.00
Moderate-severe distress
1
0.43
Patient/family concerns regarding decision making
1
0.14
Additional criteria
Team needs assistance with decision making
1
1.00
Patient/family requests PC consult
1
1.00
Prolonged length of stay
1
0.70
Total
0-14
Abbreviations: COPD, chronic obstructive pulmonary disease; CVR, content validity ratio; ECOG, Eastern Cooperative Oncology Group; KPS, Karnofsky performance score; NCCN, National Comprehensive Cancer Network; PC, palliative care; PCCB, Palliative Care Center of the Bluegrass (Lexington, KY). * Scoring system recommended by PCCB; cut point for consult recommended by PCCB was score ⬎ or ⬍ 3. † Examples given in NCCN guideline include: ECOG of 3 or KPS of 50, hypercalcemia, brain or cerebrospinal fluid metastasis, delirium, superior vena cava syndrome, spinal cord compression, cachexia, malignant effusions, bilirubin ⬎ 2.5 mg/dL, and creatinine ⬎ 3 mg/dL. ‡ NCCN guideline does not specify these conditions. PCCB suggests: liver disease, moderate or end-stage renal disease, moderate or advanced cardiac disease, moderate or advanced COPD, stroke with loss of ⱖ 50% of function, other life-limiting illnesses, or other conditions complicating care.
was younger than usual, with only one third age ⬎ 65 years. More than 90% had advanced disease, and ⬎ 60% were still pursuing active treatment. Admissions were mostly for the management of the complications of advanced cancer or treatment-related toxicities; almost none were for administration of chemotherapy. A waiver to use existing patient data was previously obtained from the MSKCC Institutional Review Board to analyze and disseminate the results of the quality improvement project for which they were collected. A separate waiver was not sought to reuse the same data for the purposes of validating the tool used in the project.
Determination of Tool Validity The screening tool is summarized in Table 1. It comprises 11 items covering five clinical dimensions, which define the need for PC in patients with cancer: extent of disease, performance status (PS), prognosis, comorbidities (with two subparts), and PC-specific problems. Three versions of validity of the tool were evaluated: content validity, construct validity, and the two forms of criterion validity.8 Internal consistency of the individual items was also calculated, but other aspects of reliability (test-retest reliability and inter-rater reliability) could not be evaluated from this data. All calculations were performed in Excel (Microsoft; Redmond, WA) using the data analysis add-in function. e82
JOURNAL
OF
ONCOLOGY PRACTICE
•
Content validity evaluates if a scale or other measurement incorporates the various domains of the phenomenon it purports to measure. In this case, the screening tool would have content validity if its items were considered to be relevant for capturing the kind of problems that are within the scope of practice of a specialist PC program. Although this may be done as a subjective exercise, we attempted to quantify content validity by calculating the content validity ratio (CVR), being the extent of agreement among content experts on how essential a particular item in the tool is, using the following categories: essential to be included; useful, but not essential to be included; or not necessary to be included. The CVR score for an item can range from ⫺1 to ⫹1; a positive value for an item in the CVR accords it at least some content validity. The mean CVR across items may be used as an indicator of overall content validity. Seven physicians and nurse practitioners from our 13-member interdisciplinary team (IDT), all certified in palliative care, returned completed surveys, which rated the essentiality of the items in the tool and the NCCN guideline screening items and referral criteria. Construct validity evaluates the extent to which a measurement corresponds to theoretic concepts concerning the phenomenon under study. PS falls and symptom burden rises as death approaches in patients with advanced cancer,9 so the screening tool score ought to be higher in patients with a shorter survival, if the tool has construct validity. Analysis of variance was used to compare the scores on the screening tool for patients who survived ⬍ 3, 3 to 6, and ⬎ 6 months. Internal consistency, which assesses the consistency of results across the items of a tool, was evaluated as another marker of construct validity using Cronbach’s alpha. Criterion validity evaluates the extent to which the measurement correlates with an external criterion or gold standard for the phenomenon under concern. This is a challenging concept for health care services that are not disease based, such as PC. In this case, symptom scores and consultation requests were used to test for concurrent validity and predictive validity, which together satisfy criterion validity. Concurrent validity was tested by comparing patient self-reports for pain intensity and symptom distress, using the condensed Memorial Symptom Assessment Schedule (CMSAS),10 with the identification of uncontrolled symptoms and moderate-severe distress by the screening tool. Predictive validity was evaluated by testing if the total score on the tool was higher in patients identified as needing consultation—whether based on the clinical judgment of the on-service oncologist or triggered by meeting ⱖ one NCCN referral criteria.
Results Content Validity The items in the screening tool and the referral criteria came directly from a national panel of experts convened by the NCCN and so would be expected to have strong content validity, although no data are available for how this list was developed or if a CVR was calculated for the NCCN panelists. The
V O L . 11, I S S U E 1
Copyright © 2014 by American Society of Clinical Oncology
Simple Screening Tool for Identifying Unmet Palliative Care Needs
Average Score
6 5 4 3 2 1 0
≤ 90 days
91-180 days
> 180 days
Figure 1. Construct validity: higher tool score associated with shorter survival.
CVR for the individual items in the screening tool is listed in Table 1. Nine of the 11 items in the tool were rated as essential by the survey respondents; two items had negative CVRs (ie, useful but nonessential); no member of the IDT rated any of the items as unnecessary for inclusion. When the NCCN guideline lists were evaluated, each of the screening items was rated as essential by a majority of the IDT members, but only eight of the 24 items on referral criteria were rated essential. There was a large degree of overlap between the eight essential referral criteria and the screening items. Nine items on referral criteria were considered unnecessary by at least one member.
Construct Validity The average score on the tool was significantly higher in patients who survived ⬍ 6 months (5.0 v 3.2; t ⫽ 6.1131; df ⫽ 228; P ⬍ .001). Furthermore, as shown in Figure 1, the average screening tool scores were also significantly different across the three survival groups (ⱕ 90, 91 to 180, and ⬎ 180 days), being 5.3, 3.9, and 3.3 points, respectively (SS ⫽ 203.93; df ⫽ 2; MS ⫽ 101.97; P ⬍ .001). The internal consistency of the tool was high, with the extent of agreement in scores across the tool exceeding 80% ( ⫽ 12; E variance ⫽ 1.32; variance ⫽ 5.083; Cronbach’s alpha ⫽ .808), further supporting that the items in the tool are addressing a unified construct.
Criterion Validity Concurrent validity. Complete pain scores and CMSAS scores were available for 122 admissions, including 48 of 100 admissions of patients who were identified as having uncontrolled symptoms on screening and 74 of 129 who did not. Although a lot of data were missing, those screening positive for uncontrolled symptoms had a higher average pain intensity score (⫾ standard deviation [SD]; 4.75 ⫾ 0.54 v 3.74 ⫾ 0.17; t ⫽ 2.08; df ⫽ 136; P ⫽ .04) and a higher physical symptom distress score on the CMSAS (2.00 ⫾ 0.13 v 1.45 ⫾ 0.11; t ⫽ 3.29; df ⫽ 120; P ⫽ .001). However, the average psychological symptom score (⫾ SD) was no higher in patients who screened positive for being distressed than those who were not (1.71 ⫾ 0.36 v 1.60 ⫾ 0.1; P ⫽ .7). Predictive validity. The score on the tool was higher in those patients who were identified as in need of a consult by both Copyright © 2014 by American Society of Clinical Oncology
methods (ie, oncologist judgment and NCCN referral criteria). The score (⫾ SD) was 5.5 ⫾ 0.7 versus 3.6 ⫾ 0.2 (P ⫽ .01) when oncologists used their clinical acumen and 4.9 ⫾ 0.3 versus 3.5 ⫾ 0.2 (P ⫽ .001) for the NCCN referral criteria. Using a score of 5 on the tool as the cut point for triggering a consult, a consult would have been recommended for 34% of screened patients. A score of ⱖ 5 had a high positive predictive value (PPV; 80%) that a patient would meet ⱖ one of the NCCN referral criteria but only a moderate negative predictive value (NPV; 44%), meaning that almost half the patients meeting the NCCN criteria would be missed. On the basis the oncologists’ decision to call a consult, a score of 5 had a PPV of only 25%, suggesting that the clinicians were missing many of the patients who needed a consult. However, the NPV in this setting was 89%, indicating that if the score was ⬍ 5, it was unlikely a consult would be needed.
Discussion This evaluation indicates that this one-step, scored screening tool based on the NCCN PC guideline is a valid method for identifying patients with cancer who might benefit from a PC consult. At a time when workforce issues in both oncology and palliative care are becoming more pressing,3 this is an important tool for busy oncologists to have at their disposal. Validation was performed from multiple perspectives. A majority of a panel of local PC experts concurred that most of the items in the tool were essential to be included. Patients who were closer to death had higher scores on the tool. Patients who had more pain and other symptoms had higher scores, although there was no association between psychological symptoms and screening positive for distress. Scores were higher on the tool in patients for whom a consult was requested, whereas a low score on the tool had a high NPV for consultation. Using a cut point of 5, approximately one third of patients would be referred to a PC specialist—twice as many as currently occurs when oncologists rely on their subjective judgment. The implementation study indicated that these consults were called upstream, occurring at an earlier time and when patients were less distressed.7 Earlier integration of PC with oncology care allows more time for the benefits of PC to be achieved, compared with a PC consult being called in reaction to a pain crisis or a goals-of-care emergency at the time of hospitalization. Other PC screening tools have been developed,4,11-13 including the tool from which this one was adapted,4 but few have undergone validation or other kinds of evaluation. An exception is the NEST (Needs of a Social Nature, Existential Concerns, Symptoms, and Therapeutic Interaction) 13⫹ tool. Although it facilitated greater documentation of illness-related needs than routine clinical assessment, it was less clear that using the NEST 13⫹ produced changes in clinician response, such as calling more PC consults.12 A study presented at the 2013 American Society of Clinical Oncology Quality of Care Symposium14 described a similar screening tool to that reported here, further supporting the content validity. Widespread implementation of a validated screening tool could have significant implications for the PC workforce, which
J A N U A R Y 2015
•
jop.ascopubs.org
e83
Glare and Chow
is already strained (Appendix, online only). Extrapolating data from Ontario, Canada, where all ambulatory patients attending cancer centers have their PS and symptom burden scored at each visit,15 to the United States (1.5 million new cancer diagnoses per year), approximately one in eight new visits (187,500 patients) will result in a score ⱖ 5 (ie, positive screening). Approximately half of these patients would be expected to die within the next 6 months.9 If these patients are seen in clinic monthly,2 screening would result in an extra 1.5 million physician-patient encounters per year. Assuming a full-time equivalent (FTE) PC specialist may see 15 patients per day, approximately 400 additional FTE PC specialists would be needed to be employed for this purpose alone, aggravating the existing specialist PC physician FTE shortage of 8,000. This strengthens the case for a division of labor between generalists and PC specialists.3 Aside from workforce issues, a potential clinical limitation of setting the cut point at 5 is that all patients with metastatic cancer and an Eastern Cooperative Oncology (ECOG) PS of 3 would screen positive. However, it is important to emphasize that this is a screening tool, not an automatic consult trigger, so it does not mean that all patients with an ECOG PS of 3 must have a consult. Rather, it should prompt the oncologist to consider a PC consult if he or she has not already done so and discuss this with the patient and family. In the Japanese study of outpatient screening, only 50% of patients who screened positive proceeded to PC consultation.13 Nevertheless, as a group, patients with metastatic cancer and an ECOG PS of 3 have a survival typically measured in a small number of months16 and increasing symptom burden and distress,9 so it is likely that a PC specialist would be able to assist in the care of many of these patients.13 If oversensitivity remains a concern, another option would be to borrow from cardiology and divide the items into major (ie, extent of disease, PS, complications, comorbidities) and minor criteria (ie, physical symptoms, distress, goals of care), as is done for rheumatic fever,17 and develop rules for referral.17 Beyond PC to the broader issue of tools for the systematic needs assessment of patients with cancer, a review published in 2000 identified nine questionnaires for the assessment of patients’ needs and six for the assessment of needs of family members. Most of the instruments were carefully constructed, with satisfactory validity and reliability.18 However, the need for care was often confounded by satisfaction with care and the problems experienced by patients. Only one questionnaire for patients specifically addressed the need for help; none for family members was so specific. Data on the feasibility of questionnaires for use in regular care were scarce. Issues frequently omitted were spiritual issues, the personal needs of family members, and the continuity of care. It also seemed that most instruments had been constructed for research purposes and had not been tested for use in practical care. Seven years later, another review identified 15 tools that were appraised for psychometric properties, such as validity, reliability, responsiveness to change, and
e84
JOURNAL
OF
ONCOLOGY PRACTICE
•
feasibility, including acceptability to patients.19 Each tool met some of these criteria, but none was found to be complete for all of them. Again, data on how feasible they were to use in practice were scarce, and it was concluded that most were insufficiently tested to recommend for use in routine care. Now that our tool has been validated, its other psychometric properties need to be tested. The previous study of its use in hospitalized patients indicated that although it was quick, easy to use, and effective, there were concerns regarding its acceptability by nurses and oncologists and regarding the accuracy and reliability of the scoring, because many of the items in the tool are subjective or vaguely defined. These issues need further evaluation; they could potentially be resolved by developing an electronic version of the tool that self-populates with clinical data from the electronic health record, supplemented with patient and family self-reports of symptoms, distress, and other concerns. This screening tool could also be adapted for use in special areas, such as the intensive care unit, by replacing the items related to cancer and its complications and comorbidities with commonly used physiology scores and prognostic tools paired with a prolonged length of stay without clinical progress, although revalidation would be required. In conclusion, this simple 11-item screening tool has been shown to be a valid method for screening for PC problems in patients with cancer hospitalized at a comprehensive cancer center. Consistent with the concept of screening in health care, its scoring system provides clinicians with a means to identify which of their large volume of patients should be considered for PC referral. Using a score of ⱖ 5 as the trigger, 34% of patients who were screened would be recommended to receive a consult, with a high PPV that they would meet ⱖ one of the NCCN referral criteria. Now that it has been shown to be valid and feasible to use in routine caregiving, the impact of screening with the tool on quality of care needs to be established. Acknowledgment We thank our physician and nurse practitioner colleagues who participated anonymously in evaluating the content validity of the tool. Authors’ Disclosures of Potential Conflicts of Interest The authors indicated no potential conflicts of interest.
Author Contributions Conception and design: Paul A. Glare Collection and assembly of data: Paul A. Glare Data analysis and interpretation: All authors Manuscript writing: All authors Final approval of manuscript: All authors Corresponding author: Paul A. Glare, MD, Palliative Medicine Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York NY 10065; e-mail: [email protected].
DOI: 10.1200/JOP.2014.001487; published online ahead of print at jop.ascopubs.org on November 12, 2014.
V O L . 11, I S S U E 1
Copyright © 2014 by American Society of Clinical Oncology
Simple Screening Tool for Identifying Unmet Palliative Care Needs
References 1. Hui D, Elsayem A, De la Cruz M, et al: Availability and integration of palliative care at US cancer centers. JAMA 303:1054-1061, 2010 2. Temel JS, Greer JA, Muzikansky A, et al: Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 363:733-742, 2010 3. Quill TE, Abernethy AP: Generalist plus specialist palliative care: Creating a more sustainable model. N Engl J Med 368:1173-1175, 2013 4. Meier DE, Sieger CE: A Guide to Building a Hospital Based Palliative Care Program. New York, NY, Center to Advance Palliative Care, 2004 5. Levy MH, Back A, Benedetti C, et al: NCCN clinical practice guidelines in oncology: Palliative care. J Natl Compr Canc Netw 7:436-473, 2009 6. Glare PA, Semple D, Stabler SM, et al: Palliative care in the outpatient oncology setting: Evaluation of a practical set of referral criteria. J Oncol Pract 7:366-370, 2011 7. Glare P, Plakovic K, Schloms A, et al: Study using the NCCN guidelines for palliative care to screen patients for palliative care needs and referral to palliative care specialists. J Natl Compr Canc Netw 11:1087-1096, 2013 8. Last JM: A Dictionary of Epidemiology (ed 4). Oxford, United Kingdom, Oxford University Press, 2001 9. Seow H, Barbera L, Sutradhar R, et al: Trajectory of performance status and symptom scores for patients with cancer during the last 6 months of life. J Clin Oncol 29:1151-1158, 2011 10. Chang VT, Hwang SS, Kasimis B, et al: Shorter symptom assessment instruments: The Condensed Memorial Symptom Assessment Scale (CMSAS). Cancer Invest 22:526-536, 2004
Copyright © 2014 by American Society of Clinical Oncology
11. Weissman DE, Meier DE: Identifying patients in need of a palliative care assessment in the hospital setting: A consensus report from the Center to Advance Palliative Care. J Palliat Med 14:17-23, 2011 12. Scandrett KG, Reitschuler-Cross EB, Nelson L, et al: Feasibility and effectiveness of the NEST13⫹ as a screening tool for advanced illness care needs. J Palliat Med 13:161-169, 2010 13. Morita T, Fujimoto K, Namba M, et al: Palliative care needs of cancer outpatients receiving chemotherapy: An audit of a clinical screening project. Support Care Cancer 16:101-107, 2008 14. Adelson K, Paris J, Smith C, et al: Standardized criteria for required palliative care consultation on the solid tumor oncology service. Presented at the American Society of Clinical Oncology Quality Care Symposium, November 1, 2013 (abstr 37) 15. Barbera L, Seow H, Howell D, et al: Symptom burden and performance status in a population-based cohort of ambulatory cancer patients. Cancer 116: 5767-5776, 2010 16. Miller RJ: Predicting survival in the advanced cancer patient. Henry Ford Hosp Med J 39:81-84, 1991 17. Ferrieri P: Proceedings of the Jones criteria workshop. Circulation 106:25212523, 2002 18. Osse BH, Vernooij-Dassen MJ, de Vree BP, et al: Assessment of the need for palliative care as perceived by individual cancer patients and their families: A review of instruments for improving patient participation in palliative care. Cancer 88:900-911, 2000 19. Richardson A, Medina J, Brown V, et al: Patients’ needs assessment in cancer care: A review of assessment tools. Support Care Cancer 15:1125-1144, 2007
J A N U A R Y 2015
•
jop.ascopubs.org
e85
Copyright of Journal of Oncology Practice is the property of American Society of Clinical Oncology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
