Screening Instruments for the Early Detection of Cognitive Impairment in Patients with Multiple Sclerosis Sonya Kim, PhD, CRC; Vance Zemon, PhD; Joseph F. Rath, PhD; MaryAnn Picone, MD; Elizabeth S. Gromisch, MA; Heather Glubo, PhD; Lucia Smith-Wexler, PhD; Frederick W. Foley, PhD Background: Cognitive impairments are common in individuals with MS and adversely affect functioning. Early detection of cognitive impairment, therefore, would enable earlier, and possibly more effective, treatment. We sought to compare self-reports with a short neuropsychological test as possible screening tools for cognitive impairment. Methods: One hundred patients with MS were tested with the Minimal Assessment of Cognitive Function in Multiple Sclerosis; z scores were used to derive the Cognitive Index (CI). Receiver operator characteristic curve analyses were performed, with criteria for impairment set at −1.5 and −2.0 SD below the mean. Scores from two self-reports (the Multiple Sclerosis Neuropsychological Screening Questionnaire–Patient Version and the Behavior Rating Inventory of Executive Function–Adult Version [BRIEF-A]) and a neuropsychological test (the Symbol Digit Modalities Test [SDMT]) were entered as test variables. Exploratory regression analyses were conducted with 1) CI and self-reports and 2) CI and the Problem-Solving Inventory (PSI). Results: Classification accuracy was high or moderately high for SDMT when the criterion was −2.0 or −1.5 SD, respectively, but low for the self-reports. Hierarchical linear regression showed that the SDMT alone was the best predictor of cognitive impairment; adding the self-reports did not improve the model. Exploratory analyses indicated that certain self-reports (BRIEF-A, PSI) provided some explanatory power in separate models. Conclusions: The SDMT is a more accurate screening tool for cognitive impairment; however, self-reports provide additional information and may complement objective testing. Results suggest that screening for cognitive impairment may require a multidimensional approach. Int J MS Care. 2017;19:1–10.
M
ultiple sclerosis (MS) is a demyelinating disease of the central nervous system that affects cognitive and physical functioning.1-4 Cognitive impairments affect 45% to 65% of patients with MS.3 Over time, changes in a patient’s cognition can have greater effects on the patient’s life. Early screening
to detect impaired cognition, followed by a comprehensive neuropsychological evaluation, could allow for earlier treatment,5 better education, and counseling for individuals with MS and their families. Screening for cognitive impairments has relied on rating scales completed by patients and their infor-
From the Department of Neurology (SK), Department of Rehabilitation Medicine (SK), and Rusk Institute of Rehabilitation Medicine (JFR, HG, LS-W), New York University School of Medicine, New York, NY, USA; Ferkauf Graduate School of Psychology, Yeshiva University, Albert Einstein College of Medicine, Bronx, NY, USA (VZ, ESG, FWF); Multiple Sclerosis Comprehensive Care Center, Holy Name Medical Center, Teaneck, NJ, USA (MP, FWF); and Department of Psychology, VA Connecticut Healthcare System, West Haven, CT, USA (ESG). Correspondence: Sonya Kim, PhD, CRC, Department of Rehabilitation Medicine, New York University School of Medicine, 240 E. 38th St., 17th Fl., New York, NY 10016; e-mail: [email protected]. Note: Supplementary material for this article is available on IJMSC Online at ijmsc.org. DOI: 10.7224/1537-2073.2015-001 © 2017 Consortium of Multiple Sclerosis Centers.
International Journal of MS Care 1
Kim et al.
Methods
mants, along with brief standardized tests. Currently available brief standardized tests include the Neuropsychological Screening Battery for Multiple Sclerosis and the Screening Examination for Cognitive Impairment.6 The Brief International Cognitive Assessment for Multiple Sclerosis has recently been recommended as a less time-consuming option.7 Although relatively brief in administration, these batteries still take 15 to 30 minutes to administer, require training,6 and do not include self-report measures, which can be important predictors of functional outcome.8 In contrast, the Symbol Digit Modalities Test (SDMT), which typically measures processing speed,9 takes only 5 minutes to administer. Because processing speed is a primary deficit in individuals with MS, the SDMT may be a particularly good candidate for cognitive screening purposes; previous studies have found it to be effective.10 Similar to the batteries mentioned previously herein, however, the SDMT does not provide information on real-life outcomes. The Multiple Sclerosis Neuropsychological Screening Questionnaire (MSNQ) was developed to screen patients quickly for cognitive impairment in everyday activities. The results of the self-report version, however, were more related to measures of depression.11 In addition to depression, other factors can influence subjective cognitive concerns.12 Levels of anxiety, fatigue, and selfefficacy12 have been found to shape perceived cognitive abilities, which suggests that attitudes, beliefs, and expectations regarding cognitive functioning are important variables that need to be addressed in treatment.8 Particularly, perceptions of cognition and depression tend to overlap, as depression can influence one’s perception of cognitive difficulties.2 In contrast to the self-report version of the MSNQ, the informant version of the MSNQ had a significant relationship to objective testing. Not all patients, however, will have an informant available to accompany them to their clinic visit, and some may not have an informant at all. Because questionnaires mailed to informants cause delays, we evaluated two alternative self-reports of cognition. The objective of this study was to identify the most accurate and efficient means to screen for cognitive impairment in patients with MS by comparing various self-reports, both with each other and with the SDMT. Raw scores for the SDMT were used so that clinic personnel could readily administer the test.
One hundred fifteen consenting participants with a definite diagnosis of MS were recruited from an MS clinic in New Jersey. The study was approved by the institutional review board of Albert Einstein College of Medicine (Bronx, NY). Neuropsychological testing was conducted by a postdoctoral fellow in the clinic. Seven patients failed the effort test (Test of Memory Malingering13 or the Forced Choice of the California Verbal Learning Test, Second Edition14) and were excluded. Eight additional participants were excluded because they were unable to complete all the study questionnaires in the allotted time. Descriptive statistics of the variables of interest were similar for completers versus noncompleters. Table 1 presents the demographic data of the final sample. All the participants completed an evaluation consisting of a clinical interview, self-reports, informant reports, and neuropsychological testing.
Measures The Cognitive Index: Composite Z Score Index Derived from the Minimal Assessment of Cognitive Function in Multiple Sclerosis All the patients were given the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS), a 90-minute battery of tests validated for individuals with MS.15 The MACFIMS includes the following: Controlled Oral Word Association Test16; Table 1. Characteristics of the 100 study participants Characteristic Sex, M/F, No. Age, mean (SD) [range], y Education, mean (SD) [range], y Multiple sclerosis diagnosis, No. Relapsing-remitting Primary progressive Secondary progressive Vocational status, No. Unemployed/disabled (objectively) Unemployed/disabled (subjectively) Unemployed/not disabled Homemaker/student/volunteer (>10 h) Part-time, reduced capacity (subjectively), 10–20 h Full-time, reduced capacity or responsibility Full-time, unchanged capacity Retired due to age
International Journal of MS Care 2
Values 25/75 46.61 (11.72) [18–77] 14.51 (2.34) [9–20] 88 5 7 36 11 10 7 5 1 26 4
Cognitive Impairment Screening in MS
Judgment of Line Orientation Test16; California Verbal Learning Test, Second Edition14; Brief Visuospatial Memory Test–Revised17; Paced Auditory Serial Addition Test18; SDMT9; and Delis-Kaplan Executive Function System Sorting Test.19 Scores from the MACFIMS were converted to z scores; 12 of 13 measures of the MACFIMS were averaged to derive a single metric. The composite z score was termed the Cognitive Index (CI) and represented patients’ overall cognitive function. (Because the Brief Visuospatial Memory Test–Revised Recognition Trial has a low ceiling—the highest score that can be attained is greater than the 16th percentile—the score for this measure was not included in the CI.) Table 2 presents the patients’ performance and frequency of impairment on the MACFIMS. Impairment was defined by the CI as 2 or more test failures, using cutoff values of less than or equal to −1.5 and −2.0 SD below the mean. A dichotomous measure of cognitive impair-
ment, based on the CI (pass/fail), was defined by a criterion set at either −2.0 or −1.5 SD below the mean. This CI was also used in the analysis as a continuous variable. The SDMT is part of the CI; therefore, an inflated correlation with the CI is expected if the SDMT is not removed when calculating the CI. Because the SDMT is a measure of processing speed (a primary deficit in individuals with MS20), it was not removed from the CI in analyses that involved only the self-reports as predictors of impairment. However, all analyses that included the SDMT as a predictor were performed with the SDMT excluded from the CI. Neuropsychological Test One neuropsychological test, the SDMT, was used as a screening tool for comparative evaluation. It typically measures processing speed9 and is one of seven tests that compose the MACFIMS. Because this test takes only 5 minutes to administer and is effective for screening purposes,10 it was a good candidate for determining whether this single test could accurately correspond to the derived CI. Raw scores were used to make it easier for nurses to administer and interpret results. Standard scores, adjusted for age and education,21 were analyzed separately to confirm that any clinically significant findings would not be affected by their use.
Table 2. Performance on the MACFIMS: frequency of failures (N = 100) Failures, No. Variable Cognitive Indexa Failed ≥2 tests of the MACFIMSa F-A-S CVLT-II total recall CVLT-II short delay free recall CVLT-II long delay free recall CVLT-II recognition (hits) PASAT-3”b PASAT-2”b SDMT BVMT-R Total Recall BVMT-R Delayed Recall JLOT DKEFS-Free Sorting Description scorec
z score of z score of Missing ≤ −1.5 ≤ −2.0 data, No. 22 56 29 25 36 39 26 50 30 21 47 36 15 9
13 37 9 10 23 22 24 34 14 14 35 26 12 5
0 0 0 0 0 0 0 2 2 0 0 0 0 5
Neuropsychological Self-reports Patients completed self-report measures related to self-appraisals of neuropsychological functioning. The MSNQ–Patient Version. The MSNQ–Patient Version (MSNQ-P) is a 15-item questionnaire, validated for the MS population,22 of cognitive and neuropsychiatric dysfunction. Patients rate themselves from 0 (never; does not occur) to 4 (very often; very disrupted) on specific cognitive and behavioral problems that may arise in daily life. For example, they may be asked, “Do you lose your thoughts while listening to somebody speak? Do you have difficulty controlling your impulses?” Behavior Rating Inventory of Executive Function–Adult Version, Self-Report. The Behavior Rating Inventory of Executive Function–Adult Version (BRIEF-A) captures individuals’ views of their own executive functioning (see the study by Roth et al.23 for psychometric properties). It contains 70 items and yields an overall score (Global Executive Composite) that combines two indices: the Metacognitive Index (MI) and the Behavioral Regulation Index (BRI). The MI (regulation of cognition, eg, I lie around the house a lot; I forget what I
Abbreviations: BVMT-R, Brief Visuospatial Memory Test–Revised; CVLT-II, California Verbal Learning Test, Second Edition (adult version); DKEFS, Delis-Kaplan Executive Function System; F-A-S, Controlled Oral Word Association Test F-A-S; JLOT, Judgment of Line Orientation Test; MACFIMS, Minimal Assessment of Cognitive Function in Multiple Sclerosis; PASAT-3” and -2”, Paced Auditory Serial Addition Test-3” [3-second] and -2” [2-second]; SDMT, Symbol Digit Modalities Test. a The SDMT was excluded from the MACFIMS battery. b Subsequent to the practice trials of the PASAT-2” and -3”, two patients refused to complete the PASAT. For these patients, the Cognitive Index was computed without the test. c Owing to fine motor impairments, five patients were unable to complete the DKEFS. For these patients, the Cognitive Index was computed without the DKEFS.
International Journal of MS Care 3
Kim et al.
Statistical Analysis
am doing in the middle of things) and the BRI (regulation of behavior, eg, I have problems waiting my turn; I have emotional outbursts for little reason) were recently validated for individuals with MS.24 Because one aim of the study was to identify a screening tool that clinic personnel could administer and interpret, raw scores were used, and only the MI and BRI summary indices are reported. Problem-Solving Inventory. Consisting of 32 items rated using a 6-point Likert format (see the manual by Heppner25 for the psychometric properties), the Problem-Solving Inventory (PSI) is a self-appraisal measurement of how well people believe they can handle problems. The PSI probes individuals’ global beliefs about everyday problems and their perception of their problem-solving ability.26 In this regard, the PSI has been interpreted as an indicator of confidence in cognitive abilities and self-efficacy in managing real-life problems.8,27 A total score is derived from three subscales: problem-solving confidence (PSC; self-assurance, belief and trust in problem-solving skills), approach-avoidance style (tendency to avoid or withdraw from problems), and personal control (belief that one is in control of emotions and behavior when solving problems). Higher scores reflect a negative self-appraisal. The PSI was added to the study because Rath et al.27 demonstrated that the way in which an individual self-appraised his or her capabilities, using the PSI, accounts for a significant proportion of the variance in community functioning after traumatic brain injury. The PSI was added late in the present study, and, thus, only 41 participants completed this measure; descriptive statistics, however, summarized for completers versus noncompleters of the PSI, and a multivariate analysis of variance of the two groups indicate that the two groups were not significantly different. In this study, the adolescent version of the PSI was used. Self-reported Mood: the Beck Depression Inventory–II. The Beck Depression Inventory–II (BDI-II) is a 21-item multiple-choice questionnaire that measures the presence and severity of depressive symptoms. Each item is scored from 0 to 3 in terms of intensity, with total scores ranging from 0 to 63.28 The BDI-II is widely used in MS research and shows high internal consistency in psychiatric clinic outpatients (Cronbach α = 0.92)28 and in the MS population (Cronbach α = 0.86).29 The BDIII was added as a covariate in this study to investigate the role of depression in objective cognitive performance and self-reports of cognition for individuals with MS.
Statistical analyses were performed using IBM SPSS Statistics for Windows, version 22.0 (IBM Corp, Armonk, NY). Descriptive statistics were used to determine whether the data met the necessary assumptions for parametric statistical analysis. Collinearity diagnostic tests were performed. Receiver operating characteristic (ROC) curve analyses were performed to test each of the self-report measures and the SDMT for classification accuracy in terms of identifying cognitive impairment based on the criterion-determined dichotomous outcome measure of the CI. Hierarchical linear regression analyses were conducted to determine predictors that best explain impairment as expressed by the continuous measure of the CI. The PSI was excluded from this set of analyses because of the small subset of the sample that completed this measure. A separate, secondary analysis was performed to explore the linear relation between the PSI and the CI.
Results A visual inspection of the frequency histogram and the P-P plot for the outcome measure (the CI) confirms that the data were normally distributed. Diagnostic statistics to assess collinearity (variance inflation factor and tolerance) demonstrated that multicollinearity was not a problem with the given set of predictors, and application of the Durbin-Watson statistic showed independence of serial residuals. Table 3 presents the descriptive statistics of the outcome measure, five predictor/classification variables, and a covariate (BDI-II); Table 4 presents the Pearson correlations among these variables. Also examined were the linear relations between depression and self-reports, which were moderately strong (BDI-II/ BRIEF-A BRI: R2 = 0.32; BDI-II/BRIEF-A MI: R2 = 0.39; BDI-II/MSNQ: R2 = 0.43). The more depressed the patients were, the more negatively they appraised their cognition. The CI was not significantly related to depression (R2 = 0.03). There was also a strong positive linear relation among all the self-reports (BRI/MSNQP: R2 = 0.48; BRI/MI: R2 = 0.52; MI/MSNQ-P: R2 = 0.58). The PSI was excluded from this intercorrelation analysis because it was not included in the hierarchical regression analysis on grounds of the small number of 41 patients associated with this measure. When a separate correlation analysis with the PSI and the other measures was performed, however, the results showed that the PSI scales were mostly significantly correlated with the other measures with moderate coefficients (Table 5).
International Journal of MS Care 4
Cognitive Impairment Screening in MS
the mean, the MSNQ-P was significant, whereas the two summary indices of the BRIEF-A were just shy of significance. The ROC curve analyses of the SDMT (excluding the SDMT from the CI) were significant given both criteria. Classification accuracy, as measured by area under the ROC curve (AUC), was high for the SDMT when the criterion was −2.0 SD and was moderately high when the criterion was −1.5 SD. Specificity was high when the SD criterion was set at −2.0 and was moderately high when it was set at −1.5 SD. Moderate sensitivity was obtained for both criteria, with a somewhat lower cutoff score given the −2.0 SD criterion. Minimal change was observed regardless of whether the SDMT was included in the CI; excluding the SDMT from the CI resulted in a higher SDMT cutoff score (≤44 vs. ≤39) when the criterion was set at −1.5 SD. There was no significant difference when SDMT-transformed scores, as opposed to raw scores, were used in the analyses. Classification accuracy, sensitivity, and specificity were not high for any of the other measures. The PSI was not statistically significant for classifying impairment according to either criterion. Table 6 presents the diagnostic accuracy of the SDMT, MSNQ-P, and BRIEF-A.
Table 3. Descriptive statistics of variables used in the analyses Variable
Patients, No. Mean (SD)
Range
Cognitive Index (z score composite) with SDMT
100
−0.78 (0.88) −3.04 to 0.78
Cognitive Index (z score composite) without SDMT
100
−0.78 (0.89) −2.89 to 0.85
SDMT raw score MSNQ-P BRIEF-A MIa BRIEF-A BRIa PSI-PSCb BDI-II
100 100 100 100 41 100
47.91 (11.9) 31.56 (12.53) 79.14 (16.3) 54.9 (11.61) 30.98 (10.81) 19.24 (11.05)
13 to 73 0 to 58 40 to 116 30 to 85 12 to 59 0 to 53
Abbreviations: BDI-II, Beck Depression Inventory–II; BRI, Behavioral Regulation Index; BRIEF-A, Behavior Rating Inventory of Executive Function–Adult Version; MI, Metacognition Index; MSNQ-P, Multiple Sclerosis Neuropsychological Screening Questionnaire–Patient Version; PSC, problem-solving confidence subscale; PSI, ProblemSolving Inventory; SDMT, Symbol Digit Modalities Test. Note: The Cognitive Index, the outcome measure, is used for classification purposes; the BDI-II is used as a covariate; and the remaining variables are used as predictors. a The BRI is based on published norms of healthy adults: mean (SD) z score = −1.48 (1.33); the MI is based on published norms of healthy adults: mean (SD) z score = −1.89 (1.39). b The PSI-PSC is based on norms of healthy adults: mean (SD) z score = −0.73 (1.28).
Models to Predict the CI Scatterplots of each predictor versus the outcome variable were reviewed. The CI versus the SDMT illustrates a moderately strong positive linear relation (R2 = 0.50, P < .001). Weak negative linear relations exist between the CI and the self-reports (CI/MSNQ-P: R2 = 0.05, P = .026; CI/BRI: R2 = 0.07, P = .008; and CI/MI: R2 = 0.03, P = .073). Hierarchical linear regression analyses were performed to identify the best model for predicting impairment,
Impairment Classification and the Accuracy of Self-reports According to ROC curve analyses with the CI criterion set at −2.0 SD below the mean, none of the self-report measures were significant predictors of the CI (MSNQ-P: AUC = 0.607, P = .231; BRIEF-A BRI: AUC = 0.629, P = .149; BRIEF-A MI: AUC = 0.627, P = .125). With the CI criterion set at −1.5 SD below
Table 4. Intercorrelations among predictor variables, depression, and the CI Measure 1. SDMT 2. MSNQ-P 3. BRIEF-A MIb 4. BRIEF-A BRIb 5. CI 6. BDI-II
1
2
3
4
— −0.214a −0.225a −0.257c 0.748c,d 0.704c,e −0.082
— 0.761c 0.693c −0.223a,d −0.212a,e 0.654c
— 0.721c −0.180d −0.167e 0.624c
— −0.263c,d −0.256a,e 0.564c
5
6
— −0.168
—
Abbreviations: BDI-II, Beck Depression Inventory–II; BRI, Behavioral Regulation Index; BRIEF-A, Behavior Rating Inventory of Executive Function–Adult Version; CI, Cognitive Index; MI, Metacognition Index; MSNQ-P, Multiple Sclerosis Neuropsychological Screening Questionnaire–Patient Version; SDMT, Symbol Digit Modalities Test. a P < .05, two-tailed. b Higher score indicates greater difficulty with behavioral regulation (BRI) and metacognition (MI). c P < .01, two-tailed. d Correlations with SDMT included in the Cognitive Index. e Correlations with SDMT excluded from the Cognitive Index.
International Journal of MS Care 5
Kim et al.
SDMT was entered at the second step (model 2), followed by the self-reports (BRIEF-A BRI, MSNQ-P, and BRIEF-A MI). The SDMT alone emerged as a significant predictor of impairment: the BDI-II in model 1 yielded an adjusted R2 of 0.02 (P = .088). Adding the SDMT in model 2 yielded an adjusted R2 of 0.50; this change in R2 of 0.48 was significant (P < .001). The self-reports did not provide significant explanatory power (ΔR2 of 0.01 to
M
ultiple sclerosis (MS) is a demyelinating disease of the central nervous system that affects cognitive and physical functioning.1-4 Cognitive impairments affect 45% to 65% of patients with MS.3 Over time, changes in a patient’s cognition can have greater effects on the patient’s life. Early screening
to detect impaired cognition, followed by a comprehensive neuropsychological evaluation, could allow for earlier treatment,5 better education, and counseling for individuals with MS and their families. Screening for cognitive impairments has relied on rating scales completed by patients and their infor-
From the Department of Neurology (SK), Department of Rehabilitation Medicine (SK), and Rusk Institute of Rehabilitation Medicine (JFR, HG, LS-W), New York University School of Medicine, New York, NY, USA; Ferkauf Graduate School of Psychology, Yeshiva University, Albert Einstein College of Medicine, Bronx, NY, USA (VZ, ESG, FWF); Multiple Sclerosis Comprehensive Care Center, Holy Name Medical Center, Teaneck, NJ, USA (MP, FWF); and Department of Psychology, VA Connecticut Healthcare System, West Haven, CT, USA (ESG). Correspondence: Sonya Kim, PhD, CRC, Department of Rehabilitation Medicine, New York University School of Medicine, 240 E. 38th St., 17th Fl., New York, NY 10016; e-mail: [email protected]. Note: Supplementary material for this article is available on IJMSC Online at ijmsc.org. DOI: 10.7224/1537-2073.2015-001 © 2017 Consortium of Multiple Sclerosis Centers.
International Journal of MS Care 1
Kim et al.
Methods
mants, along with brief standardized tests. Currently available brief standardized tests include the Neuropsychological Screening Battery for Multiple Sclerosis and the Screening Examination for Cognitive Impairment.6 The Brief International Cognitive Assessment for Multiple Sclerosis has recently been recommended as a less time-consuming option.7 Although relatively brief in administration, these batteries still take 15 to 30 minutes to administer, require training,6 and do not include self-report measures, which can be important predictors of functional outcome.8 In contrast, the Symbol Digit Modalities Test (SDMT), which typically measures processing speed,9 takes only 5 minutes to administer. Because processing speed is a primary deficit in individuals with MS, the SDMT may be a particularly good candidate for cognitive screening purposes; previous studies have found it to be effective.10 Similar to the batteries mentioned previously herein, however, the SDMT does not provide information on real-life outcomes. The Multiple Sclerosis Neuropsychological Screening Questionnaire (MSNQ) was developed to screen patients quickly for cognitive impairment in everyday activities. The results of the self-report version, however, were more related to measures of depression.11 In addition to depression, other factors can influence subjective cognitive concerns.12 Levels of anxiety, fatigue, and selfefficacy12 have been found to shape perceived cognitive abilities, which suggests that attitudes, beliefs, and expectations regarding cognitive functioning are important variables that need to be addressed in treatment.8 Particularly, perceptions of cognition and depression tend to overlap, as depression can influence one’s perception of cognitive difficulties.2 In contrast to the self-report version of the MSNQ, the informant version of the MSNQ had a significant relationship to objective testing. Not all patients, however, will have an informant available to accompany them to their clinic visit, and some may not have an informant at all. Because questionnaires mailed to informants cause delays, we evaluated two alternative self-reports of cognition. The objective of this study was to identify the most accurate and efficient means to screen for cognitive impairment in patients with MS by comparing various self-reports, both with each other and with the SDMT. Raw scores for the SDMT were used so that clinic personnel could readily administer the test.
One hundred fifteen consenting participants with a definite diagnosis of MS were recruited from an MS clinic in New Jersey. The study was approved by the institutional review board of Albert Einstein College of Medicine (Bronx, NY). Neuropsychological testing was conducted by a postdoctoral fellow in the clinic. Seven patients failed the effort test (Test of Memory Malingering13 or the Forced Choice of the California Verbal Learning Test, Second Edition14) and were excluded. Eight additional participants were excluded because they were unable to complete all the study questionnaires in the allotted time. Descriptive statistics of the variables of interest were similar for completers versus noncompleters. Table 1 presents the demographic data of the final sample. All the participants completed an evaluation consisting of a clinical interview, self-reports, informant reports, and neuropsychological testing.
Measures The Cognitive Index: Composite Z Score Index Derived from the Minimal Assessment of Cognitive Function in Multiple Sclerosis All the patients were given the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS), a 90-minute battery of tests validated for individuals with MS.15 The MACFIMS includes the following: Controlled Oral Word Association Test16; Table 1. Characteristics of the 100 study participants Characteristic Sex, M/F, No. Age, mean (SD) [range], y Education, mean (SD) [range], y Multiple sclerosis diagnosis, No. Relapsing-remitting Primary progressive Secondary progressive Vocational status, No. Unemployed/disabled (objectively) Unemployed/disabled (subjectively) Unemployed/not disabled Homemaker/student/volunteer (>10 h) Part-time, reduced capacity (subjectively), 10–20 h Full-time, reduced capacity or responsibility Full-time, unchanged capacity Retired due to age
International Journal of MS Care 2
Values 25/75 46.61 (11.72) [18–77] 14.51 (2.34) [9–20] 88 5 7 36 11 10 7 5 1 26 4
Cognitive Impairment Screening in MS
Judgment of Line Orientation Test16; California Verbal Learning Test, Second Edition14; Brief Visuospatial Memory Test–Revised17; Paced Auditory Serial Addition Test18; SDMT9; and Delis-Kaplan Executive Function System Sorting Test.19 Scores from the MACFIMS were converted to z scores; 12 of 13 measures of the MACFIMS were averaged to derive a single metric. The composite z score was termed the Cognitive Index (CI) and represented patients’ overall cognitive function. (Because the Brief Visuospatial Memory Test–Revised Recognition Trial has a low ceiling—the highest score that can be attained is greater than the 16th percentile—the score for this measure was not included in the CI.) Table 2 presents the patients’ performance and frequency of impairment on the MACFIMS. Impairment was defined by the CI as 2 or more test failures, using cutoff values of less than or equal to −1.5 and −2.0 SD below the mean. A dichotomous measure of cognitive impair-
ment, based on the CI (pass/fail), was defined by a criterion set at either −2.0 or −1.5 SD below the mean. This CI was also used in the analysis as a continuous variable. The SDMT is part of the CI; therefore, an inflated correlation with the CI is expected if the SDMT is not removed when calculating the CI. Because the SDMT is a measure of processing speed (a primary deficit in individuals with MS20), it was not removed from the CI in analyses that involved only the self-reports as predictors of impairment. However, all analyses that included the SDMT as a predictor were performed with the SDMT excluded from the CI. Neuropsychological Test One neuropsychological test, the SDMT, was used as a screening tool for comparative evaluation. It typically measures processing speed9 and is one of seven tests that compose the MACFIMS. Because this test takes only 5 minutes to administer and is effective for screening purposes,10 it was a good candidate for determining whether this single test could accurately correspond to the derived CI. Raw scores were used to make it easier for nurses to administer and interpret results. Standard scores, adjusted for age and education,21 were analyzed separately to confirm that any clinically significant findings would not be affected by their use.
Table 2. Performance on the MACFIMS: frequency of failures (N = 100) Failures, No. Variable Cognitive Indexa Failed ≥2 tests of the MACFIMSa F-A-S CVLT-II total recall CVLT-II short delay free recall CVLT-II long delay free recall CVLT-II recognition (hits) PASAT-3”b PASAT-2”b SDMT BVMT-R Total Recall BVMT-R Delayed Recall JLOT DKEFS-Free Sorting Description scorec
z score of z score of Missing ≤ −1.5 ≤ −2.0 data, No. 22 56 29 25 36 39 26 50 30 21 47 36 15 9
13 37 9 10 23 22 24 34 14 14 35 26 12 5
0 0 0 0 0 0 0 2 2 0 0 0 0 5
Neuropsychological Self-reports Patients completed self-report measures related to self-appraisals of neuropsychological functioning. The MSNQ–Patient Version. The MSNQ–Patient Version (MSNQ-P) is a 15-item questionnaire, validated for the MS population,22 of cognitive and neuropsychiatric dysfunction. Patients rate themselves from 0 (never; does not occur) to 4 (very often; very disrupted) on specific cognitive and behavioral problems that may arise in daily life. For example, they may be asked, “Do you lose your thoughts while listening to somebody speak? Do you have difficulty controlling your impulses?” Behavior Rating Inventory of Executive Function–Adult Version, Self-Report. The Behavior Rating Inventory of Executive Function–Adult Version (BRIEF-A) captures individuals’ views of their own executive functioning (see the study by Roth et al.23 for psychometric properties). It contains 70 items and yields an overall score (Global Executive Composite) that combines two indices: the Metacognitive Index (MI) and the Behavioral Regulation Index (BRI). The MI (regulation of cognition, eg, I lie around the house a lot; I forget what I
Abbreviations: BVMT-R, Brief Visuospatial Memory Test–Revised; CVLT-II, California Verbal Learning Test, Second Edition (adult version); DKEFS, Delis-Kaplan Executive Function System; F-A-S, Controlled Oral Word Association Test F-A-S; JLOT, Judgment of Line Orientation Test; MACFIMS, Minimal Assessment of Cognitive Function in Multiple Sclerosis; PASAT-3” and -2”, Paced Auditory Serial Addition Test-3” [3-second] and -2” [2-second]; SDMT, Symbol Digit Modalities Test. a The SDMT was excluded from the MACFIMS battery. b Subsequent to the practice trials of the PASAT-2” and -3”, two patients refused to complete the PASAT. For these patients, the Cognitive Index was computed without the test. c Owing to fine motor impairments, five patients were unable to complete the DKEFS. For these patients, the Cognitive Index was computed without the DKEFS.
International Journal of MS Care 3
Kim et al.
Statistical Analysis
am doing in the middle of things) and the BRI (regulation of behavior, eg, I have problems waiting my turn; I have emotional outbursts for little reason) were recently validated for individuals with MS.24 Because one aim of the study was to identify a screening tool that clinic personnel could administer and interpret, raw scores were used, and only the MI and BRI summary indices are reported. Problem-Solving Inventory. Consisting of 32 items rated using a 6-point Likert format (see the manual by Heppner25 for the psychometric properties), the Problem-Solving Inventory (PSI) is a self-appraisal measurement of how well people believe they can handle problems. The PSI probes individuals’ global beliefs about everyday problems and their perception of their problem-solving ability.26 In this regard, the PSI has been interpreted as an indicator of confidence in cognitive abilities and self-efficacy in managing real-life problems.8,27 A total score is derived from three subscales: problem-solving confidence (PSC; self-assurance, belief and trust in problem-solving skills), approach-avoidance style (tendency to avoid or withdraw from problems), and personal control (belief that one is in control of emotions and behavior when solving problems). Higher scores reflect a negative self-appraisal. The PSI was added to the study because Rath et al.27 demonstrated that the way in which an individual self-appraised his or her capabilities, using the PSI, accounts for a significant proportion of the variance in community functioning after traumatic brain injury. The PSI was added late in the present study, and, thus, only 41 participants completed this measure; descriptive statistics, however, summarized for completers versus noncompleters of the PSI, and a multivariate analysis of variance of the two groups indicate that the two groups were not significantly different. In this study, the adolescent version of the PSI was used. Self-reported Mood: the Beck Depression Inventory–II. The Beck Depression Inventory–II (BDI-II) is a 21-item multiple-choice questionnaire that measures the presence and severity of depressive symptoms. Each item is scored from 0 to 3 in terms of intensity, with total scores ranging from 0 to 63.28 The BDI-II is widely used in MS research and shows high internal consistency in psychiatric clinic outpatients (Cronbach α = 0.92)28 and in the MS population (Cronbach α = 0.86).29 The BDIII was added as a covariate in this study to investigate the role of depression in objective cognitive performance and self-reports of cognition for individuals with MS.
Statistical analyses were performed using IBM SPSS Statistics for Windows, version 22.0 (IBM Corp, Armonk, NY). Descriptive statistics were used to determine whether the data met the necessary assumptions for parametric statistical analysis. Collinearity diagnostic tests were performed. Receiver operating characteristic (ROC) curve analyses were performed to test each of the self-report measures and the SDMT for classification accuracy in terms of identifying cognitive impairment based on the criterion-determined dichotomous outcome measure of the CI. Hierarchical linear regression analyses were conducted to determine predictors that best explain impairment as expressed by the continuous measure of the CI. The PSI was excluded from this set of analyses because of the small subset of the sample that completed this measure. A separate, secondary analysis was performed to explore the linear relation between the PSI and the CI.
Results A visual inspection of the frequency histogram and the P-P plot for the outcome measure (the CI) confirms that the data were normally distributed. Diagnostic statistics to assess collinearity (variance inflation factor and tolerance) demonstrated that multicollinearity was not a problem with the given set of predictors, and application of the Durbin-Watson statistic showed independence of serial residuals. Table 3 presents the descriptive statistics of the outcome measure, five predictor/classification variables, and a covariate (BDI-II); Table 4 presents the Pearson correlations among these variables. Also examined were the linear relations between depression and self-reports, which were moderately strong (BDI-II/ BRIEF-A BRI: R2 = 0.32; BDI-II/BRIEF-A MI: R2 = 0.39; BDI-II/MSNQ: R2 = 0.43). The more depressed the patients were, the more negatively they appraised their cognition. The CI was not significantly related to depression (R2 = 0.03). There was also a strong positive linear relation among all the self-reports (BRI/MSNQP: R2 = 0.48; BRI/MI: R2 = 0.52; MI/MSNQ-P: R2 = 0.58). The PSI was excluded from this intercorrelation analysis because it was not included in the hierarchical regression analysis on grounds of the small number of 41 patients associated with this measure. When a separate correlation analysis with the PSI and the other measures was performed, however, the results showed that the PSI scales were mostly significantly correlated with the other measures with moderate coefficients (Table 5).
International Journal of MS Care 4
Cognitive Impairment Screening in MS
the mean, the MSNQ-P was significant, whereas the two summary indices of the BRIEF-A were just shy of significance. The ROC curve analyses of the SDMT (excluding the SDMT from the CI) were significant given both criteria. Classification accuracy, as measured by area under the ROC curve (AUC), was high for the SDMT when the criterion was −2.0 SD and was moderately high when the criterion was −1.5 SD. Specificity was high when the SD criterion was set at −2.0 and was moderately high when it was set at −1.5 SD. Moderate sensitivity was obtained for both criteria, with a somewhat lower cutoff score given the −2.0 SD criterion. Minimal change was observed regardless of whether the SDMT was included in the CI; excluding the SDMT from the CI resulted in a higher SDMT cutoff score (≤44 vs. ≤39) when the criterion was set at −1.5 SD. There was no significant difference when SDMT-transformed scores, as opposed to raw scores, were used in the analyses. Classification accuracy, sensitivity, and specificity were not high for any of the other measures. The PSI was not statistically significant for classifying impairment according to either criterion. Table 6 presents the diagnostic accuracy of the SDMT, MSNQ-P, and BRIEF-A.
Table 3. Descriptive statistics of variables used in the analyses Variable
Patients, No. Mean (SD)
Range
Cognitive Index (z score composite) with SDMT
100
−0.78 (0.88) −3.04 to 0.78
Cognitive Index (z score composite) without SDMT
100
−0.78 (0.89) −2.89 to 0.85
SDMT raw score MSNQ-P BRIEF-A MIa BRIEF-A BRIa PSI-PSCb BDI-II
100 100 100 100 41 100
47.91 (11.9) 31.56 (12.53) 79.14 (16.3) 54.9 (11.61) 30.98 (10.81) 19.24 (11.05)
13 to 73 0 to 58 40 to 116 30 to 85 12 to 59 0 to 53
Abbreviations: BDI-II, Beck Depression Inventory–II; BRI, Behavioral Regulation Index; BRIEF-A, Behavior Rating Inventory of Executive Function–Adult Version; MI, Metacognition Index; MSNQ-P, Multiple Sclerosis Neuropsychological Screening Questionnaire–Patient Version; PSC, problem-solving confidence subscale; PSI, ProblemSolving Inventory; SDMT, Symbol Digit Modalities Test. Note: The Cognitive Index, the outcome measure, is used for classification purposes; the BDI-II is used as a covariate; and the remaining variables are used as predictors. a The BRI is based on published norms of healthy adults: mean (SD) z score = −1.48 (1.33); the MI is based on published norms of healthy adults: mean (SD) z score = −1.89 (1.39). b The PSI-PSC is based on norms of healthy adults: mean (SD) z score = −0.73 (1.28).
Models to Predict the CI Scatterplots of each predictor versus the outcome variable were reviewed. The CI versus the SDMT illustrates a moderately strong positive linear relation (R2 = 0.50, P < .001). Weak negative linear relations exist between the CI and the self-reports (CI/MSNQ-P: R2 = 0.05, P = .026; CI/BRI: R2 = 0.07, P = .008; and CI/MI: R2 = 0.03, P = .073). Hierarchical linear regression analyses were performed to identify the best model for predicting impairment,
Impairment Classification and the Accuracy of Self-reports According to ROC curve analyses with the CI criterion set at −2.0 SD below the mean, none of the self-report measures were significant predictors of the CI (MSNQ-P: AUC = 0.607, P = .231; BRIEF-A BRI: AUC = 0.629, P = .149; BRIEF-A MI: AUC = 0.627, P = .125). With the CI criterion set at −1.5 SD below
Table 4. Intercorrelations among predictor variables, depression, and the CI Measure 1. SDMT 2. MSNQ-P 3. BRIEF-A MIb 4. BRIEF-A BRIb 5. CI 6. BDI-II
1
2
3
4
— −0.214a −0.225a −0.257c 0.748c,d 0.704c,e −0.082
— 0.761c 0.693c −0.223a,d −0.212a,e 0.654c
— 0.721c −0.180d −0.167e 0.624c
— −0.263c,d −0.256a,e 0.564c
5
6
— −0.168
—
Abbreviations: BDI-II, Beck Depression Inventory–II; BRI, Behavioral Regulation Index; BRIEF-A, Behavior Rating Inventory of Executive Function–Adult Version; CI, Cognitive Index; MI, Metacognition Index; MSNQ-P, Multiple Sclerosis Neuropsychological Screening Questionnaire–Patient Version; SDMT, Symbol Digit Modalities Test. a P < .05, two-tailed. b Higher score indicates greater difficulty with behavioral regulation (BRI) and metacognition (MI). c P < .01, two-tailed. d Correlations with SDMT included in the Cognitive Index. e Correlations with SDMT excluded from the Cognitive Index.
International Journal of MS Care 5
Kim et al.
SDMT was entered at the second step (model 2), followed by the self-reports (BRIEF-A BRI, MSNQ-P, and BRIEF-A MI). The SDMT alone emerged as a significant predictor of impairment: the BDI-II in model 1 yielded an adjusted R2 of 0.02 (P = .088). Adding the SDMT in model 2 yielded an adjusted R2 of 0.50; this change in R2 of 0.48 was significant (P < .001). The self-reports did not provide significant explanatory power (ΔR2 of 0.01 to
