Psychiatric Rehabilitation Journal 2014, Vol. 37, No. 1, 17–23
In the public domain DOI: 10.1037/prj0000035
Brief Version of the Internalized Stigma of Mental Illness (ISMI) Scale: Psychometric Properties and Relationship to Depression, Self Esteem, Recovery Orientation, Empowerment, and Perceived Devaluation and Discrimination Jennifer E. Boyd
Poorni G. Otilingam
San Francisco VA Medical Center and University of California, San Francisco
San Francisco VA Medical Center
Bruce R. DeForge School of Social Work, University of Maryland, Baltimore Objective: The internalized stigma of mental illness impedes recovery and is associated with increased depression, reduced self-esteem, reduced recovery orientation, reduced empowerment, and increased perceived devaluation and discrimination. The Internalized Stigma of Mental Illness (ISMI) scale is a 29-item self-report questionnaire developed with consumer input that includes the following subscales: Alienation, Discrimination Experience, Social Withdrawal, Stereotype Endorsement, and Stigma Resistance. Here we present a 10-item version of the ISMI containing the two strongest items from each subscale. Method: Participants were all outpatient veterans with serious mental illness. Following the rigorous scale-reduction methods set forth by Stanton and colleagues (2002), we selected the 10 items, tested the psychometrics of the shortened scale in the original validation sample (N ⫽ 127), and cross-checked the results in a second dataset (N ⫽ 760). Results: As expected, the ISMI-10 retained the essential properties of the ISMI-29, including adequate internal consistency reliability and external validity in relation to depression, self-esteem, recovery orientation, perceived devaluation and discrimination, and empowerment. The ISMI-10 scores are normally distributed and have similar descriptive statistics to the ISMI-29. The reliability and depression findings were replicated in a cross-validation sample. Conclusions and Implications for Practice: We conclude that the ISMI-10 has strong psychometric properties and is a practical, reliable, and valid alternative to the original ISMI-29. Future work should test the ISMI-10 in more diverse samples. This shorter version should reduce respondent burden in program evaluation projects that seek to determine whether participation in psychosocial rehabilitation programming reduces internalized stigma. Keywords: stigma of mental illness, questionnaire, depression, self-esteem, empowerment
objective effects such as lost housing or social ostracism, stigma also has subjective psychological effects on people with mental illness. Internalized stigma is the psychological point of impact of societal stigma on people with mental illness. As members of the society in which we all live, people with mental illness often apply negative stereotypes, biases, and discrimination to themselves, which impedes their recovery process by eroding their morale (Corrigan & Watson, 2002; Link, Cullen, Struening, Shrout, & Dohrenwend, 1989; Ritsher (Boyd) & Phelan, 2004; West, Yanos, Smith, Roe, & Lysaker, 2011). Also known as self-stigma, internalized stigma is associated with increased depression, reduced self-esteem, reduced recovery orientation, reduced empowerment, and increased perceived devaluation and discrimination (Drapalski et al., 2013; Livingston & Boyd, 2010; Lucksted et al., 2011; Ritsher (Boyd), Otilingam, & Grajales, 2003). Several measures of internalized stigma have been developed (Brohan, Slade, Clement, & Thornicroft, 2010; Livingston & Boyd, 2010). One of the most widely used measures is the Internalized Stigma of Mental Illness (ISMI) scale, which was developed with consumer input and contains 29 items and 5 subscales (Boyd, Adler, Otilingam & Peters, 2014; Ritsher et al., 2003). The
The stigma of mental illness is the negative stereotyping, biases, and discrimination often directed at people with mental illness (Hinshaw, 2007). Such stigma is widespread across societies around the world and has been shown to have a variety of negative effects (Dickerson, Sommerville, Origoni, Ringel, & Parente, 2002; Hinshaw, 2007; Overton & Medina, 2008; Wahl, 1999). In addition to
Jennifer E. Boyd, Department of Psychiatry, San Francisco VA Medical Center and University of California, San Francisco; Poorni G. Otilingam, Mental Health Service, San Francisco VA Medical Center; and Bruce R. DeForge, School of Social Work, University of Maryland, Baltimore. We thank the Department of Veterans Affairs, Northeast Program Evaluation Center and the Office of Patient Care Services Data Transfer Agreement SNCMI (2011-02), for providing the cross-validation sample. Thanks also go to Jeanne McPhee for assistance with manuscript preparation. Correspondence concerning this article should be addressed to Jennifer E. Boyd, Department of Psychiatry, San Francisco VA Medical Center and University of California/San Francisco, 4150 Clement Street (116A), San Francisco, CA 94121. E-mail: [email protected] or [email protected] 17
BOYD, OTILINGAM, AND DEFORGE
18
ISMI is included in the Mental Health Assistant (MHA) software in the Veterans Affairs (VA) health care system. Over the decade since the ISMI was published, the first author (Boyd, formerly Ritsher) has received numerous requests for a shortened version, from researchers, clinicians, clients, and survey participants. Scale reduction that uses careful and rigorous methods can counter “survey fatigue” without compromising quality (Pather & Uys, 2008). Here we present a 10-item version of the ISMI containing two items from each subscale. Following a set of best practices for scale reduction (Stanton, Sinar, Balzer, & Smith, 2002), we selected the 10 items, tested the psychometrics of the shortened scale in the original validation sample (N ⫽ 127), and cross-checked the results in a second dataset (N ⫽ 760). In addition to presenting psychometric findings, we also present more substantively applied results regarding the relationship of ISMI-10 scores with indicators of depression, self-esteem, recovery orientation, devaluation and discrimination, and empowerment. Based on the original results from the ISMI-29 (Ritsher et al., 2003), a recent meta-analysis (Livingston & Boyd, 2010), and other recent work (Drapalski et al., 2013; Lucksted et al., 2011), we hypothesized that the ISMI-10 scores would be statistically significantly associated with each of these.
Methods Analytic Strategy Following the rigorous scale-reduction methods set forth by Stanton and colleagues (Stanton et al., 2002), we used data from the original ISMI validation study (Ritsher et al., 2003) to select a set of 10 items representing the two strongest items from each subscale. More specifically, we calculated indicators of each item’s external, internal, and judgmental item quality, and used these scores to select the items (Stanton et al., 2002). This method is in contrast to the commonly used method of selecting items simply to maximize internal consistency reliability, which can lead to alpha inflation (Kopalle & Lehmann, 1997). We chose to include two items from each subscale in order to make sure that the full domain of constructs measured by the original ISMI-29 was preserved in the new ISMI-10. Next, we used the same dataset to assess the validity correlations between the ISMI-10 and the ISMI-29 and between the ISMI-10 and relevant other correlates. We also computed the internal consistency reliability of the reduced set of items. Next, we crosschecked these results in a second dataset (Kasprow & Rosenheck, 2008). We expected the validity and reliability coefficients in the ISMI-10 to be somewhat smaller than those found in the ISMI-29, due to the “shrinkage” that typically results from shortening a questionnaire (Stanton et al., 2002). As noted above, we hypothesized that the ISMI-10 scores would be statistically significantly associated with indicators of depression, self-esteem, recovery orientation, devaluation and discrimination, and empowerment. Each of these scores was available in the validation sample, but only depression scores were available in the cross-validation sample. We include them all in the article because of their prominence in the literature and because their associations in the validation sample could be calculated for both the ISMI-29 and the ISMI-10. We expected the correlation of the
ISMI-29 with ISMI-10 to indicate convergent validity (r ⬎ .85). Based on the original results from the validation of the ISMI-29, we expected that the correlations between ISMI-10 and the other scales would be statistically significant and the following size and direction: depression (positive, medium); self-esteem (negative, large); recovery orientation (negative, large); empowerment (negative, large); and devaluation and discrimination (positive, medium). Using Cohen’s convention, a medium effect size is r ⬎ .3 and a large effect size is r ⬎ .5(Cohen, 1988). Due to the differences between constructs measured, we also expected that none of these large correlations would approach the level of convergent validity (all would be well under r ⫽ .85).
Participants Participants for the scale-reduction analyses were the same as in the original validation study for the ISMI-29 (Ritsher et al., 2003). Participants included 127 mental health outpatients from a U.S. Department of Veterans Affairs (VA) facility, and their demographics were typical of those served by that facility, as detailed in the original article (Ritsher et al., 2003). Participants for the cross-validation analyses included 760 mental health outpatients from 12 VA facilities. These are the baseline data from a longitudinal study called the “Special NeedsChronically Mentally Ill (SN-CMI) Program for Homeless Veterans Discharged from VA Inpatient Care,” which was directed by the North East Program Evaluation Center (NEPEC) (Kasprow & Rosenheck, 2008). To be included in that study, participants must have a serious psychiatric diagnosis, recent psychiatric hospitalization at one of 12 VA facilities, and recent or imminent homelessness (Kasprow & Rosenheck, 2008). For the present article, participants must also have provided nonmissing data on the ISMI-10 items. In both samples, all participants provided informed consent to study procedures approved by the appropriate Institutional Review Board.
Measures Internalized stigma. The Internalized Stigma of Mental Illness (ISMI) scale is a 29-item self-report questionnaire developed with consumer input that includes the following subscales: Alienation, Discrimination Experience, Social Withdrawal, Stereotype Endorsement, and Stigma Resistance (Ritsher et al., 2003). Items were originally developed in partnership with research team members and focus group members who had a diagnosis of a serious mental illness (SMI) (Ritsher et al., 2003). Items were written to be applicable to all respondents and oriented to the present. Answers are coded on the following 4-point anchored Likert scale: 1 (strongly disagree), 2 (disagree), 3 (agree), and 4 (strongly agree). The ISMI-29 has high internal consistency reliability (␣ ⫽ .90). The ISMI-29 has been widely used around the world (Boyd et al., 2014; Brohan et al., 2010; Livingston & Boyd, 2010), and has been validated in multiple samples in a variety of languages (Boyd et al., 2014; Ersoy & Varan, 2007; Ghanean, Nojomi, & Jacobsson, 2011; Hwang, Lee, Han, & Kwon, 2006; Li, Gao, Bai, & Long, 2009; Sibitz et al., 2006). There are two methods of score interpretation (Lysaker, Roe, & Yanos, 2007; Ritsher & Phelan, 2004). As shown in the Appendix, the 4-category method divides scores
10-ITEM ISMI SCALE
into the following categories: 1.00 –2.00 (minimal to no internalized stigma), 2.01–2.50 (mild internalized stigma), 2.51–3.00 (moderate internalized stigma), and 3.01– 4.00 (severe internalized stigma; Lysaker et al., 2007). The 2-category method simply divides scores according to whether they are above or below the midpoint: 1.00 –2.50 (does not report high internalized stigma) and 2.51– 4.00 (reports high internalized stigma; Ritsher & Phelan, 2004). Of course, the score can also be used as a continuous variable. The scale typically takes about 4 –5 minutes to complete. Depression. Both samples presented in this article included a measure of depression, but not the same one. In the validation sample, the level of depressive symptoms was measured using the widely used Center for Epidemiological Studies-Depression (CES-D) scale (Radloff, 1977). The CES-D uses a 4-point Likert scale and had an alpha of 0.88 in the validation sample. Although the CES-D scores were significantly correlated with the ISMI-29 scores, they loaded onto two distinct factors in a two-factor analysis (Ritsher et al., 2003), supporting the notion that these are two distinct constructs that may co-occur. In the cross-validation sample, the level of depressive symptoms was measured using the depression scale of the Symptom Checklist 90 Revised (SCL90 –R) (Derogatis, Rickels, & Rock, 1976). The questionnaire asks participants to rate how distressed they were in the past month by 90 different symptoms, such as “crying easily.” There are 13 items in the depression subscale. Response anchors range from 0 (not at all) to 4 (extremely). Self-esteem. The level of self-esteem was measured with the widely used Rosenberg self-esteem scale (SES) (Rosenberg, 1979, 1989), which consists of 10 items scored on the same 4-point scale as that of the ISMI, and in the validation sample, it had an internal consistency reliability coefficient of ␣ ⫽ .87. Recovery orientation. The Recovery Assessment Scale (RAS) (Corrigan, Giffort, Rashid, Leary, & Okeke, 1999) was used to measure the belief among people with serious mental illness that mental health recovery is possible. Items include, for example, “I can handle it if I get sick again.” Responses are coded using a five-point scale ranging from “strongly disagree” to “strongly agree.” We used the 41-item version of the scale (Corrigan et al., 1999). In the validation sample, it had high internal consistency reliability (␣ ⫽ .96). Empowerment. As detailed in the validation study for the ISMI-29 (Ritsher et al., 2003), we measured the level of empowerment in two different ways. First, we used the 17 items from Factor 1 in the validation study for the Boston University Empowerment Scale (BUES-17; Rogers, Chamberlin, Ellison, & Crean, 1997). With the permission of the scale’s authors, the first factor was used rather than the full set of items because our pilot study of the full scale in a different sample showed more support for using this set of items than the full scale (Ritsher et al., 2003). Items include, for example, “I see myself as a capable person,” and they have a four-point response format ranging from “strongly disagree” to “strongly agree.” This group of items had an internal consistency reliability of ␣ ⫽ .85. Second, we measured empowerment using the Personal Empowerment Scale (PES; Segal, Silverman, & Temkin, 1995). This scale consists of 10 items such as, “How much choice do you have in deciding who stays in your living space at night?” and has a four-point response scale ranging from “no choice” to “a lot of choice.” The PES had an alpha of 0.84 in the validation sample.
19
Perceived devaluation and discrimination. We measured respondents’ perception of societal stigma using the widely used 12-item Perceived Devaluation and Discrimination scale (PDD; Link, Struening, Neese-Todd, Asmussen, & Phelan, 2001; Link, Struening, Rahav, Phelan, & Nuttbrock, 1997; Perlick et al., 2001). Items include, for example, “most people think less of a person who has been in a mental hospital.” Responses are coded on a four-point scale ranging from “strongly disagree” to “strongly agree.” This scale measures an aspect of stigma, but it does not focus on internalized stigma. Therefore, we expected the PDD to be strongly positively correlated with ISMI-10 scores but not redundantly so (i.e., r over 0.50 but less than 0.85). In the validation sample the PDD had good internal consistency (␣ ⫽ .84).
Results Sample As detailed in the ISMI-29 validation study, (Ritsher et al., 2003), the average age of the 127 veterans in the validation sample was 49.5; the sample was 94% male; 62% white; 63% had higher than a high school education; 60% had income from government sources; and 100% had at least one of the following ICD-9 serious mental illness diagnoses according to central VA records: schizophrenia, paranoid psychosis, affective psychosis, depression, posttraumatic stress disorder, anxiety disorder, or personality disorder. In the cross-validation sample, the average age of these 760 veterans was 49.6; the sample was 94% male; 54% white; 65% had more than 12 years of education, and 100% had a serious mental illness (as defined by Kasprow & Rosenheck, 2008).
External Item Quality Using raw data from the validation sample (N ⫽ 127), we calculated a score for each of the 29 items averaging the correlations between that item and each of the correlates used in the original validation study (Ritsher et al., 2003). Thus, scores for each item were correlated with total scores for each of the following six measures: Perceived Devaluation and Discrimination (Link et al., 1997), a 41-item version of the Recovery Assessment Scale (Corrigan et al., 1999), Factor 1 of the Boston University Empowerment Scale (Rogers et al., 1997), the Personal Empowerment Scale (Segal et al., 1995), the Center for Epidemiologic Studies Depression Scale (CES-D) (Radloff, 1977), and the Rosenberg Self-Esteem Scale (Rosenberg, 1979, 1989). The resulting six correlation coefficients were averaged. The external item quality of the 29 items averaged across the six scores ranged from r ⫽ .04 to .35 (detailed results not shown but available upon request). This means that on average across all available indicators, the 29 items ranged from not correlated to a medium level of correlation with the indicators. This is consistent with the idea that some aspects of the domain measured by the scale, particularly Stigma Resistance, are less overlapping with the other constructs measured. We continued to consider the inclusion of the lower-scoring items if they had higher scores on internal and/or judgmental item quality.
Internal Item Quality Next, the internal item quality was evaluated by calculating the corrected item-total correlation for each of the 29 items. Because
BOYD, OTILINGAM, AND DEFORGE
20
our objective was to maximize coverage across the domains represented by the subscales, we used the subscale total as the total score for these analyses. The internal item quality in the 29 items ranged from r ⫽ .26 to 0.67 (detailed results not shown but available upon request).
Judgmental Item Quality Following the approach described by Stanton and colleagues (Stanton et al., 2002), the three authors independently rated face validity (Bornstein, 1996), according to the degree to which each of the 29 items corresponded to the construct measured by the subscale. We used the following 3-point Likert scale: 0 (poor fit with the construct), 1 (adequate fit with the construct), and 2 (excellent fit with the construct). Our three scores were averaged into a judgmental item quality score for each of the 29 items, with scores ranging from 1.00 to 2.00 (detailed results not shown but available upon request).
Item Sort and Scale Reduction As recommended (Stanton et al., 2002), two of the original ISMI-29 authors sorted the items and selected the reduced scale. Items were sorted within subscales. When sorting the items, we gave first preference to external item quality, next preference to internal item quality, and last preference to judgmental item quality (detailed results not shown but available upon request). The resulting ISMI-10 scale is shown in the Appendix. The distribution of the ISMI-10 had a skewness statistic of 0.28 (SE ⫽ 0.22) and a kurtosis statistic of 0.56 (SE ⫽ 0.43), indicating a relatively normal distribution.
Validity Correlations Next, we used the raw data to calculate the validity correlations between the ISMI-10 and the ISMI-29 (r ⫽ .94, p ⬍ .001, N ⫽ 127) and the six other correlates. All results were in the expected direction, statistically significant, and in the expected size range, as shown in Table 1. The ISMI-10 had a range of 1.30 –3.60, with a mean of 2.32 and an SD of .41. These results are comparable to the ISMI-29, which had a range of 1.48 –3.66, with a mean of 2.32 and an SD of .39. Both versions categorized 29.1% of the sample as having average scores over the midpoint, indicating a high level of internalized stigma. Because some individuals had scores just above that threshold according to one version and just below it
according to another version, the sensitivity was .87 and the specificity was .94.
Internal Consistency Reliability The ISMI-29 had an internal consistency reliability of ␣ ⫽ .90, and the ISMI-10 had an alpha of 0.75.
Cross-Validation Next, following the best practices set forth by (Stanton et al., 2002), we ran a new set of analyses using the ISMI-10 in a separate dataset. In the cross-validation sample, the internal consistency reliability of the ISMI-10 was ␣ ⫽ .81. The correlation between the ISMI-10 score and the SCL-90 –R depression scale was r ⫽ .50, p ⬍ .001, N ⫽ 756. The correlation between ISMI-10 and ISMI-29 was r ⫽ .94, p ⬍ .001, N ⫽ 760. The descriptive statistics for the cross-validation sample were similar to those listed above for the validation sample, with a range of 1.00 –3.80, a mean of 2.27, and an SD of 0.51. In addition, 30.1% of the cross-validation sample had a mean score over the midpoint of 2.5.
Ease of Use In a small clinical focus group of five consumers with serious mental illness, the ISMI-10 took 1–2 minutes to complete and did not generate any questions or difficulties.
Discussion As expected, the ISMI-10 retained the essential properties of the ISMI-29, including adequate internal consistency reliability and external validity in relation to depression, self-esteem, recovery orientation, perceived devaluation and discrimination, and empowerment. The ISMI-10 scores are normally distributed and have similar descriptive statistics to the ISMI-29. The reliability and depression findings were replicated in a cross-validation sample. These results are in line with findings of a recent review of 127 stigma articles and meta-analysis including 15 studies using the ISMI-29 (Livingston & Boyd, 2010). That article showed a striking and robust pattern of relationships between elevated internalized stigma and higher depression, lower self-esteem, and lower empowerment (Livingston & Boyd, 2010). The ISMI-10 results are in full accord with these results. In addition, we found that higher stigma levels on the ISMI-10 are associated with signifi-
Table 1 Correlations in the validation sample (N ⫽ 127)
10-item ISMI 29-item ISMI
ISMI-29 Internalized stigma
CES-D Depression
SES Self-esteem
RAS-41 Recovery
BUES-17 Empowerment
PES Empowerment
PDD Societal stigma
.94ⴱⴱ 1.0
.55ⴱⴱ .53ⴱⴱ
⫺.64ⴱⴱ ⫺.59ⴱⴱ
⫺.54ⴱⴱ ⫺.50ⴱⴱ
⫺.54ⴱⴱ ⫺.52ⴱⴱ
⫺.36ⴱⴱ ⫺.34ⴱⴱ
.31ⴱⴱ .35ⴱⴱ
Note. ISMI-29 ⫽ Internalized Stigma of Mental Illness (ISMI) scale (29-item version) (Ritsher et al., 2003); CES-D ⫽ Center for Epidemiological Studies-Depression (CES-D) scale (Radloff, 1977); SES ⫽ Rosenberg Self-Esteem Scale (Rosenberg, 1979, 1989); RAS-41 ⫽ The Recovery Assessment Scale (RAS) (41-item version) (Corrigan et al., 1999); BUES-17 ⫽ The 17 items from Factor 1 in the validation study for the Boston University Empowerment Scale (Rogers et al., 1997); PES ⫽ Personal Empowerment Scale (PES) (Segal et al., 1995); PDD ⫽ Perceived Devaluation and Discrimination scale (Link et al., 2001; Link et al., 1997; Perlick et al., 2001). ⴱⴱ p ⬍ .001.
10-ITEM ISMI SCALE
cantly higher scores on perceived devaluation and discrimination and lower scores on recovery orientation. As noted above, we expected only a medium effect size for devaluation and discrimination due to the differences between the two constructs (internal vs. societal stigma) and our past results (Ritsher et al., 2003). Similarly, we did not expect extremely high correlations with any of the constructs due to differences between them. Indeed, the original ISMI-29 validation study included a series of 2-factor factor analyses that showed that the items from each pair of scales sorted into separate factors (Ritsher et al., 2003). The fact that longitudinal studies show changes in outcome variables, such as depression, holding the baseline levels constant also argues for the distinction between constructs (Ritsher & Phelan, 2004). Although the ISMI-10 covers all five dimensions of internalized stigma measured by the ISMI-29, it was not our intention to create five two-item subscales. Thus, we recommend using the total ISMI-10 score rather than dividing it into subscales. Researchers and others wishing to differentiate or examine the subdomains of Alienation, Discrimination Experience, Social Withdrawal, Stereotype Endorsement, and Stigma Resistance should consider using the items from the full ISMI (Ritsher et al., 2003). Two-item scales are likely to be unstable. The brief ISMI-10 is intended for using as part of a larger series of scales where response burden is an issue. The ISMI-29 takes about 5 minutes to complete and is somewhat challenging to self-score, whereas the ISMI-10 takes 1–2 minutes to complete, and can be easily self-scored (reverse code 2 items, total the score, divide by 10). Anecdotally, there are indications that the ISMI-10 has utility for clinical and program evaluation purposes. Clinically, it has been used in the Psychosocial Rehabilitation and Recovery Center (PRRC) at the San Francisco VA Medical Center as the basis of productive discussions about stigma in the context of group therapy for veterans with serious mental illness. The ISMI-10 provides a simpler focus for this type of discussion than the ISMI-29. In terms of program evaluation, the ISMI-10 has been included in a Northeast Program Evaluation Center (NEPEC) evaluation of PRRC programs nationwide. Pilot data at an initial set of sites has been successfully collected and analyzed (Hunt & Smith, 2012; Wong, Boyd, & Hunt 2013).
Limitations The data available for the cross-validation sample did not allow us to confirm the relationship between the ISMI-10 and selfesteem or empowerment. Moreover, it was beyond the scope of the study to conduct the optional 10th step proposed by Stanton and colleagues (Stanton et al., 2002), which was multigroup structural equation modeling. We were also not able to use a cross-validation sample that had originally used only the 10-item version rather than the full ISMI-29 (Stanton et al., 2002). In other words, we did not test the brief version on its own. An adequate research dataset is not yet available for this purpose. This would be an important step in the establishment of the ISMI-10 that should be completed in future work. Also, we did not test a brief version of any of the more than 60 other language and disorder versions of the ISMI-29 (Boyd et al., 2014). Furthermore, this article considers only internal consistency and convergent validity, due to the nature of the “best practices” steps that we followed (Stanton et al., 2002), and the limitations of the available datasets. One of the most notewor-
21
thy limitations of these datasets is the narrow range of demographics among respondents (all were veterans, and most were male). Longitudinal studies reviewed by Livingston and Boyd (2010) show that the ISMI-29 is able to show meaningful changes over time. The fact that it has been so widely used in at least 70 countries around the world shows that the ISMI-29 has wide acceptability (Boyd et al., 2014). We can only assume that because the ISMI-10 is a subset of the same items, it will have similar properties. Although informal focus group results show that the ISMI-10 takes less than half the amount of time to complete as the ISMI-29, this has not been formally tested. Future studies will be needed to investigate these points.
Conclusions We conclude that the ISMI-10 has strong psychometric properties and is a practical, reliable, and valid alternative to the original ISMI-29. As expected, the ISMI-10 results showed that internalized stigma is significantly related to indicators of depression, self-esteem, recovery orientation, empowerment, and perceived devaluation and discrimination. We hope that the advent of the brief ISMI-10 will stimulate further longitudinal studies and program evaluation projects to shed more light on ways to reduce internalized stigma.
References Bornstein, R. F. (1996). Face validity in psychological assessment: Implications for a unified model of validity. American Psychologist, 51, 983–984. doi:10.1037/0003-066X.51.9.983 Boyd, J. E., Adler, E. P., Otilingam, P. G., & Peters, T. (2014). Internalized Stigma of Mental Illness (ISMI) Scale: A multinational review. Comprehensive Psychiatry, 55, 221–231. doi:10.1016/j.comppsych.2013.06 .005. Epub 2013 Sep 21. Brohan, E., Slade, M., Clement, S., & Thornicroft, G. (2010). Experiences of mental illness stigma, prejudice and discrimination: A review of measures. BMC Health Service Research, 10, 80. doi:10.1186/14726963-10-80 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Corrigan, P. W., Giffort, D., Rashid, F., Leary, M., & Okeke, I. (1999). Recovery as a psychological construct. Community Mental Health Journal, 35, 231–239. doi:10.1023/A:1018741302682 Corrigan, P. W., & Watson, A. C. (2002). The paradox of self-stigma and mental illness. Clinical Psychology: Science and Practice, 9, 35–53. doi:10.1093/clipsy.9.1.35 Derogatis, L. R., Rickels, K., & Rock, A. F. (1976). The SCL-90 and the MMPI: A step in the validation of a new self-report scale. British Journal of Psychiatry, 128, 280 –289. doi:10.1192/bjp.128.3.280 Dickerson, F. B., Sommerville, J., Origoni, A. E., Ringel, N. B., & Parente, F. (2002). Experiences of stigma among outpatients with schizophrenia. Schizophrenia Bulletin, 28, 143–155. doi:10.1093/oxfordjournals.schbul .a006917 Drapalski, A. L., Lucksted, A., Perrin, P. B., Aakre, J. M., Brown, C. H., DeForge, B. R., & Boyd, J. E. (2013). A model of internalized stigma and its effects on people with mental illness. Psychiatric Services, 64, 264 –269. doi:10.1176/appi.ps.001322012 Ersoy, M. A., & Varan, A. (2007). Ruhsal hastalıklarda içselles¸tirilmis¸ damgalanma ölçeg˘ i Türkçe formu’nun güvenilirlik ve geçerlik çalıs¸ması. [Reliability and validity of the Turkish version of the internalized stigma of mental illness scale]. Turk Psikiyatri Derg, 18, 163– 171.
22
BOYD, OTILINGAM, AND DEFORGE
Ghanean, H., Nojomi, M., & Jacobsson, L. (2011). Internalized stigma of mental illness in Tehran, Iran. Stigma Research and Action, 1, 11–17. doi:10.5463/sra.v1i1.10 Hinshaw, S. P. (2007). The mark of shame: stigma of mental illness and an agenda for change. New York, NY: Oxford University Press. Hunt, M., & Smith, T. (2012). Psychosocial Rehabilitation and Recovery Centers (PRRC): Program evaluation pilot data, SMI Committee. Washington, DC: Department of Veterans Affairs. Hwang, T. Y., Lee, W. K., Han, E. S., & Kwon, E. J. (2006). A study on the reliability and validity of the Korean Version of Internalized Stigma of Mental Illness Scale (K-ISMI). Journal of the Korean Neuropsychiatric Association, 45, 418 – 426. Kasprow, W. J., & Rosenheck, R. A. (2008). Interim report on the special needs - chronically mentally ill (SN-CMI) program for hospitalized homeless veterans. West Haven, CT: Northeast Program Evaluation Center. Kopalle, P. K., & Lehmann, D. R. (1997). Alpha inflation? The impact of eliminating scale items on Cronbach’s Alpha. Organizational Behavior and Human Decision Processes, 70, 189 –197. doi:10.1006/obhd.1997 .2702 Li, Q., Gao, W.-J., Bai, B.-Q., & Long, J. (2009). Preliminary application of Internalized Stigma of Mental Illness scale. Chinese Journal of Clinical Psychology, 17, 127–130. Link, B. G., Cullen, F. T., Struening, E., Shrout, P. E., & Dohrenwend, B. P. (1989). A modified labeling theory approach to mental disorders: An empirical assessment. American Sociological Review, 54, 400 – 423. doi:10.2307/2095613 Link, B. G., Struening, E. L., Neese-Todd, S., Asmussen, S., & Phelan, J. C. (2001). Stigma as a barrier to recovery: The consequences of stigma for the self-esteem of people with mental illnesses. Psychiatric Services, 52, 1621–1626. doi:10.1176/appi.ps.52.12.1621 Link, B. G., Struening, E. L., Rahav, M., Phelan, J. C., & Nuttbrock, L. (1997). On stigma and its consequences: Evidence from a longitudinal study of men with dual diagnoses of mental illness and substance abuse. Journal of Health and Social Behavior, 38, 177–190. doi:10.2307/ 2955424 Livingston, J. D., & Boyd, J. E. (2010). Correlates and consequences of internalized stigma for people living with mental illness: A systematic review and meta-analysis. Social Science and Medicine, 71, 2150 –2161. doi:10.1016/j.socscimed.2010.09.030 Lucksted, A., Drapalski, A., Calmes, C., Forbes, C., DeForge, B., & Boyd, J. (2011). Ending self-stigma: Pilot evaluation of a new intervention to reduce internalized stigma among people with mental illnesses. Psychiatric Rehabilitation Journal, 35, 51–54. doi:10.2975/35.1.2011.51.54 Lysaker, P. H., Roe, D., & Yanos, P. T. (2007). Toward understanding the insight paradox: Internalized stigma moderates the association between insight and social functioning, hope, and self-esteem among people with schizophrenia spectrum disorders. Schizophrenia Bulletin, 33, 192–199. doi:10.1093/schbul/sbl016
Overton, S. L., & Medina, S. L. (2008). The stigma of mental illness. Journal of Counseling and Development, 86, 143–151. doi:10.1002/j .1556-6678.2008.tb00491.x Pather, S., & Uys, C. S. (2008). Using scale reduction techniques for improved quality of survey information. South African Journal of Information Management, 10, 1–7. Perlick, D. A., Rosenheck, R. A., Clarkin, J. F., Sirey, J. A., Salahi, J., Struening, E. L., & Link, B. G. (2001). Stigma as a barrier to recovery: Adverse effects of perceived stigma on social adaptation of persons diagnosed with bipolar affective disorder. Psychiatric Services, 52, 1627–1632. doi:10.1176/appi.ps.52.12.1627 Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385– 401. doi:10.1177/014662167700100306 Ritsher (Boyd), J. B., Otilingam, P. O., & Grajales, M. (2003). Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research, 121, 31– 49. doi:10.1016/j.psychres.2003.08.008 Ritsher (Boyd), J. B., & Phelan, J. (2004). Internalized stigma predicts erosion of morale among psychiatric outpatients. Psychiatry Research, 129, 257–265. doi:10.1016/j.psychres.2004.08.003 Rogers, E. S., Chamberlin, J., Ellison, M. L., & Crean, T. (1997). A consumer-constructed scale to measure empowerment among users of mental health services. Psychiatric Services, 48, 1042–1047. Rosenberg, M. (1979). Conceiving the self. New York, NY: Basic Books. Rosenberg, M. (1989). Society and the adolescent self-image (Rev. ed.). Middletown, CT: Wesleyan University Press. Segal, S. P., Silverman, C., & Temkin, T. (1995). Measuring empowerment in client-run self-help agencies. Community Mental Health Journal, 31, 215–227. doi:10.1007/BF02188748 Sibitz, I., Amering, M., Unger, A., Seyringer, M., Bachmann, A., & Benesch, T. (2006). Internalisiertes Stigma bei Personen mit einer Erkrangung aus dem schizophrenen Formenkreis: Validierung der ISMI (Internalized Stigma of Mental Illness) Skala [Internalized stigma among persons with a schizophrenia spectrum disorder: Validation of the ISMI (Internalized Stigma of Mental Illness) scale.] (No. Projekt Nr. 2396.). Vienna: Forschungsbericht an den Medizinisch-Wissenschaftlichen Fonds des Burgermeisters der Bundeshauptstadt Wien. Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167–194. doi:10.1111/j.1744-6570.2002.tb00108.x Wahl, O. F. (1999). Mental health consumers’ experience of stigma. Schizophrenia Bulletin, 25, 467– 478. doi:10.1093/oxfordjournals.schbul .a033394 West, M. L., Yanos, P. T., Smith, S. M., Roe, D., & Lysaker, P. H. (2011). Prevalence of internalized stigma among persons with severe mental illness. Stigma Research and Action, 1, 3–10. doi:10.5463/sra.v1i1.9 Wong, J., Boyd, J. E., & Hunt, M. (2013). Poster presented at the 16th VA Psychology Leadership Conference: Internalized stigma among PRRC veterans. San Antonio, TX: the American Psychological Association, Association of VA Psychologist Leaders (AVAPL) and APA Division 18-Psychologists in Public Service.
(Appendix follows)
10-ITEM ISMI SCALE
23
Appendix Internalized Stigma of Mental Illness Inventory – 10-item Version (ISMI-10) We are going to use the term “mental illness” in the rest of this questionnaire, but please think of it as whatever you feel is the best term for it. For each question, please mark whether you strongly disagree (1), disagree (2), agree (3), or strongly agree (4). Strongly disagree
Disagree
Agree
Strongly agree
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1. Mentally ill people tend to be violent. 2. People with mental illness make important contributions to society. 3. I don’t socialize as much as I used to because my mental illness might make me look or behave “weird.” 4. Having a mental illness has spoiled my life. 5. I stay away from social situations in order to protect my family or friends from embarrassment. 6. People without mental illness could not possibly understand me. 7. People ignore me or take me less seriously just because I have a mental illness. 8. I can’t contribute anything to society because I have a mental illness. 9. I can have a good, fulfilling life, despite my mental illness. 10. Others think that I can’t achieve much in life because I have a mental illness.
Scoring Key The ISMI-10 contains 10 items which produce a total score. Reverse-code items 2 and 9 before calculating the total score. Add the item scores together and then divide by the total number of answered items. The resulting score should range from 1-4. For example, if someone answers 8 of the 10 items, the total score is produced by adding together the 8 answered items and dividing by 8.
Interpretation of Scores 4-category method (following the method used by Lysaker et al., 2007): 1.00-2.00: minimal to no internalized stigma 2.01-2.50: mild internalized stigma 2.51-3.00: moderate internalized stigma 3.01-4.00: severe internalized stigma 2-category method (following the method used by Ritsher [Boyd] & Phelan, 2004). 1.00-2.50: does not report high internalized stigma 2.51-4.00: reports high internalized stigma Received July 29, 2013 Revision received October 21, 2013 Accepted October 22, 2013 䡲
In the public domain DOI: 10.1037/prj0000035
Brief Version of the Internalized Stigma of Mental Illness (ISMI) Scale: Psychometric Properties and Relationship to Depression, Self Esteem, Recovery Orientation, Empowerment, and Perceived Devaluation and Discrimination Jennifer E. Boyd
Poorni G. Otilingam
San Francisco VA Medical Center and University of California, San Francisco
San Francisco VA Medical Center
Bruce R. DeForge School of Social Work, University of Maryland, Baltimore Objective: The internalized stigma of mental illness impedes recovery and is associated with increased depression, reduced self-esteem, reduced recovery orientation, reduced empowerment, and increased perceived devaluation and discrimination. The Internalized Stigma of Mental Illness (ISMI) scale is a 29-item self-report questionnaire developed with consumer input that includes the following subscales: Alienation, Discrimination Experience, Social Withdrawal, Stereotype Endorsement, and Stigma Resistance. Here we present a 10-item version of the ISMI containing the two strongest items from each subscale. Method: Participants were all outpatient veterans with serious mental illness. Following the rigorous scale-reduction methods set forth by Stanton and colleagues (2002), we selected the 10 items, tested the psychometrics of the shortened scale in the original validation sample (N ⫽ 127), and cross-checked the results in a second dataset (N ⫽ 760). Results: As expected, the ISMI-10 retained the essential properties of the ISMI-29, including adequate internal consistency reliability and external validity in relation to depression, self-esteem, recovery orientation, perceived devaluation and discrimination, and empowerment. The ISMI-10 scores are normally distributed and have similar descriptive statistics to the ISMI-29. The reliability and depression findings were replicated in a cross-validation sample. Conclusions and Implications for Practice: We conclude that the ISMI-10 has strong psychometric properties and is a practical, reliable, and valid alternative to the original ISMI-29. Future work should test the ISMI-10 in more diverse samples. This shorter version should reduce respondent burden in program evaluation projects that seek to determine whether participation in psychosocial rehabilitation programming reduces internalized stigma. Keywords: stigma of mental illness, questionnaire, depression, self-esteem, empowerment
objective effects such as lost housing or social ostracism, stigma also has subjective psychological effects on people with mental illness. Internalized stigma is the psychological point of impact of societal stigma on people with mental illness. As members of the society in which we all live, people with mental illness often apply negative stereotypes, biases, and discrimination to themselves, which impedes their recovery process by eroding their morale (Corrigan & Watson, 2002; Link, Cullen, Struening, Shrout, & Dohrenwend, 1989; Ritsher (Boyd) & Phelan, 2004; West, Yanos, Smith, Roe, & Lysaker, 2011). Also known as self-stigma, internalized stigma is associated with increased depression, reduced self-esteem, reduced recovery orientation, reduced empowerment, and increased perceived devaluation and discrimination (Drapalski et al., 2013; Livingston & Boyd, 2010; Lucksted et al., 2011; Ritsher (Boyd), Otilingam, & Grajales, 2003). Several measures of internalized stigma have been developed (Brohan, Slade, Clement, & Thornicroft, 2010; Livingston & Boyd, 2010). One of the most widely used measures is the Internalized Stigma of Mental Illness (ISMI) scale, which was developed with consumer input and contains 29 items and 5 subscales (Boyd, Adler, Otilingam & Peters, 2014; Ritsher et al., 2003). The
The stigma of mental illness is the negative stereotyping, biases, and discrimination often directed at people with mental illness (Hinshaw, 2007). Such stigma is widespread across societies around the world and has been shown to have a variety of negative effects (Dickerson, Sommerville, Origoni, Ringel, & Parente, 2002; Hinshaw, 2007; Overton & Medina, 2008; Wahl, 1999). In addition to
Jennifer E. Boyd, Department of Psychiatry, San Francisco VA Medical Center and University of California, San Francisco; Poorni G. Otilingam, Mental Health Service, San Francisco VA Medical Center; and Bruce R. DeForge, School of Social Work, University of Maryland, Baltimore. We thank the Department of Veterans Affairs, Northeast Program Evaluation Center and the Office of Patient Care Services Data Transfer Agreement SNCMI (2011-02), for providing the cross-validation sample. Thanks also go to Jeanne McPhee for assistance with manuscript preparation. Correspondence concerning this article should be addressed to Jennifer E. Boyd, Department of Psychiatry, San Francisco VA Medical Center and University of California/San Francisco, 4150 Clement Street (116A), San Francisco, CA 94121. E-mail: [email protected] or [email protected] 17
BOYD, OTILINGAM, AND DEFORGE
18
ISMI is included in the Mental Health Assistant (MHA) software in the Veterans Affairs (VA) health care system. Over the decade since the ISMI was published, the first author (Boyd, formerly Ritsher) has received numerous requests for a shortened version, from researchers, clinicians, clients, and survey participants. Scale reduction that uses careful and rigorous methods can counter “survey fatigue” without compromising quality (Pather & Uys, 2008). Here we present a 10-item version of the ISMI containing two items from each subscale. Following a set of best practices for scale reduction (Stanton, Sinar, Balzer, & Smith, 2002), we selected the 10 items, tested the psychometrics of the shortened scale in the original validation sample (N ⫽ 127), and cross-checked the results in a second dataset (N ⫽ 760). In addition to presenting psychometric findings, we also present more substantively applied results regarding the relationship of ISMI-10 scores with indicators of depression, self-esteem, recovery orientation, devaluation and discrimination, and empowerment. Based on the original results from the ISMI-29 (Ritsher et al., 2003), a recent meta-analysis (Livingston & Boyd, 2010), and other recent work (Drapalski et al., 2013; Lucksted et al., 2011), we hypothesized that the ISMI-10 scores would be statistically significantly associated with each of these.
Methods Analytic Strategy Following the rigorous scale-reduction methods set forth by Stanton and colleagues (Stanton et al., 2002), we used data from the original ISMI validation study (Ritsher et al., 2003) to select a set of 10 items representing the two strongest items from each subscale. More specifically, we calculated indicators of each item’s external, internal, and judgmental item quality, and used these scores to select the items (Stanton et al., 2002). This method is in contrast to the commonly used method of selecting items simply to maximize internal consistency reliability, which can lead to alpha inflation (Kopalle & Lehmann, 1997). We chose to include two items from each subscale in order to make sure that the full domain of constructs measured by the original ISMI-29 was preserved in the new ISMI-10. Next, we used the same dataset to assess the validity correlations between the ISMI-10 and the ISMI-29 and between the ISMI-10 and relevant other correlates. We also computed the internal consistency reliability of the reduced set of items. Next, we crosschecked these results in a second dataset (Kasprow & Rosenheck, 2008). We expected the validity and reliability coefficients in the ISMI-10 to be somewhat smaller than those found in the ISMI-29, due to the “shrinkage” that typically results from shortening a questionnaire (Stanton et al., 2002). As noted above, we hypothesized that the ISMI-10 scores would be statistically significantly associated with indicators of depression, self-esteem, recovery orientation, devaluation and discrimination, and empowerment. Each of these scores was available in the validation sample, but only depression scores were available in the cross-validation sample. We include them all in the article because of their prominence in the literature and because their associations in the validation sample could be calculated for both the ISMI-29 and the ISMI-10. We expected the correlation of the
ISMI-29 with ISMI-10 to indicate convergent validity (r ⬎ .85). Based on the original results from the validation of the ISMI-29, we expected that the correlations between ISMI-10 and the other scales would be statistically significant and the following size and direction: depression (positive, medium); self-esteem (negative, large); recovery orientation (negative, large); empowerment (negative, large); and devaluation and discrimination (positive, medium). Using Cohen’s convention, a medium effect size is r ⬎ .3 and a large effect size is r ⬎ .5(Cohen, 1988). Due to the differences between constructs measured, we also expected that none of these large correlations would approach the level of convergent validity (all would be well under r ⫽ .85).
Participants Participants for the scale-reduction analyses were the same as in the original validation study for the ISMI-29 (Ritsher et al., 2003). Participants included 127 mental health outpatients from a U.S. Department of Veterans Affairs (VA) facility, and their demographics were typical of those served by that facility, as detailed in the original article (Ritsher et al., 2003). Participants for the cross-validation analyses included 760 mental health outpatients from 12 VA facilities. These are the baseline data from a longitudinal study called the “Special NeedsChronically Mentally Ill (SN-CMI) Program for Homeless Veterans Discharged from VA Inpatient Care,” which was directed by the North East Program Evaluation Center (NEPEC) (Kasprow & Rosenheck, 2008). To be included in that study, participants must have a serious psychiatric diagnosis, recent psychiatric hospitalization at one of 12 VA facilities, and recent or imminent homelessness (Kasprow & Rosenheck, 2008). For the present article, participants must also have provided nonmissing data on the ISMI-10 items. In both samples, all participants provided informed consent to study procedures approved by the appropriate Institutional Review Board.
Measures Internalized stigma. The Internalized Stigma of Mental Illness (ISMI) scale is a 29-item self-report questionnaire developed with consumer input that includes the following subscales: Alienation, Discrimination Experience, Social Withdrawal, Stereotype Endorsement, and Stigma Resistance (Ritsher et al., 2003). Items were originally developed in partnership with research team members and focus group members who had a diagnosis of a serious mental illness (SMI) (Ritsher et al., 2003). Items were written to be applicable to all respondents and oriented to the present. Answers are coded on the following 4-point anchored Likert scale: 1 (strongly disagree), 2 (disagree), 3 (agree), and 4 (strongly agree). The ISMI-29 has high internal consistency reliability (␣ ⫽ .90). The ISMI-29 has been widely used around the world (Boyd et al., 2014; Brohan et al., 2010; Livingston & Boyd, 2010), and has been validated in multiple samples in a variety of languages (Boyd et al., 2014; Ersoy & Varan, 2007; Ghanean, Nojomi, & Jacobsson, 2011; Hwang, Lee, Han, & Kwon, 2006; Li, Gao, Bai, & Long, 2009; Sibitz et al., 2006). There are two methods of score interpretation (Lysaker, Roe, & Yanos, 2007; Ritsher & Phelan, 2004). As shown in the Appendix, the 4-category method divides scores
10-ITEM ISMI SCALE
into the following categories: 1.00 –2.00 (minimal to no internalized stigma), 2.01–2.50 (mild internalized stigma), 2.51–3.00 (moderate internalized stigma), and 3.01– 4.00 (severe internalized stigma; Lysaker et al., 2007). The 2-category method simply divides scores according to whether they are above or below the midpoint: 1.00 –2.50 (does not report high internalized stigma) and 2.51– 4.00 (reports high internalized stigma; Ritsher & Phelan, 2004). Of course, the score can also be used as a continuous variable. The scale typically takes about 4 –5 minutes to complete. Depression. Both samples presented in this article included a measure of depression, but not the same one. In the validation sample, the level of depressive symptoms was measured using the widely used Center for Epidemiological Studies-Depression (CES-D) scale (Radloff, 1977). The CES-D uses a 4-point Likert scale and had an alpha of 0.88 in the validation sample. Although the CES-D scores were significantly correlated with the ISMI-29 scores, they loaded onto two distinct factors in a two-factor analysis (Ritsher et al., 2003), supporting the notion that these are two distinct constructs that may co-occur. In the cross-validation sample, the level of depressive symptoms was measured using the depression scale of the Symptom Checklist 90 Revised (SCL90 –R) (Derogatis, Rickels, & Rock, 1976). The questionnaire asks participants to rate how distressed they were in the past month by 90 different symptoms, such as “crying easily.” There are 13 items in the depression subscale. Response anchors range from 0 (not at all) to 4 (extremely). Self-esteem. The level of self-esteem was measured with the widely used Rosenberg self-esteem scale (SES) (Rosenberg, 1979, 1989), which consists of 10 items scored on the same 4-point scale as that of the ISMI, and in the validation sample, it had an internal consistency reliability coefficient of ␣ ⫽ .87. Recovery orientation. The Recovery Assessment Scale (RAS) (Corrigan, Giffort, Rashid, Leary, & Okeke, 1999) was used to measure the belief among people with serious mental illness that mental health recovery is possible. Items include, for example, “I can handle it if I get sick again.” Responses are coded using a five-point scale ranging from “strongly disagree” to “strongly agree.” We used the 41-item version of the scale (Corrigan et al., 1999). In the validation sample, it had high internal consistency reliability (␣ ⫽ .96). Empowerment. As detailed in the validation study for the ISMI-29 (Ritsher et al., 2003), we measured the level of empowerment in two different ways. First, we used the 17 items from Factor 1 in the validation study for the Boston University Empowerment Scale (BUES-17; Rogers, Chamberlin, Ellison, & Crean, 1997). With the permission of the scale’s authors, the first factor was used rather than the full set of items because our pilot study of the full scale in a different sample showed more support for using this set of items than the full scale (Ritsher et al., 2003). Items include, for example, “I see myself as a capable person,” and they have a four-point response format ranging from “strongly disagree” to “strongly agree.” This group of items had an internal consistency reliability of ␣ ⫽ .85. Second, we measured empowerment using the Personal Empowerment Scale (PES; Segal, Silverman, & Temkin, 1995). This scale consists of 10 items such as, “How much choice do you have in deciding who stays in your living space at night?” and has a four-point response scale ranging from “no choice” to “a lot of choice.” The PES had an alpha of 0.84 in the validation sample.
19
Perceived devaluation and discrimination. We measured respondents’ perception of societal stigma using the widely used 12-item Perceived Devaluation and Discrimination scale (PDD; Link, Struening, Neese-Todd, Asmussen, & Phelan, 2001; Link, Struening, Rahav, Phelan, & Nuttbrock, 1997; Perlick et al., 2001). Items include, for example, “most people think less of a person who has been in a mental hospital.” Responses are coded on a four-point scale ranging from “strongly disagree” to “strongly agree.” This scale measures an aspect of stigma, but it does not focus on internalized stigma. Therefore, we expected the PDD to be strongly positively correlated with ISMI-10 scores but not redundantly so (i.e., r over 0.50 but less than 0.85). In the validation sample the PDD had good internal consistency (␣ ⫽ .84).
Results Sample As detailed in the ISMI-29 validation study, (Ritsher et al., 2003), the average age of the 127 veterans in the validation sample was 49.5; the sample was 94% male; 62% white; 63% had higher than a high school education; 60% had income from government sources; and 100% had at least one of the following ICD-9 serious mental illness diagnoses according to central VA records: schizophrenia, paranoid psychosis, affective psychosis, depression, posttraumatic stress disorder, anxiety disorder, or personality disorder. In the cross-validation sample, the average age of these 760 veterans was 49.6; the sample was 94% male; 54% white; 65% had more than 12 years of education, and 100% had a serious mental illness (as defined by Kasprow & Rosenheck, 2008).
External Item Quality Using raw data from the validation sample (N ⫽ 127), we calculated a score for each of the 29 items averaging the correlations between that item and each of the correlates used in the original validation study (Ritsher et al., 2003). Thus, scores for each item were correlated with total scores for each of the following six measures: Perceived Devaluation and Discrimination (Link et al., 1997), a 41-item version of the Recovery Assessment Scale (Corrigan et al., 1999), Factor 1 of the Boston University Empowerment Scale (Rogers et al., 1997), the Personal Empowerment Scale (Segal et al., 1995), the Center for Epidemiologic Studies Depression Scale (CES-D) (Radloff, 1977), and the Rosenberg Self-Esteem Scale (Rosenberg, 1979, 1989). The resulting six correlation coefficients were averaged. The external item quality of the 29 items averaged across the six scores ranged from r ⫽ .04 to .35 (detailed results not shown but available upon request). This means that on average across all available indicators, the 29 items ranged from not correlated to a medium level of correlation with the indicators. This is consistent with the idea that some aspects of the domain measured by the scale, particularly Stigma Resistance, are less overlapping with the other constructs measured. We continued to consider the inclusion of the lower-scoring items if they had higher scores on internal and/or judgmental item quality.
Internal Item Quality Next, the internal item quality was evaluated by calculating the corrected item-total correlation for each of the 29 items. Because
BOYD, OTILINGAM, AND DEFORGE
20
our objective was to maximize coverage across the domains represented by the subscales, we used the subscale total as the total score for these analyses. The internal item quality in the 29 items ranged from r ⫽ .26 to 0.67 (detailed results not shown but available upon request).
Judgmental Item Quality Following the approach described by Stanton and colleagues (Stanton et al., 2002), the three authors independently rated face validity (Bornstein, 1996), according to the degree to which each of the 29 items corresponded to the construct measured by the subscale. We used the following 3-point Likert scale: 0 (poor fit with the construct), 1 (adequate fit with the construct), and 2 (excellent fit with the construct). Our three scores were averaged into a judgmental item quality score for each of the 29 items, with scores ranging from 1.00 to 2.00 (detailed results not shown but available upon request).
Item Sort and Scale Reduction As recommended (Stanton et al., 2002), two of the original ISMI-29 authors sorted the items and selected the reduced scale. Items were sorted within subscales. When sorting the items, we gave first preference to external item quality, next preference to internal item quality, and last preference to judgmental item quality (detailed results not shown but available upon request). The resulting ISMI-10 scale is shown in the Appendix. The distribution of the ISMI-10 had a skewness statistic of 0.28 (SE ⫽ 0.22) and a kurtosis statistic of 0.56 (SE ⫽ 0.43), indicating a relatively normal distribution.
Validity Correlations Next, we used the raw data to calculate the validity correlations between the ISMI-10 and the ISMI-29 (r ⫽ .94, p ⬍ .001, N ⫽ 127) and the six other correlates. All results were in the expected direction, statistically significant, and in the expected size range, as shown in Table 1. The ISMI-10 had a range of 1.30 –3.60, with a mean of 2.32 and an SD of .41. These results are comparable to the ISMI-29, which had a range of 1.48 –3.66, with a mean of 2.32 and an SD of .39. Both versions categorized 29.1% of the sample as having average scores over the midpoint, indicating a high level of internalized stigma. Because some individuals had scores just above that threshold according to one version and just below it
according to another version, the sensitivity was .87 and the specificity was .94.
Internal Consistency Reliability The ISMI-29 had an internal consistency reliability of ␣ ⫽ .90, and the ISMI-10 had an alpha of 0.75.
Cross-Validation Next, following the best practices set forth by (Stanton et al., 2002), we ran a new set of analyses using the ISMI-10 in a separate dataset. In the cross-validation sample, the internal consistency reliability of the ISMI-10 was ␣ ⫽ .81. The correlation between the ISMI-10 score and the SCL-90 –R depression scale was r ⫽ .50, p ⬍ .001, N ⫽ 756. The correlation between ISMI-10 and ISMI-29 was r ⫽ .94, p ⬍ .001, N ⫽ 760. The descriptive statistics for the cross-validation sample were similar to those listed above for the validation sample, with a range of 1.00 –3.80, a mean of 2.27, and an SD of 0.51. In addition, 30.1% of the cross-validation sample had a mean score over the midpoint of 2.5.
Ease of Use In a small clinical focus group of five consumers with serious mental illness, the ISMI-10 took 1–2 minutes to complete and did not generate any questions or difficulties.
Discussion As expected, the ISMI-10 retained the essential properties of the ISMI-29, including adequate internal consistency reliability and external validity in relation to depression, self-esteem, recovery orientation, perceived devaluation and discrimination, and empowerment. The ISMI-10 scores are normally distributed and have similar descriptive statistics to the ISMI-29. The reliability and depression findings were replicated in a cross-validation sample. These results are in line with findings of a recent review of 127 stigma articles and meta-analysis including 15 studies using the ISMI-29 (Livingston & Boyd, 2010). That article showed a striking and robust pattern of relationships between elevated internalized stigma and higher depression, lower self-esteem, and lower empowerment (Livingston & Boyd, 2010). The ISMI-10 results are in full accord with these results. In addition, we found that higher stigma levels on the ISMI-10 are associated with signifi-
Table 1 Correlations in the validation sample (N ⫽ 127)
10-item ISMI 29-item ISMI
ISMI-29 Internalized stigma
CES-D Depression
SES Self-esteem
RAS-41 Recovery
BUES-17 Empowerment
PES Empowerment
PDD Societal stigma
.94ⴱⴱ 1.0
.55ⴱⴱ .53ⴱⴱ
⫺.64ⴱⴱ ⫺.59ⴱⴱ
⫺.54ⴱⴱ ⫺.50ⴱⴱ
⫺.54ⴱⴱ ⫺.52ⴱⴱ
⫺.36ⴱⴱ ⫺.34ⴱⴱ
.31ⴱⴱ .35ⴱⴱ
Note. ISMI-29 ⫽ Internalized Stigma of Mental Illness (ISMI) scale (29-item version) (Ritsher et al., 2003); CES-D ⫽ Center for Epidemiological Studies-Depression (CES-D) scale (Radloff, 1977); SES ⫽ Rosenberg Self-Esteem Scale (Rosenberg, 1979, 1989); RAS-41 ⫽ The Recovery Assessment Scale (RAS) (41-item version) (Corrigan et al., 1999); BUES-17 ⫽ The 17 items from Factor 1 in the validation study for the Boston University Empowerment Scale (Rogers et al., 1997); PES ⫽ Personal Empowerment Scale (PES) (Segal et al., 1995); PDD ⫽ Perceived Devaluation and Discrimination scale (Link et al., 2001; Link et al., 1997; Perlick et al., 2001). ⴱⴱ p ⬍ .001.
10-ITEM ISMI SCALE
cantly higher scores on perceived devaluation and discrimination and lower scores on recovery orientation. As noted above, we expected only a medium effect size for devaluation and discrimination due to the differences between the two constructs (internal vs. societal stigma) and our past results (Ritsher et al., 2003). Similarly, we did not expect extremely high correlations with any of the constructs due to differences between them. Indeed, the original ISMI-29 validation study included a series of 2-factor factor analyses that showed that the items from each pair of scales sorted into separate factors (Ritsher et al., 2003). The fact that longitudinal studies show changes in outcome variables, such as depression, holding the baseline levels constant also argues for the distinction between constructs (Ritsher & Phelan, 2004). Although the ISMI-10 covers all five dimensions of internalized stigma measured by the ISMI-29, it was not our intention to create five two-item subscales. Thus, we recommend using the total ISMI-10 score rather than dividing it into subscales. Researchers and others wishing to differentiate or examine the subdomains of Alienation, Discrimination Experience, Social Withdrawal, Stereotype Endorsement, and Stigma Resistance should consider using the items from the full ISMI (Ritsher et al., 2003). Two-item scales are likely to be unstable. The brief ISMI-10 is intended for using as part of a larger series of scales where response burden is an issue. The ISMI-29 takes about 5 minutes to complete and is somewhat challenging to self-score, whereas the ISMI-10 takes 1–2 minutes to complete, and can be easily self-scored (reverse code 2 items, total the score, divide by 10). Anecdotally, there are indications that the ISMI-10 has utility for clinical and program evaluation purposes. Clinically, it has been used in the Psychosocial Rehabilitation and Recovery Center (PRRC) at the San Francisco VA Medical Center as the basis of productive discussions about stigma in the context of group therapy for veterans with serious mental illness. The ISMI-10 provides a simpler focus for this type of discussion than the ISMI-29. In terms of program evaluation, the ISMI-10 has been included in a Northeast Program Evaluation Center (NEPEC) evaluation of PRRC programs nationwide. Pilot data at an initial set of sites has been successfully collected and analyzed (Hunt & Smith, 2012; Wong, Boyd, & Hunt 2013).
Limitations The data available for the cross-validation sample did not allow us to confirm the relationship between the ISMI-10 and selfesteem or empowerment. Moreover, it was beyond the scope of the study to conduct the optional 10th step proposed by Stanton and colleagues (Stanton et al., 2002), which was multigroup structural equation modeling. We were also not able to use a cross-validation sample that had originally used only the 10-item version rather than the full ISMI-29 (Stanton et al., 2002). In other words, we did not test the brief version on its own. An adequate research dataset is not yet available for this purpose. This would be an important step in the establishment of the ISMI-10 that should be completed in future work. Also, we did not test a brief version of any of the more than 60 other language and disorder versions of the ISMI-29 (Boyd et al., 2014). Furthermore, this article considers only internal consistency and convergent validity, due to the nature of the “best practices” steps that we followed (Stanton et al., 2002), and the limitations of the available datasets. One of the most notewor-
21
thy limitations of these datasets is the narrow range of demographics among respondents (all were veterans, and most were male). Longitudinal studies reviewed by Livingston and Boyd (2010) show that the ISMI-29 is able to show meaningful changes over time. The fact that it has been so widely used in at least 70 countries around the world shows that the ISMI-29 has wide acceptability (Boyd et al., 2014). We can only assume that because the ISMI-10 is a subset of the same items, it will have similar properties. Although informal focus group results show that the ISMI-10 takes less than half the amount of time to complete as the ISMI-29, this has not been formally tested. Future studies will be needed to investigate these points.
Conclusions We conclude that the ISMI-10 has strong psychometric properties and is a practical, reliable, and valid alternative to the original ISMI-29. As expected, the ISMI-10 results showed that internalized stigma is significantly related to indicators of depression, self-esteem, recovery orientation, empowerment, and perceived devaluation and discrimination. We hope that the advent of the brief ISMI-10 will stimulate further longitudinal studies and program evaluation projects to shed more light on ways to reduce internalized stigma.
References Bornstein, R. F. (1996). Face validity in psychological assessment: Implications for a unified model of validity. American Psychologist, 51, 983–984. doi:10.1037/0003-066X.51.9.983 Boyd, J. E., Adler, E. P., Otilingam, P. G., & Peters, T. (2014). Internalized Stigma of Mental Illness (ISMI) Scale: A multinational review. Comprehensive Psychiatry, 55, 221–231. doi:10.1016/j.comppsych.2013.06 .005. Epub 2013 Sep 21. Brohan, E., Slade, M., Clement, S., & Thornicroft, G. (2010). Experiences of mental illness stigma, prejudice and discrimination: A review of measures. BMC Health Service Research, 10, 80. doi:10.1186/14726963-10-80 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Corrigan, P. W., Giffort, D., Rashid, F., Leary, M., & Okeke, I. (1999). Recovery as a psychological construct. Community Mental Health Journal, 35, 231–239. doi:10.1023/A:1018741302682 Corrigan, P. W., & Watson, A. C. (2002). The paradox of self-stigma and mental illness. Clinical Psychology: Science and Practice, 9, 35–53. doi:10.1093/clipsy.9.1.35 Derogatis, L. R., Rickels, K., & Rock, A. F. (1976). The SCL-90 and the MMPI: A step in the validation of a new self-report scale. British Journal of Psychiatry, 128, 280 –289. doi:10.1192/bjp.128.3.280 Dickerson, F. B., Sommerville, J., Origoni, A. E., Ringel, N. B., & Parente, F. (2002). Experiences of stigma among outpatients with schizophrenia. Schizophrenia Bulletin, 28, 143–155. doi:10.1093/oxfordjournals.schbul .a006917 Drapalski, A. L., Lucksted, A., Perrin, P. B., Aakre, J. M., Brown, C. H., DeForge, B. R., & Boyd, J. E. (2013). A model of internalized stigma and its effects on people with mental illness. Psychiatric Services, 64, 264 –269. doi:10.1176/appi.ps.001322012 Ersoy, M. A., & Varan, A. (2007). Ruhsal hastalıklarda içselles¸tirilmis¸ damgalanma ölçeg˘ i Türkçe formu’nun güvenilirlik ve geçerlik çalıs¸ması. [Reliability and validity of the Turkish version of the internalized stigma of mental illness scale]. Turk Psikiyatri Derg, 18, 163– 171.
22
BOYD, OTILINGAM, AND DEFORGE
Ghanean, H., Nojomi, M., & Jacobsson, L. (2011). Internalized stigma of mental illness in Tehran, Iran. Stigma Research and Action, 1, 11–17. doi:10.5463/sra.v1i1.10 Hinshaw, S. P. (2007). The mark of shame: stigma of mental illness and an agenda for change. New York, NY: Oxford University Press. Hunt, M., & Smith, T. (2012). Psychosocial Rehabilitation and Recovery Centers (PRRC): Program evaluation pilot data, SMI Committee. Washington, DC: Department of Veterans Affairs. Hwang, T. Y., Lee, W. K., Han, E. S., & Kwon, E. J. (2006). A study on the reliability and validity of the Korean Version of Internalized Stigma of Mental Illness Scale (K-ISMI). Journal of the Korean Neuropsychiatric Association, 45, 418 – 426. Kasprow, W. J., & Rosenheck, R. A. (2008). Interim report on the special needs - chronically mentally ill (SN-CMI) program for hospitalized homeless veterans. West Haven, CT: Northeast Program Evaluation Center. Kopalle, P. K., & Lehmann, D. R. (1997). Alpha inflation? The impact of eliminating scale items on Cronbach’s Alpha. Organizational Behavior and Human Decision Processes, 70, 189 –197. doi:10.1006/obhd.1997 .2702 Li, Q., Gao, W.-J., Bai, B.-Q., & Long, J. (2009). Preliminary application of Internalized Stigma of Mental Illness scale. Chinese Journal of Clinical Psychology, 17, 127–130. Link, B. G., Cullen, F. T., Struening, E., Shrout, P. E., & Dohrenwend, B. P. (1989). A modified labeling theory approach to mental disorders: An empirical assessment. American Sociological Review, 54, 400 – 423. doi:10.2307/2095613 Link, B. G., Struening, E. L., Neese-Todd, S., Asmussen, S., & Phelan, J. C. (2001). Stigma as a barrier to recovery: The consequences of stigma for the self-esteem of people with mental illnesses. Psychiatric Services, 52, 1621–1626. doi:10.1176/appi.ps.52.12.1621 Link, B. G., Struening, E. L., Rahav, M., Phelan, J. C., & Nuttbrock, L. (1997). On stigma and its consequences: Evidence from a longitudinal study of men with dual diagnoses of mental illness and substance abuse. Journal of Health and Social Behavior, 38, 177–190. doi:10.2307/ 2955424 Livingston, J. D., & Boyd, J. E. (2010). Correlates and consequences of internalized stigma for people living with mental illness: A systematic review and meta-analysis. Social Science and Medicine, 71, 2150 –2161. doi:10.1016/j.socscimed.2010.09.030 Lucksted, A., Drapalski, A., Calmes, C., Forbes, C., DeForge, B., & Boyd, J. (2011). Ending self-stigma: Pilot evaluation of a new intervention to reduce internalized stigma among people with mental illnesses. Psychiatric Rehabilitation Journal, 35, 51–54. doi:10.2975/35.1.2011.51.54 Lysaker, P. H., Roe, D., & Yanos, P. T. (2007). Toward understanding the insight paradox: Internalized stigma moderates the association between insight and social functioning, hope, and self-esteem among people with schizophrenia spectrum disorders. Schizophrenia Bulletin, 33, 192–199. doi:10.1093/schbul/sbl016
Overton, S. L., & Medina, S. L. (2008). The stigma of mental illness. Journal of Counseling and Development, 86, 143–151. doi:10.1002/j .1556-6678.2008.tb00491.x Pather, S., & Uys, C. S. (2008). Using scale reduction techniques for improved quality of survey information. South African Journal of Information Management, 10, 1–7. Perlick, D. A., Rosenheck, R. A., Clarkin, J. F., Sirey, J. A., Salahi, J., Struening, E. L., & Link, B. G. (2001). Stigma as a barrier to recovery: Adverse effects of perceived stigma on social adaptation of persons diagnosed with bipolar affective disorder. Psychiatric Services, 52, 1627–1632. doi:10.1176/appi.ps.52.12.1627 Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385– 401. doi:10.1177/014662167700100306 Ritsher (Boyd), J. B., Otilingam, P. O., & Grajales, M. (2003). Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research, 121, 31– 49. doi:10.1016/j.psychres.2003.08.008 Ritsher (Boyd), J. B., & Phelan, J. (2004). Internalized stigma predicts erosion of morale among psychiatric outpatients. Psychiatry Research, 129, 257–265. doi:10.1016/j.psychres.2004.08.003 Rogers, E. S., Chamberlin, J., Ellison, M. L., & Crean, T. (1997). A consumer-constructed scale to measure empowerment among users of mental health services. Psychiatric Services, 48, 1042–1047. Rosenberg, M. (1979). Conceiving the self. New York, NY: Basic Books. Rosenberg, M. (1989). Society and the adolescent self-image (Rev. ed.). Middletown, CT: Wesleyan University Press. Segal, S. P., Silverman, C., & Temkin, T. (1995). Measuring empowerment in client-run self-help agencies. Community Mental Health Journal, 31, 215–227. doi:10.1007/BF02188748 Sibitz, I., Amering, M., Unger, A., Seyringer, M., Bachmann, A., & Benesch, T. (2006). Internalisiertes Stigma bei Personen mit einer Erkrangung aus dem schizophrenen Formenkreis: Validierung der ISMI (Internalized Stigma of Mental Illness) Skala [Internalized stigma among persons with a schizophrenia spectrum disorder: Validation of the ISMI (Internalized Stigma of Mental Illness) scale.] (No. Projekt Nr. 2396.). Vienna: Forschungsbericht an den Medizinisch-Wissenschaftlichen Fonds des Burgermeisters der Bundeshauptstadt Wien. Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167–194. doi:10.1111/j.1744-6570.2002.tb00108.x Wahl, O. F. (1999). Mental health consumers’ experience of stigma. Schizophrenia Bulletin, 25, 467– 478. doi:10.1093/oxfordjournals.schbul .a033394 West, M. L., Yanos, P. T., Smith, S. M., Roe, D., & Lysaker, P. H. (2011). Prevalence of internalized stigma among persons with severe mental illness. Stigma Research and Action, 1, 3–10. doi:10.5463/sra.v1i1.9 Wong, J., Boyd, J. E., & Hunt, M. (2013). Poster presented at the 16th VA Psychology Leadership Conference: Internalized stigma among PRRC veterans. San Antonio, TX: the American Psychological Association, Association of VA Psychologist Leaders (AVAPL) and APA Division 18-Psychologists in Public Service.
(Appendix follows)
10-ITEM ISMI SCALE
23
Appendix Internalized Stigma of Mental Illness Inventory – 10-item Version (ISMI-10) We are going to use the term “mental illness” in the rest of this questionnaire, but please think of it as whatever you feel is the best term for it. For each question, please mark whether you strongly disagree (1), disagree (2), agree (3), or strongly agree (4). Strongly disagree
Disagree
Agree
Strongly agree
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1. Mentally ill people tend to be violent. 2. People with mental illness make important contributions to society. 3. I don’t socialize as much as I used to because my mental illness might make me look or behave “weird.” 4. Having a mental illness has spoiled my life. 5. I stay away from social situations in order to protect my family or friends from embarrassment. 6. People without mental illness could not possibly understand me. 7. People ignore me or take me less seriously just because I have a mental illness. 8. I can’t contribute anything to society because I have a mental illness. 9. I can have a good, fulfilling life, despite my mental illness. 10. Others think that I can’t achieve much in life because I have a mental illness.
Scoring Key The ISMI-10 contains 10 items which produce a total score. Reverse-code items 2 and 9 before calculating the total score. Add the item scores together and then divide by the total number of answered items. The resulting score should range from 1-4. For example, if someone answers 8 of the 10 items, the total score is produced by adding together the 8 answered items and dividing by 8.
Interpretation of Scores 4-category method (following the method used by Lysaker et al., 2007): 1.00-2.00: minimal to no internalized stigma 2.01-2.50: mild internalized stigma 2.51-3.00: moderate internalized stigma 3.01-4.00: severe internalized stigma 2-category method (following the method used by Ritsher [Boyd] & Phelan, 2004). 1.00-2.50: does not report high internalized stigma 2.51-4.00: reports high internalized stigma Received July 29, 2013 Revision received October 21, 2013 Accepted October 22, 2013 䡲
