Psychological Assessment 2014, Vol. 26. No. I. 321-325
© 2013 American Psychological Association 1040-3590/14/$12.00 DOI: 10.1037/a0034889
BRIEF REPORT
Diagnostic Accuracy of the Posttraumatic Stress Disorder ChecklistCivilian Version in a Representative Military Sample Karen-Inge Karstoft
S0ren Bo Andersen, Mette Bertelsen, and Trine IVIadsen
University of Southern Denmark
Research and Knowledge Centre, Danish Veteran Centre, Ringsted, Denmark This study aimed to assess the diagnostic accuracy of the Posttraumatic Stress Disorder ChecklistCivilian Version (PCL-C; Weathers, Litz, Herman, Huska, & Keane, 1993) and to establish the most accurate cutoff for prevalence estimation of posttraumalic stress disorder (PTSD) in a representative military sample compared lo a clinical interview. Danish soldiers (A^ = 415; 94.4% male, mean age 26.6 years) were assessed with the PCL-C and the Structured Clinical Interview for the DSM-IV (SCID; First, Spitzer, Gibbon, & Williams, 2002) 2.5 years after their return from deployment to Afghanistan. Diagnostic accuracy of the PCL-C was assessed through receiver operating charactedstic curve analysis. The PCL-C displayed high overall accuracy (area under the curve = .95, confidence interval [.92, .98]) and performed well (sensitivity > .70 and specificity a .90), with cutoff scores ranging from 37 to 44. When including sensitivity values a little below .70 (.69), the PCL-C performed well for cutoff levels up to 53. Prevalence of PTSD vaded considerably with the application of different cutoff values and scodng methods. Our results show that the PCL-C is a relevant and valid tool for screening for probable PTSD in active military samples. However, it is of great importance that cutoff scores be chosen based on the sample and the purpose of the particular study or screening. Keywords: posttraumatic stress disorder, prevalence, sensitivity and specificity, PTSD Checklist, diagnostic accuracy
The Posttraumatic Stress Disorder Checklist (PCL; Weathers, Litz, Herman, Huska, & Keane, 1993) is a widely applied measure for the general assessment and screening of posttraumatic stress disorder (PTSD). The PCL has previously been used to estimate the prevalence of PTSD in military samples (e.g.. Fear et al., 2010; Hoge et al., 2004; Iversen et al., 2009; Smith et al, 2009). The presence or absence of PTSD is generally determined via cutoff scores. However, the exact value of such a cutoff score is under debate. Indeed, the extant research has yet to establish which cutoff score is most accurate in distinguishing cases of PTSD from noncases in representative military samples (McDonald & Calhoun, 2010). Optimal values for cutoff scores for the PCL have been found to vary across different trauma samples and settings. Generally,
higher cutoff scores perform optimally in highly traumatized samples (44-60; Andrykowski, Cordova, Studts, & Miller, 1998; Blanchard, Jones-Alexander, Buckley, & Fomeds, 1996; Forbes, Creamer, & Biddle, 2001; Keen, Kutter, Niles, & Kdnsley, 2008; Weathers et al., 1993), while substantially lower cutoff scores have been suggested for screening use in pdmary care settings (28-38; Dobie et al., 2002; Lang, Laffaye, Satz, Dresselhaus, & Stein, 2003; Walker, Newman, Dobie, Ciechanowski, & Katon, 2002; Yeager, Magruder, Knapp, Nicholas, & Frueh, 2007). However, when the actual prevalence of PTSD is low, the overestimation of PTSD prevalence dedved from the PCL is greater than in samples with high PTSD prevalence (McDonald & Calhoun, 2010). This underlines the importance on basing cutoff scores on characteristics of the sample in question. To the best of our knowledge, only one study has investigated the diagnostic accuracy of the PCL in an active military sample. This study was conducted by Bliese et al. (2008), who identified cutoff values for the PCL by means of signal detection theory. By using the Mini International Neuropsychiatdc Interview (MINI; Sheehan et al., 1998) as the gold standard, the authors found that the highest specificity and sensitivity for the PCL was found with cutoff scores ranging from 30-34. While this is in line with, but in the low range of, suggested values for screening in pdmary care settings, it is much lower than the cutoff values that are often applied in studies of PTSD prevalence in military samples (see, for example. Fear et al., 2010; Iversen et al., 2009). Clearly, the
This article was published Online First November 4, 2013. Karen-Inge Karstoft, The National Centre for Psychotraumatology, Department of Psychology, University of Southern Denmark, Odense, Funen, Denmark; S0ren Bo Andersen, Mette Bertelsen, and Tdne Madsen, Research and Knowledge Centre, Danish Veteran Centre, Ringsted, Denmark. This study received financial support from TrygFonden. Correspondence concerning this article should be addressed to KarenInge Karstoft, The National Centre for Psychotraumatology, Department of Psychology, University of Southern Denmark, Campusvej 55, Odense, Funen, Denmark. E-mail: [email protected] 321
KARSTOFT, ANDERSEN, BERTELSEN, AND MADSEN
322
defined cutoff value for the PCL significantly impacts on the estimated prevalence of PTSD. It is therefore important that PCL cutoff values be further validated in representative military samples. Recent studies have based estimates of PTSD prevalence on individuals scoring above 50 on the PCL and meeting the criteria of the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) of at least one intrusion symptom, at least three avoidance symptoms, and at least one hyperarousal symptom (Hoge et al., 2004; Smith et al., 2008, 2009). This is expected to increase specificity of the test and decrease the number of false positives but might, however, increase the number of false negatives (McDonald & Calhoun, 2010). Moreover, this scoring method of combining DSM criteria with a cutoff has not yet been validated. In a representative sample of active Danish soldiers who were recently deployed to Afghanistan, we aimed to compare the accuracy of different scoring methods of the Civilian Version of the PCL (PCL-C; Weathers et al., 1993) with diagnosis established through a validated clinical interview for psychiatric disorders, the Structured Clinical Interview for the DSM-IV (SCID; First, Spitzer, Gibbon, & Williams, 2002). Furthermore, we assessed the diagnostic accuracy of the PCL-C based on different cutoffs.
Method
Participants The current sample is part of a longitudinal study of Danish Intemational Security Assistance Force (ISAF) combat soldiers who deployed to Afghanistan for 6 months. The study was initiated in 2009 prior to deployment and encompasses the whole cohort of Danish soldiers who deployed to Afghanistan at the time. The soldiers deployed to the Helmand province in Afghanistan and participated in joint operations with nations in ISAF, especially with and under the command of the United Kingdom. The current study utilizes data from two postdeployment measurements on this sample, namely an assessment conducted within the first weeks after homecoming and a postdeployment follow-up conducted 3 years after deployment. In total, 743 soldiers deployed to Afghanistan, of which three died in combat and 51 were repatriated due to injuries, illness, or for psychological or other reasons. Of the remaining soldiers, 415 (60.0%) participated in the questionnaire study as well as a clinical interview 3 years after deployment; thus, our effective sample size is 415. The sample consisted mainly of male participants (n = 401; 94.4%), who were on average 26.6 years (SD = 7.20) of age. Of the effective sample, 78.4% reported having participated in combat patrols, while 87.7% experienced being under fire from the enemy. Furthermore, 69.1% reported having been in danger of getting injured or killed, and 66.3% reported having shot at the enemy. Finally, 48.4% watched someone get shot, while 57.3% experienced someone from the unit getting injured, killed, or disappearing during patrol. When compared to nonresponders from the original sample, responders were significantly older (p = .038). There were no differences between responders and nonresponders in gender distribution or marital status, no differences of years served in the military, no difference in predeployment personality traits (as measured by the NEO Personality Inventory-Revised; Costa &
McCrae, 2008), and no significant difference in predeployment PCL-C score.
Procedure Data for the first postdeployment assessment were collected at a standard homecoming meeting 1-3 weeks after homecoming. Soldiers were contacted again via mail 3 years after deployment for the follow-up. Reminders were sent to nonresponders after 3 months, and nonresponders were eventually contacted by phone. They were able to participate in one or both features of the study; (1) the questionnaire-based part or (2) the questionnaire-based part and a clinical interview. Participants received travel reimbursement and a gift certificate (value of 500 Danish Kroner «» 90 USD) for participation. The clinical interviews were performed by six graduate psychology students at the soldiers' current or former military workplace. The interviewers went through intensive SCID training, including a certification course. At this course, the interviewers individually scored multiple videotaped SCID interviews and discussed their ratings with an expert interviewer. One month into the interviews, interrater reliability was assessed through individual ratings of recorded SCID interviews. The overall interrater reliability for the SCID interviews was .99, whereas interrater reliability for the PTSD module was .73. While this can be considered relatively low, two things should be highhghted. First, interrater reliability of the SCID is based on only eight items, namely the two Stressor (A) criteria, the five intrusion criteria, and the overall rating of fulfillment of the intrusion criteria. Overall, the interviewers all agreed on the two A-criterion items. For the intrusion items, all interviewers agreed on three out of five items. For one item (B4, psychological distress when exposed to cues), one interviewer disagreed with the rest of the interviewers. Overall disagreement was present for only one item, namely that of fiashbacks (B3). Importantly, for the overall decision on presence or absence of the intrusion cluster, all interviewers agreed, and as such, no differences in diagnostic decision were present across interviewers.
Measures Combat Exposure Scale (CES; Keane et al., 1989). The CES consists of seven questions assessing the level of combat exposure. For the purpose of this study, CES was employed at the first postdeployment assessment. Each item was dichotomized and used individually to indicate specific combat exposure. Posttraumatic Stress Disorder Checklist-Civilian Version (PCL-C; Weathers et al., 1993). The PCL-C is a 17-item selfreport measure developed to capture the PTSD symptoms as described in the DSM-IV (American Psychiatric Association, 1994). The civilian version of the PCL does not limit the traumatic event to being military-related and was chosen so that traumas endorsed outside the military context were also included. The PCL-C has demonstrated high intemal validity (a = .94; Blanchard et al., 1996), which was also found in our sample (a = .94). Each item is scored from 1 (not at all) to 5 (extremely), and from these, a total score is computed. Structured Cliuicai Interview for the DSM-IV (SCID; First, Spitzer, Gihbon, & Wiliiams, 2002). The SCID covers Axis I disorders of the DSM-IV. The PTSD section includes questions on
PCL-C ACCURACY IN A MILITARY SAMPLE
323
the index trauma as well as the 17 DSM-IVVTSD symptoms. The SCID also assesses duration and current severity of symptoms. Each symptom is scored as absent, subthreshold, or threshold, and presence of current PTSD is based on the DSM-IV criteria.
Analysis Prevalence of true positives, false positives, tme negatives, and false negafives were calculated for relevant PCL-C cutoff scores as well as for DSM scoring criteria and combinations. The diagnostic accuracy of the PCL-C was assessed by plotting the receiver operating characteristic (ROC) curve, which plots sensitivity versus 1 - specificity. Presence or absence of PTSD based on the SCID interview was used as the state variable against which PCL-C performance was tested. The general accuracy of the test was estimated as the area under the curve (AUC; Shapiro, 1999), and pattems of sensitivity and specificity, positive, and negative predictive value for a range of different cutoffs were then calculated. A nonparametric approach was taken due to nonnormality of the PCL-C: Kolmogorov-Smimov, D (415) = .196,/? < .001. All data analyses were conducted in SPSS Version 19.
Results Prevalence of PTSD based on different cutoff values and approaches is presented in Table 1. Our gold standard, the SCID interview, revealed a prevalence of 7.0%. The cutoff-based approaches for the PCL-C found prevalence ranging from 6.5% (cutoff 50) to 18.8% (cutoff 34), whereas the combined cutoff and DSM approach found prevalence ranging from 6.0% (cutoff 50 and DSM-IV) to 9.4% (cutoff 34 and DSM-IV). The number of true positives ranged from 19 (cutoff 50 and DSM-IV) to 25 (cutoff 34). The number of false positives for different approaches ranged from 6 (cutoff 50 and DSM-IV) to 53 (cutoff 34). True negatives were in the range from 333 (cutoff 34) to 380 (cutoff 50 and DSM-IV), and false negatives fell between 4 (cutoff 34) and 10 (cutoff 50 and DSM-IV)Figure 1 shows the ROC curve of the PCL-C validated against the SCID. The AUC for the analysis was .95 (95% confidence interval [95% CI = .92, .98]), suggesting high overall accuracy of the PCL-C. Table 2 displays the diagnostic efficiency of the PCL-C for different cutoff values. As seen from Table 2, the
0.4 0.6 False positive rat«
Figure I. Receiving operator characteristic curve for the Posttraumatic Stress Disorder Checklist- .70 and specificity > .90) over a wide range of cutoff values; namely from 37-44. Within this range, sensifivity is optimized with a cutoff at 37, while specificity is optimized with a cutoff value of 44. When including sensitivity values a little below .70 (.69), the PCL-C performed well for cutoff levels up to 53.
Discussion Altogether, the PCL-C showed high overall accuracy, as demonstrated by an AUC of the ROC curve of .95. When including measures of sensitivity, specificity, positive predictive value, and negative predictive value, the statistical performance was optimal in the range from 37 to 44 (good performance being defined by sensitivity > .70 and specificity > .90). However, if accepting
Table 1 Prevalence of PTSD for Different Scoring Methods and Cutoff Values Measure
Prevalence
True positives"
False positives'
True negatives"
False negatives"
SCID, DSM-IV PCL-C, DSM-IV PCL-C, cutoff 34 PCL-C, cutoff 44 PCL-C, cutoff 50 PCL-C, cutoff 34, and DSM-IV PCL-C, cutoff 44, and DSM-IV PCL-C, cutoff 50, and DSM-IV
29 (7.0%) 39 (9.4%) 78(18.8%) 36 (8.7%) 27 (6.5%) 39 (9.4%) 32 (7.7%) 25 (6.0%)
29 20 25 21 20 20 20 19
19 53 15 7 19 12 6
399 367 333 371 379 367 374 380
9 4 8 9 9 9 10
Note. PTSD = posttraumatic stress disorder; SCID = Structured Clinical Interview for the DSM-IV\ DSMIV = Diagnostic and Statistical Manual of Mental Disorders (4th ed.); PCL-C = Posttraumatic Stress Disorder Checklist-Civilian Version. " Compared with the SCID.
KARSTOFT, ANDERSEN, BERTELSEN, AND MADSEN
324 Table 2
Sensitivity, Specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV) for Different Cutoff Values for the PCL-C Cutoff
Sensitivity
Specificity
PPV
NPV
36 37 38 40 41 42 43 44
.83 .79 .79 .79 .76 .76 .72 .72 .69 .69 .69 .69 .69 .69 .69 .69 .69 .59
.89 .91 .92 .93 .94 .95 .96 .96 .97 .97 .97 .97 .98 .98 .98 .98 .98 .98
.36 .40 .43 .46 .49 .53 .58 .58 .63 .63 .63 .63 .72 .72 .72 .72 .72 .69
.99 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .97
45 46 47 48 49 50 51 52 53 54 Note. sion.
PCL-C = Posttraumatic Stress Disorder Checklist-Civilian Ver-
sensitivity scores slightly below .70 (.69), cutoff scores as high as 53 performed well, including high positive predictive value and high negative predictive value. Not only do a wide range of cutoff values perform well for the PCL-C, but these different cutoff values entail great variance in PTSD prevalence, ranging from 6% to 18.8%. Hence, it is pertinent to base the selection of a cutoff score on the purpose of the study. Many prevalence studies of PTSD in active military samples have applied a cutoff score of 50 (see, for example, Hoge et al., 2004; Stnith et al., 2009). In this study we found that a PCL-C score of 50 resulted in a PTSD prevalence that most closely matched the PTSD prevalence as reported by the SCID (6.5% with PCL-C cutoff 50, 7% based on the SCID). Therefore, a PCL-C eutoff value of 50 may indicate reasonable prevalence estimates in active military samples. However, whereas the specificity of the PCL-C with a cutoff at 50 is very high (.98), the sensitivity is slightly below .70 (.69). This suggests that there is a risk of underestimating the prevalence and that the number of false negatives may become too high. Our study suggests that a eutoff score of 44 displays good specificity (.96) while maintaining acceptable sensitivity (.72), so this eutoff value presents a viable altemative if false negatives are a concem for the study purpose. For screening procedures where the goal is to identify as many cases as possible for referral to further assessment, our results suggest that a significantly lower eutoff value is appropriate. This is in line with earlier studies of nontreatment-seeking samples, such as samples from primary care settings (Dobie et al., 2002; Lang et al., 2003; Walker et al, 2002; Yeager et al., 2007), that have found statistically superior performance of the PCL with cutoff scores ranging between 30 and 38. Indeed, based on our results, we suggest that 37 might be an appropriate cutoff for screening in active military samples. It is clear from our results that in spite of the overall accuracy of the PCL-C, basing prevalence of PTSD in a sample solely on a
PCL-C score has several caveats, the major one being that the most exact cutoff score for a specific sample cannot be determined without knowledge of the actual prevalence of PTSD in that sample (McDonald & Calhoun, 2010). Clearly, validated diagnostic interviews should, when possible, be implemented for accurate estimation of PTSD prevalence. A limitation of the current study is that we used the version of the PCL that corresponds to the PTSD criteria in the DSM-TV. While this was the standard when the study commenced, it has recently been replaced by a PCL version corresponding to PTSD criteria in the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5: American Psychiatric Association, 2013). The results obtained in this study are, however, very general in that they address overall issues with the application of cutoff scores and show that cutoff scores must be chosen based on sample characteristics and study purpose. As such, even with the introduction of the DSM-5, this study provides valuable information on the general application of cutoff scores. In conclusion, we find that the PCL-C is a relevant and valid tool for screening for probable PTSD in active military samples. However, we also demonstrate that it is of the utmost importance to choose cutoff scores based on the sample and the purpose of the particular study or screening. Since our study is based on a representative sample, we suggest that our results can be applied to other military samples, especially those returning home from the wars in Iraq and Afghanistan. As such, this study provides useful knowledge on the application, usefulness, and diagnostic accuracy of the PCL-C.
References American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: American Psychiatric Publishing. Andrykowski, M. A., Cordova, M. J., Studts, J. L., & Miller, T. W. (1998). Posttraumatic stress disorder after treatment for breast cancer: Prevalence of diagnosis and use of the PTSD Checklist-Civilian Version (PCL-C) as a screening instrument. Joumal of Consulting and Clinical Psychology, 66, 586-590. doi:10.1037/0022-006X.66.3.586 Blanchard, E. B., Jones-Alexander, J., Buckley, T. C , & Forneris, C. A. (1996). Psychometric properties of the PTSD Checklist (PCL). Behaviour Research and Therapy, 34, 669-673. Bliese, P. D., Wright, K. M., Adler, A. B., Cabrera, O., Castro, C. A., & Hoge, C. W. (2008). Validating the primary care posttraumatic stress disorder screen and the Posttraumatic Stress Disorder Checklist with soldiers retuming from combat. Joumal of Consulting and Clinical Psychology, 76, 272-281. doi:10.1037/0022-006X.76.2.272 Costa, P., & McCrae, R. (2008). The Revised NEO Personality Inventory (NEO-PI-R). In G. Boyle, G. Matthews, & D. Saklofske (Eds.), The Sage handbook ofpersonality theory and assessment: Vol. 2. Personality measurement and testing (pp. 179-198). London, England: Sage. Dobie, D. J., Kivlahan, D. R., Maynard, C , Bush, K. R., McFall, M., Epler, A. J., & Bradley, K. A. (2002). Screening for post-traumatic stress disorder in female Veteran's Affairs patients: Validation of the PTSD Checklist. General Hospital Psychiatry, 24, 367-374. Fear, N. T., Jones, M., Murphy, D., Hull, L., Iversen, A. C , Coker, B Wessely, S. (2010, May 22-28). What are the consequences of deployment to Iraq and Afghanistan on the mental health of the UK armed forces? A cohort study. Lancet, 375, 1783-1797. doi:10.1016/S01406736(10)60672-1
PCL-C ACCURACY IN A MILITARY SAMPLE First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (2002). Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Non-patient Edition (SCÍD-I/NP). New York, NY: Biometrics Research Department, New York State Psychiatric Institute. Forbes, D., Creamer, M., & Biddle, D. (2001). The validity of the PTSD Checklist as a measure of symptomatic change in combat-related PTSD. Behaviour Research and Therapy, 39, 977-986. Hoge, C. W., Castro, C. A., Messer, S. C , McGurk, D., Cotting, D. I., & Koffman, R. L. (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. New England Joumal of Medicine, 351, 13-22. doi:10.1056/NEJMoa040603 Iversen, A. C , van Staden, L., Hughes, J. H., Browne, T., Hull, L., Hall, J., .. . Fear, N. T. (2009). The prevalence of common mental disorders and PTSD in the UK military: Using data from a clinical interview-based study. BMC Psychiatry, 9, 68. doi:10.1186/1471-244X-9-68 Keane, T. M.. Fairbank, J. A., Caddell, J., Zimering, R., Taylor, K., & Mora, C. (1989). Clinical evaluation of a measure to assess combat exposure. Psychological Assessment, I, 53-55. doi: 10.1037/1040-3590 .1.1.53 Keen, S. M., Kutter, C. J., Niles, B. L., & Krinsley, K. E. (2008). Psychometric properties of PTSD Checklist in sample of male veterans. Joumal of Rehabilitation Research and Development, 45, 465-474. doi:10.1682/JRRD.2007.09.0138 Lang, A. J., Laffaye, C , Satz, L. E., Dresselhaus, T. R., & Stein, M. B. (2003). Sensitivity and specificity of the PTSD Checklist in detecting PTSD in female veterans in primary care. Joumal of Trauma Stress, 16, 257-264. doi: 10.1023/A: 1023796007788 McDonald, S. D., & Calhoun, P. S. (2010). The diagnostic accuracy ofthe PTSD Checklist: A critical review. Clinical Psychology Review, 30, 976-987. doi:10.1016/j.cpr.2010.06.012 Shapiro, D. E. (1999). The interpretation of diagnostic tests. Statistical
Methods in Medical Research, 8, 113-134. doi:10.1191/ 096228099666928387
325
Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E Dunbar, G. C. (1998). The Mini-International Neuropsychiatrie Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Joumal of Clinical Psychiatry, 59(Suppl. 20), 22-57. Smith, T. C , Ryan, M. A., Wingard, D. L., Slymen, D. J., Sallis, J. F., & Kritz-Silverstein, D. (2008). New onset and persistent symptoms of post-traumatic stress disorder self-reported after deployment and combat exposures: Prospective population based US military cohort study. British Medical Joumal, 336, 366-371. doi:10.1136/bmj.39430.638241.AE Smith, T. C , Wingard, D. L., Ryan, M. A. K., Kritz-Silverstein, D., Slymen, D. J., & Sallis, J. F. (2009). PTSD prevalence, associated exposures, and functional health outcomes in a large, population-based military cohort. Public Health Reports, 124, 90-102. Walker, E. A., Newman, E., Dobie, D. J., Ciechanowski, P., & Katon, W. (2002). Validation of the PTSD Checklist in an HMO sample of women. General Hospital Psychiatry, 24, 375-380. Weathers, F. W., Litz, B. T., Herman, D., Huska, J., &. Keane, T. M. (1993). The PTSD Checklist (PCL): Reliability, validity, and diagnostic utility. Paper presented at the Annual Convention of the Intemational Society for Traumatic Stress Studies, San Antonio, TX. Yeager, D. E., Magruder, K. M., Knapp, R. G., Nicholas, J. S., & Fmeh, B. C. (2007). Performance characteristics of the Posttraumatic Stress Disorder Checklist and SPAN in Veterans Affairs primary care settings. General Hospital Psychiatry, 29, 294-301. doi:10.1016/j.genhosppsych .2007.03.004
Received April 23, 2013 Revision received September 23, 2013 Accepted September 23, 2013
Copyright of Psychological Assessment is the property of American Psychological Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
© 2013 American Psychological Association 1040-3590/14/$12.00 DOI: 10.1037/a0034889
BRIEF REPORT
Diagnostic Accuracy of the Posttraumatic Stress Disorder ChecklistCivilian Version in a Representative Military Sample Karen-Inge Karstoft
S0ren Bo Andersen, Mette Bertelsen, and Trine IVIadsen
University of Southern Denmark
Research and Knowledge Centre, Danish Veteran Centre, Ringsted, Denmark This study aimed to assess the diagnostic accuracy of the Posttraumatic Stress Disorder ChecklistCivilian Version (PCL-C; Weathers, Litz, Herman, Huska, & Keane, 1993) and to establish the most accurate cutoff for prevalence estimation of posttraumalic stress disorder (PTSD) in a representative military sample compared lo a clinical interview. Danish soldiers (A^ = 415; 94.4% male, mean age 26.6 years) were assessed with the PCL-C and the Structured Clinical Interview for the DSM-IV (SCID; First, Spitzer, Gibbon, & Williams, 2002) 2.5 years after their return from deployment to Afghanistan. Diagnostic accuracy of the PCL-C was assessed through receiver operating charactedstic curve analysis. The PCL-C displayed high overall accuracy (area under the curve = .95, confidence interval [.92, .98]) and performed well (sensitivity > .70 and specificity a .90), with cutoff scores ranging from 37 to 44. When including sensitivity values a little below .70 (.69), the PCL-C performed well for cutoff levels up to 53. Prevalence of PTSD vaded considerably with the application of different cutoff values and scodng methods. Our results show that the PCL-C is a relevant and valid tool for screening for probable PTSD in active military samples. However, it is of great importance that cutoff scores be chosen based on the sample and the purpose of the particular study or screening. Keywords: posttraumatic stress disorder, prevalence, sensitivity and specificity, PTSD Checklist, diagnostic accuracy
The Posttraumatic Stress Disorder Checklist (PCL; Weathers, Litz, Herman, Huska, & Keane, 1993) is a widely applied measure for the general assessment and screening of posttraumatic stress disorder (PTSD). The PCL has previously been used to estimate the prevalence of PTSD in military samples (e.g.. Fear et al., 2010; Hoge et al., 2004; Iversen et al., 2009; Smith et al, 2009). The presence or absence of PTSD is generally determined via cutoff scores. However, the exact value of such a cutoff score is under debate. Indeed, the extant research has yet to establish which cutoff score is most accurate in distinguishing cases of PTSD from noncases in representative military samples (McDonald & Calhoun, 2010). Optimal values for cutoff scores for the PCL have been found to vary across different trauma samples and settings. Generally,
higher cutoff scores perform optimally in highly traumatized samples (44-60; Andrykowski, Cordova, Studts, & Miller, 1998; Blanchard, Jones-Alexander, Buckley, & Fomeds, 1996; Forbes, Creamer, & Biddle, 2001; Keen, Kutter, Niles, & Kdnsley, 2008; Weathers et al., 1993), while substantially lower cutoff scores have been suggested for screening use in pdmary care settings (28-38; Dobie et al., 2002; Lang, Laffaye, Satz, Dresselhaus, & Stein, 2003; Walker, Newman, Dobie, Ciechanowski, & Katon, 2002; Yeager, Magruder, Knapp, Nicholas, & Frueh, 2007). However, when the actual prevalence of PTSD is low, the overestimation of PTSD prevalence dedved from the PCL is greater than in samples with high PTSD prevalence (McDonald & Calhoun, 2010). This underlines the importance on basing cutoff scores on characteristics of the sample in question. To the best of our knowledge, only one study has investigated the diagnostic accuracy of the PCL in an active military sample. This study was conducted by Bliese et al. (2008), who identified cutoff values for the PCL by means of signal detection theory. By using the Mini International Neuropsychiatdc Interview (MINI; Sheehan et al., 1998) as the gold standard, the authors found that the highest specificity and sensitivity for the PCL was found with cutoff scores ranging from 30-34. While this is in line with, but in the low range of, suggested values for screening in pdmary care settings, it is much lower than the cutoff values that are often applied in studies of PTSD prevalence in military samples (see, for example. Fear et al., 2010; Iversen et al., 2009). Clearly, the
This article was published Online First November 4, 2013. Karen-Inge Karstoft, The National Centre for Psychotraumatology, Department of Psychology, University of Southern Denmark, Odense, Funen, Denmark; S0ren Bo Andersen, Mette Bertelsen, and Tdne Madsen, Research and Knowledge Centre, Danish Veteran Centre, Ringsted, Denmark. This study received financial support from TrygFonden. Correspondence concerning this article should be addressed to KarenInge Karstoft, The National Centre for Psychotraumatology, Department of Psychology, University of Southern Denmark, Campusvej 55, Odense, Funen, Denmark. E-mail: [email protected] 321
KARSTOFT, ANDERSEN, BERTELSEN, AND MADSEN
322
defined cutoff value for the PCL significantly impacts on the estimated prevalence of PTSD. It is therefore important that PCL cutoff values be further validated in representative military samples. Recent studies have based estimates of PTSD prevalence on individuals scoring above 50 on the PCL and meeting the criteria of the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) of at least one intrusion symptom, at least three avoidance symptoms, and at least one hyperarousal symptom (Hoge et al., 2004; Smith et al., 2008, 2009). This is expected to increase specificity of the test and decrease the number of false positives but might, however, increase the number of false negatives (McDonald & Calhoun, 2010). Moreover, this scoring method of combining DSM criteria with a cutoff has not yet been validated. In a representative sample of active Danish soldiers who were recently deployed to Afghanistan, we aimed to compare the accuracy of different scoring methods of the Civilian Version of the PCL (PCL-C; Weathers et al., 1993) with diagnosis established through a validated clinical interview for psychiatric disorders, the Structured Clinical Interview for the DSM-IV (SCID; First, Spitzer, Gibbon, & Williams, 2002). Furthermore, we assessed the diagnostic accuracy of the PCL-C based on different cutoffs.
Method
Participants The current sample is part of a longitudinal study of Danish Intemational Security Assistance Force (ISAF) combat soldiers who deployed to Afghanistan for 6 months. The study was initiated in 2009 prior to deployment and encompasses the whole cohort of Danish soldiers who deployed to Afghanistan at the time. The soldiers deployed to the Helmand province in Afghanistan and participated in joint operations with nations in ISAF, especially with and under the command of the United Kingdom. The current study utilizes data from two postdeployment measurements on this sample, namely an assessment conducted within the first weeks after homecoming and a postdeployment follow-up conducted 3 years after deployment. In total, 743 soldiers deployed to Afghanistan, of which three died in combat and 51 were repatriated due to injuries, illness, or for psychological or other reasons. Of the remaining soldiers, 415 (60.0%) participated in the questionnaire study as well as a clinical interview 3 years after deployment; thus, our effective sample size is 415. The sample consisted mainly of male participants (n = 401; 94.4%), who were on average 26.6 years (SD = 7.20) of age. Of the effective sample, 78.4% reported having participated in combat patrols, while 87.7% experienced being under fire from the enemy. Furthermore, 69.1% reported having been in danger of getting injured or killed, and 66.3% reported having shot at the enemy. Finally, 48.4% watched someone get shot, while 57.3% experienced someone from the unit getting injured, killed, or disappearing during patrol. When compared to nonresponders from the original sample, responders were significantly older (p = .038). There were no differences between responders and nonresponders in gender distribution or marital status, no differences of years served in the military, no difference in predeployment personality traits (as measured by the NEO Personality Inventory-Revised; Costa &
McCrae, 2008), and no significant difference in predeployment PCL-C score.
Procedure Data for the first postdeployment assessment were collected at a standard homecoming meeting 1-3 weeks after homecoming. Soldiers were contacted again via mail 3 years after deployment for the follow-up. Reminders were sent to nonresponders after 3 months, and nonresponders were eventually contacted by phone. They were able to participate in one or both features of the study; (1) the questionnaire-based part or (2) the questionnaire-based part and a clinical interview. Participants received travel reimbursement and a gift certificate (value of 500 Danish Kroner «» 90 USD) for participation. The clinical interviews were performed by six graduate psychology students at the soldiers' current or former military workplace. The interviewers went through intensive SCID training, including a certification course. At this course, the interviewers individually scored multiple videotaped SCID interviews and discussed their ratings with an expert interviewer. One month into the interviews, interrater reliability was assessed through individual ratings of recorded SCID interviews. The overall interrater reliability for the SCID interviews was .99, whereas interrater reliability for the PTSD module was .73. While this can be considered relatively low, two things should be highhghted. First, interrater reliability of the SCID is based on only eight items, namely the two Stressor (A) criteria, the five intrusion criteria, and the overall rating of fulfillment of the intrusion criteria. Overall, the interviewers all agreed on the two A-criterion items. For the intrusion items, all interviewers agreed on three out of five items. For one item (B4, psychological distress when exposed to cues), one interviewer disagreed with the rest of the interviewers. Overall disagreement was present for only one item, namely that of fiashbacks (B3). Importantly, for the overall decision on presence or absence of the intrusion cluster, all interviewers agreed, and as such, no differences in diagnostic decision were present across interviewers.
Measures Combat Exposure Scale (CES; Keane et al., 1989). The CES consists of seven questions assessing the level of combat exposure. For the purpose of this study, CES was employed at the first postdeployment assessment. Each item was dichotomized and used individually to indicate specific combat exposure. Posttraumatic Stress Disorder Checklist-Civilian Version (PCL-C; Weathers et al., 1993). The PCL-C is a 17-item selfreport measure developed to capture the PTSD symptoms as described in the DSM-IV (American Psychiatric Association, 1994). The civilian version of the PCL does not limit the traumatic event to being military-related and was chosen so that traumas endorsed outside the military context were also included. The PCL-C has demonstrated high intemal validity (a = .94; Blanchard et al., 1996), which was also found in our sample (a = .94). Each item is scored from 1 (not at all) to 5 (extremely), and from these, a total score is computed. Structured Cliuicai Interview for the DSM-IV (SCID; First, Spitzer, Gihbon, & Wiliiams, 2002). The SCID covers Axis I disorders of the DSM-IV. The PTSD section includes questions on
PCL-C ACCURACY IN A MILITARY SAMPLE
323
the index trauma as well as the 17 DSM-IVVTSD symptoms. The SCID also assesses duration and current severity of symptoms. Each symptom is scored as absent, subthreshold, or threshold, and presence of current PTSD is based on the DSM-IV criteria.
Analysis Prevalence of true positives, false positives, tme negatives, and false negafives were calculated for relevant PCL-C cutoff scores as well as for DSM scoring criteria and combinations. The diagnostic accuracy of the PCL-C was assessed by plotting the receiver operating characteristic (ROC) curve, which plots sensitivity versus 1 - specificity. Presence or absence of PTSD based on the SCID interview was used as the state variable against which PCL-C performance was tested. The general accuracy of the test was estimated as the area under the curve (AUC; Shapiro, 1999), and pattems of sensitivity and specificity, positive, and negative predictive value for a range of different cutoffs were then calculated. A nonparametric approach was taken due to nonnormality of the PCL-C: Kolmogorov-Smimov, D (415) = .196,/? < .001. All data analyses were conducted in SPSS Version 19.
Results Prevalence of PTSD based on different cutoff values and approaches is presented in Table 1. Our gold standard, the SCID interview, revealed a prevalence of 7.0%. The cutoff-based approaches for the PCL-C found prevalence ranging from 6.5% (cutoff 50) to 18.8% (cutoff 34), whereas the combined cutoff and DSM approach found prevalence ranging from 6.0% (cutoff 50 and DSM-IV) to 9.4% (cutoff 34 and DSM-IV). The number of true positives ranged from 19 (cutoff 50 and DSM-IV) to 25 (cutoff 34). The number of false positives for different approaches ranged from 6 (cutoff 50 and DSM-IV) to 53 (cutoff 34). True negatives were in the range from 333 (cutoff 34) to 380 (cutoff 50 and DSM-IV), and false negatives fell between 4 (cutoff 34) and 10 (cutoff 50 and DSM-IV)Figure 1 shows the ROC curve of the PCL-C validated against the SCID. The AUC for the analysis was .95 (95% confidence interval [95% CI = .92, .98]), suggesting high overall accuracy of the PCL-C. Table 2 displays the diagnostic efficiency of the PCL-C for different cutoff values. As seen from Table 2, the
0.4 0.6 False positive rat«
Figure I. Receiving operator characteristic curve for the Posttraumatic Stress Disorder Checklist- .70 and specificity > .90) over a wide range of cutoff values; namely from 37-44. Within this range, sensifivity is optimized with a cutoff at 37, while specificity is optimized with a cutoff value of 44. When including sensitivity values a little below .70 (.69), the PCL-C performed well for cutoff levels up to 53.
Discussion Altogether, the PCL-C showed high overall accuracy, as demonstrated by an AUC of the ROC curve of .95. When including measures of sensitivity, specificity, positive predictive value, and negative predictive value, the statistical performance was optimal in the range from 37 to 44 (good performance being defined by sensitivity > .70 and specificity > .90). However, if accepting
Table 1 Prevalence of PTSD for Different Scoring Methods and Cutoff Values Measure
Prevalence
True positives"
False positives'
True negatives"
False negatives"
SCID, DSM-IV PCL-C, DSM-IV PCL-C, cutoff 34 PCL-C, cutoff 44 PCL-C, cutoff 50 PCL-C, cutoff 34, and DSM-IV PCL-C, cutoff 44, and DSM-IV PCL-C, cutoff 50, and DSM-IV
29 (7.0%) 39 (9.4%) 78(18.8%) 36 (8.7%) 27 (6.5%) 39 (9.4%) 32 (7.7%) 25 (6.0%)
29 20 25 21 20 20 20 19
19 53 15 7 19 12 6
399 367 333 371 379 367 374 380
9 4 8 9 9 9 10
Note. PTSD = posttraumatic stress disorder; SCID = Structured Clinical Interview for the DSM-IV\ DSMIV = Diagnostic and Statistical Manual of Mental Disorders (4th ed.); PCL-C = Posttraumatic Stress Disorder Checklist-Civilian Version. " Compared with the SCID.
KARSTOFT, ANDERSEN, BERTELSEN, AND MADSEN
324 Table 2
Sensitivity, Specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV) for Different Cutoff Values for the PCL-C Cutoff
Sensitivity
Specificity
PPV
NPV
36 37 38 40 41 42 43 44
.83 .79 .79 .79 .76 .76 .72 .72 .69 .69 .69 .69 .69 .69 .69 .69 .69 .59
.89 .91 .92 .93 .94 .95 .96 .96 .97 .97 .97 .97 .98 .98 .98 .98 .98 .98
.36 .40 .43 .46 .49 .53 .58 .58 .63 .63 .63 .63 .72 .72 .72 .72 .72 .69
.99 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .98 .97
45 46 47 48 49 50 51 52 53 54 Note. sion.
PCL-C = Posttraumatic Stress Disorder Checklist-Civilian Ver-
sensitivity scores slightly below .70 (.69), cutoff scores as high as 53 performed well, including high positive predictive value and high negative predictive value. Not only do a wide range of cutoff values perform well for the PCL-C, but these different cutoff values entail great variance in PTSD prevalence, ranging from 6% to 18.8%. Hence, it is pertinent to base the selection of a cutoff score on the purpose of the study. Many prevalence studies of PTSD in active military samples have applied a cutoff score of 50 (see, for example, Hoge et al., 2004; Stnith et al., 2009). In this study we found that a PCL-C score of 50 resulted in a PTSD prevalence that most closely matched the PTSD prevalence as reported by the SCID (6.5% with PCL-C cutoff 50, 7% based on the SCID). Therefore, a PCL-C eutoff value of 50 may indicate reasonable prevalence estimates in active military samples. However, whereas the specificity of the PCL-C with a cutoff at 50 is very high (.98), the sensitivity is slightly below .70 (.69). This suggests that there is a risk of underestimating the prevalence and that the number of false negatives may become too high. Our study suggests that a eutoff score of 44 displays good specificity (.96) while maintaining acceptable sensitivity (.72), so this eutoff value presents a viable altemative if false negatives are a concem for the study purpose. For screening procedures where the goal is to identify as many cases as possible for referral to further assessment, our results suggest that a significantly lower eutoff value is appropriate. This is in line with earlier studies of nontreatment-seeking samples, such as samples from primary care settings (Dobie et al., 2002; Lang et al., 2003; Walker et al, 2002; Yeager et al., 2007), that have found statistically superior performance of the PCL with cutoff scores ranging between 30 and 38. Indeed, based on our results, we suggest that 37 might be an appropriate cutoff for screening in active military samples. It is clear from our results that in spite of the overall accuracy of the PCL-C, basing prevalence of PTSD in a sample solely on a
PCL-C score has several caveats, the major one being that the most exact cutoff score for a specific sample cannot be determined without knowledge of the actual prevalence of PTSD in that sample (McDonald & Calhoun, 2010). Clearly, validated diagnostic interviews should, when possible, be implemented for accurate estimation of PTSD prevalence. A limitation of the current study is that we used the version of the PCL that corresponds to the PTSD criteria in the DSM-TV. While this was the standard when the study commenced, it has recently been replaced by a PCL version corresponding to PTSD criteria in the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5: American Psychiatric Association, 2013). The results obtained in this study are, however, very general in that they address overall issues with the application of cutoff scores and show that cutoff scores must be chosen based on sample characteristics and study purpose. As such, even with the introduction of the DSM-5, this study provides valuable information on the general application of cutoff scores. In conclusion, we find that the PCL-C is a relevant and valid tool for screening for probable PTSD in active military samples. However, we also demonstrate that it is of the utmost importance to choose cutoff scores based on the sample and the purpose of the particular study or screening. Since our study is based on a representative sample, we suggest that our results can be applied to other military samples, especially those returning home from the wars in Iraq and Afghanistan. As such, this study provides useful knowledge on the application, usefulness, and diagnostic accuracy of the PCL-C.
References American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: American Psychiatric Publishing. Andrykowski, M. A., Cordova, M. J., Studts, J. L., & Miller, T. W. (1998). Posttraumatic stress disorder after treatment for breast cancer: Prevalence of diagnosis and use of the PTSD Checklist-Civilian Version (PCL-C) as a screening instrument. Joumal of Consulting and Clinical Psychology, 66, 586-590. doi:10.1037/0022-006X.66.3.586 Blanchard, E. B., Jones-Alexander, J., Buckley, T. C , & Forneris, C. A. (1996). Psychometric properties of the PTSD Checklist (PCL). Behaviour Research and Therapy, 34, 669-673. Bliese, P. D., Wright, K. M., Adler, A. B., Cabrera, O., Castro, C. A., & Hoge, C. W. (2008). Validating the primary care posttraumatic stress disorder screen and the Posttraumatic Stress Disorder Checklist with soldiers retuming from combat. Joumal of Consulting and Clinical Psychology, 76, 272-281. doi:10.1037/0022-006X.76.2.272 Costa, P., & McCrae, R. (2008). The Revised NEO Personality Inventory (NEO-PI-R). In G. Boyle, G. Matthews, & D. Saklofske (Eds.), The Sage handbook ofpersonality theory and assessment: Vol. 2. Personality measurement and testing (pp. 179-198). London, England: Sage. Dobie, D. J., Kivlahan, D. R., Maynard, C , Bush, K. R., McFall, M., Epler, A. J., & Bradley, K. A. (2002). Screening for post-traumatic stress disorder in female Veteran's Affairs patients: Validation of the PTSD Checklist. General Hospital Psychiatry, 24, 367-374. Fear, N. T., Jones, M., Murphy, D., Hull, L., Iversen, A. C , Coker, B Wessely, S. (2010, May 22-28). What are the consequences of deployment to Iraq and Afghanistan on the mental health of the UK armed forces? A cohort study. Lancet, 375, 1783-1797. doi:10.1016/S01406736(10)60672-1
PCL-C ACCURACY IN A MILITARY SAMPLE First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (2002). Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Non-patient Edition (SCÍD-I/NP). New York, NY: Biometrics Research Department, New York State Psychiatric Institute. Forbes, D., Creamer, M., & Biddle, D. (2001). The validity of the PTSD Checklist as a measure of symptomatic change in combat-related PTSD. Behaviour Research and Therapy, 39, 977-986. Hoge, C. W., Castro, C. A., Messer, S. C , McGurk, D., Cotting, D. I., & Koffman, R. L. (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. New England Joumal of Medicine, 351, 13-22. doi:10.1056/NEJMoa040603 Iversen, A. C , van Staden, L., Hughes, J. H., Browne, T., Hull, L., Hall, J., .. . Fear, N. T. (2009). The prevalence of common mental disorders and PTSD in the UK military: Using data from a clinical interview-based study. BMC Psychiatry, 9, 68. doi:10.1186/1471-244X-9-68 Keane, T. M.. Fairbank, J. A., Caddell, J., Zimering, R., Taylor, K., & Mora, C. (1989). Clinical evaluation of a measure to assess combat exposure. Psychological Assessment, I, 53-55. doi: 10.1037/1040-3590 .1.1.53 Keen, S. M., Kutter, C. J., Niles, B. L., & Krinsley, K. E. (2008). Psychometric properties of PTSD Checklist in sample of male veterans. Joumal of Rehabilitation Research and Development, 45, 465-474. doi:10.1682/JRRD.2007.09.0138 Lang, A. J., Laffaye, C , Satz, L. E., Dresselhaus, T. R., & Stein, M. B. (2003). Sensitivity and specificity of the PTSD Checklist in detecting PTSD in female veterans in primary care. Joumal of Trauma Stress, 16, 257-264. doi: 10.1023/A: 1023796007788 McDonald, S. D., & Calhoun, P. S. (2010). The diagnostic accuracy ofthe PTSD Checklist: A critical review. Clinical Psychology Review, 30, 976-987. doi:10.1016/j.cpr.2010.06.012 Shapiro, D. E. (1999). The interpretation of diagnostic tests. Statistical
Methods in Medical Research, 8, 113-134. doi:10.1191/ 096228099666928387
325
Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E Dunbar, G. C. (1998). The Mini-International Neuropsychiatrie Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Joumal of Clinical Psychiatry, 59(Suppl. 20), 22-57. Smith, T. C , Ryan, M. A., Wingard, D. L., Slymen, D. J., Sallis, J. F., & Kritz-Silverstein, D. (2008). New onset and persistent symptoms of post-traumatic stress disorder self-reported after deployment and combat exposures: Prospective population based US military cohort study. British Medical Joumal, 336, 366-371. doi:10.1136/bmj.39430.638241.AE Smith, T. C , Wingard, D. L., Ryan, M. A. K., Kritz-Silverstein, D., Slymen, D. J., & Sallis, J. F. (2009). PTSD prevalence, associated exposures, and functional health outcomes in a large, population-based military cohort. Public Health Reports, 124, 90-102. Walker, E. A., Newman, E., Dobie, D. J., Ciechanowski, P., & Katon, W. (2002). Validation of the PTSD Checklist in an HMO sample of women. General Hospital Psychiatry, 24, 375-380. Weathers, F. W., Litz, B. T., Herman, D., Huska, J., &. Keane, T. M. (1993). The PTSD Checklist (PCL): Reliability, validity, and diagnostic utility. Paper presented at the Annual Convention of the Intemational Society for Traumatic Stress Studies, San Antonio, TX. Yeager, D. E., Magruder, K. M., Knapp, R. G., Nicholas, J. S., & Fmeh, B. C. (2007). Performance characteristics of the Posttraumatic Stress Disorder Checklist and SPAN in Veterans Affairs primary care settings. General Hospital Psychiatry, 29, 294-301. doi:10.1016/j.genhosppsych .2007.03.004
Received April 23, 2013 Revision received September 23, 2013 Accepted September 23, 2013
Copyright of Psychological Assessment is the property of American Psychological Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
