Journal of Anxiety Disorders 28 (2014) 51–56

Contents lists available at ScienceDirect

Journal of Anxiety Disorders

Diagnostic utility of CPSS vs. CAPS-CA for assessing posttraumatic stress symptoms in children and adolescents Silje Hukkelberg a,b,∗ , Silje M. Ormhaug a , Tonje Holt a , Tore Wentzel-Larsen a,c , Tine K. Jensen a,d a

Norwegian Centre for Violence and Traumatic Stress Studies, Kirkeveien 166, Building 48, 0450 Oslo, Norway Norwegian Center for Child Behavioral Development, P.O. Box 7053 Majorstuen, 0306 Oslo, Norway c Centre for Child and Adolescent Mental Health, Eastern and Southern Norway, Oslo, Norway d Department of Psychology, University of Oslo, Postbox 1094 Blindern, 0317 Oslo, Norway b

a r t i c l e

i n f o

Article history: Received 26 February 2013 Received in revised form 31 October 2013 Accepted 1 November 2013 Keywords: PTSD CPSS CAPS-CA Children ROC analysis

a b s t r a c t Objectives: This study compared the diagnostic utility of the symptom part of the child PTSD symptom scale (CPSS) screening instrument with the clinician-administered PTSD scale for children and adolescents (CAPS-CA). Methods: The study included a clinical sample of traumatized children and adolescents (mean age 15.1, range 10–18) living in Norway, who were assessed for posttraumatic stress symptoms using the CPSS and the CAPS-CA. Diagnostic utility was investigated using receiver operating characteristic analyses. Results: The results showed that CPSS reached medium effect sizes (AUC from .63 to .76). The sensitivity was good (.80), but the specificity was relatively low (.56). Kappa between CPSS and CAPS-CA was low ( = .27). Conclusions: Findings suggests that CPSS is a good tool for screening purposes, but not as a diagnostic instrument in an early phase of assessment. Implications and limitations of the findings are discussed. Crown Copyright © 2013 Published by Elsevier Ltd. All rights reserved.

1. Introduction Every year, a significant number of children and adolescents are exposed to potentially traumatizing events (Breslau, 2009; Cuffe et al., 1998; Finkelhor, Ormrod, Turner, & Hamby, 2005; Mossige & Stefansen, 2007; Schou, Dyb, & Graff-Iversen, 2007). In a Norwegian study it was found that among high school seniors, 25% reported at least one episode of physical abuse, and 15% of the girls and 7% of the boys reported exposure to sexual assault or abuse (Mossige & Stefansen, 2007). In a younger sample, 24% of the boys and 12% of the girls reported exposure to peer violence during the last year (Schou, Dyb, & Graff-Iversen, 2007). These are events that put young people at a high risk of developing psychopathology in general and posttraumatic stress symptoms (PTSS) in particular (Dube, Felitti, Dong, Giles, & Anda, 2003; Pynoos, Steinberg, & Piacentini, 1999). Figures indicate that approximately 30–40% of children and adolescents who experience any type of traumatic event will later satisfy the diagnostic criteria for posttraumatic stress disorder (PTSD) (Fletcher, 1996).

∗ Corresponding author at: Norwegian Centre for Violence and Traumatic Stress Studies, Kirkeveien 166, Building 48, 0450 Oslo, Norway. Tel.: +47 928 68 553. E-mail address: [email protected] (S. Hukkelberg).

The diagnostic and statistical manual of mental disorders (DSMIV) (American Psychiatric Association, 1994), describes PTSD as consisting of a range of symptoms in the aftermath of a traumatizing event that may cause clinically significant distress or impairment in social, occupational or other areas of functioning. Specifically, 17 symptoms of PTSD are assumed to cluster into three correlated factors: re-experiencing, avoidance, and hyperarousal. Although this tripartite factor structure has been questioned (e.g., Hukkelberg & Jensen, 2011), several confirmatory and exploratory factor analyses have provided support for these factors as representing common reactions to traumas in children and adolescents (Bal & Jensen, 2007; Ford, Elhai, Ruggiero, & Frueh, 2009; Giannopoulou et al., 2006). The high prevalence of exposure to traumatic events among children and adolescents emphasizes the need for validated instruments to effectively screen for PTSS and PTSD, e.g. to determine which children will be in need of trauma-focused interventions. Although a full PTSD diagnosis is seldom needed to recommend trauma-focused treatment, knowledge of how a screening instrument performs compared to a diagnostic instrument is of clinical value. There are currently a number of instruments available to screen for PTSS in youth, but none of these have been validated against a diagnostic interview to investigate their diagnostic utility (Ohan, Myers, & Collett, 2002). The clinician-administered PTSD scale for children and adolescents (CAPS-CA; Nader et al.,

0887-6185/$ – see front matter. Crown Copyright © 2013 Published by Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.janxdis.2013.11.001

52

S. Hukkelberg et al. / Journal of Anxiety Disorders 28 (2014) 51–56

1996) is considered the gold standard for assessing and diagnosing PTSD in children and adolescents (Griffin, Uhlmansiek, Resick, & Mechanic, 2004). It has been widely used for both clinical and research purposes, and has repeatedly demonstrated good psychometric properties (Harrington, 2009; Ohan et al., 2002). The interview, however, requires special training and is time consuming to administer. Thus, clinicians may find themselves in need of shorter and more efficient screening tools. One such instrument is the child PTSD symptom scale (CPSS; Foa, Johnson, Feeny, & Treadwell, 2001). This readily available self-rating screening instrument assesses PTSS in children and adolescents from age 8 to 18, and it takes less than 30 min to administer. The scale maps directly onto the 17 symptom items as described in the DSM-IV, and has demonstrated good psychometric properties in terms of convergent and divergent validity, and internal consistency (Foa et al., 2001; Ohan et al., 2002). As such, the CPSS may represent an effective tool for screening PTSS in young people. It remains unclear, however, whether it has utility as a diagnostic instrument. A study of an adult sample of women exposed to domestic violence found that the adult version of the CPSS (Foa, Cashman, Jaycox, & Perry, 1997), the Posttraumatic Diagnostic Scale, performed generally well compared to the adult version of CAPS-CA, although it had a tendency to over-diagnose PTSD (Griffin et al., 2004). Due to a different level of development, young people may show different posttraumatic stress reactions compared to adults (Pynoos et al., 2009). Thus, there are reasons to believe that the findings from adult samples may not necessarily generalize to young samples. To our knowledge, no instrument has been validated against an established gold standard for assessing PTSD in children and adolescents. The aim of the present study was to investigate the comparative utility of the interpretation of the test scores of CPSS against the CAPS-CA to establish its sensitivity, specificity and predictive power in a clinical sample of traumatized children.

2. Method 2.1. Setting and sample The participants were part of a larger study on the treatment of traumatized children and were recruited from eight child guidance clinics in different cities in Norway from April 2008 to February 2011. All youth were referred to the clinics through standard procedures (i.e., from a general practitioner or through the child welfare system). The eligibility criteria for the study required the children to be 10–18 years old, have experienced one or more potentially traumatic events, speak Norwegian, and have a CPSS total score ≥15. Children were excluded if they suffered from psychosis, had mental retardation, or presented symptoms of severe untreated conduct disorder before the traumatic experience. All children were to be treated for PTSS by means of trauma-focused cognitive behavioral therapy (TF-CBT; Cohen & Mannarino, 2008) or therapy as usual (TAU). For more information about the treatment conditions and procedures, see Jensen et al. (2013). Of 454 children, 217 had a CPSS < 15, and were therefore excluded from further participation. Eighty-one children declined to participate or satisfied one or more of the exclusion criteria. The remaining 156 children were assessed for posttraumatic stress symptoms. They were first assessed by CPSS, and approximately two weeks later by the CAPS-CA. The participants’ mean age was 15.08 (SD = 2.18), and 125 (80.1%) were girls. Most of the children had two (n = 114, 73.1%) or one (n = 14, 9.0%) Norwegian parent(s). Sixteen (10.3%) participants were Asian; 7 (4.5%) were from other European countries; 3 were African (1.9%); and 2 (1.3%) participants

were from South and Central America. The number of traumas reported by the sample ranged from 1 to 10 (M = 3.6, SD = 1.8), whereof most children (n = 86.4%) had experienced multiple traumatic events. The children reported the following events: 60.9% (n = 95) sudden death or severe illness of a close person, 59% (n = 92) violence or threats of violence outside the family context, 45.5% (n = 71) physical abuse within the family, 42.9% (n = 67) witnessing violence within the family, 27.6% (n = 43) witnessing violence outside the family, 27.6% (n = 43) sexual abuse outside the family, 20.5% (n = 32) severe accident, 16% (n = 25) extremely painful or frightening medical procedures, 10.9% (n = 17) robbery or assault, 7.7% (n = 12) sexual abuse within the family, 5.8% (n = 9) natural disaster, 5.1% (n = 8) kidnapping, and 30.8% (n = 48) other frightening or overwhelming experiences. The Norwegian Regional Ethical Committee approved the study.

2.2. Measures Child PTSD symptom scale (CPSS). The CPSS (Foa et al., 2001) is a self-report questionnaire whose wording and format makes it especially appropriate for children and adolescents 8 to 18 years. The instrument consists of two parts: one that assesses the 17 items as described in the DSM-IV manual PTSD symptoms (Criteria B, C and D) and a second part that measures daily functioning. The present study addresses the first part only. The instructions were as follows: “Circle the number that describes how often the problem has bothered you in the last 2 weeks”. Answers were evaluated using a 4-point scale from 0 to 3 (0 = Not at all, 1 = Once a week or less/once in a while, 2 = 2 to 4 times a week/half the time, 3 = 5 or more times a week/almost always). Assessments of the CPSS were conducted individually at the clinic, and a clinical psychologist was present to help read the questions or facilitate comprehension, if necessary. To facilitate the screening and scoring process, a computer version of the CPSS was used that allowed the youth to easily respond to the questions. Because most of the children had experienced more than one traumatic event, posttraumatic symptoms were assessed based on their self-reported worst experience. The CPSS was assessed ≥4 weeks after the trauma. The Norwegian version has shown good internal consistency in a screening sample (Hukkelberg & Jensen, 2011; re-experiencing ˛ = ·84; avoidance ˛ = .80 and hyperarousal ˛ = .76). Cronbach’s alphas for the sub- and total scales in the current sample are presented in Table 1. The clinician-administered PTSD scale for children and adolescents (CAPS-CA). The CAPS-CA (Nader et al., 2004) is a semi-structured clinical interview that determines trauma exposure and assesses the frequency and intensity of the 17 DSM-IV-defined symptoms of PTSD for ages 8 and older. The present study addresses the symptom part of the CAPS-CA only, on which items are scored on 5-point frequency scales (e.g., from 0 = None of the time to 4 = Most of the time) and 5-point intensity rating scales (e.g., from 0 = Not a problem to 4 = A big problem, I have to stop what I am doing) for the past month. Items are scored based on children’s answers and clinical judgment during the interview. It is recommended that the “F1/I2 Rule” is used to determine a diagnosis; that is, a frequency score of 1 and an intensity score of 2 are required for a particular symptom to meet the criterion (Weathers et al., 2004; Weathers, Ruscio, & Keane, 1999). A severity score for each symptom is calculated by summing the frequency and intensity scores, which can then be summed for all 17 symptoms questions and/or for the three symptom clusters. The interviews were administered and scored by two equally experienced clinical psychologists and took approximately 40–90 min to administer, depending on the level of psychopathology. Cronbach’s alphas for the sub- and total scales are presented in Table 1. Inter-rater agreement was  = .60. Kappa between CPSS and CAPS-CA was .27.

S. Hukkelberg et al. / Journal of Anxiety Disorders 28 (2014) 51–56

2.3. Data analyses

53

Statistical Computing, Vienna, Austria), with the R package pROC for AUC comparisons.

First, we calculated how many of the participants would be diagnosed with PTSD using the CAPS-CA based on the DSM-IV algorithm (at least 1 symptom of the B criterion, 3 for the C criterion, and 2 for the D criterion). In addition, a total score above 45 was required. The CPSS is not a diagnostic instrument per se, but because it contains all 17 symptoms, we applied the DSM-IV algorithm regarding the number of B, C and D symptoms necessary to be present for a diagnosis to be fulfilled. Because the symptom items of the CPSS and CAPS-CA are measured on different scales (CPSS items are assessed on a 4-point scale, whereas CAPS-CA are assessed on a 5-point scale), we transformed the scale scores into similar rating scales (ranging from 0 to 100). These transformed variables were used as input data in the subsequent comparative and receiver operating characteristic analyses. To provide information about how well the three factors and the total score of CPSS were able to match the CAPS-CA, we conducted receiver operating characteristic (ROC) curve analyses. The area under the curve (AUC) was estimated for each comparison, to determine the discriminative ability of the CPSS to predict CAPSCA. AUC can be interpreted as an effect size, and represents the probability that a randomly selected subject diagnosed with PTSD according to the CAPS-CA has a higher CPSS score than a subject not diagnosed with PTSD. An area of 1.0 indicates the most accurate scale possible, whereas an area of 0.5 reflects accuracy no greater than chance. We tested whether the AUC is significantly different from 0.5. Also, we tested pairwise differences in AUCs for the CPSS subscale and total score by the DeLong procedure with Holm adjustment for multiple comparisons (DeLong, DeLong, & Clarke-Pearson, 1988). The efficacy of the instrument can be given by the area under the curve (AUC) of the receiver operating characteristic. Using standard conventions, we consider AUCs > .56 small effects, AUCs > 64 medium effects, and AUCs > .71 large effects (Rice & Harris, 2005). The number of respondents with PTSD who were diagnosed using the scale (i.e., sensitivity) and the number of those without these conditions who were correctly classified using the scale (i.e., specificity) are computed for each value of the scale. These values are then used to compute an ROC curve that plots sensitivity versus 1 − specificity. The coordinates of the ROC curves for CPSS are used to determine the respective cutoff values necessary to achieve high sensitivity. Ideal values of sensitivity and specificity are 1.0. Here, we find cut-points for achieving sensitivity of at least 0.80 and report the corresponding sensitivity, specificity and positive and negative predictive power, together with the level (Q) of the test (proportion of positive tests) and total efficiency (proportion of concordance between CPS and CAPS-CA). Statistical analyses were performed in SPSS version 18.0 (IBM SPSS Statistics, 2011) and R (The R Foundation for

3. Results 3.1. Descriptives One hundred and thirty-two subjects (84.6%) were diagnosed with PTSD using the CPSS, whereas 97 subjects (62.2%) were diagnosed with PTSD using the CAPS-CA (based on the DSM-IV algorithm for symptoms and a total score above 45). The intertest agreement between CPSS and CAPS-CA, in terms of kappa, was low ( = .27). Table 1 shows correlations, means (SD) and reliability coefficients for the sub-scales and the total scale for PTSS assessed by the CPSS and CAPS-CA. The correlation between same factors, as measured by the CPSS and CAPS-CA, was moderate: r = .55 for reexperiencing, r = .47 for avoidance, and r = .65 for hyperarousal. The total score of PTSS as measured by the CAPS-CA was significantly correlated with the total score as measured by the CPSS (r = .63, p < .001). For the CPSS, mean scores for re-experiencing and avoidance were both around the mid-point, whereas the mean score for hyperarousal was somewhat higher. For CAPS-CA, Re-experiencing and Avoidance were both below the mid-point, whereas Hyperarousal was above the midpoint. Overall, the factors showed poor to good reliability coefficients for the sub-scales (˛ = .51–.79). 3.2. Diagnostic utility statistics Fig. 1 and Table 2 show the results of the ROC analyses for the CPSS against the CAPS-CA. Fig. 1 displays the ROC curves for the three subscales (re-experiencing, avoidance and hyperarousal) the total scale, and the predicted curve for the CPSS compared to the CAPS-CA diagnosis of PTSD. As shown, both the total sum score of the CPSS and the Hyperarousal factor performed quite well. As shown in Table 2, the AUC were from .63 (re-experiencing) to.75 (total scale) compared to .76 for the predicted probability curve. AUCs (excluding the predicted probability curve) were significantly different between re-experiencing and the total scale (unadjusted p = .001, adjusted p = .008), all other pairwise AUC comparisons were insignificant (unadjusted p ≥ .085, adjusted p ≥ .443). The latter curve represents a combined curve where the three factors are weighted differently and, as such, is an “optimal” representation of the CPSS. All values were significantly different from 0.5 at p < .001 except for the re-experiencing factor, which was significant at p = .011. To determine possible cutoff scores for each scale, we tried to maximize specificity while keeping sensitivity ≥.80. Based on this criterion, the empirical cut-points were 42.16 (based on the

Table 1 Correlations, means (SD) and alphas for CPSS and CAPS-CA factors and sum scores. 1 1. CPSS re-experiencing 2. CPSS avoidance 3. CPSS  hyperarousal CPSS 4. 5. CAPS re-experiencing 6. CAPS avoidance 7. CAPS  hyperarousal CAPS 8. Mean SD Alpha

2

3

4

5

6

7

8

– .36 .44 .74 .55 .29 .32 .48

– .40 .81 .27 .47 .31 .44

– .76 .43 .36 .65 .58

– .52 .49 .53 .63

– .51 .52 .82

– .48 .83

– .81

50.64 20.00 .67

48.17 19.06 .58

62.86 19.47 .51

53.22 15.09 .75

42.06 22.04 .79

38.99 17.55 .58

57.32 21.41 .61

– 45.37 16.43 .83

All correlations were significant at the p < 0.01 level (2-tailed). Means and standard deviations are based on transformed scales, range 0–100. Note: CAPS was the CAPS-CA version.

54

S. Hukkelberg et al. / Journal of Anxiety Disorders 28 (2014) 51–56

Table 2 Diagnostic utility for CPSS against CAPS-CA.

Re-experiencing Avoidance Hyperarousal CPSS Pred. probability

AUC

p

95% CI

Cut-pointc

Cut-pointd

Sensitivity

Specificity

Level (Q)a

Total efficiencyb

.63 (.74) .73 (.78) .71 (.77) .75 (.79) .76

.007