Original Paper Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
Received: July 3, 2013 Accepted after revision: March 19, 2014 Published online: July 3, 2014
Reliability and Validity of Teacher-Rated Symptoms of Oppositional Defiant Disorder and Conduct Disorder in a Clinical Sample Elena Ise a Anja Görtz-Dorten a, b Manfred Döpfner a, b a b
Department of Child and Adolescent Psychiatry and Psychotherapy, Medical Faculty, University of Cologne, and Institute for Child Psychotherapy of the Christoph Dornier Foundation, Cologne, Germany
Key Words Attention deficit and disruptive behavior disorders · Conduct disorder · Symptom assessment · Factor analysis, statistical · Reproducibility of results
Abstract Background/Aims: It is recommended to use information from multiple informants when making diagnostic decisions concerning oppositional defiant disorder (ODD) and conduct disorder (CD). The purpose of this study was to investigate the reliability and validity of teacher-rated symptoms of ODD and CD in a clinical sample. Methods: The sample comprised 421 children (84% boys; 6–17 years) diagnosed with ODD, CD, and/or attention deficit hyperactivity disorder (ADHD). Teachers completed a standardized ODD/CD symptom rating scale and the Teacher Report Form (TRF). Results: The reliability (internal consistency) of the symptom rating scale was high (α = 0.90). Convergent and divergent validity were demonstrated by substantial correlations with similar TRF syndrome scales and low-to-moderate correlations with dissimilar TRF scales. Discriminant validity was shown by the ability of the symptom rating scale to differentiate between children with ODD/CD and those with ADHD. Factorial validity was demonstrated by principal component analysis,
© 2014 S. Karger AG, Basel 0254–4962/14/0475–0312$39.50/0 E-Mail [email protected] www.karger.com/psp
which produced a two-factor solution that is largely consistent with the two-dimensional model of ODD and CD proposed by the Diagnostic and Statistical Manual of Mental Disorders (DSM)-IV-TR, although some CD symptoms representing aggressive behavior loaded on the ODD dimension. Conclusion: These findings suggest that DSM-IV-TR-based teacher rating scales are useful instruments for assessing disruptive behavior problems in children and adolescents. © 2014 S. Karger AG, Basel
Rating scales are widely used tools for the diagnosis of disruptive behavior disorders in children and adolescents. Many of these are directly linked to the diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders (DSM) [1] or the ICD-10 Classification of Mental and Behavioural Disorders [2] and ask the respondent to rate the presence and intensity of symptoms of oppositional defiant disorder (ODD) and conduct disorder (CD). Although parent-completed symptom rating scales are reliable and valid measures of disruptive behav-
E.I. and A.G.-D. contributed equally to this study and should both be regarded as first authors.
Manfred Döpfner Department of Child and Adolescent Psychiatry and Psychotherapy University of Cologne, Robert-Koch-Str. 10 DE–50931 Cologne (Germany) E-Mail manfred.doepfner @ uk-koeln.de
ior [3, 4], several authors recommended to use additional information from teachers when making diagnostic decisions concerning ODD/CD [5]. It has, for example, been demonstrated that combined parent and teacher ratings yield a more accurate prediction of children’s diagnostic status (presence vs. absence of ODD/CD) than parent ratings alone [6]. This might at least in part be due to the fact that parents and teachers often disagree about children’s disruptive behavior [7, 8]. These discrepancies reflect not only rater bias but also meaningful differences in children’s behavior across contexts [9], and there is some evidence that children who show disruptive behavior in more than one context have more severe behavior problems [7, 10]. The psychometric properties of ODD and CD subscales of DSM-based teacher rating scales have been shown to be good. For example, the ODD and CD symptom categories of the teacher version of the Child Symptom Inventory-4 (CSI-4 [11]) are internally consistent (α = 0.78–0.94), show convergent and divergent validity with corresponding scales of the Teacher Report Form (TRF [12]), and discriminate between children diagnosed with ODD/CD and those without ODD/CD [13, 14]. Similar results have been reported for the oppositional subscale of the Swanson, Nolan, and Pelham IV Scale teacher version (SNAP-IV [15–17]). Using confirmatory factor analysis, several studies found that teacher-rated symptoms of ODD and CD form two distinct factors [18, 19]. An advantage of exploratory factor analysis (EFA) is its ability to detect deviations from the two-dimensional structure proposed by the DSM-IV-TR and ICD-10. For example, Rowe et al. [20] conducted EFA of symptoms of ODD and identified irritable and headstrong subdimensions within these symptoms. There is also evidence from EFA that CD symptoms can be distinguished in aggressive and rulebreaking behaviors [21]. Only few studies have used EFA to simultaneously examine the structure of teacher-reported symptoms of ODD and CD. EFA conducted on teacher ratings of children attending regular classrooms [22] or special education [23] revealed that DSM-III-R-defined ODD symptoms load on a single factor, but CD symptoms either load on the ODD factor or on a separate CD factor. A similar finding was reported for teachers’ responses to structured interviews based on the DSM-III-R in a clinical sample [24]. One CD symptom was consistently associated with ODD symptoms in all three studies (‘initiates physical fights’). Other symptoms representing open aggressive behavior toward others (e.g., ‘bullies’ and ‘is physically
cruel to people’) loaded on the ODD factor in some, but not all, studies. The FBB-SSV (German: Fremdbeurteilungsbogen für Störungen des Sozialverhaltens) [25] is a standardized measure of disruptive behavior that is frequently used in clinical and research settings in Germany [26, 27]. The rating scale has a proxy rating form that can be completed by parents or teachers, and a self-report form that can be completed by patients aged 11 years or older. The 9-item ODD subscale assesses symptoms of ODD including irritability, disobedience, and defiance toward authority figures. The 16-item CD subscale assesses behavior that violates the basic rights of others or societal norms (e.g., fighting, bullying, stealing, and vandalism). Previous research [28] suggests that the parent-completed FBB-SSV is reliable (internally consistent) and has a twofactor structure that supports the distinction between ODD and CD proposed by the DSM-IV-TR and ICD-10. The psychometric properties of FBB-SSV teacher ratings have not yet been evaluated. This study examines the reliability and validity of the teacher-rated version of the FBB-SSV in a clinical sample of children and adolescents diagnosed with disruptive behavior disorders including ODD, CD, and attention deficit hyperactivity disorder (ADHD). Three tests of validity were employed. Convergent and divergent validity were assessed by examining correlations with other constructs. Discriminant validity was evaluated by comparing the scale scores of children with ODD/CD with those of children with ADHD. Factorial validity was tested using EFA. A strength of the present study is that the sample (n = 421) was substantially larger than the clinical samples used in previous studies [13, 17]. Another strength is that this study is the first to examine the factor structure of teacher-completed rating scales assessing symptoms of ODD and CD in children diagnosed with ODD, CD, and/or ADHD using EFA (principal component analysis, PCA). Based on previous findings [24], it is expected that PCA will identify two components, one reflecting ODD and the more aggressive CD symptoms, and one comprising nonaggressive (rule-breaking) CD symptoms.
Teacher Ratings of ODD and CD
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
Materials and Methods Sample and Procedure All children were referred to the child psychiatry outpatient unit of the University Hospital in Cologne, Germany. During the intake interview, the parents were informed about the study’s purpose and provided their informed consent. They completed a series of questionnaires, including the FBB-SSV, and were asked to
313
pass the FBB-SSV and the TRF to their child’s teacher. Further diagnostic sessions were conducted as necessary on an individual basis. The diagnostic status of each child was determined by a child psychologist or psychiatrist within 8 weeks after referral based on a semi-structured interview using the Diagnostic Checklist for ODD/CD (DCL-SSV) and the Diagnostic Checklist for ADHD (DCL-ADHS) [25] according to the diagnostic criteria of the DSM-IV-TR and ICD-10. All interviewers were trained in the use of the interview schedule by A.G.-D. The training was done individually or in small groups. The DCL have been shown to be reliable instruments that provide a categorical assessment of mental disorders in children and adolescents [29]. Children were selected for the current analyses if (a) complete datasets of TRF and teacher-completed FBB-SSV were available, and (b) if they had a diagnosis of ODD/CD and/or ADHD. The sample consisted of 421 school children and adolescents (84% boys) who ranged in age from 6 to 17 years (mean = 10.1, SD = 2.4). Ninety-four children had a diagnosis of ODD/CD (ICD10 codes F91 ‘conduct disorders’ and F92 ‘mixed disorders of conduct and emotions’), 191 had a diagnosis of hyperkinetic conduct disorder (F90.1), and 136 had a diagnosis of ADHD (F90 ‘hyperkinetic disorder’ except F90.1). The Ethics Committee at the University of Cologne, Germany, approved all procedures. The FBB-SSV The FBB-SSV assesses the occurrence of symptoms of ODD and CD. The scale is part of a diagnostic system for the assessment of mental disorders in children and adolescents [25] based on the DSM-IV-TR and ICD-10. There are two versions of the FBB-SSV, one for caretakers (parents and teachers) and one for children aged 11 years and older. Both versions consist of 25 items that are rated on a 4-point Likert scale ranging from 0 (‘not at all’) to 3 (‘very much’). The 9-item subscale ODD evaluates the 9 symptom criteria for ODD (e.g., ‘often argues with adults’). The 16 items of the subscale CD correspond to the symptom criteria for CD (e.g., ‘has broken into someone else’s house, building, or car’). Respondents are instructed to choose 0 (‘not at all’) if they do not know whether the behavior has occurred. All FBB-SSV questionnaires with more than 10% missing items were excluded from the analyses. For questionnaires with less than 10% missing items, each missing item was given a value of 0 (‘not at all’).
sistency [32]. Convergent and divergent validity were evaluated by calculating bivariate correlations (Pearson) between the FBB-SSV scales and the TRF scales. To investigate whether the FBB-SSV can differentiate between ODD/CD and ADHD, ANOVA were conducted with the FBBSSV total and subscale scores as dependent variables and ‘diagnostic group’ as the between-subjects variable. Children with an ICD10 diagnosis of ODD or CD (F91 and F92) were allocated to the ‘ODD/CD’ group (n = 94). Children with hyperkinetic conduct disorder (F90.1) were allocated to the ‘ODD/CD + ADHD’ group (n = 191). Those with ADHD (F90 except F90.1) were allocated to the ‘ADHD’ group (n = 136). PCA with promax rotation were performed to test the validity of the structure of the FBB-SSV. Promax rotation was chosen because it is an oblique rotation method that does not require factors to be uncorrelated. Following Achenbach’s criterion [33], items with low variance and high skewness (i.e., items coded 0 in more than 95% of the cases) were excluded from the PCA (items 13, 16, 18–22, 24, and 25). Two solutions were computed. First, a forced two-factor solution was computed because this is the predicted number of factors on theoretical and empirical grounds. Second, an unforced solution was computed using the ‘eigenvalue greater than 1.0’ rule to select factors for rotation. All analyses were performed with SPSS (IBM SPSS Statistics for Windows version 21.0; IBM Deutschland GmbH, Ehningen, Germany).
Results
The TRF The TRF 6-18 [12] (German version by Döpfner et al. [30]) measures teacher-reported behavior and emotional problems in school-aged children and adolescents. The items describe typical behavior and emotional problems and are rated 0 (‘not true’), 1 (‘somewhat or sometimes true’), or 2 (‘very true or often true’). It contains 113 items and yields 8 syndrome scales: withdrawn, somatic complaints, anxious/depressed, social problems, thought problems, attention problems, rule-breaking behavior, and aggressive behavior. There are 2 broad-band syndromes: externalizing and internalizing problems. The German version has good reliability and validity [30, 31].
Reliability The estimates of internal consistency were high for the FBB-SSV total scale (α = 0.90) and ODD subscale (α = 0.93). The internal consistency of the CD subscale (α = 0.672) was below the recommended minimum value of 0.70 [32]. Excluding items with low variance and high skewness reduced the number of items on the CD scale (retained items: 10–12, 14, 15, 17, and 23). The internal consistency of the reduced CD scale (α = 0.665) was comparable to that of the full CD scale. Next, an item analysis was conducted separately for each subscale. The corrected item-total correlations for the 9 ODD items ranged from 0.63 to 0.83. Consequently, all ODD items were retained. The corrected item-total correlations were below the recommended threshold of 0.3 [32] for 9 CD items (items 10, 13, 16, 17, 19, 21, 22, 24, and 25). Removing these items did not substantially improve the internal consistency of the CD subscale (α = 0.71). Because each CD item represents an important aspect of conduct behavior, all CD items were retained.
Data Analysis The internal consistencies of the FBB-SSV total scale and the two subscales (ODD and CD) were determined by computing Cronbach’s alpha. An alpha coefficient of 0.70 is generally assumed to be the acceptable minimum value for good internal con-
Convergent and Divergent Validity The bivariate correlation (Pearson) between the two subscales was r = 0.67 (p < 0.001). Table 1 shows the bivariate correlations between the FBB-SSV scales and the
314
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
Ise/Görtz-Dorten/Döpfner
Table 1. Correlations between FBB-SSV scales rated by teachers and TRF scales (n = 421)
Table 3. Factor loadings for the FBB-SSV teacher ratings (promaxrotated two-factor solution; loadings >0.25; n = 421) Item
FBB-SSV, r Total scale TRF broad-band scales Internalizing Externalizing TRF syndrome scales Withdrawn Somatic complaints Anxious/depressed Social problems Thought problems Attention problems Rule-breaking behavior Aggressive behavior
ODD scale
CD scale
0.35** 0.79**
0.36** 0.78**
0.22** 0.61**
0.15** 0.13* 0.41** 0.49** 0.28** 0.29** 0.63** 0.79**
0.14** 0.13* 0.43** 0.48** 0.27** 0.26** 0.58** 0.79**
0.12* 0.10* 0.26** 0.38** 0.24** 0.28** 0.60** 0.60**
* p < 0.05; ** p < 0.001.
Table 2. Means ± SD of the FBB-SSV total and subscale scores for the three diagnostic groups and the total sample
ODD/CD (n = 94) ODD/CD + ADHD (n = 191) ADHD (n = 136) Total (n = 421)
Total scale
ODD scale
CD scale
18.3±10.0 14.7±9.5 8.0±7.9 13.3±9.9
14.0±7.4 11.4±7.1 6.5±6.3 10.4±7.5
4.2±3.8 3.3±3.1 1.6±2.2 3.0±3.2
TRF scales. As expected, the FBB-SSV scales correlated more highly with the TRF externalizing scale than with the TRF internalizing scale. The ODD subscale correlated most strongly with the narrow-band syndrome scale ‘aggressive behavior’ (r = 0.79). The CD subscale correlated highly with the scales ‘aggressive behavior’ (r = 0.60) and ‘rule-breaking behavior’ (r = 0.60). The correlations between the FBB-SSV subscales and dissimilar TRF syndrome scales (including the ‘attention problems’ scale) were low. Moderate correlations were found between the FBB-SSV scales and the ‘social problems’ scale. Discriminant Validity Table 2 represents the mean scores ± SD of the FBBSSV teacher ratings for each of the three diagnostic groups. ANOVA revealed a significant effect of diagnosTeacher Ratings of ODD and CD
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Loses temper Gets angry with others Argues with adults Refuses to obey adults Annoys people Blames others for his or her mistakes Touchy or easily annoyed by others Is angry and resentful Is spiteful or vindictive Starts fights with siblings Starts fights with other children Bullies, threatens, or intimidates others Is physically cruel to animals1 Lies Steals secretly Stays out at night1 Skips school Has used weapons1 Has been physically cruel to others1 Has stolen while confronting a victim1 Has forced someone into sexual activity1 Has engaged in fire setting1 Has destroyed others’ property Has broken into someone’s house, building, or car1 25 Ran away from home1 Eigenvalue
Factor 1 ‘ODD’
0.95 1.00 0.76 0.68 0.53 0.67 0.93 0.98 0.51 0.65 0.50
Factor 2 ‘CD’
–0.26 0.37
0.35 0.38 0.27 0.40 0.72 0.70
–
–
0.59
7.03
3.84
1
Item was not included in the analysis due to low variance and high skewness.
tic group on the FBB-SSV total scale [F(2, 418) = 38.55; p < 0.001], the ODD subscale [F(2, 418) = 37.00; p < 0.001], and the CD subscale [F(2, 418) = 23.39; p < 0.001]. Post hoc tests (Tukey) demonstrated that the children with ODD/CD had significantly higher scores than the children with ADHD (all scales: p < 0.001) or ODD/CD + ADHD (total scale: p = 0.005; ODD subscale: p = 0.006; CD subscale: p = 0.044). The children with ODD/CD + ADHD had significantly higher scores than the children with ADHD (all scales: p < 0.001). Factor Structure The two-factor solution explained 54.6% of the total variance. Table 3 shows the promax-rotated pattern matrix loadings. All 9 ODD items (items 1–9) had their highest loadings on the first factor, labelled ‘ODD’ (loadings: Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
315
0.51–1.00). Seven CD items were included in the analysis. One CD item (item 17) did not load above 0.25 on any factor. The remaining 6 CD items all had substantial loadings on the second factor, labelled ‘CD’ (loadings: 0.27– 0.72), although 2 items had their highest loading on the first factor (items 11 and 12). The subsequent unforced solution generated 4 factors with eigenvalues greater than 1.00 which accounted for 68.7% of the variance. After promax rotation, the ODD items had substantial loadings either on the first factor (items 1–4, 7, and 8; loadings: 0.57–0.98) or on the second factor (items 5, 6, and 9; loadings: 0.54–0.83). The CD items had their highest loading either on the second factor (items 11, 12, and 14; loadings: 0.68–0.82), on the third factor (items 10, 15, and 23; loadings: 0.46–0.76), or on the fourth factor (item 17; loading: 0.96).
Discussion
Because children’s disruptive behavior differs across contexts, teacher ratings are an essential component of the diagnostic process [5]. Broad-range behavior rating scales such as the TRF [12] are commonly used. Symptom rating scales have the advantage of being not only shorter and quicker to complete but also directly linked to the diagnostic criteria for ODD and CD. The present study provides evidence for the reliability and validity of a teacher-completed ODD/CD symptom rating scale (the FBB-SSV) in a German clinical sample. The internal consistency estimates of reliability were high for the FBB-SSV total scale and the ODD subscale (Cronbach’s α = 0.90 and 0.93, respectively), but not satisfactory for the CD subscale (α = 0.67). The obtained alpha values are comparable to those reported previously for ODD and CD symptom categories of teacher-completed DSM-based rating scales [13, 14, 16]. A number of items assessing CD symptoms were found to have poor corrected item-total correlations. Removing these items slightly improved the internal consistency of the CD scale (α = 0.71). Because each CD item represents an important aspect of conduct behavior, all items should be retained if the rating scale is used for clinical purposes. A short (7-item) version of the CD scale might be used in research settings. The convergent validity of teacher ratings was demonstrated by substantial correlations between the FBB-SSV scales and TRF syndrome scales assessing similar constructs (externalizing scale, rule-breaking behavior, and aggressive behavior). Divergent validity was confirmed by low-to-moderate correlations with dissimilar TRF 316
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
scales. Comparable correlations between teacher ratings of ODD/CD symptoms and TRF scale scores have been reported in an US sample [13]. In addition, the discriminant validity of teacher ratings was confirmed by significant differences in the FBB-SSV scale scores between the diagnostic groups. On both subscales, the teachers gave significantly higher ratings to children with ODD/CD (with or without comorbid ADHD) than to children with ADHD alone. Finally, factorial validity was examined by PCA. A promax-rotated two-factor solution revealed a two-dimensional structure that largely confirms the a priori defined ODD and CD subscales and supports the two-dimensional model of ODD and CD suggested by the DSM-IV-TR and ICD-10. As expected, all ODD symptoms loaded on the same factor, while most CD symptoms loaded on a separate CD factor. Also in line with our expectation, 2 CD symptoms representing aggressive behavior had their highest loading on the ODD factor (‘starts fights with other children’ and ‘bullies, threatens, or intimidates others’). These 2 symptoms have previously been found to load on a factor composed of ODD symptoms when teachers’ responses to structured interviews were analyzed in a clinical sample [24]. The association between CD symptoms representing open aggressive behavior and ODD symptoms might at least partly explain the high rates of comorbidity between ODD and CD [34]. A major limitation of the present study is that the FBBSSV was part of the standard clinical intake assessment. As a consequence, the clinical diagnoses are not fully independent of the FBB-SSV and it is possible that the present results overestimate the discriminant validity of the FBB-SSV teacher rating. However, it is important to note that all children in the sample were diagnosed with ODD/ CD and/or ADHD. These disorders have a high rate of comorbidity [34], and bidirectional halo effects in ratings of ADHD and ODD have been observed [35]. It is therefore likely that estimates of discriminant validity obtained from more heterogeneous samples will be higher than those observed in the present study. Future work may clarify this issue by evaluating the discriminant validity of the FBB-SSV in different samples and with a reference standard that is independent of the FBB-SSV. It should also be highlighted that information obtained from rating scales is not sufficient for diagnostic purposes [36] and that additional information should be gathered for clinical evaluations. Despite its limitations, the present study supports previous evidence for the reliability and validity of teacherIse/Görtz-Dorten/Döpfner
completed ODD/CD symptom rating scales. In addition, the study provides initial evidence for the factorial validity of teacher ratings of DSM-IV-TR-defined symptoms of ODD and CD in clinical populations. Since similar findings were reported for parent-completed symptom ratings [3, 4, 25, 28], it can be concluded that DSM-based rating scales are useful instruments for assessing disruptive behavior problems in children and adolescents across different contexts.
Acknowledgements The authors wish to thank all families that participated in this study.
Disclosure Statement A.G.-D. and M.D. are authors of the symptom rating scale for ODD and CD (FBB-SSV). They receive royalties from the publisher.
References 1 American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, ed 4, rev. Washington, American Psychiatric Association, 2000. 2 World Health Organization: The ICD-10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research. Geneva, World Health Organization, 1994. 3 Hommersen P, Murray C, Ohan JL, Johnston C: Oppositional defiant disorder rating scale: preliminary evidence of reliability and validity. J Emot Behav Disord 2006;14:118–125. 4 Sprafkin J, Gadow KD, Salisbury H, Schneider J, Loney J: Further evidence of reliability and validity of the Child Symptom Inventory-4: parent checklist in clinically referred boys. J Clin Child Adolesc Psychol 2002; 31: 513–524. 5 Dirks MA, de los Reyes A, Briggs-Gowan M, Cella D, Wakschlag LS: Annual research review: embracing not erasing contextual variability in children’s behavior – theory and utility in the selection and use of methods and informants in developmental psychopathology. J Child Psychol Psychiatry 2012;3:1–17. 6 Becker A, Hagenberg N, Roessner V, Woerner W, Rothenberger A: Evaluation of the selfreported SDQ in a clinical setting: do self-reports tell us more than ratings by adult informants? Eur Child Adolesc Psychiatry 2004; 13(suppl 2):II17–II24. 7 Munkvold L, Lundervold A, Lie SA, Manger T: Should there be separate parent- and teacher-based categories of ODD? Evidence from a general population. J Child Psychol Psychiatry 2009;50:1264–1272. 8 Salbach-Andrae H, Lenz K, Lehmkuhl U: Patterns of agreement among parent, teacher and youth ratings in a referred sample. Eur Psychiatry 2009;24:345–351. 9 Dumenci L, Achenbach TM, Windle M: Measuring context-specific and cross-contextual components of hierarchical constructs. J Psychopathol Behav Assess 2011;33:3–10. 10 Rettew DC, van Oort FVA, Verhulst FC, Buitelaar JK, Ormel J, Hartman C, Veenstra R, Althoff RR, Hudziak JJ: When parent and teacher ratings don’t agree: the Tracking Ado-
Teacher Ratings of ODD and CD
11 12 13
14
15
16
17
18
19
lescents’ Individual Lives Survey (TRAILS). J Child Adolesc Psychopharmacol 2011; 21: 389–397. Gadow KD, Sprafkin J: Child Symptom Inventory-4 Screening and Norms Manual. Stony Brook, Checkmate Plus, 2002. Achenbach TM, Rescorla LA: Manual for the ASEBA School-Age Forms and Profiles. Burlington, University of Vermont, 2001. Gadow KD, Sprafkin J, Salisbury H, Schneider J, Loney J: Further validity evidence for the teacher version of the Child Symptom Inventory-4. Sch Psychol Q 2004;19:50–71. Mattison RE, Gadow KD, Sprafkin J, Nolan EE, Schneider J: A DSM-IV-referenced teacher rating scale for use in clinical management. J Am Acad Child Adolesc Psychiatry 2003;42: 442–449. Swanson JM, Kraemer HC, Hinshaw SP, Arnold LE, Conners CK, Abikoff HB, et al: Clinical relevance of the primary findings of the MTA: success rates based on severity of ADHD and ODD symptoms at the end of treatment. J Am Acad Child Adolesc Psychiatry 2001;40:168–179. Bussing R, Fernandez M, Harwood M, Wei H, Garvan CW, Eyberg SM, Swanson JM: Parent and teacher SNAP-IV ratings of attention deficit hyperactivity disorder symptoms: psychometric properties and normative ratings from a school district sample. Assessment 2008;15: 317–328. Gau SS, Lin CH, Hu FC, Shang CY, Swanson JM, Liu YC, Liu SK: Psychometric properties of the Chinese version of the Swanson, Nolan, and Pelham, version IV scaleTeacher Form. J Pediatr Psychol 2009; 34: 850–861. Burns GL, Walsh JA, Owen SM, Snell J: Internal validity of attention deficit hyperactivity disorder, oppositional defiant disorder, and overt conduct disorder symptoms in young children: implications from teacher ratings for a dimensional approach to symptom validity. J Clin Child Adolesc Psychol 1997; 26: 266–275. Zuddas A, Marzocchi GM, Oosterlaan J, Cavolina P, Ancilletta B, Sergeant J: Factor structure and cultural factors of disruptive behav-
20
21
22
23
24
25
26
27 28
ior disorders symptoms in Italian children. Eur Psychiatry 2006;21:410–418. Rowe R, Costello EJ, Angold A, Copeland WE, Maughan B: Developmental pathways in oppositional defiant disorder and conduct disorder. J Abnorm Psychol 2010;119:726–738. Tackett JL, Krueger RF, Iacono WG, McGue M: Symptom-based subfactors of DSM-defined conduct disorder: evidence for etiologic distinctions. J Abnorm Psychol 2005; 114: 483–487. Pelham WE, Gnagy EM, Greenslade KE, Milich R: Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. J Am Acad Child Adolesc Psychiatry 1992;31:210–218. Pelham WE, Evans SW, Gnagy EM, Greenslade KE: Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders: prevalence, factor analyses, and conditional probabilities in a special education sample. Sch Psych Rev 1992;21:285–299. Frick PJ, Lahey BB, Loeber R, StouthamerLoeber M, Green S, Hart EL, Christ MAG: Oppositional defiant disorder and conduct disorder in boys: patterns of behavioral covariation. J Clin Child Psychol 1991; 20: 202– 208. Döpfner M, Görtz-Dorten A, Lehmkuhl G, Breuer D, Goletz H: Diagnostik-System für psychische Störungen nach ICD-10 und DSM-IV für Kinder und Jugendliche-II (DISYPS-II). Bern, Huber, 2008. Kierfeld F, Ise E, Hanisch C, Görtz-Dorten A, Döpfner M: Effectiveness of telephone-assisted parent-administered behavioural family intervention for preschool children with externalizing problem behaviour: a randomized controlled trial. Eur Child Adolesc Psychiatry 2013;22:553–565. Mayer K, Wyckoff SN, Strehl U: One size fits all? Slow cortical potentials neurofeedback – a review. J Atten Disord 2013;17:393–409. Görtz-Dorten A, Ise E, Hautmann C, Walter D, Döpfner M: Psychometric properties of a German parent rating scale for oppositional defiant and conduct disorder (FBB-SSV) in clinical and community samples. Child Psychiatry Hum Dev 2014;45:388–397.
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
317
29 Görtz-Dorten A, Döpfner M: DiagnoseChecklisten aus dem Diagnostik-System für Psychische Störungen im Kindes- und Jugendalter (DISYPS-II): Gütekriterien und klinische Anwendung. Klin Diagn Eval 2008; 1: 378–394. 30 Döpfner M, Plück J, Kinnen C; Arbeitsgruppe Deutsche Child Behavior Checklist: CBCL Handbuch-Schulalter: Manual zum Elternfragebogen über das Verhalten von Kindern und Jugendlichen (CBCL/618R), zum Lehrerfragebogen über das Verhalten von Kindern und Jugendlichen (TRF/6-18R) und zum Fragebogen für Jugendliche (YSR/11-18R). Göttingen, Hogrefe, 2013.
318
31 Döpfner M, Berner W, Schmeck K, Lehmkuhl G, Poustka F: Internal consistency and validity of the CBCL and the TRF in a German sample: a cross-cultural comparison; in Sergeant J (ed): Eunethydis: European Approaches to Hyperkinetic Disorder. Zurich, Fotorotar Egg, 1995, pp 51–81. 32 Nunnally JC, Bernstein IH: Psychometric Theory, ed 3. New York, McGraw-Hill, 1994. 33 Achenbach TM: Manual for the Child Behavior Checklist/4-18 and 1991 Profile. Burlington, University of Vermont, 1991.
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
34 Nock MK, Kazdin AE, Hiripi E, Kessler RC: Lifetime prevalence, correlates, and persistence of oppositional defiant disorder: results from the National Comorbidity Survey Replication. J Child Psychol Psychiatry 2007; 48: 703–713. 35 Hartung CM, Lefler EK, Tempel AB, Armendariz ML, Sigel BA, Little CS: Halo effects in ratings of ADHD and ODD: identification of susceptible symptoms. J Psychopathol Behav Assess 2010;32:128–137. 36 Myers K, Winters NC: Ten-year review of rating scales. I. Overview of scale functioning, psychometric properties, and selection. J Am Acad Child Adolesc Psychiatry 2002;41:114– 122.
Ise/Görtz-Dorten/Döpfner
Copyright: S. Karger AG, Basel 2014. Reproduced with the permission of S. Karger AG, Basel. Further reproduction or distribution (electronic or otherwise) is prohibited without permission from the copyright holder.
Received: July 3, 2013 Accepted after revision: March 19, 2014 Published online: July 3, 2014
Reliability and Validity of Teacher-Rated Symptoms of Oppositional Defiant Disorder and Conduct Disorder in a Clinical Sample Elena Ise a Anja Görtz-Dorten a, b Manfred Döpfner a, b a b
Department of Child and Adolescent Psychiatry and Psychotherapy, Medical Faculty, University of Cologne, and Institute for Child Psychotherapy of the Christoph Dornier Foundation, Cologne, Germany
Key Words Attention deficit and disruptive behavior disorders · Conduct disorder · Symptom assessment · Factor analysis, statistical · Reproducibility of results
Abstract Background/Aims: It is recommended to use information from multiple informants when making diagnostic decisions concerning oppositional defiant disorder (ODD) and conduct disorder (CD). The purpose of this study was to investigate the reliability and validity of teacher-rated symptoms of ODD and CD in a clinical sample. Methods: The sample comprised 421 children (84% boys; 6–17 years) diagnosed with ODD, CD, and/or attention deficit hyperactivity disorder (ADHD). Teachers completed a standardized ODD/CD symptom rating scale and the Teacher Report Form (TRF). Results: The reliability (internal consistency) of the symptom rating scale was high (α = 0.90). Convergent and divergent validity were demonstrated by substantial correlations with similar TRF syndrome scales and low-to-moderate correlations with dissimilar TRF scales. Discriminant validity was shown by the ability of the symptom rating scale to differentiate between children with ODD/CD and those with ADHD. Factorial validity was demonstrated by principal component analysis,
© 2014 S. Karger AG, Basel 0254–4962/14/0475–0312$39.50/0 E-Mail [email protected] www.karger.com/psp
which produced a two-factor solution that is largely consistent with the two-dimensional model of ODD and CD proposed by the Diagnostic and Statistical Manual of Mental Disorders (DSM)-IV-TR, although some CD symptoms representing aggressive behavior loaded on the ODD dimension. Conclusion: These findings suggest that DSM-IV-TR-based teacher rating scales are useful instruments for assessing disruptive behavior problems in children and adolescents. © 2014 S. Karger AG, Basel
Rating scales are widely used tools for the diagnosis of disruptive behavior disorders in children and adolescents. Many of these are directly linked to the diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders (DSM) [1] or the ICD-10 Classification of Mental and Behavioural Disorders [2] and ask the respondent to rate the presence and intensity of symptoms of oppositional defiant disorder (ODD) and conduct disorder (CD). Although parent-completed symptom rating scales are reliable and valid measures of disruptive behav-
E.I. and A.G.-D. contributed equally to this study and should both be regarded as first authors.
Manfred Döpfner Department of Child and Adolescent Psychiatry and Psychotherapy University of Cologne, Robert-Koch-Str. 10 DE–50931 Cologne (Germany) E-Mail manfred.doepfner @ uk-koeln.de
ior [3, 4], several authors recommended to use additional information from teachers when making diagnostic decisions concerning ODD/CD [5]. It has, for example, been demonstrated that combined parent and teacher ratings yield a more accurate prediction of children’s diagnostic status (presence vs. absence of ODD/CD) than parent ratings alone [6]. This might at least in part be due to the fact that parents and teachers often disagree about children’s disruptive behavior [7, 8]. These discrepancies reflect not only rater bias but also meaningful differences in children’s behavior across contexts [9], and there is some evidence that children who show disruptive behavior in more than one context have more severe behavior problems [7, 10]. The psychometric properties of ODD and CD subscales of DSM-based teacher rating scales have been shown to be good. For example, the ODD and CD symptom categories of the teacher version of the Child Symptom Inventory-4 (CSI-4 [11]) are internally consistent (α = 0.78–0.94), show convergent and divergent validity with corresponding scales of the Teacher Report Form (TRF [12]), and discriminate between children diagnosed with ODD/CD and those without ODD/CD [13, 14]. Similar results have been reported for the oppositional subscale of the Swanson, Nolan, and Pelham IV Scale teacher version (SNAP-IV [15–17]). Using confirmatory factor analysis, several studies found that teacher-rated symptoms of ODD and CD form two distinct factors [18, 19]. An advantage of exploratory factor analysis (EFA) is its ability to detect deviations from the two-dimensional structure proposed by the DSM-IV-TR and ICD-10. For example, Rowe et al. [20] conducted EFA of symptoms of ODD and identified irritable and headstrong subdimensions within these symptoms. There is also evidence from EFA that CD symptoms can be distinguished in aggressive and rulebreaking behaviors [21]. Only few studies have used EFA to simultaneously examine the structure of teacher-reported symptoms of ODD and CD. EFA conducted on teacher ratings of children attending regular classrooms [22] or special education [23] revealed that DSM-III-R-defined ODD symptoms load on a single factor, but CD symptoms either load on the ODD factor or on a separate CD factor. A similar finding was reported for teachers’ responses to structured interviews based on the DSM-III-R in a clinical sample [24]. One CD symptom was consistently associated with ODD symptoms in all three studies (‘initiates physical fights’). Other symptoms representing open aggressive behavior toward others (e.g., ‘bullies’ and ‘is physically
cruel to people’) loaded on the ODD factor in some, but not all, studies. The FBB-SSV (German: Fremdbeurteilungsbogen für Störungen des Sozialverhaltens) [25] is a standardized measure of disruptive behavior that is frequently used in clinical and research settings in Germany [26, 27]. The rating scale has a proxy rating form that can be completed by parents or teachers, and a self-report form that can be completed by patients aged 11 years or older. The 9-item ODD subscale assesses symptoms of ODD including irritability, disobedience, and defiance toward authority figures. The 16-item CD subscale assesses behavior that violates the basic rights of others or societal norms (e.g., fighting, bullying, stealing, and vandalism). Previous research [28] suggests that the parent-completed FBB-SSV is reliable (internally consistent) and has a twofactor structure that supports the distinction between ODD and CD proposed by the DSM-IV-TR and ICD-10. The psychometric properties of FBB-SSV teacher ratings have not yet been evaluated. This study examines the reliability and validity of the teacher-rated version of the FBB-SSV in a clinical sample of children and adolescents diagnosed with disruptive behavior disorders including ODD, CD, and attention deficit hyperactivity disorder (ADHD). Three tests of validity were employed. Convergent and divergent validity were assessed by examining correlations with other constructs. Discriminant validity was evaluated by comparing the scale scores of children with ODD/CD with those of children with ADHD. Factorial validity was tested using EFA. A strength of the present study is that the sample (n = 421) was substantially larger than the clinical samples used in previous studies [13, 17]. Another strength is that this study is the first to examine the factor structure of teacher-completed rating scales assessing symptoms of ODD and CD in children diagnosed with ODD, CD, and/or ADHD using EFA (principal component analysis, PCA). Based on previous findings [24], it is expected that PCA will identify two components, one reflecting ODD and the more aggressive CD symptoms, and one comprising nonaggressive (rule-breaking) CD symptoms.
Teacher Ratings of ODD and CD
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
Materials and Methods Sample and Procedure All children were referred to the child psychiatry outpatient unit of the University Hospital in Cologne, Germany. During the intake interview, the parents were informed about the study’s purpose and provided their informed consent. They completed a series of questionnaires, including the FBB-SSV, and were asked to
313
pass the FBB-SSV and the TRF to their child’s teacher. Further diagnostic sessions were conducted as necessary on an individual basis. The diagnostic status of each child was determined by a child psychologist or psychiatrist within 8 weeks after referral based on a semi-structured interview using the Diagnostic Checklist for ODD/CD (DCL-SSV) and the Diagnostic Checklist for ADHD (DCL-ADHS) [25] according to the diagnostic criteria of the DSM-IV-TR and ICD-10. All interviewers were trained in the use of the interview schedule by A.G.-D. The training was done individually or in small groups. The DCL have been shown to be reliable instruments that provide a categorical assessment of mental disorders in children and adolescents [29]. Children were selected for the current analyses if (a) complete datasets of TRF and teacher-completed FBB-SSV were available, and (b) if they had a diagnosis of ODD/CD and/or ADHD. The sample consisted of 421 school children and adolescents (84% boys) who ranged in age from 6 to 17 years (mean = 10.1, SD = 2.4). Ninety-four children had a diagnosis of ODD/CD (ICD10 codes F91 ‘conduct disorders’ and F92 ‘mixed disorders of conduct and emotions’), 191 had a diagnosis of hyperkinetic conduct disorder (F90.1), and 136 had a diagnosis of ADHD (F90 ‘hyperkinetic disorder’ except F90.1). The Ethics Committee at the University of Cologne, Germany, approved all procedures. The FBB-SSV The FBB-SSV assesses the occurrence of symptoms of ODD and CD. The scale is part of a diagnostic system for the assessment of mental disorders in children and adolescents [25] based on the DSM-IV-TR and ICD-10. There are two versions of the FBB-SSV, one for caretakers (parents and teachers) and one for children aged 11 years and older. Both versions consist of 25 items that are rated on a 4-point Likert scale ranging from 0 (‘not at all’) to 3 (‘very much’). The 9-item subscale ODD evaluates the 9 symptom criteria for ODD (e.g., ‘often argues with adults’). The 16 items of the subscale CD correspond to the symptom criteria for CD (e.g., ‘has broken into someone else’s house, building, or car’). Respondents are instructed to choose 0 (‘not at all’) if they do not know whether the behavior has occurred. All FBB-SSV questionnaires with more than 10% missing items were excluded from the analyses. For questionnaires with less than 10% missing items, each missing item was given a value of 0 (‘not at all’).
sistency [32]. Convergent and divergent validity were evaluated by calculating bivariate correlations (Pearson) between the FBB-SSV scales and the TRF scales. To investigate whether the FBB-SSV can differentiate between ODD/CD and ADHD, ANOVA were conducted with the FBBSSV total and subscale scores as dependent variables and ‘diagnostic group’ as the between-subjects variable. Children with an ICD10 diagnosis of ODD or CD (F91 and F92) were allocated to the ‘ODD/CD’ group (n = 94). Children with hyperkinetic conduct disorder (F90.1) were allocated to the ‘ODD/CD + ADHD’ group (n = 191). Those with ADHD (F90 except F90.1) were allocated to the ‘ADHD’ group (n = 136). PCA with promax rotation were performed to test the validity of the structure of the FBB-SSV. Promax rotation was chosen because it is an oblique rotation method that does not require factors to be uncorrelated. Following Achenbach’s criterion [33], items with low variance and high skewness (i.e., items coded 0 in more than 95% of the cases) were excluded from the PCA (items 13, 16, 18–22, 24, and 25). Two solutions were computed. First, a forced two-factor solution was computed because this is the predicted number of factors on theoretical and empirical grounds. Second, an unforced solution was computed using the ‘eigenvalue greater than 1.0’ rule to select factors for rotation. All analyses were performed with SPSS (IBM SPSS Statistics for Windows version 21.0; IBM Deutschland GmbH, Ehningen, Germany).
Results
The TRF The TRF 6-18 [12] (German version by Döpfner et al. [30]) measures teacher-reported behavior and emotional problems in school-aged children and adolescents. The items describe typical behavior and emotional problems and are rated 0 (‘not true’), 1 (‘somewhat or sometimes true’), or 2 (‘very true or often true’). It contains 113 items and yields 8 syndrome scales: withdrawn, somatic complaints, anxious/depressed, social problems, thought problems, attention problems, rule-breaking behavior, and aggressive behavior. There are 2 broad-band syndromes: externalizing and internalizing problems. The German version has good reliability and validity [30, 31].
Reliability The estimates of internal consistency were high for the FBB-SSV total scale (α = 0.90) and ODD subscale (α = 0.93). The internal consistency of the CD subscale (α = 0.672) was below the recommended minimum value of 0.70 [32]. Excluding items with low variance and high skewness reduced the number of items on the CD scale (retained items: 10–12, 14, 15, 17, and 23). The internal consistency of the reduced CD scale (α = 0.665) was comparable to that of the full CD scale. Next, an item analysis was conducted separately for each subscale. The corrected item-total correlations for the 9 ODD items ranged from 0.63 to 0.83. Consequently, all ODD items were retained. The corrected item-total correlations were below the recommended threshold of 0.3 [32] for 9 CD items (items 10, 13, 16, 17, 19, 21, 22, 24, and 25). Removing these items did not substantially improve the internal consistency of the CD subscale (α = 0.71). Because each CD item represents an important aspect of conduct behavior, all CD items were retained.
Data Analysis The internal consistencies of the FBB-SSV total scale and the two subscales (ODD and CD) were determined by computing Cronbach’s alpha. An alpha coefficient of 0.70 is generally assumed to be the acceptable minimum value for good internal con-
Convergent and Divergent Validity The bivariate correlation (Pearson) between the two subscales was r = 0.67 (p < 0.001). Table 1 shows the bivariate correlations between the FBB-SSV scales and the
314
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
Ise/Görtz-Dorten/Döpfner
Table 1. Correlations between FBB-SSV scales rated by teachers and TRF scales (n = 421)
Table 3. Factor loadings for the FBB-SSV teacher ratings (promaxrotated two-factor solution; loadings >0.25; n = 421) Item
FBB-SSV, r Total scale TRF broad-band scales Internalizing Externalizing TRF syndrome scales Withdrawn Somatic complaints Anxious/depressed Social problems Thought problems Attention problems Rule-breaking behavior Aggressive behavior
ODD scale
CD scale
0.35** 0.79**
0.36** 0.78**
0.22** 0.61**
0.15** 0.13* 0.41** 0.49** 0.28** 0.29** 0.63** 0.79**
0.14** 0.13* 0.43** 0.48** 0.27** 0.26** 0.58** 0.79**
0.12* 0.10* 0.26** 0.38** 0.24** 0.28** 0.60** 0.60**
* p < 0.05; ** p < 0.001.
Table 2. Means ± SD of the FBB-SSV total and subscale scores for the three diagnostic groups and the total sample
ODD/CD (n = 94) ODD/CD + ADHD (n = 191) ADHD (n = 136) Total (n = 421)
Total scale
ODD scale
CD scale
18.3±10.0 14.7±9.5 8.0±7.9 13.3±9.9
14.0±7.4 11.4±7.1 6.5±6.3 10.4±7.5
4.2±3.8 3.3±3.1 1.6±2.2 3.0±3.2
TRF scales. As expected, the FBB-SSV scales correlated more highly with the TRF externalizing scale than with the TRF internalizing scale. The ODD subscale correlated most strongly with the narrow-band syndrome scale ‘aggressive behavior’ (r = 0.79). The CD subscale correlated highly with the scales ‘aggressive behavior’ (r = 0.60) and ‘rule-breaking behavior’ (r = 0.60). The correlations between the FBB-SSV subscales and dissimilar TRF syndrome scales (including the ‘attention problems’ scale) were low. Moderate correlations were found between the FBB-SSV scales and the ‘social problems’ scale. Discriminant Validity Table 2 represents the mean scores ± SD of the FBBSSV teacher ratings for each of the three diagnostic groups. ANOVA revealed a significant effect of diagnosTeacher Ratings of ODD and CD
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Loses temper Gets angry with others Argues with adults Refuses to obey adults Annoys people Blames others for his or her mistakes Touchy or easily annoyed by others Is angry and resentful Is spiteful or vindictive Starts fights with siblings Starts fights with other children Bullies, threatens, or intimidates others Is physically cruel to animals1 Lies Steals secretly Stays out at night1 Skips school Has used weapons1 Has been physically cruel to others1 Has stolen while confronting a victim1 Has forced someone into sexual activity1 Has engaged in fire setting1 Has destroyed others’ property Has broken into someone’s house, building, or car1 25 Ran away from home1 Eigenvalue
Factor 1 ‘ODD’
0.95 1.00 0.76 0.68 0.53 0.67 0.93 0.98 0.51 0.65 0.50
Factor 2 ‘CD’
–0.26 0.37
0.35 0.38 0.27 0.40 0.72 0.70
–
–
0.59
7.03
3.84
1
Item was not included in the analysis due to low variance and high skewness.
tic group on the FBB-SSV total scale [F(2, 418) = 38.55; p < 0.001], the ODD subscale [F(2, 418) = 37.00; p < 0.001], and the CD subscale [F(2, 418) = 23.39; p < 0.001]. Post hoc tests (Tukey) demonstrated that the children with ODD/CD had significantly higher scores than the children with ADHD (all scales: p < 0.001) or ODD/CD + ADHD (total scale: p = 0.005; ODD subscale: p = 0.006; CD subscale: p = 0.044). The children with ODD/CD + ADHD had significantly higher scores than the children with ADHD (all scales: p < 0.001). Factor Structure The two-factor solution explained 54.6% of the total variance. Table 3 shows the promax-rotated pattern matrix loadings. All 9 ODD items (items 1–9) had their highest loadings on the first factor, labelled ‘ODD’ (loadings: Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
315
0.51–1.00). Seven CD items were included in the analysis. One CD item (item 17) did not load above 0.25 on any factor. The remaining 6 CD items all had substantial loadings on the second factor, labelled ‘CD’ (loadings: 0.27– 0.72), although 2 items had their highest loading on the first factor (items 11 and 12). The subsequent unforced solution generated 4 factors with eigenvalues greater than 1.00 which accounted for 68.7% of the variance. After promax rotation, the ODD items had substantial loadings either on the first factor (items 1–4, 7, and 8; loadings: 0.57–0.98) or on the second factor (items 5, 6, and 9; loadings: 0.54–0.83). The CD items had their highest loading either on the second factor (items 11, 12, and 14; loadings: 0.68–0.82), on the third factor (items 10, 15, and 23; loadings: 0.46–0.76), or on the fourth factor (item 17; loading: 0.96).
Discussion
Because children’s disruptive behavior differs across contexts, teacher ratings are an essential component of the diagnostic process [5]. Broad-range behavior rating scales such as the TRF [12] are commonly used. Symptom rating scales have the advantage of being not only shorter and quicker to complete but also directly linked to the diagnostic criteria for ODD and CD. The present study provides evidence for the reliability and validity of a teacher-completed ODD/CD symptom rating scale (the FBB-SSV) in a German clinical sample. The internal consistency estimates of reliability were high for the FBB-SSV total scale and the ODD subscale (Cronbach’s α = 0.90 and 0.93, respectively), but not satisfactory for the CD subscale (α = 0.67). The obtained alpha values are comparable to those reported previously for ODD and CD symptom categories of teacher-completed DSM-based rating scales [13, 14, 16]. A number of items assessing CD symptoms were found to have poor corrected item-total correlations. Removing these items slightly improved the internal consistency of the CD scale (α = 0.71). Because each CD item represents an important aspect of conduct behavior, all items should be retained if the rating scale is used for clinical purposes. A short (7-item) version of the CD scale might be used in research settings. The convergent validity of teacher ratings was demonstrated by substantial correlations between the FBB-SSV scales and TRF syndrome scales assessing similar constructs (externalizing scale, rule-breaking behavior, and aggressive behavior). Divergent validity was confirmed by low-to-moderate correlations with dissimilar TRF 316
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
scales. Comparable correlations between teacher ratings of ODD/CD symptoms and TRF scale scores have been reported in an US sample [13]. In addition, the discriminant validity of teacher ratings was confirmed by significant differences in the FBB-SSV scale scores between the diagnostic groups. On both subscales, the teachers gave significantly higher ratings to children with ODD/CD (with or without comorbid ADHD) than to children with ADHD alone. Finally, factorial validity was examined by PCA. A promax-rotated two-factor solution revealed a two-dimensional structure that largely confirms the a priori defined ODD and CD subscales and supports the two-dimensional model of ODD and CD suggested by the DSM-IV-TR and ICD-10. As expected, all ODD symptoms loaded on the same factor, while most CD symptoms loaded on a separate CD factor. Also in line with our expectation, 2 CD symptoms representing aggressive behavior had their highest loading on the ODD factor (‘starts fights with other children’ and ‘bullies, threatens, or intimidates others’). These 2 symptoms have previously been found to load on a factor composed of ODD symptoms when teachers’ responses to structured interviews were analyzed in a clinical sample [24]. The association between CD symptoms representing open aggressive behavior and ODD symptoms might at least partly explain the high rates of comorbidity between ODD and CD [34]. A major limitation of the present study is that the FBBSSV was part of the standard clinical intake assessment. As a consequence, the clinical diagnoses are not fully independent of the FBB-SSV and it is possible that the present results overestimate the discriminant validity of the FBB-SSV teacher rating. However, it is important to note that all children in the sample were diagnosed with ODD/ CD and/or ADHD. These disorders have a high rate of comorbidity [34], and bidirectional halo effects in ratings of ADHD and ODD have been observed [35]. It is therefore likely that estimates of discriminant validity obtained from more heterogeneous samples will be higher than those observed in the present study. Future work may clarify this issue by evaluating the discriminant validity of the FBB-SSV in different samples and with a reference standard that is independent of the FBB-SSV. It should also be highlighted that information obtained from rating scales is not sufficient for diagnostic purposes [36] and that additional information should be gathered for clinical evaluations. Despite its limitations, the present study supports previous evidence for the reliability and validity of teacherIse/Görtz-Dorten/Döpfner
completed ODD/CD symptom rating scales. In addition, the study provides initial evidence for the factorial validity of teacher ratings of DSM-IV-TR-defined symptoms of ODD and CD in clinical populations. Since similar findings were reported for parent-completed symptom ratings [3, 4, 25, 28], it can be concluded that DSM-based rating scales are useful instruments for assessing disruptive behavior problems in children and adolescents across different contexts.
Acknowledgements The authors wish to thank all families that participated in this study.
Disclosure Statement A.G.-D. and M.D. are authors of the symptom rating scale for ODD and CD (FBB-SSV). They receive royalties from the publisher.
References 1 American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, ed 4, rev. Washington, American Psychiatric Association, 2000. 2 World Health Organization: The ICD-10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research. Geneva, World Health Organization, 1994. 3 Hommersen P, Murray C, Ohan JL, Johnston C: Oppositional defiant disorder rating scale: preliminary evidence of reliability and validity. J Emot Behav Disord 2006;14:118–125. 4 Sprafkin J, Gadow KD, Salisbury H, Schneider J, Loney J: Further evidence of reliability and validity of the Child Symptom Inventory-4: parent checklist in clinically referred boys. J Clin Child Adolesc Psychol 2002; 31: 513–524. 5 Dirks MA, de los Reyes A, Briggs-Gowan M, Cella D, Wakschlag LS: Annual research review: embracing not erasing contextual variability in children’s behavior – theory and utility in the selection and use of methods and informants in developmental psychopathology. J Child Psychol Psychiatry 2012;3:1–17. 6 Becker A, Hagenberg N, Roessner V, Woerner W, Rothenberger A: Evaluation of the selfreported SDQ in a clinical setting: do self-reports tell us more than ratings by adult informants? Eur Child Adolesc Psychiatry 2004; 13(suppl 2):II17–II24. 7 Munkvold L, Lundervold A, Lie SA, Manger T: Should there be separate parent- and teacher-based categories of ODD? Evidence from a general population. J Child Psychol Psychiatry 2009;50:1264–1272. 8 Salbach-Andrae H, Lenz K, Lehmkuhl U: Patterns of agreement among parent, teacher and youth ratings in a referred sample. Eur Psychiatry 2009;24:345–351. 9 Dumenci L, Achenbach TM, Windle M: Measuring context-specific and cross-contextual components of hierarchical constructs. J Psychopathol Behav Assess 2011;33:3–10. 10 Rettew DC, van Oort FVA, Verhulst FC, Buitelaar JK, Ormel J, Hartman C, Veenstra R, Althoff RR, Hudziak JJ: When parent and teacher ratings don’t agree: the Tracking Ado-
Teacher Ratings of ODD and CD
11 12 13
14
15
16
17
18
19
lescents’ Individual Lives Survey (TRAILS). J Child Adolesc Psychopharmacol 2011; 21: 389–397. Gadow KD, Sprafkin J: Child Symptom Inventory-4 Screening and Norms Manual. Stony Brook, Checkmate Plus, 2002. Achenbach TM, Rescorla LA: Manual for the ASEBA School-Age Forms and Profiles. Burlington, University of Vermont, 2001. Gadow KD, Sprafkin J, Salisbury H, Schneider J, Loney J: Further validity evidence for the teacher version of the Child Symptom Inventory-4. Sch Psychol Q 2004;19:50–71. Mattison RE, Gadow KD, Sprafkin J, Nolan EE, Schneider J: A DSM-IV-referenced teacher rating scale for use in clinical management. J Am Acad Child Adolesc Psychiatry 2003;42: 442–449. Swanson JM, Kraemer HC, Hinshaw SP, Arnold LE, Conners CK, Abikoff HB, et al: Clinical relevance of the primary findings of the MTA: success rates based on severity of ADHD and ODD symptoms at the end of treatment. J Am Acad Child Adolesc Psychiatry 2001;40:168–179. Bussing R, Fernandez M, Harwood M, Wei H, Garvan CW, Eyberg SM, Swanson JM: Parent and teacher SNAP-IV ratings of attention deficit hyperactivity disorder symptoms: psychometric properties and normative ratings from a school district sample. Assessment 2008;15: 317–328. Gau SS, Lin CH, Hu FC, Shang CY, Swanson JM, Liu YC, Liu SK: Psychometric properties of the Chinese version of the Swanson, Nolan, and Pelham, version IV scaleTeacher Form. J Pediatr Psychol 2009; 34: 850–861. Burns GL, Walsh JA, Owen SM, Snell J: Internal validity of attention deficit hyperactivity disorder, oppositional defiant disorder, and overt conduct disorder symptoms in young children: implications from teacher ratings for a dimensional approach to symptom validity. J Clin Child Adolesc Psychol 1997; 26: 266–275. Zuddas A, Marzocchi GM, Oosterlaan J, Cavolina P, Ancilletta B, Sergeant J: Factor structure and cultural factors of disruptive behav-
20
21
22
23
24
25
26
27 28
ior disorders symptoms in Italian children. Eur Psychiatry 2006;21:410–418. Rowe R, Costello EJ, Angold A, Copeland WE, Maughan B: Developmental pathways in oppositional defiant disorder and conduct disorder. J Abnorm Psychol 2010;119:726–738. Tackett JL, Krueger RF, Iacono WG, McGue M: Symptom-based subfactors of DSM-defined conduct disorder: evidence for etiologic distinctions. J Abnorm Psychol 2005; 114: 483–487. Pelham WE, Gnagy EM, Greenslade KE, Milich R: Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. J Am Acad Child Adolesc Psychiatry 1992;31:210–218. Pelham WE, Evans SW, Gnagy EM, Greenslade KE: Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders: prevalence, factor analyses, and conditional probabilities in a special education sample. Sch Psych Rev 1992;21:285–299. Frick PJ, Lahey BB, Loeber R, StouthamerLoeber M, Green S, Hart EL, Christ MAG: Oppositional defiant disorder and conduct disorder in boys: patterns of behavioral covariation. J Clin Child Psychol 1991; 20: 202– 208. Döpfner M, Görtz-Dorten A, Lehmkuhl G, Breuer D, Goletz H: Diagnostik-System für psychische Störungen nach ICD-10 und DSM-IV für Kinder und Jugendliche-II (DISYPS-II). Bern, Huber, 2008. Kierfeld F, Ise E, Hanisch C, Görtz-Dorten A, Döpfner M: Effectiveness of telephone-assisted parent-administered behavioural family intervention for preschool children with externalizing problem behaviour: a randomized controlled trial. Eur Child Adolesc Psychiatry 2013;22:553–565. Mayer K, Wyckoff SN, Strehl U: One size fits all? Slow cortical potentials neurofeedback – a review. J Atten Disord 2013;17:393–409. Görtz-Dorten A, Ise E, Hautmann C, Walter D, Döpfner M: Psychometric properties of a German parent rating scale for oppositional defiant and conduct disorder (FBB-SSV) in clinical and community samples. Child Psychiatry Hum Dev 2014;45:388–397.
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
317
29 Görtz-Dorten A, Döpfner M: DiagnoseChecklisten aus dem Diagnostik-System für Psychische Störungen im Kindes- und Jugendalter (DISYPS-II): Gütekriterien und klinische Anwendung. Klin Diagn Eval 2008; 1: 378–394. 30 Döpfner M, Plück J, Kinnen C; Arbeitsgruppe Deutsche Child Behavior Checklist: CBCL Handbuch-Schulalter: Manual zum Elternfragebogen über das Verhalten von Kindern und Jugendlichen (CBCL/618R), zum Lehrerfragebogen über das Verhalten von Kindern und Jugendlichen (TRF/6-18R) und zum Fragebogen für Jugendliche (YSR/11-18R). Göttingen, Hogrefe, 2013.
318
31 Döpfner M, Berner W, Schmeck K, Lehmkuhl G, Poustka F: Internal consistency and validity of the CBCL and the TRF in a German sample: a cross-cultural comparison; in Sergeant J (ed): Eunethydis: European Approaches to Hyperkinetic Disorder. Zurich, Fotorotar Egg, 1995, pp 51–81. 32 Nunnally JC, Bernstein IH: Psychometric Theory, ed 3. New York, McGraw-Hill, 1994. 33 Achenbach TM: Manual for the Child Behavior Checklist/4-18 and 1991 Profile. Burlington, University of Vermont, 1991.
Psychopathology 2014;47:312–318 DOI: 10.1159/000362373
34 Nock MK, Kazdin AE, Hiripi E, Kessler RC: Lifetime prevalence, correlates, and persistence of oppositional defiant disorder: results from the National Comorbidity Survey Replication. J Child Psychol Psychiatry 2007; 48: 703–713. 35 Hartung CM, Lefler EK, Tempel AB, Armendariz ML, Sigel BA, Little CS: Halo effects in ratings of ADHD and ODD: identification of susceptible symptoms. J Psychopathol Behav Assess 2010;32:128–137. 36 Myers K, Winters NC: Ten-year review of rating scales. I. Overview of scale functioning, psychometric properties, and selection. J Am Acad Child Adolesc Psychiatry 2002;41:114– 122.
Ise/Görtz-Dorten/Döpfner
Copyright: S. Karger AG, Basel 2014. Reproduced with the permission of S. Karger AG, Basel. Further reproduction or distribution (electronic or otherwise) is prohibited without permission from the copyright holder.
