The DSM·!!!·R Field Trial of Disruptive Behavior Disorders ROBERT L. SPITZER, M.D., MARK DAVIES, M.P.H.,
AND
RUSSELL A. BARKLEY, PH.D.
Abstract. The members of the DSM-Il/-R Advisory Committee responsible for the diagnostic criteria for the disruptive behavior disorders (attention deficit hyperactivity disorder, oppositional defiant disorder, and conduct disorder) were able to reach agreement on potential items to be included in the final diagnostic criteria. However, there was considerable disagreement about the relative utility of different items for the three disorders and no agreement on how many items should be required from a final list of discriminating items to establish each of the diagnoses. This article describes the method and results of a national field trial of the proposed criteria. Using as a standard the diagnosis of these disorders made by expert clinicians with experience with these disorders, the diagnostic criteria that were finally included in DSM-Il/-R demonstrated high sensitivity, specificity, and internal consistency. J. Am. Acad. ChildAdolesc. Psychiatry. 1990, 29, 5:690-697. Key Words: disruptive, attention deficit, oppositional, conduct, field trial.
The rationale for the development of the items for the diagnostic criteria for the disruptive behavior disorders, attention deficit hyperactivity disorder (ADHD), oppositional defiant disorder (ODD) and conduct disorder (CD) is described by Barkley et al. (unpublished manuscript). The members of the DSM-III-R Advisory Committee responsible for the diagnostic criteria for these disorders were able to reach agreement on potential items to be included in the final diagnostic criteria. They were also able to reach agreement, in the case of ADHD, on the value of abandoning the grouping of the component items under three separate rubrics (inattention, impulsivity, hyperactivity) and instead to treat the items polythetically, consistent with a general trend in DSM-III-R towards a polythetic format for diagnostic criteria. However, there was considerable disagreement within the Advisory Committee about the relative utility of different items for the three disorders and no agreement on how many items should be required from a final list of discriminating items to establish each of the diagnoses. In order to provide am empirical basis for resolving these issues, a national field trial of the proposed items sets was conducted in early 1985. The field trial was designed to answer the following questions: 1. Which items in the proposed criteria sets for the disruptive behavior disorders have sufficient discriminating power to be included in the final item sets? 2. What is the internal consistency of the three item sets? This addresses the issue of the extent to which the criteria for each of the three disorders are representative of its particular domain of psychopathology. 3. What threshold (minimum number of items) for making the diagnosis should be selected to maximize its sensitivity and specificity, using a clinical diagnosis made without ref-
AcceptedJanuary Si lvvt). Dr. Spitzer and Mr. Davies are from the New York State Psychiatric institute and the Department of Psychiatry. Columbia University. Dr. Barkley isfrom the Department ofPsychiatry• University ofMassachusetts Medical Center. Worchester. MA. The authors thank Dr. William E. Pelham for his many helpful comments in reviewing the initial manuscript. 0890-8567/90/2905-0690$02.00/0© 1990 by the American Acad-
emy of Child and Adolescent Psychiatry.
690
erence to the diagnostic criteria as the validity criterion? This addresses the descriptive validity of the diagnoses, that is, the extent to which the clinical features of the disorders are unique to them. 4. Is the prevalence of the disorders, diagnosed clinically without using the DSM-III-R diagnostic criteria, similar to the prevalence based on the final DSM-III-R criteria? 5. What is the level of agreement between clinical diagnoses made without using the DSM-III-R diagnostic criteria and diagnoses based on the final DSM-III-R criteria? 6. Is the pattern of comorbidity of the disruptive behavior disorders similar with diagnoses made clinically and diagnoses made with the DSM-III-R criteria? 7. Is there empirical support for theDSM -III-R convention that CD preempts a diagnosis of ODD? In other words, does the pattern of psychopathology indicate a Guttman type (1) of relationship between CD and ODD in which the majority of cases of CD also exhibit the defining symptoms of ODD? Method
Clinical facilities that evaluated large numbers of children who received diagnoses of one or more of the disruptive behavior disorders were recruited for the field trial. Table 1 lists the 10 field trial sites with demographic and diagnostic data for each site. (Note: This list is correct. Because of clerical error, the listing in Appendix F of DSM-III-R is incorrect.) In all, 550 children were evaluated by 72 clinicians. As can be seen, the average age of the subjects is similar across sites as is the usual preponderance of male subjects in child psychiatry clinics. Each facility was asked to evaluate at least 50 subjects from all consecutive referrals until at least 10 cases had been evaluated for each of the following five groups: ADHD, ODD, CD, other mental disorders (e.g. , specific developmental, major depression, overanxious disorder), and no mental disorder. Cases with a diagnosis of a pervasive developmental disorder were excluded, since this diagnosis precludes the diagnosis of ADHD. If a case had a diagnosis from more than one of the five groups, the same case would go towards filling the cells of each of the appropriate diagnoses. The clinicians were asked to make a diagnosis "as you
ordinarily do in clinical practice" andwithout consulting the
DISRUPTIVE BEHAVIOR DISORDERS FIELD TRIAL TABLE 1. Field Trial Sites', Demographic Data, and Diagnoses' ODD %
CD %
Other %
33
27
32
42
64
43
20
8
53
9.6
75
70
22
10
49
59
8.6
80
83
44
24
35
54
8.3
94
96
43
26
20
49
9.6
71
29
24
26
51
44
8.6
59
36
11
18
59
44
8.8
77
68
18
20
62
38
11.2
72
42
13
55
32
14 550
8.0 9.2
100 73
100 56
21 26
64 24
0 44
Mean Age
Male %
85
10.2
65
83
8.2
80
N
Site and (No. ofInterviewer) 1. Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine , Pittsburgh, PA (23) 2. Department of Pediatrics and Child Study Center, Yale University School of Medicine, New Haven, CT (6) 3. Division of Child Psychiatry, New York State Psychiatric Institute, and Babies Hospital, Columbia-Presbyterian Medical Center, New York (6) 4. Child Study Center, Summer Treatment Program, Dept. of Psychology, Florida State University, Tallahassee, FL (1) 5. Department of Pediatrics, Child Development Center, University of California, Irvine, CA (2) 6. Division of Child Psychiatry, Washington University School of Medicine, St. Louis, MO (3) 7. Center for Children, Youth & Families, Dept. of Psychiatry, University of Vermont, Burlington, VT (2) 8. Neuropsychology Clinic, Medical College of Wisconsin, Milwaukee, WI (3) 9. Department of Child Psychiatry University of California, Los Angeles, CA(24) 10. Child Psychiatry Branch, National Institute of Mental Health, Bethesda, MD(2) Total (72)
ADHD %
"The coordinators for the sites were: 1. Anthony Costello, M.D., 2. Bennet A. Shaywitz, M.D., 3. William Chambers, M.D., 4. William Pelham, Ph.D. ,5. JamesSwanson,M.D.,6. Felton Earls, M.D. , 7. ThomasAchenbach,Ph.D., 8. RussellA. Barkley, Ph.D. , 9. Dennis Cantwell, M.D. and 10. Judith Rapoport, M.D. bADHD = Attention deficit hyperactivity disorder, ODD = oppositional defiant disorder, CD = conduct disorder, Other = nondisruptive diagnosis (e.g., anxiety disorder, mental retardation) and no mental disorder.
ratings that the clinician made on the proposed DSM-III-R diagnostic criteria for the disorder. In making this "clinical" diagnosis, the clinicians were encouraged to consider factors that were not in DSM-III or the DSM-III-R proposed criteria, such as response to treatment, family history, symptoms of the disorder that are not in these item lists, and laboratory measures of attentional disturbance. The diagnostic assessment was predominantly based on a clinical interview of the parent (95%) and the child (72%). Less frequently, the assessment made use of a structured interview with the child (30%), parent (38%), school records (37%), and psychometric tests orlab measures (42%). At the time of evaluation, 12% of the children were receiving a psychotropic drug. The quality and completeness of the information was judged by the raters to be "good" or "excellent" in 82% of the cases. Following the evaluation, the clinician completed a symptom check list that listed all of the items proposed for the diagnostic criteria for the three disruptive behavior disorders. The items were not identified with their corresponding diagnoses but, presumably, most of the clinicians realized to which diagnoses they referred. Each item was rated as present, absent, or no information. For purposes of data analysis, different rules were ruled to handle missing information about the diagnostic criteria. For calculating sensitivity and specificity of individual diagnostic criteria, all cases were considered valid that had information on the particular item (Tables 3, 5, and 7). Thus, a case for which the clinician did not have sufficient information to l.Am.Acad. Child Adolesc. Psychiatry, 29:5, September 1990
make a judgment that the item was either present or absent (not an uncommon occurrence in the real world of clinical assessment), was not included in this data analysis. For calculating the psychometric descriptors of the DSM-III-R diagnoses at different thresholds, a valid case had to contain information on all of the diagnostic, criteria (Tables 4,6, and 8). In calculating agreement between clinical and DSM-III-R diagnoses, a valid case had to have valid information for both diagnoses (Table 9). For calculating the comorbidity of clinical and DSM-III-R diagnoses, valid diagnoses had to be available for all clinical and DSM-III-R diagnoses. Because of these rules for handling missing information, the number ofsubjects reported in the various tables vary. Table 1, which reports the clinical diagnoses (no missing information), has the largest NS, whereas, Table 10, which has the most restrictive definition of a valid case, has the smallest NS.
Results As expected, there was a high degree ofcomorbidity within the group of disruptive behavior disorders. As can be seen in Table 2, which presents the comorbidity of the clinical diagnoses, most of the cases with a diagnosis of a disruptive behavior disorder also had another mental disorder. Over half of the cases of ADHD also had a diagnosis of either ODD or CD. Similarly, over half of the cases of CD also had a diagnosis of ADHD, and over half of the cases of ODD also had a diagnosis of ADHD. Although the degree of comorbidity may represent referral biases to the particular clinics that were involved, the results suggest that each ofthese disruptive 691
SPITZER ET AL.
TABLE 2. Comorbidity ofClinical Diagnoses
N Attention deficit hyperactivity disorder (ADHD) (N = 311) ADHDonly ADHD and ODD (with or without nondisruptive diagnosis) ADHD and CD (with or without nondisruptive diagnosis) ADHD and nondisruptive mental disorder Oppositional defiant disorder (ODD) (N = 140) ODD only ODD and ADHD (with or without nondisrupti ve diagnosis) ODD and nondisruptive mental disorder Conduct disorder (CD) (N = 130) Conduct disorder (CD) only CD and ADHD (with or without nondisruptive diagnosis) CD and nondisruptive mental disorder Nondisruptive mental disorder only (N = 104) No mental disorder (N = 30) Total
%
TABLE 3. Sensitivity, Specificity, and Odds Ratio for Proposed Criteria for Clinical Diagnosis ofAttention Deficit Hyperactivity Disorder Sensitivity
109
35
94
30
71 37
23 12
29
21
94 17
67 12
45
35
71 14
55 11
550
100
disorders (particularly ODD) are usually seen in association with another disruptive behavior disorder. ADHD. The sensitivity, specificity, and odds ratio for the proposed criteria for ADHD were calculated, using the clinical diagnosis of ADHD as the criterion. This data is presented in Table 3 with the items ordered, as in the DSM-III-R manual, by the magnitude of the odds ratio (odds of the item in cases diagnosed ADHD divided by the odds of nonADHD cases). Generally, the odds ratios are quite high, indicating that the items have considerable discriminating power. On the basis of this analysis, the item, "Often extremely messy or sloppy" was dropped because of its comparatively low odds ratio. The odds ratios indicate that clinicians gave more weight to the items, "Fidgets or squirms" and" Difficulty remaining seated" -symptoms of overactivity-than they gave to items describing inattentiveness and impulsivity. The item with the best combination of sensitivity and specificity was an item with face-validity for tapping impulsivity-"Difficulty awaiting turns in games or group situations. " Also of interest, the items with face-validity for assessing inattention all had low specificity (e.g., "Doesn't listen," "Easily distracted," and "Difficulty following instructions"). The internal consistency ofthe item set was very high, 0.90. Using the fourteen items, Table 4 presents the psychometric properties of the items set by different thresholds for making the diagnosis, as can be seen, requiring at least eight of the 14 items maximizes the total predictive value and yields sensitivity and specificity above 0.80. Therefore, the DSM-IlI-R criteria for ADHD require at least eight of the 14 items. ODD. Table 5 presents the sensitivity, specificity, and odds ratio for the proposed criteria for ODD, again using the clinical diagnosis of ODD as the criterion. Cases with a clinical diagnosis of CD were excluded from the analysis as these cases would be expected to have a large number of the
Criterion (Abbreviated)
1. Fidgets or squirms 2. Difficulty remaining seated 3. Easily distracted 4. Difficulty awaiting tum 5. Blurts out answers to questions 6. Difficulty following instructions 7. Difficulty sustaining attention 8. Shifts between uncompleted tasks 9. Difficulty playing quietly 10. Talks excessively 11. Interrupts or intrudes on others 12. Doesn't listen 13. Loses things 14. Physically dangerous activities Extremely messy or sloppy' a
qv = 291-310)
Specificity Odds = 222-236) Ratio
(N
91
54
11.85
85 91
65 50
10.67 10.59
82
79
9.18
65
83
9.03
85
61
8.92
86
57
8.12
84
59
7.56
64 73
79 68
6.65 5.70
74 85 66
66 44 69
5.59 4.52 4.37
47
79
3.35
54
70
2.81
Item not included in final item set.
TABLE 4. Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), and Total Predictive Value (TPV) ofAttention Deficit Hyperactivity Disorder Criteria by Different
Thresholdsfor Making the Diagnosis = 260; NonADHD, N = 291)
(ADHD, N
Threshold (Minimum No. of Symptoms)
Sensitivity
Specificity
PPV
NPV
TPV
5 6 7 8 9 10 11
95 91 87 85 76 69 60
53 61 71 80 83 91 94
73 76 80 85 86 91 93
88 83 80 79 72 68 63
77 78 80 83 79 78 74
items for ODD. As with ADHD, the odds ratios are quite high, indicating that the items have considerable discriminatingpower. The item, "Bullies oris mean to other children (other than physically cruel)" had a reasonably high odds ratio. However, it was deleted from the final ODD DSM-III-R item set because it seemed to describe behavior that was closer to the underlying construct of CD than to the construct of ODD, and it was as correlated with the clinical diagnosis of CD as with l,
692
l.Am.Acad. Child Adolesc.Psychiatry, 29:5, September 1990
DISRUPTIVE BEHAVIOR DISORDERS FIELD TRIAL TABLE
5. Sensitivity, Specificity, and Odds Ratiofor Proposed Criteria for Clinical Diagnosis ofOppositional Defiant Disorder" Sensitivity
Specificity = 268-278)
Odds Ratio
Criterion (Abbreviated)
(IV = 136-139)
1. Loses temper 2. Argues with adults 3. Actively defies requests or rules 4. Deliberately annoys other people 5. Blames others for his mistakes Bullies or is mean to other children" 6. Touchy or easily annoyed 7. Angry or resentful 8. Spiteful or vindictive 9. Swears or uses obscene language
87 86
59 59
9.61 9.22
84
62
8.15
77
69
7.30
73
65
5.09
48
86
5.40
73 67 44
61 68 89
4.16 4.34 6.06
36
87
3.78
(N
the clinical diagnosis of ODD. The internal consistency of the total item set was high, 0.85. Using the nine items, Table 6 presents the psychometric properties of the full item set by different thresholds for making the diagnosis. As can be seen, requiring at least five of the nine items maximizes the total predictive value and yields sensitivity and specificity of approximately 80. Therefore, the DSM-III-R criteria for ODD require at least five of the nine items. TABLE 6. Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), and Total Predictive Value (TPV) ofOppositional Defiant Disorder Criterid' by Different Thresholds for Making the Diagnosis (ODD, N = 130, NonODD, N = 249)
2 3 4 5 6
Sensitivity
Specificity
PPV
NPV
TPV
96 93 88 80 67
40 59 70 79 86 92 96
45 54 61 67 72
95 94 92 88 83
59 71 76 79 80
7
51
8
33
7. Sensitivity, Specificity, and Odds Ratio for Proposed Criteria for Clinical Diagnosis ofConduct Disorder Sensitivity
Criterion (Abbreviated) Unusually early sexual activity" I. Stolen without confrontation 2. Runaway from home at least twice 3. Often lies Early use of tobacco ordrug~
" Cases with a clinical diagnosis of conduct disorder have been excluded from the analysis as these cases would be expected to have a large number of the items for oppositional defiant disorder. b Item not included in final item set.
Threshold (Minimum No. of Symptoms)
TABLE
77
78
78
80
73
74
"Cases with a clinical diagnosis of conduct disorder have been excluded from the analysis as these cases would be expected to have a large number of the items for oppositional defiant disorder.
CD. The sensitivity, specificity, and odds ratio for the proposed criteria from CD are presented in Table 7. Interestingly, the item, "often lies," is the item with the best combination of both sensitivity and specificity. Although all of the J.Am.Acad. Child Adolesc.Psychiatry, 29:5, September 1990
4. Deliberately set fires 5. Truant 6. Broken into someone's house or car 7. Deliberately destroyed property 8. Physically cruel to animals 9. Forced someone to have sex Borrows without permission" 10. Used a weapon in fights Cheats in games" 11. Initiates physical fights 12. Stolen with confrontation 13. Physically cruel to other people
Specificity
(IV = 112-130) (N = 386-408)
Odds Ratio
13
99
14.97
43
94
11.34
18 80
98 72
10.85 10.23
19 35 24
97 93 95
8.63 7.37 6.20
9
98
6.22
51
86
6.12
23
94
5.07
5
99
4.93
40
86
4.25
16 40
95 85
3.76 3.72
53
77
3.84
100
3.17
94
3.14
16
"Item not included in final item set.
items had considerable discriminating power, four items were eliminated. The items describing unusually early sexually activity and use of tobacco or drugs were dropped because they did not describe behavior that was inherently antisocial, even though the behaviors are risk factors for the diagnosis. The items describing borrowing without permission and cheating in games were eliminated because of the high frequency with which such symptoms are also commonly seen in Ol) and ADHD. The internal consistency of the total item set was acceptable, 0.73, although lower than ADHS and ODD. Using the 13 finalDSM -III-R items, Table 8 presents the psychometric properties of the full item set by different thresholds for making the diagnosis. A threshold of at least three items was chosen as this maximized the total predictive value (0.85) and yields a sensitivity of 0.71 and a specificity of 0.90. Prevalenceofdisorder by method ofdiagnosis. The prevalence of the three disorders, by clinical diagnosis and by DSM-III-R criteria, was calculated for each site and for the total sample (Table 9). This was done in order to determine if, for some sites, there are statistically significant discrepancies between the prevalence of the disorders according to whether the diagnosis was clinical or based on theDSM-III-R
693
SPITZER ET AL.
TABLE 10. Comorbidity ofClinical and DSM-IlI-R Diagnoses
TABLE 8. Sensitivity , Specificity, Positive Predi ctive Value (PPV ), Negative, Predi ctive Value (NPV), and Total Predi ctive Value (TPV) of Conduct Disord er Criteria by Different Thresholdsfor Making the Diagnosis (CD , N = 113, NonCD, N = 372) Threshold (Minimum No. of Symptoms)
Clinical N
%
N
106 32 39
26 8 9
80 46 27
II
7
0.01 0.06 0.05
75 59
18 14
75 78
18 19
NS 0.01
NeitherADHD,ODD, or CD
103
25
108
26
NS
Total
414
100
414
100
ADHDalone ODD alone CD alone Sensitivity
Specificity
PPV
NPV
TPV
97 90 71 50 30 12
54 73 90 95 97 99
39 50 68 74 76 74
99 96 91 86 82 79
64 77 85 84 81 79
I
2 3 4 5 6
ADHDandODD ADHD and CD
criteria. Althou gh for some sites the prevalence of the disorders differs markedly accordin g to whether the diagnosis was clinical or based on the DSM-lII -R criteria, when the total sample is examined, the prevalences are virtually ident ical for ADHD and CD , and differ only slightly for ODD (p < NS) . Agreement betwe en clinical diagnosis and DSM-III-R diagnosis. Table 9 also presents the kappa agreement between the clinical diagno sis and the DSM-lII -R diagno sis for each site and for the total sample . There is considerable variability across the sites and for the total sample the agreement is only fair for ADHD and CD , and poor for ODD. Psychometric properties of item sets by age and sex . The sensitivity, specificity, and total predictive value were calculated for three age groups (2 to 7, 8 to 11, and older) and by sex. The only system atic effect of age was slightly decreased sensitivity in the older children for ADHD (0.92, 0.82, and 0 .72, respectively ,p < 0 .05) and a corresponding increase in specificity (0.74, 0 .77 , and 0.89, respectively, p
AND
RUSSELL A. BARKLEY, PH.D.
Abstract. The members of the DSM-Il/-R Advisory Committee responsible for the diagnostic criteria for the disruptive behavior disorders (attention deficit hyperactivity disorder, oppositional defiant disorder, and conduct disorder) were able to reach agreement on potential items to be included in the final diagnostic criteria. However, there was considerable disagreement about the relative utility of different items for the three disorders and no agreement on how many items should be required from a final list of discriminating items to establish each of the diagnoses. This article describes the method and results of a national field trial of the proposed criteria. Using as a standard the diagnosis of these disorders made by expert clinicians with experience with these disorders, the diagnostic criteria that were finally included in DSM-Il/-R demonstrated high sensitivity, specificity, and internal consistency. J. Am. Acad. ChildAdolesc. Psychiatry. 1990, 29, 5:690-697. Key Words: disruptive, attention deficit, oppositional, conduct, field trial.
The rationale for the development of the items for the diagnostic criteria for the disruptive behavior disorders, attention deficit hyperactivity disorder (ADHD), oppositional defiant disorder (ODD) and conduct disorder (CD) is described by Barkley et al. (unpublished manuscript). The members of the DSM-III-R Advisory Committee responsible for the diagnostic criteria for these disorders were able to reach agreement on potential items to be included in the final diagnostic criteria. They were also able to reach agreement, in the case of ADHD, on the value of abandoning the grouping of the component items under three separate rubrics (inattention, impulsivity, hyperactivity) and instead to treat the items polythetically, consistent with a general trend in DSM-III-R towards a polythetic format for diagnostic criteria. However, there was considerable disagreement within the Advisory Committee about the relative utility of different items for the three disorders and no agreement on how many items should be required from a final list of discriminating items to establish each of the diagnoses. In order to provide am empirical basis for resolving these issues, a national field trial of the proposed items sets was conducted in early 1985. The field trial was designed to answer the following questions: 1. Which items in the proposed criteria sets for the disruptive behavior disorders have sufficient discriminating power to be included in the final item sets? 2. What is the internal consistency of the three item sets? This addresses the issue of the extent to which the criteria for each of the three disorders are representative of its particular domain of psychopathology. 3. What threshold (minimum number of items) for making the diagnosis should be selected to maximize its sensitivity and specificity, using a clinical diagnosis made without ref-
AcceptedJanuary Si lvvt). Dr. Spitzer and Mr. Davies are from the New York State Psychiatric institute and the Department of Psychiatry. Columbia University. Dr. Barkley isfrom the Department ofPsychiatry• University ofMassachusetts Medical Center. Worchester. MA. The authors thank Dr. William E. Pelham for his many helpful comments in reviewing the initial manuscript. 0890-8567/90/2905-0690$02.00/0© 1990 by the American Acad-
emy of Child and Adolescent Psychiatry.
690
erence to the diagnostic criteria as the validity criterion? This addresses the descriptive validity of the diagnoses, that is, the extent to which the clinical features of the disorders are unique to them. 4. Is the prevalence of the disorders, diagnosed clinically without using the DSM-III-R diagnostic criteria, similar to the prevalence based on the final DSM-III-R criteria? 5. What is the level of agreement between clinical diagnoses made without using the DSM-III-R diagnostic criteria and diagnoses based on the final DSM-III-R criteria? 6. Is the pattern of comorbidity of the disruptive behavior disorders similar with diagnoses made clinically and diagnoses made with the DSM-III-R criteria? 7. Is there empirical support for theDSM -III-R convention that CD preempts a diagnosis of ODD? In other words, does the pattern of psychopathology indicate a Guttman type (1) of relationship between CD and ODD in which the majority of cases of CD also exhibit the defining symptoms of ODD? Method
Clinical facilities that evaluated large numbers of children who received diagnoses of one or more of the disruptive behavior disorders were recruited for the field trial. Table 1 lists the 10 field trial sites with demographic and diagnostic data for each site. (Note: This list is correct. Because of clerical error, the listing in Appendix F of DSM-III-R is incorrect.) In all, 550 children were evaluated by 72 clinicians. As can be seen, the average age of the subjects is similar across sites as is the usual preponderance of male subjects in child psychiatry clinics. Each facility was asked to evaluate at least 50 subjects from all consecutive referrals until at least 10 cases had been evaluated for each of the following five groups: ADHD, ODD, CD, other mental disorders (e.g. , specific developmental, major depression, overanxious disorder), and no mental disorder. Cases with a diagnosis of a pervasive developmental disorder were excluded, since this diagnosis precludes the diagnosis of ADHD. If a case had a diagnosis from more than one of the five groups, the same case would go towards filling the cells of each of the appropriate diagnoses. The clinicians were asked to make a diagnosis "as you
ordinarily do in clinical practice" andwithout consulting the
DISRUPTIVE BEHAVIOR DISORDERS FIELD TRIAL TABLE 1. Field Trial Sites', Demographic Data, and Diagnoses' ODD %
CD %
Other %
33
27
32
42
64
43
20
8
53
9.6
75
70
22
10
49
59
8.6
80
83
44
24
35
54
8.3
94
96
43
26
20
49
9.6
71
29
24
26
51
44
8.6
59
36
11
18
59
44
8.8
77
68
18
20
62
38
11.2
72
42
13
55
32
14 550
8.0 9.2
100 73
100 56
21 26
64 24
0 44
Mean Age
Male %
85
10.2
65
83
8.2
80
N
Site and (No. ofInterviewer) 1. Western Psychiatric Institute and Clinic, University of Pittsburgh School of Medicine , Pittsburgh, PA (23) 2. Department of Pediatrics and Child Study Center, Yale University School of Medicine, New Haven, CT (6) 3. Division of Child Psychiatry, New York State Psychiatric Institute, and Babies Hospital, Columbia-Presbyterian Medical Center, New York (6) 4. Child Study Center, Summer Treatment Program, Dept. of Psychology, Florida State University, Tallahassee, FL (1) 5. Department of Pediatrics, Child Development Center, University of California, Irvine, CA (2) 6. Division of Child Psychiatry, Washington University School of Medicine, St. Louis, MO (3) 7. Center for Children, Youth & Families, Dept. of Psychiatry, University of Vermont, Burlington, VT (2) 8. Neuropsychology Clinic, Medical College of Wisconsin, Milwaukee, WI (3) 9. Department of Child Psychiatry University of California, Los Angeles, CA(24) 10. Child Psychiatry Branch, National Institute of Mental Health, Bethesda, MD(2) Total (72)
ADHD %
"The coordinators for the sites were: 1. Anthony Costello, M.D., 2. Bennet A. Shaywitz, M.D., 3. William Chambers, M.D., 4. William Pelham, Ph.D. ,5. JamesSwanson,M.D.,6. Felton Earls, M.D. , 7. ThomasAchenbach,Ph.D., 8. RussellA. Barkley, Ph.D. , 9. Dennis Cantwell, M.D. and 10. Judith Rapoport, M.D. bADHD = Attention deficit hyperactivity disorder, ODD = oppositional defiant disorder, CD = conduct disorder, Other = nondisruptive diagnosis (e.g., anxiety disorder, mental retardation) and no mental disorder.
ratings that the clinician made on the proposed DSM-III-R diagnostic criteria for the disorder. In making this "clinical" diagnosis, the clinicians were encouraged to consider factors that were not in DSM-III or the DSM-III-R proposed criteria, such as response to treatment, family history, symptoms of the disorder that are not in these item lists, and laboratory measures of attentional disturbance. The diagnostic assessment was predominantly based on a clinical interview of the parent (95%) and the child (72%). Less frequently, the assessment made use of a structured interview with the child (30%), parent (38%), school records (37%), and psychometric tests orlab measures (42%). At the time of evaluation, 12% of the children were receiving a psychotropic drug. The quality and completeness of the information was judged by the raters to be "good" or "excellent" in 82% of the cases. Following the evaluation, the clinician completed a symptom check list that listed all of the items proposed for the diagnostic criteria for the three disruptive behavior disorders. The items were not identified with their corresponding diagnoses but, presumably, most of the clinicians realized to which diagnoses they referred. Each item was rated as present, absent, or no information. For purposes of data analysis, different rules were ruled to handle missing information about the diagnostic criteria. For calculating sensitivity and specificity of individual diagnostic criteria, all cases were considered valid that had information on the particular item (Tables 3, 5, and 7). Thus, a case for which the clinician did not have sufficient information to l.Am.Acad. Child Adolesc. Psychiatry, 29:5, September 1990
make a judgment that the item was either present or absent (not an uncommon occurrence in the real world of clinical assessment), was not included in this data analysis. For calculating the psychometric descriptors of the DSM-III-R diagnoses at different thresholds, a valid case had to contain information on all of the diagnostic, criteria (Tables 4,6, and 8). In calculating agreement between clinical and DSM-III-R diagnoses, a valid case had to have valid information for both diagnoses (Table 9). For calculating the comorbidity of clinical and DSM-III-R diagnoses, valid diagnoses had to be available for all clinical and DSM-III-R diagnoses. Because of these rules for handling missing information, the number ofsubjects reported in the various tables vary. Table 1, which reports the clinical diagnoses (no missing information), has the largest NS, whereas, Table 10, which has the most restrictive definition of a valid case, has the smallest NS.
Results As expected, there was a high degree ofcomorbidity within the group of disruptive behavior disorders. As can be seen in Table 2, which presents the comorbidity of the clinical diagnoses, most of the cases with a diagnosis of a disruptive behavior disorder also had another mental disorder. Over half of the cases of ADHD also had a diagnosis of either ODD or CD. Similarly, over half of the cases of CD also had a diagnosis of ADHD, and over half of the cases of ODD also had a diagnosis of ADHD. Although the degree of comorbidity may represent referral biases to the particular clinics that were involved, the results suggest that each ofthese disruptive 691
SPITZER ET AL.
TABLE 2. Comorbidity ofClinical Diagnoses
N Attention deficit hyperactivity disorder (ADHD) (N = 311) ADHDonly ADHD and ODD (with or without nondisruptive diagnosis) ADHD and CD (with or without nondisruptive diagnosis) ADHD and nondisruptive mental disorder Oppositional defiant disorder (ODD) (N = 140) ODD only ODD and ADHD (with or without nondisrupti ve diagnosis) ODD and nondisruptive mental disorder Conduct disorder (CD) (N = 130) Conduct disorder (CD) only CD and ADHD (with or without nondisruptive diagnosis) CD and nondisruptive mental disorder Nondisruptive mental disorder only (N = 104) No mental disorder (N = 30) Total
%
TABLE 3. Sensitivity, Specificity, and Odds Ratio for Proposed Criteria for Clinical Diagnosis ofAttention Deficit Hyperactivity Disorder Sensitivity
109
35
94
30
71 37
23 12
29
21
94 17
67 12
45
35
71 14
55 11
550
100
disorders (particularly ODD) are usually seen in association with another disruptive behavior disorder. ADHD. The sensitivity, specificity, and odds ratio for the proposed criteria for ADHD were calculated, using the clinical diagnosis of ADHD as the criterion. This data is presented in Table 3 with the items ordered, as in the DSM-III-R manual, by the magnitude of the odds ratio (odds of the item in cases diagnosed ADHD divided by the odds of nonADHD cases). Generally, the odds ratios are quite high, indicating that the items have considerable discriminating power. On the basis of this analysis, the item, "Often extremely messy or sloppy" was dropped because of its comparatively low odds ratio. The odds ratios indicate that clinicians gave more weight to the items, "Fidgets or squirms" and" Difficulty remaining seated" -symptoms of overactivity-than they gave to items describing inattentiveness and impulsivity. The item with the best combination of sensitivity and specificity was an item with face-validity for tapping impulsivity-"Difficulty awaiting turns in games or group situations. " Also of interest, the items with face-validity for assessing inattention all had low specificity (e.g., "Doesn't listen," "Easily distracted," and "Difficulty following instructions"). The internal consistency ofthe item set was very high, 0.90. Using the fourteen items, Table 4 presents the psychometric properties of the items set by different thresholds for making the diagnosis, as can be seen, requiring at least eight of the 14 items maximizes the total predictive value and yields sensitivity and specificity above 0.80. Therefore, the DSM-IlI-R criteria for ADHD require at least eight of the 14 items. ODD. Table 5 presents the sensitivity, specificity, and odds ratio for the proposed criteria for ODD, again using the clinical diagnosis of ODD as the criterion. Cases with a clinical diagnosis of CD were excluded from the analysis as these cases would be expected to have a large number of the
Criterion (Abbreviated)
1. Fidgets or squirms 2. Difficulty remaining seated 3. Easily distracted 4. Difficulty awaiting tum 5. Blurts out answers to questions 6. Difficulty following instructions 7. Difficulty sustaining attention 8. Shifts between uncompleted tasks 9. Difficulty playing quietly 10. Talks excessively 11. Interrupts or intrudes on others 12. Doesn't listen 13. Loses things 14. Physically dangerous activities Extremely messy or sloppy' a
qv = 291-310)
Specificity Odds = 222-236) Ratio
(N
91
54
11.85
85 91
65 50
10.67 10.59
82
79
9.18
65
83
9.03
85
61
8.92
86
57
8.12
84
59
7.56
64 73
79 68
6.65 5.70
74 85 66
66 44 69
5.59 4.52 4.37
47
79
3.35
54
70
2.81
Item not included in final item set.
TABLE 4. Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), and Total Predictive Value (TPV) ofAttention Deficit Hyperactivity Disorder Criteria by Different
Thresholdsfor Making the Diagnosis = 260; NonADHD, N = 291)
(ADHD, N
Threshold (Minimum No. of Symptoms)
Sensitivity
Specificity
PPV
NPV
TPV
5 6 7 8 9 10 11
95 91 87 85 76 69 60
53 61 71 80 83 91 94
73 76 80 85 86 91 93
88 83 80 79 72 68 63
77 78 80 83 79 78 74
items for ODD. As with ADHD, the odds ratios are quite high, indicating that the items have considerable discriminatingpower. The item, "Bullies oris mean to other children (other than physically cruel)" had a reasonably high odds ratio. However, it was deleted from the final ODD DSM-III-R item set because it seemed to describe behavior that was closer to the underlying construct of CD than to the construct of ODD, and it was as correlated with the clinical diagnosis of CD as with l,
692
l.Am.Acad. Child Adolesc.Psychiatry, 29:5, September 1990
DISRUPTIVE BEHAVIOR DISORDERS FIELD TRIAL TABLE
5. Sensitivity, Specificity, and Odds Ratiofor Proposed Criteria for Clinical Diagnosis ofOppositional Defiant Disorder" Sensitivity
Specificity = 268-278)
Odds Ratio
Criterion (Abbreviated)
(IV = 136-139)
1. Loses temper 2. Argues with adults 3. Actively defies requests or rules 4. Deliberately annoys other people 5. Blames others for his mistakes Bullies or is mean to other children" 6. Touchy or easily annoyed 7. Angry or resentful 8. Spiteful or vindictive 9. Swears or uses obscene language
87 86
59 59
9.61 9.22
84
62
8.15
77
69
7.30
73
65
5.09
48
86
5.40
73 67 44
61 68 89
4.16 4.34 6.06
36
87
3.78
(N
the clinical diagnosis of ODD. The internal consistency of the total item set was high, 0.85. Using the nine items, Table 6 presents the psychometric properties of the full item set by different thresholds for making the diagnosis. As can be seen, requiring at least five of the nine items maximizes the total predictive value and yields sensitivity and specificity of approximately 80. Therefore, the DSM-III-R criteria for ODD require at least five of the nine items. TABLE 6. Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), and Total Predictive Value (TPV) ofOppositional Defiant Disorder Criterid' by Different Thresholds for Making the Diagnosis (ODD, N = 130, NonODD, N = 249)
2 3 4 5 6
Sensitivity
Specificity
PPV
NPV
TPV
96 93 88 80 67
40 59 70 79 86 92 96
45 54 61 67 72
95 94 92 88 83
59 71 76 79 80
7
51
8
33
7. Sensitivity, Specificity, and Odds Ratio for Proposed Criteria for Clinical Diagnosis ofConduct Disorder Sensitivity
Criterion (Abbreviated) Unusually early sexual activity" I. Stolen without confrontation 2. Runaway from home at least twice 3. Often lies Early use of tobacco ordrug~
" Cases with a clinical diagnosis of conduct disorder have been excluded from the analysis as these cases would be expected to have a large number of the items for oppositional defiant disorder. b Item not included in final item set.
Threshold (Minimum No. of Symptoms)
TABLE
77
78
78
80
73
74
"Cases with a clinical diagnosis of conduct disorder have been excluded from the analysis as these cases would be expected to have a large number of the items for oppositional defiant disorder.
CD. The sensitivity, specificity, and odds ratio for the proposed criteria from CD are presented in Table 7. Interestingly, the item, "often lies," is the item with the best combination of both sensitivity and specificity. Although all of the J.Am.Acad. Child Adolesc.Psychiatry, 29:5, September 1990
4. Deliberately set fires 5. Truant 6. Broken into someone's house or car 7. Deliberately destroyed property 8. Physically cruel to animals 9. Forced someone to have sex Borrows without permission" 10. Used a weapon in fights Cheats in games" 11. Initiates physical fights 12. Stolen with confrontation 13. Physically cruel to other people
Specificity
(IV = 112-130) (N = 386-408)
Odds Ratio
13
99
14.97
43
94
11.34
18 80
98 72
10.85 10.23
19 35 24
97 93 95
8.63 7.37 6.20
9
98
6.22
51
86
6.12
23
94
5.07
5
99
4.93
40
86
4.25
16 40
95 85
3.76 3.72
53
77
3.84
100
3.17
94
3.14
16
"Item not included in final item set.
items had considerable discriminating power, four items were eliminated. The items describing unusually early sexually activity and use of tobacco or drugs were dropped because they did not describe behavior that was inherently antisocial, even though the behaviors are risk factors for the diagnosis. The items describing borrowing without permission and cheating in games were eliminated because of the high frequency with which such symptoms are also commonly seen in Ol) and ADHD. The internal consistency of the total item set was acceptable, 0.73, although lower than ADHS and ODD. Using the 13 finalDSM -III-R items, Table 8 presents the psychometric properties of the full item set by different thresholds for making the diagnosis. A threshold of at least three items was chosen as this maximized the total predictive value (0.85) and yields a sensitivity of 0.71 and a specificity of 0.90. Prevalenceofdisorder by method ofdiagnosis. The prevalence of the three disorders, by clinical diagnosis and by DSM-III-R criteria, was calculated for each site and for the total sample (Table 9). This was done in order to determine if, for some sites, there are statistically significant discrepancies between the prevalence of the disorders according to whether the diagnosis was clinical or based on theDSM-III-R
693
SPITZER ET AL.
TABLE 10. Comorbidity ofClinical and DSM-IlI-R Diagnoses
TABLE 8. Sensitivity , Specificity, Positive Predi ctive Value (PPV ), Negative, Predi ctive Value (NPV), and Total Predi ctive Value (TPV) of Conduct Disord er Criteria by Different Thresholdsfor Making the Diagnosis (CD , N = 113, NonCD, N = 372) Threshold (Minimum No. of Symptoms)
Clinical N
%
N
106 32 39
26 8 9
80 46 27
II
7
0.01 0.06 0.05
75 59
18 14
75 78
18 19
NS 0.01
NeitherADHD,ODD, or CD
103
25
108
26
NS
Total
414
100
414
100
ADHDalone ODD alone CD alone Sensitivity
Specificity
PPV
NPV
TPV
97 90 71 50 30 12
54 73 90 95 97 99
39 50 68 74 76 74
99 96 91 86 82 79
64 77 85 84 81 79
I
2 3 4 5 6
ADHDandODD ADHD and CD
criteria. Althou gh for some sites the prevalence of the disorders differs markedly accordin g to whether the diagnosis was clinical or based on the DSM-lII -R criteria, when the total sample is examined, the prevalences are virtually ident ical for ADHD and CD , and differ only slightly for ODD (p < NS) . Agreement betwe en clinical diagnosis and DSM-III-R diagnosis. Table 9 also presents the kappa agreement between the clinical diagno sis and the DSM-lII -R diagno sis for each site and for the total sample . There is considerable variability across the sites and for the total sample the agreement is only fair for ADHD and CD , and poor for ODD. Psychometric properties of item sets by age and sex . The sensitivity, specificity, and total predictive value were calculated for three age groups (2 to 7, 8 to 11, and older) and by sex. The only system atic effect of age was slightly decreased sensitivity in the older children for ADHD (0.92, 0.82, and 0 .72, respectively ,p < 0 .05) and a corresponding increase in specificity (0.74, 0 .77 , and 0.89, respectively, p
