J Abnorm Child Psychol DOI 10.1007/s10802-014-9923-4

Integrating Autism-Related Symptoms into the Dimensional Internalizing and Externalizing Model of Psychopathology. The TRAILS Study Arjen Noordhof & Robert F. Krueger & Johan Ormel & Albertine J. Oldehinkel & Catharina A. Hartman

# Springer Science+Business Media New York 2014

Abstract Problems associated with Autism Spectrum Disorder (ASD) occur frequently in the general population and often co-occur with problems in other domains of psychopathology. In the research presented here these co-occurrence patterns were investigated by integrating a dimensional approach to ASDs into the more general dimensional framework of internalizing and externalizing psychopathology. Factor Analysis was used to develop hierarchical and bi-factor models covering multiple domains of psychopathology in three measurement waves of a longitudinal general population sample (N=2,230, ages 10–17, 50.8 % female). In all adequately fitting models, autism related problems were part of a specific domain of psychopathology that could be distinguished from the internalizing and externalizing domains. Optimal model fit was found for a bi-factor model with one non-specific factor and four specific factors related to internalizing, externalizing, autism spectrum problems and problems related to attention and orientation. Autism-related problems constitute a specific domain of psychopathology that can be distinguished from the internalizing and externalizing

Electronic supplementary material The online version of this article (doi:10.1007/s10802-014-9923-4) contains supplementary material, which is available to authorized users. A. Noordhof : J. Ormel : A. J. Oldehinkel : C. A. Hartman University Medical Center Groningen, Groningen, Netherlands A. Noordhof (*) University of Amsterdam, Weesperplein 4, Amsterdam 1018, XA, The Netherlands e-mail: [email protected] R. F. Krueger University of Minnesota, Minneapolis, USA A. J. Oldehinkel University of Groningen, Groningen, Netherlands

domains. In addition, the co-occurrence patterns in the data indicate the presence of a strong general factor. Keywords Autism spectrum disorder . Comorbidity . Internalizing psychopathology . Externalizing psychopathology . Confirmatory factor analysis In the recently published DSM-5 (American Psychiatric Association 2013), the DSM-IV system of categorical diagnosis was basically retained (as the major focus of Section II, labeled Diagnostic Criteria and Codes), but some steps have been made towards a more dimensional representation of psychopathology. For example, previous dichotomous distinctions between Asperger’s Syndrome, Autistic Disorder and Pervasive Developmental Disorder-Not Otherwise Specified (PDD-NOS) have been removed and these disorders have been collapsed into the single diagnosis Autism Spectrum Disorder with additional severity-ratings to capture within-category heterogeneity. This hybrid approach to diagnosis was based on multiple studies showing validity and utility of both categorical (e.g., Frazier et al. 2010) and dimensional (e.g., Constantino 2011) representations of this symptom domain, and was validated by two recent studies (Frazier et al. 2012; Mandy, Charman, and Skuse 2012). Also, a set of optional transdiagnostic dimensional measures (DSM5) have been described (in Section III, labeled Emerging Measures and Model), which can be applied regardless of the categorical diagnosis an individual receives. Furthermore, in the introduction to DSM-5 it is explained that many specific disorders can be grouped into the well-established structure of internalizing and externalizing psychopathology (e.g., Eaton, South, and Krueger 2010; Krueger 1999; Markon 2010). Autism-related symptoms are not explicitly included in the DSM-5 description of the internalizing and externalizing groupings, and may represent a separable domain of

J Abnorm Child Psychol

psychopathology. However, just like internalizing and externalizing problems, autism-related symptoms appear very useful for transdiagnostic purposes. For example, dimensionally charting autism-related problems may be clinically relevant for children who do not meet DSM-5 criteria of Autism Spectrum Disorder (ASD), because their sub-threshold problems may still be important in the course of another disorder or with regard to impairments in daily life (Greaves-Lord et al. 2013; St Pourcain et al. 2011). In the current paper we therefore aim to integrate autism-related symptoms into the dimensional m odel of internalizing a nd ex ternalizing psychopathology. The distinction between internalizing and externalizing problems has a solid empirical base. As first described by Achenbach (1966), psychopathology can be summarized by two higher-order factors of covarying problems: internalizing and externalizing psychopathology. Below those broad dimensions there are narrow-band dimensions that describe specific features (e.g., withdrawn behavior, anxiety, and somatoform symptoms; Achenbach 1991). Accommodating both lumping and splitting at different levels, this hierarchical model provides a basis for describing heterogeneity and comorbidity of a wide range of problems. The distinction between internalizing and externalizing psychopathology has been replicated and extended in clinical and general populations samples for both childhood (e.g., Lahey et al. 2008, 2012) and adult psychopathology (Krueger 1999; Markon 2010; Vollebergh et al. 2001). A number of studies have shown the relevance of dimensional representations of autism related problems in clinical and general population samples (Constantino et al. 2006; Constantino and Todd 2003; Frazier et al. 2010, 2012; Mandy et al. 2012; Mulligan et al. 2009; Reiersen, Constantino, Grimmer, Martin, and Todd 2008) as well as their genetic independence (Constantino, Hudziak, and Todd 2003; Hoekstra, Bartels, Hudziak, van Beijsterveldt, and Boomsma 2007). From this perspective, autism-related problems constitute a general dimension of individual differences rather than a specific dimension only relevant for a subsample of patients, similar to how internalizing and externalizing problems are conceived of. In order to understand how autism spectrum problems relate to internalizing and externalizing problems, these three broad domains need to be studied simultaneously, preferably in the general population. An advantage of using general population samples to study the dimensional structure of psychopathology is that they represent a clearly defined population, in contrast to clinical samples which depend on specific selection criteria of the institution from which patients are sampled. There are multiple non-exclusive ways in which autismrelated problems could be included into the structure of psychopathology in the general population. First, these problems could load on either the Internalizing or the Externalizing

factor. This would imply that autism-related problems are specific subdomains of internalizing or externalizing psychopathology. Second, autism-related problems could load on both the Internalizing and Externalizing factor. This would imply that autism-related dimensions comprise a mix of internalizing and externalizing features. Third, covariance between autism-related problems may not be captured well by the Internalizing and Externalizing factors. In that case a third higher-order dimension could be introduced, specifically capturing autism-related problems. Fourth, covariance between autism-related problems and internalizing and externalizing psychopathology may be captured by a non-specific general factor. This perspective on covariance of problems can be modeled by using a bi-factor model, as for example advocated by Lahey et al. (2012) and Caspi et al. (2014) for, respectively, childhood and adult psychopathology. Which of these ways of including autism-related problems into a transdiagnostic model is best depends on how well the proposed structure fits with observed data. Therefore, in the current paper multiple alternative models will be tested against data from a general population sample. While developing and comparing these models, two empirical questions will be central. First, do we find in the general population a specific dimension related to autism-related problems, or does it suffice to capture covariation of problems in the general population by Internalizing and Externalizing factors? Second, if such a dimension of autism-related problems can be discerned, how is this dimension situated relative to internalizing and externalizing psychopathology?

Method Sample Subjects were participants in the Tracking Adolescents’ Individual Lives Survey (TRAILS), a prospective multi-cohort study of Dutch (pre)adolescents. The study involved a representative sample from the general population and is described in detail in De Winter et al. (2005), Huisman et al. (2008) and Ormel et al. (2012). Briefly, the target sample involved all 10to 11-year-old children living in the three largest cities and some rural areas in the North of The Netherlands. Of the eligible children, 76.0 % (n=2,230, mean age = 11.09, SD =0.55, 50.8 % female) were enrolled in the study. Responders and non-responders did not differ regarding the prevalence of teacher-rated problem behavior and associations between sociodemographic variables and mental health indicators (De Winter et al. 2005). To date, the population cohort has been assessed four times (T1: March 2000- July 2001, T2: September 2003- December 2004, T3: September 2005-December 2007, T4: October 2008-September 2010). T1, T2, and T3 data are used in the present study. Participation rates were

J Abnorm Child Psychol

96.4 % at T2 (mean age = 13.55, SD=0.53), and 81.4 % at T3 (mean age = 16.25, SD=0.73). After complete description of the study to the subjects, written informed consent was obtained from the parents at each assessment wave and from the adolescents at T2 and T3. Instruments Child Behavior Checklist (CBCL) The Dutch version of the Child Behavior Checklist (CBCL; Achenbach 1991; Verhulst, van der Ende, and Koot 1996) was used to assess internalizing and externalizing psychopathology. The CBCL is a 112-item questionnaire on which parents rate descriptions of emotions and behaviors on a 3-point scale (0 (not), 1 (sometimes), or 2 (very often)). These items comprise 8 subscales, 6 of which were used for the current study: Anxious-Depressed (Anx, 13 items, α=.78), Somatic complaints (Sc, 11 items, α=0.69), Withdrawn-Depressed (Wd, 8 items, α=0.71), Aggressive Behavior (Agg, 18 items, α=0.88), Rule-Breaking behavior (Rb, 17 items, α=0.68), and Attention Problems (Att, 10 items, α=0.82). These subscales were chosen, because they are assumed to be indicators of internalizing (Anx, Sc, Wd) and externalizing (Agg, Rb) psychopathology. Attention problems also strongly covary with autism-related symptoms (St Pourcain et al. 2011) and are regarded as externalizing psychopathology by some authors (e.g., Lahey et al. 2008). Therefore, we chose to include the Att scale in our analyses. Yet, in the CBCL Att is not regarded as an indicator of externalizing psychopathology (e.g., Achenbach 1991). Therefore, we chose to include it in our models as a separate dimension. We did not include the Thought Problems (TP) and Social Problems (SP) scales in these analyses. The main reason for this choice was that we aimed at integrating ASD in a structure of internalizing and externalizing psychopathology, not so much into the full CBCL structure. Although these scales have been found correlated with an autism-related measure (Social Responsiveness Scale; e.g., Reiersen and Todorov 2013) and specific items from the SP and TP scales have been found to be predictive of Autism Spectrum Disorders (e.g., Ooi, Rescorla, Ang, Woo, and Fung 2011; So et al. 2013), we believe these scales are measures of different domains (see Constantino et al. 2003). Social Problems appears to tap into a wide area of contextual problems (e.g., being teased) which are prevalent in children with both internalizing, externalizing and neurodevelopment problems. With regard to TP, we do think understanding relations between internalizing and externalizing psychopathology and psychotic experiences (e.g., hears things) is an interesting topic in itself, but this was not the goal of the current analysis and we have some doubt whether the small and heterogeneous TP-scale of the CBCL is appropriate to fully investigate this.

Child Social Behavior Questionnaire (CSBQ) The parentrated Child Social Behavior Questionnaire (Hartman, De Bildt, and Minderaa 2013; Hartman, Luteijn, Moorlag, de Bildt, A., and Minderaa, R. 2007; Hartman, Luteijn, Serra, and Minderaa 2006; Luteijn, Jackson, Volkmar, and Minderaa 1998) was used to assess problems that are commonly found in children diagnosed with an ASD. Children with an Autism Spectrum Disorder (ASD) form a heterogeneous group. The Children’s Social Behavior Questionnaire (CSBQ) charts this heterogeneous behavior through 49 items rated by parents. The instrument has a 3-point rating-scale that is equal to the CBCL-format (0 (not), 1 (sometimes), or 2 (very often)). Factor analysis revealed a 6-factor structure: Behavior/ Emotions not Optimally Tuned to the Social Situation (Not tuned, 11 items, α=0.84), Reduced Contact and Social Interests (Reduced Contact, 12 items, α=0.76), Orientation Problems in Time, Place or Activity (Orientation, 8 items, α= 0.78), Difficulties in Understanding Social Information (Social Understanding, 7 items, α=0.75), Stereotyped Behavior (Stereotyped, 8 items, α=0.69), and Fear and Resistance to Change (Fear of Change, 3 items, α=0.74). Several studies have supported the factorial structure (Hartman, Luteijn, Serra, and Minderaa 2006; Luteijn et al. 2000a) and criterion validity of the CSBQ, in children with and without mental retardation (De Bildt et al. 2005; de Bildt et al. 2009; Hartman et al. 2006; Luteijn et al. 2000b). The CSBQ differentiates between (DSM-IV based) autism, PDD-NOS and ADHD, with decreasing scores for these three conditions, respectively (Hartman et al. 2006). It has proven useful in populations other than ASD such as ADHD for characterizing subthreshold ASD problems (Nijmeijer et al. 2008) and delinquent groups (‘t Hart-Kerkhoffs, Jansen, Doreleijers, Vermeiren, Minderaa, and Hartman 2009; Geluk et al. 2012). The validity of the CSBQ also follows from genetic (Nijmeijer et al. 2010), neurocognitive (Blijd-Hoogewys, van Geert, Serra, and Minderaa 2008; Geurts, Luman, and van Meel 2008; Rommelse et al. 2009) and behavioral (de Bildt et al. 2005; Jaspers et al. 2013) studies. Statistical Analyses MPlus version 5.2 was used to explore and test latent variable models of the subscales of the CBCL and the CSBQ. To accommodate for skewness, we transformed all scales by taking their natural logarithms and used maximum likelihood estimation with robust standard errors (MLR), which is relatively robust to deviations from normality. We used the Root Mean Square Error of Approximation (RMSEA) and Comparative Fit Index (CFI) as indicators of model fit. Furthermore, the Bayesian Information Criterion (BIC) was used to compare different models with satisfactory RMSEA (0.95).

J Abnorm Child Psychol

Using both Confirmatory Factor Analysis (CFA) and Exploratory Factor Analysis (EFA), we developed higher-order and bi-factor models which will be explained and illustrated in the results section. We started by developing models based on data from one measurement wave, selected those models that showed adequate model fit, and replicated these models in data from the other measurement waves. We chose T2 as the measurement wave in which to develop models, because we aimed at a model that would be valid in both childhood and adolescence. Choosing the measurement-wave in the middle (T2) would most likely give the best chance of generalizing to the younger (T1) and older age groups (T3). Additional Analyses at Item Level The aim of the main analyses outlined in the former paragraph was to find a theory-based higher order structure of psychopathology based on existing lower order scales of the CBCL and CSBQ. However, at the item level, some overlap between the CBCL and CSBQ exists, and therefore, part of the covariance modeled in higher-order analyses at scale level might actually be due to overlapping item-content rather than truly reflecting higher-order structure. A fully exploratory approach starting at item level data might be useful to investigate the lower order structure further by testing how autism-related symptoms fit with symptoms of internalizing and externalizing problem domains in a less theory-driven way. We performed post hoc additional exploratory factor analyses (EFA) on the basis of items rather than scales to determine if the conclusions of our main analyses would be supported when based on this bottom-up approach. Due to space constraints a brief summary of the results of these explorative analyses will be given here (see Appendix A – online Supplementary Material for complete results).

Results Developing a Higher-Order Model on T2-Data We used CFA and EFA to test specific hypotheses regarding ASD-related problems and their co-occurrence with other problem domains. The analyses proceeded in four steps. Fit indices of these analyses are shown in Table 1. In step one, we tested the hypotheses that the CSBQscales measure a single distinct higher order dimension, which we designated the Autism Spectrum factor (AS), and that comorbidity of AS with internalizing and externalizing problems can be captured by correlations with the Internalizing and Externalizing higher-order factors and the dimension of attention problems. To this end the basic higher-order model (Fig. 1) was tested with the use

of CFA. As shown in Table 1, this model did not fit well to the T2-data (RMSEA>0.05; CFI0.20) that were found in the EFA. A first refinement was to include loadings of CSBQscales on the higher-order factors Internalizing (INT) and Externalizing (EXT). Based on EFA-factors F1 and F2, free loadings were added of the CSBQ-scale Fear of Change on INT and Not Tuned on EXT. As a second refinement, to adapt for the loadings on F3 (see Table 2), a free correlation was added between the residual variance of the CSBQ-scale Reduced Contact and that of the CBCL-scale Withdrawn-Depressed. The third refinement accommodated the fourth EFAfactor in CFA by adding a higher-order factor with loadings of five subscales. This EFA-factor captured covariance between two CBCL (Rule-Breaking Behavior, and Attention Problems) and three CSBQscales (Orientation, Stereotyped, and Social Understanding). We coined this factor AO for Attention and Orientation Problems, as these scales loaded most strongly on the factor. These three additions resulted in the refined higher-order model illustrated in Fig. 2. As shown in Table 1, this model (factor-loadings are shown in Table 3) fitted well to the T2-data (RMSEA = 0.05, CFI > 0.95) and had a lower BIC-value than the basic model. In step four, we tested the hypothesis that the six CSBQscales constitute a separate domain in the general population. For this aim we tested whether a model without a higher order AS-factor fitted the data. This was not the case (Table 1: RMSEA>0.05; CFI 0.90, TLI > 0.90) and distinguished three broad domains of psychopathology: internalizing problems, externalizing problems and a domain including autism-related problems as well as problems in attention and orientation. This is consistent with the

Agg Anx Rb Sc Wd Att Beh Dif Fea Ori Con Ste

F1

F2

F3

F4

F5

0.88 0.80 0.53

0.31

0.48 0.38

0.69 0.64 0.71 0.32

0.34 0.58 0.46 0.21

0.34 0.40 0.42 0.42 0.46 0.44

Only loadings >0.2 are reported. Factor Fx refers to the x-th factor derived in the EFA-solution; Abbreviations are explained in Fig. 1

Fitting Models on T1 and T3 Data After selection of a higher-order model (Fig. 2) and a bifactor model (Fig. 3), we tested their fit to T1 and T3 data. As shown in Table 1, CFI fit indices were adequate for all models (CFI > 0.95). Only the refined bi-factor model with AO* showed completely satisfactory RMSEA indices (RMSEA < 0.05). Furthermore, for this model BIC-values were lowest at all occasions. As can be observed in Table 4, standardized loadings of subscales on the NS-factor were substantial to high (between 0.40 and 0.77). As an indicator of the importance of this non-specific factor we computed McDonald’s omega (Revelle and Zinbarg 2009) which is an indicator of general factor saturation. A rather high value was found (Ωh=0.75), indicating that indeed general factor variance is rather important in these data. Subsequently, we tested for model-invariance over these three measurement waves, assuming equal factor-loadings and thresholds (i.e., strong invariance). The bi-factor model showed adequate fit-indices, while the higher-order model did not show completely satisfactory fit (RMSEA = 0.064) and higher BICvalues (see Table 1). Altogether, these results indicate a small, but consistent, superiority of the bi-factor model. Fig. 2 The refined higher order model with inclusion of a thirdorder factor. Note: Abbreviations are explained in Fig. 1, AO = Attention and Orientation Problems

EXT

agg

rb

AO

INT

wd

anx

sc

att

AS

beh

con

ori

ste

fea

dif

J Abnorm Child Psychol Table 3 Factor loadings of the hierarchical model for T2-data and explained variance for each subscale EXT Loadings Agg 1.00 Anx Rb 0.45 Sc Wd Att Beh 0.61 Dif Fea Ori Con Ste Correlations EXT INT AO AS

INT

AO

R2

AS

0.97 1.00

0.82

0.76 0.56

0.50 0.69

0.54

0.54

1.00

0.23

0.63 0.64 0.54

0.94 0.67 0.54

0.26

0.85

0.62 0.65

0.32

0.75

0.38

0.19

0.12

0.32

0.56 0.63

analyses at scale level by showing that 1) in a 3factor model autism-related problems load onto a specific domain separate from internalizing and externalizing problems, and 2) further refinement up to a 9factor model yields clearly distinguishable factors related to internalizing problems, externalizing problems, autism-spectrum problems and problems in attention and orientation.

1.00 1.33 0.62 1.18 1.95 1.01

0.32 0.51 0.37 0.47 0.69 0.51

0.25 0.48 0.73 0.71 0.54 0.38 0.57 0.48 0.34

Discussion In this study problems related to the autism-spectrum were integrated into the model of internalizing and externalizing psychopathology. In all adequately fitting models autism related problems were part of a specific domain of psychopathology that could be distinguished from the internalizing and externalizing domains. Optimal model fit was found for a bifactor model with one non-specific factor and four specific factors related to internalizing, externalizing, attention and orientation, and autism spectrum problems. This held for childhood, early adolescence, and late adolescence. The first goal of these analyses was to test whether autismrelated problems in the general population could be regarded as part of the internalizing or externalizing domains of psychopathology, or alternatively constituted a distinguishable third domain. The results clearly point out that neither Internalizing or Externalizing factors, nor a non-specific general factor, were sufficient for capturing autism-related problems. Instead an Autism-Spectrum factor was found, supporting the idea that these symptoms are indeed part of a specific domain. The second goal was to find out how a dimension of autism-related problems was related to internalizing and externalizing psychopathology. The most important result pertaining to this question was the finding of a very substantial non-specific factor. This finding is comparable to results of Lahey et al. (2012) and Caspi et al. (2014). This large general factor, present in many measures of psychopathology, implies

0.56

Residual Correlation Wd with Con = 0.48 Standardized loadings and correlations are shown in italics, EXT Externalizing, INT Internalizing, AO Attention and Orientation Problems, AS Broader Autism Phenotype, Other abbreviations are explained in Fig. 1

hypothesis that autism-related problems are part of a specific domain of psychopathology that can be distinguished from the internalizing and externalizing domains. However, it also suggests that this domain might be broader than just autism-related symptoms. In all, the results of these additional analyses at the item level in the general population were highly consistent with previous simultaneous analyses of CBCL and CSBQ items in a clinical sample (Hartman et al. 2006). They additionally supported our current main Fig. 3 The bi-factor model based on adding an NS-factor to the ‘refined higher-order model’ and fixing correlations between factors at zero. Note: Abbreviations are explained in Figure 1. A * refers to a factor in the bi-factor that corresponds to, but is not equivalent to, the factor with the same name in the higherorder models, AO = Attention and Orientation Problems

EXT*

agg

rb

AO

INT*

wd

anx

sc

att

AS*

beh

NS

con

ori

ste

fea

dif

J Abnorm Child Psychol Table 4 Factor loadings of the final bi-factor model for T2-data and explained variance for each subscale NS Agg Anx Rb Sc Wd Att Beh Dif Fea Ori Con Ste

1.00 0.71 0.67 0.41 0.60 0.41 0.88 0.68 0.30 0.58 0.63 0.36

EXT* 0.77 0.63 0.64 0.40 0.59 0.68 0.70 0.65 0.45 0.58 0.55 0.45

1.00 0.52

0.74

INT*

AO*

AS*

R2

1.00 1.01 0.76 1.27 1.39 0.88

0.89 0.71 0.58 0.25 0.47 0.57 0.74 0.53 0.43 0.73 0.46 0.34

0.55 1.00

0.56

0.47 0.56

0.35 0.29

0.35

1.15

0.22

1.00

0.33

0.42 0.33

0.71

0.13

2.40

0.48

0.53

0.13

0.31

0.26 0.31 0.37 0.41 0.39 0.35

Residual Correlation Wd with Con = 0.56 Standardized loadings are shown in italics, NS Non Specific, EXT* Externalizing, INT* Internalizing, AO* Attention and Orientation problems, AS* Broader Autism Phenotype, Other abbreviations are explained in Fig. 1

that a substantial part of the variance in item-responses may not capture a specific kind of psychopathology, but rather indicates non-specific (parent-reported) problem-severity. In addition to general problem severity, there are multiple other possible mechanisms underlying such a general factor (see also Lahey et al. 2012), including non-specific item-content (e.g., feeling unhappy - being a result of most specific types of psychopathology), direct causal interactions between symptoms (e.g., Cramer, Waldorp, van der Maas, and Borsboom 2010; e.g., worry leading to sleep problems leading to anger and so forth), or the effect of using the perspective of one specific informant (e.g., the non-specific factor may actually measure parental concern). Although not the prime target of this paper, a third conclusion that can be drawn is that the CBCL-scale Attention Problems is not captured by the externalizing nor the internalizing domain, in accordance with the scale-structure of the CBCL (Achenbach 1991). In the scale-based confirmatory models, the Attention Problem scale was part of a fourth factor, which we labeled Attention and Orientation problems on the basis of its highest loading scales. A hypothetical interpretation of this factor based on the item content of both scales is that it captures individual differences in cognitive control problems. Note however, that in the post-hoc itemlevel analyses, the distinction between an Autism Spectrum factor (AS) and an Attention and Orientation factor (AO) only appeared in models with eight factors or more, suggesting that AO and AS are strongly intertwined. This is in line with a large body of literature on the (genetic and neurocognitive) overlap between ASD and ADHD (Rommelse, Franke, Geurts, Hartman, and Buitelaar 2010; Rommelse, Geurts, Franke, Buitelaar, and Hartman 2011).

The utility of the bi-factor model for clinical practice is that it provides a model of psychopathology that corresponds to empirical data and can be used to organize diagnostic descriptions. A well-known problem with syndrome-specific approaches is that they are either too specific, thereby losing track of those transdiagnostic features that are shared by large groups of patients, or too broad, thereby insufficient for capturing within-group heterogeneity. In the proposed bi-factor model a balance is found between these issues by the inclusion of a general factor, as well as broad domains of psychopathology (internalizing, externalizing, Autism Spectrum, Attention and Orientation). Importantly, the presence of broad higher-order factors by no means implies that lowerorder factors are unimportant in the general population and certainly not that making fine-grained distinction would be unnecessary in clinical practice. On the contrary: the Autism-spectrum, just like the internalizing and externalizing domains, was modelled as a multidimensional construct, in accordance with the multidimensional structure of the CSBQ. In the general population, broad domains of psychopathology only captured a proportion of the variance in more specific scales (see Table 4). For clinical populations (e.g., specialized care for Autism Spectrum Disorder) it can be expected that specific factors become even more important relative to broad factors. We believe it may be advantageous to include autismrelated problems, or a broader domain of neurodevelopmental problems, into the model of internalizing and externalizing psychopathology, as well as in the DSM-5 cross-dimensional system (Andrews et al. 2009). When using a fully

J Abnorm Child Psychol

transdiagnostic dimensional approach, all individuals, whether they meet certain thresholds or not, are described in terms of the extent to which they are affected in broad domains of psychopathology. With regard to autism-related problems and ASD this has some advantages. First, the dimension of autism-related problems may be relevant for children who do not pass the diagnostic threshold for autism-spectrum disorder (ASD). For example, social communication problems have been shown to be highly prevalent in children with ADHD. The existence of such problems may be of importance for treatment planning. Second, specifically with regard to children with a diagnosis of ASD, it is important to have a realistic perspective on possibilities for change. While complete remission from ASD seems unlikely, moderate changes may be feasible along autism-related dimensions, and substantial progress may be achieved in other psychopathology domains, such as aggression, anxiety, or attention problems. Therefore, it may be useful to describe progress and stability along multiple, hierarchically organized, dimensions. The results should be interpreted with some limitations of the research-design and the models used in mind. First, the results are based on parent-reported complaints using questionnaires, which only provide one source of information that may be influenced by parent-specific perspectives, contexts and biases. Using additional informants or interview-data could be useful in subsequent research. Second, the results are based on analyses in only one general population sample of (pre)adolescents (10–17 years of age) in the north of the Netherlands. While the model replicated in different age samples, it should be tested whether findings generalize to other samples (e.g., different ages, clinical samples). Third, a disadvantage of factor analytic models is the assumption that domains of psychopathology are only correlated due to common factors. This assumption may be at odds with possible direct causal relations between psychiatric symptoms (see Cramer, Waldorp, van der Maas, and Borsboom 2010). Therefore, a strong causal interpretation (e.g., problems are directly caused by a general factor) of our factor-analytic results may not be warranted. These cautionary remarks with regard to underlying causal structure notwithstanding, we nevertheless believe the bi-factor model presented in this paper offers a parsimonious approach for capturing co-occurrence and heterogeneity in reported problems in the general population.

Conflict of Interest The authors declare that they have no conflict of interest.

References Achenbach, T. M. (1966). Classification of children’s psychiatric symptoms - a factor-analytic study. Psychological Monographs, 80, 1–37.

Achenbach, T. M. (1991). Manual for the child behavior checklist/4-18 and 1991 profile. Burlington: University of Vermont, Department of Psychiatry. American Pychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th edition). VA: Arlington. Andrews, G., Goldberg, D. P., Krueger, R. F., Carpenter, W. T., Hyman, S. E., Sachdev, P., et al. (2009). Exploring the feasibility of a metastructure for DSM-V and ICD-11: could it improve utility and validity? Psychological Medicine, 39, 1993–2000. Blijd-Hoogewys, E. M., van Geert, P. L., Serra, M., & Minderaa, R. B. (2008). Measuring theory of mind in children. Psychometric properties of the ToM storybooks. Journal of Autism and Developmental Disorders, 38, 1907–1930. Caspi, A., Houts, R. M., Belsky, D. W., Goldman-Mellor, S. J., Harrington, H. L., Israel, A., et al. (2014). The p Factor: one general psychopathology factor in the structure of psychiatric disorders? Clinical Psychological Science, 2, 119–137. Constantino, J. N. (2011). The quantitative nature of autistic social impairment. Pediatric Research, 69, 55R–62R. doi:10.1203/PDR. 0b013e318212ec6e. Constantino, J. N., & Todd, R. D. (2003). Autistic traits in the general population — a twin study. Archives of General Psychiatry, 60, 524–530. Constantino, J. N., Hudziak, J. J., & Todd, R. D. (2003). Deficits in reciprocal social behavior in male twins: Evidence for a genetically independent domain of psychopathology. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 458–467. doi:10. 1097/01.chi.0000046811.95464.21. Constantino, J. N., Lajonchere, C., Lutz, M., Gray, T., Abbacchi, A., McKenna, K., et al. (2006). Autistic social impairment in the siblings of children with pervasive developmental disorders. American Journal of Psychiatry, 163, 294–296. Cramer, A. O. J., Waldorp, L. J., Van der Maas, H. L. J., & Borsboom, D. (2010). Comorbidity: A network perspective. Behavioral and Brain Sciences, 33, 137–150. doi:10.1017/s0140525x09991567. De Bildt, A., Serra, M., Luteijn, E., Kraijer, D., Sytema, S., & Minderaa, R. (2005). Social skills in children with intellectual disabilities with and without autism. Journal of Intellectual Disability Research, 49, 317–328. doi:10.1111/j.1365-2788.2005.00655.x. De Bildt, A., Mulder, E. J., Hoekstra, P. J., van Lang, N. D. J., Minderaa, R. B., & Hartman, C. A. (2009). Validity of the children’s social behavior questionnaire (CSBQ) in children with intellectual disability: Comparing the CSBQ with ADI-R, ADOS, and clinical DSMIV-TR classification. Journal of Autism and Developmental Disorders, 39, 1464–1470. doi:10.1007/s10803-009-0764-x. De Winter, A., Oldehinkel, A. J., Veenstra, R., Brunnekreef, J. A., Verhulst, F. C., & Ormel, J. (2005). Evaluation of non-response bias in mental health determinants and outcomes in a large s a m p l e o f p r e - a d o l e s c e n t s . E u ro p e a n J o u r n a l o f Epidemiology, 20, 173–181. Eaton, N., South, S., & Krueger, R. F. (2010). The meaning of comorbidity among common mental disorders. In T. Millon, R. Krueger, & E. Simonsen (Eds.), Contemporary directions in psychopathology: Scientific foundations of the DSM-V and ICD-11 (pp. 223–241). New York: Guilford Press. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299. doi:10.1037// 1082-989x.4.3.272. Frazier, T. W., Youngstrom, E. A., Sinclair, L., Kubu, C. S., Law, P., Rezai, A., et al. (2010). Autism spectrum disorders as a qualitatively distinct category from typical behavior in a large, clinically ascertained sample. Assessment, 17, 308–320. doi:10.1177/ 1073191109356534. Frazier, T. W., Youngstrom, E. A., Speer, L., Embacher, R., Law, P., Constantino, J., et al. (2012). Validation of proposed DSM-5 criteria

J Abnorm Child Psychol for autism spectrum disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 51, 28–40. doi:10.1016/j.jaac. 2011.09.021. Geluk, C. A. M. L., Jansen, L. M. C., Vermeiren, R., Doreleijers, T. A. H., van Domburgh, L., de Bildt, A., et al. (2012). Autistic symptoms in childhood arrestees: Longitudinal association with delinquent behavior. Journal of Child Psychology and Psychiatry, 53, 160–167. Geurts, H. M., Luman, M., & van Meel, C. S. (2008). What’s in a game: The effect of social motivation on interference control in boys with ADHD and autism spectrum disorders. Journal of Child Psychology and Psychiatry, 49, 848–857. Greaves-Lord, K., Eussen, M. L., Verhulst, F. C., Minderaa, R. B., Mandy, W., Hudziak, J. J., et al. (2013). Empirically based phenotypic profiles of children with pervasive developmental disorders: Interpretation in the light of the DSM-5. Journal of Autism and Developmental Disorders, 43, 1784–1797. Hartman, C. A., Luteijn, E., Serra, M., & Minderaa, R. (2006). Refinement of the children’s social behavior questionnaire (CSBQ): An instrument that describes the diverse problems seen in milder forms of PDD. Journal of Autism and Developmental Disorders, 36, 325–342. doi:10.1007/s10803-005-0072-z. Hartman, C. A., Luteijn, E., Moorlag, A., de Bildt, A., & Minderaa, R. (2007). Manual for the CSBQ [Handleiding voor de VISK]. Amsterdam: Harcourt. Hartman, C. A., De Bildt, A., & Minderaa, R. B. (2013). CSBQ children’s social behavior questionnaire. In F. R. Volkmar (Ed.), Encyclopedia of Autism Spectrum Disorders (pp. 825–829). New York: Springer. Hoekstra, R. A., Bartels, M., Hudziak, J. J., van Beijsterveldt, T., & Boomsma, D. I. (2007). Genetic and environmental covariation between autistic traits and Behavioral problems. Twin Research and Human Genetics, 10, 853–860. Huisman, M., Oldehinkel, A. J., de Winter, A., Minderaa, R. B., de Bildt, A., Huizink, A. C., et al. (2008). Cohort profile: The Dutch TRacking Adolescents Individual Lives Survey; TRAILS. International Journal of Epidemiology, 37, 1227–1235. doi:10.1093/ije/dym273. Jaspers, M., de Winter, A. F., Buitelaar, J. K., Verhulst, F. C., Reijneveld, S. A., & Hartman, C. A. (2013). Early childhood assessments of community pediatric professionals predict autism spectrum and attention deficit hyperactivity problems. Journal of Abnormal Child Psychology, 41, 71–80. Krueger, R. F. (1999). The structure of common mental disorders. Archives of General Psychiatry, 56, 921–926. Lahey, B. B., Rathouz, P. J., Van Hulle, C., Urbano, R. C., Krueger, R. F., Applegate, B., et al. (2008). Testing structural models of DSM-IV symptoms of common forms of child and adolescent psychopathology. Journal of Abnormal Child Psychology, 36, 187–206. doi:10. 1007/s10802-007-9169-5. Lahey, B. B., Applegate, B., Hakes, J. K., Zald, D. H., Hariri, A. R., & Rathouz, P. J. (2012). Is there a general factor of prevalent psychopathology during Adulthood? Journal of Abnormal Psychology, 121, 971–977. doi:10.1037/a0028355. Luteijn, E. F., Jackson, A. E., Volkmar, F. R., & Minderaa, R. B. (1998). The development of the children’s social behavior questionnaire: Preliminary data. Journal of Autism and Developmental Disorders, 28, 559–565. Luteijn, E., Luteijn, F., Jackson, S., Volkmar, F., & Minderaa, R. (2000a). The children’s social behavior questionnaire for milder variants of PDD problems: Evaluation of the psychometric characteristics. Journal of Autism and Developmental Disorders, 30, 317–330. Luteijn, E. F., Serra, M., Jackson, A. E., Steenhuis, M. P., Althaus, M., Volkmar, F. R., et al. (2000b). How unspecified are disorders of children with a pervasive developmental disorder not otherwise specified? A study of social problems in children with PDDNOS and or ADHD. European Child and Adolescent Psychiatry, 9, 168–179.

Mandy, W. P. L., Charman, T., & Skuse, D. H. (2012). Testing the construct validity of proposed criteria for DSM-5 Autism Spectrum Disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 51, 41–50. doi:10.1016/j.jaac.2011.10.013. Markon, K. E. (2010). Modeling psychopathology structure: A symptomlevel analysis of Axis I and II disorders. Psychological Medicine, 40, 273–288. doi:10.1017/s0033291709990183. Mulligan, A., Anney, R. J. L., O’Regan, M., Chen, W., Butler, L., Fitzgerald, M., et al. (2009). Autism symptoms in attention-deficit/ hyperactivity disorder: A familial trait which correlates with conduct, oppositional defiant, language and motor disorders. Journal of Autism and Developmental Disorders, 39, 197–209. doi:10.1007/ s10803-008-0621-3. Nijmeijer, J. S., Hoekstra, P. J., Minderaa, R. B., Buitelaar, J. K., Altink, M. E., Buschgens, H., et al. (2008). PDD symptoms in ADHD, an independent familial trait? Journal of Abnormal Child Psychology, 37, 443–453. Nijmeijer, J. S., Hartman, C. A., Rommelse, N. J., Altink, M. E., Buschens, C. J. M., Fliers, H., et al. (2010). Perinatal risk factors interacting with Catechol O-Methyltransferase and the Serotonin transporter gene predict ASD symptoms in children with ADHD. Journal of Child Psychology and Psychiatry, 51, 1242–1250. Ooi, Y. P., Rescorla, L., Ang, R. P., Woo, B., & Fung, D. S. S. (2011). Identification of autism spectrum disorders using the child behavior checklist in Singapore. Journal of Autism and Developmental Disorders, 41, 1147–1156. Ormel, J., Oldehinkel, A. J., Sijtsema, J., van Oort, F., Raven, D., Veenstra, R., et al. (2012). The TRacking Adolescents’ Individual Lives Survey (TRAILS): design, current status, and selected findings. Journal of the American Academy of Child and Adolescent Psychiatry, 51, 1020–1036. Reiersen, A. M., & Todorov, A. (2013). Exploration of ADHD subtype definitions and co-occurring psychopathology in a Missouri population-based large sibship sample. Scandinavian Journal of Child and Adolescent Psychiatry and Psychology, 1, 3–13. Reiersen, A. M., Constantino, J. N., Grimmer, M., Martin, N. G., & Todd, R. D. (2008). Evidence for shared genetic influences on selfreported ADHD and autistic symptoms in young adult Australian twins. Twin Research and Human Genetics, 11, 579–585. Revelle, W., & Zinbarg, R. E. (2009). Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145–154. Rommelse, N., Altink, M. E., Fliers, E. A., Martin, N. C., Buschgens, C. J., Hartman, C. A., et al. (2009). Comorbid problems in ADHD: Degree of association, shared endophenotypes, and formation of distinct subtypes. Implications for a future DSM. Journal of Abnormal Child Psychology, 37, 793–804. Rommelse, N. N., Franke, B., Geurts, H. M., Hartman, C. A., & Buitelaar, J. K. (2010). Shared heritability of attention-deficit/hyperactivity disorder and autism spectrum disorder. European Child and Adolescent Psychiatry, 19, 281–295. Rommelse, N. N., Geurts, H. M., Franke, B., Buitelaar, J. K., & Hartman, C. A. (2011). Review on cognitive and brain endophenotypes that may be common in autism spectrum disorder and attention-deficit/hyperactivity disorder and facilitate the search for pleiotropic genes. Neuroscience and Biobehavioral Reviews, 35, 1363–1396. So, P., Greaves-Lord, K., van der Ende, J., Verhulst, F. C., Rescorla, L., & de Nijs, P. F. A. (2013). Using the Child Behavior Checklist and the Teacher’s Report Form for identification of children with autism spectrum disorders. Autism, 17, 595–607. doi:10.1177/ 1362361312448855. St Pourcain, B., Mandy, W. P., Heron, J., Golding, J., Smith, G. D., & Skuse, D. H. (2011). Links Between Co-occurring social-communication and hyperactive-inattentive trait trajectories. Journal of the American Academy of Child and Adolescent Psychiatry, 50, 892– 902. doi:10.1016/j.jaac.2011.05.015.

J Abnorm Child Psychol ’t Hart-Kerkhoffs, L. A., Jansen, L. M., Doreleijers, T. A., Vermeiren, R. A., Minderaa, R. B., & Hartman, C. A. (2009). Autism spectrum disorder symptoms in juvenile suspects of sex offenses. The Journal of Clinical Psychiatry, 70, 266–272. Verhulst, F. C., van der Ende, J., & Koot, H. (1996). Handleiding voor de CBCL/4-18. [Manual for the CBCL/4-18]. Rotterdam: Afdeling Kinder- en jeugdpsychiatrie, Sophia Kinderziekenhuis/Academisch Ziekenhuis Rotterdam/Erasmus. Vollebergh, W. A. M., Iedema, J., Bijl, R. V., de Graaf, R., Smit, F., & Ormel, J. (2001). The structure and stability of common mental disorders - The NEMESIS study. Archives of General Psychiatry, 58, 597–603.

Author Note Arjen Noordhof, dep. Interdisciplinary Center for Psychiatric Epidemiology, University Medical Center Groningen, dep. Clinical Psychology, University of Amsterdam; Robert F. Krueger, dep. Psychology, University of Minnesota; Johan Ormel, dep. Interdisciplinary Center for Psychiatric Epidemiology, University Medical Center Groningen; Albertine J. Oldehinkel, dep. Interdisciplinary Center for Psychiatric Epidemiology University Medical Center Groningen, dep. Psychology, University of Groningen; Catharina A. Hartman, dep. Child and Adolescent Psychiatry, University Medical Center Groningen.

The Interdisciplinary Center for Psychiatric Epidemiology has been renamed as Interdisciplinary Center Psychopathology and Emotion regulation. Arjen Noordhof is no longer affiliated to the University Medical Center Groningen. This research is part of the TRacking Adolescents’ Individual Lives Survey (TRAILS). Participating centers of TRAILS include various departments of the University Medical Center and University of Groningen, the Erasmus University Medical Center Rotterdam, the University of Utrecht, the Radboud Medical Center Nijmegen, and the Trimbos Institute, all in the Netherlands. Principal investigators are prof. dr. J. Ormel (University Medical Center Groningen) and prof. dr. F.C. Verhulst (Erasmus University Medical Center). TRAILS has been financially supported by various grants from the Netherlands Organization for Scientific Research NWO (Medical Research Council program grant GBMW 940-38-011; ZonMW Brainpower grant 100-001-004; ZonMw Risk Behavior and Dependence grants 60-60600-98-018 and 60-60600-97118; ZonMw Culture and Health grant 261-98-710; Social Sciences Council medium-sized investment grants GB-MaGW 480-01-006 and GB-MaGW 480-07-001; Social Sciences Council project grants GBMaGW 457-03-018, GB-MaGW 452-04-314, an GB-MaGW 452-06004; NWO large-sized investment grant 175.010.2003.005); the Sophia Foundation for Medical Research (projects 301 and 393), the Dutch Ministry of Justice (WODC), and the participating universities. We are grateful to all adolescents, their parents and teachers who participated in this research and to everyone who worked on this project and made it possible. The authors have no competing interests.