Psychological Bulletin 2015, Vol. 141, No. 4, 858 –900

© 2015 American Psychological Association 0033-2909/15/$12.00 http://dx.doi.org/10.1037/a0038498

The Validity of the Multi-Informant Approach to Assessing Child and Adolescent Mental Health Andres De Los Reyes and Tara M. Augenstein

Mo Wang

University of Maryland

University of Florida

Sarah A. Thomas

Deborah A. G. Drabick, Darcy E. Burgers, and Jill Rabinowitz

University of Maryland This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Temple University Child and adolescent patients may display mental health concerns within some contexts and not others (e.g., home vs. school). Thus, understanding the specific contexts in which patients display concerns may assist mental health professionals in tailoring treatments to patients’ needs. Consequently, clinical assessments often include reports from multiple informants who vary in the contexts in which they observe patients’ behavior (e.g., patients, parents, teachers). Previous meta-analyses indicate that informants’ reports correlate at low-to-moderate magnitudes. However, is it valid to interpret low correspondence among reports as indicating that patients display concerns in some contexts and not others? We meta-analyzed 341 studies published between 1989 and 2014 that reported cross-informant correspondence estimates, and observed low-to-moderate correspondence (mean internalizing: r ⫽ .25; mean externalizing: r ⫽ .30; mean overall: r ⫽ .28). Informant pair, mental health domain, and measurement method moderated magnitudes of correspondence. These robust findings have informed the development of concepts for interpreting multi-informant assessments, allowing researchers to draw specific predictions about the incremental and construct validity of these assessments. In turn, we critically evaluated research on the incremental and construct validity of the multi-informant approach to clinical child and adolescent assessment. In so doing, we identify crucial gaps in knowledge for future research, and provide recommendations for “best practices” in using and interpreting multi-informant assessments in clinical work and research. This article has important implications for developing personalized approaches to clinical assessment, with the goal of informing techniques for tailoring treatments to target the specific contexts where patients display concerns. Keywords: construct validity, incremental validity, informant discrepancies, multiple informants, Operations Triad Model

Luthar, Cicchetti, & Becker, 2000; Sanislow et al., 2010). However, not all contexts elicit mental health concerns to the same degree (e.g., Carey, 1998; Kazdin & Kagan, 1994; Mischel & Shoda, 1995). Therefore, patients may display concerns within some contexts, such as home and school, but not others, such as within peer interactions. In fact, these contextual variations in displays of mental health concerns occur within a variety of mental health domains including social anxiety, attention and hyperactivity, and conduct problems (e.g., Bögels et al., 2010; Dirks, De Los Reyes, Briggs-Gowan, Cella, & Wakschlag, 2012; Drabick, Gadow, & Loney, 2007, 2008; Kraemer et al., 2003). Thus, identifying the specific contexts in which patients display concerns may facilitate treatment planning and boost treatment efficacy (e.g., De Los Reyes, 2013; National Institute of Mental Health [NIMH], 2008). The most prevalent strategy for assessing contextual variations in mental health is the multi-informant assessment approach (Kraemer et al., 2003). Specifically, this approach involves taking reports from informants who share close relationships with the patients about whom they are providing reports, or at minimum spend a significant amount of time observing patients’ behavior (Achenbach, 2006). By considering reports from informants who

Child and adolescent mental health patients lead complex lives (i.e., collectively referred to as “children” unless otherwise specified). Indeed, mental health concerns arise out of an interplay among biological, psychological, and sociocultural factors that pose risk for, or offer protection against, developing maladaptive reactions to environmental or social contexts (e.g., Cicchetti, 1984;

This article was published Online First April 27, 2015. Andres De Los Reyes and Tara M. Augenstein, Comprehensive Assessment and Intervention Program, Department of Psychology, University of Maryland; Mo Wang, Department of Management, University of Florida; Sarah A. Thomas, Comprehensive Assessment and Intervention Program, Department of Psychology, University of Maryland; Deborah A. G. Drabick, Darcy E. Burgers, and Jill Rabinowitz, Department of Psychology, Temple University. This work was partially supported by a predoctoral National Research Service Award to Sarah A. Thomas from the National Institute on Drug Abuse (F31-DA033913). Correspondence concerning this article should be addressed to Andres De Los Reyes, Comprehensive Assessment and Intervention Program, Department of Psychology, University of Maryland, Biology/Psychology Building, Room 3123H, College Park, MD 20742. E-mail: [email protected] 858

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

vary among each other in the specific contexts in which they observe patients’ behavior (e.g., home vs. school vs. peer interactions), mental health professionals may gain an understanding as to how consistently or inconsistently patients display concerns across contexts (Dirks et al., 2012). For child patients, these informants most often include parents, teachers, and patients themselves (Hunsley & Mash, 2007). Further, trained raters might also complete reports, such as clinical interviewers and independent observers of patients’ behavior on standardized clinical tasks (e.g., structured social interactions) or unstandardized home or school observations (Groth-Marnat, 2009). Clinicians may use informants’ reports to make decisions about mental health care, such as assigning diagnoses and planning treatment (e.g., Hawley & Weisz, 2003). Researchers may also use these reports to draw conclusions from empirical work on such topics as identifying treatments that successfully ameliorate mental health concerns (i.e., evidence-based treatments; Weisz, Doss, & Hawley, 2005). In both practice and research settings, collecting multiple informants’ reports generates a great deal of information about patients’ concerns. However, the individual reports often yield inconsistent conclusions (i.e., informant discrepancies; Achenbach, 2006; De Los Reyes & Kazdin, 2004, 2005, 2006; Goodman, De Los Reyes, & Bradshaw, 2010).1 For instance, a female adolescent patient in a pretreatment assessment may be identified as experiencing “low” positive mood based on a parent or teacher report whereas the adolescent self-reports her mood as “elevated.” Informant discrepancies often create considerable uncertainties in delivering services to patients and drawing conclusions from research (De Los Reyes, Kundey, & Wang, 2011). A key reason for these uncertainties originates from the near-exclusive focus in mental health research on whether informant discrepancies reflect measurement error or reporting biases (e.g., De Los Reyes, 2011; Richters, 1992). Consequently, what remains unclear is whether multi-informant approaches to assessment validly capture contextual variations in displays of patients’ mental health concerns, and which informants ought to be included in mental health assessments. As we explain below, these decisions might involve considering data from multiple assessment literatures, chief among them research on multi-informant clinical child assessments and their (a) correspondence, (b) incremental validity, and (c) construct validity. Prior work has addressed each of these issues as they relate to clinical child assessment broadly (e.g., Achenbach, McConaughy, & Howell, 1987; Johnston & Murray, 2003; Mash & Hunsley, 2005), and within assessments in other clinical literatures (e.g., Achenbach, 2006; Campbell & Fiske, 1959; Dawes, 1999; Garb, 2003; Hunsley & Mash, 2005). However, with regard to multi-informant clinical child assessments, these literatures have advanced largely in isolation of one another.

The Present Review The purpose of this article is to review and synthesize research on using and interpreting multiple informants’ reports in clinical child assessments. We expand the literature on this topic in five ways. First, as an update to the seminal meta-analysis on crossinformant correspondence of clinical child assessments carried out by Achenbach, McConaughy, and Howell (1987), we conduct a quantitative review of 341 studies of cross-informant correspon-

859

dence in reports of children’s mental health published over the last 25 years (1989 –2014). Second, we use this updated quantitative review as a backdrop for discussing recent work on the conceptual rationale for conducting multi-informant assessments (De Los Reyes, Thomas, Goodman, & Kundey, 2013). We expand upon this conceptual work by developing specific predictions about the incremental and construct validity of the multi-informant approach. Third, we review research on the incremental validity of multi-informant assessments, or work examining whether a multiinformant approach yields reports that, relative to one another, incrementally contribute information in the prediction of relevant criterion variables (e.g., diagnostic status or treatment response; see Dawes, 1999; Garb, 2003; Hunsley & Meyer, 2003). Fourth, we summarize recent work on the construct validity of multiinformant assessments. By construct validity, we mean research examining whether it is valid to interpret patterns of convergence and divergence among multiple informants’ reports as accurate reflections of contextual variations in children’s mental health concerns (e.g., Borsboom, Mellenbergh, & van Heerden, 2004; Campbell & Fiske, 1959). Lastly, we synthesize work on multiinformant assessments to provide recommendations for both future research and “best practices” for using and interpreting multiinformant assessments administered in clinic settings.

Correspondence in Multi-Informant Clinical Child Assessments One of the most normative observations in assessments of child patients is that of low correspondence between informants’ reports about patients’ mental health (Achenbach, 2011). Over five decades ago, Lapouse and Monk (1958) were the first to report about this phenomenon. Their primary aim was to estimate the prevalence of children’s mental health concerns in an epidemiological study of 482 6- to 12-year-old children representative of the community from which they were sampled (i.e., Buffalo, New York). To estimate prevalence, the researchers relied on structured clinical interviews administered to mothers about a single child from the household. In this study, the researchers also wanted to address what they termed “the problem of validity” with respect to mothers’ interview responses. Thus, the researchers conducted a secondary study using a separate convenience sample of 193 children ages 8 –12 and their mothers, recruited from local outpatient hospitals and pediatricians’ offices. In this sample, researchers conducted structured clinical interviews with the mothers and the identified child separately and simultaneously, using two different interviewers. To assess correspondence between mother and child reports, researchers examined responses on a subset of be1 We use the term informant discrepancies to generally denote studies of differences and similarities among informants’ reports. Yet, researchers have used other terms to characterize this research (e.g., informant agreement and informant correspondence). Importantly, these terms are not interchangeable and, where possible, we use different terms to denote disparate lines of research. That is, these terms distinguish examinations of divergence between reports (discrepancies) from examinations of convergence between reports (agreement and correspondence), and researchers often use disparate statistical techniques within and between studies (e.g., statistical interactions between reports to study divergence vs. correlations between reports to study convergence; De Los Reyes, Salas, Menzer, & Daruwala, 2013).

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

860

DE LOS REYES ET AL.

havioral domains assessed in the interview, including fears, nightmares, bed wetting, restlessness, and repetitive behaviors (e.g., tics, thumb sucking, skin picking). Using percent agreement to assess correspondence, the authors reported agreement between 46% (amount of food intake) and 84% (bed wetting). Broadly, the authors observed greater correspondence between reports of relatively more observable behaviors (e.g., bed wetting, stuttering, thumb sucking) than relatively less observable behaviors (e.g., fears and worries, nightmares, restlessness). Further, with two exceptions (bed wetting and overactivity), lack of correspondence was primarily due to children self-endorsing behaviors that the mother did not endorse about the child. Highlighting the robust nature of cross-informant correspondence patterns, in their seminal meta-analysis of 119 studies of crossinformant correspondence in reports of children’s mental health, Achenbach et al. (1987) reported, similar to Lapouse and Monk (1958), that correspondence levels exhibited a range from low-tomoderate in magnitude (i.e., rs ranging from .20 to .60). Importantly, Achenbach et al. (1987) noted that the low correlations they observed among informants’ reports of children’s mental health did not necessarily indicate that such reports carried poor psychometric qualities, as they also observed satisfactory test–retest reliability estimates for the informants’ reports they examined. In fact, levels of correspondence systematically varied as a function of three key factors. First, pairs of informants who observed children in the same context (e.g., pairs of parents or pairs of teachers) tended to exhibit greater levels of correspondence than pairs of informants who observed children in different contexts (e.g., parent and teacher). Similarly, a second key finding was that greater levels of cross-informant correspondence occurred when informants provided reports about younger children relative to reports about older children. Younger children may be relatively more constrained than older children in the contexts in which they display mental health concerns (e.g., De Los Reyes & Kazdin, 2005; Smetana, 2008). Additionally, Achenbach et al. (1987) conducted their meta-analysis at a time when cross-informant correspondence studies predominantly focused on comparing adults’ reports of children’s concerns (e.g., parent vs. teacher; mother vs. father; see Table 2 of Achenbach et al., 1987). Thus, this age effect might have also reflected the idea that less variation among informants in contexts of observation pointed to greater correspondence between informants’ reports, particularly for adult informants’ reports of children’s concerns. Third, Achenbach et al. (1987) observed larger correspondence levels between informants’ reports of children’s externalizing (e.g., aggression and hyperactivity concerns) versus internalizing (e.g., anxiety and mood) concerns. This third finding may have reflected greater correspondence between reports of directly observable concerns, relative to concerns that are internally experienced by the child and thus, relatively less observable in nature.

anxiety symptoms (Rapee, Barrett, Dadds, & Evans, 1994), and disruptive behavior assessments taken from representative nonclinic samples (Offord et al., 1996). Further, the Lapouse and Monk (1958) finding of greater endorsements among children’s self-reports relative to mothers’ reports was corroborated by comparisons between mother and child behavioral checklist reports from 25 countries (Rescorla et al., 2013). Since the Achenbach et al. (1987) review, researchers have expanded the range of domains examined for cross-informant correspondence (e.g., anxiety, conduct problems, hyperactivity, mood; De Los Reyes, 2011). Further, studies conducted after the Achenbach et al. (1987) review collectively examined to a greater extent variations in cross-informant correspondence across multiple (a) measurement and scaling methods (e.g., behavioral checklists, clinical interviews, symptom rating scales); (b) patients’ developmental periods; and (c) informant pairs (e.g., greater focus on children’s self-reports for assessing internalizing concerns; e.g., Achenbach, 2006; De Los Reyes & Kazdin, 2005). In light of the continued attention to estimating cross-informant correspondence, have the findings of Achenbach et al. (1987) stood the test of time? To address this question, we conducted a meta-analysis of the last 25 years of cross-informant correspondence studies for assessments of children’s internalizing and externalizing mental health concerns. We focused on these domains because they encompass the most commonly assessed and treated childhood concerns (i.e., anxiety, attention and hyperactivity, conduct, mood; Hunsley & Mash, 2007; Weisz et al., 2005). Further, we took two approaches to identifying studies for our review. First, we sampled studies included in meta-analyses of cross-informant correspondence published since Achenbach et al. (1987), which is a variant of the second-order meta-analytic approach (e.g., Butler, Chapman, Forman, & Beck, 2006; Cuijpers & Dekker, 2005; Lipsey & Wilson, 1993; Møller & Jennions, 2002; Peterson, 2001; Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011). Second, we searched for studies published in the years (i.e., 2000 – 2014) following the meta-analyses sampled in our review. Importantly, Achenbach et al. (1987) reviewed 119 studies published over roughly a quarter-century (i.e., 1960 –1986). Similarly, as we explain below, we reviewed 341 studies published in the most recent quarter-century (i.e., 1989 –2014). Thus, we were wellpositioned to assess whether more recent work replicated the findings of Achenbach et al. (1987). To this end, we focused our review on studies that examined correspondence among parent, teacher, and child reports of children’s mental health. We focused on these three informants because these are the reporters on which mental health professionals most commonly rely when administering and interpreting the outcomes of clinical child assessments (e.g., Hunsley & Mash, 2007; Kraemer et al., 2003).

Method Meta-Analysis of the Last Quarter-Century of Cross-Informant Correspondence Studies

Literature Review

Since the seminal work of Lapouse and Monk (1958) and Achenbach et al. (1987), researchers have conducted many additional cross-informant correspondence studies. As in earlier work, more recent studies find that low-to-moderate levels of crossinformant correspondence characterize such varied assessment settings and mental health domains as inpatients’ depressive mood symptoms (Frank, Van Egeren, Fortier, & Chase, 2000), outpatients’

We identified meta-analyses and empirical studies published since Achenbach et al. (1987). We conducted two searches. First, to identify relevant meta-analyses we searched via Google Scholar of all peer-reviewed scholarly work citing the Achenbach et al. (1987) review (N ⫽ 3,978 citations; search conducted March 2, 2014). We searched within these cited articles using the search terms “informant” and “quantitative review.” We augmented this

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

search with an additional Google Scholar search of cited articles using the terms “meta-analysis OR quantitative review OR systematic review,” and conducted this same literature search using the Web of Science search engine. Combined across searches, we identified 1,799 articles. This search yielded an initial set of four quantitative reviews (i.e., Crick et al., 1998; Duhig, Renk, Epstein, & Phares, 2000; Meyer et al., 2001; Renk & Phares, 2004), to which we applied the inclusion and exclusion criteria described below. As an additional check, we conducted Google Scholar searches using the above-listed terms of all work citing these four quantitative reviews, yielding an additional set of 858 articles. This search yielded no additional quantitative reviews. Second, to identify empirical articles of cross-informant correspondence conducted in the years (i.e., between 2000 and 2014) following the meta-analyses included in our review, we searched via Google Scholar for all peer-reviewed scholarly work citing the Achenbach et al. (1987) review between 2000 and 2014 using the search terms “internalizing symptoms/problems/difficulties OR externalizing symptoms/problems/difficulties” (N ⫽ 1,440 citations; search conducted September 9, 2014). Additionally, we searched the reference lists of narrative reviews of cross-informant correspondence research published since 2000.

Inclusion and Exclusion Criteria To be included in our quantitative review, meta-analyses and studies must have (a) focused on informants’ reports of children at or under the age of 18 years; (b) examined correspondence between informants’ reports of children’s mental health concerns (i.e., internalizing and/or externalizing concerns); (c) examined correspondence between reports completed by pairs of parents, teachers, and/or children (i.e., mother–father, parent– child, parent–teacher, teacher– child); and (d) been published in English. In addition to these criteria, included meta-analyses must have provided a list of individual studies used to calculate metrics of cross-informant correspondence. We employed this criterion to ensure that articles examined in our review were published after 1987 (i.e., not included in the original review by Achenbach et al., 1987). Further, knowledge of individual studies within each of the reviews allowed us to identify any articles examined by more than one review and multiple articles that examined the same sample or cohort of children. This step allowed us to ensure that effects observed did not occur because reviews examined the same studies or sample(s) across studies. Among individual empirical studies identified either via the reference lists of meta-analyses or our additional search for individual studies published between 2000 and 2014, to be included in our review studies must have provided sufficient data to code estimates of cross-informant correspondence. Specifically, we required studies to report between-informant correspondence metrics (e.g., Pearson correlations for dimensional measures or Kappa coefficients for categorical measures) on measures completed by one or more informant pairs (e.g., mother–father, parent– child, parent–teacher, teacher– child). Measures completed by these informants must have assessed the same construct at the same time point (e.g., measures about internalizing problems completed by parent and child when the child was 10-years-old). We excluded studies that only reported correspondence metrics as an average or range across multiple informant pairs (e.g., correlations ranged

861

from .15–.40 depending on informant pair), as this information did not allow us to code moderator variables. Similarly, we excluded studies that only reported correspondence metrics as an average or range across informants’ reports of different mental health domains. Finally, studies needed to report cross-informant correspondence estimates for measures of mental health on the internalizing (e.g., anxiety and mood) and/or externalizing (e.g., aggression and hyperactivity) spectrum. We excluded studies focused on related constructs, namely studies on risk and protective factors of mental health (e.g., emotion regulation, parenting practices, personality traits, resiliency, self-esteem). For meta-analyses identified in our search, employing inclusion criteria led to our excluding the Crick et al. (1998) and Renk and Phares (2004) reviews. We excluded Crick et al. (1998) because it did not list the individual studies the authors used to calculate cross-informant correspondence estimates. We excluded the Renk and Phares (2004) review because it focused on correspondence between reports on a construct (i.e., social competence) that fell outside of the spectra of internalizing and externalizing mental health concerns. Thus, we identified relevant studies via quantitative review based on two meta-analyses that met our inclusion criteria: Duhig et al. (2000) and Meyer et al. (2001). Across studies identified via these meta-analyses and our search of individual studies published between 2000 and 2014, collectively we coded effect sizes on a final sample of 1218 data points taken from 341 studies published between 1989 and 2014. A complete study list can be retrieved online at https://sites.google.com/site/caipumaryland/ Home/people/director.

Data Extraction and Covariates Coded From Each Study Two doctoral graduate students served as coders for our quantitative review, and received coding training by the first author and fifth author through discussion and practice coding. After the two coders completed study coding, the fifth author served as an independent assessor on 50% of the studies coded. This independent assessment resulted in 100% interrater agreement on effect sizes coded (i.e., effect size magnitude and metrics), and Cohen’s Kappa coefficients of 1.0 in terms of interrater agreement on coding of each of the covariates described below (i.e., child age, informant pair, measurement method, mental health domain). Outcomes representing mean estimates of cross-informant correspondence were presented as Pearson r correlations, Cohen’s Kappa coefficients, Cohen’s d, Hedge’s g, means and standard deviations of informants’ reports, or odds ratios that we converted to r for the meta-analyses. In addition to mean estimates of cross-informant correspondence, we also coded for four covariates, consistent with effects observed by Achenbach et al. (1987): (a) mental health domain (internalizing vs. externalizing); (b) informant pair (mother–father, parent– child, parent–teacher, teacher– child); (c) child age (younger [10 years and younger] vs. older [11 years and older]); and (d) measurement method (categorical vs. dimensional). We coded studies based on measurement method because recent work indicates that, relative to categorical scaling, dimensional scaling results in measures that evidence greater estimates of reliability and validity (Markon, Chmielewski, & Miller, 2011). Thus, we would expect greater levels of crossinformant correspondence for reports taken using dimensional

DE LOS REYES ET AL.

862

scales (e.g., behavioral checklists), relative to categorical scales (e.g., diagnostic interview endorsements).

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Data-Analytic Strategy As described below, we observed significant differences between cross-informant correspondence estimates for reports of internalizing concerns versus externalizing concerns (see Table 1). Thus, we performed two primary meta-analyses, one for internalizing concerns and one for externalizing concerns, using published or calculated rs to estimate the precision of the mean for all included studies, using Comprehensive Meta-analysis Version 2 (Biostat, Englewood, NJ, n.d.) software. Because the studies included in the meta-analyses varied in methodology and design, we calculated a random-effects model. In addition, for some studies and samples, we observed multiple effect sizes that varied across our moderator variables (e.g., multiple effect sizes reported for different informant pairs, categorical vs. dimensional approaches). We accounted for this nesting in the data by calculating (a) effect sizes for each cohort or sample, and then (b) an overall effect size. Specifically, we computed effect sizes and variances for each cohort. Next, we calculated a weighted mean for each study cohort (i.e., effect sizes drawn from the same sample). We based these weights on the inverse of the total variance associated with each of the data points. Lastly, we computed a weighted mean of the effect sizes for each of the study cohorts, which were based on both within-cohort error and between-cohort variance, to produce an overall summary effect (Borenstein, Hedges, Higgins, & Rothstein, 2009). This method allowed us to capitalize on the multiple sources of variance present both within and across studies, rather than alternative methods that would have resulted in lost sources of variance (e.g., taking a simple average of correspondence estimates for a study that included multiple informant pairs). Additionally, we addressed the issue of statistical stability in two ways. First, we individually excluded each study from the analysis and recalculated the pooled r and 95% confidence interval (CI). If an individual study contributed heavily to the pooled r, we would observe a change in the magnitude or significance of the pooled r. Second, given the possibility of publication bias (i.e., significant findings are more likely to be published), we calculated Orwins’ Fail-safe N (Borenstein et al., 2009), which provides an index of the number of data points necessary to make the overall effect size trivial (i.e., defined as an r of .10). Following determination of the combined effect size across studies, we considered heterogeneity or observed variance across studies. Specifically, we calculated tau, a metric that is sensitive to the unit of measurement and acts as the standard deviation of the summary effect, and I2, which provides an index of the proportion of observed variability that is attributable to heterogeneity among

the data points and reflects “real” differences among studies (Borenstein et al., 2009; Higgins, Thompson, Deeks, & Altman, 2003). To explore the impact of the categorical variables or covariates on this observed variability among studies, we conducted ancillary, subgroup analyses to determine effect sizes (rs) for each level of the categorical variable (e.g., separately for studies that used categorical vs. dimensional assessment approaches or examined younger vs. older children). For these analyses, we calculated rs and associated p values for each level of the categorical variable (Borenstein et al., 2009). We then examined the Q statistic, which is based on the weighted sum of squares for each level of the covariate and thus provides an index of dispersion (Borenstein et al., 2009), to determine whether the magnitude of correspondence differed between levels of the covariates.

Results and Discussion Overall Effect Sizes Combined across 341 studies and 1,218 data points, we observed an overall cross-informant correlation of .28 (95% CI [.22, .33]; p ⬍ .001). These estimates are virtually identical to the mean cross-informant correlations reported by Achenbach et al. (1987). In Table 1, we report cross-informant correlations for reports of children’s internalizing and externalizing concerns. Consistent with Achenbach et al. (1987), we observed low-to-moderate and statistically significant magnitudes of cross-informant correspondence for both domains. As seen in Table 1, we observed nonoverlapping 95% CI for crossinformant correspondence estimates for reports of children’s internalizing concerns versus externalizing concerns. Overall, informant correspondence rates tended to be larger for reports of children’s externalizing concerns, relative to reports of children’s internalizing concerns. Thus, below we report effects of covariates on levels of cross-informant correspondence separately for reports of children’s internalizing concerns and externalizing concerns.

Heterogeneity in Effect Sizes and Stability of Results Consistent with Achenbach et al. (1987), we observed an overall large variance in results for internalizing and externalizing concerns, respectively (I2 ⫽ 99.25 and 99.40), suggesting heterogeneity among studies in effect sizes. That is, the percentage of total variability that is attributable to heterogeneity among the data points included in the meta-analysis is approximately 99% for reports of internalizing and externalizing concerns. Further, removal of any individual study from the analysis did not affect relations between magnitudes of cross-informant correspon-

Table 1 Meta-Analytic Findings Regarding Cross-Informant Correspondence in Reports of Children’s Mental Health Mean cross-informant correlations Symptom type

# of data points

Overall effect size (95% CI)

P-C

P-T

T-C

M-F

Internalizing Externalizing

697 521

.25 [.24, .25] (p ⬍ .001) .30 [.30, .31] (p ⬍ .001)

.26 (n ⫽ 372) .32 (n ⫽ 198)

.21 (n ⫽ 155) .28 (n ⫽ 162)

.20 (n ⫽ 62) .29 (n ⫽ 53)

.48 (n ⫽ 109) .58 (n ⫽ 109)

Note. Correlations are in the Pearson r metric. P-C ⫽ Parent–Child; P-T ⫽ Parent–Teacher; T-C ⫽ Teacher–Child; M-F ⫽ Mother–Father.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

dence and covariates (rs with each individual study removed ranged from .27 to .28 for reports of internalizing concerns; r with each individual study removed was .36 for reports of externalizing concerns; all ps ⬍ .001). Using the respective effect sizes for these data points, defining a trivial effect size as an r of .10, and a threshold of p ⬍ .05, Orwin’s Fail-safe N was 1,875 for reports of internalizing concerns and 2,292 for reports of externalizing concerns. This indicates that one would have to include in the metaanalysis over 1,800 data points of reports of children’s internalizing concerns and well over 2,000 data points of reports of children’s externalizing concerns with a mean r of .00 before the cumulative effects would become trivial.

Effects of Covariates To explore whether the covariates could account for some of the variability among the effect sizes for each of the data points (697 for internalizing, 521 for externalizing), we conducted ancillary analyses where we calculated separate r and p values for each level of the categorical covariates. Because the data points within studies and cohorts often differed in these covariates (e.g., child age, measurement method), these supplementary analyses considered the data points independently. Beyond the effects of mental health domain reported previously, we observed significant effects for two other covariates. First, pairs of mother and father informants (i.e., informants who observe child in the same setting) yielded larger magnitudes of cross-informant correspondence, relative to all other informant pairs (i.e., parent– child, parent–teacher, teacher– child). We observed this effect for both reports of internalizing concerns (rs: .48 vs. .24; Q ⫽ 72.42, p ⬍ .001) and externalizing concerns (rs: .58 vs. .30; Q ⫽ 96.26; p ⬍ .001). Second, informants completing reports on a dimensional scale tended to yield higher magnitudes of correspondence, relative to informants completing reports on a categorical scale. We observed this effect of measurement method for both reports of internalizing concerns (rs: .29 vs. .06; Q ⫽ 38.54, p ⬍ .001) and externalizing concerns (rs: .37 vs. .06; Q ⫽ 30.86, p ⬍ .001). Further, we observed nonsignificant effects of child age on levels of correspondence for reports about younger children (i.e., 10 years and younger) versus older children (i.e., 11 years and older) for both reports of internalizing concerns (rs: .32 vs. .26; Q ⫽ 1.42, p ⫽ .23) and externalizing concerns (rs: .38 vs. .35; Q ⫽ 0.48, p ⫽ .49). In sum, we made three findings consistent with previous reviews (Achenbach et al., 1987; Markon et al., 2011). First, informants’ reports of relatively more observable or externalizing concerns tended to evidence greater levels of cross-informant correspondence, relative to reports of internalizing concerns. Second, informant pairs for which both informants observed the child in the same setting (i.e., mother– father) tended to yield the highest levels of correspondence, relative to all other informant pairs. Third, when informants completed reports using measures that tended to evidence greater reliability and validity estimates (i.e., dimensional), we observed greater levels of crossinformant correspondence, relative to when informants completed reports using categorical measures. One finding from our review that was inconsistent with the Achenbach et al. (1987) meta-analysis was our null finding regarding the effects of child age on magnitudes of cross-informant correspondence. These inconsistencies may reflect changes since Achenbach et al. (1987) with regard to evidence-based assessment

863

practices for clinical child assessments. Specifically, the Achenbach et al. (1987) review consisted of a relatively small proportion of studies comparing parent and teacher reports to children’s self-reports (see Table 2 of Achenbach et al., 1987). Since 1987, a proliferation of evidence-based assessment research focused on understanding children’s perspectives on their own mental health concerns, and this increased focus resulted in an increased number of options available for taking self-reports of mental health, particularly for older children and adolescents (for reviews, see Klein, Dougherty, & Olino, 2005; McMahon & Frick, 2005; Silverman & Ollendick, 2005). Consequently, we based our cross-informant correspondence estimates for reports of children’s mental health concerns on a sample of studies in which over 50% compared children’s self-reports with reports from other informants (see Table 1). For some mental health domains and contexts (e.g., worry and anxiety displayed within peer interactions; covert delinquent behaviors displayed within peer interactions), children may be in a unique position to observe displays of these concerns, relative to parents and teachers (e.g., Comer & Kendall, 2004; McMahon & Frick, 2005). Thus, it may be that the null effects of child age on magnitudes of cross-informant correspondence that we observed in fact reflected a greater use since Achenbach et al. (1987) of self-reports to assess children’s mental health concerns.

Conceptual Foundations of Multi-Informant Mental Health Assessments Operations Triad Model Collectively, our meta-analytic findings indicated that clinical child assessments tend to yield low-to-moderate levels of crossinformant correspondence, with higher levels of correspondence occurring when informants complete reports about mental health concerns that both informants either have relatively greater opportunities to observe (e.g., externalizing vs. internalizing) or observe within the same context (e.g., mother–father vs. parent–teacher). The main findings from the Achenbach et al. (1987) review, as well as the stability of these findings in research published since their review (see Table 1), have greatly informed the development of theoretical principles about using and interpreting multiinformant assessment outcomes. Indeed, an underlying assumption of the multi-informant approach is that informants each carry a unique and valid perspective of the patients about whom they provide reports (De Los Reyes, 2013). Thus, to maximize the clinical value offered by the multi-informant approach, the informants selected to provide reports ought to differ in their opportunities for observing patients’ mental health concerns (e.g., observations that vary across home and school contexts; Kraemer et al., 2003). Therefore, if patients contextually vary in where they display concerns, then discrepancies among informants’ reports should, in part, reflect these contextual variations. These basic assumptions underlying the multi-informant assessment approach have been elaborated upon and codified in recent theoretical work. Specifically, the idea that integrating multiple distinct pieces of information (e.g., research evidence and clinical expertise) improves clinical decision-making relative to relying on limited information (e.g., only clinical expertise) is a foundational principle of evidence-based practice (e.g., Sackett, Rosenberg,

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

864

DE LOS REYES ET AL.

Gray, Haynes, & Richardson, 1996). Yet, much of multi-informant assessment research has focused on whether informant discrepancies reflect error or rater biases (for a review, see De Los Reyes, 2013), and this research often conflicts with the key rationale for taking a multi-informant approach (De Los Reyes, Thomas et al., 2013). To address these inconsistencies between theory and interpretations of multi-informant assessments, researchers developed the Operations Triad Model (OTM; Figure 1; De Los Reyes, Thomas et al., 2013). The OTM addresses a key issue underlying these inconsistencies, namely, that research methodology in clinical psychology has largely relied on the concept of converging operations to interpret research conclusions and assessment outcomes in practice. Converging operations is a set of measure-

ment conditions within which one draws inferences of the veracity of multiple methodologically distinct research observations based on whether the observations yielded similar conclusions (Garner, Hake, & Eriksen, 1956). Under this concept, one draws stronger inferences from studies within which observations converged on a common conclusion (e.g., multiple informants’ reports all supported treatment efficacy), and in turn, one draws weaker inferences from studies within which observations diverged toward different conclusions (De Los Reyes, Thomas et al., 2013). To expand upon the converging operations concept, the OTM delineates conditions for two additional research concepts for interpreting discrepancies among informants’ reports. First, the

Figure 1. Graphical representation of the research concepts that comprise the Operations Triad Model. The top half (Figure 1a) represents converging operations: a set of measurement conditions for interpreting patterns of findings based on the consistency within which findings yield similar conclusions. The bottom half denotes two circumstances within which researchers identify discrepancies across empirical findings derived from multiple informants’ reports and thus discrepancies in the research conclusions drawn from these reports. On the left (Figure 1b) is a graphical representation of diverging operations: a set of measurement conditions for interpreting patterns of inconsistent findings based on hypotheses about variations in the behavior(s) assessed. The solid lines linking informants’ reports, empirical findings derived from these reports, and conclusions based on empirical findings denote the systematic relations among these three study components. Further, the presence of dual arrowheads in the figure representing diverging operations conveys the idea that one ties meaning to the discrepancies among empirical findings and research conclusions and thus how one interprets informants’ reports to vary as a function of variation in the behaviors being assessed. Lastly, on the right (Figure 1c) is a graphical representation of compensating operations: a set of measurement conditions for interpreting patterns of inconsistent findings based on methodological features of the study’s measures or informants. The dashed lines denote the lack of systematic relations among informants’ reports, empirical findings, and research conclusions. Originally published in De Los Reyes, Thomas et al. (2013). © Annual Review of Clinical Psychology. Copyright, 2012 Annual Reviews. All rights reserved. The Annual Reviews logo, and other Annual Reviews products referenced herein are either registered trademarks or trademarks of Annual Reviews. All other marks are the property of their respective owner and/or licensor.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

OTM includes a concept, diverging operations, developed for interpreting instances in which multiple informants’ reports diverge from each other for meaningful reasons. An example of diverging operations may involve a circumstance in which a clinician expects two informants’ reports about a patient’s mental health concerns to meaningfully diverge because (a) the informants observe the patient’s behavior within completely different contexts (e.g., home vs. school), and (b) the patient displays mental health concerns to a greater degree in one context than the other (e.g., home to a greater extent than school). Second, the OTM also includes conditions for a research concept, compensating operations, reflecting those circumstances in which multiple informants’ reports diverge from each other because of measurement error or some other methodological process. For instance, two informants’ reports may yield divergent information because they completed reports using distinct measures (e.g., different item content and scaling) and the measures may have varied in their psychometric properties (e.g., internal consistency estimates). Thus, within instances that reflect compensating operations, methodological features of the assessment process provide a parsimonious account for why informants’ reports yielded different outcomes. In sum, the OTM promotes an evidence-based approach to testing whether meaningful data can be gleaned from interpreting the consistencies and discrepancies observed among multiple informants’ reports.

Predictions About the Incremental and Construct Validity of Multi-Informant Assessments Mental health professionals can make a strong conceptual case for taking a multi-informant approach to assessing child patients. Further, the OTM may facilitate linking the concepts underlying administering multi-informant assessments with interpreting their outcomes (see Figure 1). Yet, unknown is how to use the OTM to make specific predictions about the incremental and construct validity of multi-informant assessments. That is, when multiinformant assessments yield low correspondence among informants’ reports, how should the evidence appear if the low correspondence reflects contextual variations in patients’ mental health? An illustrative example of multi-informant assessment outcomes may be helpful.2 For the purposes of this illustration, we focus on interpreting reports from informants who differ in the contexts in which they observe patients, specifically parents at home versus teachers at school. We also focus on assessments of a mental health domain for which patients’ concerns may significantly vary across home and school contexts, namely preschoolers’ disruptive behavior (for a review, see Wakschlag, Tolan, & Leventhal, 2010). To assess a sample of patients experiencing disruptive behavior, consider a research team conducting a multi-informant assessment that involved collecting parent and teacher reports. In Figure 2, we present graphical depictions of the outcomes of these assessments. Figure 2 demonstrates links among parent and teacher reports of disruptive behavior and “actual” disruptive behavior as displayed by the patient within home and/or school contexts. We include this measure of “actual” disruptive behavior in Figure 2 to illustrate the construct validity of parent and teacher reports. Specifically, we graphically depict three different patients, or patients who display concerns in home but not school (Figure 2a), school but not home (Figure 2b), and across home and school (Figure 2c). For each example, arrows link informants’ reports to patients’ behavior.

865

These arrows depict the strength of relations between informants’ reports and patients’ behavior. That is, solid-line arrows linking informants’ reports to patients’ behavior indicate stronger relations between reports and behavior, relative to the dashed lines that indicate weaker relations between reports and behavior. Thus, Figures 2a and 2b depict examples of diverging operations, in that the parent and teacher reports differ because the patient displays concerns in either the home context or school context, but not both contexts. In contrast, Figure 2c depicts an example of converging operations, in that the parent and teacher reports corroborate each other because the patient displays concerns across home and school contexts. In Figure 2, we also describe the strength of relations between parent and teacher reports and an independently administered criterion variable; an example of such a variable might be a consensus diagnostic assessment by an experienced clinical team. We include this criterion variable in Figure 2 to illustrate the incremental validity of parent and teacher reports. Here, we use the same “solid-line/dashed-line” approach as with the links between informants’ reports and measures of “actual” patients’ behavior that we used to illustrate construct validity. That is, in illustrating relations between parent and teacher reports and the criterion variable, solid-line (relative to dashed-line) arrows linking informants’ reports to each other indicate stronger relations between reports. Further, solid-line (relative to dashed-line) arrows linking an informant’s report to the validity criterion indicate stronger incremental validity for the “solid-line” informant’s report, relative to the “dashed-line” informant’s report. Therefore, in Figure 2a parent (and not teacher) reports incrementally predict variance in the criterion. In contrast, in Figure 2b teacher (and not parent) reports incrementally predict variance in the criterion. In Figure 2c, the overlap between parent and teacher reports results in neither report incrementally predicting variance in the criterion. If low correspondence among reports validly reflected contextual variations in patients’ disruptive behavior, the patterns of assessment outcomes would be likely to reflect two predictions, each of which are graphically depicted in Figure 2. The first prediction is that each informant’s report should “build upon the other” in terms of predicting or relating to relevant clinical metrics (e.g., consensus diagnoses, referral status, treatment response). The uniqueness of informants’ reports cannot merely reflect “noise” and must reveal something important about patients’ clinical presentations, prognoses, and/or responses to treatment protocols (see also Youngstrom, 2008). Thus, in Figure 2 one can see how conducting a study on the incremental validity of the multiinformant approach depends heavily on the contexts in which patients displayed concerns. Specifically, a sample of patients whose concerns were specific to the home context may result in teacher reports that did not yield incrementally valid information relative to parent reports (Figure 2a). Conversely, a sample of patients who exhibited school-specific concerns may yield parent reports that did not evidence incremental validity relative to 2 The predictions we highlight below may also apply to interpreting distinctions between observer reports (e.g., parent, teacher, and clinician reports) and patient self-reports, with the one caveat that the contexts in which patients observe their own behavior may overlap with those of other informants’ reports (see De Los Reyes, Bunnell et al., 2013; Kraemer et al., 2003).

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

866

DE LOS REYES ET AL.

Figure 2. Graphical depiction of incremental and construct validity predictions that follow from the OTM and conceptual rationale for taking a multi-informant approach to mental health assessment.

teacher reports (Figure 2b). Thus, for multiple informants’ reports to be considered incrementally valid relative to each other, patients in a sample should be heterogeneous in terms of the contexts in which they displayed concerns (i.e., a sample of Figure 2a, 2b, and 2c patients). That is, if a sample of patients exhibited individual differences in the context(s) in which they displayed concerns, then each kind of informant who provided reports about these patients (e.g., parents and teachers) would have had the opportunity to observe at least a subgroup of patients’ concerns in the sample. Without an informant having the opportunity to observe patients’ concerns, one cannot expect that informant to provide incrementally valid reports about such concerns. The second prediction is that patterns of cross-informant correspondence should reflect contextual variations inherent in patients’ concerns. For instance, a pattern of parents reporting high levels of disruptive behavior for which teachers’ reports do not corroborate should indicate that patients exhibited disruptive behavior to a greater extent at home than school (Figure 2a). Alternatively, consistently high levels of disruptive behavior reported by both parents and teachers should indicate that patients experienced concerns across contexts (Figure 2c).

Considering Constructs Other Than Context When Interpreting Informants’ Reports One key issue warrants comment. Specifically, a great deal of research has focused on testing whether low versus high crossinformant correspondence reflects constructs other than contextual variations in displays of patients’ concerns (for reviews, see Achenbach, 2006; De Los Reyes & Kazdin, 2005; Kraemer et al., 2003). Thus, drawing inferences about patients’ concerns from multi-informant assessments ought to involve examining plausible rival hypotheses for why informants’ reports may converge or diverge.

Clinical severity. First, one interpretation of the patterns of mental health concerns illustrated in Figure 2 is that consistencies and inconsistencies between informants’ reports reflect individual differences in severity of patients’ clinical presentations, and not necessarily contextual variations in patients’ concerns. Importantly, the literature is quite equivocal as to whether crosscontextual consistency in patients’ mental health concerns can be treated as a proxy for high levels of clinical severity. Indeed, prior work identified relations between informant discrepancies and context-specific displays of mental health concerns, even when accounting for participants’ clinical severity (e.g., De Los Reyes, Bunnell, & Beidel, 2013; De Los Reyes, Henry, Tolan, & Wakschlag, 2009). Additionally, studies of patients from a variety of clinical domains (e.g., attention and hyperactivity, conduct problems, social anxiety) are quite inconsistent as to whether patients evidencing concerns across contexts also evidence significantly greater clinical severity or impairment levels than patients who display concerns within specific contexts (e.g., Bögels et al., 2010; Dirks et al., 2012; Drabick et al., 2007). In contrast, studies of informant discrepancies in assessments of large samples of psychiatric inpatients (Carlson & Youngstrom, 2003) and outpatients (Thuppal, Carlson, Sprafkin, & Gadow, 2002) found that patients whose concerns were endorsed by at least two informants evidenced greater levels of clinical impairment, relative to patients whose concerns were endorsed by a single informant. Overall, in informant discrepancies studies, one ought to statistically account for the clinical severity of patients’ concerns when examining the relations between informants’ reports and contextual variations in patients’ concerns. Informants’ perspectives and rater biases. Perhaps the most frequently studied set of constructs in informant discrepancies research are those that reflect variations in informants’ perspectives about patients’ concerns and possible biases in informants’

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

reports. Research on these informant characteristics has been extensively reviewed in prior work (e.g., De Los Reyes, 2013; De Los Reyes & Kazdin, 2005). Consequently, we will focus on the evidence regarding the most often-studied characteristics. For instance, informants may vary in their perspectives as to which mental health concerns in child patients warrant care (BrookmanFrazee, Haine, Gabayan, & Garland, 2008; Hawley & Weisz, 2003; Jensen-Doss & Weisz, 2008; Yeh & Weisz, 2001). In fact, informants may vary in perceiving that a child’s mental health concerns distress the child and thus warrant care, even when all informants converge on endorsing that concerns exist (e.g., Phares & Compas, 1990; Phares & Danforth, 1994). Consequently, informants may vary considerably in their perspectives of patients’ mental health concerns, and these discrepant perspectives may impact such aspects of mental health care as access to services and therapeutic engagement (see also Hawley & Weisz, 2003; Yeh & Weisz, 2001). Similarly, informants from different ethnic or racial backgrounds may vary in whether or which behaviors reflect mental health concerns and warrant care, although research on the relation between such backgrounds and informant discrepancies has yielded inconsistent findings (see De Los Reyes & Kazdin, 2005; Duhig et al., 2000). Further, low-to-moderate correspondence levels and directional differences in reporting (e.g., whether children self-report greater concerns than parents report about children) similarly characterize cross-informant correspondence in reports examined in over 20 countries (Rescorla et al., 2014; Rescorla et al., 2013). Few characteristics in the informant discrepancies literature have been given more research attention than the impact of informants’ mental health concerns on their reports of children’s mental health. Broadly, the conceptual rationale for this work can be encapsulated in what researchers have termed the depression ¡ distortion hypothesis: When an informant experiences low mood, this causes the informant to attend to, encode, and thus rate children’s behavior with greater negative descriptors, relative to use of positive or neutral descriptors (see Richters, 1992; Youngstrom, Izard, & Ackerman, 1999). Importantly, depression ¡ distortion effects should not be seen as mutually exclusive from the previously described characteristics. For instance, depression ¡ distortion effects may result in depressed informants reporting greater levels of patients’ mental health concerns than nondepressed informants. These effects potentially impact patients’ access to care if mental health professionals encounter discrepant reports, and these discrepant reports result in uncertainties when making clinical decisions. Among the informants providing reports, parents’ reports have been the most frequently studied in terms of depression ¡ distortion effects, perhaps because a common risk factor of children’s mental health concerns is a family history of such concerns (e.g., Goodman & Gotlib, 1999). Thus, within clinic samples of child patients, parents providing reports of patients’ concerns may often experience a key characteristic thought to bias informants’ reports. A key concern with research on the depression ¡ distortion hypothesis is that the hypothesis does not have strong empirical support on its behalf. As mentioned previously, many studies have tested the depression ¡ distortion hypothesis, with some studies finding that informants experiencing depressive symptoms provide reports that indicate greater levels of children’s mental health concerns, relative to reports taken from other informants (De Los

867

Reyes & Kazdin, 2005). However, a number of studies have found no such support (e.g., Conrad & Hammen, 1989; De Los Reyes, Goodman, Kliewer, & Reid-Quiñones, 2010; De Los Reyes, Youngstrom et al., 2011a; Hawley & Weisz, 2003; Weissman et al., 1987). Further, most depression ¡ distortion studies do not include independent ratings of children’s behavior in the same situation (see also Richters, 1992), and interactions between informants (e.g., parents and teachers) and the subject of the assessment (e.g., children receiving treatment) are often context-dependent (e.g., occur exclusively in home and/or school contexts). Consequently, observing a child behaving differently with their depressed parent at home than they did with teachers at school might be a logical consequence of the nature of interactions between patients and informants. Among those studies that have used experimental designs and constrained other possible confounding factors (e.g., context of observation), researchers have observed, at best, modest support for the depression ¡ distortion hypothesis. For instance, Jouriles and Thompson (1993) experimentally induced negative mood states in parents before they viewed a previously recorded task involving a “cleanup” activity with their child, and had parents and independent observers rate the child’s behavior during the task. The researchers observed nonsignificant differences between parents’ and independent observers’ reports of children’s behavior during the task. Other studies using mood induction procedures have also failed to support the depression ¡ distortion hypothesis (Youngstrom, Kahana, Starr, & Schwartz, 2004). The depression ¡ distortion hypothesis’s strongest empirical support comes from a quasi-experimental study examining the relation between mothers’ depressive mood symptoms and their reports of children’s behavior during completion of a frustrating task (Youngstrom et al., 1999). Youngstrom, Izard, and Ackerman (1999) controlled for the context on which informants based their reports by comparing mothers’ reports of children completing a frustrating task with independent observers’ reports of children during the same task. The researchers observed statistically significant depression ¡ distortion effects, and yet these effects only accounted for between 2%–20% of incremental variance in mothers’ reports, relative to independent observers’ reports. Consequently, one would be hard-pressed to discard mothers’ reports based on even the highest observed figure of incremental variance (i.e., 20%), as even this figure constitutes a minority of the variance in the mothers’ report that could possibly be “afflicted” with rater bias. Overall, the lack of strong support for depression ¡ distortion effects indicate that these effects cannot fully account for the presence of informant discrepancies. Measurement error. A construct related to that of informants’ perspectives and rater biases is measurement error. Measurement error has key implications for research on the validity of the multi-informant approach. Indeed, measurement error truncates the ability of informants’ reports to predict criterion variables, thus hindering the ability of multi-informant assessments to evidence incremental and construct validity (see Dirks, Boyle, & Georgiades, 2011; Dirks et al., 2012). Thus, if measurement error explained informant discrepancies, then mental health professionals could not validly infer that discrepancies among informants’ reports reflect changes in displays of patients’ mental health concerns across contexts.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

868

DE LOS REYES ET AL.

The psychometrics literature points to three important components of measurement error that have implications for the multiinformant assessment approach: (a) transient error, (b) random error, and (c) systematic error. The first, transient error refers to characteristics of the rater that might hinder measurement reliability (Schmidt & Hunter, 1996). In many respects, the literature reviewed previously on the depression ¡ distortion hypothesis represents a case of research on transient error in multi-informant assessment. Second, randomly distributed error variance, particularly within the individual informants’ reports in a multi-informant assessment, may pose challenges to reliably assessing discrepancies among informants’ reports. This is because assessments of discrepancies between informants’ reports can only be as reliable as the individual informants’ reports from which one assesses discrepancies (e.g., Rogosa, Brandt, & Zimowski, 1982; Rogosa & Willett, 1983). Third and with regard to systematic error, recent work indicates that multi-informant assessments may yield consistent differences between informants’ reports across multiple measures (De Los Reyes, Alfano, & Beidel, 2010, 2011; De Los Reyes, Bunnell et al., 2013; De Los Reyes, Goodman et al., 2008, 2010; De Los Reyes, Youngstrom et al., 2011a, 2011b). Yet, the presence of systematic error may result in giving consistent differences between informants’ reports the appearance of meaningful differences (see also De Los Reyes, 2013). Overall, measurement error may pose challenges to validly interpreting the outcomes of multi-informant assessments as reflecting contextual variations in patients’ concerns. Thus, researchers have examined components of the measurement process that either constrain levels of such error or improve interpretations of informants’ reports. Two research literatures appear particularly pertinent in this regard. First, even subtle changes in measurement scaling can have profound effects on informants’ reports. For example, on a positively valenced self-report measure of life success, scaling items on a “⫺5 to 5” range produced greater proportions of respondents rating high life success than the proportion observed when the scale ranged from “0 to 10” (for a review, see Schwarz, 1999). Along these lines, a key element of best practices in multi-informant assessment involves ensuring that assessors hold such measurement features as item content, scaling, and response labeling constant across multiple informants’ reports. In this way, an assessor can decrease the likelihood that discrepancies among reports are the result of methodological artifacts of the measurement process (for a review, see De Los Reyes, Thomas et al., 2013). In fact, when researchers hold these measurement factors constant across reports, and the measures informants complete exhibit the same factor structures and similar levels of internal consistency, informants nevertheless provide reports that correspond with each other at low-to-moderate magnitudes (e.g., Achenbach & Rescorla, 2001; Baldwin & Dadds, 2007). At the same time, holding measurement factors constant across informants’ reports does not guarantee that the resulting discrepancies reflect contextual variations in patients’ concerns. Indeed, informants may nonetheless react differently to the same measurement instrument, and the reasons for these different reactions may or may not reflect differences in the contexts in which informants observe patients’ behavior. For instance, in recent work researchers administered assessments of parental knowledge of adolescents’ whereabouts and activities (i.e., a risk factor for adolescent delinquency) to a community-based sample of parents and adoles-

cents (De Los Reyes, Ehrlich et al., 2013). Researchers randomly assigned parents and adolescents to the order in which they completed two different assessments of parental knowledge. Within one assessment protocol, parents and adolescents received training on how to use the number of contexts in which they tend to observe behaviors indicative of parental knowledge to provide Likert-type ratings on survey reports about such knowledge (i.e., greater number of contexts ¡ greater ratings; De Los Reyes, Ehrlich et al., 2013). Within a second protocol, parents and adolescents received instructions to complete the assessments that did not involve any training (i.e., a typical questionnaire completion protocol). Interestingly, the training received by parents and adolescents increased the differences between their reports, relative to the no-training condition. These findings suggest that parents and adolescents differ in reports about parental knowledge, in part, because they view behaviors indicative of parental knowledge in different contexts. In contrast to the findings of De Los Reyes, Ehrlich et al. (2013), in another study clinicians with experience in assessing and treating child patients read vignettes of hypothetical children receiving a screening evaluation for conduct disorder (De Los Reyes & Marsh, 2011). The clinicians read vignettes that described contextual features in a child’s life that the researchers randomly manipulated to reflect either risks for conduct disorder (e.g., associating with deviant peers) or no such risks. The vignettes also described the child as evidencing a single symptom within the fourth edition of the Diagnostic and statistical manual of mental disorders (DSM–IV) diagnostic criteria for conduct disorder (American Psychiatric Association [APA], 2000). Researchers prompted clinicians to make likelihood ratings of the children that reflected their impressions of whether the children would meet DSM–IV criteria for conduct disorder if a clinician were to administer a full diagnostic evaluation to the child. Further, the clinicians made judgments for 30 hypothetical children, in order to make likelihood judgments for all 15 DSM–IV conduct disorder symptoms rated in both high-risk and low-risk contexts. Thus, all clinicians in the sample made ratings about the same hypothetical children and conduct disorder symptoms, were exposed to the same rating instructions, and had access to the same information about the hypothetical children’s environments. Overall, clinicians made greater likelihood ratings for children described in high-risk contexts versus those described in low-risk contexts. These findings suggest that clinicians tended to apply contextual information to their judgments about the children’s conduct disorder symptoms. However, clinicians exhibited little correspondence in terms of the specific symptoms for which they applied contextual information. For instance, whereas high-risk contextual information might have influenced one clinician’s judgments about a child’s truancy, the same high-risk contextual information might have had little influence on another clinician’s judgments about the child’s truancy. These findings have since been replicated when examining judgments about attention-deficit/hyperactivity concerns, as well as judgments completed by laypeople (Marsh, De Los Reyes, & Wallerstein, 2014). The findings of De Los Reyes, Ehrlich et al. (2013) and De Los Reyes and Marsh (2011) indicate that holding measurement factors constant alone does not guarantee interpretability of any one informant’s report or the discrepancies between informants’ reports. Thus, a second literature involves identifying and using

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

validity criteria for assessing context-specific displays of mental health concerns. Using context-specific criterion measures, mental health professionals might improve understanding of the reasons why informants provide discrepant reports. For example, to assess cross-contextual variations in preschool children’s disruptive behavior, researchers recently developed the Disruptive Behavior Diagnostic Observation Schedule (DB-DOS; Wakschlag et al., 2010). This behavioral observation measure consists of assessments of children’s disruptive behavior as displayed within interactions between children and adult authority figures. The adult authority figures consist of the assessed child’s parent and an unfamiliar clinical examiner. Assessors hold the nature of the interactions constant across adult– child interactions. For instance, the parent prompts the child in one interaction to help with cleaning up toys, and in a separate interaction the clinical examiner administers a similar cleanup prompt to the child. The consistency in the structure of adult– child interactions on the DB-DOS allows one to assess in an analogue sense disruptive behavior as displayed in home contexts (e.g., with parental adults) and/or nonhome contexts (e.g., with nonparental adults such as teachers). Thus, the DB-DOS might serve as an independent assessment of whether patients display disruptive behavior consistently across contexts or within specific contexts. Such an assessment could serve as a criterion for assessing the validity in interpreting patterns of informants’ reports (e.g., parent and teacher reports) as reflective of contextual variations in patients’ concerns. Below, we describe a study that took exactly this approach. Another example of a context-sensitive validity criterion comes from research on the social interactions that tend to elicit displays of children’s behavioral and emotional concerns (Hartley, Zakriski, & Wright, 2011). In this work, researchers administered behavioral checklists for informants (e.g., parents and teachers) to complete about children. The informants also completed measures about how the same children tend to react (e.g., aggressive behavior) within interactions with peers (e.g., peer bosses child around) and adult authority figures (e.g., adult giving child instructions). This second measure allows researchers to examine whether convergence between informants’ reports about children’s behavioral and emotional concerns signals similarities in the kinds of interactions encountered across contexts (e.g., child encounters teasing by peers at home and school). Alternatively, one could examine whether divergence between informants’ reports about children’s concerns reflects the idea that interactions that tend to elicit children’s concerns are present in one context (e.g., school) and not another (e.g., home). The previous examples of context-sensitive validity criteria relied on behavioral observations or survey methods. One final example leverages the time- and context-sensitive properties of physiological arousal. Specifically, individuals experiencing social anxiety vary in that some experience social anxiety in different ways, depending on the context (e.g., Bögels et al., 2010). Direct assessments of physiological arousal may distinguish social anxiety patients’ experiences within and across these contexts (e.g., arousal during public speaking vs. arousal in anticipation of public speaking; De Los Reyes et al., 2015). These assessments have great potential as validity indicators to interpret informants’ reports (De Los Reyes & Aldao, 2015). This is because technology now allows mental health professionals to assess arousal using wireless, ambulatory devices (e.g., heart rate monitors), and im-

869

plement them in vivo within the laboratory as well as routine clinic settings (see also Thomas, Aldao, & De Los Reyes, 2012). For example, in one study researchers recruited adolescents who met DSM–IV criteria for social anxiety disorder as well as adolescents who did not meet criteria for any mental disorder (Anderson & Hope, 2009). In this study, adolescents engaged in a series of laboratory social interaction tasks totaling 20 minutes (e.g., oneon-one social interaction and public speaking tasks with trained confederates), to assess adolescents’ self-reported changes in arousal in reference to arousal indices of physiological habituation taken during these interactions (i.e., wireless, ambulatory heart rate monitors). Regardless of diagnostic status, adolescents experienced physiological habituation to the social interactions (i.e., sharp increase in heart rate at beginning of task, followed by gradual decrease in heart rate over course of task). Yet, adolescents’ self-reports varied as a function of diagnostic status. Specifically, adolescents experiencing social anxiety disorder were more likely than adolescents not meeting criteria for any mental disorder to self-report stable and high levels of arousal from pre-to-post tasks. This observation is consistent with key components of exposure-based therapies for social anxiety. Indeed, these therapeutic approaches often involve training patients to subjectively perceive decreases in physiological arousal within and across social situations that at the outset of therapy patients find anxiety provoking (e.g., Beidel et al., 2007). Overall, these three examples illustrate how mental health professionals might use independent behavioral, survey, and/or physiological assessments to use and interpret informants’ clinical reports of patients’ concerns. False-positive and false-negative reports. A set of constructs related to measurement error but with different implications for the multi-informant approach involves concepts drawn from judgment and decision-making research, namely signal detection theory (SDT; for a review, see Swets, Dawes, & Monahan, 2000). Specifically, let us assume that the parent and teacher reports illustrated in Figures 2a and 2b reflected error-free representations of children’s behavior, and the parents and teachers systematically varied in terms of exclusive access to observations of children’s behavior as displayed at home and school, respectively. If so, then parents and not teachers would endorse every instance of disruptive behavior displayed at home and not school, whereas teachers and not parents would endorse every instance of disruptive behavior displayed at school and not home. Stated another way, both parents and teachers would evidence perfect rates of positive endorsements of “true cases” of disruptive behavior as displayed within their own context of observation. Parents and teachers would also refrain from endorsing disruptive behavior that only occurred outside of their observational context. Within SDT, researchers refer to true cases of endorsement as true positives and true cases of nonendorsement as true negatives (see also Swets, 1992). However, one should not expect any measure (e.g., informant’s report), to provide an error-free representation of the construct being assessed. Thus, SDT delineates concepts for representing incorrect positive endorsements for noncases (i.e., false positives) and incorrect nonendorsements for true cases (i.e., false negatives). For instance, in a case in which a parent endorsed disruptive behavior that a teacher did not endorse, if in reality the patient did not display any disruptive behavior in any context, the parent report would reflect a false positive. Con-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

870

DE LOS REYES ET AL.

versely, both parent and teacher might fail to endorse disruptive behavior in a case in which the patient actually did display disruptive behavior in both contexts; in this case both reports would reflect false negatives. As in addressing measurement error, one might address false positive and false negative outcomes with an independent, contextsensitive assessment of patients’ concerns. Figure 3 provides an illustration of this approach. For example, consider a hypothetical sample of children in which teachers and parents completed standardized behavioral checklists of children’s disruptive behavior, and researchers also administered an independent observation of children’s disruptive behavior in the home context. In Figure 3, the left-most distribution of scores relates to teacher reports, the center distribution to parent reports, and the right-most distribution to the independent home observation. The x-axis denotes a standardized score (i.e., T score) derived from each assessment to represent children’s disruptive behavior. Further, we included a demarcation at the T score “65” to denote the clinical cutoff at which researchers would identify cases of disruptive behavior. The y-axis denotes the frequency for scores at a given T score level. Given standardization of the disruptive behavior scores, all three assessment modalities in the example follow a normal distribution. In Figure 3, the independent assessment occurred in the home context. Thus, one would expect greater overlap between identified cases of disruptive behavior concerns using the independent assessment and identified cases using parent report (i.e., relative to teacher report). Thus, using the independent assessment as a reference point, researchers could examine whether patients identified via the independent assessment were more likely to be those patients positively endorsed by parents relative to teachers. Stated another way, the independent assessment allows the research team to identify the proportions of parent and teacher reports evidencing true positives and negatives, as well as false positives and negatives. With these proportions, one could estimate how sensitive (i.e., proportion of true positives relative to false positives) and specific (i.e., proportion of true negatives to false negatives) the parent reports were in detecting home-specific observations of patients’ concerns, relative to an informant who observed patients in a context other than home (see also Youngstrom, 2013). One

could also structure an assessment similar to that described in Figure 3 to gauge the context-sensitivity of teachers’ reports (i.e., independent school observation). Thus, one might incorporate methods from SDT with the OTM to improve use and interpretation of multi-informant assessments. Informants’ cognitive abilities and social desirability. Lastly, how might one interpret the unique qualities of informants’ reports, beyond that they reflect contextual variations in patients’ concerns, levels of clinical severity, and/or rater biases? That is, discrepancies between adult informants such as parents and teachers might originate from patients’ concerns varying across contexts. However, they may also vary in light of the relationship that the patient shares with the informant and the context in which this relationship develops. For instance, a parent may provide a report based on extensive experiences with the patient, and although these experiences may develop over the course of several years, they may not be perceived within the context of normative childhood behavior (De Los Reyes et al., 2009). Conversely, a teacher’s relationship with the patient may develop within the context of a single academic year and a large classroom of other students. That is, a teacher’s report may reflect a relatively limited time range of experiences with a patient. However, the teacher may have the opportunity to calibrate their report based on normative classroom behavior (see also Drabick et al., 2007). When researchers compare adult and child informants, informant discrepancies may be attributable, in part, to differences in developing cognitive abilities (e.g., Smetana, Campione-Barr, & Metzger, 2006; Spear, 2000). In the context of patient care, differences between adult reports about patients and patient selfreports may occur as a result of social desirability concerns on the part of the patient, although studies yield inconsistent support for these concerns as contributors to informant discrepancies (for a review, see De Los Reyes & Kazdin, 2005). These differences between adult informants and child informants may indicate, for instance, that adult reports reliably and validly reflect patients’ concerns to a greater extent than patient self-reports. Recent evidence indicates that differences between parent reports and child reports may also reflect meaningful variation between their perceptions, in addition to variations in cognitive

Figure 3. Graphical depiction of T score distributions of teacher reports (left, labeled “T”), parent reports (center, labeled “P”), and independent home-based assessments of disruptive behavior (right, labeled “H”), in a hypothetical sample of young children.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

abilities and social desirability concerns. For instance, in structured diagnostic interview assessments of child anxiety, interviews based on parent reports and child reports exhibit greater correspondence when reports are about directly observable anxiety behaviors displayed in nonschool contexts (e.g., behavioral avoidance displayed at home) relative to internal anxiety behaviors such as worry displayed in school contexts (Comer & Kendall, 2004). Further, recent work in an age- and gender-matched sample of clinic-referred adolescents and community control adolescents found that clinic-referred adolescents self-reported lower levels of social anxiety relative to their parents’ reports, and adolescents’ self-reports exhibited little to no correspondence with objective measures of psychophysiology (i.e., during a baseline psychophysiological assessment; De Los Reyes, Aldao et al., 2012). Yet, in this study adolescents nonetheless provided internally consistent self-reports that evidenced convergent validity and differentiated clinic-referred from community control adolescents. This study is an important contribution to the literature on social desirability. Indeed, clinic samples of anxiety patients comprise key settings within which patients’ social desirability has been of primary concern to mental health professionals (e.g., Dadds, Perrin, & Yule, 1998; DiBartolo, Albano, Barlow, & Heimberg, 1998; Grills & Ollendick, 2003; Rapee et al., 1994; Silverman & Rabian, 1995). Importantly, De Los Reyes, Aldao et al. (2012) did not directly assess social desirability. Yet, the authors made a key observation that researchers often interpret as indicating social desirability—relative underreporting via patient self-reports relative to adult informants’ reports about patients—while also finding support for the internal consistency and validity of scores taken from patient self-reports. Thus, informant discrepancies alone should not be interpreted as reflecting developmental differences in cognitive abilities or social desirability in informants’ reports. Rather, mental health professionals should evaluate whether these discrepancies relate to independent assessments of informants’ cognitive abilities or social desirability (see Table 2).

Incremental Validity in Multi-Informant Mental Health Assessments Theoretical and Methodological Approaches to Incremental Validity A full account of the validity of multi-informant assessments involves examining the unique contribution of one informant’s report over another’s, or the incremental validity of each report in the prediction of clinical outcomes (see Figure 2). Yet, despite repeated appeals for more investigations of incremental validity in psychological assessment, a dearth of information in this area remains (Hunsley & Meyer, 2003). Nevertheless, prior work on incremental validity in such areas as test construction (Haynes & Lench, 2003; Smith, Fischer, & Fister, 2003), adult psychopathology (Garb, 2003), and use of projective tests (e.g., Dawes, 1999) has key implications for our discussion of incremental validity. In particular, two ideas informed our examination of the incremental validity of multi-informant clinical child assessments. Expertise in assessment. First, the Rorschach Inkblot Test is a projective test that involves an assessor soliciting patients’ openended, verbal descriptions of ambiguous stimuli (i.e., a series of

871

blotches of black ink set against a plain white background; for a review, see Hunsley & Lee, 2010). Patients’ responses are thought to reveal important aspects of their personality functioning from a psychoanalytic perspective, and thus scoring responses often involves use of complex procedures that require a substantial amount of expertise on the part of the assessor (i.e., knowledge of psychoanalytic theory and practice; see Blais, Hilsenroth, Castlebury, Fowler, & Baity, 2001). However, Rorschach responses can also be scored using less onerous procedures that require little expertise to carry out (e.g., assessing the “match” between a patient’s response about the shape of an inkblot and the actual shape of the inkblot). Further, personality functioning can be assessed with paper-and-pencil inventories that can be administered, scored, and interpreted by personnel with little training. Consequently, Dawes (1999) argues that a proper evaluation of incremental validity of the Rorschach involves assessing whether scores that require a great deal of expertise to obtain hold incremental value above-and-beyond the explanatory value of scores that require less expertise to obtain. In clinical child assessments, the multi-informant approach poses incremental validity issues similar to that of the Rorschach and delineated by Dawes (1999). In essence, do clinical child assessments require the collective expertise of multiple informants for one to obtain an adequate sense of the context(s) in which a patient displays mental health concerns? Indeed, the multi-informant approach is a comprehensive assessment procedure that incurs considerable resources in terms of time needed to administer, score, and interpret instruments, and staffing and equipment needed to carry out the assessments (see also Yates & Taub, 2003). A less onerous and comprehensive assessment might effectively address the goals of assessing a patient, such as use of a single informant’s report. Thus, demonstrating that the multi-informant approach yields incremental value over, for instance, a single informant’s report involves testing whether each informant observes patients’ concerns in a particular context that other informants have relatively fewer or no opportunities to observe. That is, the expertise of each informant should result in reports from which scores reflect different facets of patients’ concerns (i.e., different contexts in which patients display concerns; see also Smith et al., 2003). Criterion variables and criterion contamination. A second core idea driving our examination of the incremental validity of multi-informant clinical child assessments follows from ideas about expertise advanced by Dawes (1999). Specifically, as in other cases of validity (e.g., construct and predictive), the incremental validity of scores from a given measure may vary as a function of the criterion variable(s) examined (e.g., diagnostic status and treatment response) and the other measures to which it is compared (e.g., Haynes & Lench, 2003). These two factors give rise to two additional challenges to using and interpreting multi-informant assessments. The first challenge is to design incremental validity studies involving multiple predictor variables (e.g., multiple informants’ reports). Predictors should evidence bivariate relations with the criterion variable(s) but share little overlapping variance with each other (see also Blais et al., 2001). For instance, use of parent reports and teacher reports to predict clinicians’ impressions of patients’ behavior across home and school contexts capitalizes on use of the context-specific expertise of each informant (i.e., home-specific vs. school-specific) to predict clinicians’ reports. However, addressing challenges with overlapping variance among predictor variables raises a second challenge. Specifically, if the

DE LOS REYES ET AL.

872

Table 2 Summary of Exemplary Findings Regarding the Incremental Validity of Cross-Informant Reports When Assessing Children’s Psychosocial Functioning Symptom/Disorder

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

ADHD

Author group (year)

Informants

Power et al. (1998)

P, T

Smith, Pelham, Gnagy, Molina, & Evans (2000)

C, T

ADHD subtypes and ODD

Owens & Hoza (2003)

P, T

Anxiety

Villabø, Gere, Torgersen, March, & Kendall (2012)

C, P

Bipolar Disorder

Youngstrom et al. (2004a)

C, P, T

ODD

Angold & Costello (1996)

C, P

Relational Adjustment

Ladd & KochenderferLadd (2002)

C, P, T, Peer

Social Skills and Academic Competence

Kwon, Kim, & Sheridan (2012)

T, Peer

Description Researchers recommended use of a single parent report or teacher report to rule out ADHD in children, but to combine parent report and teacher report to identify children meeting criteria for an ADHD diagnosis. On average, adolescent self-reports contributed less than a 1% unique variance above teacher ratings in the assessment of ADHD. In contrast, teacher reports contributed, on average, 8% unique variance relative to self-report. However, teacher reports evidenced domain-specific incremental validity (i.e., higher unique variance contributed by teachers for behaviors displayed in the classroom vs. outside the classroom). When assessing inattentive or hyperactive/impulsive subtypes of ADHD, combining parent and teacher reports did not provide any additional information above relying on parent report or teacher report alone. In contrast, when assessing ADHD, combined type, or ODD, teacher reports alone and the combination of teacher reports and parent reports proved superior to parent reports alone in predicting a diagnosis. Adding parent reports to child self-reports (i.e., on the MASC) increased prediction of children’s anxiety disorder diagnoses. Specifically, parent reports incrementally predicted children’s separation anxiety disorder diagnoses, whereas child self-reports incrementally predicted children’s social phobia diagnoses. Across estimates of diagnostic efficiency for six different rating scales, parent reports evidenced superior prediction of a pediatric bipolar disorder diagnosis relative to both self-reports and teacher-reports. Additionally, combining self-reports or teacher reports with parent reports did not significantly enhance the prediction of a diagnosis of pediatric bipolar disorder. Children tended to self-report fewer oppositional symptoms relative to parents’ reports about children. However, children self-reported certain oppositional symptoms (i.e., frequent loss of temper) at greater levels relative to parents’ reports about children. Researchers recommended both self-reports and parent reports whenever possible when considering an ODD diagnosis. No single informant’s report of victimization consistently predicted relational adjustment across Grades 2–4. The informant whose report explained the most variance in relational adjustment varied as a function of grade: (a) teacher reports in Grade 2, (b) teacher reports and self-reports in Grade 3, and (c) a combination of self-reports and teacher reports in Grade 4. Overall, parent reports explained little unique variance. Teacher-reported social skills for children in Grades 3–5 did not uniquely or significantly predict social status indicators (e.g., likeability and popularity) above information obtained from peer reports. In contrast, both peerreported and teacher-reported social skills demonstrated incremental validity in predicting academic competence and children’s positive school functioning.

Note. C ⫽ Child; P ⫽ Parent; T ⫽ Teacher; ADHD ⫽ attention-deficit/hyperactivity disorder; MASC ⫽ Multidimensional Anxiety Scale for Children; ODD ⫽ oppositional defiant disorder; MDD ⫽ major depressive disorder; CD ⫽ conduct disorder.

outcomes of incremental validity studies vary depending on the criterion variable, then these studies should be designed such that differences in findings among the predictor variables should to the greatest extent possible reflect “true differences” among predictors in their incremental validity. For instance, when comparing reports within a multi-informant assessment, one should ensure that measurement content be held constant across the informants’ reports. Yet, holding these measurement factors constant across reports may do little to address a key issue raised by Garb (2003): criterion contamination. In incremental validity research, criterion contamination refers to differences in incremental validity among predictors that arises, in part, because the criterion variable was based on information from one or more of the predictors. For instance, if a study’s validity

criterion consisted of clinicians’ diagnostic judgments about patients, the study might suffer from criterion contamination if prior to forming their judgments, clinicians had access to information from the predictors evaluated for incremental validity (see also Bossuyt et al., 2003; Whiting et al., 2011).

Incremental Validity in Multi-Informant Clinical Child Assessments We evaluate the incremental validity of multi-informant clinical child assessments in light of key theoretical and methodological contributions from prior work. As in the incremental validity literature reviewed previously, the incremental validity of reports

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

about child patients may be influenced by measurement methodology, the mental health domain assessed, and criterion contamination (e.g., Johnston & Murray, 2003; Pelham, Fabiano, & Massetti, 2005). In Table 2, we illustrate key findings from prior work. Importantly, few incremental validity studies exist for internalizing concerns such as anxiety, relative to externalizing concerns such as attention and hyperactivity and conduct problems (for reviews, see Dirks et al., 2012; Silverman & Ollendick, 2005; Tulbure, Szentagotai, Dobrean, & David, 2012). Thus, we focus our attention on the incremental validity of multi-informant assessments of children’s externalizing concerns, and where possible we incorporate work on children’s internalizing concerns. Measurement method of informants’ reports evaluated for incremental validity. Across both clinical and research settings, informants may provide reports via a variety of measurement methods, including rating scales and structured or semistructured interviews. Under certain circumstances (e.g., diagnosing childhood attention-deficit/hyperactivity disorder [ADHD]), research supports the construct validity of these approaches (Pelham et al., 2005). Yet, scores from an informant’s report may evidence construct validity and still fail to incrementally contribute information beyond that contributed by other measures within an assessment battery (i.e., incremental validity; Johnston & Murray, 2003). For example, in a review of evidence-based assessments for ADHD, Pelham, Fabiano, and Massetti (2005) examined the incremental validity of rating scales and structured interviews for the assessment of childhood ADHD. Despite evidence of the construct validity of each measure, Pelham et al. (2005) found rating scales to be the most incrementally valid for assessing ADHD, and use of a DSM–IV-based structured interview did not contribute unique information beyond rating scales. Yet, as we discuss below, issues with criterion contamination hinder the interpretability of prior work on the incremental validity of measures used in clinical child and adolescent assessments. Mental health domain or focus of assessments. Another factor that can affect the incremental validity of a predictor is the domain being assessed or the goal of assessment. The incremental validity of an informant’s report may vary dramatically when assessing externalizing symptoms versus internalizing symptoms, or when the goal of the assessment is to reach a single diagnosis versus ruling out a diagnosis (see Table 2). For instance, returning to Pelham et al.’s (2005) review, research suggests that a single teacher rating of ADHD sufficiently captures the presence of ADHD for research purposes. However, ratings from both parents and teachers are important if the goal of the assessment is to reach a clinical diagnosis, given the diagnostic requirement of symptom displays across multiple contexts. Similarly, rating scales may suffice for identifying ADHD symptoms, and yet ruling out diagnoses other than ADHD (i.e., for purposes of differential diagnosis) requires gathering additional information regarding other mental health domains (e.g., anxiety and conduct problems; Pelham et al., 2005).3 As with incremental validity research on measurement methods (e.g., rating scales vs. structured interviews), we discuss below issues regarding how criterion contamination hinders the interpretability of prior work examining incremental validity and use of multiple informants’ reports (e.g., parents vs. teachers) and measurement of multiple mental health domains. Criterion variables and criterion contamination. Crucially, estimates of incremental validity for multi-informant clinical child

873

assessments may vary as a function of the validity criterion or predictors (e.g., Johnston & Murray, 2003; Pelham et al., 2005). How might the choice of the validity criterion and predictors influence conclusions about the incremental validity of multiinformant assessments? We address this question in two ways. First, how do multi-informant assessments perform if one holds measurement method, item content, and an independently assessed correlate constant across informants’ reports? To address this, we present in Table 3 findings reported by Achenbach and Rescorla (2001) for cross-informant correspondence for parent report, teacher report, and child self-report forms of the Achenbach System of Empirically Based Assessments (ASEBA), as well as relations among these reports and children’s referral status. Briefly, the ASEBA battery consists of parent, teacher, and child reports that exhibit an identical format (i.e., behavioral checklist) and response scaling (i.e., response values of none, some, and a lot), with some minor modifications to item content to match each informant’s perspective. Each form assesses syndromes reflecting various children’s mental health domains that cluster around two broadband problem domains: internalizing (e.g., anxiety, mood, somatic complaints) and externalizing (e.g., aggressive and rulebreaking behaviors). Further, Table 3 reports relations between the informants’ reports and a single correlate, namely children’s referral status (i.e., differentiating clinic referred vs. nonreferred children). In Table 3 we report two sets of findings regarding the ASEBA forms. Specifically, we present statistical estimates reported by Achenbach and Rescorla (2001). Specifically, we report Pearson r correlation coefficients among parent, teacher, and child reports, which we derived from Table 9-2 from Achenbach and Rescorla (2001). As reported in Achenbach and Rescorla (2001), each value denotes a significant relation between informants’ reports, p ⬍ .05. Consistent with meta-analytic findings reviewed previously, levels of informant correspondence (a) ranged from low-to-moderate in magnitude, and (b) were larger for correspondence on reports of externalizing relative to internalizing problem behaviors. Based on these cross-informant correspondence estimates, we calculated the common variance shared by multiple informants for the internalizing, externalizing, and total problem scores on the ASEBA forms. We made these calculations based on regression commonality analyses illustrated by Cohen, Cohen, West, and Aiken (2003) and Nimon (2010). These estimates (ranging from 0.8% to 3.0%; Table 3) reveal that the proportion of variance in problem scores that is captured by all informants is very small. We also report the percentages of variances in informants’ reports that are explained by children’s referral status. We derived these percentages of variance from Table 10-3 from Achenbach and Rescorla (2001). Specifically, the authors conducted multiple 3 Additional factors such as the patient’s age and gender, cultural characteristics of the assessment setting, and the base rates of the symptoms may affect the generalizability of information regarding the incremental validity of assessment methods. Therefore, these other issues must be taken into account before generalizing findings from one study to new samples or patient populations (Johnston & Murray, 2003). An example of this effect can be seen in work examining the incremental validity of multiple informants’ reports of peer victimization in the prediction of relational adjustment in elementary school-age children (Ladd & Kochenderfer-Ladd, 2002). In this study, the unique information contributed by reporters varied as a function of children’s grade level (see Table 2).

DE LOS REYES ET AL.

874

Table 3 Cross-Informant Correspondence and Relations Between Referral Status and Informants’ Reports, Based on Parent, Teacher, and Youth Forms of the Achenbach System of Empirically Based Assessments (Achenbach & Rescorla, 2001) Correlation estimates of cross-informant correspondence

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Psychopathology domain Internalizing problems Externalizing problems Total problems Psychopathology domain Internalizing problems Externalizing problems Total problems Psychopathology domain Internalizing problems Externalizing problems Total problems Psychopathology domain Internalizing problems Externalizing problems Total problems

CBCL-YSR CBCL-TRF YSR-TRF .48 .21 .17 .56 .36 .28 .54 .35 .21 Estimated common variance attributable to multiple informantsa 1.3% 3.0% 0.8% Estimated variance in informant accounted for by referral status, converted from point biserial Pearson r correlations CBCL YSR TRF 26% 11% 14% 33% 17% 19% 36% 15% 26% Estimated correlation with referral status for a single informant CBCL YSR TRF .51 .33 .37 .57 .41 .44 .60 .39 .51 Estimated maximum possible correlation with referral status for multiple informantsb

Psychopathology domain Internalizing problems Externalizing problems Total problems

.72 .85 .88

Note. CBCL ⫽ Child Behavior Checklist (i.e., Parent Report Form); YSR ⫽ Youth Self-Report; TRF ⫽ Teacher Report Form. a Estimates based on regression commonality analyses illustrated by Cohen et al. (2003) and Nimon (2010), using cross-informant correlations reported above. b By “maximum,” we mean that we estimated an upper limit correlation between informants’ reports and referral status, assuming unique variance among the informants’ reports.

regression analyses using referral status as a predictor variable along with a series of children’s demographic variables as predictor variables (i.e., age, socioeconomic status, ethnicity). In separate analyses, each of the parent, teacher, and child reports served as dependent variables. Following these analyses, the authors converted the regression coefficients representing unique effects of children’s referral status on informants’ reports to Fisher’s z, averaged these Fisher’s zs across gender and age groups, and converted these final Fisher’s zs to Pearson rs. The percentages of variances reported in Table 3 are squared estimates of these Pearson rs. These values denote significant differentiation between referred and nonreferred children for all informants’ reports (p ⬍ .01), with lower scores for nonreferred children relative to referred children. That is, each report (parent, teacher, child) significantly related to children’s referral status. Based on these percentages of variances estimated by Achenbach and Rescorla (2001), we estimated correlations between referral status and the individual informants’ reports, as well as a total correlation between referral status and the three informants’ reports. Specifically, to estimate correlations for individual informants, we took the square root of the percentage of variance accounted for by referral status separately for parent report, teacher report, and child report. To estimate maximum possible correlations for the multi-informant reports, we took the square root of the sum of the common variance

across multiple informants and the percentage of variance accounted for by referral status for the parent, teacher, and child reports (Cohen et al., 2003). By “maximum,” we mean that we estimated an upper limit correlation between informants’ reports and referral status, assuming unique variance among the informants’ reports. In comparing the coefficients observed for individual and multiinformant correlations, a clear pattern of findings emerges. Specifically, one observes greater magnitudes of correlations between referral status and an approach based on the multi-informant parent, teacher, and child reports, relative to an approach based on use of any one of the individual parent, teacher, or child reports. Overall, when using reports that hold item content and scaling constant across informants and in relation to assessments of children’s referral status, multiple informants’ reports of children’s internalizing and externalizing concerns may each relate to referral status. However, in each case (i.e., relations between referral status and internalizing, externalizing, or total broadband problems), a multi-informant assessment approach yields greater correlations with referral status than an approach based on any one individual informant’s report. In light of our findings, our approach to illustrating the incremental validity of multi-informant assessments evidenced a number of key strengths. Specifically, each informant’s report was assessed using the same reporting format (i.e.,

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

survey checklist) and item content, and incremental validity was evaluated using a criterion measure that evidenced relatively little criterion contamination. Yet, one limitation is that the findings in Table 3 were based on the scale scores reported by Achenbach and Rescorla (2001) for the complete parent, teacher, and child ASEBA scales (Leslie Rescorla, personal communication, September 10, 2014). Although these scales share many items in common, each contain items that the other scales do not (e.g., teacher report contains items to assess attentional concerns that the other reports do not contain). Consequently, we suspect the crossinformant correspondence estimates reported in Table 3 are conservative and would be slightly higher if these estimates were based strictly on items common across the informants’ instruments. Nevertheless, previous work examining cross-informant correspondence specifically on ASEBA items that were common across informants’ reports found low-to-moderate correspondence between informants’ reports (e.g., De Los Reyes, Youngstrom et al., 2011a, 2011b). In light of the findings in Table 3, a second question arises: What happens when a validity criterion suffers from criterion contamination, as described by Garb (2003)? Indeed, in some of the incremental validity studies in Table 2, the validity criterion consisted of clinicians’ diagnoses based, in part, on information provided by multiple informants, indicating nonindependence between the informants’ reports and the criterion measure. In addition, in some of the studies in Table 2, when clinicians gathered information from multiple informants they also knew which informant referred the patient for clinical services and which informants served as “collateral” informants. Importantly, in these studies clinicians would have been likely to encounter instances in which informants’ reports pointed to inconsistent conclusions regarding patients’ concerns and whether these concerns warranted additional clinical services (see also Yeh & Weisz, 2001). Two lines of research indicate that the designs of the incremental validity studies described in Table 2 may compromise interpretations of research on the incremental validity of multiinformant assessments. First, mental health professionals may view specific informants as “optimal” for assessing specific mental health concerns, such as viewing teachers as “better” informants than parents for assessing child hyperactivity (e.g., Bird, Gould, & Staghezza, 1992; Loeber, Green, Lahey, & Stouthamer-Loeber, 1989; Loeber, Green, & Lahey, 1990). Second, compounding issues about “optimal informants” is the fact that a single informant (e.g., parent) usually initiates a patient’s clinical services (Hunsley & Lee, 2010). Among other pieces of clinical information, the referral informant provides an assessor with a referral question (e.g., Does the patient evidence concerns with depressive mood symptoms?; Garb, 1998). This referral question typically informs assessors’ initial hypotheses as to the patients’ concerns and thus the key goal of the assessment (Croskerry, 2003). Thus, when clinicians make decisions based on multi-informant assessment outcomes, their decisions may align with referral informants’ reports more so than other informants’ reports, even without compelling evidence to indicate the validity of this approach. Note that clinicians’ prior knowledge of which informant served as the referral source and which informants served as collateral informants should be distinguished from clinicians having access to the outcomes of multi-informant assessments prior to making clinical decisions. Studies rarely note whether

875

clinicians were kept blind to all clinical information collected apart from their own clinical ratings or tools used to make clinical decisions (e.g., diagnostic interview information; for an exception, see Carlson & Blader, 2011). That being said, studies using clinicians’ reports as a criterion variable when clinicians also had access to the multiinformant assessments evaluated for incremental validity would have been likely to compound the interpretative problems raised by clinicians having knowledge of the identity of the referral source (e.g., Bossuyt et al., 2003; Whiting et al., 2011). Regardless of the mechanisms underlying reliance on a single informant to make clinical decisions, this practice has important implications for clinical decision-making. That is, presumably approaches to clinical decision-making that rely on one informant’s report more so than other informants’ reports most often take the form of disjunctive decision-making strategies (i.e., “OR” rule; for a review see Piacentini, Cohen, & Cohen, 1992), whereby the informant relied upon most, for instance, to plan treatment for a patient might also be identifying mental health concerns that go uncorroborated based on other informants’ reports. A strength of this approach is that it can increase access to mental health services for patients with impairing symptoms because only a single informant would have to report concerns for the patient to receive care. However, relying on a single report introduces considerable measurement error in decision-making and may lead to the false positive test outcomes described previously (e.g., Hunsley, 2003; Swets et al., 2000; Youngstrom, Findling, & Calabrese, 2003). Yet, to what extent do the data indicate that such clinical decisions occur? In Table 4, we summarize the findings of several studies on the relation between informant discrepancies and decisions that clinicians make regarding ratings of children’s functional impairment, diagnosis, treatment response, and treatment planning. For four of the studies described in Table 4 (De Los Reyes, Alfano et al., 2011; DiBartolo et al., 1998; Grills & Ollendick, 2003; Youngstrom, Findling, & Calabrese, 2004b), an overall pattern of findings emerges: When correspondence between parent reports and child self-reports is low, clinicians tend to make diagnostic decisions and ratings of treatment response that correspond more with parent reports than child self-reports. For three studies (BrownJacobsen, Wallace, & Whiteside, 2011; Hawley & Weisz, 2003; Kramer et al., 2004), whether clinicians corresponded in their clinical decisions more with parent reports or child self-reports depended on the measurement method or problem domain assessed. For instance, given high disagreement among parents, patients, and clinicians in terms of which problems to target in therapy, clinicians’ decisions regarding therapeutic targets agreed more with parent reports than patient self-reports when the target concerned a patient problem (e.g., children’s anxiety, mood, attention concerns; Hawley & Weisz, 2003). However, in this same study, clinicians agreed more with patient self-reports than parent reports when the target concerned problems with the family or the environment. When interpreting the findings summarized in Table 4, it is important to further scrutinize the studies and the quality of the informants’ reports. Indeed, one possible interpretation of these findings is that clinicians rightfully identified parents as the “most valid” informants for the particular mental health concerns assessed. A key characteristic of two of the studies in Table 4 is that

DE LOS REYES ET AL.

876

Table 4 Summary of Studies on the Relation Between Informant Discrepancies in Parent and Child Reports of Children’s Mental Health and Clinical Decision-Making

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Author group (year)

Mean patient age in years (age range)

Brown-Jacobsen, Wallace, & Whiteside (2011)

12.3 (7–18)

De Los Reyes, Alfano, & Beidel (2011)

11.6 (7–16)

DiBartolo, Albano, Barlow, & Heimberg (1998) Grills & Ollendick (2003)

13.9 (8–17)

10.7 (7–17)

Hawley & Weisz (2003)

11.5 (7–17)

Kramer et al. (2004)

13.7 (11–18)

Youngstrom, Findling, & Calabrese (2004b)

14.1 (11–17)

Description When assessing children’s anxiety symptoms, both parent- and child selfreported symptoms were significantly related to the clinician’s diagnostic impression. For parent and child questionnaire reports, considered simultaneously, only parent reports uniquely related to the clinician’s diagnostic impression of the child. Conversely, when only the parent or the child endorsed an anxiety disorder diagnosis, the clinician made a diagnosis approximately 30% of the time, with the prevalence based on parent-reported symptoms (28.6%) slightly higher than child-reported symptoms (26.8%). When assessing pre- and post-treatment social functioning in a sample of children, clinician ratings of treatment improvement more strongly agreed with parent-reported post-treatment functioning, relative to children’s self-reported functioning. Overall, clinicians weighted parent reports more strongly than they did child self-reports when determining the severity of child social phobia. When assessing children’s anxiety symptoms, a clinicians’ consensus diagnoses more highly correlated with parent reports, relative to child self-reports. When asked to identify target problems in outpatient treatment, low agreement was found within the child–parent–therapist triad. More than three fourths of the participating triads disagreed on target problems at the beginning of treatment. Therapists exhibited significantly higher agreement with parents versus children on the majority of target problems except for family and environmental problems where the therapists agreed more with child self-report than parent report. Parent–adolescent agreement on reports of functional impairment was low, and clinician agreement was variable. Overall, clinicians rated impairment as more severe when both the parent and adolescent endorsed the problem; however, when a problem was reported a single informant, the weight given to the report by the clinician varied by problem type. For example, for problems endorsed by a parent alone, clinicians generally weighted the severity as equal to a problem endorsed by both the parent and adolescent. In contrast, some areas of impairment such as sibling issues or school work were rated as less severe when only reported by the adolescent versus by both the parent and adolescent, whereas adolescent-only reports of illegal behavior were rated more serious than parent-only reports. Clinician’s ratings of adolescent manic symptoms were more strongly correlated with parent reports relative to adolescent self-reports.

the measures administered to parents and children were identical to or slightly modified versions of those used in our illustration of the incremental validity of multi-informant assessments (i.e., Hawley & Weisz, 2003; Youngstrom et al., 2004b). One of the other studies administered functional impairment measures to parents and adolescents that share many of the characteristics of the measures used in our Table 3 illustration (e.g., similar item content, focus on last 6 months of functioning, Likert-type scaling; Kramer et al., 2004). Thus, differential measurement validity between parent and child reports may not parsimoniously explain findings from Hawley and Weisz (2003) and Youngstrom and colleagues (2004b). The remainder of the studies described in Table 4 focused on assessments of child anxiety symptoms and diagnoses (Brown-

Did clinicians’ decisions correspond to a greater extent with one informant relative to other informants? Yes, but clinicians’ decisions corresponded more with parent reports relative to child selfreports, depending on the measurement methodology

Yes (Parent ⬎ Child)

Yes (Parent ⬎ Child)

Yes (Parent ⬎ Child) Yes, but clinicians’ decisions corresponded more with parent reports or child self-reports, depending on the problem domain Yes, but clinicians’ decisions corresponded more with parent reports or adolescent self-reports, depending on the problem domain

Yes (Parent ⬎ Child)

Jacobsen et al., 2011; De Los Reyes, Alfano et al., 2011; DiBartolo et al., 1998; Grills & Ollendick, 2003). With child anxiety concerns, it is important to note that decades of research attest to the reliability and validity of child self-reports for assessing these concerns (for reviews, see Silverman & Ollendick, 2005; Tulbure et al., 2012). As mentioned previously, the lack of correspondence between parent reports and patient self-reports of child anxiety does not necessarily indicate that patients provide self-reports of poor quality (e.g., Comer & Kendall, 2004; De Los Reyes, Aldao et al., 2012). In fact, even when parent reports and child selfreports of child anxiety yield low correspondence, these reports may all exhibit identical factor structures (Baldwin & Dadds, 2007). Most crucially, recent work suggests that children provide incrementally valid self-reports of anxiety, relative to parent re-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

ports, in the prediction of child anxiety disorder diagnoses (Villabø et al., 2012). Overall, when mental health professionals encounter discrepant assessment outcomes from multiple informants’ reports, they make clinical decisions that tend to correspond with a single informant’s report to a greater extent than the reports of other informants involved in the assessment. Sometimes in the cases of child patients, this informant may be a nonparental informant (e.g., patient); however, most often this informant is the parent, or the informant who mental health professionals rely on most to initiate children’s clinical services. Consequently, study design considerations in incremental validity research preclude firm conclusions as to the support for use of the multi-informant approach in clinical child assessments. To further illustrate these issues, in Table 5 we describe the characteristics of the incremental validity studies reviewed in Table 2. These characteristics highlight three issues concerning prior work. First, studies on use and interpretation of multiple informants’ reports require, at a minimum, informants to complete reports that hold measurement content, scaling, and response options constant across reports to control the methodology through which informants provide reports (De Los Reyes, Thomas et al., 2013). However, at times incremental validity studies involve informants completing reports via measures that contain different items, scaling, and response options (see Table

877

5). Here, findings may reflect differential psychometric properties among measures, rather than evidence supportive of the incremental validity of reports. Second, studies have not adequately controlled for criterion contamination (see Table 5). For instance, many studies evaluate incremental validity using clinicians’ diagnostic decisions (i.e., via structured interviews administered to informants) as the key criterion by which to compare informants’ reports. Yet, these criterion measures were derived through clinical interviews made by evaluators who had access to the reports of the same informants being evaluated for incremental validity (see Table 5). Based on the findings reported in Table 4, this is an important consideration: In the presence of discrepant informants’ reports, clinical evaluators tend to make clinical decisions that most closely correspond with the informant initiating clinical services. In these studies, one interpretation is that the referral informant (e.g., parent) provided incrementally valid information not because of the validity of their reports per se, but because they informed assessors’ hypotheses regarding the referral question and thus rationale and structure of the clinical evaluation (Croskerry, 2003; Garb, 1998). Third, contextual variations in patients’ mental health concerns may influence the extent to which a given informant’s report is found to be incrementally valid (see Figure 2). For example, consider the idea that a given mental health concern, such as social

Table 5 Summary of Characteristics in Studies Examining the Incremental Validity of a Multi-Informant Approach to the Assessment of Mental Health

Descriptive information

Mental health domain ADHD

Author group (year)

Informants

Contextual variations in assessed behavior

Criterion measures

Criterion Independent Use of the same independent from assessment of measure format informants prevalence of to gather Information sources examined for Objective contextual variations informants’ used for criterion incremental criterion in participants’ reports measures validity useda behavior

Power et al. (1998)

P, T



Smith, Pelham, Gnagy, Molina, & Evans (2000)

C, T

/

P, T C, P

✓ ✓

C, P, T C, P

Ladd & KochenderferLadd (2002) Kwon, Kim, & Sheridan (2012)

ADHD subtypes and ODD Owens & Hoza (2003) Anxiety Villabø, Gere, Torgersen, March, & Kendall (2012) Bipolar Disorder Youngstrom et al. (2004a) ODD Angold & Costello (1996)

Relational Adjustment Social Skills and Academic Competence

Informants examined for incremental validity

Combined diagnostic interview and T report Independent observers

X

X

X





X

P, T Diagnostic interview

X X

X X

X X

/ ✓

Diagnostic interview Diagnostic interview; P and C interview reports of impairment, parental burden, and service use

X X

X X

X X

C, P, T, Peer

/

C, P, T, Peer

X

X

X

T, Peer

X

C, T, Peer

/

X

X

Note. ✓ ⫽ study utilized characteristic described by column header; / ⫽ study utilized characteristic described by column header for some but not all measures; X ⫽ study did not utilize characteristic described by column header; C ⫽ Child; P ⫽ Parent; T ⫽ Teacher; ADHD ⫽ attention-deficit/ hyperactivity disorder; ODD ⫽ oppositional defiant behaviors. a Criterion based on objective measure (e.g., record reviews, behavioral observations, or behavioral measures).

DE LOS REYES ET AL.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

878

anxiety (Bögels et al., 2010), may manifest itself differently from one patient to the next (see Figures 2a, 2b, and 2c). Further, consider that a researcher conducts an incremental validity study using three informants’ reports, namely parents, teachers, and peers. In this study, patients in the sample only displayed mental health concerns within a single context (i.e., home), and thus only one of the three informants (i.e., parents) primarily observed patients in this context. The results of this study may yield incremental validity evidence in support of parent reports and not the teacher and peer reports, but only because parents had access to observing the behaviors assessed, and not teachers and peers. Unfortunately, although some of the incremental validity studies in Table 5 included independent measures (i.e., beyond the informants’ reports examined) of patients’ mental health concerns, none assessed and reported the prevalence rate of context-specific displays of patients’ mental health concerns. Stated another way, previous incremental validity studies have not reported estimates of which patients displayed their concerns within specific contexts (e.g., home or school) or across multiple contexts. Without the availability of these data, one cannot address the methodological confound that the study’s findings in support of a given informant’s report may have arisen because of a lack of variation in the contexts in which patients displayed concerns (cf. Figures 2a and 2b). Consequently, findings from such a study might not generalize to other samples of patients who display qualitatively distinct contextual variations in mental health concerns. In sum, study design issues preclude our ability to discern whether multiinformant clinical child assessments demonstrate incremental validity.

Construct Validity of Multi-Informant Mental Health Assessments Theoretical and Methodological Approaches to Construct Validity Our perspective on the construct validity of multi-informant assessments draws from the definitions provided by Borsboom, Mellenbergh, and van Heerden (2004). We also draw from core principles of convergent and divergent validity, and the multitrait/ multimethod (MTMM) matrix (e.g., Campbell & Fiske, 1959; Westen & Rosenthal, 2003, 2005). Specifically, in a multiinformant assessment of a given domain of patients’ concerns (e.g., aggressive behavior), scores from any one informant’s report may validly reflect levels of concerns on that domain. However, one informant’s report alone cannot estimate whether patients display concerns across contexts (e.g., Kraemer et al., 2003); knowledge of multiple contexts requires multiple informants. If we were to use MTMM matrix terminology (see Campbell & Fiske, 1959), our focus is on assessments in which (a) multiple informants complete reports about the same domain (i.e., mono-domain); (b) each informant’s report reflects observations within a single context (i.e., mono-context); and (c) patterns of scores across multiple informants reflect contextual variations in patients’ concerns (i.e., multicontext). By examining patterns of scores from these “mono-domain/multicontext” assessments, one can assess the construct validity of multi-informant assessments. Specifically, if an independent assessment of patients’ concerns reveals that

they contextually vary in where they display concerns, then one should observe large discrepancies among the reports of informants who observe patients’ behavior in different contexts. Alternatively, if an independent assessment reveals that patients display concerns consistently across contexts, then one should observe high correspondence among informants’ reports of patients’ concerns. These patterns would, in effect, support the construct validity of interpreting multi-informant assessments as reflecting the extent to which patients contextually vary in displays of their concerns. To this end, we review research taking exactly this approach to understanding the construct validity of scores taken from multi-informant clinical child assessments.

Construct Validity in Multi-Informant Clinical Child Assessments Until recently, the best evidence in support of the construct validity of multi-informant assessments came from the Achenbach et al. (1987) meta-analysis, as described previously. However, these findings provide an incomplete picture of construct validity. That is, similar to the incremental validity research reviewed previously, strong construct validity evidence would involve studies that compare the points of convergence and divergence between informants’ reports to independent and contextually sensitive measures of patients’ concerns (e.g., DB-DOS; see Wakschlag et al., 2010). With this independent assessment, one can examine whether an informant’s report relates most strongly with behavioral observations of patients within the observational context that most closely matches their own context of observation (e.g., parent at home vs. teacher at school; see Figure 2). Interestingly, recent research has expanded upon work by Achenbach et al. (1987) by examining multi-informant assessments in reference to independent assessments of contextual variations in patients’ mental health concerns. Two studies support the construct validity of the multiinformant approach. First, a study of 327 preschool children aged 3–5 years found that children varied in the contexts in which they displayed disruptive behavior symptoms (De Los Reyes et al., 2009). Specifically, on the DB-DOS described previously, 29.4% of the children displayed disruptive behavior exclusively within interactions with parental adults, and 15% displayed disruptive behavior exclusively within interactions with nonparental adults (i.e., unfamiliar clinical examiners). Further, 8.8% of the children displayed disruptive behavior across interactions with parental and nonparental adults, and 46.8% did not display disruptive behavior within any of these interactions. Thus, De Los Reyes, Henry, Tolan, and Wakschlag (2009) examined a heterogeneous sample of children whose disruptive behavior was specific to home rather than nonhome contexts (Figure 2a), specific to nonhome rather than home contexts (Figure 2b), or nonspecific or displayed across home and nonhome contexts (Figure 2c), in addition to children who were unlikely to display disruptive behavior in any of the contexts. Further, in this study, parents’ reports of children’s disruptive behavior more closely “matched” how children behaved within interactions with parental adults than with nonparental adults, whereas teachers’ reports more closely matched how children behaved within interactions with nonparental adults than with parental adults.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

Second, findings consistent with those of De Los Reyes et al. (2009) have been observed in a study of early adolescents (N ⫽ 123; mean age ⫽ 13.30), which focused on links between parents’ and teachers’ reports of aggressive behavior and the social experiences that tended to elicit these behaviors (Hartley et al., 2011). Specifically, Hartley, Zakriski, and Wright (2011) found that as parents and teachers provided increasingly similar reports of children’s aggressive behavior, the similarities increased between the kinds of social experiences in which parents and teachers reported observing the children exhibiting aggressive behavior (e.g., peer interactions or receiving instructions from adult authority figures). Overall, when assessing children for displays of observable mental health concerns (i.e., aggressive and disruptive behavior), preliminary evidence indicates that informants who differ in where they observe children (i.e., parents and teachers) provide reports that map onto contextual variations in such concerns.

Discussion Main Findings In this article, we reviewed research on the validity of the multi-informant approach to clinical child assessment. We conducted a quantitative review of 341 studies reporting estimates of cross-informant correspondence in reports about children’s mental health (see Table 1), representing the last quarter-century of research on the topic. We used the findings from this review as a backdrop for discussing recent conceptual work on how multiinformant assessments may reveal information about the specific contexts in which patients display mental health concerns. Guided by this conceptual work, we made specific predictions about the incremental and construct validity of multi-informant assessments. Based on these predictions, we evaluated whether the evidence supports using multi-informant assessments to measure contextspecific displays of children’s concerns. We sought to inform future research, as well as make recommendations for clinical practice on using and interpreting multi-informant assessments. We made three findings. First, the strongest evidence in support of the multi-informant approach comes from studies of crossinformant correspondence. Our quantitative review indicated that multiple informants’ reports share little variance with each other (see Table 1). Further, moderators of cross-informant correspondence indicated that informants’ reports tend to correspond at higher levels when informants (a) provide reports about concerns that are relatively easy to observe (e.g., externalizing vs. internalizing concerns); (b) observe children’s concerns from the same setting (e.g., mother and father reports vs. parent and teacher reports); and (c) complete reports using dimensional measures (i.e., relative to categorical measures). Second, research on the incremental validity of multi-informant assessments is relatively underdeveloped relative to research on cross-informant correspondence. Importantly, our review should not be taken as an indication that multi-informant assessments cannot be incrementally valid. Rather, incremental validity studies generally have been constructed to address research questions such as, Which specific informant(s) provide incrementally valid reports within assessments of mental health domain X and patient Group Y? In fact, there is a dearth of literature systematically evaluating which sets of informants provide incrementally valid reports. Such work

879

might take the form of studies constructed to address research questions such as, For mental health domain X and patient Group Y, do multi-informant assessments consisting of parent, teacher, and child reports provide incrementally valid information, relative to use of any one informant’s report? Third, our review of the construct validity of multi-informant assessments focused on studies examining the “match” between informant discrepancies in reports of patients’ concerns and independent, context-sensitive assessments of patients’ behavior. Two studies support the ability of multi-informant assessments to reflect contextual variations in patients’ behavior for parent and teacher reports of (a) preschoolers’ disruptive behavior, and (b) early adolescents’ aggressive behavior. Thus, studies support the construct validity of the multiinformant approach, and yet for a limited set of informants, mental health domains, and patients’ developmental periods.

Recommendations for Clinical Practice A key goal of this article was to provide individuals conducting clinical work guidance on when the evidence supports interpreting the outcomes of multi-informant assessments as reflections of contextual variations in patients’ mental health concerns. To this end, we identified evidence indicating that at times, multiinformant clinical child assessments pinpoint the specific contexts in which patients display mental health concerns. However, currently we do not have “best practices” for using and interpreting multi-informant assessments in clinical work. In line with these gaps in best practices, we can make three recommendations for applying the multi-informant approach to clinical work. Using multi-informant assessments to personalize patient care. First, a key implication of the work reviewed above is that practitioners may observe significant within- and between-patient variations in terms of the specific contexts in which patients display concerns (see Figure 2). This points to a key idea: Not all patients require or may be suited for the same mental health assessment procedures administered in a single way (e.g., NIMH, 2008). In line with this view, providing proper patient care may involve practitioners seeking to personalize assessments to optimally fit or address patients’ unique needs (see Figure 2). In an effort to innovate personalized approaches to mental health care, the NIMH recently released a strategic plan for its funding priorities, which includes broadening outcome measurement so as to tailor assessments and interventions to patients’ unique needs (Strategy 3.2; NIMH, 2008). Consequently, if a clinician wishes to administer a mental health assessment that is personalized to the unique needs of each patient, the work reviewed above indicates that this assessment should be able to capture information about the specific contexts in which the patient requires care. In this respect, future work may inform the development of contextually sensitive assessments that improve patient care. For example, as mentioned previously social anxiety patients vary in that some experience social anxiety across social contexts and others within specific contexts (i.e., public speaking; Bögels et al., 2010). This creates the potential for patients to display anxiety in some contexts but not others. Consequently, a practitioner ought to administer treatment planning assessments personalized to the unique needs of the patient. For instance, assessments can be tailored to classify different patients who display social

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

880

DE LOS REYES ET AL.

anxiety mostly during public speaking (i.e., performance anxiety), or both preceding (i.e., anticipatory anxiety) and during public speaking. Based on these assessments, treatments can be tailored to patients experiencing predominantly performance anxiety (e.g., behavioral exposure) and/or anticipatory anxiety (e.g., emotion regulation strategies). Thus, practitioners might leverage assessment methods to identify and implement treatment techniques tailored to each patient. Such assessments may result in costeffective symptom reduction and improved functioning, relative to existing treatments that may or may not effectively target the unique needs of each patient. Developing new multi-informant assessment procedures. Second, new methods for using and interpreting multi-informant assessments might assist in developing the personalized assessment approaches we just described. To this end, recent theoretical work on interpreting multi-informant assessment outcomes in research (e.g., OTM; De Los Reyes, Thomas et al., 2013) may assist in future efforts toward developing evidence-based practices in interpreting multi-informant assessment outcomes. Specifically, a key feature of the OTM is based on the idea that practitioners have a great deal of evidence indicating that multiple informants’ reports will often yield discrepant outcomes and in predictable ways (e.g., Achenbach et al., 1987; De Los Reyes & Kazdin, 2005; Kraemer et al., 2003). As a result, the OTM guides practitioners toward (a) hypothesizing patterns of convergence and divergence among informants’ reports, and then (b) constructing assessments that directly test these hypothesized patterns. For example, a practitioner with a referral question involving an initial assessment for a child’s hyperactivity might look to the research literature on typical rates of reporting discrepancies between parent and teacher hyperactivity reports. The practitioner could also conduct a short unstructured phone interview with the parent and teacher before conducting the assessment. Taking precautions not to ask leading questions that might bias informants’ reports (Groth-Marnat, 2009), the practitioner could probe for similarities and differences between the home and school contexts, such as in how parent and teacher manage the child’s behavior within and across contexts. Using this contextual information as a guide, the practitioner would predict the patterns of information to be gleaned from parent and teacher reports. The practitioner would then structure the assessment to make sense of the parent and teacher reports. Specifically, based on the qualitative data the practitioner might expect the patient’s hyperactivity to vary by contextual demands. These variations might reflect that adult authority figures within the home and school contexts (i.e., parents vs. teachers) differ in the strategies used to manage the patient’s behavior, with more effective management strategies present in the school context. These variations might result in the practitioner predicting informants’ reports across contexts to vary in levels of hyperactivity, with greater levels of hyperactivity reported in the home context, relative to the school context. The result may be collecting reports of children’s hyperactivity from parents and teachers. To corroborate the meaning behind convergence or divergence between parent and teacher reports, the practitioner might also administer independent observational assessments of the child’s behavior. These observational assessments might include having the child complete a set of structured and unstructured tasks in the home and school contexts. These observational assessments might resemble

the contextually sensitive observational assessments developed to assess disruptive behavior among preschool children (Wakschlag et al., 2010). Let us assume that the data confirmed the practitioner’s hypotheses: observations of greater parent-reported levels of hyperactivity, relative to teacher report. If so, the independent assessment would allow the practitioner to make an informed decision as to whether the parent and teacher reports varied for meaningful reasons. For instance, the independent assessment might have revealed the presence of higher levels of hyperactivity and less effective behavior management strategies at home relative to school. The key point to be taken from this example is that assessment procedures guiding clinical decision-making would greatly benefit from a priori predictions of patterns of convergence and divergence between informants’ reports. These predictions can inform the gathering of independent qualitative and quantitative data about the contexts in which patients display concerns. In turn, these data and predictions may result in greater certainty when interpreting the outcomes of multi-informant assessments, resulting in increased reliability and validity in clinical decision-making. Integrating the OTM with approaches from evidence-based medicine. Developing personalized assessment procedures that leverage multiple informants’ reports could be further facilitated by incorporating knowledge from other literatures focused on patient-centered models of clinical decision-making. Specifically, assessment methods within evidence-based medicine (EBM) often involve forming likelihood judgments for an assessment case, based on whether one or more pieces of information collected by an assessor support or fail to support a given conclusion about that patient’s clinical presentation (for a review, see Youngstrom, 2013). As an example, consider a practitioner who collects clinical information about a patient who may evidence concerns with oppositional defiant disorder (ODD). The difference between EBM assessments and traditional psychological assessment procedures is that in EBM, one assigns specific values to data derived from information sources, based on whether these data support one outcome, relative to an alternative outcome. For instance, to assess a patient for the presence versus absence of an ODD diagnosis, a practitioner might begin by consulting prior research on risk and associated features of ODD. Based on prior research, the practitioner would pose a clinical question to inform gathering clinical information about the patient, such as, What are the base rates of inconsistent parenting practices among the parents of patients diagnosed with ODD and among nonpatients? The practitioner would administer to the parent a short survey assessment of inconsistent parenting. The practitioner would compare the outcomes of this survey against known base rates of inconsistent parenting. Essentially, the practitioner examines whether the parent’s inconsistent parenting outcomes “matched” the outcomes identified in the literature for parents of patients diagnosed with ODD, relative to an alternative outcome (i.e., outcomes of inconsistent parenting for parents of nonpatients). The resulting comparisons, referred to as diagnostic likelihood ratios (DLRs) are essentially estimates of the ratio of true positives-to-false positives (e.g., ratio of the base rate of inconsistent parenting practices among the parents of ODD patients/ratio for parents of nonpatients). These DLRs carry with them interpretative conventions similar to odds ratios (OR) calculated within logistic regression (see Straus, Glasziou, Richardson, & Haynes, 2011). That is, DLRs at or below 2 carry little value in terms of

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

guiding clinical decisions, DLRs of approximately 5 carry relatively greater value, and DLRs of 10 or above may ensure sound confidence in making the particular clinical decision in question (e.g., ruling in an ODD diagnosis; see also Youngstrom, 2013). With this ODD example, if the parenting information about the patient yielded a high DLR, the practitioner might conclude that the patient is quite likely to meet diagnostic criteria for ODD and begin treatment for the condition. The practitioner may conduct additional assessments for the presence of commonly comorbid conditions (e.g., ADHD; see Drabick & Kendall, 2010). Within an EBM assessment, multiple informants’ reports might prove particularly useful for treatment planning. To continue with our previous example, perhaps DLRs for the presence of an ODD diagnosis were based primarily on useful, easy-to-collect information about inconsistent parenting. However, the information that informed initial diagnosis did not yield sensitive data on whether the patient evidences ODD concerns specific to particular contexts (e.g., home only vs. home and school). Yet, knowledge about the context(s) within which the patient evidences concerns would greatly facilitate treatment planning. In this sense, a multiinformant assessment consisting of behavioral checklists administered to the patient’s parents and teachers could yield great utility in revealing context-specificity in displays of the patient’s ODD concerns. Here, the clinical question might be, What is the likelihood that the patient’s concerns are pervasive across contexts or specific to the home context? To address this question, the practitioner would administer survey reports to the patient’s parent and teacher. Additionally, the practitioner would consult prior work to identify estimates of patients exhibiting disruptive behavior in interactions with parental and nonparental adults, perhaps based on data from the DB-DOS (e.g., De Los Reyes et al., 2009). Within these estimates, the practitioner would identify the base rates of patients exhibiting cross-contextual displays of disruptive behavior on the DB-DOS who also evidenced disruptive behavior across parent and teacher reports versus only parent report (i.e., 29% vs. 10%; see Table 5 of De Los Reyes et al., 2009). These estimates would translate to a DLR of approximately 2.9. Based on this DLR figure, if the patient in fact evidenced disruptive behavior concerns across parent and teacher reports, the data would point to the patient displaying concerns across contexts. Treatment would then progress based on the implementation of techniques at both home and school. One additional issue warrants comment. In this EBM example, the only data collected on behalf of the patient were the parent and teacher reports. That is, the practitioner relied on parent and teacher report data to make clinical decisions (i.e., determine whether patient displayed concerns across home and school contexts). However, the practitioner turned to independent behavioral assessment data collected in prior published work to construct the DLR that was used to interpret the parent and teacher reports (De Los Reyes et al., 2009). The value in this approach is that the practitioner consulted prior published work to interpret the survey reports they collected from the patient’s parent and teacher, and gauge whether the reports contained information that should guide decision-making. This process is quite similar to current practices in interpreting clinic data, such as when a mental health professional consults published clinical cutoff values for a symptom scale to determine whether a patient evidences clinically relevant mental health concerns. Thus, this illustration highlights that meth-

881

ods taken from EBM hold promise for improving the use and interpretation of multi-informant assessments.

Research and Theoretical Implications Addressing methodological issues raised by incremental validity research. This review has important research and theoretical implications. Specifically, future multi-informant assessment research would benefit greatly from addressing the methodological confounds we discussed previously with regard to prior incremental validity research. We have three recommendations for addressing particularly pressing methodological issues. First, future incremental validity studies should involve use of measures completed by multiple informants that hold item content, response labeling, and scaling constant, in order to rule out methodological differences among reports when drawing inferences about incremental validity. Relatedly, as in research on preschool disruptive behavior (i.e., DB-DOS; Wakschlag et al., 2010), future research should focus on developing measures that can assess contextual variations in patients’ mental health concerns, independently from the informants completing reports about patients. Using these measures, researchers can gauge, for instance, whether findings supporting the incremental validity of parent but not teacher reports were driven by the propensity of patients in the sample to display mental health concerns only observable by parents (Figure 2a). Second, a benchmark or standard does not exist for identifying an incrementally valid informant for assessing particular mental health concerns for particular patient groups. That is, mental health researchers have developed a number of guidelines for making evidence-based decisions, most notably with regard to identifying efficacious treatments (for a review, see Roth & Fonagy, 2005). However, there does not exist specific criteria for determining how much incremental validity (e.g., predictive power, represented in effect size metrics) is sufficient to conclude that a given informant’s report should be included in an assessment, let alone whether multiple informants’ reports ought to be collected versus only a single report. This is an important consideration, in that one tends to observe small effects for the incremental validity of a given informant’s report in relation to other informants’ reports (Hunsley & Meyer, 2003). Third, incremental validity research would benefit from basic efficacy studies of whether informants can provide incrementally valid reports when predicting validity criterion measures constructed blind or independent of all of the informants’ reports. This work would focus on asking the question, All things being equal, are scores taken from multi-informant assessments incrementally valid, relative to scores taken from any one informant? However, like treatment efficacy research, this type of incremental validity research might involve use of validity criterion measures that do not reflect clinical indices or decisions made in “real-world” settings. Thus, in tandem with basic efficacy studies, we recommend a focus on incremental validity effectiveness research. This form of incremental validity work would focus on asking the question, Are scores taken from multi-informant assessments incrementally valid in predicting “real-world” clinically meaningful criterion measures, relative to scores taken from any one informant? Essentially, these incremental validity studies would share many elements with prior incremental validity work, most notably the focus on such validity criterion measures as consensus diag-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

882

DE LOS REYES ET AL.

noses and treatment response. However, the difference between prior research and the efficacy and effectiveness variants of incremental validity research we propose would be in systematically ruling out the effects of criterion contamination on the outcomes of incremental validity studies. Using basic mental health research to inform incremental validity research. Much of our discussion of the multiinformant approach makes a fundamental assumption that mental health concerns manifest such that patients display signs of these concerns within some contexts and not others (Figures 2a and 2b) or across contexts (Figure 2c). The importance of understanding these contextual variations may be relevant to some (e.g., social anxiety, attention and hyperactivity), but not all, mental health concerns. Indeed, displays of some concerns may rarely vary across contexts (e.g., schizophrenia). Our findings highlight the importance of basic research on contextual variations in mental health for informing the use and interpretation of multiple informants’ reports, particularly with regard to incremental validity. Indeed, we discussed previously how lack of knowledge on contextual variations in mental health concerns may hinder our ability to draw conclusions on the generalizability of findings from incremental validity studies. That is, the composition of contextual variations in mental health concerns in a sample may often dictate the extent to which any one informant provides incrementally valid reports (see Figure 2). Consequently, future research must link our basic understanding of contextual variations in patients’ mental health concerns with understanding and interpreting findings in incremental validity research. We previously described observational measures used in multiinformant research on preschool disruptive behavior that have revealed individual differences among patients in the specific contexts in which they displayed disruptive behavior concerns (De Los Reyes et al., 2009). We recommend that future incremental validity studies implement independent, contextually sensitive measures such as these to estimate contextual variations in patients’ concerns. Using these measures, researchers may identify, for example, whether the sample was predominantly represented by patients who displayed mental health concerns exclusively within the home context (Figure 2a). In assessments of preschoolers’ disruptive behavior, such a patient sample would be likely to result in parent reports evidencing incrementally valid reports to a far greater extent than other informants’ reports, such as teachers. Yet, we previously discussed research indicating that preschoolers may evidence mental health concerns within and across a variety of contexts (e.g., Dirks et al., 2012). Thus, samples of patients displaying heterogeneity in mental health concerns within and across contexts may yield distinct estimates of incremental validity for parent and teacher reports (see Figures 2a and 2b). Consequently, the availability of contextually sensitive measures independent of the informants providing reports about patients would provide researchers with important data to rule out important methodological confounds. Few measures exist for assessing contextually sensitive variations in patients’ mental health concerns in a way that is independent from the informants providing mental health reports (e.g., using trained laboratory observers or clinical evaluators). This lack of contextually sensitive behavioral assessments may pose challenges to future research. To this end, we have two recommendations on the development and testing of these

independent measures. First, previous work illustrates measures for assessing contextual variations in patients’ reactions to authority figures (e.g., DB-DOS; see De Los Reyes et al., 2009). Importantly, a great deal of work points to the relevance of assessing patients’ interactions with authority figures such as parents and teachers for a number of highly prevalent mental health concerns, such as attention and hyperactivity, depressive mood, and disruptive behavior (APA, 2013). This indicates the potential for researchers to develop versions of the DB-DOS tailored to assessing contextual variations in concerns displayed by patients of varying clinical presentations. Further, deficits in communicating and interacting with others in social settings (i.e., social competence deficits) characterize multiple mental health domains assessed and diagnosed among patients across the life span, including attention and hyperactivity, schizophrenia, and social anxiety (APA, 2013). Along these lines, recent work with adult social anxiety patients indicates that the administration of laboratory tasks across varying social interaction contexts (e.g., one-on-one social interactions and public speaking) allow for the detection of patients who display social competence deficits across social interactions or only within specific interactions (Beidel, Rao, Scharfstein, Wong, & Alfano, 2010; De Los Reyes, Bunnell et al., 2013). Thus, we encourage future work on whether independent assessments tailored to assess contextual variations in social competence deficits improve interpretability of multiple informants’ reports of patients of varying developmental periods and clinical presentations. Second, much of our discussion about using contextually sensitive independent measures to test the validity of the multiinformant approach involved use of structured behavioral observations of patients’ behavior. This limited view of independent assessments extends from the fact that tests reviewed previously of the construct validity of the multi-informant approach relied on these or similar approaches (De Los Reyes et al., 2009; Hartley et al., 2011). However, other modalities may prove as useful, if not more useful, than behavioral observations. For instance, rather than take behavioral ratings of patients’ performance within clinically relevant tasks, one could assess patients’ physiology using wireless, ambulatory devices (e.g., heart rate monitors; for a review, see De Los Reyes & Aldao, 2015). The principles underlying this approach would function quite similarly to that of previous examples. Specifically, one might assess changes in physiological arousal across different clinically meaningful contexts to test whether these contextual changes in physiology “match” the differences observed between multiple informants’ reports. For example, one might assess a patient’s physiological arousal when completing a frustrating task indicative of the home context (e.g., discussing a topic that tends to elicit conflict between parent and patient) and also of the school context (e.g., completing a difficult mathematics problem in a noisy room). Here, one might test whether patients who display relatively high levels of arousal during the “math problem” task and relatively low levels during the “conflict discussion” task also tend to be the patients whose teachers report mental health concerns that go uncorroborated by reports completed by parents. Venturing outside use of tasks administered in traditional clinic or research settings, there exist a variety of modalities with great potential for testing the validity of the multi-informant approach. For instance, mental health professionals often seek to integrate multi-informant assessments of mental health with such modalities

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT

as estimates of environmental versus genetic contributions to mental health (Baker, Jacobson, Raine, Lozano, & Bezdjian, 2007; Bartels, Boomsma, Hudziak, van Beijsterveldt, & van den Oord, 2007; Derks, Hudziak, Van Beijsterveldt, Dolan, & Boomsma, 2006), neurobiological assessments (e.g., electroencephalography; Curtis & Cicchetti, 2007), salivary assays of the cortisol awakening response (Bitsika, Sharpley, Sweeney, & McFarlane, 2014), social network analyses (Neal, Cappella, Wagner, & Atkins, 2011), geographical assessments of neighborhood risks (e.g., highcrime and economically disadvantaged areas; Odgers, Caspi, Bates, Sampson, & Moffitt, 2012), and performance-based neuropsychological tests (e.g., tests of executive functioning; Silver, 2014). Future research might involve constructing batteries consisting of multiple independent assessments that vary in their propensity to reflect one informant’s report to a greater extent than other informants’ reports. For example, a research team interested in assessing externalizing concerns (e.g., disruptive behavior disorders and ADHD) might predict that geographic assessments of neighborhood disadvantage near the vicinity of patients’ homes reflect parents’ reports about patients’ concerns to a greater extent than teachers’ reports about patients. Conversely, performancebased assessments of patients’ executive functioning may include use of structured tasks akin to school activities (e.g., completing in-class assignments), and thus these assessments may reflect teachers’ reports about patients to a greater extent than parents’ reports about patients. Armed with this battery, the research team could examine whether the discrepancies between parent and teacher reports represent ecologically valid reflections of contextual variations in patients’ mental health concerns. For example, instances in which parent reports indicate disruptive behavior concerns that go uncorroborated based on teacher reports may reflect patients who evidence both a relatively high likelihood of living in an economically disadvantaged neighborhood and scores in the normative range on standardized tests of executive functioning. In sum, multimodal independent assessments may result in a richer understanding of whether mental health professionals can use the outcomes from multi-informant assessments to draw inferences about the context(s) in which patients display mental health concerns. Expanding construct validity research on multi-informant assessments to additional clinical domains and developmental periods. Research on the construct validity of multi-informant assessments has yielded some promising findings. Yet, the findings are limited in their relevance to assessments focused on a circumscribed set of informants, mental health domains, and patient developmental periods. Clearly, the mental health domains for which patients’ concerns may contextually vary far outnumber the domains within which the available evidence supports the construct validity of the multi-informant approach. Thus, we recommend that in the child literature, future research should focus on testing the construct validity of multi-informant assessments: (a) of children’s mood and anxiety concerns; (b) that incorporate informants other than parent and teacher reports (e.g., clinician, patient, peer reports); and (c) that focus on developmental periods other than preschool and early adolescence. Importantly, mental health researchers have made inroads in recent years on understanding multi-informant approaches to assessing adult mental health (e.g., Oltmanns & Turkheimer, 2009; van der Ende, Verhulst, & Tiemeier, 2012). When assessing adult

883

mental health, self-reports and clinician ratings may be the most often used sources, yet recently researchers also have focused on collecting reports from collateral informants such as spouses or adult caregivers, as well as trained raters (e.g., clinical interviewers and laboratory observers; Achenbach, 2006). Further, metaanalytic reviews of adult mental health assessments have revealed correspondence levels at low-to-moderate magnitudes, similar to the correspondence levels observed in assessments of children (Achenbach, Krukowski, Dumenci, & Ivanova, 2005). As with clinical child assessments, commonly used informants in adult mental health assessments may vary in their unique perspectives on adult patients’ mental health concerns. For example, in multi-informant assessments using patients’ self-reports and clinicians’ reports about patients, reports might also contextually vary. That is, patients may base their reports on how they behave in work and home contexts; clinicians may take this information into account, but their reports also may reflect their trained observations of patients’ behaviors in clinic settings (Achenbach et al., 2005; Groth-Marnat, 2009). Therefore, when assessing adult patients, informants selected to provide reports may differ in their opportunities for observing concerns. To this end, there exists preliminary support for the incremental and construct validity of multi-informant approaches to assessing adult mental health. For example, in the assessment of personality pathology, collateral informants’ reports may yield incremental information over-and-above patients’ self-reports (e.g., Galione & Oltmanns, 2013; Miller, Pilkonis, & Clifton, 2005; Ready, Watson, & Clark, 2002). Further, one study supports the construct validity of multi-informant approaches to assessing adult social anxiety. Specifically, in this study patients completed a series of social interaction tasks from which independent raters assessed patients’ social competence deficits (i.e., one-on-one interactions and a public speaking task; De Los Reyes, Bunnell et al., 2013). Patients and clinicians also completed reports about patients’ internalizing symptoms. Consistent with previous work with children (De Los Reyes et al., 2009; Hartley et al., 2011), patients and clinicians were more likely to agree than disagree that the patient experienced moderate-to-high internalizing symptoms when patients exhibited social competence deficits consistently (i.e., rather than inconsistently) across social tasks. Despite promising findings about the use of multi-informant approaches in adult mental health assessments, this area of work is far underdeveloped relative to multi-informant approaches to clinical child assessments. Indeed, the quantitative review conducted by Achenbach, Krukowski, Dumenci, and Ivanova (2005) evaluated a sample of 108 studies that provided sufficient information to estimate levels of cross-informant correspondence, accounting for 0.2% of the 51,000 articles identified for the review. Additionally, little work exists to estimate the consistency of low-to-moderate levels of correspondence among different developmental periods in adulthood (e.g., emerging adulthood and mid- to late-adulthood; for an exception, see van der Ende et al., 2012). Thus, future research should focus on construct validity as it relates to (a) assessments of other concerns such as mood; (b) use of collateral informants such as spouses and adult caregivers (i.e., in the case of elderly patients); and (c) specific developmental periods.

DE LOS REYES ET AL.

884

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Concluding Comments We provided a conceptual overview of the multi-informant assessment approach, and synthesized cross-informant correspondence, incremental validity, and construct validity research literatures to assess whether the extant data support taking this approach to assessing children’s mental health. Findings from the last 25 years of research on cross-informant correspondence in reports of children’s mental health (see Table 1) essentially mirror findings from the preceding 25 years (Achenbach et al., 1987). These robust findings across 50 years of research have allowed us to develop theoretical principles for interpreting the outcomes of multiinformant assessments (De Los Reyes, Thomas et al., 2013). We used these theoretical principles to critically evaluate research on the incremental and construct validity of the multi-informant assessment approach. In so doing, we identified a series of limitations in the research designs of previous studies testing the incremental and construct validity of multi-informant approaches to assessing children’s mental health. In turn, we must limit our conclusions to stating that the best evidence in support of the validity of the multi-informant approach exists largely for assessments of observable mental health concerns (e.g., aggressive and disruptive behavior). The issues we raised in our review merit further study in patients exhibiting varied mental health concerns (e.g., anxiety, attention problems and hyperactivity, depressive symptoms) and across a wider range of developmental periods (e.g., childhood, adolescence, emerging adulthood, older adulthood). Future research on multi-informant assessments may inform the development of contextually sensitive assessment paradigms that mental health professionals can use to personalize assessment techniques and interventions to fit patients’ unique needs. Identifying when multi-informant assessments reveal information about the specific contexts in which patients display mental health concerns may improve how mental health professionals use this information to make clinical decisions, provide patient care, and evaluate the outcomes of such care.

References References marked with an asterisk indicate studies included in the meta-analysis. Achenbach, T. M. (2011). Commentary: Definitely more than measurement error: But how should we understand and deal with informant discrepancies? Journal of Clinical Child and Adolescent Psychology, 40, 80 – 86. http://dx.doi.org/10.1080/15374416.2011.533416 Achenbach, T. M., Krukowski, R. A., Dumenci, L., & Ivanova, M. Y. (2005). Assessment of adult psychopathology: Meta-analyses and implications of cross-informant correlations. Psychological Bulletin, 131, 361–382. http://dx.doi.org/10.1037/0033-2909.131.3.361 Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/ adolescent behavioral and emotional problems: Implications of crossinformant correlations for situational specificity. Psychological Bulletin, 101, 213–232. http://dx.doi.org/10.1037/0033-2909.101.2.213 Achenbach, T. M., & Rescorla, L. (2001). Manual for the ASEBA schoolage forms and profiles. Burlington, VT: University of Vermont Research Center for Children, Youth, and Families. Achenbach, T. M. (2006). As others see us: Clinical and research implications of cross-informant correlations for psychopathology. Current Directions in Psychological Science, 15, 94 –98. http://dx.doi.org/10.1111/ j.0963-7214.2006.00414.x



Allen, J. P., Manning, N., & Meyer, J. (2010). Tightly linked systems: Reciprocal relations between maternal depressive symptoms and maternal reports of adolescent externalizing behavior. Journal of Abnormal Psychology, 119, 825– 835. http://dx.doi.org/10.1037/a0021081 American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders-text revision (4th ed.) (DSM–IV–TR). Washington, DC: American Psychiatric Association. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.) (DSM-V). Washington, DC: American Psychiatric Association. Anderson, E. R., & Hope, D. A. (2009). The relationship among social phobia, objective and perceived physiological reactivity, and anxiety sensitivity in an adolescent population. Journal of Anxiety Disorders, 23, 18 –26. http://dx.doi.org/10.1016/j.janxdis.2008.03.011 Angold, A., & Costello, J. (1996). The relative diagnostic utility of child and parent reports of oppositional defiant behaviors. International Journal of Methods in Psychiatric Research, 6, 253–259. http://dx.doi.org/ 10.1002/(SICI)1234-988X(199612)6:4⬍253::AID-MPR170⬎3.3.CO; 2-O ⴱ Armistead, L., Klein, K., Forehand, R., & Wierson, M. (1997). Disclosure of parental HIV infection to children in the families of men with hemophilia: Description, outcomes, and the role of family processes. Journal of Family Psychology, 11, 49 – 61. http://dx.doi.org/10.1037/ 0893-3200.11.1.49 ⴱ Arseneault, L., Kim-Cohen, J., Taylor, A., Caspi, A., & Moffit, T. E. (2005). Psychometric evaluation of 5- and 7-year-old children’s selfreports of conduct problems. Journal of Abnormal Child Psychology, 33, 537–550. http://dx.doi.org/10.1007/s10802-005-6736-5 ⴱ Asher, R. A., & Wakefield, J. A., Jr. (1990). The relationship between parents’ personality and their ratings of their preschool children’s behaviors. Personality and Individual Differences, 11, 1131–1136. http:// dx.doi.org/10.1016/0191-8869(90)90024-L ⴱ Auerbach, J. G., Gross-Tsur, V., Manor, O., & Shalev, R. S. (2008). Emotional and behavioral characteristics over a six-year period in youths with persistent and nonpersistent dyscalculia. Journal of Learning Disabilities, 41, 263–273. Retrieved from http://eric.ed.gov/?id⫽EJ791343 ⴱ Auger, R. W. (2004). The accuracy of teacher reports in the identification of middle school students with depressive symptomatology. Psychology in the Schools, 41, 379 –389. http://dx.doi.org/10.1002/pits.10164 ⴱ Baker, B. L., Blacher, J., Crnic, K. A., & Edelbrock, C. (2002). Behavior problems and parenting stress in families of three-year-old children with and without developmental delays. American Journal on Mental Retardation, 107, 433– 444. http://dx.doi.org/10.1352/0895-8017(2002) 107⬍0433:BPAPSI⬎2.0.CO;2 ⴱ Baker, B. L., & Heller, T. L. (1996). Preschool children with externalizing behaviors: Experience of fathers and mothers. Journal of Abnormal Child Psychology, 24, 513–532. http://dx.doi.org/10.1007/BF01441572 ⴱ Baker, B. L., McIntyre, L. L., Blacher, J., Crnic, K., Edelbrock, C., & Low, C. (2003). Pre-school children with and without developmental delay: Behaviour problems and parenting stress over time. Journal of Intellectual Disability Research, 47, 217–230. http://dx.doi.org/10.1046/ j.1365-2788.2003.00484.x Baker, L. A., Jacobson, K. C., Raine, A., Lozano, D. I., & Bezdjian, S. (2007). Genetic and environmental bases of childhood antisocial behavior: A multi-informant twin study. Journal of Abnormal Psychology, 116, 219 –235. http://dx.doi.org/10.1037/0021-843X.116.2.219 ⴱ Baldwin, J. S., & Dadds, M. R. (2007). Reliability and validity of parent and child versions of the Multidimensional Anxiety Scale for Children in community samples. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 252–260. http://dx.doi.org/10.1097/01.chi .0000246065.93200.a1 ⴱ Banez, G. A., & Compas, B. E. (1990). Children’s and parents’ daily stressful events and psychological symptoms. Journal of Abnormal Child Psychology, 18, 591– 605. http://dx.doi.org/10.1007/BF01342749

VALIDITY OF MULTI-INFORMANT ASSESSMENT

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



Barker, E. T., Bornstein, M. H., Putnick, D. L., Hendricks, C., & Suwalsky, J. T. D. (2007). Adolescent-mother agreement about adolescent problem behaviors: Direction and predictors of disagreement. Journal of Youth and Adolescence, 36, 950 –962. http://dx.doi.org/10.1007/s10964006-9164-0 ⴱ Bartels, M., Boomsma, D. I., Hudziak, J. J., Rietveld, M. J. H., van Beijsterveldt, T. C. E. M., & van den Oord, E. J. C. G. (2004). Disentangling genetic, environmental, and rater effects on internalizing and externalizing problem behavior in 10-year-old twins. Twin Research, 7, 162–175. http://dx.doi.org/10.1375/136905204323016140 Bartels, M., Boomsma, D. I., Hudziak, J. J., van Beijsterveldt, T. C., & van den Oord, E. J. (2007). Twins and the study of rater (dis)agreement. Psychological Methods, 12, 451– 466. http://dx.doi.org/10.1037/1082989X.12.4.451 ⴱ Bartels, M., Hudziak, J. J., Boomsma, D. I., Rietveld, M. J. H., Van Beijsterveldt, T. C. E. M., & Van den Oord, E. J. C. G. (2003). A study of parent ratings of internalizing and externalizing problem behavior in 12-year-old twins. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 1351–1359. http://dx.doi.org/10.1097/01.CHI .0000085755.71002.5d ⴱ Bass, J. K., Ayash, C., Betancourt, T. S., Haroz, E. E., Verdeli, H., Neugebauer, R., . . . Bolton, P. A. (2014). Mental health problems of displaced war-affected adolescents in northern Uganda: Patterns of agreement between self and caregiver assessment. Journal of Child and Family Studies, 23, 1000 –1010. http://dx.doi.org/10.1007/s10826-0139755-9 Beidel, D. C., Rao, P. A., Scharfstein, L., Wong, N., & Alfano, C. A. (2010). Social skills and social phobia: An investigation of DSM–IV subtypes. Behaviour Research and Therapy, 48, 992–1001. http://dx.doi .org/10.1016/j.brat.2010.06.005 Beidel, D. C., Turner, S. M., Young, B. J., Ammerman, R. T., Sallee, F. R., & Crosby, L. (2007). Psychopathology of adolescent social phobia. Journal of Psychopathology and Behavioral Assessment, 29, 47–54. http://dx.doi.org/10.1007/s10862-006-9021-1 ⴱ Bein, L. A., Petrik, M. L., Saunders, S. M., & Wojcik, J. V. (2014). Discrepancy between parents and children in reporting of distress and impairment: Association with critical symptoms. Clinical Child Psychology and Psychiatry. Advance online publication. http://dx.doi.org/ 10.1177/1359104514532185 ⴱ Bendersky, M., Bennett, D., & Lewis, M. (2006). Aggression at age 5 as a function of prenatal exposure to cocaine, gender, and environmental risk. Journal of Pediatric Psychology, 31, 71– 84. http://dx.doi.org/ 10.1093/jpepsy/jsj025 ⴱ Benjamin, C. L., Puleo, C. M., & Kendall, P. C. (2011). Informant agreement in treatment gains for child anxiety. Child & Family Behavior Therapy, 33, 199 –216. http://dx.doi.org/10.1080/07317107.2011 .595987 ⴱ Bennett, D. S., Snooks, Q., Llera, S., Vogel, K., Conklin, D., & Varlotta, L. (2008). Monitoring and internalizing symptoms among youths with cystic fibrosis. Children’s Health Care, 37, 278 –292. http://dx.doi.org/ 10.1080/02739610802437426 ⴱ Berg-Nielsen, T. S., Solheim, E., Belsky, J., & Wichstrom, L. (2012). Preschoolers’ psychosocial problems: In the eyes of the beholder? Adding teacher characteristics as determinants of discrepant parent–teacher reports. Child Psychiatry & Human Development, 43, 393– 413. http:// dx.doi.org/10.1007/s10578-011-0271-0 ⴱ Berkhout, L., Hoekman, J., & Goorhuis-Brouwer, S. M. (2012). Psychosocial health of 4-year-old boys and girls, as observed by parents and teachers. European Journal of Developmental Psychology, 9, 500 –507. http://dx.doi.org/10.1080/17405629.2012.662590 ⴱ Bickman, L., Wighton, L. G., Lambert, E. W., Karver, M. S., & Steding, L. (2012). Problems in using diagnosis in child and adolescent mental health services research. Journal of Methods and Measurement in the

885

Social Sciences, 3, 1–26. Retrieved from https://journals.uair.arizona .edu/index.php/jmmss/article/view/16110/16035 Bird, H. R., Gould, M. S., & Staghezza, B. (1992). Aggregating data from multiple informants in child psychiatry epidemiological research. Journal of the American Academy of Child and Adolescent Psychiatry, 31, 78 – 85. http://dx.doi.org/10.1097/00004583-199201000-00012 Bitsika, V., Sharpley, C. F., Sweeney, J. A., & McFarlane, J. R. (2014). HPA and SAM axis responses as correlates of self- vs. parental ratings of anxiety in boys with an Autistic Disorder. Physiology & Behavior, 127, 1–7. http://dx.doi.org/10.1016/j.physbeh.2013.12.011 Blais, M. A., Hilsenroth, M. J., Castlebury, F., Fowler, J. C., & Baity, M. R. (2001). Predicting DSM-IV cluster B personality disorder criteria from MMPI-2 and Rorschach data: A test of incremental validity. Journal of Personality Assessment, 76, 150 –168. http://dx.doi.org/10.1207/ S15327752JPA7601_9 ⴱ Blakeley-Smith, A., Reaven, J., Ridge, K., & Hepburn, S. (2012). Parent– child agreement of anxiety symptoms in youth with autism spectrum disorders. Research in Autism Spectrum Disorders, 6, 707–716. http:// dx.doi.org/10.1016/j.rasd.2011.07.020 Bögels, S. M., Alden, L., Beidel, D. C., Clark, L. A., Pine, D. S., Stein, M. B., & Voncken, M. (2010). Social anxiety disorder: Questions and answers for the DSM-V. Depression and Anxiety, 27, 168 –189. http:// dx.doi.org/10.1002/da.20670 ⴱ Bohnert, A. M., Kane, P., & Garber, J. (2008). Organized activity participation and internalizing and externalizing symptoms: Reciprocal relations during adolescence. Journal of Youth and Adolescence, 37, 239 – 250. http://dx.doi.org/10.1007/s10964-007-9195-1 Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley. http://dx.doi.org/ 10.1002/9780470743386 Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061–1071. http://dx.doi.org/ 10.1037/0033-295X.111.4.1061 Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., . . . Lijmer, J. G. (2003). The STARD statement for reporting studies of diagnostic accuracy: Explanation and elaboration. Clinical Chemistry, 49, 7–18. http://dx.doi.org/10.1373/49.1.7 ⴱ Braaten, E. B., Biederman, J., DiMauro, A., Mick, E., Monuteaux, M. C., Muehl, K., & Faraone, S. V. (2001). Methodological complexities in the diagnosis of major depression in youth: An analysis of mother and youth self-reports. Journal of Child and Adolescent Psychopharmacology, 11, 395– 407. http://dx.doi.org/10.1089/104454601317261573 Brookman-Frazee, L., Haine, R. A., Gabayan, E. N., & Garland, A. F. (2008). Predicting frequency of treatment visits in community-based youth psychotherapy. Psychological Services, 5, 126 –138. http://dx.doi .org/10.1037/1541-1559.5.2.126 ⴱ Brody, G. H., Stoneman, Z., Flor, D., McCrary, C., Hastings, L., & Conyers, O. (1994). Financial resources, parent psychological functioning, parent co-caregiving, and early adolescent competence in rural two-parent African-American families. Child Development, 65, 590 – 605. http://dx.doi.org/10.2307/1131403⬙10.2307/1131403 ⴱ Brown, J. D., Wissow, L. S., Gadomski, A., Zachary, C., Bartlett, E., & Horn, I. (2006). Parent and teacher mental health ratings of children using primary-care services: Interrater agreement and implications for mental health screening. Ambulatory Pediatrics, 6, 347–351. http://dx .doi.org/10.1016/j.ambp.2006.09.004 Brown-Jacobsen, A. M., Wallace, D. P., & Whiteside, S. P. H. (2011). Multimethod, multi-informant agreement, and positive predictive value in the identification of child anxiety disorders using the SCAS and ADIS-C. Assessment, 18, 382–392. http://dx.doi.org/10.1177/ 1073191110375792 ⴱ Burt, S. A., McGue, M., Krueger, R. F., & Iacono, W. G. (2005). How are parent– child conflict and childhood externalizing symptoms related over time? Results from a genetically informative cross-lagged study. Devel-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

886

DE LOS REYES ET AL.

opment and Psychopathology, 17, 145–165. http://dx.doi.org/10.1017/ S095457940505008X ⴱ Buschgens, C. J. M., Aken, M. A. van Swinkels, S. H. N., Ormel, J., Verhulst, F. C., & Buitelaar, J. K. (2010). Externalizing behaviors in preadolescents: Familial risk to externalizing behaviors and perceived parenting styles. European Child & Adolescent Psychiatry, 19, 567–575. http://dx.doi.org/10.1007/s00787-009-0086-8 ⴱ Buschgens, C. J. M., Swinkels, S. H. N., Aken, M. A. van Ormel, J., Verhulst, F. C., & Buitelaar, J. K. (2009). Externalizing behaviors in preadolescents: Familial risk to externalizing behaviors, prenatal and perinatal risks, and their interactions. European Child & Adolescent Psychiatry, 18, 65–74. http://dx.doi.org/10.1007/s00787-008-0704-x Butler, A. C., Chapman, J. E., Forman, E. M., & Beck, A. T. (2006). The empirical status of cognitive-behavioral therapy: A review of metaanalyses. Clinical Psychology Review, 26, 17–31. http://dx.doi.org/ 10.1016/j.cpr.2005.07.003 ⴱ Campbell, S. B., Pierce, E. W., Moore, G., Marakovitz, S., & Newby, K. (1996). Boys’ externalizing problems at elementary school age: Pathways from early behavior problems, maternal control, and family stress. Development and Psychopathology, 8, 701–719. http://dx.doi.org/ 10.1017/S0954579400007379 Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. http://dx.doi.org/10.1037/h0046016 Carey, W. B. (1998). Temperament and behavior problems in the classroom. School Psychology Review, 27, 522–533. Retrieved from http:// psycnet.apa.org/psycinfo/1999-00154-005 Carlson, G. A., & Blader, J. C. (2011). Diagnostic implications of informant disagreement for manic symptoms. Journal of Child and Adolescent Psychopharmacology, 21, 399 – 405. http://dx.doi.org/10.1089/cap .2011.0007 Carlson, G. A., & Youngstrom, E. A. (2003). Clinical implications of pervasive manic symptoms in children. Biological Psychiatry, 53, 1050 –1058. http://dx.doi.org/10.1016/S0006-3223(03)00068-4 ⴱ Carlston, D. L., & Ogles, B. M. (2009). Age, gender, and ethnicity effects on parent– child discrepancy using identical item measures. Journal of Child and Family Studies, 18, 125–135. http://dx.doi.org/10.1007/ s10826-008-9213-2 ⴱ Carter, A. S., Grigorenko, E. L., & Pauls, D. L. (1995). A Russian adaption of the Child Behavior Checklist: Psychometric properties and associations with child and maternal affective symptomatology and family functioning. Journal of Abnormal Child Psychology, 23, 661– 684. http://dx.doi.org/10.1007/BF01447471 ⴱ Carter, R., Jaccard, J., Silverman, W. K., & Pina, A. A. (2009). Pubertal timing and its link to behavioral and emotional problems among ‘at-risk’ African American adolescent girls. Journal of Adolescence, 32, 467– 481. http://dx.doi.org/10.1016/j.adolescence.2008.07.005 ⴱ Caye, A., Machado, J. D., & Rohde, L. A. (2013). Evaluating parental disagreement in ADHD diagnosis: Can we rely on a single report from home? Journal of Attention Disorders. Advance online publication. http://dx.doi.org/10.1177/1087054713504134 ⴱ Choudhury, M. S., Pimentel, S. S., & Kendall, P. C. (2003). Childhood anxiety disorders: Parent– child (dis) agreement using a structured interview for the DSM-IV. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 957–964. http://dx.doi.org/10.1097/01.CHI .0000046898.27264.A2 Cicchetti, D. (1984). The emergence of developmental psychopathology. Child Development, 55, 1–7. http://dx.doi.org/10.2307/1129830 Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Hillsdale, NJ: Erlbaum. ⴱ Cohen, M., Becker, M. G., & Campbell, R. (1990). Relationships among four methods of assessment of children with attention deficit-

hyperactivity disorder. Journal of School Psychology, 28, 189 –202. http://dx.doi.org/10.1016/0022-4405(90)90011-U ⴱ Cohen, N. J., Coyne, J., & Duvall, J. (1993). Adopted and biological children in the clinic: Family, parental and child characteristics. Child Psychology & Psychiatry & Allied Disciplines, 34, 545–562. http://dx .doi.org/10.1111/j.1469-7610.1993.tb01035.x ⴱ Cole, D. A., Hoffman, K., Tram, J. M., & Maxwell, S. E. (2000). Structural differences in parent and child reports of children’s symptoms of depression and anxiety. Psychological Assessment, 12, 174 –185. http://dx.doi.org/10.1037/1040-3590.12.2.174 ⴱ Cole, D. A., & Martin, N. C. (2005). The longitudinal structure of the Children’s Depression Inventory: Testing a latent trait-state model. Psychological Assessment, 17, 144 –155. http://dx.doi.org/10.1037/ 1040-3590.17.2.144 ⴱ Cole, D. A., Peeke, L. G., Martin, J. M., Truglio, R., & Seroczynski, A. D. (1998). A longitudinal look at the relation between depression and anxiety in children and adolescents. Journal of Consulting and Clinical Psychology, 66, 451– 460. http://dx.doi.org/10.1037/0022-006X.66.3 .451 ⴱ Cole, D. A., Truglio, R., & Peeke, L. (1997). Relation between symptoms of anxiety and depression in children: A multitrait-multimethodmultigroup assessment. Journal of Consulting and Clinical Psychology, 65, 110 –119. http://dx.doi.org/10.1037/0022-006X.65.1.110 ⴱ Collishaw, S., Goodman, R., Ford, T., Rabe-Hesketh, S., & Pickles, A. (2009). How far are associations between child, family and community factors and child psychopathology informant-specific and informantgeneral? Journal of Child Psychology and Psychiatry, 50, 571–580. http://dx.doi.org/10.1111/j.1469-7610.2008.02026.x ⴱ Comer, J. S., & Kendall, P. C. (2004). A symptom-level examination of parent-child agreement in the diagnosis of anxious youths. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 878 – 886. http://dx.doi.org/10.1097/01.chi.0000125092.35109.c5 ⴱ Connell, A. M., & Dishion, T. J. (2006). The contribution of peers to monthly variation in adolescent depressed mood: A short-term longitudinal study with time-varying predictors. Development and Psychopathology, 18, 139 –154. http://dx.doi.org/10.1017/S0954579406060081 ⴱ Connolly, A., & Vance, A. (2010). Psychosocial factors associated with parent and teacher reports of aggression in children and adolescents with attention deficit hyperactivity disorder. Australian and New Zealand Journal of Psychiatry, 44, 667– 675. http://dx.doi.org/10.3109/ 00048671003664697 ⴱ Connor, J., & Rueter, M. (2009). Predicting adolescent suicidality: Comparing multiple informants and assessment techniques. Journal of Adolescence, 32, 619 – 631. http://dx.doi.org/10.1016/j.adolescence.2008.06 .005 ⴱ Connors, L. L., Connolly, J., & Toplak, M. E. (2012). Self-reported inattention in early adolescence in a community sample. Journal of Attention Disorders, 16, 60 –70. http://dx.doi.org/10.1177/ 1087054710379734 Conrad, M., & Hammen, C. (1989). Role of maternal depression in perceptions of child maladjustment. Journal of Consulting and Clinical Psychology, 57, 663– 667. http://dx.doi.org/10.1037/0022-006X.57.5 .663 Crick, N. R., Werner, N. E., Casas, J. F., O’Brien, K. M., Nelson, D. A., Grotpeter, J. K., & Markon, K. (1998). Childhood aggression and gender; A new look at an old problem. In D. Bernstein (Ed.), Nebraska Symposium on Motivation (pp. 75–141). Lincoln, NE: University of Nebraska Press. ⴱ Crockenberg, S., & Lourie, A. (1996). Parents’ conflict strategies with children and children’s conflict strategies with peers. Merrill-Palmer Quarterly, 42, 495–518. Retrieved from http://www.jstor.org/stable/ 23087467

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT Croskerry, P. (2003). The importance of cognitive errors in diagnosis and strategies to minimize them. Academic Medicine, 78, 775–780. http:// dx.doi.org/10.1097/00001888-200308000-00003 ⴱ Crowley, S. L., Worchel, F. F., & Ash, M. J. (1992). Self-report, peerreport, and teacher-report measures of childhood depression: An analysis by item. Journal of Personality Assessment, 59, 189 –203. http://dx.doi .org/10.1207/s15327752jpa5901_16 Cuijpers, P., & Dekker, J. (2005). Psychologische behandeling van depressie; een systematisch overzicht van meta-analysen. [Psychological treatment for depression: A systematic review of meta-analyses]. Nederlands Tijdschrift voor Geneeskunde, 149, 1892–1897. Retrieved from http://hdl.handle.net/1871/17373 Curtis, W. J., & Cicchetti, D. (2007). Emotion and resilience: A multilevel investigation of hemispheric electroencephalogram asymmetry and emotion regulation in maltreated and nonmaltreated children. Development and Psychopathology, 19, 811– 840. http://dx.doi .org/10.1017/S0954579407000405 Dadds, M. R., Perrin, S., & Yule, W. (1998). Social desirability and self-reported anxiety in children: An analysis of the RCMAS Lie scale. Journal of Abnormal Child Psychology, 26, 311–317. http://dx.doi.org/ 10.1023/A:1022610702439 ⴱ Daley, T. C. (2006). Perceptions and congruence of symptoms and communication among second-generation Cambodian youth and parents: A matched-control design. Child Psychiatry and Human Development, 37, 39 –53. http://dx.doi.org/10.1007/s10578-006-0018-5 Dawes, R. M. (1999). Two methods for studying the incremental validity of a Rorschach variable. Psychological Assessment, 11, 297–302. http:// dx.doi.org/10.1037/1040-3590.11.3.297 ⴱ Deater-Deckard, K., & Scarr, S. (1996). Parenting stress among dualearner mothers and fathers: Are there gender differences? Journal of Family Psychology, 10, 45–59. http://dx.doi.org/10.1037/0893-3200.10 .1.45 ⴱ deKeyser, L., Svedin, C. G., Agnafors, S., Bladh, M., & Sydsjo, G. (2014). Multi-informant reports of mental health in Swedish-born children of immigrants and children born to non-immigrants – the SESBiC-study. BMC Pediatrics, 14: 95. http://dx.doi.org/10.1186/1471-2431-14-95 ⴱ de la Osa, N., Ezpelata, L., Granero, R., Olaya, B., & Domenech, J. M. (2011). Diagnostic value of the dominic interactive assessment with children exposed to intimate partner violence. Psicothema, 23, 648 – 653. Retrieved from http://www.unioviedo.es/reunido/index.php/PST/article/ viewFile/9139/9003 De Los Reyes, A. (2011). Introduction to the special section: More than measurement error: Discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. Journal of Clinical Child and Adolescent Psychology, 40, 1–9. http://dx.doi.org/ 10.1080/15374416.2011.533405 De Los Reyes, A. (2013). Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Development and Psychopathology, 25, 669 – 682. http://dx.doi.org/ 10.1017/S0954579413000096 De Los Reyes, A., & Aldao, A. (2015). Introduction to the special issue: Toward implementing physiological measures in clinical child and adolescent assessments. Journal of Clinical Child and Adolescent Psychology, 44, 221–237. http://dx.doi.org/10.1080/15374416.2014.891227 ⴱ De Los Reyes, A., Aldao, A., Thomas, S. A., Daruwala, S. E., Swan, A. J., Van Wie, M., . . . Lechner, W. V. (2012). Adolescent self-reports of social anxiety: Can they disagree with objective psychophysiological measures and still be valid? Journal of Psychopathology and Behavioral Assessment, 34, 308 –322. http://dx.doi.org/10.1007/s10862-012-9289-2 ⴱ De Los Reyes, A., Alfano, C. A., & Beidel, D. C. (2010). The relations among measurements of informant discrepancies within a multisite trial of treatments for childhood social phobia. Journal of Abnormal Child Psychology, 38, 395– 404. http://dx.doi.org/10.1007/s10802-009-9373-6



887

De Los Reyes, A., Alfano, C. A., & Beidel, D. C. (2011). Are clinicians’ assessments of improvements in children’s functioning “global”? Journal of Clinical Child and Adolescent Psychology, 40, 281–294. http:// dx.doi.org/10.1080/15374416.2011.546043 De Los Reyes, A., Augenstein, T. M., Aldao, A., Thomas, S. A., Daruwala, S. E., Kline, K., & Regan, T. (2015). Implementing psychophysiology in clinical assessments of adolescent social anxiety: Use of rater judgments based on graphical representations of psychophysiology. Journal of Clinical Child and Adolescent Psychology, 44, 264 –279. http://dx.doi .org/10.1080/15374416.2013.859080 De Los Reyes, A., Bunnell, B. E., & Beidel, D. C. (2013). Informant discrepancies in adult social anxiety disorder assessments: Links with contextual variations in observed behavior. Journal of Abnormal Psychology, 122, 376 –386. http://dx.doi.org/10.1037/a0031150 De Los Reyes, A., Ehrlich, K. B., Swan, A. J., Luo, T., Van Wie, M., & Pabón, S. C. (2013). An experimental test of whether informants can report about child and family behavior based on settings of behavioral expression. Journal of Child and Family Studies, 22, 177–191. http://dx .doi.org/10.1007/s10826-012-9567-3 De Los Reyes, A., Goodman, K. L., Kliewer, W., & Reid-Quiñones, K. (2008). Whose depression relates to discrepancies? Testing relations between informant characteristics and informant discrepancies from both informants’ perspectives. Psychological Assessment, 20, 139 –149. http://dx.doi.org/10.1037/1040-3590.20.2.139 De Los Reyes, A., Goodman, K. L., Kliewer, W., & Reid-Quiñones, K. (2010). The longitudinal consistency of mother-child reporting discrepancies of parental monitoring and their ability to predict child delinquent behaviors two years later. Journal of Youth and Adolescence, 39, 1417– 1430. http://dx.doi.org/10.1007/s10964-009-9496-7 ⴱ De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009). Linking informant discrepancies to observed variations in young children’s disruptive behavior. Journal of Abnormal Child Psychology, 37, 637– 652. http://dx.doi.org/10.1007/s10802-009-9307-3 ⴱ De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrepancies in clinical child research. Psychological Assessment, 16, 330 – 334. http://dx.doi.org/10.1037/1040-3590.16.3.330 De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483–509. http://dx.doi.org/10.1037/0033-2909.131.4.483 De Los Reyes, A., & Kazdin, A. E. (2006). Conceptualizing changes in behavior in intervention research: The range of possible changes model. Psychological Review, 113, 554 –583. http://dx.doi.org/10.1037/0033295X.113.3.554 ⴱ De Los Reyes, A., & Kazdin, A. E. (2006). Informant discrepancies in assessing child dysfunction relate to dysfunction within mother-child interactions. Journal of Child and Family Studies, 15, 645– 663. http:// dx.doi.org/10.1007/s10826-006-9031-3 De Los Reyes, A., Kundey, S. M. A., & Wang, M. (2011). The end of the primary outcome measure: A research agenda for constructing its replacement. Clinical Psychology Review, 31, 829 – 838. http://dx.doi.org/ 10.1016/j.cpr.2011.03.011 De Los Reyes, A., & Marsh, J. K. (2011). Patients’ contexts and their effects on clinicians’ impressions of conduct disorder symptoms. Journal of Clinical Child and Adolescent Psychology, 40, 479 – 485. http:// dx.doi.org/10.1080/15374416.2011.563471 De Los Reyes, A., Salas, S., Menzer, M. M., & Daruwala, S. E. (2013). Criterion validity of interpreting scores from multi-informant statistical interactions as measures of informant discrepancies in psychological assessments of children and adolescents. Psychological Assessment, 25, 509 –519. http://dx.doi.org/10.1037/a0032081 De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. A. (2013). Principles underlying the use of multiple informants’ reports.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

888

DE LOS REYES ET AL.

Annual Review of Clinical Psychology, 9, 123–149. http://dx.doi.org/ 10.1146/annurev-clinpsy-050212-185617 ⴱ De Los Reyes, A., Youngstrom, E. A., Pabón, S. C., Youngstrom, J. K., Feeny, N. C., & Findling, R. L. (2011a). Internal consistency and associated characteristics of informant discrepancies in clinic referred youths age 11 to 17 years. Journal of Clinical Child and Adolescent Psychology, 40, 36 –53. http://dx.doi.org/10.1080/15374416.2011 .533402 De Los Reyes, A., Youngstrom, E. A., Swan, A. J., Youngstrom, J. K., Feeny, N. C., & Findling, R. L. (2011b). Informant discrepancies in clinical reports of youths and interviewers’ impressions of the reliability of informants. Journal of Child and Adolescent Psychopharmacology, 21, 417– 424. http://dx.doi.org/10.1089/cap.2011.0011 ⴱ Dempsey, A. G., Llorens, A., Brewton, C., Mulchandani, S., & GoinKochel, R. P. (2012). Emotional and behavioral adjustment in typically developing siblings of children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42, 1393–1402. http://dx.doi .org/10.1007/s10803-011-1368-9 ⴱ Deng, S., Liu, X., & Roosa, M. W. (2004). Agreement between parent and teacher reports on behavioral problems among Chinese children. Journal of Developmental and Behavioral Pediatrics, 25, 407– 417. http://dx.doi .org/10.1097/00004703-200412000-00004 ⴱ De Nijs, P. F. A., Ferdinand, R. F., de Bruin, E. I., Dekker, M. C. J., van Duijn, C. M., & Verhulst, D. C. (2004). Attention-deficit/hyperactivity disorder (ADHD): Parents’ judgment about school, teachers’ judgment about home. European Child & Adolescent Psychiatry, 13, 315–320. http://dx.doi.org/10.1007/s00787-004-0405-z ⴱ De Pauw, S. S. W., Mervielde, I., De Clercq, B. J., De Fruyt, F., Tremmery, S., & Deboutte, D. (2009). Personality symptoms and selfesteem as correlates of psychopathology in child psychiatric patients: Evaluating multiple informant data. Child Psychiatry and Human Development, 40, 499 –515. http://dx.doi.org/10.1007/s10578-009-0140-2 ⴱ Derks, E. M., Hudziak, J. J., Van Beijsterveldt, C. E., Dolan, C. V., & Boomsma, D. I. (2006). Genetic analyses of maternal and teacher ratings on attention problems in 7-year-old Dutch twins. Behavior Genetics, 36, 833– 844. http://dx.doi.org/10.1007/s10519-006-9084-5 ⴱ Diamond, G. M., Izzard, M. C., Kedar, T., Hutlzer, A., & Mell, H. (2005). Psychological symptoms and drug use severity among Israeli adolescents presenting for outpatient drug abuse treatment. Journal of Adolescence, 28, 495–505. http://dx.doi.org/10.1016/j.adolescence.2004.10 .005 DiBartolo, P. M., Albano, A. M., Barlow, D. H., & Heimberg, R. G. (1998). Cross-informant agreement in the assessment of social phobia in youth. Journal of Abnormal Child Psychology, 26, 213–220. http://dx .doi.org/10.1023/A:1022624318795 ⴱ DiBartolo, P. M., & Grills, A. E. (2006). Who is best at predicting children’s anxiety in response to a social evaluative task? A comparison of child, parent, and teacher reports. Journal of Anxiety Disorders, 20, 630 – 645. http://dx.doi.org/10.1016/j.janxdis.2005.06.003 ⴱ Dirks, M. A., Boyle, M. H., & Georgiades, K. (2011). Psychological symptoms in youth and later socioeconomic functioning: Do associations vary by informant? Journal of Clinical Child and Adolescent Psychology, 40, 10 –22. http://dx.doi.org/10.1080/15374416.2011 .533403 Dirks, M. A., De Los Reyes, A., Briggs-Gowan, M. J., Cella, D., & Wakschlag, L. S. (2012). Embracing not erasing contextual variability in children’s behavior–theory and utility in the selection and use of methods and informants in developmental psychopathology. Journal of Child Psychology and Psychiatry, 53, 558 –574. http://dx.doi.org/10.1111/j .1469-7610.2012.02537.x ⴱ Doctoroff, G. L., & Arnold, D. H. (2004). Parent-rated externalizing behavior in preschoolers: The predictive utility of structured interviews, teacher reports, and classroom observations. Journal of Clinical Child &

Adolescent Psychology, 33, 813– 818. http://dx.doi.org/10.1207/ s15374424jccp3304_16 ⴱ Donenberg, G. R., Emerson, E., Bryant, F. B., Wilson, H., & WeberShifrin, E. (2001). Understanding AIDS-risk behavior among adolescents in psychiatric care: Links to psychopathology and peer relationships. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 642– 653. http://dx.doi.org/10.1097/00004583200106000-00008 ⴱ Douma, J. C. H., Dekker, M. C., Verhulst, F. C., & Koot, H. M. (2006). Self-reports on mental health problems of youth with moderate to borderline intellectual disabilities. Journal of the American Academy of Child & Adolescent Psychiatry, 45, 1224 –1231. http://dx.doi.org/ 10.1097/01.chi.0000233158.21925.95 ⴱ Drabick, D. A. G., Bubier, J., Chen, D., Price, J., & Lanza, I. H. (2011). Source-specific oppositional defiant disorder among inner-city children: Prospective prediction and moderation. Journal of Clinical Child and Adolescent Psychology, 40, 23–35. http://dx.doi.org/10.1080/15374416 .2011.533401 Drabick, D. A. G., Gadow, K. D., & Loney, J. (2007). Source-specific oppositional defiant disorder: Comorbidity and risk factors in referred elementary schoolboys. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 92–101. http://dx.doi.org/10.1097/01.chi .0000242245.00174.90 Drabick, D. A. G., Gadow, K. D., & Loney, J. (2008). Co-occurring ODD and GAD symptom groups: Source-specific syndromes and crossinformant comorbidity. Journal of Clinical Child and Adolescent Psychology, 37, 314 –326. http://dx.doi.org/10.1080/15374410801955862 Drabick, D. A. G., & Kendall, P. C. (2010). Developmental psychopathology and the diagnosis of mental health problems among youth. Clinical Psychology: Science and Practice, 17, 272–280. http://dx.doi.org/10.1111/ j.1468-2850.2010.01219.x ⴱ Du, Y., Kou, J., & Coghill, D. (2008). The validity, reliability and normative scores of the parent, teacher and self report versions of the Strengths and Difficulties Questionnaire in China. Child and Adolescent Psychiatry and Mental Health, 2:8. http://dx.doi.org/10.1186/17532000-2-8 ⴱ Dubi, K., & Schneider, S. (2009). The Picture Anxiety Test (PAT): A new pictorial assessment of anxiety symptoms in young children. Journal of Anxiety Disorders, 23, 1148 –1157. http://dx.doi.org/10.1016/j.janxdis .2009.07.020 Duhig, A. M., Renk, K., Epstein, M. K., & Phares, V. (2000). Interparental agreement on internalizing, externalizing, and total behavior problems: A meta-analysis. Clinical Psychology: Science and Practice, 7, 435– 453. http://dx.doi.org/10.1093/clipsy.7.4.435 ⴱ Duke, N., Ireland, M., & Borowsky, I. W. (2005). Identifying psychosocial problems among youth: Factors associated with youth agreement on a positive parent-completed PSC-17. Child: Care, Health and Development, 31, 563–573. http://dx.doi.org/10.1111/j.1365-2214.2005.00551.x ⴱ Ebesutani, C., Bernstein, A., Martinez, J. I., Chorpita, B. F., & Weisz, J. R. (2011). The youth self report: Applicability and validity across younger and older youths. Journal of Clinical Child and Adolescent Psychology, 40, 338 –346. http://dx.doi.org/10.1080/15374416.2011 .546041 ⴱ Ebesutani, C., Okamura, K., Higa-McMillan, C., & Chorpita, B. F. (2011). A psychometric analysis of the Positive and Negative Affect Schedule for Children–Parent Version in a school sample. Psychological Assessment, 23, 406 – 416. http://dx.doi.org/10.1037/a0022057 ⴱ Efstratopoulou, M., Janssen, R., & Simons, J. (2012). Agreement among physical educators, teachers and parents on children’s behaviors: A multitrait–multimethod design approach. Research in Developmental Disabilities, 33, 1343–1351. http://dx.doi.org/10.1016/j.ridd.2012.03 .015 ⴱ Efstratopoulou, M., Simons, J., & Janssen, R. (2013). Concordance among physical educators’, teachers’, and parents’ perceptions of attention

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT problems in children. Journal of Attention Disorders, 17, 437– 443. http://dx.doi.org/10.1177/1087054711431698 ⴱ Ehrlich, K. B., Cassidy, J., & Dykas, M. J. (2011). Reporter discrepancies among parents, adolescents, and peers: Adolescent attachment and informant depressive symptoms as explanatory factors. Child Development, 82, 999 –1012. http://dx.doi.org/10.1111/j.1467-8624.2010 .01530.x ⴱ Eisenberg, N., Cumberland, A., Spinrad, T. L., Fabes, R. A., Shepard, S. A., Reiser, M., . . . Guthrie, I. K. (2001). The relations of regulation and emotionality to children’s externalizing and internalizing problem behavior. Child Development, 72, 1112–1134. http://dx.doi.org/10.1111/ 1467-8624.00337 ⴱ Eisenberg, N., Valiente, C., Morris, A. S., Fabes, R. A., Cumberland, A., Reiser, M., . . . Losoya, S. (2003). Longitudinal relations among parental emotional expressivity, children’s regulation, and quality of socioemotional functioning. Developmental Psychology, 39, 3–19. http://dx.doi .org/10.1037/0012-1649.39.1.3 ⴱ Eiser, C., Havermans, T., Pancer, M., & Eiser, J. R. (1992). Adjustment to chronic disease in relation to age and gender: Mothers’ and fathers’ reports of their children’s behavior. Journal of Pediatric Psychology, 17, 261–275. http://dx.doi.org/10.1093/jpepsy/17.3.261 ⴱ Epkins, C. C, & Meyers, A. W. (1994). Assessment of childhood depression, anxiety, and aggression: Convergent and discriminant validity of self-, parent-, teacher-, and peer-report measures. Journal of Personality Assessment, 62, 364 –381. http://dx.doi.org/10.1207/ s15327752jpa6202_16 ⴱ Eremsoy, C. E., Karanci, A. N., & Berument, S. K. (2010). How do the predictors of conduct problems/hyperactivity and callous-unemotional (CU) traits in children differ according to mothers’ and teachers’ ratings? Procedia - Social and Behavioral Sciences, 5, 471– 476. http://dx .doi.org/10.1016/j.sbspro.2010.07.126 ⴱ Escobar, M. J., Pereira, X., & Santelices, M. P. (2014). Behavior problems and attachment in adopted and non-adopted adolescents. Children and Youth Services Review, 42, 59 – 66. http://dx.doi.org/10.1016/j .childyouth.2014.04.001 ⴱ Fagot, B. I. (1995). Classification of problem behaviors in young children: A comparison of four systems. Journal of Applied Developmental Psychology, 16, 95–106. http://dx.doi.org/10.1016/0193-3973(95)90018-7 ⴱ Feeney-Kettler, K. A., Kratochwill, T. R., & Kettler, R. J. (2011). Identification of preschool children at risk for emotional and behavioral disorders: Development and validation of a universal screening system. Journal of School Psychology, 49, 197–216. http://dx.doi.org/10.1016/j .jsp.2011.02.001 ⴱ Fellinger, J., Holzinger, D., Sattel, H., & Laucht, M. (2008). Mental health and quality of life in deaf pupils. European Child & Adolescent Psychiatry, 17, 414 – 423. http://dx.doi.org/10.1007/s00787-008-0683-y ⴱ Ferdinand, R. F., van der Ende, J., & Verhulst, F. C. (2004). Parentadolescent disagreement regarding psychopathology in adolescents from the general population as a risk factor for adverse outcome. Journal of Abnormal Psychology, 113, 198 –206. http://dx.doi.org/10.1037/0021843X.113.2.198 ⴱ Ferdinand, R. F., van der Ende, J., & Verhulst, F. C. (2006). Prognostic value of parent–adolescent disagreement in a referred sample. European Child & Adolescent Psychiatry, 15, 156 –162. http://dx.doi.org/10.1007/ s00787-005-0518-z ⴱ Fisher, S. L., Bucholz, K. K., Reich, W., Fox, L., Kuperman, S., Kramer, J., . . . Bierut, L. J. (2006). Teenagers are right—Parents do not know much: An analysis of adolescent–parent agreement on reports of adolescent substance use, abuse, and dependence. Alcoholism: Clinical and Experimental Research, 30, 1699 –1710. http://dx.doi.org/10.1111/j .1530-0277.2006.00205.x ⴱ Foley, D., Rutter, M., Pickles, A., Angold, A., Maes, H., Silberg, J., & Eaves, L. (2004). Informant disagreement for separation anxiety disorder. Journal of the American Academy of Child & Adolescent Psychia-

889

try, 43, 452– 460. http://dx.doi.org/10.1097/00004583-20040400000011 ⴱ Forehand, R., Frame, C. L., Wierson, M., Armistead, L., & Kempton, T. (1991). Assessment of incarcerated juvenile delinquents: Agreement across raters and approaches to psychopathology. Journal of Psychopathology and Behavioral Assessment, 13, 17–25. http://dx.doi.org/ 10.1007/BF00960736 ⴱ Forman, E. M., & Patrick, T. (2005). Assessing children’s appraisals of security in the family system: The development of the Security in the Family System (SIFS) scales. Journal of Child Psychology and Psychiatry, 46, 900 –916. http://dx.doi.org/10.1111/j.1469-7610.2004.00385.x Frank, S. J., Van Egeren, L. A., Fortier, J. L., & Chase, P. (2000). Structural, relative, and absolute agreement between parents’ and adolescent inpatients’ reports of adolescent functional impairment. Journal of Abnormal Child Psychology, 28, 395– 402. http://dx.doi.org/10.1023/ A:1005125211187 ⴱ Frick, P. J., Silverthorn, P., & Evans, C. (1994). Assessment of childhood anxiety using structured interviews: Patterns of agreement among informants and association with maternal anxiety. Psychological Assessment, 6, 372–379. http://dx.doi.org/10.1037/1040-3590.6.4.372 ⴱ Frigerio, A., Cattaneo, C., Cataldo, M. G., Schiatti, A., Molteni, M., & Battaglia, M. (2004). Behavioral and emotional problems among Italian children and adolescents aged 4 to 18 years as reported by parents and teachers. European Journal of Psychological Assessment, 20, 124 –133. http://dx.doi.org/10.1027/1015-5759.20.2.124 ⴱ Frosch, C. A., & Mangelsdorf, S. C. (2001). Marital behavior, parenting behavior, and multiple reports of preschoolers’ behavior problems: Mediation or moderation? Developmental Psychology, 37, 502–519. http:// dx.doi.org/10.1037/0012-1649.37.4.502 ⴱ Funderburk, B. W., Eyberg, S. M., Rich, B. A., & Behar, L. (2003). Further psychometric evaluation of the Eyberg and Behar rating scales for parents and teachers of preschoolers. Early Education and Development, 14, 67– 82. http://dx.doi.org/10.1207/s15566935eed1401_5 ⴱ Fung, J. J., & Lau, A. S. (2010). Factors associated with parent– child (dis)agreement on child behavior and parenting problems in Chinese immigrant families. Journal of Clinical Child & Adolescent Psychology, 39, 314 –327. http://dx.doi.org/10.1080/15374411003691693 ⴱ Gagnon-Oosterwaal, N., Cossette, L., Smolla, N., Pomerleau, A., Malcuit, G., Chicoine, J.-F., . . . Berthiaume, C. (2012). Pre-adoption adversity and self-reported behavior problems in 7 year-old international adoptees. Child Psychiatry & Human Development, 43, 648 – 660. http://dx.doi .org/10.1007/s10578-011-0279-5 Galione, J. N., & Oltmanns, T. F. (2013). Identifying personality pathology associated with major depressive episodes: Incremental validity of informant reports. Journal of Personality Assessment, 95, 625– 632. http:// dx.doi.org/10.1080/00223891.2013.825624 ⴱ Gao, W., Paterson, J., Carter, S., Iustini, L., & Sundborn, G. (2011). Agreement and discordance of parents’ and teachers’ reports of behavioural problems among Pacific children living in New Zealand. Pacific Health Dialog, 17, 65–77. Retrieved from http://forum.nuama.org/ attachments/article/220/132299%20PHD%20Journal.pdf#page⫽65 Garb, H. N. (1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association. Garb, H. N. (2003). Incremental validity and the assessment of psychopathology in adults. Psychological Assessment, 15, 508 –520. http://dx.doi .org/10.1037/1040-3590.15.4.508 ⴱ Garner, A. A., Mrug, S., Hodgens, B., & Patterson, C. (2013). Do symptoms of sluggish cognitive tempo in children with ADHD symptoms represent comorbid internalizing difficulties? Journal of Attention Disorders, 17, 510 –518. http://dx.doi.org/10.1177/1087054711431456 Garner, W. R., Hake, H. W., & Eriksen, C. W. (1956). Operationism and the concept of perception. Psychological Review, 63, 149 –159. http:// dx.doi.org/10.1037/h0042992

890

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



DE LOS REYES ET AL.

Germán, M., Gonzales, N. A., & Dumka, L. (2009). Familism values as a protective factor for Mexican-origin adolescents exposed to deviant peers. The Journal of Early Adolescence, 29, 16 – 42. http://dx.doi.org/ 10.1177/0272431608324475 ⴱ Ghassabian, A., Bongers-Schokking, J. J., de Rijke, Y. B., van Mil, N., Jaddoe, V. W., de Muinck Keizer-Schrama, S. M., . . . Tiemeier, H. (2012). Maternal thyroid autoimmunity during pregnancy and the risk of attention deficit/hyperactivity problems in children: The Generation R study. Thyroid: Official Journal of the American Thyroid Association, 22, 178 –186. http://dx.doi.org/10.1089/thy.2011.0318 ⴱ Ghassabian, A., Bongers-Schokking, J. J., Henrichs, J., Jaddoe, V. W., Visser, T. J., Visser, W., . . . Tiemeier, H. (2011). Maternal thyroid function during pregnancy and behavioral problems in the offspring: The generation R study. Pediatric Research, 69, 454 – 459. http://dx.doi.org/ 10.1203/PDR.0b013e3182125b0c ⴱ Gizer, I. R., Waldman, I. D., Abramowitz, A., Barr, C., Feng, Y., Wigg, K. G., . . . Rowe, D. C. (2008). Relations between multi-informant assessments of ADHD symptoms, DAT1, and DRD4. Journal of Abnormal Psychology, 117, 869 – 880. http://dx.doi.org/10.1037/a0013297 ⴱ Gomez, R. (2007). Australian parent and teacher ratings of the DSM-IV ADHD symptoms: Differential symptom functioning, and parentteacher agreement and differences. Journal of Attention Disorders, 11, 17–27. http://dx.doi.org/10.1177/1087054706295665 ⴱ Gomez, R., Burns, G. L., Walsh, J. A., & de Moura, M. A. (2003). A multitrait-multisource confirmatory factor analytic approach to the construct validity of ADHD rating scales. Psychological Assessment, 15, 3–16. http://dx.doi.org/10.1037/1040-3590.15.1.3 ⴱ Goodman, A., Lamping, D. L., & Ploubidis, G. B. (2010). When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the strengths and difficulties questionnaire (SDQ): Data from British parents, teachers and children. Journal of Abnormal Child Psychology, 38, 1179 –1191. http://dx.doi.org/10.1007/ s10802-010-9434-x Goodman, K. L., De Los Reyes, A., & Bradshaw, C. P. (2010). Understanding and using informants’ reporting discrepancies of youth victimization: A conceptual model and recommendations for research. Clinical Child and Family Psychology Review, 13, 366 –383. http://dx.doi.org/ 10.1007/s10567-010-0076-x ⴱ Goodman, R. (2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1337–1345. http://dx.doi.org/10.1097/ 00004583-200111000-00015 Goodman, S. H., & Gotlib, I. H. (1999). Risk for psychopathology in the children of depressed mothers: A developmental model for understanding mechanisms of transmission. Psychological Review, 106, 458 – 490. http://dx.doi.org/10.1037/0033-295X.106.3.458 ⴱ Gordis, E. B., Granger, D. A., Susman, E. J., & Trickett, P. K. (2006). Asymmetry between salivary cortisol and ␣-amylase reactivity to stress: Relation to aggressive behavior in adolescents. Psychoneuroendocrinology, 31, 976 –987. http://dx.doi.org/10.1016/j.psyneuen.2006.05.010 ⴱ Gordis, E. B., Margolin, G., & John, R. S. (2001). Parents’ hostility in dyadic marital and triadic family settings and children’s behavior problems. Journal of Consulting and Clinical Psychology, 69, 727–734. http://dx.doi.org/10.1037/0022-006X.69.4.727 ⴱ Greenbaum, R. L., Stevens, S. A., Nash, K., Koren, G., & Rovet, J. (2009). Social cognitive and emotion processing abilities of children with fetal alcohol spectrum disorders: A comparison with attention deficit hyperactivity disorder. Alcoholism: Clinical and Experimental Research, 33, 1656 –1670. http://dx.doi.org/10.1111/j.1530-0277.2009 .01003.x ⴱ Grietens, H., Onghena, P., Prinzie, P., Gadeyne, E., Van Assche, C., Ghesquière, P., & Hellinckx, W. (2004). Comparison of mothers’, fathers’, and teachers’ reports on problem behavior in 5- to 6-year-old

children. Journal of Psychopathology and Behavioral Assessment, 26, 137–146. http://dx.doi.org/10.1023/B:JOBA.0000013661.14995.59 ⴱ Grigorenko, E. L., Geiser, C., Slobodskaya, H. R., & Francis, D. J. (2010). Cross-informant symptoms from CBCL, TRF, and YSR: Trait and method variance in a normative sample of Russian youths. Psychological Assessment, 22, 893–911. http://dx.doi.org/10.1037/a0020703 Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the Anxiety Disorders Interview Schedule for parents and children. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 30 – 40. http://dx.doi.org/10.1097/00004583-200301000-00008 Groth-Marnat, G. (2009). Handbook of psychological assessment (5th ed.). Hoboken, NJ: Wiley. ⴱ Gustafsson, A., Arnup, K., Broberg, A. G., Bodin, L., & Berggren, U. (2010). Child dental fear as measured with the Dental Subscale of the Children’s Fear Survey Schedule: The impact of referral status and type of informant (child versus parent). Community Dentistry and Oral Epidemiology, 38, 256 –266. http://dx.doi.org/10.1111/j.1600-0528.2009 .00521.x ⴱ Haine, R. A., Ayers, T. S., Sandler, I. N., Wolchik, S. A., & Weyer, J. L. (2003). Locus of control and self-esteem as stress-moderators or stressmediators in parentally bereaved children. Death Studies, 27, 619 – 640. http://dx.doi.org/10.1080/07481180302894 ⴱ Handwerk, M. L., Larzelere, R. E., Soper, S. H., & Friman, P. C. (1999). Parent and child discrepancies in reporting severity of problem behaviors in three out-of-home settings. Psychological Assessment, 11, 14 –23. http://dx.doi.org/10.1037/1040-3590.11.1.14 ⴱ Hanssen-Bauer, K., Langsrud, O., Kvernmo, S., & Heyerdahl, S. (2010). Clinician-rated mental health in outpatient child and adolescent mental health services: Associations with parent, teacher and adolescent ratings. Child and Adolescent Psychiatry and Mental Health, 4:29. http://dx.doi .org/10.1186/1753-2000-4-29 ⴱ Harding, H. G., Morelen, D., Thomassin, K., Bradbury, L., & Shaffer, A. (2013). Exposure to maternal- and paternal-perpetrated intimate partner violence, emotion regulation, and child outcomes. Journal of Family Violence, 28, 63–72. http://dx.doi.org/10.1007/s10896-012-9487-4 ⴱ Hartley, A. G., Zakriski, A. L., & Wright, J. C. (2011). Probing the depths of informant discrepancies: Contextual influences on divergence and convergence. Journal of Clinical Child and Adolescent Psychology, 40, 54 – 66. http://dx.doi.org/10.1080/15374416.2011.533404 ⴱ Hartos, J. L., & Power, T. G. (2000). Association between mother and adolescent reports for assessing relations between parent–adolescent communication and adolescent adjustment. Journal of Youth and Adolescence, 29, 441– 450. http://dx.doi.org/10.1023/A:1005158425861 ⴱ Harvey, E. A., Fischer, C., Weieneth, J. L., Hurwitz, S. D., & Sayer, A. G. (2013). Predictors of discrepancies between informants’ ratings of preschool-aged children’s behavior: An examination of ethnicity, child characteristics, and family functioning. Early Childhood Research Quarterly, 28, 668 – 682. http://dx.doi.org/10.1016/j.ecresq.2013.05.002 Hawley, K. M., & Weisz, J. R. (2003). Child, parent, and therapist (dis)agreement on target problems in outpatient therapy: The therapist’s dilemma and its implications. Journal of Consulting and Clinical Psychology, 71, 62–70. http://dx.doi.org/10.1037/0022-006X.71.1.62 Haynes, S. N., & Lench, H. C. (2003). Incremental validity of new clinical assessment measures. Psychological Assessment, 15, 456 – 466. http:// dx.doi.org/10.1037/1040-3590.15.4.456 ⴱ Hemmelgarn, A. L., Glisson, C., & Sharp, S. R. (2003). The validity of the Shortform Assessment for Children (SAC). Research on Social Work Practice, 13, 510 –530. http://dx.doi.org/10.1177/ 1049731503013004006 ⴱ Hendriks, A. H. C., de Moor, J. M. H., Oud, J. H. L., Franken, W. M., & Savelberg, M. M. H. W. (2001). Behaviour problems of young motor disabled children at home and in the therapeutic toddler class. European Journal of Special Needs Education, 16, 15–28. http://dx.doi.org/ 10.1080/08856250150501770

VALIDITY OF MULTI-INFORMANT ASSESSMENT

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



Henggeler, S. W., Watson, S. M., & Whelan, J. P. (1990). Peer relations of hearing-impaired adolescents. Journal of Pediatric Psychology, 15, 721–731. http://dx.doi.org/10.1093/jpepsy/15.6.721 ⴱ Henrich, C. C., Kuperminc, G. P., Sack, A., Blatt, S. J., & Leadbeater, B. J. (2000). Characteristics and homogeneity of early adolescent friendship groups: A comparison of male and female clique and nonclique members. Applied Developmental Science, 4, 15–26. http://dx.doi.org/ 10.1207/S1532480XADS0401_2 ⴱ Henry, B., Moffitt, T. E., Caspi, A., Langley, J., & Silva, P. A. (1994). On the “remembrance of things past”: A longitudinal evaluation of the retrospective method. Psychological Assessment, 6, 92–101. http://dx .doi.org/10.1037/1040-3590.6.2.92 ⴱ Henry, D. B., & Metropolitan Area Child Study Research Group US. (2006). Associations between peer nominations, teacher ratings, selfreports, and observations of malicious and disruptive behavior. Assessment, 13, 241–252. http://dx.doi.org/10.1177/1073191106287668 ⴱ Higa, C. K., Daleiden, E. L., & Chorpita, B. F. (2002). Psychometric properties and clinical utility of the school refusal assessment scale in a multiethnic sample. Journal of Psychopathology and Behavioral Assessment, 24, 247–258. http://dx.doi.org/10.1023/A:1020727016113 ⴱ Higa, C. K., Fernandez, S. N., Nakamura, B. J., Chorpita, B. F., & Daleiden, E. L. (2006). Parental assessment of childhood social phobia: Psychometric properties of the Social Phobia and Anxiety Inventory for Children–Parent Report. Journal of Clinical Child & Adolescent Psychology, 35, 590 –597. http://dx.doi.org/10.1207/s15374424jccp3504_11 Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557–560. http://dx.doi.org/10.1136/bmj.327.7414.557 ⴱ Hogue, A., Dauber, S., Lichvar, E., & Spiewag, G. (2014). Adolescent and caregiver reports of ADHD symptoms among inner-city youth agreement, perceived need for treatment, and behavioral correlates. Journal of Attention Disorders, 18, 212–225. http://dx.doi.org/10.1177/ 1087054712443160 ⴱ Hollocks, M. J., Papadopoulos, A. S., Khondoker, M., & Simonoff, E. (2014). Differences in HPA-axis and heart rate responsiveness to psychosocial stress in children with autism spectrum disorders with and without co-morbid anxiety. Psychoneuroendocrinology, 46, 32– 45. http://dx.doi.org/10.1016/j.psyneuen.2014.04.004 ⴱ Hommersen, P., Murray, C., Ohan, J. L., & Johnston, C. (2006). Oppositional defiant disorder rating scale preliminary evidence of reliability and validity. Journal of Emotional and Behavioral Disorders, 14, 118 – 125. http://dx.doi.org/10.1177/10634266060140020201 ⴱ Honomichi, R. D., & Donnellan, M. B. (2012). Dimensions of temperament in preschoolers predict risk taking and externalizing behaviors in adolescents. Social Psychological and Personality Science, 3, 14 –22. http://dx.doi.org/10.1177/1948550611407344 ⴱ Huberty, T. J., Austin, J. K., Harezlak, J., Dunn, D. W., & Ambrosius, W. T. (2000). Informant agreement in behavior ratings for children with epilepsy. Epilepsy & Behavior, 1, 427– 435. http://dx.doi.org/10.1006/ ebeh.2000.0119 ⴱ Hughes, E. K., & Gullone, E. (2010). Discrepancies between adolescent, mother, and father reports of adolescent internalizing symptom levels and their association with parent symptoms. Journal of Clinical Psychology, 66, 978 –995. http://dx.doi.org/10.1002/jclp.20695 Hunsley, J. (2003). Introduction to the special section on incremental validity and utility in clinical assessment. Psychological Assessment, 15, 443– 445. http://dx.doi.org/10.1037/1040-3590.15.4.443 Hunsley, J., & Lee, C. M. (2010). Introduction to clinical psychology. Hoboken, NJ: Wiley. Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical Psychology, 3, 29 –51. http://dx.doi.org/10.1146/ annurev.clinpsy.3.022806.091419 Hunsley, J., & Mash, E. J. (Eds.). (2005). Developing guidelines for the evidence-based assessment (EBA) of adult disorders [Special section].

891

Psychological Assessment, 17. http://dx.doi.org/10.1037/1040-3590.17 .3.251 Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issues. Psychological Assessment, 15, 446 – 455. http://dx.doi.org/ 10.1037/1040-3590.15.4.446 ⴱ Hwang, J., & Lee, B. R. (2013). Response agreement rates between child welfare–involved youth and their caregivers. Journal of Social Service Research, 39, 218 –232. http://dx.doi.org/10.1080/01488376.2012 .746768 ⴱ Jaffee, S. R., Caspi, A., Moffitt, T. E., Polo-Tomas, M., Price, T. S., & Taylor, A. (2004). The limits of child effects: Evidence for genetically mediated child effects on corporal punishment but not on physical maltreatment. Developmental Psychology, 40, 1047–1058. http://dx.doi .org/10.1037/0012-1649.40.6.1047 ⴱ Jaffee, S. R., Moffitt, T. E., Caspi, A., Taylor, A., & Arseneault, L. (2002). Influence of adult domestic violence on children’s internalizing and externalizing problems: An environmentally informative twin study. Journal of the American Academy of Child & Adolescent Psychiatry, 41, 1095–1103. http://dx.doi.org/10.1097/00004583-200209000-00010 ⴱ Janssens, J. M. A. M., De Bruyn, E. E. J., Manders, W. A., & Scholte, R. H. J. (2005). The multitrait-multimethod approach in family assessment: Mutual parent-child relationships assessed by questionnaires and observations. European Journal of Psychological Assessment, 21, 232– 239. http://dx.doi.org/10.1027/1015-5759.21.4.232 ⴱ Jaser, S. S., Langrock, A. M., Keller, G., Merchant, M. J., Benson, M. A., Reeslund, K., . . . Compas, B. E. (2005). Coping with the stress of parental depression II: Adolescent and parent reports of coping and adjustment. Journal of Clinical Child and Adolescent Psychology, 34, 193–205. http://dx.doi.org/10.1207/s15374424jccp3401_18 ⴱ Jaycox, L. H., & Repetti, R. L. (1993). Conflict in families and the psychological adjustment of preadolescent children. Journal of Family Psychology, 7, 344 –355. http://dx.doi.org/10.1037/0893-3200.7.3.344 Jensen-Doss, A., & Weisz, J. R. (2008). Diagnostic agreement predicts treatment process and outcomes in youth mental health clinics. Journal of Consulting and Clinical Psychology, 76, 711–722. http://dx.doi.org/ 10.1037/0022-006X.76.5.711 ⴱ Joffe, B. I., Van Lieshout, R. J., Duncan, L., & Boyle, M. H. (2014). Suicidal ideation and behavior in adolescents aged 12–16 years: A 17-year follow-up. Suicide and Life-Threatening Behavior. Advance online publication. http://dx.doi.org/10.1111/sltb.12077 Johnston, C., & Murray, C. (2003). Incremental validity in the psychological assessment of children and adolescents. Psychological Assessment, 15, 496 –507. http://dx.doi.org/10.1037/1040-3590.15.4.496 ⴱ Jouriles, E. M., Norwood, W. D., McDonald, R., Vincent, J. P., & Mahoney, A. (1996). Physical violence and other forms of marital aggression: Links with children’s behavior problems. Journal of Family Psychology, 10, 223–234. http://dx.doi.org/10.1037/0893-3200.10.2.223 ⴱ Jouriles, E. N., Rosenfield, D., McDonald, R., & Mueller, V. (2014). Child involvement in interparental conflict and child adjustment problems: A longitudinal study of violent families. Journal of Abnormal Child Psychology, 42, 693–704. http://dx.doi.org/10.1007/s10802-0139821-1 Jouriles, E. N., & Thompson, S. M. (1993). Effects of mood on mothers’ evaluations of children’s behavior. Journal of Family Psychology, 6, 300 –307. http://dx.doi.org/10.1037/0893-3200.6.3.300 ⴱ Jung, E., Raikes, H. H., & Chazan-Cohen, R. (2013). Maternal depressive symptoms and behavior problems in preschool children from lowincome families: Comparison of reports from mothers and teachers. Journal of Child and Family Studies, 22, 757–768. http://dx.doi.org/ 10.1007/s10826-012-9630-0 ⴱ Karustis, J. L., Power, T. J., Rescorla, L. A., Eiraldi, R. B., & Gallagher, P. R. (2000). Anxiety and depression in children with ADHD: Unique

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

892

DE LOS REYES ET AL.

associations with academic and social functioning. Journal of Attention Disorders, 4, 133–149. http://dx.doi.org/10.1177/108705470000400301 ⴱ Karver, M. S. (2006). Determinants of multiple informant agreement on child and adolescent behavior. Journal of Abnormal Child Psychology, 34, 251–262. http://dx.doi.org/10.1007/s10802-005-9015-6 ⴱ Kassam-Adams, N., García-España, J. F., Miller, V. A., & Winston, F. (2006). Parent-child agreement regarding children’s acute stress: The role of parent acute stress reactions. Journal of the American Academy of Child & Adolescent Psychiatry, 45, 1485–1493. http://dx.doi.org/ 10.1097/01.chi.0000237703.97518.12 ⴱ Kazak, A. E., Christakis, D., Alderfer, M., & Coiro, M. J. (1994). Young adolescent cancer survivors and their parents: Adjustment, learning problems, and gender. Journal of Family Psychology, 8, 74 – 84. http:// dx.doi.org/10.1037/0893-3200.8.1.74 Kazdin, A. E., & Kagan, J. (1994). Models of dysfunction in developmental psychopathology. Clinical Psychology: Science and Practice, 1, 35–52. http://dx.doi.org/10.1111/j.1468-2850.1994.tb00005.x ⴱ Kearney, C. A., & Silverman, W. K. (1993). Measuring the function of school refusal behavior: The School Refusal Assessment scale. Journal of Clinical Child Psychology, 22, 85–96. http://dx.doi.org/10.1207/ s15374424jccp2201_9 ⴱ Kemper, T. S., Gerhardstein, R., Repper, K. K., & Kistner, J. A. (2003). Mother– child agreement on reports of internalizing symptoms among children referred for evaluation of ADHD. Journal of Psychopathology and Behavioral Assessment, 25, 239 –250. http://dx.doi.org/10.1023/A: 1025847012201 ⴱ Kendall, P. C., Puliafico, A. C., Barmish, A. J., Choudhury, M. S., Henin, A., & Treadwell, K. S. (2007). Assessing anxiety with the Child Behavior Checklist and the Teacher Report Form. Journal of Anxiety Disorders, 21, 1004 –1015. http://dx.doi.org/10.1016/j.janxdis.2006.10.012 ⴱ Keown, L. J. (2012). Predictors of boys’ ADHD symptoms from early to middle childhood: The role of father-child and mother-child interactions. Journal of Abnormal Child Psychology, 40, 569 –581. http://dx.doi.org/ 10.1007/s10802-011-9586-3 ⴱ King, K., Reschly, A. L., & Appleton, J. J. (2012). An examination of the validity of the behavioral and emotional screening system in a rural elementary school validity of the BESS. Journal of Psychoeducational Assessment, 30, 527–538. http://dx.doi.org/10.1177/0734282912440673 ⴱ Klaus, N. M., Mobilio, A., & King, C. A. (2009). Parent–adolescent agreement concerning adolescents’ suicidal thoughts and behaviors. Journal of Clinical Child & Adolescent Psychology, 38, 245–255. http:// dx.doi.org/10.1080/15374410802698412 Klein, D. N., Dougherty, L. R., & Olino, T. M. (2005). Toward guidelines for evidence-based assessment of depression in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 412– 432. http://dx.doi.org/10.1207/s15374424jccp3403_3 ⴱ Knox, M., King, C., Hanna, G. L., Logan, D., & Ghaziuddin, N. (2000). Aggressive behavior in clinically depressed adolescents. Journal of the American Academy of Child & Adolescent Psychiatry, 39, 611– 618. http://dx.doi.org/10.1097/00004583-200005000-00015 ⴱ Kraatz Keiley, M., Bates, J. E., Dodge, K. A., & Pettit, G. S. (2000). A cross-domain growth analysis: Externalizing and internalizing behaviors during 8 years of childhood. Journal of Abnormal Child Psychology, 28, 161–179. http://dx.doi.org/10.1023/A:1005122814723 Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. The American Journal of Psychiatry, 160, 1566 –1577. http://dx.doi.org/10.1176/appi.ajp.160.9.1566 ⴱ Krain, A. L., & Kendall, P. C. (2000). The role of parental emotional distress in parent report of child anxiety. Journal of Clinical Child Psychology, 29, 328 –335. http://dx.doi.org/10.1207/ S15374424JCCP2903_4

Kramer, T. L., Phillips, S. D., Hargis, M. B., Miller, T. L., Burns, B. J., & Robbins, J. M. (2004). Disagreement between parent and adolescent reports of functional impairment. Journal of Child Psychology and Psychiatry, 45, 248 –259. http://dx.doi.org/10.1111/j.1469-7610.2004 .00217.x ⴱ Kugler, B. B., Burke, N. L., Bloom, M., Truax, T. V., Kaercher, L. B., & Storch, E. A. (2013). Caregiver–teacher agreement on emotional and behavioral problems in traumatized youth in residential treatment. Residential Treatment for Children & Youth, 30, 151–167. http://dx.doi.org/ 10.1080/0886571X.2013.813343 ⴱ Kuijpers, R. C. W. M., Otten, R., Krol, N. P. C. M., Vermulst, A. A., & Engels, R. C. M. E. (2013). The reliability and validity of the Dominic interactive: A computerized child report instrument for mental health problems. Child & Youth Care Forum, 42, 35–52. http://dx.doi.org/ 10.1007/s10566-012-9185-7 ⴱ Kumpulainen, K., Rasanen, E., Heuttonen, L., Moilanen, I., Piha, J., Puura, K., . . . Almqvist, F. (1999). Children’s behavioral/emotional problems: A comparison of parents’ and teachers’ reports for elementary school-aged children. European Child and Adolescent Psychiatry, 8(Supplement), S41–S47. http://dx.doi.org/10.1007/BF03159641 Kwon, K., Kim, E. M., & Sheridan, S. M. (2012). A contextual approach to social skills assessment in the peer group: Who is the best judge? School Psychology Quarterly, 27, 121–133. http://dx.doi.org/10.1037/ a0028696 ⴱ Ladd, G. W., & Ettekal, I. (2013). Peer-related loneliness across early to late adolescence: Normative trends, intra-individual trajectories, and links with depressive symptoms. Journal of Adolescence, 36, 1269 – 1282. http://dx.doi.org/10.1016/j.adolescence.2013.05.004 Ladd, G. W., & Kochenderfer-Ladd, B. (2002). Identifying victims of peer aggression from early to middle childhood: Analysis of cross-informant data for concordance, estimation of relational adjustment, prevalence of victimization, and characteristics of identified victims. Psychological Assessment, 14, 74 –96. http://dx.doi.org/10.1037/1040-3590.14.1.74 ⴱ Lagattuta, K. H., Sayfan, L., & Bamford, C. (2012). Do you know how I feel? Parents underestimate worry and overestimate optimism compared to child self-report. Journal of Experimental Child Psychology, 113, 211–232. http://dx.doi.org/10.1016/j.jecp.2012.04.001 ⴱ Lahey, B. B., Applegate, B., Waldman, I. D., Loft, J. D., Hankin, B. L., & Rick, J. (2004). The structure of child and adolescent psychopathology: Generating new hypotheses. Journal of Abnormal Psychology, 113, 358 –385. http://dx.doi.org/10.1037/0021-843X.113.3.358 ⴱ Laird, R. D., & Weems, C. F. (2011). The equivalence of regression models using difference scores and models using separate scores for each informant: Implications for the study of informant discrepancies. Psychological Assessment, 23, 388 –397. http://dx.doi.org/10.1037/ a0021926 ⴱ Langberg, J. M., Epstein, J. N., Simon, J. O., Loren, R. E. A., Arnold, L. E., Hechtman, L., . . . Wigal, T. (2010). Parent agreement on ratings of children’s attention deficit/hyperactivity disorder and broadband externalizing behaviors. Journal of Emotional and Behavioral Disorders, 18, 41–50. http://dx.doi.org/10.1177/1063426608330792 ⴱ Langeveld, J. H., Gundersen, K. K., & Svartdal, F. (2012). Social competence as a mediating factor in reduction of behavioral problems. Scandinavian Journal of Educational Research, 56, 381–399. http://dx .doi.org/10.1080/00313831.2011.594614 Lapouse, R., & Monk, M. A. (1958). An epidemiologic study of behavior characteristics in children. American Journal of Public Health, 48, 1134 –1144. http://dx.doi.org/10.2105/AJPH.48.9.1134 ⴱ Lauth, B., Arnkelsson, G. B., Magnusson, P., Skarpheoinsson, G. A., Ferrari, P., & Petursson, H. (2010). Parent–youth agreement on symptoms and diagnosis: Assessment with a diagnostic interview in an adolescent inpatient clinical population. Journal of Physiology-Paris, 104, 315–322. http://dx.doi.org/10.1016/j.jphysparis.2010.09.004

VALIDITY OF MULTI-INFORMANT ASSESSMENT

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



Lavigne, J. V., Dahl, K. P., Gouze, K. R., LeBailly, S. A., & Hopkins, J. (2014). Multi-domain predictors of oppositional defiant disorder symptoms in preschool children: Cross-informant differences. Child Psychiatry & Human Development. Advance online publication. http://dx.doi .org/10.1007/s10578-014-0472-4 ⴱ Lee, H., & Schepp, K. G. (2011). Multiple informants in assessing stress and symptoms in adolescents with schizophrenia. Archives of Psychiatric Nursing, 25, 120 –128. http://dx.doi.org/10.1016/j.apnu.2010.06.004 ⴱ Lee, S. W., Elliott, J., & Barbour, J. D. (1994). A comparison of crossinformant behavior ratings in school-based diagnosis. Behavioral Disorders, 19, 87–97. Retrieved from http://www.jstor.org/stable/23887490 ⴱ Leeuwen, K. V., Meerschaert, T., Bosmans, G., De Medts, L., & Braet, C. (2006). The Strengths and Difficulties Questionnaire in a community sample of young children in Flanders. European Journal of Psychological Assessment, 22, 189 –197. http://dx.doi.org/10.1027/1015-5759.22 .3.189 ⴱ Lehti, V., Niemelä, S., Heinze, M., Sillanmäki, L., Helenius, H., Piha, J., . . . Sourander, A. (2012). Childhood predictors of becoming a teenage mother among Finnish girls. Acta Obstetricia et Gynecologica Scandinavica, 91, 1319 –1325. http://dx.doi.org/10.1111/j.1600-0412.2012 .01517.x ⴱ Lengua, L. J. (2003). Associations among emotionality, self-regulation, adjustment problems, and positive adjustment in middle childhood. Journal of Applied Developmental Psychology, 24, 595– 618. http://dx .doi.org/10.1016/j.appdev.2003.08.002 Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. American Psychologist, 48, 1181–1209. http://dx.doi.org/10.1037/0003066X.48.12.1181 ⴱ Little, S. A., & Garber, J. (2005). The role of social stressors and interpersonal orientation in explaining the longitudinal relation between externalizing and depressive symptoms. Journal of Abnormal Psychology, 114, 432– 443. http://dx.doi.org/10.1037/0021-843X.114.3.432 ⴱ Liu, J., Cheng, H., & Leung, P. W. L. (2011). The application of the Preschool Child Behavior Checklist and the Caregiver–Teacher Report Form to mainland Chinese children: Syndrome structure, gender differences, country effects, and inter-informant agreement. Journal of Abnormal Child Psychology, 39, 251–264. http://dx.doi.org/10.1007/ s10802-010-9452-8 ⴱ Liu, X., Sun, Z., Neiderhiser, J. M., Uchiyama, M., Okawa, M., & Rogan, W. (2001). Behavioral and emotional problems in Chinese adolescents: Parent and teacher reports. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 828 – 836. http://dx.doi.org/10.1097/ 00004583-200107000-00018 Loeber, R., Green, S. M., & Lahey, B. B. (1990). Mental health professionals’ perception of the utility of children, mothers, and teachers as informants of childhood psychopathology. Journal of Clinical Child Psychology, 19, 136 –143. http://dx.doi.org/10.1207/ s15374424jccp1902_5 Loeber, R., Green, S. M., Lahey, B. B., & Stouthamer-Loeber, M. (1989). Optimal informants on childhood disruptive behaviors. Development and Psychopathology, 1, 317–337. http://dx.doi.org/10.1017/ S095457940000050X ⴱ Loomans, E. M., van der Stelt, O., van Eijsden, M., Gemke, R. J., Vrijkotte, T., & den Bergh, B. R. (2011). Antenatal maternal anxiety is associated with problem behaviour at age five. Early Human Development, 87, 565–570. http://dx.doi.org/10.1016/j.earlhumdev.2011.04.014 ⴱ Lopata, C., Toomey, J. A., Fox, J. D., Volker, M. A., Chow, S. Y., Thomeer, M. L., . . . Smerbeck, A. M. (2010). Anxiety and depression in children with HFASDs: Symptom levels and source differences. Journal of Abnormal Child Psychology, 38, 765–776. http://dx.doi.org/ 10.1007/s10802-010-9406-1 ⴱ Low, S. M., & Stocker, C. (2005). Family functioning and children’s adjustment: Associations among parents’ depressed mood, marital hos-

893

tility, parent-child hostility, and children’s adjustment. Journal of Family Psychology, 19, 394 – 403. http://dx.doi.org/10.1037/0893-3200.19.3 .394 ⴱ Luby, J. L., Belden, A., Sullivan, J., & Spitznagel, E. (2007). Preschoolers’ contribution to their diagnosis of depression and anxiety: Uses and limitations of young child self-report of symptoms. Child Psychiatry and Human Development, 38, 321–338. http://dx.doi.org/10.1007/s10578007-0063-8 Luthar, S. S., Cicchetti, D., & Becker, B. (2000). The construct of resilience: A critical evaluation and guidelines for future work. Child Development, 71, 543–562. http://dx.doi.org/10.1111/1467-8624.00164 ⴱ Lyneham, H. J., Street, A. K., Abbott, M. J., & Rapee, R. M. (2008). Psychometric properties of the School Anxiety Scale—Teacher report (SAS-TR). Journal of Anxiety Disorders, 22, 292–300. http://dx.doi.org/ 10.1016/j.janxdis.2007.02.001 ⴱ Mahone, E. M., Zabel, T. A., Levey, E., Verda, M., & Kinsman, S. (2002). Parent and self-report ratings of executive function in adolescents with myelomeningocele and hydrocephalus. Child Neuropsychology, 8, 258 – 270. http://dx.doi.org/10.1076/chin.8.4.258.13510 ⴱ Maikovich-Fong, A. K., & Jaffee, S. R. (2010). Sex differences in childhood sexual abuse characteristics and victims’ emotional and behavioral problems: Findings from a national sample of youth. Child Abuse & Neglect, 34, 429 – 437. http://dx.doi.org/10.1016/j.chiabu.2009.10.006 ⴱ Mann, B. J., & MacKenzie, E. P. (1996). Pathways among marital functioning, parental behaviors, and child behavior problems in schoolage boys. Journal of Clinical Child Psychology, 25, 183–191. http://dx .doi.org/10.1207/s15374424jccp2502_7 Markon, K. E., Chmielewski, M., & Miller, C. J. (2011). The reliability and validity of discrete and continuous measures of psychopathology: A quantitative review. Psychological Bulletin, 137, 856 – 879. http://dx.doi .org/10.1037/a0023678 Marsh, J. K., De Los Reyes, A., & Wallerstein, A. (2014). The influence of contextual information on lay judgments of childhood mental health concerns. Psychological Assessment, 26, 1268 –1280. http://dx.doi.org/ 10.1037/pas0000012 ⴱ Martin, J. L., Ford, C. B., Dyer-Friedman, J., Tang, J., & Huffman, L. C. (2004). Patterns of agreement between parent and child ratings of emotional and behavioral problems in an outpatient clinical setting: When children endorse more problems. Journal of Developmental and Behavioral Pediatrics, 25, 150 –155. http://dx.doi.org/10.1097/ 00004703-200406000-00002 ⴱ Martin-Storey, A., Serbin, L. A., Stack, D. M., & Schwartzman, A. E. (2009). The behaviour style observation system for young children predicts teacher-reported externalizing behaviour in middle childhood. Infant and Child Development, 18, 337–350. http://dx.doi.org/10.1002/ icd.601 ⴱ Mascendaro, P. M., Herman, K. C., & Webster-Stratton, C. (2012). Parent discrepancies in ratings of young children’s co-occurring internalizing symptoms. School Psychology Quarterly, 27, 134 –143. http://dx.doi .org/10.1037/a0029320 Mash, E. J., & Hunsley, J. (Eds.). (2005). Evidence-based assessment of child and adolescent disorders: Issues and challenges [Special section]. Journal of Clinical Child and Adolescent Psychology. http://dx.doi.org/ 10.1207/s15374424jccp3403_1 ⴱ Mattison, R. E., Carlson, G. A., Cantwell, D. P., & Asarnow, J. R. (2007). Teacher and parent ratings of children with depressive disorders. Journal of Emotional and Behavioral Disorders, 15, 184 –192. http://dx.doi.org/ 10.1177/10634266070150030501 ⴱ McConaughy, S. H., Stanger, C., & Achenbach, T. M. (1992). Three-year course of behavioral/emotional problems in a national sample of 4- to 16-year-olds: I. Agreement among informants. Journal of the American Academy of Child & Adolescent Psychiatry, 31, 932–940. http://dx.doi .org/10.1097/00004583-199209000-00023

894

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



DE LOS REYES ET AL.

McLaughlin, K. A., Alves, S., & Sheridan, M. A. (2014). Vagal regulation and internalizing psychopathology among adolescents exposed to childhood adversity. Developmental Psychobiology, 56, 1036 –1051. http:// dx.doi.org/10.1002/dev.21187 McMahon, R. J., & Frick, P. J. (2005). Evidence-based assessment of conduct problems in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 477–505. http://dx.doi.org/10.1207/ s15374424jccp3403_6 ⴱ Meesters, C., Muris, P., Ghys, A., Reumerman, T., & Rooijmans, M. (2003). The Children’s Somatization Inventory: Further evidence for its reliability and validity in a pediatric and a community sample of Dutch children and adolescents. Journal of Pediatric Psychology, 28, 413– 422. http://dx.doi.org/10.1093/jpepsy/jsg031 ⴱ Mellor, D. (2004). Furthering the use of the Strengths and Difficulties Questionnaire: Reliability with younger child respondents. Psychological Assessment, 16, 396 – 401. http://dx.doi.org/10.1037/1040-3590.16.4 .396 ⴱ Mellor, D., Wong, J., & Xu, X. (2011). Interparent agreement on the strengths and difficulties questionnaire: A Chinese study. Journal of Clinical Child & Adolescent Psychology, 40, 890 – 896. http://dx.doi .org/10.1080/15374416.2011.614580 Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., . . . Reed, G. M. (2001). Psychological testing and psychological assessment. A review of evidence and issues. American Psychologist, 56, 128 –165. http://dx.doi.org/10.1037/0003-066X.56.2.128 ⴱ Mikami, A. Y., Lee, S. S., Hinshaw, S. P., & Mullin, B. C. (2008). Relationships between social information processing and aggression among adolescent girls with and without ADHD. Journal of Youth and Adolescence, 37, 761–771. http://dx.doi.org/10.1007/s10964-0079237-8 ⴱ Mikami, A. Y., & Pfiffner, L. J. (2008). Sibling relationships among children with ADHD. Journal of Attention Disorders, 11, 482– 492. http://dx.doi.org/10.1177/1087054706295670 Miller, J. D., Pilkonis, P. A., & Clifton, A. (2005). Self- and other-reports of traits from the five-factor model: Relations to personality disorder. Journal of Personality Disorders, 19, 400 – 419. http://dx.doi.org/10.1521/ pedi.2005.19.4.400 Mischel, W., & Shoda, Y. (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102, 246 – 268. http://dx.doi.org/10.1037/0033-295X.102.2.246 ⴱ Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn, J. H., & Halperin, J. M. (2000). Parent–teacher concordance for DSM-IV attention-deficit/ hyperactivity disorder in a clinic-referred sample. Journal of the American Academy of Child & Adolescent Psychiatry, 39, 308 –313. http:// dx.doi.org/10.1097/00004583-200003000-00012 Møller, A. P., & Jennions, M. D. (2002). How much variance can be explained by ecologists and evolutionary biologists? Oecologia, 132, 492–500. http://dx.doi.org/10.1007/s00442-002-0952-2 ⴱ Montgomery, E. (2008). Self- and parent assessment of mental health: Disagreement on externalizing and internalizing behaviour in young refugees from the Middle East. Clinical Child Psychology and Psychiatry, 13, 49 – 63. http://dx.doi.org/10.1177/1359104507086341 ⴱ Morrow, M. T., Hubbard, J. A., Rubin, R. M., & McAuliffe, M. D. (2008). The relation between childhood aggression and depressive symptoms: The unique and joint mediating roles of peer rejection and peer victimization. Merrill-Palmer Quarterly, 54, 316 –340. http://dx.doi.org/ 10.1353/mpq.0.0000 ⴱ Moss, E., Smolla, N., Cyr, C., Dubois-Comtois, K., Mazzarello, T., & Berthiaume, C. (2006). Attachment and behavior problems in middle childhood as reported by adult and child informants. Development and Psychopathology, 18, 425– 444. http://dx.doi.org/10.1017/ S0954579406060238



Mulvaney, M. K., Mebert, C. J., & Flint, J. (2007). Parental affect and childrearing beliefs uniquely predict mothers’ and fathers’ ratings of children’s behavior problems. Journal of Applied Developmental Psychology, 28, 445– 457. http://dx.doi.org/10.1016/j.appdev.2007.06.001 ⴱ Munkvold, L., Lundervold, A., Lie, S. A., & Manger, T. (2009). Should there be separate parent and teacher-based categories of ODD? Evidence from a general population. Journal of Child Psychology and Psychiatry, 50, 1264 –1272. http://dx.doi.org/10.1111/j.1469-7610.2009.02091.x ⴱ Muris, P., & Maas, A. (2004). Strengths and difficulties as correlates of attachment style in institutionalized and non-institutionalized children with below-average intellectual abilities. Child Psychiatry and Human Development, 34, 317–328. http://dx.doi.org/10.1023/B:CHUD .0000020682.55697.4f ⴱ Muris, P., & Meesters, C. (2003). The validity of attention deficit hyperactivity and hyperkinetic disorder symptom domains in nonclinical Dutch children. Journal of Clinical Child & Adolescent Psychology, 32, 460 – 466. http://dx.doi.org/10.1207/S15374424JCCP3203_14 ⴱ Muris, P., Meesters, C., & Rompelberg, L. (2007). Attention control in middle childhood: Relations to psychopathological symptoms and threat perception distortions. Behaviour Research and Therapy, 45, 997–1010. http://dx.doi.org/10.1016/j.brat.2006.07.010 ⴱ Muris, P., Meesters, C., & Timmermans, A. (2013). Some youths have a gloomy side: Correlates of the dark triad personality traits in non-clinical adolescents. Child Psychiatry & Human Development, 44, 658 – 665. http://dx.doi.org/10.1007/s10578-013-0359-9 ⴱ Murray, D. W., Kollins, S. H., Hardy, K. K., Abikoff, H. B., Swanson, J. M., Cunningham, C., . . . Chuang, S. Z. (2007). Parent versus teacher ratings of attention-deficit/hyperactivity disorder symptoms in the Preschoolers with Attention-Deficit/Hyperactivity Disorder Treatment Study (PATS). Journal of Child and Adolescent Psychopharmacology, 17, 605– 619. http://dx.doi.org/10.1089/cap.2007.0060 ⴱ Najman, J. M., Williams, G. M., Nikles, J., Spence, S., Bor, W., O’Callaghan, M., . . . Andersen, M. J. (2000). Mothers’ mental illness and child behavior problems: Cause– effect association or observation bias? Journal of the American Academy of Child & Adolescent Psychiatry, 39, 592– 602. http://dx.doi.org/10.1097/00004583-20000500000013 National Institute of Mental Health. (2008). The National Institute of Mental Health strategic plan (NIH Publication No. 08 – 6368). Retrieved from http://www.nimh.nih.gov/about/strategic-planning-reports/index .shtml ⴱ Natsuaki, M. N., Klimes-Dougan, B., Ge, X., Shirtcliff, E. A., Hastings, P. D., & Zahn-Waxler, C. (2009). Early pubertal maturation and internalizing problems in adolescence: Sex differences in the role of cortisol reactivity to interpersonal stress. Journal of Clinical Child and Adolescent Psychology, 38, 513–524. http://dx.doi.org/10.1080/ 15374410902976320 ⴱ Nauta, M. H., Scholing, A., Rapee, R. M., Abbott, M., Spence, S. H., & Waters, A. (2004). A parent-report measure of children’s anxiety: Psychometric properties and comparison with child-report in a clinic and normal sample. Behaviour Research and Therapy, 42, 813– 839. http:// dx.doi.org/10.1016/S0005-7967(03)00200-6 Neal, J. W., Cappella, E., Wagner, C., & Atkins, M. S. (2011). Seeing eye to eye: Predicting teacher-student agreement on classroom social networks. Social Development, 20, 376 –393. http://dx.doi.org/10.1111/j .1467-9507.2010.00582.x ⴱ New, M., Razzino, B., Lewin, A., Schlumpf, K., & Joseph, J. (2002). Mental health service use in a community head start population. Archives of Pediatrics and Adolescent Medicine, 156, 721–727. http://dx .doi.org/10.1001/archpedi.156.7.721 ⴱ Nickerson, A. B., & Fishman, C. (2009). Convergent and divergent validity of the Devereux Student Strengths Assessment. School Psychology Quarterly, 24, 48 –59. http://dx.doi.org/10.1037/a0015147

VALIDITY OF MULTI-INFORMANT ASSESSMENT

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



Niditch, L. A., & Varela, R. E. (2011). Mother– child disagreement in reports of child anxiety: Effects of child age and maternal anxiety. Journal of Anxiety Disorders, 25, 450 – 455. http://dx.doi.org/10.1016/j .janxdis.2010.11.009 Nimon, K. (2010). Regression commonality analysis: Demonstration of an SPSS solution. Multiple Linear Regression Viewpoints, 36, 10 –17. Retrieved from http://mlrv.ua.edu/2010/vol36_1/nimon_proof_2.pdf ⴱ O’Brien, J., Margolin, G., & John, R. S. (1995). Relation among marital conflict, child coping, and child adjustment. Journal of Clinical Child Psychology, 24, 346 –361. http://dx.doi.org/10.1207/ s15374424jccp2403_12 Odgers, C. L., Caspi, A., Bates, C. J., Sampson, R. J., & Moffitt, T. E. (2012). Systematic social observation of children’s neighborhoods using Google Street View: A reliable and cost-effective method. Journal of Child Psychology and Psychiatry, 53, 1009 –1017. http://dx.doi.org/ 10.1111/j.1469-7610.2012.02565.x ⴱ Offord, D. R., Boyle, M. H., Racine, Y., Szatmari, P., Fleming, J. E., Sanford, M., & Lipman, E. L. (1996). Integrating assessment data from multiple informants. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 1078 –1085. http://dx.doi.org/10.1097/ 00004583-199608000-00019 ⴱ Oldehinkel, A. J., Hartman, C. A., De Winter, A. F., Veenstra, R., & Ormel, J. (2004). Temperament profiles associated with internalizing and externalizing problems in preadolescence. Development and Psychopathology, 16, 421– 440. http://dx.doi.org/10.1017/ S0954579404044591 ⴱ Oldehinkel, A. J., Veenstra, R., Ormel, J., de Winter, A. F., & Verhulst, F. C. (2006). Temperament, parenting, and depressive symptoms in a population sample of preadolescents. Journal of Child Psychology and Psychiatry, 47, 684 – 695. http://dx.doi.org/10.1111/j.1469-7610.2005 .01535.x ⴱ O’Leary-Barrett, M., Topper, L., Al-Khudhairy, N., Pihl, R. O., Castellanos-Ryan, N., Mackie, C. J., & Conrod, P. J. (2013). Two-year impact of personality-targeted, teacher-delivered interventions on youth internalizing and externalizing problems: A cluster-randomized trial. Journal of the American Academy of Child & Adolescent Psychiatry, 52, 911–920. http://dx.doi.org/10.1016/j.jaac.2013.05.020 Oltmanns, T. F., & Turkheimer, E. (2009). Person perception and personality pathology. Current Directions in Psychological Science, 18, 32–36. http://dx.doi.org/10.1111/j.1467-8721.2009.01601.x Owens, J., & Hoza, B. (2003). Diagnostic utility of DSM–IV–TR symptoms in the prediction of DSM–IV–TR ADHD subtypes and ODD. Journal of Attention Disorders, 7, 11–27. http://dx.doi.org/10.1177/ 108705470300700102 ⴱ Pastorelli, C., Barbaranelli, C., Cermak, I., Rozsa, S., & Caprara, G. V. (1997). Measuring emotional instability, prosocial behavior and aggression in pre-adolescents: A cross-national study. Personality and Individual Differences, 23, 691–703. http://dx.doi.org/10.1016/S01918869(97)00056-1 ⴱ Pearson, D. A., Aman, M. G., Arnold, L. E., Lane, D. M., Loveland, K. A., Santos, C. W., . . . Cleveland, L. A. (2012). High concordance of parent and teacher attention-deficit/hyperactivity disorder ratings in medicated and unmedicated children with autism spectrum disorders. Journal of Child and Adolescent Psychopharmacology, 22, 284 –291. http://dx.doi .org/10.1089/cap.2011.0067 Pelham, W. E., Jr., Fabiano, G. A., & Massetti, G. M. (2005). Evidencebased assessment of attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 449 – 476. http://dx.doi.org/10.1207/s15374424jccp3403_5 ⴱ Penney, S. R., & Skilling, T. A. (2012). Moderators of informant agreement in the assessment of adolescent psychopathology: Extension to a forensic sample. Psychological Assessment, 24, 386 – 401. http://dx.doi .org/10.1037/a0025693



895

Perry-Parrish, C., Waasdorp, T. E., & Bradshaw, C. P. (2012). Peer nominations of emotional expressivity among urban children: Social and psychological correlates. Social Development, 21, 88 –108. http://dx.doi .org/10.1111/j.1467-9507.2011.00615.x Peterson, R. A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 450 – 461. http://dx.doi.org/10.1086/323732 ⴱ Peterson, R. L., Kirkwood, M. W., Taylor, H. G., Stancin, T., Brown, T. M., & Wade, S. L. (2013). Adolescents’ internalizing problems following traumatic brain injury are related to parents’ psychiatric symptoms. Journal of Head Trauma Rehabilitation, 28. http://dx.doi.org/ 10.1097/HTR.0b013e318263f5ba ⴱ Petot, D., Rescorla, L., & Petot, J.-M. (2011). Agreement between parentand self-reports of Algerian adolescents’ behavioral and emotional problems. Journal of Adolescence, 34, 977–986. http://dx.doi.org/10.1016/j .adolescence.2010.11.011 ⴱ Phares, V., & Compas, B. E. (1990). Adolescents’ subjective distress over their emotional/behavioral problems. Journal of Consulting and Clinical Psychology, 58, 596 – 603. http://dx.doi.org/10.1037/0022-006X.58.5 .596 ⴱ Phares, V., Compas, B. E., & Howell, D. C. (1989). Perspectives on child behavior problems: Comparisons of children’s self-reports with parent and teacher reports. Psychological Assessment, 1, 68 –71. http://dx.doi .org/10.1037/1040-3590.1.1.68 Phares, V., & Danforth, J. S. (1994). Adolescents’, parents’, and teachers’ distress over adolescents’ behavior. Journal of Abnormal Child Psychology, 22, 721–732. http://dx.doi.org/10.1007/BF02171998 ⴱ Phillips, B. M., Lonigan, C. J., Driscoll, K., & Hooe, E. S. (2002). Positive and negative affectivity in children: A multitrait-multimethod investigation. Journal of Clinical Child and Adolescent Psychology, 31, 465– 479. http://dx.doi.org/10.1207/153744202320802142 ⴱ Phillips, P. L., Greenson, J. N., Collett, B. R., & Gimpel, G. A. (2002). Assessing ADHD symptoms in preschool children: Use of the ADHD symptoms rating scale. Early Education and Development, 13, 283–299. http://dx.doi.org/10.1207/s15566935eed1303_3 Piacentini, J. C., Cohen, P., & Cohen, J. (1992). Combining discrepant diagnostic information from multiple sources: Are complex algorithms better than simple ones? Journal of Abnormal Child Psychology, 20, 51– 63. http://dx.doi.org/10.1007/BF00927116 ⴱ Pierrehumbert, B., Bader, M., Thevoz, S., Kinal, A., & Halfon, O. (2006). Hyperactivity and attention problems in a Swiss sample of school-aged children: Effects of school achievement, child gender, and informants. Journal of Attention Disorders, 10, 65–76. http://dx.doi.org/10.1177/ 1087054705286050 Power, T. J., Andrews, T. J., Eiraldi, R. B., Doherty, B. J., Ikeda, M. J., DuPaul, G. J., & Landau, S. (1998). Evaluating attention deficit hyperactivity disorder using multiple informants: The incremental utility of combining teacher with parent reports. Psychological Assessment, 10, 250 –260. http://dx.doi.org/10.1037/1040-3590.10.3.250 ⴱ Puura, K., Almqvist, F., Tamminen, T., Piha, J., Rasanen, E., Kumpulainen, K., . . . Koivisto, A.-M. (1998). Psychiatric disturbances among pre-pubertal children in Southern Finland. Social Psychiatry and Psychiatric Epidemiology, 33, 310 –318. http://dx.doi.org/10.1007/ s001270050060 ⴱ Randazzo, K. V., Landsverk, J., & Ganger, W. (2003). Three informants’ reports of child behavior: Parents, teachers, and foster parents. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 1343– 1350. http://dx.doi.org/10.1097/01.chi.0000085753.71002.da ⴱ Rapee, R. M., Barrett, P. M., Dadds, M. R., & Evans, L. (1994). Reliability of the DSM–III–R childhood anxiety disorders using structured interview: Interrater and parent-child agreement. Journal of the American Academy of Child and Adolescent Psychiatry, 33, 984 –992. http:// dx.doi.org/10.1097/00004583-199409000-00008

896

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.



DE LOS REYES ET AL.

Raviv, A., Erel, O., Fox, N. A., Leavitt, L. A., Raviv, A., Dar, I., . . . Greenbaum, C. W. (2001). Individual measurement of exposure to everyday violence among elementary schoolchildren across various settings. Journal of Community Psychology, 29, 117–140. http://dx.doi.org/ 10.1002/1520-6629(200103)29:2⬍117::AID-JCOP1009⬎3.0.CO;2-2 Ready, R. E., Watson, D., & Clark, L. A. (2002). Psychiatric patientand informant-reported personality: Predicting concurrent and future behavior. Assessment, 9, 361–372. http://dx.doi.org/10.1177/ 1073191102238157 ⴱ Redmond, S. M., & Rich, M. L. (2002). Stability of behavioral ratings of children with SLI. Journal of Speech, Language, and Hearing Research, 45, 190 –201. http://dx.doi.org/10.1044/1092-4388(2002/014) ⴱ Reeve, E. A., Bernstein, G. A., & Christenson, G. A. (1992). Clinical characteristics and psychiatric comorbidity in children with trichotillomania. Journal of the American Academy of Child & Adolescent Psychiatry, 31, 132–138. http://dx.doi.org/10.1097/00004583-19920100000020 ⴱ Reidler, E. B., & Swenson, L. P. (2012). Discrepancies between youth and mothers’ perceptions of their mother– child relationship quality and self-disclosure: Implications for youth- and mother-reported youth adjustment. Journal of Youth and Adolescence, 41, 1151–1167. http://dx .doi.org/10.1007/s10964-012-9773-8 Renk, K., & Phares, V. (2004). Cross-informant ratings of social competence in children and adolescents. Clinical Psychology Review, 24, 239 –254. http://dx.doi.org/10.1016/j.cpr.2004.01.004 ⴱ Rescorla, L. A., Achenbach, T. M., Ivanova, M. Y., Bilenberg, N., Bjarnadottir, G., Denner, S., . . . Verhulst, F. C. (2012). Behavioral/ emotional problems of preschoolers: Caregiver/teacher reports from 15 societies. Journal of Emotional and Behavioral Disorders, 20, 68 – 81. http://dx.doi.org/10.1177/1063426611434158 ⴱ Rescorla, L. A., Bochicchio, L., Achenbach, T. M., Ivanova, M. Y., Almqvist, F., Begovac, I., . . . Verhulst, F. C. (2014). Parent-teacher agreement on children’s problems in 21 societies. Journal of Clinical Child and Adolescent Psychology, 43, 627– 642. http://dx.doi.org/ 10.1080/15374416.2014.900719 ⴱ Rescorla, L. A., Ginzburg, S., Achenbach, T. M., Ivanova, M. Y., Almqvist, F., Begovac, I., . . . Verhulst, F. C. (2013). Cross-informant agreement between parent-reported and adolescent self-reported problems in 25 societies. Journal of Clinical Child and Adolescent Psychology, 42, 262–273. http://dx.doi.org/10.1080/15374416.2012.717870 ⴱ Rettew, D. C., van Oort, F. V. A., Verhulst, F. C., Buitelaar, J. K., Ormel, J., Hartman, C., . . . Hudziak, J. J. (2011). When parent and teacher ratings don’t agree: The Tracking Adolescents’ Individual Lives Survey (TRAILS). Journal of Child and Adolescent Psychopharmacology, 21, 389 –397. http://dx.doi.org/10.1089/cap.2010.0153 ⴱ Reuterskiöld, L., Ost, L.-G., & Ollendick, T. (2008). Exploring child and parent factors in the diagnostic agreement on the Anxiety Disorders Interview Schedule. Journal of Psychopathology and Behavioral Assessment, 30, 279 –290. http://dx.doi.org/10.1007/s10862-008-9081-5 Richters, J. E. (1992). Depressed mothers as informants about their children: A critical review of the evidence for distortion. Psychological Bulletin, 112, 485– 499. http://dx.doi.org/10.1037/0033-2909.112.3.485 ⴱ Rockhill, C. M., Russo, J. E., McCauley, E., Katon, W. J., Richardson, L. P., & Lozano, P. (2007). Agreement between parents and children regarding anxiety and depression diagnoses in children with asthma. Journal of Nervous & Mental Disease, 195, 897–904. http://dx.doi.org/ 10.1097/NMD.0b013e318159289c ⴱ Rodriguez, E. M., Nichols, S. R., Javdani, S., Emerson, E., & Donenberg, G. R. (2013). Economic hardship, parent positive communication and mental health in urban adolescents seeking outpatient psychiatric care. Journal of Child and Family Studies. Advance online publication. http:// dx.doi.org/10.1007/s10826-013-9872-5

Rogosa, D., Brandt, D., & Zimowski, M. (1982). A growth curve approach to the measurement of change. Psychological Bulletin, 92, 726 –748. http://dx.doi.org/10.1037/0033-2909.92.3.726 Rogosa, D. R., & Willett, J. B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20, 335–343. http://dx.doi.org/10.1111/j.1745-3984.1983 .tb00211.x ⴱ Rosenbaum, M., & Ronen, T. (1997). Parents’ and children’s appraisals of each other’s anxiety while facing a common threat. Journal of Clinical Child Psychology, 26, 43–52. http://dx.doi.org/10.1207/ s15374424jccp2601_5 ⴱ Rosenberg, L. A., Brown, J., & Singer, H. S. (1994). Self-reporting of behavior problems in patients with tic disorders. Psychological Reports, 74, 653– 654. http://dx.doi.org/10.2466/pr0.1994.74.2.653 ⴱ Rosenthal, J. A., & Curiel, H. F. (2006). Modeling behavioral problems of children in the child welfare system: Caregiver, youth, and teacher perceptions. Children and Youth Services Review, 28, 1391–1408. http:// dx.doi.org/10.1016/j.childyouth.2006.02.007 ⴱ Roskam, I., & Stievenart, M. (2014). Is there a common pathway to maladjustment for internationally adopted and non-adopted adolescents? Journal of Applied Developmental Psychology, 35, 215–222. http://dx .doi.org/10.1016/j.appdev.2013.12.005 ⴱ Rosnati, R., Montirosso, R., & Barni, D. (2008). Behavioral and emotional problems among Italian international adoptees and non-adopted children: Father’s and mother’s reports. Journal of Family Psychology, 22, 541–549. http://dx.doi.org/10.1037/0893-3200.22.3.541 Roth, A., & Fonagy, P. (2005). What works for whom?: A critical review of psychotherapy research (2nd ed.). New York, NY: Guilford Press. ⴱ Rothbaum, F., Rosen, K. S., Pott, M., & Beatty, M. (1995). Early parentchild relationships and later problem behavior: A longitudinal study. Merrill-Palmer Quarterly, 41, 133–151. Retrieved from http://www .jstor.org/stable/23090528 ⴱ Rubio-Stipec, M., Canino, G. J., Shrout, P., Dulcan, M., Freeman, D., & Bravo, M. (1994). Psychometric properties of parents and children as informants in child psychiatry epidemiology with the Spanish Diagnostic Interview Schedule for Children (DISC-2). Journal of Abnormal Child Psychology, 22, 703–720. http://dx.doi.org/10.1007/BF02171997 ⴱ Ruchkin, V., Koposov, R., Vermeiren, R., & Schwab-Stone, M. (2012). The Strength and Difficulties Questionnaire: Russian validation of the teacher version and comparison of teacher and student reports. Journal of Adolescence, 35, 87–96. http://dx.doi.org/10.1016/j.adolescence.2011 .06.003 Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71–72. http://dx.doi.org/10.1136/bmj.312 .7023.71 ⴱ Salbach-Andrae, H., Klinkowski, N., Lenz, K., & Lehmkuhl, U. (2009). Agreement between youth-reported and parent-reported psychopathology in a referred sample. European Child & Adolescent Psychiatry, 18, 136 –143. http://dx.doi.org/10.1007/s00787-008-0710-z ⴱ Salbach-Andrae, H., Klinkowski, N., Lenz, K., Pfeiffer, E., Lehmkuhl, U., & Ehrlich, S. (2008). Correspondence between self-reported and parentreported psychopathology in adolescents with eating disorders. Psychopathology, 41, 307–312. http://dx.doi.org/10.1159/000146068 ⴱ Salbach-Andrae, H., Lenz, K., & Lehmkuhl, U. (2009). Patterns of agreement among parent, teacher and youth ratings in a referred sample. European Psychiatry, 24, 345–351. http://dx.doi.org/10.1016/j.eurpsy .2008.07.008 Sanislow, C. A., Pine, D. S., Quinn, K. J., Kozak, M. J., Garvey, M. A., Heinssen, R. K., . . . Cuthbert, B. N. (2010). Developing constructs for psychopathology research: Research domain criteria. Journal of Abnormal Psychology, 119, 631– 639. http://dx.doi.org/10.1037/a0020909 ⴱ Satake, H., Yoshida, K., Yamashita, H., Kinukawa, N., & Takagishi, T. (2003). Agreement between parents and teachers on behavioral/

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT emotional problems in Japanese school children using the Child Behavior Checklist. Child Psychiatry and Human Development, 34, 111–126. http://dx.doi.org/10.1023/A:1027342007736 ⴱ Saudino, K. J., Ronald, A., & Plomin, R. (2005). The etiology of behavior problems in 7-year-old twins: Substantial genetic influence and negligible shared environmental influence for parent ratings and ratings by same and different teachers. Journal of Abnormal Child Psychology, 33, 113–130. http://dx.doi.org/10.1007/s10802-005-0939-7 ⴱ Saunders, C. D., Voelker, S. L., & Shore, D. L. (2002). Comparison of mother and father report on the personality inventory for children in a non referred Canadian sample. Canadian Journal of School Psychology, 17, 85–95. http://dx.doi.org/10.1177/082957350201700107 ⴱ Sawyer, M. G., Baghurst, P., & Clark, J. (1992). Differences between reports from children, parents and teachers: Implications for epidemiological studies. Australian and New Zealand Journal of Psychiatry, 26, 652– 660. http://dx.doi.org/10.3109/00048679209072102 ⴱ Sawyer, M. G., Baghurst, P., & Mathias, J. (1992). Differences between informants’ reports describing emotional and behavioural problems in community and clinic-referred children: A research note. Child Psychology & Psychiatry & Allied Disciplines, 33, 441– 449. http://dx.doi.org/ 10.1111/j.1469-7610.1992.tb00878.x ⴱ Schenck, C. E., Braver, S. L., Wolchik, S. A., Saenz, D., Cookston, J. T., & Fabricius, W. V. (2009). Relations between mattering to step- and non-residential fathers and adolescent mental health. Fathering, 7, 70 – 90. http://dx.doi.org/10.3149/fth.0701.70 Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199 –223. http://dx.doi.org/10.1037/1082-989X.1.2.199 ⴱ Schroeder, J. F., Hood, M. M., & Hughes, H. M. (2010). Inter-parent agreement on the syndrome scales of the Child Behavior Checklist (CBCL): Correspondence and discrepancies. Journal of Child and Family Studies, 19, 646 – 653. http://dx.doi.org/10.1007/s10826-010-9352-0 Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. http://dx.doi.org/10.1037/0003066X.54.2.93 ⴱ Seiffge-Krenke, I. (2000). Causal links between stressful events, coping style, and adolescent symptomatology. Journal of Adolescence, 23, 675– 691. http://dx.doi.org/10.1006/jado.2000.0352 ⴱ Selbo-Bruns, A., Nakamura, B. J., Higa-McMillan, C. K., Kentaro, H., & Chorpita, B. F. (2012). A multi-informant multi-measure approach to the two factor model of emotion with youth. Journal of Psychopathology and Behavioral Assessment, 34, 520 –530. http://dx.doi.org/10.1007/ s10862-012-9298-1 ⴱ Self-Brown, S., LeBlanc, M., & Kelley, M. L. (2004). Effects of violence exposure and daily stressors on psychological outcomes in urban adolescents. Journal of Traumatic Stress, 17, 519 –527. http://dx.doi.org/ 10.1007/s10960-004-5801-0 ⴱ Sentse, M., & Laird, R. D. (2010). Parent– child relationships and dyadic friendship experiences as predictors of behavior problems in early adolescence. Journal of Clinical Child and Adolescent Psychology, 39, 873– 884. http://dx.doi.org/10.1080/15374416.2010.517160 ⴱ Sentse, M., Lindenberg, S., Omvlee, A., Ormel, J., & Veenstra, R. (2010). Rejection and acceptance across contexts: Parents and peers as risks and buffers for early adolescent psychopathology. The TRAILS study. Journal of Abnormal Child Psychology, 38, 119 –130. http://dx.doi.org/ 10.1007/s10802-009-9351-z ⴱ Sharp, C., Mosko, O., Chang, B., & Ha, C. (2011). The cross-informant concordance and concurrent validity of the Borderline Personality Features Scale for Children in a community sample of boys. Clinical Child Psychology and Psychiatry, 16, 335–349. http://dx.doi.org/10.1177/ 1359104510366279 ⴱ Sharp, C., Venta, A., Marais, L., Skinner, D., Lenka, M., & Serekoane, J. (2014). First evaluation of a population-based screen to detect emotional-behavior disorders in orphaned children in Sub-Saharan Af-

897

rica. AIDS and Behavior, 18, 1174 –1185. http://dx.doi.org/10.1007/ s10461-014-0739-6 ⴱ Shemmassian, S. K., & Lee, S. S. (2014). Comparative validity of DSM-IV and alternative empirically derived approaches for the assessment of ADHD. Journal of Attention Disorders. Advance online publication. http://dx.doi.org/10.1177/1087054714522511 Silver, C. H. (2014). Sources of data about children’s executive functioning: Review and commentary. Child Neuropsychology, 20, 1–13. http:// dx.doi.org/10.1080/09297049.2012.727793 Silverman, W. K., & Ollendick, T. H. (2005). Evidence-based assessment of anxiety and its disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 380 – 411. http://dx.doi .org/10.1207/s15374424jccp3403_2 Silverman, W. K., & Rabian, B. (1995). Test–retest reliability of the DSM–III–R childhood anxiety disorders symptoms using the Anxiety Disorders Interview Schedule for children. Journal of Anxiety Disorders, 9, 139 –150. http://dx.doi.org/10.1016/0887-6185(94)00032-8 ⴱ Simmel, C., Lee, I., Kim, S., & Miles, J. (2014). Multiple assessors of child welfare youths’ mental-health functioning: Comparing perceptions of adolescents, caregivers and teachers. Child & Family Social Work, 19, 343–354. http://dx.doi.org/10.1111/cfs.12024 Smetana, J. G. (2008). “It’s 10 o’clock: Do you know where your children are?” Recent advances in understanding parental monitoring and adolescents’ information management. Child Development Perspectives, 2, 19 –25. http://dx.doi.org/10.1111/j.1750-8606.2008.00036.x Smetana, J. G., Campione-Barr, N., & Metzger, A. (2006). Adolescent development in interpersonal and societal contexts. Annual Review of Psychology, 57, 255–284. http://dx.doi.org/10.1146/annurev.psych.57 .102904.190124 ⴱ Smider, N. A., Essex, M. J., Kalin, N. H., Buss, K. A., Klein, M. H., Davidson, R. J., & Goldsmith, H. H. (2002). Salivary cortisol as a predictor of socioemotional adjustment during kindergarten: A prospective study. Child Development, 73, 75–92. http://dx.doi.org/10.1111/ 1467-8624.00393 Smith, B. H., Pelham, W. E., Jr., Gnagy, E., Molina, B., & Evans, S. (2000). The reliability, validity, and unique contributions of self-report by adolescents receiving treatment for attention-deficit/hyperactivity disorder. Journal of Consulting and Clinical Psychology, 68, 489 – 499. http://dx.doi.org/10.1037/0022-006X.68.3.489 Smith, G. T., Fischer, S., & Fister, S. M. (2003). Incremental validity principles in test construction. Psychological Assessment, 15, 467– 477. http://dx.doi.org/10.1037/1040-3590.15.4.467 ⴱ Smith, M. A., & Jenkins, J. M. (1991). The effects of marital disharmony on prepubertal children. Journal of Abnormal Child Psychology, 19, 625– 644. http://dx.doi.org/10.1007/BF00918903 ⴱ Smith Carter, J., Garber, J., Jeffrey, A., & Cole, D. A. (2006). Modeling relations between hassles and internalizing and externalizing symptoms in adolescents: A four-year prospective study. Journal of Abnormal Psychology, 115, 428 – 442. http://dx.doi.org/10.1037/0021-843X.115.3 .428 ⴱ Snyder, H. T., Bilboul, M. J., & Pope, A. W. (2005). Psychosocial adjustment in adolescents with craniofacial anomalies: A comparison of parent and self-reports. The Cleft Palate-Craniofacial Journal, 42, 548 – 555. http://dx.doi.org/10.1597/04-078R.1 ⴱ Sollie, H., Larsson, B., & Morch, W. T. (2013). Comparison of mother, father, and teacher reports of ADHD core symptoms in a sample of child psychiatric outpatients. Journal of Attention Disorders, 17, 699 –710. http://dx.doi.org/10.1177/1087054711436010 ⴱ Soper, A. C., Wolchik, S. A., Sandler, I. N., Tein, J.-Y., & Lustig, J. L. (2007). Predictors of threat to self-appraisals for children from divorced homes. Journal of Divorce & Remarriage, 48, 1–27. http://dx.doi.org/ 10.1300/J087v48n01_01 ⴱ Sorlie, M.-A., Hagen, K. A., & Ogden, T. (2008). Social competence and antisocial behavior: Continuity and distinctiveness across early adoles-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

898

DE LOS REYES ET AL.

cence. Journal of Research on Adolescence, 18, 121–144. http://dx.doi .org/10.1111/j.1532-7795.2008.00553.x ⴱ Sourander, A., Fossum, S., Rønning, J. A., Elonheimo, H., Ristkari, T., Kumpulainen, K., . . . Almqvist, F. (2012). What is the long-term outcome of boys who steal at age eight? Findings from the Finnish nationwide “From A Boy To A Man” birth cohort study. Social Psychiatry and Psychiatric Epidemiology, 47, 1391–1400. http://dx.doi.org/ 10.1007/s00127-011-0455-8 ⴱ Southam-Gerow, M. A., Flannery-Schroeder, E. C., & Kendall, P. C. (2003). A psychometric evaluation of the parent report form of the State-Trait Anxiety Inventory for Children—Trait Version. Journal of Anxiety Disorders, 17, 427– 446. http://dx.doi.org/10.1016/S08876185(02)00223-2 Spear, L. P. (2000). The adolescent brain and age-related behavioral manifestations. Neuroscience and Biobehavioral Reviews, 24, 417– 463. http://dx.doi.org/10.1016/S0149-7634(00)00014-2 ⴱ Spiker, D., Kraemer, H. C., Constantine, N. A., & Bryant, D. (1992). Reliability and validity of behavior problem checklists as measures of stable traits in low birth weight, premature preschoolers. Child Development, 63, 1481–1496. http://dx.doi.org/10.2307/1131570 ⴱ Stanger, C., & Lewis, M. (1993). Agreement among parents, teachers, and children on internalizing and externalizing behavior problems. Journal of Clinical Child Psychology, 22, 107–115. http://dx.doi.org/10.1207/ s15374424jccp2201_11 ⴱ Steele, R. G., Forehand, R., & Armistead, L. (1997). The role of family processes and coping strategies in the relationship between parental chronic illness and childhood internalizing problems. Journal of Abnormal Child Psychology, 25, 83–94. http://dx.doi.org/10.1023/A: 1025771210350 ⴱ Sterling, L., Renno, P., Storch, E. A., Ehrenreich-May, J., Lewin, A. B., Arnold, E., . . . Wood, J. (2014). Validity of the Revised Children’s Anxiety and Depression Scale for youth with autism spectrum disorders. Autism. Advance online publication. http://dx.doi.org/10.1177/ 1362361313510066 ⴱ Sternberg, K. J., Lamb, M. E., Greenbaum, C., Cicchetti, D., Dawud, S., Cortes, R. M., . . . Lorey, F. (1993). Effects of domestic violence on children’s behavior problems and depression. Developmental Psychology, 29, 44 –52. http://dx.doi.org/10.1037/0012-1649.29.1.44 ⴱ Sternberg, K. J., Lamb, M. E., Guterman, E., & Abbott, C. B. (2006). Effects of early and later family violence on children’s behavior problems and depression: A longitudinal, multi-informant perspective. Child Abuse & Neglect, 30, 283–306. http://dx.doi.org/10.1016/j.chiabu.2005 .10.008 ⴱ Stice, E., & Barrera, M., Jr. (1995). A longitudinal examination of the reciprocal relations between perceived parenting and adolescent’s substance use and externalizing behaviors. Developmental Psychology, 31, 322–334. http://dx.doi.org/10.1037/0012-1649.31.2.322 ⴱ Stocker, C. M., Richmond, M. K., Low, S. M., Alexander, E. K., & Elias, N. M. (2003). Marital conflict and children’s adjustment: Parental hostility and children’s interpretations as mediators. Social Development, 12, 149 –161. http://dx.doi.org/10.1111/1467-9507.00226 ⴱ Stocker, C. M., Richmond, M. K., Rhoades, G. K., & Kiang, L. (2007). Family emotional processes and adolescents’ adjustment. Social Development, 16, 310 –325. http://dx.doi.org/10.1111/j.1467-9507.2007 .00386.x ⴱ Stone, L. L., van Daal, C., van der Maten, M., Engels, R. C. M. E., Janssens, J. M. A. M., & Otten, R. (2014). The Berkeley Puppet Interview: A screening instrument for measuring psychopathology in young children. Child & Youth Care Forum, 43, 211–225. http://dx.doi .org/10.1007/s10566-013-9235-9 ⴱ Stover, C. S., Hahn, H., Im, J. J. Y., & Berkowitz, S. (2010). Agreement of parent and child reports of trauma exposure and symptoms in the early aftermath of a traumatic event. Psychological Trauma: Theory, Re-

search, Practice, and Policy, 2, 159 –168. http://dx.doi.org/10.1037/ a0019156 Straus, S. E., Glasziou, P., Richardson, W. S., & Haynes, R. B. (2011). Evidence-based medicine: How to practice and teach EBM (4th ed.). New York, NY: Churchill Livingstone. ⴱ Strickland, J., Hopkins, J., & Keenan, K. (2012). Mother-teacher agreement on preschoolers’ symptoms of ODD and CD: Does context matter? Journal of Abnormal Child Psychology, 40, 933–943. http://dx.doi.org/ 10.1007/s10802-012-9622-y ⴱ Strijker, J., van Oijen, S., & Knot-Dickscheit, J. (2011). Assessment of problem behaviour by foster parents and their foster children. Child & Family Social Work, 16, 93–100. http://dx.doi.org/10.1111/j.1365-2206 .2010.00717.x ⴱ Stuewig, J., Tangney, J. P., Heigel, C., Harty, L., & McCloskey, L. (2010). Shaming, blaming, and maiming: Functional links among the moral emotions, externalization of blame, and aggression. Journal of Research in Personality, 44, 91–102. http://dx.doi.org/10.1016/j.jrp .2009.12.005 Swets, J. A. (1992). The science of choosing the right decision threshold in high-stakes diagnostics. American Psychologist, 47, 522–532. http://dx .doi.org/10.1037/0003-066X.47.4.522 Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26. http://dx.doi.org/10.1111/1529-1006.001 ⴱ Tackett, J. L., Herzhoff, K., Reardon, K. W., Smack, A. J., & Kushner, S. C. (2013). The relevance of informant discrepancies for the assessment of adolescent personality pathology. Clinical Psychology: Science and Practice, 20, 378 –392. http://dx.doi.org/10.1111/cpsp.12048 Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4 –28. http://dx.doi.org/ 10.3102/0034654310393361 ⴱ Termine, C., Luoni, C., Selvini, C., Bandera, V., Balottin, U., Eddy, C. M., & Cavanna, A. E. (2014). Mother-child agreement on behavioral ratings in Tourette syndrome: A controlled study. Journal of Child Neurology, 29, 79 – 83. http://dx.doi.org/10.1177/0883073812470003 Thomas, S. A., Aldao, A., & De Los Reyes, A. (2012). Implementing clinically feasible psychophysiological measures in evidence-based assessments of adolescent social anxiety. Professional Psychology: Research and Practice, 43, 510 –519. http://dx.doi.org/10.1037/a0029183 ⴱ Thuppal, M., Carlson, G. A., Sprafkin, J., & Gadow, K. D. (2002). Correspondence between adolescent report, parent report, and teacher report of manic symptoms. Journal of Child and Adolescent Psychopharmacology, 12, 27–35. http://dx.doi.org/10.1089/10445460252943542 ⴱ Thurber, S., & Osborn, R. A. (1993). Comparisons of parent and adolescent perspectives on deviance. The Journal of Genetic Psychology: Research and Theory on Human Development, 154, 25–32. http://dx.doi .org/10.1080/00221325.1993.9914718 ⴱ Tinoco, V. M., Lagares, I. J., Moreno, C., Tessier, N. G., & Schneider, B. H. (2009). The nature and basis of teachers’ concerns about Spanish secondary-school students. School Psychology International, 30, 585– 602. http://dx.doi.org/10.1177/0143034309107078 ⴱ Trzesniewski, K. H., Moffitt, T. E., Caspi, A., Taylor, A., & Maughan, B. (2006). Revisiting the association between reading achievement and antisocial behavior: New evidence of an environmental explanation from a twin study. Child Development, 77, 72– 88. http://dx.doi.org/10.1111/ j.1467-8624.2006.00857.x ⴱ Tu, K. M., & Erath, S. A. (2013). Social discomfort in preadolescence: Predictors of discrepancies between preadolescents and their parents and teachers. Child Psychiatry & Human Development, 44, 201–216. http:// dx.doi.org/10.1007/s10578-012-0319-9 Tulbure, B. T., Szentagotai, A., Dobrean, A., & David, D. (2012). Evidence based clinical assessment of child and adolescent social phobia: A

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

VALIDITY OF MULTI-INFORMANT ASSESSMENT critical review of rating scales. Child Psychiatry and Human Development, 43, 795– 820. http://dx.doi.org/10.1007/s10578-012-0297-y ⴱ Turchik, J. A., Karpenko, V., & Ogles, B. M. (2007). Further evidence of the utility and validity of a measure of outcome for children and adolescents. Journal of Emotional and Behavioral Disorders, 15, 119 – 128. http://dx.doi.org/10.1177/10634266070150020201 ⴱ van der Ende, J., & Verhulst, F. C. (2005). Informant, gender and age differences in ratings of adolescent problem behaviour. European Child & Adolescent Psychiatry, 14, 117–126. http://dx.doi.org/10.1007/ s00787-005-0438-y ⴱ van der Ende, J., Verhulst, F. C., & Tiemeier, H. (2012). Agreement of informants on emotional and behavioral problems from childhood to adulthood. Psychological Assessment, 24, 293–300. http://dx.doi.org/ 10.1037/a0025500 ⴱ van der Oord, S., Prins, P. J., Oosterlaan, J., & Emmelkamp, P. M. (2006). The association between parenting stress, depressed mood and informant agreement in ADHD and ODD. Behaviour Research and Therapy, 44, 1585–1595. http://dx.doi.org/10.1016/j.brat.2005.11.011 ⴱ van der Sluis, C. M., van der Bruggen, C. O., Brechman-Toussaint, M. L., Thissen, M. A., & Bögels, S. M. (2012). Parent-directed cognitive behavioral therapy for young anxious children: A pilot study. Behavior Therapy, 43, 583–592. http://dx.doi.org/10.1016/j.beth.2011.10.002 ⴱ van der Valk, J. C., van den Oord, E. J. C. G., Verhulst, F. C., & Boomsma, D. I. (2001). Using parental ratings to study the etiology of 3-year-old twins’ problem behaviors: Different views or rater bias? Journal of Child Psychology and Psychiatry, 42, 921–931. http://dx.doi .org/10.1111/1469-7610.00788 ⴱ van der Valk, J. C., van den Oord, E. J. C. G., Verhulst, F. C., & Boomsma, D. I. (2003). Using shared and unique parental views to study the etiology of 7-year-old twins’ internalizing and externalizing problems. Behavior Genetics, 33, 409 – 420. http://dx.doi.org/10.1023/A: 1025369525924 ⴱ van Dulmen, M. H. M., & Egeland, B. (2011). Analyzing multiple informant data on child and adolescent behavior problems: Predictive validity and comparison of aggregation procedure. International Journal of Behavioral Development, 35, 84 –92. http://dx.doi.org/10.1177/ 0165025410392112 ⴱ van Hoeke, E., Hoebeke, P., Braet, C., & Walle, J. V. (2004). An assessment of internalizing problems in children with enuresis. The Journal of Urology, 171, 2580 –2583. http://dx.doi.org/10.1097/01.ju .0000110521.20103.14 ⴱ Veenstra, R., Lindenberg, S., Oldehinkel, A. J., De Winter, A. F., Verhulst, F. C., & Ormel, J. (2008). Prosocial and antisocial behavior in preadolescence: Teachers’ and parents’ perceptions of the behavior of girls and boys. International Journal of Behavioral Development, 32, 243–251. http://dx.doi.org/10.1177/0165025408089274 ⴱ Veenstra, R., Oldehinkel, A. J., De Winter, A. F., Lindenberg, S., & Ormel, J. (2006). Temperament, environment, and antisocial behavior in a population sample of preadolescent boys and girls. International Journal of Behavioral Development, 30, 422– 432. http://dx.doi.org/ 10.1177/0165025406071490 ⴱ Verhulst, F. C., & Akkerhuis, G. W. (1989). Agreement between parents’ and teachers’ ratings of behavioral/emotional problems of children aged 4-12. Journal of Child Psychology & Psychiatry & Allied Disciplines, 30, 123–136. http://dx.doi.org/10.1111/j.1469-7610.1989.tb00772.x ⴱ Verhulst, F. C., & van der Ende, J. (1991). Assessment of child psychopathology: Relationships between different methods, different informants and clinical judgment of severity. Acta Psychiatrica Scandinavica, 84, 155–159. http://dx.doi.org/10.1111/j.1600-0447.1991 .tb03120.x ⴱ Verhulst, F. C., & van der Ende, J. (1992). Agreement between parents’ reports and adolescents’ self-reports of problem behavior. Journal of Child Psychology & Psychiatry & Allied Disciplines, 33, 1011–1023. http://dx.doi.org/10.1111/j.1469-7610.1992.tb00922.x



899

Vierhaus, M., & Lohaus, A. (2008). Children and parents as informants of emotional and behavioural problems predicting female and male adolescent risk behaviour: A longitudinal cross-informant study. Journal of Youth and Adolescence, 37, 211–224. http://dx.doi.org/10.1007/s10964007-9193-3 ⴱ Villabø, M., Gere, M., Torgersen, S., March, J. S., & Kendall, P. C. (2012). Diagnostic efficiency of the child and parent versions of the Multidimensional Anxiety Scale for Children. Journal of Clinical Child and Adolescent Psychology, 41, 75– 85. http://dx.doi.org/10.1080/ 15374416.2012.632350 ⴱ Visser, A., Huizinga, G. A., Hoekstra, H. J., Van Der Graaf, W. T. A., Klip, E. C., Pras, E., & Hoekstra-Weebers, J. E. H. M. (2005). Emotional and behavioural functioning of children of a parent diagnosed with cancer: A cross-informant perspective. Psycho-Oncology, 14, 746 –758. http://dx.doi.org/10.1002/pon.902 ⴱ Vitiello, B., Malone, R., Buschle, P. R., Delaney, M. A., & Behar, D. (1990). Reliability of DSM-III diagnoses of hospitalized children. Hospital and Community Psychiatry, 41, 63– 67. http://dx.doi.org/10.1176/ ps.41.1.63 Wakschlag, L. S., Tolan, P. H., & Leventhal, B. L. (2010). Research review: “Ain’t misbehavin:” Towards a developmentally-specified nosology for preschool disruptive behavior. Journal of Child Psychology and Psychiatry, 51, 3–22. http://dx.doi.org/10.1111/j.1469-7610.2009 .02184.x ⴱ Walker, K. C., & Bracken, B. A. (1996). Inter-parent agreement on four preschool behavior rating scales: Effects of parent and child gender. Psychology in the Schools, 33, 273–281. http://dx.doi.org/10.1002/ (SICI)1520-6807(199610)33:4⬍273::AID-PITS2⬎3.0.CO;2-O ⴱ Wang, J., Liu, L., Wu, H., Yang, X., Wang, Y., & Wang, L. (2014). Agreement between parents and adolescents on emotional and behavioral problems and its associated factors among Chinese school adolescents: A cross-sectional study. BMC Psychiatry, 14:114. http://dx.doi .org/10.1186/1471-244X-14-114 ⴱ Waschbusch, D. A., & Elgar, F. J. (2007). Development and validation of the Conduct Disorder Rating Scale. Assessment, 14, 65–74. http://dx.doi .org/10.1177/1073191106289908 ⴱ Waschbusch, D. A., & Willoughby, M. T. (2008). Parent and teacher ratings on the IOWA Conners Rating Scale. Journal of Psychopathology and Behavioral Assessment, 30, 180 –192. http://dx.doi.org/10.1007/ s10862-007-9064-y ⴱ Watts, S. E., & Weems, C. F. (2006). Associations among selective attention, memory bias, cognitive errors and symptoms of anxiety in youth. Journal of Abnormal Child Psychology, 34, 841– 852. http://dx .doi.org/10.1007/s10802-006-9066-3 Weissman, M. M., Wickramaratne, P., Warner, V., John, K., Prusoff, B. A., Merikangas, K. R., & Gammon, G. D. (1987). Assessing psychiatric disorders in children. Discrepancies between mothers’ and children’s reports. Archives of General Psychiatry, 44, 747–753. http://dx.doi.org/ 10.1001/archpsyc.1987.01800200075011 Weisz, J. R., Doss, A. J., & Hawley, K. M. (2005). Youth psychotherapy outcome research: A review and critique of the evidence base. Annual Review of Psychology, 56, 337–363. http://dx.doi.org/10.1146/annurev .psych.55.090902.141449 ⴱ Weisz, J. R., Southam-Gerow, M. A., & McCarty, C. A. (2001). Controlrelated beliefs and depressive symptoms in clinic-referred children and adolescents: Developmental differences and model specificity. Journal of Abnormal Psychology, 110, 97–109. http://dx.doi.org/10.1037/0021843X.110.1.97 ⴱ Weitkamp, K., Romer, G., Rosenthal, S., Wiegand-Grefe, S., & Daniels, J. (2010). German Screen for Child Anxiety Related Emotional Disorders (SCARED): Reliability, validity, and cross-informant agreement in a clinical sample. Child and Adolescent Psychiatry and Mental Health, 4:19. http://dx.doi.org/10.1186/1753-2000-4-19

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

900

DE LOS REYES ET AL.

Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: Two simple measures. Journal of Personality and Social Psychology, 84, 608 – 618. http://dx.doi.org/10.1037/0022-3514.84.3.608 Westen, D., & Rosenthal, R. (2005). Improving construct validity: Cronbach, Meehl, and Neurath’s ship. Psychological Assessment, 17, 409 – 412. http://dx.doi.org/10.1037/1040-3590.17.4.409 ⴱ Westerman, M. A., & Schonholtz, J. (1993). Marital adjustment, joint parental support in a triadic problem-solving task, and child behavior problems. Journal of Clinical Child Psychology, 22, 97–106. http://dx .doi.org/10.1207/s15374424jccp2201_10 ⴱ Whiteside, S. P. (2009). Adapting the Sheehan Disability Scale to assess child and parent impairment related to childhood anxiety disorders. Journal of Clinical Child & Adolescent Psychology, 38, 721–730. http:// dx.doi.org/10.1080/15374410903103551 Whiting, P. F., Rutjes, A. W., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., . . . Bossuyt, P. M.. (2011). QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine, 155, 529 –536. http://dx.doi.org/10.7326/0003-4819155-8-201110180-00009 ⴱ Will, M. N., & Wilson, B. J. (2014). A longitudinal analysis of parent and teacher ratings of problem behavior in boys with and without developmental delays. Journal of Intellectual Disabilities, 18, 176 –187. http:// dx.doi.org/10.1177/1744629514528828 ⴱ Winsler, A., & Wallace, G. L. (2002). Behavior problems and social skills in preschool children: Parent-teacher agreement and relations with classroom observations. Early Education and Development, 13, 41–58. http:// dx.doi.org/10.1207/s15566935eed1301_3 ⴱ Wong, H. Y., Jenvey, V., & Lill, A. (2012). Children’s self-reports and parents’ reports of internalising and externalising problems in Chinese and Anglo-Celtic children in Australia. Australian Journal of Psychology, 64, 155–163. http://dx.doi.org/10.1111/j.1742-9536.2011.00038.x ⴱ Wren, F. J., Bridge, J. A., & Birmaher, B. (2004). Screening for childhood anxiety symptoms in primary care: Integrating child and parent reports. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 1364 –1371. http://dx.doi.org/10.1097/01.chi.0000138350.60487.d3 ⴱ Xu, X., Mellor, D., & Wong, J. (2011). Childhood cruelty to animals: Do mothers’ and fathers’ reports agree? Anthrozoos: A Multidisciplinary Journal of the Interactions of People & Animals, 24, 263–271. http:// dx.doi.org/10.2752/175303711X13045914865105 Yates, B. T., & Taub, J. (2003). Assessing the costs, benefits, costeffectiveness, and cost-benefit of psychological assessment: We should, we can, and here’s how. Psychological Assessment, 15, 478 – 495. http:// dx.doi.org/10.1037/1040-3590.15.4.478 Yeh, M., & Weisz, J. R. (2001). Why are we here at the clinic? Parent-child (dis)agreement on referral problems at outpatient treatment entry. Journal of Consulting and Clinical Psychology, 69, 1018 –1025. http://dx.doi .org/10.1037/0022-006X.69.6.1018 Youngstrom, E. (2008). Evidence-based strategies for the assessment of developmental psychopathology: Measuring prediction, prescription, and process. In W. E. Craighead & D. J. Milkowitz (Eds.), Psychopathology. History, diagnosis, and empirical foundations (pp. 34 –77). Hoboken, NJ: Wiley.



Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68, 1038 –1050. http://dx.doi.org/ 10.1037/0022-006X.68.6.1038 Youngstrom, E. A. (2013). Future directions in psychological assessment: Combining evidence-based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Clinical Child and Adolescent Psychology, 42, 139 –159. http://dx.doi.org/10.1080/15374416 .2012.736358 ⴱ Youngstrom, E. A., Findling, R. L., & Calabrese, J. R. (2003). Who are the comorbid adolescents? Agreement between psychiatric diagnosis, youth, parent, and teacher report. Journal of Abnormal Child Psychology, 31, 231–245. http://dx.doi.org/10.1023/A:1023244512119 Youngstrom, E. A., Findling, R. L., & Calabrese, J. R. (2004b). Effects of adolescent manic symptoms on agreement between youth, parent, and teacher ratings of behavior problems. Journal of Affective Disorders, 82, S5–S16. http://dx.doi.org/10.1016/j.jad.2004.05.016 Youngstrom, E. A., Findling, R. L., Calabrese, J. R., Gracious, B. L., Demeter, C., . . . Price, M. (2004a). Comparing the diagnostic accuracy of six potential screening instruments for bipolar disorder in youths aged 5 to 17 years. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 847– 858. http://dx.doi.org/10.1097/01.chi.0000125091 .35109.1e Youngstrom, E., Izard, C., & Ackerman, B. (1999). Dysphoria-related bias in maternal ratings of children. Journal of Consulting and Clinical Psychology, 67, 905–916. http://dx.doi.org/10.1037/0022-006X.67.6 .905 Youngstrom, E. A., Kahana, S. Y., Starr, M., & Schwartz, C. (2004, November). Emotion states and bias in clinical ratings: Causal but complicated. In A. De Los Reyes & L. M. Saavedra (Chairs), Informant discrepancies in clinical child research: Processes of informant discrepancies and distortions and impact on clinical decision-making. Symposium conducted at the annual meeting of the Association for the Advancement of Behavior Therapy, New Orleans. ⴱ Zahn-Waxler, C., Schmitz, S., Fulker, D., Robinson, J., & Emde, R. (1996). Behavior problems in five-year-old monozygotic and dizygotic twins: Genetic and environmental influences, patterns of regulation, and internalization of control. Development and Psychopathology, 8, 103– 122. http://dx.doi.org/10.1017/S0954579400006994 ⴱ Zhou, Q., Hofer, C., Eisenberg, N., Reiser, M., Spinrad, T. L., & Fabes, R. A. (2007). The developmental trajectories of attention focusing, attentional and behavioral persistence, and externalizing problems during school-age years. Developmental Psychology, 43, 369 –385. http:// dx.doi.org/10.1037/0012-1649.43.2.369 ⴱ Zhou, Q., Lengua, L. J., & Wang, Y. (2009). The relations of temperament reactivity and effortful control to children’s adjustment problems in China and the United States. Developmental Psychology, 45, 724 –739. http://dx.doi.org/10.1037/a0013776

Received March 14, 2013 Revision received November 1, 2014 Accepted November 7, 2014 䡲

The validity of the multi-informant approach to assessing child and adolescent mental health.

Child and adolescent patients may display mental health concerns within some contexts and not others (e.g., home vs. school). Thus, understanding the ...
858KB Sizes 1 Downloads 11 Views